S1 - Sites are having issues loading properly
Incident Report for IOFFICE
Postmortem

On the morning of January 21st, 2022, iOFFICE started to receive reports that client sites are not able to load or experiencing a refresh the site message.
Root Cause:
iOFFICE investigation identified that smoke tests sporadically kept failing and passing. One of our application services was failing to fully process requests with a large response, which caused clients login page to function sporadically.
Remediation:
iOFFICE identified the host running the service which caused large requests, we manually restarted the service and during our monitoring the functionality returned to normal and the sporadic smoke test failures stopped.
Timeline:
iOFFICE system alerts began around 11:32pm CST, Monitoring alerts started firing, Customer Support started to receive reports that clients were experiencing a refresh screen upon logon as well as intermittent access to sites. At 11:40pm we narrowed the issue to one specific part of code that required a hot fix, the fix was tested and released at 12:30pm. Functionality returned to normal and after monitoring the systems the incident was resolved 1:37pm
Preventative Action and Analysis:
iOFFICE engineering quickly identified and released a fix.

Posted Feb 04, 2022 - 17:30 UTC

Resolved
The issue has been resolved and the Engineering team has set into place a fix. We will be monitoring through the weekend. We've identified the exact issue that caused this outage.
Posted Jan 21, 2022 - 21:54 UTC
Monitoring
The issue is now resolved. We are implementing a fix that will prevent this same issue from happening.
Posted Jan 21, 2022 - 21:21 UTC
Investigating
Our team is currently experiencing issues with the website's function. Our team is investigating and we will be updating in one hour at 4:16 PM CST.
Posted Jan 21, 2022 - 21:16 UTC
This incident affected: System Status.