On the morning of January 21st, 2022, iOFFICE started to receive reports that client sites are not able to load or experiencing a refresh the site message.
Root Cause:
iOFFICE investigation identified that smoke tests sporadically kept failing and passing. One of our application services was failing to fully process requests with a large response, which caused clients login page to function sporadically.
Remediation:
iOFFICE identified the host running the service which caused large requests, we manually restarted the service and during our monitoring the functionality returned to normal and the sporadic smoke test failures stopped.
Timeline:
iOFFICE system alerts began around 11:32pm CST, Monitoring alerts started firing, Customer Support started to receive reports that clients were experiencing a refresh screen upon logon as well as intermittent access to sites. At 11:40pm we narrowed the issue to one specific part of code that required a hot fix, the fix was tested and released at 12:30pm. Functionality returned to normal and after monitoring the systems the incident was resolved 1:37pm
Preventative Action and Analysis:
iOFFICE engineering quickly identified and released a fix.