iOFFICE Detailed Root Cause Analysis – Severity 2 – 05/11/2023

iOffice Automated Processes, including AUP's, Data Exports, and Scheduled Tasks

Description:

On May 11, 2023, at approximately 7:22am, MST multiple customers reported issues of the iOffice Automated Processes, including AUP's, Data Exports, and Scheduled Tasks have not been processed from the previous day.

Type of Event:

Performance Degradation

Services\Modules Impacted:

Production

Timeline:

On May 11, 2023, 7:22am, MST, internal and external customers notified the Support Team that Automated Processes, including AUP's, Data Exports, and Scheduled Tasks have not been processed from the previous day. At 9:13am, MST our customers were notified via Status Page that our Dev Ops Team are in an Investigation Phase. At approximately, 10:30am, MST, our Dev Ops Team had identified and implemented the resolution. We updated the Status Page and notified customers that we will go into a Monitoring phase. At 11:47am, MST, multiple customers were able to verify the fix and the Status Page was updated from Monitoring to Resolved.

Total Duration of Event:

4hrs 25mins

Root Cause Analysis:

The Dev Ops Team identified the Root Cause in the program and a minor code fix was implemented to correct it.

Preventative Action:

An install and provisioning of internal programs were implemented to prevent this from happening again.