iOFFICE Detailed Root Cause Analysis – Severity 2 – July 12, 2023

Description:

On July 12, 2023, at approximately 12:11pm EDT, internal teams and customers support started to receive excessive reports of issues throughout the iOffice platform with the inability to use the Checkout and Delete functionality in the Visitor Module. End users would experience errors when clicking on the checkout function in the Visitor queue.

Type of Event:

Service Disruption

Services\Modules Impacted:

Production – Visitor Module

Timeline:

On Wednesday, July 12, 2023, at approximately 12:11pm EDT, internal teams and customers support started to receive excessive reports of issues throughout the iOffice platform with the inability to use the Checkout and Delete functionality in the Visitor Module. End users would experience errors when clicking on the checkout function in the Visitor queue. All customers were alerted via Status Page about the disruption. Investigation continued through Thursday, July 13, 2023, till 6:13pm EDT when engineers had identified the issue. Our engineering team had Implemented a fix at approximately 9:10pm and monitored the system for the next hour. Friday, July 14, 2023, at 7:56am EDT support began reaching out to customers for confirmation. At 10:40am EDT the Status Page was moved from Monitoring to Resolved.

‌

Total Duration of Event:

1 Day, 22 Hours, 29 Minutes

Remediation:

Change critical variable to be re-calculated at every use, instead of on class instantiation. This resolves the malformed database connection and records are found as needed.

Root Cause Analysis:

Internal teams found the root cause was introduced while resolving 2 iOffice issues. The modified Visitor Module logic to set a critical variable needed by the visitor service at the class level. In production usage volume, it was found that this class is at times reused instead of created per request as expected, which created a malformed database connection within the visitor service. The service then could not find the record to modify and would return an error.

Preventative Action:

No further steps are needed to take preventative action.