Eptura Workplace detailed Root Cause Analysis | May 30, 2024 

S2 – Service Request Notifications 

We are grateful for your continued support and loyalty. We value your feedback and appreciate your patience as we worked to resolve this incident. 

Description: 

When Service Requests were being created, notifications were not being sent to Requesters and Operators. 

Type of Event: 

Eptura Workplace was experiencing a functionality issue. 

Services/Modules impacted: 

Production/ Service Request Modules 

Timeline (reported in CST):  

Thursday, May 30, 2024 

• 9:05am: Our Support team has received multiple reports from customers that Service Request emails are not being sent to Requesters and Operators. Because of this, our open ticket with the engineering team was upgraded from a Severity 3 to a Severity 2. All customers were alerted of the upgrade through the Status Page.  

• 11:20am: After some investigation by our Clops team, a restart of the Preference service was requested. There were signs that the root cause could be pointing to the connection from iOfficeConnect in k8s to Preferences. All customers were notified that we have identified the issue.  

• 11:46am: The Preference service was restarted, and support began to confirm the fix with customers.  

• 1:26pm: Since the restart of the Preference service, customers report that the issue remains, and the status page was moved back to an Investigation phase.  

• 4:58pm: Investigation is now being reviewed by our engineering team. The issue could be caused by a release item (IOHOU-8987) on May 22, 2024. 

• 9:24pm: The status page was updated, and all customers were made aware that the investigation phase will continue into the next morning.  

Friday, May 31, 2024 

• 6:08am: QA confirmed that after reverting the code related to the release item, Service Request notifications were being received. The team continues to test in QA and Staging environments.  

• 7:28am: The status page was updated from an Investigation to an Identified Phase.  

• 10:31am: The hotfix release to roll back the prior build was completed. Customers were notified to confirm the fix. All customers were alerted via Status Page that we are now moving into a Monitoring Phase.  

• 12:23pm: As we have not seen further service disruptions after the fix was implemented, we have moved the Status Page to the Resolved phase.  

Total Duration of Event: 

1 Day, 3 Hours 

Root Cause:  

On May 22, 2024, a release to product, Eptura Workplace, caused the notifications for service requests not to be sent. 

Remediation: 

To resolve this issue, the decision was made to roll back the release on the prior build for Service Requests.   

Preventative Action:  

 To enhance the stability and performance of our iOffice Connect modules, our engineering team is implementing robust measures to prevent future disruptions. We are automating email verification across all modules to ensure consistency and reliability. Additionally, until these automations are fully operational and integrated into our Automation Regression suite, we will continue to manually verify all email generation scenarios before each software release. This meticulous approach guarantees that you receive the most reliable and efficient service possible.