Eptura Workplace Status - S2 - Space - Not able to upload large floor plans

S2 - Space - Not able to upload large floor plans

Incident Report for Eptura Workplace

Postmortem

iOFFICE Detailed Root Cause Analysis – Severity 2 – June 8, 2023

Space - Unable to Load Large Floor Plans

‌

Description:

On June 8, 2023, at approximately 11:46am MDT, the Support team received reports of Space - Not able to upload large floor plans through SoftSpace or Admin Space.

Type of Event:

Service Disruption

Services\Modules Impacted:

Production – SWF Drawing Service

Timeline:

Thursday June 8, 2023

On June 8, 2023, at approximately 11:46am MDT, the Support team received reports of Space - Not able to upload large floor plans through SoftSpace or Admin Space. The issue was reported to engineering and they acknowledged that they are investigating the issue. All customers were alerted about the disruption at 12:13pm MDT through our Status Page that we are in an Investigation Phase on our Status Page. 7:28pm MDT Support is notified by our dev ops team for testing on a possible resolution.

Friday June 9, 2023

Our support team begins to test the possibility of the fix at 6:15am MDT and the Status Page is updated to Monitoring Phase. At 7:13am MDT support confirms the disruption is ongoing. The engineering team continues to investigate, and customers are notified that we are moving back into an Investigation Phase which continues till approximately 3:15pm MDT. Our engineering team has found a temporary fix by upgrading internal systems, so that customers will be able to upload plans through the weekend. Meanwhile, the engineering team works on a permanent fix through the weekend. They will also monitor the temporary fix for any issues. Customers were alerted that monitoring will continue through the weekend.

Monday June 12, 2023

Investigation and Monitoring continues.

Tuesday June 13, 2023

At 8:09am MDT the Engineering team confirms that they have identified a more permanent solution for the issue and a hotfix should be released as soon as it passed QA. At 10:51pm MDT Dev Ops confirms that the hot fix has been released to production.

Wednesday June 13, 2023

At 8:55am MDT the Support and Drafting Teams have confirmed the fix and at 9:16am MDT Support confirms that as of June 9th, no additional reports were made to the support team about this issue. All customers have confirmed that this issue has been resolved and the Status Page is moved into a Resolved Phase.

Total Duration of Event:

5 days, 21 hours, 30 mins

Remediation:

We upgraded to the latest version of Java SWF drawing service. We also increased the memory and CPU power for the cluster.

Posted Jun 22, 2023 - 21:21 UTC

Resolved

This incident has been resolved.

Posted Jun 21, 2023 - 21:35 UTC

Update

After days of monitoring we have not seen further service disruptions after the fix was implemented, we have moved to the Resolved Phase.
A Detailed RCA will be posted in this incident in 10 business days. Please stay subscribed to the page to receive updates automatically.

Posted Jun 14, 2023 - 15:16 UTC

Update

Monitoring continues for Space - Not able to upload large floor plans through this evening. We will continue to post further updates in the morning on the progress our internal teams have made. Thank you for your continued patience.

Posted Jun 14, 2023 - 01:05 UTC

Update

Monitoring continues today for full resolution. Thank you for your continued patience as we work through this disruption.

Posted Jun 13, 2023 - 15:02 UTC

Update

We are continuing to monitor for any further issues.

Posted Jun 12, 2023 - 22:25 UTC

Update

On Friday, June 9th, there were upgrades that were made to improve internal systems. Our internal team continue to monitor the initial "S2 - Space - Not able to upload large floor plans". This disruption to floorplans continues to be a work in progress. We appreciate your patience while we work through this.

Posted Jun 12, 2023 - 18:35 UTC

Update

We are continuing to monitor for any further issues.

Posted Jun 12, 2023 - 14:54 UTC

Update

We are continuing to monitor for any further issues.

Posted Jun 10, 2023 - 03:10 UTC

Monitoring

A fix has been implemented and we are monitoring the results.

Posted Jun 09, 2023 - 22:44 UTC

Update

We are continuing to work on a fix for this issue.

Posted Jun 09, 2023 - 21:13 UTC

Identified

The team has identified the issue and are working on a implementing a fix for S2 - Space - Not able to upload large floor plans.

Posted Jun 09, 2023 - 18:50 UTC

Update

We are continuing to investigate this issue.

Posted Jun 09, 2023 - 17:54 UTC

Investigating

After monitoring, we are moving back into an Investigating phase. Our internal teams continue to navigate the root cause of this issue, "Space - Not able to upload large floor plans". We appreciate your continued patience.

Posted Jun 09, 2023 - 13:59 UTC

Monitoring

A fix has been implemented and we are monitoring the results.

Posted Jun 09, 2023 - 12:20 UTC

Update

A fix has been implemented for "Space - Not able to upload large floor plans". We are moving into the Monitoring Phase for the next 2 hours.

Posted Jun 09, 2023 - 12:15 UTC

Update

We are continuing to investigate this issue.

Posted Jun 08, 2023 - 22:45 UTC

Update

We are continuing to investigate this issue.

Posted Jun 08, 2023 - 20:21 UTC

Investigating

We are currently investigating an issue with Space - Not able to upload large floor plans. Our Engineering team is currently investigating to determine the cause of the disruption.

Posted Jun 08, 2023 - 18:13 UTC

This incident affected: Eptura Workplace Modules (Space Module).