iOFFICE is currently investigating an issue with customers site access.
Incident Report for IOFFICE
Postmortem

Root Cause:
iOFFICE investigation has pointed to a degradation caused by server node(s) not resolving correctly to DNS.
Remediation:
iOFFICE received system alerts notifying both the Customer Support and Engineering teams of nodes becoming unstable. Once identified, they began restart sequences to bring services back online. These restarts initially failed due to a service becoming out of sequence, which required a production change of DNS configurations before engineering team could restart the services successfully.
Timeline:
iOFFICE system alerts began at 8:15am CST, our Engineering team begins investigation as Customer Support started to receive reports of site outages. Engineering at 8:49am CST conducted a full restart attempt to restore services quickly. However, not all services came back online. As Customer reports continue and internal testing has confirmed all sites are affected, at 8:50am CST iOFFICE declared Severity 1 incident. Engineering continued to investigate the cause of the service restart failure and noticed that the nodes were not resolving correctly. At 9:25am and 9:37am CST nodes were restarted and services attempted to pick up new DNS configurations. These restarts failed to bring in the required DNS configuration. Engineering investigation continued at 9:50am. At 11:50 am CST, Engineering turned focus to server component that is responsible for leader elections. At 12:28pm CST, they discovered that cache didn’t match on all servers and cleared it. After that, knowing that not all services were running, Engineering attempted to bring these back into alignment by conducting another restart around 12:45pm CST. It was decided at 1:15pm CST that a configuration would be required and needs to be released. At 1:50 pm CST, the configuration changes are tested and deployed, and iOFFICE was able to begin a monitoring phase at 2:10pm CST. iOFFICE was confidently able to declare an all clear after a lengthy monitoring phase at 4:06:pm CST
Preventative Action and Analysis:
The prolonged downtime was preceded by a lack of restart sequence documentation as well as required configuration changes needed to bring systems back online. iOFFICE engineering is conducting a review
and enhancing our system restart documentation. Continued efforts to add appropriate monitoring and alerting for our systems, as well as system evaluation of services used are also being considered.

Posted Sep 24, 2021 - 16:29 UTC

Resolved
All systems have returned to a stable state and customers should experience a return to regular functionality. We appreciate your patience as our team worked to identify a resolution. Reminder, As per our standard procedure we will continue to monitor operations, and will conduct a Root Cause Analysis and distribute the findings.
Posted Sep 16, 2021 - 21:07 UTC
Update
As we continue to monitor system stability, we are able to report that the Hummingbird App is back and operational. Please do not hesitate to reach out to Customer Support if you are still experiencing any issues.
Posted Sep 16, 2021 - 20:01 UTC
Monitoring
We are happy to report that Desktop systems have returned to a stable state and customers should experience a return to regular functionality. We are continuing to investigate a subsequent issue with the Hummingbird app and logins. We appreciate your patience as our team worked to identify a resolution. As per our standard procedure we will continue to monitor operations, and will conduct a Root Cause Analysis and distribute the findings.
Posted Sep 16, 2021 - 19:53 UTC
Update
Our engineering team has identified the issue causing interrupted platform access and is committed to implementing a resolution quickly. Restoring your service is our top priority and we will continue to provide additional information related to this situation. Next update will be at 2:50pm CST or as information become available.
Posted Sep 16, 2021 - 18:59 UTC
Update
As our engineering teams continue to diligently work to restore services, we thank you for your patience. We will continue to update our customers. Restoring service is our highest priority. Next update will be at 1:50pm CST or as information become available.
Posted Sep 16, 2021 - 17:59 UTC
Update
Our Engineering team is still working to get services back to a functioning state. The issue is identified and we're working diligently for the next update at 12:50 PM CST
Posted Sep 16, 2021 - 16:50 UTC
Update
We are continuing to work on a fix for this issue. Next update will be at 11:50am CST or as information become available,
Posted Sep 16, 2021 - 15:50 UTC
Identified
Our incident response team has identified the issue and is working hard to bring services back online. Our next update will be at 10:50am CST or as new information comes available.
Posted Sep 16, 2021 - 14:51 UTC
Investigating
iOFFICE is currently investigating an issue with customers site access.
Posted Sep 16, 2021 - 13:54 UTC
This incident affected: iOFFICE Modules (Space Module, Move Module, Reservation Module, Service Request Module, Asset Module, Inventory Module, Mail Module, Copy Module, Visitor Module, Insights Module) and Apps (Hummingbird App).