GUIDEcx - Users unable to access projects – Incident details

Root Cause Analysis

Issue Summary:

On the morning of October 4, 2024, at 10:00 AM MST, an error spike occurred on project plan loading after the release of an improved database view designed to enhance project statistics load time and overall database performance. The issue was resolved by 10:40 AM MST.

Root Cause:

The new database view was incompatible with old pods, which caused errors when they accessed the updated view. Our automated rollback process was triggered in response to the error spike, but it only rolled back the application deployment, not the database schema. As a result, all pods continued to access the incompatible view, extending the period of disruption.

Resolution:

At 10:30 AM MST, we redeployed the latest version of the application, ensuring that all pods were compatible with the new database view, resolving the issue.

Preventive Measures:

• Improve the canary release cycle to isolate better and test database changes before full production rollouts.

• Enhance the automated rollback process to include database schema rollbacks when necessary.

This issue has been resolved.

We implemented a fix and are currently monitoring the result. The initial results are positive and it appears that access to projects has been restored.

We are currently investigating this incident. The behavior we are seeing is that users access the Project page, click on a project and rather than being directed to the Plan view are redirected back to the Project page.