GUIDEcx - Notice history

Summary

We recently encountered an issue related to a new feature release, which temporarily affected project creation and led to slower performance in some areas of our platform. Specifically, customers saw projects stuck in the "Creating" state with a loading screen that wouldn't go away.

What Happened

During the release of a new feature, a part of our project creation process encountered an unexpected issue. While projects were being created successfully, a background process responsible for completing the setup encountered an error, causing it to retry the process multiple times. This led to the creation of duplicate templates within projects and prevented some projects from completing their setup.

Additionally, the repeated attempts to finalize project creation increased the workload on our system, leading to slower response times for certain actions, such as syncing data with external tools like Jira and Salesforce.

How We Fixed It

1. Rapid Response: Within 15 minutes of the issue being reported, our team implemented a fix to ensure that new projects could complete their setup process without any issues.

2. Clean-Up: The cleanup of impacted jobs followed this schedule:

   • 2:30 PM ET: We completed tests of a script to remove duplicate templates. A total of 735 duplicates were removed, along with duplicate milestones, tasks, and attachments.

   • 5:46 PM ET: All projects stuck in a "Creating" state were updated, and we confirmed that known customer projects were accessible.

   • 5:56 PM ET: The system processed the remaining background tasks, and overall performance returned to normal as the load on our database decreased throughout the day.
     

What We've Done to Prevent This in the Future

To ensure this doesn't happen again, we took several steps:

1. Enhanced Quality Control: We reviewed our release processes and introduced additional quality checks and safeguards, ensuring that similar issues would be caught before they reach production.

2. Improved Monitoring: We reinstated and enhanced our monitoring systems to quickly detect and alert us to any issues affecting critical processes like project creation.

3. System Improvements: We also made improvements to how our background processes handle spikes in activity, ensuring that our system remains responsive even during high-demand periods.
   

We understand the impact this may have had on your experience, and we are committed to learning from this event to provide you with a more reliable service moving forward.

The fix was confirmed as successful. Any duplicate templates that were inadvertently added to projects created over the past few minutes will be automatically cleaned up this morning.

The fix has been applied and we are monitoring to confirm it was fully effective.

The cause has been diagnosed and we are in the process of applying a fix.

This incident has been resolved.

We implemented a fix and are currently monitoring the result. Most users should be able to access the Projects page. If the 500 Error occurs a simple refresh page appears to be resolving it.

Access is fully restored

The fix has been deployed and access has been restored. We will continue monitoring to confirm the system is stable for all users in all components.

The temporary instability is being driven by a project sync being used to improve reporting. We are putting that on hold and will run the update during a maintenance window after hours.

The fix has proven stable. This is now fully resolved and all custom field validation is working consistently.

The fix has been deployed and we are no longer able to reproduce the issue. We will continue to monitor to confirm the system remains stable.

The cause has been identified and we are deploying a fix now.

This is confirmed as resolved. All API endpoints are operational.

A hotfix is in the process of being pushed now and we will monitor to confirm its effectiveness for the get project detail by ID, get task detail by ID, and patch custom field endpoints. All other endpoints are operational.

Load balancer rules in the API need to be modified

This incident has been resolved. We apologize for the back and forth on this but have tried to be as transparent and proactive as possible. At this point we do believe the issue is resolved and our testing has not indicated any major issues. However, we have found that there might be a few recipes that might need modifications. We will handle these on a case by case basis, but at this time there isn't any degraded performance of the API for our customer base as a whole.

We are continuing to work on a fix for this incident. We are running into one issue during project creation through the API. The rest of the API has been restored to working properly.

This incident has been resolved. We will continue to monitor, functionality has been restored. Thank you for your patience.

This was caused by an edge case around our load balancer. Applying a fix now.

We are still investigating and are continuing to troubleshoot.

This has been resolved. All projects are reliably accessible.

While we haven't found a root cause for the issue this morning we did a rollback to a previous version and it appears that access has been restored. We will continue to monitor to ensure we have a proper resolution.

This has been resolved. The API is consistently working on all endpoints.

We have rolled back the change while we complete additional load testing before the GA release of new custom fields. All API access has been restored but some recipes might need to be re-run.

This was caused by a load balancer change which temporarily impacted rate limits.