Duration: 48 minutes

Affected Components: , Advanced Time Tracking, Report Navigator and Report Builder, Compass Customer Portal, Resource Management, Project Management, Web Application →

Mar 26, 01:00:01 GMT+0
Identified - Maintenance is now in progress.

Mar 26, 01:47:41 GMT+0
Completed - Maintenance has completed successfully. .

Mar 26, 01:00:00 GMT+0
Identified - We are planning for a scheduled maintenance during that time. We don't anticipate having any down time within the application, but wanted everyone to be made aware. .

Duration: 22 minutes

Affected Components: , Advanced Time Tracking, Report Navigator and Report Builder, Compass Customer Portal, Resource Management, Project Management, Web Application →

Jan 30, 17:44:31 GMT+0
Investigating - We are currently investigating reports of timeout issues for SAML SSO impacting some customers.

Jan 30, 18:00:45 GMT+0
Monitoring - We implemented a fix and are currently monitoring the result..

Jan 30, 18:06:54 GMT+0
Resolved - This incident has been resolved..

Jan 30, 18:28:57 GMT+0
Postmortem - This morning, between approximately 10:00 and 10:45 MST, a small number of users experienced intermittent issues signing in using SAML SSO. Because the impact was limited and overall error rates remained within SLA thresholds, automated monitoring did not alert the operations team. After receiving customer reports, our team promptly investigated, identified a configuration issue affecting one of the services supporting SAML authentication, and resolved it. Full SAML SSO functionality was restored by 10:45 MST. We apologize for the inconvenience to the affected users..

Duration: 50 minutes

Jan 27, 17:05:52 GMT+0
Investigating - We’re aware of an issue affecting the legacy (1.0) platform and are actively investigating. The team is fully engaged and working to resolve this as quickly as possible. We’ll share updates as they become available..

Jan 27, 17:15:05 GMT+0
Monitoring - We have identified the issue and it has been resolved. We'll continue to monitor the issue..

Jan 27, 17:55:43 GMT+0
Resolved - **RCA – Root Cause Analysis: Latency Issues with 1.0 Platform** **Status: Resolved** **Date:** January 27, 2026 **Incident Window:** 9:06 AM – 10:15 AM MST **Summary** Between 9:06 AM and 10:15 AM MST, the 1.0 platform experienced elevated latency and intermittent API timeouts. The root cause was identified as storage exhaustion within the Redis cache cluster. This saturation forced the platform to fall back to primary database queries, leading to system congestion and slower response times. **Timeline** * **9:06 AM:** Incident start; Redis storage reaches capacity. * **9:26 AM:** **System alerts received by IT and Operations teams.** * **9:33 AM:** Formal incident ticket submitted and investigation intensified. * **9:50 AM:** Root cause confirmed as Redis cluster saturation. * **9:55 AM:** Cluster upgrade initiated to increase storage capacity. * **10:06 AM:** Upgrade complete; cache layer fully operational. * **10:15 AM:** **Resolution:** API latency returned to baseline. **Resolution & Prevention** To resolve the incident, the Redis cluster was vertically scaled to increase available storage. To prevent a recurrence, we are **recalibrating our monitoring and alerting thresholds.** These updates will ensure that alerts are triggered more appropriately and earlier in the utilization cycle, allowing our team to intervene before the system reaches saturation..

Duration: 1 hour and 28 minutes

Affected Components: API v1, API v2, Advanced Time Tracking, Report Navigator and Report Builder, Compass Customer Portal, Resource Management, Project Management

Dec 13, 03:30:00 GMT+0
Identified - We are planning for a scheduled maintenance during that time. The product may be offline for a portion of the time. .

Dec 13, 03:30:01 GMT+0
Identified - Maintenance is now in progress.

Dec 13, 04:21:57 GMT+0
Identified - Maintenance is now in progress. Starting at 9:19 PM MST, you may experience latency or an inability to login. The team is actively working on deploying this infrastructure update. .

Dec 13, 04:29:54 GMT+0
Identified - Maintenance is still in progress. Services were restored at 9:23 PM MST and the team is now validating changes. Any downtime or latency due to the maintenance was limited to that 4 minute window of 9:19-9:23\. We will provide a final update once testing and validation is completed. .

Dec 13, 04:57:38 GMT+0
Completed - Maintenance has completed successfully. All of our testing and validation been completed and the results have been promising. Thank you for your patience. We hope you enjoy your weekend!.

Duration: 31 minutes

Affected Components: Advanced Time Tracking, Report Navigator and Report Builder, Compass Customer Portal, Resource Management, Project Management

Dec 1, 20:30:00 GMT+0
Investigating - We are currently investigating this incident..

Dec 1, 20:44:00 GMT+0
Monitoring - We implemented a fix and are currently monitoring the result..

Dec 1, 21:01:18 GMT+0
Resolved - This incident has been resolved. Thank you for your patience. .

Dec 1, 22:28:20 GMT+0
Postmortem - **Root Cause Analysis – Redis Cache Storage Exhaustion** Date of incident: 2025-12-01 Incident window: \~1:10 PM–1:45 PM MST Impact: Elevated latency and intermittent API timeouts on the 1.0 platform 1\. Summary At 1:10 PM MST, the Redis cache supporting the 1.0 platform reached 100% storage utilization, causing cache write failures and forcing the platform to fall back to slower database queries. This resulted in significant API latency increases, ultimately leading to timeouts for some users. An alert was triggered at 1:17 PM, the issue was identified as Redis storage exhaustion, and the Redis instance was resized. Normal operation resumed at 1:45 PM. 2\. Impact Affected system: 1.0 Platform APIs User impact: Some API requests experienced elevated latency A subset of users experienced full timeouts Business impact: Temporary performance degradation Increased system load during fallback operations Duration: Approximately 35 minutes 3\. Root Cause The Redis cache used to optimize 1.0 platform queries ran out of available storage. Once storage was exhausted: * Redis evictions and failures occurred * Systems reliant on cached query results began performing full database queries * Resulting load contributed to latency increases and some request timeouts _Contributing factor_: A faulty Redis storage monitoring alert failed to notify the operations team before the cache reached saturation. 4\. Timeline Time (MST) Event * 1:10 PM Redis cache reaches 100% storage capacity; platform begins falling back to database queries, increasing latency. * 1:17 PM Operations alert triggered due to API timeouts. * 1:20–1:40 PM Investigation identifies Redis storage exhaustion; Redis instance is resized. * 1:45 PM Resized Redis instance becomes fully operational; traffic and latency return to normal ranges. 5\. Resolution Redis instance was resized, increasing available storage capacity. Platform traffic and API latency returned to baseline levels after the upgrade was completed. 6\. Preventative Measures Completed Fixed the faulty Redis storage monitor, ensuring future alerts will correctly trigger before storage is fully consumed..

Duration: 4 hours

Affected Components: API v1, API v2, Advanced Time Tracking, Report Navigator and Report Builder, Compass Customer Portal, Resource Management, Project Management

Nov 14, 02:00:00 GMT+0
Identified - We are planning for a scheduled maintenance during that time. Services may be disrupted or degraded during this time. .

Nov 14, 02:00:01 GMT+0
Identified - Maintenance is now in progress.

Nov 14, 06:00:00 GMT+0
Completed - Maintenance has completed successfully.

Duration: 1 hour and 13 minutes

Affected Components: , Project Management, Web Application →

Sep 16, 23:01:16 GMT+0
Investigating - We are currently investigating this incident. .

Sep 16, 23:18:09 GMT+0
Investigating - We are currently investigating this incident. This is our top priority right now and we have our engineering team actively investigate potential solutions. A root cause has yet to be identified. .

Sep 16, 23:34:01 GMT+0
Investigating - We are currently investigating this incident and login access has been restored, although it can take up to a minute or so to login in. Once a user is logged in they can navigate the app as expected and are experiencing normal navigation speeds. Users should be able to log in now, but again will experience a minor delay after entering your password. .

Sep 17, 00:08:08 GMT+0
Monitoring - We implemented a fix and are currently monitoring the result. The fix has been deployed and users should be able to log in without any delay. Any users that were currently logged in during this degraded performance didn't experience any additional slowness while logged in. .

Sep 17, 00:14:19 GMT+0
Resolved - This incident has been resolved. Thank you for your patience as we navigated restoring the log in flow. .

Sep 17, 23:25:36 GMT+0
Postmortem - ## **Post-Mortem: Service Provider (Atlas) MongoDB Connection Incident (September 16, 2025)** **Summary** On September 16, 2025, the access-audit service experienced widespread connection failures to MongoDB Atlas, causing major latency in the login flow and service disruptions. The MongoDB was taking too long to respond, resulting in timeouts for requests sent there. MongoDB was making some infrastructure changes that affected us and other clients. The issue was resolved through implementation of a workaround to handle MongoDB timeouts more gracefully. **Resolution** The issue was resolved by: Deploying a change to the access-audit service to Gracefully handle the timeouts. Incident Timeline | **Time (MDT)** | **Date** | **Status** | | -------------- | -------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | 4:34 PM | Sep 16 | Engineering was alerted through our automated alerts that we were seeing unexpected latency. Posts were also made to the engineering channel to alert other engineers. | | 4:47 PM | Sep 16 | Support raises alarm that they are seeing customer impact in the Product Support channel. | | 4:53 PM | Sep 16 | Engineering assembles in a war room and communicates to the org they are investigating. | | 5:01 PM | Sep 16 | The situation is deemed an incident and the status page is updated to indicate investigation is underway. | | 5:18 PM | Sep 16 | Engineering team actively investigating, root cause not yet identified | | 5:34 PM | Sep 16 | Login access restored with \~1 minute delay, users can navigate normally once logged in | | 6:08 PM | Sep 16 | Fix implemented and deployed, users should be able to log in without delay | | 6:14 PM | Sep 16 | Incident resolved, login flow restored to normal operation | **Root Causes** * Atlas experienced an issue when implementing a feature flag for serverless MongoDB databases, causing latency for us and others of their customers. **Observed Evidence**: **Contributing Factors:** * Service did not gracefully handle MongoDB connection timeouts, causing complete service failures instead of degraded operation. * Authentication endpoints were dependent on the response from audit calls, though successful completion or error did not impede user login. A fix was implemented to cease awaiting that response, thereby allowing the continued processing of login, logout, and other requests (e.g., SendEmail) irrespective of the audit's response. **Additional Notes** To prevent similar issues in the future, we will be implementing the following: * Implementing better timeout handling and retry logic for MongoDB connections * Adding in better logging to indicate connection issues. * Code adjustments to Mongo follow our existing database connection processes..

Duration: 2 hours

Affected Components: Report Navigator and Report Builder, Resource Management, Project Management, API v1, API v2, Advanced Time Tracking, Compass Customer Portal

Jun 22, 02:00:00 GMT+0
Identified - We are planning for a scheduled maintenance during that time..

Jun 22, 04:00:00 GMT+0
Completed - Maintenance has completed successfully.

Jun 22, 02:00:01 GMT+0
Identified - Maintenance is now in progress.

Duration: 49 minutes

Affected Components: , Report Navigator and Report Builder, Resource Management, Project Management, Advanced Time Tracking, Compass Customer Portal, Web Application →

Jun 10, 20:59:55 GMT+0
Investigating - We are currently investigating this incident. Users are seeing slowness or errors when it comes to Custom Fields, Messages, and general navigation. Thank you for your patience and we will continue to update as more information becomes available. .

Jun 10, 21:13:26 GMT+0
Identified - We are continuing to work on a fix for this incident. We believe we have identified the issue and are deploying a potential fix. If that works we will update to Monitoring status otherwise we will revert back to Investigating status. .

Jun 10, 21:48:33 GMT+0
Resolved - The issue has been resolved and users should be able to navigate without significant lag or latency. .

Jun 11, 15:57:10 GMT+0
Postmortem - # **Post-Mortem: image limit (June 10th 2024)** ## **Summary** Following increased system load today, users experienced delays in messaging and email processing. Image registry rate limiting prevented services from restarting properly. This caused cascading dependent systems to fall behind in processing, resulting in delayed message and email delivery. ## **Resolution** The issues were resolved by taking two key actions: * Monitoring existing services - Monitored existing services to make sure that they were running as performant as possible while investigating the underlying image pull failures. * Image registry migration - Converted affected services to pull images from a more reliable image registry location that doesn't have the same rate limiting constraints. ## **Incident Timeline** | Time | Event | | ------- | -------------------------------------------------------------------- | | 1:15 PM | First error detected | | 1:17 PM | Engineering team notified of the incident | | 1:34 PM | Root cause identified as image registry rate limiting | | 2:50 PM | Fix deployed to production (services migrated to new image registry) | | 3:46 PM | All issues resolved, system returned to normal operation | ## **Root Causes** * Image Registry Rate Limiting: The primary image registry had restrictive rate limits that prevented services from pulling server images during restart events, blocking the successful restart of services that were failing health checks. * Service Health Check Failures: services were restarting due to health check failures, and the subsequent server image pull failures during restart created a cascading effect that impacted message and email processing across the platform. ## **Additional Notes** To prevent similar issues in the future, the following measures will be implemented: * Image pre-caching - Implement image pre-caching on nodes to reduce dependency on external registry pulls during scaling events. * Enhanced monitoring - Add specific monitoring for image pull failures and registry rate limit warnings. * Root Cause Analysis: Examination into the persistent health check failures experienced by the service..

Duration: 2 hours and 57 minutes

Affected Components: Report Navigator and Report Builder, Resource Management, Project Management, Advanced Time Tracking, Compass Customer Portal

May 28, 15:54:17 GMT+0
Investigating - We are currently investigating this incident. We have already boosted resources to reduce lag, but still haven't found the root issue. We have a couple of leads we are still investigating. .

May 28, 16:24:52 GMT+0
Investigating - We are currently investigating this incident. Again, we have a few ideas about a potential root cause, but haven't identified the culprit yet. .

May 28, 17:16:35 GMT+0
Investigating - Our team is actively investigating the issue impacting performance. We've identified a likely root cause and are preparing a fix that will be deployed shortly. We’re closely monitoring the situation and will provide updates as soon as more information becomes available. Thank you for your patience as we work to resolve this quickly and thoroughly..

May 28, 18:04:52 GMT+0
Monitoring - We implemented a fix and are currently monitoring the result. Typical navigation speed, project creation, task creation, etc. are all seeing improved speed. .

May 28, 18:50:50 GMT+0
Resolved - This incident has been resolved. Thank you for your patience. We will be reviewing what caused the issue and share with you our learnings and the steps we have taken to help prevent future disruptions. Please reach out to Support if you experience any future latency. .

May 28, 20:18:35 GMT+0
Postmortem - **Post-Mortem: Messaging Service Incident (May 28, 2024)** # **Summary** Following the release of a new messaging feature early on May 28, 2024, users experienced general slowness in messaging related requests. The new messaging feature increased the number of requests hitting our gateway, triggering rate limit issues that hadn't been encountered before. Additionally, messaging failed to load in the task drawer due to invalid feature flag configurations related to customers, and channel loading was slow due to an expensive database query. # **Resolution** The issues were resolved by taking three key actions: 1. Increasing resources on affected services 2. Temporarily scaling gateway services to handle the increased request volume while implementing a permanent fix that adjusted rate limits to more reasonable values 3. Optimizing the expensive database query to retrieve only necessary data, reducing contention on the database # **Incident Timeline** | **Time** | **Event** | | -------------------- | -------------------------------------------------------- | | 2:30 AM MDT, May 28 | New messaging feature released | | 8:12 AM MDT, May 28 | Customer reports of issues received | | 8:30 AM MDT, May 28 | Resources adjusted on affected services | | 9:00 AM MDT, May 28 | War Room initiated for coordinated response | | 11:33 AM MDT, May 28 | Optimized query deployed | | 11:35 AM MDT, May 28 | Adjusted the feature flag configuration | | 12:34 PM MDT, May 28 | Adjusted rate limit configuration deployed | | 12:50 PM MDT, May 28 | All issues resolved, system returned to normal operation | # **Root Causes** * Resource Constraints: Insufficient resources allocated to select services to handle the increased load from the new messaging feature * Gateway Rate Limiting: Rate limits on gateway services were too restrictive, causing legitimate requests to be denied when traffic increased * Inefficient Database Queries: Certain queries were retrieving excessive data, causing database contention and slowing down channel loading # **Additional Notes** To prevent similar issues in the future, we will be looking into the following: * Proactive monitoring for rate limit thresholds, especially during new feature releases * Load testing with realistic traffic patterns prior to major feature releases * Database query optimization reviews as part of the deployment checklist * Automated scaling policies for critical gateway servicesEnhanced existing * monitoring to include coverage for areas primarily affected by the incident, such as messaging request latency and error rates..

Duration: 2 hours

Affected Components: Recipe Webhook ingestion, Compass Customer Portal, Report Navigator and Report Builder, Resource Management, Project Management, Recipe API gateway, API v2, Advanced Time Tracking, Recipe runtime for job execution

May 25, 05:00:00 GMT+0
Completed - Maintenance has completed successfully.

May 25, 03:00:00 GMT+0
Identified - We are planning for a scheduled maintenance during that time. While the risk is low for having any down time in the application, it is a possibility and want you to be aware. .

May 25, 03:00:01 GMT+0
Identified - Maintenance is now in progress.

Duration: 2 hours

Affected Components: Report Navigator and Report Builder, Workbot for Slack, Resource Management, Recipe Webhook ingestion, Workato Email notifications, Project Management, Compass Customer Portal, Workato Website, API v1, Recipe API gateway, API v2, Workbot for Teams, Advanced Time Tracking, Recipe runtime for job execution

May 11, 13:00:01 GMT+0
Identified - Maintenance is now in progress.

May 8, 15:00:32 GMT+0
Identified - GUIDEcx is planning routine server cluster maintenance this Sunday. **_This will be done early in the morning (mountain time) to avoid disrupting international teams in APAC._** This important work will improve your experience by refining our autoscaling. The web interface will be temporarily unavailable during that time. The API will also be temporarily unresponsive so any automated recipe jobs running during that short period will need to be retried again outside of this window. .

May 11, 15:00:00 GMT+0
Completed - Maintenance has completed successfully.

May 11, 13:00:00 GMT+0
Identified - GUIDEcx is planning routine server cluster maintenance this Sunday. This will be done early in the morning of mountain time to avoid disrupting international teams in APAC. This important work will improve your experience by refining our autoscaling. The web interface will be temporarily unavailable during that time. The API will also be temporarily unresponsive so any automated recipe jobs running during that short period will need to be retried again outside of this window. .

Duration: 37 minutes

Affected Components: Report Navigator and Report Builder, Project Management, Resource Management, Compass Customer Portal, Advanced Time Tracking

Apr 30, 20:55:30 GMT+0
Investigating - We are currently investigating this incident..

Apr 30, 21:32:24 GMT+0
Resolved - The fix was successful. All logins are working well. .

Apr 30, 20:56:40 GMT+0
Identified - The root cause has been identified. An index is being applied that will help. .

Apr 30, 21:03:57 GMT+0
Monitoring - The improvements have been applied and we are monitoring the impact to ensure all logins are being processed quickly. .

Duration: 9 minutes

Affected Components: Report Navigator and Report Builder, Project Management, Resource Management, Compass Customer Portal, Advanced Time Tracking

Jan 13, 22:53:57 GMT+0
Resolved - This incident has been resolved..

Jan 13, 22:45:17 GMT+0
Identified - We are continuing to work on a fix for this incident. We deployed an update and are seeing some odd behavior in some of our labels. Users should still be able to login and navigate, but may have some weirdness in the UI until some additional deployments finish. .

Jan 13, 22:49:31 GMT+0
Monitoring - We implemented a fix and are currently monitoring the result..

Duration: 7 minutes

Affected Components: Report Navigator and Report Builder, Project Management, Resource Management, Compass Customer Portal, Advanced Time Tracking

Jan 13, 17:12:19 GMT+0
Identified - The root cause has been diagnosed. The access service is being updated. This is expected to be resolved very soon..

Jan 13, 17:10:03 GMT+0
Investigating - We are currently investigating this incident. The system is still available, but login credential processing is slower than usual. .

Jan 13, 17:16:00 GMT+0
Monitoring - The access service has been restarted and the pending backlog of requests are being processed. Authentication performance will be monitored for the next few minutes until login credentials are processed within 3-5 seconds as usual. .

Jan 13, 17:17:11 GMT+0
Resolved - This issue has been fully resolved. Login should be fast and consistent now for all organizations..

Duration: 2 hours and 41 minutes

Affected Components: API v1, API v2, Project Management

Nov 20, 16:13:33 GMT+0
Investigating - We are currently investigating this incident..

Nov 20, 16:44:51 GMT+0
Investigating - We are still investigating this incident. We have found some insights from our monitoring tools and are chasing down a few theories. We also understand that some issues are leaking into the UI experience and that it isn't just related to work done via the API, including issues with custom fields, new Team page, etc. .

Nov 20, 18:08:32 GMT+0
Identified - The team has identified a fix and has started working on implementing a fix. We will continue to update as progress is made. .

Nov 20, 18:54:10 GMT+0
Resolved - This incident has been resolved. Our team will continue to monitor the situation but users are no longer receiving errors within the API and the Team page and Custom Fields are loading properly. .

Nov 21, 00:26:20 GMT+0
Resolved - ### **RCA** **Summary:** The latest GUIDEcx API services run on AWS EKS Clusters. On the evening of November 19, 2024, a scheduled deployment introduced changes to a particular service related to new project and task-related features. This service also manages custom fields, certain application pages, and interfaces with Open API requests concerning custom fields. A specific database query, in certain instances, returned more data than necessary. Although these queries executed within acceptable time limits, the excessive amount of data returned created a bottleneck for other queries requesting similar data. While it can sometimes be acceptable for queries to return significant amounts of data, this query was intended to return smaller amounts, as it was being run frequently across multiple distributed services. This led to a backlog of requests, which in turn created pressure on messages, events, and other services reliant on the same data. This pressure caused the slowness and inconsistencies experienced during the incident. --- **Resolution:** The issue was resolved by identifying the cases where more data was being requested than intended and modifying those queries to only return the necessary data. Once this adjustment was made, queries executed efficiently, and the backlog of messages, events, and services was processed, allowing the system to return to normal operations. --- **Incident Timeline (in EST):** * **6:30 PM, November 19**: Code released for feature updates. Services appeared healthy and scaled appropriately. * **6:48 PM, November 19**: Automated monitoring detects minor instability, but the incident resolves itself before a support representative can investigate further. * **8:00 AM, November 20**: Engineering support discovers errors related to requests failing to connect to services. A backlog of requests causes some services to restart, resulting in intermittent failures. * **10:52 AM, November 20**: Support receives reports of API errors (later identified as timeouts), incomplete page loads, and missing custom fields on projects. Open API traffic spikes during this time (an expected occurrence, but it increases the frequency of the problematic query execution). * **12:24 PM, November 20**: The root cause is identified, and a resolution is implemented. * **1:54 PM, November 20**: The incident is fully resolved. System back pressure is processed successfully, and operations return to normal. --- **Additional Notes:** Three specific actions are being implemented to mitigate similar issues in the future: 1. Automated alerts will flag queries that return larger-than-expected payloads to clients. Use cases will define reasonable maximum thresholds for data loads. 2. Automated alerts will detect system backlogs related to messaging. Use cases will define reasonable maximum thresholds for such backlogs. 3. APIs associated with AWS EKS services will require filterable criteria for database queries unless explicitly defined otherwise..

Duration: 25 minutes

Affected Components: API v1, API v2, Report Navigator and Report Builder, Workbot for Teams, Workato Website, Workbot for Slack, Recipe API gateway, Project Management, Resource Management, Compass Customer Portal, Advanced Time Tracking, Recipe runtime for job execution, Recipe Webhook ingestion, Workato Email notifications

Oct 26, 01:24:58 GMT+0
Completed - The deployment went smoothly and was completed by 7:20 PM MDT. .

Oct 26, 01:00:00 GMT+0
Identified - We are planning for a scheduled maintenance during that time. We don't anticipate there being any disruption to your workflow, but we want you to be aware that it is happening. If you do see any latency during this time it may be due to this maintenance. .

Oct 26, 01:00:01 GMT+0
Identified - Maintenance is now in progress.

Duration: 16 minutes

Affected Components: API v1, API v2, Report Navigator and Report Builder, Workbot for Teams, Workato Website, Workbot for Slack, Recipe API gateway, Project Management, Resource Management, Compass Customer Portal, Advanced Time Tracking, Recipe runtime for job execution, Recipe Webhook ingestion, Workato Email notifications

Oct 9, 14:00:23 GMT+0
Monitoring - Fix has been applied. We are monitoring to ensure it stays stable. Access is up for all services. .

Oct 9, 13:52:28 GMT+0
Identified - AWS ECS Cluster is not auto-provisioning new instances, preventing API services from autoscaling. .

Oct 9, 07:00:00 GMT+0
Investigating - We are investigating this incident where core API services are unavailable in all regions. This is resulting in outages for the web application and integrations. .

Oct 9, 14:04:16 GMT+0
Resolved - Issue is fully resolved.

Oct 9, 23:43:35 GMT+0
Resolved - # RCA ## Summary: The GUIDEcx API services run on AWS ECS Clusters, supported by Auto Scaling Group (ASG) configurations for managing EC2 instances and ECS task placement. On October 8, 2024, the engineering team noticed that the ECS Cluster was not provisioning or de-provisioning EC2 instances correctly, which could potentially impact the ability of API services to scale according to demand. After refreshing the EC2 instances for the primary cluster, services appeared healthy and were scaling as expected. However, around 12:00 AM EDT, the clusters returned to an unhealthy state, causing ECS to remove EC2 instances that had active services running on them. This issue compounded as the cluster's unhealthy state prevented GUIDEcx services from automatically scaling back up, leaving all services stuck in a "Pending" status and resulting in a system-wide outage of GUIDEcx API services. ## Resolution: The issue was resolved by refreshing the EC2 instances in the ASG and ensuring that “Scale In Protection” was being applied correctly to new instances. This allowed the "Pending" ECS Cluster apps to start on the newly restarted instances. ## Incident Timeline (in EDT): * **11:45 PM, October 8**: Configuration changes were made to the ASG to address down-scaling issues. * **12:00 AM, October 8**: Automated monitoring first detected instability. * **3:00 AM, October 9**: Incident was first reported by customers, but most systems were still functioning. * **8:00 AM, October 9**: A scaling event caused ECS to remove EC2 instances running primary services, leading to a complete outage as ECS services had zero running tasks. * **9:45 AM, October 9**: The cause was identified, and the resolution was implemented. * **10:00 AM, October 9**: Incident fully resolved. ## Additional Notes: Our automated monitoring and alerting system detected the initial signs of instability at 12:00 AM EDT. However, the team's response to these alerts did follow our standard incident response program, leading to a delayed resolution. As a result, we have reinforced training for on-call engineers and improved our escalation policies to ensure timely responses to these automated alerts in the future. Additionally, since the auto-scaling configurations for these ECS Clusters have not been modified in over 18 months, we believe the root cause of this unhealthy behavior is vendor-related, and we are currently engaging with the vendor to ensure this issue does not recur..

Duration: 37 minutes

Affected Components: Report Navigator and Report Builder, Project Management, Resource Management, Compass Customer Portal, Advanced Time Tracking

Oct 4, 16:15:26 GMT+0
Investigating - We are currently investigating this incident..

Oct 4, 16:33:40 GMT+0
Investigating - We are currently investigating this incident. The behavior we are seeing is that users access the Project page, click on a project and rather than being directed to the Plan view are redirected back to the Project page. .

Oct 4, 16:44:59 GMT+0
Monitoring - We implemented a fix and are currently monitoring the result. The initial results are positive and it appears that access to projects has been restored. .

Oct 4, 16:52:54 GMT+0
Resolved - This issue has been resolved. .

Oct 4, 17:50:47 GMT+0
Resolved - ## Root Cause Analysis ### **Issue Summary:** On the morning of October 4, 2024, at 10:00 AM MST, an error spike occurred on project plan loading after the release of an improved database view designed to enhance project statistics load time and overall database performance. The issue was resolved by 10:40 AM MST. ### **Root Cause:** The new database view was incompatible with old pods, which caused errors when they accessed the updated view. Our automated rollback process was triggered in response to the error spike, but it only rolled back the application deployment, not the database schema. As a result, all pods continued to access the incompatible view, extending the period of disruption. ### **Resolution:** At 10:30 AM MST, we redeployed the latest version of the application, ensuring that all pods were compatible with the new database view, resolving the issue. ### **Preventive Measures:** * Improve the canary release cycle to isolate better and test database changes before full production rollouts. * Enhance the automated rollback process to include database schema rollbacks when necessary..

Duration: 1 hour and 14 minutes

Affected Components: Report Navigator and Report Builder, Project Management, Resource Management, Compass Customer Portal, Advanced Time Tracking

Aug 19, 18:21:44 GMT+0
Investigating - We are currently investigating this incident that's causing intermittent login issues..

Aug 19, 18:53:26 GMT+0
Identified - Diagnosis is complete. We are working on solving the main login issue now..

Aug 19, 19:12:22 GMT+0
Monitoring - We implemented a fix and are currently monitoring the result..

Aug 19, 19:35:30 GMT+0
Resolved - The login fix was successful. Monitoring has proven stable. All access is restored..

Aug 19, 23:00:00 GMT+0
Resolved - ## Summary During a routine upgrade of our system infrastructure, we encountered an issue related to the rate-limiting of image downloads from an external service. This rate limit disrupted the startup of essential services, leading to a temporary outage that affected the availability of certain features. ## What Happened The issue occurred during the upgrade process when the rapid and simultaneous restarting of multiple system components led to a higher-than-usual number of download requests within a short time frame. This exceeded the limits set by the external service provider, disrupting the startup of critical services. ## How We Fixed It * **Enhanced Access:** At **3:10 PM ET**, we upgraded our access to the external service, allowing for higher download limits. A new access credential was created and applied, which allowed the impacted services to restart successfully. * **Configuration Update:** We updated our system configurations to ensure more reliable access to required components in the future, reducing the likelihood of similar issues. ## What We've Done to Prevent This in the Future To prevent this from happening again, we took several steps: * **Image Caching & Version Control:** We implemented changes to cache frequently used components and pin them to specific versions, reducing the need to download them from external sources repeatedly and avoiding future rate limits. * **Upgraded Service Plan:** We upgraded our plan with the external service provider to a higher tier, increasing our allowed download capacity and providing more robust support for future operations..

Duration: 31 minutes

Affected Components: API v1, API v2, Report Navigator and Report Builder, Workbot for Teams, Workato Website, Workbot for Slack, Recipe API gateway, Project Management, Resource Management, Compass Customer Portal, Advanced Time Tracking, Recipe runtime for job execution, Recipe Webhook ingestion, Workato Email notifications

Aug 7, 19:44:40 GMT+0
Monitoring - Access is back again. We will monitor to ensure it remains stable. .

Aug 7, 19:20:38 GMT+0
Investigating - We are investigating unavailability of the web interface. The API and recipes are still working. No data has been lost. .

Aug 7, 19:21:00 GMT+0
Identified - Vercel is having an outage. GUIDEcx uses that system for hosting of the front end web interace. We are escalating with them now. .

Aug 7, 19:25:44 GMT+0
Monitoring - Vercel is back up again. The web interface is available. We will monitor the status for a few more minutes. .

Aug 7, 19:28:45 GMT+0
Monitoring - Vercel is volatile right now and access is temporarily lost again. We are continuing to escalate with them..

Aug 7, 19:37:21 GMT+0
Monitoring - Vercel status has been integrated into our status page for the top section called "Web Application". Real time updates on their progress fixing the hosting of the front end interface can be monitored in the "Edge Functions" and "Edge Middleware" components. .

Aug 7, 19:42:03 GMT+0
Monitoring - Vercel has identified the deeper root cause on their end and they are working on resolving the issue. Thousands of sites and systems around the world are having the same issue so the urgency drives the expectation this should be resolved within the next few minutes. .

Aug 7, 19:51:53 GMT+0
Resolved - Fix has been applied by Vercel. Access is consistently restored. All systems operational. .

Duration: 30 minutes

Affected Components: API v2, Report Navigator and Report Builder, Workbot for Teams, Workato Website, Workbot for Slack, Recipe API gateway, Project Management, Resource Management, Compass Customer Portal, Advanced Time Tracking, Recipe runtime for job execution, Recipe Webhook ingestion, Workato Email notifications

Jul 21, 05:00:00 GMT+0
Identified - We are planning for scheduled maintenance during this time that will require restarting some of our database clusters. During this time you may experience some intermittent availability issues with custom fields..

Jul 21, 05:30:00 GMT+0
Completed - Maintenance has completed successfully.

Jul 21, 05:00:01 GMT+0
Identified - Maintenance is now in progress.

Duration: 1 minute

Affected Components: Report Navigator and Report Builder

Jul 15, 23:59:43 GMT+0
Completed - Custom reports have now been migrated to a new version that supports scheduled email delivery..

Jul 15, 15:15:34 GMT+0
Identified - To add support for scheduled reports, custom reports will be unavailable this morning. In the meantime, all standard reports are still available and all data across both report types are still safe and secure. .

Jul 15, 23:59:12 GMT+0
Identified - The bulk update script for custom report IDs is running.

Duration: 11 minutes

Affected Components: Project Management

Jun 28, 15:43:39 GMT+0
Investigating - We are currently investigating this incident..

Jun 28, 15:44:04 GMT+0
Identified - The cause has been diagnosed and we are in the process of applying a fix. .

Jun 28, 15:49:24 GMT+0
Monitoring - The fix has been applied and we are monitoring to confirm it was fully effective. .

Jun 28, 15:55:00 GMT+0
Resolved - The fix was confirmed as successful. Any duplicate templates that were inadvertently added to projects created over the past few minutes will be automatically cleaned up this morning. .

Jun 29, 15:55:00 GMT+0
Resolved - ## Summary We recently encountered an issue related to a new feature release, which temporarily affected project creation and led to slower performance in some areas of our platform. Specifically, customers saw projects stuck in the "Creating" state with a loading screen that wouldn't go away. ## What Happened During the release of a new feature, a part of our project creation process encountered an unexpected issue. While projects were being created successfully, a background process responsible for completing the setup encountered an error, causing it to retry the process multiple times. This led to the creation of duplicate templates within projects and prevented some projects from completing their setup. Additionally, the repeated attempts to finalize project creation increased the workload on our system, leading to slower response times for certain actions, such as syncing data with external tools like Jira and Salesforce. ## How We Fixed It 1. Rapid Response: Within 15 minutes of the issue being reported, our team implemented a fix to ensure that new projects could complete their setup process without any issues. 2. Clean-Up: The cleanup of impacted jobs followed this schedule: * 2:30 PM ET: We completed tests of a script to remove duplicate templates. A total of 735 duplicates were removed, along with duplicate milestones, tasks, and attachments. * 5:46 PM ET: All projects stuck in a "Creating" state were updated, and we confirmed that known customer projects were accessible. * 5:56 PM ET: The system processed the remaining background tasks, and overall performance returned to normal as the load on our database decreased throughout the day. ## What We've Done to Prevent This in the Future To ensure this doesn't happen again, we took several steps: 1. Enhanced Quality Control: We reviewed our release processes and introduced additional quality checks and safeguards, ensuring that similar issues would be caught before they reach production. 2. Improved Monitoring: We reinstated and enhanced our monitoring systems to quickly detect and alert us to any issues affecting critical processes like project creation. 3. System Improvements: We also made improvements to how our background processes handle spikes in activity, ensuring that our system remains responsive even during high-demand periods. We understand the impact this may have had on your experience, and we are committed to learning from this event to provide you with a more reliable service moving forward..

Duration: 23 minutes

Affected Components: Report Navigator and Report Builder, Project Management, Resource Management, Compass Customer Portal, Advanced Time Tracking

Jun 13, 21:08:50 GMT+0
Investigating - We are currently investigating this incident..

Jun 13, 21:22:17 GMT+0
Monitoring - We implemented a fix and are currently monitoring the result. Most users should be able to access the Projects page. If the 500 Error occurs a simple refresh page appears to be resolving it. .

Jun 13, 21:31:27 GMT+0
Resolved - This incident has been resolved. .

Duration: 18 minutes

Affected Components: Report Navigator and Report Builder, Project Management, Resource Management, Compass Customer Portal, Advanced Time Tracking

Jun 12, 21:39:18 GMT+0
Resolved - Access is fully restored.

Jun 12, 21:21:32 GMT+0
Investigating - We are currently investigating this incident..

Jun 12, 21:25:19 GMT+0
Identified - The temporary instability is being driven by a project sync being used to improve reporting. We are putting that on hold and will run the update during a maintenance window after hours. .

Jun 12, 21:34:56 GMT+0
Monitoring - The fix has been deployed and access has been restored. We will continue monitoring to confirm the system is stable for all users in all components. .

Duration: 30 minutes

Affected Components: API v2, Project Management

Jun 9, 03:00:01 GMT+0
Identified - Maintenance is now in progress.

Jun 9, 03:30:00 GMT+0
Completed - Maintenance has completed successfully.

Jun 9, 03:00:00 GMT+0
Identified - Engineering is performing scheduled maintenance that will improve reporting datasets, custom field flexibility, and system health checks. This is only expected to take a few minutes..

Duration: 21 minutes

Affected Components: Project Management

Jun 5, 18:59:40 GMT+0
Investigating - We are currently investigating an incident where error messages appear in the interface with the warning "some fields are invalid"..

Jun 5, 19:11:21 GMT+0
Identified - The cause has been identified and we are deploying a fix now. .

Jun 5, 19:15:59 GMT+0
Monitoring - The fix has been deployed and we are no longer able to reproduce the issue. We will continue to monitor to confirm the system remains stable. .

Jun 5, 19:21:07 GMT+0
Resolved - The fix has proven stable. This is now fully resolved and all custom field validation is working consistently. .

Duration: 1 hour and 21 minutes

Affected Components: Report Navigator and Report Builder, Project Management, Resource Management, Compass Customer Portal, Advanced Time Tracking

May 29, 16:42:52 GMT+0
Identified - Intercom (a third party system used by GUIDEcx to enable Ticket/Chat Support as well as Help Center articles) is experiencing intermittent latency and loading issues. Their team is aware and working on a resolution. We will make updates here or you can follow along directly at <http://status.intercom.com/>. If you need immediate assistance and Intercom is having issues please reach out to your CSM. .

May 29, 16:48:33 GMT+0
Monitoring - We implemented a fix and are currently monitoring the result..

May 29, 18:03:58 GMT+0
Resolved - Intercom is stable again for chat and help articles.

Duration: 48 minutes

Affected Components: API v2

May 29, 15:25:00 GMT+0
Investigating - We are currently investigating this incident..

May 29, 15:52:57 GMT+0
Identified - Load balancer rules in the API need to be modified.

May 29, 16:04:19 GMT+0
Monitoring - A hotfix is in the process of being pushed now and we will monitor to confirm its effectiveness for the get project detail by ID, get task detail by ID, and patch custom field endpoints. All other endpoints are operational..

May 29, 16:12:38 GMT+0
Resolved - This is confirmed as resolved. All API endpoints are operational..

Duration: 2 hours and 12 minutes

Affected Components: API v1, API v2

May 24, 16:23:48 GMT+0
Investigating - We are currently investigating this incident..

May 24, 17:52:21 GMT+0
Resolved - This incident has been resolved. We will continue to monitor, functionality has been restored. Thank you for your patience. .

May 24, 16:57:27 GMT+0
Identified - This was caused by an edge case around our load balancer. Applying a fix now. .

May 24, 18:13:56 GMT+0
Identified - We are continuing to work on a fix for this incident. We are running into one issue during project creation through the API. The rest of the API has been restored to working properly. .

May 24, 18:35:53 GMT+0
Resolved - This incident has been resolved. We apologize for the back and forth on this but have tried to be as transparent and proactive as possible. At this point we do believe the issue is resolved and our testing has not indicated any major issues. However, we have found that there might be a few recipes that might need modifications. We will handle these on a case by case basis, but at this time there isn't any degraded performance of the API for our customer base as a whole. .

May 24, 16:54:55 GMT+0
Investigating - We are still investigating and are continuing to troubleshoot. .

Duration: 45 minutes

Affected Components: Report Navigator and Report Builder, Project Management, Resource Management, Compass Customer Portal, Advanced Time Tracking

May 23, 16:12:46 GMT+0
Monitoring - While we haven't found a root cause for the issue this morning we did a rollback to a previous version and it appears that access has been restored. We will continue to monitor to ensure we have a proper resolution. .

May 23, 16:47:39 GMT+0
Resolved - This has been resolved. All projects are reliably accessible. .

May 23, 16:02:25 GMT+0
Investigating - We are currently investigating this incident..

Duration: 9 hours and 4 minutes

Affected Components: API v2

May 23, 14:51:00 GMT+0
Monitoring - We have rolled back the change while we complete additional load testing before the GA release of new custom fields. All API access has been restored but some recipes might need to be re-run..

May 23, 07:48:00 GMT+0
Investigating - We are currently investigating this incident..

May 23, 08:10:00 GMT+0
Identified - This was caused by a load balancer change which temporarily impacted rate limits. .

May 23, 15:18:00 GMT+0
Resolved - This has been resolved. The API is consistently working on all endpoints. .

Duration: 3 hours and 35 minutes

Affected Components: Report Navigator and Report Builder, Resource Management

Feb 29, 18:02:08 GMT+0
Investigating - We are currently investigating this incident..

Feb 29, 18:04:23 GMT+0
Identified - Standard and custom dashboards in the Reports tab are temporarily unavailable while engineering updates filtering options for Report Navigator. Report Builder is still fully available. .

Feb 29, 21:08:33 GMT+0
Monitoring - This maintenance has been completed and we are monitoring to confirm the filter options are even more flexible. Both Report Navigator and Report Builder are fully available. .

Feb 29, 21:36:43 GMT+0
Resolved - Everything has been validated. Report Navigator and Report Builder are fully available with the upgraded filters. .

Duration: 15 days, 21 hours and 5 minutes

Affected Components: Project Management

Feb 26, 19:08:55 GMT+0
Investigating - We are currently investigating this incident. Some users are experiencing issues of not being able to navigate to the Team tab, save a note, change a task status or update a template. .

Feb 26, 19:41:25 GMT+0
Investigating - We are still currently investigating this incident. We will continue to update this page either at regular intervals or when progress is achieved. .

Feb 26, 20:44:26 GMT+0
Investigating - We are currently investigating this incident. We have implemented some changes that should help users that were being prompted to log in frequently, but are still investigating the root cause of the other issues regarding navigation. .

Feb 26, 22:37:02 GMT+0
Monitoring - We are implementing a fix around access tokens and are currently monitoring the result to confirm that sessions do not expire sooner than expected. .

Feb 27, 01:30:00 GMT+0
Resolved - This incident has been resolved..

Duration: 4 hours

Affected Components: API v1, API v2, Project Management, Resource Management, , Compass Customer Portal, Advanced Time Tracking, Integration Recipe Builder →

Aug 9, 06:30:00 GMT+0
Completed - Maintenance has completed successfully.

Aug 9, 02:30:00 GMT+0
Identified - We are planning for a scheduled database maintenance. This may affect the application's performance, but the application will remain fully operational during this maintenance window. .

Aug 9, 02:30:01 GMT+0
Identified - Maintenance is now in progress.

Duration: 12 hours

Apr 1, 05:00:00 GMT+0
Identified - GUIDEcx reporting services will be unavailable on Friday, April 7 at 11:00 PM to Saturday April 8, 3:00 AM during scheduled maintenance on reporting systems. During these few hours you will still be able to access the GUIDEcx web application and API services, but pages in the Reports section will be unavailable. The team will announce when maintenance officially begins and again when reporting services have been restored. This maintenance is necessary for reporting performance optimization. We apologize for any inconvenience..

Apr 1, 17:00:00 GMT+0
Completed - Maintenance has completed successfully.

Apr 1, 05:00:01 GMT+0
Identified - Maintenance is now in progress.

Duration: 12 hours

Affected Components: API v1, API v2, Project Management, Compass Customer Portal, , Advanced Time Tracking, Resource Management, Integration Recipe Builder →

Mar 23, 23:00:01 GMT+0
Identified - Maintenance is now in progress.

Mar 24, 11:00:00 GMT+0
Completed - Maintenance has completed successfully.

Mar 23, 23:00:00 GMT+0
Identified - We are planning to conduct security testing on our application. You may notice some slowness while we conduct our testing, but the application will remain functional throughout the testing..

Duration: 12 hours

Affected Components: API v1, API v2, Project Management, Compass Customer Portal, , Advanced Time Tracking, Resource Management, Integration Recipe Builder →

Mar 23, 11:00:00 GMT+0
Completed - Maintenance has completed successfully.

Mar 22, 23:00:01 GMT+0
Identified - Maintenance is now in progress.

Mar 22, 23:00:00 GMT+0
Identified - We are planning to conduct security testing on our application. You may notice some slowness while we conduct our testing, but the application will remain functional throughout the testing..

Duration: 12 hours

Affected Components: API v1, API v2, Project Management, Compass Customer Portal, , Advanced Time Tracking, Resource Management, Integration Recipe Builder →

Mar 22, 11:00:00 GMT+0
Completed - Maintenance has completed successfully.

Mar 21, 23:00:00 GMT+0
Identified - We are planning to conduct security testing on our application. You may notice some slowness while we conduct our testing, but the application will remain functional throughout the testing..

Mar 21, 23:00:01 GMT+0
Identified - Maintenance is now in progress.

Duration: 30 minutes

Affected Components: API v1, Project Management, API v2

Nov 12, 01:00:01 GMT+0
Identified - Maintenance is now in progress.

Nov 12, 01:30:00 GMT+0
Completed - Maintenance has completed successfully.

Nov 12, 01:00:00 GMT+0
Identified - We are planning some minor database maintenance that will start on Nov 11, 8:00PM EST, and will be completed by 8:30PM EST. Most maintenance can be performed without downtime. However, this change does require a restart of our database servers, which will make GUIDEcx unavailable for a few minutes as the databases come back online. This maintenance is required to wrap up some data architecture projects to increase read throughput. Our apologies for the short notice and any inconvenience this may cause your team..

Duration: 2 days and 12 hours

Affected Components: Compass Customer Portal, API v1, Resource Management, Advanced Time Tracking, Project Management, API v2

Nov 5, 00:00:00 GMT+0
Identified - Please be aware that the GUIDEcx web application and API services will be unavailable at November 5, 1:00AM UTC to November 5, 2:00AM UTC as the team will be performing scheduled maintenance at this time. During this time, you will be unable to access the GUIDEcx web application or API services. Before maintenance takes place, please make sure you save any changes to your projects as edits will not be possible during the maintenance window. The team will announce when maintenance begins and again when maintenance is complete and services have been restored. This maintenance is necessary for data migrations. We apologize for any inconvenience..

Nov 5, 00:00:01 GMT+0
Identified - Maintenance is now in progress.

Nov 5, 01:00:00 GMT+0
Completed - Maintenance has completed successfully and all services are back to normal. Thank for your patience and enjoy the faster databases! 🎉.

Duration: 1 hour

Affected Components: Project Management

Mar 3, 01:00:00 GMT+0
Identified - We are planning for a scheduled maintenance during this time. Access to the platform may be disrupted or you may experience some degraded performance..

Mar 3, 02:00:00 GMT+0
Completed - Maintenance has completed successfully.

Mar 3, 01:00:01 GMT+0
Identified - Maintenance is now in progress.

Mar 3, 02:00:00 GMT+0
Completed - Maintenance has completed successfully.

Duration: 3 hours and 30 minutes

Affected Components: Project Management

Jan 8, 09:45:00 GMT+0
Identified - Running next phase of maintenance. Platform availability may be impacted..

Jan 8, 07:30:00 GMT+0
Identified - We will be performing required database maintenance during this time. During this maintenance, the platform may be periodically unavailable..

Jan 8, 11:00:00 GMT+0
Completed - Maintenance has completed successfully..