GUIDEcx - Notice history

The issue has been resolved, and all systems once again appear functional!

We implemented a fix and currently monitoring the result.

SSO services are operational for all customers

Testing has completed and fix is being rolled out.

Issue has been found and fix is being tested

Requests times and database query times have returned to normal.

We implemented a fix and currently monitoring the result.

Team has resolved the immediate performance issues and performance is returning to normal. Team will continue working on the long-term solution.

Queue latency has returned to normal bounds and project creation and adding template performance has returned to normal.

Root cause identified and fix deployed. Queued jobs are now working through the backlog.

Root issue addressed, and remaining services returned to normal operations.

Root Cause

A database that supports platform jobs and authentication exceeded allocated storage due to a higher-than-normal volume. Service provider alerting on this resource also failed to warn of pending storage limits, which prevented GuideCX platform resources from autoscaling correctly.

Solution

The GuideCX Engineering team added more autoscaling resources to address the capacity issue. Queued jobs (e.g. project creation, task status updates) were then moved to a new queue to speed up the process of working through the pending tasks. The supporting database will be moved to a new service provider with more reliable alerting.

Login restored for affected users

New containers are autoscaling successfully now. An outdated docker agent on an autoscaling group caused duplicate host mappings, leading one target group to fail health checks, which caused the minority of the tasks in the autoscaling cluster to continually restart. If requests were routed to the healthy tasks, experience was normal. But if traffic was routed to one of these “unhealthy” tasks it only had 5 minutes that it would work – hence the sporadic success or failure of requests.

A breaking API change to Docker required updating the ECS agents from 1.55.1 to 1.55.3. Once upgrade was completed, autoscaling and host port mapping worked correctly.

Fix for stability issues continues to work. Working with IaaS support on autoscaling issues.

Instability of some requests succeeding and others failing has been resolved. Team is working on root cause analysis and to reenable autoscaling.

Instability is caused by new containers failing health checks 5 to 10 minutes after starting. Load balancers updated to ensure requests are routed to old containers that are healthy.