GUIDEcx - Notice history

Root issue addressed, and remaining services returned to normal operations.

Root Cause

A database that supports platform jobs and authentication exceeded allocated storage due to a higher-than-normal volume. Service provider alerting on this resource also failed to warn of pending storage limits, which prevented GuideCX platform resources from autoscaling correctly.

Solution

The GuideCX Engineering team added more autoscaling resources to address the capacity issue. Queued jobs (e.g. project creation, task status updates) were then moved to a new queue to speed up the process of working through the pending tasks. The supporting database will be moved to a new service provider with more reliable alerting.

Login restored for affected users

New containers are autoscaling successfully now. An outdated docker agent on an autoscaling group caused duplicate host mappings, leading one target group to fail health checks, which caused the minority of the tasks in the autoscaling cluster to continually restart. If requests were routed to the healthy tasks, experience was normal. But if traffic was routed to one of these “unhealthy” tasks it only had 5 minutes that it would work – hence the sporadic success or failure of requests.

A breaking API change to Docker required updating the ECS agents from 1.55.1 to 1.55.3. Once upgrade was completed, autoscaling and host port mapping worked correctly.

Fix for stability issues continues to work. Working with IaaS support on autoscaling issues.

Instability of some requests succeeding and others failing has been resolved. Team is working on root cause analysis and to reenable autoscaling.

Instability is caused by new containers failing health checks 5 to 10 minutes after starting. Load balancers updated to ensure requests are routed to old containers that are healthy.