Login and Project Creation Issues
- Resolved · UTC
Root issue addressed, and remaining services returned to normal operations.
Root Cause
A database that supports platform jobs and authentication exceeded allocated storage due to a higher-than-normal volume. Service provider alerting on this resource also failed to warn of pending storage limits, which prevented GuideCX platform resources from autoscaling correctly.
Solution
The GuideCX Engineering team added more autoscaling resources to address the capacity issue. Queued jobs (e.g. project creation, task status updates) were then moved to a new queue to speed up the process of working through the pending tasks. The supporting database will be moved to a new service provider with more reliable alerting.
- Monitoring · UTC
Login restored for affected users
- Investigating · UTC
Intermittent stability issues, characterized by the partial failure of trying to login or perform some actions like create templates, projects, or change task statuses, returning generic errors messages.