GUIDEcx - Login and Project Creation Issues – Incident details

All systems operational

Login and Project Creation Issues

Resolved
Degraded performance
Started about 4 years agoLasted about 2 hours
Updates
  • Resolved
    Resolved

    Root issue addressed, and remaining services returned to normal operations.

    Root Cause

    A database that supports platform jobs and authentication exceeded allocated storage due to a higher-than-normal volume. Service provider alerting on this resource also failed to warn of pending storage limits, which prevented GuideCX platform resources from autoscaling correctly.

    Solution

    The GuideCX Engineering team added more autoscaling resources to address the capacity issue. Queued jobs (e.g. project creation, task status updates) were then moved to a new queue to speed up the process of working through the pending tasks. The supporting database will be moved to a new service provider with more reliable alerting.

  • Monitoring
    Monitoring

    Login restored for affected users

  • Investigating
    Investigating

    Intermittent stability issues, characterized by the partial failure of trying to login or perform some actions like create templates, projects, or change task statuses, returning generic errors messages.