GUIDEcx - Duplicate Email Issues for Milestone Complete - GUIDEcx 1.0 – Incident details

All systems operational

Duplicate Email Issues for Milestone Complete - GUIDEcx 1.0

Resolved
Major outage
Started 6 days agoLasted less than a minute
Updates
  • Postmortem
    UTC
    Postmortem

    August 31, 2026

    Summary

    On August 31, 2026, an issue occurred with our email notification queue that affected providers using GUIDEcx 1.0. A legacy email service outage over the weekend caused outbound jobs to accumulate, resulting in duplicate email notifications being sent when the jobs were restarted. The duplicate sending was contained, and email services are operating normally.

    Impact

    Some providers associated with GUIDEcx 1.0 accounts received multiple duplicate email notifications during the morning hours.

    The incident was identified and contained earlier this morning. System monitoring confirms that no additional duplicate emails are being generated or sent.

    Resolution

    Our engineering team halted the queued outbound email jobs to stop additional duplicate messages from sending.

    Once contained, the team verified queue states and stabilized the email dispatch pipeline to ensure normal message flow resumed safely without further duplicate processing.

    Root Cause

    Our legacy email service went offline over the weekend, leading to a backlog of queued email jobs. When the service was restarted this morning, the backed-up jobs executed simultaneously, triggering duplicate email transmissions to affected providers on GUIDEcx 1.0.

    Corrective and Preventive Actions

    • Legacy email stack has been updated to alert if this issue arises again.

    • GUIDE 2.0 email service already contains duplicate email filtering logic.

    • Communicate improved stability benefits in GUIDE 2.0 to customers still using GUIDE 1.0 projects.

    We apologize for any confusion or inconvenience this may have caused your providers and appreciate your patience as our team worked to resolve the issue.

  • Resolved
    UTC
    Resolved