GUIDEcx - Notice history

Root Cause Analysis – Redis Cache Storage Exhaustion
Date of incident: 2025-12-01
Incident window: ~1:10 PM–1:45 PM MST
Impact: Elevated latency and intermittent API timeouts on the 1.0 platform

1. Summary

At 1:10 PM MST, the Redis cache supporting the 1.0 platform reached 100% storage utilization, causing cache write failures and forcing the platform to fall back to slower database queries. This resulted in significant API latency increases, ultimately leading to timeouts for some users. An alert was triggered at 1:17 PM, the issue was identified as Redis storage exhaustion, and the Redis instance was resized. Normal operation resumed at 1:45 PM.

2. Impact
Affected system: 1.0 Platform APIs

User impact:
Some API requests experienced elevated latency
A subset of users experienced full timeouts

Business impact:
Temporary performance degradation
Increased system load during fallback operations
Duration: Approximately 35 minutes

3. Root Cause
The Redis cache used to optimize 1.0 platform queries ran out of available storage. Once storage was exhausted:

• Redis evictions and failures occurred

• Systems reliant on cached query results began performing full database queries

• Resulting load contributed to latency increases and some request timeouts

Contributing factor:
A faulty Redis storage monitoring alert failed to notify the operations team before the cache reached saturation.

4. Timeline
Time (MST) Event

• 1:10 PM Redis cache reaches 100% storage capacity; platform begins falling back to database queries, increasing latency.

• 1:17 PM Operations alert triggered due to API timeouts.

• 1:20–1:40 PM Investigation identifies Redis storage exhaustion; Redis instance is resized.

• 1:45 PM Resized Redis instance becomes fully operational; traffic and latency return to normal ranges.

5. Resolution
Redis instance was resized, increasing available storage capacity. Platform traffic and API latency returned to baseline levels after the upgrade was completed.

6. Preventative Measures
Completed
Fixed the faulty Redis storage monitor, ensuring future alerts will correctly trigger before storage is fully consumed.

This incident has been resolved. Thank you for your patience.