Post-Mortem: image limit (June 10th 2024)

Summary

Following increased system load today, users experienced delays in messaging and email processing. Image registry rate limiting  prevented services from restarting properly. This caused cascading dependent systems to fall behind in processing, resulting in delayed message and email delivery.

Resolution

The issues were resolved by taking two key actions:

• Monitoring existing services - Monitored existing services to make sure that they were running as performant as possible while investigating the underlying image pull failures.

• Image registry migration - Converted affected services to pull images from a more reliable image registry location that doesn't have the same rate limiting constraints.

Incident Timeline

Root Causes

• Image Registry Rate Limiting: The primary image registry had restrictive rate limits that prevented services from pulling server images during restart events, blocking the successful restart of services that were failing health checks.

• Service Health Check Failures: services were restarting due to health check failures, and the subsequent server image pull failures during restart created a cascading effect that impacted message and email processing across the platform.

Additional Notes

To prevent similar issues in the future, the following measures will be implemented:

• Image pre-caching - Implement image pre-caching on nodes to reduce dependency on external registry pulls during scaling events.

• Enhanced monitoring - Add specific monitoring for image pull failures and registry rate limit warnings.

• Root Cause Analysis: Examination into the persistent health check failures experienced by the service.

The issue has been resolved and users should be able to navigate without significant lag or latency.

We are continuing to work on a fix for this incident. We believe we have identified the issue and are deploying a potential fix. If that works we will update to Monitoring status otherwise we will revert back to Investigating status.

Post-Mortem: Messaging Service Incident (May 28, 2024)

Summary

Following the release of a new messaging feature early on May 28, 2024, users experienced general slowness in messaging related requests. The new messaging feature increased the number of requests hitting our gateway, triggering rate limit issues that hadn't been encountered before. Additionally, messaging failed to load in the task drawer due to invalid feature flag configurations related to customers, and channel loading was slow due to an expensive database query.

Resolution

The issues were resolved by taking three key actions:

1. Increasing resources on affected services

2. Temporarily scaling gateway services to handle the increased request volume while implementing a permanent fix that adjusted rate limits to more reasonable values

3. Optimizing the expensive database query to retrieve only necessary data, reducing contention on the database

Incident Timeline

Root Causes

• Resource Constraints: Insufficient resources allocated to select services to handle the increased load from the new messaging feature

• Gateway Rate Limiting: Rate limits on gateway services were too restrictive, causing legitimate requests to be denied when traffic increased

• Inefficient Database Queries: Certain queries were retrieving excessive data, causing database contention and slowing down channel loading

Additional Notes

To prevent similar issues in the future, we will be looking into the following:

• Proactive monitoring for rate limit thresholds, especially during new feature releases

• Load testing with realistic traffic patterns prior to major feature releases

• Database query optimization reviews as part of the deployment checklist

• Automated scaling policies for critical gateway servicesEnhanced existing

• monitoring to include coverage for areas primarily affected by the incident, such as messaging request latency and error rates.

This incident has been resolved. Thank you for your patience. We will be reviewing what caused the issue and share with you our learnings and the steps we have taken to help prevent future disruptions. Please reach out to Support if you experience any future latency.

We implemented a fix and are currently monitoring the result. Typical navigation speed, project creation, task creation, etc. are all seeing improved speed.

Our team is actively investigating the issue impacting performance. We've identified a likely root cause and are preparing a fix that will be deployed shortly. We’re closely monitoring the situation and will provide updates as soon as more information becomes available. Thank you for your patience as we work to resolve this quickly and thoroughly.

We are currently investigating this incident. Again, we have a few ideas about a potential root cause, but haven't identified the culprit yet.

The fix was successful. All logins are working well.

The improvements have been applied and we are monitoring the impact to ensure all logins are being processed quickly.

The root cause has been identified. An index is being applied that will help.