Post-Mortem: Messaging Service Incident (May 28, 2024)

Summary

Following the release of a new messaging feature early on May 28, 2024, users experienced general slowness in messaging related requests. The new messaging feature increased the number of requests hitting our gateway, triggering rate limit issues that hadn't been encountered before. Additionally, messaging failed to load in the task drawer due to invalid feature flag configurations related to customers, and channel loading was slow due to an expensive database query.

Resolution

The issues were resolved by taking three key actions:

1. Increasing resources on affected services

2. Temporarily scaling gateway services to handle the increased request volume while implementing a permanent fix that adjusted rate limits to more reasonable values

3. Optimizing the expensive database query to retrieve only necessary data, reducing contention on the database

Incident Timeline

Root Causes

• Resource Constraints: Insufficient resources allocated to select services to handle the increased load from the new messaging feature

• Gateway Rate Limiting: Rate limits on gateway services were too restrictive, causing legitimate requests to be denied when traffic increased

• Inefficient Database Queries: Certain queries were retrieving excessive data, causing database contention and slowing down channel loading

Additional Notes

To prevent similar issues in the future, we will be looking into the following:

• Proactive monitoring for rate limit thresholds, especially during new feature releases

• Load testing with realistic traffic patterns prior to major feature releases

• Database query optimization reviews as part of the deployment checklist

• Automated scaling policies for critical gateway servicesEnhanced existing

• monitoring to include coverage for areas primarily affected by the incident, such as messaging request latency and error rates.

This incident has been resolved. Thank you for your patience. We will be reviewing what caused the issue and share with you our learnings and the steps we have taken to help prevent future disruptions. Please reach out to Support if you experience any future latency.

We implemented a fix and are currently monitoring the result. Typical navigation speed, project creation, task creation, etc. are all seeing improved speed.

Our team is actively investigating the issue impacting performance. We've identified a likely root cause and are preparing a fix that will be deployed shortly. We’re closely monitoring the situation and will provide updates as soon as more information becomes available. Thank you for your patience as we work to resolve this quickly and thoroughly.

We are currently investigating this incident. Again, we have a few ideas about a potential root cause, but haven't identified the culprit yet.