← All posts
Incidents
Post-mortem: elevated purge latency on 28 October
For 26 minutes, purges took up to 40 seconds to reach every location. Here is what happened and what we changed.
On 28 October between 14:02 and 14:28 UTC, purge requests took up to 40 seconds to propagate instead of the usual 150 ms. Cached content kept being served normally; only purges were slow.
What happened
A customer sent a purge request containing a tag with an unexpected character. One consumer of our purge log failed to parse it and retried indefinitely, blocking every purge behind it for the locations it served.
What we changed
- Malformed messages now go to a dead-letter queue after three attempts.
- A watchdog restarts any consumer that has not advanced in ten seconds.
- Tag validation now happens at the API, before anything is written to the log.
We are sorry for the disruption. The full timeline is on our status page.