A partial slowdown turned into a full outage after clients retried — diagnose and fix it
Service A calls service B. One of B's database replicas was lost, so B slowed and started erroring. Minutes later the whole platform was down. Below is the incident timeline from the golden-signal dashboards. Explain what turned a partial degradation into a total outage, and give the concrete fix so it cannot happen again.
14:02 svc-B p99 latency 200ms→3s, error rate 2%→15% (one DB replica lost)
14:03 svc-A retries every failed call immediately, up to 3x — no backoff, no jitter
14:03 inbound QPS to svc-B: 8k → 41k (retries stacked across A, gateway, mobile)
14:04 svc-B thread pool saturated, p99 → timeout, error rate → 100%
14:05 svc-C (shares svc-A's thread pool) also failing — cascade
14:07 the DB replica recovers, but retry traffic keeps svc-B pinned at 100% errorsDiagnose the cause and give the fix.
This is a retry storm. B lost a replica and slowed; A retried immediately with no backoff or jitter, and retries stacked across layers, so B's load jumped 5x (8k→41k QPS) — a thundering herd that saturated it and cascaded via a shared thread pool. Even after recovery, retry traffic kept it pinned. Fix: exponential backoff with jitter, a retry budget that caps and stops retries, and a circuit breaker (open on errors, half-open to probe), plus timeouts and bulkheads; retry only idempotent calls.
- ✗Blaming only the replica loss and missing the retry amplification
- ✗Fixing it by retrying harder or removing timeouts, worsening the storm
- ✗Forgetting jitter, a retry budget, or that retries need idempotency
- →Why does jitter matter on top of exponential backoff?
- →How does a retry budget differ from a per-request retry limit?
Analysis
What happened
Losing one DB replica was only the trigger, not the cause of the total outage. The cause was a retry storm: when B slowed and began erroring, A retried every failed call immediately, with no backoff and no jitter, up to 3 times. Retries stacked at every layer (mobile → gateway → A), and inbound traffic to B jumped nearly fivefold — from 8k to 41k QPS. That is a thundering herd: a then-healthy service was drowned by the retries themselves.
partial degradation (15% errors) ──retries×N──▶ 100% errors + cascadeB's thread pool saturated, and because svc-C shared a thread pool with A, the failure cascaded to it. Even when the replica returned at 14:07, the retry avalanche kept B pinned at 100% errors — the system could not climb out on its own.
The fix
- Exponential backoff + jitter — spread retries over time and desynchronize clients so they do not fire in lockstep.
- Retry budget — cap retries to a small fraction of requests (say < 10%) and stop retrying once that fraction is exceeded.
- Circuit breaker — on a high error rate, open the circuit and fail fast, probing recovery periodically in the half-open state.
- Per-call timeout + bulkheads — stop a slow dependency from pinning every thread, and isolate pools so B's failure cannot take down svc-C.
- Retry only idempotent operations.
