A partial slowdown turned into a full outage after clients retried — diagnose and fix it
Service A calls service B. One of B's database replicas was lost, so B slowed and started erroring. Minutes later the whole platform was down. Below is the incident timeline from the golden-signal dashboards. Explain what turned a partial degradation into a total outage, and give the concrete fix so it cannot happen again.
14:02 svc-B p99 latency 200ms→3s, error rate 2%→15% (one DB replica lost)
14:03 svc-A retries every failed call immediately, up to 3x — no backoff, no jitter
14:03 inbound QPS to svc-B: 8k → 41k (retries stacked across A, gateway, mobile)
14:04 svc-B thread pool saturated, p99 → timeout, error rate → 100%
14:05 svc-C (shares svc-A's thread pool) also failing — cascade
14:07 the DB replica recovers, but retry traffic keeps svc-B pinned at 100% errorsDiagnose the cause and give the fix.
This is a retry storm. B lost a replica and slowed; A retried immediately with no backoff or jitter, and retries stacked across layers, so B's load jumped 5x (8k→41k QPS) — a thundering herd that saturated it and cascaded via a shared thread pool. Even after recovery, retry traffic kept it pinned. Fix: exponential backoff with jitter, a retry budget that caps and stops retries, and a circuit breaker (open on errors, half-open to probe), plus timeouts and bulkheads; retry only idempotent calls.
- ✗Blaming only the replica loss and missing the retry amplification
- ✗Fixing it by retrying harder or removing timeouts, worsening the storm
- ✗Forgetting jitter, a retry budget, or that retries need idempotency
- →Why does jitter matter on top of exponential backoff?
- →How does a retry budget differ from a per-request retry limit?
Analysis
What happened
Losing one DB replica was only the trigger, not the cause of the total outage. The cause was a retry storm: when B slowed and began erroring, A retried every failed call immediately, with no backoff and no jitter, up to 3 times. Retries stacked at every layer (mobile → gateway → A), and inbound traffic to B jumped nearly fivefold — from 8k to 41k QPS. That is a thundering herd: a then-healthy service was drowned by the retries themselves.
partial degradation (15% errors) ──retries×N──▶ 100% errors + cascadeB's thread pool saturated, and because svc-C shared a thread pool with A, the failure cascaded to it. Even when the replica returned at 14:07, the retry avalanche kept B pinned at 100% errors — the system could not climb out on its own.
The fix
clients so they do not fire in lockstep.
stop retrying once that fraction is exceeded.
recovery periodically in the half-open state.
and isolate pools so B's failure cannot take down svc-C.
- Exponential backoff + jitter — spread retries over time and desynchronize
- Retry budget — cap retries to a small fraction of requests (say < 10%) and
- Circuit breaker — on a high error rate, open the circuit and fail fast, probing
- Per-call timeout + bulkheads — stop a slow dependency from pinning every thread,
- Retry only idempotent operations.
