Resilience and Latency
A service that flies on an empty bench lives in a different world in production: requests fan out to dozens of dependencies, one of them slows down or falls over, load spikes — and the question is not whether partial failure happens, but whether the service turns it into controlled degradation or a collapse. This topic is about the techniques that keep a Go service fast and stable under exactly these conditions: how to measure latency, how to bound call time, how to retry and when to stop calling a dead dependency, how to isolate resources, and how to shed excess without going down whole.
The core trap of the topic is optimistic thinking: "the average is fine, we'll add a timeout later, on an error we'll just retry." The average hides the tail, which under fan-out catches a significant share of users; the absence of a timeout and deadline propagation piles up stuck goroutines; a naive retry without backoff and jitter finishes off a recovering dependency with a synchronized storm. Beyond that it comes down to failure discipline: a circuit breaker that stops calling a dead dependency and lets it recover; a bulkhead that isolates resources so one dependency can't sink everything; backpressure and load shedding that slow the source and return 503/429 fast instead of a silent OOM; and finally hedged requests and request coalescing that cut the tail and dedupe identical calls. The topic is laid out in layers — from measuring latency to protecting a hot key.
Topic map
- Latency and percentiles — latency versus throughput and concurrency (Little's law), measuring p50/p95/p99/p999 percentiles instead of the average, and tail dominance under fan-out.
- Timeouts and deadlines — every outbound call is bounded by a timeout, while a deadline in
context.Contextpropagates down the chain, and each hop subtracts its own time without resetting the budget. - Retries, backoff and jitter — retry only idempotent work on a transient error, with exponential backoff and jitter against a synchronized storm, capped by a retry budget.
- Circuit breaker — a closed → open → half-open machine that trips open when the error rate climbs and fails fast locally, periodically probing the dependency's recovery.
- Bulkhead — isolating resources per dependency (separate goroutine and connection pools, bounded queues) so one misbehaving dependency can't exhaust all the service's resources.
- Backpressure — under overload, signal "slow down" upstream instead of an unbounded buffer: bounded channels and queues absorb spikes and block or shed once full.
- Load shedding — under overload, deliberately drop excess and lower-priority work to protect the core path, returning
503/429fast before the system collapses. - Hedged requests — after waiting around p95, send a second copy of the request to another replica and take the first answer, cutting the tail at the cost of a capped amount of extra load.
- Request coalescing — collapse concurrent identical in-flight requests into one call (
singleflight), protecting a hot key from a cache stampede and thundering herd.
Common mistakes and traps
| Mistake | Consequence |
|---|---|
| Measuring latency by the average | A low mean hides the tail that a significant share of users land in |
| Ignoring the tail under fan-out | One rare slow dependency makes the whole request's p99 slow |
| No timeout on an outbound call | A stuck call holds a goroutine, connection and memory until the service drowns |
| Resetting the deadline at each hop instead of propagating | The chain runs past the client's budget after the client has already left |
| Retrying a non-idempotent operation | A payment retry charges the money twice |
| Retrying without backoff and jitter | A synchronized retry storm finishes off a recovering dependency |
| Calling an already-downed dependency | Cascading failure: calls pile up on timeout and keep it from recovering |
| Not isolating resources per dependency | One misbehaving dependency eats all goroutines and connections — the whole service falls |
| Buffering incoming work unbounded | Memory leaks to OOM instead of a "slow down" signal upstream |
| Drowning under overload while serving everyone | A slow death of the whole service instead of a fast 503/429 to excess requests |
| Sending N identical requests for a hot key | Thundering herd / cache stampede crushes the dependency under one-hit load |
Why it matters for interviews
Resilience and latency are the spine of the senior part of a Go system-design interview: what's tested is not knowing the pattern names but understanding how a service behaves under load and partial failure. The interviewer watches whether you measure percentiles rather than the average, remember tail dominance under fan-out, put a timeout on every outbound call and propagate the deadline, tell an idempotent retry from a dangerous one, and know when to stop calling a dead dependency, how to isolate resources, and how to degrade in a controlled way instead of collapsing.
What they usually check:
- How latency differs from throughput and concurrency and why you measure p50/p95/p99/p999 percentiles, not the average.
- Why every outbound call needs a timeout and how to propagate the deadline through
context.Contextdown the chain. - What is safe to retry, why exponential backoff with jitter is needed, and why a retry budget guards against amplification.
- How a circuit breaker works (closed → open → half-open) and why failing fast stops cascading failure.
- How a bulkhead isolates resources and why without it one dependency sinks the whole service.
- How backpressure differs from load shedding and why the buffer must not grow without bound.
- How hedged requests and request coalescing cut the tail and protect a hot key, and what they cost.
The typical wrong answer: "on an error we'll just retry, and our average is fine." That opens up a discussion of how the average hides the tail, while under fan-out it is the tail that sets the p99; that a retry without backoff and jitter finishes off a recovering dependency, while a non-idempotent retry charges the money twice; that without a timeout and circuit breaker calls to a dead dependency pile up and cause cascading failure; and that under overload you must deliberately shed excess and return 503/429 fast rather than drown while serving everyone.