Reliability & SRE

An error budget is the unreliability an SLO permits: budget = 100% − SLO. A 99.9% monthly SLO allows 0.1% failure — about 43 minutes of downtime. It is a shared currency: while budget remains you ship fast; once spent you freeze risky releases. So 100% is the wrong target — you budget for failure.

An SLI is a measured signal of service health — good events over valid events, like the fraction of requests served under 300ms. An SLO is the internal target set on that SLI over a window, say 99.9% monthly. An SLA is an external contract promising customers a level, with consequences if missed; keep the SLO stricter than the SLA.

The four golden signals from Google SRE are latency (how long requests take, split by success versus error), traffic (demand on the system, like requests per second), errors (the rate of failed requests), and saturation (how full the most constrained resource is). RED and USE are close cousins.

Availability is uptime over total time, so 99.9% permits 0.1% downtime: about 8.76 hours a year, 43 minutes a 30-day month, or 10 minutes a week. Each added nine cuts the budget roughly tenfold — 99.99% is about 52 minutes a year. This downtime budget is exactly the SLO's error budget.

Burn rate is how fast you spend the error budget versus the SLO window: rate 1 spends the whole budget over the window, 14x spends it 14× faster. A burn-rate alert fires on that speed, usually over a long window for signal plus a short one to confirm. Unlike a static threshold it tracks SLO impact: it pages on fast burns, ignores blips.

MTTR (mean time to recovery) is the mean time from incident to recovery; MTBF (mean time between failures) is the average uptime between incidents. Availability rises with MTBF, falls with MTTR. MTTR = detect + acknowledge + diagnose + fix, and faster detection via the golden signals wins most. Severity sets response urgency.

Degrade instead of failing: serve a reduced but useful experience — cached or default responses, or hide the non-critical feature — when the dependency is down. Put a timeout on every call so a slow dependency cannot pin threads, a circuit breaker to fail fast (probing in half-open), and bulkheads to isolate its pool. Under overload, shed low-priority traffic early, prioritizing by user value.

Pick SLIs from user-facing success and latency, set an SLO like 99.9% over a rolling 28 days, and define the error budget as 1 − SLO. While budget remains, ship weekly; as it nears zero, slow risky changes; once exhausted, freeze feature releases (reliability and security fixes only) until it recovers, per a rule dev and SRE agreed up front. The target sits below 100% to budget for failure.

A postmortem is the record after a significant incident: a timeline reconstructed from monitoring and the golden signals, the user impact, causes, and action items with owners. Blameless means it targets systemic and process gaps, not individuals — everyone acted reasonably given what they knew — so people report honestly and the team learns.

Toil is manual, repetitive, automatable, tactical work that scales linearly with the service and adds no lasting value — like hand-running the same restart. Meetings are not toil. The SRE model caps it near 50% to fund engineering that removes future toil.

Estimate peak demand (baseline times the 5x multiplier plus growth), then load-test one instance for its max throughput at target latency — the per-unit model. Provision for the peak with headroom (N+1 plus room for a lost zone and spikes), not 100% use. Set autoscaling limits, confirm downstreams (DB, quotas, third parties) scale, and load-test before the event.

Run in at least two regions so one region's loss is survivable. Active-active serves from both behind a global load balancer with health-checked failover — highest availability, but needs replication; active-passive fails over to a warm standby — cheaper, with RTO and data loss up to the RPO. Sync replication tightens RPO, async adds lag; watch latency and split-brain, and rehearse failover.

Page only on actionable, user-facing symptoms — SLO burn via burn-rate alerts — and route causes and low-priority signals to tickets or dashboards. Run a primary/secondary rotation with escalation, map severities to a response SLA, give each alert a runbook, and use an incident commander for major incidents. Prune noisy alerts, because fatigue makes on-call miss real pages and raises MTTR.

This is a retry storm. B lost a replica and slowed; A retried immediately with no backoff or jitter, and retries stacked across layers, so B's load jumped 5x (8k→41k QPS) — a thundering herd that saturated it and cascaded via a shared thread pool. Even after recovery, retry traffic kept it pinned. Fix: exponential backoff with jitter, a retry budget that caps and stops retries, and a circuit breaker (open on errors, half-open to probe), plus timeouts and bulkheads; retry only idempotent calls.