Resilience & Scalability

Fault tolerance is keeping a system serving correct results while parts of it fail. Resilience patterns — timeouts, retries, circuit breakers, bulkheads, fallbacks — contain a single failure so it degrades one feature instead of cascading and taking the whole distributed system down.

A load balancer spreads requests across identical instances so none is overloaded and dead ones are skipped. Round-robin rotates evenly, least-connections picks the least-busy instance, and hash routes by a client key. A sticky session pins a client to one instance — good for in-memory state but bad for balance and failover.

Scaling up (vertical) means a bigger machine for one instance — simple but capped, and a single point of failure. Scaling out (horizontal) means more instances behind a load balancer, so capacity grows almost linearly and survives an instance loss. A service scales out only if it is stateless — session and data in a shared store — so any instance serves any request.

Caching stacks across layers. The client caches assets per Cache-Control; a CDN caches content at edge nodes near users; the API gateway caches common responses to shield the backend; the server caches query results in a store like Redis. Layers nearer the user cut latency; nearer the data cut recomputation.

First, pre-scale ahead of the event — add instances and read replicas while load is still low. Second, protect the money path: wrap payment and inventory calls in timeouts and circuit breakers so a slow dependency fails fast instead of stalling checkout. Third, shed non-essential load by value — turn off recommendations, suggestions, and heavy analytics first. Spend scarce capacity on the checkout path.

A circuit breaker wraps a dependency's calls and stops them once it looks unhealthy. Closed counts failures; Open fails calls fast past a failure-rate threshold; Half-open lets a few trial calls through after a cooldown, closing on success or re-opening on failure. It saves resources on a dead service and stops cascading overload.

The core content — price, description, add-to-cart — must render independently of recommendations. Wrap the recommendation call in a short timeout and a circuit breaker; when it trips, return a fallback instead of an error: a cached list, best-sellers, or a hidden block. Checkout stays fully functional while one optional widget degrades — silently to the buyer, never a failed page.

Enforce the limit at the API gateway, before the backend, scoped per API key (the partner), not globally. Use a token-bucket or sliding-window counter to cap requests per second while allowing short bursts. On excess, return HTTP 429 Too Many Requests with a Retry-After header — never a silent drop or 500. Each client has its own bucket, so throttling one partner spares the rest.

On a transient failure the client retries but waits longer each attempt — exponential backoff (1s, 2s, 4s…) — so it does not hammer a struggling service. Jitter randomizes each delay so clients that failed together do not retry in sync. A retry is dangerous when the operation is not idempotent (a retried payment charges twice) or when it amplifies the overload the retries caused.

RTO is how long you may be down after a failure; RPO is how much recent data you may lose. For payments both must be tight — RTO of minutes, RPO near zero — because lost transactions are money and trust. Near-zero RPO needs synchronous replication to a second region; low RTO needs a warm/hot standby with automated failover, not a manual restore from nightly backups.

Named after a ship's watertight compartments, the bulkhead pattern partitions a shared resource so one overloaded consumer cannot drain it for everyone. Each dependency gets its own pool — separate thread and connection pools or queues — with fixed limits, so a hung service saturates only its pool while others keep capacity. It isolates resource contention, not the failure itself.

Choosing AP keeps the cart writable during a partition, so replicas can diverge — a removed item might briefly reappear or a count go stale until they reconcile. That is fine: a cart is a private, re-editable draft, and blocking adds loses sales. At checkout you switch to CP — price, stock, and payment must be strongly consistent — so you never buy out of stock or at a stale price.

CAP says a partition forces a choice between consistency and availability; with a healthy network you get both, so all three hold only when not partitioned. When both data centres take writes but cannot sync, they either reject writes (CP) or accept divergent ones (AP). For order placement I choose CP — queue or refuse minority-side writes — because a conflicting order costs more than a brief retryable outage.

Use an expand-migrate-contract rollout, not one blocking ALTER. Expand: add the new column additively so old and new coexist. Migrate: dual-write to both, then backfill existing rows in small batches so no long lock is held. Once new is populated and reads switch to it, contract: stop writing the old column and drop it later. No step takes a global lock, so writers never block.