Distributed Systems
A distributed system is one whose parts run on different machines and talk over a network that is slow, can drop messages, and can split in two. For a system analyst the task is rarely to build one from scratch — it is to specify how the pieces scale, exchange data, and keep working when something fails, and then to defend those trade-offs in non-functional requirements. Get the model wrong and you promise a stakeholder something the CAP theorem forbids, or you write a retry that quietly double-charges a customer.
This topic moves from the data store outward. First, how a database scales — read replicas, sharding, and the consistency-versus-availability choice. Then how services exchange messages reliably — queues, logs, delivery guarantees, idempotency. Then the transports that carry the calls — REST, gRPC, GraphQL, WebSockets. Finally, how the whole thing is designed to keep serving when a piece dies. Named up front are the traps that end interviews: calling exactly-once a broker setting, expecting all three CAP guarantees at once, and retrying a non-idempotent operation.
Topic map
- Database scaling — vertical vs horizontal, read replicas, sharding and shard-key choice, sync vs async replication, caching and CQRS, and
CAPfor a data store. - Messaging reliability — queue vs log, delivery guarantees, per-partition ordering, consumer groups and offsets, the outbox pattern, dead-letter queues, and consumer lag.
- Realtime & RPC — sync vs async, REST vs gRPC vs GraphQL, WebSockets/SSE/long-polling, retries and idempotency, timeouts and circuit breakers, versioning, and the fallacies of distributed computing.
- Resilience — retries with backoff and jitter, circuit breakers, bulkheads, timeouts, graceful degradation, health checks, redundancy, and availability vs reliability.
Common traps
| Mistake | Consequence |
|---|---|
| Promising consistency, availability, and partition tolerance at once | You specified an impossible store; CAP forces a choice during a partition |
| Treating exactly-once as a broker flag | Duplicates in production; end-to-end needs at-least-once plus idempotency |
| Retrying a non-idempotent write | A retried payment charges twice |
| Expecting ordering across a whole topic | Kafka orders only within a partition, by key |
| Expecting a synchronous call to always answer | It can time out with an unknown outcome; the network is not reliable |
| Adding shards to fix a hotspot | A skewed key still maps a heavy tenant to one shard |
Interview relevance
Distributed-systems questions test whether you can reason about scale and failure precisely, not recite buzzwords. A candidate who says "during a partition the store keeps consistency or availability, never both — I would choose CP for order placement and justify it in business terms" instantly outranks one who lists "Kafka, Redis, microservices".
Typical checks:
- Where a database scales and what
CAPforces you to give up during a partition. - Which delivery guarantee a broker really provides and why exactly-once is not a flag.
- When to pick synchronous RPC over asynchronous messaging, and which call is safe to retry.
- Which resilience patterns keep one failure from cascading, and how availability differs from reliability.
Common wrong answer: "we'll use a broker with exactly-once, so messages are never lost or duplicated". In reality exactly-once holds only inside the broker's own transactions; the moment a consumer writes to an external system it is back to at-least-once, and correctness comes from an idempotent consumer, not a setting.