Backend Design
Resilience patterns, caching strategy, rate limiting, idempotency, circuit breakers, graceful shutdown, and choosing between REST, gRPC and messaging.
8 questions
JuniorTheoryVery commonWhat are cache-aside, write-through, and TTL eviction?
What are cache-aside, write-through, and TTL eviction?
With cache-aside the application checks the cache first and, on a miss, loads from the database and fills the cache itself. With write-through every write goes through the cache, which updates the database synchronously, so the two stay in step. TTL eviction drops an entry after a fixed lifetime, bounding how stale a value can get.
Common mistakes
- ✗Expecting the cache to fill itself on a
cache-asidemiss, when the application code has to load and populate it - ✗Thinking
write-throughspeeds writes up, when it only keeps the cache in step and still waits for the database - ✗Treating a
TTLas a consistency guarantee rather than a bound on how stale a value may be
Follow-up questions
- →What happens under
cache-asidewhen two requests miss on the same key at the same moment? - →When can a
write-throughcache still hand a reader a stale value?
JuniorTheoryCommonWhat failure does each of timeout, retry and circuit breaker address?
What failure does each of timeout, retry and circuit breaker address?
A timeout caps how long you wait on a slow call so a thread isn't blocked forever. A retry re-issues a call that failed for a transient reason (a network blip, a 503) to recover quietly. A circuit breaker stops calling a downstream that is already failing — after enough errors it opens and fails fast until the dependency recovers.
Common mistakes
- ✗Treating a
retryas safe for non-idempotent calls, so a retried write is applied twice - ✗Setting no
timeout, so one hung downstream ties up every worker thread - ✗Thinking a
circuit breakerrecovers a failing dependency rather than just shielding callers from it
Follow-up questions
- →Why is retrying a non-idempotent request dangerous, and how do you make it safe?
- →How does a
circuit breakerdecide when to move from open back to closed?
MiddleDesignCommonLast night checkout was down for forty minutes, and the cause was not checkout. A downstream pricing service began answering in 30 seconds instead of 30 milliseconds; every checkout thread parked waiting on it, the servlet thread pool filled, health checks started failing, and requests that never touch pricing died with the rest. Pricing was degraded rather than dead, and it recovered on its own once the load drained. You must make checkout survive the next occurrence. Constraints: requests that do not need pricing must keep being served throughout; while pricing is unhealthy the checkout page should still render with a cached or default price instead of an error; and traffic has to be let back in automatically once pricing is healthy, with no operator flipping a switch. How would you apply the resilience pattern circuit breaker here — what does it trip on, what happens to a call while it is open, and how does it decide the dependency has recovered?
Last night checkout was down for forty minutes, and the cause was not checkout. A downstream pricing service began answering in 30 seconds instead of 30 milliseconds; every checkout thread parked waiting on it, the servlet thread pool filled, health checks started failing, and requests that never touch pricing died with the rest. Pricing was degraded rather than dead, and it recovered on its own once the load drained. You must make checkout survive the next occurrence. Constraints: requests that do not need pricing must keep being served throughout; while pricing is unhealthy the checkout page should still render with a cached or default price instead of an error; and traffic has to be let back in automatically once pricing is healthy, with no operator flipping a switch. How would you apply the resilience pattern circuit breaker here — what does it trip on, what happens to a call while it is open, and how does it decide the dependency has recovered?
Wrap the pricing call in a breaker that counts failures and slow calls over a window and opens once their share crosses a threshold — a timeout must count, since the incident was latency, not errors. While open, the call fails instantly into a fallback with a cached or default price, so no thread parks. After a cool-down it half-opens, admits a few trial calls, and closes only if they pass.
Common mistakes
- ✗Counting only exceptions and not timeouts, so a dependency that is slow rather than failing never trips the breaker
- ✗Queueing or retrying calls while the breaker is open, which keeps threads parked and defeats the point of failing fast
- ✗Requiring a manual reset instead of half-open trial calls, so the outage lasts until someone notices
Follow-up questions
- →Why must a
timeouton the pricing client be in place before the breaker can work at all? - →What goes wrong if the half-open state admits the full traffic instead of a few trial calls?
MiddleDesignCommonYour public REST API runs on three stateless Spring Boot instances behind a load balancer. One integration partner is hammering the search endpoint and starving every other client, so you must cap each API key at 100 requests per second while still letting a short legitimate burst through. Constraints: the cap is per key across the whole fleet, not per instance, and it must survive an instance being added or restarted; a throttled caller must be told plainly that it was throttled and when it may come back; the limiter must not itself become the thing that takes the API down when its backing store is slow or unreachable; and the check may add no more than a couple of milliseconds to a request. How would you design the rate limiter, where do the counters live, what algorithm decides whether a call is admitted, and what does the API answer a caller that is over the limit?
Your public REST API runs on three stateless Spring Boot instances behind a load balancer. One integration partner is hammering the search endpoint and starving every other client, so you must cap each API key at 100 requests per second while still letting a short legitimate burst through. Constraints: the cap is per key across the whole fleet, not per instance, and it must survive an instance being added or restarted; a throttled caller must be told plainly that it was throttled and when it may come back; the limiter must not itself become the thing that takes the API down when its backing store is slow or unreachable; and the check may add no more than a couple of milliseconds to a request. How would you design the rate limiter, where do the counters live, what algorithm decides whether a call is admitted, and what does the API answer a caller that is over the limit?
Hold the counters in a store every instance shares, such as Redis, so the cap is fleet-wide rather than per node, and admit calls by a token bucket — a steady refill rate with a depth that absorbs a short burst. Answer an over-limit caller with 429 Too Many Requests and a Retry-After header. Guard the store call with a tight timeout and fail open, so a sick limiter degrades the cap, not the API.
Common mistakes
- ✗Counting per instance and dividing the cap by the fleet size, so the real limit shifts every time a node is added or restarted
- ✗Blocking the request thread until the window rolls over instead of rejecting fast, which turns a throttle into a thread-pool outage
- ✗Failing closed when the counter store is unreachable, so a limiter incident becomes a full API outage
Follow-up questions
- →Why does a fixed one-second window let a client push through twice the cap around the boundary?
- →What would you have to change to give a paying partner a higher cap than a free one?
MiddleDesignCommonYou are carving a monolith into three services. checkout is called by the browser on every page view. pricing is called by checkout on the hot path — tens of millions of calls a day, between two internal Java services, on a latency budget of a few milliseconds. invoicing must run for every completed order, can take seconds, and must never fail an order just because it is down. Your architect wants one transport everywhere 'for consistency' and has picked REST over HTTP. Make the case for choosing per link instead: for each of the three edges, say which of REST over HTTP, the binary RPC framework gRPC, or an asynchronous message broker you would use, and defend the pick on latency and payload size, on how tightly it couples caller to callee, on how each side evolves its contract, and above all on what happens to the caller when the callee is unavailable.
You are carving a monolith into three services. checkout is called by the browser on every page view. pricing is called by checkout on the hot path — tens of millions of calls a day, between two internal Java services, on a latency budget of a few milliseconds. invoicing must run for every completed order, can take seconds, and must never fail an order just because it is down. Your architect wants one transport everywhere 'for consistency' and has picked REST over HTTP. Make the case for choosing per link instead: for each of the three edges, say which of REST over HTTP, the binary RPC framework gRPC, or an asynchronous message broker you would use, and defend the pick on latency and payload size, on how tightly it couples caller to callee, on how each side evolves its contract, and above all on what happens to the caller when the callee is unavailable.
Browser to checkout stays REST: it is public, cacheable, debuggable, and every client already speaks it. checkout to pricing is a hot internal call between two Java services, so gRPC wins on a compact binary payload and a generated contract. invoicing is asynchronous by nature — publish an event to a broker, so the order completes while invoicing is down.
Common mistakes
- ✗Picking one transport for the whole system and then arguing the requirements to fit it, instead of choosing per link
- ✗Calling a slow, must-not-block dependency like
invoicingsynchronously and relying on retries to cover its downtime - ✗Putting a broker on a hot low-latency path, paying its hop and its complexity for a decoupling that link does not need
Follow-up questions
- →What does an asynchronous edge give the caller that a synchronous call with a retry cannot?
- →How does adding a field to the contract differ between
gRPCand REST over HTTP?
MiddleDesignOccasionalEvery deploy of your Spring Boot service produces a burst of 502s and a few orphaned jobs. The container orchestrator Kubernetes sends SIGTERM to the pod and force-kills it thirty seconds later. In those thirty seconds three things go wrong: the load balancer keeps routing new requests to the pod for a second or two after the signal, in-flight HTTP requests are cut off mid-response, and a @Scheduled batch task that was halfway through its work simply vanishes. Constraints: a request the pod has already accepted must never be dropped; the pod must stop receiving new traffic before it stops serving; the executor running background jobs must get a bounded chance to finish or hand its work back; and the whole sequence has to fit inside the termination grace period, because after it the process is killed regardless. What has to happen between SIGTERM and process exit, and in what order?
Every deploy of your Spring Boot service produces a burst of 502s and a few orphaned jobs. The container orchestrator Kubernetes sends SIGTERM to the pod and force-kills it thirty seconds later. In those thirty seconds three things go wrong: the load balancer keeps routing new requests to the pod for a second or two after the signal, in-flight HTTP requests are cut off mid-response, and a @Scheduled batch task that was halfway through its work simply vanishes. Constraints: a request the pod has already accepted must never be dropped; the pod must stop receiving new traffic before it stops serving; the executor running background jobs must get a bounded chance to finish or hand its work back; and the whole sequence has to fit inside the termination grace period, because after it the process is killed regardless. What has to happen between SIGTERM and process exit, and in what order?
On SIGTERM, fail the readiness probe while still serving: the load balancer takes the pod out of rotation and in-flight requests finish normally. Only then stop accepting connections and let the server drain, bounded by a timeout. Shut the executors down with shutdown() and awaitTermination so a running batch finishes or is handed back. The whole drain must fit inside the grace period.
Common mistakes
- ✗Closing the server before traffic is removed, so requests routed a moment earlier are cut off mid-response
- ✗Calling
shutdownNow()on the executor, which interrupts a batch halfway instead of letting it finish or hand back - ✗Waiting for jobs with no timeout, so the drain overruns the grace period and the process is force-killed anyway
Follow-up questions
- →Why does the readiness probe have to fail before the connector is closed, and not after?
- →How do you pick the drain timeout relative to the termination grace period?
MiddleDesignOccasionalA Spring Boot consumer reads payment.captured events from a topic in the distributed log Kafka. For each event it inserts a row into a ledger table and sends the customer a receipt email. Delivery is at-least-once: after a rebalance, or when the consumer dies between finishing the work and committing the offset, the very same event arrives again. Support is now reporting duplicated ledger rows and customers receiving the same receipt twice. Constraints: you cannot change the broker's delivery guarantee; the ledger row and the committed offset live in different systems, so they cannot be written atomically together; a redelivery may show up minutes later on a different consumer instance; and throughput must stay in the thousands of events per second. How do you make this handler idempotent — what identifies a repeat, where does the state that recognises it live, and what happens to the email on the second delivery?
A Spring Boot consumer reads payment.captured events from a topic in the distributed log Kafka. For each event it inserts a row into a ledger table and sends the customer a receipt email. Delivery is at-least-once: after a rebalance, or when the consumer dies between finishing the work and committing the offset, the very same event arrives again. Support is now reporting duplicated ledger rows and customers receiving the same receipt twice. Constraints: you cannot change the broker's delivery guarantee; the ledger row and the committed offset live in different systems, so they cannot be written atomically together; a redelivery may show up minutes later on a different consumer instance; and throughput must stay in the thousands of events per second. How do you make this handler idempotent — what identifies a repeat, where does the state that recognises it live, and what happens to the email on the second delivery?
Key the event by a stable business identifier — the payment id, never the offset — and write that key into a processed-events table inside the same transaction as the ledger row. A unique constraint turns a redelivery into a no-op, and the transaction means a crash cannot leave one written without the other. The email goes through an outbox that de-duplicates on the same key.
Common mistakes
- ✗Using the
Kafkaoffset or the delivery attempt as the deduplication key instead of a stable business identifier - ✗Checking whether the event was processed and then writing it in two separate steps, so a crash between them still double-processes
- ✗De-duplicating the database write but leaving the email outside the transaction, so the receipt still goes out twice
Follow-up questions
- →Why can the offset commit and the ledger insert never be made atomic across the two systems?
- →How would you stop the processed-keys table from growing without bound?
SeniorDesignOccasionalA product-catalogue API serves 40k reads per second from twelve JVMs, and 95% of that traffic lands on 2% of the products. Today every read goes to PostgreSQL and p99 is 180 ms. You are asked to design a two-tier cache — the in-process cache library Caffeine in front of the shared store Redis, in front of the database. Constraints: an edit to a product must be visible on every instance within a few seconds, so you cannot simply wait out a long TTL; the twelve local caches must not each hold a different version of the same hot product; a Redis failover lasting several seconds must not turn into a database stampede or an outage; and when a hot key does expire, the thousands of readers that miss on it at the same instant must not all hit the database for it. How do you lay the two tiers out, how is each invalidated, and what does the design do while Redis is unavailable?
A product-catalogue API serves 40k reads per second from twelve JVMs, and 95% of that traffic lands on 2% of the products. Today every read goes to PostgreSQL and p99 is 180 ms. You are asked to design a two-tier cache — the in-process cache library Caffeine in front of the shared store Redis, in front of the database. Constraints: an edit to a product must be visible on every instance within a few seconds, so you cannot simply wait out a long TTL; the twelve local caches must not each hold a different version of the same hot product; a Redis failover lasting several seconds must not turn into a database stampede or an outage; and when a hot key does expire, the thousands of readers that miss on it at the same instant must not all hit the database for it. How do you lay the two tiers out, how is each invalidated, and what does the design do while Redis is unavailable?
Caffeine holds the hot keys per JVM under a short TTL, Redis is the shared tier behind it, and only a miss in both reaches the database. On a write, evict in Redis and publish the key so all twelve instances drop their local copy in a second — the local TTL is only a backstop. Collapse concurrent misses on a key into one database load, and if Redis is unavailable fail open past it behind a breaker.
Common mistakes
- ✗Leaning on a short
TTLas the invalidation mechanism, so the twelve local caches still diverge for the length of that window - ✗Evicting the shared tier on a write but not the local ones, leaving each JVM serving its own stale copy of a hot product
- ✗Letting every concurrent miss on a hot key go to the database, so one expiry turns into thousands of identical queries
Follow-up questions
- →What is the worst-case staleness of a product edit in your design, and which step sets it?
- →Why is a single database load per missed key not enough on its own during a
Redisfailover?
