Scaling
Distributed rate limiting, caching and stampede protection, key generation, inter-service communication, and realtime transports.
10 questions
JuniorDesignVery commonDesign a request rate limiter for a Go HTTP service that runs as several identical instances behind a load balancer. It must cap how often each client (identified by API key or IP) may call the service. Requirements:
- The enforced limit is per-client and global across the whole fleet — N instances must not each allow the full quota, letting a client send N times the intended rate.
- The check-and-account step is correct under concurrency: two simultaneous requests from one client must not both slip through when only one slot remains (no lost-update race on the shared counter).
- A client may spend a short burst up to a cap, then be limited to a steady refill rate, rather than hard-cut at a fixed window boundary.
- A rejected request gets a clear, standard signal telling the client it was throttled and roughly when to retry.
Specify the limiting algorithm, the per-client state, and where that state lives so all instances share one limit.
Design a request rate limiter for a Go HTTP service that runs as several identical instances behind a load balancer. It must cap how often each client (identified by API key or IP) may call the service. Requirements: - The enforced limit is per-client and global across the whole fleet — N instances must not each allow the full quota, letting a client send N times the intended rate. - The check-and-account step is correct under concurrency: two simultaneous requests from one client must not both slip through when only one slot remains (no lost-update race on the shared counter). - A client may spend a short burst up to a cap, then be limited to a steady refill rate, rather than hard-cut at a fixed window boundary. - A rejected request gets a clear, standard signal telling the client it was throttled and roughly when to retry. Specify the limiting algorithm, the per-client state, and where that state lives so all instances share one limit.
Use a token bucket per client key (API key or IP): each bucket holds tokens that refill at a fixed rate up to a cap, and a request is allowed only if it can take a token. Keep the bucket state in Redis (an atomic Lua script) so all instances share one limit; wrap it in middleware that returns 429 with a Retry-After header when the bucket is empty.
Common mistakes
- ✗Keeping the counter in process memory, so each instance enforces its own limit and the real cap is N times higher
- ✗Doing a read-then-write to the shared store non-atomically, which races under concurrency and lets requests slip through
- ✗Using a fixed window instead of a bucket, allowing a double burst at the window boundary
Follow-up questions
- →How do you make the check-and-decrement against
Redisatomic? - →What key do you limit by, and how do you avoid penalising users behind a shared NAT?
JuniorDesignVery commonDesign a URL-shortener service in Go: clients submit a long URL and get back a short link, and visiting that short link redirects a browser to the original URL. Design it to satisfy these requirements:
- Each generated short key is unique — no two long URLs ever collide onto the same key, even under many concurrent create requests.
- Keys stay short (a handful of characters) and are usable in a path like https://sho.rt/{key}.
- The redirect path is on the hot path and must be fast: looking up a key and returning the original URL should be cheap and not require scanning.
- A browser hitting an existing short link is actually navigated to the long URL (a real HTTP redirect), while an unknown key returns a clear not-found.
Cover how you generate the key (and why it cannot collide), where and how you store the key → long URL mapping, and how the redirect endpoint behaves for both known and unknown keys.
Design a URL-shortener service in Go: clients submit a long URL and get back a short link, and visiting that short link redirects a browser to the original URL. Design it to satisfy these requirements:
- Each generated short key is unique — no two long URLs ever collide onto the same key, even under many concurrent create requests.
- Keys stay short (a handful of characters) and are usable in a path like https://sho.rt/{key}.
- The redirect path is on the hot path and must be fast: looking up a key and returning the original URL should be cheap and not require scanning.
- A browser hitting an existing short link is actually navigated to the long URL (a real HTTP redirect), while an unknown key returns a clear not-found.
Cover how you generate the key (and why it cannot collide), where and how you store the key → long URL mapping, and how the redirect endpoint behaves for both known and unknown keys.
Generate a unique key by base62-encoding a monotonic id (a DB sequence or Snowflake-style id), which guarantees no collisions and keeps keys short. Store key → long URL in a table indexed on key, optionally cached in Redis. Serve redirects from GET /{key} with a 301/302 and a not-found path for unknown keys.
Common mistakes
- ✗Hashing the URL and truncating, which collides and silently maps two different URLs to one key
- ✗Random keys with a check-then-insert that races, so two requests can grab the same key under load
- ✗Returning the target in a
200body instead of an HTTP redirect, so browsers do not actually navigate
Follow-up questions
- →Why does encoding a sequence avoid the collision retries that random keys need?
- →Would you use a
301or302, and how does that choice affect analytics and caching?
JuniorDesignCommonDesign a Go HTTP handler that returns a value computed by forecast(), a call that takes about one second. The handler serves a hot endpoint at roughly 10k requests per second, so it must not run forecast() on the request path. Requirements:
- A request must respond fast — it reads a cached value, never calling the slow function inline.
- The cached value is refreshed in the background so it stays reasonably fresh as the input changes over time.
- Concurrent reads must be safe and cheap, since reads vastly outnumber writes.
- When the process shuts down, the background refresh must stop cleanly.
Specify where the cached value lives, how reads and the refresh coordinate, and how you avoid recomputing the same value many times at once (a cache stampede).
Design a Go HTTP handler that returns a value computed by forecast(), a call that takes about one second. The handler serves a hot endpoint at roughly 10k requests per second, so it must not run forecast() on the request path. Requirements:
- A request must respond fast — it reads a cached value, never calling the slow function inline.
- The cached value is refreshed in the background so it stays reasonably fresh as the input changes over time.
- Concurrent reads must be safe and cheap, since reads vastly outnumber writes.
- When the process shuts down, the background refresh must stop cleanly.
Specify where the cached value lives, how reads and the refresh coordinate, and how you avoid recomputing the same value many times at once (a cache stampede).
Cache the result in a shared variable guarded by an RWMutex. A single background goroutine recomputes it on a ticker and writes under Lock; the handler reads under RLock, so the slow forecast() never blocks a request. Run the goroutine off a context so shutdown cancels it cleanly.
Common mistakes
- ✗Calling the 1s
forecast()on the request path, so every request pays the full latency - ✗Reading and writing the cached value without synchronisation, which is a data race under load
- ✗Letting many requests recompute the same stale value at once instead of a single background refresher
Follow-up questions
- →How does an
RWMutexlet many readers proceed while still serialising the writer? - →If the cache is keyed per city, how do you avoid a stampede when several keys expire together?
JuniorTheoryCommonWhat is a WebSocket and how does it differ from a normal HTTP request?
What is a WebSocket and how does it differ from a normal HTTP request?
A WebSocket is a persistent, full-duplex TCP connection between client and server, opened by upgrading an initial HTTP request via the Upgrade: websocket header. Unlike request/response HTTP — where the client must ask before the server can reply — either side can push messages at any time over the one open connection, which suits live feeds and chat.
Common mistakes
- ✗Thinking a WebSocket is just fast HTTP polling rather than one persistent connection
- ✗Believing only the server can send — WebSockets are full-duplex, both sides push
- ✗Forgetting the connection starts as an HTTP request upgraded via the
Upgradeheader
Follow-up questions
- →What does the HTTP
Upgradehandshake exchange to switch the connection to a WebSocket? - →How does a WebSocket differ from Server-Sent Events?
MiddleTheoryCommonSynchronous RPC, a message queue, or long-polling — how do you choose between them for inter-service communication?
Synchronous RPC, a message queue, or long-polling — how do you choose between them for inter-service communication?
Synchronous RPC (HTTP/gRPC) fits request/response where the caller needs the result immediately and tight coupling is acceptable. A message queue (Kafka/RabbitMQ) decouples services, absorbs bursts and enables retries, but is eventually consistent. Long-polling pushes server-side events to simple HTTP clients.
Common mistakes
- ✗Treating a message queue as a low-latency synchronous call instead of an async, eventually-consistent channel
- ✗Assuming gRPC is asynchronous and decoupled just because it rides HTTP/2 streams
- ✗Confusing long-polling with a full-duplex WebSocket connection
Follow-up questions
- →When would you put a message queue in front of a synchronous endpoint, and how does back-pressure work?
- →How does the outbox pattern keep a database write and a queue publish consistent?
MiddleTheoryCommonWhat are the common rate-limiting algorithms, and how do token bucket and sliding window differ?
What are the common rate-limiting algorithms, and how do token bucket and sliding window differ?
Fixed window counts requests per clock interval — simple, but it allows a double burst across the boundary. Sliding window weights the previous window to smooth that edge. Leaky bucket drains a queue at a constant rate, shaping output. Token bucket refills tokens at a steady rate up to a cap, allowing a bounded burst then a steady rate — the usual default.
Common mistakes
- ✗Confusing token bucket (allows a burst) with a fixed-window counter (hard cap)
- ✗Thinking a fixed window prevents the boundary double-burst
- ✗Swapping the roles of leaky bucket and token bucket regarding bursts
Follow-up questions
- →Why does a fixed window let a client send nearly double the limit around the reset instant?
- →How would you implement token bucket atomically in
Redisacross many instances?
MiddleTheoryOccasionalHow does long polling work, and how does it differ from a regular HTTP request and from WebSockets?
How does long polling work, and how does it differ from a regular HTTP request and from WebSockets?
In long polling the client sends an HTTP request that the server holds open until it has data or a timeout fires, then responds; the client immediately reopens a new request. Unlike a normal request it does not return at once, and unlike a WebSocket it stays one-directional request/response with HTTP overhead per message.
Common mistakes
- ✗Confusing long polling with short polling on a fixed timer
- ✗Thinking the held request is full-duplex like a WebSocket
- ✗Equating long polling with HTTP/2 server push
Follow-up questions
- →Is long polling possible over HTTP/3, and what changes about the held connection?
- →When would you choose Server-Sent Events over long polling for one-way streams?
MiddleTheoryOccasionalWhat are the trade-offs of enforcing a rate limit on the client versus on the server?
What are the trade-offs of enforcing a rate limit on the client versus on the server?
A client-side limit (the caller throttles itself) saves bandwidth and protects a downstream third-party API you do not control, but a buggy or malicious client can ignore it — it is only advisory. A server-side limit is authoritative and protects the service from any client, yet it still spends resources receiving and rejecting each request. Real systems usually do both.
Common mistakes
- ✗Treating a client-side limit as authoritative rather than advisory
- ✗Assuming a server-side reject costs nothing because traffic still arrives
- ✗Thinking only one side needs to enforce the limit
Follow-up questions
- →How do you protect a rate-limited third-party API when your own clients misbehave?
- →Why does a server-side limit still consume resources even on a rejected request?
MiddleTheoryOccasionalWhen do you choose WebSockets over the server-push standard Server-Sent Events or long-polling?
When do you choose WebSockets over the server-push standard Server-Sent Events or long-polling?
Choose WebSockets for low-latency bidirectional traffic — chat, multiplayer, live trading — where the client also sends often. Server-Sent Events suit one-way server→client streams (notifications, dashboards): they ride plain HTTP with auto-reconnect, so they are simpler and proxy-friendly. Long-polling is the fallback when neither fits, paying repeated-request overhead.
Common mistakes
- ✗Reaching for WebSockets for one-way notifications where SSE is simpler and proxy-friendly
- ✗Believing SSE is bidirectional — it is server→client only
- ✗Choosing the transport by language rather than by traffic shape and direction
Follow-up questions
- →Why does SSE survive proxies and load balancers more easily than a WebSocket?
- →How does each option behave when a connection drops and must reconnect?
SeniorDesignRareA Go service running in a container with 1 GB of RAM watches for an incoming file of newline-separated 8-character strings, sorts the lines, and writes the sorted result to a new file. It now must sort files up to 2 GB — larger than the whole memory budget — and you cannot increase the RAM. Disk is plentiful. How do you sort a file that does not fit in memory? Describe the phases, what bounds peak memory during the merge, and name the algorithm.
A Go service running in a container with 1 GB of RAM watches for an incoming file of newline-separated 8-character strings, sorts the lines, and writes the sorted result to a new file. It now must sort files up to 2 GB — larger than the whole memory budget — and you cannot increase the RAM. Disk is plentiful. How do you sort a file that does not fit in memory? Describe the phases, what bounds peak memory during the merge, and name the algorithm.
External merge sort. Phase 1: read the file in chunks that each fit in RAM, sort each chunk in memory, and write it out as a sorted run file. Phase 2: k-way merge the sorted runs by reading only a small buffer from each run at a time and repeatedly emitting the smallest current line — peak memory is bounded by the number of runs plus the buffer sizes, not the file size. The result streams to the output file.
Common mistakes
- ✗Trying to load the whole file and relying on GC or compaction to fit it
- ✗Believing
mmapremoves the memory bound rather than just deferring paging - ✗Forgetting that the merge reads bounded buffers, not whole runs, into memory
Follow-up questions
- →How does the chunk size trade off the number of runs against per-run memory?
- →Why does a k-way merge with a min-heap beat repeated two-way merges for many runs?
