LLM Serving & Inference
Quantization, KV-cache memory, speculative decoding, FlashAttention, continuous batching, decoding strategies, latency vs throughput and serving cost.
11 questions
JuniorTheoryVery commonWhy is prefill compute-bound but decode bandwidth-bound, and how does that map onto time-to-first-token?
Why is prefill compute-bound but decode bandwidth-bound, and how does that map onto time-to-first-token?
Prefill pushes all prompt tokens through large parallel matmuls, so arithmetic dominates and it sets time-to-first-token. Decode emits one token per pass, re-reading every weight for little math, so bandwidth dominates and it sets inter-token latency.
Common mistakes
- ✗Assuming a GPU with more FLOPs will fix slow decode
- ✗Treating time-to-first-token and inter-token latency as one number
- ✗Forgetting that decode re-reads every weight to produce a single token
Follow-up questions
- →Why does batching raise decode throughput without improving single-stream inter-token latency?
- →How does the scheduling technique chunked prefill trade time-to-first-token against decode smoothness?
MiddlePerformanceVery commonHow does continuous in-flight batching differ from static batching, and why does throughput jump?
How does continuous in-flight batching differ from static batching, and why does throughput jump?
Static batching fixes the group and holds every slot until the longest sequence ends, so finished rows idle and newcomers wait. Continuous batching schedules per decode step — done sequences retire and queued ones take their slots, keeping the GPU busy.
Common mistakes
- ✗Thinking the win comes from a bigger batch rather than from per-step scheduling
- ✗Assuming padding to the longest sequence is harmless
- ✗Expecting continuous batching to lower single-request latency rather than raise throughput
Follow-up questions
- →How does the scheduler choose between admitting a new prefill and continuing decode?
- →What does preemption cost when the KV budget runs out mid-batch?
JuniorTheoryCommonWhat does INT8 or INT4 quantization actually shrink, what does it speed up, and what does it cost?
What does INT8 or INT4 quantization actually shrink, what does it speed up, and what does it cost?
It shrinks the bytes per weight, so the model needs less VRAM and moves less data per token. That speeds up memory-bound decode far more than compute-bound prefill. INT8 weight-only is near-lossless; INT4 costs accuracy on hard reasoning.
Common mistakes
- ✗Expecting prefill to speed up as much as decode does
- ✗Treating INT4 as free — assuming any quantization is lossless
- ✗Confusing quantization with pruning or distillation
Follow-up questions
- →When does the training-time method quantization-aware training beat post-training quantization?
- →Why does quantizing the KV cache help differently from quantizing the weights?
JuniorPerformanceCommonHow do temperature, top-k and top-p reshape sampling, and what do you set for strict JSON output?
How do temperature, top-k and top-p reshape sampling, and what do you set for strict JSON output?
Temperature scales the logits — low sharpens, high flattens. Top-k keeps the k likeliest tokens; top-p keeps the smallest set whose mass reaches p, so its width adapts. Strict JSON wants temperature 0, creative text a higher one with top-p.
Common mistakes
- ✗Thinking top-p is just top-k with a fixed-size candidate set
- ✗Setting temperature 0 and expecting top-k or top-p to still matter
- ✗Believing a fixed seed alone makes generation reproducible
Follow-up questions
- →What does the repetition-penalty knob do that temperature cannot?
- →Why can batching break bit-exact reproducibility even at temperature 0?
MiddlePerformanceCommonThe KV cache runs out of memory at long context — rank the levers by what each costs in quality.
The KV cache runs out of memory at long context — rank the levers by what each costs in quality.
Free levers first — paged blocks and prefix sharing only remove waste, and chunked prefill merely reschedules. Grouped-query attention is free if the model has it, INT8 KV quantization costs a little, and truncating context costs real quality, so it goes last.
Common mistakes
- ✗Reaching for context truncation before the free memory levers
- ✗Treating INT4 KV quantization as quality-neutral
- ✗Thinking chunked prefill changes what the model outputs
Follow-up questions
- →Why does chunked prefill help tail latency as well as memory pressure?
- →Where does prefix caching pay off most in a chat workload?
JuniorPerformanceOccasionalWhy does beam search cost roughly k times greedy decoding, and why do chat models rarely use it?
Why does beam search cost roughly k times greedy decoding, and why do chat models rarely use it?
It keeps k live hypotheses, so every step decodes k sequences and holds k KV caches — roughly k times the compute, memory and bandwidth. Chat models skip it because maximising likelihood yields bland, repetitive text and kills sampling diversity.
Common mistakes
- ✗Thinking beam search returns the globally optimal sequence
- ✗Assuming higher likelihood means a better chat answer
- ✗Forgetting that each beam carries its own KV cache
Follow-up questions
- →Where is beam search still the right default, such as in machine translation?
- →How does a length penalty change what beam search finally returns?
MiddlePerformanceOccasionalThe attention kernel FlashAttention is exact, so where does its speedup come from and what does it save?
The attention kernel FlashAttention is exact, so where does its speedup come from and what does it save?
It is an exact kernel, not an approximation. It tiles queries, keys and values into on-chip SRAM and uses a running softmax, so the full attention matrix never reaches HBM. What it saves is memory traffic, not FLOPs — it even recomputes some work.
Common mistakes
- ✗Calling FlashAttention an approximate or sparse attention method
- ✗Expecting the FLOP count to go down
- ✗Assuming the same speedup carries over to an already compute-bound regime
Follow-up questions
- →Why does the running-softmax trick keep the result numerically exact?
- →How much of the gain survives when the batch is already compute-bound?
MiddlePerformanceOccasionalWhich KV-cache fragmentation does the memory scheme PagedAttention fix, and what does it enable?
Which KV-cache fragmentation does the memory scheme PagedAttention fix, and what does it enable?
Reserving a contiguous buffer per request at maximum length wastes most of the KV cache to internal and external fragmentation. PagedAttention holds it in fixed-size blocks behind a block table, so blocks are allocated on demand, shared and evicted.
Common mistakes
- ✗Confusing paged KV storage with compressing or quantizing the KV cache
- ✗Thinking the block table sits on the CPU critical path
- ✗Missing that a shared prefix can reuse the very same physical blocks
Follow-up questions
- →How does prefix sharing between requests reuse the same physical blocks?
- →What happens when the block pool is exhausted in the middle of decode?
MiddlePerformanceOccasionalHow do draft and verify keep the target distribution unchanged, and when does speculation stop paying off?
How do draft and verify keep the target distribution unchanged, and when does speculation stop paying off?
A small draft proposes tokens; the target verifies them in one parallel pass; a rejection rule accepts a prefix and corrects the rest, so the output distribution is exactly the target's. It stops paying when acceptance drops or the batch is compute-bound.
Common mistakes
- ✗Believing speculative decoding trades quality for latency
- ✗Expecting a win at large batch sizes where the GPU is already compute-bound
- ✗Ignoring that a poorly aligned draft model can make things slower
Follow-up questions
- →How does the acceptance rate translate into an end-to-end speedup?
- →What changes when the draft comes from the target's own early layers instead of a separate model?
SeniorDesignOccasionalYou must serve a 70B-parameter chat model to 200 concurrent users on rented GPUs. Prompts average 2000 tokens, answers 300 tokens, and the product requires p95 end-to-end latency under two seconds with the answer streamed. The budget is fixed, and traffic triples during working hours and nearly vanishes at night. Describe the serving design — what precision you run the weights and the KV cache at, how you split the model across GPUs, what the batching and scheduling policy is, how you size the KV-cache budget against concurrency, and which latency lever you pull first when p95 slips. Say where the money actually goes and what you would measure to prove the design holds.
You must serve a 70B-parameter chat model to 200 concurrent users on rented GPUs. Prompts average 2000 tokens, answers 300 tokens, and the product requires p95 end-to-end latency under two seconds with the answer streamed. The budget is fixed, and traffic triples during working hours and nearly vanishes at night. Describe the serving design — what precision you run the weights and the KV cache at, how you split the model across GPUs, what the batching and scheduling policy is, how you size the KV-cache budget against concurrency, and which latency lever you pull first when p95 slips. Say where the money actually goes and what you would measure to prove the design holds.
Weights run at INT8 or FP8 and the KV cache at FP8, with tensor parallelism sized so the model and its KV budget fit with headroom. Serve with continuous batching over paged KV, cap concurrency by the KV budget, and exhaust latency levers before buying GPUs.
Common mistakes
- ✗Sizing GPUs for the weights only and forgetting the KV-cache budget
- ✗Optimising average latency when the requirement is a p95 tail
- ✗Adding replicas before fixing precision, batching and admission control
Follow-up questions
- →How would you split the two-second budget between time-to-first-token and the streaming tail?
- →What changes if traffic is bursty rather than a smooth daily curve?
SeniorDesignOccasionalYour product spends a fast-growing amount on a hosted LLM API and leadership asks whether to self-host. Traffic today is 40 million input and 8 million output tokens per month, doubling every quarter, with a sharp daytime peak and almost no night traffic. Latency requirements are moderate, and the workload is a mix of short chat turns and long document summaries. Build the cost model both ways — what you must measure on your own hardware to get a defensible dollars-per-million-tokens number, how GPU rental price, achievable tokens per second and realistic utilisation combine, and what fixed costs the API price already covers for you. Then say at what volume and under what conditions self-hosting wins, what it costs in engineering and risk, and which hybrid you would ship first.
Your product spends a fast-growing amount on a hosted LLM API and leadership asks whether to self-host. Traffic today is 40 million input and 8 million output tokens per month, doubling every quarter, with a sharp daytime peak and almost no night traffic. Latency requirements are moderate, and the workload is a mix of short chat turns and long document summaries. Build the cost model both ways — what you must measure on your own hardware to get a defensible dollars-per-million-tokens number, how GPU rental price, achievable tokens per second and realistic utilisation combine, and what fixed costs the API price already covers for you. Then say at what volume and under what conditions self-hosting wins, what it costs in engineering and risk, and which hybrid you would ship first.
Cost per million tokens is the GPU hourly price divided by tokens actually produced per hour, measured under real batching and utilisation rather than at peak. Self-hosting wins only at high, steady volume, because idle GPUs and on-call engineering are paid regardless.
Common mistakes
- ✗Using vendor peak throughput instead of measured tokens per second under real batching
- ✗Ignoring night-time idle capacity and treating GPU hours as consumption-billed
- ✗Leaving engineering, on-call and evaluation work out of the self-hosted side
Follow-up questions
- →Which hybrid would you ship first — self-hosting the bulk traffic and keeping the API for spikes?
- →How does the break-even move when the workload is long summaries rather than short chat turns?
