LLM Architecture
Transformer internals for large language models: autoregressive generation, the KV cache, MQA/GQA, RoPE, Mixture-of-Experts, tokenizer training and scaling laws.
10 questions
JuniorTheoryVery commonHow does an LLM generate text autoregressively, and how does prefill differ from decode?
How does an LLM generate text autoregressively, and how does prefill differ from decode?
A loop: one forward pass turns the sequence into a distribution over the vocabulary, a token is sampled and appended, and the loop repeats. Prefill runs the whole prompt in one parallel pass; decode emits one token per pass, so it is sequential.
Common mistakes
- ✗Thinking the whole answer is produced by one forward pass
- ✗Treating prefill and decode as equally parallel and equally cheap
- ✗Forgetting that each decode step conditions on the full sequence so far
Follow-up questions
- →Why is decode usually memory-bandwidth bound while prefill is compute bound?
- →How does the sampling strategy sit on top of this generation loop?
MiddlePerformanceVery commonWhat does the KV cache store, and how does its size scale with batch, layers and context?
What does the KV cache store, and how does its size scale with batch, layers and context?
It stores the Key and Value vectors already computed for every past token in every layer, so a new token attends without recomputing them. Its size is batch times sequence length times layers times KV heads times head dimension times two, growing linearly with context.
Common mistakes
- ✗Thinking the cache holds queries or logits rather than keys and values
- ✗Assuming the growth in context length is quadratic rather than linear
- ✗Dropping the per-layer factor and badly underestimating the total footprint
Follow-up questions
- →Which term would you attack first to fit a longer context on the same card?
- →How does paged attention memory management change this footprint in practice?
JuniorTheoryCommonWhat is a decoder-only LLM trained to predict, and what does causal masking do?
What is a decoder-only LLM trained to predict, and what does causal masking do?
Next-token prediction — the loss is cross-entropy over the vocabulary, averaged across positions. Causal masking blocks attention to future positions, so each prediction sees only the prefix and every position of a sequence trains in one pass.
Common mistakes
- ✗Confusing the causal next-token objective with BERT-style masked language modelling
- ✗Thinking causal masking is an inference-time trick rather than a training constraint
- ✗Believing only the last position of a sequence contributes to the loss
Follow-up questions
- →Why can a causal model train all positions of a sequence in a single pass?
- →What does the masked language modelling (MLM) objective buy that a causal one cannot?
MiddleTheoryCommonEncoder-only, encoder-decoder, decoder-only — what is each good at, and why did decoder-only win?
Encoder-only, encoder-decoder, decoder-only — what is each good at, and why did decoder-only win?
Encoder-only is bidirectional, trained with masked language modelling, so it suits classification and embeddings. Encoder-decoder fits a fixed input-to-output mapping like translation. Decoder-only trains causally on raw text, so one model covers all tasks.
Common mistakes
- ✗Assigning generation to the encoder-only family and classification to the encoder-decoder one
- ✗Assuming all three families share the same pretraining objective
- ✗Explaining the decoder-only win by serving cost alone, ignoring task generality
Follow-up questions
- →When would you still reach for an encoder-only model such as BERT today?
- →What does the encoder-decoder cross-attention give that a prompted decoder cannot?
MiddleTheoryCommonHow does Mixture-of-Experts routing make active parameters far smaller than total parameters?
How does Mixture-of-Experts routing make active parameters far smaller than total parameters?
A router sends each token to its top one or two experts, so only those feed-forward experts run. Active parameters and compute stay near a small dense model while total parameters grow many times. The cost — all experts stay in memory and routing needs load balancing.
Common mistakes
- ✗Assuming all experts run for every token and are merely weighted
- ✗Confusing total parameter count with the active parameters per token
- ✗Forgetting that unbalanced routing needs an auxiliary load-balancing loss
Follow-up questions
- →Why does an MoE model still need memory proportional to its total parameter count?
- →What does the auxiliary load-balancing loss actually penalize during training?
MiddleTheoryCommonHow do multi-query and grouped-query attention differ from multi-head, and what do they buy?
How do multi-query and grouped-query attention differ from multi-head, and what do they buy?
Multi-head gives every query head its own Key and Value heads. Multi-query keeps one K/V head shared by all query heads; grouped-query keeps one per group. Fewer K/V heads shrink the KV cache proportionally and speed up decode, and GQA keeps quality close to multi-head.
Common mistakes
- ✗Believing the sharing applies to the Query projections instead of Key and Value
- ✗Missing that the KV cache shrinks in proportion to the number of K/V heads
- ✗Treating GQA as a lossy approximation rather than a quality-preserving middle ground
Follow-up questions
- →How would you choose the number of key-value groups for a given serving budget?
- →Can a multi-head checkpoint be converted to grouped-query without full retraining?
MiddleTheoryCommonWalk through one transformer block, and say where most of its parameters live.
Walk through one transformer block, and say where most of its parameters live.
A block is attention then a feed-forward network, each wrapped in a residual connection with normalization. Attention mixes across tokens; the feed-forward layer acts per position. With the usual fourfold expansion it holds about two thirds of the parameters.
Common mistakes
- ✗Believing the attention projections hold most of a block's parameters
- ✗Swapping the roles — thinking the feed-forward layer is what mixes tokens
- ✗Treating the residual connection or normalization as a large parameter store
Follow-up questions
- →How does the expansion factor of the feed-forward layer change that parameter split?
- →What does the pre-normalization versus post-normalization placement change about training?
MiddleTheoryOccasionalHow does rotary positional embedding (RoPE) encode relative position between tokens?
How does rotary positional embedding (RoPE) encode relative position between tokens?
RoPE rotates query and key vectors by an angle proportional to absolute position. A dot product of two rotated vectors depends only on the angle difference, so scores see relative distance. No position table is learned, so positions past training length stay defined.
Common mistakes
- ✗Thinking RoPE is added to the embedding like a sinusoidal or learned encoding
- ✗Applying the rotation to Values or to post-softmax logits instead of queries and keys
- ✗Assuming RoPE extrapolates arbitrarily far without any scaling of its frequencies
Follow-up questions
- →What do frequency-scaling tricks such as position interpolation change about RoPE?
- →How does RoPE compare with a relative attention bias such as ALiBi?
MiddleTheoryOccasionalWhat does a larger tokenizer vocabulary buy and cost, and how would you size it for Russian plus code?
What does a larger tokenizer vocabulary buy and cost, and how would you size it for Russian plus code?
A larger vocabulary means fewer tokens per text, so sequences shorten, attention gets cheaper and more content fits. It costs bigger embedding and output matrices and gives rare tokens less signal. For Russian plus code, train byte-pair encoding on that mixture.
Common mistakes
- ✗Thinking a bigger vocabulary lengthens rather than shortens sequences
- ✗Ignoring that the embedding and output matrices scale with vocabulary size
- ✗Assuming rare tokens improve with a bigger vocabulary rather than getting less signal
Follow-up questions
- →Why does an English-trained tokenizer inflate the token count on Russian text?
- →What measurement on a held-out corpus would you use to compare two vocabulary sizes?
SeniorDesignOccasionalYou have a fixed pretraining compute budget for a new general-purpose text model and must split it between model size and training tokens. The budget is locked and cannot be extended; the data team can supply either a small carefully filtered corpus or a much larger noisier one; and whatever you train will be served to users for at least two years at high request volume, on the same hardware you trained on. Using scaling laws, decide how you would trade parameters against training tokens, justify the point on that trade-off you would pick given the serving horizon, and state explicitly which parts of this decision the scaling laws do not price in.
You have a fixed pretraining compute budget for a new general-purpose text model and must split it between model size and training tokens. The budget is locked and cannot be extended; the data team can supply either a small carefully filtered corpus or a much larger noisier one; and whatever you train will be served to users for at least two years at high request volume, on the same hardware you trained on. Using scaling laws, decide how you would trade parameters against training tokens, justify the point on that trade-off you would pick given the serving horizon, and state explicitly which parts of this decision the scaling laws do not price in.
Loss falls predictably when parameters and tokens grow together, so a compute-optimal run grows both. With a two-year serving horizon I would go smaller and train longer, since inference is paid per request. The laws price in neither data quality nor serving cost.
Common mistakes
- ✗Spending the budget on parameters alone and treating token count as a leftover
- ✗Shipping the compute-optimal point without weighing the two-year inference bill
- ✗Assuming the laws cover data quality and serving economics
Follow-up questions
- →How would the answer change if the model were for internal batch use rather than live traffic?
- →What would you measure during training to detect that the chosen split was wrong?
