LLM Training & Adaptation
Fine-tuning vs RAG vs prompting, LoRA/QLoRA, catastrophic forgetting, SFT/RLHF/DPO alignment, instruction tuning and prompting strategies.
11 questions
MiddleDesignVery commonYour team ships an internal assistant and three needs land in the same sprint. First, it must answer from a policy wiki that the compliance team edits every week. Second, every reply must come back as a strict object the downstream service parses, with no prose around it. Third, it must write in the company house style, which reviewers recognise on sight but cannot fully write down. Constraints: the budget allows at most one training run per quarter, the wiki owners will not wait a quarter for an edit to take effect, and answers must cite the wiki page they came from. Choose retrieval-augmented generation, fine-tuning or prompt engineering for each of the three needs, and justify each choice against these constraints.
Your team ships an internal assistant and three needs land in the same sprint. First, it must answer from a policy wiki that the compliance team edits every week. Second, every reply must come back as a strict object the downstream service parses, with no prose around it. Third, it must write in the company house style, which reviewers recognise on sight but cannot fully write down. Constraints: the budget allows at most one training run per quarter, the wiki owners will not wait a quarter for an edit to take effect, and answers must cite the wiki page they came from. Choose retrieval-augmented generation, fine-tuning or prompt engineering for each of the three needs, and justify each choice against these constraints.
Weekly-changing facts go to RAG — retrieval injects context at inference and cites its source. The strict output shape is prompt engineering with constrained decoding. House style is the fine-tuning case — a behaviour taught by examples, not a fact.
Common mistakes
- ✗Reaching for fine-tuning to add facts, when it neither refreshes nor cites them
- ✗Treating RAG as a way to teach style or output discipline instead of supplying context
- ✗Assuming a long context is equivalent to a weight update for stable behaviour
Follow-up questions
- →How would you evaluate the three tracks separately rather than as one system?
- →What changes if the wiki grows past what the context window can hold?
JuniorTheoryCommonWhat is instruction tuning, and how does its dataset differ from pretraining data?
What is instruction tuning, and how does its dataset differ from pretraining data?
Instruction tuning is supervised fine-tuning on instruction-response pairs, so the model follows a request rather than just continuing text. Its data is curated and labelled — thousands of examples, against trillions of raw pretraining tokens.
Common mistakes
- ✗Calling instruction tuning another pretraining pass rather than supervised fine-tuning
- ✗Assuming the SFT set must be huge, when curation beats raw volume at this stage
- ✗Confusing a prompt template applied at inference with an actual weight update
Follow-up questions
- →How does the quality of the written responses bound the tuned model's ceiling?
- →Why does reinforcement learning from human feedback (
RLHF) come after instruction tuning?
MiddleTheoryCommonHow does catastrophic forgetting show up after a full fine-tune, and what mitigates it?
How does catastrophic forgetting show up after a full fine-tune, and what mitigates it?
The update overwrites weights carrying general ability, so the model gains on the new task while losing instruction-following or reasoning. LoRA freezes the base and confines the change to a small adapter; replay data and a lower learning rate shrink each step.
Common mistakes
- ✗Judging a fine-tune only on the target task and never re-testing general ability
- ✗Thinking forgetting hits facts alone, when instruction-following degrades too
- ✗Raising the learning rate or epoch count to force the new behaviour in faster
Follow-up questions
- →Which held-out suites would you re-run to detect forgetting before release?
- →How much replay data from the original mix is usually enough in practice?
MiddleTheoryCommonIn adapter method LoRA, what is trained, what stays frozen, and what does the rank control?
In adapter method LoRA, what is trained, what stays frozen, and what does the rank control?
Base weights stay frozen; LoRA trains a low-rank product B·A beside chosen projections, typically attention. Rank r sets adapter capacity. Gradients and optimizer state exist only for the adapter, and B·A merges into the base afterwards.
Common mistakes
- ✗Thinking LoRA updates the base weights rather than a separate low-rank pair
- ✗Attributing the memory saving to parameter count alone, not optimizer state and gradients
- ✗Believing the adapter must stay separate, missing that B·A can be merged into the base
Follow-up questions
- →How do you pick the rank r, and what breaks when it is far too small?
- →Why does merging several adapters into one base weight interfere between tasks?
MiddleTheoryCommonIn reinforcement learning from human feedback (RLHF), where do preference pairs come from and what is reward hacking?
In reinforcement learning from human feedback (RLHF), where do preference pairs come from and what is reward hacking?
Annotators rank sampled responses to one prompt, producing preference pairs. A reward model trains on them to score a response as humans would, and the policy is optimized against that score. Reward hacking is the policy exploiting flaws in that proxy.
Common mistakes
- ✗Thinking the reward model predicts tokens rather than a human preference score
- ✗Assuming preferences are absolute ratings instead of ranked comparisons on one prompt
- ✗Reading a rising reward curve as proof that real answer quality improved
Follow-up questions
- →Why is a KL penalty against the reference policy used during optimization?
- →How would you detect reward hacking that the reward curve alone would hide?
SeniorDesignCommonYou fine-tuned an assistant on support conversations and the team wants to ship it. The only evidence so far is that training loss fell and that three engineers liked the sample replies. You have production traffic, a labelling budget of roughly 2000 human judgements, and a base model still serving live. Design the evaluation that would actually justify the release: what you hold out, what you compare against, how you use a stronger model as an automatic judge, and how you run an online experiment. For each method name the failure mode that could make it declare a win that is not real.
You fine-tuned an assistant on support conversations and the team wants to ship it. The only evidence so far is that training loss fell and that three engineers liked the sample replies. You have production traffic, a labelling budget of roughly 2000 human judgements, and a base model still serving live. Design the evaluation that would actually justify the release: what you hold out, what you compare against, how you use a stronger model as an automatic judge, and how you run an online experiment. For each method name the failure mode that could make it declare a win that is not real.
Freeze a held-out set the tune never saw and compare against the base model, not against itself. Combine human preference judgements, a stronger model as a judge for scale, and an online A/B. Judges carry position bias, held-out sets drift from production, and A/B needs traffic.
Common mistakes
- ✗Treating a falling training loss as evidence that answer quality improved
- ✗Evaluating on prompts that overlap the training set, which guarantees a win
- ✗Trusting an automatic judge without checking it against human labels or its biases
Follow-up questions
- →How would you calibrate the judge model against your human labels before trusting it?
- →Which guardrail metrics would stop the online experiment even if the primary metric rose?
JuniorTheoryOccasionalWhen do few-shot examples and chain-of-thought prompting help, and what does CoT cost?
When do few-shot examples and chain-of-thought prompting help, and what does CoT cost?
Zero-shot suits simple tasks. Few-shot examples pin down format and edge cases, helping when output shape is unusual. Chain-of-thought asks for intermediate steps and aids multi-step reasoning, but extra tokens cost latency and money.
Common mistakes
- ✗Thinking few-shot examples teach new knowledge instead of demonstrating format
- ✗Applying chain-of-thought to trivial tasks where it only adds tokens and latency
- ✗Ignoring that generated reasoning tokens are billed and slow the response down
Follow-up questions
- →How does example ordering inside a few-shot prompt change the answer?
- →When is a smaller model with chain-of-thought worse than a larger one without it?
JuniorTheoryOccasionalWhat belongs in a system prompt, and why can an instruction work better after a long context?
What belongs in a system prompt, and why can an instruction work better after a long context?
A system prompt holds stable per-application rules — role, tone, format, refusal policy — not per-request data. Attention is most reliable at the start and end of the context, so an instruction before a long document is lost; repeating it after helps.
Common mistakes
- ✗Stuffing per-request data into the system prompt instead of the user turn
- ✗Assuming attention is uniform, so instruction placement cannot matter
- ✗Forgetting that system prompt tokens are part of the context and are billed
Follow-up questions
- →How would you measure the lost-in-the-middle effect on your own prompts?
- →When does moving rules from the system prompt into a tool schema work better?
MiddleDebuggingOccasionalA fine-tune on 500 examples now parrots the training set and lost general ability — diagnose and fix it.
A fine-tune on 500 examples now parrots the training set and lost general ability — diagnose and fix it.
It is overfitting plus forgetting on a tiny set — train loss near zero while held-out loss climbs after the first epoch. Cut to one or two epochs with early stopping, lower the learning rate, switch to LoRA, and diversify the data with replay examples.
Open full question →Common mistakes
- ✗Reading a near-zero training loss as success instead of memorisation
- ✗Watching only the target metric while the general benchmark quietly collapses
- ✗Adding epochs or learning rate to a 500-example set instead of shrinking the update
Follow-up questions
- →How would you size a held-out split when only 500 labelled examples exist?
- →Why does a modest LoRA rank often beat a full fine-tune on data this small?
MiddleTheoryOccasionalIn tuning method QLoRA, what is quantized to 4-bit, what receives gradients, and why does it fit one GPU?
In tuning method QLoRA, what is quantized to 4-bit, what receives gradients, and why does it fit one GPU?
The frozen base weights are stored in 4-bit NF4 and dequantized block-wise in the forward pass. Gradients reach only the LoRA adapters, kept at higher precision. A frozen base means no optimizer state for billions of parameters, and that fits one GPU.
Common mistakes
- ✗Believing the adapters rather than the base weights are the quantized part
- ✗Assuming 4-bit storage means 4-bit arithmetic, missing block-wise dequantization
- ✗Crediting quantization alone, when the frozen base removes optimizer state too
Follow-up questions
- →What quality regression would you measure before shipping a 4-bit-base tune?
- →Why is merging an adapter back into a 4-bit base weight not straightforward?
SeniorDesignOccasionalYou own alignment for an assistant that already has a strong instruction-tuned checkpoint. You have 40k human preference comparisons, a small annotation team that can keep producing them, one training cluster shared with other teams, and a release every six weeks. Leadership wants the assistant to refuse a defined set of requests reliably and to stop rambling. Lay out the alignment pipeline end to end. Then compare the classic route — a separate reward model plus a policy-gradient stage such as PPO — against direct preference optimization (DPO) on the same comparisons: state exactly what DPO removes from the pipeline, what you give up by removing it, and which route you would run under these constraints.
You own alignment for an assistant that already has a strong instruction-tuned checkpoint. You have 40k human preference comparisons, a small annotation team that can keep producing them, one training cluster shared with other teams, and a release every six weeks. Leadership wants the assistant to refuse a defined set of requests reliably and to stop rambling. Lay out the alignment pipeline end to end. Then compare the classic route — a separate reward model plus a policy-gradient stage such as PPO — against direct preference optimization (DPO) on the same comparisons: state exactly what DPO removes from the pipeline, what you give up by removing it, and which route you would run under these constraints.
SFT first on demonstrations. RLHF then fits a separate reward model on preference pairs and optimizes the policy with PPO against it. DPO removes the reward model and the RL loop, training directly on the same pairs — cheaper, but with no live reward.
Common mistakes
- ✗Believing DPO drops the preference data rather than the reward model and RL loop
- ✗Ordering the stages wrongly, putting SFT after the preference-optimization stage
- ✗Treating DPO as strictly better, ignoring the loss of an auditable reward model
Follow-up questions
- →How does the KL penalty against the reference policy change what each route can drift to?
- →Which failure would push you back to a reward model after shipping the direct route?
