Retrieval-Augmented Generation
RAG pipeline anatomy, chunking strategies, hybrid search, vector databases and HNSW, grounding/refusal, and separating retrieval from generation quality.
9 questions
JuniorTheoryVery commonWhy pair the lexical ranking function BM25 with dense vectors, and what does each miss?
Why pair the lexical ranking function BM25 with dense vectors, and what does each miss?
BM25 matches exact tokens, so it nails rare literals — error codes, part numbers, names — but misses paraphrase. Dense vectors match meaning, catching paraphrase but blurring rare literals. Hybrid runs both and fuses the two ranked lists.
Common mistakes
- ✗Expecting dense embeddings to retrieve exact identifiers as reliably as keyword scoring
- ✗Fusing scores from two retrievers without normalising their incomparable scales
- ✗Treating hybrid search as a latency optimisation rather than a recall one
Follow-up questions
- →How does the rank-fusion method Reciprocal Rank Fusion merge two lists without calibrating scores?
- →In multi-turn chat, why rewrite the follow-up question before either retriever sees it?
JuniorTheoryVery commonWalk the stages of a retrieval-augmented generation pipeline and what each can break.
Walk the stages of a retrieval-augmented generation pipeline and what each can break.
Ingest chunks the documents, embeds them and builds an index. A query is embedded, top-k chunks retrieved, reranked, then pasted into the prompt. Chunking splits answers, embeddings miss matches, retrieval returns wrong chunks, generation invents.
Common mistakes
- ✗Treating RAG as fine-tuning — it injects context at inference and never touches the weights
- ✗Blaming the model for a wrong answer when retrieval never put the fact in the prompt
- ✗Skipping the index build and re-embedding the whole corpus on every incoming query
Follow-up questions
- →When does fine-tuning beat RAG on a knowledge-heavy task, and when is it the reverse?
- →Where does a query-rewriting step fit in multi-turn chat, and what does it fix?
MiddleDesignCommonYou are indexing 40k pages of product documentation — deep heading hierarchies, long API reference tables, and short release notes. Retrieval is dense plus keyword, the generator gets roughly 4k tokens of context, and answers frequently need a definition plus the constraint stated one paragraph below it. Choose between fixed-size, recursive, semantic and parent-document chunking for this corpus, justify the chunk size and the overlap you would set, and state concretely what a too-small chunk costs you at answer time and what a too-large chunk costs you at retrieval time.
You are indexing 40k pages of product documentation — deep heading hierarchies, long API reference tables, and short release notes. Retrieval is dense plus keyword, the generator gets roughly 4k tokens of context, and answers frequently need a definition plus the constraint stated one paragraph below it. Choose between fixed-size, recursive, semantic and parent-document chunking for this corpus, justify the chunk size and the overlap you would set, and state concretely what a too-small chunk costs you at answer time and what a too-large chunk costs you at retrieval time.
Split on structure — headings, then paragraphs — with a fixed window as fallback. Aim for one self-contained idea per chunk, a few hundred tokens, with light overlap so no boundary halves an answer. Too small loses context; too large dilutes embeddings.
Common mistakes
- ✗Picking one chunk size for the whole corpus regardless of heading depth or table structure
- ✗Setting overlap to zero and losing every answer that straddles a chunk boundary
- ✗Assuming a bigger chunk always improves recall, ignoring how it dilutes the embedding
Follow-up questions
- →How does parent-document retrieval let you embed small but hand the generator large context?
- →What signal tells you the chunk size is wrong rather than the embedding model being weak?
MiddleDesignCommonYour hybrid retriever returns 50 candidates in about 30 ms, and the generator streams its first token roughly 900 ms later; the product's end-to-end budget is 2 seconds. Offline measurement shows the gold chunk lands inside the top 50 about 94% of the time but inside the top 5 — the slice that actually reaches the prompt — only 61% of the time. Decide whether to insert a cross-encoder reranker between retrieval and generation: explain why it can order candidates better than the vector scores already available, and account for the latency it adds against the recall it buys.
Your hybrid retriever returns 50 candidates in about 30 ms, and the generator streams its first token roughly 900 ms later; the product's end-to-end budget is 2 seconds. Offline measurement shows the gold chunk lands inside the top 50 about 94% of the time but inside the top 5 — the slice that actually reaches the prompt — only 61% of the time. Decide whether to insert a cross-encoder reranker between retrieval and generation: explain why it can order candidates better than the vector scores already available, and account for the latency it adds against the recall it buys.
A bi-encoder embeds query and chunk apart, so the corpus is precomputed but the pair is not read together. A cross-encoder reads the pair jointly and ranks better, at one model pass per candidate. Retrieve wide and cheap, then rerank only the top tens.
Common mistakes
- ✗Assuming cross-encoder scores can be precomputed the way bi-encoder embeddings are
- ✗Reranking hundreds of candidates and spending the whole latency budget before generation
- ✗Reading a high top-50 recall as proof that the prompt already contains the answer
Follow-up questions
- →How would you pick the rerank depth from a measured recall-versus-latency curve?
- →When does a smaller distilled reranker beat dropping the rerank stage altogether?
SeniorDebuggingCommonThe bot answers confidently and wrongly about a fact that is in the corpus — how do you localise it?
The bot answers confidently and wrongly about a fact that is in the corpus — how do you localise it?
Walk the stages in order and ask where the gold chunk drops. Confirm it was ingested and embedded, then its rank in retrieval, then after reranking, then whether it survived into the prompt. Only if it reached the prompt is generation at fault.
Open full question →Common mistakes
- ✗Calling every confident wrong answer a hallucination without checking what the prompt held
- ✗Treating presence in the corpus as proof the chunk reached the generator's context
- ✗Skipping the rerank stage when bisecting, so a reordering regression stays invisible
Follow-up questions
- →How would you log each stage so this bisection takes minutes instead of a day?
- →What would the same trace look like if the embedding model were genuinely at fault?
JuniorDesignOccasionalYou are specifying a customer-support bot that answers strictly from a versioned help-centre corpus. Compliance requires every claim to be traceable to a source, and the product owner would rather ship a refusal than a confident guess. Some incoming questions concern features the corpus does not document at all; others are documented only in an article that was superseded two releases ago. Define the answer contract — when the bot must refuse instead of answering, what a citation has to point at, and how you would test in CI before a release that grounding actually holds rather than merely looking plausible.
You are specifying a customer-support bot that answers strictly from a versioned help-centre corpus. Compliance requires every claim to be traceable to a source, and the product owner would rather ship a refusal than a confident guess. Some incoming questions concern features the corpus does not document at all; others are documented only in an article that was superseded two releases ago. Define the answer contract — when the bot must refuse instead of answering, what a citation has to point at, and how you would test in CI before a release that grounding actually holds rather than merely looking plausible.
Answer only from retrieved chunks. If nothing clears the score floor, refuse and name what was missing. Every claim cites the chunk id and document version. CI runs a labelled set of answerable and unanswerable questions and fails on uncited claims.
Common mistakes
- ✗Treating a refusal as a product failure, so the bot guesses whenever retrieval is thin
- ✗Citing a live document URL instead of the exact chunk and version that was actually read
- ✗Testing grounding only on answerable questions, never on ones the corpus cannot answer
Follow-up questions
- →How do you keep a refusal useful — what should it tell the user besides declining?
- →How does document versioning change what a citation must pin down over time?
MiddlePerformanceOccasionalIn the approximate-nearest-neighbour graph index HNSW, what do M and ef_search trade off?
In the approximate-nearest-neighbour graph index HNSW, what do M and ef_search trade off?
HNSW is an approximate graph index. M, the neighbours per node, is fixed at build time and drives memory and graph quality. ef_search, the query-time candidate list, buys recall for latency. Raise ef_search until measured recall@k hits target.
Common mistakes
- ✗Believing an approximate index returns the true nearest neighbours at any setting
- ✗Tuning
ef_searchby feel instead of against a measured recall@k on a labelled set - ✗Forgetting that
Mis baked in at build time and needs a full reindex to change
Follow-up questions
- →How do you build the ground-truth set that makes a recall@k measurement meaningful?
- →When does a quantised index beat lowering
ef_searchto hit the same latency target?
SeniorDesignOccasionalA RAG assistant that shipped three months ago is drawing complaints — answers read fluently but users report them as wrong or unsupported, and the only metric on the dashboard is a thumbs-up rate that has drifted from 78% to 61%. You have the query logs, the corpus, and budget for a small labelled set. Design the evaluation: which retrieval metrics you would compute, which generation metrics you would compute, how each one is labelled, and say which half you would fix first when both look degraded — and why that order is not arbitrary.
A RAG assistant that shipped three months ago is drawing complaints — answers read fluently but users report them as wrong or unsupported, and the only metric on the dashboard is a thumbs-up rate that has drifted from 78% to 61%. You have the query logs, the corpus, and budget for a small labelled set. Design the evaluation: which retrieval metrics you would compute, which generation metrics you would compute, how each one is labelled, and say which half you would fix first when both look degraded — and why that order is not arbitrary.
Score the halves apart. Retrieval gets recall@k and MRR on labelled chunks — did the fact reach the prompt. Generation gets faithfulness to those chunks and answer relevance. Fix retrieval first: grading generation over context missing the fact is meaningless.
Common mistakes
- ✗Reporting a single end-to-end score that cannot say which half of the pipeline regressed
- ✗Tuning prompts and the generator while the gold chunk never enters the context at all
- ✗Reading retrieval similarity scores as a quality metric without any relevance labels
Follow-up questions
- →How would you build the labelled relevance set cheaply from the existing query logs?
- →What does a high faithfulness score with low answer relevance tell you about the system?
SeniorDesignRareDesign RAG over a 10M-document corpus shared by 400 corporate tenants, where each user may read only the documents their role grants and some documents are re-issued daily. Answers must stay under a two-second budget, and a single leak of one tenant's text into another tenant's answer is a contract-ending incident. Cover how you shard and index the vectors, where access control is enforced relative to retrieval, how freshness reaches the index without a full rebuild, and how you would prove to an auditor that cross-tenant leakage is structurally impossible rather than merely unobserved.
Design RAG over a 10M-document corpus shared by 400 corporate tenants, where each user may read only the documents their role grants and some documents are re-issued daily. Answers must stay under a two-second budget, and a single leak of one tenant's text into another tenant's answer is a contract-ending incident. Cover how you shard and index the vectors, where access control is enforced relative to retrieval, how freshness reaches the index without a full rebuild, and how you would prove to an auditor that cross-tenant leakage is structurally impossible rather than merely unobserved.
Enforce access at retrieval, never in the prompt. Put an ACL on every chunk and pre-filter inside the index, or shard per tenant so a query cannot reach foreign vectors. Post-filtering after top-k loads forbidden text into memory and silently drops recall.
Common mistakes
- ✗Enforcing tenant isolation with prompt instructions instead of at the retrieval boundary
- ✗Post-filtering the top-k, which silently shrinks recall and still reads forbidden text
- ✗Rebuilding the whole index for freshness instead of upserting and tombstoning by document
Follow-up questions
- →How does per-tenant sharding change your recall and latency budget versus one filtered index?
- →What would you show an auditor as evidence of isolation beyond a passing test suite?
