The bot answers confidently and wrongly about a fact that is in the corpus — how do you localise it?
A support assistant answers "the retention window is 30 days" while the documented value is 90. The sentence stating 90 days exists in the corpus as chunk doc-4471#c12.
Constraints: you may add instrumentation and replay the query, but you cannot retrain the generator and you cannot re-chunk the corpus during the investigation.
query : "how long is data retained?"
gold chunk : doc-4471#c12 indexed=yes embedded=yes
dense top-50 : rank 7 bm25 top-50 : rank 3
after rerank : rank 22
prompt (top-5): doc-1180#c03, doc-4471#c02, doc-9002#c41, doc-1180#c07, doc-3355#c19Name the ordered checks that localise the fault, and say which line already settles it.
Walk the stages in order and ask where the gold chunk drops. Confirm it was ingested and embedded, then its rank in retrieval, then after reranking, then whether it survived into the prompt. Only if it reached the prompt is generation at fault.
- ✗Calling every confident wrong answer a hallucination without checking what the prompt held
- ✗Treating presence in the corpus as proof the chunk reached the generator's context
- ✗Skipping the rerank stage when bisecting, so a reordering regression stays invisible
- →How would you log each stage so this bisection takes minutes instead of a day?
- →What would the same trace look like if the embedding model were genuinely at fault?
The line after rerank : rank 22 already settles it. The chunk is indexed, embedded and found well by both retrievers (dense 7, bm25 3), but the reranker pushes it down to rank 22, so only the top five reach the prompt and the gold chunk is not among them. The generator answered confidently from what it was handed; nothing was invented.
Check in input-to-output order, never backwards:
- Is the chunk in the index and embedded at all (
indexed=yes embedded=yes). - Its rank in each retriever separately — dense and BM25.
- Its rank after reranking, compared against the rank before.
- The exact prompt text: which chunks actually made it in.
- Only then the generation step itself.
stage rank in_prompt
dense 7 -
bm25 3 -
after rerank 22 no <-- fault localised hereThe fix then targets the reranker: check whether it was trained on a different domain, whether the chunk text is truncated on input, and whether short definitions are penalised by length. Prompt edits and a lower temperature cannot help — the fact is simply not in the context.
