Retrieval & RAG
Most RAG failures are retrieval failures
Short answer
In the RAG systems I audit, most wrong answers are cases where the correct passage never reached the model. That is a retrieval bug, not a hallucination, and prompt engineering cannot fix it. This piece explains how to diagnose the difference and what to change.
Almost every RAG audit I run starts the same way. The team reports that the model hallucinates. They have iterated on the system prompt for weeks. They are considering fine-tuning.
I ask to see the retrieved context for twenty failed answers.
In most cases, the correct information was not there.
This is not hallucination
A language model given context that does not contain the answer, and asked to answer from context, is in an impossible position. It will usually produce something plausible from whatever it was given. That is not the model malfunctioning. It is the model doing exactly what a bad retrieval step set it up to do.
The distinction matters because the fixes are entirely different. Retrieval problems are fixed with chunking, hybrid search, reranking and query rewriting. None of those live in the prompt.
The diagnostic
Log retrieved passages with every response. Then sample failures and answer one question for each: was the correct information in the context window?
Sort the failures into two piles. The proportions tell you where to spend the next month. In my experience the retrieval pile is consistently larger, and frequently much larger.
Why teams look at the prompt first
The prompt is visible, editable and gives immediate feedback. Retrieval is opaque. You cannot see what the index returned without instrumenting it, and most teams have not.
There is also a measurement gap. Prompt changes get judged by trying a few questions and forming an impression. Retrieval quality needs a labelled set to measure at all. Teams optimise what they can see.
What actually moves retrieval quality
Hybrid search. Pure vector search reliably misses exact strings: part numbers, error codes, proper nouns, statutory references. Adding BM25 closes a category of failure that no amount of embedding tuning addresses.
Reranking. A cross-encoder over the top 50 candidates is, in my benchmarks, the single highest-return change available to most systems. It routinely moves recall@5 by five to fifteen points.
Chunk context headers. Prepending the document title and heading path to every chunk before embedding costs nothing and consistently improves retrieval on corpora where passages contain ambiguous references.
Query rewriting. Users do not phrase questions the way documents phrase answers. Rewriting the query, or generating several variants and merging results, helps materially on conversational interfaces.
Measure it, then argue about it
None of the above is worth doing on faith. Build a labelled set of 150 to 300 real questions with known-correct passages, then measure recall@5 before and after each change.
Teams that do this stop arguing about which model is better and start making decisions with numbers attached. That shift is usually worth more than any individual technique on the list.
Frequently asked questions
How do we know if our problem is retrieval or generation?
Log the retrieved passages alongside every answer, then take twenty failures and read the context that was supplied. If the correct information was not in the context, it is a retrieval problem and no prompt change will fix it. If the information was there and the answer contradicted it, it is a generation problem. This takes an afternoon and it redirects most teams away from work that would not have helped.