---
title: Most RAG failures are retrieval failures
source: https://rubradigital.com/insights/most-rag-failures-are-retrieval-failures
site: Rubra Digital
description: When teams report hallucination in a RAG system, the model is usually not the problem. Here is how to tell the difference and what to do about it.
published: 2026-07-02
updated: 2026-07-02
author: Your Name
topic: rag
---

# Most RAG failures are retrieval failures

**Summary:** In the RAG systems I audit, most wrong answers are cases where the correct passage never reached the model. That is a retrieval bug, not a hallucination, and prompt engineering cannot fix it. This piece explains how to diagnose the difference and what to change.

Almost every RAG audit I run starts the same way. The team reports that the
model hallucinates. They have iterated on the system prompt for weeks. They are
considering fine-tuning.

I ask to see the retrieved context for twenty failed answers.

In most cases, the correct information was not there.

## This is not hallucination

A language model given context that does not contain the answer, and asked to
answer from context, is in an impossible position. It will usually produce
something plausible from whatever it was given. That is not the model
malfunctioning. It is the model doing exactly what a bad retrieval step set it
up to do.

The distinction matters because the fixes are entirely different. Retrieval
problems are fixed with chunking, hybrid search, reranking and query rewriting.
None of those live in the prompt.

## The diagnostic

Log retrieved passages with every response. Then sample failures and answer one
question for each: **was the correct information in the context window?**

Sort the failures into two piles. The proportions tell you where to spend the
next month. In my experience the retrieval pile is consistently larger, and
frequently much larger.

## Why teams look at the prompt first

The prompt is visible, editable and gives immediate feedback. Retrieval is
opaque. You cannot see what the index returned without instrumenting it, and
most teams have not.

There is also a measurement gap. Prompt changes get judged by trying a few
questions and forming an impression. Retrieval quality needs a labelled set to
measure at all. Teams optimise what they can see.

## What actually moves retrieval quality

**Hybrid search.** Pure vector search reliably misses exact strings: part
numbers, error codes, proper nouns, statutory references. Adding BM25 closes a
category of failure that no amount of embedding tuning addresses.

**Reranking.** A cross-encoder over the top 50 candidates is, in my benchmarks,
the single highest-return change available to most systems. It routinely moves
recall@5 by five to fifteen points.

**Chunk context headers.** Prepending the document title and heading path to
every chunk before embedding costs nothing and consistently improves retrieval
on corpora where passages contain ambiguous references.

**Query rewriting.** Users do not phrase questions the way documents phrase
answers. Rewriting the query, or generating several variants and merging
results, helps materially on conversational interfaces.

## Measure it, then argue about it

None of the above is worth doing on faith. Build a labelled set of 150 to 300
real questions with known-correct passages, then measure recall@5 before and
after each change.

Teams that do this stop arguing about which model is better and start making
decisions with numbers attached. That shift is usually worth more than any
individual technique on the list.

## Frequently asked questions

### How do we know if our problem is retrieval or generation?

Log the retrieved passages alongside every answer, then take twenty failures and read the context that was supplied. If the correct information was not in the context, it is a retrieval problem and no prompt change will fix it. If the information was there and the answer contradicted it, it is a generation problem. This takes an afternoon and it redirects most teams away from work that would not have helped.


---

Source: https://rubradigital.com/insights/most-rag-failures-are-retrieval-failures
Rubra Digital. Independent LLM and RAG consulting for regulated and document-heavy organisations in Europe and North America.
Contact: hello@rubradigital.com