How do you evaluate a RAG system?

Separate the two failure modes

When an answer is wrong, there are exactly two possibilities: the right information never reached the model, or it reached the model and the model handled it badly. These need different fixes, so they need different measurements.

In my experience, the majority of “hallucination” complaints in production RAG systems are retrieval failures. The model was asked to answer from context that did not contain the answer, and it obliged.

Retrieval metrics

Build a set of questions paired with the passage IDs that answer them. Then:

• Recall@k. Is a correct passage in the top k results? The headline number. If recall@5 is 0.7, three in ten questions are unanswerable no matter what the model does.
• Mean reciprocal rank (MRR). How high up is the first correct passage? Rank matters because passages far down the list compete for attention with irrelevant ones.
• nDCG. Accounts for multiple relevant passages at different usefulness levels. Worth adding when questions typically need several sources.

These are deterministic, cost nothing to run, and take seconds. Run them on every change.

Generation metrics

• Groundedness. Is every factual claim in the answer supported by the retrieved context? The most important generation metric by a wide margin.
• Citation accuracy. Do the citations point at passages that actually contain the claim? Systems that cite plausibly but wrongly are more dangerous than systems that do not cite at all.
• Task completion. Did it answer the question that was asked, rather than a nearby one?
• Appropriate refusal. When the context does not support an answer, does the system say so? Measure this with a deliberate subset of unanswerable questions.

Building the test set

Use real questions. Pull them from search logs, support tickets, or a week of asking the people who will use the system to write down what they actually want to know. Invented questions cluster on the easy cases.

Stratify deliberately. Simple lookups, multi-document synthesis, questions with near-miss distractors in the corpus, ambiguous questions, and unanswerable questions. Aim for roughly balanced coverage rather than natural frequency, which over-weights the easy tail.

150 to 300 curated cases beats 5,000 scraped ones. The curated set exercises the failure modes; the scraped set mostly re-confirms that easy questions work.

Get the ground truth from people who know. A subject-matter expert marking which passage is correct is the expensive, unavoidable part. Budget for it.

Calibrating an LLM judge

Using a model to grade output is practical and scales. Uncalibrated, it produces confident numbers with unknown error.

Calibrate it: have humans score a stratified sample of 80–150 outputs, have the judge score the same sample, and compute agreement. Then use the judge only for metrics where agreement is high enough to act on. Groundedness usually calibrates well. Subjective quality usually does not, and where it does not, say so rather than shipping a metric nobody should trust.

Report the agreement rate alongside the metric. It tells everyone how much weight the gate deserves.

Wire it into CI

An evaluation suite that runs when someone remembers is not a gate. Run retrieval metrics on every pull request, the fuller suite nightly, and block merges on regression beyond a threshold you agree in advance. This is what makes it safe for more than one person to change prompts.