---
title: How do you evaluate a RAG system?
source: https://rubradigital.com/answers/how-to-evaluate-a-rag-system
site: Rubra Digital
description: A practical method for evaluating RAG: which retrieval and generation metrics matter, how large a test set needs to be, and how to calibrate an LLM judge.
updated: 2026-07-02
topic: evaluation
also_asked: What metrics should we use for RAG? | How many test cases do we need for LLM evaluation? | Is LLM-as-a-judge reliable?
---

# How do you evaluate a RAG system?

**Summary:** Evaluate retrieval and generation separately, because they fail for different reasons. For retrieval, build a set of real questions paired with the passages that actually answer them, then measure recall@k, mean reciprocal rank and nDCG. This catches most of what teams misdiagnose as hallucination. For generation, measure groundedness (is every claim supported by the retrieved context), citation accuracy, and task completion. A stratified set of 150 to 300 curated cases detects meaningful regressions reliably; run it in CI and block merges on regression.

## Separate the two failure modes

When an answer is wrong, there are exactly two possibilities: the right
information never reached the model, or it reached the model and the model
handled it badly. These need different fixes, so they need different
measurements.

In my experience, the majority of "hallucination" complaints in production RAG
systems are retrieval failures. The model was asked to answer from context that
did not contain the answer, and it obliged.

## Retrieval metrics

Build a set of questions paired with the passage IDs that answer them. Then:

- **Recall@k.** Is a correct passage in the top _k_ results? The headline
  number. If recall@5 is 0.7, three in ten questions are unanswerable no matter
  what the model does.
- **Mean reciprocal rank (MRR).** How high up is the first correct passage? Rank
  matters because passages far down the list compete for attention with
  irrelevant ones.
- **nDCG.** Accounts for multiple relevant passages at different usefulness
  levels. Worth adding when questions typically need several sources.

These are deterministic, cost nothing to run, and take seconds. Run them on
every change.

## Generation metrics

- **Groundedness.** Is every factual claim in the answer supported by the
  retrieved context? The most important generation metric by a wide margin.
- **Citation accuracy.** Do the citations point at passages that actually
  contain the claim? Systems that cite plausibly but wrongly are more dangerous
  than systems that do not cite at all.
- **Task completion.** Did it answer the question that was asked, rather than a
  nearby one?
- **Appropriate refusal.** When the context does not support an answer, does the
  system say so? Measure this with a deliberate subset of unanswerable
  questions.

## Building the test set

**Use real questions.** Pull them from search logs, support tickets, or a week
of asking the people who will use the system to write down what they actually
want to know. Invented questions cluster on the easy cases.

**Stratify deliberately.** Simple lookups, multi-document synthesis, questions
with near-miss distractors in the corpus, ambiguous questions, and unanswerable
questions. Aim for roughly balanced coverage rather than natural frequency,
which over-weights the easy tail.

**150 to 300 curated cases beats 5,000 scraped ones.** The curated set exercises
the failure modes; the scraped set mostly re-confirms that easy questions work.

**Get the ground truth from people who know.** A subject-matter expert marking
which passage is correct is the expensive, unavoidable part. Budget for it.

## Calibrating an LLM judge

Using a model to grade output is practical and scales. Uncalibrated, it produces
confident numbers with unknown error.

Calibrate it: have humans score a stratified sample of 80–150 outputs, have the
judge score the same sample, and compute agreement. Then use the judge only for
metrics where agreement is high enough to act on. Groundedness usually
calibrates well. Subjective quality usually does not, and where it does not, say
so rather than shipping a metric nobody should trust.

Report the agreement rate alongside the metric. It tells everyone how much
weight the gate deserves.

## Wire it into CI

An evaluation suite that runs when someone remembers is not a gate. Run
retrieval metrics on every pull request, the fuller suite nightly, and block
merges on regression beyond a threshold you agree in advance. This is what makes
it safe for more than one person to change prompts.

---

Source: https://rubradigital.com/answers/how-to-evaluate-a-rag-system
Rubra Digital. Independent LLM and RAG consulting for regulated and document-heavy organisations in Europe and North America.
Contact: hello@rubradigital.com