---
title: How do you reduce hallucinations in an LLM system?
source: https://rubradigital.com/answers/how-to-reduce-llm-hallucinations
site: Rubra Digital
description: Practical techniques for reducing LLM hallucinations in production: fixing retrieval first, constraining generation, verification passes and measured refusal.
updated: 2026-06-22
topic: evaluation
also_asked: Why does our RAG chatbot make things up? | Can hallucinations be eliminated completely? | How do you make an LLM say we do not know?
---

# How do you reduce hallucinations in an LLM system?

**Summary:** Most hallucinations in production systems are retrieval failures rather than model failures. The model was asked to answer from context that did not contain the answer. Fix retrieval first and measure recall@k, because no prompt change compensates for the right passage never arriving. Then constrain generation to cite retrieved passages and to refuse when context is insufficient, add a verification pass that checks each claim against the source for high-stakes answers, and measure groundedness on every change so regressions are caught in CI rather than by users.

## Diagnose before you treat

The first question is always: **was the correct information in the context
window?**

Log the retrieved passages alongside every answer and check a sample of
failures. Teams that do this are usually surprised. In most RAG systems I audit,
the majority of wrong answers had wrong or missing context. That is a retrieval
bug wearing a hallucination costume, and no amount of prompt tuning fixes it.

Only once retrieval is measurably good does it make sense to work on generation.

## Retrieval-side fixes

- **Hybrid search.** Pure vector search misses exact terms: product codes, error
  numbers, proper nouns. Combining it with BM25 keyword search closes a
  surprising share of failures.
- **Reranking.** A cross-encoder reranker over the top 50 candidates
  consistently improves precision at the top of the list, and it is one of the
  highest-return changes available.
- **Chunking that respects structure.** A clause split from its heading loses
  the context that made it findable.
- **Query rewriting.** Users ask questions that do not match how documents are
  written. Rewriting the query before search, or generating several variants and
  merging the results, helps materially on conversational systems.

## Generation-side fixes

- **Instruct for grounding and refusal explicitly.** The system prompt should
  require answers to come only from the provided context and require an explicit
  "the provided documents do not cover this" when they do not.
- **Require citations per claim.** Attribution pressure measurably reduces
  invention, and it makes errors visible when they happen.
- **Lower temperature for factual tasks.** Not a cure, but free.
- **Do not overfill the context.** Stuffing thirty marginal passages in makes
  the answer worse, not better, because relevant material competes with noise.

## Verification for high-stakes answers

Where an error is expensive, add a second pass: take each claim in the generated
answer and check it against the cited passage. This roughly doubles cost and
latency, so reserve it for the paths that warrant it. It catches the residual
errors that grounding alone does not.

## Make refusal a first-class outcome

A system that says "I do not know" is more useful than one that guesses, and
users calibrate their trust accordingly. Build a subset of deliberately
unanswerable questions into your evaluation set and measure the refusal rate on
them. If it is low, your system is confidently wrong on exactly the questions
where being wrong matters most.

## The honest limit

Hallucination cannot be reduced to zero with current models. It can be reduced
to a measured, acceptable rate for a defined task, with a known failure mode and
a human checkpoint where the stakes require one. Any vendor claiming elimination
is describing a demo, not a production system.

The useful goal is not perfection. It is knowing your groundedness number,
having it under regression control, and designing the surrounding process for
the residual error rate.

---

Source: https://rubradigital.com/answers/how-to-reduce-llm-hallucinations
Rubra Digital. Independent LLM and RAG consulting for regulated and document-heavy organisations in Europe and North America.
Contact: hello@rubradigital.com