---
title: Should we use RAG or fine-tuning?
source: https://rubradigital.com/answers/rag-vs-fine-tuning
site: Rubra Digital
description: RAG vs fine-tuning compared on cost, speed, accuracy and maintenance, with a clear rule for choosing and when to use both together.
updated: 2026-07-08
topic: rag
also_asked: Is RAG better than fine-tuning? | Can you use RAG and fine-tuning together? | When is fine-tuning worth the cost?
---

# Should we use RAG or fine-tuning?

**Summary:** Use RAG when the model needs access to knowledge it does not have: private documents, current information, or facts that change. Use fine-tuning when the model needs to behave differently: a specific output format, a house tone, a narrow classification task, or a domain vocabulary it handles poorly. The distinction is knowledge versus behaviour. RAG is cheaper, updates instantly and gives citations; fine-tuning changes the model itself and requires retraining to update. Most production systems that use fine-tuning also use RAG, because the two solve different problems.

## The one-line rule

**Knowledge goes in retrieval. Behaviour goes in weights.**

If your complaint is _the model does not know X_, that is RAG. If your complaint
is _the model knows X but says it wrong_, that is fine-tuning.

## Side by side

|                | RAG                               | Fine-tuning                            |
| -------------- | --------------------------------- | -------------------------------------- |
| Solves         | Missing knowledge                 | Wrong behaviour                        |
| Update cost    | Re-index a document, minutes      | Retrain, hours to days                 |
| Setup cost     | Moderate: pipeline and evaluation | High: labelled data, training, hosting |
| Citations      | Natural, passages are retrieved   | Not possible from weights alone        |
| Data freshness | Immediate                         | Frozen at training time                |
| Access control | Enforceable at retrieval time     | Not enforceable; knowledge is baked in |
| Fails by       | Retrieving the wrong passage      | Confidently generalising wrongly       |

## Why teams reach for fine-tuning too early

Fine-tuning feels like the more serious engineering answer, so it attracts teams
who want to be doing something substantial. In practice it is usually the wrong
first move:

- **Knowledge fine-tuned into weights cannot be cited**, which fails immediately
  in any regulated or high-trust context.
- **It cannot respect permissions.** If a document is in the training set, the
  model may surface it to anyone. Retrieval can filter by the user's access
  rights before the model ever sees a passage.
- **It goes stale.** Every material change to the underlying information means
  another training run.
- **It needs labelled data**, typically several hundred to a few thousand good
  examples, which most teams do not have and underestimate the cost of creating.

## When fine-tuning earns its place

- **Strict output structure** that prompting cannot hold reliably at volume.
- **A narrow, high-volume classification task** where a small fine-tuned model
  matches a large general one at a fraction of the cost and latency.
- **Domain language** the base model handles badly: specialised clinical, legal
  or industrial vocabulary.
- **Latency or cost ceilings** that a large model cannot meet, where a small
  fine-tuned model can.

## The realistic sequence

1. Prompt engineering with a strong model. Establish the quality ceiling.
2. Add RAG if the failures are knowledge failures. This resolves most cases.
3. Build evaluation. You cannot judge step four without it.
4. Fine-tune only if measured failures are behavioural, and only after you have
   a labelled set worth training on.

Teams that skip to step four spend three months and arrive back at step two.

---

Source: https://rubradigital.com/answers/rag-vs-fine-tuning
Rubra Digital. Independent LLM and RAG consulting for regulated and document-heavy organisations in Europe and North America.
Contact: hello@rubradigital.com