Retrieval & RAG
Should we use RAG or fine-tuning?
Short answer
Use RAG when the model needs access to knowledge it does not have: private documents, current information, or facts that change. Use fine-tuning when the model needs to behave differently: a specific output format, a house tone, a narrow classification task, or a domain vocabulary it handles poorly. The distinction is knowledge versus behaviour. RAG is cheaper, updates instantly and gives citations; fine-tuning changes the model itself and requires retraining to update. Most production systems that use fine-tuning also use RAG, because the two solve different problems.
Last reviewed
The one-line rule
Knowledge goes in retrieval. Behaviour goes in weights.
If your complaint is the model does not know X, that is RAG. If your complaint is the model knows X but says it wrong, that is fine-tuning.
Side by side
| RAG | Fine-tuning | |
|---|---|---|
| Solves | Missing knowledge | Wrong behaviour |
| Update cost | Re-index a document, minutes | Retrain, hours to days |
| Setup cost | Moderate: pipeline and evaluation | High: labelled data, training, hosting |
| Citations | Natural, passages are retrieved | Not possible from weights alone |
| Data freshness | Immediate | Frozen at training time |
| Access control | Enforceable at retrieval time | Not enforceable; knowledge is baked in |
| Fails by | Retrieving the wrong passage | Confidently generalising wrongly |
Why teams reach for fine-tuning too early
Fine-tuning feels like the more serious engineering answer, so it attracts teams who want to be doing something substantial. In practice it is usually the wrong first move:
- Knowledge fine-tuned into weights cannot be cited, which fails immediately in any regulated or high-trust context.
- It cannot respect permissions. If a document is in the training set, the model may surface it to anyone. Retrieval can filter by the user’s access rights before the model ever sees a passage.
- It goes stale. Every material change to the underlying information means another training run.
- It needs labelled data, typically several hundred to a few thousand good examples, which most teams do not have and underestimate the cost of creating.
When fine-tuning earns its place
- Strict output structure that prompting cannot hold reliably at volume.
- A narrow, high-volume classification task where a small fine-tuned model matches a large general one at a fraction of the cost and latency.
- Domain language the base model handles badly: specialised clinical, legal or industrial vocabulary.
- Latency or cost ceilings that a large model cannot meet, where a small fine-tuned model can.
The realistic sequence
- Prompt engineering with a strong model. Establish the quality ceiling.
- Add RAG if the failures are knowledge failures. This resolves most cases.
- Build evaluation. You cannot judge step four without it.
- Fine-tune only if measured failures are behavioural, and only after you have a labelled set worth training on.
Teams that skip to step four spend three months and arrive back at step two.
People also ask this as
- Is RAG better than fine-tuning?
- Can you use RAG and fine-tuning together?
- When is fine-tuning worth the cost?