Retrieval & RAG
What is retrieval-augmented generation (RAG)?
Short answer
Retrieval-augmented generation (RAG) is a technique where a language model answers a question using documents fetched from your own data at the moment the question is asked, rather than relying only on what it learned during training. A retrieval step searches your corpus for relevant passages, those passages are placed into the model prompt as context, and the model generates an answer grounded in them. RAG lets a model use private, current or proprietary information it was never trained on, and makes answers citable back to a source document.
Last reviewed
The pipeline, step by step
A RAG system has two phases: one that runs ahead of time, and one that runs per question.
Indexing (ahead of time)
- Ingest. Documents are collected from their sources: file shares, a CMS, a ticketing system, a database.
- Parse. Each document is converted to text. This is where most quality is won or lost, particularly with PDFs, tables and scanned pages.
- Chunk. Text is split into passages small enough to be useful as context but large enough to remain self-contained.
- Embed. Each chunk is converted into a vector that represents its meaning.
- Index. Vectors and their source text are stored in a searchable index.
Retrieval and generation (per question)
- Embed the question using the same model used for the chunks.
- Search the index for the most similar chunks, usually combining vector similarity with keyword search.
- Rerank the candidates with a model that scores relevance more precisely than the initial search can.
- Assemble the prompt. The question plus the top passages, with instructions to answer only from them.
- Generate the answer, with citations pointing back to the source passages.
Why it exists
A language model’s knowledge is fixed at training time and contains nothing private to your organisation. Three problems follow: it does not know your internal information, it does not know what changed last week, and it cannot show you where an answer came from.
RAG addresses all three by moving the knowledge out of the model’s weights and into a retrievable store. Update the store and the system’s knowledge updates immediately, with no retraining.
Where RAG works well
Questions whose answers already exist in writing but are expensive to find: policy and regulatory libraries, technical manuals, contract portfolios, support histories, research literature.
Where RAG struggles
- Aggregation. “How many contracts expire this quarter?” is a database query, not a retrieval problem. RAG will find some contracts, not all of them.
- Answers spread thinly across many documents. Retrieval returns a handful of passages; if the answer needs fifty, it will not fit.
- Information that was never written down. No retrieval strategy recovers what does not exist.
- Reasoning over structure. Comparisons, calculations and trend analysis are better served by giving the model a tool that queries structured data.
The most common production architecture is a hybrid: retrieval for the document-shaped questions, tool calls for the structured ones, and a routing step that decides which is which.
People also ask this as
- What does RAG stand for in AI?
- How does retrieval-augmented generation work?
- What is a RAG pipeline?
- Why use RAG instead of just prompting a model?