Question 1

What is retrieval-augmented generation (RAG)?

Accepted Answer

Retrieval-augmented generation (RAG) is a technique where a language model answers a question using documents fetched from your own data at the moment the question is asked, rather than relying only on what it learned during training. A retrieval step searches your corpus for relevant passages, those passages are placed into the model prompt as context, and the model generates an answer grounded in them. RAG lets a model use private, current or proprietary information it was never trained on, and makes answers citable back to a source document.

Question 2

Should we use RAG or fine-tuning?

Accepted Answer

Use RAG when the model needs access to knowledge it does not have: private documents, current information, or facts that change. Use fine-tuning when the model needs to behave differently: a specific output format, a house tone, a narrow classification task, or a domain vocabulary it handles poorly. The distinction is knowledge versus behaviour. RAG is cheaper, updates instantly and gives citations; fine-tuning changes the model itself and requires retraining to update. Most production systems that use fine-tuning also use RAG, because the two solve different problems.

Question 3

How much does it cost to build a RAG system?

Accepted Answer

A production RAG system typically costs between €60,000 and €180,000 to build with an external partner, or roughly three to six months of a two-person internal team. Running costs are much lower than most teams expect: €500 to €4,000 per month for a typical internal deployment serving a few thousand queries a day, covering inference, embeddings, vector storage and hosting. The build cost is driven far more by document complexity and the number of source systems than by query volume. Scanned PDFs and table-heavy documents can double the engineering effort.

Question 4

Is a RAG system GDPR compliant?

Accepted Answer

RAG can be GDPR compliant, and is often easier to make compliant than fine-tuning, because personal data stays in a retrievable store you control rather than being absorbed into model weights. That structure means you can honour erasure requests by deleting from the index, enforce access controls at retrieval time, and show exactly which records informed an answer. The obligations that still apply are a documented lawful basis, a data processing agreement with your model provider, a transfer mechanism if inference happens outside the EEA, and a DPIA where the processing is likely to be high risk.

Question 5

How do you evaluate a RAG system?

Accepted Answer

Evaluate retrieval and generation separately, because they fail for different reasons. For retrieval, build a set of real questions paired with the passages that actually answer them, then measure recall@k, mean reciprocal rank and nDCG. This catches most of what teams misdiagnose as hallucination. For generation, measure groundedness (is every claim supported by the retrieved context), citation accuracy, and task completion. A stratified set of 150 to 300 curated cases detects meaningful regressions reliably; run it in CI and block merges on regression.

Question 6

How do you reduce hallucinations in an LLM system?

Accepted Answer

Most hallucinations in production systems are retrieval failures rather than model failures. The model was asked to answer from context that did not contain the answer. Fix retrieval first and measure recall@k, because no prompt change compensates for the right passage never arriving. Then constrain generation to cite retrieved passages and to refuse when context is insufficient, add a verification pass that checks each claim against the source for high-stakes answers, and measure groundedness on every change so regressions are caught in CI rather than by users.

Question 7

Which vector database should we use for RAG?

Accepted Answer

If your corpus is under roughly one million chunks and you already run PostgreSQL, use pgvector. It removes a system from your stack, keeps vectors transactionally consistent with your metadata, and performs well at that scale. A dedicated vector database earns its place when you need very large scale, sub-50ms retrieval at high concurrency, complex metadata filtering, or built-in hybrid search and reranking you would otherwise build yourself. Choose on operational fit rather than benchmark charts: retrieval quality is determined far more by chunking, embedding choice and reranking than by which index you store the vectors in.

Question 8

How should we chunk documents for RAG?

Accepted Answer

Chunk along the document structure rather than at a fixed character count: split on headings, sections and natural boundaries so each chunk stays self-contained. For most prose corpora, target 400 to 800 tokens with 10 to 15 percent overlap, and prepend the document title and section heading to every chunk so a retrieved passage carries its own context. Tables and lists should stay intact rather than being split mid-structure. There is no universally best size, so benchmark two or three configurations against a labelled evaluation set on your own corpus. The right answer differs by document type.

Question 9

Is our AI system high-risk under the EU AI Act?

Accepted Answer

High-risk classification under the EU AI Act follows the use case, not the technology. Annex III lists the high-risk categories: biometrics, critical infrastructure, education and vocational training, employment and worker management, access to essential private and public services including creditworthiness and insurance pricing, law enforcement, migration and border control, and administration of justice. An internal knowledge assistant is normally minimal or limited risk, carrying mainly transparency obligations. The same underlying technology used to screen job applicants is high-risk. Classify each use case separately and document the reasoning, because the reasoning is what you have to defend.

Question 10

How can we reduce our LLM API costs?

Accepted Answer

The four techniques that move LLM costs most are caching, model routing, context trimming and prompt-prefix reuse. Exact-match plus semantic caching typically removes 25 to 45 percent of calls in assistant workloads. Routing simple requests to a smaller model and reserving the largest model for hard synthesis often halves the remaining spend. Trimming retrieved context from ten passages to four usually improves answer quality while cutting input tokens. Applied together these commonly reduce inference bills by 60 to 80 percent, but each change has to be verified against an evaluation set, because cheaper routing quietly degrades some task types.

Question 11

How long does it take to build a production LLM application?

Accepted Answer

A working prototype takes two to three weeks. A production system takes eight to sixteen weeks, and the difference is not polish. It is document parsing, evaluation, access control, observability and the failure handling that a demo never needs. The largest single cause of overrun is document quality discovered late: scanned PDFs, table-heavy files and inconsistent formats routinely double the ingestion effort. Teams that build a labelled evaluation set in week two finish faster than those that skip it, because every subsequent decision stops being guesswork.

Question 12

Should we build or buy an AI solution?

Accepted Answer

Buy when the use case is generic and a vendor already solves it well: meeting notes, coding assistance, general document search over common formats. Build when the system depends on your proprietary data, your specific workflow, or a domain judgement that no vendor can encode, and when the capability is close enough to your core business that owning it matters. The most common outcome is hybrid: buy the horizontal tools, build the two or three systems that touch your differentiating data. Before choosing, run a short feasibility probe on your real data, because most build-versus-buy arguments are really disagreements about whether the data supports the use case at all.

Questions I get asked, answered properly

Retrieval & RAG