--- title: Answers source: https://rubradigital.com/answers site: Rubra Digital description: Direct answers to the questions I am most often asked about LLM and RAG work. --- # Answers **Summary:** Direct answers to the questions I am most often asked about LLM and RAG work. ## What is retrieval-augmented generation (RAG)? Retrieval-augmented generation (RAG) is a technique where a language model answers a question using documents fetched from your own data at the moment the question is asked, rather than relying only on what it learned during training. A retrieval step searches your corpus for relevant passages, those passages are placed into the model prompt as context, and the model generates an answer grounded in them. RAG lets a model use private, current or proprietary information it was never trained on, and makes answers citable back to a source document. Full page: https://rubradigital.com/answers/what-is-rag ## Should we use RAG or fine-tuning? Use RAG when the model needs access to knowledge it does not have: private documents, current information, or facts that change. Use fine-tuning when the model needs to behave differently: a specific output format, a house tone, a narrow classification task, or a domain vocabulary it handles poorly. The distinction is knowledge versus behaviour. RAG is cheaper, updates instantly and gives citations; fine-tuning changes the model itself and requires retraining to update. Most production systems that use fine-tuning also use RAG, because the two solve different problems. Full page: https://rubradigital.com/answers/rag-vs-fine-tuning ## How much does it cost to build a RAG system? A production RAG system typically costs between €60,000 and €180,000 to build with an external partner, or roughly three to six months of a two-person internal team. Running costs are much lower than most teams expect: €500 to €4,000 per month for a typical internal deployment serving a few thousand queries a day, covering inference, embeddings, vector storage and hosting. The build cost is driven far more by document complexity and the number of source systems than by query volume. Scanned PDFs and table-heavy documents can double the engineering effort. Full page: https://rubradigital.com/answers/how-much-does-a-rag-system-cost ## Is a RAG system GDPR compliant? RAG can be GDPR compliant, and is often easier to make compliant than fine-tuning, because personal data stays in a retrievable store you control rather than being absorbed into model weights. That structure means you can honour erasure requests by deleting from the index, enforce access controls at retrieval time, and show exactly which records informed an answer. The obligations that still apply are a documented lawful basis, a data processing agreement with your model provider, a transfer mechanism if inference happens outside the EEA, and a DPIA where the processing is likely to be high risk. Full page: https://rubradigital.com/answers/is-rag-gdpr-compliant ## How do you evaluate a RAG system? Evaluate retrieval and generation separately, because they fail for different reasons. For retrieval, build a set of real questions paired with the passages that actually answer them, then measure recall@k, mean reciprocal rank and nDCG. This catches most of what teams misdiagnose as hallucination. For generation, measure groundedness (is every claim supported by the retrieved context), citation accuracy, and task completion. A stratified set of 150 to 300 curated cases detects meaningful regressions reliably; run it in CI and block merges on regression. Full page: https://rubradigital.com/answers/how-to-evaluate-a-rag-system ## How do you reduce hallucinations in an LLM system? Most hallucinations in production systems are retrieval failures rather than model failures. The model was asked to answer from context that did not contain the answer. Fix retrieval first and measure recall@k, because no prompt change compensates for the right passage never arriving. Then constrain generation to cite retrieved passages and to refuse when context is insufficient, add a verification pass that checks each claim against the source for high-stakes answers, and measure groundedness on every change so regressions are caught in CI rather than by users. Full page: https://rubradigital.com/answers/how-to-reduce-llm-hallucinations ## Which vector database should we use for RAG? If your corpus is under roughly one million chunks and you already run PostgreSQL, use pgvector. It removes a system from your stack, keeps vectors transactionally consistent with your metadata, and performs well at that scale. A dedicated vector database earns its place when you need very large scale, sub-50ms retrieval at high concurrency, complex metadata filtering, or built-in hybrid search and reranking you would otherwise build yourself. Choose on operational fit rather than benchmark charts: retrieval quality is determined far more by chunking, embedding choice and reranking than by which index you store the vectors in. Full page: https://rubradigital.com/answers/which-vector-database-should-i-use ## How should we chunk documents for RAG? Chunk along the document structure rather than at a fixed character count: split on headings, sections and natural boundaries so each chunk stays self-contained. For most prose corpora, target 400 to 800 tokens with 10 to 15 percent overlap, and prepend the document title and section heading to every chunk so a retrieved passage carries its own context. Tables and lists should stay intact rather than being split mid-structure. There is no universally best size, so benchmark two or three configurations against a labelled evaluation set on your own corpus. The right answer differs by document type. Full page: https://rubradigital.com/answers/rag-chunking-strategy ## Is our AI system high-risk under the EU AI Act? High-risk classification under the EU AI Act follows the use case, not the technology. Annex III lists the high-risk categories: biometrics, critical infrastructure, education and vocational training, employment and worker management, access to essential private and public services including creditworthiness and insurance pricing, law enforcement, migration and border control, and administration of justice. An internal knowledge assistant is normally minimal or limited risk, carrying mainly transparency obligations. The same underlying technology used to screen job applicants is high-risk. Classify each use case separately and document the reasoning, because the reasoning is what you have to defend. Full page: https://rubradigital.com/answers/is-my-ai-system-high-risk-under-the-eu-ai-act ## How can we reduce our LLM API costs? The four techniques that move LLM costs most are caching, model routing, context trimming and prompt-prefix reuse. Exact-match plus semantic caching typically removes 25 to 45 percent of calls in assistant workloads. Routing simple requests to a smaller model and reserving the largest model for hard synthesis often halves the remaining spend. Trimming retrieved context from ten passages to four usually improves answer quality while cutting input tokens. Applied together these commonly reduce inference bills by 60 to 80 percent, but each change has to be verified against an evaluation set, because cheaper routing quietly degrades some task types. Full page: https://rubradigital.com/answers/how-to-reduce-llm-api-costs ## How long does it take to build a production LLM application? A working prototype takes two to three weeks. A production system takes eight to sixteen weeks, and the difference is not polish. It is document parsing, evaluation, access control, observability and the failure handling that a demo never needs. The largest single cause of overrun is document quality discovered late: scanned PDFs, table-heavy files and inconsistent formats routinely double the ingestion effort. Teams that build a labelled evaluation set in week two finish faster than those that skip it, because every subsequent decision stops being guesswork. Full page: https://rubradigital.com/answers/how-long-does-it-take-to-build-an-llm-application ## Should we build or buy an AI solution? Buy when the use case is generic and a vendor already solves it well: meeting notes, coding assistance, general document search over common formats. Build when the system depends on your proprietary data, your specific workflow, or a domain judgement that no vendor can encode, and when the capability is close enough to your core business that owning it matters. The most common outcome is hybrid: buy the horizontal tools, build the two or three systems that touch your differentiating data. Before choosing, run a short feasibility probe on your real data, because most build-versus-buy arguments are really disagreements about whether the data supports the use case at all. Full page: https://rubradigital.com/answers/should-we-build-or-buy-ai --- Rubra Digital. Independent LLM and RAG consulting for regulated and document-heavy organisations in Europe and North America. Contact: hello@rubradigital.com · https://rubradigital.com/contact