# Rubra Digital: full site content > Independent LLM and RAG consulting for regulated and document-heavy organisations in Europe and North America. Canonical site: https://rubradigital.com Contact: hello@rubradigital.com Generated: 2026-08-23 This file contains the complete text of every content page on rubradigital.com. It is provided so that language models and retrieval pipelines can ingest my writing directly rather than scraping rendered HTML. Attribution appreciated. ======================================================================== # ANSWERS ======================================================================== ## What is retrieval-augmented generation (RAG)? Source: https://rubradigital.com/answers/what-is-rag Summary: Retrieval-augmented generation (RAG) is a technique where a language model answers a question using documents fetched from your own data at the moment the question is asked, rather than relying only on what it learned during training. A retrieval step searches your corpus for relevant passages, those passages are placed into the model prompt as context, and the model generates an answer grounded in them. RAG lets a model use private, current or proprietary information it was never trained on, and makes answers citable back to a source document. ## The pipeline, step by step A RAG system has two phases: one that runs ahead of time, and one that runs per question. **Indexing (ahead of time)** 1. **Ingest.** Documents are collected from their sources: file shares, a CMS, a ticketing system, a database. 2. **Parse.** Each document is converted to text. This is where most quality is won or lost, particularly with PDFs, tables and scanned pages. 3. **Chunk.** Text is split into passages small enough to be useful as context but large enough to remain self-contained. 4. **Embed.** Each chunk is converted into a vector that represents its meaning. 5. **Index.** Vectors and their source text are stored in a searchable index. **Retrieval and generation (per question)** 6. **Embed the question** using the same model used for the chunks. 7. **Search** the index for the most similar chunks, usually combining vector similarity with keyword search. 8. **Rerank** the candidates with a model that scores relevance more precisely than the initial search can. 9. **Assemble the prompt.** The question plus the top passages, with instructions to answer only from them. 10. **Generate** the answer, with citations pointing back to the source passages. ## Why it exists A language model's knowledge is fixed at training time and contains nothing private to your organisation. Three problems follow: it does not know your internal information, it does not know what changed last week, and it cannot show you where an answer came from. RAG addresses all three by moving the knowledge out of the model's weights and into a retrievable store. Update the store and the system's knowledge updates immediately, with no retraining. ## Where RAG works well Questions whose answers already exist in writing but are expensive to find: policy and regulatory libraries, technical manuals, contract portfolios, support histories, research literature. ## Where RAG struggles - **Aggregation.** "How many contracts expire this quarter?" is a database query, not a retrieval problem. RAG will find some contracts, not all of them. - **Answers spread thinly across many documents.** Retrieval returns a handful of passages; if the answer needs fifty, it will not fit. - **Information that was never written down.** No retrieval strategy recovers what does not exist. - **Reasoning over structure.** Comparisons, calculations and trend analysis are better served by giving the model a tool that queries structured data. The most common production architecture is a hybrid: retrieval for the document-shaped questions, tool calls for the structured ones, and a routing step that decides which is which. ## Should we use RAG or fine-tuning? Source: https://rubradigital.com/answers/rag-vs-fine-tuning Summary: Use RAG when the model needs access to knowledge it does not have: private documents, current information, or facts that change. Use fine-tuning when the model needs to behave differently: a specific output format, a house tone, a narrow classification task, or a domain vocabulary it handles poorly. The distinction is knowledge versus behaviour. RAG is cheaper, updates instantly and gives citations; fine-tuning changes the model itself and requires retraining to update. Most production systems that use fine-tuning also use RAG, because the two solve different problems. ## The one-line rule **Knowledge goes in retrieval. Behaviour goes in weights.** If your complaint is _the model does not know X_, that is RAG. If your complaint is _the model knows X but says it wrong_, that is fine-tuning. ## Side by side | | RAG | Fine-tuning | | -------------- | --------------------------------- | -------------------------------------- | | Solves | Missing knowledge | Wrong behaviour | | Update cost | Re-index a document, minutes | Retrain, hours to days | | Setup cost | Moderate: pipeline and evaluation | High: labelled data, training, hosting | | Citations | Natural, passages are retrieved | Not possible from weights alone | | Data freshness | Immediate | Frozen at training time | | Access control | Enforceable at retrieval time | Not enforceable; knowledge is baked in | | Fails by | Retrieving the wrong passage | Confidently generalising wrongly | ## Why teams reach for fine-tuning too early Fine-tuning feels like the more serious engineering answer, so it attracts teams who want to be doing something substantial. In practice it is usually the wrong first move: - **Knowledge fine-tuned into weights cannot be cited**, which fails immediately in any regulated or high-trust context. - **It cannot respect permissions.** If a document is in the training set, the model may surface it to anyone. Retrieval can filter by the user's access rights before the model ever sees a passage. - **It goes stale.** Every material change to the underlying information means another training run. - **It needs labelled data**, typically several hundred to a few thousand good examples, which most teams do not have and underestimate the cost of creating. ## When fine-tuning earns its place - **Strict output structure** that prompting cannot hold reliably at volume. - **A narrow, high-volume classification task** where a small fine-tuned model matches a large general one at a fraction of the cost and latency. - **Domain language** the base model handles badly: specialised clinical, legal or industrial vocabulary. - **Latency or cost ceilings** that a large model cannot meet, where a small fine-tuned model can. ## The realistic sequence 1. Prompt engineering with a strong model. Establish the quality ceiling. 2. Add RAG if the failures are knowledge failures. This resolves most cases. 3. Build evaluation. You cannot judge step four without it. 4. Fine-tune only if measured failures are behavioural, and only after you have a labelled set worth training on. Teams that skip to step four spend three months and arrive back at step two. ## How much does it cost to build a RAG system? Source: https://rubradigital.com/answers/how-much-does-a-rag-system-cost Summary: A production RAG system typically costs between €60,000 and €180,000 to build with an external partner, or roughly three to six months of a two-person internal team. Running costs are much lower than most teams expect: €500 to €4,000 per month for a typical internal deployment serving a few thousand queries a day, covering inference, embeddings, vector storage and hosting. The build cost is driven far more by document complexity and the number of source systems than by query volume. Scanned PDFs and table-heavy documents can double the engineering effort. ## Build cost: what the range actually depends on The two things that move build cost most are not the ones teams expect. **Document complexity.** A corpus of clean, digital, text-first documents is a straightforward ingestion job. A corpus of scanned PDFs, engineering drawings, documents where the meaning lives in tables, or files exported from a system that lost the structure can double or triple the parsing effort. This is the single most common source of budget overrun, and it is knowable in week one if anyone looks. **Number of source systems.** One SharePoint library is one integration. Six systems with different auth models, permission schemes and update cadences is a different project, and permission-aware retrieval across all of them is harder still. Rough bands, for delivery with an experienced partner: | Scope | Typical range | | ---------------------------------------------------- | ------------------ | | Prototype on a clean single corpus | €15,000 – €30,000 | | Production system, one corpus, internal users | €60,000 – €95,000 | | Production system, several sources, permission-aware | €95,000 – €180,000 | | Customer-facing, regulated, multilingual | €180,000+ | An internal build is not free: the same work is roughly three to six months of two capable engineers, plus the learning curve if this is their first such system. ## Running cost: usually the smaller number For a system handling around 3,000 queries a day over a corpus of a few hundred thousand chunks, a representative monthly bill in 2026: - **Generation.** €300 to €2,500, depending heavily on model choice and how much context is stuffed into each prompt - **Embeddings.** €20 to €150. Re-embedding on document change is minor - **Vector storage.** €0 if you use pgvector on a database you already run, up to €700 for a managed vector service at this scale - **Reranking.** €50 to €400 - **Hosting and observability.** €100 to €500 Teams routinely over-provision here. The most common waste is sending 8,000 tokens of context when 2,000 would answer the question equally well, and routing every request to the largest available model regardless of difficulty. ## What reduces cost without hurting quality - **Caching.** Exact-match plus semantic caching typically removes 25–45% of calls in assistant-style workloads. - **Right-sizing the model per request.** Route simple lookups to a small model, hard synthesis to a large one. Verify with evaluation, because this quietly degrades some task types. - **Tighter retrieval.** Better reranking means fewer passages needed, which means shorter prompts and lower cost per call. - **Prompt-prefix caching** where the provider supports it, for the static instruction block. ## The cost nobody budgets Evaluation. Building the labelled set and the harness is typically 15–25% of the project, and it is the line most often cut. It is also the line that determines whether you can safely change anything after launch. Cutting it does not save money, it defers a larger cost to month six. ## Is a RAG system GDPR compliant? Source: https://rubradigital.com/answers/is-rag-gdpr-compliant Summary: RAG can be GDPR compliant, and is often easier to make compliant than fine-tuning, because personal data stays in a retrievable store you control rather than being absorbed into model weights. That structure means you can honour erasure requests by deleting from the index, enforce access controls at retrieval time, and show exactly which records informed an answer. The obligations that still apply are a documented lawful basis, a data processing agreement with your model provider, a transfer mechanism if inference happens outside the EEA, and a DPIA where the processing is likely to be high risk. _This is engineering guidance, not legal advice. Your data protection officer or counsel should own the legal position._ ## Why RAG is structurally easier than fine-tuning Under GDPR, personal data must be deletable, correctable and access-controlled. Once personal data is fine-tuned into model weights, none of those are cleanly achievable. You cannot surgically remove one person's data from a trained model, and you cannot stop it surfacing to an unauthorised user. In a RAG system the personal data stays in an index: - **Erasure** is a delete from the index and the source, and it takes effect immediately. - **Rectification** is an update and re-index. - **Access control** is enforced at retrieval, before the model sees anything, so so a user only ever gets answers grounded in documents they were entitled to read. - **Traceability** falls out of the citations: you can show which records contributed to a given output. That is a far better starting position. ## What you still have to get right **Lawful basis.** Usually legitimate interests for internal knowledge tools, with a documented balancing test. Consent is rarely the right basis for a workplace system, and it is fragile because it can be withdrawn. **Processor agreements.** Your model provider is a processor. You need an Article 28 data processing agreement, and you need to check what the provider does with inputs: whether prompts are retained, for how long, and whether they may be used for training. Enterprise tiers of the major providers generally offer zero-retention and no-training options; the default consumer tiers often do not. **International transfers.** If inference happens outside the EEA you need a valid transfer mechanism, typically Standard Contractual Clauses or the EU-US Data Privacy Framework where the provider is certified. The simplest path is often to avoid the question by using EU-region inference endpoints, which every major provider now offers. **DPIA.** Required where processing is likely to result in high risk to individuals. For systems touching employee data, health data, or automated decisions affecting people, assume you need one. **Purpose limitation.** Data collected for one purpose cannot be silently repurposed as AI training or retrieval material. This catches a lot of internal projects that index a CRM or a support archive without revisiting the original notice. ## Design choices that make this tractable 1. **Filter at retrieval, not in the prompt.** Apply the user's permissions to the search query itself so unauthorised passages are never retrieved. 2. **Keep the index authoritative and derived.** Treat it as a projection of the source system so deletions propagate on a known schedule, and document that schedule. 3. **Redact before egress.** Strip identifiers that the model does not need to answer the question. 4. **Choose EU-region inference** for EU personal data. It removes the hardest part of the transfer analysis. 5. **Log what was retrieved, not just what was answered.** That log is what lets you respond to a subject access request truthfully. 6. **Set a retention period on prompt logs** and enforce it. Conversation history is personal data too, and it is the part teams most often forget. ## How do you evaluate a RAG system? Source: https://rubradigital.com/answers/how-to-evaluate-a-rag-system Summary: Evaluate retrieval and generation separately, because they fail for different reasons. For retrieval, build a set of real questions paired with the passages that actually answer them, then measure recall@k, mean reciprocal rank and nDCG. This catches most of what teams misdiagnose as hallucination. For generation, measure groundedness (is every claim supported by the retrieved context), citation accuracy, and task completion. A stratified set of 150 to 300 curated cases detects meaningful regressions reliably; run it in CI and block merges on regression. ## Separate the two failure modes When an answer is wrong, there are exactly two possibilities: the right information never reached the model, or it reached the model and the model handled it badly. These need different fixes, so they need different measurements. In my experience, the majority of "hallucination" complaints in production RAG systems are retrieval failures. The model was asked to answer from context that did not contain the answer, and it obliged. ## Retrieval metrics Build a set of questions paired with the passage IDs that answer them. Then: - **Recall@k.** Is a correct passage in the top _k_ results? The headline number. If recall@5 is 0.7, three in ten questions are unanswerable no matter what the model does. - **Mean reciprocal rank (MRR).** How high up is the first correct passage? Rank matters because passages far down the list compete for attention with irrelevant ones. - **nDCG.** Accounts for multiple relevant passages at different usefulness levels. Worth adding when questions typically need several sources. These are deterministic, cost nothing to run, and take seconds. Run them on every change. ## Generation metrics - **Groundedness.** Is every factual claim in the answer supported by the retrieved context? The most important generation metric by a wide margin. - **Citation accuracy.** Do the citations point at passages that actually contain the claim? Systems that cite plausibly but wrongly are more dangerous than systems that do not cite at all. - **Task completion.** Did it answer the question that was asked, rather than a nearby one? - **Appropriate refusal.** When the context does not support an answer, does the system say so? Measure this with a deliberate subset of unanswerable questions. ## Building the test set **Use real questions.** Pull them from search logs, support tickets, or a week of asking the people who will use the system to write down what they actually want to know. Invented questions cluster on the easy cases. **Stratify deliberately.** Simple lookups, multi-document synthesis, questions with near-miss distractors in the corpus, ambiguous questions, and unanswerable questions. Aim for roughly balanced coverage rather than natural frequency, which over-weights the easy tail. **150 to 300 curated cases beats 5,000 scraped ones.** The curated set exercises the failure modes; the scraped set mostly re-confirms that easy questions work. **Get the ground truth from people who know.** A subject-matter expert marking which passage is correct is the expensive, unavoidable part. Budget for it. ## Calibrating an LLM judge Using a model to grade output is practical and scales. Uncalibrated, it produces confident numbers with unknown error. Calibrate it: have humans score a stratified sample of 80–150 outputs, have the judge score the same sample, and compute agreement. Then use the judge only for metrics where agreement is high enough to act on. Groundedness usually calibrates well. Subjective quality usually does not, and where it does not, say so rather than shipping a metric nobody should trust. Report the agreement rate alongside the metric. It tells everyone how much weight the gate deserves. ## Wire it into CI An evaluation suite that runs when someone remembers is not a gate. Run retrieval metrics on every pull request, the fuller suite nightly, and block merges on regression beyond a threshold you agree in advance. This is what makes it safe for more than one person to change prompts. ## How do you reduce hallucinations in an LLM system? Source: https://rubradigital.com/answers/how-to-reduce-llm-hallucinations Summary: Most hallucinations in production systems are retrieval failures rather than model failures. The model was asked to answer from context that did not contain the answer. Fix retrieval first and measure recall@k, because no prompt change compensates for the right passage never arriving. Then constrain generation to cite retrieved passages and to refuse when context is insufficient, add a verification pass that checks each claim against the source for high-stakes answers, and measure groundedness on every change so regressions are caught in CI rather than by users. ## Diagnose before you treat The first question is always: **was the correct information in the context window?** Log the retrieved passages alongside every answer and check a sample of failures. Teams that do this are usually surprised. In most RAG systems I audit, the majority of wrong answers had wrong or missing context. That is a retrieval bug wearing a hallucination costume, and no amount of prompt tuning fixes it. Only once retrieval is measurably good does it make sense to work on generation. ## Retrieval-side fixes - **Hybrid search.** Pure vector search misses exact terms: product codes, error numbers, proper nouns. Combining it with BM25 keyword search closes a surprising share of failures. - **Reranking.** A cross-encoder reranker over the top 50 candidates consistently improves precision at the top of the list, and it is one of the highest-return changes available. - **Chunking that respects structure.** A clause split from its heading loses the context that made it findable. - **Query rewriting.** Users ask questions that do not match how documents are written. Rewriting the query before search, or generating several variants and merging the results, helps materially on conversational systems. ## Generation-side fixes - **Instruct for grounding and refusal explicitly.** The system prompt should require answers to come only from the provided context and require an explicit "the provided documents do not cover this" when they do not. - **Require citations per claim.** Attribution pressure measurably reduces invention, and it makes errors visible when they happen. - **Lower temperature for factual tasks.** Not a cure, but free. - **Do not overfill the context.** Stuffing thirty marginal passages in makes the answer worse, not better, because relevant material competes with noise. ## Verification for high-stakes answers Where an error is expensive, add a second pass: take each claim in the generated answer and check it against the cited passage. This roughly doubles cost and latency, so reserve it for the paths that warrant it. It catches the residual errors that grounding alone does not. ## Make refusal a first-class outcome A system that says "I do not know" is more useful than one that guesses, and users calibrate their trust accordingly. Build a subset of deliberately unanswerable questions into your evaluation set and measure the refusal rate on them. If it is low, your system is confidently wrong on exactly the questions where being wrong matters most. ## The honest limit Hallucination cannot be reduced to zero with current models. It can be reduced to a measured, acceptable rate for a defined task, with a known failure mode and a human checkpoint where the stakes require one. Any vendor claiming elimination is describing a demo, not a production system. The useful goal is not perfection. It is knowing your groundedness number, having it under regression control, and designing the surrounding process for the residual error rate. ## Which vector database should we use for RAG? Source: https://rubradigital.com/answers/which-vector-database-should-i-use Summary: If your corpus is under roughly one million chunks and you already run PostgreSQL, use pgvector. It removes a system from your stack, keeps vectors transactionally consistent with your metadata, and performs well at that scale. A dedicated vector database earns its place when you need very large scale, sub-50ms retrieval at high concurrency, complex metadata filtering, or built-in hybrid search and reranking you would otherwise build yourself. Choose on operational fit rather than benchmark charts: retrieval quality is determined far more by chunking, embedding choice and reranking than by which index you store the vectors in. ## The uncomfortable answer The vector store is rarely what determines whether your RAG system works. Chunking strategy, embedding model choice, hybrid search and reranking all move retrieval quality far more than the choice of index. Teams spend weeks on this decision and days on chunking, which is exactly backwards. Pick the option that fits your operations and move on to the things that matter. ## Start with pgvector if you can If you already run PostgreSQL, `pgvector` is the default answer: - **No new system to operate**, monitor, back up, secure or pay for. - **Vectors and metadata in one transaction.** No consistency gap between your documents table and your index, which is a real source of bugs in two-system setups. - **Filtering is just SQL.** Permission checks and metadata filters compose naturally, which matters a lot for access-controlled retrieval. - **Performance is fine** for corpora into the low millions of chunks with HNSW indexing. The limits are real but arrive later than most teams expect: index build time grows awkward at very large scale, and heavy vector query load competes with your transactional workload unless you use a read replica. ## When a dedicated vector database earns its place - **Scale beyond a few million chunks**, where purpose-built index management and sharding matter. - **Tight latency at high concurrency.** Consistent sub-50ms retrieval under sustained load. - **Complex metadata filtering at scale**, where naive pre- or post-filtering degrades recall badly. - **Built-in hybrid search and reranking**, saving you from assembling BM25 and a cross-encoder yourself. - **You do not run PostgreSQL** and do not want to start. ## What actually differentiates the options Ignore recall benchmarks. At typical configurations the mature options are within noise of each other. Judge on: 1. **Filtering semantics.** How does filtering interact with the approximate index? Some options degrade recall sharply under selective filters. Test this with your real filter patterns. 2. **Update behaviour.** How costly are inserts and deletes? If your corpus changes hourly this dominates. 3. **Operational model.** Managed or self-hosted, and does the managed region list include the jurisdiction your data must stay in? For European clients this frequently eliminates otherwise-attractive options. 4. **Hybrid support.** If it does not do keyword search, you will run a second system anyway. 5. **Cost at your scale.** Per-vector pricing looks cheap until you re-embed a large corpus. ## How I choose in practice I benchmark two candidates against the client's actual corpus and their actual filter patterns, using the labelled evaluation set built earlier in the project. It takes about two days and replaces an argument with a number. Nine times out of ten for European mid-market clients the answer is pgvector, and the two days are still worth spending, because they close the question for good. ## How should we chunk documents for RAG? Source: https://rubradigital.com/answers/rag-chunking-strategy Summary: Chunk along the document structure rather than at a fixed character count: split on headings, sections and natural boundaries so each chunk stays self-contained. For most prose corpora, target 400 to 800 tokens with 10 to 15 percent overlap, and prepend the document title and section heading to every chunk so a retrieved passage carries its own context. Tables and lists should stay intact rather than being split mid-structure. There is no universally best size, so benchmark two or three configurations against a labelled evaluation set on your own corpus. The right answer differs by document type. ## Why chunking decides more than it should A chunk is the unit of retrieval. If the answer spans two chunks, retrieval returns half of it. If a chunk contains five topics, its embedding represents none of them well. Most retrieval quality problems that get blamed on the embedding model are chunking problems. ## The strategies, in ascending order of effort **Fixed-size.** Split every _n_ characters. Trivial to implement and reliably poor: it cuts sentences, separates clauses from headings, and splits tables down the middle. Use it only as a baseline to beat. **Recursive.** Split on paragraph breaks, then sentences, then characters, until chunks fit the target size. A large improvement over fixed-size for very little work, and a reasonable default for undifferentiated prose. **Structure-aware.** Parse the document's actual structure (headings, sections, list items, table boundaries) and chunk along it. This is what I use for most production systems. Section boundaries are semantic boundaries that the author already provided; ignoring them is throwing away free signal. **Semantic.** Use embeddings to detect topic shifts and split there. Sometimes better than structure-aware on documents with no usable structure: meeting transcripts, long unformatted text. It costs an embedding pass over the corpus and, in my benchmarks, rarely beats good structure-aware chunking on documents that have real structure. Try it, but measure before adopting it. ## Practical parameters **Size: 400–800 tokens for prose.** Small enough to be specific, large enough to be self-contained. Question-answer pairs and reference entries work well much smaller. Narrative or legal text often needs the upper end. **Overlap: 10–15%.** Enough to catch answers that straddle a boundary. More than about 20% mostly inflates your index and returns near-duplicate results that crowd out the genuinely different sources. **Always prepend context.** Every chunk should carry the document title and its heading path, for example `Employee Handbook > Leave > Parental leave`. This is the highest-return, lowest-effort improvement available: it makes a passage retrievable by terms that appear in its heading rather than its body, and it gives the model orientation when the passage arrives out of context. **Keep structures intact.** A table split across chunks is useless in both halves. Extract tables as units; if a table is too large, repeat the header row in each part. **Attach metadata.** Source, section, date, document type, and whatever your permission model needs. Metadata filters are often more effective than any tuning of the vector search itself. ## Two techniques worth the extra effort **Small-to-big.** Embed and search over small, precise chunks, then return the larger parent section to the model. You get the precision of small chunks in retrieval and the completeness of large ones in generation. **Contextual chunk headers.** Generate a one-sentence summary of what each chunk covers in the context of its document, and prepend it before embedding. It costs one cheap model call per chunk at index time and measurably improves retrieval on corpora full of ambiguous references. ## How to decide Take your labelled evaluation set. Run three configurations: recursive at 512 tokens, structure-aware at 512, and structure-aware at 1024 with parent retrieval. Then compare recall@5. This takes an afternoon and settles the question with evidence rather than with whatever the last blog post recommended. ## Is our AI system high-risk under the EU AI Act? Source: https://rubradigital.com/answers/is-my-ai-system-high-risk-under-the-eu-ai-act Summary: High-risk classification under the EU AI Act follows the use case, not the technology. Annex III lists the high-risk categories: biometrics, critical infrastructure, education and vocational training, employment and worker management, access to essential private and public services including creditworthiness and insurance pricing, law enforcement, migration and border control, and administration of justice. An internal knowledge assistant is normally minimal or limited risk, carrying mainly transparency obligations. The same underlying technology used to screen job applicants is high-risk. Classify each use case separately and document the reasoning, because the reasoning is what you have to defend. _Engineering guidance, not legal advice. Your counsel should own the legal classification; I help produce the technical evidence behind it._ ## The four tiers **Unacceptable risk, prohibited.** Social scoring by public authorities, manipulative techniques exploiting vulnerabilities, untargeted scraping of facial images to build recognition databases, emotion inference in workplaces and schools, and certain biometric categorisation. These have been prohibited since February 2025. **High risk, the substantive obligations.** Two routes in. Annex I covers AI as a safety component of products already subject to EU product legislation such as medical devices, machinery, vehicles and lifts. Annex III lists standalone high-risk use cases: - Biometric identification and categorisation - Critical infrastructure management and operation - Education and vocational training: admission, evaluation, proctoring - Employment and worker management: recruitment, screening, promotion, task allocation, monitoring - Access to essential private and public services: creditworthiness, benefits eligibility, emergency dispatch, life and health insurance pricing - Law enforcement - Migration, asylum and border control - Administration of justice and democratic processes There is a narrowing provision: a system in an Annex III area may fall outside high-risk if it only performs a narrow procedural task, improves the result of a previously completed human activity, or does preparatory work, but not if it profiles individuals. Relying on this requires documented assessment, not an assumption. **Limited risk, transparency.** Chatbots must disclose that a user is interacting with an AI system. Synthetic image, audio, video and text must be marked as artificially generated in machine-readable form. Deepfakes must be disclosed. Most customer-facing assistants land here. **Minimal risk.** Everything else. No mandatory obligations. Most internal productivity tooling sits here. ## Applying it honestly The question "is my RAG system high-risk?" has no answer. The system is not the unit of classification. The use case is. One retrieval platform inside a company might simultaneously power: - An HR policy assistant for staff. **Limited risk**, disclosure only - A CV screening tool for recruiters. **High risk**, Annex III employment - A customer support assistant. **Limited risk**, disclosure - A credit memo drafting aid. **Likely high risk**, creditworthiness Same infrastructure, four different classifications. Inventory by use case. ## What high-risk actually requires Risk management across the lifecycle (Art. 9); data governance covering provenance, representativeness and bias examination (Art. 10); technical documentation to Annex IV (Art. 11); automatic logging (Art. 12); transparency and instructions for use (Art. 13); effective human oversight (Art. 14); accuracy, robustness and cybersecurity with declared metrics (Art. 15); a quality management system (Art. 17); conformity assessment and CE marking; and registration in the EU database. That is a substantial programme. It is also, for the engineering half, largely the same evidence a well-run evaluation and observability practice already produces. ## Timeline In force since August 2024, applying in stages: prohibitions and AI literacy from February 2025, general-purpose AI model obligations from August 2025, the main Annex III high-risk obligations from August 2026, and Annex I embedded systems from August 2027. The Commission has proposed adjustments to parts of this schedule, so confirm the current position with your counsel before planning around specific dates. ## Extraterritorial reach The Act applies if you place a system on the EU market, put it into service in the EU, **or** if the system's output is used in the EU, wherever your company is established. That final limb catches a large number of US and Canadian companies serving European customers who assume they are out of scope. ## How can we reduce our LLM API costs? Source: https://rubradigital.com/answers/how-to-reduce-llm-api-costs Summary: The four techniques that move LLM costs most are caching, model routing, context trimming and prompt-prefix reuse. Exact-match plus semantic caching typically removes 25 to 45 percent of calls in assistant workloads. Routing simple requests to a smaller model and reserving the largest model for hard synthesis often halves the remaining spend. Trimming retrieved context from ten passages to four usually improves answer quality while cutting input tokens. Applied together these commonly reduce inference bills by 60 to 80 percent, but each change has to be verified against an evaluation set, because cheaper routing quietly degrades some task types. ## Measure before optimising Almost every team I audit finds that a small fraction of requests accounts for most of the spend. Common culprits: a background job re-summarising documents that have not changed, a retrieval step returning twenty passages when four would do, a debug code path left in production, or one internal power user running bulk queries through an interactive endpoint. Attribute cost per request, per feature and per user before changing anything. The first week of visibility usually pays for the work. ## Caching **Exact-match caching** on identical prompts. Trivially cheap, and in assistant-style products the repeat rate is higher than anyone expects, because the same twelve questions dominate. **Semantic caching** on near-identical questions, using an embedding similarity threshold. Effective, but set the threshold carefully and evaluate it: too loose and you serve the answer to a subtly different question, which is a correctness bug rather than a saving. **Prompt-prefix caching** where the provider supports it. Your system prompt and instruction block are identical on every call; providers charge substantially less for cached prefix tokens. This is usually a configuration change with no quality risk. Typical combined effect: **25–45% fewer billed calls.** ## Model routing Not every request needs the largest model. A classifier, or often just a heuristic on query type and length, routes straightforward lookups to a small model and reserves the expensive one for genuine synthesis. This is the largest single lever, frequently halving remaining spend. It is also the one that most needs evaluation: quality degradation from routing is invisible in aggregate metrics and shows up only in the task types you did not check. Run the full evaluation suite per route. ## Context trimming Retrieval systems tend to over-fetch. Sending ten passages when four contain the answer costs you on input tokens and usually produces a _worse_ answer, because relevant material competes with noise. Better reranking lets you send fewer passages with higher confidence. This is one of the rare changes that improves quality and cost simultaneously. Also check: are you resending the full conversation history every turn? Summarise older turns instead. ## Output length Output tokens usually cost several times more than input tokens. Instructing for concision, and setting sensible `max_tokens`, is free money on verbose systems. ## Structural options - **Batch APIs** for anything not interactive, typically around half price at the cost of latency. - **Self-hosting a small model** for high-volume narrow tasks. Only worth it at sustained volume; below roughly €8,000 a month of inference the engineering and operational cost usually exceeds the saving. - **Provider negotiation.** At meaningful volume, committed-use pricing is available and rarely offered unprompted. ## The rule Every cost optimisation is a quality experiment. Run it against your evaluation set, compare the numbers, and keep the change only if quality holds. A 70% saving that costs five points of groundedness is not a saving. It is a decision to ship a worse product, and it should be made deliberately if at all. ## How long does it take to build a production LLM application? Source: https://rubradigital.com/answers/how-long-does-it-take-to-build-an-llm-application Summary: A working prototype takes two to three weeks. A production system takes eight to sixteen weeks, and the difference is not polish. It is document parsing, evaluation, access control, observability and the failure handling that a demo never needs. The largest single cause of overrun is document quality discovered late: scanned PDFs, table-heavy files and inconsistent formats routinely double the ingestion effort. Teams that build a labelled evaluation set in week two finish faster than those that skip it, because every subsequent decision stops being guesswork. ## The two-week demo and the four-month system You can build something impressive over a corpus of clean documents in two weeks. It will answer questions well, demo beautifully, and create an expectation that production is a fortnight away. It is not, and the reason is worth being precise about. The demo works because it was run on the good documents, by people who knew what to ask, with no permissions, no logging, no error handling and no measurement. Production has to work on the whole corpus, for people who ask badly, under access control, with evidence that it works. ## Where the time actually goes For a typical 12-week production build: **Weeks 1 to 2. Corpus reality check and evaluation set.** What the documents really look like, and a labelled set of real questions with known-correct passages. Skipping this is the most expensive shortcut available. **Weeks 3 to 5. Ingestion and parsing.** Consistently the largest and most underestimated block. Scanned pages need OCR. Tables need structural extraction. Documents exported from legacy systems arrive with the structure stripped out. **Weeks 5 to 8. Retrieval engineering.** Chunking, embedding choice, hybrid search weighting and metadata filters, all run as experiments against the labelled set rather than as preferences. **Weeks 7 to 10. Application layer.** Generation with citations and refusal behaviour, conversation handling, permission enforcement at retrieval time, the interface. **Weeks 9 to 12. Hardening and handover.** Observability, cost controls, rate limiting, failure paths, runbook, load testing, and the sessions that leave your team able to change it. The overlaps are intentional; these tracks run in parallel. ## What causes overrun 1. **Document quality found late.** Knowable in week one if anyone samples the corpus properly. Frequently is not. 2. **Permission requirements found late.** "Everyone can see everything" becomes "actually, four access tiers and regional restrictions" in week eight, and that reshapes retrieval. 3. **No evaluation set.** Without one, retrieval tuning becomes an endless subjective argument nobody can close. 4. **Scope drift after the demo.** The demo is so encouraging that three more departments want their corpus in it. 5. **Waiting on source system access.** Nearly always the long pole, and nearly always started too late. Begin the access requests in week one. ## Compressing it honestly You can go faster by narrowing scope, not by skipping stages. One corpus, one user group, one clearly defined question type, shipped in eight weeks, beats four corpora shipped in twenty-four, and it produces the evidence you need to justify the next phase. The stages that look optional are the ones that determine whether the system survives its first month with real users. ## Should we build or buy an AI solution? Source: https://rubradigital.com/answers/should-we-build-or-buy-ai Summary: Buy when the use case is generic and a vendor already solves it well: meeting notes, coding assistance, general document search over common formats. Build when the system depends on your proprietary data, your specific workflow, or a domain judgement that no vendor can encode, and when the capability is close enough to your core business that owning it matters. The most common outcome is hybrid: buy the horizontal tools, build the two or three systems that touch your differentiating data. Before choosing, run a short feasibility probe on your real data, because most build-versus-buy arguments are really disagreements about whether the data supports the use case at all. ## Start from differentiation, not cost The cost comparison is usually the least useful frame, because the two options are rarely doing the same job. The better question: **does this system encode something specific to how I operate?** If the answer is no, buy. Nobody gains advantage from a bespoke meeting transcription tool. If the answer is yes, meaning the system depends on your document corpus, your approval workflow, your domain rules or your data model, building becomes defensible, because vendor products necessarily generalise and generalising away your specifics is exactly what breaks them. ## Buy when - The use case is horizontal and well served: transcription, coding assistance, general enterprise search, translation. - Your requirements are close to the vendor's defaults. Every deviation is configuration debt. - Time to value matters more than fit. - You have no engineering capacity to maintain a system after launch. **This is the most commonly ignored criterion.** An unmaintained internal AI system degrades quickly as its data drifts. ## Build when - The system's value comes from proprietary data or a proprietary process. - Data residency, sovereignty or access control requirements that vendors cannot meet. Common in European regulated sectors and public bodies. - The domain judgement required cannot be encoded by someone outside your field. - Per-seat vendor pricing becomes irrational at your scale. - The capability is close enough to your core business that outsourcing it means outsourcing your advantage. ## The hybrid outcome Most organisations that do this well end up buying horizontal tooling for productivity, buying platform components rather than writing them (gateways, observability, vector stores), and building the two or three systems that touch their differentiating data. That last category is small on purpose. Organisations that try to build everything ship nothing; organisations that buy everything end up with a generic capability set and a lot of vendors. ## The step that resolves most arguments Before deciding, spend a week testing feasibility against real data. Can the information needed actually be retrieved from these documents? Is the source data clean enough? Does the accuracy bar look reachable? In my experience, most build-versus-buy disagreements are not really about build versus buy. They are unspoken disagreements about whether the use case is viable at all, and a feasibility probe settles that in days for a fraction of what either path costs. ## If you buy, ask these Where does inference run and can it be restricted to a region? Is my data used for training, and is that contractual or a setting? Can I export my data and configuration if I leave? What evaluation evidence supports your accuracy claims? How do you handle access control against my existing permission model? Vendors who answer these precisely are worth talking to. Vendors who deflect are telling you something. ======================================================================== # SERVICES ======================================================================== ## RAG System Development & Consulting Source: https://rubradigital.com/services/rag-system-development Summary: Rubra builds production retrieval-augmented generation systems: document ingestion and parsing, chunking and embedding strategy, hybrid retrieval with reranking, grounded generation, and an evaluation harness that proves accuracy before launch. Typical engagements run 8–14 weeks and deliver a system your own engineers can operate. Engagement: 8–14 weeks, from €65,000, Two engineers plus a fractional architect. Most retrieval-augmented generation projects do not fail at the model. They fail at the boring parts: a PDF parser that silently drops tables, a chunking strategy that splits a clause from its heading, an index that returns plausible-looking passages that do not actually contain the answer. I build the boring parts properly. ## What I actually do ### Start with the corpus, not the model The first week is spent on your documents. I measure what your corpus really looks like: how many documents are scanned rather than digital, how much of the meaning lives in tables, and how often a single answer requires stitching two documents together. Those facts determine the architecture far more than the choice of model does. ### Build a labelled evaluation set before writing retrieval code I sit with your subject-matter experts and assemble a set of real questions with known-correct source passages. It is unglamorous work and it is the single most valuable thing in the project: without it, every subsequent decision is guesswork, and every "this feels better" is unfalsifiable. ### Tune retrieval against that set Chunk size, overlap strategy, embedding model, hybrid weighting between semantic and keyword search, reranking, metadata filters. Each of these is an experiment with a number attached. I typically run thirty to sixty configurations and report recall@k and mean reciprocal rank for each, so the final architecture is a decision you can audit rather than a preference you have to trust. ### Constrain generation and measure groundedness The generation layer cites the passages it used and declines to answer when the retrieved context does not support one. Refusal behaviour is a feature, and in regulated settings it is often the most important one. I measure groundedness and citation accuracy on every build. ### Hand it over properly You get the evaluation harness, the architecture decision records explaining why each choice was made, a runbook for the failure modes I found, and working sessions with your engineers. The goal is that the next change to the system is made by your team, not by me. ## Where this pays back Retrieval works best where the answer already exists in writing but is expensive to find: regulatory and policy libraries, technical documentation and service manuals, contract portfolios, historical support tickets, clinical or research literature, and internal knowledge that currently lives with three people who are always busy. It works poorly where the answer requires calculation, judgement under uncertainty, or data that was never written down. I will tell you which one you have during discovery, before you have spent a budget finding out. ## LLM Evaluation & Quality Engineering Source: https://rubradigital.com/services/llm-evaluation Summary: Rubra builds LLM evaluation systems: labelled test sets drawn from real usage, task-specific metrics for retrieval and generation quality, LLM-as-judge pipelines calibrated against human raters, and regression gates in CI. The result is that every prompt, model or retrieval change ships with evidence rather than a hunch. Engagement: 4–8 weeks, from €38,000, One senior engineer plus an evaluation specialist. Teams ship LLM changes the way they shipped code in 2004: someone tries a few prompts, it looks better, it goes out. Then a user finds the case where it now confidently states the opposite of the policy document, and nobody can say when that broke or which change caused it. Evaluation is what turns an LLM feature into an engineering artefact. ## The three layers ### Retrieval evaluation Deterministic, fast and cheap. Given a question and the passage that answers it, does the retriever return that passage in the top _k_ results? This one measurement catches the majority of what teams misdiagnose as "hallucination". If the right passage never reaches the model, no amount of prompt engineering will fix the answer. ### Generation evaluation Given the retrieved context, is the answer supported by it? Are the citations real and do they point at the passage that actually contains the claim? Did the system complete the task the user asked for, or answer a nearby question? These need a mix of deterministic checks and judgement. ### Online evaluation Offline sets go stale. Real users ask things nobody anticipated, and the distribution of questions drifts as the product changes. I wire in feedback capture, sample production traffic for review, and monitor for the drift that tells you the offline set needs refreshing. ## Calibrating the judge Using a language model to grade another language model is practical and, done carefully, reliable. Done carelessly it produces confident numbers that mean nothing. I calibrate: human raters score a stratified sample, the judge scores the same sample, and I report the agreement. Where agreement is strong, the judge runs unsupervised in CI. Where it is weak, I say so and keep humans in the loop rather than reporting a metric I do not trust. The agreement rate itself is a deliverable, because it tells you how much weight the gate deserves. ## What this unlocks Once evaluation exists, several things become possible that were not before. You can switch models on evidence rather than vendor marketing. You can let more engineers touch the prompts, because the gate catches mistakes. You can answer an auditor asking how you know the system performs as claimed. And you can tell the difference between a change that helped and a change that felt like it helped. For clients under the EU AI Act, this layer also does double duty: the accuracy and robustness testing that Article 15 expects is largely the same evidence a good evaluation harness produces anyway. ## AI Agent Development Services Source: https://rubradigital.com/services/ai-agent-development Summary: Rubra builds LLM agent systems that take actions rather than just answering questions: tool and API integration, multi-step planning, state management, human approval checkpoints, and full execution tracing. I am deliberately conservative about where agents belong, because most tasks marketed as agentic are better served by a deterministic workflow. Engagement: 10–16 weeks, from €75,000, Two engineers plus a fractional architect. Agents are the most oversold and most under-engineered category in applied AI right now. The demos are extraordinary. The production deployments are, mostly, narrow, carefully bounded and much less exciting than the demos. That is precisely why they work. ## My bias: fewer agents, better bounded Before building an agent I ask whether the task actually needs one. Most do not. A large share of what gets pitched as agentic automation is a workflow with a language model at two or three decision points, and building it that way makes it cheaper to run, faster, and possible to debug at 3am. When the path really is unknowable in advance, an agent is the right tool. I then spend my effort on the things that determine whether it survives contact with reality. ## What determines whether an agent works **Tool design, not prompt design.** An agent is only as good as the operations available to it. Tools with narrow, typed inputs and informative error messages produce reliable agents; tools that accept free-form strings and fail silently produce agents that loop. Most of my engineering time goes here. **Bounded autonomy.** Every agent runs inside a budget covering steps, wall-clock time and spend, and every irreversible action passes through an approval gate. The gate is a product decision as much as a safety one: users trust a system more when they can see what it is about to do. **Observability from the first commit.** Every run is fully traced and replayable: what the model saw, what it decided, which tool it called, what came back. Without this, debugging an agent is archaeology. **Failure containment.** The failure modes get enumerated during design: wrong tool, right tool with wrong arguments, plausible-but-wrong intermediate conclusion, infinite loop. Each one gets its own containment. The question is never whether the model will get something wrong, only what happens when it does. ## Where I have seen agents deliver Support triage that reads a ticket, checks three internal systems and either resolves it or routes it with a written summary. Vendor and contract review that pulls a document apart against a checklist and flags the clauses a human needs to read. Data reconciliation across systems that disagree, where the agent investigates the discrepancy and proposes the correction for approval. The pattern in all of them: bounded scope, reversible actions, a human on the consequential decision, and a measurable completion rate. ## LLMOps & AI Platform Engineering Source: https://rubradigital.com/services/llmops-platform Summary: Rubra builds the platform layer beneath AI features: a model gateway with failover, prompt and config versioning, distributed tracing, per-team cost attribution and budgets, semantic caching, and deployment pipelines with evaluation gates. It is what turns three separate AI prototypes into a capability the whole organisation can build on. Engagement: 6–12 weeks, from €55,000, One platform engineer plus a fractional architect. The first AI feature ships from a notebook. The third one reveals that three teams have three sets of provider credentials, nobody can explain last month's invoice, and the prompt that runs in production exists only in a deployed container image. The platform layer is what stops that. ## What I build ### A gateway in front of every model call One path to every provider, with routing, failover, rate limiting and credential management in one place. It gives you a point of control for redaction and audit logging, and it makes switching providers a configuration change rather than a migration. ### Prompts as versioned artefacts Prompts, model selection, temperature and retrieval parameters belong in version control with review and staged rollout. Not hard-coded, and not in a database row someone can edit at 5pm on a Friday. Every production response should be traceable to the exact configuration that produced it. ### Tracing that spans the whole request A single user request may touch retrieval, reranking, two model calls and three tools. When it goes wrong, you need the whole path in one view: inputs, outputs, latency and cost at each hop. I build this on OpenTelemetry so it lands in the observability stack you already run. ### Cost as a first-class metric Per-request cost attributed to team, feature and customer; budgets with alerting before the invoice arrives; and the data to answer whether a feature is worth what it costs. Most organisations discover at this point that a small number of requests account for most of the spend, and that a large share of it is avoidable. ### Deployment with evaluation gates Changes flow through CI, run against the evaluation set, and are blocked on regression. Canary rollout for anything touching a live surface. This is the piece that lets you move quickly without the quality of the system depending on who happened to review the pull request. ## When to build it Not for your first AI feature, which would be premature. The right moment is when a second or third team starts building, when spend crosses roughly €3,000 a month, or when your first provider incident makes it clear that a hard-coded endpoint is a single point of failure. In practice most organisations reach that point about six months after their first successful pilot. ## EU AI Act Compliance Consulting Source: https://rubradigital.com/services/eu-ai-act-compliance Summary: Rubra helps organisations classify their AI systems under the EU AI Act, identify obligations by risk tier, and produce the technical evidence the regulation requires: risk management records, data governance documentation, accuracy and robustness testing, logging, and human oversight design. I cover the engineering half; I work alongside your counsel on the legal half. Engagement: 5–10 weeks, from €42,000, One AI architect plus a governance specialist. The EU AI Act is the first comprehensive AI regulation anywhere, and it is already shaping procurement across Europe, including for companies that are not themselves in scope, because their European customers now ask. Most of the obligations are engineering obligations. That is the part I do. ## The classification problem comes first Almost every conversation starts in the wrong place: _is my AI high-risk?_ The question cannot be answered about a technology, only about a use case. The same retrieval system is minimal risk when it helps staff find a policy document and high-risk when it screens candidates for a role. I inventory every AI system you operate, classify each use case against the Act's tiers, and then write down why. That last step is the one that matters. A classification you cannot justify is worse than no classification, because it looks like a decision was made and then cannot be defended. ## Then the obligations, article by article For each system I map exactly which obligations attach and what evidence satisfies them: - **Article 9.** A risk management system that operates across the lifecycle - **Article 10.** Data governance: provenance, representativeness, bias examination - **Article 11 and Annex IV.** Technical documentation - **Article 12.** Automatic logging with sufficient traceability - **Article 13.** Transparency and instructions for use - **Article 14.** Human oversight that is effective, not nominal - **Article 15.** Accuracy, robustness and cybersecurity, with declared metrics For limited-risk systems the burden is much lighter: disclosure that the user is interacting with an AI system, and marking synthetic content. I will tell you plainly when that is all you need. ## The overlap nobody mentions If you have already built a serious evaluation harness, you are most of the way to Article 15 evidence. If you have proper tracing, Article 12 is largely handled. Good engineering practice and AI Act compliance converge to a surprising degree, which means the work is rarely wasted even if your classification later turns out to be lighter than feared. ## What I do not do I am not your lawyers, and I will not give you a legal opinion on classification. I give you a technical classification with documented reasoning, the engineering evidence, and a gap analysis with real effort estimates. Your counsel owns the legal position and I work alongside them. When the Act's harmonised standards and Commission guidance move, I will tell you what changes for your systems. ## AI Strategy & Opportunity Assessment Source: https://rubradigital.com/services/ai-strategy Summary: Rubra runs three to five week AI discovery engagements that produce a ranked portfolio of use cases with feasibility assessments, cost and effort estimates, data readiness findings, and a sequenced roadmap. The most valuable output is usually the list of things I recommend you do not build. Engagement: 3–5 weeks, from €22,000, One principal consultant plus an AI architect. Most organisations do not have an AI capability problem. They have a prioritisation problem: twenty ideas, no shared basis for comparing them, and a budget that will fund three. Discovery is a short engagement that replaces opinion with evidence. ## How it runs **Week one. Inventory.** Structured interviews across the functions that would use or be affected by these systems. I am looking for tasks that are repetitive, text-heavy, currently slow, and where the information needed already exists somewhere in writing. I am also listening for the constraints people mention in passing, which are usually the ones that matter. **Week two. Feasibility probes.** For the strongest candidates I test the riskiest assumption directly against your data. Not a build: a probe. Can I actually extract this field from these documents? Does retrieval find the right passage for these fifty real questions? Is this dataset as complete as everyone believes? Two days of this routinely changes the ranking. **Week three. Economics.** Effort, cost, running cost, time to value, and the organisational change required. The last is the one most often underestimated, and it is why technically successful projects sometimes deliver nothing. **Weeks four and five. Roadmap and readout.** A sequenced plan with decision points, a written recommendation for each candidate, and a session with your leadership team where I present the case against the weak ones as clearly as the case for the strong ones. ## What makes this different from a strategy deck The people running discovery are the people who would build the system. That changes the estimates: an architect who has shipped six retrieval systems knows what the scanned-PDF corpus is going to cost, and will say so in week two rather than discovering it in month four. It also means the recommendation is accountable. I am proposing work I would have to deliver. ## The most valuable output Usually the no list. Nearly every discovery I run identifies at least one use case with visible executive enthusiasm that will not work, because the data does not exist, the accuracy bar cannot be met, or the process around it would have to change in ways nobody has agreed to. Finding that in week three costs a fraction of finding it in month nine. ======================================================================== # INDUSTRIES ======================================================================== ## AI & RAG Consulting for Financial Services Source: https://rubradigital.com/industries/financial-services Summary: Rubra builds LLM and RAG systems for banks, insurers and asset managers across Europe and North America, designed from the start to meet model risk governance, DORA operational resilience, and supervisory expectations on explainability and audit trails. Financial services was the first sector to take LLM governance seriously, largely because it already had the machinery. Model risk management, validation cycles, three lines of defence. The frameworks exist. What they were not built for is a model you did not train, whose behaviour can change when a vendor ships an update. ## What changes with foundation models **You do not own the model.** Traditional validation assumes you can inspect training data and reproduce results. With a hosted foundation model you can do neither. Validation shifts to behavioural evidence: a fixed evaluation set, run continuously, with declared thresholds and alerting when performance moves. **The failure mode is fluent and wrong.** A traditional model fails visibly, producing a number outside plausible bounds. An LLM fails by producing an articulate, well-formatted answer that is incorrect. Controls have to assume plausibility is not evidence. **Behaviour drifts without a release.** Your provider can change model behaviour without your deployment changing at all. Continuous evaluation is not a nicety here; it is the only way you would find out. ## How I work in this sector I build the evaluation harness before the application, because it is the evidence your second line will ask for and the thing that lets you keep shipping afterwards. I keep a human decision point on anything that affects a customer outcome. I instrument retrieval logging so a supervisor can be shown exactly which documents informed a given output. And I design for provider substitutability, because a single hard-coded endpoint is a DORA problem as much as an engineering one. Most banking engagements start with the internal policy assistant: low risk, fast payback, and it builds the governance muscle on a use case where a mistake is inexpensive. The high-risk use cases come second, once the organisation has learned what evidence it actually needs to produce. ## AI & RAG Consulting for Law Firms and Legal Teams Source: https://rubradigital.com/industries/legal Summary: Rubra builds retrieval systems for law firms and in-house legal teams: contract analysis, precedent search, regulatory monitoring and matter research, engineered so that every assertion carries a verifiable citation and the system refuses rather than guesses. Legal is the sector where retrieval quality and citation integrity matter most, and where the consequences of getting them wrong have been most public. ## Citations are a correctness requirement In most applications, a wrong citation is embarrassing. In legal work it is sanctionable. So the system is built so that fabrication is structurally impossible rather than statistically unlikely: - The model may only cite from passages actually retrieved in this request. - Every citation is resolved and validated against the retrieved text before the response is returned. - An unresolvable citation fails the response rather than degrading it. - Refusal is a designed, measured behaviour. The system says the corpus does not cover a question rather than assembling something that reads as if it does. ## Confidentiality has to be architectural Matter-level access control is enforced at retrieval time, so a passage the user is not entitled to read is never retrieved, never enters the prompt, and cannot appear in an answer. Where the firm requires it, inference runs inside the firm's own infrastructure and nothing leaves the tenancy. ## Where the value is The highest-return use case in most firms is not sophisticated: it is finding the firm's own prior work. Decades of precedent, memoranda and negotiated positions sit in a document management system that is nearly impossible to search usefully, and the knowledge of what is in there lives with a handful of senior people. Making that corpus properly retrievable is a lower-risk, higher-payback project than anything involving drafting, and it is the one that partners notice. ## AI & RAG Consulting for Healthcare and Life Sciences Source: https://rubradigital.com/industries/healthcare-life-sciences Summary: Rubra builds LLM and retrieval systems for pharmaceutical, medtech and healthcare organisations: regulatory document search, safety literature review, clinical documentation support and medical information response, designed around validation, traceability and human oversight. Life sciences organisations have a specific and unusual advantage in this work: they already know how to validate a computerised system. The vocabulary of intended use, risk assessment, acceptance criteria and change control is established, and the quality function knows how to apply it. The adjustment is that language models are not deterministic, so acceptance criteria have to be expressed as measured performance on a fixed validation set rather than as exact-output tests. In my experience quality functions accept this readily once it is framed in terms they already use. The conversation goes badly only when engineering presents it as a new category of thing that existing frameworks cannot cover. ## Where I start Almost never with anything clinical. The fastest payback and lowest risk sit in regulatory and quality operations: dossier navigation, SOP retrieval, deviation investigation support. These are document-heavy, currently slow, and a wrong answer is caught by a human who was going to read the source anyway. Clinical use cases come later, if at all, and with the device question resolved first. ## Traceability is the deliverable In this sector the audit trail is not a supporting artefact. It is a large part of what you are buying. Every answer records which documents were retrieved, which versions of them, who asked, and what the system returned. That record is what makes the system defensible in an inspection, and it is designed in from the first sprint rather than added when someone asks for it. ## AI Consulting for B2B SaaS Companies Source: https://rubradigital.com/industries/saas Summary: Rubra helps B2B SaaS companies ship AI features that hold up commercially: multi-tenant retrieval with strict isolation, evaluation that survives customer-specific data, unit economics that work at your price point, and the security evidence enterprise buyers demand. For SaaS companies the hard part of AI features is rarely the model. It is shipping something that works across thousands of tenants whose data you have never seen, at a cost that survives your price point, with security evidence that clears enterprise procurement. ## Evaluation when you cannot see the data Your internal evaluation set is drawn from your own documents. Your customers' data looks nothing like it: different vocabulary, different structure, different quality. A feature that scores well internally can fail badly on a real tenant. The workable pattern is a synthetic evaluation corpus deliberately built to span the variation you see across customers, plus opt-in evaluation with design partners on their real data, plus online metrics that tell you per-tenant quality in production. Without the last one you will hear about failures from churn rather than from monitoring. ## Unit economics before launch Instrument cost per request, per tenant and per feature before the feature is generally available, not after the first invoice surprises someone. Set per-tenant ceilings. Expect the distribution to be heavily skewed. In every deployment I have measured, a small number of accounts drove most of the spend, and knowing which ones changes both pricing and product decisions. ## Isolation is architectural Tenant filtering happens server-side in the retrieval query, derived from the authenticated session. It is tested adversarially in CI. A cross-tenant leak is the one failure in this category that ends customer relationships, and the control for it cannot live in a prompt. ## AI & RAG Consulting for Manufacturing and Energy Source: https://rubradigital.com/industries/manufacturing Summary: Rubra Digital builds retrieval systems for manufacturers and energy companies over technical documentation, maintenance histories, engineering drawings and inspection reports, including the scanned and legacy material where document parsing rather than model choice decides whether the project works. Industrial retrieval projects are decided in the document parsing stage. Model choice barely matters if the maintenance manual is a 1997 scan with the torque specification in a table that OCR renders as a column of unrelated numbers. ## What makes this sector distinct **The documents are hostile.** Scans of varying quality, engineering drawings where meaning lives in callouts and title blocks, tables that carry the actual answer, and mixed languages within one document set. Layout-aware extraction is mandatory, not an optimisation. **Versioning is the real question.** "What is the torque specification?" has a different answer depending on the revision, the serial range and whether a service bulletin superseded it. A system that returns the right value from the wrong revision is worse than one that returns nothing, so version and effective date are retrieval metadata, not presentation detail. **The stakes are physical.** A wrong answer about a procedure can injure someone. These systems are designed to cite precisely and refuse clearly, and to be positioned as a way to find the authoritative document rather than as a substitute for reading it. **Connectivity is not guaranteed.** Plant floors, offshore platforms and remote sites need graceful degradation rather than an error page. ## How I scope it I ask for a sample of your worst documents in week one, not the clean ones. The recoverable proportion of that sample sets the scope of the whole project, and it is far better to know in week one than in month three. Where OCR quality is not good enough, I say so and scope the remediation separately rather than quietly building a system that will return confident answers from garbled source text. ## AI & RAG Consulting for Government and Public Bodies Source: https://rubradigital.com/industries/public-sector Summary: Rubra delivers AI and retrieval systems for government departments, agencies and public bodies across Europe and Canada, designed around data sovereignty, algorithmic transparency obligations, procurement requirements and the explainability that public accountability demands. Public sector AI work carries a constraint private sector work does not: the people affected by the system did not choose to interact with it, and cannot go elsewhere. That changes what an acceptable error rate means, and it changes how much of the system's reasoning has to be visible. ## Design principles I hold to here **Decision support, never decision making.** Where an outcome affects a citizen, a human makes it, sees what informed it, and can override it. The system's job is to bring the right material to that person's attention. **Explainability that a citizen could follow.** Not a feature importance chart, but a plain statement of which documents informed an answer, in language a member of the public could read. Retrieval systems are well suited to this, because the citations are the explanation. **Sovereignty settled before architecture.** EU-only, nationally qualified, or fully on-premise are different systems, not different deployment settings. **Built to be published.** Assume the evaluation results, the limitations and the design will be disclosed. Document accordingly. ## Where public bodies get the most value Consistently, the least controversial use cases: making existing guidance findable. Departments hold decades of policy, precedent and internal interpretation that new staff take years to learn and retiring staff take with them. Retrieval over that corpus improves consistency of decisions without touching the decisions themselves. It is also the use case where oversight bodies raise the fewest objections, because nothing about the decision process changes except how quickly the right guidance reaches the person making it. ======================================================================== # INSIGHTS ======================================================================== ## The EU AI Act: what engineering teams actually have to build Source: https://rubradigital.com/insights/eu-ai-act-what-engineering-teams-actually-need-to-do Summary: Most EU AI Act coverage is written for lawyers. This is the engineering view: which articles translate into logging, evaluation, documentation and oversight work, what each one concretely requires, and how much of it a well-run team is already doing. _Engineering guidance, not legal advice._ Most writing about the EU AI Act is aimed at legal and compliance teams. That leaves engineers with a set of obligations expressed in a vocabulary they do not use and no clear sense of what to build. Here is the translation. ## Article 9: risk management system **What it says:** a continuous, iterative risk management process across the system's lifecycle. **What you build:** a living risk register tied to your actual system, listing identified risks, the mitigations in place, and the residual risk. Updated when the system changes, not annually. In practice this works best as a document in the repository, reviewed in the same cadence as architecture decisions. ## Article 10: data governance **What it says:** training, validation and testing data must be relevant, representative, and examined for bias. **What you build:** documented provenance for every corpus in your retrieval index: where it came from, what it covers, what it does not cover, and the known gaps. For retrieval systems, the important and often-missed part is representativeness of your _evaluation_ set: if it only contains questions from one department, you cannot claim system-wide performance. ## Article 11 and Annex IV: technical documentation **What it says:** documentation sufficient to assess conformity. **What you build:** a system description, architecture, design choices and their rationale, performance metrics, and known limitations. Teams that keep architecture decision records are most of the way there already. Teams that do not will find this the most tedious part of the programme. ## Article 12: logging **What it says:** automatic recording of events over the system's lifetime, enabling traceability. **What you build:** for a retrieval system, that means the query, the retrieved passages and their identifiers, the model and configuration version, the output, the user and the timestamp. With a defined retention period that survives contact with your privacy team. Good tracing largely satisfies this. ## Article 13: transparency and instructions for use **What it says:** deployers must be able to interpret and use the output appropriately. **What you build:** documentation of what the system does, what it does not do, its measured accuracy, its known failure modes, and the conditions under which it should not be relied on. The failure modes section is the one that matters and the one most likely to be written vaguely. ## Article 14: human oversight **What it says:** effective oversight by natural persons. **What you build:** a human decision point on consequential outputs, with enough information presented for that person to actually exercise judgement, which means showing the retrieved sources rather than only the answer. Design against automation bias: an interface that shows a confident answer and an approve button produces rubber-stamping, and a regulator will say so. ## Article 15: accuracy, robustness, cybersecurity **What it says:** appropriate levels of accuracy, with declared metrics, plus robustness and security. **What you build:** the evaluation harness. Declared metrics, a fixed test set, continuous measurement, adversarial testing including prompt injection, and monitoring for degradation. **If you have built serious evaluation, this article is largely covered**, and that is the strongest argument for doing evaluation work regardless of your classification. ## The convergence Read as a whole, the technical articles describe a system that is measured, logged, documented and supervised. That is not a regulatory invention; it is what a well-engineered system looks like. The organisations finding the Act painful are generally the ones who shipped AI features without evaluation, tracing or documentation. The work they are doing now is work they needed anyway. The regulation set a deadline for it. ## Most RAG failures are retrieval failures Source: https://rubradigital.com/insights/most-rag-failures-are-retrieval-failures Summary: In the RAG systems I audit, most wrong answers are cases where the correct passage never reached the model. That is a retrieval bug, not a hallucination, and prompt engineering cannot fix it. This piece explains how to diagnose the difference and what to change. Almost every RAG audit I run starts the same way. The team reports that the model hallucinates. They have iterated on the system prompt for weeks. They are considering fine-tuning. I ask to see the retrieved context for twenty failed answers. In most cases, the correct information was not there. ## This is not hallucination A language model given context that does not contain the answer, and asked to answer from context, is in an impossible position. It will usually produce something plausible from whatever it was given. That is not the model malfunctioning. It is the model doing exactly what a bad retrieval step set it up to do. The distinction matters because the fixes are entirely different. Retrieval problems are fixed with chunking, hybrid search, reranking and query rewriting. None of those live in the prompt. ## The diagnostic Log retrieved passages with every response. Then sample failures and answer one question for each: **was the correct information in the context window?** Sort the failures into two piles. The proportions tell you where to spend the next month. In my experience the retrieval pile is consistently larger, and frequently much larger. ## Why teams look at the prompt first The prompt is visible, editable and gives immediate feedback. Retrieval is opaque. You cannot see what the index returned without instrumenting it, and most teams have not. There is also a measurement gap. Prompt changes get judged by trying a few questions and forming an impression. Retrieval quality needs a labelled set to measure at all. Teams optimise what they can see. ## What actually moves retrieval quality **Hybrid search.** Pure vector search reliably misses exact strings: part numbers, error codes, proper nouns, statutory references. Adding BM25 closes a category of failure that no amount of embedding tuning addresses. **Reranking.** A cross-encoder over the top 50 candidates is, in my benchmarks, the single highest-return change available to most systems. It routinely moves recall@5 by five to fifteen points. **Chunk context headers.** Prepending the document title and heading path to every chunk before embedding costs nothing and consistently improves retrieval on corpora where passages contain ambiguous references. **Query rewriting.** Users do not phrase questions the way documents phrase answers. Rewriting the query, or generating several variants and merging results, helps materially on conversational interfaces. ## Measure it, then argue about it None of the above is worth doing on faith. Build a labelled set of 150 to 300 real questions with known-correct passages, then measure recall@5 before and after each change. Teams that do this stop arguing about which model is better and start making decisions with numbers attached. That shift is usually worth more than any individual technique on the list. ## Your evaluation set is the real product Source: https://rubradigital.com/insights/evaluation-is-the-product Summary: Models change, frameworks are replaced, architectures get rewritten. A well-built labelled evaluation set survives all of it and makes every future decision cheaper. It is the highest-leverage artefact in an AI project and the one most often cut. Ask a team what their AI project produced and they will describe an application. Ask them in eighteen months what still has value and the honest answer is usually: the evaluation set. ## What outlasts what In three years the model you launched on will be several generations old. The framework will have been replaced or abandoned. The retrieval architecture will have been rewritten at least once. The set of 250 real questions, each paired with the passage that answers it and validated by someone who knows the domain, is still correct. It was correct before your architecture and it will be correct after it. It is the only artefact in the project that does not depreciate. ## What it makes possible **Model migration becomes an afternoon.** A new model appears. Run the suite, compare the numbers, decide. Without it, migration is a multi-week exercise in subjective comparison that nobody trusts. **More people can safely change things.** With a gate in CI, a junior engineer can adjust a prompt. Without one, changes are bottlenecked on whoever has the best intuition, which does not scale and does not survive that person leaving. **Vendor claims become checkable.** Someone will tell you their platform is more accurate. You can find out in an hour. **Regulators and auditors get a real answer.** "How do you know it performs as claimed?" has a document behind it. ## Why it gets cut It is expensive in the currency projects are shortest on: subject-matter expert attention. Getting a specialist to label which passage correctly answers each of 250 questions is days of a busy person's time, and it produces nothing demonstrable. Meanwhile the prototype already looks impressive. The pressure to move to features is real, and evaluation is the easiest thing to defer. ## Build it early, deliberately **Real questions.** From search logs, tickets, or a week of asking users what they actually want to know. Invented questions cluster on the easy cases. **Stratified.** Simple lookups, multi-document synthesis, near-miss distractors, ambiguous phrasings, and deliberately unanswerable questions. Roughly balanced, not naturally distributed. **Expert-validated ground truth.** The expensive, unavoidable part. **Version controlled.** It changes as the product changes; you want the history. **150 to 300 cases.** Curated beats scraped by a wide margin. ## The test If your team cannot answer "did last week's change make the system better?" with a number, you do not have an evaluation set. You have opinions with a deployment pipeline attached.