--- title: LLM Evaluation & Quality Engineering source: https://rubradigital.com/services/llm-evaluation site: Rubra Digital description: Turn "it seems better" into a number. I build evaluation harnesses, labelled test sets and CI gates so you can change an LLM system without breaking it. updated: 2026-07-02 duration: 4–8 weeks starting_price: from €38,000 team: One senior engineer plus an evaluation specialist --- # LLM Evaluation & Quality Engineering **Summary:** Rubra builds LLM evaluation systems: labelled test sets drawn from real usage, task-specific metrics for retrieval and generation quality, LLM-as-judge pipelines calibrated against human raters, and regression gates in CI. The result is that every prompt, model or retrieval change ships with evidence rather than a hunch. Teams ship LLM changes the way they shipped code in 2004: someone tries a few prompts, it looks better, it goes out. Then a user finds the case where it now confidently states the opposite of the policy document, and nobody can say when that broke or which change caused it. Evaluation is what turns an LLM feature into an engineering artefact. ## The three layers ### Retrieval evaluation Deterministic, fast and cheap. Given a question and the passage that answers it, does the retriever return that passage in the top _k_ results? This one measurement catches the majority of what teams misdiagnose as "hallucination". If the right passage never reaches the model, no amount of prompt engineering will fix the answer. ### Generation evaluation Given the retrieved context, is the answer supported by it? Are the citations real and do they point at the passage that actually contains the claim? Did the system complete the task the user asked for, or answer a nearby question? These need a mix of deterministic checks and judgement. ### Online evaluation Offline sets go stale. Real users ask things nobody anticipated, and the distribution of questions drifts as the product changes. I wire in feedback capture, sample production traffic for review, and monitor for the drift that tells you the offline set needs refreshing. ## Calibrating the judge Using a language model to grade another language model is practical and, done carefully, reliable. Done carelessly it produces confident numbers that mean nothing. I calibrate: human raters score a stratified sample, the judge scores the same sample, and I report the agreement. Where agreement is strong, the judge runs unsupervised in CI. Where it is weak, I say so and keep humans in the loop rather than reporting a metric I do not trust. The agreement rate itself is a deliverable, because it tells you how much weight the gate deserves. ## What this unlocks Once evaluation exists, several things become possible that were not before. You can switch models on evidence rather than vendor marketing. You can let more engineers touch the prompts, because the gate catches mistakes. You can answer an auditor asking how you know the system performs as claimed. And you can tell the difference between a change that helped and a change that felt like it helped. For clients under the EU AI Act, this layer also does double duty: the accuracy and robustness testing that Article 15 expects is largely the same evidence a good evaluation harness produces anyway. ## What you get - Labelled evaluation set built from real user queries - Retrieval metrics: recall@k, MRR, nDCG against known-correct passages - Generation metrics: groundedness, citation accuracy, task completion - LLM-as-judge pipeline calibrated against human raters - CI integration with pass/fail gates and trend dashboards - Online evaluation: user feedback capture and drift monitoring ## Outcomes - Every change measured before it reaches users - Regression caught in CI, not in production - A defensible quality record for auditors and regulators ## Frequently asked questions ### How do you evaluate an LLM system objectively? By splitting the problem. Retrieval is evaluated against known-correct passages using recall@k, mean reciprocal rank and nDCG, all of which are deterministic and cheap to run. Generation is evaluated on groundedness, citation accuracy and task completion, using a mix of deterministic checks and an LLM judge calibrated against human ratings on a sample. The calibration step is what makes the judge trustworthy. Without it you have a second opinion with no error bars. ### How many test cases do we need? Fewer than most teams assume. A well-chosen set of 150 to 300 cases, stratified across question types and difficulty, detects meaningful regressions reliably. Thousands of cases scraped without curation are worse than a few hundred chosen deliberately, because they cluster on easy queries and hide failures in the tail that actually matter. ### Is LLM-as-judge reliable enough for production gates? It is once you calibrate it. I measure agreement between the judge and human raters on a held-out sample, report the agreement rate honestly, and only use the judge for the metrics where agreement is high enough to act on. Where agreement is poor, which is usually on subjective quality rather than factual grounding, a human stays in the loop and I say so rather than pretending the number means something. ### Can you evaluate a system you did not build? Yes. Roughly half of my evaluation work is on systems built in-house or by another supplier. I often start with an audit: run a fresh evaluation set against the current system and report where it actually stands, which is frequently the first honest measurement the team has had. --- Source: https://rubradigital.com/services/llm-evaluation Rubra Digital. Independent LLM and RAG consulting for regulated and document-heavy organisations in Europe and North America. Contact: hello@rubradigital.com