---
title: Your evaluation set is the real product
source: https://rubradigital.com/insights/evaluation-is-the-product
site: Rubra Digital
description: The labelled evaluation set outlasts your model, your framework and probably your architecture. It is the most durable asset an AI project produces.
published: 2026-05-12
updated: 2026-06-30
author: Your Name
topic: evaluation
---

# Your evaluation set is the real product

**Summary:** Models change, frameworks are replaced, architectures get rewritten. A well-built labelled evaluation set survives all of it and makes every future decision cheaper. It is the highest-leverage artefact in an AI project and the one most often cut.

Ask a team what their AI project produced and they will describe an application.
Ask them in eighteen months what still has value and the honest answer is
usually: the evaluation set.

## What outlasts what

In three years the model you launched on will be several generations old. The
framework will have been replaced or abandoned. The retrieval architecture will
have been rewritten at least once.

The set of 250 real questions, each paired with the passage that answers it and
validated by someone who knows the domain, is still correct. It was correct
before your architecture and it will be correct after it.

It is the only artefact in the project that does not depreciate.

## What it makes possible

**Model migration becomes an afternoon.** A new model appears. Run the suite,
compare the numbers, decide. Without it, migration is a multi-week exercise in
subjective comparison that nobody trusts.

**More people can safely change things.** With a gate in CI, a junior engineer
can adjust a prompt. Without one, changes are bottlenecked on whoever has the
best intuition, which does not scale and does not survive that person leaving.

**Vendor claims become checkable.** Someone will tell you their platform is more
accurate. You can find out in an hour.

**Regulators and auditors get a real answer.** "How do you know it performs as
claimed?" has a document behind it.

## Why it gets cut

It is expensive in the currency projects are shortest on: subject-matter expert
attention. Getting a specialist to label which passage correctly answers each of
250 questions is days of a busy person's time, and it produces nothing
demonstrable.

Meanwhile the prototype already looks impressive. The pressure to move to
features is real, and evaluation is the easiest thing to defer.

## Build it early, deliberately

**Real questions.** From search logs, tickets, or a week of asking users what
they actually want to know. Invented questions cluster on the easy cases.

**Stratified.** Simple lookups, multi-document synthesis, near-miss distractors,
ambiguous phrasings, and deliberately unanswerable questions. Roughly balanced,
not naturally distributed.

**Expert-validated ground truth.** The expensive, unavoidable part.

**Version controlled.** It changes as the product changes; you want the history.

**150 to 300 cases.** Curated beats scraped by a wide margin.

## The test

If your team cannot answer "did last week's change make the system better?" with
a number, you do not have an evaluation set. You have opinions with a deployment
pipeline attached.

## Frequently asked questions

### How much of an AI project budget should go to evaluation?

Between 15 and 25 percent for a first build. It is consistently the line item that gets cut when budgets tighten, and consistently the one that determines whether the team can safely change anything after launch. Cutting it does not save money. It defers a larger cost to the point where someone needs to change a prompt and nobody can tell whether it helped.


---

Source: https://rubradigital.com/insights/evaluation-is-the-product
Rubra Digital. Independent LLM and RAG consulting for regulated and document-heavy organisations in Europe and North America.
Contact: hello@rubradigital.com