ADR-0022: An LLM feature changes only against a measurement — fixed scenarios, a held-out set, a judge that sees the evidence

  • Status: Accepted
  • Date: 2026-11-06
  • Scope: core (any backend feature whose behaviour is a prompt, a tool description, a model or a
    reasoning setting)

Context

The behaviour of a language-model feature lives in text and settings: the system prompt, each
tool’s description, the model, its effort. None of it is covered by a compiler or a unit test, and
a single conversation proves nothing, because the same question gets a different answer on the next
run.

Expenses’ assistant went through several rounds of prompt and model changes (expenses
0031-the-assistant-investigates-before-it-answers). What it learned about judging those changes
holds for any model, provider or product:

  • a rule added to fix one bad answer routinely broke others, and only a fixed set of questions,
    re-run, showed it;
  • once the authors had read a scenario’s answers and tuned against them, its score stopped meaning
    anything about the questions nobody wrote;
  • a judge that saw only the answer passed confident answers built on numbers the assistant never
    looked up;
  • the same suite, run twice on the same code, moved by several points, so a small gain was
    usually noise;
  • a change that improved answers could also raise the cost of a turn by a quarter, and nothing
    showed it unless the run reported it.

Decision

1. Every behaviour change is run against a fixed suite, before and after

A change to the prompt, a tool description, the model or its effort is measured on the same
scenarios before and after. The scenarios run against a seeded account whose data is known, so
each one has an answer key. Multi-turn scenarios, and ones where the data changes between turns,
belong in the suite: they catch what a single question cannot.

2. The suite is split: TRAIN to tune against, TEST held out

The authors read TRAIN answers and tune against them. TEST answers are not read; only TEST’s
aggregate is looked at. A TEST scenario whose answers were read moves to TRAIN, and gets a
replacement. TEST scenarios put the intended behaviour in situations no TRAIN scenario has, so
they measure whether a change generalises.

3. The judge sees the answer key and what the model looked up

A grader compares the answer with the key and with the tool calls and results behind it. An
answer is correct only if its figures came from a lookup or from arithmetic on looked-up figures.
The pass is strict: a right conclusion with an invented figure fails.

4. A result is read against its spread and its cost

Run each scenario more than once and know the run-to-run spread before calling a difference real.
Every run reports cost per turn alongside the score; a gain is weighed against what it costs on
every turn in production.

5. Evaluation traces never reach production

The suite needs to see tool inputs and results (the evidence in §3). Whatever exposes them is on
in development only: it exposes queries and rows.

Consequences

  • A prompt edit is a measured change, not a hunch. Trimming a section that looks redundant is
    measured too: in Expenses the general method section moved the scenarios more than any single
    rule did.
  • Runs cost real money. Run a subset (one scenario, one suite) while iterating and the full suite
    before merging.
  • Accepted: TEST shrinks as it gets read and must be replenished. Writing a scenario that tests
    a behaviour in a new situation is the expensive part.
  • The suite and its seed are code: they live in the repo and change with it.

Amendment — 2026-11-11: the eval’s own cost is budgeted, and the judge is most of it

A day of tuning the agents platform ran nineteen rounds and spent more than the account held: the grader, a top-tier
model at high effort over every run, was about 60 % of it — more than the agents it graded. The suite is a
production-grade spender and is treated as one:

  • A round states its estimated cost before it starts and stops at a cap set for it; it never runs past the
    account’s balance unseen.
  • While iterating, run only the scenarios the change touches; the whole suite, TRAIN and TEST, runs before the
    change merges.
  • The judge’s model and effort are chosen like any model’s (core 0029-a-models-tier-is-chosen-by-its-job-cheapest-first):
    a cheaper judge is adopted after re-grading a past round with it and finding the same verdicts.
  • Every call the suite makes is metered with the rest (core 0026-llm-usage-is-metered-in-money-pinned-to-a-reference-model),
    the judge included; a spender outside the meter is how a balance runs out by surprise.