ADR-0029: A model’s tier is chosen by its job, cheapest first, and moved up only on evidence
- Status: Accepted
- Date: 2026-11-11
- Scope: core (any product that runs language-model features on more than one model tier)
Context
The agents platform started every customer-facing agent on a mid tier because it felt safer. Measured on one bounded
job — a pizzeria taking orders, the same real customer messages replayed on both tiers — the cheapest tier priced
every order right, kept every rule and invented nothing, at about a twentieth of the cost; it was weaker only at the
edges (an unavailable item left out without saying so). It works because what must never go wrong — prices, rules,
confirmations — is enforced by code (core 0019-llm-features-prompts-shape-code-enforces), not by the model’s
judgement (agents 0024-a-customer-agents-model-is-chosen-by-its-job). That does not hold where judgement is the
product, where a cheaper model fails quietly and no tool catches it. The tier is the largest single factor in a feature’s
running cost (core 0026-llm-usage-is-metered-in-money-pinned-to-a-reference-model).
Decision
1. Each feature’s tier is a decision about its job
A bounded job whose risky parts code guards starts on the cheapest tier. A higher tier is for jobs where judgement
is the product (advice, diagnosis, many interacting conditions, long documents) or long work over many steps. The
choice and its reason are written down with the feature.
2. Moving up is a measured change
A tier changes like a prompt does: on the eval, before and after (core
0022-llm-changes-are-measured-on-a-held-out-eval). When a cheap tier answers wrong, first look for the cause in
data, a rule or a tool description; only when none is the cause, run the same messages on the next tier and keep
what works.
3. Production signals suggest; they never switch
Signals from real use (a share of conversations handed to a person, failed runs) may raise a suggestion to the
person who owns the feature. Nothing changes a tier on its own.
4. People see tiers in their own words
Wherever a person chooses or sees a tier, it is named by what it means to them (economical, balanced, advanced),
never by a model id, which changes with every model release.
Consequences
- Most features run at a fraction of the cost, and a tier upgrade has a measured reason.
- A model swap is configuration: the tier names stay, the ids behind them change.
- Accepted: a cheap tier is a little more variable run to run; the eval’s spread (core
0022-llm-changes-are-measured-on-a-held-out-eval §4) is what tells noise from a regression.
Related
- Agents 0024-a-customer-agents-model-is-chosen-by-its-job — FAST by default,
MODEL_STRUGGLINGas a suggestion - 0026-llm-usage-is-metered-in-money-pinned-to-a-reference-model — why a cheaper tier costs fewer credits