ADR-0027: Instructions teach a method, tools carry their own rules, and output checks accept correct work
- Status: Accepted
- Date: 2026-11-06
- Scope: core (any backend feature built on a tool-using language model)
Context
0019-llm-features-prompts-shape-code-enforces settled what a prompt cannot do: hold a
boundary. This ADR records what Expenses learned about what a prompt should do, and about the
checks and runtime around it (expenses 0020-assistant-truthfulness-guards,
0031-the-assistant-investigates-before-it-answers):
- a “which tool, when” decision tree in the prompt routed the model down one path whatever the
question, and duplicated the tool descriptions until the two disagreed; - a rule added for one bad answer fixed that answer and rarely carried to the next question;
general habits of a careful analyst did, and measured better than any single rule; - a rule stated in one place (the extractor’s prompt) leaked through another (the assistant’s own
tool), until it was stated in each place the behaviour could happen; - the model answered follow-ups badly because it kept its prose but not what it had looked up;
- a check that required a source for every derived number fired on half of all turns, mostly
correct ones, and raised cost per turn by a quarter; - a thinking model rejects a turn whose reasoning blocks were rebuilt, and is silent for longer
than a proxy’s idle timeout while it reasons.
Decision
1. The prompt teaches a method; it does not route
The system prompt holds only what every answer needs: how to work (investigate before answering,
take a total apart, compare like with like, chase a disagreement, look for the record another
implies, say only what was checked, reread the draft and correct openly), what the domain’s terms
mean, and safety. It never holds a tool-routing table. A new failure is first a sharper habit,
and only then a rule.
2. A rule about one tool lives in that tool’s description
When to call it, how to read its fields, its procedure. A rule that must bind in several places is
stated in each, and says so.
3. The model keeps what it looked up, for a few turns
An answer is stored with the tool calls and results behind it, cut to a budget, and replayed as
real tool pairs for the last few exchanges, dropped in steps so the cache is rewritten rarely.
It is retained for days, not forever, never sent to a client, and erased with the data it copied.
4. An output check accepts correct work and fires once
A check on the draft (for example: every exact figure is in a result or is arithmetic on figures
the reply and the user gave) accepts what a careful reader would accept. When it fires, the draft
goes back once, unseen by the user, with a note saying what to fix. Measure its firing rate and its
cost (core 0022-llm-changes-are-measured-on-a-held-out-eval); a check that fires on correct
answers is a cost, not a guard.
5. Thinking is replayed exactly within a turn and never across turns
Within a tool loop the model’s turn goes back block for block, reasoning included. Across turns it
is dropped, since a block is bound to the prefix that produced it. A provider refusal fails the
turn visibly. A streaming turn sends keep-alives on its own clock while the model reasons.
6. A reasoning trace is kept only where the privacy policy allows it
Storing a summary of the model’s reasoning helps debugging and is a copy of the user’s data and
of what was inferred about them. Keep it for the operator’s own sessions, or for others only once
the privacy policy says so and access is audited.
Consequences
- Prompts get shorter and more general; behaviour carries to questions nobody wrote a rule for.
- Tool descriptions grow, and sit in the cached prefix
(0025-prompt-cache-spend-every-breakpoint-most-stable-first). - Accepted: kept evidence costs context on every follow-up; the step-wise cut bounds it.
- Accepted: a tolerant check misses some wrong arithmetic that happens to match a sum of nearby
figures. Catching the gross errors at a small cost beat catching all of them at a large one.
Amendment — 2026-11-11: a failure is fixed at the lowest level that removes it
On the agents platform, rewriting the instructions to fix observed failures measured flat on the held-out set
(core 0022-llm-changes-are-measured-on-a-held-out-eval), while changes lower down moved the numbers. A failure is
fixed at the first level of this ladder that removes it, and measured:
- Code and data — a guard, a rule enforced at the tool that acts, a fact stored as data. A fact the model needs on
every turn (is the business open now?) is written into the turn’s context by code, not left in a document the
model must think to open. - What the tool result says — the result states what happened in words the model cannot misread: “sent” was read
as “booked” until the result said whether the other system answered and what that means. A result shows the state,
not what is still missing: a list of missing items reads as the next question to ask. - The tool’s description (§2).
- The method (§1), last, and capped: a new habit needs a failure no habit covers, seen in more than one
situation, and a habit whose removal moves nothing goes.
A problem any user of the feature would hit belongs to the feature, never patched into one account’s own
instructions, where it would drift from the platform.
Related
- 0019-llm-features-prompts-shape-code-enforces — what a prompt cannot do
- 0022-llm-changes-are-measured-on-a-held-out-eval — how each of these was judged
- 0023-model-facing-data-is-a-catalog — where tool rules attach
- Expenses 0031-the-assistant-investigates-before-it-answers and
0020-assistant-truthfulness-guards — the worked examples (DraftFigures,ChatEvidence,
ChatHistoryWindow, keep-alive)