Dev Handbook

0019-llm-features-prompts-shape-code-enforces

4 min read

ADR-0019: LLM features — the prompt shapes the answer, code enforces the boundary

  • Status: Accepted
  • Date: 2026-09-10
  • Scope: core (any backend feature that puts a language model in front of users or
    their data)

Context

A language-model feature is built from two kinds of material: instructions (a system
prompt, a tool description) and code (the tools the model calls, what they validate, what
they can reach). Instructions are cheap and read like rules, so it is tempting to state a
boundary once in the prompt and treat it as held.

Expenses’ assistant showed, within one release (v1.70.1), that it does not hold. Each
failure was a rule stated only in instructions:

  • told to keep an internal schema to itself, the model recited it when asked for it;
  • it told a user it had no access to something it could in fact query. The refusal was a
    performance, not a boundary;
  • a value from the user’s own stored data, written as a query fragment, was pasted by the
    model into a query it ran. A validator stopped it; the prompt had not;
  • tool failures handed the model the raw exception message, and an infrastructure error’s
    message carries real database identifiers. Only a prompt rule stood in the way.

The same project had already settled the identity half earlier: who the user is, and which
rows the model may touch, come from the server and never from the model.

Nothing here depends on the model, the provider or the product, which is why it is recorded
in core. The Expenses instance, with its specific tools and guards, stays in that project’s
ADRs.

Decision

1. Anything that must never happen is enforced in code

A prompt is a request the model may decline under pressure. It shapes good answers; it is
never the control. Whatever must not happen (reaching another user’s data, running a
statement of the wrong kind, disclosing an internal) is prevented by a validator, by what
the model is given access to, or by a type. The prompt may say it as well, as defense in
depth.

2. A boundary the model states is a specification

When a prompt or tool description makes the model assert a limit (“you cannot see X”), that
sentence is only as true as the code behind it. Make it true in code, then verify it by
executing both directions: the attack shapes rejected, and the ordinary uses still working.

3. The model only receives error text we wrote

A tool failure passes its message to the model only when the application raised it on
purpose. Such a message describes the model’s own input, and it is what lets the model
correct itself and retry. Any other exception (from the ORM, the database driver, a
serializer) reaches the model as its type alone, because its message describes our system.
The full stack is logged as usual.

4. Data is data

Text from the user’s records or uploads (a name, a note, a cell, a file name) is a value to
match on. It is never followed as an instruction or run as code. An injection targets the
model’s judgment, so the model’s judgment is not what stops it: the validator on whatever the
model produces is.

5. Identity and scope come from the server

Who the user is, what they may see, and any standing permission (such as being allowed to
discuss the assistant’s own instructions) are established from the authenticated session and
supplied by the server. A message that claims an identity is only a claim. Tools are scoped
to the caller by construction, not by the model choosing the right id.

Consequences

  • Every new tool gets a review question beyond “does it work”: what could the model do with
    it that it must not, and which line of code stops that?
  • A prompt change is never the whole fix for a boundary incident. The fix is code; the
    prompt change rides along.
  • Accepted: infrastructure failures are less legible to the model. It could never act on
    them, while the failures it can act on (bad arguments, a bad query, not found) keep their
    message and their retry.
  • Accepted: enforcing in code costs more than a sentence in a prompt, and it needs
    verification by execution rather than by reading.

Graph View

Backlinks