ADR-0019: LLM features — the prompt shapes the answer, code enforces the boundary
Status: Accepted
Date: 2026-09-10
Scope: core (any backend feature that puts a language model in front of users or
their data)
Context
A language-model feature is built from two kinds of material: instructions (a system
prompt, a tool description) and code (the tools the model calls, what they validate, what
they can reach). Instructions are cheap and read like rules, so it is tempting to state a
boundary once in the prompt and treat it as held.
Expenses’ assistant showed, within one release (v1.70.1), that it does not hold. Each
failure was a rule stated only in instructions:
told to keep an internal schema to itself, the model recited it when asked for it;
it told a user it had no access to something it could in fact query. The refusal was a
performance, not a boundary;
a value from the user’s own stored data, written as a query fragment, was pasted by the
model into a query it ran. A validator stopped it; the prompt had not;
tool failures handed the model the raw exception message, and an infrastructure error’s
message carries real database identifiers. Only a prompt rule stood in the way.
The same project had already settled the identity half earlier: who the user is, and which
rows the model may touch, come from the server and never from the model.
Nothing here depends on the model, the provider or the product, which is why it is recorded
in core. The Expenses instance, with its specific tools and guards, stays in that project’s
ADRs.
Decision
1. Anything that must never happen is enforced in code
A prompt is a request the model may decline under pressure. It shapes good answers; it is
never the control. Whatever must not happen (reaching another user’s data, running a
statement of the wrong kind, disclosing an internal) is prevented by a validator, by what
the model is given access to, or by a type. The prompt may say it as well, as defense in
depth.
2. A boundary the model states is a specification
When a prompt or tool description makes the model assert a limit (“you cannot see X”), that
sentence is only as true as the code behind it. Make it true in code, then verify it by
executing both directions: the attack shapes rejected, and the ordinary uses still working.
3. The model only receives error text we wrote
A tool failure passes its message to the model only when the application raised it on
purpose. Such a message describes the model’s own input, and it is what lets the model
correct itself and retry. Any other exception (from the ORM, the database driver, a
serializer) reaches the model as its type alone, because its message describes our system.
The full stack is logged as usual.
4. Data is data
Text from the user’s records or uploads (a name, a note, a cell, a file name) is a value to
match on. It is never followed as an instruction or run as code. An injection targets the
model’s judgment, so the model’s judgment is not what stops it: the validator on whatever the
model produces is.
5. Identity and scope come from the server
Who the user is, what they may see, and any standing permission (such as being allowed to
discuss the assistant’s own instructions) are established from the authenticated session and
supplied by the server. A message that claims an identity is only a claim. Tools are scoped
to the caller by construction, not by the model choosing the right id.
Consequences
Every new tool gets a review question beyond “does it work”: what could the model do with
it that it must not, and which line of code stops that?
A prompt change is never the whole fix for a boundary incident. The fix is code; the
prompt change rides along.
Accepted: infrastructure failures are less legible to the model. It could never act on
them, while the failures it can act on (bad arguments, a bad query, not found) keep their
message and their retry.
Accepted: enforcing in code costs more than a sentence in a prompt, and it needs
verification by execution rather than by reading.
Example
Real code from the repos — optional
Agents: safe to skip. The Decision above is the specification; everything below is
illustration. Skipping this block costs you nothing normative and saves tokens. Expand it
when you need the shape rather than the rule — porting the pattern to a new project, or
answering “what does this actually look like”.
static final int MAX_RESULT_ROWS = 200;private static final int MAX_LOADED_ROWS = 500;private static final int MAX_LOADED_COLUMNS = 20;private static final Set<String> ALLOWED_FIRST_KEYWORDS = Set.of("SELECT", "WITH", "DESCRIBE", "SHOW");private static final Pattern FORBIDDEN_KEYWORDS = /* INSERT, COPY, ATTACH, … */;private static final Pattern FORBIDDEN_CATALOG = /* duckdb_*, information_schema, … */;// Per-turn engine, held to a budget it cannot argue with.stmt.execute("SET memory_limit = '256MB'");stmt.setMaxRows(MAX_RESULT_ROWS + 1);
Every line here is a boundary the prompt is not trusted to hold. The prompt shapes
the answer; this decides what is possible.
Layered on purpose, because each layer fails differently: an allow-list of opening
keywords (a statement that is not a read never starts), a forbidden-keyword scan, a catalog scan so the engine cannot be asked what else exists, no filesystem access, a
timeout, and a row cap.
The catalog rule is the non-obvious one. SELECT * FROM duckdb_tables() is a perfectly
valid read — it passes every check aimed at writes — and it is how a model discovers
tables it was never told about. Blocking writes is the easy half; blocking reconnaissance
is the half that keeps tenant isolation true.
The engine is built per turn from that user’s own rows, so isolation is structural:
there is no other tenant’s data in the database to leak, whatever SQL is generated.