ADR-0025: Prompt caching — order the request most-stable-first, spend every breakpoint, keep the bytes stable
- Status: Accepted
- Date: 2026-11-06
- Scope: core (any backend that calls a language model with a long or repeated prefix)
Context
A model request re-sends its whole prefix every time: tools, system prompt, shared reference,
the user’s context, the conversation. A tool loop multiplies that, since each step re-sends
everything before it. Providers cache a request by its rendered prefix up to a marked
breakpoint, with a small, hard cap on the number of breakpoints.
Expenses found five ways to waste that cache, none of which fails visibly (expenses
0022-prompt-cache-breakpoint-budget):
- one breakpoint after the system prompt left the growing conversation uncached on every call;
- text identical for every user placed after a per-user byte became a per-user entry;
- a request built from a hash map with a per-process iteration order rendered different bytes
after every restart, discarding the cache on each deploy and between instances; - a user’s context rebuilt mid-conversation (a memory saved in the background) rewrote the whole
conversation into the cache; - after a pause longer than a five-minute TTL, the first message rewrote the whole conversation.
Decision
1. The application states stability; only the gateway knows the wire
The provider-agnostic port takes context by how stable it is: shared by everyone, this user’s,
this conversation’s. Only the provider adapter turns that into cache markers.
2. Most stable first, every breakpoint spent
Order: tools and system prompt and shared reference → the user’s context → the message that
opened the turn → the latest message. Spend every breakpoint the provider allows; a new block
takes a slot, never adds one past the cap. A longer TTL renders before a shorter one. Use the long
TTL where the next read is likely to come after a pause.
3. Nothing user-specific in the shared half; tool definitions are user-independent
One user-specific byte in the shared prefix costs every other user their hit. Tool descriptions
and schemas are part of that prefix, so they are built once from code, never from a caller’s data
(unless a per-user prefix is the point, and the ADR says so).
4. Byte-stable rendering, and context frozen for the conversation
The request body is built from ordered maps with canonicalised schemas. A context block that can
change mid-conversation is snapshotted when the conversation starts (or when its cache would have
expired anyway), so it does not change under the cache.
5. Breakpoints below the minimum are harmless; regressions are found in the usage log
A breakpoint on a prefix under the model’s minimum cacheable size is a no-op, not an error; leave
it. Nothing fails when caching regresses, so log cache reads and writes per call, persist them, and
check them after any change to how a request is built.
Consequences
- Cost per turn grows with the change since the previous call, not with the conversation.
- A deployment shares one entry for the shared prefix, across instances.
- Accepted: the long TTL costs more to write; it pays back when users pause.
- Any meter that counts tokens must account for cache reads and writes, or caching silently
changes what it measures (0026-llm-usage-is-metered-in-money-pinned-to-a-reference-model).
Related
- 0023-model-facing-data-is-a-catalog — the index lives in the shared prefix
- 0026-llm-usage-is-metered-in-money-pinned-to-a-reference-model — what caching did to the quota
- Expenses 0022-prompt-cache-breakpoint-budget — the worked example (four slots,
ordered()/
canonical(), memory snapshots, one-hour writes priced apart)