ADR-0025: Prompt caching — order the request most-stable-first, spend every breakpoint, keep the bytes stable

  • Status: Accepted
  • Date: 2026-11-06
  • Scope: core (any backend that calls a language model with a long or repeated prefix)

Context

A model request re-sends its whole prefix every time: tools, system prompt, shared reference,
the user’s context, the conversation. A tool loop multiplies that, since each step re-sends
everything before it. Providers cache a request by its rendered prefix up to a marked
breakpoint, with a small, hard cap on the number of breakpoints.

Expenses found five ways to waste that cache, none of which fails visibly (expenses
0022-prompt-cache-breakpoint-budget):

  • one breakpoint after the system prompt left the growing conversation uncached on every call;
  • text identical for every user placed after a per-user byte became a per-user entry;
  • a request built from a hash map with a per-process iteration order rendered different bytes
    after every restart, discarding the cache on each deploy and between instances;
  • a user’s context rebuilt mid-conversation (a memory saved in the background) rewrote the whole
    conversation into the cache;
  • after a pause longer than a five-minute TTL, the first message rewrote the whole conversation.

Decision

1. The application states stability; only the gateway knows the wire

The provider-agnostic port takes context by how stable it is: shared by everyone, this user’s,
this conversation’s. Only the provider adapter turns that into cache markers.

2. Most stable first, every breakpoint spent

Order: tools and system prompt and shared reference → the user’s context → the message that
opened the turn → the latest message. Spend every breakpoint the provider allows; a new block
takes a slot, never adds one past the cap. A longer TTL renders before a shorter one. Use the long
TTL where the next read is likely to come after a pause.

3. Nothing user-specific in the shared half; tool definitions are user-independent

One user-specific byte in the shared prefix costs every other user their hit. Tool descriptions
and schemas are part of that prefix, so they are built once from code, never from a caller’s data
(unless a per-user prefix is the point, and the ADR says so).

4. Byte-stable rendering, and context frozen for the conversation

The request body is built from ordered maps with canonicalised schemas. A context block that can
change mid-conversation is snapshotted when the conversation starts (or when its cache would have
expired anyway), so it does not change under the cache.

5. Breakpoints below the minimum are harmless; regressions are found in the usage log

A breakpoint on a prefix under the model’s minimum cacheable size is a no-op, not an error; leave
it. Nothing fails when caching regresses, so log cache reads and writes per call, persist them, and
check them after any change to how a request is built.

Consequences

  • Cost per turn grows with the change since the previous call, not with the conversation.
  • A deployment shares one entry for the shared prefix, across instances.
  • Accepted: the long TTL costs more to write; it pays back when users pause.
  • Any meter that counts tokens must account for cache reads and writes, or caching silently
    changes what it measures (0026-llm-usage-is-metered-in-money-pinned-to-a-reference-model).