All posts

Why drifting prompts double your AI bills: KV cache locking

Nov 13, 2026 2 min read Oleg Savelev
costs caching architecture

Model providers bill cached input tokens at a fraction of full price, provided your prompt prefix matches what they have already processed. The catch: the match must be byte-exact from the very first token. One volatile field sorted differently, one timestamp injected mid-prompt, or one tool schema reordered, and the cache dies for the entire conversation. You pay full price again for context you already bought.

In a 20-turn live replay over src/*.py, locking the prefix produced a 90.8% byte-identical KV-cache hit rate (cached reads at 0.10x). Combined with -84.5% lower input volume, that cut a Claude Sonnet 4.5 session total bill by -96.5% (DeepSeek V3: -96.9%) against an uncached baseline, and by -84.5% against a naive "mid-trim but cache-blind" baseline.

What actually breaks the cache

Three silent issues show up in almost every agent stack:

  1. Volatile metadata inside the prompt: timestamps, run IDs, and counters injected before the stable body.
  2. Unstable tool schemas: identical tools serialized in a different key order each turn.
  3. Mid-conversation rewrites: attempts to optimize context by editing its middle instead of appending.

The third issue is the most expensive trap. Trimming context to save tokens often increases the bill: every edit to the middle of the prefix re-bills everything after it at full input rates.

The locking rules

  • Append-only governor: the conversation prefix grows monotonically up to ~80% of the context window and is never edited in place.
  • Prefix normalizer: volatile metadata is stripped from the prefix zone; tool schemas are sorted so byte-order is stable across turns.
  • Multi-breakpoint cache control: explicit Anthropic cache_control and OpenAI prompt_cache_key markers injected at several stable breakpoints, so a miss at one point does not discard the entire prefix.

The economics per 20-turn session (Sonnet 4.5, measured on the replay harness):

Scenario Input spend Total bill
Uncached baseline baseline baseline
Cache-blind mid-trim -84.5% -84.5%
Prefix locking + slicing -96.8% -96.5%

Cached read-heavy and graph-discovery workflows reach up to 99%; typical end-to-end sessions land at 80-97%.

The one rule to remember

Never edit the middle of a conversation. If you manage prompts today: append, do not rewrite; keep tool definitions in a fixed order; keep volatile fields out of the head of the prompt. That alone moves your cache hit rate more than prompt compression tricks. Compression that rewrites history is billed as hidden cache misses.

TokenSafe enforces this automatically inside the local proxy: the agent stack talks to localhost:8765/v1 and the locking happens before anything leaves the machine. You can pair prefix locking with AST code slicing, explore the local proxy architecture, or start a 7-day team trial to lock prompt caches across your engineering team.

Every cache miss is a second purchase of context you already bought. Lock the prefix, and the provider stops charging you for it.