Cache Optimization
Stabilize message prefixes for provider KV cache hits and configure provider-specific caching strategies.
LLM providers cache prompt prefixes to avoid reprocessing identical input on repeated calls. Headroom's CacheAligner is detector-only, so it surfaces prefix drift, reports observability data, and leaves message assembly to the caller.
CacheAligner is opt-in and detector-only
CacheAligner is disabled by default and hard-disabled inside the proxy — it never runs on your traffic unless you explicitly enable it, and even then it only reports metrics; it never repairs a prefix. In proxy mode, prefix-cache stability comes from cache mode (--mode cache, the default), which compresses only the newest delta and forwards prior turns byte-faithfully. See Savings Profiles.
What CacheAligner reports
System prompts often contain dynamic content, such as dates, session IDs, and timestamps, that changes between requests. Even a single character difference at the start of a prompt invalidates the entire provider cache.
CacheAligner does not extract, move, normalize, reorder, strip, compress, or rewrite content. It detects volatile content and reports the stable prefix hash plus cache metrics so you can fix the prefix at the source:
| Signal | Meaning |
|---|---|
warnings | The prefix contains unstable content |
cache_metrics.stable_prefix_bytes | Stable prefix size in bytes |
cache_metrics.stable_prefix_tokens_est | Stable prefix size in estimated tokens |
cache_metrics.stable_prefix_hash | Stable prefix hash for repeated-wake comparison |
cache_metrics.prefix_changed | The prefix drifted since the previous wake |
markers | The emitted stable_prefix_hash marker for observability |
The prefix must stay byte-identical across requests for provider KV caches to reuse previously computed attention states.
Provider-specific strategies
Each LLM provider implements caching differently. Headroom applies the optimal strategy for each.
Anthropic
Anthropic supports explicit cache_control blocks that mark content as cacheable. Cached input tokens cost 90% less than regular input tokens.
Keep the stable prefix byte-identical, then place provider cache markers where your client or orchestrator already assembles the request. Headroom's job is to surface prefix instability, not repair it.
| Metric | Value |
|---|---|
| Cache read discount | 90% off input price |
| Cache write cost | 25% premium on first write |
| Cache TTL | 5 minutes (extended on hit) |
OpenAI
OpenAI uses automatic prefix caching. If consecutive requests share the same message prefix, the provider reuses cached KV states. No explicit API markers are needed, but the prefix must be byte-identical.
CacheAligner tells you when the prefix changed, which is the only signal you need to keep OpenAI prefix caching effective.
| Metric | Value |
|---|---|
| Cache read discount | 50% off input price |
| Activation | Automatic (prefix match) |
| Min prefix length | 1024 tokens |
Google provides the CachedContent API, which lets you explicitly cache large context (system instructions, documents, tools) and reference it across requests. Cached tokens cost 75% less.
Keep the prefix stable in your integration layer; Headroom reports when the cacheable zone drifts. The CachedContent lifecycle itself also stays in your integration layer.
| Metric | Value |
|---|---|
| Cache read discount | 75% off input price |
| Mechanism | Explicit CachedContent API objects |
| Min cache size | 32,768 tokens |
What this means in practice
Keep the stable prefix first, keep volatile content out of it, and treat CacheAligner warnings as a signal that the caller needs to move assembly logic.
CacheAligner surfaces prefix instability, provider caches reward byte-identical prefixes, and the caller owns the actual message layout.
Cold-prefix recompaction (when the cache lapses)
Byte-identical forwarding only pays off while the prompt cache is warm. When a session goes idle past the provider's cache TTL, that cache is dead — so re-sending the prefix verbatim buys nothing, and it's the one safe moment to rewrite it. The cold-prefix hook detects a lapsed cache and recompacts the whole prefix (cross-turn dedupe + superseded-read drop + lossless folds) instead, then re-caches the smaller result. It only fires on a confirmed-cold turn — a wrong call would bust a warm cache — so it reads the real TTL (exact for Claude Code via its cache-control env vars; learned over time for other providers).
For models that re-send reasoning as plain text every turn (Kimi/GLM/DeepSeek), a companion hook Kompresses that reasoning on warm turns and drops it on cold ones.
Both are off by default. See Cold-prefix hook & reasoning compaction for the exact flags and when each is safe to enable.