A Smaller Prompt Can Cost More After a Cache Rewrite

Removing 2,000 tokens looks useful until the edit sits before a warm 50,000-token suffix. Under Headroom's default cost-model inputs, that mutation loses 55,500 plain-input-token cost units. The same reduction with a cold cache gains 4,300 units.

These are executed synthetic calculations, not measured bills. The useful finding is the decision boundary: a compression layer needs to account for the cache it invalidates, alongside the tokens it removes. Headroom has an explicit gate for that tradeoff, including a batch rule that charges a shared invalidation only once.

I examined revision e995eb9f1dacac7ed548758406de6c1508caef95, whose package configuration declares headroom-ai version 0.41.0. I ran its Python cost policy and an isolated router method with Python 3.12.14 on Linux. I did not run its Rust compressor, tokenizer, proxy server, or provider requests. The token counts, cache state, and expected reuse below are fixture inputs.

A Compression Candidate Has a Position

Headroom contains a native compression core, Python integration layers, provider adapters, and tests for routing and cache behavior. Its build configuration uses Maturin; this is more than a wrapper around a summarization prompt. For example, the Python diff-compressor interface delegates execution to the native core.

My subject is narrower: the economic decision after a candidate's original and compressed token counts are available. The Python cost policy mirrors a canonical Rust policy. I inspected both implementations and their tests; runtime results here come from Python, not a cross-language parity run.

The model distinguishes four inputs:

Input	Meaning in the examined code
delta_t	Tokens removed from the candidate message
suffix_tokens	Token total in messages after that slot
expected_reads	Modeled remaining reuse count, R
p_alive	Estimated probability that the existing cache is still useful

Write and read prices are relative to an ordinary input token. The code's five-minute tier uses w = 1.25 and r = 0.1; its one-hour tier uses w = 2.0. These are constants in the pinned model, not a fresh pricing survey or a dollar-denominated benchmark.

The Warm Case Changes the Arithmetic

The general gain expression is:

gain = delta_t * (w + r * (R - 1))
       - p_alive * (w - r) * (S + delta_t)

Here S is the suffix token count. With a cold cache, removing tokens avoids their initial write and subsequent reads. With a warm cache, those removed tokens have already been written. Keeping them would require reads; changing the earlier message also makes the later suffix need another write instead of a read.

Setting p_alive = 1 reduces the expression to:

gain = delta_t * r * R - (w - r) * S

That reduction explains the S + delta_t penalty in the general formula. Charging only S there would incorrectly count a fresh write saving for tokens that were already cached.

The policy admits a mutation only when gain is strictly positive. Its warm-cache break-even helper returns:

R = ((w - r) / r) * (S / delta_t)

For a 2,000-token cut before a 50,000-token suffix, the computed boundary is 287.5 at the five-minute write tier and 475.0 at the one-hour tier. These are model boundaries, not forecasts of how long an agent session will run.

Executed Cases

The replay probe verifies SHA-256 digests of four pinned source files. It loads the unchanged policy modules through isolated package namespaces, bypassing SDK initializers. For the router gate, it selects the original _net_cost_allows() method and its log-bucket helper from the source AST, compiles their unchanged bodies, and supplies the real policy on a small receiver object.

This isolates the decision logic from native compression and provider dependencies. It does not test importing the complete SDK, actual token counting, request dispatch, or the surrounding router workflow. Obtain the four files listed in the probe from the pinned revision, preserving their paths, to replay it.

From the portfolio checkout, I ran:

python --version
python public/blog/token-savings-cache-rewrite/cost-probe.py /tmp/headroom-note24

Both commands exited zero. The five cost cases used R = 10:

Tokens removed	Suffix tokens	Cache alive	Write tier	Computed gain	Decision
2,000	50,000	1	Five-minute	-55,500	Skip
2,000	50,000	0	Five-minute	4,300	Mutate
50,000	10,000	1	Five-minute	38,500	Mutate
2,000	1,500	1	Five-minute	275	Mutate
2,000	1,500	1	One-hour	-850	Skip

All gains use plain-input-token cost units. The last two rows hold the candidate and reuse assumptions fixed: changing only the write tier reverses the decision.

Selected stdout:

small-warm: gain=-55500.0; mutate=False
small-cold: gain=4300.0; mutate=True
large-warm: gain=38500.0; mutate=True
near-5m: gain=275.0; mutate=True
near-1h: gain=-850.0; mutate=False
small-warm break-even, ttl=300: R=287.5
small-warm break-even, ttl=3600: R=475.0
PASS: 5 cost cases, 2 break-even boundaries, 4 isolated gate scenarios

A Batch Must Remember What It Already Invalidated

Evaluating every edit independently would overcharge overlapping suffixes. In the examined router gate, batch_state["floor"] records the shallowest admitted full-suffix mutation. A candidate at a strictly deeper slot then uses S = 0 in its incremental calculation: the earlier edit has already paid for invalidating that region.

The isolated replay checked four scenarios:

Gate scenario	Observed result
Small warm-cache edit rejected	Batch floor stayed unset
Large shallow edit admitted, followed by a small deeper edit	Both admitted; deeper edit recorded router:netcost_batch_admit
Earlier edit evaluated with only a deeper floor	Earlier edit still rejected
Small edit with an explicit p_alive_override = 0	Admitted with router:netcost_idle_compaction

The floor therefore represents an admitted change, not merely a candidate that the compressor considered. Its direction matters too: invalidating a later prefix does not subsidize an earlier edit.

The surrounding apply() code enables this gate only when HEADROOM_NET_COST_POLICY=1. It builds suffix totals, prefers a request-level cache TTL, and can derive cache survival as max(0, 1 - idle_seconds / ttl). Those integration details are code inspection. My probe calls the gate directly; it does not establish that a deployed proxy enables it or forwards those inputs correctly. The upstream integration tests exercise broader routing behavior, but I did not execute that suite.

The remaining uncertainty is operational. Expected reuse and cache survival are estimates, and the suffix size depends on the tokenizer and request adapter. A linear idle-time estimate does not observe the provider's cache. Compression quality is another independent constraint: a positive modeled gain says nothing about whether the compressed tool result retained the evidence an agent needs. This gate answers a useful, smaller question—whether a proposed token saving is worth the cache rewrite under explicit assumptions.