## Description
Running headroom as the proxy for Claude Code destroys Anthropic
prompt-cache
reuse (#2085: ~4.4x cache-creation inflation, 2.5–3x net cost). Tracing
live
Claude Code traffic through the proxy shows **two independent
session-identity
defects**, both of which orphan or thrash the frozen-prefix state; this
PR
fixes both.
### Defect 1: `<system-reminder>` turns rotate the fallback session id
mid-conversation
Claude Code interleaves reminder turns into the history as actual
`role:"system"` messages (hook output, skills lists, file-truncation
notices).
`compute_session_id` hashed **every** system message, so the id rotated
each
time a reminder landed. Live trace (subagent reading two 80KB files; sid
changes exactly when the truncation reminder appears, and the tracker
restarts
at turn 0):
```
REQ#2 sid=68d4ee666990 nmsg=3 [0]SYSTEM<<top-level system>> [1]user [2]SYSTEM<<skills reminder>>
REQ#3 sid=6944948c9fb2 nmsg=6 ... [5]SYSTEM<<Truncated: PARTIAL view ...>> <- id rotated
```
Everything keyed on the session id is orphaned at that moment: the
prefix
tracker (freeze never survives past a reminder-bearing turn),
beta-header
stickiness, the CCR and memory-tool registries, and the compression
cache.
**Fix:** hash only the **leading run** of system messages (everything
before
the first non-system turn) — the top-level system prompt on the
Anthropic path
(folded in as the synthetic first message), the conventional leading
system
message(s) on the OpenAI path. Stable for the life of a conversation;
mid-history system turns are content, not identity.
### Defect 2: conversations sharing a (now stable) id thrash one tracker
With ids stable, the fallback tuple `model + system prompt` is identical
across every same-type parallel subagent (and any sessions reusing one
system
prompt) — all of them collapse onto one `PrefixCacheTracker`, and their
interleaved histories cross-contaminate the freeze state: the forwarded
prefix
is byte-unstable on nearly every turn and the provider cache is
re-written
instead of read. Reproduced against the real code paths (script below):
```
1) fallback session ids: A=3dc639aaf4f48fa1 B=3dc639aaf4f48fa1 -> COLLIDE=True
2) single conversation, legacy : stable prefix on 4/4 later turns, trackers=1
2) interleaved (subagents), legacy : stable prefix on 0/9 later turns, trackers=1
2) interleaved, lineage resolution : stable prefix on 8/8 later turns, trackers=2
```
**Fix:** `SessionTrackerStore.resolve_tracker` — within a session id,
reuse
the tracker whose previous request messages are a prefix of the incoming
history (client histories are append-only, so a conversation's next
request
always extends its previous one); a diverging or rewritten history
(client-side compaction) starts a fresh lineage. Matching uses the
repo's
existing canonical cross-turn equivalence
(`_canonicalize_for_prefix_compare`,
the same one the cache-stable delta path uses) on the **original client
bytes**, so moved cache breakpoints, string<->block sugar, transport
annotations, or a tail-mutating `pre_compress` hook never read as a
rewrite.
Byte-identical histories (templated fan-outs before they diverge)
intentionally share a tracker — their provider cache line is identical
too.
### Both fixes together, on live Claude Code traffic (sonnet, 2 parallel
Explore agents)
```
main conversation: sid=5b7e245a... one tracker, turns 0->4, id stable across reminders
agents (collide): sid=2bdffc9e... -> lineage bare (alpha) turns 0->1->2
-> lineage "~1" (beta) turns 0->1->2
```
Before: the agents' ids rotated per reminder (every tracker stuck at
turn 0),
and whenever they did share an id they thrashed one tracker (`0/9`
stable
prefixes in the repro).
### Why not key the session id on conversation content?
Draft #1912 folds the first user turn into the fallback id; this change
composes with it, but identity-level keying alone can't close#2085:
identical
first turns (templated fan-outs) still collide, and everything keyed on
the
session id rotates with it when the client rewrites history. The
"session"
(client/workspace grouping) and the "conversation" (positional cache
lineage)
are different identities; only the tracker holds positional per-turn
state
that thrashes under collision — beta stickiness is a monotone union and
the
compression cache is content-addressed — so lineage resolution lives one
level below the session id and leaves the id semantics (and every other
consumer) untouched.
## Changes Made
- `headroom/cache/prefix_tracker.py`:
- `compute_session_id`: harvest only the leading system run (defect 1).
- `SessionTrackerStore.resolve_tracker`: conversation-lineage resolution
(defect 2). First lineage lives under the bare session id —
single-conversation sessions behave byte-identically to before; degrades
to `get_or_create` when messages are absent or prefix freeze is
disabled.
- Lineages are capped per session id
(`PrefixFreezeConfig.max_lineages_per_session`, default 32). **Over-cap
conversations share one overflow tracker instead of evicting an
established lineage** — any eviction policy degrades every conversation
once the working set exceeds the cap (under round-robin the victim is
always the conversation about to arrive), while overflow sharing
degrades
only the over-cap tail, to exactly the pre-lineage shared behavior; `0`
disables lineage splitting. Chains are stored as structural snapshots
that normalize `NaN` (`json.loads` accepts bare NaN, and `NaN != NaN`
would read a byte-identical resend as a rewrite). Synthetic lineage keys
use a `\x00` separator, which cannot appear in an HTTP header value, so
they can never collide with a client-supplied `x-headroom-session-id`.
- `headroom/proxy/handlers/anthropic.py`, `openai.py`: the session id
and
the lineage both derive from the **same original client bytes** (a
turn-dependent hook rewrite can no longer rotate one without the other);
anthropic folds in its synthetic system message so explicit-header
clients
with different system prompts stay separate. Plus a docstring correction
in `streaming.py` that falsely claimed its coarse mid-turn key "mirrors"
`compute_session_id`.
- `tests/test_cache/test_prefix_tracker.py`: 24 new test cases —
reminder-rotation regression; interleaved isolation + per-conversation
turn state; identical-first-turn share-then-split; cache_control
movement
(3 cases); representation churn (string<->block sugar / streaming
`index`
/ Bedrock cachePoint); rewritten history → fresh lineage (compacted /
middle-edited / truncated); legacy no-messages / freeze-disabled /
empty-canonical fallbacks; NaN-in-tool-payload stability; overflow
sharing, established-lineages-survive-cap, and a cap+1 round-robin
no-cliff guard; TTL cleanup; session-id-not-rotated-by-lineage guard.
One
existing test renamed (`uses_all_system_messages` →
`distinguishes_leading_system_run`) to match the new contract.
- Three SimpleNamespace stub stores in existing tests gained a
`resolve_tracker` field (handlers call it unconditionally — a silent
`hasattr` fallback would degrade to the pre-fix behavior with no
signal).
One of them is the cold-start fast-pass suite (#2073), which landed
while
this branch was in review.
- `CHANGELOG.md` entry.
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
## Testing
- [x] Unit tests pass (`pytest`) — 11 failed, 8652 passed, 528 skipped
in 4:37 (the 11 are pre-existing on unmodified `main` — verified by
rerunning the same node ids on a clean checkout:
gh-CLI/onnx/PID-reuse/deadline flakes and order-dependent cases, none
touching session/cache/proxy paths)
- [x] Linting passes (`ruff check .`) — All checks passed (ruff 0.15.17,
CI-pinned; `ruff format --check .` clean)
- [x] Type checking passes (`mypy headroom`) — Success: no issues found
in 471 source files
- [x] New tests added for new functionality — 24 test cases; the
rotation/isolation/no-cliff ones fail on `main`
- [x] Manual testing performed — live Claude Code end-to-end, below
### Test Output
```text
$ python -m pytest tests/ -q
11 failed, 8652 passed, 528 skipped, 5857 warnings in 276.68s (0:04:36)
# same 11 fail on unmodified main (env/order-dependent: test_wrap_claude_base_url pid-reuse,
# copilot_auth gh-cli fallback, image_compression onnx, content_router deadline, rtk/output-shaper/dedup order flakes)
$ python -m pytest tests/test_cache/test_prefix_tracker.py -q
63 passed
$ uvx ruff@0.15.17 check . && uvx ruff@0.15.17 format --check .
All checks passed! / 1208 files already formatted
$ mypy headroom
Success: no issues found in 471 source files
$ python repro_2085.py
1) fallback session ids: A=3dc639aaf4f48fa1 B=3dc639aaf4f48fa1 -> COLLIDE=True
2) single conversation, legacy : stable prefix on 4/4 later turns, trackers=1
2) interleaved (subagents), legacy : stable prefix on 0/9 later turns, trackers=1
2) interleaved, lineage resolution : stable prefix on 8/8 later turns, trackers=2
```
## Real Behavior Proof
- Environment: macOS arm64, Python 3.13, `uv sync --extra dev --extra
proxy`;
real Claude Code CLI pointed at the proxy via
`ANTHROPIC_BASE_URL=http://127.0.0.1:8790`, real Anthropic backend.
- Exact command / steps: ran Claude Code sessions that launch 2–3
parallel Explore subagents
(each reading multi-KB JSON files, several tool-loop turns each), with
an
observability wrapper printing each request's resolved session id,
tracker
identity, and turn counter inside the proxy.
- Observed result: on `main`, subagent session ids rotate on
reminder-bearing turns
(trackers permanently stuck at turn 0); when conversations do share an
id
they share one tracker whose turn counter interleaves all of them.
On this branch: ids stable for the life of each conversation;
colliding subagents resolve to separate lineages (`bare`, `~1`) with
clean
per-conversation turn progressions (trace above). Unit-level repro shows
forwarded-prefix stability going 0/9 → 8/8 for the interleaved shape.
- Not tested: reporter-scale cache-economics (his 4.4x needs his
long-session
workload against a paid backend); happy to coordinate with
@RomanAlexanderW
on a before/after — the number to watch is the cache-read ratio in
Claude
Code transcripts recovering toward ~96%.
<details>
<summary>repro_2085.py</summary>
```python
"""Repro for #2085: concurrent conversations sharing a fallback session id
(same model + system prompt — e.g. a Claude Code session and its parallel
subagents) collapse onto one PrefixCacheTracker and thrash its frozen-prefix
state -> byte-unstable forwarded prefixes -> the provider prompt cache is
re-written on nearly every call. Uses headroom's real code paths.
Run from the repo root: python ../repro_2085.py
"""
from headroom.cache.prefix_tracker import PrefixFreezeConfig, SessionTrackerStore
MODEL = "claude-sonnet-5"
# Claude Code system prompt: long, static, identical across the main session
# and every parallel subagent of the same type.
SYSTEM = ("You are Claude Code, Anthropic's official CLI for Claude. " * 40)[:2000]
def convo(name: str, turns: int) -> list[dict]:
msgs = [{"role": "system", "content": SYSTEM}]
for t in range(turns):
msgs.append({"role": "user", "content": f"[{name}] user turn {t}: " + ("x" * 800)})
msgs.append(
{"role": "assistant", "content": f"[{name}] tool_result {t}: " + ('{"data": 1}' * 200)}
)
return msgs
class _Req: # request stub: no x-headroom-session-id header
headers: dict = {}
# --- Part 1: identity collision (real derivation) ----------------------------
store = SessionTrackerStore(PrefixFreezeConfig())
id_a = store.compute_session_id(_Req(), MODEL, convo("A", 3))
id_b = store.compute_session_id(_Req(), MODEL, convo("B", 5))
print(f"1) fallback session ids: A={id_a} B={id_b} -> COLLIDE={id_a == id_b}")
# --- Part 2: interleaved conversations thrash the freeze state ---------------
def run(interleave: bool, lineage_resolution: bool) -> tuple[int, int, int]:
store = SessionTrackerStore(PrefixFreezeConfig())
stable_turns = 0
later_turns = 0
seq = []
for t in range(1, 6):
seq.append(("A", convo("A", t)))
if interleave:
seq.append(("B", convo("B", t)))
for _name, msgs in seq:
sid = store.compute_session_id(_Req(), MODEL, msgs)
if lineage_resolution:
tracker = store.resolve_tracker(sid, "anthropic", messages=msgs)
else:
tracker = store.get_or_create(sid, "anthropic")
if tracker._turn_number > 0:
later_turns += 1
if tracker._forwarded_prefix_stable(msgs):
stable_turns += 1
tracker.update_from_response(
cache_read_tokens=5000 * len(msgs),
cache_write_tokens=2000,
messages=msgs,
)
return stable_turns, later_turns, store.active_sessions
for label, interleave, fixed in (
("single conversation, legacy ", False, False),
("interleaved (subagents), legacy ", True, False),
("interleaved, lineage resolution ", True, True),
):
stable, later, sessions = run(interleave, fixed)
print(f"2) {label}: stable prefix on {stable}/{later} later turns, trackers={sessions}")
```
</details>
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
## Checklist
- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation (CHANGELOG
only — no docs describe the tracker store)
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I have updated the CHANGELOG.md if applicable
## Additional Notes
- Addresses the session-identity mechanisms of #2085; intentionally does
not
`Closes` it — the reporter should confirm the cache-read ratio recovers
on
live traffic first.
- Composes with draft #1912 (first-user-turn fallback id).
- Known bounded tradeoffs (all strictly milder than the per-turn thrash
this
fixes): a fork-style branch that resends a parent's full history adopts
the
parent's lineage, costing the parent one cold restart at its next turn;
a
request that aborts before the response and is retried with different
bytes
starts a fresh lineage; history truncation/tail-edit starts a fresh
lineage
even though the shorter provider prefix may still be warm.
- Hot-path cost, measured on a 199-message/2.1MB agentic history:
canonical
projection 0.21ms + structural snapshot 0.92ms + match loop 0.06ms with
32 candidate lineages (2.27ms absolute worst case) ≈ **1.3ms per
request** — same order as the handler's existing request deepcopy
(0.80ms) and below one `json.dumps` of the body (2.9ms). Chain memory is
structure-only (~180-330KB per lineage; message strings are shared with
state the tracker already retains).
- Known semantic shift to flag: hashing only the leading system run
means
conversations distinguished ONLY by mid-list system messages (e.g.
clients injecting a per-conversation system context late in the list)
now
share a fallback id. The tracker is protected by lineage resolution; the
residual sharing concentrates in the CCR sticky-tool registry and the
monotone beta union — the same pre-existing class as same-system-prompt
conversations today. Happy to file the CCR-stickiness scoping as a
follow-up.
- Out of scope, observed while tracing: `SessionCcrTracker.has_done_ccr`
mildly cross-contaminates conversations sharing an id (monotone, no
thrash) — can file separately if useful.
---------
Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
Andrei Boldyrev
7bfb1d7f38