fix(reporting): show net vs gross savings, real skip thresholds, and the effective profile (#3123)
## Description
Six reporting/config defects found while investigating a user reporting
~1% savings on Claude Code. **None of these changes how much Headroom
compresses** — all of them change whether an operator can tell what it
did. Every one was found by reading that user's own 227,777 lines of
proxy logs against the code.
## Changes Made
- **`perf/analyzer`: parse and render `tok_inflated`.** Every PERF line
carried it; nothing downstream read it. The report could print
`321,239,562 -> 313,274,727` directly above `8,455,763 saved` — two
figures that differ by exactly the 490,928 tokens of inflation it
omitted.
- **`content_router`: report the real skip thresholds.** The routing
summary hardcoded `skipped (<50 words)` regardless of what was in force.
Wrong number (the message gate is `min_tokens`, 10–250 by profile),
wrong unit (tokens and characters, never words), and it merged two
different gates under one label.
- **`perf/analyzer`: disclose that Transform Effectiveness is partial.**
It is built only from `pipeline.py`'s `Transform NAME:` lines.
`compression_units.py` / `compression_batches.py` contain zero logging
calls, so the table read `content_router: 189,783 saved` against a PERF
total 44x larger. Reports the divergence rather than a coverage ratio —
the two are different populations and neither contains the other (those
lines carry no request_id, fire per stage, and are emitted before the
forwarder decides).
- **`perf/analyzer`: disclose the routing denominator.** Percentages
were taken over 4 of the router's 17 outcome buckets, silently dropping
buckets larger than several it displayed.
- **`savings_tracker`: stop dropping tool-schema dollars.**
`estimate_request_savings_usd` prices four buckets; `record_request`
read three. `tool_schema` was computed and discarded, so a quarter of
the token headline never reached "Cost saved". The two inputs are
disjoint (verified at the call site), so this is additive, not
double-counting.
- **`agent_savings`: an unknown profile no longer degrades to
`balanced`.** `balanced` is a different product posture from the default
`coding`: cache→token mode, dedup off, tool-search off, user messages
uncompressed, message floor 25x higher, block floor 20x higher. A typo
in `HEADROOM_SAVINGS_PROFILE` silently reconfigured the whole proxy. Now
degrades to `DEFAULT_PROFILE` and names the resolved profile in the
warning.
- **`agent_savings`: give `min_chars_for_block` a config-object path.**
Every other router pipeline kwarg travels on the config object; this one
alone was env-only, so an unseeded proxy applied every sibling `coding`
knob while this floor stayed at 500 instead of 25.
- **`server`: log the resolved compression posture at startup**, reading
cross-turn dedup off the constructed router rather than the environment
(the router resolves it as `config OR env`, so reading env alone would
be a guess).
## Testing
- [x] Unit tests pass, [x] ruff, [x] mypy, [x] new tests added
```text
uv run pytest tests/ -k "content_router or agent_savings or perf or analyzer or savings or proxy_server or cli_perf or prometheus"
620 passed, 25 skipped
uv run mypy headroom # Success
uv run ruff check . && ruff format --check . # clean
```
## Real behavior proof
- **Setup:** macOS arm64, Python 3.12, this branch. Input: 60 MB /
227,777 lines of real proxy logs from the reporting user (6 rotated
files, 2,792 PERF lines, 2026-08-17 → 2026-08-19).
- **Steps:** pointed `headroom.perf.analyzer.LOG_DIR` at that directory
and rendered the report before and after the patch.
- **After-fix output (real data, unmodified):**
```text
Requests: 2792
Tokens: 321,288,161 -> 313,323,326 (2.6% messages)
Tokens saved: 11,158,901 (3.4% reduction)
· inflated 490,928 (net message reduction 7,964,835)
· messages 8,455,763
· tool schemas 2,703,138
! stage-level total 190,641 != PERF message total 8,455,763 — this table sees only
engines that emit a Transform line, counts per stage, and does not check whether
the mutation shipped
Skipped: 44641 (77%) — below size floor
(shares are of these 4 buckets only, n=58319; see `[router] route_counts=` for the
full outcome space)
```
The arithmetic now closes on the page: `8,455,763 - 490,928 =
7,964,835`, matching the token delta exactly. Before the patch none of
the three annotated lines existed and the `Skipped` line claimed `<50
words`.
- **Profile resolution verified by execution**, not inspection —
subprocesses with controlled env:
```text
vanilla (nothing set) mode=cache dedupe=1 tool_search=1 min_tokens=10 min_chars=25
HEADROOM_SAVINGS_PROFILE=coding mode=cache dedupe=1 tool_search=1 min_tokens=10 min_chars=25
unknown profile name (before) mode=token dedupe=0 tool_search=0 min_tokens=250 min_chars=500
unknown profile name (after) -> resolves to `coding`, warning names it
coding, seeding never runs min_chars=25 (was 500 before this patch)
```
- **Not tested:** live paid Anthropic traffic. These are
reporting/config surfaces; the wire path is untouched by this PR.
## Review readiness
- [x] Self-reviewed. Three overclaims in my own first draft were
corrected before this PR: a false subset claim in the Transform
Effectiveness note, a comment asserting `min_chars_for_block` was the
*only* env-only field (it is the only env-only *router pipeline kwarg*;
`cross_turn_dedup`, `tool_search`, `protect_reads`, `code_aware`,
`effort_router`, `lossless` remain env-only via a different mechanism
and are **not** fixed here), and a money-path expression that relied on
`a + b if c else d` grouping.
## Known remaining (deliberately out of scope)
- `Requests: N` still overcounts: the Codex WS forwarder reuses one
`request_id` across every turn (one observed 156x), plus ~18 duplicate
PERF emissions.
- `compression_units.py` / `compression_batches.py` remain unlogged —
this PR *discloses* the blind spot rather than closing it.
- The headline stays **gross**. True net is `11,158,901 - 490,928 =
10,667,973` (3.3%, not 3.4%). Making net the headline lowers every
user's reported savings ~4.4%; that is a product call, not mine, so the
inflation is surfaced beside it instead.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-authored-by: Tejas Chopra <tejas@Tejass-MacBook-Pro.local>
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
Tejas Chopra
@@ -0,0 +1,223 @@
|
||||
# Headroom Metrics — Dashboard Guide
|
||||
|
||||
What each metric shows, so you can build panels against it.
|
||||
|
||||
**Two endpoints.** Both are on the proxy (default `:8787`).
|
||||
|
||||
| Surface | How to get it | Use it for |
|
||||
|---|---|---|
|
||||
| **Prometheus** — `GET /metrics` | Always on, no config | Everything below. Start here. |
|
||||
| **OpenTelemetry** — OTLP/HTTP | `HEADROOM_OTEL_METRICS_ENABLED=1` + `pip install "headroom-ai[proxy,otel]"` | Same data, dotted names, plus per-tenant labels |
|
||||
|
||||
Names differ between them: Prometheus uses `headroom_tokens_saved_total` (**milliseconds** for timings), OTel uses `headroom.proxy.tokens.saved` (**seconds**). Both are listed below.
|
||||
|
||||
---
|
||||
|
||||
## The savings panel — start here
|
||||
|
||||
**`headroom.proxy.tokens.saved`** is the headline number. It already combines compression + tool-schema deferral — no need to add anything to it.
|
||||
|
||||
| Metric | What it shows |
|
||||
|---|---|
|
||||
| **`headroom.proxy.tokens.saved`** *(OTel)* | **Total input tokens Headroom kept out of the request.** Compression + tool savings, combined. This is your hero number. |
|
||||
| `headroom.proxy.savings.usd{source}` *(OTel)* | **Dollars saved**, split by layer: `compression`, `tool_schema`, `output_shaping`, `provider_cache`. Sum for the total. |
|
||||
| `headroom_persistent_savings_tokens_saved_total` | Same tokens-saved number, but **survives proxy restarts**. Use for "lifetime saved" tiles. |
|
||||
| `headroom_persistent_savings_compression_savings_usd_total` | **Lifetime dollars saved**, durable across restarts. |
|
||||
| `headroom_tokens_input_total` | Input tokens actually sent upstream (post-compression). The denominator for a reduction %. |
|
||||
| `headroom_tokens_output_total` | Output tokens returned by the provider. |
|
||||
|
||||
```promql
|
||||
# Hero tile: tokens saved per second
|
||||
rate(headroom_tokens_saved_total[5m])
|
||||
+ sum(rate(headroom_savings_attributed_tokens_total{source="tool_search",realized="true"}[5m]))
|
||||
|
||||
# Context reduction %
|
||||
100 * rate(headroom_tokens_saved_total[5m])
|
||||
/ clamp_min(rate(headroom_tokens_input_total[5m]) + rate(headroom_tokens_saved_total[5m]), 1)
|
||||
|
||||
# Lifetime tiles (survive restart)
|
||||
headroom_persistent_savings_tokens_saved_total
|
||||
headroom_persistent_savings_compression_savings_usd_total
|
||||
```
|
||||
|
||||
> **One catch on the Prometheus side.** `headroom_tokens_saved_total` is compression **only** — it leaves out tool-schema deferral. The OTel `headroom.proxy.tokens.saved` includes both. That's why the query above adds the `tool_search` term back in. On tool-heavy workloads the gap is large.
|
||||
|
||||
---
|
||||
|
||||
## Latency panel
|
||||
|
||||
All Prometheus timings are in **milliseconds**, exposed as `_sum` / `_count` / `_min` / `_max`. Build means with `rate(sum)/rate(count)`.
|
||||
|
||||
| Metric | What it shows |
|
||||
|---|---|
|
||||
| **`headroom_overhead_ms_*`** | **Latency Headroom itself adds.** Handler entry → end of compression. Excludes the LLM call. This is the "what does this cost us" number. |
|
||||
| `headroom_latency_ms_*` | Total request duration, including the provider. |
|
||||
| `headroom_ttfb_ms_*` | Time to first byte from upstream. Streaming requests only. |
|
||||
| `headroom_stage_timing_ms_*{path,stage}` | Where time went inside the handler — `compression_first_stage`, `upstream_connect`, `memory_context`, etc. |
|
||||
| `headroom_transform_timing_ms_*{transform}` | Time per compression transform. Use to find a slow transform. |
|
||||
|
||||
```promql
|
||||
# Headroom's added overhead, mean ms
|
||||
rate(headroom_overhead_ms_sum[5m]) / rate(headroom_overhead_ms_count[5m])
|
||||
|
||||
# End-to-end, mean ms
|
||||
rate(headroom_latency_ms_sum[5m]) / rate(headroom_latency_ms_count[5m])
|
||||
|
||||
# Slowest stages
|
||||
topk(5, rate(headroom_stage_timing_ms_sum[5m]) / rate(headroom_stage_timing_ms_count[5m]))
|
||||
```
|
||||
|
||||
> **No percentiles are available.** There are no histogram buckets on `/metrics`, and the OTel histograms ship with default buckets that put every request into one bucket, so `histogram_quantile()` returns nonsense. **Means work fine.** For real p95/p99 today, use the `headroom perf` CLI.
|
||||
>
|
||||
> Also: divide each `_sum` by **its own** `_count`. Overhead and TTFB are only sampled when > 0, so their counts are smaller than the latency count.
|
||||
|
||||
---
|
||||
|
||||
## Cache panel
|
||||
|
||||
| Metric | What it shows |
|
||||
|---|---|
|
||||
| `headroom_provider_cache_hit_requests_total{provider}` | Requests that read from the provider's prompt cache. |
|
||||
| `headroom_provider_cache_requests_total{provider}` | Requests with any cache activity. **The correct denominator for hit rate.** |
|
||||
| `headroom_cache_read_tokens_total{provider}` | Tokens served from cache (the discounted ones). |
|
||||
| `headroom_cache_write_tokens_total{provider}` | Tokens written into cache (these carry a premium). |
|
||||
| `headroom_cache_write_ttl_tokens_total{provider,ttl}` | Cache writes split by TTL — `5m` vs `1h`. |
|
||||
| `headroom_uncached_input_tokens_total{provider}` | Input tokens that missed cache entirely. |
|
||||
| `headroom_cache_bust_total` | Requests where compression broke a cached prefix. **Should stay near zero.** |
|
||||
| `headroom_cache_miss_attribution_total{provider,reason}` | Why a cached prefix missed — `ttl_expiry`, `prefix_change`, `unknown`. |
|
||||
|
||||
```promql
|
||||
# Cache hit rate by provider
|
||||
sum by (provider) (rate(headroom_provider_cache_hit_requests_total[5m]))
|
||||
/ sum by (provider) (rate(headroom_provider_cache_requests_total[5m]))
|
||||
|
||||
# Compression breaking cache — alert if this rises
|
||||
rate(headroom_cache_bust_total[5m])
|
||||
```
|
||||
|
||||
> **Don't use `headroom_requests_cached_total` as a hit rate.** It mixes the provider's prompt cache with Headroom's own response cache into one boolean, so it measures neither.
|
||||
|
||||
---
|
||||
|
||||
## Traffic & health panel
|
||||
|
||||
| Metric | What it shows |
|
||||
|---|---|
|
||||
| `headroom_requests_total` | Requests handled. Unlabelled. |
|
||||
| `headroom_requests_by_provider{provider}` | Traffic split by provider — `anthropic`, `openai`, `gemini`, `bedrock`… |
|
||||
| `headroom_requests_by_model{model}` | Traffic split by model. Capped at 1024 distinct; overflow lands in `model="other"`. |
|
||||
| `headroom_requests_failed_total` | Upstream 5xx errors. |
|
||||
| `headroom_requests_rate_limited_total` | Requests **Headroom** rejected via its own rate limiter (not upstream 429s). |
|
||||
| `headroom_compression_failed_total{reason}` | Compression failures — `timeout` or `error`. Fails open, so traffic keeps flowing but savings quietly stop. **Worth an alert.** |
|
||||
| `headroom_compression_quarantine_total{event}` | Compression disabled after repeated timeouts — `activated`, `skipped`, `released`. |
|
||||
| `headroom_inbound_requests_active` | In-flight requests, gauge. Counts all HTTP including `/metrics`. |
|
||||
| `headroom_active_ws_sessions` | Live Codex WebSocket sessions, gauge. |
|
||||
|
||||
```promql
|
||||
# Failure rate
|
||||
rate(headroom_requests_failed_total[5m])
|
||||
/ clamp_min(rate(headroom_requests_total[5m]) + rate(headroom_requests_failed_total[5m]), 1)
|
||||
|
||||
# Savings silently stopped
|
||||
sum by (reason) (rate(headroom_compression_failed_total[5m]))
|
||||
|
||||
# Traffic mix
|
||||
sum by (provider) (rate(headroom_requests_by_provider[5m]))
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Anthropic subscription panel
|
||||
|
||||
Only if you're on an Anthropic OAuth/subscription plan. OTel only, gauges, no labels.
|
||||
|
||||
| Metric | What it shows |
|
||||
|---|---|
|
||||
| `headroom.subscription.5h_utilization_pct` | How much of the 5-hour rate-limit window is used (0–100). |
|
||||
| `headroom.subscription.7d_utilization_pct` | Same for the 7-day window. |
|
||||
| `headroom.subscription.5h_seconds_to_reset` | Seconds until the 5-hour window resets. |
|
||||
| `headroom.subscription.7d_seconds_to_reset` | Seconds until the 7-day window resets. |
|
||||
| `headroom.subscription.overage_usd` | Extra-usage credits consumed, in dollars. |
|
||||
|
||||
---
|
||||
|
||||
## Attribution — where savings came from
|
||||
|
||||
| Metric | What it shows |
|
||||
|---|---|
|
||||
| `headroom_savings_attributed_tokens_total{source,realized}` | Tokens saved, broken out by named source. `source="tool_search"` is tool-schema deferral. |
|
||||
| `headroom_savings_attributed_usd_total{source,realized}` | Dollars saved by source. **Gauge, can go negative** — don't `rate()` it. |
|
||||
| `headroom_savings_attribution_events_total{source,realized}` | How often each source contributed. |
|
||||
| `headroom_waste_signal_tokens_total{signal}` | Wasteful patterns *detected* in the input — `json_bloat`, `base64`, `repetition`, `reread`… This is diagnosis, **not savings**. |
|
||||
|
||||
These rows *explain* the headline total — they are never added to it.
|
||||
|
||||
---
|
||||
|
||||
## Compression internals
|
||||
|
||||
| Metric | What it shows |
|
||||
|---|---|
|
||||
| `headroom.compression.tokens.input` *(OTel)* | Tokens going into the compression pipeline. |
|
||||
| `headroom.compression.tokens.output` *(OTel)* | Tokens coming out. |
|
||||
| `headroom.compression.tokens.saved` *(OTel)* | The difference. Pipeline-level view of compression only. |
|
||||
| `headroom.compression.runs` *(OTel)* | Pipeline executions. Note: **per pipeline run, not per request.** |
|
||||
| `headroom.compression.pipeline.duration` *(OTel, seconds)* | How long the pipeline took. |
|
||||
| `headroom.compression.transforms{transform}` *(OTel)* | Which transforms fired. **High cardinality — drop or aggregate at the collector.** |
|
||||
|
||||
---
|
||||
|
||||
## Five things that will break a dashboard
|
||||
|
||||
1. **Only savings counters survive a restart.** 55 of 60 Prometheus families reset to zero when the proxy restarts. Only `headroom_persistent_savings_*` is durable, and it needs `HEADROOM_WORKSPACE_DIR` on a persistent volume — otherwise it resets on every deploy.
|
||||
|
||||
2. **No percentiles anywhere.** Use means. See the latency section.
|
||||
|
||||
3. **`headroom_latency_ms` measures differently for streaming.** On streaming requests the timer starts *after* compression, so end-to-end is `latency + overhead`. On non-streaming it's just `latency`. Don't mix both in one panel.
|
||||
|
||||
4. **A 5xx erases its own savings.** Requests that fail upstream are dropped from every savings and token counter. During a provider incident, savings rates look artificially clean while throughput falls.
|
||||
|
||||
5. **`/metrics` needs auth if you set a proxy token.** With `HEADROOM_PROXY_TOKEN` set, any non-loopback scraper must send `Authorization: Bearer <token>`. Loopback is always exempt.
|
||||
|
||||
---
|
||||
|
||||
## Metrics the docs mention that don't exist
|
||||
|
||||
If panels came back empty, this is probably why. These names appear in the published docs but not in the code:
|
||||
|
||||
`headroom_compression_ratio` · `headroom_latency_seconds` (and `_bucket`) · `headroom_cache_hits_total` · `headroom_cache_misses_total` · `headroom_cost_usd_total` · the `mode="optimize"` label on `headroom_requests_total`
|
||||
|
||||
The shipped `examples/grafana/headroom-dashboard.json` also filters every panel on `pool` and `hook` labels that no metric emits — the dropdowns will be permanently empty. Its metric names are otherwise correct.
|
||||
|
||||
---
|
||||
|
||||
## Setup reference
|
||||
|
||||
```bash
|
||||
# Prometheus — nothing to do, GET /metrics is always on
|
||||
|
||||
# OpenTelemetry
|
||||
pip install "headroom-ai[proxy,otel]"
|
||||
export HEADROOM_OTEL_METRICS_ENABLED=1
|
||||
export HEADROOM_OTEL_METRICS_ENDPOINT=https://otel.corp.example/v1/metrics
|
||||
export HEADROOM_OTEL_METRICS_HEADERS="authorization=Bearer XXX"
|
||||
export HEADROOM_OTEL_RESOURCE_ATTRIBUTES="service.instance.id=$HOSTNAME"
|
||||
```
|
||||
|
||||
| Variable | Default | Notes |
|
||||
|---|---|---|
|
||||
| `HEADROOM_OTEL_METRICS_ENABLED` | `0` | Master switch |
|
||||
| `HEADROOM_OTEL_METRICS_EXPORTER` | `otlp_http` | Or `console`. No gRPC exporter exists. |
|
||||
| `HEADROOM_OTEL_METRICS_ENDPOINT` | unset | Passed verbatim — `/v1/metrics` is **not** appended |
|
||||
| `HEADROOM_OTEL_METRICS_HEADERS` | unset | `k=v,k2=v2` |
|
||||
| `HEADROOM_OTEL_METRICS_EXPORT_INTERVAL_MS` | `10000` | |
|
||||
| `HEADROOM_OTEL_SERVICE_NAME` | `headroom-proxy` | |
|
||||
| `HEADROOM_OTEL_RESOURCE_ATTRIBUTES` | unset | **Set `service.instance.id` here** — Headroom doesn't, and replicas will collide |
|
||||
|
||||
Verify with `curl -s localhost:8787/stats | jq .otel`.
|
||||
|
||||
**Multi-tenant labels:** `register_otel_metric_attribute_provider()` adds request-scoped attributes (tenant, team, cost centre) to every OTel datapoint. Max 16 attributes, 256 chars each.
|
||||
|
||||
**Air-gapped deployments:** `HEADROOM_OFFLINE=1` disables all outbound traffic — the anonymous usage beacon (which is **on by default**), the update check, and model downloads.
|
||||
|
||||
---
|
||||
+48
-10
@@ -207,27 +207,43 @@ _PROFILES: dict[str, AgentSavingsProfile] = {
|
||||
def get_agent_savings_profile(name: str | None = None) -> AgentSavingsProfile:
|
||||
"""Return a named agent savings profile.
|
||||
|
||||
An unrecognized name falls back to the ``balanced`` profile with a warning
|
||||
instead of raising. The savings profile is a soft config knob, but it is
|
||||
resolved during proxy startup (``proxy_pipeline_kwargs`` -> ``create_app``),
|
||||
so raising here takes the whole proxy down before it can open its port. That
|
||||
happens on desktop/runtime version skew: a newer client requests a profile
|
||||
(e.g. ``coding``) that an older pinned or fallback runtime predates. Degrade
|
||||
to ``balanced`` rather than leaving the user with no proxy at all.
|
||||
An unrecognized name degrades with a warning instead of raising. The savings
|
||||
profile is a soft config knob, but it is resolved during proxy startup
|
||||
(``proxy_pipeline_kwargs`` -> ``create_app``), so raising here takes the
|
||||
whole proxy down before it can open its port. That happens on desktop/runtime
|
||||
version skew: a newer client requests a profile (e.g. ``coding``) that an
|
||||
older pinned or fallback runtime predates. Degrade rather than leaving the
|
||||
user with no proxy at all.
|
||||
|
||||
**Where it degrades to matters.** This used to land on ``balanced``
|
||||
unconditionally, which is a drastically different posture from the
|
||||
out-of-box default: cache->token mode, cross-turn dedup off, tool-search
|
||||
off, user messages uncompressed, the message floor 25x higher (250 vs 10)
|
||||
and the block floor 20x higher (500 vs 25). A single typo in
|
||||
``HEADROOM_SAVINGS_PROFILE`` therefore silently reconfigured the whole
|
||||
proxy, and the only trace was one WARNING at startup that operators read
|
||||
past. Prefer :data:`DEFAULT_PROFILE` — the documented out-of-box posture and
|
||||
the same thing an unset variable resolves to, so a typo now costs nothing.
|
||||
``balanced`` remains the last resort for the genuine version-skew case,
|
||||
where an older runtime has no ``DEFAULT_PROFILE`` entry to fall back to.
|
||||
"""
|
||||
|
||||
key = (name or DEFAULT_PROFILE).strip().lower()
|
||||
profile = _PROFILES.get(key)
|
||||
if profile is not None:
|
||||
return profile
|
||||
fallback_name = DEFAULT_PROFILE if DEFAULT_PROFILE in _PROFILES else FALLBACK_PROFILE
|
||||
valid = ", ".join(sorted(_PROFILES))
|
||||
logger.warning(
|
||||
"unknown savings profile %r; falling back to %r (known: %s)",
|
||||
"unknown savings profile %r; falling back to %r (known: %s). "
|
||||
"Set HEADROOM_SAVINGS_PROFILE to one of the known names, or unset it to "
|
||||
"get %r explicitly.",
|
||||
name,
|
||||
FALLBACK_PROFILE,
|
||||
fallback_name,
|
||||
valid,
|
||||
DEFAULT_PROFILE,
|
||||
)
|
||||
return _PROFILES[FALLBACK_PROFILE]
|
||||
return _PROFILES[fallback_name]
|
||||
|
||||
|
||||
def apply_agent_savings_env_defaults(
|
||||
@@ -300,6 +316,28 @@ def proxy_pipeline_kwargs(config: object) -> dict[str, object]:
|
||||
# unset → Kompress decides / ambient default applies).
|
||||
if profile.target_ratio is not None:
|
||||
kwargs["target_ratio"] = profile.target_ratio
|
||||
# Block-compression char floor. Every OTHER router pipeline kwarg in this
|
||||
# function travels on the config object; this one alone was populated
|
||||
# only from ``HEADROOM_MIN_CHARS_FOR_BLOCK`` (read below), so a proxy
|
||||
# whose config carries ``savings_profile="coding"`` but whose process env
|
||||
# was never seeded applied every sibling coding knob while this floor
|
||||
# silently stayed at ``ContentRouterConfig.min_chars_for_block_compression``
|
||||
# (500) instead of the profile's 25 — a 20x gap on the gate that governs
|
||||
# tool_result blocks, the dominant content type in agent traffic.
|
||||
#
|
||||
# NOTE: this does not make the profile fully config-deliverable. The
|
||||
# profile's ``cross_turn_dedup`` / ``tool_search`` / ``lossless_then_lossy``
|
||||
# / ``protect_reads`` / ``code_aware`` / ``effort_router`` / ``lossless``
|
||||
# fields are still env-only, but by a different mechanism: their consumers
|
||||
# read ``os.environ`` directly (ContentRouter.__init__ for HEADROOM_DEDUPE,
|
||||
# the Anthropic handler for HEADROOM_TOOL_SEARCH) and never pass through
|
||||
# this function at all. Those remain seed-dependent and are the reason a
|
||||
# profile can still be half-applied; fixing them means threading each
|
||||
# consumer, which is a larger change than this one.
|
||||
#
|
||||
# The env read below still wins, since it is an explicit operator override.
|
||||
if profile.min_chars_for_block is not None:
|
||||
kwargs["min_chars_for_block_compression"] = profile.min_chars_for_block
|
||||
|
||||
if getattr(config, "compress_user_messages", False):
|
||||
kwargs["compress_user_messages"] = True
|
||||
|
||||
@@ -157,6 +157,12 @@ class PerfRecord:
|
||||
tokens_before: int = 0
|
||||
tokens_after: int = 0
|
||||
tokens_saved: int = 0
|
||||
# Tokens the forwarded request GREW by (PERF ``tok_inflated``). Both
|
||||
# endpoints are clamped — ``tok_saved`` at zero and ``tok_inflated`` at zero
|
||||
# — so a turn that left the proxy bigger reports ``tok_saved=0`` and hides
|
||||
# its growth in a field nothing downstream read. Carrying it here is what
|
||||
# lets the report state net alongside gross instead of implying they agree.
|
||||
tokens_inflated: int = 0
|
||||
tool_saved: int = 0
|
||||
cache_read: int = 0
|
||||
cache_write: int = 0
|
||||
@@ -397,6 +403,7 @@ def parse_log_files(last_n_hours: float = 168.0) -> PerfReport:
|
||||
tokens_before=int(kv.get("tok_before", 0)),
|
||||
tokens_after=int(kv.get("tok_after", 0)),
|
||||
tokens_saved=int(kv.get("tok_saved", 0)),
|
||||
tokens_inflated=int(kv.get("tok_inflated", 0)),
|
||||
tool_saved=int(kv.get("tool_saved", 0)),
|
||||
savings_breakdown=_decode_perf_savings(kv.get("savings", "none")),
|
||||
cache_read=int(kv.get("cache_read", 0)),
|
||||
@@ -546,6 +553,19 @@ def format_report(report: PerfReport) -> str:
|
||||
# include tool bytes), so it used to render as a rival "Tool saved" line — which
|
||||
# read as a side metric and hid the win on tool-heavy turns where tok_saved=0.
|
||||
lines.append(f"Tokens saved: {total_headline_saved:,} ({headline_pct:.1f}% reduction)")
|
||||
# Gross vs net. ``tok_saved`` is clamped at zero per request, so turns
|
||||
# where Headroom made the body BIGGER (CCR proactive expansion, memory
|
||||
# injection) contribute nothing negative to the headline — their growth
|
||||
# lands in ``tok_inflated`` instead, which nothing here used to read.
|
||||
# Printing "321,239,562 -> 313,274,727" directly above "8,455,763 saved"
|
||||
# implies the two reconcile; they differ by exactly the inflation. Show
|
||||
# it whenever it is non-zero so the arithmetic closes on the page.
|
||||
total_inflated = sum(r.tokens_inflated for r in records)
|
||||
if total_inflated > 0:
|
||||
lines.append(
|
||||
f" · inflated {total_inflated:,} "
|
||||
f"(net message reduction {total_before - total_after:,})"
|
||||
)
|
||||
if total_tool_saved > 0:
|
||||
lines.append(f" · messages {max(0, total_saved):,}")
|
||||
lines.append(f" · tool schemas {total_tool_saved:,}")
|
||||
@@ -698,6 +718,31 @@ def format_report(report: PerfReport) -> str:
|
||||
lines.append(
|
||||
f" {name}: {avg_pct:.1f}% avg reduction, {len(recs)} uses, {total_s:,} saved"
|
||||
)
|
||||
# This table is built ONLY from "Transform NAME: B -> A tokens (saved N)"
|
||||
# lines, which just one engine emits (transforms/pipeline.py). The
|
||||
# OpenAI-Responses engine (transforms/compression_units.py +
|
||||
# compression_batches.py) applies the same strategies and contains no
|
||||
# logging calls at all, so none of its work appears above. On real
|
||||
# traffic that hid ~7M of ~8.5M message-token savings — the table read
|
||||
# "content_router: 189,783 saved" against a PERF total 44x larger, which
|
||||
# invites exactly the wrong conclusion about which compressors work.
|
||||
#
|
||||
# State the divergence, NOT a coverage ratio. The two totals are
|
||||
# different populations and neither strictly contains the other: the
|
||||
# Transform lines carry no request_id, fire once per pipeline STAGE (so
|
||||
# several can describe one request), and are emitted before the forwarder
|
||||
# decides anything — a mutation later discarded by the signed-thinking
|
||||
# byte-lock still logs its "saved" here while the request's PERF line
|
||||
# correctly reports 0. So "table covers X of Y" would be a false subset
|
||||
# claim in both directions; report the two sums and let the reader judge.
|
||||
table_total = sum(r.tokens_saved for r in report.transform_records)
|
||||
perf_total = sum(r.tokens_saved for r in report.perf_records)
|
||||
if table_total != perf_total:
|
||||
lines.append(
|
||||
f" ! stage-level total {table_total:,} != PERF message total {perf_total:,} "
|
||||
"— this table sees only engines that emit a Transform line, counts "
|
||||
"per stage, and does not check whether the mutation shipped"
|
||||
)
|
||||
lines.append("")
|
||||
|
||||
# Router routing breakdown
|
||||
@@ -717,11 +762,24 @@ def format_report(report: PerfReport) -> str:
|
||||
f" Excluded: {total_excluded} ({total_excluded / total_all * 100:.0f}%) — Read/Glob outputs"
|
||||
)
|
||||
lines.append(
|
||||
f" Skipped: {total_skipped} ({total_skipped / total_all * 100:.0f}%) — <50 words"
|
||||
f" Skipped: {total_skipped} ({total_skipped / total_all * 100:.0f}%) — below size floor"
|
||||
)
|
||||
lines.append(
|
||||
f" Unchanged: {total_unchanged} ({total_unchanged / total_all * 100:.0f}%) — ratio too high"
|
||||
)
|
||||
# These four buckets are NOT the router's full outcome space — the
|
||||
# `[router] route_counts=` line carries 17 keys, and the ones omitted
|
||||
# here (cache_hit, system_msg, error_protected, already_compressed,
|
||||
# …) are individually larger than "Excluded". Percentages taken over
|
||||
# this subset therefore overstate every share: on real traffic the
|
||||
# "skipped" bucket read 77% here against 49.5% of actual terminal
|
||||
# fates, which reads as a mis-set threshold rather than a narrow
|
||||
# denominator. Say what the denominator is instead of implying it is
|
||||
# everything.
|
||||
lines.append(
|
||||
f" (shares are of these 4 buckets only, n={total_all}; "
|
||||
"see `[router] route_counts=` for the full outcome space)"
|
||||
)
|
||||
if total_excluded > total_compressed * 3:
|
||||
lines.append(" ! Excluded tools dominate — consider compressing stale Read outputs")
|
||||
lines.append("")
|
||||
@@ -803,6 +861,7 @@ PERF_RECORD_FIELDS = [
|
||||
# Appended last so every existing CSV column keeps its position; a reader
|
||||
# that indexes by name is unaffected either way.
|
||||
"from_response_cache",
|
||||
"tokens_inflated",
|
||||
]
|
||||
|
||||
|
||||
|
||||
@@ -768,11 +768,31 @@ class SavingsTracker:
|
||||
delta_output_tokens_saved = max(_coerce_int(output_tokens_saved), 0)
|
||||
delta_cache_read_tokens = _coerce_int(cache_read_tokens)
|
||||
priced = estimated_savings_usd
|
||||
delta_savings_usd = (
|
||||
max(_coerce_float(priced.get("compression")), 0.0)
|
||||
if priced is not None
|
||||
else _estimate_compression_savings_usd(model, delta_tokens_saved)
|
||||
)
|
||||
# ``estimate_request_savings_usd`` prices FOUR buckets, but this method
|
||||
# only ever read three — ``tool_schema`` was computed and dropped on the
|
||||
# floor. Tool-schema deferral is a quarter of the token headline on real
|
||||
# traffic (2.7M of 11.2M), and ``tokens_saved`` here is the bare
|
||||
# message-level figure (the caller folds deferral in separately for the
|
||||
# ledger, see prometheus_metrics.record_request), so the dollars were
|
||||
# simply missing rather than counted elsewhere. That is why "Cost saved"
|
||||
# read materially below the token-savings percent on the same traffic.
|
||||
# Add it to the compression bucket, matching how the PERF headline and
|
||||
# perf/analyzer fold deferral into one number.
|
||||
if priced is not None:
|
||||
# Two DISJOINT buckets: the caller passes bare message savings as
|
||||
# ``compression_tokens_saved`` and deferral separately as
|
||||
# ``tool_schema_tokens_saved`` (see prometheus_metrics.record_request),
|
||||
# so summing them is additive, not double counting. Written as a
|
||||
# statement rather than folded into the ternary below — a money path
|
||||
# should not depend on the reader knowing that ``a + b if c else d``
|
||||
# groups as ``(a + b) if c else d``.
|
||||
delta_savings_usd = max(_coerce_float(priced.get("compression")), 0.0) + max(
|
||||
_coerce_float(priced.get("tool_schema")), 0.0
|
||||
)
|
||||
else:
|
||||
# No priced breakdown available: only message savings are known here,
|
||||
# so this path stays message-only exactly as before.
|
||||
delta_savings_usd = _estimate_compression_savings_usd(model, delta_tokens_saved)
|
||||
delta_output_savings_usd = (
|
||||
max(_coerce_float(priced.get("output_shaping")), 0.0)
|
||||
if priced is not None
|
||||
|
||||
@@ -66,7 +66,7 @@ except ImportError:
|
||||
sys.path.insert(0, str(Path(__file__).parent.parent.parent))
|
||||
|
||||
from headroom._version import __version__
|
||||
from headroom.agent_savings import proxy_pipeline_kwargs
|
||||
from headroom.agent_savings import DEFAULT_PROFILE, proxy_pipeline_kwargs
|
||||
from headroom.cache.compression_feedback import get_compression_feedback
|
||||
from headroom.cache.compression_store import format_retrieval_miss_detail, get_compression_store
|
||||
from headroom.ccr import (
|
||||
@@ -1737,6 +1737,38 @@ class HeadroomProxy(
|
||||
if self.config.mode == PROXY_MODE_CACHE:
|
||||
logger.info(" Prefix freeze: strict (all prior turns immutable)")
|
||||
logger.info(" Mutations: latest turn only")
|
||||
# Effective compression posture, resolved (not merely requested). The
|
||||
# savings profile reaches the router through two paths — the config
|
||||
# object and the seeded process env — and `setdefault` semantics mean a
|
||||
# stale HEADROOM_* value silently overrides the profile that names it.
|
||||
# A deployment can therefore log `savings_profile=coding` while actually
|
||||
# running balanced's thresholds, and the only prior evidence was a
|
||||
# single easily-missed WARNING. Print what is actually in force so
|
||||
# "which profile am I really running?" is answerable from the banner.
|
||||
try:
|
||||
_eff = proxy_pipeline_kwargs(self.config)
|
||||
# Read cross-turn dedup off the CONSTRUCTED router, not off the env.
|
||||
# ContentRouter resolves it as `config.enable_cross_turn_dedup OR
|
||||
# $HEADROOM_DEDUPE`, so reporting the env alone would be a guess that
|
||||
# happens to be right only while nothing sets the config field. A
|
||||
# banner line exists to be trusted; it must read what was resolved.
|
||||
_dedupe: object = "unknown"
|
||||
for _t in getattr(self.anthropic_pipeline, "transforms", []):
|
||||
if isinstance(_t, ContentRouter):
|
||||
_dedupe = bool(getattr(_t, "_cross_turn_dedup_enabled", False))
|
||||
break
|
||||
logger.info(
|
||||
"Savings profile: %s (effective: min_tokens=%s min_chars_block=%s "
|
||||
"compress_user=%s dedupe=%s tool_search=%s)",
|
||||
self.config.savings_profile or DEFAULT_PROFILE,
|
||||
_eff.get("min_tokens_to_compress", "default"),
|
||||
_eff.get("min_chars_for_block_compression", "default(500)"),
|
||||
_eff.get("compress_user_messages", False),
|
||||
_dedupe,
|
||||
os.environ.get("HEADROOM_TOOL_SEARCH", "1") in ("1", "true", "yes", "on", "auto"),
|
||||
)
|
||||
except Exception: # never let a banner line block startup
|
||||
logger.debug("effective savings-profile banner skipped", exc_info=True)
|
||||
logger.info(f"Caching: {'ENABLED' if self.config.cache_enabled else 'DISABLED'}")
|
||||
logger.info(f"Rate Limiting: {'ENABLED' if self.config.rate_limit_enabled else 'DISABLED'}")
|
||||
logger.info(
|
||||
|
||||
@@ -5552,7 +5552,16 @@ class ContentRouter(Transform):
|
||||
if route_counts["user_msg"]:
|
||||
parts.append(f"{route_counts['user_msg']} skipped (user)")
|
||||
if route_counts["small"]:
|
||||
parts.append(f"{route_counts['small']} skipped (<50 words)")
|
||||
# Report the thresholds actually in force, not a literal. This line
|
||||
# used to read "skipped (<50 words)" unconditionally: wrong number
|
||||
# (the message gate is `min_tokens`, which profiles set anywhere from
|
||||
# 10 to 250), wrong unit (tokens and characters, never words), and it
|
||||
# merged two different gates under one label. Operators read it as
|
||||
# evidence of a mis-set threshold and tuned the wrong knob.
|
||||
parts.append(
|
||||
f"{route_counts['small']} skipped "
|
||||
f"(<{min_tokens} tok msg / <{min_chars_for_block_compression} chars block)"
|
||||
)
|
||||
if route_counts["recent_code"]:
|
||||
parts.append(f"{route_counts['recent_code']} protected (recent code)")
|
||||
if route_counts["analysis_ctx"]:
|
||||
|
||||
@@ -10,6 +10,7 @@ from click.testing import CliRunner
|
||||
|
||||
from headroom.agent_savings import (
|
||||
AGENT_90_PROFILE,
|
||||
DEFAULT_PROFILE,
|
||||
apply_agent_savings_env_defaults,
|
||||
apply_agent_savings_profile,
|
||||
get_agent_savings_profile,
|
||||
@@ -170,17 +171,30 @@ def test_agent_savings_env_defaults_preserve_user_overrides() -> None:
|
||||
assert env["HEADROOM_SMART_CRUSHER_COMPACTION"] == "0"
|
||||
|
||||
|
||||
def test_unknown_agent_savings_profile_falls_back_to_balanced(
|
||||
def test_unknown_agent_savings_profile_falls_back_to_default(
|
||||
caplog: pytest.LogCaptureFixture,
|
||||
) -> None:
|
||||
# An unknown profile must NOT raise: it's resolved during proxy startup, so
|
||||
# raising takes the whole proxy down before it opens its port (desktop asked
|
||||
# for a profile a fallback runtime predates). Degrade to "balanced" instead.
|
||||
# for a profile a fallback runtime predates).
|
||||
#
|
||||
# It must degrade to the DEFAULT profile, not to "balanced". The two are not
|
||||
# interchangeable: balanced flips cache->token mode, turns cross-turn dedup
|
||||
# and tool-search off, stops compressing user messages, and raises the
|
||||
# message floor 25x (250 vs 10) and the block floor 20x (500 vs 25). A typo
|
||||
# in HEADROOM_SAVINGS_PROFILE used to silently reconfigure the entire proxy
|
||||
# into that posture, which is strictly worse than behaving as if the
|
||||
# variable were unset.
|
||||
with caplog.at_level(logging.WARNING):
|
||||
profile = get_agent_savings_profile("missing")
|
||||
assert profile is get_agent_savings_profile("balanced")
|
||||
assert profile is get_agent_savings_profile(None)
|
||||
assert profile.name == DEFAULT_PROFILE
|
||||
assert profile is not get_agent_savings_profile("balanced")
|
||||
assert "unknown savings profile" in caplog.text
|
||||
assert "missing" in caplog.text
|
||||
# The warning has to name the resolved profile, so an operator reading it
|
||||
# knows what they actually got rather than only what they asked for.
|
||||
assert DEFAULT_PROFILE in caplog.text
|
||||
|
||||
|
||||
def test_with_target_savings_recomputes_target_ratio() -> None:
|
||||
@@ -784,3 +798,45 @@ def test_agent_savings_smoke_fixture_passes_real_gate(tmp_path) -> None:
|
||||
assert "codex: 91.0% savings meets 90.0%" in gate_result.output
|
||||
assert "cursor: 93.0% savings meets 90.0%" in gate_result.output
|
||||
assert "100.0% accuracy meets 90.0%" in gate_result.output
|
||||
|
||||
|
||||
def test_coding_profile_min_chars_block_reaches_router_without_env_seeding(
|
||||
monkeypatch: pytest.MonkeyPatch,
|
||||
) -> None:
|
||||
"""The block-char floor must travel on the config object, not env only.
|
||||
|
||||
Every other router pipeline kwarg this function builds travels on the config
|
||||
object; ``min_chars_for_block`` alone was populated only from
|
||||
``HEADROOM_MIN_CHARS_FOR_BLOCK`` (emitted by ``proxy_env()``). A proxy whose
|
||||
config carried ``savings_profile="coding"`` but whose process env was never
|
||||
seeded applied every sibling coding knob while this floor silently stayed at
|
||||
``ContentRouterConfig.min_chars_for_block_compression`` (500) instead of the
|
||||
profile's 25 — a 20x gap on the gate that governs tool_result blocks.
|
||||
|
||||
Note this does not make the profile fully config-deliverable: fields whose
|
||||
consumers read ``os.environ`` directly (``cross_turn_dedup`` via
|
||||
ContentRouter, ``tool_search`` via the Anthropic handler) never pass through
|
||||
this function and remain seed-dependent.
|
||||
"""
|
||||
monkeypatch.delenv("HEADROOM_MIN_CHARS_FOR_BLOCK", raising=False)
|
||||
|
||||
class _Config:
|
||||
savings_profile = "coding"
|
||||
min_tokens_to_crush = 500
|
||||
|
||||
kwargs = proxy_pipeline_kwargs(_Config())
|
||||
assert kwargs["min_chars_for_block_compression"] == 25
|
||||
assert kwargs["min_tokens_to_compress"] == 10
|
||||
|
||||
|
||||
def test_explicit_min_chars_block_env_overrides_the_profile(
|
||||
monkeypatch: pytest.MonkeyPatch,
|
||||
) -> None:
|
||||
"""An explicit operator override still wins over the profile value."""
|
||||
monkeypatch.setenv("HEADROOM_MIN_CHARS_FOR_BLOCK", "120")
|
||||
|
||||
class _Config:
|
||||
savings_profile = "coding"
|
||||
min_tokens_to_crush = 500
|
||||
|
||||
assert proxy_pipeline_kwargs(_Config())["min_chars_for_block_compression"] == 120
|
||||
|
||||
Reference in New Issue
Block a user