fix(beacon): report all-layers savings, not context-compression only (#2796)

## Description

`rates.saved_pct` and `rates.yield_pct` in the beacon payload divide
`tokens_saved` by `original` / `attempted`. Tool-schema deferral never
lands in either denominator — `outcome.py` says so explicitly, and
`tokens.tool_saved` exists precisely because of it — so every beacon
rate silently reports context compression only.

On a tool-heavy fleet that is not a rounding difference. Across the
first 516 sessions in the corpus the beacon reads **2.80%** where the
dashboard headline for the same traffic reads **12.82%**: 157.6M context
tokens vs 803.9M all-layers, with 646.3M of tool-schema deferral missing
from the ratio.

`headroom/proxy/server.py` already resolved this for the dashboard in
#2737 — `savings_percent` is `all_layers_saved / (input +
all_layers_saved)` and `active_savings_percent` puts tool savings on
both sides of the ratio. The beacon was never brought along. This does
that.

Closes #

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- `headroom/telemetry/session.py`: add `rates.all_layers_saved_pct` and
`rates.all_layers_yield_pct`, computed the way `server.py` builds
`savings_percent` / `active_savings_percent` — tool savings added to
**both** sides, since deferred schemas were attempted work that
succeeded whole.
- `headroom/telemetry/session.py`: extend the `demo()` self-test to
assert both new rates against the existing tool-heavy fixture.
- `deploy/beacon/query.sh`: add an `all_layers_pct` column to the fleet
summary, so the reader stops showing the understated number too.

**Kept alongside `saved_pct` rather than folded into it.** Every row
already in the corpus means context-only under that name; redefining it
would make old and new rows non-comparable with no field to tell them
apart.

**`SCHEMA_VERSION` deliberately stays at 1.** The change is purely
additive, nothing reads the field, and the query uses `union_by_name =
true`, so old and new rows mix cleanly. Happy to bump it if maintainers
want the marker.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

```text
$ python -m headroom.telemetry.session
ok

$ python -m pytest tests/test_savings_tool_search_aggregation.py tests/test_outcome_dual_ruler_funnel.py -q
FAILED tests/test_outcome_dual_ruler_funnel.py::test_ledger_delta_stays_on_the_local_ruler
FAILED tests/test_outcome_dual_ruler_funnel.py::test_ledger_falls_back_to_billed_when_local_omitted
2 failed, 5 passed, 3 warnings in 4.42s

$ ruff check headroom/
All checks passed!

$ mypy --python-version 3.12 headroom/telemetry/session.py
Success: no issues found in 1 source file
```

The two `test_outcome_dual_ruler_funnel.py` failures are **pre-existing
on `main`, not caused by this PR** — verified by `git stash`-ing the
change and re-running:

```text
$ git stash && python -m pytest tests/test_outcome_dual_ruler_funnel.py -q
FAILED tests/test_outcome_dual_ruler_funnel.py::test_ledger_delta_stays_on_the_local_ruler
FAILED tests/test_outcome_dual_ruler_funnel.py::test_ledger_falls_back_to_billed_when_local_omitted
2 failed, 3 passed, 3 warnings in 3.71s
```

## Real Behavior Proof

- **Environment:** macOS 25.4.0 arm64, Python 3.12.6, repo `.venv`,
branch rebased on `upstream/main` @ `d0a86d40`.

- **Exact command / steps:** drive a real `SessionAggregator` with a
tool-heavy outcome (`original=1000`, `attempted=400`,
`tokens_saved=300`, plus `tool_search_deferred_tokens=800` and
`turn_hook_tools_saved_tokens=200`) and print the emitted payload's
`rates` block:

```text
tokens: {"original": 1000, "attempted": 400, "saved": 300, "tool_saved": 1000}
rates:  {
  "saved_pct": 30.0,
  "eligible_pct": 40.0,
  "yield_pct": 75.0,
  "all_layers_saved_pct": 65.0,
  "all_layers_yield_pct": 92.86,
  "cache_read_pct": 50.0,
  "overhead_pct": 5.0
}
```

- **Observed result:** the pre-existing rates are byte-identical (30.0 /
40.0 / 75.0 / 50.0 / 5.0 — no regression), and the two new fields report
the all-layers view: 1300 saved of 2000 sent = **65.0%**, 1300 of 1400
attempted = **92.86%**. Both denominators grow with the numerator,
matching `server.py`.

Cross-checked against the live corpus with DuckDB over the R2 bucket —
restating all 516 sessions both ways reproduces the gap this PR closes:

```text
┌───────────┬────────────┬──────────────────┬──────────────────┬─────────────────────┐
│ ctx_saved │ tool_saved │ all_layers_saved │ beacon_saved_pct │ dashboard_saved_pct │
├───────────┼────────────┼──────────────────┼──────────────────┼─────────────────────┤
│ 157561051 │ 646293457  │ 803854508        │ 2.8              │ 12.82               │
└───────────┴────────────┴──────────────────┴──────────────────┴─────────────────────┘
```

- **Not tested:** no live proxy run was made against a real provider —
the payload above comes from `SessionAggregator` driven directly, which
is the same code path the proxy feeds. The 516 sessions already in R2
are **not backfilled**: they carry `tool_saved`, so the corrected rate
is computable from them today, but their own `rates` block stays
context-only. Only sessions emitted from the next release carry the new
fields.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md`

## Additional Notes

- **Documentation:** N/A — no doc references the beacon `rates` field
names (`grep -rn "saved_pct\|yield_pct" docs/` returns nothing). The
field semantics are documented inline in `session.py`, which this PR
extends.
- **Worth a maintainer opinion:** the dashboard formula adds
`tool_saved` to the *denominator* as well. That is defensible — deferred
schemas were attempted work that succeeded 100% — but it does mean a
tool-heavy session's ratio is partly measuring a layer that is
near-always at full yield. This PR matches the dashboard rather than
inventing a third convention; if the convention should change, it should
change in both places at once.
- **Follow-up:** nothing here changes what the fleet actually saved,
only what the beacon admits to. The 4.6x gap was reporting, not
performance.
This commit is contained in:
Tejas Chopra
2026-08-05 08:33:04 -07:00
committed by GitHub
parent b6f9877c78
commit e9a24f3ec1
2 changed files with 31 additions and 0 deletions
+9
View File
@@ -74,6 +74,15 @@ case "${1:-summary}" in
/ nullif(sum(tokens.attempted), 0), 2) AS yield_pct,
round(sum(tokens.saved) * 100.0
/ nullif(sum(tokens.original), 0), 2) AS saved_pct,
-- saved_pct/yield_pct above are context-compression only, because
-- tool_saved never lands in original/attempted. This is the
-- dashboard headline (server.py `savings_percent`): tool-schema
-- savings on BOTH sides, since deferred schemas were attempted work
-- that succeeded whole. On a tool-heavy fleet the two differ several-
-- fold, so say which one you are quoting.
round(sum(tokens.saved + tokens.tool_saved) * 100.0
/ nullif(sum(tokens.original + tokens.tool_saved), 0), 2)
AS all_layers_pct,
sum(failures) AS failures
FROM sessions;"
;;
+22
View File
@@ -325,6 +325,24 @@ class _Session:
# Of what we touched, how much did we remove? THIS is the
# compressor quality number, and the one Kompress moves.
"yield_pct": _pct(self.tokens_saved, self.attempted_tokens),
# The two above are context-compression only, because
# `tool_saved` never lands in original/attempted. On a
# tool-heavy fleet that understates the product several-fold —
# observed 2.80% vs 12.82% across the first 516 sessions.
# These two are what the dashboard headline shows
# (`tokens.savings_percent` / `active_savings_percent` in
# server.py): tool-schema savings added to BOTH sides, since
# deferred schemas were attempted work that succeeded whole.
# Kept alongside rather than folded into `saved_pct`, which
# already means context-only in every row of the corpus.
"all_layers_saved_pct": _pct(
self.tokens_saved + self.tool_saved_tokens,
self.original_tokens + self.tool_saved_tokens,
),
"all_layers_yield_pct": _pct(
self.tokens_saved + self.tool_saved_tokens,
self.attempted_tokens + self.tool_saved_tokens,
),
# Provider prompt cache participation. Headroom freezes prefixes
# to protect this, so it is the other side of eligible_pct.
"cache_read_pct": _pct(self.cache_read_tokens, self.original_tokens),
@@ -811,6 +829,10 @@ def demo() -> None:
assert r["rates"]["overhead_pct"] == 5.0, r["rates"]
# Tool savings are invisible in `saved` by design; they must not be lost.
assert r["tokens"]["tool_saved"] == 1000, r["tokens"]
# ...and the all-layers rates are the ones that do count them: 1300 saved
# of 2000 sent, 1300 of 1400 attempted. Both denominators grow too.
assert r["rates"]["all_layers_saved_pct"] == 65.0, r["rates"]
assert r["rates"]["all_layers_yield_pct"] == 92.86, r["rates"]
assert r["compression"]["response_cache_hits"] == 1
assert r["tokens"]["cache_write"] == 100 and r["tokens"]["uncached"] == 400