fix(beacon): report all-layers savings, not context-compression only (#2796)
## Description `rates.saved_pct` and `rates.yield_pct` in the beacon payload divide `tokens_saved` by `original` / `attempted`. Tool-schema deferral never lands in either denominator — `outcome.py` says so explicitly, and `tokens.tool_saved` exists precisely because of it — so every beacon rate silently reports context compression only. On a tool-heavy fleet that is not a rounding difference. Across the first 516 sessions in the corpus the beacon reads **2.80%** where the dashboard headline for the same traffic reads **12.82%**: 157.6M context tokens vs 803.9M all-layers, with 646.3M of tool-schema deferral missing from the ratio. `headroom/proxy/server.py` already resolved this for the dashboard in #2737 — `savings_percent` is `all_layers_saved / (input + all_layers_saved)` and `active_savings_percent` puts tool savings on both sides of the ratio. The beacon was never brought along. This does that. Closes # ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/telemetry/session.py`: add `rates.all_layers_saved_pct` and `rates.all_layers_yield_pct`, computed the way `server.py` builds `savings_percent` / `active_savings_percent` — tool savings added to **both** sides, since deferred schemas were attempted work that succeeded whole. - `headroom/telemetry/session.py`: extend the `demo()` self-test to assert both new rates against the existing tool-heavy fixture. - `deploy/beacon/query.sh`: add an `all_layers_pct` column to the fleet summary, so the reader stops showing the understated number too. **Kept alongside `saved_pct` rather than folded into it.** Every row already in the corpus means context-only under that name; redefining it would make old and new rows non-comparable with no field to tell them apart. **`SCHEMA_VERSION` deliberately stays at 1.** The change is purely additive, nothing reads the field, and the query uses `union_by_name = true`, so old and new rows mix cleanly. Happy to bump it if maintainers want the marker. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text $ python -m headroom.telemetry.session ok $ python -m pytest tests/test_savings_tool_search_aggregation.py tests/test_outcome_dual_ruler_funnel.py -q FAILED tests/test_outcome_dual_ruler_funnel.py::test_ledger_delta_stays_on_the_local_ruler FAILED tests/test_outcome_dual_ruler_funnel.py::test_ledger_falls_back_to_billed_when_local_omitted 2 failed, 5 passed, 3 warnings in 4.42s $ ruff check headroom/ All checks passed! $ mypy --python-version 3.12 headroom/telemetry/session.py Success: no issues found in 1 source file ``` The two `test_outcome_dual_ruler_funnel.py` failures are **pre-existing on `main`, not caused by this PR** — verified by `git stash`-ing the change and re-running: ```text $ git stash && python -m pytest tests/test_outcome_dual_ruler_funnel.py -q FAILED tests/test_outcome_dual_ruler_funnel.py::test_ledger_delta_stays_on_the_local_ruler FAILED tests/test_outcome_dual_ruler_funnel.py::test_ledger_falls_back_to_billed_when_local_omitted 2 failed, 3 passed, 3 warnings in 3.71s ``` ## Real Behavior Proof - **Environment:** macOS 25.4.0 arm64, Python 3.12.6, repo `.venv`, branch rebased on `upstream/main` @ `d0a86d40`. - **Exact command / steps:** drive a real `SessionAggregator` with a tool-heavy outcome (`original=1000`, `attempted=400`, `tokens_saved=300`, plus `tool_search_deferred_tokens=800` and `turn_hook_tools_saved_tokens=200`) and print the emitted payload's `rates` block: ```text tokens: {"original": 1000, "attempted": 400, "saved": 300, "tool_saved": 1000} rates: { "saved_pct": 30.0, "eligible_pct": 40.0, "yield_pct": 75.0, "all_layers_saved_pct": 65.0, "all_layers_yield_pct": 92.86, "cache_read_pct": 50.0, "overhead_pct": 5.0 } ``` - **Observed result:** the pre-existing rates are byte-identical (30.0 / 40.0 / 75.0 / 50.0 / 5.0 — no regression), and the two new fields report the all-layers view: 1300 saved of 2000 sent = **65.0%**, 1300 of 1400 attempted = **92.86%**. Both denominators grow with the numerator, matching `server.py`. Cross-checked against the live corpus with DuckDB over the R2 bucket — restating all 516 sessions both ways reproduces the gap this PR closes: ```text ┌───────────┬────────────┬──────────────────┬──────────────────┬─────────────────────┐ │ ctx_saved │ tool_saved │ all_layers_saved │ beacon_saved_pct │ dashboard_saved_pct │ ├───────────┼────────────┼──────────────────┼──────────────────┼─────────────────────┤ │ 157561051 │ 646293457 │ 803854508 │ 2.8 │ 12.82 │ └───────────┴────────────┴──────────────────┴──────────────────┴─────────────────────┘ ``` - **Not tested:** no live proxy run was made against a real provider — the payload above comes from `SessionAggregator` driven directly, which is the same code path the proxy feeds. The 516 sessions already in R2 are **not backfilled**: they carry `tool_saved`, so the corrected rate is computable from them today, but their own `rates` block stays context-only. Only sessions emitted from the next release carry the new fields. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I did **not** edit `CHANGELOG.md` ## Additional Notes - **Documentation:** N/A — no doc references the beacon `rates` field names (`grep -rn "saved_pct\|yield_pct" docs/` returns nothing). The field semantics are documented inline in `session.py`, which this PR extends. - **Worth a maintainer opinion:** the dashboard formula adds `tool_saved` to the *denominator* as well. That is defensible — deferred schemas were attempted work that succeeded 100% — but it does mean a tool-heavy session's ratio is partly measuring a layer that is near-always at full yield. This PR matches the dashboard rather than inventing a third convention; if the convention should change, it should change in both places at once. - **Follow-up:** nothing here changes what the fleet actually saved, only what the beacon admits to. The 4.6x gap was reporting, not performance.
This commit is contained in:
@@ -74,6 +74,15 @@ case "${1:-summary}" in
|
||||
/ nullif(sum(tokens.attempted), 0), 2) AS yield_pct,
|
||||
round(sum(tokens.saved) * 100.0
|
||||
/ nullif(sum(tokens.original), 0), 2) AS saved_pct,
|
||||
-- saved_pct/yield_pct above are context-compression only, because
|
||||
-- tool_saved never lands in original/attempted. This is the
|
||||
-- dashboard headline (server.py `savings_percent`): tool-schema
|
||||
-- savings on BOTH sides, since deferred schemas were attempted work
|
||||
-- that succeeded whole. On a tool-heavy fleet the two differ several-
|
||||
-- fold, so say which one you are quoting.
|
||||
round(sum(tokens.saved + tokens.tool_saved) * 100.0
|
||||
/ nullif(sum(tokens.original + tokens.tool_saved), 0), 2)
|
||||
AS all_layers_pct,
|
||||
sum(failures) AS failures
|
||||
FROM sessions;"
|
||||
;;
|
||||
|
||||
@@ -325,6 +325,24 @@ class _Session:
|
||||
# Of what we touched, how much did we remove? THIS is the
|
||||
# compressor quality number, and the one Kompress moves.
|
||||
"yield_pct": _pct(self.tokens_saved, self.attempted_tokens),
|
||||
# The two above are context-compression only, because
|
||||
# `tool_saved` never lands in original/attempted. On a
|
||||
# tool-heavy fleet that understates the product several-fold —
|
||||
# observed 2.80% vs 12.82% across the first 516 sessions.
|
||||
# These two are what the dashboard headline shows
|
||||
# (`tokens.savings_percent` / `active_savings_percent` in
|
||||
# server.py): tool-schema savings added to BOTH sides, since
|
||||
# deferred schemas were attempted work that succeeded whole.
|
||||
# Kept alongside rather than folded into `saved_pct`, which
|
||||
# already means context-only in every row of the corpus.
|
||||
"all_layers_saved_pct": _pct(
|
||||
self.tokens_saved + self.tool_saved_tokens,
|
||||
self.original_tokens + self.tool_saved_tokens,
|
||||
),
|
||||
"all_layers_yield_pct": _pct(
|
||||
self.tokens_saved + self.tool_saved_tokens,
|
||||
self.attempted_tokens + self.tool_saved_tokens,
|
||||
),
|
||||
# Provider prompt cache participation. Headroom freezes prefixes
|
||||
# to protect this, so it is the other side of eligible_pct.
|
||||
"cache_read_pct": _pct(self.cache_read_tokens, self.original_tokens),
|
||||
@@ -811,6 +829,10 @@ def demo() -> None:
|
||||
assert r["rates"]["overhead_pct"] == 5.0, r["rates"]
|
||||
# Tool savings are invisible in `saved` by design; they must not be lost.
|
||||
assert r["tokens"]["tool_saved"] == 1000, r["tokens"]
|
||||
# ...and the all-layers rates are the ones that do count them: 1300 saved
|
||||
# of 2000 sent, 1300 of 1400 attempted. Both denominators grow too.
|
||||
assert r["rates"]["all_layers_saved_pct"] == 65.0, r["rates"]
|
||||
assert r["rates"]["all_layers_yield_pct"] == 92.86, r["rates"]
|
||||
assert r["compression"]["response_cache_hits"] == 1
|
||||
assert r["tokens"]["cache_write"] == 100 and r["tokens"]["uncached"] == 400
|
||||
|
||||
|
||||
Reference in New Issue
Block a user