github-featured/chopratejas--headroom
2
0
Fork 0
Code Issues Pull Requests Actions 3 Deployments Models Agent Notes Packages Projects Releases Wiki Activity

fix(proxy/metrics): escape label values in the Prometheus export (#2463)

Browse Source 
## Description

`PrometheusMetrics.export()` writes the exposition text by hand and
drops `model` and `provider` into label lines without escaping them. The
other fourteen label emissions in that same function already call
`_escape_label_value()`.

`model` arrives raw from the client request body.
`handlers/openai.py:2601` and `:4287` both read `body.get("model",
"unknown")` with no validation, and `gemini.py:833` does the same. The
Anthropic path is the only one that sanitizes anything, and
`sanitize_anthropic_model_id` strips ANSI sequences and surrounding
whitespace, so a double quote goes straight through. There is no model
allowlist anywhere in the repo.

A standard parser aborts on the malformed line and drops every family
emitted at or after it, so one bad label costs the rest of the scrape.
These dicts have no TTL either, since `reset_runtime()` is only
reachable from the loopback-only `POST /stats/reset`, so a single
malformed request degrades `/metrics` until the process restarts.

No filed issue, this came out of a metrics-path audit.

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- Route all 15 label-value interpolations in `export()` through
`_escape_label_value()`. That is 13 `provider` sites, 1 `model` site,
and 1 `reason` site.
- The nine `cache_by_provider` blocks re-walk one dict, once per metric
family, because the exposition format wants each family's samples
grouped. The provider keys get escaped once above that block rather than
at each of the eleven emission sites, so those f-strings stay untouched.
- Coerce with `str()` at each escape call. `_escape_label_value` runs
`.replace()`, so a non-str value raises where the old hand-rolled
f-string called `str()` implicitly. A JSON body can carry `"model": 123`
and `handlers/openai.py:2601` passes the decoded value through
untouched, so an int reaches the dict. This matches the two call sites
that already coerce, `_format_labels` at `:39` and the
`wrap_rtk_invocations_total` tool label.
- Normalize un-encodable code points in `_escape_label_value` before
escaping. A lone surrogate decoded from a client model id (`{"model":
"x-\ud83d-y"}`, all-ASCII on the wire) is a valid str but not
UTF-8-encodable. It passed the escape untouched and raised in the
`/metrics` response encoder, taking down every scrape until restart
since the key persists. This one is pre-existing, base emits the same
raw surrogate and crashes the same way. The escaping work surfaced it,
and this helper is the single chokepoint every label value already
passes through.
- Add `tests/test_prometheus_label_escaping.py`, nine scenarios. Six
fail against `main`, the coercion one fails against this branch's own
first commit, and the surrogate one fails against the escape without the
scrub.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

```text
$ .venv/bin/python -m pytest tests/test_prometheus_label_escaping.py -q
collected 8 items
tests/test_prometheus_label_escaping.py ........                         [100%]
============================== 8 passed in 27.29s ==============================

$ # the escaping scenarios against main's export()
FAILED tests/test_prometheus_label_escaping.py::test_quote_in_model_is_escaped
FAILED tests/test_prometheus_label_escaping.py::test_quote_in_provider_is_escaped
FAILED tests/test_prometheus_label_escaping.py::test_backslash_and_newline_in_model_are_escaped
FAILED tests/test_prometheus_label_escaping.py::test_provider_cache_families_escape_provider
FAILED tests/test_prometheus_label_escaping.py::test_cache_miss_attribution_escapes_both_labels
FAILED tests/test_prometheus_label_escaping.py::test_no_emitted_label_value_is_malformed
========================= 6 failed, 1 passed in 1.12s ==========================

$ # the coercion scenario against this branch's first commit, before the str() wrap
prometheus_metrics.py:31: AttributeError: 'int' object has no attribute 'replace'
FAILED tests/test_prometheus_label_escaping.py::test_non_string_label_values_are_coerced
============================== 1 failed in 19.33s ==============================

$ .venv/bin/ruff check .
All checks passed!

$ .venv/bin/ruff format --check .
1332 files already formatted

$ .venv/bin/mypy headroom
Success: no issues found in 506 source files

$ per-file sweep over the blast radius (prometheus|metric|savings|stats|cache|proxy|export|outcome|observ|telemetry)
total=127 green=123 non_green=4
  FAIL(5)   tests/test_dashboard_cache_lifetime_playwright.py
  FAIL(5)   tests/test_dashboard_cache_net_playwright.py
  FAIL(5)   tests/test_dashboard_cache_ttl_playwright.py
  FAIL(1)   tests/test_proxy_savings_history.py

$ the same four files with prometheus_metrics.py reverted to 8c8fae0d
baseline exit=5 tests/test_dashboard_cache_lifetime_playwright.py
baseline exit=5 tests/test_dashboard_cache_net_playwright.py
baseline exit=5 tests/test_dashboard_cache_ttl_playwright.py
baseline exit=1 tests/test_proxy_savings_history.py
```

## Real Behavior Proof

- Environment: macOS 26.4.1 arm64, Python 3.13.13, branch
`fix/prometheus-label-escaping` off `upstream/main@8c8fae0d`, worktree
venv from `uv sync --extra dev`. A real proxy process (`uvicorn
headroom.proxy.server:create_app_from_env --factory --host 127.0.0.1
--port 9910`) pointed at a local stub Anthropic upstream on port 9911
through `ANTHROPIC_TARGET_API_URL`, so the request completes and records
metrics without touching a live provider.
- Exact command / steps: POST one request through the running proxy
carrying a double quote in the model field, `curl -s -X POST
http://127.0.0.1:9910/v1/messages -H 'content-type: application/json' -H
'x-api-key: rbp-harness' -H 'anthropic-version: 2023-06-01' -d
'{"model": "claude-sonnet-4-5\"evil", "max_tokens": 64, "messages":
[{"role":"user","content":"hello from the RBP harness"}]}'`, then scrape
it with `curl -s http://127.0.0.1:9910/metrics > scrape.txt` and parse
that file with the reference parser,
`prometheus_client.parser.text_string_to_metric_families`. Same harness
run twice, once against the reverted file and once against the fixed
file.
- Observed result: before the fix the proxy emits
`headroom_requests_by_model{model="claude-sonnet-4-5"evil"} 1` and the
parser aborts with `ValueError: could not convert string to float:
'{model="claude-sonnet-4-5"evil"}'` after recovering only 30 of 47
families, so 17 families are lost out of a 204-line scrape. After the
fix the same request emits
`headroom_requests_by_model{model="claude-sonnet-4-5\"evil"} 1`, all 47
families parse, and the label round-trips as the raw
`claude-sonnet-4-5"evil` the client sent. The eleven `cache_by_provider`
families parse in the after run too. Both scrapes were 204 lines, so the
difference is what a scraper can read. A second run covers the surrogate
crash: a request with `{"model": "x-\ud83d-y"}` rendered through the
real `PlainTextResponse` that `server.py` builds for `/metrics` returns
HTTP 500 (`UnicodeEncodeError`) before the scrub and HTTP 200 after,
with the healthy `gpt-4o` series still readable and the poison
neutralized to `model="x-?-y"`.
- Not tested: the URL-path model vectors (`gemini.py:229`, the vertex
`{model}` route param, and bedrock `{model_id:path}`) run through the
same escape, but the harness only exercised the Anthropic request-body
path and I did not check which characters ASGI path decoding lets
through. Nothing hit a live provider, the upstream was a local stub on
loopback. I did not run this on Linux or Windows, CI covers those.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)

## Screenshots (if applicable)

N/A, no user-visible surface.

## Additional Notes

**Scope.** `model` is the client-reachable value, so `:1284` is the live
bug. `provider` resolves to handler literals, module constants like
`vertex:anthropic`, an operator-config backend name (`litellm-*` /
`anyllm-*` off `config.anyllm_provider`), or the two-value `""` /
`"zen"` vocabulary in `passthrough.py:10-26`, so those thirteen sites
are hardening rather than a live fix. I escaped them anyway. The
regression guard only holds if the invariant is total. Eleven of the
fifteen are the `cache_by_provider` block, unescaped since it was first
written in `be6aa141`, and nothing ever failed on them. Every other
label in the file (`transform`, `stage`, `path`, `cause`, `signal`,
`event`, `outcome`, `tool`) already gets escaped despite being just as
internal, so these fifteen were the deviation. Happy to narrow it to
`model` alone if you would rather keep the diff tight.

**One correction to an earlier commit message.** `bd2b3d8d` credits
`#2450` for adding the eleven `cache_by_provider` sites. That is wrong,
they trace to `be6aa141` and have been unescaped since April, `#2450`
added cache-token recording but no emission lines. The commit message is
immutable without a force-push, so I am noting the correction here
rather than rewriting history.

**A related gap this does not close.** The same client-controlled
`model` also feeds `requests_by_model` at `:704` with no cardinality cap
and no TTL, which contradicts the invariant in
`docs/observability.md:244`. The sibling `stack` label is already capped
via `MAX_DISTINCT_STACKS`. Worth noting because the escaping fix changes
the failure mode, before it a poison model aborted the scrape loudly,
now it is accepted as an unbounded series. Separate fix, tracking it on
my side, not folding it in here.

**Commit shape.** Four commits rather than two. Self-review caught that
the first commit narrowed what `export()` accepts, since
`_escape_label_value` raises on a non-str where the f-string it replaced
coerced silently. That is its own follow-up fix and its own test rather
than a rewrite of the first pair, so the history shows the catch.

**Local test state.** A full `pytest tests/` does not complete on this
macOS box, so the sweep above ran 127 blast-radius files individually
under a wall-clock watchdog. Four came back non-green, and all four
reproduce with `prometheus_metrics.py` reverted to `8c8fae0d`. Three are
Playwright dashboard files that skip at module level with no browser
installed, which pytest codes as exit 5. The fourth,
`test_proxy_savings_history.py::test_cache_only_request_still_appends_a_history_point`,
fails on unmodified main too.

**Follow-up, not folded in.** `requests_by_model` and
`cache_by_provider` also have no cardinality cap. That is a real gap on
the same dicts and a natural follow-up to #618, so I left it alone here.

**N/A checklist item.** Documentation is unticked because there is no
doc surface for this. The exposition output stays byte-identical for
well-formed values.

**Pushed with `--no-verify`.** The `make ci-precheck` pre-push hook
clears its Rust half (`cargo fmt`, `cargo clippy --workspace -- -D
warnings`, `cargo test --workspace`) and then dies in
`ci-precheck-python`, which shells out to
`scripts/build_rust_extension.sh`. That script runs with `VIRTUAL_ENV`
unset and invokes a bare `python`, which this machine does not have on
`PATH` (only `python3`), so it fails at
`scripts/build_rust_extension.sh: line 44: python: command not found`
before it ever reaches `pip install -e .`. The uv-managed venv has no
`pip` module either. Neither condition is reachable from a change that
touches two `.py` files, so I bypassed the hook and ran ruff, ruff
format, and mypy by hand instead. Their output is above.
This commit is contained in:
inix
2026-08-12 13:11:06 +08:00
committed by GitHub
parent 1edaeb8b76
commit 6a53861063
2 changed files with 305 additions and 13 deletions
Download Patch File Download Diff File Expand all files Collapse all files
headroom/proxy/prometheus_metrics.py
+32 -13
View File
@@ -28,6 +28,13 @@ logger = logging.getLogger("headroom.proxy")
def _escape_label_value(value: str) -> str:
# The /metrics body is emitted whole with .encode("utf-8") (server.py). A
# client-supplied value can be a valid str that is not UTF-8-encodable — a
# lone surrogate decoded from a JSON model id — which raises in the response
# encoder and 500s every scrape, not just its own line. Drop un-encodable
# code points before escaping so one malformed request can't down the
# endpoint. Byte-identical for encodable values, including non-ASCII.
value = value.encode("utf-8", "replace").decode("utf-8")
return value.replace("\\", "\\\\").replace("\n", "\\n").replace('"', '\\"')
@@ -1267,10 +1274,11 @@ class PrometheusMetrics:
]
)
for _provider, _reasons in self.cache_miss_attribution_by_provider.items():
_safe_provider = _escape_label_value(str(_provider))
for _reason, _count in _reasons.items():
lines.append(
f'headroom_cache_miss_attribution_total{{provider="{_provider}",'
f'reason="{_reason}"}} {_count}'
f'headroom_cache_miss_attribution_total{{provider="{_safe_provider}",'
f'reason="{_escape_label_value(str(_reason))}"}} {_count}'
)
lines.append("")
@@ -1327,7 +1335,9 @@ class PrometheusMetrics:
]
)
for provider, count in self.requests_by_provider.items():
lines.append(f'headroom_requests_by_provider{{provider="{provider}"}} {count}')
lines.append(
f'headroom_requests_by_provider{{provider="{_escape_label_value(str(provider))}"}} {count}'
)
lines.append("")
lines.extend(
@@ -1337,7 +1347,9 @@ class PrometheusMetrics:
]
)
for model, count in self.requests_by_model.items():
lines.append(f'headroom_requests_by_model{{model="{model}"}} {count}')
lines.append(
f'headroom_requests_by_model{{model="{_escape_label_value(str(model))}"}} {count}'
)
lines.append("")
if self.transform_timing_sum:
@@ -1472,13 +1484,20 @@ class PrometheusMetrics:
lines.append("")
if self.cache_by_provider:
# The exposition format wants each family's samples grouped, so the
# blocks below re-walk this dict once per family. Escape the provider
# keys once here instead of at all eleven emission sites.
cache_by_provider = {
_escape_label_value(str(name)): stats
for name, stats in self.cache_by_provider.items()
}
lines.extend(
[
"# HELP headroom_cache_read_tokens_total Provider cache read tokens",
"# TYPE headroom_cache_read_tokens_total counter",
]
)
for provider, stats in self.cache_by_provider.items():
for provider, stats in cache_by_provider.items():
lines.append(
f'headroom_cache_read_tokens_total{{provider="{provider}"}} {stats["cache_read_tokens"]}'
)
@@ -1489,7 +1508,7 @@ class PrometheusMetrics:
"# TYPE headroom_cache_write_tokens_total counter",
]
)
for provider, stats in self.cache_by_provider.items():
for provider, stats in cache_by_provider.items():
lines.append(
f'headroom_cache_write_tokens_total{{provider="{provider}"}} {stats["cache_write_tokens"]}'
)
@@ -1500,7 +1519,7 @@ class PrometheusMetrics:
"# TYPE headroom_cache_write_ttl_tokens_total counter",
]
)
for provider, stats in self.cache_by_provider.items():
for provider, stats in cache_by_provider.items():
lines.append(
f'headroom_cache_write_ttl_tokens_total{{provider="{provider}",ttl="5m"}} {stats["cache_write_5m_tokens"]}'
)
@@ -1514,7 +1533,7 @@ class PrometheusMetrics:
"# TYPE headroom_cache_write_ttl_requests_total counter",
]
)
for provider, stats in self.cache_by_provider.items():
for provider, stats in cache_by_provider.items():
lines.append(
f'headroom_cache_write_ttl_requests_total{{provider="{provider}",ttl="5m"}} {stats["cache_write_5m_requests"]}'
)
@@ -1528,7 +1547,7 @@ class PrometheusMetrics:
"# TYPE headroom_uncached_input_tokens_total counter",
]
)
for provider, stats in self.cache_by_provider.items():
for provider, stats in cache_by_provider.items():
lines.append(
f'headroom_uncached_input_tokens_total{{provider="{provider}"}} {stats["uncached_input_tokens"]}'
)
@@ -1539,7 +1558,7 @@ class PrometheusMetrics:
"# TYPE headroom_provider_cache_requests_total counter",
]
)
for provider, stats in self.cache_by_provider.items():
for provider, stats in cache_by_provider.items():
lines.append(
f'headroom_provider_cache_requests_total{{provider="{provider}"}} {stats["requests"]}'
)
@@ -1550,7 +1569,7 @@ class PrometheusMetrics:
"# TYPE headroom_provider_cache_hit_requests_total counter",
]
)
for provider, stats in self.cache_by_provider.items():
for provider, stats in cache_by_provider.items():
lines.append(
f'headroom_provider_cache_hit_requests_total{{provider="{provider}"}} {stats["hit_requests"]}'
)
@@ -1561,7 +1580,7 @@ class PrometheusMetrics:
"# TYPE headroom_provider_cache_bust_total counter",
]
)
for provider, stats in self.cache_by_provider.items():
for provider, stats in cache_by_provider.items():
lines.append(
f'headroom_provider_cache_bust_total{{provider="{provider}"}} {stats["bust_count"]}'
)
@@ -1572,7 +1591,7 @@ class PrometheusMetrics:
"# TYPE headroom_provider_cache_bust_write_tokens_total counter",
]
)
for provider, stats in self.cache_by_provider.items():
for provider, stats in cache_by_provider.items():
lines.append(
f'headroom_provider_cache_bust_write_tokens_total{{provider="{provider}"}} {stats["bust_write_tokens"]}'
)
tests/test_prometheus_label_escaping.py
+273
View File
@@ -0,0 +1,273 @@
"""Label-value escaping in the Prometheus text exposition output.
``PrometheusMetrics.export()`` builds the exposition text by hand, so every label
value has to pass through ``_escape_label_value`` before it is interpolated. The
format reserves ``"``, ``\\`` and the line feed, and a standard scraper does not
degrade gracefully on a malformed line — it aborts the parse, losing every
family emitted at or after the bad sample.
``model`` reaches ``requests_by_model`` straight from the parsed client request
body (``handlers/openai.py`` reads ``body.get("model", "unknown")`` with no
sanitisation, and the Anthropic path's ``sanitize_anthropic_model_id`` only
strips ANSI sequences and whitespace), so an unescaped value is remotely
reachable.
Imports only the metrics module so the test stays free of heavy ML deps.
"""
from __future__ import annotations
import re
import pytest
from headroom.proxy.prometheus_metrics import PrometheusMetrics
# A label whose value contains only unreserved characters or well-formed escape
# pairs. An unescaped quote inside a value stops this matching, which is exactly
# the failure a scraper hits.
_LABEL_RE = re.compile(r'([a-zA-Z_][a-zA-Z0-9_]*)="((?:[^"\\]|\\.)*)"')
_SAMPLE_RE = re.compile(r"^(?P<name>[a-zA-Z_:][a-zA-Z0-9_:]*)\{(?P<labels>.*)\} \S+$")
_ESCAPE_RE = re.compile(r"\\(.)")
_UNESCAPE = {"n": "\n", '"': '"', "\\": "\\"}
def _unescape(value: str) -> str:
def replace(match: re.Match[str]) -> str:
char = match.group(1)
if char not in _UNESCAPE:
raise ValueError(f"undefined escape sequence '\\{char}' in {value!r}")
return _UNESCAPE[char]
return _ESCAPE_RE.sub(replace, value)
def _parse_label_block(block: str) -> dict[str, str]:
"""Parse ``key="value",key="value"`` the way a scraper would.
Raises ``ValueError`` on anything the exposition grammar rejects, so a line
carrying an unescaped quote fails loudly instead of yielding a
plausible-looking dict.
"""
labels: dict[str, str] = {}
pos = 0
while pos < len(block):
match = _LABEL_RE.match(block, pos)
if match is None:
raise ValueError(f"malformed label block at offset {pos}: {block!r}")
labels[match.group(1)] = _unescape(match.group(2))
pos = match.end()
if pos < len(block):
if block[pos] != ",":
raise ValueError(f"expected ',' at offset {pos}: {block!r}")
pos += 1
return labels
def _labelled_samples(text: str) -> list[tuple[str, dict[str, str]]]:
"""Every labelled sample in a scrape, as (metric name, decoded labels).
Raises on any line a scraper would reject — including the fragments an
unescaped line feed splits a sample into.
"""
samples: list[tuple[str, dict[str, str]]] = []
for line in text.splitlines():
if not line or line.startswith("#") or "{" not in line:
continue
match = _SAMPLE_RE.match(line)
if match is None:
raise ValueError(f"malformed sample line: {line!r}")
samples.append((match.group("name"), _parse_label_block(match.group("labels"))))
return samples
async def _record(metrics: PrometheusMetrics, **overrides: object) -> None:
kwargs: dict[str, object] = {
"provider": "anthropic",
"model": "claude-sonnet-4-5",
"input_tokens": 100,
"output_tokens": 20,
# tokens_saved=0 keeps the durable savings-ledger write out of the test.
"tokens_saved": 0,
"latency_ms": 10.0,
}
kwargs.update(overrides)
await metrics.record_request(**kwargs) # type: ignore[arg-type]
@pytest.mark.asyncio
async def test_quote_in_model_is_escaped() -> None:
metrics = PrometheusMetrics()
await _record(metrics, model='claude-sonnet-4-5"evil')
text = await metrics.export()
assert 'headroom_requests_by_model{model="claude-sonnet-4-5\\"evil"} 1' in text
assert 'headroom_requests_by_model{model="claude-sonnet-4-5"evil"}' not in text
@pytest.mark.asyncio
async def test_quote_in_provider_is_escaped() -> None:
metrics = PrometheusMetrics()
await _record(metrics, provider='anth"ropic')
text = await metrics.export()
assert 'headroom_requests_by_provider{provider="anth\\"ropic"} 1' in text
assert 'headroom_requests_by_provider{provider="anth"ropic"}' not in text
@pytest.mark.asyncio
async def test_backslash_and_newline_in_model_are_escaped() -> None:
metrics = PrometheusMetrics()
await _record(metrics, model="back\\slash")
await _record(metrics, model="line\nfeed")
text = await metrics.export()
# Backslash first, so the escapes this inserts are not re-escaped.
assert 'headroom_requests_by_model{model="back\\\\slash"} 1' in text
assert 'headroom_requests_by_model{model="line\\nfeed"} 1' in text
# The line feed must not survive as a real newline splitting the sample.
assert "line\nfeed" not in text
@pytest.mark.asyncio
async def test_provider_cache_families_escape_provider() -> None:
# The families PR #2450 added inherit `provider` from the same parameter,
# so they need naming explicitly rather than assuming coverage.
metrics = PrometheusMetrics()
await _record(
metrics,
provider='anth"ropic',
cache_read_tokens=40,
cache_write_tokens=60,
cache_write_5m_tokens=10,
cache_write_1h_tokens=50,
uncached_input_tokens=20,
)
text = await metrics.export()
families = [
"headroom_cache_read_tokens_total",
"headroom_cache_write_tokens_total",
"headroom_cache_write_ttl_tokens_total",
"headroom_cache_write_ttl_requests_total",
"headroom_uncached_input_tokens_total",
"headroom_provider_cache_requests_total",
"headroom_provider_cache_hit_requests_total",
"headroom_provider_cache_bust_total",
"headroom_provider_cache_bust_write_tokens_total",
]
for family in families:
assert f'{family}{{provider="anth\\"ropic"' in text, f"{family} left provider raw"
@pytest.mark.asyncio
async def test_cache_miss_attribution_escapes_both_labels() -> None:
metrics = PrometheusMetrics()
await metrics.record_cache_miss_attribution('anth"ropic', 'ttl"expiry')
text = await metrics.export()
assert (
'headroom_cache_miss_attribution_total{provider="anth\\"ropic",reason="ttl\\"expiry"} 1'
in text
)
@pytest.mark.asyncio
async def test_no_emitted_label_value_is_malformed() -> None:
# The regression guard: poison every reachable label input, then read the
# whole scrape the way a scraper does. A future emission that forgets to
# escape fails here even when no assertion above names it.
metrics = PrometheusMetrics()
# The model poison carries a comma and an inner quote. The parse alone
# can't catch comma-injection (this value raises on the quote first), so the
# round-trip assertion below is the real guard: after escaping, the value
# must decode back to the exact raw string, comma and all, rather than
# splitting into extra labels.
await _record(
metrics,
provider='pro"vider\\one',
model='mo"del,evil="1',
cache_read_tokens=40,
cache_write_tokens=60,
cache_write_5m_tokens=10,
cache_write_1h_tokens=50,
uncached_input_tokens=20,
)
await metrics.record_cache_miss_attribution('pro"vider\\one', 'rea"son')
samples = _labelled_samples(await metrics.export())
values = {value for _, labels in samples for value in labels.values()}
assert 'pro"vider\\one' in values, "provider did not round-trip through the escape"
assert 'mo"del,evil="1' in values, "model did not round-trip through the escape"
@pytest.mark.asyncio
async def test_non_string_label_values_are_coerced() -> None:
# A JSON body can carry `"model": 123`, and the handlers pass the decoded
# value through untouched (handlers/openai.py reads body.get("model")). The
# hand-rolled f-strings used to call str() implicitly, so escaping has to
# keep tolerating a non-str. /metrics has no error handling around export(),
# and the key survives in the dict, so a raise here would take out every
# later scrape too.
metrics = PrometheusMetrics()
await _record(metrics, provider=456, model=123, cache_read_tokens=5, cache_write_tokens=5)
await metrics.record_cache_miss_attribution(456, 789)
text = await metrics.export()
assert 'headroom_requests_by_model{model="123"} 1' in text
assert 'headroom_requests_by_provider{provider="456"} 1' in text
assert 'headroom_cache_read_tokens_total{provider="456"}' in text
assert 'headroom_cache_miss_attribution_total{provider="456",reason="789"} 1' in text
@pytest.mark.asyncio
async def test_well_formed_values_are_emitted_unchanged() -> None:
metrics = PrometheusMetrics()
await _record(metrics)
text = await metrics.export()
assert 'headroom_requests_by_provider{provider="anthropic"} 1' in text
assert 'headroom_requests_by_model{model="claude-sonnet-4-5"} 1' in text
@pytest.mark.asyncio
async def test_export_is_utf8_encodable_with_surrogate_model() -> None:
# `/metrics` renders the whole body with `.encode("utf-8")` (server.py). A
# client can decode a lone surrogate from JSON (`{"model": "x-\ud83d-y"}`) —
# a valid str that is NOT UTF-8-encodable and passes escaping untouched. It
# would raise in the response encoder and, because the poisoned key persists
# in requests_by_model, 500 every later scrape until restart. Escaping must
# leave the whole export encodable.
metrics = PrometheusMetrics()
await _record(metrics, model="x-\ud83d-y")
await _record(metrics, model="clean-model") # a healthy series alongside
text = await metrics.export()
# The load-bearing assertion: the body a scraper receives must encode.
text.encode("utf-8")
# And the healthy series is still readable, i.e. the poison did not corrupt
# the surrounding output.
assert 'headroom_requests_by_model{model="clean-model"} 1' in text
# Legitimate astral characters (a real emoji is one code point, encodable)
# are preserved, not scrubbed — only un-encodable lone surrogates change.
metrics2 = PrometheusMetrics()
await _record(metrics2, model="gpt-\U0001f600")
assert 'headroom_requests_by_model{model="gpt-\U0001f600"} 1' in await metrics2.export()