Files

T

Andrey Kumanyaev e1df92ae32 bench/wire-format: dual-tokenizer scorecard (cl100k + Claude Opus 4.7)

Adds a second token-cost column to the GCX1 scorecard scored against
the Claude Opus 4.7 tokenizer. The cl100k_base column is preserved
unchanged — Claude 3 / Opus 4 / Sonnet 4 / Haiku 4.5 / GPT-4o all
still use it as their budget reference.

Two data sources for the Opus 4.7 column, picked at flag time:

  - Scalar estimate (default, offline). cl100k_base count × empirical
    1.35 inflation factor. Per-fixture variance 28-42%; the median
    across the suite is honest. Labeled "estimated" in the scorecard.

  - Exact counts via Anthropic count_tokens (--use-api). Requires
    ANTHROPIC_API_KEY. Successful calls populate
    bench/wire-format/opus47-counts.json so subsequent runs are
    deterministic without re-hitting the API. Network failures
    degrade gracefully to the scalar with a single warning on
    stderr — harness must keep running when offline.

New flags:
  --tokenizer cl100k|opus47|both   (default both)
  --use-api                        (opt-in API-backed counting)
  --opus47-cache PATH              (exact-count sidecar)
  --opus47-model NAME              (model id for count_tokens)

Scorecard now renders one or two stacked markdown tables with
per-row "exact" vs "est." source markers and tokenizer-specific
summary lines. docs/wire-format.md gets a new "Dual tokenizer
scorecard" section. README headline updated: "−27.4% under cl100k
and −27.3% under Opus 4.7" — the wire format's advantage compounds
with the tokenizer change rather than being amplified by it.

13 opus47 tests cover the scalar arithmetic (round-half-to-even),
cache round-trip + partial-entry fallback, missing-file tolerance,
parseTokenizerMode aliases, API success + caches, API error → scalar
fallback path with httptest. Full repo: 4526 / 121 packages green
(+13 new). go vet + golangci-lint clean.

2026-05-18 21:42:09 +02:00

3.1 KiB

Raw Permalink Blame History

GCX1 wire-format benchmark

Reproducible benchmark comparing the GCX1 compact wire format against JSON (and, when JCODEMUNCH=/path/to/jcm is set, against jCodeMunch MUNCH) on representative MCP tool responses.

What it measures

For each fixture case the harness captures, for JSON and GCX:

Bytes — raw UTF-8 byte length.
tiktoken (cl100k_base) — LLM-relevant token count via github.com/pkoukk/tiktoken-go (same loader Gortex uses at runtime). Matches Claude 3 / Opus 4 / Sonnet 4 / Haiku 4.5 / GPT-4o budgets.
Claude Opus 4.7 input tokens — second column populated either by scaling cl100k_base (×1.35 empirical inflation factor, the default; labeled estimated) or by calling Anthropic's messages/count_tokens endpoint with --use-api (requires ANTHROPIC_API_KEY; results cached to opus47-counts.json for deterministic reruns; labeled exact).
gzip — gzip-compressed byte length, fair comparison when the transport would compress anyway.
Round-trip integrity — encode → decode → re-marshal, compare to the canonical JSON normalisation. Must be 100 % for the format to be considered lossless.

Results land in scorecard.md with per-case rows and a summary of medians / totals.

Running

go run ./bench/wire-format -cases ./bench/wire-format/cases -out ./bench/wire-format/scorecard.md

Flags:

-cases DIR — directory of fixture case files (default ./bench/wire-format/cases).
-out FILE — output scorecard markdown path (default stdout).
-json FILE — emit raw per-case metrics as JSON too.
-tokenizer cl100k|opus47|both — which token-cost column(s) to render (default both).
-use-api — call Anthropic count_tokens for exact Opus 4.7 numbers (requires ANTHROPIC_API_KEY; degrades to the scalar on network failure with a single warning).
-opus47-cache PATH — exact-count sidecar (default bench/wire-format/opus47-counts.json).
-opus47-model NAME — model id passed to the API (default claude-opus-4-20250514).

Fixture format

Each case is a Go struct literal decoded from a YAML file with two sections:

tool: search_symbols
description: 20 search hits on a medium-size repo
input: |
  [{...JSON rows as the tool would return...}]

The harness encodes input via the canonical Go encoder for the specified tool and scores the two outputs.

Target

GCX1 targets ≥20 % token savings vs JSON on the median case with 100 % round-trip integrity. Current baseline (20 cases):

Median token savings (cl100k_base): −27.4 %
Median token savings (Opus 4.7, scalar): −27.3 %
Median byte savings: −26.8 %
Round-trip integrity: 20/20

Tabular list payloads (search_symbols, analyze_hotspots, find_usages_large, smart_context) hit −30 to −38 %. Small scalar-heavy records (graph_stats, find_cycles) can flip positive — GCX1's header overhead exceeds the savings when there are fewer than ~5 rows and no repeated field names. Payloads with long inline bodies (get_symbol_source) are roughly neutral — the source body dominates and neither encoding compresses it.

3.1 KiB Raw Permalink Blame History Unescape Escape