diff --git a/.claude/skills/agent-eval/SKILL.md b/.claude/skills/agent-eval/SKILL.md index 2e894a7..8d06ac7 100644 --- a/.claude/skills/agent-eval/SKILL.md +++ b/.claude/skills/agent-eval/SKILL.md @@ -58,6 +58,12 @@ scripts/agent-eval/audit.sh "" codegraph-tool calls, duration, **total cost**. - Interactive (`parse-session.mjs`): the `VERDICT: codegraph_explore used Nx | Read N | Grep/Bash N` and `TOKENS:` lines. +- Both paths also print the three feedback metrics — residual context occupancy, + explore sufficiency, allocation efficiency — and a headless A/B ends with a + side-by-side `ARM COMPARISON` table. Report that table, and check its + contamination row first: `CLI calls that RETURNED output` > 0 means the arm + reached codegraph through Bash and its numbers are void. How to read the rest: + `docs/benchmarks/agent-eval-feedback-metrics.md`. Lead with cost + tool/Read counts — they are the reliable signals; raw token in/out are confounded by subagent delegation and prompt caching. State whether diff --git a/CLAUDE.md b/CLAUDE.md index a063956..1ae8ae9 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -138,6 +138,8 @@ For each **language × framework**, validate on **small, medium, and large** rea 1. **Pick the canonical flow** for the framework ("how does X reach Y": state→render, request→handler→view, query→SQL, action→reducer→store…). 2. **Deterministic probes** (`scripts/agent-eval/probe-{node,explore}.mjs` against the built `dist/`): `codegraph_explore` with the flow's symbol names connects from→to end-to-end with no break (its Flow section shows the path); **no node explosion** (`select count(*) from nodes` stable before/after re-index); synthesized-edge **precision** spot-check (`select … where provenance='heuristic'`). 3. **Agent A/B** (`scripts/agent-eval/run-all.sh ""`): with vs without codegraph, **≥2 runs/arm** (run-to-run variance is large — never conclude from n=1). Record **duration, total tool calls, Read, Grep**. Optional forced-Read-0 sufficiency proof via the block-read hook (`scripts/agent-eval/hook-settings.json`). + - **Every run also reports three feedback metrics** — residual context occupancy, explore sufficiency (what the agent did NEXT after each explore), and allocation efficiency (share of returned bytes the answer cited) — under each run, plus a side-by-side arm table (`compare-arms.mjs`). Entry point: `docs/benchmarks/agent-eval-feedback-metrics.md`. Reading them: `Read a file we returned` is an allocation miss, `Read a file we did NOT return`/`Grep` is recall; allocation efficiency is **relative** (attribution is by citation) so it is only valid between builds on the same question; occupancy *shares* are Claude Code / 200k and don't transfer to another host — the arm ratio does. + - **The `codegraph` CLI is blocked in every arm** (`no-cli-shim.sh`: sanitized PATH + a PreToolUse hook, shared by both harnesses). Without it 14 of 15 without-arm runs in one 7-repo pass reached codegraph through Bash. Check the contamination row before believing any number: `CLI calls that RETURNED output` > 0 invalidates the run (in a new-vs-baseline A/B it silently drops calls from all three metrics, since a CLI explore is not a tool call). - **Model policy — every A/B arm runs Claude with `--model sonnet --effort high`. Always. Never Opus/Fable.** All `scripts/agent-eval/*.sh` default to this (`MODEL`/`EFFORT` env override exists — don't raise it without an explicit reason from the maintainer). Two reasons, and the second matters more than cost: (a) Sonnet doesn't burn tokens; (b) **Sonnet is the deliberate floor model** — codegraph's real users attach it to whatever agent they already run (Cursor Composer, Gemini, etc.), so we validate on a "dumber" model on purpose: a stronger model's tool-use covers up the salience/sufficiency problems a weaker one exposes. An affordance that lands on Sonnet generalizes up to every host; one that only works on Opus/Fable doesn't generalize down to the agents most users actually have. Both arms always use the same model. - **MCP attach is a startup-latency issue, not a hard block.** On a multi-step task the agent dives into Read/grep before codegraph finishes its ~2-3s startup (worse when the eval is itself run nested inside a Claude session, under CPU contention), so it runs with no codegraph. Fix: **pre-warm a persistent daemon** for the target (`CODEGRAPH_DAEMON_IDLE_TIMEOUT_MS` high; spawn `serve --mcp --path "" [baseline-ref]` (it bakes in the pre-warm). 4. **Pass bar:** a normal flow question reaches **~0 Read/Grep within the repo's explore-call budget**, runs **faster** than without-codegraph, and shows **no regression on a control repo**. Record the numbers in `docs/design/dynamic-dispatch-coverage-playbook.md` (the coverage matrix). diff --git a/docs/benchmarks/agent-eval-feedback-metrics.md b/docs/benchmarks/agent-eval-feedback-metrics.md new file mode 100644 index 0000000..797804d --- /dev/null +++ b/docs/benchmarks/agent-eval-feedback-metrics.md @@ -0,0 +1,218 @@ +# The three explore feedback metrics — start here + +The agent-eval harness reports three metrics on every run. They are not three +views of one number; each answers a different question, and a retrieval change +can move one without moving the others. This page says which is which, which +harness to run, and how to read the output. The per-metric docs carry the +derivations and the caveats — read the one that matters once a number moves. + +| Metric | The question it answers | Doc | +|---|---|---| +| **Residual context occupancy** (CG-7) | How much of the window does this arm's retrieval still hold when the run ends — i.e. what does every following turn have to work in? | [`residual-context-occupancy.md`](residual-context-occupancy.md) | +| **Explore sufficiency** (CG-8) | Was a response *enough*? Read off what the agent did next: explored again, read a file, or answered. | [`explore-sufficiency.md`](explore-sufficiency.md) | +| **Allocation efficiency** (CG-9) | Of the bytes a response spent, what share went to files the answer actually drew on? | [`explore-allocation-efficiency.md`](explore-allocation-efficiency.md) | + +All three are **harness-only**: parsed out of transcripts we already write. +Nothing is emitted from the product and nothing leaves the machine. + +--- + +## Which harness + +Pick by the question you are actually asking. All three metrics print in both. + +**Isolating a retrieval change — `ab-new-vs-baseline.sh`.** New build (HEAD) vs +a baseline build (a git ref), **both arms codegraph-on**, same task. This is +the harness the three metrics were built for: with codegraph on in both arms, +every number is measuring the change rather than adoption. + +```bash +RUNS=3 scripts/agent-eval/ab-new-vs-baseline.sh /tmp/codegraph-corpus/express \ + "Add a charset option to res.send and wire it through" main +``` + +It builds each arm, indexes a throwaway copy of the target, **pre-warms a +codegraph daemon per run**, runs the task `RUNS` times per arm, prints the three +metric blocks under each run, and ends with the side-by-side table below. The +pre-warm is load-bearing and must not be removed: without it the agent dives +into Read/grep before codegraph finishes its ~2–3s startup, and the run measures +attach latency instead of retrieval. + +**With vs without codegraph — `run-all.sh`.** Codegraph-on against an empty MCP +config. A different question: displacement and adoption, not the effect of a +change. Multi-turn is where occupancy is actually charged, so separate turns +with `||`. + +```bash +scripts/agent-eval/run-all.sh /tmp/codegraph-corpus/gin \ + "How does gin route requests through its middleware chain?||\ +Where is the 404 / no-route case handled in that same chain?" +``` + +`CG_ARMS=with|without` re-runs one arm without redoing the other; the comparison +table still renders against whichever arm's logs are already in `$AGENT_EVAL_OUT`. + +**A campaign — `bench-readme.sh`.** The 7 README repos, three turns each, +`RUNS` per arm, through `run-all.sh` — so every run in a campaign carries all +three metrics. Aggregate with `parse-bench-readme.mjs`. + +**A log you already have.** `parse-run.mjs [run.tN.jsonl …]` prints +the three blocks for any stream-json log; `--brief` drops the numbered call +transcript. `parse-session.mjs ` does sufficiency and allocation +for an *interactive* session. `compare-arms.mjs