test(agent-eval): report residual direction by sign, not by hope (CG-13)

The occupancy summary hardcoded "% lower with codegraph". pct(w, wo) is the
reduction going with->without, so a negative value means the with-arm's
residual is LARGER -- and the line printed "-82% lower with codegraph" for the
case where codegraph in fact occupies 82% MORE. A double negative that reads as
a win and inverts the headline of the whole metric.

Direction now follows the sign in words, and the negative case says what the
shape actually is: codegraph front-loads one large verbatim payload that stays
resident, where Read/Grep churn many small results that evict. Fewer total
tokens processed and a larger persistent footprint are both true at once --
that pair is the axis issue #1500 reported.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
Colby McHenry
2026-08-05 13:03:08 -05:00
parent 382791f11e
commit 520ed9d933
+19 -1
View File
@@ -146,7 +146,25 @@ for (const { repo, W, WO } of rows) {
`${wSw.toFixed(1)}%→${woSw.toFixed(1)}%`
);
}
console.log(`\nAVERAGE: retrieval residual ${avg(occ.resid)}% lower with codegraph · share-of-context ${avg(occ.shareCtx)}% lower`);
// Direction must follow the SIGN, not the hope. `pct(w, wo)` is the reduction
// going with→without, so a NEGATIVE value means the with-arm's residual is
// LARGER. Hardcoding "lower" printed "-82% lower with codegraph" for the case
// where codegraph in fact occupies 82% MORE — a double negative that reads as a
// win and inverts the headline. Say which way it went, in words.
const dir = (v) => (v < 0 ? 'HIGHER' : 'lower');
const magn = (v) => Math.abs(v);
console.log(
`\nAVERAGE: retrieval residual ${magn(avg(occ.resid))}% ${dir(avg(occ.resid))} with codegraph` +
` · share-of-context ${magn(avg(occ.shareCtx))}% ${dir(avg(occ.shareCtx))}`
);
if (avg(occ.resid) < 0) {
console.log(
` ^ codegraph front-loads one large verbatim payload that STAYS resident, where Read/Grep\n` +
` churn many small results that evict. Read alongside the cost/token table above: fewer\n` +
` total tokens processed can coexist with a larger persistent footprint. This is the axis\n` +
` issue #1500 reported.`
);
}
console.log(
`FIXED overhead: codegraph's tool schema + MCP instructions cost ${avg(occ.fixed) >= 0 ? '+' : ''}${avg(occ.fixed)} tok\n` +