test(agent-eval): report residual direction by sign, not by hope (CG-13)
The occupancy summary hardcoded "% lower with codegraph". pct(w, wo) is the reduction going with->without, so a negative value means the with-arm's residual is LARGER -- and the line printed "-82% lower with codegraph" for the case where codegraph in fact occupies 82% MORE. A double negative that reads as a win and inverts the headline of the whole metric. Direction now follows the sign in words, and the negative case says what the shape actually is: codegraph front-loads one large verbatim payload that stays resident, where Read/Grep churn many small results that evict. Fewer total tokens processed and a larger persistent footprint are both true at once -- that pair is the axis issue #1500 reported. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
@@ -146,7 +146,25 @@ for (const { repo, W, WO } of rows) {
|
||||
`${wSw.toFixed(1)}%→${woSw.toFixed(1)}%`
|
||||
);
|
||||
}
|
||||
console.log(`\nAVERAGE: retrieval residual ${avg(occ.resid)}% lower with codegraph · share-of-context ${avg(occ.shareCtx)}% lower`);
|
||||
// Direction must follow the SIGN, not the hope. `pct(w, wo)` is the reduction
|
||||
// going with→without, so a NEGATIVE value means the with-arm's residual is
|
||||
// LARGER. Hardcoding "lower" printed "-82% lower with codegraph" for the case
|
||||
// where codegraph in fact occupies 82% MORE — a double negative that reads as a
|
||||
// win and inverts the headline. Say which way it went, in words.
|
||||
const dir = (v) => (v < 0 ? 'HIGHER' : 'lower');
|
||||
const magn = (v) => Math.abs(v);
|
||||
console.log(
|
||||
`\nAVERAGE: retrieval residual ${magn(avg(occ.resid))}% ${dir(avg(occ.resid))} with codegraph` +
|
||||
` · share-of-context ${magn(avg(occ.shareCtx))}% ${dir(avg(occ.shareCtx))}`
|
||||
);
|
||||
if (avg(occ.resid) < 0) {
|
||||
console.log(
|
||||
` ^ codegraph front-loads one large verbatim payload that STAYS resident, where Read/Grep\n` +
|
||||
` churn many small results that evict. Read alongside the cost/token table above: fewer\n` +
|
||||
` total tokens processed can coexist with a larger persistent footprint. This is the axis\n` +
|
||||
` issue #1500 reported.`
|
||||
);
|
||||
}
|
||||
|
||||
console.log(
|
||||
`FIXED overhead: codegraph's tool schema + MCP instructions cost ${avg(occ.fixed) >= 0 ? '+' : ''}${avg(occ.fixed)} tok\n` +
|
||||
|
||||
Reference in New Issue
Block a user