Reproducible suite measuring the managed CodeGraph AI offload and the front-load
UserPromptSubmit hook (approach 1) vs raw codegraph and no-codegraph, across repo
sizes, on time / main-session tokens+cost / CodeGraph-AI tokens+cost / accuracy.
All agent arms run claude -p sonnet --effort high; eval-only, nothing shipped.
- offload-eval-setup.sh: clone + index 4 memory-probe-verified "not-trained-on" repos
(mtkruto/postybirb/shapeshift/trezor — small→large) so the no-codegraph baseline is honest.
- offload-eval-3arm.sh / -frontload.sh: one repo, the arms (offload/raw/nocg, frontload).
- offload-eval-matrix.sh / -frontload-matrix.sh: drive all 4 tiers.
- offload-eval-hook.mjs: the front-load hook (self-locates its engine; CG_FRONTLOAD_DEBUG to log).
- offload-eval-metrics.mjs / -judge.mjs (Sonnet) / -summarize.mjs: extract, score, aggregate.
- offload-eval-ground-truth.json: source-verified canonical flows (the judge's reference).
- offload-eval.md: usage + the 2026-06 findings (raw = the win; offload least-accurate;
front-load solves adoption but exposes explore's dynamic-dispatch gaps).
Scripts are path-portable (self-locating $HERE/$ENGINE; AGENT_EVAL_OUT scratch dir).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Colby McHenry