Colby McHenry
7ddd3fa7eb
Reproducible suite measuring the managed CodeGraph AI offload and the front-load UserPromptSubmit hook (approach 1) vs raw codegraph and no-codegraph, across repo sizes, on time / main-session tokens+cost / CodeGraph-AI tokens+cost / accuracy. All agent arms run claude -p sonnet --effort high; eval-only, nothing shipped. - offload-eval-setup.sh: clone + index 4 memory-probe-verified "not-trained-on" repos (mtkruto/postybirb/shapeshift/trezor — small→large) so the no-codegraph baseline is honest. - offload-eval-3arm.sh / -frontload.sh: one repo, the arms (offload/raw/nocg, frontload). - offload-eval-matrix.sh / -frontload-matrix.sh: drive all 4 tiers. - offload-eval-hook.mjs: the front-load hook (self-locates its engine; CG_FRONTLOAD_DEBUG to log). - offload-eval-metrics.mjs / -judge.mjs (Sonnet) / -summarize.mjs: extract, score, aggregate. - offload-eval-ground-truth.json: source-verified canonical flows (the judge's reference). - offload-eval.md: usage + the 2026-06 findings (raw = the win; offload least-accurate; front-load solves adoption but exposes explore's dynamic-dispatch gaps). Scripts are path-portable (self-locating $HERE/$ENGINE; AGENT_EVAL_OUT scratch dir). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
25 lines
1.3 KiB
Bash
Executable File
25 lines
1.3 KiB
Bash
Executable File
#!/usr/bin/env bash
|
|
# Clone + index the 4 "not-trained-on" eval repos into $AGENT_EVAL_OUT/repos. These were
|
|
# selected via a no-tools memory-probe gate (Sonnet cannot answer their flow questions from
|
|
# memory — so the no-codegraph baseline is honest). Env: AGENT_EVAL_OUT=<scratch dir>
|
|
set -uo pipefail
|
|
HERE="$(cd "$(dirname "$0")" && pwd)"
|
|
ENGINE="$(cd "$HERE/../.." && pwd)"
|
|
BIN="$ENGINE/dist/bin/codegraph.js"
|
|
OUT="${AGENT_EVAL_OUT:-/tmp/cg-offload-eval}"
|
|
ROOT="$OUT/repos"; mkdir -p "$ROOT"
|
|
export CODEGRAPH_TELEMETRY=0 DO_NOT_TRACK=1
|
|
[ -f "$BIN" ] || { echo "engine not built: run 'npm run build' in $ENGINE first"; exit 1; }
|
|
|
|
clone_index() { # url name
|
|
echo "=== $2: clone ==="; rm -rf "$ROOT/$2"
|
|
git clone --quiet --depth 1 "$1" "$ROOT/$2" || { echo " clone FAILED"; return 1; }
|
|
echo "=== $2: index ==="
|
|
node "$BIN" init "$ROOT/$2" 2>&1 | grep -iE 'indexed|nodes|edges|error' | tail -2
|
|
}
|
|
clone_index https://github.com/MTKruto/MTKruto.git mtkruto # small (~322 TS)
|
|
clone_index https://github.com/mvdicarlo/postybirb-plus.git postybirb # medium (~608 TS)
|
|
clone_index https://github.com/shapeshift/web.git shapeshift # complex (~3.2k TS, 35-pkg monorepo)
|
|
clone_index https://github.com/trezor/trezor-suite.git trezor # large (~8k TS monorepo)
|
|
echo "###### SETUP DONE -> $ROOT"
|