Colby Mchenry
0682681175
All agent A/B arms now run claude --model sonnet --effort high by default (MODEL/EFFORT env overrides exist). Sonnet is the deliberate floor model: codegraph's users attach whatever host they already run (Cursor Composer, Gemini, ...), and a stronger model's tool-use masks the salience problems a weaker one exposes — what lands on Sonnet generalizes up; Opus/Fable-only wins don't generalize down. Policy recorded in CLAUDE.md's validation methodology; 11 hardcoded --model opus call sites across 9 eval scripts switched to the env-overridable default. Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
35 lines
1.3 KiB
Bash
Executable File
35 lines
1.3 KiB
Bash
Executable File
#!/usr/bin/env bash
|
|
# Headless Claude Code run against a repo with codegraph MCP, capturing the
|
|
# full stream-json so we can see tool calls + token usage. Complements the
|
|
# interactive itrun.sh: headless gives a clean per-tool breakdown + exact
|
|
# tokens/cost, but defaults to the general-purpose subagent (not Explore).
|
|
# To force the Explore path, ask for it in the prompt.
|
|
#
|
|
# Usage: run-agent.sh <repo-path> <label> "<prompt>"
|
|
# Env: AGENT_EVAL_OUT (default /tmp/agent-eval), CG_BIN (codegraph dist binary)
|
|
set -uo pipefail
|
|
|
|
REPO="$1"; LABEL="$2"; PROMPT="$3"
|
|
CG_BIN="${CG_BIN:-$(command -v codegraph || echo /usr/local/bin/codegraph)}"
|
|
OUT_DIR="${AGENT_EVAL_OUT:-/tmp/agent-eval}"; mkdir -p "$OUT_DIR"
|
|
OUT="$OUT_DIR/run-${LABEL}.jsonl"
|
|
|
|
MCP_CONFIG=$(cat <<JSON
|
|
{"mcpServers":{"codegraph":{"command":"${CG_BIN}","args":["serve","--mcp","--path","${REPO}"]}}}
|
|
JSON
|
|
)
|
|
|
|
echo "→ running [$LABEL] in $REPO"
|
|
cd "$REPO" || exit 1
|
|
|
|
claude -p "$PROMPT" \
|
|
--output-format stream-json --verbose \
|
|
--permission-mode bypassPermissions \
|
|
--model "${MODEL:-sonnet}" --effort "${EFFORT:-high}" \
|
|
--max-budget-usd 2 \
|
|
--strict-mcp-config --mcp-config "$MCP_CONFIG" \
|
|
> "$OUT" 2>"$OUT_DIR/run-${LABEL}.err"
|
|
|
|
echo "exit: $? | wrote $OUT ($(wc -l < "$OUT") lines)"
|
|
node "$(cd "$(dirname "$0")" && pwd)/parse-run.mjs" "$OUT" 2>/dev/null || true
|