diff --git a/.cursor/rules/codegraph.mdc b/.cursor/rules/codegraph.mdc index 00a3f81..17d144a 100644 --- a/.cursor/rules/codegraph.mdc +++ b/.cursor/rules/codegraph.mdc @@ -1,37 +1,22 @@ --- -description: CodeGraph MCP usage guide — when to use which tool +description: CodeGraph MCP usage guide — one tool, codegraph_explore alwaysApply: true --- ## CodeGraph -This project has a CodeGraph MCP server (`codegraph_*` tools) configured. CodeGraph is a tree-sitter-parsed knowledge graph of every symbol, edge, and file. Reads are sub-millisecond and return structural information grep cannot. +This project has a CodeGraph MCP server configured, exposing a single tool: `codegraph_explore`. CodeGraph is a tree-sitter-parsed knowledge graph of every symbol, edge, and file. Reads are sub-millisecond and return structural information grep cannot. -### When to prefer codegraph over native search +### Use codegraph_explore instead of reading files -Use codegraph for **structural** questions — what calls what, what would break, where is X defined, what is X's signature. Use native grep/read only for **literal text** queries (string contents, comments, log messages) or after you already have a specific file open. - -| Question | Tool | -|---|---| -| "Where is X defined?" / "Find symbol named X" | `codegraph_search` | -| "What calls function Y?" | `codegraph_callers` | -| "What does Y call?" | `codegraph_callees` | -| "How does X reach/become Y? / trace the flow from X to Y" | `codegraph_trace` (one call = the whole path, incl. callback/React/JSX dynamic hops) | -| "What would break if I changed Z?" | `codegraph_impact` | -| "Show me Y's signature / source / docstring" | `codegraph_node` | -| "Give me focused context for a task/area" | `codegraph_context` | -| "See several related symbols' source at once" | `codegraph_explore` | -| "What files exist under path/" | `codegraph_files` | -| "Is the index healthy?" | `codegraph_status` | +Reach for `codegraph_explore` before grep/find or Read for any **structural** question — how does X work, how does X reach Y, what calls what, where is X defined, or surveying an area. It takes a natural-language question or a bag of symbol/file names and returns the relevant symbols' **verbatim, line-numbered source** grouped by file (the same `\t` shape Read gives you, safe to Edit from), plus the call paths between them — including dynamic-dispatch hops (callbacks, React re-render, JSX children) grep can't follow — and a blast-radius summary of what depends on them. Name a file or symbol in the query to read its current source. ### Rules of thumb -- **Answer directly — don't delegate exploration.** For "how does X work" / architecture questions, answer with 2-3 codegraph calls: `codegraph_context` first, then ONE `codegraph_explore` for the source of the symbols it surfaces. For a specific **flow** ("how does X reach Y") start with `codegraph_trace` from→to — one call returns the whole path with dynamic hops bridged — then ONE `codegraph_explore` for the bodies; don't rebuild the path with `codegraph_search` + `codegraph_callers`. Codegraph IS the pre-built index, so spawning a separate file-reading sub-task/agent — or running a grep + read loop — repeats work codegraph already did and costs more for the same answer. +- **Answer directly — don't delegate exploration.** ONE `codegraph_explore` usually answers the whole question; follow up with another `codegraph_explore` naming more specific symbols if you need more. Codegraph IS the pre-built index, so spawning a separate file-reading sub-task/agent — or running a grep + read loop — repeats work codegraph already did and costs more for the same answer. - **Trust codegraph results.** They come from a full AST parse. Do NOT re-verify them with grep — that's slower, less accurate, and wastes context. -- **Don't grep first** when looking up a symbol by name. `codegraph_search` is faster and returns kind + location + signature in one call. -- **Don't chain `codegraph_search` + `codegraph_node`** when you just want context — `codegraph_context` is one call. -- **Don't loop `codegraph_node` over many symbols** — one `codegraph_explore` call returns several symbols' source grouped in a single capped call, while each separate node/Read call re-reads the whole context and costs far more. -- **Index lag — check the staleness banner, don't guess a wait.** When a codegraph response starts with "⚠️ Some files referenced below were edited since the last index sync…", the listed files are pending re-index — Read those specific files for accurate content. Files NOT in that banner are fresh and codegraph is authoritative for them. `codegraph_status` also lists pending files under "Pending sync". +- **Don't grep or Read first** to find or understand indexed code — one `codegraph_explore` returns the relevant source in a single round-trip. Reach for raw Read/Grep only to confirm a specific detail codegraph didn't cover, or for what it doesn't index (configs, docs). +- **Index lag — check the staleness banner, don't guess a wait.** When a codegraph response starts with "⚠️ Some files referenced below were edited since the last index sync…", the listed files are pending re-index — Read those specific files for accurate content. Files NOT in that banner are fresh and codegraph is authoritative for them. ### If `.codegraph/` doesn't exist diff --git a/README.md b/README.md index c182f45..b900c3a 100644 --- a/README.md +++ b/README.md @@ -262,7 +262,7 @@ agent writes src/Widget.ts → next agent query sees it ``` -**Verify any time** with `codegraph_status` (via MCP) or `codegraph status` (CLI). If anything is pending, you'll see a `### Pending sync:` section naming the files and their edit age. +**Verify any time** with `codegraph status` (CLI). If anything is pending, you'll see a `### Pending sync:` section naming the files and their edit age. The handful of cases where manual `codegraph sync` makes sense: the watcher is disabled (sandboxed environments, or `CODEGRAPH_NO_DAEMON=1`), or you're scripting against the index outside an agent session and want a pre-flight sync at the start of your script. @@ -300,7 +300,7 @@ CodeGraph detects web-framework routing files and emits `route` nodes linked by ## Mixed iOS / React Native / Expo bridging -Real iOS and React Native codebases live across multiple languages — a Swift caller invokes an Objective-C selector that's been auto-bridged, a JS file calls into a native module via the React Native bridge, a JSX component delegates to a native view manager. Static tree-sitter extraction stops at each language boundary. CodeGraph bridges them so `trace`, `callers`, `callees`, and `impact` connect end-to-end across the gap. +Real iOS and React Native codebases live across multiple languages — a Swift caller invokes an Objective-C selector that's been auto-bridged, a JS file calls into a native module via the React Native bridge, a JSX component delegates to a native view manager. Static tree-sitter extraction stops at each language boundary. CodeGraph bridges them so `codegraph_explore` connects the flow end-to-end across the gap — call paths and blast radius cross the boundary instead of stopping at it. | Boundary | JS / Swift side | Native side | How | |---|---|---|---| @@ -339,7 +339,7 @@ The installer will: - Ask which agent(s) to configure — auto-detects installed ones from: **Claude Code**, **Cursor**, **Codex CLI**, **opencode**, **Hermes Agent**, **Gemini CLI**, **Antigravity IDE**, **Kiro** - Prompt to install `codegraph` on your PATH (so agents can launch the MCP server) - Ask whether configs apply to all your projects or just this one -- Write each chosen agent's MCP server config, plus a small marker-fenced CodeGraph section in the agent's instructions file (`CLAUDE.md` / `AGENTS.md` / `GEMINI.md`) — that's how subagents and non-MCP agents learn the `codegraph explore` / `codegraph node` commands, since the MCP server's own guidance only reaches the main agent. Removed cleanly by `codegraph uninstall`. +- Write each chosen agent's MCP server config, plus a small marker-fenced CodeGraph section in the agent's instructions file (`CLAUDE.md` / `AGENTS.md` / `GEMINI.md`) — that's how subagents and non-MCP agents learn the `codegraph explore` command, since the MCP server's own guidance only reaches the main agent. Removed cleanly by `codegraph uninstall`. - Set up auto-allow permissions when Claude Code is one of the targets - Initialize your current project (local installs only) @@ -401,19 +401,14 @@ npm install -g @colbymchenry/codegraph { "permissions": { "allow": [ - "mcp__codegraph__codegraph_search", - "mcp__codegraph__codegraph_explore", - "mcp__codegraph__codegraph_callers", - "mcp__codegraph__codegraph_callees", - "mcp__codegraph__codegraph_impact", - "mcp__codegraph__codegraph_node", - "mcp__codegraph__codegraph_status", - "mcp__codegraph__codegraph_files" + "mcp__codegraph__*" ] } } ``` +One wildcard auto-approves every CodeGraph tool — `codegraph_explore` is the only one listed by default, but if you re-enable others via `CODEGRAPH_MCP_TOOLS` they're already permitted, no prompt. +
@@ -422,11 +417,11 @@ npm install -g @colbymchenry/codegraph CodeGraph's MCP server delivers its usage guidance to your agent **automatically**, in the MCP `initialize` response. In short, it tells the agent to: - **Answer structural questions directly with CodeGraph** — it *is* the pre-built index, so a grep/read loop just repeats work it already did. Treat the returned source as already read. -- **Pick the tool by intent:** `codegraph_explore` for almost anything — "how does X work", a flow/"how does X reach Y", or surveying an area (one call returns the relevant symbols' source grouped by file); `codegraph_search` to just locate a symbol; `codegraph_callers` for every call site (including callback registrations); `codegraph_node` for one symbol's full source + callers, or to read a file like the Read tool. +- **Reach for `codegraph_explore` for almost anything** — "how does X work", a flow/"how does X reach Y", or surveying an area. One call returns the relevant symbols' verbatim source grouped by file, the call paths between them (dynamic-dispatch hops included), and a blast-radius summary. Name a file or symbol in the query to read its current line-numbered source. - **Trust the results — don't re-verify with grep**, and check the staleness banner after edits. - In a workspace with no index, CodeGraph announces itself inactive and serves no tools — indexing stays your decision. -The exact text is `src/mcp/server-instructions.ts` — the single source of truth for the main agent. Because subagents and non-MCP harnesses never see the MCP guidance, the installer also writes a four-line marker-fenced section into the agent's instructions file pointing at the `codegraph explore` / `codegraph node` CLI equivalents. +The exact text is `src/mcp/server-instructions.ts` — the single source of truth for the main agent. Because subagents and non-MCP harnesses never see the MCP guidance, the installer also writes a short marker-fenced section into the agent's instructions file pointing at the `codegraph explore` CLI equivalent.
@@ -447,7 +442,7 @@ The exact text is `src/mcp/server-instructions.ts` — the single source of trut ┌───────────────────────────────────────────────────────────────────┐ │ CodeGraph MCP Server │ │ │ -│ explore · search · callers · callees · impact · node │ +│ explore · one call → verbatim source + call flow + blast radius │ │ │ │ │ ▼ │ │ SQLite knowledge graph │ @@ -524,16 +519,13 @@ fi ## MCP Tools -When running as an MCP server, CodeGraph exposes a focused set of four tools — measured agent behavior showed a leaner list steers agents to the right tool and saves context every session: +When running as an MCP server, CodeGraph exposes a **single tool** — `codegraph_explore`. Measured agent behavior showed that one strong tool steers agents better than a menu of narrower ones — fewer mis-picks, and it saves context every session: | Tool | Purpose | |------|---------| -| `codegraph_explore` | **Primary.** Answer almost any question in one call — "how does X work", a flow ("how does X reach Y"), or surveying an area — returning the relevant symbols' verbatim source grouped by file, plus a relationship map and blast radius. Surfaces dynamic-dispatch hops (callbacks, React re-render, interface→impl) grep can't follow. | -| `codegraph_node` | One symbol's full source + caller/callee trail (every overload for an ambiguous name) — or pass a file path to **read a whole file like the Read tool** (same line-numbered output, `offset`/`limit`), with its dependents attached. | -| `codegraph_search` | Find symbols by name across the codebase | -| `codegraph_callers` | Every call site of a function — including where it's registered as a callback — with one section per definition when several share a name | +| `codegraph_explore` | Answer almost any question in one call — "how does X work", a flow ("how does X reach Y"), or surveying an area — returning the relevant symbols' verbatim source grouped by file, plus the call paths between them and a blast-radius summary. Surfaces dynamic-dispatch hops (callbacks, React re-render, interface→impl) grep can't follow. Name a file or symbol in the query to read its current line-numbered source, the same shape the Read tool gives you. | -Four more tools (`codegraph_callees`, `codegraph_impact`, `codegraph_files`, `codegraph_status`) stay fully functional but unlisted by default — measured across eval runs, agents never or rarely picked them, and their information already arrives inline on the four above (explore's blast-radius section, node's dependents note, a symbol's body as its callee list). Re-enable any of them with the `CODEGRAPH_MCP_TOOLS` environment variable (e.g. `CODEGRAPH_MCP_TOOLS=explore,node,search,callers,impact`), or use their CLI equivalents (`codegraph callees` / `impact` / `files` / `status`). +The other tools (`codegraph_node`, `codegraph_search`, `codegraph_callers`, `codegraph_callees`, `codegraph_impact`, `codegraph_files`, `codegraph_status`) stay fully functional but **unlisted by default** — everything they return already arrives inline on `codegraph_explore` (its blast-radius section, the relationship map, a symbol's body as its callee list). Re-enable any of them for the MCP surface with the `CODEGRAPH_MCP_TOOLS` environment variable (e.g. `CODEGRAPH_MCP_TOOLS=explore,node,search,callers`), or use their CLI equivalents (`codegraph node` / `query` / `callers` / `callees` / `impact` / `files` / `status`). In a workspace with no `.codegraph/` index, the server announces itself inactive and lists **no** tools — agents work normally with their built-in tools, and indexing stays your decision. diff --git a/__tests__/installer-targets.test.ts b/__tests__/installer-targets.test.ts index 91753c3..eb15da0 100644 --- a/__tests__/installer-targets.test.ts +++ b/__tests__/installer-targets.test.ts @@ -1031,7 +1031,7 @@ describe('Installer targets — partial-state idempotency', () => { // The unrelated GitKraken hook survives untouched. expect(stopCommands.some((c: string) => c.includes('gk') && c.includes('ai hook run'))).toBe(true); // Permissions still written as normal alongside the cleanup. - expect(after.permissions?.allow).toContain('mcp__codegraph__codegraph_search'); + expect(after.permissions?.allow).toContain('mcp__codegraph__*'); }); it('claude: cleanupLegacyHooks preserves a sibling hook sharing our matcher group', () => { diff --git a/__tests__/mcp-tool-allowlist.test.ts b/__tests__/mcp-tool-allowlist.test.ts index 08067c9..579e8b8 100644 --- a/__tests__/mcp-tool-allowlist.test.ts +++ b/__tests__/mcp-tool-allowlist.test.ts @@ -17,18 +17,13 @@ describe('CODEGRAPH_MCP_TOOLS allowlist', () => { const listed = () => new ToolHandler(null).getTools().map(t => t.name).sort(); - it('exposes the default 4-tool surface when unset', () => { + it('exposes ONLY codegraph_explore by default when unset', () => { delete process.env[ENV]; - // The default set (see DEFAULT_MCP_TOOLS): explore + node are the - // validated workhorses, search the cheap lookup, callers the one - // irreplaceable enumerator. callees/impact/files/status stay defined - // and executable but unlisted — impact appeared in ZERO recorded runs. - expect(listed()).toEqual([ - 'codegraph_callers', - 'codegraph_explore', - 'codegraph_node', - 'codegraph_search', - ]); + // The default set (see DEFAULT_MCP_TOOLS) is pared to explore alone — the one + // tool that earns its place (verbatim source grouped by file, plus the reasoned + // flow map under the offload). node/search/callers/callees/impact/files/status + // stay defined and executable but unlisted; CODEGRAPH_MCP_TOOLS re-enables them. + expect(listed()).toEqual(['codegraph_explore']); }); it('re-enables an unlisted tool via the allowlist (impact)', () => { @@ -48,8 +43,7 @@ describe('CODEGRAPH_MCP_TOOLS allowlist', () => { it('treats an empty/whitespace value as unset (default surface)', () => { process.env[ENV] = ' '; - expect(listed()).toHaveLength(4); - expect(listed()).toContain('codegraph_explore'); + expect(listed()).toEqual(['codegraph_explore']); }); it('rejects a disabled tool on execute (defense in depth)', async () => { diff --git a/__tests__/mcp-unindexed.test.ts b/__tests__/mcp-unindexed.test.ts index 2b0019d..5194193 100644 --- a/__tests__/mcp-unindexed.test.ts +++ b/__tests__/mcp-unindexed.test.ts @@ -116,7 +116,7 @@ describe('Unindexed-workspace session policy', () => { expect(instructions).toMatch(/inactive/i); expect(instructions).toMatch(/codegraph init/); // The full playbook must NOT be sent into a session where every call fails - expect(instructions).not.toMatch(/Tool selection by intent/); + expect(instructions).not.toMatch(/How to query/); expect(instructions).not.toMatch(/codegraph_explore/); }); @@ -128,7 +128,7 @@ describe('Unindexed-workspace session policy', () => { expect((res.result as { tools: unknown[] }).tools).toEqual([]); }); - it('an INDEXED workspace still gets the full playbook and all tools', async () => { + it('an INDEXED workspace still gets the full playbook and the explore tool', async () => { fs.writeFileSync(path.join(tempDir, 'index.ts'), 'export function hello(): string { return "hi"; }\n'); const cg = await CodeGraph.init(tempDir, { index: true }); cg.close(); @@ -136,15 +136,15 @@ describe('Unindexed-workspace session policy', () => { child = spawnServer(tempDir); const init = await request(child, { id: 0, method: 'initialize', params: initializeParams(tempDir) }); const instructions = (init.result as { instructions: string }).instructions; - expect(instructions).toMatch(/Tool selection by intent/); + expect(instructions).toMatch(/How to query/); expect(instructions).not.toMatch(/inactive/i); const list = await request(child, { id: 1, method: 'tools/list' }); const tools = (list.result as { tools: Array<{ name: string }> }).tools; - // A 1-file project triggers the pre-existing tiny-repo tool gating (a - // reduced core set) — the contract under test is "indexed → tools are - // PRESENT", in contrast to the unindexed empty list above. - expect(tools.length).toBeGreaterThanOrEqual(3); + // The default surface is pared to explore alone (see DEFAULT_MCP_TOOLS) — the + // contract under test is "indexed → tools are PRESENT", in contrast to the + // unindexed empty list above. + expect(tools.length).toBeGreaterThanOrEqual(1); expect(tools.map((t) => t.name)).toContain('codegraph_explore'); }); }); diff --git a/__tests__/redux-thunk-synthesizer.test.ts b/__tests__/redux-thunk-synthesizer.test.ts new file mode 100644 index 0000000..4494193 --- /dev/null +++ b/__tests__/redux-thunk-synthesizer.test.ts @@ -0,0 +1,82 @@ +import { describe, it, expect, beforeEach, afterEach } from 'vitest'; +import * as fs from 'node:fs'; +import * as path from 'node:path'; +import * as os from 'node:os'; +import { CodeGraph } from '../src'; + +/** + * End-to-end test for the redux-thunk dispatch-chain synthesizer. + * + * `createAsyncThunk(prefix, async (a, api) => {...})` passes the async body as an argument, so + * tree-sitter never makes it its own function node — the thunk `constant`'s body calls (incl. + * `dispatch(nextThunk(...))`) are orphaned and `callees(thunk)` is empty. Verify the synthesizer + * body-scans each thunk constant and links it → each dispatched thunk, so the chain + * `outer → inner → deep` connects end-to-end; and that a non-thunk constant is skipped. + */ +describe('redux-thunk synthesizer', () => { + let dir: string; + beforeEach(() => { + dir = fs.mkdtempSync(path.join(os.tmpdir(), 'redux-thunk-fixture-')); + }); + afterEach(() => { + fs.rmSync(dir, { recursive: true, force: true }); + }); + + it('links each thunk constant to the thunks it dispatches, and skips non-thunks', async () => { + fs.writeFileSync( + path.join(dir, 'package.json'), + JSON.stringify({ name: 'app', dependencies: { '@reduxjs/toolkit': '^2' } }) + ); + fs.writeFileSync( + path.join(dir, 'thunks.ts'), + `import { createAsyncThunk } from '@reduxjs/toolkit'; + +export const deepThunk = createAsyncThunk('app/deep', async (n: number) => { + return n * 2; +}); + +export const innerThunk = createAsyncThunk('app/inner', async (n: number, { dispatch }) => { + return dispatch(deepThunk(n)); +}); + +export const outerThunk = createAsyncThunk('app/outer', async (n: number, { dispatch }) => { + await dispatch(innerThunk(n)); +}); + +// Non-thunk constant that only MENTIONS dispatch in a string — must be skipped. +export const notAThunk = 'dispatch(innerThunk())'; +` + ); + + const cg = await CodeGraph.init(dir, { silent: true }); + await cg.indexAll(); + + const db = (cg as any).db.db; + const rows = db + .prepare( + `SELECT s.name source_name, s.kind source_kind, t.name target_name, + json_extract(e.metadata,'$.via') via, + json_extract(e.metadata,'$.registeredAt') registeredAt + FROM edges e + JOIN nodes s ON s.id = e.source + JOIN nodes t ON t.id = e.target + WHERE json_extract(e.metadata,'$.synthesizedBy') = 'redux-thunk'` + ) + .all(); + cg.close?.(); + + // The dispatch chain connects: outer → inner → deep. + const pairs = new Set(rows.map((r: any) => `${r.source_name}>${r.target_name}`)); + expect(pairs.has('outerThunk>innerThunk')).toBe(true); + expect(pairs.has('innerThunk>deepThunk')).toBe(true); + + // Sources are thunk constants; the non-thunk string constant is never a source. + expect(rows.every((r: any) => r.source_kind === 'constant')).toBe(true); + expect(rows.some((r: any) => r.source_name === 'notAThunk')).toBe(false); + + // Edges are 'calls' with the wiring site surfaced for the agent. + const outer = rows.find((r: any) => r.source_name === 'outerThunk'); + expect(outer.via).toBe('innerThunk'); + expect(outer.registeredAt).toMatch(/thunks\.ts:\d+/); + }); +}); diff --git a/scripts/agent-eval/offload-eval-effort.mjs b/scripts/agent-eval/offload-eval-effort.mjs new file mode 100644 index 0000000..c6d275e --- /dev/null +++ b/scripts/agent-eval/offload-eval-effort.mjs @@ -0,0 +1,108 @@ +#!/usr/bin/env node +// Effort A/B — does CODEGRAPH_OFFLOAD_EFFORT=high improve offload SYNTHESIS FIDELITY vs low? +// Probe-based (no agent): for each repo × effort × rep, run codegraph_explore with the offload +// ON on the canonical question, capture the synthesized answer + AI tokens/cost/latency, then +// Sonnet-judge that answer's fidelity vs source-verified ground truth. Isolates the synthesis +// from agent/adoption noise. Requires `codegraph login` (managed offload) + indexed repos. +// +// Env: REPS (default 3) · CG_ENGINE (engine repo) · AGENT_EVAL_OUT (repos under /repos) · CONC (judge concurrency) +import { pathToFileURL, fileURLToPath } from 'node:url'; +import { resolve, dirname, join } from 'node:path'; +import { readFileSync, writeFileSync, existsSync, rmSync } from 'node:fs'; +import { execFile } from 'node:child_process'; +import { tmpdir } from 'node:os'; + +const HERE = dirname(fileURLToPath(import.meta.url)); +const ENGINE = process.env.CG_ENGINE || resolve(HERE, '..', '..'); +const OUT = process.env.AGENT_EVAL_OUT || '/tmp/cg-offload-eval'; +const REPOS = join(OUT, 'repos'); +const GT = JSON.parse(readFileSync(resolve(HERE, 'offload-eval-ground-truth.json'), 'utf8')); +const REPS = Number(process.env.REPS || 3); +const CONC = Number(process.env.CONC || 4); +const EFFORTS = (process.env.EFFORTS_FILTER || 'low,high').split(','); +const ONLY = process.env.REPOS_FILTER ? new Set(process.env.REPOS_FILTER.split(',')) : null; +const TIER = { mtkruto: 'small', postybirb: 'medium', shapeshift: 'complex', trezor: 'large' }; + +const load = async (rel) => import(pathToFileURL(resolve(ENGINE, rel)).href); +const idx = await load('dist/index.js'); +const toolsMod = await load('dist/mcp/tools.js'); +const CodeGraph = idx.default?.default ?? idx.default ?? idx.CodeGraph; +const ToolHandler = toolsMod.ToolHandler ?? toolsMod.default?.ToolHandler; +if (typeof CodeGraph?.openSync !== 'function' || typeof ToolHandler !== 'function') { + console.error('could not load engine from', ENGINE); process.exit(2); +} + +const fidPrompt = (gt, ans) => `You are scoring the FIDELITY of a machine-synthesized code-exploration answer against verified ground truth. Do NOT use any tools. + +QUESTION: ${gt.question} + +VERIFIED GROUND TRUTH (the actual call path + files): +${gt.truth} + +SYNTHESIZED ANSWER (to score): +${ans || '(empty)'} + +Judge: (1) is the traced call path correct vs ground truth? (2) are the cited files/symbols correct (not fabricated)? (3) if it gave a "Coverage:" verdict, was it honest? A confident WRONG trace is the worst outcome — penalize it harder than an honest partial. +Output ONLY minified JSON: {"verdict":"pass|partial|fail","score":<0-100>,"fabrication":,"coverageHonest":,"note":"<=20 words"}`; + +const askJudge = (prompt) => new Promise((res) => { + execFile('claude', ['-p', prompt, '--model', 'sonnet', '--effort', 'high', '--max-budget-usd', '0.5', + '--strict-mcp-config', '--mcp-config', '{"mcpServers":{}}'], + { cwd: OUT, maxBuffer: 1 << 24, timeout: 120000 }, (err, stdout) => { + const m = (stdout || '').match(/\{[\s\S]*\}/); + if (!m) return res({ verdict: 'error', score: null, note: (err ? err.message : 'no json').slice(0, 60) }); + try { res(JSON.parse(m[0])); } catch { res({ verdict: 'error', score: null }); } + }); +}); + +// ---- 1. Probe: collect synthesized answers at each effort ------------------- +const records = []; +for (const repo of Object.keys(GT)) { + if (ONLY && !ONLY.has(repo)) continue; + const dir = join(REPOS, repo); + if (!existsSync(join(dir, '.codegraph'))) { console.error('skip (not indexed):', repo); continue; } + const cg = CodeGraph.openSync(dir); + const h = new ToolHandler(cg); + for (const effort of EFFORTS) { + for (let rep = 1; rep <= REPS; rep++) { + process.env.CODEGRAPH_OFFLOAD_EFFORT = effort; + const usageLog = join(tmpdir(), `effort-${repo}-${effort}-${rep}.jsonl`); + try { rmSync(usageLog); } catch { /* none */ } + process.env.CODEGRAPH_OFFLOAD_USAGE_LOG = usageLog; + let answer = ''; + try { answer = (await h.execute('codegraph_explore', { query: GT[repo].question }))?.content?.[0]?.text ?? ''; } + catch (e) { console.error(` ${repo}/${effort}#${rep} explore failed: ${e?.message}`); } + const fired = /Synthesized by CodeGraph/.test(answer); + const ai = { tokens: 0, cost: 0, ms: 0 }; + if (existsSync(usageLog)) for (const e of readFileSync(usageLog, 'utf8').split('\n').filter(Boolean).map(JSON.parse)) { + ai.tokens += e.totalTokens || 0; ai.cost += e.costUsd || 0; ai.ms += e.ms || 0; + } + records.push({ repo, tier: TIER[repo], effort, rep, fired, ai, answer }); + console.error(` ${repo}/${effort}#${rep}: fired=${fired} ${ai.tokens}tok $${ai.cost.toFixed(4)} ${ai.ms}ms`); + } + } + try { cg.close?.(); } catch { /* none */ } +} + +// ---- 2. Judge fidelity (concurrency) --------------------------------------- +console.error(`\njudging ${records.length} answers (concurrency ${CONC})...`); +let done = 0; +const q = [...records]; +async function worker() { while (q.length) { const r = q.shift(); r.fid = await askJudge(fidPrompt(GT[r.repo], r.answer)); console.error(` [${++done}/${records.length}] ${r.repo}/${r.effort}#${r.rep}: ${r.fid.verdict} ${r.fid.score ?? ''}`); } } +await Promise.all(Array.from({ length: CONC }, worker)); +writeFileSync(join(OUT, 'effort-results.jsonl'), records.map((r) => JSON.stringify(r)).join('\n') + '\n'); + +// ---- 3. Aggregate: low vs high per repo ------------------------------------ +const med = (a) => { a = a.filter((x) => x != null).sort((x, y) => x - y); return a.length ? (a.length % 2 ? a[(a.length - 1) / 2] : (a[a.length / 2 - 1] + a[a.length / 2]) / 2) : null; }; +console.log(`\n${'='.repeat(80)}\nEFFORT A/B — offload synthesis fidelity (probe, n=${REPS}/cell)\n${'='.repeat(80)}`); +console.log(`${'repo'.padEnd(11)} ${'tier'.padEnd(8)} ${'effort'.padEnd(6)} fired ${'fid(med)'.padStart(8)} ${'fab%'.padStart(5)} ${'AItok'.padStart(7)} ${'AIcost'.padStart(8)} ${'ms(med)'.padStart(8)}`); +for (const repo of Object.keys(GT)) { + for (const effort of EFFORTS) { + const rs = records.filter((r) => r.repo === repo && r.effort === effort); + if (!rs.length) continue; + const fids = rs.map((r) => r.fid?.score).filter((x) => x != null); + const fab = rs.filter((r) => r.fid?.fabrication === true).length; + console.log(`${repo.padEnd(11)} ${TIER[repo].padEnd(8)} ${effort.padEnd(6)} ${rs.filter((r) => r.fired).length}/${rs.length} ${String(med(fids) ?? '—').padStart(8)} ${String(Math.round(100 * fab / rs.length) + '%').padStart(5)} ${String(Math.round(med(rs.map((r) => r.ai.tokens)) / 1000) + 'k').padStart(7)} ${('$' + (med(rs.map((r) => r.ai.cost)) ?? 0).toFixed(4)).padStart(8)} ${String(med(rs.map((r) => r.ai.ms)) ?? '—').padStart(8)}`); + } +} +console.log(''); diff --git a/scripts/agent-eval/offload-eval-metrics.mjs b/scripts/agent-eval/offload-eval-metrics.mjs index 916453c..97cb35f 100644 --- a/scripts/agent-eval/offload-eval-metrics.mjs +++ b/scripts/agent-eval/offload-eval-metrics.mjs @@ -43,7 +43,11 @@ for (const line of lines) { const text = Array.isArray(b.content) ? b.content.map(c => (typeof c === 'string' ? c : c.text || '')).join('') : (typeof b.content === 'string' ? b.content : ''); - if (/Synthesized by CodeGraph/.test(text)) { offloadAnswers.push(text); exploreResults++; } + // An offload answer is either the 'plain'/'report' synthesis (carries the + // "Synthesized by CodeGraph" footer) or a 'refs' answer (carries the re-expanded + // "### Referenced source — verbatim" appendix). A refs call that cited nothing + // valid falls back to RAW source, which is correctly counted as a raw explore below. + if (/Synthesized by CodeGraph|### Referenced source — verbatim/.test(text)) { offloadAnswers.push(text); exploreResults++; } else if (/Found \d+ symbols? across|## Exploration:/.test(text)) exploreResults++; } } diff --git a/scripts/agent-eval/offload-eval-refs1.sh b/scripts/agent-eval/offload-eval-refs1.sh new file mode 100755 index 0000000..aac6fb1 --- /dev/null +++ b/scripts/agent-eval/offload-eval-refs1.sh @@ -0,0 +1,50 @@ +#!/usr/bin/env bash +# ONE offload run on ONE indexed repo at a given offload STYLE (plain|refs), so we can +# watch a single agent transcript at a time (the user's one-run-at-a-time methodology). +# The OFFLOAD reasoning runs in the prewarmed DAEMON process, so the style env must be +# set on BOTH the daemon and the client MCP config. Writes one metrics line to RESULTS +# and leaves the raw stream-json at $RUNS/-