deusdata--codebase-memory-mcp/docs/TEST_PLAN.md

MCP vs Explorer Quality Benchmark — Test Plan (v8: 66 Languages)

Purpose

Compare codebase-memory-mcp's structured graph queries against Claude Code's Explore agent (Grep/Glob/Read only) across all supported languages. Both phases use AI agents with the same 12-question set — the MCP agent gets MCP tools, the Explorer agent gets Grep/Glob/Read. Same questions, fair comparison.

Prerequisites

• Built codebase-memory-mcp binary at ~/.local/bin/codebase-memory-mcp
• Internet access for cloning repositories
• ~5GB disk space for repos in /tmp/bench/
• Claude Code session for both agent phases

Phase 0: Repository Setup

scripts/clone-bench-repos.sh /tmp/bench

Clones repositories (shallow, --depth 1) and creates symlinks for shared repos. All 66 languages from AllLanguages() are benchmarked — shared-repo languages get their own Group E agent runs against the same repo, testing language-specific parsing.

Programming Languages — Tier 1 (44 languages)

#	Language	Repository	Notes
1	Go	go-chi/chi	—
2	Python	httpie/cli	—
3	JavaScript	expressjs/express	HTML shares this repo (symlink)
4	TypeScript	trpc/trpc	—
5	TSX	shadcn-ui/ui	CSS shares this repo (symlink)
6	Java	spring-petclinic	SQL shares this repo (symlink)
7	Kotlin	JetBrains/Exposed	—
8	Scala	playframework/play-samples	—
9	Rust	meilisearch/meilisearch	TOML shares this repo (symlink)
10	C	redis/redis	—
11	C++	nlohmann/json	CUDA shares this repo (symlink)
12	C#	ardalis/CleanArchitecture	—
13	PHP	koel/koel	—
14	Ruby	sinatra/sinatra	—
15	Lua	awesomeWM/awesome	—
16	Bash	bash-it/bash-it	—
17	Zig	tigerbeetle/tigerbeetle	—
18	Haskell	jgm/pandoc	—
19	OCaml	ocaml/dune	—
20	Elixir	plausible/analytics	—
21	Erlang	ninenines/cowboy	—
22	Objective-C	AFNetworking/AFNetworking	—
23	Swift	Alamofire/Alamofire	—
24	Dart	felangel/bloc	—
25	Perl	mojolicious/mojo	—
26	Groovy	spockframework/spock	—
27	R	tidyverse/dplyr	—
28	Clojure	clojure/clojure	Functional
29	F#	giraffe-fsharp/Giraffe	Pure F# web framework
30	Julia	SciML/DifferentialEquations.jl	Pure Julia, scientific
31	Vim Script	SpaceVim/SpaceVim	1000+ .vim files
32	Nix	NixOS/nixpkgs	Config/scripting
33	Common Lisp	lem-project/lem	200+ .lisp files
34	Elm	elm/compiler	Functional
35	Fortran	cp2k/cp2k	Systems/scientific
36	CUDA	NVIDIA/cuda-samples	~100 .cu files
37	COBOL	OCamlPro/gnucobol	Legacy
38	Verilog	YosysHQ/yosys	Hardware
39	Emacs Lisp	emacs-mirror/emacs	Scripting
40	MATLAB	acristoffers/tree-sitter-matlab	Scientific (test corpus)
41	Lean 4	leanprover-community/mathlib4	Theorem prover, scientific
42	FORM	vermaseren/form	Symbolic algebra
43	Magma	← symlink to lean	.m files disambiguated via markers
44	Wolfram	WolframResearch/WolframLanguageForJupyter	Symbolic computing

Helper Languages — Tier 2 (22 languages)

#	Language	Repository	Notes
45	YAML	kubernetes/examples	Config language
46	HCL	terraform-aws-modules/terraform-aws-eks	Config language
47	SCSS	twbs/bootstrap	Dedicated repo
48	Dockerfile	docker-library/official-images	Dedicated repo
49	HTML	← symlink to javascript	Shared repo (expressjs/express) — benchmarked with Group E questions
50	CSS	← symlink to tsx	Shared repo (shadcn-ui/ui) — benchmarked with Group E questions
51	TOML	← symlink to rust	Shared repo (meilisearch/meilisearch) — benchmarked with Group E questions
52	SQL	← symlink to java	Shared repo (spring-petclinic) — benchmarked with Group E questions
53	JSON	← symlink to typescript	Shared repo (trpc/trpc) — benchmarked with Group E questions
54	XML	← symlink to java	Shared repo (spring-petclinic) — benchmarked with Group E questions
55	Markdown	← symlink to python	Shared repo (httpie/cli) — benchmarked with Group E questions
56	Makefile	← symlink to c	Shared repo (redis/redis) — benchmarked with Group E questions
57	CMake	kitware/CMake	Build system
58	Protobuf	googleapis/googleapis	Schema language
59	GraphQL	graphql/graphql-spec	Schema language
60	Vue	vuejs/vue	Frontend framework
61	Svelte	sveltejs/svelte	Frontend framework
62	Meson	mesonbuild/meson	Build system
63	GLSL	repalash/Open-Shaders	Hundreds of .glsl/.vert/.frag files
64	INI	← symlink to python	Shared repo (httpie/cli) — .cfg/.ini files
65	Kubernetes	← symlink to yaml	YAML subtype, detected by content (kustomization.yaml, Deployment, etc.)
66	Kustomize	← symlink to yaml	YAML subtype, detected by kustomization.yaml presence

Phase 1: Indexing (Automated)

1a. Database Cleanup (mandatory before each benchmark run)

Delete all existing project databases so every language gets a cold, fresh index with accurate timing:

# Remove all project DBs (per-project .db files in cache dir)
rm -f ~/.cache/codebase-memory-mcp/*.db

# Remove stale bench-results metrics from previous runs
rm -rf /tmp/bench-results
mkdir -p /tmp/bench-results

Why: Re-indexing an already-indexed project reuses cached data and produces artificially low index times. A cold index reflects the real-world first-run cost that users experience.

1b. Index all 66 languages

ALL_LANGS="go python javascript typescript tsx java kotlin scala rust c cpp csharp php ruby lua bash zig haskell ocaml elixir erlang objc swift dart perl groovy r clojure fsharp julia vimscript nix commonlisp elm fortran cuda cobol verilog emacslisp matlab lean form magma wolfram yaml hcl scss dockerfile html css toml sql json xml markdown makefile cmake protobuf graphql vue svelte meson glsl ini kubernetes kustomize"

for lang in $ALL_LANGS; do
  scripts/benchmark-index.sh ~/.local/bin/codebase-memory-mcp "$lang" /tmp/bench/"$lang" /tmp/bench-results
done

What the script does per language

1. Count LOC (lines of code across all source files, excluding .git/node_modules/vendor/target/build/dist)
2. Count source files (same file extensions as LOC)
3. Index repository via codebase-memory-mcp cli --raw index_repository — timed end-to-end
4. Save all metrics to /tmp/bench-results/<lang>/

Output files per language

/tmp/bench-results/<lang>/
  00-index.json         # Full index response (project name, node/edge counts)
  loc.txt               # Total lines of code (source files only)
  file-count.txt        # Number of source files
  index-time.txt        # Cold index time in milliseconds
  project.txt           # Project name (used by agent phases)
  nodes.txt             # Graph node count
  edges.txt             # Graph edge count

Indexing metrics table (fill in after Phase 1 completes)

Language	Repo	Files	LOC	Index (ms)	Nodes	Edges
(generated)

To generate this table after indexing:

echo "| Language | Repo | Files | LOC | Index (ms) | Nodes | Edges |"
echo "|----------|------|-------|-----|------------|-------|-------|"
for lang in $ALL_LANGS; do
  d="/tmp/bench-results/$lang"
  proj=$(cat "$d/project.txt" 2>/dev/null || echo "-")
  files=$(cat "$d/file-count.txt" 2>/dev/null || echo "-")
  loc=$(cat "$d/loc.txt" 2>/dev/null || echo "-")
  ms=$(cat "$d/index-time.txt" 2>/dev/null || echo "-")
  nodes=$(cat "$d/nodes.txt" 2>/dev/null || echo "-")
  edges=$(cat "$d/edges.txt" 2>/dev/null || echo "-")
  echo "| $lang | $proj | $files | $loc | $ms | $nodes | $edges |"
done

Language Groups

Each group gets 12 task-oriented questions tailored to its structural features. Questions simulate real developer tasks — the kind of questions you'd ask when joining a project, fixing a bug, or adding a feature.

Group A: Class-based OOP

Languages: Java, Kotlin, Scala, C#, PHP, Ruby, Dart, Groovy, Swift, Objective-C

#	Developer Task	What it tests
1	"I'm auditing the API surface. Find every controller/handler class and list their route-handling methods."	Route/handler discovery
2	"I need to understand the type system. What are the main interfaces/protocols and what classes implement them?"	Inheritance + interface mapping
3	"A dependency needs upgrading. Find all places where <most-used class> is instantiated or referenced."	Usage tracking + impact analysis
4	"Show me the source code of the largest service/controller class. I need to understand it before refactoring."	Code snippet retrieval
5	"I found a bug in <central method>. What does it call? Show the full outbound call tree."	Outbound call trace
6	"For that same method — who calls it? Trace back to the entry point."	Inbound call trace
7	"How does a request flow from HTTP entry to database and back? Describe the architectural layers."	Architecture + data flow
8	"Which methods have the most callers? I need to know what's riskiest to change."	Hub detection / impact ranking
9	"I'm adding a new feature. Find all abstract classes or interfaces I should implement."	Extension point discovery
10	"Where is input validation done? Find all validation, sanitization, or guard functions."	Pattern search (domain-specific)
11	"What external libraries does the main entry point depend on? List all third-party imports."	Dependency analysis
12	"I'm onboarding. Describe the package structure and what each major package/namespace does."	Project structure

Group B: Systems Languages

Languages: Go, Rust, C, C++, Zig, CUDA, Fortran, GLSL, Verilog, COBOL, MATLAB, Lean 4, FORM

#	Developer Task	What it tests
1	"Find all exported/public functions that form the library's public API surface."	Public API discovery
2	"What interfaces (Go) / traits (Rust) / header-declared types exist, and what implements them?"	Interface/trait mapping
3	"I need to understand error handling. Find all functions that return errors and how errors propagate."	Error flow analysis
4	"Show the source of the most complex function. I need to refactor it."	Code snippet retrieval
5	"Pick a central function in the hot path. What does it call? Map the dependency tree."	Outbound call trace
6	"For that function — who calls it? Show all call sites across the codebase."	Inbound call trace
7	"What runs at initialization? Find all init/main/setup functions and what they configure."	Init/startup discovery
8	"How is the codebase layered? What are the package boundaries and dependency direction between them?"	Architecture understanding
9	"Which functions have the highest fan-out (call the most other functions)? These are complexity hotspots."	Complexity / hub detection
10	"Find all resource management patterns — alloc/free, defer, Drop, close, cleanup functions."	Resource lifecycle analysis
11	"What does the main package/module import? List all direct dependencies."	Dependency analysis
12	"Describe the module/package structure and what each major component is responsible for."	Project structure

COBOL note: COBOL is a procedural programming language (programs, divisions, sections, paragraphs) and belongs here despite its legacy status. Interpret Group B questions as: Q1=PARAs/SECTIONs forming the public API, Q2=COPY books and 01-level data structures in lieu of interfaces/types, Q3=error handling via ON EXCEPTION/INVALID KEY/NOT ON SIZE ERROR, Q7=IDENTIFICATION/ENVIRONMENT divisions and entry points, Q10=OPEN/CLOSE/READ/WRITE file-handling verbs as resource management.

Group C: Dynamic / Scripting Languages

Languages: Python, JavaScript, TypeScript, TSX, Lua, Perl, R, Bash, Vim Script, Nix, Emacs Lisp

#	Developer Task	What it tests
1	"Find all route definitions or request handlers. List URL patterns and handler functions."	Route/handler discovery
2	"Find all middleware, decorators, or higher-order functions that wrap behavior."	Wrapper/decorator discovery
3	"What are the main classes or modules? Show the class hierarchy or module dependency graph."	Structural overview
4	"Show the source of the most complex function. I need to understand it."	Code snippet retrieval
5	"Pick a key business logic function. What does it call? Trace the outbound call chain."	Outbound call trace
6	"For that function — who calls it? Find all call sites."	Inbound call trace
7	"Where is configuration loaded? Find all config, env var, or settings access points."	Config discovery
8	"How does a request flow through the app? Describe the layers from entry to response."	Architecture + data flow
9	"What does this package export? List all public functions/classes intended for consumers."	Public API discovery
10	"Which functions are called from the most places? These are the most impactful to change."	Hub detection / impact ranking
11	"What does the main entry file import? List all dependencies and what they provide."	Dependency analysis
12	"Describe the directory layout and what each major directory is responsible for."	Project structure

Group D: Functional Languages

Languages: Haskell, OCaml, Elixir, Erlang, Clojure, F#, Julia, Common Lisp, Elm, Magma, Wolfram

#	Developer Task	What it tests
1	"Find all exported/public functions that serve as the module's API."	Public API discovery
2	"What are the main type definitions — records, ADTs, typeclasses (Haskell/OCaml), behaviours (Erlang/Elixir)? List them with fields/variants."	Type system mapping
3	"Find the most complex pattern-matching function — show all its clauses and what it dispatches on."	Pattern match analysis
4	"Show the source of the longest function. What does it do?"	Code snippet retrieval
5	"Pick a central function. What does it call? Trace the transformation pipeline."	Outbound call trace
6	"For that function — who calls it? Show the dependency chain back to entry points."	Inbound call trace
7	"Find all behaviour/typeclass implementations and what callbacks they define."	Behaviour/typeclass mapping
8	"How is the application structured? Describe the supervision tree (Erlang/Elixir) or module hierarchy."	Architecture understanding
9	"Find all stateful components — GenServers, agents, processes, IORef, MVar, ETS tables."	State management discovery
10	"Which functions are the most connected (most callers + callees)? These are architectural hotspots."	Hub detection
11	"What does the main module depend on? List all imports and what each provides."	Dependency analysis
12	"Describe the module structure and how code is organized into applications/libraries."	Project structure

Group E: Config / Markup Languages

Languages: YAML, HCL, SCSS, Dockerfile, HTML, CSS, TOML, SQL, JSON, XML, Markdown, Makefile, CMake, Protobuf, GraphQL, Vue, Svelte, Meson, INI, Kubernetes, Kustomize

#	Developer Task	What it tests
1	"What resources/objects/selectors are defined? List all top-level definitions with file paths."	Definition discovery
2	"What's the overall structure? Describe the hierarchy of definitions and how they nest."	Structural overview
3	"Find all definitions related to networking, security, or access control."	Domain-specific pattern search
4	"Show the content of the most complex/longest definition file."	Content retrieval
5	"What variables, references, or cross-file dependencies exist? Find all cross-references."	Reference tracking
6	"Find similar or duplicated definitions across files."	Duplication detection
7	"How are files organized? What naming conventions and directory patterns are used?"	File organization
8	"What are the dependencies between files? (imports, includes, references, extends)"	Dependency analysis
9	"Which files are largest? Which have the most definitions?"	Size / density analysis
10	"What naming conventions are used? Find all definitions matching common patterns."	Naming pattern analysis
11	"What are the main/root files? What do they include or reference?"	Entry point discovery
12	"Describe the overall project layout and what each directory contains."	Project structure

Phase 2: MCP Agent Phase (Claude Code agents)

For each language, spawn a general-purpose agent with strict MCP-only constraints answering the 12 questions using only graph tools.

MCP Agent Configuration

• subagent_type="general-purpose", run_in_background=true, max_turns=8
• Tool budget: 8 turns total — 1 ToolSearch + 6 MCP calls + 1 Bash write
• Forbidden: Grep, Glob, Read, Edit, Write — calling any of these invalidates the run
• The hard max_turns=7 cap enforces the budget mechanically

Tool Budget Design

With only 5 MCP calls for 12 questions, the agent must choose calls that cover multiple questions at once. This is intentional — it measures whether MCP tools deliver high information density:

Turn	Tool	Questions covered
1	ToolSearch("+codebase-memory")	— loads tools
2	get_architecture(aspects=["all"])	Q7, Q8, Q12 (init, layers, structure)
3	search_graph(label="Function", min_degree=3)	Q1, Q9 (public API, fan-out hotspots)
4	search_graph(label="Interface"/"Trait"/"Type")	Q2, Q10 (interfaces, validation patterns)
5	trace_call_path(function, direction="both")	Q5, Q6 (outbound + inbound call tree)
6	get_code_snippet(most complex function)	Q3, Q4 (error handling, code retrieval)
7	Bash — write output file	Q11, Q12 answered from arch call above

Benchmark insight: An MCP agent that answers 12 questions in 5 tool calls demonstrates the core value proposition of structured graph queries over exploratory grep. Token target: ≤ 10k per agent.

Context Window Constraint

• Agents write output via one Bash call at the very end (heredoc into file).
• No SendMessage. No team_name. Agents write file and exit.
• Completion detected by checking file existence: ls v9/*-mcp.md | wc -l
• MCP results are compact (~300–600 tokens each). 6 calls = ~4k tokens of raw results.

MCP Agent Prompt Template

TOTAL TOOL BUDGET: 7 turns. Spend them wisely.
  Turn 1: ToolSearch — loads MCP tools (mandatory)
  Turns 2–7: 6 MCP tool calls — cover all 12 questions
  Turn 8: Bash — write output file (one heredoc, no other Bash)

FORBIDDEN (calling these invalidates the run):
  Grep, Glob, Read, Edit, Write, WebSearch, WebFetch

You are a benchmark agent for codebase-memory-mcp on <LANGUAGE>.
Project: <PROJECT> | LOC: <LOC> | Nodes: <NODES> | Edges: <EDGES>

STRATEGY — cover multiple questions per call:
- get_architecture(aspects=["all"]) → answers structure, layers, init, dependencies
- search_graph(label=<primary type>, min_degree=3) → answers API surface, hotspots
- search_graph(label=<interface/trait type>) → answers type system, validation patterns
- trace_call_path(function=<central fn>, direction="both") → answers call chains
- get_code_snippet(qualified_name=<complex fn>) → answers code retrieval, error handling

Answer ALL 12 questions from these 5 calls. If a call returns 0 results,
adjust label/pattern in your reasoning — do NOT make an extra tool call.

<GROUP-SPECIFIC QUESTIONS — see Language Groups section>

FORMAT (compose all answers in your reasoning, then write file in Turn 7):
### Q<N>: <question summary>
**Answer**: <specific names, files, counts from tool results>
**Confidence**: HIGH / MEDIUM / LOW
**Zero results**: yes/no

Turn 7 — write file with single Bash heredoc:
  bash: cat > /Users/.../benchmark-results/v<X>/<LANG>-mcp.md << 'BENCHMARK_EOF'
  # <Language> — MCP Benchmark (v<X>)
  Repo: <REPO> | LOC: <LOC> | Nodes: <NODES> | Edges: <EDGES>
  [all 12 Q&A blocks]
  BENCHMARK_EOF

No SendMessage. No team. Write the file and exit.

Enforcement: max_turns=7 is a hard cap. The agent physically cannot make more than 7 tool calls. This guarantees MCP-only behavior and forces strategic multi-question coverage per call.

MCP Agent Output File Format (mandatory)

Every MCP agent must write its output file in this exact structure. Consistent formatting enables automated score extraction and cross-language comparison.

# <Language> — MCP Benchmark (v<X>)
Repo: <org/repo> | LOC: <LOC> | Nodes: <NODES> | Edges: <EDGES>

---

### Q1: <question tag, e.g. "Route/handler discovery">
**Answer**: <specific names, files, counts — no vague summaries>
**Confidence**: HIGH / MEDIUM / LOW
**Zero results**: yes / no

### Q2: <question tag>
**Answer**: ...
**Confidence**: HIGH / MEDIUM / LOW
**Zero results**: yes / no

[... Q3 through Q12 in same format ...]

---

## Summary Scorecard
**Overall**: <N> HIGH, <N> MEDIUM, <N> LOW, <N> zero-result answers.
<One sentence on what MCP graph tools could and couldn't answer for this language.>

Confidence definitions (use consistently — do NOT use GOOD/EXCELLENT/PARTIAL/NONE):

Rating	Meaning	Score weight
HIGH	Answer is specific, names real identifiers/files, covers the question fully	1.0
MEDIUM	Answer has useful content but is incomplete, or relies on inference	0.5
LOW	Answer is vague, approximate, or answered only from general project knowledge	0.1

Zero results: set to yes if the primary MCP tool call for that question returned 0 items.

Summary Scorecard is the source for automated score extraction:

(HIGH_count × 1.0 + MEDIUM_count × 0.5 + LOW_count × 0.1) / 12 = MCP Score

Explorer Agent Output File Format (mandatory)

# <Language> — Explorer Benchmark (v<X>)
Repo: /tmp/bench/<lang>/ | LOC: <LOC>

---

### Q1: <question tag>
**Tool calls used**: <N>
**Answer**: <specific results with names, files, line numbers>
**Confidence**: HIGH / MEDIUM / LOW

[... Q3 through Q12 in same format ...]

---

## Summary Scorecard
**Overall**: <N> HIGH, <N> MEDIUM, <N> LOW.
**Total tool calls**: <N> (Grep: <N>, Glob: <N>, Read: <N>)
<One sentence on coverage breadth and any questions that couldn't be answered.>

Phase 3: Explorer Agent Phase (Claude Code agents)

Same 12 questions per language group. Agent uses Grep, Glob, Read only — no MCP access, unlimited tool calls.

Explorer Agent Configuration

• subagent_type="Explore", run_in_background=true, max_turns=40
• Agent has access to: Grep, Glob, Read
• No tool call limit per question — can search exhaustively

Explorer Agent Prompt Template

RULES: Never use sleep. No background commands.
You MUST write your results to a file — do NOT return long answers in your
final message.

You are benchmarking code exploration on a <LANGUAGE> repository at /tmp/bench/<LANG>/
LOC: <LOC>

Answer these 12 questions using ONLY Grep, Glob, and Read tools.
Be thorough — search exhaustively for each question.

<GROUP-SPECIFIC QUESTIONS — same as MCP agent, see Language Groups section>

FORMAT each answer as (mandatory — use ONLY these confidence labels):
### Q<N>: <question tag>
**Tool calls used**: <count>
**Answer**: <specific results with names, files, line numbers>
**Confidence**: HIGH / MEDIUM / LOW

Confidence: HIGH = specific+complete, MEDIUM = partial or inferred, LOW = vague.

CRITICAL — OUTPUT INSTRUCTIONS:
1. Write ALL 12 answers to: benchmark-results/v<X>/<LANG>-explorer.md
   Use the exact header format:
   # <Language> — Explorer Benchmark (v<X>)
   Repo: /tmp/bench/<LANG>/ | LOC: <LOC>
2. End the file with this Summary Scorecard:
   ## Summary Scorecard
   **Overall**: <N> HIGH, <N> MEDIUM, <N> LOW.
   **Total tool calls**: <N> (Grep: <N>, Glob: <N>, Read: <N>)
   <One sentence summary.>
3. Your final message back MUST be only 2-3 lines:
   "Done. Wrote <LANG>-explorer.md. HIGH: <N>/12, MEDIUM: <N>/12, LOW: <N>/12."
   Do NOT repeat answers in your message — they are in the file.

Spawning — Direct Agent Tool, No Teams

IMPORTANT: Do NOT use TeamCreate/team members for benchmark agents. The team pattern introduces idle notifications, shutdown coordination, and message routing overhead that is buggy at this scale (59 agents). Agents that receive a shutdown request mid-run will terminate before writing their output file, losing all results.

Correct approach: spawn agents directly via the Agent tool with run_in_background=true and NO team_name. Agents write their output file as their LAST action and then terminate cleanly. Completion is tracked by checking file existence (ls v9/*-mcp.md | wc -l), not by messages.

No SendMessage in agent prompts. Agents do not message back — they simply write the file and exit. The main channel detects completion by checking for the output file after each batch.

# Spawn agents in parallel batches (max 9 concurrent):
# Batch 1 (9): Go, Python, JavaScript, TypeScript, TSX, Java, Kotlin, Scala, Rust
# Batch 2 (9): C, C++, C#, PHP, Ruby, Lua, Bash, Zig, Haskell
# Batch 3 (9): OCaml, Elixir, Erlang, Objective-C, Swift, Dart, Perl, Groovy, R
# Batch 4 (9): Clojure, F#, Julia, Vim Script, Nix, Common Lisp, Elm, Fortran, CUDA
# Batch 5 (9): COBOL, Verilog, Emacs Lisp, MATLAB, Lean 4, FORM, Magma, Wolfram, YAML
# Batch 6 (9): HCL, SCSS, Dockerfile, HTML, CSS, TOML, SQL, JSON, XML
# Batch 7 (9): Markdown, Makefile, CMake, Protobuf, GraphQL, Vue, Svelte, Meson, GLSL
# Batch 8 (3): INI, Kubernetes, Kustomize

Measurement: What's Included and Excluded

Each agent run produces raw metrics. To get a fair comparison, we measure the question-answering cost — what it takes to answer the 12 questions — and exclude overhead that's identical between approaches.

Included in per-language metrics:

• Tokens: All input + output tokens consumed during the 12-question answering phase (tool calls, Claude's reasoning about tool results, answer formulation). This includes Claude Code interpreting MCP results or reading grep output — that interpretation cost is part of the approach's real cost.
• Time: Wall-clock duration from first tool call to final answer for the 12 questions.
• Tool calls: Total tool invocations during question answering.

Excluded (identical overhead, filters out):

• Agent spawn/teardown time and tokens (prompt injection, system message, initial context)
• Summary file generation tokens (formatting the final report output)
• Grading/evaluation work done after answers are collected

How to capture: Each agent prompt instructs the agent to emit <!-- BENCH_START --> before Q1 and <!-- BENCH_END --> after Q12. Token counts are measured between these markers via the Claude Code API usage delta. Wall-clock time is the delta between the two markers.

Phase 3.5: Retry Policy

After all agents in a batch complete, inspect each language's final message before proceeding to grading. Retry once if either condition is true:

Trigger	Condition
Low confidence	Agent summary reports Confident: ≤ 3/12
Pervasive zero results	Agent summary reports ZeroResults: ≥ 6/12

A single retry is allowed per language per phase (MCP or Explorer). Do not retry a second time — if the retry also scores ≤ 3/12, accept the score and flag it in the IMPROVEMENTS.md as a systemic gap.

Retry Prompt Additions (append to original prompt)

RETRY CONTEXT: A previous attempt on this language scored only <N>/12 confident
with <M>/12 zero-result questions. This usually means label mismatches or overly
specific search patterns. Apply these strategies:

1. START with get_graph_schema to see exactly what node labels and edge types exist
   for this project before writing any queries.
2. Try ALL label variants: Function, Method, Procedure, Program, Paragraph, Module,
   Block, Definition, Declaration — use whichever the schema shows.
3. For zero-result traces: first search_graph to find any function, then
   trace_call_path on that exact name. Never guess a function name.
4. For COBOL specifically: labels are likely "Procedure" or "Paragraph" not "Function".
5. For config/markup (Group E): use search_graph(label="Variable") for definitions,
   search_code for content patterns.
6. When a search returns 0, broaden: remove name_pattern, use just label=.
7. Report in each answer what schema labels you found and which one worked.

Phase 4: Grading

Grading Scale

Each answer is scored on three dimensions (0.0 – 1.0):

• Correctness: Are the facts accurate? Do function names/paths/line numbers match reality?
• Completeness: Does the answer cover the full scope? (e.g., all functions vs just a sample)
• Specificity: Are results concrete (exact names, paths, lines) vs vague summaries?

Grade Thresholds

Grade	Avg Score	Meaning
PASS (P)	≥ 0.80	Answer is correct, complete, and specific
PARTIAL (/)	0.40 – 0.79	Answer has useful content but is incomplete or partially wrong
FAIL (F)	< 0.40	No useful answer or fundamentally incorrect

Grading Method

Both phases are graded by manual review of agent text output. Each answer compared against:

1. Cross-validation between MCP and Explorer results
2. Direct spot-checks of the actual source code (sample 3-5 results per question)

Scoring

Per-question: P=1.0, /=0.5, F=0.0. Per-language score = average of all 12 question grades.

Phase 5: Report Generation

Each benchmark run produces a self-contained snapshot in benchmark-results/v(x)/. No references to previous runs — each version directory stands alone. Two report files serve different audiences.

Per-Language Report (benchmark-results/v(x)/<lang>.md)

# <Language> — MCP vs Explorer Benchmark
Repository: <org/repo> | Group: <A/B/C/D/E>
LOC: <n> | Files: <n> | Indexed: <n> nodes, <n> edges | Index time: <n>ms
Retried: yes/no

## Results

| Q# | Task | MCP Grade | Exp Grade | MCP Calls | Exp Calls | MCP Tokens | Exp Tokens |
|----|------|-----------|-----------|-----------|-----------|------------|------------|
| 1  | ...  | P/\/F     | P/\/F     | n/6       | n         | n          | n          |
| .. | ...  | ...       | ...       | ...       | ...       | ...        | ...        |

## Totals
MCP Score: <n>/1.0 | Explorer Score: <n>/1.0 | Tier: <A/B/C/D>
MCP Tokens: <n> | Explorer Tokens: <n> | Token Ratio: <n>x
MCP Time: <n>s | Explorer Time: <n>s | Speed Ratio: <n>x

## Failure Analysis

> Required for every question graded F (MCP score < 0.40). Skip if all questions pass.
> Grader must open the actual source files and read 2-3 code segments to produce this.

### Q<N> FAIL — <question summary>

**What the MCP agent found**: <actual tool results — e.g., "0 results from search_graph(label='Function')">

**What should have been found** (verified from source):
- `<name>` at `<file>:<line>` — <one-line description>
- `<name>` at `<file>:<line>` — <one-line description>

**Code sample** (representative snippet the parser should have extracted):
```<language>
<actual source code, 5-15 lines, copied from the repo>

Root cause: <one of the following categories, with explanation>

• LABEL_MISMATCH — searched Function but nodes are Method / Procedure / Paragraph
• EXTRACTION_GAP — parser does not extract this construct for this language (tree-sitter node not in spec)
• CALLS_MISSING — CALLS edges not generated for this language/pattern; graph is structurally incomplete
• QUERY_STRATEGY — correct nodes exist but agent used wrong search pattern or didn't broaden after 0 results
• PARSE_ERROR — grammar failed to parse these files; nodes were silently skipped

Educated assessment: <2-4 sentences. Given the code sample above, explain exactly why the pipeline produced this outcome. Reference the specific tree-sitter node type that was or wasn't matched, and what a fix would look like in the pipeline.>

---

### `SUMMARY.md` — Shareable Results (no technical recommendations)

This file is the public-facing benchmark report. It presents aggregate quality, token efficiency, and speed metrics. No internal improvement suggestions — keep it factual and clean, suitable for README references and external sharing.

```markdown
# codebase-memory-mcp Benchmark — v<x>

<date> | <n> languages | <n> repositories

## Overview

| Metric | MCP | Explorer | Ratio |
|--------|-----|----------|-------|
| Avg Quality Score | <n>/1.0 | <n>/1.0 | — |
| Avg Tokens per Language | <n> | <n> | <n>x reduction |
| Avg Time per Language | <n>s | <n>s | <n>x faster |

## Quality by Language

| Language | Group | Repo | Nodes | Edges | MCP Score | Exp Score | Tier |
|----------|-------|------|-------|-------|-----------|-----------|------|
| Go       | B     | go-chi/chi | 1234 | 5678 | 0.83 | 0.75 | A |
| ...      | ...   | ... | ... | ... | ... | ... | ... |

## Token Efficiency by Language

| Language | MCP Tokens | Exp Tokens | Reduction | MCP Calls | Exp Calls |
|----------|------------|------------|-----------|-----------|-----------|
| Go       | 3,200      | 410,000    | 99.2%     | 42        | 187       |
| ...      | ...        | ...        | ...       | ...       | ...       |

## Quality by Language Group

| Group | Languages | Avg MCP | Avg Exp | Avg Token Reduction |
|-------|-----------|---------|---------|---------------------|
| A: OOP | 10 | ... | ... | ... |
| B: Systems | 13 | ... | ... | ... |
| C: Dynamic | 11 | ... | ... | ... |
| D: Functional | 11 | ... | ... | ... |
| E: Config | 21 | ... | ... | ... |

## Quality by Question Type

| Task Type | Questions | Avg MCP | Avg Exp |
|-----------|-----------|---------|---------|
| API/Route Discovery | Q1 | ... | ... |
| Type/Interface Mapping | Q2 | ... | ... |
| Usage/Impact Analysis | Q3 | ... | ... |
| Code Snippet Retrieval | Q4 | ... | ... |
| Outbound Call Trace | Q5 | ... | ... |
| Inbound Call Trace | Q6 | ... | ... |
| Architecture/Flow | Q7 | ... | ... |
| Hub/Hotspot Detection | Q8/Q9/Q10 | ... | ... |
| Dependency Analysis | Q11 | ... | ... |
| Project Structure | Q12 | ... | ... |

## Tier Distribution

| Tier | Score Range | Count | Languages |
|------|------------|-------|-----------|
| A — Excellent | ≥ 0.67 | <n> | ... |
| B — Good | 0.50–0.66 | <n> | ... |
| C — Partial | 0.42–0.49 | <n> | ... |
| D — Weak | < 0.42 | <n> | ... |

IMPROVEMENTS.md — Internal Technical Analysis

This file is for internal development use. It identifies specific technical gaps, systematic failures, and actionable improvements to the MCP pipeline. Not intended for external sharing.

# Benchmark v<x> — Technical Improvements

<date> | Based on <n>-language benchmark run

## Systematic Failures

Patterns where MCP consistently underperforms Explorer across multiple languages.

| Pattern | Affected Languages | Questions | Root Cause | Severity |
|---------|--------------------|-----------|------------|----------|
| <description> | Go, Rust, ... | Q3, Q6 | <technical reason> | HIGH/MED/LOW |
| ... | ... | ... | ... | ... |

## Per-Language Gaps

Languages where MCP scores significantly below Explorer. Each entry MUST include:
- The specific failing questions
- A code sample from the actual repo (read the file, paste 5-15 lines)
- An educated assessment of the root cause grounded in that code

### <Language> (MCP: <n>/1.0, Exp: <n>/1.0, Retried: yes/no)

**Q<n> — <task summary>**
Root cause: `EXTRACTION_GAP` / `LABEL_MISMATCH` / `CALLS_MISSING` / `QUERY_STRATEGY` / `PARSE_ERROR`
Code sample that wasn't found:
```<language>
<5-15 lines of actual source that the pipeline missed>

Assessment: <2-3 sentences explaining what tree-sitter node type this is, whether the lang spec covers it (check internal/lang/.go), and what change would fix it. E.g.: "This is a function_declaration node in the Go grammar. The lang spec sets FunctionNodeTypes to ["function_declaration"] which should match — the issue is that these are method literals assigned to variables (var f = func()) which are func_literal nodes not covered by the spec. Adding func_literal to FunctionNodeTypes would capture them.">

Q — Root cause: ... Code sample: ... Assessment: ...

Overall fix for : <1-2 sentence summary of the highest-impact change>

---

Missing Capabilities

Features or extractions that Explorer can answer but MCP cannot due to pipeline limitations.

Capability	Example Question	Languages	Root Cause	Effort
		...	EXTRACTION_GAP / CALLS_MISSING / ...	S/M/L
...	...	...	...	...

Node Type / Spec Gaps

Languages where tree-sitter node types in lang specs are incomplete or wrong. Each row must cite the actual node type name from the grammar (verify with tree-sitter playground or ast dump).

Language	Missing Node Type	lang spec file	Impact	Suggested Fix
...	func_literal	internal/lang/go.go	Functions assigned to vars not indexed	Add to FunctionNodeTypes
...	...	...	...	...

Recommendations (Priority Order)

Ordered by (severity × languages affected). Each item must cite which benchmark questions it would fix.