The legacy "Deploy from a branch" Pages build ran Jekyll on every push to
main and failed on the planning docs under /docs — Jekyll's Liquid parser
hit `Unknown tag 'data'` in EVALUATION_PLAN.md, so the website build went
red after nearly every commit.
/docs is a hand-written static site (index.html + robots/sitemap/llms.txt),
so it never needed Jekyll. Deploy it with a GitHub Actions workflow that
uploads /docs as-is and only triggers on docs/** changes, and add a
.nojekyll guard. Switch the Pages source to "GitHub Actions" for this to
take over from the legacy build.
Co-authored-by: DeusData <joerge@datadice.io>
Address peer-review feedback on question provenance/external validity.
- Anchor the five dimensions to Sillito, Murphy & De Volder's catalogue of
questions programmers ask during software-evolution tasks (FSE'06/TSE'08),
with LaToza & Myers (2010) and Ko et al. (ICSE'07) as support; add a
D1-D5 -> Sillito-group mapping table
- Adopt published repo-level QA benchmarks for the major languages: reuse and
compare against SWE-QA (primary), with CoReQA, CodeRepoQA and RepoQA as
references; note snippet-level sets (CodeQA/CS1QA/CoSQA) as the contrast
that motivates structural retrieval
- Policy: anchor to Sillito; reuse SWE-QA for major languages; generate the
multilingual remainder from independent LSP/git ground truth, never from
the model under test
Rename TEST_PLAN.md to EVALUATION_PLAN.md and rebuild it as a peer-review
evaluation specification covering all 159 supported languages.
- Graph (MCP) vs Explorer (grep/glob/read) per language, 5 bespoke questions
mapped to 5 universal dimensions (D1-D5) for cross-language aggregation
- Sequential, main-channel execution: one language at a time, Explorer the
only subagent, per-language index lifecycle (cold index -> eval -> delete)
to avoid concurrent-indexing memory blowups; index time is a key metric
- LLM-as-a-Judge grading (single disclosed model, blind A/B, median of passes)
- Per-language node-type and edge-type histograms with every one of the 32
edge types listed, zeros kept (a zero is a real extraction-gap signal)
- Cross-repo intelligence + semantic/similarity deep-dive for the 9 LSP
languages, with recall/precision ground truth
- Single result tree (eval-results/), retiring the versioned-dir scheme
- Master repo assignment for all 159 languages; per-language chapters in
Appendix A
Surface previously undocumented capabilities on the site, README, and
package metadata: semantic vector search (bundled nomic-embed-code
embeddings, fully local), SEMANTICALLY_RELATED / SIMILAR_TO clone
detection, cross-repo intelligence, data-flow tracing, change-impact
analysis, and ADR management. Broaden the cross-service entry to cover
gRPC/GraphQL/tRPC and pub/sub channels. Add a "star on GitHub" nav CTA.
GEO/SEO: add a question-style semantic-search section, extend the
JSON-LD featureList and FAQPage, add semantic keywords, and refresh the
sitemap lastmod.
Correct the language count to 159 (was 155/158) everywhere while keeping
the 157 vendored-grammar figure, update the Linux-kernel index stats to
the measured 4.81M nodes / 7.72M edges, and make trace_path the canonical
tool name (trace_call_path noted as an alias).
Resolved 5 conflicts where the php-lsp branch and main both added LSP
infrastructure but with different APIs:
- Makefile.cbm: kept both TEST_PHP_LSP_SRCS and main's py_lsp/scope/
type_rep/stack_overflow test source variables in ALL_TEST_SRCS.
- internal/cbm/cbm.c: kept both #include lsp/php_lsp.h and lsp/py_lsp.h;
the per-language dispatch already supports stacking PHP after PYTHON.
- internal/cbm/lsp_all.c: kept both lsp/php_lsp.c + php_stdlib_data.c
AND lsp/py_lsp.c + python_stdlib_data.c in the compilation unit.
- src/pipeline/pass_calls.c: took main's version entirely.
find_lsp_resolution + resolve_single_call(ctx, call, lsp_calls, ...)
is language-agnostic — it matches any CBMResolvedCall whose caller_qn
hits the textual call's enclosing function and whose callee_qn
short-name hits the textual callee. PHP LSP populates the same
result->resolved_calls array via cbm_run_php_lsp, so the unified
resolver handles PHP for free. Dropped the php-branch's
qn_short_name + lsp_override_resolution helpers as superseded.
Locked confidence floor at main's CBM_LSP_CONFIDENCE_FLOOR = 0.6f
(was 0.5f on the php branch).
- tests/test_main.c: kept all suite externs and RUN_SUITE invocations
for php_lsp + py_lsp/py_lsp_bench/py_lsp_stress/py_lsp_scale +
scope + type_rep.
Auto-merged: src/pipeline/pass_parallel.c.
Brings in 52 commits from main since the branch was forked at c768b61
(#207 watcher follow-up): #208 through #281, dependabot bumps, Pine
Script, AUR docs, INHERITS edges, get_architecture wiring, codeql
revert, the entire python-lsp subsystem.
Suite: 3249 passed, 0 failed (was 2971 on main; +278 new tests from
php-lsp matches the branch's claimed test count).
BENCHMARK_PYTHON.md updated through Round 10. The stress-test suite
went from 22 patterns (19 hard / 3 KNOWN GAP) to 43 patterns (all
hard-asserted PASS). Bench held at 100% (52/52) across the surface
expansion.
Patterns added in Round 10 cover the framework / real-world cases
that distinguish parity-with-LSP from passing-toy-fixtures:
SQLAlchemy 2.0 Mapped[T], Pydantic BaseModel, async with / async
for, Callable[..., R], next(iter), **kwargs annotation, dict-literal
dispatch tables, lambda call-site inference, recursive types,
classmethod-to-method chains, dataclass default_factory, nested
match patterns.
The original Phase 11 plan called this category "compiler-grade".
The actual line where compiler-rebuild starts is now narrower:
constraint solving for ParamSpec, code-flow narrowing across
basic blocks, custom metaclass __call__, and full descriptor
protocol. None of those are needed for what's resolved on the
bench fixture or the stress suite.
BENCHMARK_PYTHON.md ends the parity push with the in-process bench
results table (81% -> 79% -> 95% across rounds), an explicit
stopping-point list of features that would require building a
constraint solver / code-flow engine / plugin system, and an
"achievable next steps" list for future iterations that stay below
the compiler-rebuild line.
The 81% -> 79% jump going from a 65-line fixture to 136 lines is
deliberate: the smaller fixture was biased toward the resolver's
strengths. The 79% -> 95% jump comes from instance-attribute typing
(Round 4) which closed the largest single gap on real-world code
shapes.
Total: 11 LSP commits on this branch (Phases 0-11 plus Rounds 1-4),
2892 / 0 passing, 95% resolution on the parity fixture.
Edge counts (laravel/framework):
- Total edges: 196,979 (baseline) -> 152,614 (-22%)
- CALLS edges: ~83,000 (baseline) -> 50,150 (-40%)
The drop is from suppressing name-fallback misroutes when the receiver
is statically typed to a vendor class that isn't indexed. The Phase-4
capabilities (generic templates, narrowing, property tracking, full
parent-walk method dispatch, expanded stdlib) cover most of phpactor's
type-resolution surface for in-project receivers.
100 unit tests, all passing. Total project tests: 2913 / 0 failed.
Final phase of Python LSP integration. Adds:
- docs/BENCHMARK_PYTHON.md — methodology document covering target
metrics (per-file LSP overhead < 10 ms, resolution ratio ≥ 40% on
application code, ≥ 70% with stdlib, < 5% spot-check false positive
rate), the declared accepted-failure-class ceiling (metaclass
__call__, __getattr__/__getattribute__, dynamic setattr, runtime
importlib, monkey-patching, eval/exec, descriptors beyond the
standard property/classmethod/staticmethod, dynamic type() classes),
and instructions for running scripts/benchmark-index.sh against the
largest Python repos in datadice (falkemedia: 232 files, api-hub:
188 files).
- tests/test_py_lsp_bench.c — in-process benchmark wired into
scripts/test.sh. Runs cbm_extract_file on a 65-line Python fixture
covering imports, dataclasses, classmethods, single-inheritance with
super(), typed parameters, attribute chains, and stdlib calls.
Asserts soft floor (resolved >= calls/4) and hard time budget
(< 50 ms under sanitizers). Initial run reports:
bench: 65 lines, 16 calls, 13 resolved (81%), 2.45 ms
81% beats the 40% application-code target and the 70%
application-plus-stdlib target. 2.45 ms under ASan + UBSan is well
inside the 10 ms (no-sanitizer) production budget.
This wraps the 11-phase plan documented in PYTHON_LSP_PLAN.md. Total
test count: 2874 / 0 (100% pass). Repository-level benchmarks via
scripts/benchmark-index.sh remain a manual operation against a fresh
production binary; methodology is now in place for when that runs.
PHP score moves from 9/12 (75%) to 10/12 (83%); the two remaining
PARTIALs (Q6 search recall, Q8 trace tool disambiguation) are non-LSP
issues tracked separately per PHP_LSP_PRE_FLIGHT.md §3 and §4.1.
Headline change is a graph-correctness win on laravel/framework: the
collide-set $prompt->value() misroute is gone, and unindexed-vendor
receiver calls no longer produce wrong name-fallback edges (total
edges drop ~14% from 196,979 to 169,268, trading recall for precision).
Pre-flight against laravel/framework establishes that the PHP graph has
systemic call-attribution drift on short names shared between a global
function and a method (e.g. value(), make(), get()). Spot-check shows
~20-30% of typed-receiver method calls misroute to the global helper
because the existing extractor cannot resolve receiver type.
Plan adds internal/cbm/lsp/php_lsp.{c,h} as a third sibling to go_lsp.c
and c_lsp.c. In-process C type resolver, no PHP runtime, no new
dependencies. Uses already-vendored tree-sitter-php grammar and the
existing CBMArena / CBMScope / CBMType / CBMTypeRegistry primitives.
Phase 1 success metric is collide-set attribution correctness, not
benchmark tier alone. Q6 (text search) and Q8 (trace tool
disambiguation) tracked as separate tickets.
- Delete cmd/codebase-memory-mcp/assets/ — skills and codex instructions
are embedded as C string literals in src/cli/cli.c since v0.5.0
- Move BENCHMARK.md to docs/ to reduce root clutter
- Fix setup scripts: replace stale go build with scripts/build.sh
- THIRD_PARTY.md: replace Go module dependencies with vendored C
libraries (sqlite3, mimalloc, mongoose, yyjson, xxhash, tre),
fix CGo reference to static compilation
The snippet handler now uses exact QN + suffix matching only (no fuzzy/auto-resolve).
Updated 3 tests to match: fuzzy_suggestions, fuzzy_last_segment, auto_resolve_enabled.
- Add field('path', ...) to Magma load_statement grammar rule so
parse_generic_imports() finds the import path via field lookup
instead of the broken text fallback (which only extracted 1 per file)
- Fix passImports() to resolve file-path imports (e.g. "utils.mag",
"lib/helpers.h") via fqn.ModuleQN() when raw path doesn't match
any node QN — general fix benefiting any file-path-based import
- Add TestMagmaImport_Regression and TestMagmaCall_Regression
- Update language count 59 → 63 in README, docs/index.html, marketing
- Languages: 35 -> 59
- Add Gemini CLI, VS Code, Zed to supported clients list
- Update benchmark description to match v7 methodology
- Update meta tags and comparison table
DeusData