A byte-identity audit of all vendored license files against their
upstream repositories found 113 copies differing only by a missing
trailing newline (vendoring artifact) — replaced with the exact
upstream bytes. Five needed real corrections: fennel carried an
unfilled MIT template although its upstream is CC0-1.0 at the pinned
commit; ron now carries the repo's LICENSE-MIT (dual MIT/Apache
upstream); python, wit, and verilog were refreshed to current upstream
bytes. The manifest's first-party table was also corrected: six of the
twelve grammars are self-maintained forks whose retained upstream
licenses are now byte-verified (assembly's upstream has been deleted
from GitHub; the retained MIT copy is the surviving grant). The audit
tool is kept as scripts/audit-license-provenance.py for future sweeps.
Removes the heaviest vendored grammar (66 MB) along with its language
wiring and test fixtures; the supported-language count moves to 158
across README, site, and manifest. The grammar manifest additionally
records the canonical-source decisions for the five
registry-disagreement grammars and the per-directory license files
restored earlier this week.
The alloc.h/array.h/parser.h headers in
internal/cbm/vendored/common/tree_sitter/ are part of the tree-sitter C
runtime (MIT, (c) 2018 Max Brunsfeld) but had no LICENSE file alongside
them. Copy the upstream MIT license into that directory and clarify in
THIRD_PARTY.md the split between the tree-sitter-html scanner helpers
(common/, (c) 2014) and the core runtime headers (common/tree_sitter/,
(c) 2018).
Restore upstream LICENSE files for every vendored grammar and C library
across both vendored trees (19 grammar dirs, lz4, zstd, simplecpp,
verstable, wyhash, ts_runtime, common, mongoose, xxhash, yyjson; SQLite
public-domain notice). Add the nomic-embed-code Apache-2.0 license plus a
NOTICE describing the embedding derivation. First-party grammars carry
the project MIT license.
THIRD_PARTY.md now defers grammar provenance to the verified
MANIFEST.md instead of a stale hand-written table (which misattributed
clojure as EPL-1.0 — it is CC0-1.0), covers all vendored libraries, and
documents the Hybrid LSP reference servers and stdlib type-data
provenance (typeshed, Go stdlib introspection, hand-curated specs).
Profiling symfony (416 s wall, one core pinned) convicted cbm_registry_lookup_type -> strcmp at 95% of samples: the PHP and Python def-registration loops probe the registry per Method (receiver-stub check) BEFORE the hash indexes exist, i.e. a linear scan over the growing type table - O(methods x types) per file, and the cross resolvers run per file. Java and Kotlin never called cbm_registry_finalize at all, leaving every lookup during their file walks linear over the whole cross registry. Fixes: (1) two-phase registration in php/py (types, finalize, then funcs/methods); (2) finalize added to the java/kotlin cross entry points; (3) lookups now tail-scan entries added after finalize - previously such entries were silently INVISIBLE to the hashed path (reproduce-first: tslsp_hash_registry_post_finalize_adds was red); (4) cbm_registry_finalize_into allocates the index from a per-call scratch arena - putting buckets into the pipeline-lifetime result arena per file accumulated +1.1 GB RSS on the FastAPI incremental test (incr_full_index caught it), and the py tier-2 builder (one def per call) skips mid-build finalize entirely. symfony: 416 s -> 12 s, identical node count; suite 5577/0.
The hand-rolled dump writer stored index cells fully inline regardless of payload size. SQLite's format spills index payloads above the max-local threshold (16,422 bytes at the 64 KB page size) to overflow pages, so a long key written inline makes the reader interpret key bytes as an overflow page number: PRAGMA integrity_check reported 'invalid page number 0x43654C6C' (ASCII from the key text) in idx_nodes_name on elasticsearch (very long Section names), and name lookups on that index silently returned nothing. write_index_btree now rewrites oversized cells (varint + local prefix + overflow chain via write_overflow_pages) before page building; promoted separators carry their chains with them. Reproduce-first: sw_long_index_keys_overflow (20 KB name; integrity_check failed pre-fix, ok post-fix). Also escapes the remaining raw text interpolations in DATA_FLOWS route props and HANDLES handler QNs (the ~51-edge malformed tail after 6c04ab7).
c_process_function's template-param attach wrote rf->type_param_names on the registry — in the Tier-2 cross-LSP phase that registry is shared READ-ONLY across all resolve workers, so the write raced concurrent readers AND stored a pointer into the writing worker's per-file arena; once that arena was recycled the shared registry held dangling memory, crashing other workers intermittently (bitcoin: SIGSEGV in c_process_function strcmp via cbm_run_c_lsp_cross_with_registry, ~1-in-2 runs). A registry_shared flag now marks the cross context and the attach is skipped there (cross-phase template deduction falls back to positional names — graceful degradation); the single-file extract phase keeps full fidelity. Real-repo-tier reproduction: 10x bitcoin index loop, 0 failures post-fix (pre-fix ~50% crash rate); suite 5575/0.
bind_lambda_args and the method-reference arity path indexed the resolved method signature's NULL-terminated param_types array by the CALL-SITE argument index. A call with more arguments than the resolved overload declares (overload mismatch, varargs) read past the terminator and dereferenced whatever followed in the arena as a CBMType* - SIGSEGV while indexing elasticsearch (same out-of-bounds family as #427 and the bitcoin targs fix). Both sites now bound the index by the walked param count. Reproduce-first: 6-line fixture - run(Runnable) invoked with six lambdas - crashed rc=134 pre-fix; fork+exit-signal regression test lsp_java_lambda_args_exceed_params_no_crash added; suite 5574/0.
The C/C++ LSP's explicit-template-argument call path (template_function, e.g. Using<Fmt>(v)) filled a stack array const CBMType *targs[16] without NULL-terminating it before cbm_type_substitute. With more declared type params than explicit args (bitcoin src/serialize.h: Using's Wrapper<Formatter, T&> with 2 params, 1 arg), the bounded args walk read uninitialized stack; any non-NULL garbage got bound to the type param and woven into the REGISTERED return-type graph, then dereferenced at a later call site - SIGSEGV (crash shape: string bytes misread as a CBMType*). All c_lsp targs arrays are now zero-filled, and cbm_type_substitute's args walk treats implausible values (misaligned / below the first page) as the terminator so contract violations can never leak garbage into a type graph. Reproduce-first: deterministic poison-args unit test (typerep_substitute_rejects_garbage_args_entries) + real-repo verification (bitcoin indexes clean under ASan; was a deterministic SIGSEGV).
Indexing large OSS during the perf sweep crashed three LSP resolvers, all by stack overflow in their recursive resolve walks (macOS crash reports): elasticsearch — SIGSEGV in bind_lambda_args under deep recursive java_resolve_calls_in_node frames; bitcoin — SIGSEGV in cbm_type_substitute via c_adl_resolve under deep c_resolve_calls_in_node frames; microsoft/TypeScript — SIGBUS under an unbounded lookup_member_type cycle (cyclic type graph in reallyLargeFile.ts). The expression-eval guards (eval_depth) did not cover these walks. Each resolver now has a depth-guarded entry (walk_depth cap 512 for the Java/C AST walks, member_depth cap 64 for TS member lookup); past the cap the subtree resolves as unknown — graceful degradation instead of a crash. Reproduce-first: prod CLI probes crashed pre-fix (nested-call Java rc=139, nested-call C++ rc=139); fork+exit-signal regression tests added (TS cyclic shape needs a real cross-file registry, so it is verified at the real-repo tier — microsoft/TypeScript now indexes clean). Side-finding documented in the sweep report: Java BLOCK nesting >=3000 is minutes-slow (separate pathology, not this fix).
C++/CUDA out-of-line method definitions (Foo::bar in a .cpp; the class body lives declaration-only in a header) were recorded as free Functions with no link to their class. A new helper cpp_out_of_line_parent_class() detects a qualified_identifier declarator on a function_definition and resolves the immediate enclosing class (descending nested scopes ns::Foo::bar to the direct parent); extract_func_def then promotes the def to label Method with parent_class set to the class QN so DEFINES_METHOD edges resolve. Distilled (reimplemented) from PR #428. Reproduce-first: cpp_out_of_line_method_issue428 (single-level out-of-line defs; the nested-namespace path isn't isolation-testable because tree-sitter-cpp parses a synthetic doubly-qualified def as an ERROR node, so it is exercised by real codebases).
The C++ LSP passes a class's declared template-param names (e.g. template<class T, class Alloc = ...>) alongside an instance's fewer template args (vector<int>). The substitution loop is bounded by type_params, so a param matching beyond the supplied args indexed type_args[i] past the array — reading adjacent arena memory (a type-name string byte) as a CBMType* that c_eval_expr_type later dereferenced -> SEGV (ASan: stack-buffer-overflow in cbm_type_substitute via c_eval_expr_type). Deterministic, all platforms; distinct from the #424 allocator crash. Fix at the single chokepoint: bound the args length by the (NULL-terminated) param count and gate every type_args[i] access by it, so no caller can over-read regardless of language. Reproduce-first: typerep_substitute_short_args_no_oob_issue427.
Defense-in-depth follow-up to the tree-sitter bind: route sqlite3 and libgit2
allocations through mimalloc explicitly so a correct binary never depends on the
fragile MI_OVERRIDE symbol override (where the Windows static-MinGW link can
resolve a library's malloc to mimalloc but its free to the CRT, corrupting the
heap). Prod-only, guarded by the existing CBM_BIND_TS_ALLOCATOR; the CRT+ASan
test build compiles these as no-ops and stays unchanged.
New cbm_alloc_init() consolidates all three binds behind a static idempotency
guard. main() calls it as its very first statement — before cbm_mcp_server_new
opens the in-memory store — because SQLITE_CONFIG_MALLOC must run before the
first sqlite3_open*/sqlite3_initialize, else sqlite3_config returns SQLITE_MISUSE
silently and the bind is ignored (the return code is asserted). cbm_init() also
calls cbm_alloc_init() so pipeline-pass entry points get the binds too; the
ts_set_allocator bind moved out of cbm_init into cbm_alloc_init.
sqlite3: a static sqlite3_mem_methods backed by mi_malloc/mi_free/mi_realloc,
xSize via mi_usable_size, xRoundup to an 8-byte boundary, xInit returning
SQLITE_OK, xShutdown a no-op (int-sized sqlite signatures wrapped with size_t
casts). libgit2: a static git_allocator (gmalloc/grealloc/gfree → mi_*) installed
via git_libgit2_opts(GIT_OPT_SET_ALLOCATOR). The allocator is set BEFORE any
git_libgit2_init: git_allocator_global_init keeps a custom allocator if one is
already set (git__allocator.gmalloc != git_failalloc_malloc), the pre-init global
is the fail-allocator, and there is no allocator reset on git_libgit2_shutdown —
so binding once at startup survives pass_githistory's per-call init/shutdown
pairs. libgit2 code is additionally guarded by HAVE_LIBGIT2.
Test build 5564/0 ASan-clean; prod build clean with CBM_BIND_TS_ALLOCATOR=1; the
HAVE_LIBGIT2 branch type-checked against the libgit2 sys/alloc.h struct shape;
lint-ci clean.
Root cause (corrected from the report): not a tree-sitter null-subtree sentinel — 0xffffffffffffffff is a freed/heap-corrupted pointer. The vendored ts runtime allocates via its overridable ts_current_malloc/free; under the prod build's MI_OVERRIDE=1, the Windows static-MinGW link (--allow-multiple-definition) resolves ts_malloc to mimalloc but ts_free can resolve to the CRT free (or vice-versa), corrupting the heap freelist and crashing mid-parse on large templated C++ headers (scales with parse churn, not syntax). cbm_init now calls ts_set_allocator(mi_malloc, mi_calloc, mi_realloc, mi_free) so ts allocate+free go through one allocator on every platform. Guarded to CFLAGS_PROD (MI_OVERRIDE=1); the test build stays CRT+ASan to avoid an alloc/free mismatch there. Tests in test_stack_overflow.c: allocator-binding mechanism + large-templated-C++ extraction guard (the C++ gap noted in #424).
Remove leftover getenv(CBM_DBG_*) debug fprintf blocks in extract_calls.c/pass_calls.c (cppcheck always-true). Convert parse_lisp_imports to a stack-based walk (no recursion → no NOLINT, avoids deep-nesting overflow). Apply Homebrew clang-format to all touched sources. Local suite 5562/0, lint-ci clean.
fwcd/tree-sitter-kotlin wraps a whole class in an ERROR node when a member uses companion object / inner class combined with a delegation specifier. recover_kotlin_error_classes scans an ERROR node's flat tokens for class/object keyword + identifier, emits the Class def (respecting enclosing class QN) and collects bases from delegation_specifier siblings. Strictly additive (only fires on ERROR regions). Fixes inherit_kotlin (MyClass companion + Node inner).
pass_semantic + pass_parallel: when a decorator name resolves to no local def, upsert a synthetic '<decorator:NAME>' node (label Decorator) and emit DECORATES+USAGE — fixes rust #[derive]/#[allow], swift @discardableResult, scala @deprecated (deduped per project; pipeline-level so extraction histograms unaffected). py_lsp: inject a synthetic CBMCall for resolved subscript/operator dunders on typed receivers so they reach the CALLS pipeline — fixes python s[0]->Seq.__getitem__.
Imports: per-grammar parsers for hare/pascal/powershell/scheme/racket/starlark/tcl/teal/zsh + css/html/cmake/bitbake/kconfig/gn/just/nix/jsonnet/pkl/nickel/thrift/capnp/d/tablegen/crystal/fsharp/ada; node-extraction for D/PowerShell/Pascal/Julia/F#; INHERITS for those. Pipeline edges: register Variable/Field defs for WRITES/READS resolution; CONFIGURES pass consuming env_accesses; DEPENDS_ON from go.mod/requirements.txt; SQS dot-form QN. Golden updates: julia +Class, just +Function (correct new extraction).
Four overload-scoring lookups (cbm_registry_lookup_method_by_types/_by_args, _symbol_by_types/_by_args) linear-scanned the entire project registry on every call site, ignoring the pre-built method_buckets/func_qn_buckets hash index that the by-name lookups already use. On a large C++ header (e.g. 10989 defs) that is O(calls x project_funcs) per file (~34s/file), so big codebases never finish indexing. Add a hashed fast path (O(overloads)) when the registry is finalized; preserve first-match semantics via lowest registration index (equivalence verified over 1606 lookups, 0 divergence); keep the linear fallback for un-finalized single-file registries so normal files are unaffected. Adds tests/test_cpp_index_hang.sh (scale-tier subprocess+timeout repro, opt-in via CBM_RUN_HANG_TEST).
scan_start_tag_name moves scan_tag_name's owned tag_name into a Tag, but the CF_SET/CF_RETURN/CF_ELSEIF/CF_ELSE cases return without pushing the tag to the scanner stack and (unlike CF_VOID) without freeing it — leaking the tag_name buffer under LeakSanitizer. Free the tag in those return paths, matching CF_VOID. Removes the need for the __lsan_default_suppressions(scan_tag_name) workaround in the test runner.
Both vendored generated parsers declared the entrypoint with an empty parameter list, so calling them through the const TSLanguage *(*)(void) factory pointer is an undefined function-type mismatch (UBSan, clang). Add void to match every other grammar and the extern declaration.
tree_sitter_sql_external_scanner_deserialize set state->start_tag = NULL and re-malloc'd it without freeing the previous allocation. tree-sitter calls deserialize repeatedly during a parse, leaking the prior start_tag each time (LeakSanitizer, Linux x64). Free it first — safe since create() NULL-inits it and destroy() frees+NULLs it.
cbm_kind_in_set builds a thread-local cache of calloc'd symbol bitsets (ks_cache) that was never freed, so LeakSanitizer (Linux x64) reported the per-worker-thread caches as leaks once the previously-empty grammars gained real node-type sets. Add cbm_kind_in_set_free_cache() and call it at worker-thread teardown (extract_worker, beside cbm_slab_destroy_thread) and main-thread exit (test runner, beside sqlite3_shutdown).
Seven REG_TYPE registrations in the curated Java stdlib (GregorianCalendar, FileNotFoundException, Closeable, UnaryOperator, BinaryOperator, CompletableFuture, ReentrantLock) passed their parent list as an inline compound literal. A compound literal has block storage duration, and cbm_registry_add_type shallow-copies the embedded_types pointer, so the registry kept a dangling pointer into cbm_java_stdlib_register's stack once each statement's block ended. The inheritance walk later read rt->embedded_types[0] for such a type (e.g. java.io.Closeable on a closeable.close() dispatch) — an AddressSanitizer stack-use-after-scope. Hoist the seven lists to static arrays, matching the other parent lists in the same function.
The derive-method synthesis loop tested di_entry->methods[mi].short_name before the mi < 4 bound, so at mi==4 it read methods[4] — one past the 4-element array (UBSan: index 4 out of bounds for DeriveMethod[4]). Evaluate the mi < 4 bound first so the dereference is guarded.
Whitelist find_first_descendant_by_kind and find_first_descendant_of in recursion_whitelist.h (bounded AST-depth DFS, like the existing r_collect_imports entry), and narrow two cppcheck-flagged variable scopes in sqlite_writer.c: use w->project at its single use, and fclose the value-copied wc.fp after free(w) instead of a pre-captured fp.
The CI lint (clang-format-20) flagged formatting violations across these files (pre-existing drift plus the new extraction code). Reformat to satisfy the check; no logic changes.
msys2-clang on Windows lacks the GNU/BSD memmem extension, breaking the Kotlin LSP compile (kotlin_lsp.c called memmem at 4 sites). Add a portable hand-rolled cbm_memmem in helpers.c/.h and use it instead. kotlin_lsp.c also picks up clang-format-20 reflow.
Make the per-grammar regression suite fully green. (1) Add def/name extraction for 22 IDL/config/niche grammars that previously yielded only a Module node (agda, pony, move, cobol, janet, pine, smali, verilog, vhdl, systemverilog, protobuf, graphql, thrift, capnp, smithy, wit, prisma, cmake, puppet, tablegen, assembly, nasm): corrected function/class node-type sets in lang_specs.c and per-grammar name resolution in extract_defs.c (these grammars define no tree-sitter 'name' field, so names are read from typed child nodes; a depth-bounded descendant search handles the FIELD_COUNT-0 (System)Verilog wrappers). (2) Resolve same-file CALLS for groovy, solidity, and ada (callee handlers in extract_calls.c plus descending into Ada subprograms). Golden label histograms and fixtures are updated to the real new extraction output (min_defs raised, never lowered).
Add per-language callee extraction so a same-file caller->callee resolves to a CALLS edge for the lisp dialects (Clojure, Common Lisp, Scheme, Fennel, Racket, Emacs Lisp — head symbol of a list/list_lit), F# (application_expression long_identifier_or_op head), PowerShell (command_name child of a command), WGSL (identifier under type_constructor_or_function_call_expression), and Dart (identifier sibling of the invocation selector); and accept ReScript's value_identifier function field in the shared field resolver. Reduces contract_calls_breadth from 14 to 3 (groovy, solidity, ada still need dedicated branches).
parse_generic_imports only scans direct children of the root, which missed three grammars: Dart wraps imports in import_or_export (the old dispatch matched the never-emitted import_declaration), Haskell nests import nodes under an imports container, and Zig's @import is a builtin_function nested inside a variable_declaration. Add dedicated parsers (dart finds the nested string_literal URI, haskell descends the imports container reusing the module field, zig DFS-finds @import/@cImport builtin_functions) so all three extract their imports.
collect_bases_from_field ignored the bare identifier (and dotted attribute) node that tree-sitter-python uses for a base class inside the superclasses argument_list, so it fell through to the raw-text fallback and captured "(Base)" with parentheses. That never resolved, leaving every Python subclass with zero INHERITS edges. Accept identifier/attribute children so the bare base name is extracted. Turns python_class_base_extracted_bare green.
Several grammars (clojure/racket/scheme, ada/pascal/fortran/fsharp, cairo/d/odin/squirrel/rescript, and the hlsl/ispc/slang C-family) extracted only a Module node. Add per-grammar def-name resolution in extract_defs.c with matching lang_specs entries so their functions reach the graph, and descend Kotlin import_list/import_header in extract_imports.c so Kotlin imports are captured. The vendored grammar MANIFEST documents each custom case.
Split cbm_write_db's monolithic build into an incremental API:
cbm_writer_open / cbm_writer_append_nodes / cbm_writer_finalize. Node
rows are written to the table B-tree across append calls via a
persistent page builder, so callers can flush and free node batches
(including the heavy properties column) without materializing every
record at once. Everything after the nodes table — edges, vectors,
metadata, indexes, and the sqlite_master page — is emitted at finalize,
as before.
cbm_write_db is now a one-shot wrapper (open, append all nodes,
finalize) producing byte-identical output; the full test suite and
PRAGMA integrity_check confirm the on-disk format is unchanged.
state_deserialize() dereferenced &buffer[0] and passed it to
indent_vec_deserialize()'s nonnull argument even when buffer was NULL /
buffer_len was 0 (the initial empty-state deserialize), tripping UBSan. Return
early after clearing state when there is nothing to deserialize.
Hybrid tree-sitter + type-resolution call resolver for Rust, alongside the
Go/C/PHP/C#/Python/TypeScript/Java/Kotlin resolvers. Resolves method dispatch
on typed receivers, UFCS and associated-function calls, trait/impl methods,
generics, closures, the `?` operator, and iterator adapters; seeds the standard
library and well-known crates, parses Cargo.toml for dependency context, and
expands a curated set of derive/proc-macro generated methods.
Includes rust_cargo (manifest parsing), rust_proc_macros (derive expansion),
rust_rustdoc (doc extraction), a generated stdlib + crates seed, and a large
test suite (registered as suite rust_lsp). Wired into the per-file dispatch.
Field-access chains through Self-returning calls remain a documented coverage
gap. Full suite green (4666 tests).
Hybrid tree-sitter + type-resolution call resolver for Kotlin, matching the
existing Go/C/PHP/C#/Python/TypeScript/Java resolvers. Resolves intra-file
and imported calls, method dispatch on typed receivers, extension functions,
operator conventions (plus/compareTo/contains/get/unary), scope functions and
trailing lambdas (it/this), smart-casts (is / when), super dispatch,
constructor-val properties, and stdlib defaults.
Targets the refreshed tree-sitter-kotlin grammar: discovers declarations via
simple_identifier/type_identifier, walks the statements wrapper, extracts
method names from navigation_suffix, descends call_suffix for trailing
lambdas, reads delegation_specifier for inheritance, class_parameter for
constructor fields, and import_list for imports.
Wires the resolver into the per-file dispatch and registers its test suite
(78 tests). Full suite green (4160 tests).
Update 128 vendored grammars to the commits pinned by nvim-treesitter and
Helix (cross-verified canonical sources), normalize their tree_sitter
includes (angle-bracket -> quoted) and external-scanner deserialize
signatures, and restore missing upstream LICENSE files.
Add internal/cbm/vendored/grammars/MANIFEST.md as a provenance record:
upstream repo, pinned commit, current/latest ABI, and license status for
all 157 grammars. 12 first-party (self-maintained) grammars are marked and
excluded from upstream refresh; vhdl and fennel are kept at their prior
revisions (scanner ABI / added-scanner incompatibilities).
Adapt the shared extraction layer to node-shape changes in the refreshed
grammars: resolve declaration names from simple_identifier/type_identifier
children when the grammar drops the `name` field (kotlin, swift), and
recognize multiline_comment as a doc comment.
Integrates the type-aware Java semantic resolver from
worktree-adding-java-lsp-support (based on 673ac4e): java_lsp.c + generated
java.lang/util stdlib registry, wired into the per-file LSP pass, with 94
java_lsp tests + ~170 java_lsp_coverage tests.
Wiring conflicts (the shared files diverged 146 commits on main) resolved by
keeping main's existing LSP registrations (go/c/php/py/ts/cs) and adding the
Java entries in lsp_all.c, cbm.c (cbm_run_java_lsp dispatch), test_main.c and
Makefile.cbm.
Adapted to the current unity-build: renamed java_lsp.c's static helpers
node_text/return_type_of to java_node_text/java_return_type_of so they no
longer collide with ts_lsp.c inside the shared lsp_all.c translation unit.
4082 tests pass (incl. java_lsp + java_lsp_coverage); ASan/UBSan clean.
Martin Vogel