Commit Graph

267 Commits

Author SHA1 Message Date
pcristin 1519f8619a feat(git): add worktree branch roots
Signed-off-by: pcristin <xxxokzxxx@protonmail.com>
2026-06-19 13:56:17 +02:00
Martin Vogel f714bf3de1 Extend the notices bundle to the UI dependencies and the Glama image
The with-ui archives now append the per-package license texts of the
frontend bundle's production dependency tree to THIRD_PARTY_NOTICES.md
(platform-specific native build tooling is listed but excluded — its
code never reaches the browser bundle). The Glama check image carries
LICENSE and the notices file alongside the binary. Also renames a
TypeScript LSP test to describe its generic ODM-interface fixture.
2026-06-12 10:51:11 +02:00
Martin Vogel 3ee9d5d112 Drop the nim grammar and refresh language counts
Removes the heaviest vendored grammar (66 MB) along with its language
wiring and test fixtures; the supported-language count moves to 158
across README, site, and manifest. The grammar manifest additionally
records the canonical-source decisions for the five
registry-disagreement grammars and the per-directory license files
restored earlier this week.
2026-06-12 02:17:39 +02:00
Martin Vogel 630bd40a90 Replace the graph-UI HTTP server with a first-party implementation
The web server behind the graph UI is now a purpose-built in-house
module (src/ui/httpd.c): localhost-only listener, strict HTTP/1.1
parsing with fixed request caps, a per-connection receive deadline,
and Connection: close semantics. http_server.c keeps the routing and
handlers, rewritten against the new transport API; the public server
API and main.c are unchanged. The previously vendored third-party
server is removed entirely.

Comes with a new 28-test transport + routing suite (tests/test_httpd.c)
covering parsing edge cases (strict CRLF, Content-Length limits, raw
path matching, percent-decode rules) and live-socket behavior (CORS
policy, RPC dispatch, receive deadline, clean shutdown). The security
audit scripts now check the new file layout and treat any network call
in vendored code as a failure.
2026-06-12 02:17:25 +02:00
Martin Vogel 39517860ff Hash cross-LSP registries; two-phase def registration
Profiling symfony (416 s wall, one core pinned) convicted cbm_registry_lookup_type -> strcmp at 95% of samples: the PHP and Python def-registration loops probe the registry per Method (receiver-stub check) BEFORE the hash indexes exist, i.e. a linear scan over the growing type table - O(methods x types) per file, and the cross resolvers run per file. Java and Kotlin never called cbm_registry_finalize at all, leaving every lookup during their file walks linear over the whole cross registry. Fixes: (1) two-phase registration in php/py (types, finalize, then funcs/methods); (2) finalize added to the java/kotlin cross entry points; (3) lookups now tail-scan entries added after finalize - previously such entries were silently INVISIBLE to the hashed path (reproduce-first: tslsp_hash_registry_post_finalize_adds was red); (4) cbm_registry_finalize_into allocates the index from a per-call scratch arena - putting buckets into the pipeline-lifetime result arena per file accumulated +1.1 GB RSS on the FastAPI incremental test (incr_full_index caught it), and the py tier-2 builder (one def per call) skips mid-build finalize entirely. symfony: 416 s -> 12 s, identical node count; suite 5577/0.
2026-06-11 14:25:11 +02:00
Martin Vogel 7f78f936ec Spill oversized index keys to overflow pages; escape remaining edge-props text
The hand-rolled dump writer stored index cells fully inline regardless of payload size. SQLite's format spills index payloads above the max-local threshold (16,422 bytes at the 64 KB page size) to overflow pages, so a long key written inline makes the reader interpret key bytes as an overflow page number: PRAGMA integrity_check reported 'invalid page number 0x43654C6C' (ASCII from the key text) in idx_nodes_name on elasticsearch (very long Section names), and name lookups on that index silently returned nothing. write_index_btree now rewrites oversized cells (varint + local prefix + overflow chain via write_overflow_pages) before page building; promoted separators carry their chains with them. Reproduce-first: sw_long_index_keys_overflow (20 KB name; integrity_check failed pre-fix, ok post-fix). Also escapes the remaining raw text interpolations in DATA_FLOWS route props and HANDLES handler QNs (the ~51-edge malformed tail after 6c04ab7).
2026-06-10 16:42:20 +02:00
Martin Vogel 6c04ab7f61 Escape sliced source text in edge properties JSON
Edge properties interpolated raw source-text slices into JSON via snprintf without escaping: decorator text (quotes + raw newlines, e.g. @register.tag("block") or multi-line @override_settings) in DECORATES props - django produced 3826 malformed edges - plus usage ref_name and route-registration callee/url in USAGE/CALLS props. Malformed edge JSON aborts every json_extract consumer, including the url_path_gen generated-column evaluation that runs during PRAGMA integrity_check. All such sites now route through cbm_json_escape (DECORATES twins in pass_parallel.c/pass_semantic.c, usage emit twins, route-registration calls). Regression: pipeline_edge_props_valid_json (register.tag fixture; real-repo red recorded on the django index).
2026-06-10 13:44:09 +02:00
Martin Vogel 0442cceb80 Bound lambda/method-ref expected-param lookup by declared param count
bind_lambda_args and the method-reference arity path indexed the resolved method signature's NULL-terminated param_types array by the CALL-SITE argument index. A call with more arguments than the resolved overload declares (overload mismatch, varargs) read past the terminator and dereferenced whatever followed in the arena as a CBMType* - SIGSEGV while indexing elasticsearch (same out-of-bounds family as #427 and the bitcoin targs fix). Both sites now bound the index by the walked param count. Reproduce-first: 6-line fixture - run(Runnable) invoked with six lambdas - crashed rc=134 pre-fix; fork+exit-signal regression test lsp_java_lambda_args_exceed_params_no_crash added; suite 5574/0.
2026-06-10 13:03:00 +02:00
Martin Vogel bb7341dc07 Fix uninitialized template-arg arrays binding stack garbage into type graphs
The C/C++ LSP's explicit-template-argument call path (template_function, e.g. Using<Fmt>(v)) filled a stack array const CBMType *targs[16] without NULL-terminating it before cbm_type_substitute. With more declared type params than explicit args (bitcoin src/serialize.h: Using's Wrapper<Formatter, T&> with 2 params, 1 arg), the bounded args walk read uninitialized stack; any non-NULL garbage got bound to the type param and woven into the REGISTERED return-type graph, then dereferenced at a later call site - SIGSEGV (crash shape: string bytes misread as a CBMType*). All c_lsp targs arrays are now zero-filled, and cbm_type_substitute's args walk treats implausible values (misaligned / below the first page) as the terminator so contract violations can never leak garbage into a type graph. Reproduce-first: deterministic poison-args unit test (typerep_substitute_rejects_garbage_args_entries) + real-repo verification (bitcoin indexes clean under ASan; was a deterministic SIGSEGV).
2026-06-10 12:11:02 +02:00
Martin Vogel 9f981a1612 Bound LSP resolve recursion depth (stack-overflow crashes indexing real repos)
Indexing large OSS during the perf sweep crashed three LSP resolvers, all by stack overflow in their recursive resolve walks (macOS crash reports): elasticsearch — SIGSEGV in bind_lambda_args under deep recursive java_resolve_calls_in_node frames; bitcoin — SIGSEGV in cbm_type_substitute via c_adl_resolve under deep c_resolve_calls_in_node frames; microsoft/TypeScript — SIGBUS under an unbounded lookup_member_type cycle (cyclic type graph in reallyLargeFile.ts). The expression-eval guards (eval_depth) did not cover these walks. Each resolver now has a depth-guarded entry (walk_depth cap 512 for the Java/C AST walks, member_depth cap 64 for TS member lookup); past the cap the subtree resolves as unknown — graceful degradation instead of a crash. Reproduce-first: prod CLI probes crashed pre-fix (nested-call Java rc=139, nested-call C++ rc=139); fork+exit-signal regression tests added (TS cyclic shape needs a real cross-file registry, so it is verified at the real-repo tier — microsoft/TypeScript now indexes clean). Side-finding documented in the sweep report: Java BLOCK nesting >=3000 is minutes-slow (separate pathology, not this fix).
2026-06-10 10:44:19 +02:00
Martin Vogel 359f37cd71 Escape control bytes in properties-JSON array items (raw newlines from multi-line C params)
The array appenders (param_types/param_names/decorators/base_classes) in both props-builder twins escaped only quote and backslash, so items sliced from multi-line C parameter declarations carried raw newline/tab bytes — invalid inside JSON strings (the remaining 118 malformed kernel rows after the truncation fix; e.g. param_types:["struct\n\t\t..."]). Array items now go through the full escape function, and both escapers degrade any other raw control byte (e.g. form feed) to a space. Reproduced via the prod CLI on a multi-line-param fixture (nl_fn json_valid=0 pre-fix); the sweep regression test gained that fixture (61st case).
2026-06-09 23:54:25 +02:00
Martin Vogel 9f465e5109 Apply atomic properties-JSON appends to the parallel pipeline twin
The parallel extraction path (taken above MIN_FILES_FOR_PARALLEL=50 files — i.e. every real repo) has its own copy of the props appenders in pass_parallel.c, so the previous fix in pass_definitions.c did not cover it: a kernel reindex still produced the 135 malformed rows and get_architecture's hotspots aspect returned empty (its json scan aborts on the first malformed row). Same atomic-append change as the serial twin. The reproduce-first sweep test now writes one function per file (60 files) to force the parallel pipeline; RED via pass_parallel pre-fix, GREEN now.
2026-06-09 23:30:51 +02:00
Martin Vogel 2712756492 test: make arch_boundaries perf guard self-relative (CI machine spread)
The absolute 8s bound failed on the shared ubuntu-arm UBSan runner, which measured 25.9s for work that takes 207ms locally (125x spread). The guard now times the boundaries aspect at 1x and 2x graph size and asserts the ratio stays below 3 (linear ~2x, the old quadratic ~4x), with an absolute fast-path short-circuit on machines where the small size completes in under 2s.
2026-06-09 23:07:47 +02:00
Martin Vogel 4b317c8779 Make definition-properties JSON appends atomic (no mid-value truncation)
build_def_props serialized into a fixed 2KB buffer and cut fields mid-value when full: the closing quote/brace got skipped, leaving malformed properties JSON (Linux kernel: 135 nodes, 50-param functions cut at 2047 bytes inside param_types — '["enum'). Malformed properties abort every json_extract()-based consumer (arch_entry_points, user Cypher on properties) mid-scan. Appends are now atomic: a field is emitted only when its whole serialized form fits (closing-brace reserve included); oversized optional fields are dropped whole and the JSON stays valid. Reproduce-first: pipeline_def_props_valid_json_when_oversized (param-count sweep deterministically hits the truncation window; RED 2047-byte malformed pre-fix, GREEN post-fix).
2026-06-09 22:59:33 +02:00
Martin Vogel 66bf33a003 Cap name-resolution candidate lists at 256 (unresolvably ambiguous)
resolve_name_lookup walked the full by_name candidate array (reachability check + score per candidate, re-done per file via the per-file caches) for every cache-miss name. On identifier-dense repos this dominated usage resolution: the Linux kernel has 274 names with >256 registered definitions (list_head 7188, flags 5520, dev 4374, ...) and resolve_usages burned 987s CPU. candidate_count_penalty already floors confidence to ~3/count (<=0.006 at 256), so these matches were noise edges. Resolution now bails out as unresolved when a name has >256 candidates (same cap in fuzzy_resolve); same-module and import-map strategies still resolve such names exactly. Kernel re-measure (same files): resolve_usages CPU 987s->177s, parallel_resolve 134s->53s, wall 325s->251s, edges -485k noise (-5%); suite 5568/0.
2026-06-09 21:53:26 +02:00
Martin Vogel b904cf0ab5 Fix O(E*N) scan in arch_boundaries: binary-search the id->package map
get_architecture's arch_boundaries looked up the package of every CALLS edge endpoint with a linear scan over all Function/Method/Class nodes (lookup_pkg over parallel arrays) — O(edges * nodes). On the Linux kernel graph (~1.4M defs, ~1.4M CALLS edges) the boundaries aspect spun >10 minutes at 100% CPU; latent until C call extraction was fixed (the kernel previously produced almost no CALLS edges). The node query now orders by id and lookup_pkg binary-searches: 80k nodes/160k edges 16.6s -> 207ms in the regression test; kernel-graph get_architecture(all) >10min -> 23s. Reproduce-first: arch_boundaries_no_quadratic_scan (red 16566ms pre-fix vs 8000ms bound, green 207ms post-fix).
2026-06-09 21:32:16 +02:00
Martin Vogel a50b086809 Attribute out-of-line C++ method definitions to their class (distill #428)
C++/CUDA out-of-line method definitions (Foo::bar in a .cpp; the class body lives declaration-only in a header) were recorded as free Functions with no link to their class. A new helper cpp_out_of_line_parent_class() detects a qualified_identifier declarator on a function_definition and resolves the immediate enclosing class (descending nested scopes ns::Foo::bar to the direct parent); extract_func_def then promotes the def to label Method with parent_class set to the class QN so DEFINES_METHOD edges resolve. Distilled (reimplemented) from PR #428. Reproduce-first: cpp_out_of_line_method_issue428 (single-level out-of-line defs; the nested-namespace path isn't isolation-testable because tree-sitter-cpp parses a synthetic doubly-qualified def as an ERROR node, so it is exercised by real codebases).
2026-06-09 14:32:26 +02:00
Martin Vogel fb17573279 Fix out-of-bounds read in cbm_type_substitute when type_args is shorter than type_params (#427)
The C++ LSP passes a class's declared template-param names (e.g. template<class T, class Alloc = ...>) alongside an instance's fewer template args (vector<int>). The substitution loop is bounded by type_params, so a param matching beyond the supplied args indexed type_args[i] past the array — reading adjacent arena memory (a type-name string byte) as a CBMType* that c_eval_expr_type later dereferenced -> SEGV (ASan: stack-buffer-overflow in cbm_type_substitute via c_eval_expr_type). Deterministic, all platforms; distinct from the #424 allocator crash. Fix at the single chokepoint: bound the args length by the (NULL-terminated) param count and gate every type_args[i] access by it, so no caller can over-read regardless of language. Reproduce-first: typerep_substitute_short_args_no_oob_issue427.
2026-06-09 14:32:26 +02:00
Martin Vogel 0d2da15d4e Bind tree-sitter runtime to mimalloc in prod build to fix C++ indexing crash (#424)
Root cause (corrected from the report): not a tree-sitter null-subtree sentinel — 0xffffffffffffffff is a freed/heap-corrupted pointer. The vendored ts runtime allocates via its overridable ts_current_malloc/free; under the prod build's MI_OVERRIDE=1, the Windows static-MinGW link (--allow-multiple-definition) resolves ts_malloc to mimalloc but ts_free can resolve to the CRT free (or vice-versa), corrupting the heap freelist and crashing mid-parse on large templated C++ headers (scales with parse churn, not syntax). cbm_init now calls ts_set_allocator(mi_malloc, mi_calloc, mi_realloc, mi_free) so ts allocate+free go through one allocator on every platform. Guarded to CFLAGS_PROD (MI_OVERRIDE=1); the test build stays CRT+ASan to avoid an alloc/free mismatch there. Tests in test_stack_overflow.c: allocator-binding mechanism + large-templated-C++ extraction guard (the C++ gap noted in #424).
2026-06-09 09:20:15 +02:00
Martin Vogel 6f3b24df4f Fix CI lint: drop debug scaffolding, make lisp import walker iterative, apply clang-format
Remove leftover getenv(CBM_DBG_*) debug fprintf blocks in extract_calls.c/pass_calls.c (cppcheck always-true). Convert parse_lisp_imports to a stack-based walk (no recursion → no NOLINT, avoids deep-nesting overflow). Apply Homebrew clang-format to all touched sources. Local suite 5562/0, lint-ci clean.
2026-06-08 22:55:23 +02:00
Martin Vogel 53f1bf2ebd Annotate python dunder_getitem receiver (sound subscript-dunder resolution)
The unannotated param could not be resolved soundly (would require an unsound sole-class guess). Annotating s: Seq exercises real type-based subscript-dunder resolution; py_lsp synthetic-call injection completes it.
2026-06-08 22:55:23 +02:00
Martin Vogel 6afb45427c Add grammar-only import parsers + pipeline edges (WRITES/CONFIGURES/DEPENDS_ON)
Imports: per-grammar parsers for hare/pascal/powershell/scheme/racket/starlark/tcl/teal/zsh + css/html/cmake/bitbake/kconfig/gn/just/nix/jsonnet/pkl/nickel/thrift/capnp/d/tablegen/crystal/fsharp/ada; node-extraction for D/PowerShell/Pascal/Julia/F#; INHERITS for those. Pipeline edges: register Variable/Field defs for WRITES/READS resolution; CONFIGURES pass consuming env_accesses; DEPENDS_ON from go.mod/requirements.txt; SQS dot-form QN. Golden updates: julia +Class, just +Function (correct new extraction).
2026-06-08 13:04:12 +02:00
Martin Vogel 2d772b01bf Fix extraction/resolution across languages: routing, Go methods, cross-LSP, inheritance, constructors, node kinds
WIP checkpoint (local). File-index routing for dotenv/gitattributes/regex; Go receiver-method parent_class; cross-LSP dispatch + :: QN split (Rust/Java/C#/Kotlin); base_classes for TS/PHP/Kotlin/Python + Java/C++ qualified; new T() constructors (java/c#/php/scala/ruby/ts); per-grammar node extraction for ~17 struct/record/type grammars. Test corrections: cpp qualified-base :: guards, gitignore ignore-rules contract, julia struct golden.
2026-06-07 22:56:12 +02:00
Martin Vogel 0518274783 Complete 159-grammar coverage: IDL/template/IaC/markup/remaining-code probes; +1 new class
Final 4 probe suites (grammar_probe_d/e/f/g, 250 cases) bring node/edge-creation coverage to all 159 grammars. Triage fixed 4 fixture mislabels (objc Method label, smithy $version order, json non-ignored filename, sql DDL-only Variables) and confirmed ~46 real reds. Findings map to known classes 2/3 (import/include-edge resolution missing for ~34 grammars: agda/bicep/elm/func/move/nasm/pony/purescript/qml/smali/tlaplus/vhdl/protobuf/thrift/capnp/wit/smithy/astro/nix/nickel/pkl/jsonnet/hyprlang/devicetree/cmake/makefile/meson/gn/just/bitbake/kconfig/html/css/scss), 16 (node-extraction incompleteness: wolfram multi-def/set_top, wit world funcs, bicep resource, just recipe), plus infra gaps (INFRA_MAPS hcl/k8s only from YAML topic bindings; DEPENDS_ON only from Helm Chart.yaml — go.mod/requirements produce none). NEW class 17 = file-index routing gap: .env/.gitignore/.gitattributes/.re have no EXT_TABLE/FILENAME_TABLE entry in discover/language.c so index_repository never indexes them (direct extraction works). Vue/Svelte/Astro component imports: extraction works but JS/TS path resolver doesn't handle those extensions. All 159 grammars now have reproduce-first guards.
2026-06-07 20:05:24 +02:00
Martin Vogel 21bc98ff92 Add grammar-only probe suites (~35 code grammars); +1 new root-cause class
Extends the node/edge-creation hunt to ~35 grammar-only code languages (129 graph/extraction cases, 73 green guards / 56 real RED reproductions; triage confirmed 0 fixture errors — all reds are genuine product gaps). Findings map to: class 2 import-EXTRACTION missing (cbm_extract_imports dispatch in extract_imports.c covers only ~38 langs + default:break, so ~25 grammar-only langs' configured *_import_types are never consumed → 0 imports); class 1 INHERITS missing (crystal/julia/pascal/powershell/squirrel/solidity); and NEW class 16 = per-grammar NODE-extraction incompleteness (class_types configured + grammar node present, but extract_defs.c emits 0 type nodes for D struct/class, F# record/union, Gleam type, Julia/Odin struct, Pascal record, PowerShell class, Hare struct, ReScript type, squirrel class, sway struct/abi, tcl namespace, wgsl struct). Affected-grammar matrix in the suite comments. Brings code-grammar coverage to ~61 of 66.
2026-06-07 19:13:01 +02:00
Martin Vogel 019a33683b Add convergence + matrix probe suites; triage confirms no new root-cause classes
Final hunt batches (convergence_probe 47, matrix_known_classes 44, matrix_new_constructs 63 = 154 graph-level cases) plus triage. Result: ZERO genuinely-new root-cause classes — every confirmed red maps to the already-known taxonomy (classes 1,4,5,6,7,12,13,14). Triage greened 5 fixture mis-builds: recursion self-calls (Go/Python/Rust/Java) produce 0 CALLS because self-loop edges are SUPPRESSED BY DESIGN (pass_calls.c/pass_parallel.c src->id==dst->id) — not a bug, fixtures now add a non-self caller; Ruby bare 'describe' parses as identifier not call (fixed to describe()). 26 reals kept as reproductions with // REAL BUG: root-cause comments. Confirms the bug hunt has converged: aggressive probing across new constructs + more languages surfaces only instances of the ~15 mapped classes.
2026-06-07 18:29:05 +02:00
Martin Vogel 2aa09bf4e0 Triage probe reds: fix fixture-only failures, confirm real bugs with root causes
Classified all 34 probe reds against the detector/extractor code. Greened 9 fixture mis-builds (HANDLES express/fastify/rails/actix, HTTP fetch/nethttp/reqwest, C generic-callback) + 1 wrong assertion (Go methods are labelled Method not Function). Kept 24 as confirmed reproductions with // REAL BUG root-cause comments. Root-cause clusters: WRITES x5 langs (Variable defs never cbm_registry_add'd); Go DEFINES_METHOD+OVERRIDE/IMPLEMENTS (receiver methods lack parent_class); new T() constructor CALLS missing for Java/C#/PHP (object_creation_expression not in call_node_types) + TS not linked; cross-LSP dispatch gap + '::' path resolution (Rust/Kotlin/Java/C#); C++ scope-qualified calls; HANDLES (Spring annotation, ASP.NET, Laravel string handlers); HTTP_CALLS (RestSharp, Guzzle short-QN); ASYNC SQS-Go; Kotlin throws/raises; TS type-param USAGE. node_creation_probe now fully green.
2026-06-07 14:16:36 +02:00
Martin Vogel 0798748abf Add probe reproduction suites: LSP-pass resolution, node creation, edge-type creation
Reproduce-first probes (215 graph-level cases) hunting node/edge-creation + LSP-pass bugs across 26 languages + web frameworks. node_creation_probe: 80/81 green (node creation is solid; only Go method-on-struct under threshold). lsp_resolution_probe: 72/83 green — RED reproductions for the cross-LSP dispatch gap (Rust/Kotlin S1 cross-file call) and constructor/static/virtual/generic resolution edge cases (C/C++/TS/Java/C#/PHP S3-S7). edge_types_probe: 29/51 green — candidate RED reproductions for HANDLES (7 web frameworks), HTTP_CALLS (5 clients), WRITES (all 5 langs), Kotlin throws/raises, Go DEFINES_METHOD/OVERRIDE (these need real-bug-vs-fixture verification). Green cases retained as regression guards throughout. No unrelated suite regressed.
2026-06-07 13:36:15 +02:00
Martin Vogel e31dd24433 Add graph-level edge-creation reproduction suite (pipeline edge resolution)
Reproduce-first coverage of the pipeline step that turns extracted symbols/imports into resolved graph edges, across the 9 hybrid-LSP languages (91 graph-level cases). Live RED reproductions: IMPORTS edges are never created for Rust/Kotlin/Java/C#/PHP (all forms) despite working extraction (+ Python aliased/wildcard, TS re-export edge cases); cross-file CALLS do not resolve for Rust and Kotlin (lsp_cross pass processes 0 files for them) — missed by real-repo checks because same-file calls dominate; same-file IMPLEMENTS edges absent for Java/C# and Go implicit-interface satisfaction; same-file DECORATES absent for TS and C# attributes; cross-file USAGE absent for TS; cross-file INHERITS absent for TS/PHP/Kotlin/Python (extraction-propagated). Green guards retained for every working path (Go/Python/TS imports, cross-file CALLS for 7 langs, INHERITS for Java/C#/C++, etc.). 51 RED reproductions; no other suite affected.
2026-06-07 13:00:55 +02:00
Martin Vogel 857432b33d Add language-agnostic extraction reproduction suite (inheritance + imports)
Reproduce-first coverage of class-inheritance (base_classes) and import extraction across the 9 hybrid-LSP languages (~300 table-driven cases). Live RED reproductions: TS/TSX store the extends/implements keyword instead of the base type; PHP never populates base_classes and its use-imports parse the wrong node type; Kotlin does not parse the : supertype list; Python loses Generic[T] (paren/bracket leak); C++ drops qualified bases (std::vector) and leaks ':'; Rust misses several impl-Trait-for-T forms; C# import aliases (using F = X) and Python wildcard/__future__ imports are missed. C# inheritance and Go/TS/JS/Java/Kotlin/Rust import extraction pass as regression guards. Suite is RED until the extractors are fixed; 12 failing reproductions, no other suite affected.
2026-06-07 12:35:53 +02:00
Martin Vogel de1d58ab9e Distill PR #407 (#406) and PR #414 (#413): parent-death watchdog + CBM_LOG_LEVEL
Two reviewed external-fork contributions, re-implemented as clean maintainer code.

#406 / PR #407 (thanks @nvt-pankajsharma): a POSIX parent-death watchdog so the stdio MCP server exits when its launching parent dies instead of lingering on stdin. Refactors signal_handler into an idempotent request_shutdown(); a watchdog thread polls getppid() (500ms) and shuts down + exit(0) once the initial ppid (>1) changes; Windows unaffected (#ifndef _WIN32). The fork's getppid()<=1 startup early-exit was dropped (it could wrongly kill a legitimately-launched server during reparent races / in container launchers; the initial_ppid>1 guard already no-ops safely), and thread-create failure is now non-fatal (matches the watcher/HTTP background-thread policy). Adds tests/test_parent_watchdog.sh + scripts/test.sh wiring.

#413 / PR #414 (thanks @santanusinha): runtime log-level control via CBM_LOG_LEVEL (case-insensitive debug|info|warn|error|none, or numeric 0..4; unset/unknown leaves the level unchanged). getenv() before threads start (no race). Adds tests/test_log.c cases + README env-var row.
2026-06-06 22:30:38 +02:00
Martin Vogel 7bce843285 Add reproduction for package.json workspaces cross-package IMPORTS (#408)
A Lerna/Yarn workspaces monorepo where packages/b imports a sibling by package name (@org/a) should resolve via package.json name->dir into a cross-package IMPORTS edge. Reproduction contract; verifies whether the generic workspaces case resolves (cbm_pipeline_resolve_module already looks up bare specifiers in the pkgmap) or exposes a gap.
2026-06-06 22:16:05 +02:00
Martin Vogel ba52d6ba56 Report excluded subtrees in the index_repository result (#411)
index_repository silently dropped entire subtrees (ALWAYS_SKIP dirs in all modes; FAST_SKIP dirs like tools/scripts/bin in moderate/fast) with no record — users couldn't tell what wasn't indexed. The discover walk now records skipped directory rel-paths; cbm_discover_ex returns them; the pipeline holds + exposes them; and the index response carries a compact summary: "excluded":{"dirs":[up to 25],"count":N,"truncated":bool}. Behavior-neutral for indexing. Adds a reproduction in test_integration.c.
2026-06-06 22:16:04 +02:00
Martin Vogel 02d27743d9 Fix O(n^2) overload resolution in the C/C++ cross-LSP registry (#410)
Four overload-scoring lookups (cbm_registry_lookup_method_by_types/_by_args, _symbol_by_types/_by_args) linear-scanned the entire project registry on every call site, ignoring the pre-built method_buckets/func_qn_buckets hash index that the by-name lookups already use. On a large C++ header (e.g. 10989 defs) that is O(calls x project_funcs) per file (~34s/file), so big codebases never finish indexing. Add a hashed fast path (O(overloads)) when the registry is finalized; preserve first-match semantics via lowest registration index (equivalence verified over 1606 lookups, 0 divergence); keep the linear fallback for un-finalized single-file registries so normal files are unaffected. Adds tests/test_cpp_index_hang.sh (scale-tier subprocess+timeout repro, opt-in via CBM_RUN_HANG_TEST).
2026-06-06 22:16:04 +02:00
Martin Vogel 4207c83679 Make parallel_for_actually_parallel deterministic
The test proved parallelism with a fixed 10000-iteration busy-wait, hoping workers would overlap — but on a loaded CI runner a worker could finish before the next thread started, leaving concurrent_max=1 (a flaky FAIL, seen on ubuntu-latest). Spin until at least two workers are concurrently active (bounded, with an early exit once the overlap is proven) so the parallelism invariant is demonstrated deterministically instead of by timing luck.
2026-06-06 19:48:48 +02:00
Martin Vogel a419bc81d5 Assert git in the FILE_CHANGES_WITH contract instead of skipping
git is now installed on every CI platform (incl. the Windows msys2 env) and run_git uses portable git -C, so a failed init is a broken-env failure, not a platform skip.
2026-06-06 18:36:28 +02:00
Martin Vogel da9332ee59 Run the watcher git tests on Windows via portable git -C
The watcher tests built their git repos with POSIX bash via system() (export ...; cd ... && git ...), which the native Windows test binary runs through cmd.exe and cannot execute — so they were SKIP_PLATFORM'd there. Replace the shell with wt_git() (git -C "<dir>" -c user.* ..., no shell metacharacters) plus th_write_file/th_append_file, so all 19 git-setup blocks run identically on POSIX and Windows. Removes the SKIP_PLATFORM.
2026-06-06 18:36:28 +02:00
Martin Vogel 945e55d4d9 Fix cfml/HTML scanner tag-name leak; drop the LSan suppression
scan_start_tag_name moves scan_tag_name's owned tag_name into a Tag, but the CF_SET/CF_RETURN/CF_ELSEIF/CF_ELSE cases return without pushing the tag to the scanner stack and (unlike CF_VOID) without freeing it — leaking the tag_name buffer under LeakSanitizer. Free the tag in those return paths, matching CF_VOID. Removes the need for the __lsan_default_suppressions(scan_tag_name) workaround in the test runner.
2026-06-06 18:36:27 +02:00
Martin Vogel 6f45ac32b8 Write fixtures in binary mode; skip FILE_CHANGES_WITH without git on Windows
fopen text mode rewrites newlines to CRLF on Windows, making line-ending-sensitive grammars under-extract (below_min/calls_breadth). Open fixtures with wb. Also skip the git-history contract when git init fails (platform limitation) rather than failing.
2026-06-06 14:57:30 +02:00
Martin Vogel 7e9baa6b0b SKIP_PLATFORM watcher git tests when the POSIX shell setup is unavailable
The watcher tests build their git repo via system() with POSIX shell syntax (export ...; cd ... && git ...), which the native Windows test binary runs through cmd.exe and cannot execute. Skip them as a platform limitation there (they still run on the POSIX legs) instead of failing.
2026-06-06 14:57:29 +02:00
Martin Vogel 2e044dc9ad Normalize temp-dir paths to forward slashes for the Windows test build
cbm_mkdtemp on msys2 yields a backslash Windows path; embedding it in the JSON repo_path produced invalid escapes so index_repository failed (count=-1) for every harness contract, plus mixed-separator git -C args. Normalize to forward slashes after mkdtemp — valid in JSON and accepted by Windows file APIs and git.
2026-06-06 13:33:15 +02:00
Martin Vogel 943eb16334 Suppress the bounded cfml/HTML vendored-scanner leak under LeakSanitizer
scan_tag_name in the shared vendored HTML-family tree-sitter scanner leaks a small bounded tag-name buffer in an edge path of scan_start_tag_name. tag_for_name's ownership is grammar-specific and not verifiable without HTML-family parse coverage, so the bounded leak is suppressed via __lsan_default_suppressions rather than risking a blind double-free. The larger SQL scanner leak was fixed properly.
2026-06-06 13:33:15 +02:00
Martin Vogel 0baa92c43c Guard the fork-based crash check for the Windows build
extract_crashes uses fork/waitpid/<sys/wait.h>, which msys2-clang lacks, so the Windows test build failed with 'sys/wait.h not found'. Guard the include for _WIN32 and run extraction in-process there (fixtures are crash-free; the fork-isolated check still runs on the POSIX CI legs).
2026-06-06 12:34:59 +02:00
Martin Vogel d8d114747c Free the thread-local node-type cache to satisfy LeakSanitizer
cbm_kind_in_set builds a thread-local cache of calloc'd symbol bitsets (ks_cache) that was never freed, so LeakSanitizer (Linux x64) reported the per-worker-thread caches as leaks once the previously-empty grammars gained real node-type sets. Add cbm_kind_in_set_free_cache() and call it at worker-thread teardown (extract_worker, beside cbm_slab_destroy_thread) and main-thread exit (test runner, beside sqlite3_shutdown).
2026-06-06 12:34:59 +02:00
Martin Vogel 3d9731f861 Zero-initialize LangFile fixture arrays (GCC -Werror=maybe-uninitialized)
GCC at -O1 flags the partially-filled LangFile arrays in contract_all_grammars_in_graph and index_parallel_fixture as maybe-used-uninitialized when passed to lang_index_files (clang does not), failing the -Werror Linux test build. Zero-initialize both; only files[0..nfiles-1] are ever read, so behavior is unchanged.
2026-06-06 11:43:23 +02:00
Martin Vogel d30093aa8f Extract definitions for 22 grammars and resolve remaining CALLS gaps
Make the per-grammar regression suite fully green. (1) Add def/name extraction for 22 IDL/config/niche grammars that previously yielded only a Module node (agda, pony, move, cobol, janet, pine, smali, verilog, vhdl, systemverilog, protobuf, graphql, thrift, capnp, smithy, wit, prisma, cmake, puppet, tablegen, assembly, nasm): corrected function/class node-type sets in lang_specs.c and per-grammar name resolution in extract_defs.c (these grammars define no tree-sitter 'name' field, so names are read from typed child nodes; a depth-bounded descendant search handles the FIELD_COUNT-0 (System)Verilog wrappers). (2) Resolve same-file CALLS for groovy, solidity, and ada (callee handlers in extract_calls.c plus descending into Ada subprograms). Golden label histograms and fixtures are updated to the real new extraction output (min_defs raised, never lowered).
2026-06-05 21:53:25 +02:00
Martin Vogel 0f04003508 Reproduce Python class-inheritance base-name extraction bug
class Animal(Base) extracts the base as the argument_list text "(Base)" instead of the bare "Base", so it never resolves to the Base class node and Python subclasses get zero INHERITS edges. Capture this as a failing test ahead of the fix (root cause: collect_bases_from_field ignores the identifier node tree-sitter-python uses for the base).
2026-06-05 21:53:25 +02:00
Martin Vogel 7a43b38ef7 Add per-grammar regression suite across all 159 grammars
Index a fixture for every grammar through the full pipeline and assert graph-level invariants: golden node-label histograms, def/node breadth, CALLS and IMPORTS breadth, presence of all 26 pipeline edge types (structural, type-relationship, service/dataflow, and similarity), no-crash, and call resolution. Reproduced quality gaps (under-extraction, import-extraction, and CALLS-resolution) are kept as hard failures for the fix phase rather than skipped. Adds a real-repo scale tier (scale_contract.sh) and ignores local-only private/ scratch.
2026-06-05 21:53:25 +02:00
Martin Vogel eed87fd372 Forbid test skips and convert existing skips to hard failures
Add scripts/check-no-test-skips.sh (run from lint) which fails the lint phase on any plain SKIP() or direct tf_skip_count manipulation; only SKIP_PLATFORM() (for genuinely platform-specific tests) is tolerated. Add FAIL() and SKIP_PLATFORM() helpers to the test framework and convert the remaining SKIP()/perf-gated skips across the suite into pass-or-fail assertions, so a suite that cannot meet its preconditions reports a red failure instead of a silent skip.
2026-06-05 21:53:25 +02:00
Martin Vogel 9515446bba Expand grammar extraction regression suite to all vendored languages
Cover every CBM_LANG_* enum (157 languages) in the per-language regression
net, up from the initial 35. Code languages assert a catastrophic-break
floor (defs >= 1); data/config/markup languages assert no-crash extraction.
A future grammar refresh that breaks any language's extraction now fails
loudly with the culprit named.
2026-06-03 17:38:09 +02:00