Second piece of closure repair. At the collect_all_defs seam -- the only
moment the per-file result cache is alive -- both drivers (parallel and
sequential) now serialize each file's CBMLSPDef slice to canonical JSON,
hash it, and hand the rows to the pipeline; cbm_pipeline_publish_generation
writes them into the staging store next to the manifest, so surface data
and graph always belong to the same generation.
Canonical bytes are the point: every field is written in fixed order with
an explicit null for absent strings (NULL and "" differ in the CBMLSPDef
contract -- receiver_type NULL means "not a method"), so byte equality IS
surface equality and the sha over the bytes is the early-cutoff key.
Registry-only labels that pxc_map_label drops but the name registry serves
(Field) are folded into the hash as a separate "reg" array, or renaming
one would slip past the cutoff.
Behaviour pinned in SUITE(pipeline): a fresh full index persists a
versioned surface row per file; a BODY edit republishes the identical
surface_sha; a SIGNATURE edit changes it. That pair of properties is what
the routing layer will stand on.
cbm_pxc_collect_all_defs gains an optional per-file prefix array -- the
flat all_defs[] otherwise loses the file boundaries the serializer needs.
CORRECTION to 6c22338's scope note: it claimed cbm_pipeline_publish_
generation was reachable only behind CBM_INCREMENTAL_TEST_API. Wrong --
dump_and_persist_hashes calls it on every production full index
(pipeline.c:1863); the grep that "verified" test-only reachability had
excluded pipeline.c itself. The predictable staging name WAS in the
production publish path, which makes that fix a real production hardening,
not a test-path cleanup.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The sequential lsp_cross pass builds its shared per-language cross registries
in ctx->seq_cross_arena, which DELIBERATELY outlives the pass -- resolved_calls
and the registries carry borrowed strings that pass_calls still reads, and the
arena is destroyed only after all passes (the earlier freeing-here bug was a
pass_calls use-after-free, says the comment at the arena's creation).
But the per-file module-QN strings (def_modules[], malloc'd in
cbm_pxc_collect_all_defs and handed to every registrar as def_module_qn) were
freed at the END OF THE PASS -- the exact mistake the arena comment warns
about, one level down. Any registry-reachable structure holding one of those
pointers read freed memory in pass_calls.
AddressSanitizer caught it as a heap-use-after-free (strcmp in
cbm_pipeline_pass_calls on a string freed by the pass-end cleanup) on the
first-ever run of the real-repo determinism tier (linux/fs/xfs, 355 files) --
a tier no CI runner can execute because the corpus is local-only, which is why
it survived: bisect shows it predates today's commits (b020748 reproduces),
and main is clean on the identical suite.
Ownership now transfers to the ctx at the end of the pass and the strings are
released beside the arena, in the pipeline teardown and in test_parallel's
direct-drive harness. The parallel path is unchanged: it already destroys its
registries and module strings together, before any later pass.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Second LSan round from the Linux leg, both verified green in the container
(532 passed, 0 failed under LSan):
- dump_and_persist_hashes' two semantic-manifest abort returns leaked BOTH of
the function's strdups (db_path and db_dir, the latter otherwise freed only
further down). Same ownership rule as the previous fix: every exit releases
what the function allocated.
- test_parallel's sequential harness drives the passes directly and never
destroyed ctx->seq_cross_arena, which the cross pass fills with the shared
per-language registries (stdlib registrations included -- ~20MB per test).
Production's run_sequential_pipeline destroys it after all passes; the
harness now does the same.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The parity harness drives the pipeline passes with a bare ctx and never
ran the production teardown, so the return-type table pass_calls builds
leaked — latent until the fixture gained typed Java methods, and visible
only under Linux LeakSanitizer. Both runners now free it exactly the way
pipeline.c does.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The parallel per-file resolve hardcoded INHERITS for every base-class
relation while the sequential semantic pass made the Interface split, so
any corpus large enough for the parallel path had its explicit implements
demoted: elasticsearch carried 11,179 INHERITS edges pointing at Interface
nodes and 3 IMPLEMENTS. Both venues now share one decision keyed off the
target node's label — the graph truth the edge attaches to.
OVERRIDE edges existed only for Go's implicit interface satisfaction. A
serial full-graph tail run by both pipelines now matches class methods
against explicit bases by name and emits Method-to-Method OVERRIDE edges
for the explicit languages, with Go excluded to avoid double cover.
The venue parity fixture gains a Java interface/implements/extends trio —
IMPLEMENTS parity used to compare 0 == 0 and guarded nothing — plus an
exact-count test so the parity can never go vacuous again. At scale:
elasticsearch IMPLEMENTS 3 to 11,182 and OVERRIDE 0 to 49,695, TypeScript
0 to 4,496 and 594, dotnet/runtime 0 to 12,953 and 45,667, with INHERITS
plus IMPLEMENTS sums conserved exactly.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
F1: dot the Exporter import target so seeded CPAN exported subs resolve.
perl_collect_qw_imports built colon-form targets (Scalar::Util::blessed)
but the stdlib registry keys curated CPAN subs in dotted form
(Scalar.Util.blessed) and lookup is exact-match. Wire in perl_pkg_to_dot
to dot the module portion and drop the now-unnecessary (void) cast.
F2: add a recursion-depth guard (CBM_LSP_PERL_MAX_WALK_DEPTH=512) to both
AST walkers (perl_resolve_calls_in_node, perl_pass1_scan) via a depth-
guarded wrapper + inner split, mirroring java_lsp's JAVA_LSP_MAX_WALK_DEPTH.
Past the cap a subtree is skipped (graceful degradation, no wrong edge),
preventing stack overflow on pathologically nested input.
F3: lock the shared last-"::"-segment normalization in lsp_resolve.h with a
direct regression test over cbm_pipeline_find_lsp_resolution: a qualified
static call still resolves AND the cross-namespace mis-attribution edge case
is bounded by caller-QN equality + the confidence floor.
F4: implement SUPER:: dispatch. Populate enclosing_parent_qn from the
enclosing package's first @ISA parent and resolve $self->SUPER::method() to
that parent's method (strategy perl_method_super). No known parent or
unresolved method emits no edge (zero-edge guarantee preserved).
Tests: perllsp_cpan_exported_function, perllsp_super_dispatch,
perllsp_super_no_parent_no_edge, lsp_perl_deep_expression_no_crash,
lsp_resolve_qualified_static_call_normalizes_colons,
lsp_resolve_misattribution_is_bounded.
Signed-off-by: Shane McCarron <shane.mccarron@corvexconnect.com>
Distilled from #685 (nguyentamdat) rebased onto current main, plus two
research-driven refinements and a genuine reproduce-first guard.
The parallel extract retains each file's source text so the fused cross-file
LSP resolve can re-parse it. That retention is transient but a peak-RSS driver.
On main the caps are flat (100 MiB/file, 2 GiB total) and a file over the cap is
silently unretained AND its cross-file resolution is skipped -- a graph-quality
gap. This change bounds retention AND keeps every cross-file edge.
- Source-text cap as a FLOOR, not just a ceiling: retention total defaults to
min(cbm_mem_budget()/8, 1 GiB), per-file min(32 MiB, total). Following the
rust-analyzer memory model, the RAM-derived default is clamped to a small
absolute ceiling so a huge-RAM host does not hold tens of GB it would re-read
cheaply. Both caps env-overridable via CBM_RETAIN_TOTAL_MB /
CBM_RETAIN_PER_FILE_MB (limits.c convention); ceilings bound only the
auto-derived default, never an explicit operator/caller choice. A dropped file
emits one index.retain_capped WARN per run.
- Bounded re-read fallback (the correctness guarantee): resolve_worker re-reads
an unretained file's source on demand (bounded, freed immediately) instead of
skipping resolution, wired at every cross-LSP site that consumes source. The
cap now only trades retained RAM for a bounded re-read, never a lost edge.
- cbm_parallel_extract_ex + opts struct (cbm_parallel_extract is now a wrapper
passing NULL -> env-derived defaults); malloc/calloc NULL-check hardening.
Reproduce-first: parallel_cross_file_reread_preserves_unretained_edges uses a
Java<->Kotlin pair whose cross-file lsp edges are genuinely source-dependent;
the edges are lost when the caller is unretained and the fallback is absent
(RED), present with it (GREEN), with a retained CONTROL scenario proving
non-vacuity. #685's original Python red test was a false guard (per-file py_lsp
already resolves those calls) and is replaced. Peak-bound guards
(retained_bytes <= total_cap; retain_sources=false retains nothing) in test_mem.c.
Verify: make -f Makefile.cbm cbm && make -f Makefile.cbm lint-ci; test-runner
parallel pipeline incremental py_lsp ts_lsp java_lsp kotlin_lsp c_lsp cs_lsp
go_lsp rust_lsp mem -> 2323 passed.
Co-authored-by: nguyentamdat <nguyentamdat@gmail.com>
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Mixed Gradle/Maven source roots (src/main/java + src/main/kotlin) share
JVM packages that path-derived module QNs cannot join: cross-file LSP
emitted package-shaped names while graph nodes stayed path-derived, and
same-package defs across roots were filtered out, so Java<->Kotlin calls
fell back to weak or wrong textual edges.
The declared package is JVM ground truth. Normalize JVM defs to their
declared package (inferring it from conventional source roots only when
no package was extracted), index defs by declared namespace alongside
the path module so same-package defs survive per-file filtering, restrict
JVM callers to JVM defs, register cross-def return-type signatures for
Kotlin receiver typing, and consume qualified member-call overrides by
leaf name in the parallel lsp_idx fast path.
Distilled from #684 with one design change: the new unique-Class.method
tail-match fallbacks in lsp_resolve.h are gated to JVM callers
(cbm_pipeline_lsp_allow_tail_match, Java/Kotlin only). Tail-matching by
unique leaf is safe where class-per-file package semantics hold; in
other languages a single wrong-module coincidence would fabricate a
CALLS edge. The leaf-key lsp_idx lookup stays global: it aligns the
fast path with the build-side key that already used leaf names.
Tests: carried mixed Java/Kotlin regression (red on main, green here)
plus a gate guard asserting the tail fallbacks stay off for non-JVM
callers and on for JVM.
Co-authored-by: nguyentamdat <nguyentamdat@gmail.com>
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
A call carrying enough long arguments drove append_args_json()'s running
position past the fixed CBM_SZ_2K `props` stack buffer in
emit_normal_calls_edge(): format_call_arg() returns snprintf's *untruncated*
length, so `pos += (size_t)n` could exceed `bufsize`, after which the
trailing `buf[pos] = '\0'` (and `buf[pos++] = ']'`) wrote out of bounds. The
stack canary caught it as SIGABRT, so full-repo indexing of large TypeScript
codebases crashed the server in the parallel resolve pass
(emit_service_edge -> emit_normal_calls_edge -> finalize_and_emit ->
append_args_json). Confirmed with AddressSanitizer:
stack-buffer-overflow WRITE at pass_parallel.c:1124, 'props' (2048 B).
Fix: when an argument does not fully fit, roll back to before its separator
and stop appending (atomic field, matching append_json_string's behaviour),
so `pos` can never advance past the buffer.
Add regression test parallel_args_json_no_overflow: indexes a fixture whose
single call carries 60 long string args (args JSON well past 2 KB); under the
ASan test build it aborts without this fix and passes with it.
Signed-off-by: Andrius Skerla <1492322+rainder@users.noreply.github.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add scripts/check-no-test-skips.sh (run from lint) which fails the lint phase on any plain SKIP() or direct tf_skip_count manipulation; only SKIP_PLATFORM() (for genuinely platform-specific tests) is tolerated. Add FAIL() and SKIP_PLATFORM() helpers to the test framework and convert the remaining SKIP()/perf-gated skips across the suite into pass-or-fail assertions, so a suite that cannot meet its preconditions reports a red failure instead of a silent skip.
emit_grpc_edge produced two classes of bad output:
1. Wrong service name: the suffix list stripped "ServiceClient"/"ServiceGrpc"
before "Client"/"Grpc", so FooServiceClient collapsed to "Foo" instead of
the proto-declared "FooService" — breaking cross-repo Route matching.
2. Phantom Routes: extract_grpc_service_method returned true for any
"<recv>.<method>" call, so ordinary receiver vars (_provider.GetGroup,
_builder.AddSomeService) became __grpc__provider/... Routes matching no
.proto anywhere.
Strip only the trailing stub/client token (Client/Stub/Grpc/BlockingStub/…),
preserving "Service", and require that a recognized suffix was actually present
before emitting — the suffix match is the gRPC stub-type signal. Plain receiver
vars no longer yield Routes.
extract_grpc_service_method is no longer static (declared in pipeline_internal.h)
so it is unit-testable.
The gopls 'package summary' pattern, applied properly: build the cross-
LSP CBMTypeRegistry ONCE per language at pipeline time (in a dedicated
cross_lsp_arena), share it READ-ONLY across all resolve workers via the
CBMCrossLspRegistries struct. Per-file work then becomes just tree-
sitter parse + AST walk + O(1) hash lookups — zero per-file registry
rebuilds for languages with a pre-built registry.
Each LSP gains two new public functions:
CBMTypeRegistry *cbm_<lang>_build_cross_registry(arena, defs, def_count)
— filters all_defs to its language, init + stdlib + registration
loop + cbm_registry_finalize, returns finalized registry.
void cbm_run_<lang>_lsp_cross_with_registry(arena, source, ...,
CBMTypeRegistry *reg, ...)
— skips the per-file build AND the per-file Phase 1b/1c AST scans
(those mutate the registry, which would race on a shared read-
only reg). Trade-off: a few file-local type aliases discovered
only by Phase 1b are missed; in practice the per-file extract
LSP already captured them via embedded_types/field_defs/etc.
Tested with parallel_python_lsp_override_cross_file_emits_lsp_
strategy_edges — fused + filtered + prebuilt path stays correct.
CBMLSPDef gains a CBMLanguage lang field set in pxc_build_lsp_def so
the per-language build functions can filter the global all_defs[].
cbm_parallel_resolve signature now takes CBMCrossLspRegistries* instead
of a single CBMTypeRegistry*. The resolve_worker dispatches:
cbm_pxc_registry_for_lang(lang) returns the right pre-built reg if any;
the worker then switch()es to the matching _with_registry variant.
Languages without a _with_registry variant fall through to the existing
filter + per-file build path.
Kubernetes (Go-heavy, 12k files, 139k defs) measured perf:
parallel_resolve: 5.51 min (strdup-removal baseline) -> 1.86 min
= 2.96x speedup
parallel.resolve.lsp_cross.slow lines: 1,258 -> 2
(-99.8%; the 2 remaining are Python files, not Go)
Peak RSS: 6.29 GB -> 6.23 GB (slight decrease — pre-built registry
shared, no per-file scratch)
All 3,616 tests pass; zod/serilog graphs exact match (per-file path
unaffected).
Cumulative trajectory from the original broken state on kubernetes:
35 min -> 19 min (finalize) -> 6.24 min (module-def filter) ->
5.51 min (strdup-removal) -> 1.86 min (Tier 2 Go) = 18.8x total.
Languages wired so far: Go, Python. C/C++, TS/JS, PHP, C# still use the
per-file filter+build path until their _with_registry variants land in
follow-up commits.
Previously cbm_pipeline_pass_lsp_cross ran as a separate sequential
pass that re-read every source file from disk and re-parsed every
tree-sitter tree on a single thread — a 50x regression on the
kubernetes Go monorepo (extract 10s, lsp_cross 9+ min when killed).
Three changes:
1. CBMFileResult gains const char *source / int source_len. The
parallel extract worker copies source bytes into result->arena
(per-file 100MB cap, project-wide 2GB atomic cap) so the fused
cross-LSP step can run without disk I/O. Source pointer is
borrowed by downstream readers; arena owns the storage.
2. Cross-LSP work moves into the parallel resolve worker. Each worker
processes its file with the existing parallel infrastructure
(worker pool + work stealing). Slab is reclaimed after each
per-file cross-LSP call to bound peak slab memory. The previously
separate lsp_cross pass entry function is kept for the sequential
pipeline path; the parallel path no longer invokes it.
3. CBMModuleDefIndex (gopls 'package summary' pattern) — an inverted
index module_qn -> list of def indices built once in O(D) before
resolve_worker fires. Per file, cbm_pxc_filter_defs_for_file slices
all_defs down to the file's own_module + imported modules only,
typically 50-100x smaller than the global ~110k defs on kubernetes.
Drops the per-file registry build from O(all_defs) to O(relevant).
Other plumbing: cbm_parallel_resolve signature gains all_defs,
def_count, def_modules, module_def_index params. pipeline.c builds
the index between registry_build and parallel_resolve and frees it
after. pipeline_incremental and tests/test_parallel updated to match.
Helpers in pass_lsp_cross.c exposed via the header: cbm_pxc_has_cross_lsp,
cbm_pxc_collect_all_defs, cbm_pxc_run_one(_ts), cbm_pxc_ts_modes,
cbm_pxc_build_module_def_index, cbm_pxc_filter_defs_for_file.
Kubernetes wall-time: 35 min -> 6.24 min (~5.6x) before the followup
perf commits in this series.
Per-file LSP (run from cbm_extract_file) only sees one file's defs, so
callees whose receiver type comes from an imported module stayed
unresolved — even though Go/C/Py/TS already had cross-file entry points
(cbm_run_X_lsp_cross). Those entry points existed but no pipeline pass
ever invoked them. PHP didn't even have a cross-file entry point.
This change lands the missing pieces:
1. cbm_run_php_lsp_cross + cbm_batch_php_lsp_cross — PHP cross-file
resolver mirroring py_lsp_cross / ts_lsp_cross. Reuses PHPLSPContext,
php_lsp_init, php_lsp_add_use, php_lsp_process_file. Handles PHP-
specific labels (Class / Interface / Trait / Enum / Type) and uses
def_module_qn so cross-file return types qualify against the def's
own module rather than the importing module's.
2. pass_lsp_cross.c — new pipeline pass. For every cached file result,
builds a project-wide CBMLSPDef[] (CBMDefinition → CBMLSPDef
converter), per-file import map (from gbuf IMPORTS edges), then
dispatches to the matching cbm_run_X_lsp_cross. Output strings get
copied into the per-file arena (with O(N²) memory bug avoided by
running each file inside a fresh scratch arena that gets destroyed
after the LSP call — the LSP allocates its registry + stdlib +
project defs in scratch, not in the caller's persistent arena).
3. Wired into both pipelines:
- sequential: definitions → k8s → lsp_cross → calls → ...
- parallel: parallel_extract → registry_build → lsp_cross →
parallel_resolve → ...
4. Updated tests/test_parallel.c run_parallel helper to mirror the new
parallel ordering so existing cross-file Python/PHP tests now exercise
pass_lsp_cross.
Languages covered: Go, C/C++/CUDA, Python, TS/JS/JSX/TSX, PHP. PHP cross
file is new. The other four had functions but no pipeline integration
until this change.
Verified on a 1670-file FastAPI index: pass_lsp_cross processes 1050
Python files (525 non-LSP files skipped), collects 4070 project defs in
9.2s, and parallel_resolve subsequently reports lsp_overrides=568 — the
LSP-strategy attribution gain over per-file alone.
Per-LSP emit functions dedup by (caller_qn, callee_qn) within their own
run; pxc_append_results dedups again at append time so cross-file
entries duplicating per-file output get dropped instead of inflating
result->resolved_calls.
Per-file py_lsp emits resolved_calls.callee_qn as the raw import-module
path (e.g. `greeter.Greeter` from `from greeter import Greeter`) rather
than the project-qualified QN the gbuf stores (`<project>.greeter.Greeter`).
Before this change, the LSP match succeeded (lsp_overrides counter
incremented) but the downstream cbm_gbuf_find_by_qn lookup missed
silently and the edge was dropped — the failure mode the new
lsp_overrides telemetry was designed to surface.
The LSP layer can't tell in-project imports (qualify) from external
imports (don't qualify, e.g. `os.path`) without consulting the gbuf,
which is built downstream. So normalise at the consumer instead: try
the LSP-emitted QN as-is first; on miss, retry with
`<project>.<callee_qn>`. If that also misses, drop the edge — same as
before, target is external/unindexed.
New helper cbm_pipeline_lsp_target_node in lsp_resolve.h is shared by
both pipelines (sequential pass_calls.c and parallel pass_parallel.c)
so they continue to admit identical sets of LSP overrides. The helper
also canonicalises res.qualified_name to the gbuf node's QN, so
downstream edge property serialisation shows the project-qualified
form even when fallback resolution kicked in.
Cross-file regression test parallel_python_lsp_override_cross_file_-
emits_lsp_strategy_edges pins the two-file Greeter scenario that
originally exposed the bug.
The sequential pass_calls and the parallel cbm_parallel_extract pipelines
each carried their own copy of the LSP-resolved-call lookup with
divergent confidence floors and different match semantics. Most production
indexing runs the parallel path, so projects produced different
CALLS-edge attributions depending on which pipeline mode kicked in:
pass_calls.c find_lsp_resolution floor 0.6 (CBM_LSP_CONFIDENCE_FLOOR)
pass_parallel lsp_override_resolution_pp floor 0.5 (hardcoded)
Both pipelines correctly invoke cbm_run_*_lsp through cbm_extract_file,
so result->resolved_calls is populated identically — the bug was at the
consumer.
Changes
-------
- New src/pipeline/lsp_resolve.h: static-inline cbm_pipeline_find_lsp_resolution
and CBM_LSP_CONFIDENCE_FLOOR (0.6f). Single source of truth for the
match rule (caller_qn equality + callee_qn short-name match + floor),
with rationale and language-coverage notes inline.
- pass_calls.c: include lsp_resolve.h, drop the local copy, rename the
call site to cbm_pipeline_find_lsp_resolution.
- pass_parallel.c: include lsp_resolve.h, drop lsp_override_resolution_pp,
use cbm_pipeline_find_lsp_resolution at resolve_file_calls. Add an
int lsp_overrides counter to resolve_worker_state_t (with adjusted
cache-line padding for the new field), aggregate it per-worker,
and emit it in the parallel.resolve.done log line so divergence
between pipelines becomes observable in production telemetry.
- tests/test_parallel.c: parallel_python_lsp_override_emits_lsp_strategy_edges
builds a single-file Python scenario (Greeter class + main() in one
file), runs the parallel pipeline, walks every CALLS edge, asserts
at least one carries an "lsp_*" strategy in its properties_json.
Pins both halves of the wiring at once: the shared find helper IS
consulted in the parallel path, AND the LSP-attributed strategy
survives all the way to the gbuf edge. Single-file rather than
cross-file because per-file py_lsp's resolved_calls callee_qn uses
the import module path rather than the project-qualified QN, so
cross-file LSP edges currently get dropped by cbm_gbuf_find_by_qn —
a separate cross-file QN-mismatch bug to track elsewhere.
Full suite: 3250 passed, 0 failed (was 3249; +1 new regression test).
- cbm_mkdtemp: _mktemp + _mkdir on Windows
- cbm_setenv/cbm_unsetenv: _putenv_s on Windows
- cbm_pipe: _pipe on Windows
- Replace bare POSIX calls in 12 test files with compat wrappers
- All test files now include compat.h
Enforce WarningsAsErrors: '*' with zero suppressions of entire check categories.
All non-FP warnings fixed at source; true false positives get per-instance NOLINT
with reasoning. Fixes include: braces around statements (1679), isolate declarations
(75), explicit casts for multilevel pointer conversions, missing includes, function
parameter documentation via NOLINT, and proper NOLINTBEGIN/NOLINTEND nesting.
Complete C implementation of the indexing pipeline (src/), parallel worker
pool, graph buffer with merge support, SQLite writer, and 1893-test suite.
Linter setup: zero warnings from clang-tidy (all checks enabled, no NOLINT
suppressions), cppcheck, and clang-format. All issues fixed at source —
proper headers for external linkage, GROW_ARRAY macro restructured to
eliminate type parameter, null-deref paths guarded, named intermediates
for suspicious-argument checks.
Martin Vogel