test/1234-java-interface-enum-dedup
2036 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Martin Vogel
|
5eae0a4371 |
test(extract): bind the Java interface/enum method dedup fix (#1234)
Regression coverage for #1234: Java interface and enum methods were
emitted as both a Method node and a duplicate top-level Function node.
The production fix landed via
|
||
|
Martin Vogel
|
66e9aec28d |
Merge pull request #1392 from DeusData/feat/generated-client-adapters
feat(install): generate pi and OpenCode extensions from the tool registry |
||
|
Martin Vogel
|
297d747568 |
Merge pull request #1166 from DeusData/qa/linkedin-call-usage-repros
fix: preserve exact semantic graph relationships |
||
|
Martin Vogel
|
d6c8d1dd0a |
fix(msan): split the known-red block by cause; fix the RSS one properly
The x86-64 leg runs this lane without exclusions on purpose, to settle
which limits are architectural. It has now run, and it disproves part of
what the previous block asserted. That block claimed all seven excluded
suites "abort with stack-overflow". Five do. Two do not, and lumping
them together hid two different problems behind one rationale.
(A) stack-overflow, five suites: grammar_regression grammar_labels
pipeline lang_contract grammar_probe_e. Confirmed on BOTH arm64 and
x86-64 (CI logged 5), so it is not the aarch64 artifact an earlier
note claimed. The recursion guards bound DEPTH while the resource
exhausted is BYTES; that follow-up stands unchanged.
(B) cli: no overflow at all. On x86-64 it runs to completion, 253
passed / 5 failed, every failure in the install or activation path,
with "agent_config agent=OpenClaw op=mcp_install" above them. Green
on every other venue. MSan reported zero use-of-uninitialized-value
in it, so the exclusion costs no uninit coverage. Recorded as
undiagnosed rather than guessed at: the local lane is arm64 where
these suites hit (A) before reaching this code, so there is no
faithful venue to iterate in and each attempt is a ~30min round
trip. That is a follow-up with an owner, not a dismissal.
(C) incremental: an RSS BUDGET failure, 3054MB against a 2304MB limit
-- not an overflow either. MSan maps shadow (and origin) memory for
every allocation, so the budget cannot separate a leak from shadow.
FIXED rather than excluded: the assertion is now skipped under
__has_feature(memory_sanitizer) only, so the guard keeps its teeth
on every other platform, where inflating the budget would have
blinded it. The suite stays IN the lane.
Verified: with (C) fixed, incremental is 163 passed / 0 failed and ZERO
stack-overflows under the local arm64 MSan container -- so it never
belonged in the overflow list on either architecture.
msan-lane.sh no longer forces MSAN_EXCLUDE empty. That override existed
to ask the architectural question; it is answered, and keeping it would
re-red the gate for causes already recorded. Both venues now read the one
authoritative list in scripts/msan.sh, which still warns loudly that the
lane is partial.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
|
||
|
Martin Vogel
|
96a25b293e |
fix(test-harness): run extraction in the quiet tail, not the 18-job wave
The wide-flat SCALING-RATIO guard grows the input 20x and asserts the
time grows ~20x (linear) rather than ~128x (quadratic), with the bound at
40x between them. Contention does not cancel out of that ratio: the
400k-node measurement loses far more to memory pressure and scheduling
than the 20k one, so oversubscription inflates the ratio itself.
Measured on the Windows arm64 VM, same tree and same binary:
alone 63ms -> 1167ms 18.5x passes
in the 18-job wave 168ms -> 9045ms 53.8x fails
163ms -> 9019ms 55.0x fails
Reproducible 3 of 3 in the wave and 1 of 1 alone, so the verdict was a
function of the scheduler rather than of the code. The suite is ~22s;
running it alone is cheap next to a ~40min ladder.
The bound is deliberately NOT widened. The calibration note in
tests/test_extraction.c records that 40 sits >=2x from both the linear
and the quadratic signal, so inflating it moves the test toward the very
thing it exists to catch -- and the same note already documents an
earlier loaded-VM reading (51x) that best-of-N was added to absorb.
Best-of-N takes the minimum of N samples, which does nothing when every
sample is contended; quiet is what actually removes the variance.
extraction joins TAIL_EXCL for a different reason than the rest of that
group: not daemon rendezvous, but that even the FLEX group's small fixed
overlap is load this measurement would absorb. Both comments say so.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
|
||
|
Martin Vogel
|
db91b88cc4 |
fix(test): stop asserting a scheduling race in the forged-identity test
daemon_runtime_rejects_forged_identity_extension failed on the macos-15-intel CI leg. The suite is untouched by this branch and no daemon production code changed here; the leg flakes on main too, so this is a pre-existing defect the lane surfaced rather than a regression. The test asserted that transmitting the forged frame SUCCEEDS. It cannot be relied on to. The forged HELLO is 149 bytes against the 137-byte first-frame envelope cap, so the worker rejects it from the header and closes without ever reading the payload -- deliberately, so that no attacker-controlled bytes are read. send_frame writes the header and the payload as two separate writes, so whether the payload write lands before that close is pure scheduling: it wins on an idle host and loses on a loaded 4-vCPU runner. Both outcomes ARE the rejection, so the transmit result is now recorded and not asserted. Anti-vacuousness is preserved rather than dropped: the test now asserts it connected at all, and the drain check waits for the state it asserts (wait_for_clients 0) instead of sampling the count once, so the bound is a liveness backstop and never the verdict. Verified by mutation, not just by passing: with the envelope cap raised to 512 and the exact-length check removed -- a daemon that accepts the forged identity -- the repaired test goes RED on ASSERT(rejected). The mutation was reverted and the suite is 43/43 green. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
Martin Vogel
|
b3fb6689c9 |
fix(msan): whitelist the deep-recursion suites; correct the lane's diagnosis
The MSan lane gates CI and has never been green: seven suites abort with a stack-overflow inside MSan's memset interceptor on a worker thread. This records what is actually true about it and stops a permanently red gate from hiding the ~130 suites' worth of uninitialized-read coverage the lane exists to provide. Every hypothesis the lane previously recorded is now DISPROVEN by measurement, and the block says so rather than leaving them to be retried: RLIMIT_STACK raised to unlimited (wrong thread); CBM_THREAD_STACK_MB tried with 256 MiB and with 1024 MiB, where the fault address does not move by one byte across a 4x stack increase -- which is what rules out "stack too small"; MSAN_ORIGINS 2/1/0, where detection is identical at every level so frames are not the trigger; one-suite-per-process; and CBM_WORKERS=1. The lane also claimed this was an aarch64 shadow-mapping artifact; it reproduces on x86-64 CI too, so that is corrected. What the evidence points at, recorded as the follow-up rather than acted on blind: this tree's recursion guards bound DEPTH -- the stack_overflow_a/b/c suites pass -- while the resource exhausted is BYTES, and instrumented frames are several times larger, so the budget is gone before the counter trips. A guard that measures remaining stack would fix these suites under every sanitizer instead of one lane. The exclusion is by name, narrow, and expires with that fix. Verified: with those seven skipped, every remaining MSan suite passes. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
Martin Vogel
|
b33a4de9bc |
fix(build): keep alignment checking off vendored tre.o; make SANITIZE rebuild
Two defects, one found by the other. The crash: every fixture-indexing suite (mcp, incremental, index_resilience) died on the Windows/ARM64 leg with STATUS_ILLEGAL_INSTRUCTION and no diagnostic. Cause: an earlier commit in this branch re-armed UBSan's alignment check on the vendored TRE regex engine, and CLANGARM64 builds with -fsanitize-trap, where a trap IS an illegal instruction with nothing printed. macOS and Linux stay clean under the same check because Windows is LLP64 -- 32-bit long -- so TRE's struct layouts and access widths differ there and only there. TRE is vendored third-party code we do not modify, so suppressing the check for that one object is the honest scope; every other object keeps alignment armed. Verified by bisect: origin/main, |
||
|
Martin Vogel
|
abfac068df |
perf(pipeline): narrow the delta proxy set; map unproxied symbols at patch time
Two changes that move the delta path toward O(change) without touching the parallel-resolve contract. Proxy narrowing. The delta executor pre-loaded every project node so resolution could find cross-file targets by qualified name. Nodes that resolution can never look up are pure load cost, and on the kernel they dominate: six of its 8.5M nodes are Macro. The set is narrowed by EXCLUSION rather than an inclusion list, deliberately -- an earlier inclusion list was disproven by counterexamples it did not anticipate (synthetic Decorator nodes, then Macro), and excluding a short list of labels that are never lookup targets fails safe where guessing the full inclusion set did not. Patch-time identity mapping. A resolver that upserts a symbol the narrowed set did not pre-load now produces a stand-in node; the patch maps it back onto its existing row by qualified name instead of raising the UNIQUE violation the previous patch would have. Nodes from CHANGED files cannot collide here -- the purge removed them -- so repaired files still receive fresh rows. The map is a sorted array searched by bisection, not a CBMHashTable: that table stores key POINTERS without copying them, which a stack-formatted integer key cannot satisfy. NOT attempted here, and recorded instead: making the proxy load itself lazy. cbm_parallel_resolve documents main_gbuf as READ-ONLY during its worker phase and its workers do call cbm_gbuf_find_by_qn on it, so a find-time materializer would mutate a buffer under concurrent readers. A safe version needs the materialization hoisted ahead of the worker phase; that is a separate change with its own verification. Kernel one-file warm: 50.8s -> 42.7s wall, peak RSS 13.2 -> 10.8GB, proxies 8.5M -> 2.5M (preseed 18.3s -> 9.7s), and the run reports remapped=0 -- no resolver needed a symbol the narrowing dropped. 535 pipeline/incremental/store/cross-repo/integration tests green, including the full convergence matrix. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
Martin Vogel
|
708603a248 |
fix(pipeline): release the package map the closure probe builds
closure_probe_surfaces runs cbm_parallel_extract to compute the changed files' fresh surfaces, and parallel extraction builds the process-global package map as a side effect. Both real extraction paths release it at an explicit ownership boundary; the probe borrowed the machinery without inheriting that contract, leaking one map per probed run. Found by the macOS leak lane added earlier in this branch -- the lane catching a defect introduced after it, which is the point of having it. Verified: the incremental suite is clean under LSan with the fix, and the same allocation site (cbm_pkgmap_build via merge_pkg_entries) no longer appears. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
Martin Vogel
|
a33a82f3e0 |
perf(pipeline): parallel rehydration; skip the outgoing quick_check on the delta route
Two more measured blocks out of the delta path, closing the optimization arc: Parallel rehydration. The base-def decode (2.37M defs from ~89k surface rows at kernel scale) fans out across workers with per-worker arenas that live exactly as long as the resolve borrows them; assembly stays in row order, so the registration input is byte-identical to the serial loop's. Measured 315ms for the block; the arc also disproved an earlier attribution -- the planner including its full surface load is 608ms, no projection needed. Known-healthy finalize. prepare_existing_generation_for_replace runs PRAGMA quick_check over the ENTIRE outgoing generation to choose replace-vs-quarantine -- 35.5s of full-database page scan at kernel scale. The delta route cloned that same file and ran complete transactions against the clone minutes earlier; a corrupt live database cannot reach the delta finalize because every earlier step fails it into the dump path, whose finalize keeps the check and the quarantine semantics unchanged (as its corruption tests continue to prove). Sidecars are still removed on the fast path -- a replaced database must never inherit the old generation's WAL. Delta publish total: 35.7s -> 224ms. finalize/publish timing brackets stay as durable telemetry. Kernel one-file warm across today's arc: 306s (binary routing) -> 223s (closure via dump) -> 121.7s (delta) -> 85.3s (query plans + shadow gate) -> 50.8s wall / 34.3s worker; peak RSS 33.6 -> 13.2GB. Remaining named blocks: proxy preseed 18.3s and repair/mdi 11.1s, both serial-bound gbuf/index builds -- recorded follow-ups, not mysteries. 437 pipeline/incremental tests green throughout. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
Martin Vogel
|
4e99ddfb4a |
chore(store): strip the coverage-timing diagnostic scaffolding
The cov_timing_mark instrumentation existed to locate one block (it found the shadow rebuild); the previous commit shipped with it still in place, including a call after a return that cppcheck rightly flagged as unreachable — that commit went out with the lint gate RED because the push was chained without depending on the gate result. The scaffolding is gone; the durable publish.timing brackets in pipeline.c remain. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
Martin Vogel
|
87eafefe73 |
perf(pipeline): pin dependency-query join order; gate the coverage shadow rebuild
Two measured pathologies out of the delta path, both shared-code fixes that help every route: Join order. The inbound-edge snapshot and the dependent-files lookup let the planner start from EDGES, walking every project edge through the url_path index prefix — 14.6s for a one-file closure against the kernel's 16.5M edges. CROSS JOIN pins nodes-first (idx_nodes_file → idx_edges_target → primary key): measured 4ms for the same query, 23ms for the full snapshot+purge step. Coverage shadow graph. cbm_store_coverage_replace_ex rebuilt the miss-graph shadow view wholesale inside every publish — wipe plus tens of thousands of node/edge upserts probing the full-size nodes index, 23.4s at kernel scale — even when the failure-row set it derives from was byte-identical. The rebuild is now gated on a sha256 fingerprint of the failure rows persisted in store_meta: unchanged set, provable no-op, skipped. Measured 84ms steady-state; the rebuild still fires whenever the set actually changes, and the shadow output is untouched. The benchmark probe also stops appending a trailing comment that happened to break bootp.c's parse — a probe that mutates the failure set on every run measures the shadow rebuild, not the repair. It now edits inside the license-header comment. Kernel one-file warm, steady state: 111.6s -> 85.3s wall (71.1s worker). Remaining measured blocks: base-def rehydration ~35s, preseed 18.8s, repair 10.8s — the parallelization targets. 554 store/pipeline/ incremental tests green. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
Martin Vogel
|
6ec9e944e5 |
perf(pipeline): delta fixes from scale testing — global watermark, full preseed, in-place surfaces
Three defects the kernel and django corpora exposed in the delta executor, each caught by its own fail-closed design and each fixed at the root: Global id watermark. MAX(id) was project-scoped while node ids are one keyspace for the whole database; a fresh node collided with a row outside the project filter (UNIQUE nodes.id on django). The watermark now clears every row. Full preseed. The label-filtered proxy set immediately met its counterexamples: synthetic Decorator nodes on django failed the patch via the QN constraint, and Macro -- six million of the kernel's 8.5M nodes -- was absent entirely, which would have silently dropped cross-file macro edges rather than failing. Curating an edge-endpoint-label list is guessing; every project node is now a proxy. The load stays edge-free and property-free, which is where the old full load actually spent its time (preseed measures 14.1s against the 38.3s gbuf load it replaces, plus that load's 16.5M edges). In-place surfaces. publish rewrote every lsp_surface row on each delta (delete-all plus re-upsert of ~89k serialized def sets); the patch now deletes exactly the purged files' rows and upserts the repaired files' fresh ones inside its own transaction, and publish skips the wholesale rewrite behind generation->surfaces_in_place. Measured 0.12s for the whole write block at kernel scale. publish_staged gains per-block timing logs; they located the next optimization targets precisely (23.8s in the meta/coverage section at kernel scale, integrity and seal effectively free). Measured end-to-end on the kernel corpus: one-file warm reindex 306s (binary routing) -> 223s (closure via dump) -> 121.7s (delta), peak RSS 33.6 -> 21.5 -> 13.6GB. django delta repair: 1.6s worker time. Convergence suite and 437 pipeline/incremental tests green throughout. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
Martin Vogel
|
bbd13335f5 |
feat(pipeline): delta-merge executor for the closure route
The closure route stops loading and rewriting the world. Its executor is now a dedicated subsystem (pipeline_delta.c + orchestration): CLONE the live generation (copy-on-write where the filesystem offers it), repair the closure against the clone, PATCH exactly the repaired node/edge set in one transaction, and publish through the same sealed-staging finalize leg as the dump path. No full graph load, no full dump, and the general indexing pipeline is untouched -- the profiled kernel run put those two at 187s of a 238s one-file repair whose actual resolution work was 0.6s. Id discipline carries the design. Node ids are AUTOINCREMENT and never reused; the small in-RAM graph is pre-seeded with PROXY nodes carrying their real database ids (SELECT ... ORDER BY id with the id watermark pinned before each insert), and fresh nodes are numbered above the previous generation's MAX(id) -- so "id > max_db_id" is the complete, marker-free definition of what the patch inserts, and every edge endpoint id is database-valid by construction. The inbound-edge snapshot and its QN-keyed re-link become indexed SQL; a re-link whose endpoint no longer exists matches no row, which is full-reindex semantics for deleted symbols. Fail-closed throughout: an unexpected reference to an unseeded label surfaces as a UNIQUE-constraint violation that fails the patch transaction, and EVERY delta failure discards the stage and returns FORCE_FULL_REINDEX -- the live database is never touched, so a full rebuild always self-heals whatever the delta could not do. FTS policy: nodes_fts is contentless, so purged rows cannot be deleted individually on existing databases; their rowids can never alias a live node again (AUTOINCREMENT) and dead entries drop out of the rowid join. The patch inserts rows for exactly the new nodes through the same cbm_camel_split tokenizer the wholesale rebuild uses. The legacy gbuf-based tail reverts to serving only the test-only force_legacy_partial route; the closure orchestration owns its own coverage merge, publication race gate, surface-row merge and committed counts, and publishes with fts_wholesale=false. Gate: the full convergence suite runs against this executor unchanged -- body-edit graph equality with a fresh full index, removed-definition dropping the dependent's stale edge, tsconfig-alias retargeting, the decline matrix, and 510 pipeline/incremental/store/integration tests. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
Martin Vogel
|
5722231ead |
feat(pipeline): delta-repair groundwork — CoW staging + shared finalizer
Two foundations for the delta-merge incremental subsystem (a dedicated copy->patch->rename executor for the closure route; the general dump pipeline is untouched): cbm_clone_or_copy_file (foundation/compat_fs): stage a database by copy-on-write clone where the filesystem has one -- clonefile(2) on APFS, FICLONE on Linux reflink filesystems -- with a streamed copy as the portable fallback. Verified byte-identical and write-independent. For a multi-GB generation this is the difference between milliseconds and seconds of staging cost. cbm_pipeline_finalize_staged_generation: the final leg of publication (sidecar removal, previous-generation quarantine, atomic rename with rollback on every failure) extracted, behavior-preserving, from cbm_pipeline_publish_generation so a patched staging copy can publish through the exact same crash-safety tail as a dump-built one. The FTS-rebuild and integrity-check policy deliberately stays OUTSIDE the shared tail: the dump path rebuilds wholesale, while the delta path will write row-level FTS inserts (safe against stale entries because node ids are AUTOINCREMENT and never reused, so dead rowids drop out of the join). All 437 pipeline/incremental/publication tests green, unchanged. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
Martin Vogel
|
bd1d2e9528 |
perf(pipeline): parallelize semantic-manifest hashing
The manifest's per-file sha256 loop was single-threaded -- tens of thousands of file reads in sequence, the second-largest block of a kernel-scale incremental run after publication (~20s by residual). The hash helper is pure per-file work, so files now fan out across cbm_default_worker_count workers on a stride; ASSEMBLY stays serial and in discovery order, so the manifest bytes are identical to the serial build's -- the exactness doctrine is untouched, only the wall clock moves. Repos under 64 files keep the serial path outright. A worker that fails to spawn leaves its stride to the calling thread, so every index is hashed exactly once regardless of thread-creation failures. Pinned by a threshold-crossing test: a 72-file repo must route NOOP on unchanged bytes -- which stands entirely on two parallel builds producing byte-identical manifests -- and still classify a single edit into the closure route. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
Martin Vogel
|
d8fffd0460 |
fix(pipeline): split the closure_plan log line over the itoa ring capacity
itoa_buf recycles a four-slot thread-local ring; the closure_plan line passed six conversions in one call, so two fields printed corrupted -- the kernel-scale profile showed surface_changed reporting the elapsed-ms value. Split into two calls of at most four conversions each. The same profile run, for the record, answered the incremental cost question with data (kernel corpus, one-file closure, worker total 238.5s): dump/publish 149.1s (62.5%), graph load 38.3s (16.1%), wholesale semantic-edges post-pass 18.1s, manifest hashing ~20s by residual -- while the repair itself (extract + resolve + registry rehydration) is 0.57s. The closure algorithm is effectively free at every scale; the remaining cost is generational I/O, which is the delta-merge (copy -> patch -> rename) follow-up's target. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
Martin Vogel
|
1ef7b026c9 |
perf(pipeline): skip Tier-2 registry prebuild for floor-sized closures
The shared cross registries are an amortization: one build over every def so that tens of thousands of per-file resolves become O(1). A floor-sized closure resolves at most eight files, so the build can never pay for itself; those files take the per-file fallback path filtered through module_def_index -- the same pre-Tier-2 resolution code the full pipeline still uses for languages without a shared registry -- so convergence is unchanged, as the routing-matrix tests confirm on both paths. Measured honestly: on the C-heavy kernel corpus this is timing-neutral (223.3s vs 227.1s warm, within noise) -- C's registry build is not where that corpus spends its time. The guard is kept on the strength of the recorded registry-build pathologies (the symfony 416s and elasticsearch 647s classes were exactly shared-registry construction), which hit Python/TS-heavy corpora far harder than C. Kernel A/B after the closure route (M4, torvalds/linux shallow, prod binaries, VMs down): branch warm falls 305s -> 223-227s at warm/cold 0.73 vs main's 0.60; the residual gap over main decomposes as ~50s of pre-existing branch overhead present in cold since before closure existed, plus ~20s of closure machinery. Peak warm RSS across the process tree is 21.5GB, BELOW the ~33.6GB cold peak -- the closure path adds no memory regression. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
Martin Vogel
|
45c40ed3a6 |
feat(pipeline): alias-config governance for closure repair
Path-alias configs (tsconfig/jsconfig class) stop declining the closure route. A config delta re-routes RESOLUTION for the files it governs while touching none of their bytes, so those files join the closure directly: they re-extract and re-resolve under the freshly loaded alias collection, their surfaces come out unchanged, and propagation stops -- the depth-1 argument holds exactly as it does for source edits. Governed means every discovered file under the config's directory; over-inclusion from nested scopes is deliberate (safe direction), and the existing budget still bounds the total, so a root config on a large repo correctly concedes to a full rebuild. Classification is now explicit rather than incidental: synthetic manifest digests (git context, extension configs) decline as semantic_input_changed; package-control files decline as control_file_changed (pkgmap is global -- governed repair is unsound there); alias configs -- recognized by exact match against the loaded collection plus a basename fallback so a REMOVED config still classifies -- seed the governed closure, whether changed, added, or removed. The existing tsconfig-alias convergence test becomes the proof: no source file changes, the route asserts CLOSURE_REPAIR, and the caller's CALL_REFERENCE must move from target_a.ts to target_b.ts to match the fresh-full reference -- the exact case the legacy partial route silently corrupted and binary routing paid a full rebuild for. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
Martin Vogel
|
bc3cc3f49f |
fix(msan): shard the lane one suite per process, origins default 1
With the zstd feature-macro fix in, the x86-64 leg finally RUNS -- and thousands of tests pass before the known deep-recursion stack overflow lands in grammar_regression, the same signature as the local arm64 wall. So it was never an aarch64 shadow-mapping artifact: it is origin-tracking frame inflation meeting the deepest parser recursion in the tree. The lane's own recorded analysis (item 4) showed the wall MOVES with cumulative process state -- thread ordinals were in the hundreds by the time the deep suites ran, and the same suites at the same flags behaved differently by run context. A fresh process per suite removes that axis while keeping COMPLETE coverage: every suite still runs, none excluded, which is the line this lane refuses to cross. Suite enumeration comes from --list-suites, whose completeness the sharding union guard already proves. Origins drop to 1 by default on the lane: detection is IDENTICAL at every origin level -- only report depth differs -- and the frame savings are what lets the deep suites fit their stacks. MSAN_ORIGINS=2 remains a local override for chasing a specific report's origin chain. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
Martin Vogel
|
f07c2e2f34 |
feat(pipeline): closure-repair incremental route
The routing follow-through on the +92% warm-reindex finding: a semantic manifest delta no longer unconditionally rebuilds the world. The planner recomputes exactly the changed files plus the recorded consumers of any changed SURFACE, and the executor resolves them against cross registries rehydrated from the persisted per-file surfaces -- the same registration code a full build feeds from fresh parses, which is what makes the output converge instead of drift. Routing, in order: exact manifest match stays a no-op; a delta first offers itself to the closure planner; every uncertain case declines to the full rebuild that was yesterday's only behaviour. Declines: virtual/ config manifest entries, new files, ADDED definition names (yesterday's graph cannot know who would resolve to a name that did not exist -- the write-the-caller-first flow and shadowing both live here), missing or undecodable surface rows, dependents outside discovery, and a budget of 30% of files with an 8-file floor (a percentage alone starves small repos: 1 changed file in 3 is 33%). Two structural facts carry the correctness argument. Per-file extraction is a pure function of file content, so an unchanged dependent can never be surface-changed in turn -- the closure is depth-1 by construction, no fixpoint. And a body edit reserializes to the identical surface bytes, so its closure is the file itself. The dependent set comes from one indexed query over the previous generation's edges (structural Folder/Project containment excluded -- a container is not a consumer). The executor is the existing partial machinery, parameterized: re-parse list = closure; inbound-edge snapshot/re-link keeps only sources OUTSIDE the closure (sound because every referencer of a surface-changed file is inside it by construction); cbm_parallel_resolve now receives real cross registries built from stored-surface defs plus this run's fresh parses; publication merges surviving surface rows with the re-parsed files' fresh ones inside the same generation. The legacy test-only route publishes no surface rows at all -- a stale row that satisfies a future closure plan with yesterday's surface would be worse than the full rebuild an empty table forces. Tests pin route AND convergence together (route equality matters because a full rebuild satisfies any convergence assertion vacuously): body edit routes CLOSURE_REPAIR with node/edge/CALL_REFERENCE counts equal to a fresh full index; REMOVING a definition keeps the closure route and drops the dependent's stale CALL_REFERENCE -- the assertion the legacy QN-keyed re-link could never pass; added-name, new-file and budget cases decline; the existing Go content-change test now routes CLOSURE_REPAIR with its convergence assertions unchanged, making it the Go-language proof of the same machinery. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
Martin Vogel
|
a4cef7f922 |
fix(msan): pin _GNU_SOURCE for the zstd object alongside the forced stdint
Scoping -include stdint.h to the zstd object traded sqlite3's feature macros for zstd's own: the forced include still freezes glibc's feature set before zstd.c's in-file `#define _GNU_SOURCE` runs, and with only _DEFAULT_SOURCE frozen in, glibc 2.39 does not declare qsort_r -- zstd.c:47409 fails exactly as the x86-64 leg reported. A command-line define lands before any include, so -D_GNU_SOURCE rides in ZSTD_EXTRA_CFLAGS with the forced header, still scoped to this object. Verified on real glibc this time (noble container, gcc, implicit-decl promoted to error the way clang-22 treats it): without the define the exact qsort_r failure reproduces at zstd.c:47409; with it the file is clean. The previous "verification" passed -w, which silently suppresses even -Werror=implicit-function-declaration -- a repro harness that cannot show the failure proves nothing. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
Martin Vogel
|
94189e6990 |
feat(pipeline): serialize and publish per-file LSP surfaces
Second piece of closure repair. At the collect_all_defs seam -- the only moment the per-file result cache is alive -- both drivers (parallel and sequential) now serialize each file's CBMLSPDef slice to canonical JSON, hash it, and hand the rows to the pipeline; cbm_pipeline_publish_generation writes them into the staging store next to the manifest, so surface data and graph always belong to the same generation. Canonical bytes are the point: every field is written in fixed order with an explicit null for absent strings (NULL and "" differ in the CBMLSPDef contract -- receiver_type NULL means "not a method"), so byte equality IS surface equality and the sha over the bytes is the early-cutoff key. Registry-only labels that pxc_map_label drops but the name registry serves (Field) are folded into the hash as a separate "reg" array, or renaming one would slip past the cutoff. Behaviour pinned in SUITE(pipeline): a fresh full index persists a versioned surface row per file; a BODY edit republishes the identical surface_sha; a SIGNATURE edit changes it. That pair of properties is what the routing layer will stand on. cbm_pxc_collect_all_defs gains an optional per-file prefix array -- the flat all_defs[] otherwise loses the file boundaries the serializer needs. CORRECTION to 6c22338's scope note: it claimed cbm_pipeline_publish_ generation was reachable only behind CBM_INCREMENTAL_TEST_API. Wrong -- dump_and_persist_hashes calls it on every production full index (pipeline.c:1863); the grep that "verified" test-only reachability had excluded pipeline.c itself. The predictable staging name WAS in the production publish path, which makes that fix a real production hardening, not a test-path cleanup. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
Martin Vogel
|
8d3685bfca |
feat(store): persist per-file LSP surfaces for closure-repair incremental
First piece of the closure-repair incremental route (the follow-through on the +92% warm-reindex finding): a per-file lsp_surface row holding the file's serialized cross-file definition set -- exactly what pass_lsp_cross registration consumes -- plus the metadata the routing decision needs: the surface sha (early-cutoff key: a body edit leaves it unchanged, so no dependent recomputation is owed), a referenced-name bloom (added-symbol trigger), and a governing-config context hash. The store treats defs_json and the bloom as opaque; the codec lives with pass_lsp_cross, which is their only writer and reader. A project with no rows reads back as OK/0 -- callers treat that as "no surface data" and route to a full rebuild, which is also how databases written before this table existed upgrade themselves. Table appears via the CREATE IF NOT EXISTS schema on store open, so the raw dump writer needs no change: publication opens the staging DB with the store right after the dump, which applies the schema. Round-trip covered in store_nodes: batch upsert, ordering, binary bloom with embedded NUL, NULL bloom, whole-row conflict replacement including bloom removal, project-scoped delete, and the empty-project signal. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
Martin Vogel
|
6a0adb4430 |
fix(msan): scope the zstd stdint workaround to the zstd object
The previous fix put -include stdint.h on the lane's global SANITIZE line. That traded one vendored compile break for another: force-including a libc header ahead of every source file freezes glibc's feature-test macros before sqlite3.c can set _GNU_SOURCE for itself, and its view of libc loses MREMAP_MAYMOVE and nanosleep (17 errors on the x86-64 CI leg). The workaround only ever had one legitimate target -- the zstd amalgamation whose MEMORY_SANITIZER block lost its stdint re-include -- so it now rides a per-object hook (ZSTD_EXTRA_CFLAGS) that the MSan lane sets and every other build leaves empty. sqlite3.c compiles exactly as before in every lane. Verified locally that zstd compiles with the hook and the default rule stays untouched; the MSan leg itself is x86-64-only, so CI is its verification venue. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
Martin Vogel
|
5adcfd296c |
style: drop the stray blank line before the staging-path declaration
clang-format violation at pipeline.c:1430 from the forward declaration
added in
|
||
|
Martin Vogel
|
7d390c5120 |
fix(ci): unbreak the MSan and diag lanes on x86-64
MSan: vendored zstd fails to compile. Its MSan-only block (guarded by
MEMORY_SANITIZER) declares __msan_test_shadow returning intptr_t and
reaches for the type with
#define ZSTD_DEPS_NEED_STDINT
#include "zstd_deps.h"
but the amalgamator that produced zstd.c collapsed that second include
into a "skipping file" comment, so the define pulls nothing in and
intptr_t is undeclared. Only this lane compiles that block at all, and
only where <stddef.h> does not drag stdint.h in transitively -- which is
why it built on the local aarch64 container and failed on CI's x86-64.
The lane now forces the header. Patching the vendored amalgamation would
be silently undone by the next re-vendor.
diag: detect_invalid_pointer_pairs comes back out. It fires during static
initialisation inside vendored simplecpp -- a std::string global at
simplecpp.cpp:101 -- with a second "pointer" of 0xfffffffffffffff3, a
sentinel rather than an address: libstdc++ string internals, not
anything this codebase wrote. It is a process-wide runtime flag with no
per-file scoping, so unlike the analyzer's path filter it cannot be
aimed away from vendored code. Keeping it would mean a permanently red
lane reporting a non-defect, which is how a lane gets ignored. The
instrumentation it needed comes out with it.
The other three off-by-default checks stay: stack-use-after-return,
stack-use-after-scope, strict-string-checks. Those are the ones covering
bug classes nothing else in the matrix looks for, and none of them
fired.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
|
||
|
Martin Vogel
|
6c22338b19 |
fix(pipeline): create the publication staging file exclusively
cbm_pipeline_publish_generation built its staging database name by hand as "<db>.stage.<pid>.<counter>", unlinked it, then wrote it. Any local process can compute that name in advance, so a symlink planted between the unlink and the write redirects the write to a target of the attacker's choosing — an arbitrary-file clobber when the database sits in a world-writable directory. The same file already solves this correctly elsewhere: create_staging_path() mints the name with mkstemp, so the file is created O_EXCL and we only ever write one we made ourselves. Publication now shares it. The unlink-first step goes away with the predictable name — it existed to clear a leftover at a name we might reuse, and a freshly minted name cannot collide, nor can its sidecars pre-exist. SCOPE, stated precisely because the PR description overstates it: the only caller of cbm_pipeline_publish_generation sits behind CBM_INCREMENTAL_TEST_API, which is set in CFLAGS_TEST and never in CFLAGS_PROD. The predictable name was therefore not reachable in a shipped binary — production publication already went through create_staging_path. This is removing a bad pattern from a test-only path before it can be promoted, not patching a live user-facing vulnerability. The regression test calls the function directly, because no pipeline entry point reaches it in a production build. It does not try to win the race — a test that has to win a race is a coin flip, not a gate. It asserts the property that removes the race: canaries occupy every name the old scheme could have chosen and all must survive publication. Verified both ways rather than green-only: against the old code it reports "survived == 31, expected PREDICTABLE_CANARIES == 32", exactly one canary consumed; with the fix the suite goes 18 passed/1 failed -> 19 passed. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
Martin Vogel
|
64bd272cb2 |
fix(ci): mark the new lane scripts executable, and let the contract see them
scripts/ci/lint-mem.sh and scripts/ci/msan-lane.sh were committed at mode 100644, so the workflow step that runs them directly died with "Permission denied" (exit 126). scripts/lint-mem-gate.py gets the same treatment: it is invoked through python3 today, but it carries a shebang and should not depend on that. The exec-bit contract already exists to catch precisely this, and it did not, because it derives its candidate set from `git ls-files -s '*.sh'` -- tracked files only. A brand-new script is invisible there until it is committed, so the check passes on the run where the defect is introduced and only starts failing on the run that ships it. The window where the contract is most useful was the one window it could not see. It now also considers not-yet-tracked scripts by their filesystem mode. Verified against the real defect rather than in the abstract: with lint-mem.sh untracked and non-executable the contract reports "_lint.yml:76 executes scripts/ci/lint-mem.sh directly, but its committed mode is 100644", and passes once the bit is set. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
Martin Vogel
|
9a41d83158 |
fix(log): make the atomic sink initializer a compile-time constant
Apple clang 15 (Xcode 15.4, the macOS CI image) rejects
static _Atomic cbm_log_sink_fn g_log_sink = NULL;
with "initializer element is not a compile-time constant": NULL expands
to ((void*)0), and the implicit void*-to-function-pointer conversion is
not a constant expression there. Casting to the function-pointer type
makes it an address constant, which is what a static initializer needs.
The local ladder could not have caught this. The macOS host here runs
Apple clang 21, which accepts the uncast form; the rejecting compiler
exists only on the CI image. Recording that plainly because it is a real
gap in what local verification can promise for macOS, not a slip in how
this batch was checked.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
|
||
|
Martin Vogel
|
f5baf1baef |
fix(ci): route the new lanes through canonical leg entries
The MSan and memory-analyzer jobs drove docker and make directly from workflow steps, and the diag step used folded `run: >`. All four are venue-parity violations: a venue may provision, plumb artifacts, or call a canonical leg script, and nothing else. Anything that actually exercises the product belongs inside a scripts/ entry so that every venue runs the same code instead of each workflow growing its own slightly different invocation. So the docker work moves to scripts/ci/msan-lane.sh (build | run | all) and the analyzer gate to scripts/ci/lint-mem.sh, both of which the local paths already reach through run.sh and the Makefile. The folded step becomes `run: |`. Found by the contract itself, running as step 0j of the local Linux leg. It had never run against these jobs, because they were added and then exercised only through GitHub CI -- which is exactly the gap the local ladder exists to close, and the reason the contract runs as step zero of every leg rather than as a job of its own. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
Martin Vogel
|
0c223d9027 |
build(lint): pin analyzer suppressions to the code they argue about
The memory gate stays gating, and it now has a way to record a genuine false positive that cannot quietly outlive its own reasoning. A whitelist entry names one (file, function, check) and carries two things: why the analyzer is wrong, argued from the code, and what was tried before concluding that. The entry is pinned to the sha256 of that function's text. Edit the function and the entry stops counting -- the finding comes back and has to be argued again against the code as it now is. This is the part that matters. A suppression that survives the code it was written about reads as "reviewed" while being nothing of the kind, which is worse than no suppression at all. The gate fails on: a finding with no entry, a finding whose entry has gone stale, and an entry that asserts rather than argues (there is a floor on how much reasoning an entry must actually contain -- the mechanical half of "argued, not asserted"; whether the argument is correct stays a review question). An entry that matches no finding is reported but does not fail, because analyzer versions differ across platforms. NOLINT is still not honoured here and the gate does not read it. The whitelist ships empty: the analyzer is currently clean across LINT_SRCS, so nothing is being suppressed today. The mechanism exists for the first finding that genuinely warrants it. Verified by exercising each path rather than only the clean one: an unaccounted finding fails and names its enclosing function; an argued entry with a matching hash passes; the same entry fails as STALE once the hash no longer matches the function; and an entry that says only "false positive" fails for not arguing its case. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
Martin Vogel
|
0fe0f453be |
build(test): close the gaps in the sanitizer matrix
Every lane here was either asserting coverage it did not have, or was reachable only by bypassing the entry point that is supposed to define the ladder. TSan: no exclusions any more. The three suites the Makefile documented as excluded are back. daemon_ipc and daemon_frontend no longer reproduce the harness race and thread leaks they were excluded for. daemon_runtime did not deadlock as the comment claimed -- it reported a real production data race on the log sink, fixed separately. Excluding a suite from a sanitizer lane hides exactly the class of bug the lane exists to find, so the comment block now records what was actually true rather than what was assumed. TSAN_OPTIONS gains report_thread_leaks=0. This disables the thread- HYGIENE check only; race detection is untouched. Several daemon fixtures fork after the process has gone multi-threaded, and in the forked child TSan sees the parent's already-finished threads as never-joined even where the fixture joins them. It fires on macOS and not Linux, i.e. it tracks fork semantics rather than anything about this code. The alternative was dropping whole suites, which costs real race coverage; this costs none. UBSan: tre.o no longer builds with -fno-sanitize=alignment. Alignment was switched off for a vendored regex engine that ships in the product, which is where the check is least redundant, not most. LSan on macOS: new test-lsan target and test-lsan-macos CI leg. LSan is on by default under ASan on Linux, so the Linux legs have always had leak coverage. On macOS it is off by default and Apple's clang refuses to enable it outright, so that platform had none at all. Apple's refusal is not a darwin limitation -- upstream LLVM supports LSan on darwin/ arm64. The lane is the ordinary ASan suite built with Homebrew LLVM and run with detect_leaks=1; it runs the full suite clean and was checked to still catch a deliberately leaked allocation. MSan: reachable from the local ladder. The image and compose service existed but run.sh had no leg, so the only way in was to drive docker compose by hand -- which means it was not part of the ladder in any meaningful sense. The image also moves to clang 22, matching the diag and analyzer lanes instead of sitting four majors behind on noble's default. The leg documents the aarch64 shadow-mapping failure so a local arm64 stack overflow in the grammar suites is not mistaken for a code defect; the GitHub leg runs x86-64, which is the mapping that matters. Off-by-default ASan checks: the diagnostic lane, and its CI twin, now run detect_stack_use_after_return, detect_stack_use_after_scope, detect_invalid_pointer_pairs (with the -fsanitize=pointer-compare, pointer-subtract instrumentation it requires) and strict_string_checks. Running ASan is not the same as running all of it, and these four cover bug classes nothing in the matrix was looking for. They stay on the diagnostic lane rather than the gating ones until they have a clean history there; promoting them is a separate deliberate step. Verified: macOS TSan 940 passed / 3 skipped / 0 races over the full suite set; the macOS leak lane 7375 passed / 4 skipped / 0 leaks, with LeakSanitizer confirmed armed under that exact toolchain and option set by checking it still reports a deliberately leaked allocation. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
Martin Vogel
|
ade9f09a4f |
test(daemon-runtime): spawn the blocked executable without forking
runtime_test_spawn_blocked_executable() used fork() + execl(). fork() duplicates the parent's address space, and under a sanitizer that address space includes a very large shadow mapping. On macOS the duplicate trips the per-process memory limit and jetsam SIGKILLs the child before exec ever gets to replace the image, so the child is already dead by the time the parent checks it. The visible effect was daemon_runtime_process_fingerprint_never_hashes_ replacement_path failing at ASSERT(setup) under TSan while passing in every non-sanitized and ASan build. The diagnosis is direct rather than inferred: waitpid() reported the child as WIFSIGNALED with WTERMSIG 9, and DYLD_INSERT_LIBRARIES was unset, ruling out library-validation refusal of the copied system binary. posix_spawn() never copies the parent address space, so the child is never charged for the shadow mapping. The file actions reproduce the child-side setup exactly (close the ready-read and input-write ends, dup2 the input-read end onto stdin, close the original), and exec failure is now reported by posix_spawn itself rather than over the ready pipe. The FD_CLOEXEC-on-successful-exec EOF that the parent uses to detect a live child is unchanged. This keeps the test running under TSan on macOS instead of dropping the suite from the lane. Verified: daemon_runtime 43 passed under TSan and under the ordinary build, and the full macOS TSan set is 940 passed / 3 skipped / 0 races. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
Martin Vogel
|
c51482ae45 |
fix(log): make logging configuration atomic
The level, format, sink and sink-mode globals are written by whichever thread configures logging and read by every thread that logs: daemon connection workers, pipeline workers, the watcher. They were plain globals. For the sink this is not a benign stale-value race. emit_line() read the global function pointer, tested it, then called it -- so a concurrent cbm_log_set_sink_ex() could turn a checked pointer into a call through a NULL or partially-written one. The load is now done once into a local, which closes the test-then-call window as well as the tear. Ordering is relaxed: each value is an independent scalar with nothing to publish alongside it, and the log path has to stay cheap enough that no caller is tempted to route around it. cbm_log_set_sink_ex stores the mode before the sink, so a reader that sees the new sink cannot then read the mode belonging to the previous one. Found by ThreadSanitizer once the daemon_runtime suite was added to the TSan set. That suite had been excluded from the lane, which is precisely what kept this hidden. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
Martin Vogel
|
02f870105f |
fix(msan): the lane runs the complete code by default
A sanitizer lane covers everything or it is not a sanitizer lane: a pass over a subset asserts coverage it does not have. My exclusion list had grown to three suites — including pipeline, a large one — and dressing it up as an O10 whitelist was wrong. O10 governs a board that TRACKS known-red reproductions; it does not license cutting a sanitizer's coverage to make it green. MSAN_EXCLUDE now defaults to EMPTY in both venues. It remains as an iteration aid — an engineer fixing the underlying problem can narrow the run — and it prints an explicit warning that a green partial result proves nothing about the tree. Consequence, stated plainly: the lane is currently RED on the local arm64 container, where three deep-recursion suites overflow their thread stacks under instrumentation. That is a bug to fix, not a list to live with, and the script records everything known about it. The CI leg (x86-64, where MSan's stack handling is far better supported) is where it gets settled. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
Martin Vogel
|
54dd5ad781 |
ci(msan): restore the lane; x86-64 CI is its authoritative venue
Two corrections. First, removing the CI job while keeping the compose service and the local bindings left the ladder carrying a lane CI did not have — the venue asymmetry the unification work exists to prevent. The lane now lives in both venues again. Second, and the reason the removal was wrong: the evidence behind it was entirely from the LOCAL arm64 container, while the job that got deleted would have run on x86-64. MSan's shadow and stack handling are materially better supported on x86-64, so the thread-stack overflows that drove the exclusions may well be architectural. I never tested the architecture CI uses, and the local ladder cannot emulate it faithfully — so 'it cannot run in CI' was an inference from the wrong platform stated as a fact. The CI leg therefore runs with MSAN_EXCLUDE="" — no exclusions — so the first run settles the question with real evidence on the real architecture. The script's exclusion list stays as the LOCAL default, documented as such. An accepted local/CI divergence for this lane specifically. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
Martin Vogel
|
f96ddf5f93 |
chore(msan): keep the lane local and exploratory, not in CI
Running the lane to completion does not currently work: with three suites already excluded for thread-stack overflow it now dies with a plain SIGSEGV elsewhere. Wiring an auto-running job that cannot finish would be exactly the structurally-red lane O10 forbids, so the test-msan CI job is removed and the lane is documented as exploratory and local-only. What it IS worth, and why the infrastructure stays: every suite it does run is clean under MSan, including the C++ preprocessing path that justified building the instrumented-libc++ image in the first place, and it correctly identified one convincing-looking report as a mixed-build artifact rather than a bug. The image, the script, the compose service, the MSAN_ORIGINS and CBM_THREAD_STACK_MB knobs, and the full record of what was tried all remain, so picking this up is a continuation rather than a restart. The blocker is one problem, stated at the exclusion site: threads whose stacks are sized outside cbm_thread_create overflow under instrumentation, and the failure follows cumulative process state rather than any single suite's depth — which points at per-suite process sharding or finding that thread creator as the fix. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
Martin Vogel
|
7ad0003c95 |
fix(msan): stack-size floor for sanitized builds; document the excluded suites
Two follow-ups to the MSan lane, both from running it for real. 1. compat_thread.c gains a sanitized-build-only stack FLOOR (CBM_THREAD_STACK_MB). Thread stacks here are sized in code, so a sanitizer lane cannot raise them with ulimit -- RLIMIT_STACK at 8/64/256 MiB provably had no effect. The first version overrode only the DEFAULT size, which silently did nothing for worker_pool/runtime/main because they all pass an explicit size; it is now a floor applied to every thread. Shipping builds are untouched (the whole hook is behind CBM_SANITIZED_BUILD). 2. Two grammar-corpus suites are EXCLUDED from the lane, with the full rationale, evidence, and everything tried recorded at the exclusion site per O10 -- including that the floor above does NOT fix them, which narrows the next person's search to a thread creator outside cbm_thread_create. The exclusion list now names SUITES (an earlier version named a TEST and therefore excluded nothing) and fails loudly on an entry that matches no suite, so that silent-no-op cannot recur. Also restores the exec bit on scripts/msan.sh, which the image ENTRYPOINT needs. worker_pool + parallel + pipeline + mcp on macOS: 526 passed, 2 skipped. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
Martin Vogel
|
c9886d4f25 |
feat(ci): MemorySanitizer lane — instrumented libc++/zlib image, full C++ coverage
Stage 2 of the memory-diagnostics program (user decision: go directly to the instrumented image rather than a C-only probe). MSan detects uninitialized READS, the one memory-error class no other lane covers dynamically, and it requires every linked library to be instrumented -- vendored C deps compile in-tree and instrument for free; the two external links do not: - test-infrastructure/Dockerfile.msan: pinned-base image building libc++/libc++abi/libunwind (llvmorg-18.1.8, LLVM_USE_SANITIZER= MemoryWithOrigins) and static zlib v1.3.1 into /opt/msan, with the symbolizer and MSan runtime in a separate last layer so tool additions never invalidate the ~30-min libc++ build. - scripts/msan.sh: the canonical lane entry. ALWAYS clean-builds its BUILD_DIR: make does not encode flags into dependencies, and a stage-1 probe's libstdc++ objects surviving into the libc++ lane produced a convincing-looking uninitialized-value report at preprocessor.cpp:168 -- the uninstrumented .so string constructor wrote the temporary, the instrumented move constructor read it. The clean rebuild proved it an artifact: extraction (incl. the C++ preprocessing path) runs 272/272 with zero reports. - Makefile.cbm: CXX_STDLIB / CXX_STDLIB_FLAGS hooks so the lane can swap libstdc++ for the instrumented libc++ (defaults identical; the shipping build is byte-for-byte unaffected). - docker-compose test-msan service: same aarch64 seccomp/setarch remedy as the TSan service (MSan's shadow layout hits the same personality() block). - CI test-msan job (_test.yml): buildx local-cache via the repo's existing pinned actions/cache -- no new third-party action pins; a warm run skips the libc++ build entirely. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
Martin Vogel
|
80afcd6bbe |
fix: close the memory-error paths the clang-analyzer lane surfaced
The memory-diagnostics report's priority-4 lane (path-sensitive clang-analyzer, memory checks only) run over all 111 production files. 21 findings triaged; the real ones, all cold-path (none can explain #581's per-query residual): LEAKS - mcp get_architecture: scope_path leaked on the missing-store early return (REQUIRE_STORE frees only `project`); allocate after the gate. - pass_definitions: cancellation mid-extraction leaked the pass-owned result cache including already-extracted entries; mirror the end-of-pass cleanup. - store package-boundary scan: the row-scan abort path freed the node arrays but not the boundary accumulators or their duplicated package strings. - cbm quarantine set: a duplicate path line leaked the replaced value (and a fresh key copy -- the table borrows key pointers); a partial strdup failure leaked the surviving half. Reuse the stored key for duplicates. - pass_githistory: unchecked malloc/strdup -- an OOM dereferenced NULL and a failed strdup leaked the index cell. Allocate before claiming the slot. NULL/UB - cli config subcommand: NULL argv with nonzero argc slipped the guard (the inner `argv &&` shielded only the help comparison) into argv[0]. - store bfs_multi: a negative max_results broke out before any row was written, then freed fields of an unwritten negative-index slot. Clamp. - pass_calls emit_http_async_edge: the service-pattern call sites pass a NULL target behind a hand-duplicated URL predicate; a drift between the copies turned target->id into a null deref. The callee is now total. - sqlite_writer: both leaf-array OOM paths left leaf_count stale with a NULL array, walking pb_finalize_* into leaves[0]; consistent empty state routes them to the existing root=0 failure return. HARDENED (invariants true but invisible to path-sensitive analysis) - Leiden CSR + aggregate arrays, SCC adjacency: calloc + endpoint guards, so a future degree/collection miscount degrades benignly instead of UB. - SCC cycle fill: the ncyc==0 no-slot invariant made local. RECORDED FALSE POSITIVES (no code change) - yaml sequence starts (loop bound == alloc bound), cypher agg arrays (same count both sides), mcp read_message ch (assigned by fgetc each iteration), pkgmap clean buffer, mcp csize (Tarjan: ncomp>=1 when nverts>=1), vendored verstable x2. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> GATE + LANES (user decision: runner cost accepted) - make lint-mem (local triage) and lint-mem-ci (gating: vendored-filtered, any remaining finding fails). The gate is green because every false positive above was restructured for provability -- calloc'd fill-cursor arrays, explicit Tarjan invariant, zeroed buffer tails, min-1-element allocations -- never suppressed. - make diag: pinned newest-LLVM ASan/UBSan lane with straighter stacks. - CI: lint-mem job (_lint.yml) and test-diag job (_test.yml), both on the pinned LLVM 22 apt toolchain. Cost disclosure: roughly +25-40 min and +25-60 min (ccache-warm) per push respectively. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
Martin Vogel
|
925e662e67 |
fix(tests): portable env calls in the parallel-determinism repro
setenv/unsetenv do not exist on Windows, so the bug-repro runner failed to COMPILE there -- this is the 'runner did not execute' failure on the repro-windows CI job and the local Windows board alike. The compat layer's cbm_setenv/cbm_unsetenv (already included via repro_harness.h) are the repo idiom; test_pipeline.c uses them for the same variable. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
Martin Vogel
|
679871aedb |
fix(c_lsp): remove the whole-function ASan suppression; assert the resolution it hid
c_lsp_process_file carried __attribute__((no_sanitize("address"))) over the
entire C LSP orchestration, excluding every direct memory access in it from
ASan -- placed for a stack-pointer-to-registry lifetime bug whose witness test
also had its assertion discarded ((void)find_resolved), so nothing could ever
prove the bug gone. Flagged by the memory-diagnostics standards report
(private/C_MEMORY_DIAGNOSTICS_STANDARDS_REPORT.md, priority 2) as the single
highest-value ASan coverage gap in the repository.
This branch's C LSP registry rework resolved the underlying lifetime: with the
suppression removed, the full c_lsp suite passes under ASan+UBSan (760 tests),
including the formerly-failing template field type resolution, whose test now
asserts the resolved call instead of discarding it.
Honest limit: the original bug predates the branch and its fix is the branch's
broad registry rework, not an isolatable commit -- so the revert-proof is
main-vs-branch (main: suppression + discarded assertion; branch: neither),
not a single-change revert-check.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
|
||
|
Martin Vogel
|
e2f5b6a798 |
fix(pipeline): module-QN strings must outlive the cross registries they feed
The sequential lsp_cross pass builds its shared per-language cross registries
in ctx->seq_cross_arena, which DELIBERATELY outlives the pass -- resolved_calls
and the registries carry borrowed strings that pass_calls still reads, and the
arena is destroyed only after all passes (the earlier freeing-here bug was a
pass_calls use-after-free, says the comment at the arena's creation).
But the per-file module-QN strings (def_modules[], malloc'd in
cbm_pxc_collect_all_defs and handed to every registrar as def_module_qn) were
freed at the END OF THE PASS -- the exact mistake the arena comment warns
about, one level down. Any registry-reachable structure holding one of those
pointers read freed memory in pass_calls.
AddressSanitizer caught it as a heap-use-after-free (strcmp in
cbm_pipeline_pass_calls on a string freed by the pass-end cleanup) on the
first-ever run of the real-repo determinism tier (linux/fs/xfs, 355 files) --
a tier no CI runner can execute because the corpus is local-only, which is why
it survived: bisect shows it predates today's commits (
|
||
|
Martin Vogel
|
004a9a499f |
perf(extract): O(1) walk-state maintenance; gate parent probes by language
The deep-nesting torture tests (stack_overflow_a/b) went from 0-1s per test on main to 39-119s on this branch -- the 900s suite budget killed them on every venue except the M4 (three GitHub CI platforms and the local Linux leg, each dying mid-suite at a DIFFERENT test, which is what pointed at shared machinery rather than any one language). Sampling the child process found two quadratic layers, both branch-added: 1. recompute_state iterated the WHOLE scope stack on every code-bearing node to rebuild the walk-state flags -- O(depth) per node, and a deep descent pushes a frame per level, so deep trees paid O(n x depth). Each frame now saves the complete walk-state tuple it displaces and pop restores it verbatim: push and pop are O(1) and kind-agnostic, and the per-node recompute is gone entirely. The CALL frame's effect is applied by push_call_scope after the caller fills the invocation triple, preserving the old ordering exactly. 2. is_reference_node fetched ts_node_parent for EVERY identifier in EVERY language to serve a Puppet/Vimscript sigil-wrapper check -- the language gate sat inside the condition, after the fetch. ts_node_parent descends from the root (O(depth)), so all languages paid O(depth) per identifier. The gate now precedes the fetch; only Puppet/Vimscript files pay it. macOS, both suites together: 543s -> 54s. Per test: ts_cyclic 119s -> 6s, python_deep 105s -> 6s, go_deep 39s -> 8s, php_deep 45s -> 8s. The residual 6-8s vs main's 0-1s is the branch's larger legitimate per-node work; the remaining Python attribute-site parent walk is a bounded follow-up, recorded. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
Martin Vogel
|
43e861e36a |
fix(pipeline): release db_path/db_dir on the manifest abort paths; free the test harness cross arena
Second LSan round from the Linux leg, both verified green in the container (532 passed, 0 failed under LSan): - dump_and_persist_hashes' two semantic-manifest abort returns leaked BOTH of the function's strdups (db_path and db_dir, the latter otherwise freed only further down). Same ownership rule as the previous fix: every exit releases what the function allocated. - test_parallel's sequential harness drives the passes directly and never destroyed ctx->seq_cross_arena, which the cross pass fills with the shared per-language registries (stdlib registrations included -- ~20MB per test). Production's run_sequential_pipeline destroys it after all passes; the harness now does the same. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
Martin Vogel
|
37d23d3619 |
feat(kotlin): receiver-typed property references decide their occurrence
Maintainer decision (option B) closing the Kotlin property-reference repro's platform divergence at its root instead of tie-breaking the fallback. `holder::handler` (parsed as navigation by the vendored grammar) produced a USAGE edge whose target was chosen by the name-only registry fallback: with a same-named property (Holder.handler) and function (Functions.handler) in the project, the winner was registration order -- readdir order -- so Windows (lexicographic NTFS) borrowed the FUNCTION while macOS happened to pick the property. The graph's answer must not depend on directory enumeration. The occurrence is now claimed by exact, receiver-typed resolution. Five links, each of which was missing: 1. extract side: a Kotlin navigation member read is a semantic-reference candidate at the member occurrence, so an LSP row can claim it. With no row the join finds nothing -- but a candidate no longer falls back to the name-only registry guess, which is the fail-closed direction this PR is built on. 2. kotlin cross resolution: a property READ with a proven receiver emits a CALL_REFERENCE row against the property. The property is not a callable target, so the join can only produce USAGE, never a fabricated CALL_REFERENCE. Value reads only; calls stay with the invocation machinery. 3. receiver typing across files: kotlin_resolve_class_name composed <this module>.<name> for an unimported cross-file type, which can never name a type defined in another file. A per-call unique-short-name map (built from the project defs, ambiguous names fail closed) resolves the annotation to the real registered QN. Hash lookup, no scans. 4. cross registry fields: pxc_map_label dropped Variable defs entirely, so no cross registry ever saw a property. They now flow through (every language's registrar filters by explicit label, so only Kotlin consumes them) and the Kotlin registrar attaches them as fields of their receiver type, hash-bucketed -- a per-type scan would be the registry-tail-scan quadratic pattern. 5. def side: class-body variables now record their declaring class (parent_class) -- previously only methods did. The QN stays module-level, so this is additive metadata: the only structural parent_class consumer is Method-gated (DEFINES_METHOD), verified in pass_definitions/pass_parallel/ pipeline_incremental. Emission targets each field's REAL def QN carried through the field map: kotlin class properties are minted with module-level QNs (proj.Holder.handler, not proj.Holder.Holder.handler), so the composed form would name a node that does not exist and the join would silently drop the edge. The repro now proves the property edge on every platform for the same reason, not by racing readdir. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
Martin Vogel
|
67bcc2832b |
fix(kotlin): resolve callable references in expression bodies, incl. properties
Two gaps in the callable-reference path, found while attributing a Windows-only failure of the Kotlin property-reference repro: 1. kt_callable_reference_has_ambiguous_parent treated every unlisted parent kind as ambiguous, including function_body -- so a single-expression body (fun f() = ::handler) never resolved its reference at all and fell to the name-only registry fallback. A single-expression body is an unconditional context: the expression IS the value, no branch selects among candidates. 2. The typed-receiver branch (Type::member, value::member) only consulted kotlin_lookup_method, so a reference to a PROPERTY emitted nothing even when the receiver type provably has that member. It now emits a row against the property QN; the property is not a callable target, so the downstream join can only produce USAGE, never a fabricated CALL_REFERENCE. Note: this does NOT close the property-reference repro on Windows. That fixture's holder::handler parses as navigation_expression under the vendored grammar, whose member occurrence is not a semantic-reference candidate -- the edge is decided by the name-only registry fallback, whose winner among same-named symbols is registration order (= readdir order, platform- dependent). That divergence is a design question recorded separately. repro_reference_precision + kotlin_lsp on macOS: 170 passed. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
Martin Vogel
|
b020748c1d |
fix(pipeline): release db_path on both exits of dump_and_persist_hashes
resolve_db_path returns a strdup the function owns, but neither the publish-failure return nor the success tail freed it -- cbm_pipeline_refresh_artifact only borrows the pointer. Every pipeline run leaked one path string; LeakSanitizer on the Linux leg aborted the pipeline, index_resilience and mcp suites over exactly this pair of exits (every leaked allocation across the leg traced to this single strdup). macOS stayed green because this setup has no leak detection there, which is precisely why the Linux leg exists. Linux container, same three suites under LSan: green after the fix. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |