fix/java-enum-methods
198 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Martin Vogel
|
43ecc0988c |
feat(dbt): extract dbt model lineage from Jinja-templated SQL
A dbt model is an ordinary .sql file whose dependencies are written
{{ ref('other_model') }} or {{ source('group','table') }}, never as
literal table names. The SQL grammar cannot read those -- FROM {{ ref(
'x') }} is a parse error to it -- so the dependency structure of an
entire dbt project was invisible to the graph.
extract_dbt.c runs as a sub-extractor inside the normal indexing
pipeline. Per qualifying file it emits one Model definition (named by
the file stem, dbt's own model identity) plus one usage per ref()/
source() call; pass_usages resolves those into model -> relation
lineage edges like any other reference.
Model is a relation label alongside Table and View, which is what makes
the rest work without new plumbing: registry seeding, the central
relation veto, the incremental surface hash, search ranking and the
architecture queries all pick it up from cbm_label_is_relation. Sharing
one label class also means a model's source('raw','customers') resolves
onto a Table declared in a plain DDL migration in the same repository,
so dbt lineage and SQL lineage form one graph rather than two, while
the veto keeps model names out of every non-lineage consumer.
The pass is self-gating: SQL files only, and only those carrying a real
ref()/source() call. The dbt builtins are themselves the evidence, which
is cheaper than a dbt_project.yml lookup and more precise -- templated
SQL that is not dbt (an Airflow {{ ds }} parameter) produces no Model
node and no usages even inside a dbt repository.
{% macro %} definitions are deliberately excluded: the vendored
tree-sitter-jinja2 grammar has no node types for {% %} statements, so
they can only be recovered by a hand-written scanner that cannot see
comments or find {% endmacro %} for a correct span. Filed as follow-up
rather than approximated.
Tests cover the lineage, the last-string-argument semantics of both
builtins, the gate (against a plain-SQL control extraction), plain DDL
staying untouched, and an end-to-end pipeline case asserting model ->
model across files, model -> Table onto plain DDL, and cross-language
isolation. Disabling the pass reddens three of them; removing Model from
the relation set breaks lineage outright.
Implements the lineage half of #575.
Co-authored-by: alexisperinger-ux <alexis.peringer@iss-stoxx.com>
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
|
||
|
Martin Vogel
|
0273250c33 |
Merge pull request #1310 from Yyunozor/fix/1264-hook-diff-aware-clang-tidy
fix(hooks): scope pre-commit clang-tidy to staged changes (#1264) |
||
|
Martin Vogel
|
51770de0b6 |
perf(ts): memoize expression-type evaluation per node
ts_eval_expr_type and ts_signature_for_call are mutually recursive: resolving a call evaluates its argument expressions once per lookup path (method dispatch + namespace fallback), and in tsc-compiled spread files the first argument is itself the next nested Object.assign(...) call — the same subtree re-evaluates once per enclosing level, 2^n total. The TS suite's objectSpreadRepeatedComplexity.js (3.6 KB, 5 nodes) measured 20.4 s; with the memo its eval cost is zero within measurement noise of a one-file control, and the microsoft/TypeScript corpus drops 37.5 -> ~24 s warm (nodes byte-identical, edges within the known scheduler jitter). Expression types are position-pure within a file pass (one node = one scope path; the per-file walk is single-threaded and deterministic), so one eval per node is the correct semantics, not a cache trade-off. The memo is a per-file, arena-backed, linear-probe table keyed on TSNode.id. Results produced under a depth-cap or budget bail are never stored: both bail sites bump a degradation counter, and a store only happens when the subtree completed clean — a degraded UNKNOWN can therefore never shadow a later full evaluation. The regression guard asserts work, not wall-clock: the nested-Object.assign shape must complete without exhausting the deterministic eval budget, read back through a new CBM_ENABLE_TEST_SEAMS accessor pair. The seam lives in the lsp_all unity object, so GRAMMAR_CFLAGS_TEST/TSAN now carry the seams define (test artifacts always have seams; prod never does). Verified RED without the memo (budget exhausted, suite 47.8 s) and green with it (3.2 s). Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
Martin Vogel
|
e700b6215f |
test(complexity): deterministic O(n^2) guard on work-counter ratios
Finding #1669 took an 11-corpus A/B across two release binaries. This suite makes that bug class fail a unit test in seconds, on every platform, from tiny corpora. Method: build k and 2k REPLICATED module copies, run the full in-process pipeline on both, assert counter RATIOS. Independent copies mean every extensive quantity — nodes, edges, Σ per-file registry defs — must grow linearly (ratio ~2). A files x corpus coupling makes per-file work itself grow with k and lands at ratio ~4. Ratios expose the exponent regardless of absolute scale, so 60-120 files suffice. Verdicts are pure functions of (code, input): gates ride ONLY on deterministic work counters and data-product counts, never on wall time. Throughput (nodes/s, edges/s) is information-only, written to private/benchmarks/complexity-<ts>.json (local, gitignored; skipped under CBM_SKIP_PERF where rates are meaningless). Two corpus shapes, both needed: - Independent modules (java/py/go/ts templates): catches cross-module contamination and dedup breakage. The #1669 bug is GREEN here — fully closed modules filter perfectly, which is exactly why it survived. - The growing shared package (bigpkg): one Java package whose file count scales with k — the real-repo shape (files concentrate in large packages). The JVM namespace filter branch makes per-file work track package size, so this corpus is the honest #1669 reproducer: ratio 4.00 RED on the pre-fix tree, 4.00 RED for a module-scoped overlay, 2.00 GREEN for the own-file overlay. It discriminated the correct fix design before the fix was written. Both legs of every pair exceed MIN_FILES_FOR_PARALLEL(50): below it the sequential path runs, which builds no shared registries and would be the wrong code path to gate (its per-file cost is bounded by the 50-file ceiling). Recorded but deliberately NOT gated, with reasons at the case: tail_candidates and fallback_rows are legitimately superlinear under replication until those scans are bounded, and measured ~1 ns/unit. Every ratio gate carries a non-vacuousness floor on the base counter so broken counter wiring fails loudly instead of green-washing (cbm_pxc_count_perfile_defs feeds the overlay path into the same counter the fallback path already used; wired for Java, extend to the TS overlay when touching ts_lsp). Dynamic coverage: languages iterate CBM_LANG_COUNT; embedded templates cover the LSP-hybrid languages, and tests/fixtures/complexity/<lang>/ dirs are auto-discovered so a new language joins the guard by dropping fixtures. Uncovered languages are listed in the report with the reason. The local report additionally carries per-language node/edge counts with ratios and a per-pass elapsed_ms table per run (captured via a TEE log sink during the in-process pipeline runs) — trend data for humans, still never a gate. Refs #1669. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
Mauricio Offermann
|
c059396eb2 |
fix(build): give sanitized-build detection one spelling and a backstop
Four places ask "is this binary instrumented?" and each asked it differently.
One of them, the C# LSP bench, only recognised ASan, so TSan and MSan measured
an instrumented parse against the NATIVE 200ms budget. The other three carried
a hand-copied list of `__SANITIZE_*__` macros that nobody kept in sync.
That drift is what the TSan gap was made of. CFLAGS_TSAN never passed
SANITIZED_DEFINE, and the per-site conditions could not cover for it: they test
`__SANITIZE_THREAD__`, which is GCC's spelling. Clang — the compiler that leg
uses — announces thread instrumentation through `__has_feature(thread_sanitizer)`
only, and no site consulted it. The claim in
|
||
|
Martin Vogel
|
0a163d4f75 |
build(tsan): define CBM_SANITIZED_BUILD on the ThreadSanitizer leg
The previous commit widened the spawn-retry budget for sanitized builds, and it did nothing on TSan — the leg it was written for. `subprocess_run_spawn_failure` failed again on the very PR that was meant to fix it. CBM_SANITIZED_BUILD comes from SANITIZED_DEFINE, which keys off $(SANITIZE). TSan does not use that variable — it has its own TSAN_SANITIZE — and CFLAGS_TSAN never included SANITIZED_DEFINE. So the macro was undefined on that leg and every sanitized-budget branch compiled to its NATIVE value while running an instrumented, several-times-slower binary. The comment above SANITIZED_DEFINE already describes this exact failure for trap-UBSan: "the build system is the single source of truth for is this binary instrumented; compiler-specific probes miss clang's feature-check spelling and every non-ASan sanitizer". That lesson was recorded and the TSan leg was never wired up to it. Nor would compiler probes have saved this: clang spells thread instrumentation __has_feature(thread_sanitizer), not __SANITIZE_THREAD__. CFLAGS_TSAN now defines it unconditionally, which is honest — that flag set exists solely to build an instrumented binary. Checked the neighbours: MSan (scripts/msan.sh passes SANITIZE=) and the diag lane (passes SANITIZE= too) both already get the define. TSan was the only gap. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
Martin Vogel
|
93696aeb55 |
docs(security): publish the AV false-positive evidence; harden arm64 W^X
SECURITY.md gains an "Antivirus False Positives" section placed directly after the existing Verification block so it reuses those commands rather than restating them. It states the detection plainly, shows the measured evidence (the verdict inverts across architectures and link modes; identical macOS segment structure splits clean/flagged; entropy is 3.5-4.2 bits/byte, nowhere near packed), lists the cross-project precedent, and — deliberately — records that removing the embedded scripts and externalizing the assets did NOT move the detection. A negative result is still evidence, and publishing it is the point. It also documents the release policy, invites independent audit through the provenance/cosign/checksum path we already ship, and adds an `av-analysis` issue label route for anyone who finds something real. Tone is evidence-first: no vendor blame, and an explicit note that signing helps Windows but that no AV-honoured signing scheme exists for Linux ELF, so it is not a full answer. README carries a short pointer near Quick Start, where someone who just hit a Defender warning will actually look. Unrelated to AV, found while dissecting the artifacts: add -Wl,-z,separate-code. GNU ld enables it by default on x86-64 but not on aarch64, so arm64 emitted ONE R E PT_LOAD covering the whole image and mapped ~259 MB of tree-sitter parse tables executable at runtime. Section flags said A, not AX -- but the kernel applies segment permissions, so the section flags were never the control. The amd64 build of the same source mapped the same data R only. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
Martin Vogel
|
d58afe562d |
revert(release): re-embed runtime assets into the single shipped binary
Externalizing the integration templates (#1492/#1493) and the UI bundle
(#1501/#1503) was done to reduce the Microsoft `Wacatac.B!ml` surface. It did
not work: across dry runs the flagged artifact count stayed at ~3 and the
detections merely moved between artifacts.
Dissection of run 31286803592 shows there is no structural cause to fix. The
verdicts split across every axis at once — linux-amd64 (dynamic) flagged while
linux-amd64-portable (static) is clean, but linux-arm64 (dynamic) clean while
linux-arm64-portable (static) is flagged. The two macOS binaries have identical
segment structure and split clean/flagged. Siblings from one build landed in
different variant buckets (.B vs .C). Entropy is low everywhere
(code_vectors.bin 4.166, grammar tables 3.464 bits/byte, against 7.5-8.0 for
packed payloads), so the packed-payload hypothesis is excluded too.
So the complexity bought nothing, and installation goes back to being
self-contained: one binary that carries its own UI and agent integration
templates, with no adjacent data file that has to resolve before `install`
works. Only the UI-capable composition ships from now on, under the historical
unsuffixed archive name.
Removed: src/ui/asset_pack.{c,h}, asset_pack_stub.c, asset_manifest_stub.c,
scripts/pack-ui-assets.mjs, src/cli/integration_assets.{c,h},
assets/cbm-integrations.json, scripts/gen-integrations-hash.sh, the
--verify-runtime-assets probe (nothing adjacent left to verify), and the
composition gates A6/A7 whose property is now deliberately inverted.
Restored: scripts/embed-frontend.sh, src/ui/embedded_{assets.h,stub.c}, the
compiled-in hook/adapter template bodies, and the embed/EMBED_OBJS build path.
Kept from the reverted commits, re-applied by hand where a wholesale file
restore would have dropped them:
- cbm_module_path_utf8() in both self-path sites. GetModuleFileNameA renders
through the ANSI code page and mangles non-ASCII install paths.
- the /__cbm/ui-readiness HMAC proof, secure_random and cbm_hmac_sha256, so
`daemon start --open` still waits for a genuine CBM listener.
- X-Content-Type-Options: nosniff on served assets.
- the MinGW noexecstack gate, -lbcrypt, and the cppcheck/zip CI fixes.
Archives are now codebase-memory-mcp-<os>-<arch>[-portable] with exactly four
members (binary, LICENSE, installer, THIRD_PARTY_NOTICES.md). That restores the
names every static package manifest already points at — aur, chocolatey,
homebrew, scoop, winget and glama were all broken by the -ui- rename.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
|
||
|
Martin Vogel
|
54b5ba846b |
fix(lint): generate integration hash before cppcheck
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
Martin Vogel
|
00ed56fb77 |
fix(build): restore Windows release compile
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
Martin Vogel
|
8018561cfe |
fix(release): externalize runtime assets and harden VT verification
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
Martin Vogel
|
497c680aae |
fix(build): make the UI target depend on the generated integrations hash header
The -ui build failed on Windows with "cbm_integrations_hash.h file not found". integration_assets.c includes that generated header, and the subagent added it to PROJECT_HDRS so every single-invocation binary target picks it up — but cbm-with-ui is a phony target with its own direct recipe that does NOT list PROJECT_HDRS, so nothing generated the header before its one-shot compile. Only the UI variant hit this; the standard binary, test-runner, test-repro-runner and tsan targets all carry PROJECT_HDRS and built fine. Added $(INTEGRATIONS_HASH_HDR) as an explicit prerequisite of cbm-with-ui. Confirmed by dry-run: `make cbm-with-ui` now schedules the generator before the compile, and the header regenerates from clean. Audited every target that compiles integration_assets.c (via PROD_SRCS) — all now resolve the header, so this closes the build gap rather than moving it. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
Martin Vogel
|
83df500364 |
fix(build,ci): make the integrations-hash generator runnable and quiet a JSON-license false positive
Two CI failures on the integration-asset change, both build/gate plumbing, not the architecture: 1. scripts/gen-integrations-hash.sh was committed 100644 while Makefile.cbm invoked it directly, so a fresh CI checkout died with "Permission denied" building the hash header — which failed EVERY test leg at step 0, since the contract step builds first. It passed locally only because the generated header was already cached, so make never re-ran the generator. Fixed both ways: the recipe now runs it via `sh` (mode-independent, cannot regress from a checkout mode), and the file is committed 100755 to match its siblings. Note: test_script_exec_bit_contract.sh scans shell call sites, not Makefile recipes, so it did not catch this — the `sh` prefix is the durable guard. 2. The license gate (ScanCode) flagged scripts/package-release.sh with the SPDX 'JSON' license. The archive member lists place 'cbm-integrations.json' and 'LICENSE' adjacently, and ScanCode reads the '.json LICENSE' token adjacency as a JSON-license reference. The order is not free to change — the Windows single-binary contract locks the exact member sequence — so this is a genuine false positive on a first-party MIT build script. Added it to the policy's ignored_paths with the same justification the six existing first-party entries carry (the gate scripts, provenance auditor, and discover.c all name licenses for legitimate reasons). No allow-listed SPDX id was added; the JSON license is NOT now permitted anywhere else. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
Martin Vogel
|
a6f132bc85 |
harden: ship integration templates as a verified JSON asset, not embedded bytes
The product binary embedded nine complete shebang'd shell scripts, their
PowerShell/.cmd twins, and two node:child_process client-adapter modules as C
string literals, and wrote them into agent config dirs at 0755 on install. A
block of script-shaped bytes inside a large unsigned executable is exactly the
surface Microsoft Defender's ML scored Trojan:Script/Wacatac.B!ml — the
detection that has recurred across the last several release attempts, on the
Linux ELF and darwin-arm64 standard binaries while every UI and Windows variant
stayed clean.
This moves every integration template out of the binary into a single shipped
data file, assets/cbm-integrations.json (compact: one asset, because each
shipped file is itself scanned). The binary now carries only the templates'
identity — one embedded SHA-256, generated at build time from the JSON so it
can never drift — plus the code to load, verify, and render them.
Why a hash and not just a loose file: content compiled into the binary was
tamper-evident with the binary. Moving it to disk must not turn it into
unauthenticated code the installer blindly executes. install verifies the asset
against the embedded SHA-256 and fails closed on mismatch ("integration assets
missing or modified - reinstall from the release archive") before writing
anything. Verified end to end: a one-byte edit to the asset makes install
refuse.
Ownership, for safe upgrade/removal: the verified copy is published to
~/.cbm/assets/<version>/ — a sibling of the DB cache, never inside it, so
clearing the cache cannot strip a user's hooks. A deployed integration file is
ours to remove only if it matches a template rendered from that stored copy;
foreign files and user-edited files at the reserved paths are preserved. The
historical template bodies travel in the JSON's released[] arrays so files
written by older versions are still recognised and cleaned up on uninstall.
Removed from src/: cmm_gate/session/subagent script-prefix and suffix
constants, cbm_build_released_gate_script, the shell/PowerShell/.cmd bodies in
cli.c, and the JS/TS generators in client_adapter.c. Confirmed on the built
standard binary: 0 occurrences of '#!/usr/bin/env bash', '#!/bin/sh',
'node:child_process', and 'ExecutionPolicy' (was non-zero for each).
test_no_embedded_scripts_contract.sh makes the removal a build property, not a
reachability assumption: no production C source may contain a shebang or
node:child_process literal, and the built binary is scanned directly. It strips
comments first (a contract a comment can satisfy is a false guard), and is
revert-checked — reintroducing one shebang literal makes it fail naming the
file:line. check-binary-composition.sh (A6) enforces the same needles on every
release artifact post-strip.
This does not, by itself, prove the detection is cured: the clean UI binary is a
strict superset of the flagged standard one, so no single passage is a
deterministic trigger. It removes the most coherent malware-shaped composite
from every shipped binary, aligns with how gh/starship/zoxide externalise
integration material, and makes the templates maintainable as data rather than
escaped C literals. Causal confirmation still needs a VirusTotal ablation.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
|
||
|
Martin Vogel
|
452372eec5 |
fix(mem): route ordinary malloc through mimalloc on Linux (#1360)
The comment above MIMALLOC_OVERRIDE_DEFINE claimed Unix "relies on static-link-order override". That was never true. mimalloc emits strong malloc/free definitions only when MI_MALLOC_OVERRIDE is set — the define its source gates alloc-override.c on — and the build set it for MinGW only. MI_OVERRIDE, which the build does set everywhere, is this project's own prod/test marker that mimalloc never reads. With the override body compiled out there were no strong symbols for link order to prefer, so ordinary malloc went to libc. Measured, not inferred. The shipped v0.9.1-rc.1 artifacts report 0/6 allocator-owned size classes on linux-arm64 glibc AND musl-static, and the same A/B on ubuntu-arm64 here: main warn mem.allocator.not_owned owned_classes=0/6 with this fix info mem.allocator.owned classes=all Every purge/reclaim option cbm_mem_init sets was therefore inert on Linux, applying only to the bound sqlite/tree-sitter populations — the same class of defect as #581, where committed memory ratcheted for months because nothing asserted the wiring on a real artifact. macOS stays off deliberately and permanently: enabling the override there compiles alloc-override.c's forwarding definitions, and under the two-level namespace this binary's free becomes mi_free while system libraries keep allocating from the system allocator, so the first pointer crossing that boundary aborts with "mi_free: invalid pointer". ELF's flat namespace has no such split, which is why Linux can have this and macOS cannot. Three parts: * Makefile.cbm switches MI_MALLOC_OVERRIDE and a new CBM_MEM_GLOBAL_OVERRIDE together in one place, for MinGW and Linux. The latter tells our own sources what to expect and goes into CFLAGS_PROD only, because MIMALLOC_CFLAGS_TEST builds mimalloc with -DMI_OVERRIDE=0 and no override define: a test binary has no global override BY CONSTRUCTION and must never be told to expect one. Deriving that expectation from the platform instead would make every Linux test run warn about a correctly configured build. * mem.c splits the startup audit by what the build actually asked for. Where no override was requested, ordinary malloc reaching libc is the design, so it reports the measured ownership at INFO and names what IS bound. A warning that fires on every run of a correct build is not a tripwire, it is noise — it trains readers to ignore the one line that catches #581. The genuine warning still fires wherever an override was requested and did not take. * smoke-test.sh Phase 1b pins the SHIPPED artifact's wiring per platform and fails in BOTH directions: Windows/Linux must own all classes, macOS must not. This cannot live in a unit test — a from-source test build never has the override — which is exactly why the defect survived so long. Verified on ubuntu-arm64: prod owns all classes; the test build reports bound-populations-only at INFO (0/6) and the mem suite stays green; smoke Phase 1b passes, and fails with the expected message when the same tree is built with the override forced off. On macOS: prod reports bound_populations_only at INFO, smoke Phase 1b passes, mem suite 51/51. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
Martin Vogel
|
03e935576b |
feat(foundation): classify indexing roots by breadth and sensitivity
Adds cbm_workspace_classify_root, the breadth half of the workspace
boundary. It answers "is this path obviously too broad or too sensitive to
index as one root", not "is this caller allowed to index it" — that second
question needs the user-level grant store, since it is the only input a
caller cannot write. The header says so, because a reader who mistakes one
for the other would draw the wrong conclusion about what is enforced.
Three rules, each doing what the others cannot:
- Depth below the volume. Two components on POSIX refuses every top-level
tree in one rule with no list to maintain. Windows and UNC count
drive/share-relative and require one, because there the first component is
already user space ("D:/repos") — the system trees that sit at the same
depth ("C:/Windows") are covered by name instead, which is tractable
because that set is small and stable.
- The home directory itself, which is two deep on macOS and Linux and so is
invisible to depth.
- Credential directory names, matched against every component so both
"…/.ssh" and "…/.ssh/keys" are refused. This list is deliberately additive
so extending it needs no design authority.
A path that is or contains the cache directory is refused outright: indexing
such a tree would pull every other project's graph database into this
project's index.
Classification order is load-bearing and commented as such. The home
directory normally contains the cache directory, so testing the cache first
reported every home as "holds the cache" and made it non-overridable, which
contradicts the intent that a person may override it. Depth precedes the
cache test for the same reason: "/Users" satisfies both and "too broad" is
the reason that helps the reader.
Takes home and cache as parameters rather than reading the environment, so
the policy is a pure function and the tests need no filesystem. Twelve table
tests; three of them failed on the first implementation and drove the
ordering fix above.
Not yet wired to any caller — deliberately, so the policy can be reviewed on
its own.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
|
||
|
Martin Vogel
|
66e9aec28d |
Merge pull request #1392 from DeusData/feat/generated-client-adapters
feat(install): generate pi and OpenCode extensions from the tool registry |
||
|
Martin Vogel
|
b33a4de9bc |
fix(build): keep alignment checking off vendored tre.o; make SANITIZE rebuild
Two defects, one found by the other. The crash: every fixture-indexing suite (mcp, incremental, index_resilience) died on the Windows/ARM64 leg with STATUS_ILLEGAL_INSTRUCTION and no diagnostic. Cause: an earlier commit in this branch re-armed UBSan's alignment check on the vendored TRE regex engine, and CLANGARM64 builds with -fsanitize-trap, where a trap IS an illegal instruction with nothing printed. macOS and Linux stay clean under the same check because Windows is LLP64 -- 32-bit long -- so TRE's struct layouts and access widths differ there and only there. TRE is vendored third-party code we do not modify, so suppressing the check for that one object is the honest scope; every other object keeps alignment armed. Verified by bisect: origin/main, |
||
|
Martin Vogel
|
bbd13335f5 |
feat(pipeline): delta-merge executor for the closure route
The closure route stops loading and rewriting the world. Its executor is now a dedicated subsystem (pipeline_delta.c + orchestration): CLONE the live generation (copy-on-write where the filesystem offers it), repair the closure against the clone, PATCH exactly the repaired node/edge set in one transaction, and publish through the same sealed-staging finalize leg as the dump path. No full graph load, no full dump, and the general indexing pipeline is untouched -- the profiled kernel run put those two at 187s of a 238s one-file repair whose actual resolution work was 0.6s. Id discipline carries the design. Node ids are AUTOINCREMENT and never reused; the small in-RAM graph is pre-seeded with PROXY nodes carrying their real database ids (SELECT ... ORDER BY id with the id watermark pinned before each insert), and fresh nodes are numbered above the previous generation's MAX(id) -- so "id > max_db_id" is the complete, marker-free definition of what the patch inserts, and every edge endpoint id is database-valid by construction. The inbound-edge snapshot and its QN-keyed re-link become indexed SQL; a re-link whose endpoint no longer exists matches no row, which is full-reindex semantics for deleted symbols. Fail-closed throughout: an unexpected reference to an unseeded label surfaces as a UNIQUE-constraint violation that fails the patch transaction, and EVERY delta failure discards the stage and returns FORCE_FULL_REINDEX -- the live database is never touched, so a full rebuild always self-heals whatever the delta could not do. FTS policy: nodes_fts is contentless, so purged rows cannot be deleted individually on existing databases; their rowids can never alias a live node again (AUTOINCREMENT) and dead entries drop out of the rowid join. The patch inserts rows for exactly the new nodes through the same cbm_camel_split tokenizer the wholesale rebuild uses. The legacy gbuf-based tail reverts to serving only the test-only force_legacy_partial route; the closure orchestration owns its own coverage merge, publication race gate, surface-row merge and committed counts, and publishes with fts_wholesale=false. Gate: the full convergence suite runs against this executor unchanged -- body-edit graph equality with a fresh full index, removed-definition dropping the dependent's stale edge, tsconfig-alias retargeting, the decline matrix, and 510 pipeline/incremental/store/integration tests. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
Martin Vogel
|
94189e6990 |
feat(pipeline): serialize and publish per-file LSP surfaces
Second piece of closure repair. At the collect_all_defs seam -- the only moment the per-file result cache is alive -- both drivers (parallel and sequential) now serialize each file's CBMLSPDef slice to canonical JSON, hash it, and hand the rows to the pipeline; cbm_pipeline_publish_generation writes them into the staging store next to the manifest, so surface data and graph always belong to the same generation. Canonical bytes are the point: every field is written in fixed order with an explicit null for absent strings (NULL and "" differ in the CBMLSPDef contract -- receiver_type NULL means "not a method"), so byte equality IS surface equality and the sha over the bytes is the early-cutoff key. Registry-only labels that pxc_map_label drops but the name registry serves (Field) are folded into the hash as a separate "reg" array, or renaming one would slip past the cutoff. Behaviour pinned in SUITE(pipeline): a fresh full index persists a versioned surface row per file; a BODY edit republishes the identical surface_sha; a SIGNATURE edit changes it. That pair of properties is what the routing layer will stand on. cbm_pxc_collect_all_defs gains an optional per-file prefix array -- the flat all_defs[] otherwise loses the file boundaries the serializer needs. CORRECTION to 6c22338's scope note: it claimed cbm_pipeline_publish_ generation was reachable only behind CBM_INCREMENTAL_TEST_API. Wrong -- dump_and_persist_hashes calls it on every production full index (pipeline.c:1863); the grep that "verified" test-only reachability had excluded pipeline.c itself. The predictable staging name WAS in the production publish path, which makes that fix a real production hardening, not a test-path cleanup. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
Martin Vogel
|
6a0adb4430 |
fix(msan): scope the zstd stdint workaround to the zstd object
The previous fix put -include stdint.h on the lane's global SANITIZE line. That traded one vendored compile break for another: force-including a libc header ahead of every source file freezes glibc's feature-test macros before sqlite3.c can set _GNU_SOURCE for itself, and its view of libc loses MREMAP_MAYMOVE and nanosleep (17 errors on the x86-64 CI leg). The workaround only ever had one legitimate target -- the zstd amalgamation whose MEMORY_SANITIZER block lost its stdint re-include -- so it now rides a per-object hook (ZSTD_EXTRA_CFLAGS) that the MSan lane sets and every other build leaves empty. sqlite3.c compiles exactly as before in every lane. Verified locally that zstd compiles with the hook and the default rule stays untouched; the MSan leg itself is x86-64-only, so CI is its verification venue. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
Martin Vogel
|
7d390c5120 |
fix(ci): unbreak the MSan and diag lanes on x86-64
MSan: vendored zstd fails to compile. Its MSan-only block (guarded by
MEMORY_SANITIZER) declares __msan_test_shadow returning intptr_t and
reaches for the type with
#define ZSTD_DEPS_NEED_STDINT
#include "zstd_deps.h"
but the amalgamator that produced zstd.c collapsed that second include
into a "skipping file" comment, so the define pulls nothing in and
intptr_t is undeclared. Only this lane compiles that block at all, and
only where <stddef.h> does not drag stdint.h in transitively -- which is
why it built on the local aarch64 container and failed on CI's x86-64.
The lane now forces the header. Patching the vendored amalgamation would
be silently undone by the next re-vendor.
diag: detect_invalid_pointer_pairs comes back out. It fires during static
initialisation inside vendored simplecpp -- a std::string global at
simplecpp.cpp:101 -- with a second "pointer" of 0xfffffffffffffff3, a
sentinel rather than an address: libstdc++ string internals, not
anything this codebase wrote. It is a process-wide runtime flag with no
per-file scoping, so unlike the analyzer's path filter it cannot be
aimed away from vendored code. Keeping it would mean a permanently red
lane reporting a non-defect, which is how a lane gets ignored. The
instrumentation it needed comes out with it.
The other three off-by-default checks stay: stack-use-after-return,
stack-use-after-scope, strict-string-checks. Those are the ones covering
bug classes nothing else in the matrix looks for, and none of them
fired.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
|
||
|
Martin Vogel
|
0c223d9027 |
build(lint): pin analyzer suppressions to the code they argue about
The memory gate stays gating, and it now has a way to record a genuine false positive that cannot quietly outlive its own reasoning. A whitelist entry names one (file, function, check) and carries two things: why the analyzer is wrong, argued from the code, and what was tried before concluding that. The entry is pinned to the sha256 of that function's text. Edit the function and the entry stops counting -- the finding comes back and has to be argued again against the code as it now is. This is the part that matters. A suppression that survives the code it was written about reads as "reviewed" while being nothing of the kind, which is worse than no suppression at all. The gate fails on: a finding with no entry, a finding whose entry has gone stale, and an entry that asserts rather than argues (there is a floor on how much reasoning an entry must actually contain -- the mechanical half of "argued, not asserted"; whether the argument is correct stays a review question). An entry that matches no finding is reported but does not fail, because analyzer versions differ across platforms. NOLINT is still not honoured here and the gate does not read it. The whitelist ships empty: the analyzer is currently clean across LINT_SRCS, so nothing is being suppressed today. The mechanism exists for the first finding that genuinely warrants it. Verified by exercising each path rather than only the clean one: an unaccounted finding fails and names its enclosing function; an argued entry with a matching hash passes; the same entry fails as STALE once the hash no longer matches the function; and an entry that says only "false positive" fails for not arguing its case. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
Martin Vogel
|
0fe0f453be |
build(test): close the gaps in the sanitizer matrix
Every lane here was either asserting coverage it did not have, or was reachable only by bypassing the entry point that is supposed to define the ladder. TSan: no exclusions any more. The three suites the Makefile documented as excluded are back. daemon_ipc and daemon_frontend no longer reproduce the harness race and thread leaks they were excluded for. daemon_runtime did not deadlock as the comment claimed -- it reported a real production data race on the log sink, fixed separately. Excluding a suite from a sanitizer lane hides exactly the class of bug the lane exists to find, so the comment block now records what was actually true rather than what was assumed. TSAN_OPTIONS gains report_thread_leaks=0. This disables the thread- HYGIENE check only; race detection is untouched. Several daemon fixtures fork after the process has gone multi-threaded, and in the forked child TSan sees the parent's already-finished threads as never-joined even where the fixture joins them. It fires on macOS and not Linux, i.e. it tracks fork semantics rather than anything about this code. The alternative was dropping whole suites, which costs real race coverage; this costs none. UBSan: tre.o no longer builds with -fno-sanitize=alignment. Alignment was switched off for a vendored regex engine that ships in the product, which is where the check is least redundant, not most. LSan on macOS: new test-lsan target and test-lsan-macos CI leg. LSan is on by default under ASan on Linux, so the Linux legs have always had leak coverage. On macOS it is off by default and Apple's clang refuses to enable it outright, so that platform had none at all. Apple's refusal is not a darwin limitation -- upstream LLVM supports LSan on darwin/ arm64. The lane is the ordinary ASan suite built with Homebrew LLVM and run with detect_leaks=1; it runs the full suite clean and was checked to still catch a deliberately leaked allocation. MSan: reachable from the local ladder. The image and compose service existed but run.sh had no leg, so the only way in was to drive docker compose by hand -- which means it was not part of the ladder in any meaningful sense. The image also moves to clang 22, matching the diag and analyzer lanes instead of sitting four majors behind on noble's default. The leg documents the aarch64 shadow-mapping failure so a local arm64 stack overflow in the grammar suites is not mistaken for a code defect; the GitHub leg runs x86-64, which is the mapping that matters. Off-by-default ASan checks: the diagnostic lane, and its CI twin, now run detect_stack_use_after_return, detect_stack_use_after_scope, detect_invalid_pointer_pairs (with the -fsanitize=pointer-compare, pointer-subtract instrumentation it requires) and strict_string_checks. Running ASan is not the same as running all of it, and these four cover bug classes nothing in the matrix was looking for. They stay on the diagnostic lane rather than the gating ones until they have a clean history there; promoting them is a separate deliberate step. Verified: macOS TSan 940 passed / 3 skipped / 0 races over the full suite set; the macOS leak lane 7375 passed / 4 skipped / 0 leaks, with LeakSanitizer confirmed armed under that exact toolchain and option set by checking it still reports a deliberately leaked allocation. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
Martin Vogel
|
c9886d4f25 |
feat(ci): MemorySanitizer lane — instrumented libc++/zlib image, full C++ coverage
Stage 2 of the memory-diagnostics program (user decision: go directly to the instrumented image rather than a C-only probe). MSan detects uninitialized READS, the one memory-error class no other lane covers dynamically, and it requires every linked library to be instrumented -- vendored C deps compile in-tree and instrument for free; the two external links do not: - test-infrastructure/Dockerfile.msan: pinned-base image building libc++/libc++abi/libunwind (llvmorg-18.1.8, LLVM_USE_SANITIZER= MemoryWithOrigins) and static zlib v1.3.1 into /opt/msan, with the symbolizer and MSan runtime in a separate last layer so tool additions never invalidate the ~30-min libc++ build. - scripts/msan.sh: the canonical lane entry. ALWAYS clean-builds its BUILD_DIR: make does not encode flags into dependencies, and a stage-1 probe's libstdc++ objects surviving into the libc++ lane produced a convincing-looking uninitialized-value report at preprocessor.cpp:168 -- the uninstrumented .so string constructor wrote the temporary, the instrumented move constructor read it. The clean rebuild proved it an artifact: extraction (incl. the C++ preprocessing path) runs 272/272 with zero reports. - Makefile.cbm: CXX_STDLIB / CXX_STDLIB_FLAGS hooks so the lane can swap libstdc++ for the instrumented libc++ (defaults identical; the shipping build is byte-for-byte unaffected). - docker-compose test-msan service: same aarch64 seccomp/setarch remedy as the TSan service (MSan's shadow layout hits the same personality() block). - CI test-msan job (_test.yml): buildx local-cache via the repo's existing pinned actions/cache -- no new third-party action pins; a warm run skips the libc++ build entirely. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
Martin Vogel
|
80afcd6bbe |
fix: close the memory-error paths the clang-analyzer lane surfaced
The memory-diagnostics report's priority-4 lane (path-sensitive clang-analyzer, memory checks only) run over all 111 production files. 21 findings triaged; the real ones, all cold-path (none can explain #581's per-query residual): LEAKS - mcp get_architecture: scope_path leaked on the missing-store early return (REQUIRE_STORE frees only `project`); allocate after the gate. - pass_definitions: cancellation mid-extraction leaked the pass-owned result cache including already-extracted entries; mirror the end-of-pass cleanup. - store package-boundary scan: the row-scan abort path freed the node arrays but not the boundary accumulators or their duplicated package strings. - cbm quarantine set: a duplicate path line leaked the replaced value (and a fresh key copy -- the table borrows key pointers); a partial strdup failure leaked the surviving half. Reuse the stored key for duplicates. - pass_githistory: unchecked malloc/strdup -- an OOM dereferenced NULL and a failed strdup leaked the index cell. Allocate before claiming the slot. NULL/UB - cli config subcommand: NULL argv with nonzero argc slipped the guard (the inner `argv &&` shielded only the help comparison) into argv[0]. - store bfs_multi: a negative max_results broke out before any row was written, then freed fields of an unwritten negative-index slot. Clamp. - pass_calls emit_http_async_edge: the service-pattern call sites pass a NULL target behind a hand-duplicated URL predicate; a drift between the copies turned target->id into a null deref. The callee is now total. - sqlite_writer: both leaf-array OOM paths left leaf_count stale with a NULL array, walking pb_finalize_* into leaves[0]; consistent empty state routes them to the existing root=0 failure return. HARDENED (invariants true but invisible to path-sensitive analysis) - Leiden CSR + aggregate arrays, SCC adjacency: calloc + endpoint guards, so a future degree/collection miscount degrades benignly instead of UB. - SCC cycle fill: the ncyc==0 no-slot invariant made local. RECORDED FALSE POSITIVES (no code change) - yaml sequence starts (loop bound == alloc bound), cypher agg arrays (same count both sides), mcp read_message ch (assigned by fgetc each iteration), pkgmap clean buffer, mcp csize (Tarjan: ncomp>=1 when nverts>=1), vendored verstable x2. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> GATE + LANES (user decision: runner cost accepted) - make lint-mem (local triage) and lint-mem-ci (gating: vendored-filtered, any remaining finding fails). The gate is green because every false positive above was restructured for provability -- calloc'd fill-cursor arrays, explicit Tarjan invariant, zeroed buffer tails, min-1-element allocations -- never suppressed. - make diag: pinned newest-LLVM ASan/UBSan lane with straighter stacks. - CI: lint-mem job (_lint.yml) and test-diag job (_test.yml), both on the pinned LLVM 22 apt toolchain. Cost disclosure: roughly +25-40 min and +25-60 min (ccache-warm) per push respectively. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
Martin Vogel
|
f056aa938d |
fix(build): track internal/cbm/lsp headers in whole-program targets
The branch adds internal/cbm/lsp/type_registry.h, and PROJECT_HDRS covered $(CBM_DIR)/*.h but not the lsp/ subdirectory -- so editing an lsp header rebuilt nothing and an incremental build silently tested stale header values. The branch's own invariant (repro_make_tracks_headers.sh) caught it on the repro-unix CI jobs; the same trap class as the lsp_all.o one fixed by #662. bash tests/repro/repro_make_tracks_headers.sh: green. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
Martin Vogel
|
029416fda6 |
Merge remote-tracking branch 'origin/main' into qa/call-reference-rebase
# Conflicts: # Makefile.cbm # internal/cbm/extract_defs.c # src/foundation/compat_fs.c # src/foundation/compat_fs.h # src/pipeline/pipeline.c # src/pipeline/pipeline_incremental.c # tests/test_main.c # tests/test_pipeline.c # tests/test_store_checkpoint.c |
||
|
Martin Vogel
|
2a353b2a33 |
feat(cli): generate client extension modules from the tool registry
pi has no MCP client, and OpenCode has no declarative hook configuration -- verified against OpenCode's own plugin documentation, which states hooks are available only through JavaScript/TypeScript plugin modules. For those two clients a module is the only extension point. We do not want to ship such a module as a repository asset. #534 proposed one embedding 380 lines of TypeScript as C string literals and registering 7 of the 15 registry tools; every tool added afterwards would have been silently missing for that client. Hand-maintained copies of the tool list have already produced defects here (#1361, and the smoke-invariants count). So the module is GENERATED from the live registry instead. cbm_client_adapter_pi walks cbm_mcp_tool_count()/cbm_mcp_tool_name() and registers every tool, which makes drift structurally impossible rather than merely discouraged: adding a tool to TOOLS[] adds it to every generated adapter with no second edit. Nothing in the repository is a .ts file. Both emitters wrap their output in ownership markers so a caller can rewrite its own block and leave a user-authored file alone. Two defects from the prior proposals are closed by construction: - Path escaping. #616's template rejected the double quote but not the backslash, so a Windows home like C:\Users\urs\bin produced an invalid unicode escape and the whole auto-loaded plugin failed to parse, which is worse than an absent plugin. cbm_client_adapter_escape_js escapes backslash, quote, newline and CR, and fails closed rather than emitting a truncated literal; generation aborts if the path cannot be escaped. - The silent no-op. #616's payload omitted hook_event_name, which hook-augment requires and without which it accepts nothing, so the plugin emitted zero bytes for six weeks. The generated payload carries it and a test pins it. Recorded honestly: the OpenCode emitter hooks tool.execute.after, whose ability to modify a tool's output is NOT part of OpenCode's documented plugin contract (only tool.execute.before's argument mutation is). If they change it the augmentation stops with no error. That risk is accepted deliberately and the emitter says so in a comment, so a future reader does not have to rediscover it. This commit adds the generator and its tests only; wiring into the install and uninstall routines follows separately so the two are reviewable apart. Tests: every registry tool appears in the pi module (revert-checked by simulating the 7-of-15 subset, which fails); Windows/quote/newline escaping and its truncation boundary; the OpenCode payload carries hook_event_name and registers no tools; NULL/empty binary paths generate nothing. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
Yyunozor
|
d899bf44f7 |
fix(hooks): scope pre-commit clang-tidy to staged changes (#1264)
make -j3 -f Makefile.cbm lint runs the full lint-tidy sweep, which currently surfaces ~5,100 pre-existing clang-tidy findings across 113 files. Since the pre-commit hook (scripts/hooks/pre-commit) runs that target unconditionally, it blocks every commit for every contributor who has clang-tidy on PATH, regardless of what the commit touches. Add lint-tidy-diff (scripts/lint-tidy-diff.sh), which runs the same clang-tidy binary/config but passes clang-tidy's own -line-filter so only lines the commit actually added or modified can produce a diagnostic. Pre-existing findings on untouched lines are not reported, whether they are in an untouched file or on an untouched line of a file the commit does touch. The hook now runs lint-ci (unchanged: cppcheck + clang-format + no-suppress) plus lint-tidy-diff instead of the full lint target; make lint / make lint-tidy / scripts/lint.sh are unchanged and remain the full-tree audit. Signed-off-by: Yyunozor <yyunozor@icloud.com> |
||
|
Martin Vogel
|
619f702e88 |
build: rebuild when a project header changes
test-runner, test-repro-runner, test-runner-tsan, codebase-memory-mcp and test-foundation are each built in a single compiler invocation from their sources, and their rules listed only .c files as prerequisites. Editing a header therefore changed nothing make could see, so the target was not rebuilt and an incremental build silently kept testing the OLD header value. Found while revert-checking the label-allowlist change in this branch: the check passed, which was wrong. Shrinking CBM_SQL_TYPE_LIKE_LABELS should have turned the pin test red and did not, because nothing had recompiled. Forcing a rebuild produced the expected failure. A revert-check that cannot fail is worse than no revert-check, because it reports confidence it did not earn. This is the same class as the lsp_all.o staleness that #662 fixed with an explicit dep list; that fix was per-object, this one covers the whole-program targets. PROJECT_HDRS globs src/, internal/cbm/ and tests/ headers and is added to the five affected targets. Vendored trees are deliberately excluded: they are pinned, they already carry explicit dep lists (LSP_UNITY_DEPS, TS_RUNTIME_DEPS), and globbing tens of thousands of grammar headers would cost more than it protects. Verified: editing only src/foundation/constants.h, with no source file touched, now rebuilds and turns the pin test red -- the exact edit that was invisible to make before. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
Martin Vogel
|
1f674c805e |
harden: remove capability that should never have shipped
Microsoft's ML flagged the rc.1 release binaries. The decisive evidence is that the SAME sha256 went from 0/62 clean to Microsoft-detected in about an hour with no byte change, so the verdict lives partly in scanner-side state and no code change can promise a clean result. What code CAN do is stop shipping things that have no business in a release artifact, which is worth doing on its own merits and incidentally widens the classifier margin. Every claim below is verified against a built binary by the new gate, not by reading source. Executable stack (the worst of the findings). vendored/nomic/code_vectors_blob.S is the only assembly in the build and carried no .note.GNU-stack. An unannotated object makes ld assume the worst for the whole link, so EVERY Linux release we have ever shipped had GNU_STACK RWE. Adds the note (cause) plus ELF-only -Wl,-z,noexecstack (outcome); the gate fails the release if it returns. Test seams are now opt-in, never opt-out. TEST_SEAMS=1 defines CBM_ENABLE_TEST_SEAMS; without it the crash-orphan probe -- which forks a child that ignores SIGTERM and loops forever, then writes its pid to a caller-supplied path -- and the lease-ownership marker compile to trivial stubs, so call sites are untouched and the binary holds no fork, no signal handler and no env-var string. Opt-IN is the point: forgetting the flag yields a clean binary rather than a leaky one. scripts/test.sh requests it in the leg that consumes it, and tests/test_worker_watchdog.sh now asserts the capability up front instead of dying later with an opaque "Killed: 9". The daemon's background version check is gone. It spawned curl against api.github.com/repos/.../releases/latest on the first eligible session of every run to say "a newer version exists" -- a release URL and an outbound request in every shipped binary, for something the install scripts already report. The INJECTABLE SEAM survives: update_ops is still honoured, the fakes in tests/test_daemon_application.c still cover notice/ownership/cancellation/replay, and with no provider application_update_subscribe_locked returns early so no generation ever starts. "No network request by default" is now structural. Dead capability out of release builds. The tar.gz/zip extraction block (gzip_decompress through cbm_extract_binary_from_zip, plus its cli.h declarations) moves under CBM_CLI_ENABLE_TEST_API -- verified self-contained, zero uses of any helper outside it, only callers the excluded updater and tests/test_cli.c. Downloading an archive, decompressing it, picking an executable out of it and marking it executable is the canonical dropper composite; it is now absent rather than merely unreachable. SQLite is built with -DSQLITE_OMIT_LOAD_EXTENSION (no caller of load_extension anywhere in src/ or internal/), removing that API surface and part of the dlopen/dlsym surface. Temp files and environment scanning (S2/S3). Predictable paths in mcp.c, artifact.c and diagnostics.c are created privately and exclusively and written through the returned descriptor; pass_envscan.c no longer descends symlinked directories out of the project root, and its fixed 512-byte path buffers no longer truncate into pointer arithmetic that could land outside the buffer. Build-time entropy. mimalloc's version banner baked __DATE__/__TIME__ into every binary, so two builds of identical source seconds apart could never share a hash and no release could inherit a false-positive determination made about its predecessor. Local patch removes it (marked to survive refreshes), -Wdate-time makes any future use a build error, and -Wl,--no-insert-timestamp stops the PE header carrying the link clock. scripts/ci/check-binary-composition.sh is the proof that each removal stays removed, wired into package-release.sh after strip so the local artifact-flow smoke enforces exactly what CI does. It asserts absences plus a CANARY string, so handing it a compressed, stubbed or empty file fails instead of passing vacuously, and a missing tool is a hard error -- a skipped assertion must never look like a satisfied one. Two build-system traps found by that gate, both of which had silently defeated a fix: the product binary is compiled in one shot from sources, so a flag flip did not rebuild it (now tracked by a .build-config stamp that also removes the binary, making it independent of mtime granularity); and prod_sqlite3.o / prod_mimalloc.o depended on a single named source, so SQLITE_OMIT_LOAD_EXTENSION and the mimalloc patch BOTH compiled to nothing on the first incremental build. Source review would have called them done. Deliberately NOT changed. Three seams stay in release artifacts because scripts/smoke-test.sh runs against the real artifact and needs them: CBM_TEST_CRASH_ON and CBM_TEST_HANG_ON inject the faults that prove supervisor recovery, and CBM_TEST_WINDOWS_USER_PATH_RUN_ID is what keeps the PATH smoke from writing the tester's actual PATH. The gate treats those as an allowlist, so a NOVEL seam still fails. The true no-UI standard build is deferred rather than rushed: src/ui/* is in PROD_SRCS and four files outside src/ui reference UI symbols, including the daemon that serves the UI, so that assertion reports instead of failing until the split lands -- a gate everyone knows is red teaches people to ignore gates. No grammar is removed. ObjectScript accounts for essentially all binary growth since the last provably-clean release (+21.4MB rodata, +1.1MB text from two four-line shims), which made it the obvious ablation candidate, but a dry run performed twelve real Defender endpoint scans across standard/UI and amd64/arm64 with ObjectScript, the daemon and the expanded hooks all present and every scan was clean. Nothing there is a deterministic trigger, so cutting a community-contributed language would spend a real feature on unproven margin. Lean is not a candidate either: at 99.6MB of source it is by far the largest grammar, but it shipped in v0.9.0 which scanned 20/20 clean, so removing it would produce a novel unscanned profile instead of restoring a known-good one. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
Martin Vogel
|
a54ea95719 |
fix(windows): ship one binary — remove the launcher stub flagged as a dropper
DCO / dco (push) Has been cancelled
Windows shipped a PAIR: a small permanent launcher (codebase-memory-mcp.exe)
plus the real product binary (codebase-memory-mcp.payload.exe). The launcher
existed for exactly one reason — a running .exe cannot replace its own image
on Windows, so an in-process self-update needs a second resident binary to do
the swap.
That stub is statically indistinguishable from a dropper: a small, unsigned,
zero-prevalence PE whose whole job is verify-and-execute another binary.
Defender's ML scored it Trojan:Win32/Wacatac.B!ml and blocked the v0.9.1-rc.1
release at the VirusTotal gate. It is not fixable in our code on x64 —
bcrypt-free, stripped, VERSIONINFO'd, minimal-resource and even
resource-FREE builds on CI's own MSYS2 CLANG64 toolchain were all flagged,
while the product binary scans clean on every platform.
So remove the stub and move self-update OUT of the process into install.ps1,
which runs while cbm is NOT running: Windows' image lock only blocks a
process from replacing ITSELF. now prints the exact PowerShell
command (with the Unblock-File hint for Mark-of-the-Web); install.ps1 is
idempotent, so re-running it IS the update — it stops the daemon, renames the
running binary aside (the one mutation Windows permits on a running image),
publishes the new one, and sweeps retired copies.
Windows now matches Linux and macOS: ONE binary per platform.
* packaging, install.ps1, npm and PyPI wrappers all carry a single binary
* the launcher/payload ABI contract and ~2500 lines of stub state machinery
are deleted
* every daemon start, CLI call and hook fire loses a process spawn, a named
pipe handshake and an stdio relay
* test_windows_bundle_contract.sh is rewritten as an INVERTED contract: it
now asserts no shipped surface can reintroduce a launcher/payload pair,
and that install.ps1 retires the running binary before publishing
Verified: VirusTotal 0/67 on the packaged binary and 0/58 on install.ps1 (no
certificate involved); macOS and Linux full suites green; Windows guards all
green including the new update-handoff contract; npm 10/10; PyPI 3/3.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
|
||
|
Martin Vogel
|
375d11e3e4 |
revert(diagnostics): take the allocation profiler out of the build
The hosted windows-11-arm shards crash at process load (STATUS_ILLEGAL_INSTRUCTION, pass=0, secs=0) while main passes them, and removing the TLS detach callback did not help -- so the remaining candidate is this layer, which touches every allocation on Windows through the interposer. It declares a _Thread_local guard read on every malloc, and thread-local storage in a static MinGW aarch64 link is exactly the fragile case; nothing in it needs to ship. The profiler did its job: it produced the attribution that identified #581's mechanism, and that analysis is recorded. Sources stay in the tree, unbuilt, so a future investigation can re-enable them deliberately. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
Martin Vogel
|
412f2d23b4 |
fix(build): keep the POSIX wrap shim out of sanitized builds
The shim exists only so the allocation profiler has an observation point on Linux. Applying its --wrap flags to the test and TSan link redirected malloc into mimalloc underneath ASan's own interception, putting two allocators on the same pointers. Production keeps the shim; sanitized builds do not. Windows is unchanged -- there the wrap flags are what make mimalloc own allocations. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
Martin Vogel
|
6a483344c6 |
feat(diagnostics): deterministic memory-attribution driver
Fixed corpus, fixed request count, one process, no daemon handshake and no reindex or crash-recovery phases, so two runs differ only where the code does. Requests go through a request/response driver rather than a batched pipe: closing stdin makes the server treat it as a client disconnect and shut down before answering, which yields a census of a process that never worked. Linux gains a measurement-only --wrap shim so the profiler can observe allocations the way the Windows interposer already does; macOS has no --wrap and stays census-only, which the driver reports rather than passing off as zero. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
Martin Vogel
|
7a2376987e |
feat(diagnostics): allocation-site profiler for the #581 analysis harness
The census could say which POOL held memory but never which allocation put it there, so each hypothesis needed a code change plus an eight-minute soak to test and the investigation could only ask one yes/no question at a time. Records allocations at or above a threshold (default 32 KiB, since arena fragmentation is driven by large transient runs) against their captured stack, into fixed tables whose exhaustion is counted and reported rather than silently dropped. Emitted alongside every census sample, so pool totals and attribution always describe the same instant, plus an explicit unattributed remainder so incomplete coverage announces itself. scripts/memlab-report.py ranks sites by retained-byte growth using a linear fit and can diff two platforms' profiles against each other. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
Martin Vogel
|
d7d3ae407e |
wip: scope wrap flags, fix msize recursion
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
Martin Vogel
|
1c71b99774 |
wip: route windows allocations through mimalloc via --wrap
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
Martin Vogel
|
62821375c4 |
fix: harden daemon release paths and verification
DCO / dco (push) Has been cancelled
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
Martin Vogel
|
b98f6a99c4 |
test: sanitizer-keyed budgets and a causality-based httpd interruption proof
The build system now stamps CBM_SANITIZED_BUILD for any non-empty SANITIZE, so timing budgets key off build-system truth instead of compiler-specific __SANITIZE_ADDRESS__ probes that miss trap-UBSan and TSan; the two bench suites use it. The httpd interrupt test proved 'the interrupt worked' with a wall-clock bound, which is a lottery under sanitizers. A send-deadline test hook pins the deadline out of reach so the join itself is the causality proof, and the stop watchdog becomes a true ten-second hang detector. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
Martin Vogel
|
ece838c525 |
test: widen ThreadSanitizer coverage and add native ARM64 Windows UBSan
Close the cross-platform sanitizer gaps that were leaving real concurrency and undefined-behavior bugs uncaught, and fix a data race the first widened run surfaced. - tsan: the data-race gate ran three suites (mem, slab_alloc, parallel) over no real threaded production code. It now covers every threaded surface that runs clean and stable under TSan: the parallel-extraction worker pool (parallel, worker_pool, pipeline), the filesystem watcher, the embedded HTTP server (httpd), diagnostics sampling, the MCP server and mutation guard, subprocess supervision, and the runnable daemon-coordination paths (daemon, daemon_application). daemon_runtime (deadlocks under TSan+fork), daemon_ipc and daemon_frontend (test-harness synchronization, not production) are excluded with the reasons recorded in the Makefile. - tsan: fixed a genuine data race the widened gate immediately found — cbm_lsp_max_walk_depth's lazy cache was read and written by parallel LSP-extraction workers without synchronization. A data race is undefined behavior even when every worker computes the same value, so the cache slot is now a relaxed atomic: a plain load on the hot path, and a first-touch double-compute simply stores the same value. - tsan(ci + local): the test-tsan job now runs on Linux amd64, Linux arm64, AND native ARM64 macOS (the threading code is shared, so a race is usually caught on all three, but scheduler differences let each surface one the others miss). The local ladder gained `run.sh tsan` and `tsan-amd64` plus the matching compose services. TSan's shadow memory aborts under modern high-entropy ASLR, so the containers run under `setarch -R` with an unconfined seccomp profile (the personality syscall is otherwise blocked) and the CI Linux legs lower vm.mmap_rnd_bits first; amd64 TSan cannot run under x86_64-on-ARM translation and is a real-hardware/CI gate only (documented in run.sh). - ubsan(win/arm64): native ARM64 Windows had no sanitizer at all — AddressSanitizer ships no aarch64-w64-windows-gnu runtime. UBSan in trap mode (-fsanitize-trap=undefined) needs no runtime library, so it instruments natively and turns undefined behavior into an illegal-instruction trap; -fstack-protector-strong adds stack-smash coverage the heap tools miss. The GitHub windows-11-arm leg switches from unsanitized to this, and vm/win.sh gains trap-ubsan-build / trap-ubsan-test for local iteration (reproduce under the emulated x86_64 UBSan to see which check fired). The whole codebase builds and runs clean under it. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
Martin Vogel
|
0baddf7887 |
daemon: permanent lifecycle, daemon-backed CLI/hooks, real-Windows hardening, long-path launcher transactions
Daemon lifecycle and Windows correctness, verified on a real Windows 11 ARM64 VM through the maintained test-infrastructure/vm drivers, plus the macOS and Linux arm64 suites and the container lint gate. Daemon lifecycle: - daemon start/stop/status subcommands. `daemon start` launches a PERMANENT daemon (spawn shape is byte-exact argv; survives idle periods and session ends) and reports an already-active daemon instead of failing. Permanence is honored at every stop latch: last-committed-client disconnect, host initial-client window, coordinator release, and application final-session close — a permanent daemon also keeps admitting new sessions after its last one closes. - daemon stop refuses while sessions are active and lists the blocking peers (pid/role) that must finish first; an idle daemon drains through the activation-shutdown machinery with the ACK ordered after connection interrupts. A second stop is idempotent. The wire ops are no-cohort first-frame requests with peer fingerprint authentication, so stop/status never conflict with an exact-build admission gate. - One-shot CLI commands now execute through the daemon (index workers keep their local supervised path). A cold CLI run that had to spawn a temporary daemon prints a hint that `daemon start` removes the per-command startup tax; a warm daemon is recycled silently. - Hooks are connect-only fail-open: with no daemon present the hook emits a visible, rate-limited notice (Claude-dialect systemMessage plus stderr for other dialects) and always exits 0 — augmentation is never allowed to block the caller's tool use. - Version skew: a newer-build client automatically drains an older-build permanent daemon (strict semantic-version triples only; dev builds never auto-drain) and the build-conflict message names `cbm daemon stop` as the manual escape hatch. Windows IPC/runtime (real-VM verified): - ipc(win): persistent pending overlapped ConnectNamedPipe. The accept path used to destroy its listening pipe instance on every 20 ms poll timeout; a client attaching in the teardown window was severed or left on an orphaned pipe object whose HELLO no server handle could ever read, absorbing the connect until the client's own timeout expired. The pending connect now survives poll timeouts and nothing is destroyed while a client could be attaching. - ipc(win): drain-before-close for final responses. Closing a named-pipe server handle can discard a just-sent response before the peer reads it (POSIX stream sockets never lose buffered data on close). A bounded cbm_daemon_ipc_connection_drain (read-until-peer-EOF; no-op on POSIX, immediate on interrupted connections) now precedes close in runtime_worker_finish and runtime_reject_inline, so hello-conflict, capacity and disconnect acknowledgements reliably reach the peer. - runtime: CLOSE_INTENT wire frame. A Windows named-pipe client has no transport half-close, so close_begin now announces departure with an explicit frame (ordered after APPLICATION_CANCEL, before the local interrupt); the server releases the client's admission on receipt instead of waiting for the handle to close. Admission-drop timing is now identical to POSIX shutdown() semantics on every platform. - runtime(win): client close cancellation. close_begin serializes with request publication under the send lock, best-effort sends the active token's APPLICATION_CANCEL frame, then interrupts local I/O; the server cancels MCP/subprocess work promptly. Contract tests accept both correct outcomes (interrupted transport or decoded CANCELLED). - runtime: activation acknowledgement ordering. The activation ACK is the requester's license to act on "snapshotted and draining", so every connection interrupt is now initiated before the ACK is sent; a session could previously get one more request serviced after the requester observed the ACK. - service(win): deadline-bounded private-file prepare. The conflict-log prepare retry loop (100 x Sleep(2), which rounds up to the ~16 ms timer granularity) burned ~1.6 s against permanently obstructed paths, stalling hello rejections past the client's timeout. The retry budget is now a 250 ms deadline; transient share collisions still retry. - subprocess(win): cmd.exe /C payload encoder quotes metacharacters correctly (root cause of the git-on-Windows failure cluster). - watcher: SHA-256 buffer sizing (CBM_SZ_64 -> CBM_SZ_128) and a native Windows stop/unwatch cancellation test with exact-image verification. - httpd: send_all writes in bounded 64 KiB slices. A single giant nonblocking send() on Windows is absorbed wholesale into AFD kernel buffering regardless of SO_SNDBUF, so send deadlines and interrupts could never engage against a slow peer (and the full payload was pinned in nonpaged pool). Slicing restores a deterministic backpressure point; a test hook pins SO_SNDBUF for the deadline and interrupt tests. - ui/http: shutdown lifecycle — interrupt checks, response-wide send deadline, explicit connection states, refusal to free a server while a listener-owned connection is active. Windows long-path support: - Central path-aware wide conversion (canonicalize via GetFullPathNameW and prepend the extended-length prefix for absolute paths >=240) at the compat chokepoints (cbm_fopen/compat_fs/mkstemp/mkdtemp), sqlite store opens, and the daemon build-fingerprint/log paths. Deep managed installs (a 64-hex generation directory routinely exceeds MAX_PATH) now index, stage and activate correctly. - activation transaction: its own file APIs and the component-walking ancestry validators now operate in the extended-length namespace; the launcher path is canonicalized (and prefixed when deep) once at entry so every downstream exact-string comparison stays form-consistent. - Executable self-resolution uses the wide APIs (GetModuleFileNameW, GetFileAttributesW) so non-ASCII install paths survive argv[0] resolution. Windows launcher install/uninstall transaction: - FileRenameInfoEx names are NUL-terminated in an over-allocated buffer. FileNameLength governs per the contract, but filter drivers read FileName as NUL-terminated and appended adjacent heap bytes to created names — a flaky, garbage-suffixed rename target. Both the CLI and the launcher rename helpers are fixed. - Uninstall retires state via rename-aside (.cbm -> .cbm-retired-v1-<tag>-<pid>) with the retired tag shortened to 16 hex chars so the bare rename target stays under the FileRenameInfoEx NT-conversion ceiling at guard depths; 64 bits still uniquely identify the generation. - When the running launcher's mapped generation backings pin .cbm against rename, the backings are relocated to activation-<pid>-N .retired tombstones beside the install (a mapped image may be renamed, never deleted; the launcher's liveness-guarded sweep reclaims stale tombstones). Every relocation is recorded, and a FAILED uninstall reverses the moves after restoring .cbm — via MoveFileExW with extended-length paths on both arguments, since the deep generation target is beyond the handle-based rename's bare-path reach — so a restored install keeps its generation backings and stays runnable. - After a committed uninstall the retired tree's backings are relocated out so the tree is shallow enough for the detached cleanup's rd, and the cleanup's working directory strips the extended-length prefix (CreateProcessW lpCurrentDirectory silently ignores prefixed paths). - Files created under Administrators-default-owner directories (CopyFileW destinations, CREATE_NEW tombstones, probe directories) are explicitly owner-stamped so the exact-owner validators hold on runner images; guard fixtures stamp hand-built trees the same way. Diagnostics, tests and infra: - diagnostics: discovery is now an always-delivered JSON control record (new cbm_log_control) that survives CBM_LOG_LEVEL suppression and paths containing spaces; placement honors $TMPDIR with /tmp fallback via a diagnostics-local helper; the soak parser reads the JSON record; documented in docs/CONFIGURATION.md. Red-first coverage for suppressed log levels, TMPDIR-with-spaces, and native Windows output-contract assertions. - tests(win): daemon_ipc/daemon_frontend fixtures now build endpoint parents with production-shaped ancestry (LocalAppData on Windows, via th_secure_runtime_parent_new) — the runtime ancestry validation correctly refuses temp roots whose ancestors grant mutation rights to Authenticated Users (C:/msys64/tmp, GitHub-runner work dirs) — and drive the documented startup-owner publication flow before reading generation-bound endpoint addresses. This turns the 26 Windows failures previously visible in CI's full-test job green without weakening any validation. - tests(win): the launcher guard covers the full permanent-launcher contract including failed-uninstall restore and immediate reinstall after uninstall; new daemon lifecycle and reworked hook-augment guards run the start/recycle/stop flow end to end. - tests: CBM_SKIP_PERF is now actually consumed by the test runner (it was set by CI but never read, so perf suites ran everywhere); four throughput/bench suites are classified as perf, the heavy store_arch suite moved to the slow-timeout tier, and two wall-clock-sensitive assertions were rewritten as invariant checks with coarse hang-detector backstops. - build/test infra: build-dir safety contract, UI dev-proxy security contract, soak daemon-recovery contract, path-safety helper, the Windows VM worktree-sync contract wired into scripts/test.sh, and vm/win.sh guards building its clean embedded-UI product in an isolated BUILD_DIR so it cannot clobber the incremental test build. provision-windows.sh now installs Node.js for the guards UI build. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
Martin Vogel
|
87a0e3f74f |
fix: preserve semantic graph relationships
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
Martin Vogel
|
3bd9e872ac |
Merge origin/main into feat/shared-coordination-daemon
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> # Conflicts: # Makefile.cbm |
||
|
Martin Vogel
|
bd939fc49a |
perf(test): run suites as parallel processes — same gates, ~2.5x faster
The runner executed all 104 suites sequentially (~10 min locally, the bulk of every CI test leg). The suites are process-isolated already (per-process mkdtemp HOME sentinel; two full parallel runs produced zero cross-suite failures), so the serialization was pure convention. - test-runner --list-suites: prints every registered suite, one per line, emitted by the SAME macro table that executes suites — the list cannot drift from the run set by construction. - scripts/run-tests-parallel.sh: runs each suite as its own process (jobs = CPU count; CBM_TEST_PAR_JOBS overrides) under a ZERO-LOSS CONTRACT: a union guard fails the gate if the set of suites that produced a result differs from --list-suites (nothing can be silently dropped, a newly added suite is picked up automatically); per-suite pass/fail/skip are summed into the sequential runner's exact summary format; any suite crash, failure, or omission exits nonzero. Per-suite wall times are printed for balance tracking. - make test-par: the parallel target. make test (sequential) is unchanged and remains the escape hatch (CBM_TEST_SEQUENTIAL=1 in test.sh). - scripts/test.sh: builds, then routes through test-par — every CI test leg gets the speedup with zero workflow-topology change (same jobs, same gates, same billing; the legs just finish sooner). - The stack_overflow suite is split into a/b/c (7+6+7 of its 20 tests, pure re-registration): as one suite it was the wall-clock critical path of any parallel run — every other suite finished underneath its ~4 minutes. Measured locally (Apple Silicon, ASan runner): sequential ~10 min vs parallel 232 s, totals identical (6,361 passed / 0 failed / 1 skipped), union guard clean. The TSan leg keeps its dedicated subset runner (sequential) — out of scope here. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
Martin Vogel
|
83c137d2a5 |
feat: complete shared daemon lifecycle
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
Martin Vogel
|
0e00ef5702 |
feat: coordinate concurrent CBM sessions
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
Martin Vogel
|
feadbe1955 |
Merge pull request #467 from isc-tdyar/pr/objectscript-language-support
feat(objectscript): InterSystems IRIS ObjectScript language support |
||
|
Martin Vogel
|
f0db224140 |
feat: add coverage-aware agent integrations
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |