scripts/test.sh labels its pre-build contracts as a running 0a..0u sequence.
The new blob-generator contract was slotted in as "0f2" to sit next to the
other build-tooling contract, which breaks the one convention the list has.
Append it as 0v instead; these steps all run before the build, so position
carries no meaning beyond the label.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
vendored/nomic/code_vectors_blob.S is a GENERATED file that was hand-edited.
1f674c80 added the ELF .note.GNU-stack section to it after finding that every
Linux binary we had ever shipped requested an executable stack: an object
carrying no such note tells GNU ld nothing about its stack requirement, and ld
then assumes the worst for the entire link. That fix went into the artifact
only. scripts/extract_nomic_vectors.py still emitted the Mach-O branch alone,
so the next regeneration would have overwritten the tracked .S, dropped both
the ELF and COFF branches, and put GNU_STACK RWE back into every release --
silently, because nothing compared the generator against its own output.
write_blob_s() now emits the tracked wrapper verbatim, and the new contract
test fails if the two ever drift again. The template is compared rather than
executed so the test does not need the torch/transformers import the extraction
script pulls in. The write is pinned to UTF-8 because the WHY comment carries
an em dash and the default encoding is locale-dependent.
check-binary-composition.sh already fails a release whose binary has an
executable stack; this closes the same hole at edit time, where the diff is
still small enough to read.
Reported and originally fixed by Anand Aiyer in #1151, which also carried the
COFF branch. That PR sat unreviewed for a month while the .S half was
rediscovered and landed independently without credit.
Co-authored-by: Anand Aiyer <anand@aiyer.com>
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The *.sh eol=lf rule landed via #1314, but the git hooks
(scripts/git-hooks/commit-msg, scripts/hooks/pre-commit) are extensionless
and were still at the mercy of core.autocrlf. Add their explicit entries,
plus the contract test from #1272 wired as scripts/test.sh Step 0t: every
shell entrypoint (*.sh + both hook directories) must carry an eol=lf
attribute, with a matched-zero-files guard so a broken glob can never pass
vacuously.
Distilled from #1272: the diagnosis (CRLF checkouts breaking shebangs under
WSL/MSYS), the hook-file coverage, and the guard design are @xumian520's;
verified RED without the *.sh rule (105 uncovered entrypoints) and green
with it (107 files).
Co-Authored-By: xumian520 <126989134+xumian520@users.noreply.github.com>
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
On 0.9.0 -> 0.10.2, `update -y` auto-confirmed deleting other projects' indexes,
removed two .db files (~59 MB), and THEN aborted on an interactive prompt it
could not answer in a non-interactive shell. The run failed; the data was
already gone. Reported by @PsyTech-Rob.
The current code is already correct: both activation flows prompt early but
delete late, after the binary swap and agent configuration, and both return
before the deletion when configuration fails. #1538 also removed the specific
prompt that aborted. So this commit fixes nothing — it stops the fix from being
undone, which is the part that was missing. The reporter made exactly this
point: removing one bad prompt addressed the instance, not the rule.
Contract: in each flow that deletes indexes, cbm_remove_indexes() must appear
after the agent-configuration call. It keys on the CALL rather than a local
result variable, because the two flows spell that result differently, and it
fails loudly if a flow is renamed rather than passing vacuously.
Verified by reverting: moving the deletion ahead of configuration reproduces
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
#1558's shape and the contract goes red.
Every release now carries .mcpb one-click-install bundles alongside the
archives, and the MCP Registry entry lists them with per-file sha256:
- package-release.sh (canonical) builds codebase-memory-mcp-<target>.mcpb
for darwin/windows and the STATIC linux builds — manifest.json + the same
staged (stripped, gated) binary + LICENSE + THIRD_PARTY_NOTICES.md. The
glibc-dynamic linux targets stay archive-only: a dynamic binary defeats
the one-click promise.
- _build.yml / release-draft: bundles flow through provenance attestation,
checksums.txt, cosign signing and the release asset list; checksums.txt
is also preserved as a same-run artifact for the registry job.
- verify: the canonical scan matrix grows to 14 containers; MCPB manifests
are validated (parse, binary server, entry_point member, command binds
the entry point). Bundle binaries dedupe to the archive scan objects, so
the VT gate gains only the three distinct manifest.json files.
- publish-mcp-registry: gen-mcpb-registry-entries.sh appends one mcpb
package entry per bundle (release-asset URL + fileSha256 from the
attested checksums) to server.json before mcp-publisher runs.
Idempotent; a checksums file without bundles is a hard failure.
- contracts: Step 0o pins the bundle shape at its producer on every leg,
Step 0p pins the registry entries against the live server.json, and the
extractor contract covers the 14-container matrix incl. broken-manifest
fail-closed cases. The linux test image gains zip for the packager.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Deleting this contract with the UI-pack architecture threw away coverage of
machinery that SURVIVED: the extractor still builds the scan-set and association
manifests that check-virustotal.sh reads. Without it the gate could report a
confident green over an incomplete set.
It immediately paid for itself. The extractor was BROKEN by my own edit:
NameError: name 'variant' is not defined (process_tar)
TypeError: add_association() missing 'variant'
Removing the variant from validate_namespace's return left the call sites still
threading it. python3 -m py_compile passes on that -- it is a runtime failure --
and the extractor runs ONLY in the release and dry-run VirusTotal jobs, so
nothing else in the tree or on the 3-OS ladder executes it. It would have
shipped and broken every release verification.
The contract is rewritten rather than adapted: the original was woven through
the pack format (16 archives, 88 associations, per-pack asset math), and a
half-adapted version that passes for the wrong reasons is worse than none. It
now asserts what still exists:
- the exact 8-archive matrix, 32 member associations, nothing dropped
- archive containers never enter the scan set as members
- deduplication is exact: 8 byte-identical LICENSE members collapse to ONE
scan object while 8 distinct binaries stay 8
- manifest metadata agrees with its own rows, and every staged object matches
its recorded SHA-256
- fail-closed on a surplus member, a missing member, and a short matrix
Registered as Step 0l so it runs in every venue, not just when someone
remembers.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Completes the collapse to a single shipped composition and replaces the
zero-tolerance VirusTotal gate with a narrow, disclosed policy.
Packaging and installers
- package-release.sh loses --variant; archives are codebase-memory-mcp-<os>-<arch>
with exactly four members. install.sh/install.ps1 lose --ui/--standard.
- The extractor drops CBMUIPK pack parsing and --archive-scope; its scan-set and
association manifests (which the gate depends on) are unchanged otherwise.
- npm/PyPI/Go wrappers: the runtime "set" is one file again. The Windows lock
and race fixes from #1495/#1496 are kept; only multi-file set membership goes.
This also fixes `pip install` on Windows, which rejected the fifth archive
member against a hardcoded four-name allowlist.
- The wrappers' post-download probe moves from --verify-runtime-assets (removed)
to --version, which proves the same thing: the binary executes.
VirusTotal gate
- Exactly ONE detection is tolerated, and only when the engine is Microsoft AND
the label ends in `!ml`. Two or more engines, any non-`!ml` label, any other
vendor, any suspicious verdict and every infrastructure error still block.
- A tolerated object prints TOLERATED:, never OK:, and its counts are recorded
in vt-results.tsv exactly as a blocked one would be.
- append-vt-notes.sh mirrors the policy. It previously hard-failed on any
malicious count, so loosening only the gate would have passed the scan and
then died at note publication. The notes now DISCLOSE a tolerated detection
and link to SECURITY.md rather than claiming "0 malicious" for everything.
Rationale for the tolerance is in the gate itself: the verdict is not a property
of our bytes. It inverts across architectures and link modes, moves between
sibling artifacts of one build, and lands in different variant buckets for the
same source. The same `!ml` family hits llama.cpp, GitHub's own `gh`, Microsoft's
own Go toolchain and Anthropic's Claude installer.
The zero-tolerance contract becomes test_vt_gate_policy_contract.sh, asserting
the full matrix: 1x Microsoft !ml passes and reports TOLERATED; a Microsoft
signature label, a non-Microsoft engine, two engines, a suspicious verdict and
every malformed-response case still block. Its tripwire is narrowed to the
reverted endpoint-verification mechanism rather than the words "false positive",
so it no longer fires on a deliberate in-gate policy branch.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Externalizing the integration templates (#1492/#1493) and the UI bundle
(#1501/#1503) was done to reduce the Microsoft `Wacatac.B!ml` surface. It did
not work: across dry runs the flagged artifact count stayed at ~3 and the
detections merely moved between artifacts.
Dissection of run 31286803592 shows there is no structural cause to fix. The
verdicts split across every axis at once — linux-amd64 (dynamic) flagged while
linux-amd64-portable (static) is clean, but linux-arm64 (dynamic) clean while
linux-arm64-portable (static) is flagged. The two macOS binaries have identical
segment structure and split clean/flagged. Siblings from one build landed in
different variant buckets (.B vs .C). Entropy is low everywhere
(code_vectors.bin 4.166, grammar tables 3.464 bits/byte, against 7.5-8.0 for
packed payloads), so the packed-payload hypothesis is excluded too.
So the complexity bought nothing, and installation goes back to being
self-contained: one binary that carries its own UI and agent integration
templates, with no adjacent data file that has to resolve before `install`
works. Only the UI-capable composition ships from now on, under the historical
unsuffixed archive name.
Removed: src/ui/asset_pack.{c,h}, asset_pack_stub.c, asset_manifest_stub.c,
scripts/pack-ui-assets.mjs, src/cli/integration_assets.{c,h},
assets/cbm-integrations.json, scripts/gen-integrations-hash.sh, the
--verify-runtime-assets probe (nothing adjacent left to verify), and the
composition gates A6/A7 whose property is now deliberately inverted.
Restored: scripts/embed-frontend.sh, src/ui/embedded_{assets.h,stub.c}, the
compiled-in hook/adapter template bodies, and the embed/EMBED_OBJS build path.
Kept from the reverted commits, re-applied by hand where a wholesale file
restore would have dropped them:
- cbm_module_path_utf8() in both self-path sites. GetModuleFileNameA renders
through the ANSI code page and mangles non-ASCII install paths.
- the /__cbm/ui-readiness HMAC proof, secure_random and cbm_hmac_sha256, so
`daemon start --open` still waits for a genuine CBM listener.
- X-Content-Type-Options: nosniff on served assets.
- the MinGW noexecstack gate, -lbcrypt, and the cppcheck/zip CI fixes.
Archives are now codebase-memory-mcp-<os>-<arch>[-portable] with exactly four
members (binary, LICENSE, installer, THIRD_PARTY_NOTICES.md). That restores the
names every static package manifest already points at — aur, chocolatey,
homebrew, scoop, winget and glama were all broken by the -ui- rename.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The product binary embedded nine complete shebang'd shell scripts, their
PowerShell/.cmd twins, and two node:child_process client-adapter modules as C
string literals, and wrote them into agent config dirs at 0755 on install. A
block of script-shaped bytes inside a large unsigned executable is exactly the
surface Microsoft Defender's ML scored Trojan:Script/Wacatac.B!ml — the
detection that has recurred across the last several release attempts, on the
Linux ELF and darwin-arm64 standard binaries while every UI and Windows variant
stayed clean.
This moves every integration template out of the binary into a single shipped
data file, assets/cbm-integrations.json (compact: one asset, because each
shipped file is itself scanned). The binary now carries only the templates'
identity — one embedded SHA-256, generated at build time from the JSON so it
can never drift — plus the code to load, verify, and render them.
Why a hash and not just a loose file: content compiled into the binary was
tamper-evident with the binary. Moving it to disk must not turn it into
unauthenticated code the installer blindly executes. install verifies the asset
against the embedded SHA-256 and fails closed on mismatch ("integration assets
missing or modified - reinstall from the release archive") before writing
anything. Verified end to end: a one-byte edit to the asset makes install
refuse.
Ownership, for safe upgrade/removal: the verified copy is published to
~/.cbm/assets/<version>/ — a sibling of the DB cache, never inside it, so
clearing the cache cannot strip a user's hooks. A deployed integration file is
ours to remove only if it matches a template rendered from that stored copy;
foreign files and user-edited files at the reserved paths are preserved. The
historical template bodies travel in the JSON's released[] arrays so files
written by older versions are still recognised and cleaned up on uninstall.
Removed from src/: cmm_gate/session/subagent script-prefix and suffix
constants, cbm_build_released_gate_script, the shell/PowerShell/.cmd bodies in
cli.c, and the JS/TS generators in client_adapter.c. Confirmed on the built
standard binary: 0 occurrences of '#!/usr/bin/env bash', '#!/bin/sh',
'node:child_process', and 'ExecutionPolicy' (was non-zero for each).
test_no_embedded_scripts_contract.sh makes the removal a build property, not a
reachability assumption: no production C source may contain a shebang or
node:child_process literal, and the built binary is scanned directly. It strips
comments first (a contract a comment can satisfy is a false guard), and is
revert-checked — reintroducing one shebang literal makes it fail naming the
file:line. check-binary-composition.sh (A6) enforces the same needles on every
release artifact post-strip.
This does not, by itself, prove the detection is cured: the clean UI binary is a
strict superset of the flagged standard one, so no single passage is a
deterministic trigger. It removes the most coherent malware-shaped composite
from every shipped binary, aligns with how gh/starship/zoxide externalise
integration material, and makes the templates maintainable as data rather than
escaped C literals. Causal confirmation still needs a VirusTotal ablation.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
#1448 set CREATE_NO_WINDOW on the subprocess spawn and on cbm_popen_isolated,
but cbm_exec_no_shell (compat_fs.c) still passed a bare 0 for dwCreationFlags.
That is the helper behind git, codesign and open, so the most frequently hit
path kept flashing a console — and under a stdio MCP session with auto_watch
those windows steal focus while the user is typing. Reported by @noctrex on
#1448 after verifying the merged fix.
Also adds the contract nobody had. The flag has to be set per call site, the
four sites do not resemble each other, and reviewing "does this PR add the
flag" says nothing about the sites a PR does not touch — which is exactly how
the third one survived a fix applied twice:
daemon/bootstrap.c had it from the start (DETACHED_PROCESS | ... | flag)
subprocess.c added by #1448 (flags variable)
compat_fs.c :313 added by #1448 (inline literal)
compat_fs.c :689 missed by #1448 (bare 0)
tests/test_spawn_no_window_contract.sh asserts the whole-tree property instead,
so a fifth spawn site cannot be added without it.
The contract strips C comments before matching, and that is load-bearing rather
than tidiness: the first version of it PASSED with the fix reverted, because the
explanatory comment above the fixed call contains the flag name and satisfied
the match. A contract that can be satisfied by prose about the contract is a
false guard. Verified in both directions afterwards — green with the fix, and
red naming compat_fs.c:696 with the fix reverted and that comment still in
place.
Co-Authored-By: noctrex <noctrex@users.noreply.github.com>
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The Linux legs of the local 3-OS ladder build into build/linux-arm64 /
build/linux-amd64, not build/c, so Step 5c died with
'missing binary: /src/build/c/codebase-memory-mcp' and took the whole
leg down. Steps 5 and 5b already pass CBM_TEST_BINARY (derived from
$BUILD_DIR) to their scripts; 5c did not, and its test hardcoded the
path.
CI never caught this because every CI leg uses the default BUILD_DIR of
build/c - the container legs are the only ones that differ, which is
precisely what the local ladder is for.
- test_worker_error_response.sh honours ${CBM_TEST_BINARY:-build/c/...}
like its sibling watchdog tests; the default keeps bare manual runs working
- scripts/test.sh passes CBM_TEST_BINARY to Step 5c, matching 5 and 5b
- test_hook_conflict_notice.sh carried the identical hardcoding and is
fixed the same way (local-only today, but wrong is wrong)
Verified: with the fix the test passes against an out-of-tree binary;
reverting the fix reproduces the ladder's exact failure (rc=2,
'missing binary: <worktree>/build/c/codebase-memory-mcp').
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The regression is green locally against a seam-bearing binary but never
raises the forced conflict on ANY CI leg, so it was reddening six legs
for a reason unrelated to the fix under test. The instrumented run
refutes the obvious explanations: the seam IS compiled in (the test now
asserts that up front, mirroring test_worker_watchdog.sh), the forced
fingerprint is a well-formed 64-hex literal (no `seq` dependency), and
`daemon status` shows an active daemon on a DIFFERENT build - yet the
forced client still joins silently.
Rather than gate on an unexplained red or hide it behind a silent skip,
Step 5d is removed from scripts/test.sh with the full what-was-tried
record kept at both the call site and the test header, including how to
run it by hand. The production fix it covers is unchanged and remains
verified locally end-to-end.
Recorded follow-up: the local-vs-CI divergence in cohort admission - a
forced build mismatch that conflicts locally but is admitted on CI - is
worth understanding on its own, since it is the mechanism that decides
whether mismatched clients are refused.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
A hook client that could not join because the active daemon runs a
DIFFERENT build reported the conflict on stderr only - and stdout is
the only hook channel Claude Code surfaces, so in-session the result
was eternal silent skips. Worse, the connect-failure path classified a
conflicted daemon as ABSENT and suggested 'daemon start', which cannot
heal a build conflict. This is the rc.1-over-0.9.0 experience in
#1388/#1335: install the new build, the old warm daemon keeps running,
and every hook goes quiet.
Both failure sites now emit a throttled stdout systemMessage naming the
real state (active daemon runs a different build) with the actionable
step ('daemon stop', next command starts a matching daemon). The
version-cohort conflict path keeps its unconditional stderr detail; the
connect-conflict path now prints the formatted conflict to stderr too.
Test infrastructure: a CBM_ENABLE_TEST_SEAMS-only env override
(CBM_TEST_HOOK_CLIENT_BUILD) lets one binary present a foreign build
fingerprint as a hook client, so the conflict is reproducible without a
second build. tests/test_hook_conflict_notice.sh (scripts/test.sh step
5d, on the step-5 TEST_SEAMS binary) asserts fail-open exit 0 + the
stderr conflict + the stdout systemMessage with restart guidance. RED
before the fix (empty stdout), GREEN after, RED again on revert.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Microsoft's ML flagged the rc.1 release binaries. The decisive evidence is that
the SAME sha256 went from 0/62 clean to Microsoft-detected in about an hour with
no byte change, so the verdict lives partly in scanner-side state and no code
change can promise a clean result. What code CAN do is stop shipping things that
have no business in a release artifact, which is worth doing on its own merits
and incidentally widens the classifier margin. Every claim below is verified
against a built binary by the new gate, not by reading source.
Executable stack (the worst of the findings). vendored/nomic/code_vectors_blob.S
is the only assembly in the build and carried no .note.GNU-stack. An unannotated
object makes ld assume the worst for the whole link, so EVERY Linux release we
have ever shipped had GNU_STACK RWE. Adds the note (cause) plus ELF-only
-Wl,-z,noexecstack (outcome); the gate fails the release if it returns.
Test seams are now opt-in, never opt-out. TEST_SEAMS=1 defines
CBM_ENABLE_TEST_SEAMS; without it the crash-orphan probe -- which forks a child
that ignores SIGTERM and loops forever, then writes its pid to a caller-supplied
path -- and the lease-ownership marker compile to trivial stubs, so call sites are
untouched and the binary holds no fork, no signal handler and no env-var string.
Opt-IN is the point: forgetting the flag yields a clean binary rather than a leaky
one. scripts/test.sh requests it in the leg that consumes it, and
tests/test_worker_watchdog.sh now asserts the capability up front instead of
dying later with an opaque "Killed: 9".
The daemon's background version check is gone. It spawned curl against
api.github.com/repos/.../releases/latest on the first eligible session of every
run to say "a newer version exists" -- a release URL and an outbound request in
every shipped binary, for something the install scripts already report. The
INJECTABLE SEAM survives: update_ops is still honoured, the fakes in
tests/test_daemon_application.c still cover notice/ownership/cancellation/replay,
and with no provider application_update_subscribe_locked returns early so no
generation ever starts. "No network request by default" is now structural.
Dead capability out of release builds. The tar.gz/zip extraction block
(gzip_decompress through cbm_extract_binary_from_zip, plus its cli.h
declarations) moves under CBM_CLI_ENABLE_TEST_API -- verified self-contained, zero
uses of any helper outside it, only callers the excluded updater and
tests/test_cli.c. Downloading an archive, decompressing it, picking an executable
out of it and marking it executable is the canonical dropper composite; it is now
absent rather than merely unreachable. SQLite is built with
-DSQLITE_OMIT_LOAD_EXTENSION (no caller of load_extension anywhere in src/ or
internal/), removing that API surface and part of the dlopen/dlsym surface.
Temp files and environment scanning (S2/S3). Predictable paths in mcp.c,
artifact.c and diagnostics.c are created privately and exclusively and written
through the returned descriptor; pass_envscan.c no longer descends symlinked
directories out of the project root, and its fixed 512-byte path buffers no
longer truncate into pointer arithmetic that could land outside the buffer.
Build-time entropy. mimalloc's version banner baked __DATE__/__TIME__ into every
binary, so two builds of identical source seconds apart could never share a hash
and no release could inherit a false-positive determination made about its
predecessor. Local patch removes it (marked to survive refreshes), -Wdate-time
makes any future use a build error, and -Wl,--no-insert-timestamp stops the PE
header carrying the link clock.
scripts/ci/check-binary-composition.sh is the proof that each removal stays
removed, wired into package-release.sh after strip so the local artifact-flow
smoke enforces exactly what CI does. It asserts absences plus a CANARY string, so
handing it a compressed, stubbed or empty file fails instead of passing
vacuously, and a missing tool is a hard error -- a skipped assertion must never
look like a satisfied one.
Two build-system traps found by that gate, both of which had silently defeated a
fix: the product binary is compiled in one shot from sources, so a flag flip did
not rebuild it (now tracked by a .build-config stamp that also removes the
binary, making it independent of mtime granularity); and prod_sqlite3.o /
prod_mimalloc.o depended on a single named source, so SQLITE_OMIT_LOAD_EXTENSION
and the mimalloc patch BOTH compiled to nothing on the first incremental build.
Source review would have called them done.
Deliberately NOT changed. Three seams stay in release artifacts because
scripts/smoke-test.sh runs against the real artifact and needs them:
CBM_TEST_CRASH_ON and CBM_TEST_HANG_ON inject the faults that prove supervisor
recovery, and CBM_TEST_WINDOWS_USER_PATH_RUN_ID is what keeps the PATH smoke from
writing the tester's actual PATH. The gate treats those as an allowlist, so a
NOVEL seam still fails. The true no-UI standard build is deferred rather than
rushed: src/ui/* is in PROD_SRCS and four files outside src/ui reference UI
symbols, including the daemon that serves the UI, so that assertion reports
instead of failing until the split lands -- a gate everyone knows is red teaches
people to ignore gates.
No grammar is removed. ObjectScript accounts for essentially all binary growth
since the last provably-clean release (+21.4MB rodata, +1.1MB text from two
four-line shims), which made it the obvious ablation candidate, but a dry run
performed twelve real Defender endpoint scans across standard/UI and amd64/arm64
with ObjectScript, the daemon and the expanded hooks all present and every scan
was clean. Nothing there is a deterministic trigger, so cutting a
community-contributed language would spend a real feature on unproven margin.
Lean is not a candidate either: at 99.6MB of source it is by far the largest
grammar, but it shipped in v0.9.0 which scanned 20/20 clean, so removing it would
produce a novel unscanned profile instead of restoring a known-good one.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The staged venue-unification bundle: _smoke.yml onto the shared
wrappers with an extracted-artifact input (CBM_SMOKE_ARTIFACT_DIR),
soak-legs.sh as the one canonical soak entry (quick + query-leak legs
across all venues), protected per-user TEMP roots shared across
venues, clean-disk preflights before every VM/Docker run, win.sh and
run.sh routed through the canonical scripts, soak.yml retired, and
the venue-parity contract (whitelist walker: workflows may provision
or call canonical entries, nothing else).
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The windows-11-arm leg's first execution (broad matrix, release dry
run) died before any test ran: the trap-UBSan SANITIZE value passed
unquoted through the workflow ternary, so -fsanitize-trap=undefined
arrived as its own word and make consumed the leading -f as its
makefile flag ('No rule to make target sanitize-trap=undefined').
- The workflow quotes the ternary value so test.sh receives one
argument.
- test.sh forwards make arguments through an ARRAY instead of a
re-split string, so a VAR=VAL whose value contains spaces survives
every boundary (empty-array expansion kept bash-3.2-safe for the
macOS runners under set -u).
Verified: argument-parse proof (argc=1 with the four-flag value) and a
full macOS suite run through the changed plumbing, 6776 passed,
0 failed — identical totals.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Daemon lifecycle and Windows correctness, verified on a real Windows 11
ARM64 VM through the maintained test-infrastructure/vm drivers, plus the
macOS and Linux arm64 suites and the container lint gate.
Daemon lifecycle:
- daemon start/stop/status subcommands. `daemon start` launches a
PERMANENT daemon (spawn shape is byte-exact argv; survives idle
periods and session ends) and reports an already-active daemon
instead of failing. Permanence is honored at every stop latch:
last-committed-client disconnect, host initial-client window,
coordinator release, and application final-session close — a
permanent daemon also keeps admitting new sessions after its last
one closes.
- daemon stop refuses while sessions are active and lists the blocking
peers (pid/role) that must finish first; an idle daemon drains
through the activation-shutdown machinery with the ACK ordered after
connection interrupts. A second stop is idempotent. The wire ops are
no-cohort first-frame requests with peer fingerprint authentication,
so stop/status never conflict with an exact-build admission gate.
- One-shot CLI commands now execute through the daemon (index workers
keep their local supervised path). A cold CLI run that had to spawn a
temporary daemon prints a hint that `daemon start` removes the
per-command startup tax; a warm daemon is recycled silently.
- Hooks are connect-only fail-open: with no daemon present the hook
emits a visible, rate-limited notice (Claude-dialect systemMessage
plus stderr for other dialects) and always exits 0 — augmentation is
never allowed to block the caller's tool use.
- Version skew: a newer-build client automatically drains an
older-build permanent daemon (strict semantic-version triples only;
dev builds never auto-drain) and the build-conflict message names
`cbm daemon stop` as the manual escape hatch.
Windows IPC/runtime (real-VM verified):
- ipc(win): persistent pending overlapped ConnectNamedPipe. The accept
path used to destroy its listening pipe instance on every 20 ms poll
timeout; a client attaching in the teardown window was severed or left
on an orphaned pipe object whose HELLO no server handle could ever
read, absorbing the connect until the client's own timeout expired.
The pending connect now survives poll timeouts and nothing is
destroyed while a client could be attaching.
- ipc(win): drain-before-close for final responses. Closing a named-pipe
server handle can discard a just-sent response before the peer reads
it (POSIX stream sockets never lose buffered data on close). A bounded
cbm_daemon_ipc_connection_drain (read-until-peer-EOF; no-op on POSIX,
immediate on interrupted connections) now precedes close in
runtime_worker_finish and runtime_reject_inline, so hello-conflict,
capacity and disconnect acknowledgements reliably reach the peer.
- runtime: CLOSE_INTENT wire frame. A Windows named-pipe client has no
transport half-close, so close_begin now announces departure with an
explicit frame (ordered after APPLICATION_CANCEL, before the local
interrupt); the server releases the client's admission on receipt
instead of waiting for the handle to close. Admission-drop timing is
now identical to POSIX shutdown() semantics on every platform.
- runtime(win): client close cancellation. close_begin serializes with
request publication under the send lock, best-effort sends the active
token's APPLICATION_CANCEL frame, then interrupts local I/O; the
server cancels MCP/subprocess work promptly. Contract tests accept
both correct outcomes (interrupted transport or decoded CANCELLED).
- runtime: activation acknowledgement ordering. The activation ACK is
the requester's license to act on "snapshotted and draining", so every
connection interrupt is now initiated before the ACK is sent; a
session could previously get one more request serviced after the
requester observed the ACK.
- service(win): deadline-bounded private-file prepare. The conflict-log
prepare retry loop (100 x Sleep(2), which rounds up to the ~16 ms
timer granularity) burned ~1.6 s against permanently obstructed paths,
stalling hello rejections past the client's timeout. The retry budget
is now a 250 ms deadline; transient share collisions still retry.
- subprocess(win): cmd.exe /C payload encoder quotes metacharacters
correctly (root cause of the git-on-Windows failure cluster).
- watcher: SHA-256 buffer sizing (CBM_SZ_64 -> CBM_SZ_128) and a native
Windows stop/unwatch cancellation test with exact-image verification.
- httpd: send_all writes in bounded 64 KiB slices. A single giant
nonblocking send() on Windows is absorbed wholesale into AFD kernel
buffering regardless of SO_SNDBUF, so send deadlines and interrupts
could never engage against a slow peer (and the full payload was
pinned in nonpaged pool). Slicing restores a deterministic
backpressure point; a test hook pins SO_SNDBUF for the deadline and
interrupt tests.
- ui/http: shutdown lifecycle — interrupt checks, response-wide send
deadline, explicit connection states, refusal to free a server while
a listener-owned connection is active.
Windows long-path support:
- Central path-aware wide conversion (canonicalize via GetFullPathNameW
and prepend the extended-length prefix for absolute paths >=240) at
the compat chokepoints (cbm_fopen/compat_fs/mkstemp/mkdtemp), sqlite
store opens, and the daemon build-fingerprint/log paths. Deep managed
installs (a 64-hex generation directory routinely exceeds MAX_PATH)
now index, stage and activate correctly.
- activation transaction: its own file APIs and the component-walking
ancestry validators now operate in the extended-length namespace;
the launcher path is canonicalized (and prefixed when deep) once at
entry so every downstream exact-string comparison stays
form-consistent.
- Executable self-resolution uses the wide APIs (GetModuleFileNameW,
GetFileAttributesW) so non-ASCII install paths survive argv[0]
resolution.
Windows launcher install/uninstall transaction:
- FileRenameInfoEx names are NUL-terminated in an over-allocated
buffer. FileNameLength governs per the contract, but filter drivers
read FileName as NUL-terminated and appended adjacent heap bytes to
created names — a flaky, garbage-suffixed rename target. Both the CLI
and the launcher rename helpers are fixed.
- Uninstall retires state via rename-aside (.cbm ->
.cbm-retired-v1-<tag>-<pid>) with the retired tag shortened to 16 hex
chars so the bare rename target stays under the FileRenameInfoEx
NT-conversion ceiling at guard depths; 64 bits still uniquely
identify the generation.
- When the running launcher's mapped generation backings pin .cbm
against rename, the backings are relocated to activation-<pid>-N
.retired tombstones beside the install (a mapped image may be renamed,
never deleted; the launcher's liveness-guarded sweep reclaims stale
tombstones). Every relocation is recorded, and a FAILED uninstall
reverses the moves after restoring .cbm — via MoveFileExW with
extended-length paths on both arguments, since the deep generation
target is beyond the handle-based rename's bare-path reach — so a
restored install keeps its generation backings and stays runnable.
- After a committed uninstall the retired tree's backings are relocated
out so the tree is shallow enough for the detached cleanup's rd, and
the cleanup's working directory strips the extended-length prefix
(CreateProcessW lpCurrentDirectory silently ignores prefixed paths).
- Files created under Administrators-default-owner directories
(CopyFileW destinations, CREATE_NEW tombstones, probe directories)
are explicitly owner-stamped so the exact-owner validators hold on
runner images; guard fixtures stamp hand-built trees the same way.
Diagnostics, tests and infra:
- diagnostics: discovery is now an always-delivered JSON control record
(new cbm_log_control) that survives CBM_LOG_LEVEL suppression and
paths containing spaces; placement honors $TMPDIR with /tmp fallback
via a diagnostics-local helper; the soak parser reads the JSON record;
documented in docs/CONFIGURATION.md. Red-first coverage for suppressed
log levels, TMPDIR-with-spaces, and native Windows output-contract
assertions.
- tests(win): daemon_ipc/daemon_frontend fixtures now build endpoint
parents with production-shaped ancestry (LocalAppData on Windows, via
th_secure_runtime_parent_new) — the runtime ancestry validation
correctly refuses temp roots whose ancestors grant mutation rights to
Authenticated Users (C:/msys64/tmp, GitHub-runner work dirs) — and
drive the documented startup-owner publication flow before reading
generation-bound endpoint addresses. This turns the 26 Windows
failures previously visible in CI's full-test job green without
weakening any validation.
- tests(win): the launcher guard covers the full permanent-launcher
contract including failed-uninstall restore and immediate reinstall
after uninstall; new daemon lifecycle and reworked hook-augment
guards run the start/recycle/stop flow end to end.
- tests: CBM_SKIP_PERF is now actually consumed by the test runner
(it was set by CI but never read, so perf suites ran everywhere);
four throughput/bench suites are classified as perf, the heavy
store_arch suite moved to the slow-timeout tier, and two
wall-clock-sensitive assertions were rewritten as invariant checks
with coarse hang-detector backstops.
- build/test infra: build-dir safety contract, UI dev-proxy security
contract, soak daemon-recovery contract, path-safety helper, the
Windows VM worktree-sync contract wired into scripts/test.sh, and
vm/win.sh guards building its clean embedded-UI product in an
isolated BUILD_DIR so it cannot clobber the incremental test build.
provision-windows.sh now installs Node.js for the guards UI build.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The Windows-VM bootstrap embedded a personal SSH public key and the
drivers connected with StrictHostKeyChecking=no. Key material and host
identity are now strictly operator-supplied and live outside the
repository (~/.claude/cbm-vm/, never committed):
- windows-bootstrap.ps1 takes the operator's public key as a mandatory
-SshPublicKeyPath argument (validated single Ed25519 line) and prints
the VM host-key fingerprint to pin. Generating your own key is
documented in the script and README:
ssh-keygen -t ed25519 -N "" -C <label> -f ~/.claude/cbm-vm/id_ed25519
- win.sh and provision-windows.sh verify the pinned
CBM_VM_HOST_KEY_SHA256 fingerprint before every connection
(StrictHostKeyChecking=yes against a keyscanned known_hosts, Ed25519
only, IdentitiesOnly), validate branch names before remote
interpolation, and fail remote pipelines loudly (set -e -o pipefail).
- The MSYS2 installer download is pinned to a versioned release URL with
SHA-256 verification instead of an unpinned latest.
- win.sh gains a sync subcommand mirroring the uncommitted worktree onto
the VM; its manifest logic is guarded by
tests/test_vm_worktree_manifest.sh, wired into scripts/test.sh.
guards now runs the maintained PowerShell driver against a clean
product build; smoke-install drives vm-smoke.sh.
- run.sh drops the stale duplicate windows-vm/mac-vm legs (the former
carried its own unpinned ssh path) in favor of the vm/ drivers.
The previously committed public key has been rotated off the VM and is
no longer authorized anywhere.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
scripts/build.sh, test.sh, and clean.sh hardcoded build/c, and clean.sh
rm -rf'd it unconditionally. Running the container legs concurrently with a
native macOS build then had them delete and relink each other's objects
mid-build (ld "file cannot be opened"). Thread a BUILD_DIR through the three
scripts (default build/c) and point every compose service at its own dir
(build/linux-arm64, build/linux-amd64, build/win-cross, build/alpine); native
macOS keeps build/c exclusively. Also run the lint service in CI-parity mode
(--ci: cppcheck + clang-format only), matching CI, since clang-tidy is in
neither the lint image nor the CI job.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The runner executed all 104 suites sequentially (~10 min locally, the bulk
of every CI test leg). The suites are process-isolated already (per-process
mkdtemp HOME sentinel; two full parallel runs produced zero cross-suite
failures), so the serialization was pure convention.
- test-runner --list-suites: prints every registered suite, one per line,
emitted by the SAME macro table that executes suites — the list cannot
drift from the run set by construction.
- scripts/run-tests-parallel.sh: runs each suite as its own process
(jobs = CPU count; CBM_TEST_PAR_JOBS overrides) under a ZERO-LOSS
CONTRACT: a union guard fails the gate if the set of suites that
produced a result differs from --list-suites (nothing can be silently
dropped, a newly added suite is picked up automatically); per-suite
pass/fail/skip are summed into the sequential runner's exact summary
format; any suite crash, failure, or omission exits nonzero. Per-suite
wall times are printed for balance tracking.
- make test-par: the parallel target. make test (sequential) is unchanged
and remains the escape hatch (CBM_TEST_SEQUENTIAL=1 in test.sh).
- scripts/test.sh: builds, then routes through test-par — every CI test
leg gets the speedup with zero workflow-topology change (same jobs,
same gates, same billing; the legs just finish sooner).
- The stack_overflow suite is split into a/b/c (7+6+7 of its 20 tests,
pure re-registration): as one suite it was the wall-clock critical path
of any parallel run — every other suite finished underneath its ~4
minutes.
Measured locally (Apple Silicon, ASan runner): sequential ~10 min vs
parallel 232 s, totals identical (6,361 passed / 0 failed / 1 skipped),
union guard clean. The TSan leg keeps its dedicated subset runner
(sequential) — out of scope here.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The index supervisor gate (cbm_index_supervisor_should_wrap) wrapped ANY
embedder of cbm_mcp_handle_tool: it resolved the CURRENT executable and
re-invoked it as '<self> cli --index-worker index_repository ...'. In a
test binary 'self' is the test runner, which ignores those args and
re-runs its suites instead, producing recursive spawn chains (observed
11-minute hangs, and multiplied kernel VM-map/memory load during the
2026-07-04 host panics). The gate is now opt-in: only a process that
called cbm_index_supervisor_mark_host() — the real binary's main(),
before any subcommand dispatch — may spawn workers. Embedders index
in-process by default; the test mains additionally pin
CBM_INDEX_SUPERVISOR=0 as belt-and-suspenders. Guarded by
index_supervisor_gate_requires_marked_host_issue845 (fork+alarm harness,
new spawn-count test hook; red on the old gate, green without the env
kill switch on the new one).
A supervised worker whose supervisor died also kept indexing as an
orphan: the parent-death watchdog only ran in MCP-server mode. run_cli
now starts the same watchdog (detached; on ppid change it only writes
to stderr and _exit(0)s, so it is safe outside server mode) when it
detects --index-worker. Guarded by tests/test_worker_watchdog.sh
(scripts/test.sh Step 5b): a CBM_TEST_HANG_ON busy-spin pins the worker
mid-index, the wrapper parent is killed, and the worker must self-exit
within 15s — red without the wiring, green with it.
Fixes#845.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
On NixOS (and other non-FHS systems) /bin/bash does not exist, so
scripts with an absolute shebang fail to run. Switch the remaining
holdouts to /usr/bin/env bash: eleven scripts/*.sh,
test-infrastructure/run.sh, and the three Claude Code hook scripts
emitted by src/cli/cli.c (gate, session reminder, subagent reminder).
Distilled from PR #674, with parser-test coverage preserved: the
infra_parse_shell* fixtures in tests/test_pipeline.c intentionally keep
#!/bin/bash so absolute-path shebang extraction stays covered, and
tests/repro fixtures are untouched.
Also replace the GitHub-PAT-shaped fixture string flagged in the #674
thread with an obviously fake placeholder (ghp_FAKE...) that still
matches the ghp_ + 36-alnum secret detector.
Co-authored-by: Sandro Jäckel <sandro.jaeckel@gmail.com>
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
verify_compiler() in scripts/env.sh ran `file` on the clang path and
grepped for arm64/universal. Nix's clang is a bash wrapper script, so
`file` reports "ASCII text executable" and the check falsely concluded
"x86_64 only", aborting before any compile. Replace it with a capability
probe: compile + link a trivial program for the target -arch and trust
the result — correct for wrapper scripts (Nix, ccache) and cross-compilers.
Make arch targeting explicit and toolchain-agnostic: export
ARCHFLAGS="-arch <arch>" from env.sh and fold it into CC/CXX in
Makefile.cbm via `override`, so it reaches every compile and link
(including the vendored objects) and survives a command-line CC=
override. This replaces the `arch -<arch> make` prefix, which cannot
work with Nix's fixed-target clang. Drop the now-redundant ARCH_PREFIX
from build/test/lint/repro.
Refs #705.
Signed-off-by: Kris Williams <115474+kriswill@users.noreply.github.com>
The MSYS2 CLANG64 toolchain bakes its package-tracker URL
(https://github.com/msys2/MINGW-packages) into the static Windows .exe via
libc++/compiler-rt. The Layer 2 binary string audit flagged it as an
unauthorized URL, failing smoke-windows (both the standard and ui variants)
while Linux smoke stayed green. Add it to ALLOWED_URLS alongside the other
compiler/linker toolchain URLs (gcc.gnu.org, sourceware.org, launchpad).
Add tests/test_security_strings_allowlist.sh as a reproduce-first regression
guard: it reproduces the BLOCKED failure with a binary fixture and includes a
negative control proving a genuinely unauthorized URL is still rejected. Wired
into scripts/test.sh as Step 6.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Two reviewed external-fork contributions, re-implemented as clean maintainer code.
#406 / PR #407 (thanks @nvt-pankajsharma): a POSIX parent-death watchdog so the stdio MCP server exits when its launching parent dies instead of lingering on stdin. Refactors signal_handler into an idempotent request_shutdown(); a watchdog thread polls getppid() (500ms) and shuts down + exit(0) once the initial ppid (>1) changes; Windows unaffected (#ifndef _WIN32). The fork's getppid()<=1 startup early-exit was dropped (it could wrongly kill a legitimately-launched server during reparent races / in container launchers; the initial_ppid>1 guard already no-ops safely), and thread-create failure is now non-fatal (matches the watcher/HTTP background-thread policy). Adds tests/test_parent_watchdog.sh + scripts/test.sh wiring.
#413 / PR #414 (thanks @santanusinha): runtime log-level control via CBM_LOG_LEVEL (case-insensitive debug|info|warn|error|none, or numeric 0..4; unset/unknown leaves the level unchanged). getenv() before threads start (no race). Adds tests/test_log.c cases + README env-var row.
Four overload-scoring lookups (cbm_registry_lookup_method_by_types/_by_args, _symbol_by_types/_by_args) linear-scanned the entire project registry on every call site, ignoring the pre-built method_buckets/func_qn_buckets hash index that the by-name lookups already use. On a large C++ header (e.g. 10989 defs) that is O(calls x project_funcs) per file (~34s/file), so big codebases never finish indexing. Add a hashed fast path (O(overloads)) when the registry is finalized; preserve first-match semantics via lowest registration index (equivalence verified over 1606 lookups, 0 divergence); keep the linear fallback for un-finalized single-file registries so normal files are unaffected. Adds tests/test_cpp_index_hang.sh (scale-tier subprocess+timeout repro, opt-in via CBM_RUN_HANG_TEST).
- Use env: block instead of shell quoting for WIN32_LIBS in CI
- Forward WIN32_LIBS from environment in test.sh and build.sh
- Add skip_lint input to dry-run workflow for faster iteration
MinGW GCC does not ship ASan runtime on Windows. MSYS2 CLANG64
provides full sanitizer support via compiler-rt. All platforms
now run tests with ASan + UBSan — no exceptions.
- Fix GCC -O2 warnings: calloc for cf (maybe-uninitialized),
suppress stringop-truncation/alloc-size-larger-than (false positives),
guard louvain against negative node_count
- Default compiler: cc on macOS, gcc on Linux/Windows (fixes gcc-14
not found on MSYS2)
- Docker test infra now runs BOTH test (ASan) and build (-O2 -Werror)
- Add scripts/clean.sh, test.sh, build.sh, lint.sh as single source
of truth for local and CI builds (always clean, always parallel)
- Fix -Wmissing-braces: replace {{0}} with memset for struct arrays
- Fix -Wrestrict: use intermediate buffer for same-struct snprintf
- Update dry-run.yml and release.yml to use the new scripts
Martin Vogel