Completes the previous commit, which carried only the two mode changes because
the call-site edits were not staged when it landed (--amend is denied in this
repo, so this is additive rather than a rewrite).
Call sites now go through `bash`, which is what most of this repo already does and
which cannot break if a mode bit is lost to a patch application or a non-POSIX
checkout:
scripts/package-release.sh -> bash scripts/ci/check-binary-composition.sh
.github/workflows/release.yml -> bash scripts/ci/append-vt-notes.sh
With the 100755 modes from the previous commit, both sides are now correct, and
either alone would have been sufficient.
tests/test_script_exec_bit_contract.sh pins the class so it cannot recur: any
tracked .sh whose COMMITTED mode is non-executable must not appear as the first
word of a command in workflows, scripts, test-infrastructure or the Makefiles.
The committed mode is the thing that matters and the thing no local run can
check -- the working copy having the bit is exactly why this shipped.
Verified in both directions: passes on this tree, and fails on the exact defect
when the mode and the call site are reverted together. It joins backslash
continuations before analysing, because its own first draft reported a false
positive on
... && bash \
test-infrastructure/vm/vm-run-tests.sh --soak
where the interpreter sits on the preceding line. A contract that cries wolf
teaches people to ignore contracts, so that had to be right before it could be
useful.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Every unix build leg of release run 30499236230 died during packaging:
scripts/package-release.sh: line 190: scripts/ci/check-binary-composition.sh: Permission denied
The composition gate was committed 100644 while being invoked as a command. It
passed every local check because my WORKING COPY had the exec bit -- only the
committed mode was wrong, and nothing you can run locally reveals that.
scripts/ci/append-vt-notes.sh had the identical defect waiting in the verify
step, the last step of the release, so this would have failed a second time after
two hours of tests, build, smoke and soak.
Fixed on BOTH sides, because either alone suffices and the pair is mode-proof:
the two scripts are now 100755, and their call sites invoke them through `bash`,
which is what most of this repo already does and which cannot break if a mode bit
is ever lost to a patch application or a non-POSIX checkout.
tests/test_script_exec_bit_contract.sh pins the class: any tracked .sh whose
COMMITTED mode is non-executable must not appear as the first word of a command
in workflows, scripts, test-infrastructure or the Makefiles. Verified in both
directions -- it passes on this tree, and fails on the exact defect when the mode
and the call site are reverted. It also joins backslash continuations before
analysing, because its own first draft reported a false positive on
... && bash \
test-infrastructure/vm/vm-run-tests.sh --soak
and a contract that cries wolf teaches people to ignore contracts.
Product code is untouched: the test phase of the failed run was 27/27 green on
this exact tree, and a file mode cannot change a test outcome.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
cppcheck blocked the release run, correctly:
src/main.c:283: Condition '!main_test_worker_project_lock_marker(mutation)' is always false
src/main.c:2049: Condition '!worker_start_watchdog_test_descendant()' is always false
Compiling the seams out left behind stubs that returned true unconditionally, so
negating them was provably constant. Refactored rather than suppressed (repo
rule): the stubs are deleted and the CALL SITES are guarded, so a release build
does not contain the call at all -- which is what "the seam is not in the
artifact" should have meant in the first place.
The worker path needed care. The probe was one term in a five-term `||` chain,
and its POSITION is load-bearing: it must fork AFTER the isolated process group
exists (it inherits that group) and BEFORE the parent-death watchdog thread
starts, because forking a multithreaded process is the bug that ordering avoids.
Hoisting the call out of the chain would compile, pass tests, and silently break
that invariant. So the chain is split into three ordered steps -- process group,
then (test builds only) the probe, then the watchdog -- sharing one
worker_containment_unavailable() failure path so every step still fails
identically, with write()/_exit() rather than stdio and atexit handlers.
Verified in BOTH directions, since a guard that is always off is as wrong as one
always on: a TEST_SEAMS=1 build still contains the probe and
tests/test_worker_watchdog.sh passes against it, while the release build carries
no seam string. `make -f Makefile.cbm lint-ci` now passes locally -- the step
skipped before the last push, and the reason CI found this instead of me.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The release build blocked at Layer 8 (vendored dependency integrity):
MISMATCH: vendored/mimalloc/src/options.c
expected: 96ef01e4...
actual: 192ce06a...
That is the integrity check working exactly as designed. The hardening pass
patched mimalloc's version banner to stop baking __DATE__/__TIME__ into every
binary, which is an intentional and reviewed change to vendored code -- but an
intentional change is indistinguishable from a supply-chain edit until someone
records it, which is the entire point of the manifest. Updating the recorded
checksum is the review being written down.
Scope verified before committing: the manifest diff is exactly ONE line, the
options.c hash, and it matches the hash CI computed. No other vendored file
moved. scripts/security-vendored.sh's other layers still pass on the updated
tree -- no subprocess, network or dangerous calls in vendored code, dlopen
confined to sqlite3.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The update handoff moved out of the product on every platform, but two loose
ends were left behind.
README was actively WRONG, not merely incomplete. It still documented in-place
updates for macOS/Linux, and it promised "The MCP server also checks for updates
on startup and notifies on the first tool call" -- a claim that died with the MCP
update thread and then the daemon's curl check. The section now states the one
contract that holds everywhere (`update` validates flags and prints the install
script command), why it works that way (on Windows a running image cannot
replace itself; on POSIX a deliberate choice, because an in-process updater is
structurally a downloader), and the affirmative property that replaced it: cbm
makes no network request of its own accord and the archives carry no download
URLs at all.
`uninstall` now REPORTS the install script it finds beside the binary, printing
the path and the `rm` command, and says "would leave" under --dry-run. It does
not delete it. install.sh places itself there so `update` has something to hand
off to, but that file may equally be the user's own copy, a symlink into a
checkout, or package-manager managed -- an uninstaller must not delete a file it
cannot prove it owns, and one printed line leaves the decision where it belongs.
Windows checks install.ps1 too and tolerates either path separator.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Microsoft's ML flagged the rc.1 release binaries. The decisive evidence is that
the SAME sha256 went from 0/62 clean to Microsoft-detected in about an hour with
no byte change, so the verdict lives partly in scanner-side state and no code
change can promise a clean result. What code CAN do is stop shipping things that
have no business in a release artifact, which is worth doing on its own merits
and incidentally widens the classifier margin. Every claim below is verified
against a built binary by the new gate, not by reading source.
Executable stack (the worst of the findings). vendored/nomic/code_vectors_blob.S
is the only assembly in the build and carried no .note.GNU-stack. An unannotated
object makes ld assume the worst for the whole link, so EVERY Linux release we
have ever shipped had GNU_STACK RWE. Adds the note (cause) plus ELF-only
-Wl,-z,noexecstack (outcome); the gate fails the release if it returns.
Test seams are now opt-in, never opt-out. TEST_SEAMS=1 defines
CBM_ENABLE_TEST_SEAMS; without it the crash-orphan probe -- which forks a child
that ignores SIGTERM and loops forever, then writes its pid to a caller-supplied
path -- and the lease-ownership marker compile to trivial stubs, so call sites are
untouched and the binary holds no fork, no signal handler and no env-var string.
Opt-IN is the point: forgetting the flag yields a clean binary rather than a leaky
one. scripts/test.sh requests it in the leg that consumes it, and
tests/test_worker_watchdog.sh now asserts the capability up front instead of
dying later with an opaque "Killed: 9".
The daemon's background version check is gone. It spawned curl against
api.github.com/repos/.../releases/latest on the first eligible session of every
run to say "a newer version exists" -- a release URL and an outbound request in
every shipped binary, for something the install scripts already report. The
INJECTABLE SEAM survives: update_ops is still honoured, the fakes in
tests/test_daemon_application.c still cover notice/ownership/cancellation/replay,
and with no provider application_update_subscribe_locked returns early so no
generation ever starts. "No network request by default" is now structural.
Dead capability out of release builds. The tar.gz/zip extraction block
(gzip_decompress through cbm_extract_binary_from_zip, plus its cli.h
declarations) moves under CBM_CLI_ENABLE_TEST_API -- verified self-contained, zero
uses of any helper outside it, only callers the excluded updater and
tests/test_cli.c. Downloading an archive, decompressing it, picking an executable
out of it and marking it executable is the canonical dropper composite; it is now
absent rather than merely unreachable. SQLite is built with
-DSQLITE_OMIT_LOAD_EXTENSION (no caller of load_extension anywhere in src/ or
internal/), removing that API surface and part of the dlopen/dlsym surface.
Temp files and environment scanning (S2/S3). Predictable paths in mcp.c,
artifact.c and diagnostics.c are created privately and exclusively and written
through the returned descriptor; pass_envscan.c no longer descends symlinked
directories out of the project root, and its fixed 512-byte path buffers no
longer truncate into pointer arithmetic that could land outside the buffer.
Build-time entropy. mimalloc's version banner baked __DATE__/__TIME__ into every
binary, so two builds of identical source seconds apart could never share a hash
and no release could inherit a false-positive determination made about its
predecessor. Local patch removes it (marked to survive refreshes), -Wdate-time
makes any future use a build error, and -Wl,--no-insert-timestamp stops the PE
header carrying the link clock.
scripts/ci/check-binary-composition.sh is the proof that each removal stays
removed, wired into package-release.sh after strip so the local artifact-flow
smoke enforces exactly what CI does. It asserts absences plus a CANARY string, so
handing it a compressed, stubbed or empty file fails instead of passing
vacuously, and a missing tool is a hard error -- a skipped assertion must never
look like a satisfied one.
Two build-system traps found by that gate, both of which had silently defeated a
fix: the product binary is compiled in one shot from sources, so a flag flip did
not rebuild it (now tracked by a .build-config stamp that also removes the
binary, making it independent of mtime granularity); and prod_sqlite3.o /
prod_mimalloc.o depended on a single named source, so SQLITE_OMIT_LOAD_EXTENSION
and the mimalloc patch BOTH compiled to nothing on the first incremental build.
Source review would have called them done.
Deliberately NOT changed. Three seams stay in release artifacts because
scripts/smoke-test.sh runs against the real artifact and needs them:
CBM_TEST_CRASH_ON and CBM_TEST_HANG_ON inject the faults that prove supervisor
recovery, and CBM_TEST_WINDOWS_USER_PATH_RUN_ID is what keeps the PATH smoke from
writing the tester's actual PATH. The gate treats those as an allowlist, so a
NOVEL seam still fails. The true no-UI standard build is deferred rather than
rushed: src/ui/* is in PROD_SRCS and four files outside src/ui reference UI
symbols, including the daemon that serves the UI, so that assertion reports
instead of failing until the split lands -- a gate everyone knows is red teaches
people to ignore gates.
No grammar is removed. ObjectScript accounts for essentially all binary growth
since the last provably-clean release (+21.4MB rodata, +1.1MB text from two
four-line shims), which made it the obvious ablation candidate, but a dry run
performed twelve real Defender endpoint scans across standard/UI and amd64/arm64
with ObjectScript, the daemon and the expanded hooks all present and every scan
was clean. Nothing there is a deterministic trigger, so cutting a
community-contributed language would spend a real feature on unproven margin.
Lean is not a candidate either: at 99.6MB of source it is by far the largest
grammar, but it shipped in v0.9.0 which scanned 20/20 clean, so removing it would
produce a novel unscanned profile instead of restoring a known-good one.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The pre-release ML false-positive tolerance (single-engine Microsoft "!ml"
verdicts downgradable with Defender endpoint evidence, #1340) is reverted by
owner decision: cbm does not ship binaries carrying a VirusTotal detection,
demonstrably false or not. A "trojan" badge on a release asset is a
reputation cost the project is not willing to price in, however good the
accompanying evidence.
The gate returns to its original form: any detection, by any engine, on any
artifact, on any version blocks the release. The endpoint verification tool
and the evidence side-channel are removed with it; the notes renderer keeps
its extracted-script form but only ever states a verified "0 detections".
False positives are resolved upstream instead: verify the bytes on a real
Defender endpoint, submit a Microsoft false-positive report for the exact
hashes, wait for the detection to clear, then RE-RUN the failed verify job --
which does not rebuild, so the cleared hashes are the shipped hashes.
tests/test_vt_gate_zero_tolerance_contract.sh pins the decision: clean
passes; 1 malicious and 1 suspicious each block across stable, -rc., -pre
and -alpha versions; plus a tripwire for the specific reverted evidence
mechanism returning. Loosening this gate again has to consciously delete
that contract.
The skip_tests dispatch input and the script-extracted notes step survive
the revert -- both are orthogonal to gating policy.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Release run 30464288732 was blocked by the VirusTotal gate: three linux-amd64
binaries flagged 1/62 by Microsoft's Wacatac.B!ml -- fully stripped binaries
(0 symbols, verified on the exact artifacts), the state that scanned clean in
the two previous cycles. Meanwhile a real Defender endpoint (engine
1.1.26060.3008, signatures 1.455.410.0 updated the same day, RTP on) scans
the identical bytes clean. Four cycles of evidence now say the same thing:
this verdict is an unstable ML decision boundary, not a property of the code,
and no build-side lever moves it durably -- stripping, downloader removal and
metadata changes each "worked" only until a later build flipped it back.
So stop treating the flag as buildable-away and verify it honestly instead:
check-virustotal.sh may downgrade BLOCKED to TOLERATED only when ALL hold:
- pre-release version (-rc./-pre/-alpha/-beta); stable releases never
- every failing file flagged by exactly ONE engine
- that engine is Microsoft and the verdict ends in "!ml" (never a
signature name)
- hash-pinned Defender ENDPOINT evidence is attached to the draft release
(defender-endpoint-verification.txt) proving Microsoft's shipping
product, signature-updated at scan time, reports the exact bytes clean
av-endpoint-verify.sh (new) produces that evidence: downloads the draft
assets, scans them on the local Windows VM endpoint, refuses to attest if
RTP is off or Defender itself detects, uploads the hash-pinned result.
The gate prints the exact command when evidence is missing; re-running the
failed verify job does not rebuild, so the bytes stay fixed.
append-vt-notes.sh (new, extracted from inline YAML per venue-parity) then
renders the release-notes table honestly: a tolerated file reads "1/62 ML
false positive, endpoint-verified clean", never "0 detections".
tests/test_vt_gate_tolerance_contract.sh pins all nine decision directions
against a stubbed VT API and release store -- clean pass, stable-never,
missing/stale/DETECTED evidence, signature-named verdict, non-Microsoft
engine, multi-engine -- so the tolerance provably fails closed.
Also: release.yml gains skip_tests for re-releases of an already test-green
tree (build/smoke/soak/verify always run; lint failures still gate via
!cancelled() && !failure()).
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
All three cost real release cycles, and none is fixed by widening a budget.
daemon_application_cancels_physical_job_only_after_final_session waited for the
SUBSCRIBER COUNT to reach 2, then cancelled both sessions and asserted the
physical job had started exactly once. The job starts asynchronously after
subscription, so both cancels could land first and leave starts == 0 --
arguably the correct outcome. It now waits for the state the assertions
actually require. Verified 47/47 on Windows, the only platform it ever failed.
The parallel harness refused outright when the suite leader had already exited,
because taskkill /T cannot walk a tree from a dead PID. But the leader can exit
between the timeout decision and that call, so the harness itself lost a race:
a natural exit at the wrong moment failed the whole wave. It now proves cleanup
the only way still available -- nothing parented to that PID -- and its contract
asserts the PROPERTY rather than the phrase "tree cleanup" it used to grep for.
That string pin is what broke when the guard was reworded while behaving
correctly; the contract now checks rc==2 AND that the descendant really did
survive, which would also catch a guard that claims to fail closed while
leaking.
extract_wide_flat_file_is_linear took ONE sample per size, so the ratio carried
the noise of both. On a loaded Windows VM linear code measured 51x against a 40x
bound (184ms -> 9387ms). Best-of-N instead: timing noise only ever adds time, so
the minimum is the cheapest good estimate of the noise-free cost. The bound is
deliberately unchanged -- it sits where linear (~20x) and quadratic (~128x) are
each >=2x away, so raising it would move the test toward the very signal it
exists to catch. Now measures 19.1x on Windows, 21.4x on macOS.
Also: the smoke's `cli` helper redirected stderr to a file and discarded it, so
any of the 10 bare `VAR=$(cli ...)` assignments could kill the run under
`set -euo pipefail` printing NOTHING. One such abort cost a full Windows cycle
just to locate and still could not be attributed. It now surfaces the command
and its stderr. Neutral wording on purpose: one call site expects a non-zero
exit and must not read as a failure.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Completes the move started on Windows. `update` prints the install script's
command and exits 0 on every platform; the download/extract/chmod/exec sequence
is compiled out of release builds entirely, and the MCP background thread that
ran cbm_popen("curl ... api.github.com ...") on first tool call is gone.
The installers now place themselves beside the binary, which closes the last
gap: `update` referenced a raw.githubusercontent URL purely because an
install.sh-installed user had no local copy to point at. Two details make that
safe.
The source is the install script from the archive we JUST checksum-verified,
never "$0" -- which does not exist under `curl | bash`, and would pin the user
to the OLD installer forever. Since the script ships inside the verified
archive, it inherits that verification instead of needing its own.
And it is published by atomic rename, never written over the live path. Bash
reads a script incrementally by byte offset, so overwriting the file it is
executing continues reading NEW bytes at the OLD offset: silent, bizarre
corruption. cp to a temp name then mv -f swaps the directory entry while the
running shell keeps its original inode. Windows follows the same rule via
Copy-Item + Move-Item even though PowerShell parses up front.
Deliberately NOT done: spawning the new installer to delete and replace its
predecessor. Fetch remote code -> drop to disk -> spawn -> self-delete is a
textbook stager, and self-deletion is among the most heavily weighted
heuristics there is. It also gains nothing -- `"$DLBIN" install` already runs
the NEW release's install logic, because the binary owns the install.
Net: zero install or download URLs remain in the shipped binary, and the
allow-list entry is retired with it. Both scripts scan CLEAN (0/61) with the
self-placement code in them.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The previous change generalised `--strip-all` to every platform. Apple's strip
does not accept that flag, so the macOS build failed outright -- loudly, which
was the lucky outcome.
The unlucky version was already in the tree before that: `strip -x`. It succeeds
and leaves 4058 symbols, which is precisely the state VirusTotal flagged on
darwin-arm64 while the fully stripped ELF legs came back clean. Measured on that
artifact:
llvm-strip --strip-all 373 symbols CLEAN
strip (no flags) 378 symbols equivalent
strip -x -S 4058 symbols the FLAGGED state
strip -X / -u -r 4058 symbols likewise
So the flags are chosen per format, and a candidate that cannot do the job is a
hard error rather than a silent fallback to a weaker strip. Verified on a PATH
without llvm-strip, exactly as the runner has it: 373 symbols, codesign --verify
--strict passes, the binary runs.
Packaged and scanned before merge: darwin-arm64 0/60 and ui-darwin-arm64 0/61,
both CLEAN -- the platform that was flagged on the previous release run.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Two open security alerts.
Dependabot #13 (GHSA-r28c-9q8g-f849, high): postcss path traversal via
sourceMappingURL auto-loading. Transitive dev dependency of vite, so it never
ships in the binary, but the fix is a clean lockfile bump -- vite requires
^8.5.3 and the patch floor is 8.5.18, so 8.5.24 satisfies it with no dependency
graph change. Verified: npm ci resolves and the UI still builds.
Scorecard #76 (PinnedDependenciesID): Dockerfile.glibc22 used a floating
`ubuntu:22.04`. Every other venue image is digest-pinned; this one was missed.
It matters more here than the checkbox suggests -- that image IS the glibc-floor
assertion, the oldest userland we claim the portable binary runs on. A floating
tag means the floor silently drifts to whatever 22.04 points at, and a floor
that moves is not a floor.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The `update` handoff prints a curl one-liner when install.sh is not beside the
binary, and the security audit blocked the release on it:
BLOCKED: src/cli/cli.c: URL not on allow-list: https://raw.githubusercontent.com/DeusData/
Two things were wrong. The URL was not declared, and it was split across string
literal continuations, so the audit extracted only the first fragment -- an
allow-list entry for the real URL could never have matched it. Hoisted to a
single CBM_INSTALL_SH_URL token so the declaration means what it says.
It is the same URL as the documented one-liner install, and it is printed for
the user to paste: the binary no longer downloads anything.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Windows already handed updating to install.ps1, because a running .exe cannot
replace its own image. The remaining platforms now do the same, for a different
and better reason.
An in-process updater is structurally a downloader: it fetches a remote archive,
verifies it, extracts it, marks the result executable and runs it. That composite
lived in every shipped artifact -- along with the release URLs it fetched -- to
serve a command most users run a handful of times. The install scripts already
do all of it, are idempotent (so re-running one IS the update), and run while cbm
is not running.
`update` now prints the exact command for the running platform and exits 0:
unix bash "<dir>/install.sh" (or the curl one-liner when not adjacent)
windows powershell -ExecutionPolicy Bypass -File "<dir>\install.ps1"
Both scripts ship inside their release archive, so the printed path resolves
next to the binary. Flags are still parsed and validated, so `update --dry-run
--ui` keeps rejecting typos rather than silently accepting them.
The updater itself is EXCLUDED from release builds rather than left to dead-code
elimination: build_update_url, download_verify_install, extract_and_install_binary,
checksum fetch, detect_os/detect_arch and the cli_download_* helpers now compile
only under CBM_CLI_ENABLE_TEST_API, which release builds do not define. The C
suite still covers the flow through the activation test seam. Verified against
the release binary: `releases/latest/download` is gone entirely.
Also removes the MCP background update check. It ran on the first tool call and
did:
cbm_popen("curl -sf --max-time 5 -H 'Accept: application/vnd.github+json' "
"'https://api.github.com/repos/.../releases/latest' 2>/dev/null")
An embedded shell command plus a startup network callout, in every session, to
tell users a newer version exists. It shelled out, depended on curl being
installed, and phoned home from every agent session. The install scripts report
versions; this did not need to be in the server.
Smoke follows the same collapse. Phases 6c and 14 asserted a Windows-specific
handoff and a POSIX in-process replacement; they now assert one platform-neutral
contract selected by UPDATE_SCRIPT, and 14a's byte-identity check compares the
driver against ITSELF before and after -- comparing against "$BINARY" passed on
Windows only because Windows has no signing step, while the POSIX fixture
ad-hoc re-signs its copy and so differed before `update` ever ran.
The Linux glibc guard is RELOCATED, not retired. The standard linux asset links
glibc 2.38+ and breaks Debian 11, RHEL 8 and Ubuntu 20.04, so the installer must
fetch the static "-portable" build. That constraint moved from the binary to
install.sh, and the smoke now guards it where the behaviour lives.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The first strip landed as `strip -x` on Mach-O, out of caution that a full strip
could leave an image dyld will not load. That caution was wrong for this binary,
and it cost a release cycle.
`-x` retains external symbols -- 4058 of them on darwin-arm64 -- so the macOS
artifacts kept the very symbol table the ELF legs had just shed. Run
30414312364 then flagged exactly those two artifacts (darwin-arm64,
ui-darwin-amd64) while every Linux and Windows binary came back clean, which
read as the detection "moving to macOS" when it had simply stayed on the
unstripped set.
Measured on the flagged darwin-arm64 artifact:
strip -x 4058 symbols VirusTotal 1 malicious / 61
--strip-all 373 symbols VirusTotal 0 malicious / 61 CLEAN
The packaged binary still verifies (`codesign --verify --strict` passes, having
been re-signed after stripping) and still runs. Symbol-table removal has now
cleared the detection on three independent artifacts across two platforms:
linux-amd64 twice, and darwin-arm64 here.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Release binaries carried their full symbol table. Production compiles without
-g, but the linker keeps .symtab regardless, so every archive shipped ~536 KB of
internal function names that nothing needs: a bigger download and a free map of
the internals, by accident rather than by decision. Nothing symbolizes at
runtime -- mem_profile.c is not in the production build and never calls
backtrace_symbols -- so no diagnostics are lost.
It also had a concrete cost. Microsoft's ML scored the unstripped linux-amd64
binary Trojan:Script/Wacatac.B!ml (1 engine of 62) and blocked release run
30398064336 at the VirusTotal gate.
That verdict is a decision-boundary artifact, not a property of the code, and
the evidence is unambiguous:
* the dry-run build two days earlier (d587dea) is the same program 10 KB
larger and scans CLEAN -- a 0.003% delta flips the verdict
* the delta in that window is almost entirely DELETION of Windows-only files
that Linux never compiled
* the ui build of the same commit was never flagged
* two independently flagged builds drew different sub-variants (Wacatac.B and
Wacatac.C), which a real signature does not do
* v0.9.0, re-analysed against the same engine build, is still clean -- so this
is not model drift either
Stripping removes the symbol-name feature surface those models score. It cleared
BOTH flagged builds, and every other platform stays clean, so this fixes Linux
without trading the problem sideways. Verified before merge:
linux-amd64 (f440743) stripped 0 malicious / 62 engines
linux-amd64 (0802689) stripped 0 malicious / 62 engines
linux-arm64 stripped 0 malicious / 61 engines
linux-amd64-portable stripped 0 malicious / 62 engines
windows-amd64 stripped 0 malicious / 68 engines
darwin-arm64 stripped + re-signed 0 malicious / 60 engines
macOS needs care and gets it. The build workflow ad-hoc signs BEFORE this script
runs, so stripping invalidates that signature and the kernel then refuses to
exec the image; Mach-O is re-signed here. It also uses `strip -x` rather than
--strip-all, because a full strip can leave an image dyld will not load. The
packaged macOS binary verifies (`valid on disk`, `satisfies its Designated
Requirement`) and runs, and the full macOS smoke passes against a stripped,
re-signed binary.
Build fingerprints are derived from the file itself during execution, with
nothing embedded during compilation, so staged and target agree and activation
is unaffected.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Release run 30385988475 lost three Windows smoke jobs to one phenomenon on two
different surfaces. A Windows file that was just written or just executed can
refuse deletion and rename for a moment while a scanner reads it or while the OS
finishes reaping a child. Both conditions clear on their own within moments.
It reads as nondeterminism because it is timing: windows-2025 passed both amd64
variants while windows-latest -- the SAME image, windows-2025-vs2026 -- failed
both, and windows-11-arm failed too.
PRODUCT: uninstall abandoned a live installation.
With one binary per platform, uninstall must remove the very image it is
executing, and on Windows a running image can only be renamed, never
overwritten. That rename got a single attempt, so a transient sharing violation
surfaced to the user as
error: failed to remove ...; completed configuration/index cleanup may remain
with the install left half-torn-down. install.ps1 already retries this exact
operation ten times for this exact reason; the C path now does the same, and
only for sharing/access/lock violations, so a file that is genuinely held still
fails closed instead of spinning.
Proven with a failure-injection seam rather than by racing a real scanner:
3 injected failures must still succeed, 64 must still fail AND leave the target
in place. Deterministic, so it does not need CI to learn whether it works.
HARNESS: the smoke killed itself silently.
Phases that install a binary then `rm -rf` the directory hit the same lock. Under
`set -euo pipefail` that returns non-zero and ends the run WITHOUT printing
anything -- the logs jump straight from "OK 13h" to job cleanup with no FAIL
line, which is what made this expensive to read.
None of those 16 removals is an assertion; they are fixture cleanup, and
_smoke.yml already states the principle ("a temp dir on an ephemeral runner must
never fail the job") that smoke-test.sh did not honour. They now go through
smoke_rmtree, which retries so the disk is actually reclaimed -- these fixtures
hold ~300 MB binaries and runners are disk-tight -- then warns and continues.
Verified: Windows VM full smoke ALL PASSED (previously died after 13h), macOS
full smoke ALL PASSED, activation_transaction + cli 259 passed on Windows, all
10 shell contracts pass, lint-ci clean.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The fixture contract pinned the literal string
STALE_CMD="$UPDATE_HOME/retired-install/codebase-memory-mcp.exe"
so that Windows Phase 14 would carry an MCP entry naming a missing executable.
Two separate things retired that intent, and because the requirement only ever
matched the STRING it kept passing while the property it stood for was gone.
First, the fixture itself now copies a binary to the retired path (added with
the launcher removal), so that path stopped being missing no matter what this
string said.
Second, Windows update is a handoff to install.ps1. Nothing rewrites the entry
in-process the way the launcher-managed update did, so an entry naming a foreign
path just survives to uninstall -- which correctly refuses to remove a config
entry owned by a DIFFERENT installation. Phase 14f was then asserting that
uninstall performs exactly the removal it must never perform, and it failed for
that reason.
The requirement now pins the installed binary, which is what install.ps1 leaves
a real Windows user holding. The missing-executable classification is not
dropped, it is covered by named unit tests rather than by this fixture:
cli_editor_mcp_preserves_unrecorded_posix_absolute_entries_without_probe and
cli_editor_mcp_preserves_unsafe_windows_drive_probe, the latter covering the
Windows missing-drive case specifically.
This contract runs as step zero of every test leg, so the stale requirement
failed all 30 test jobs of release run 30384603265 -- on Windows, macOS and
Linux alike -- from one violation.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Two Windows tests needed the startup thread to be observed HOLDING the startup
lock, and looked for it by polling the lock for BUSY up to 200 times. That state
has no lower bound: whenever the rendezvous handoff does not block, acquire ->
handoff -> release completes in microseconds, so the window can close between
two polls and startup_observed asserts on a coin flip. This blocked release run
30374135923 on test-windows (windows-2025, CLANG64, x86_64).
Budget-tuning the poll was tried and rejected -- a wider budget only lowers the
failure rate, and a transient window is never a fixed test. Instead the
production path exposes a gate that fires once the startup lock is held and
before the handoff runs, matching the hook pattern already used for POSIX
publication. The test parks the startup thread there and releases it when it is
ready, so the interleaving is pinned by construction. The acquired flag is
monotonic, so the observation cannot be missed; the remaining bound only catches
a thread that never started at all. The gate is inert unless a test installs it.
Also fixes an unrelated use-after-free this exposed. ipc_forever_wait_server
closed the connection and NULLed it from the server thread, while the test was
about to call cbm_daemon_ipc_connection_interrupt on the pointer it had already
loaded -- and receive_frame can return on its own before that call lands. It
crashed roughly one Windows run in five, independent of the change above.
Ownership now stays with the test, which closes after joining, so interrupt can
only ever see a live connection.
Verified on real Windows: 5 consecutive daemon_ipc runs, no failure and no
crash. macOS 46 passed x5 under ASan/UBSan; Linux 6606 passed, 0 failed.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The Windows smoke guards that an invalid PATH seam must fail closed rather than
silently falling back to the live HKCU\Environment\Path. That guard went green
in a way that meant nothing: the seam still refused, and install still printed
"PATH configuration failed", but the process exited 0.
Windows release builds used to compile a separate managed-install path. With one
binary per platform they compile the shared cbm_cmd_install, which since 30b5f12
downgraded ANY failed dry-run plan check to CLI_OK. That commit was fixing a real
complaint -- the dry-run summary was being skipped entirely on Windows -- but it
bundled the exit status into the fix. Keeping the summary is right; reporting
success is not. --dry-run exists to answer "would this install work?", so a
caller that only sees the exit code could not tell a refused PATH probe from a
clean plan.
The summary and the triage note still print; only the status changes. A real
install already failed non-zero here, so the fail-closed behaviour itself was
never at risk -- just its observability.
The smoke also still encoded the launcher-era portable-vs-managed split, which
does not survive one-binary-per-platform:
* 6b expected uninstall to be REFUSED. That refusal existed because a portable
extracted bundle was a different artifact from the launcher-managed install
it would have torn down. The extracted binary now IS the installed one.
* 6c expected update to be REFUSED, contradicting the contract Phase 14a
asserts against a real update: Windows hands off to install.ps1, exits 0 and
prints the command. 6c now pins that same handoff.
* 14a compared binaries with cmp, which is absent from the Windows MSYS shell,
so it reported "different" for two copies of one file. Hashes instead, the
way Phase 12 already verifies the release archive.
* 14f seeded the MCP entry at the RETIRED binary. The old launcher-managed
update rewrote that entry in-process; the handoff does not, so uninstall
correctly declined to remove an entry owned by a different installation and
the phase demanded the one thing it must never do. It now seeds the
installed binary, which is what install.ps1 leaves a real user holding.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
install.ps1 did not run. It failed to PARSE, dying with a cascade of syntax
errors before its first statement, and it has been in that state on main since
61a432c.
Windows PowerShell 5.1 decodes a BOM-less .ps1 as ANSI, not UTF-8. The three
em-dashes in this file are 3 UTF-8 bytes each, and cp1252 renders them as a
sequence ENDING IN A DOUBLE QUOTE. Two sat in comments and were harmless; the
third sat inside a string literal:
"error: could not retire the existing $BinName - close all running"
where the injected quote closed the string early and left the rest of the line
as garbage expression, taking the enclosing if/foreach blocks with it.
This shipped broken because nothing executed the script. The smoke covers it in
Phase 13, but every Windows smoke run since the launcher removal aborted at an
earlier phase, and the VirusTotal check only ever scanned the bytes. Since
install.ps1 is now the sole Windows install AND update path, a parse error there
is a total loss of both.
Fixed as ASCII rather than by adding a BOM: the documented install pipes the
file straight into the parser (irm ... | iex), where a BOM would become part of
the string and break that path instead.
The bundle contract now rejects any non-ASCII codepoint in install.ps1 and names
it, so this cannot return silently.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
- Windows updates run from install.ps1, not the running binary: a running
executable cannot replace its own image on Windows. Document the exact
command, that re-running the script IS the update, and the Unblock-File
caveat for Mark-of-the-Web. npm/pip users keep updating via their package
manager on every platform.
- Build from source: state that every platform produces ONE self-contained
binary with no companion executables, and document how to run the suite
(scripts/test.sh, the same entry the CI gates use) and how to package an
archive locally with the canonical script.
- Test badge 5604 -> 6768, the current count from build/c/test-runner.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Windows shipped a PAIR: a small permanent launcher (codebase-memory-mcp.exe)
plus the real product binary (codebase-memory-mcp.payload.exe). The launcher
existed for exactly one reason — a running .exe cannot replace its own image
on Windows, so an in-process self-update needs a second resident binary to do
the swap.
That stub is statically indistinguishable from a dropper: a small, unsigned,
zero-prevalence PE whose whole job is verify-and-execute another binary.
Defender's ML scored it Trojan:Win32/Wacatac.B!ml and blocked the v0.9.1-rc.1
release at the VirusTotal gate. It is not fixable in our code on x64 —
bcrypt-free, stripped, VERSIONINFO'd, minimal-resource and even
resource-FREE builds on CI's own MSYS2 CLANG64 toolchain were all flagged,
while the product binary scans clean on every platform.
So remove the stub and move self-update OUT of the process into install.ps1,
which runs while cbm is NOT running: Windows' image lock only blocks a
process from replacing ITSELF. now prints the exact PowerShell
command (with the Unblock-File hint for Mark-of-the-Web); install.ps1 is
idempotent, so re-running it IS the update — it stops the daemon, renames the
running binary aside (the one mutation Windows permits on a running image),
publishes the new one, and sweeps retired copies.
Windows now matches Linux and macOS: ONE binary per platform.
* packaging, install.ps1, npm and PyPI wrappers all carry a single binary
* the launcher/payload ABI contract and ~2500 lines of stub state machinery
are deleted
* every daemon start, CLI call and hook fire loses a process spawn, a named
pipe handshake and an stdio relay
* test_windows_bundle_contract.sh is rewritten as an INVERTED contract: it
now asserts no shipped surface can reintroduce a launcher/payload pair,
and that install.ps1 retires the running binary before publishing
Verified: VirusTotal 0/67 on the packaged binary and 0/58 on install.ps1 (no
certificate involved); macOS and Linux full suites green; Windows guards all
green including the new update-handoff contract; npm 10/10; PyPI 3/3.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The fresh-request guard test waited for 'job terminal AND >= N/2 subscribers
still attached'. Publish flips terminal and releases the blocked prior
requests in the same locked step, so that conjunction is a transient window
whose width is thread-wakeup scheduling: the busy-spin caught it by luck,
a 1ms-yield poll (#1307) sampled past it on fast x64 runners, and no budget
can pin it (release runs 30305464193 and 30309182389 failed it from both
directions).
The production guard never consults subscriber counts —
application_find_active_job_locked skips any terminal job — and the test's
downstream assertions (starts==2, destroys==2, stale/fresh response
separation) catch a terminal-job reuse in every interleaving. So wait only
for the stable end-state (active jobs == 0) and drop the racy helper.
47/47 locally.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
app_wait_for_terminal_job_with_subscribers was the only app_wait_for_*
helper polling without cbm_usleep(1000): a sleepless spin pins a core for
up to 10s and can starve the daemon threads it is polling on scarce-CPU
runners. Release run 30305464193 hit exactly this on windows-11-arm
(test_daemon_application.c:2671 ASSERT(terminal_with_prior_subscribers));
the suite passes 47/47 locally with the yield in place.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The blocked-git child's PID-marker wait was a fixed 5s. On the 4-vCPU
windows-11-arm release runners that budget was missed twice in a row
(run 30300256783, attempts 1+2: test_watcher.c:1038 ASSERT(marker_ready))
while every faster leg stayed green — spawn latency under CPU starvation,
not a watcher defect. Raise the four spawn-marker waits to a shared 20s
budget (WATCHER_TEST_SPAWN_MARKER_BUDGET_MS). The wait stays bounded: a
genuinely wedged spawn still fails, at 20s instead of 5s. Termination,
completion, and join budgets are untouched.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
A version containing a prerelease suffix (v0.9.1-rc.1) now publishes as an
RC on every channel instead of masquerading as stable:
- GitHub release is created with prerelease: true (never marked Latest)
- npm publishes under the `next` dist-tag, so plain `npm install` keeps
serving the last stable; testers opt in with codebase-memory-mcp@next
- the MCP Registry sync is skipped (the registry has no channel concept)
- PyPI needs no change: PEP 440 already treats rc versions as pre-releases
that pip only installs with --pre
Stable releases (no dash) are byte-identical in behavior.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
v4's PASSING smoke legs already ran 12-14 minutes against the 15-min
timeout; v5 lost 13 green-path jobs to that wall (GitHub reports job
timeouts as 'cancelled', zero real failures). The unified smoke
legitimately runs more than the pre-unification one — artifact mode,
real download/update phases, a Phase 15 that actually executes.
Budget doctrine (see _soak.yml, violated twice there historically):
the timeout sits above the worst case with headroom.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Both hosted-runner Windows soak legs failed at Phase 1/diagnostics
with an empty frontend stderr while the daemon's own log showed a
healthy diagnostics.start: the workflows pass the suffix-less binary
path, msys resolves it transparently (so the old 'append .exe only
when the plain name is not executable' branch never fired), and
soak-test keys BOTH native-Windows gates — the cygpath'd
CBM_CACHE_DIR and the coproc stdio branch — off the literal .exe
suffix. A suffix-less native binary therefore received a POSIX-form
cache path: daemon logs mis-rooted (diagnostics grep zero, arm64) and
the frontend's cache path handling died silently (x64 SIGPIPE at
first write).
Reproduced on the VM's CLANG64 environment with a suffix-less
invocation (same FAIL) and red->green proven: with the normalization
the identical invocation runs the full quick soak to PASSED.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The version preflight PASSED (both images printed their version) and
then the job died in the cleanup trap: Windows keeps the just-exited
image's file busy for ~100ms (section teardown / first-touch AV), so
'rm -rf' in the EXIT trap raced it — 'Device or resource busy', exit 1
on a job whose actual work was green (observed on windows-11-arm ui in
dry run 30232946513). Temp-dir cleanup on an ephemeral runner is
best-effort with one retry, never a gate.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Martin Vogel