feat/nix-cross-dev-shell
42 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Martin Vogel
|
8eff872df5 |
fix(ci): teach the VirusTotal gate the withheld-executables manifest
The v0.10.6 release run failed deterministically at verify:
BLOCKED: expected scan object is missing:
objects/scan-3099e91c...--codebase-memory-mcp.exe
exclude-rescanned-selected-objects.sh (added after v0.10.5, first exercised
by this release) deliberately deletes the selected executables from the
surface-scan directory — their bytes were already scanned as candidates and
re-submitting identical bytes re-rolls a probabilistic classifier — and
writes binaries/virustotal-withheld.tsv. But check-virustotal.sh still
received the pre-withhold scan-set listing all sixteen objects and failed
closed on the first missing file. The rework's two halves never talked.
The gate now accepts an optional VT_WITHHELD manifest (strict parse: v1
marker, the stated reason required, sha256-keyed rows): an expected-set row
whose hash the manifest vouches for is exempt from the on-disk and
action-output contracts, while everything else keeps the strict path.
Fail-closed properties preserved and extended:
- no VT_WITHHELD -> byte-for-byte previous behavior (candidate
stage and dry-run call sites are unaffected;
verified against the original failure)
- withheld object present -> blocked (inconsistent staging)
- hash outside the set -> blocked (spurious withhold)
- everything withheld -> blocked (scan would cover nothing)
- mismatched object name -> blocked
vt-results.tsv keeps its exact shape (scanned objects only) — the release
notes table already uses the candidate results, and the withheld manifest is
now preserved with the rest of the evidence artifacts. release.yml passes
VT_WITHHELD only in the verify stage, right after the withhold step.
Verified offline with a fixture reproducing the release failure verbatim
plus the four negative cases above; the positive case passes staging and
association validation and proceeds to VT polling.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
|
||
|
Martin Vogel
|
bdb99d7750 |
fix(release): stop re-scanning bytes VirusTotal has already scanned
The verify pass submitted every extracted object, selected executables included,
on the stated grounds that "VirusTotal is content-addressed, so identical bytes
return the analysis it already holds instead of re-running 70+ engines".
That is measurably false. On v0.10.5 all EIGHT re-submissions produced a NEW
analysis - same VirusTotal file-id, timestamp 47 minutes later:
candidate: file-id=2c00f485... ts=1786795957 (12:12:37Z)
verify : file-id=2c00f485... ts=1786798758 (12:59:18Z)
Re-analysing identical bytes re-rolls a probabilistic classifier, and Microsoft's
ML engine answered differently within that hour, in BOTH directions:
82750cd1 (linux-amd64) microsoft-ml -> clean
6d3c5be6 (darwin-arm64) clean -> microsoft-ml
The published notes are generated from the candidate scan, so v0.10.5 shipped a
table calling linux-amd64 flagged when VirusTotal had it clean, and darwin-arm64
clean when VirusTotal was reporting Trojan:Script/Wacatac.B!ml. Every hash in
that table links to the page that contradicted it. Corrected in place after
publication; this removes the cause.
The second scan proved nothing the first did not. Identity is settled by hash
before this step runs: verify-release-selection.py reconciles every published
container to the selected bytes, and checksums.txt binds the same digests
publicly. A re-scan adds no assurance - only another roll.
What still gets scanned is exactly what the candidate pass never saw: install.sh,
install.ps1, LICENSE, THIRD_PARTY_NOTICES.md, the MCPB manifest.json and the
unpacked UI assets. install.sh and install.ps1 are the highest-consequence
non-executable bytes we publish - users pipe them straight into a shell - and
that coverage is untouched. Measured on the v0.10.5 object set: 16 objects in,
8 withheld, 8 still scanned.
The withheld set is recorded as evidence (cbm-virustotal-withheld-v1) naming each
sha256 and pointing at virustotal-candidate-results.tsv, so the published
evidence still accounts for every shipped object.
Fails closed three ways, each with an actionable message: no object matches a
selected sha (the containers do not carry the recorded bytes), everything is
withheld (the surface scan would be a no-op), or the selection names no shas at
all. The zero-match grep is wrapped rather than left to pipefail, because a
guard that aborts silently is not a guard - found by testing the guards rather
than assuming them.
Also drops 8 VirusTotal submissions per release.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
|
||
|
Martin Vogel
|
c360440203 |
fix(release): cover the legacy ui-* aliases in checksums.txt
The ui-* archives are byte-identical copies of the canonical ones, published after verify so they inherit the hash-bound VirusTotal verdicts. They were absent from checksums.txt, and publish-legacy-aliases.sh documented that as intentional: "checksums.txt covers the canonical names current installers request." That reasoning has a hole. The aliases exist only for 0.9.x updaters (#1538), and those verify the NAME they asked for. So the alias fixed the 404 and moved the failure one step later - the updater downloads the archive, cannot find its name in checksums.txt, and refuses: warning: codebase-memory-mcp-ui-darwin-arm64.tar.gz not found in checksums.txt error: refusing to install an unverified download Reported by AmooAti in #1134. Confirmed on the live v0.10.4 release: eight ui-* archives published, zero of them listed. Every pre-0.10 user who answered the old variant chooser with "ui" is hard-blocked from updating by any path. The same digest is now emitted under the legacy name before the attestation step, so the attested artifact covers both names. No new bytes and no new scan surface: an alias is a copy, so its sha256 is by construction the one already computed. The rule lives in scripts/ci/append-legacy-alias-checksums.sh rather than inline in the workflow, because the venue-parity contract requires it: a venue may provision, plumb artifacts, or call a canonical leg script, and text transformation is none of those. Keeping it beside publish-legacy-aliases.sh also puts the two halves of the alias rule in one place, which matters because they must stay in step - .tar.gz and .zip only, never an already-ui-* name. It fails closed when it matches nothing, since a name with no asset is as broken as an asset with no name. Validated against the real v0.10.4 checksums file: the generated set is exactly the eight ui-* assets that release published - no phantom names, none missing - and the empty case exits non-zero. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
Martin Vogel
|
5f8273222c |
fix(ci): release notes report the shipped binary, not every candidate
Release notes now carry one row per product — the bytes actually published, with their VirusTotal verdict — instead of a column per candidate. Several candidates are scanned so the selector has an alternative when an opaque classifier flags one, but a reader installing cbm cares about what they receive, not about what we discarded. The rejected candidates' verdicts remain in the published evidence TSVs for anyone auditing the selection, and they stay useful to US as a development signal; in a changelog they are noise, and listing a "microsoft-ml" verdict next to a binary nobody can download invites exactly the wrong conclusion. Also removes a claim that had become FALSE: the section asserted that every scan had a minimum of 50 decisive engines. That floor was removed deliberately (it is VirusTotal fleet availability, not a property of our binary) and the last release observed 31. Published notes would have stated something untrue. The observed range is still reported, now over the shipped binaries. The disclosure of a tolerated Microsoft `!ml` is preserved and narrowed to the binaries that ship. The contract test now pins both directions: a flagged binary that SHIPS must be disclosed, and flagged candidates that were rejected must not appear at all. Its fixture gained a target that draws the tolerated verdict on all three candidates, so the disclosure branch is actually exercised — previously every flagged candidate was rejected in favour of a clean sibling, leaving that path untested. README and SECURITY.md updated: they promised notes "link every candidate result". All five release/VT contract tests pass. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
Martin Vogel
|
07240cf45c |
fix(ci): publish VT evidence under the marker the gate actually writes
The v0.10.4 release (run 31755436916) failed at the very last step of `verify`, after all 69 other jobs had passed — full test matrix, all eight builds, smoke and soak: publish-vt-evidence: wrong evidence marker in binaries/vt-results.tsv The results format was bumped to `cbm-virustotal-results-v2` in #1596, but publish-vt-evidence.sh still demanded `-v1`. It was the only straggler: the gate, the selector, the notes renderer and all three contract fixtures were already on v2. It survived because that PR had removed the script's only caller, leaving it dead code that nothing exercised. Restoring the full-surface scan restored the caller too, and the stale expectation surfaced in the worst possible place — at the end of a real release rather than in a dry-run, since `verify` does not run in dry-runs. Also adds the guard that would have caught it: every marker publish-vt-evidence.sh validates must be one the gate actually writes. Revert-checked — reintroducing v1 fails with "publisher expects marker never written by the gate: cbm-virustotal-results-v1". The scan itself passed. This is purely the evidence-publishing step. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
Martin Vogel
|
7b4533b67d |
fix(ci): finish the three-candidate wiring and restore full-surface VT scanning
Two things, both found by the dry-run and by sweeping what it exposed.
1. The three-candidate change was incomplete. The dry-run failed at
stage-release-candidates.py, which carries its OWN copy of the transform
validation that the previous commit only fixed in the selector:
stage-release-candidates: candidate linux-amd64/debug-stripped has
invalid transform: 'strip-debug'
Sweeping for that assumption found it in six places, not two:
stage-release-candidates.py, select-release-candidates.py,
verify-release-selection.py, append-vt-notes.sh and two contract-test
fixtures — as hardcoded 16s, `len(TARGETS) * 2`, two-entry VARIANTS tuples
and two-key truth tables. All are now derived from len(VARIANTS).
Field prefixes needed care: the variant NAME keeps its hyphen because it is
the on-disk directory, while the evidence columns use underscores, so
`debug-stripped` reads `debug_stripped_sha256`. Every lookup now goes through
an explicit FIELD_KEY map instead of interpolating the variant directly.
The selection contract test now covers the truth table EXHAUSTIVELY: three
variants x two tolerated classifications is exactly eight combinations, and
there are exactly eight targets, so every case is exercised once.
2. Full-surface VirusTotal scanning is restored. This PR had moved scanning
upstream to the candidates and deleted the post-package pass, which silently
narrowed coverage from everything we ship to executables only. The 42 runtime
files across the 14 containers — install.sh, install.ps1, LICENSE,
THIRD_PARTY_NOTICES.md, the MCPB manifest.json and the unpacked UI assets —
were still extracted, structurally verified and strings-audited, but no
longer scanned at all. install.sh and install.ps1 are the highest-consequence
non-executable bytes we publish; users pipe them straight into a shell.
The verify job scans every extracted object again, under the same policy.
Re-submitting the selected executables alongside them is close to free
because VirusTotal is content-addressed and answers for identical bytes from
its own record — the same property that made the analysis-id equality check
untenable two commits ago.
The gate-chain contract asserted the opposite ("duplicate post-package
VirusTotal path remains"). That assertion is inverted: the pass is required,
and it is not a duplicate, since it covers a strictly larger set. README and
SECURITY.md updated from "archive containers are checksummed rather than
redundantly rescanned" to state the full covered surface.
All five release/VT contract tests pass.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
|
||
|
Martin Vogel
|
98d7dbab01 |
feat(ci): add a third release candidate (debug-stripped) as an extra VT draw
Release evidence from run 31744302624 shows the tolerated Microsoft `!ml` verdict is close to a coin flip per byte image rather than a property of the code. Across the eight targets the stripped and unstripped candidates of the SAME linker output disagreed on four, and in both directions: linux-amd64 stripped microsoft-ml unstripped clean darwin-arm64 stripped clean unstripped microsoft-ml linux-arm64 stripped clean unstripped microsoft-ml If the classifier were keying on something intrinsic to our code the siblings would agree; they do not. So each variant is close to an independent draw, and 5 of 16 candidates drew the flag. Two draws is not always enough. On that run linux-amd64-portable came back microsoft-ml on BOTH candidates, leaving no clean binary to ship for that target. A third independent draw at a ~31% observed per-candidate hit rate takes the both/all-flagged case from roughly 1-in-10 per target to roughly 1-in-30. The third candidate is `--strip-debug` (Apple: `-S`): debug information removed, symbol table kept. Behaviourally identical to the other two — same linker output, only metadata differs — but a distinct byte image, which is all VirusTotal needs to scan it as its own file. Verified on the real v0.10.4 candidates: linux-amd64 gives three distinct hashes (294,634,656 / 294,623,208 / 293,746,096 bytes) and darwin-arm64 likewise, with the ad-hoc signature verifying after strip. Selection is unchanged in spirit and now ordered: smallest artifact first (stripped, debug-stripped, unstripped), take the first CLEAN one, and only if every candidate drew the tolerated verdict ship the smallest flagged one. A hard verdict on any candidate still blocks the release before selection. Cost is 24 objects per release instead of 16. Derivation enforces that all three hashes differ — identical candidates would be one draw wearing three hats, and the selector would believe it had alternatives it does not have. Public claims in README, SECURITY.md and docs/index.html updated from "both stripped and unstripped" to the three candidates. All three release contract tests pass, including the native derivation test which exercises the real strip and codesign path on this host. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
Martin Vogel
|
db26e9615e |
fix(ci): stop gating the release on how many VirusTotal engines answered
The build+VT dry-run (run 31744302624) blocked all eight targets on:
BLOCKED: objects/3eb486a4... completed with only 48/75 decisive engines (< 50)
That object is the windows-arm64 STRIPPED candidate with ZERO detections. It
was classified "hard" — the same bucket as a real malware verdict — purely
because 48 engines answered instead of 50. Its own sibling scanned clean with
66.
How many engines return a decisive result is VirusTotal fleet availability on
the day, not a property of our binary. These artifacts are ~300 MB and many
engines skip or time out at that size, so the count sits near the floor and
moves run to run: on this run one object landed at 48 and the other fifteen
spread 59-68. A floor there makes shipping a lottery decided by someone else's
infrastructure, which is exactly what our CI-determinism rule forbids, and it is
not something we can influence by fixing anything on our side.
Classification now depends only on what engines FOUND:
hard - two or more engines, any non-Microsoft engine, any label that is
not `!ml`, or anything suspicious. Still blocks the release.
microsoft-ml- exactly one Microsoft `!ml`. Tolerated and disclosed.
clean - nothing flagged.
The decisive-engine count is still recorded per candidate in the results
manifest and printed as a NOTE when it is below the reference value, so the
evidence is unchanged; only its power to block is gone. The selector keeps
rejecting an incoherent response (total < completed).
Public claims updated to match in README.md, SECURITY.md and docs/index.html —
all three promised "at least 50 decisive engines" as a publication requirement,
which would no longer have been true.
Contract test updated: a clean verdict now stands regardless of engine count,
and the below-reference NOTE must still be reported. All three release contract
tests pass (vt-gate-policy, release-gate-chain, candidate-derivation).
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
|
||
|
Martin Vogel
|
4cf9579833 |
docs(ci): state the real reason the VT verdict binds to bytes, not to an id
The previous commit framed accepting VirusTotal's own analysis id as giving up
freshness. That was the wrong way round.
VirusTotal recognising our bytes is desirable: the evidence this pipeline
publishes is the hash-keyed file report — append-vt-notes.sh asserts the URL is
exactly .../gui/file/<sha256>/detection — so the report we gate on is the same
one a reader gets by looking that SHA-256 up themselves. Requiring a freshly
minted analysis id would have contradicted the evidence we publish next to it,
and made our verdict unreproducible by anyone outside the release job.
Comment-only; no behaviour change. The build+VT dry-run on
|
||
|
Martin Vogel
|
094929fddc |
fix(ci): bind the VirusTotal verdict to the bytes, not to the analysis id
The dry-run for this branch (run 31729540899) failed in `select-package`, the new job, after every build had completed and before smoke — the exact shape of failure that would have taken a release down: BLOCKED: VirusTotal response analysis id does not match the submitted analysis: objects/70fbbc37... That object is the linux-arm64 UNSTRIPPED candidate, 295,476,856 bytes. VirusTotal is content-addressed, and for bytes it already holds it may answer with its own canonical analysis rather than the one this upload created. A large release binary is exactly the artifact most likely to be already on file, so requiring the ids to be equal would have recurred on most releases rather than being a one-off. The equality check was guarding the right idea at the wrong level. What has to hold is that a verdict describes THESE bytes, and parse_completed already enforces that a few lines below against meta.file_info.sha256 and file_info.size — a strictly stronger binding than the id. It is also what keeps a tuple's two candidates apart: stripped and unstripped differ in hash by construction (70fbbc37... vs 28b3890f... on linux-arm64), so neither can be read as the other whatever ids VirusTotal hands out. The wrong-hash and wrong-size contract cases pin that and still block. The property deliberately given up is freshness: an older cached analysis of identical bytes is now accepted. For identical bytes that is the same content risk, and it buys a gate that is deterministic instead of one that fails whenever VirusTotal recognises our own binary. The submitted id is still recorded in the results manifest as evidence. The selection policy is unchanged and needs no change: stripped clean ships stripped; stripped microsoft-ml with a clean unstripped ships the clean one; both microsoft-ml ships stripped; a hard verdict on either sibling still blocks the release for analysis. Contract test updated to assert the passing direction, and revert-checked: with the fix reverted it fails with the production message, with it applied it passes. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
Martin Vogel
|
3904e59372 |
ci: select release binaries before smoke testing
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
Martin Vogel
|
141efe3766 |
fix(release): restore exec bits on the new mcpb scripts
core.filemode=false hid the +x here, and release.yml invokes the registry-entries generator directly — a 644 mode would fail the publish job the first time it runs. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
Martin Vogel
|
b6a5d2c35b |
feat(release): ship MCPB bundles and publish them to the MCP Registry (#1246)
Every release now carries .mcpb one-click-install bundles alongside the archives, and the MCP Registry entry lists them with per-file sha256: - package-release.sh (canonical) builds codebase-memory-mcp-<target>.mcpb for darwin/windows and the STATIC linux builds — manifest.json + the same staged (stripped, gated) binary + LICENSE + THIRD_PARTY_NOTICES.md. The glibc-dynamic linux targets stay archive-only: a dynamic binary defeats the one-click promise. - _build.yml / release-draft: bundles flow through provenance attestation, checksums.txt, cosign signing and the release asset list; checksums.txt is also preserved as a same-run artifact for the registry job. - verify: the canonical scan matrix grows to 14 containers; MCPB manifests are validated (parse, binary server, entry_point member, command binds the entry point). Bundle binaries dedupe to the archive scan objects, so the VT gate gains only the three distinct manifest.json files. - publish-mcp-registry: gen-mcpb-registry-entries.sh appends one mcpb package entry per bundle (release-asset URL + fileSha256 from the attested checksums) to server.json before mcp-publisher runs. Idempotent; a checksums file without bundles is a hard failure. - contracts: Step 0o pins the bundle shape at its producer on every leg, Step 0p pins the registry entries against the live server.json, and the extractor contract covers the 14-container matrix incl. broken-manifest fail-closed cases. The linux test image gains zip for the packager. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
Martin Vogel
|
8eabe191d2 |
fix(install,daemon): unbreak npx clients, group-writable homes, and legacy updaters
Five field reports in the 24 hours after v0.10.0 all pointed at the same thing: gates that were right in principle refused real, ordinary setups, and then failed to say why. Per the consolidated strictness decision, each gate keeps the protection that matters and drops the part that was refusing legitimate users — and every refusal now names what it refused and how to proceed. **Daemon image gate: npx and every ephemeral install path (#1539, #1383).** The admission check treated "the peer's image hashes differently" and "the peer's image cannot be examined at all" as one failure. The second is what `npx codebase-memory-mcp` always produces (ephemeral cache path, unfingerprintable), so every npx-invoked client was rejected — and, because the client never reported it, agents saw a transport that closed mid-handshake with zero bytes on stdout. Reported by @wassolles with the admission path already read and the fix space mapped. An unverifiable image is now admitted: the rendezvous HELLO immediately above it has already proven semantic version, build fingerprint, and protocol/store/ feature ABI, and the image check was trading that real proof for an unavailable one. It logs daemon.client_image_unverifiable_admitted so the weaker check is never invisible. A fingerprint MISMATCH — the tamper case the gate exists for — still rejects hard. Separate test seams keep the two modes testable apart. **Client bootstrap failures are no longer silent (#1539).** An MCP client that cannot reach the daemon now emits a JSON-RPC error on stdout naming the reason, plus the same text on stderr. Previously the reason sat in bootstrap_result.message and the process exited having written nothing at all. **POSIX activation: group-writable ancestors (#1535, discussion #1526).** activation_directory_secure required no group or other write bit on the install directory AND every ancestor. WSL2 ships ~ and ~/.local at 0775, as do several distro skeletons and any site using a shared primary group, so install.sh failed for a large fraction of Linux users — reporting a policy refusal as "activation transaction I/O failed", which sent reporters after disk errors and filesystem types. Root-caused by @AmirF194 in a clean ubuntu container; @shochdoerfer and @iandol confirmed independently. World-writable ancestors are still refused (any local user could swap a path component mid-transaction). Group-writable ancestors are now warned about and admitted. The LEAF directory stays strictly owner-private — that is where the binary is published, and group write there would let another account replace the executable between validation and exec. Refusals now name the directory, its mode, and which rule refused. **The obsolete ui/standard chooser (#1538, from discussion #1526).** v0.10.0 consolidated to one archive per platform with the UI always embedded, but `update` still offered a variant choice: "ui" could only 404, and "standard" quietly WAS the UI build. Reported by @iandol upgrading 0.9.0 -> 0.10.0. The chooser, its --standard/--ui flags, and the ui- URL plumbing are removed, along with the CBM_VARIANT=ui remnant in the npm installer. Already-released 0.9.x binaries cannot be fixed retroactively, so the release workflow now publishes byte-identical ui-*-named alias assets — their updaters work again with no user action. The aliases are uploaded AFTER the VirusTotal gate: they are the same bytes as archives it already cleared, and uploading them earlier would duplicate every object in the scan set and the provenance manifest. **macOS install noise and attribution (#1537).** install.sh silenced the "No such xattr: com.apple.quarantine" line, which is what happens when a curl-downloaded archive carries no quarantine attribute — harmless, and it became the title of a bug report about an unrelated failure. The session-stop refusal now points at `daemon status` to list the client processes actually holding the daemon, instead of asserting sessions exist and leaving the reader to guess. Reported by @listepo. **Riders.** hatchling is pinned in pkg/pypi (an unpinned backend resolved fresh inside `python -m build` is what emitted Metadata-Version 2.5 and broke the v0.10.1 publish); SECURITY.md's supported-versions table moves to 0.10.x. Tests: separate seams for unverifiable vs mismatched peer images with a test per outcome; activation refusal must name directory + mode + rule; a group-writable ancestor must stage successfully. The update tests drop the flag that no longer exists. Verified against each reporter's environment shape. **Open security alerts (all three, OSSF Scorecard).** - HIGH, binary artifact: an 8.8 MB compiled Go ELF wrapper had been committed at pkg/go/codebase-memory-mcp by accident. Removed, and both it and its .exe sibling are gitignored so `go build` in that directory cannot repeat it. - HIGH, GHSA-2v37-7h3g-55p8: nanoid < 3.3.17 loops forever when a custom generator is called with size 0. It reaches us transitively (postcss -> vite), so it is pinned through the existing graph-ui overrides block rather than promoted to a direct dependency; the lockfile resolves 3.3.18. - MEDIUM, unpinned pip command: the publish step installed build/twine by version only, leaving the whole transitive graph resolved at run time. pkg/pypi/requirements-publish.txt now hash-pins the complete toolchain (316 hashes), generated on a linux/amd64 python:3.12 image so the wheels match what ubuntu-latest resolves, and the step runs pip with --require-hashes. Verified by installing from it in that same image. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
Martin Vogel
|
c9b5b39c94 |
test(release): restore the archive-extractor contract, and fix what it caught
Deleting this contract with the UI-pack architecture threw away coverage of
machinery that SURVIVED: the extractor still builds the scan-set and association
manifests that check-virustotal.sh reads. Without it the gate could report a
confident green over an incomplete set.
It immediately paid for itself. The extractor was BROKEN by my own edit:
NameError: name 'variant' is not defined (process_tar)
TypeError: add_association() missing 'variant'
Removing the variant from validate_namespace's return left the call sites still
threading it. python3 -m py_compile passes on that -- it is a runtime failure --
and the extractor runs ONLY in the release and dry-run VirusTotal jobs, so
nothing else in the tree or on the 3-OS ladder executes it. It would have
shipped and broken every release verification.
The contract is rewritten rather than adapted: the original was woven through
the pack format (16 archives, 88 associations, per-pack asset math), and a
half-adapted version that passes for the wrong reasons is worse than none. It
now asserts what still exists:
- the exact 8-archive matrix, 32 member associations, nothing dropped
- archive containers never enter the scan set as members
- deduplication is exact: 8 byte-identical LICENSE members collapse to ONE
scan object while 8 distinct binaries stay 8
- manifest metadata agrees with its own rows, and every staged object matches
its recorded SHA-256
- fail-closed on a surplus member, a missing member, and a short matrix
Registered as Step 0l so it runs in every venue, not just when someone
remembers.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
|
||
|
Martin Vogel
|
39e02544cc |
fix(smoke): UI presence is the caller's claim, not every lane's assumption
Making REQUIRE_UI unconditional broke all three pr-smoke legs: FAIL 15a: SMOKE_REQUIRE_UI=1 but this binary serves no embedded UI assets The PR lane runs scripts/build.sh WITHOUT --with-ui on purpose, so an npm frontend build does not land on every product PR. Demanding embedded assets there asserts a property that lane deliberately does not produce. The guard still exists where it means something: scripts/ci/smoke-artifact.sh builds --with-ui, packages the real archive and smokes the EXTRACTED result, so it exports SMOKE_REQUIRE_UI=1 and a frontend-less binary fails there. That is the lane whose whole job is release fidelity. Deliberately NOT fixed by adding --with-ui to pr.yml: that would put `npm ci && npm run build` on three runners for every product PR to re-prove something the dry-run and release lanes already gate. PR CI cost is not free and this branch should not quietly raise it. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
Martin Vogel
|
d602eb6edb |
test(composition): gate that read-only data stays out of the executable mapping
The arm64 W^X hole regressed silently once and section flags cannot catch it: .rodata is correctly marked A (not AX), but the kernel applies SEGMENT permissions, so a merged R E PT_LOAD maps it executable anyway. A Makefile flag alone would regress the same way the next time a linker default differs. A1b sums the executable PT_LOAD segments and fails when they cover most of the file. Measured on aarch64 with a 1.2 MB const table: without -z separate-code FAIL executable segments cover 96% of the file with -z separate-code PASS executable segments cover 0% Nearly shipped this vacuous: the first version summed with awk's strtonum(), which is a gawk extension. CI's awk is mawk, where it is undefined -- the total would have been 0 and the gate would have passed on exactly the artifacts it exists to catch. It now emits the hex MemSiz fields and converts in the shell, and the numbers above were produced with the container's mawk, not gawk. Mach-O and PE report n/a; the real macOS artifact still passes all 13 assertions. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
Martin Vogel
|
a4336dc40a |
feat(release): ship one archive set and tolerate a single Microsoft !ml verdict
Completes the collapse to a single shipped composition and replaces the
zero-tolerance VirusTotal gate with a narrow, disclosed policy.
Packaging and installers
- package-release.sh loses --variant; archives are codebase-memory-mcp-<os>-<arch>
with exactly four members. install.sh/install.ps1 lose --ui/--standard.
- The extractor drops CBMUIPK pack parsing and --archive-scope; its scan-set and
association manifests (which the gate depends on) are unchanged otherwise.
- npm/PyPI/Go wrappers: the runtime "set" is one file again. The Windows lock
and race fixes from #1495/#1496 are kept; only multi-file set membership goes.
This also fixes `pip install` on Windows, which rejected the fifth archive
member against a hardcoded four-name allowlist.
- The wrappers' post-download probe moves from --verify-runtime-assets (removed)
to --version, which proves the same thing: the binary executes.
VirusTotal gate
- Exactly ONE detection is tolerated, and only when the engine is Microsoft AND
the label ends in `!ml`. Two or more engines, any non-`!ml` label, any other
vendor, any suspicious verdict and every infrastructure error still block.
- A tolerated object prints TOLERATED:, never OK:, and its counts are recorded
in vt-results.tsv exactly as a blocked one would be.
- append-vt-notes.sh mirrors the policy. It previously hard-failed on any
malicious count, so loosening only the gate would have passed the scan and
then died at note publication. The notes now DISCLOSE a tolerated detection
and link to SECURITY.md rather than claiming "0 malicious" for everything.
Rationale for the tolerance is in the gate itself: the verdict is not a property
of our bytes. It inverts across architectures and link modes, moves between
sibling artifacts of one build, and lands in different variant buckets for the
same source. The same `!ml` family hits llama.cpp, GitHub's own `gh`, Microsoft's
own Go toolchain and Anthropic's Claude installer.
The zero-tolerance contract becomes test_vt_gate_policy_contract.sh, asserting
the full matrix: 1x Microsoft !ml passes and reports TOLERATED; a Microsoft
signature label, a non-Microsoft engine, two engines, a suspicious verdict and
every malformed-response case still block. Its tripwire is narrowed to the
reverted endpoint-verification mechanism rather than the words "false positive",
so it no longer fires on a deliberate in-gate policy branch.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
|
||
|
Martin Vogel
|
d58afe562d |
revert(release): re-embed runtime assets into the single shipped binary
Externalizing the integration templates (#1492/#1493) and the UI bundle
(#1501/#1503) was done to reduce the Microsoft `Wacatac.B!ml` surface. It did
not work: across dry runs the flagged artifact count stayed at ~3 and the
detections merely moved between artifacts.
Dissection of run 31286803592 shows there is no structural cause to fix. The
verdicts split across every axis at once — linux-amd64 (dynamic) flagged while
linux-amd64-portable (static) is clean, but linux-arm64 (dynamic) clean while
linux-arm64-portable (static) is flagged. The two macOS binaries have identical
segment structure and split clean/flagged. Siblings from one build landed in
different variant buckets (.B vs .C). Entropy is low everywhere
(code_vectors.bin 4.166, grammar tables 3.464 bits/byte, against 7.5-8.0 for
packed payloads), so the packed-payload hypothesis is excluded too.
So the complexity bought nothing, and installation goes back to being
self-contained: one binary that carries its own UI and agent integration
templates, with no adjacent data file that has to resolve before `install`
works. Only the UI-capable composition ships from now on, under the historical
unsuffixed archive name.
Removed: src/ui/asset_pack.{c,h}, asset_pack_stub.c, asset_manifest_stub.c,
scripts/pack-ui-assets.mjs, src/cli/integration_assets.{c,h},
assets/cbm-integrations.json, scripts/gen-integrations-hash.sh, the
--verify-runtime-assets probe (nothing adjacent left to verify), and the
composition gates A6/A7 whose property is now deliberately inverted.
Restored: scripts/embed-frontend.sh, src/ui/embedded_{assets.h,stub.c}, the
compiled-in hook/adapter template bodies, and the embed/EMBED_OBJS build path.
Kept from the reverted commits, re-applied by hand where a wholesale file
restore would have dropped them:
- cbm_module_path_utf8() in both self-path sites. GetModuleFileNameA renders
through the ANSI code page and mangles non-ASCII install paths.
- the /__cbm/ui-readiness HMAC proof, secure_random and cbm_hmac_sha256, so
`daemon start --open` still waits for a genuine CBM listener.
- X-Content-Type-Options: nosniff on served assets.
- the MinGW noexecstack gate, -lbcrypt, and the cppcheck/zip CI fixes.
Archives are now codebase-memory-mcp-<os>-<arch>[-portable] with exactly four
members (binary, LICENSE, installer, THIRD_PARTY_NOTICES.md). That restores the
names every static package manifest already points at — aur, chocolatey,
homebrew, scoop, winget and glama were all broken by the -ui- rename.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
|
||
|
Martin Vogel
|
23e4fb0b4e |
fix(ci): scan extracted UI release files only
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
Martin Vogel
|
1ad53bb13d |
fix(ci): accept content-bound VT analysis aliases
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
Martin Vogel
|
5c18e3f2d0 |
ci: scan only UI artifacts in dry runs
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
Martin Vogel
|
b0279a010c |
fix(ci): resolve VT curl through current Bash
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
Martin Vogel
|
7a62a085cf |
fix(packages): make wrapper locks Windows-safe
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
Martin Vogel
|
8018561cfe |
fix(release): externalize runtime assets and harden VT verification
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
Martin Vogel
|
a6f132bc85 |
harden: ship integration templates as a verified JSON asset, not embedded bytes
The product binary embedded nine complete shebang'd shell scripts, their
PowerShell/.cmd twins, and two node:child_process client-adapter modules as C
string literals, and wrote them into agent config dirs at 0755 on install. A
block of script-shaped bytes inside a large unsigned executable is exactly the
surface Microsoft Defender's ML scored Trojan:Script/Wacatac.B!ml — the
detection that has recurred across the last several release attempts, on the
Linux ELF and darwin-arm64 standard binaries while every UI and Windows variant
stayed clean.
This moves every integration template out of the binary into a single shipped
data file, assets/cbm-integrations.json (compact: one asset, because each
shipped file is itself scanned). The binary now carries only the templates'
identity — one embedded SHA-256, generated at build time from the JSON so it
can never drift — plus the code to load, verify, and render them.
Why a hash and not just a loose file: content compiled into the binary was
tamper-evident with the binary. Moving it to disk must not turn it into
unauthenticated code the installer blindly executes. install verifies the asset
against the embedded SHA-256 and fails closed on mismatch ("integration assets
missing or modified - reinstall from the release archive") before writing
anything. Verified end to end: a one-byte edit to the asset makes install
refuse.
Ownership, for safe upgrade/removal: the verified copy is published to
~/.cbm/assets/<version>/ — a sibling of the DB cache, never inside it, so
clearing the cache cannot strip a user's hooks. A deployed integration file is
ours to remove only if it matches a template rendered from that stored copy;
foreign files and user-edited files at the reserved paths are preserved. The
historical template bodies travel in the JSON's released[] arrays so files
written by older versions are still recognised and cleaned up on uninstall.
Removed from src/: cmm_gate/session/subagent script-prefix and suffix
constants, cbm_build_released_gate_script, the shell/PowerShell/.cmd bodies in
cli.c, and the JS/TS generators in client_adapter.c. Confirmed on the built
standard binary: 0 occurrences of '#!/usr/bin/env bash', '#!/bin/sh',
'node:child_process', and 'ExecutionPolicy' (was non-zero for each).
test_no_embedded_scripts_contract.sh makes the removal a build property, not a
reachability assumption: no production C source may contain a shebang or
node:child_process literal, and the built binary is scanned directly. It strips
comments first (a contract a comment can satisfy is a false guard), and is
revert-checked — reintroducing one shebang literal makes it fail naming the
file:line. check-binary-composition.sh (A6) enforces the same needles on every
release artifact post-strip.
This does not, by itself, prove the detection is cured: the clean UI binary is a
strict superset of the flagged standard one, so no single passage is a
deterministic trigger. It removes the most coherent malware-shaped composite
from every shipped binary, aligns with how gh/starship/zoxide externalise
integration material, and makes the templates maintainable as data rather than
escaped C literals. Causal confirmation still needs a VirusTotal ablation.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
|
||
|
Martin Vogel
|
f3e79f976c |
ci(vt-gate): name the engine that flagged, not just the count
The gate reported "1 malicious, 0 suspicious / 59 engines" and stopped there —
withholding the one fact a detection is actionable on. A false-positive
submission goes to a SPECIFIC vendor, and triage begins with "known-noisy engine
or a new family?". Neither question is answerable from a count.
It cost exactly that in the 2026-08-07 dry run: three artifacts flagged
(linux-amd64, linux-amd64-portable, darwin-arm64), one engine each, and the logs
could not say by what. The vendor had to be recovered by hand from the web UI
afterwards — and the WDSI developer false-positive path is a web form with no
public API, so guessing the wrong vendor wastes a filing round trip.
The analysis response already carries per-engine results; the gate simply threw
them away. It now prints, under the BLOCKED line, each engine that returned
malicious or suspicious with its detection label and definition date:
BLOCKED: codebase-memory-mcp-linux-amd64 flagged (1 malicious, ...)
detected by: Microsoft = Trojan:Win32/Wacatac.B!ml (engine ..., defs ...)
Engines that did not flag are not listed, or the actionable line drowns in
sixty clean rows.
Deliberately unchanged: the gate's strictness. Zero tolerance still means any
detection on any artifact on any version blocks, and an analysis WITHOUT
per-engine results still blocks on the counts alone — the contract pins both, so
naming engines cannot become a route to tolerating one. A missing engine list
prints an explicit "<engine names unavailable>" rather than nothing, so the gap
is visible instead of reading as a detection-free block.
Verified by revert: against the old gate the new assertions fail with the
counts-only output quoted above.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
|
||
|
Martin Vogel
|
b67401a03b |
ci(dry-run): scan the archive artifacts with VirusTotal as the final step
Preparation for the next dry-run cycle. The release gate scans every shipped
artifact with VirusTotal and blocks on any detection; the dry run did not, so
the one question most likely to stop a release was the one a dry run could not
answer. v0.9.1-rc found that out the expensive way.
The scan mirrors the release job. The only difference is where the archives come
from: the release pulls them off a published release, a dry run has none, so
they come from the build job's artifacts — the same files, produced by the same
canonical package-release.sh, extracted the same way and scanned as the same
bytes a user would download and run. Same zero-tolerance gate
(scripts/ci/check-virustotal.sh), so a dry run now fails where the release would.
It runs LAST, after smoke and soak. It is the slowest job — polling can take two
hours — and its verdict does not depend on our code at all: the same unchanged
bytes can come back clean one day and flagged the next. That must not delay the
feedback that IS about our code.
Two independent skips, because they answer different questions:
skip_builds no artifacts at all "do lint and tests pass?"
skip_virustotal build + smoke + soak, no scan "do the archives pass CI?"
The second is the one asked for: during a debug cycle, when the question is
whether the archives survive smoke and soak, a two-hour scan burning API quota
is pure noise. skip_builds implies no scan, since there is nothing to scan.
Extraction lives in scripts/ci/extract-release-archives.sh rather than inline in
the YAML because the venue-parity contract requires it — a venue may provision,
plumb artifacts, or call a canonical leg script, and this is leg logic. The
contract caught the inline version.
Two defects found by testing the script instead of trusting it:
* The empty-input guard did not guard. It counted files in the output
directory, but the install scripts are copied in unconditionally, so the
count was never zero and an input with no archives reported success — a
clean VirusTotal run over nothing, the exact false green the check exists to
prevent. It now counts binaries recovered from archives, and exits 1 on
zero.
* The script was committed 100644 while the workflow invokes it directly as a
command, which is the "Permission denied" failure
test_script_exec_bit_contract.sh was written for. That contract scans
scripts/ but not .github/workflows/, so it did not catch it here.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
|
||
|
Martin Vogel
|
d6c8d1dd0a |
fix(msan): split the known-red block by cause; fix the RSS one properly
The x86-64 leg runs this lane without exclusions on purpose, to settle
which limits are architectural. It has now run, and it disproves part of
what the previous block asserted. That block claimed all seven excluded
suites "abort with stack-overflow". Five do. Two do not, and lumping
them together hid two different problems behind one rationale.
(A) stack-overflow, five suites: grammar_regression grammar_labels
pipeline lang_contract grammar_probe_e. Confirmed on BOTH arm64 and
x86-64 (CI logged 5), so it is not the aarch64 artifact an earlier
note claimed. The recursion guards bound DEPTH while the resource
exhausted is BYTES; that follow-up stands unchanged.
(B) cli: no overflow at all. On x86-64 it runs to completion, 253
passed / 5 failed, every failure in the install or activation path,
with "agent_config agent=OpenClaw op=mcp_install" above them. Green
on every other venue. MSan reported zero use-of-uninitialized-value
in it, so the exclusion costs no uninit coverage. Recorded as
undiagnosed rather than guessed at: the local lane is arm64 where
these suites hit (A) before reaching this code, so there is no
faithful venue to iterate in and each attempt is a ~30min round
trip. That is a follow-up with an owner, not a dismissal.
(C) incremental: an RSS BUDGET failure, 3054MB against a 2304MB limit
-- not an overflow either. MSan maps shadow (and origin) memory for
every allocation, so the budget cannot separate a leak from shadow.
FIXED rather than excluded: the assertion is now skipped under
__has_feature(memory_sanitizer) only, so the guard keeps its teeth
on every other platform, where inflating the budget would have
blinded it. The suite stays IN the lane.
Verified: with (C) fixed, incremental is 163 passed / 0 failed and ZERO
stack-overflows under the local arm64 MSan container -- so it never
belonged in the overflow list on either architecture.
msan-lane.sh no longer forces MSAN_EXCLUDE empty. That override existed
to ask the architectural question; it is answered, and keeping it would
re-red the gate for causes already recorded. Both venues now read the one
authoritative list in scripts/msan.sh, which still warns loudly that the
lane is partial.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
|
||
|
Martin Vogel
|
64bd272cb2 |
fix(ci): mark the new lane scripts executable, and let the contract see them
scripts/ci/lint-mem.sh and scripts/ci/msan-lane.sh were committed at mode 100644, so the workflow step that runs them directly died with "Permission denied" (exit 126). scripts/lint-mem-gate.py gets the same treatment: it is invoked through python3 today, but it carries a shebang and should not depend on that. The exec-bit contract already exists to catch precisely this, and it did not, because it derives its candidate set from `git ls-files -s '*.sh'` -- tracked files only. A brand-new script is invisible there until it is committed, so the check passes on the run where the defect is introduced and only starts failing on the run that ships it. The window where the contract is most useful was the one window it could not see. It now also considers not-yet-tracked scripts by their filesystem mode. Verified against the real defect rather than in the abstract: with lint-mem.sh untracked and non-executable the contract reports "_lint.yml:76 executes scripts/ci/lint-mem.sh directly, but its committed mode is 100644", and passes once the bit is set. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
Martin Vogel
|
f5baf1baef |
fix(ci): route the new lanes through canonical leg entries
The MSan and memory-analyzer jobs drove docker and make directly from workflow steps, and the diag step used folded `run: >`. All four are venue-parity violations: a venue may provision, plumb artifacts, or call a canonical leg script, and nothing else. Anything that actually exercises the product belongs inside a scripts/ entry so that every venue runs the same code instead of each workflow growing its own slightly different invocation. So the docker work moves to scripts/ci/msan-lane.sh (build | run | all) and the analyzer gate to scripts/ci/lint-mem.sh, both of which the local paths already reach through run.sh and the Makefile. The folded step becomes `run: |`. Found by the contract itself, running as step 0j of the local Linux leg. It had never run against these jobs, because they were added and then exercised only through GitHub CI -- which is exactly the gap the local ladder exists to close, and the reason the contract runs as step zero of every leg rather than as a job of its own. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
Martin Vogel
|
359485b6c2 |
fix(release): make the new CI scripts executable, and pin that they stay so
Every unix build leg of release run 30499236230 died during packaging:
scripts/package-release.sh: line 190: scripts/ci/check-binary-composition.sh: Permission denied
The composition gate was committed 100644 while being invoked as a command. It
passed every local check because my WORKING COPY had the exec bit -- only the
committed mode was wrong, and nothing you can run locally reveals that.
scripts/ci/append-vt-notes.sh had the identical defect waiting in the verify
step, the last step of the release, so this would have failed a second time after
two hours of tests, build, smoke and soak.
Fixed on BOTH sides, because either alone suffices and the pair is mode-proof:
the two scripts are now 100755, and their call sites invoke them through `bash`,
which is what most of this repo already does and which cannot break if a mode bit
is ever lost to a patch application or a non-POSIX checkout.
tests/test_script_exec_bit_contract.sh pins the class: any tracked .sh whose
COMMITTED mode is non-executable must not appear as the first word of a command
in workflows, scripts, test-infrastructure or the Makefiles. Verified in both
directions -- it passes on this tree, and fails on the exact defect when the mode
and the call site are reverted. It also joins backslash continuations before
analysing, because its own first draft reported a false positive on
... && bash \
test-infrastructure/vm/vm-run-tests.sh --soak
and a contract that cries wolf teaches people to ignore contracts.
Product code is untouched: the test phase of the failed run was 27/27 green on
this exact tree, and a file mode cannot change a test outcome.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
|
||
|
Martin Vogel
|
1f674c805e |
harden: remove capability that should never have shipped
Microsoft's ML flagged the rc.1 release binaries. The decisive evidence is that the SAME sha256 went from 0/62 clean to Microsoft-detected in about an hour with no byte change, so the verdict lives partly in scanner-side state and no code change can promise a clean result. What code CAN do is stop shipping things that have no business in a release artifact, which is worth doing on its own merits and incidentally widens the classifier margin. Every claim below is verified against a built binary by the new gate, not by reading source. Executable stack (the worst of the findings). vendored/nomic/code_vectors_blob.S is the only assembly in the build and carried no .note.GNU-stack. An unannotated object makes ld assume the worst for the whole link, so EVERY Linux release we have ever shipped had GNU_STACK RWE. Adds the note (cause) plus ELF-only -Wl,-z,noexecstack (outcome); the gate fails the release if it returns. Test seams are now opt-in, never opt-out. TEST_SEAMS=1 defines CBM_ENABLE_TEST_SEAMS; without it the crash-orphan probe -- which forks a child that ignores SIGTERM and loops forever, then writes its pid to a caller-supplied path -- and the lease-ownership marker compile to trivial stubs, so call sites are untouched and the binary holds no fork, no signal handler and no env-var string. Opt-IN is the point: forgetting the flag yields a clean binary rather than a leaky one. scripts/test.sh requests it in the leg that consumes it, and tests/test_worker_watchdog.sh now asserts the capability up front instead of dying later with an opaque "Killed: 9". The daemon's background version check is gone. It spawned curl against api.github.com/repos/.../releases/latest on the first eligible session of every run to say "a newer version exists" -- a release URL and an outbound request in every shipped binary, for something the install scripts already report. The INJECTABLE SEAM survives: update_ops is still honoured, the fakes in tests/test_daemon_application.c still cover notice/ownership/cancellation/replay, and with no provider application_update_subscribe_locked returns early so no generation ever starts. "No network request by default" is now structural. Dead capability out of release builds. The tar.gz/zip extraction block (gzip_decompress through cbm_extract_binary_from_zip, plus its cli.h declarations) moves under CBM_CLI_ENABLE_TEST_API -- verified self-contained, zero uses of any helper outside it, only callers the excluded updater and tests/test_cli.c. Downloading an archive, decompressing it, picking an executable out of it and marking it executable is the canonical dropper composite; it is now absent rather than merely unreachable. SQLite is built with -DSQLITE_OMIT_LOAD_EXTENSION (no caller of load_extension anywhere in src/ or internal/), removing that API surface and part of the dlopen/dlsym surface. Temp files and environment scanning (S2/S3). Predictable paths in mcp.c, artifact.c and diagnostics.c are created privately and exclusively and written through the returned descriptor; pass_envscan.c no longer descends symlinked directories out of the project root, and its fixed 512-byte path buffers no longer truncate into pointer arithmetic that could land outside the buffer. Build-time entropy. mimalloc's version banner baked __DATE__/__TIME__ into every binary, so two builds of identical source seconds apart could never share a hash and no release could inherit a false-positive determination made about its predecessor. Local patch removes it (marked to survive refreshes), -Wdate-time makes any future use a build error, and -Wl,--no-insert-timestamp stops the PE header carrying the link clock. scripts/ci/check-binary-composition.sh is the proof that each removal stays removed, wired into package-release.sh after strip so the local artifact-flow smoke enforces exactly what CI does. It asserts absences plus a CANARY string, so handing it a compressed, stubbed or empty file fails instead of passing vacuously, and a missing tool is a hard error -- a skipped assertion must never look like a satisfied one. Two build-system traps found by that gate, both of which had silently defeated a fix: the product binary is compiled in one shot from sources, so a flag flip did not rebuild it (now tracked by a .build-config stamp that also removes the binary, making it independent of mtime granularity); and prod_sqlite3.o / prod_mimalloc.o depended on a single named source, so SQLITE_OMIT_LOAD_EXTENSION and the mimalloc patch BOTH compiled to nothing on the first incremental build. Source review would have called them done. Deliberately NOT changed. Three seams stay in release artifacts because scripts/smoke-test.sh runs against the real artifact and needs them: CBM_TEST_CRASH_ON and CBM_TEST_HANG_ON inject the faults that prove supervisor recovery, and CBM_TEST_WINDOWS_USER_PATH_RUN_ID is what keeps the PATH smoke from writing the tester's actual PATH. The gate treats those as an allowlist, so a NOVEL seam still fails. The true no-UI standard build is deferred rather than rushed: src/ui/* is in PROD_SRCS and four files outside src/ui reference UI symbols, including the daemon that serves the UI, so that assertion reports instead of failing until the split lands -- a gate everyone knows is red teaches people to ignore gates. No grammar is removed. ObjectScript accounts for essentially all binary growth since the last provably-clean release (+21.4MB rodata, +1.1MB text from two four-line shims), which made it the obvious ablation candidate, but a dry run performed twelve real Defender endpoint scans across standard/UI and amd64/arm64 with ObjectScript, the daemon and the expanded hooks all present and every scan was clean. Nothing there is a deterministic trigger, so cutting a community-contributed language would spend a real feature on unproven margin. Lean is not a candidate either: at 99.6MB of source it is by far the largest grammar, but it shipped in v0.9.0 which scanned 20/20 clean, so removing it would produce a novel unscanned profile instead of restoring a known-good one. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
Martin Vogel
|
3e343a3eb2 |
ci(release): revert the VT gate to zero tolerance
The pre-release ML false-positive tolerance (single-engine Microsoft "!ml" verdicts downgradable with Defender endpoint evidence, #1340) is reverted by owner decision: cbm does not ship binaries carrying a VirusTotal detection, demonstrably false or not. A "trojan" badge on a release asset is a reputation cost the project is not willing to price in, however good the accompanying evidence. The gate returns to its original form: any detection, by any engine, on any artifact, on any version blocks the release. The endpoint verification tool and the evidence side-channel are removed with it; the notes renderer keeps its extracted-script form but only ever states a verified "0 detections". False positives are resolved upstream instead: verify the bytes on a real Defender endpoint, submit a Microsoft false-positive report for the exact hashes, wait for the detection to clear, then RE-RUN the failed verify job -- which does not rebuild, so the cleared hashes are the shipped hashes. tests/test_vt_gate_zero_tolerance_contract.sh pins the decision: clean passes; 1 malicious and 1 suspicious each block across stable, -rc., -pre and -alpha versions; plus a tripwire for the specific reverted evidence mechanism returning. Loosening this gate again has to consciously delete that contract. The skip_tests dispatch input and the script-extracted notes step survive the revert -- both are orthogonal to gating policy. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
Martin Vogel
|
48fc942418 |
ci(release): double-verify ML antivirus false positives instead of re-rolling builds
Release run 30464288732 was blocked by the VirusTotal gate: three linux-amd64
binaries flagged 1/62 by Microsoft's Wacatac.B!ml -- fully stripped binaries
(0 symbols, verified on the exact artifacts), the state that scanned clean in
the two previous cycles. Meanwhile a real Defender endpoint (engine
1.1.26060.3008, signatures 1.455.410.0 updated the same day, RTP on) scans
the identical bytes clean. Four cycles of evidence now say the same thing:
this verdict is an unstable ML decision boundary, not a property of the code,
and no build-side lever moves it durably -- stripping, downloader removal and
metadata changes each "worked" only until a later build flipped it back.
So stop treating the flag as buildable-away and verify it honestly instead:
check-virustotal.sh may downgrade BLOCKED to TOLERATED only when ALL hold:
- pre-release version (-rc./-pre/-alpha/-beta); stable releases never
- every failing file flagged by exactly ONE engine
- that engine is Microsoft and the verdict ends in "!ml" (never a
signature name)
- hash-pinned Defender ENDPOINT evidence is attached to the draft release
(defender-endpoint-verification.txt) proving Microsoft's shipping
product, signature-updated at scan time, reports the exact bytes clean
av-endpoint-verify.sh (new) produces that evidence: downloads the draft
assets, scans them on the local Windows VM endpoint, refuses to attest if
RTP is off or Defender itself detects, uploads the hash-pinned result.
The gate prints the exact command when evidence is missing; re-running the
failed verify job does not rebuild, so the bytes stay fixed.
append-vt-notes.sh (new, extracted from inline YAML per venue-parity) then
renders the release-notes table honestly: a tolerated file reads "1/62 ML
false positive, endpoint-verified clean", never "0 detections".
tests/test_vt_gate_tolerance_contract.sh pins all nine decision directions
against a stubbed VT API and release store -- clean pass, stable-never,
missing/stale/DETECTED evidence, signature-named verdict, non-Microsoft
engine, multi-engine -- so the tolerance provably fails closed.
Also: release.yml gains skip_tests for re-releases of an already test-green
tree (build/smoke/soak/verify always run; lint failures still gate via
!cancelled() && !failure()).
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
|
||
|
Martin Vogel
|
a54ea95719 |
fix(windows): ship one binary — remove the launcher stub flagged as a dropper
DCO / dco (push) Has been cancelled
Windows shipped a PAIR: a small permanent launcher (codebase-memory-mcp.exe)
plus the real product binary (codebase-memory-mcp.payload.exe). The launcher
existed for exactly one reason — a running .exe cannot replace its own image
on Windows, so an in-process self-update needs a second resident binary to do
the swap.
That stub is statically indistinguishable from a dropper: a small, unsigned,
zero-prevalence PE whose whole job is verify-and-execute another binary.
Defender's ML scored it Trojan:Win32/Wacatac.B!ml and blocked the v0.9.1-rc.1
release at the VirusTotal gate. It is not fixable in our code on x64 —
bcrypt-free, stripped, VERSIONINFO'd, minimal-resource and even
resource-FREE builds on CI's own MSYS2 CLANG64 toolchain were all flagged,
while the product binary scans clean on every platform.
So remove the stub and move self-update OUT of the process into install.ps1,
which runs while cbm is NOT running: Windows' image lock only blocks a
process from replacing ITSELF. now prints the exact PowerShell
command (with the Unblock-File hint for Mark-of-the-Web); install.ps1 is
idempotent, so re-running it IS the update — it stops the daemon, renames the
running binary aside (the one mutation Windows permits on a running image),
publishes the new one, and sweeps retired copies.
Windows now matches Linux and macOS: ONE binary per platform.
* packaging, install.ps1, npm and PyPI wrappers all carry a single binary
* the launcher/payload ABI contract and ~2500 lines of stub state machinery
are deleted
* every daemon start, CLI call and hook fire loses a process spawn, a named
pipe handshake and an stdio relay
* test_windows_bundle_contract.sh is rewritten as an INVERTED contract: it
now asserts no shipped surface can reintroduce a launcher/payload pair,
and that install.ps1 retires the running binary before publishing
Verified: VirusTotal 0/67 on the packaged binary and 0/58 on install.ps1 (no
certificate involved); macOS and Linux full suites green; Windows guards all
green including the new update-handoff contract; npm 10/10; PyPI 3/3.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
|
||
|
Martin Vogel
|
a20a50df77 |
fix(ci): executable bits on canonical entry scripts
git-apply of the staged unification recorded the new canonical entries as 100644; workflows execute them directly (run: scripts/ci/...), so a fresh CI checkout would fail at ci-ok/shard-union/soak/package steps with permission denied — the same class the parity worktree's own preflight just tripped over locally. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
Martin Vogel
|
0a40fe8bea |
feat(ci): local-remote parity lanes — artifact-flow smoke, glibc floor, Defender-ON everywhere
Closes the closable local-vs-remote venue gaps so that, outside arch legs / YAML glue / release plumbing, every local red is a remote red and vice versa on shared legs: - scripts/package-release.sh: THE canonical archive step (names + five-file Windows bundle layout defined once); _build.yml's eight inline archive blocks become calls to it. - scripts/ci/smoke-artifact.sh: the artifact-flow smoke lane — build, package, extract, then the canonical wrapper in artifact mode; wired as run.sh smoke-artifact (compose service), win.sh smoke-artifact, and directly runnable on macOS. Archive-layout bugs now surface locally instead of in a release dry run. - glibc-floor leg (Dockerfile.glibc22 + compose + run.sh): portable binary smokes on ubuntu-22.04/glibc 2.35; the dynamic binary must refuse there (2.38+ floor by design). - Defender-ON parity (user directive): scripts/ci/ensure-defender.ps1 enables + VERIFIES real-time protection, fail-closed; runs in every Windows CI job (_test x2, _soak x3, _smoke, pr.yml) AND in the VM preflight; _smoke.yml's scan engine-failure soft-skip becomes a red gate. Expected cost: slower Windows jobs (AV scanning during install/build/test I/O); the next dry run proves the runner side. - Contracts: launcher-bundle five-file check retargeted onto package-release.sh + per-archive canonical-call association; venue-parity contract requires the new lanes, counts one ensure-defender step per Windows job, adds --help probes for the new entries. Extended contract fails on the pre-change tree (verified: 20 violations on HEAD). - VM README: ephemerality/Defender posture documented — utmctl has no snapshot verb, so per-run revert stays a manual qcow2 option; the sweep preflight remains the standing mechanism. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
Martin Vogel
|
3fc93d12da |
ci: unify smoke/soak/test venues onto canonical leg entries
The staged venue-unification bundle: _smoke.yml onto the shared wrappers with an extracted-artifact input (CBM_SMOKE_ARTIFACT_DIR), soak-legs.sh as the one canonical soak entry (quick + query-leak legs across all venues), protected per-user TEMP roots shared across venues, clean-disk preflights before every VM/Docker run, win.sh and run.sh routed through the canonical scripts, soak.yml retired, and the venue-parity contract (whitelist walker: workflows may provision or call canonical entries, nothing else). Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
Martin Vogel
|
69c16cada4 |
fix: deliver the static -portable linux binary on all install/update paths
The standard linux release binary dynamically links glibc 2.38+ and GLIBCXX_3.4.32, so it fails to start on Debian 11, Ubuntu 20.04/22.04, RHEL/Rocky 8/9, Amazon Linux 2, etc. — yet install.sh, the npm and PyPI wrappers, and the binary's own self-update all fetched it by default, contradicting the "single static binary" promise. Point every linux install + self-update path at the fully-static "-portable" asset (gcc -static), which has no glibc floor. macOS/Windows are unaffected and unchanged. - install.sh, pkg/npm/install.js, pkg/pypi _cli.py: select -portable on linux - src/cli/cli.c: self-update download URL AND checksum archive name both use -portable on linux (they must match or the update fails checksum verify) - scripts/smoke-test.sh: assert linux self-update targets the -portable asset - scripts/ci/check-glibc-compat.sh: new guard — runs the binary inside debian:bullseye (glibc 2.31) and asserts it starts - _smoke.yml: run the guard on the portable binary in smoke-linux-portable Reproduced: standard binary -> "GLIBC_2.38 not found" on glibc 2.31; portable binary runs cleanly. |
||
|
Martin Vogel
|
d3aa98e7cd |
Fix VirusTotal gate: accept completed scans with < 60 engines
VT's status=completed is final — no more engines will report. The script was polling indefinitely when ARM binaries only reached 50/76 engines. Now accepts any completed scan, logs a NOTE when below MIN_ENGINES. |
||
|
Martin Vogel
|
87188913bc |
Refactor CI: split monolith workflows into reusable components
Before: 2 monolith YAMLs (904 + 1127 lines), duplicated matrices, inconsistent action versions, duplicate build-windows job. After: 5 reusable workflows + 3 lean callers (1091 total lines): - _lint.yml: lint + security-static + codeql-gate - _test.yml: tests on 5 platforms with CBM_SKIP_PERF support - _build.yml: standard + UI + portable builds, all platforms - _smoke.yml: smoke test every binary variant - _soak.yml: quick + ASan soak, parameterized duration Fixes: duplicate build-windows, missing Windows CBM_SKIP_PERF, missing timeout-minutes, inconsistent action versions, VirusTotal check extracted to scripts/ci/check-virustotal.sh. |