fix/script-exec-bits
11 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Martin Vogel
|
359485b6c2 |
fix(release): make the new CI scripts executable, and pin that they stay so
Every unix build leg of release run 30499236230 died during packaging:
scripts/package-release.sh: line 190: scripts/ci/check-binary-composition.sh: Permission denied
The composition gate was committed 100644 while being invoked as a command. It
passed every local check because my WORKING COPY had the exec bit -- only the
committed mode was wrong, and nothing you can run locally reveals that.
scripts/ci/append-vt-notes.sh had the identical defect waiting in the verify
step, the last step of the release, so this would have failed a second time after
two hours of tests, build, smoke and soak.
Fixed on BOTH sides, because either alone suffices and the pair is mode-proof:
the two scripts are now 100755, and their call sites invoke them through `bash`,
which is what most of this repo already does and which cannot break if a mode bit
is ever lost to a patch application or a non-POSIX checkout.
tests/test_script_exec_bit_contract.sh pins the class: any tracked .sh whose
COMMITTED mode is non-executable must not appear as the first word of a command
in workflows, scripts, test-infrastructure or the Makefiles. Verified in both
directions -- it passes on this tree, and fails on the exact defect when the mode
and the call site are reverted. It also joins backslash continuations before
analysing, because its own first draft reported a false positive on
... && bash \
test-infrastructure/vm/vm-run-tests.sh --soak
and a contract that cries wolf teaches people to ignore contracts.
Product code is untouched: the test phase of the failed run was 27/27 green on
this exact tree, and a file mode cannot change a test outcome.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
|
||
|
Martin Vogel
|
1f674c805e |
harden: remove capability that should never have shipped
Microsoft's ML flagged the rc.1 release binaries. The decisive evidence is that the SAME sha256 went from 0/62 clean to Microsoft-detected in about an hour with no byte change, so the verdict lives partly in scanner-side state and no code change can promise a clean result. What code CAN do is stop shipping things that have no business in a release artifact, which is worth doing on its own merits and incidentally widens the classifier margin. Every claim below is verified against a built binary by the new gate, not by reading source. Executable stack (the worst of the findings). vendored/nomic/code_vectors_blob.S is the only assembly in the build and carried no .note.GNU-stack. An unannotated object makes ld assume the worst for the whole link, so EVERY Linux release we have ever shipped had GNU_STACK RWE. Adds the note (cause) plus ELF-only -Wl,-z,noexecstack (outcome); the gate fails the release if it returns. Test seams are now opt-in, never opt-out. TEST_SEAMS=1 defines CBM_ENABLE_TEST_SEAMS; without it the crash-orphan probe -- which forks a child that ignores SIGTERM and loops forever, then writes its pid to a caller-supplied path -- and the lease-ownership marker compile to trivial stubs, so call sites are untouched and the binary holds no fork, no signal handler and no env-var string. Opt-IN is the point: forgetting the flag yields a clean binary rather than a leaky one. scripts/test.sh requests it in the leg that consumes it, and tests/test_worker_watchdog.sh now asserts the capability up front instead of dying later with an opaque "Killed: 9". The daemon's background version check is gone. It spawned curl against api.github.com/repos/.../releases/latest on the first eligible session of every run to say "a newer version exists" -- a release URL and an outbound request in every shipped binary, for something the install scripts already report. The INJECTABLE SEAM survives: update_ops is still honoured, the fakes in tests/test_daemon_application.c still cover notice/ownership/cancellation/replay, and with no provider application_update_subscribe_locked returns early so no generation ever starts. "No network request by default" is now structural. Dead capability out of release builds. The tar.gz/zip extraction block (gzip_decompress through cbm_extract_binary_from_zip, plus its cli.h declarations) moves under CBM_CLI_ENABLE_TEST_API -- verified self-contained, zero uses of any helper outside it, only callers the excluded updater and tests/test_cli.c. Downloading an archive, decompressing it, picking an executable out of it and marking it executable is the canonical dropper composite; it is now absent rather than merely unreachable. SQLite is built with -DSQLITE_OMIT_LOAD_EXTENSION (no caller of load_extension anywhere in src/ or internal/), removing that API surface and part of the dlopen/dlsym surface. Temp files and environment scanning (S2/S3). Predictable paths in mcp.c, artifact.c and diagnostics.c are created privately and exclusively and written through the returned descriptor; pass_envscan.c no longer descends symlinked directories out of the project root, and its fixed 512-byte path buffers no longer truncate into pointer arithmetic that could land outside the buffer. Build-time entropy. mimalloc's version banner baked __DATE__/__TIME__ into every binary, so two builds of identical source seconds apart could never share a hash and no release could inherit a false-positive determination made about its predecessor. Local patch removes it (marked to survive refreshes), -Wdate-time makes any future use a build error, and -Wl,--no-insert-timestamp stops the PE header carrying the link clock. scripts/ci/check-binary-composition.sh is the proof that each removal stays removed, wired into package-release.sh after strip so the local artifact-flow smoke enforces exactly what CI does. It asserts absences plus a CANARY string, so handing it a compressed, stubbed or empty file fails instead of passing vacuously, and a missing tool is a hard error -- a skipped assertion must never look like a satisfied one. Two build-system traps found by that gate, both of which had silently defeated a fix: the product binary is compiled in one shot from sources, so a flag flip did not rebuild it (now tracked by a .build-config stamp that also removes the binary, making it independent of mtime granularity); and prod_sqlite3.o / prod_mimalloc.o depended on a single named source, so SQLITE_OMIT_LOAD_EXTENSION and the mimalloc patch BOTH compiled to nothing on the first incremental build. Source review would have called them done. Deliberately NOT changed. Three seams stay in release artifacts because scripts/smoke-test.sh runs against the real artifact and needs them: CBM_TEST_CRASH_ON and CBM_TEST_HANG_ON inject the faults that prove supervisor recovery, and CBM_TEST_WINDOWS_USER_PATH_RUN_ID is what keeps the PATH smoke from writing the tester's actual PATH. The gate treats those as an allowlist, so a NOVEL seam still fails. The true no-UI standard build is deferred rather than rushed: src/ui/* is in PROD_SRCS and four files outside src/ui reference UI symbols, including the daemon that serves the UI, so that assertion reports instead of failing until the split lands -- a gate everyone knows is red teaches people to ignore gates. No grammar is removed. ObjectScript accounts for essentially all binary growth since the last provably-clean release (+21.4MB rodata, +1.1MB text from two four-line shims), which made it the obvious ablation candidate, but a dry run performed twelve real Defender endpoint scans across standard/UI and amd64/arm64 with ObjectScript, the daemon and the expanded hooks all present and every scan was clean. Nothing there is a deterministic trigger, so cutting a community-contributed language would spend a real feature on unproven margin. Lean is not a candidate either: at 99.6MB of source it is by far the largest grammar, but it shipped in v0.9.0 which scanned 20/20 clean, so removing it would produce a novel unscanned profile instead of restoring a known-good one. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
Martin Vogel
|
3e343a3eb2 |
ci(release): revert the VT gate to zero tolerance
The pre-release ML false-positive tolerance (single-engine Microsoft "!ml" verdicts downgradable with Defender endpoint evidence, #1340) is reverted by owner decision: cbm does not ship binaries carrying a VirusTotal detection, demonstrably false or not. A "trojan" badge on a release asset is a reputation cost the project is not willing to price in, however good the accompanying evidence. The gate returns to its original form: any detection, by any engine, on any artifact, on any version blocks the release. The endpoint verification tool and the evidence side-channel are removed with it; the notes renderer keeps its extracted-script form but only ever states a verified "0 detections". False positives are resolved upstream instead: verify the bytes on a real Defender endpoint, submit a Microsoft false-positive report for the exact hashes, wait for the detection to clear, then RE-RUN the failed verify job -- which does not rebuild, so the cleared hashes are the shipped hashes. tests/test_vt_gate_zero_tolerance_contract.sh pins the decision: clean passes; 1 malicious and 1 suspicious each block across stable, -rc., -pre and -alpha versions; plus a tripwire for the specific reverted evidence mechanism returning. Loosening this gate again has to consciously delete that contract. The skip_tests dispatch input and the script-extracted notes step survive the revert -- both are orthogonal to gating policy. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
Martin Vogel
|
48fc942418 |
ci(release): double-verify ML antivirus false positives instead of re-rolling builds
Release run 30464288732 was blocked by the VirusTotal gate: three linux-amd64
binaries flagged 1/62 by Microsoft's Wacatac.B!ml -- fully stripped binaries
(0 symbols, verified on the exact artifacts), the state that scanned clean in
the two previous cycles. Meanwhile a real Defender endpoint (engine
1.1.26060.3008, signatures 1.455.410.0 updated the same day, RTP on) scans
the identical bytes clean. Four cycles of evidence now say the same thing:
this verdict is an unstable ML decision boundary, not a property of the code,
and no build-side lever moves it durably -- stripping, downloader removal and
metadata changes each "worked" only until a later build flipped it back.
So stop treating the flag as buildable-away and verify it honestly instead:
check-virustotal.sh may downgrade BLOCKED to TOLERATED only when ALL hold:
- pre-release version (-rc./-pre/-alpha/-beta); stable releases never
- every failing file flagged by exactly ONE engine
- that engine is Microsoft and the verdict ends in "!ml" (never a
signature name)
- hash-pinned Defender ENDPOINT evidence is attached to the draft release
(defender-endpoint-verification.txt) proving Microsoft's shipping
product, signature-updated at scan time, reports the exact bytes clean
av-endpoint-verify.sh (new) produces that evidence: downloads the draft
assets, scans them on the local Windows VM endpoint, refuses to attest if
RTP is off or Defender itself detects, uploads the hash-pinned result.
The gate prints the exact command when evidence is missing; re-running the
failed verify job does not rebuild, so the bytes stay fixed.
append-vt-notes.sh (new, extracted from inline YAML per venue-parity) then
renders the release-notes table honestly: a tolerated file reads "1/62 ML
false positive, endpoint-verified clean", never "0 detections".
tests/test_vt_gate_tolerance_contract.sh pins all nine decision directions
against a stubbed VT API and release store -- clean pass, stable-never,
missing/stale/DETECTED evidence, signature-named verdict, non-Microsoft
engine, multi-engine -- so the tolerance provably fails closed.
Also: release.yml gains skip_tests for re-releases of an already test-green
tree (build/smoke/soak/verify always run; lint failures still gate via
!cancelled() && !failure()).
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
|
||
|
Martin Vogel
|
a54ea95719 |
fix(windows): ship one binary — remove the launcher stub flagged as a dropper
DCO / dco (push) Has been cancelled
Windows shipped a PAIR: a small permanent launcher (codebase-memory-mcp.exe)
plus the real product binary (codebase-memory-mcp.payload.exe). The launcher
existed for exactly one reason — a running .exe cannot replace its own image
on Windows, so an in-process self-update needs a second resident binary to do
the swap.
That stub is statically indistinguishable from a dropper: a small, unsigned,
zero-prevalence PE whose whole job is verify-and-execute another binary.
Defender's ML scored it Trojan:Win32/Wacatac.B!ml and blocked the v0.9.1-rc.1
release at the VirusTotal gate. It is not fixable in our code on x64 —
bcrypt-free, stripped, VERSIONINFO'd, minimal-resource and even
resource-FREE builds on CI's own MSYS2 CLANG64 toolchain were all flagged,
while the product binary scans clean on every platform.
So remove the stub and move self-update OUT of the process into install.ps1,
which runs while cbm is NOT running: Windows' image lock only blocks a
process from replacing ITSELF. now prints the exact PowerShell
command (with the Unblock-File hint for Mark-of-the-Web); install.ps1 is
idempotent, so re-running it IS the update — it stops the daemon, renames the
running binary aside (the one mutation Windows permits on a running image),
publishes the new one, and sweeps retired copies.
Windows now matches Linux and macOS: ONE binary per platform.
* packaging, install.ps1, npm and PyPI wrappers all carry a single binary
* the launcher/payload ABI contract and ~2500 lines of stub state machinery
are deleted
* every daemon start, CLI call and hook fire loses a process spawn, a named
pipe handshake and an stdio relay
* test_windows_bundle_contract.sh is rewritten as an INVERTED contract: it
now asserts no shipped surface can reintroduce a launcher/payload pair,
and that install.ps1 retires the running binary before publishing
Verified: VirusTotal 0/67 on the packaged binary and 0/58 on install.ps1 (no
certificate involved); macOS and Linux full suites green; Windows guards all
green including the new update-handoff contract; npm 10/10; PyPI 3/3.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
|
||
|
Martin Vogel
|
a20a50df77 |
fix(ci): executable bits on canonical entry scripts
git-apply of the staged unification recorded the new canonical entries as 100644; workflows execute them directly (run: scripts/ci/...), so a fresh CI checkout would fail at ci-ok/shard-union/soak/package steps with permission denied — the same class the parity worktree's own preflight just tripped over locally. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
Martin Vogel
|
0a40fe8bea |
feat(ci): local-remote parity lanes — artifact-flow smoke, glibc floor, Defender-ON everywhere
Closes the closable local-vs-remote venue gaps so that, outside arch legs / YAML glue / release plumbing, every local red is a remote red and vice versa on shared legs: - scripts/package-release.sh: THE canonical archive step (names + five-file Windows bundle layout defined once); _build.yml's eight inline archive blocks become calls to it. - scripts/ci/smoke-artifact.sh: the artifact-flow smoke lane — build, package, extract, then the canonical wrapper in artifact mode; wired as run.sh smoke-artifact (compose service), win.sh smoke-artifact, and directly runnable on macOS. Archive-layout bugs now surface locally instead of in a release dry run. - glibc-floor leg (Dockerfile.glibc22 + compose + run.sh): portable binary smokes on ubuntu-22.04/glibc 2.35; the dynamic binary must refuse there (2.38+ floor by design). - Defender-ON parity (user directive): scripts/ci/ensure-defender.ps1 enables + VERIFIES real-time protection, fail-closed; runs in every Windows CI job (_test x2, _soak x3, _smoke, pr.yml) AND in the VM preflight; _smoke.yml's scan engine-failure soft-skip becomes a red gate. Expected cost: slower Windows jobs (AV scanning during install/build/test I/O); the next dry run proves the runner side. - Contracts: launcher-bundle five-file check retargeted onto package-release.sh + per-archive canonical-call association; venue-parity contract requires the new lanes, counts one ensure-defender step per Windows job, adds --help probes for the new entries. Extended contract fails on the pre-change tree (verified: 20 violations on HEAD). - VM README: ephemerality/Defender posture documented — utmctl has no snapshot verb, so per-run revert stays a manual qcow2 option; the sweep preflight remains the standing mechanism. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
Martin Vogel
|
3fc93d12da |
ci: unify smoke/soak/test venues onto canonical leg entries
The staged venue-unification bundle: _smoke.yml onto the shared wrappers with an extracted-artifact input (CBM_SMOKE_ARTIFACT_DIR), soak-legs.sh as the one canonical soak entry (quick + query-leak legs across all venues), protected per-user TEMP roots shared across venues, clean-disk preflights before every VM/Docker run, win.sh and run.sh routed through the canonical scripts, soak.yml retired, and the venue-parity contract (whitelist walker: workflows may provision or call canonical entries, nothing else). Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
Martin Vogel
|
69c16cada4 |
fix: deliver the static -portable linux binary on all install/update paths
The standard linux release binary dynamically links glibc 2.38+ and GLIBCXX_3.4.32, so it fails to start on Debian 11, Ubuntu 20.04/22.04, RHEL/Rocky 8/9, Amazon Linux 2, etc. — yet install.sh, the npm and PyPI wrappers, and the binary's own self-update all fetched it by default, contradicting the "single static binary" promise. Point every linux install + self-update path at the fully-static "-portable" asset (gcc -static), which has no glibc floor. macOS/Windows are unaffected and unchanged. - install.sh, pkg/npm/install.js, pkg/pypi _cli.py: select -portable on linux - src/cli/cli.c: self-update download URL AND checksum archive name both use -portable on linux (they must match or the update fails checksum verify) - scripts/smoke-test.sh: assert linux self-update targets the -portable asset - scripts/ci/check-glibc-compat.sh: new guard — runs the binary inside debian:bullseye (glibc 2.31) and asserts it starts - _smoke.yml: run the guard on the portable binary in smoke-linux-portable Reproduced: standard binary -> "GLIBC_2.38 not found" on glibc 2.31; portable binary runs cleanly. |
||
|
Martin Vogel
|
d3aa98e7cd |
Fix VirusTotal gate: accept completed scans with < 60 engines
VT's status=completed is final — no more engines will report. The script was polling indefinitely when ARM binaries only reached 50/76 engines. Now accepts any completed scan, logs a NOTE when below MIN_ENGINES. |
||
|
Martin Vogel
|
87188913bc |
Refactor CI: split monolith workflows into reusable components
Before: 2 monolith YAMLs (904 + 1127 lines), duplicated matrices, inconsistent action versions, duplicate build-windows job. After: 5 reusable workflows + 3 lean callers (1091 total lines): - _lint.yml: lint + security-static + codeql-gate - _test.yml: tests on 5 platforms with CBM_SKIP_PERF support - _build.yml: standard + UI + portable builds, all platforms - _smoke.yml: smoke test every binary variant - _soak.yml: quick + ASan soak, parameterized duration Fixes: duplicate build-windows, missing Windows CBM_SKIP_PERF, missing timeout-minutes, inconsistent action versions, VirusTotal check extracted to scripts/ci/check-virustotal.sh. |