Completes the previous commit, which carried only the two mode changes because
the call-site edits were not staged when it landed (--amend is denied in this
repo, so this is additive rather than a rewrite).
Call sites now go through `bash`, which is what most of this repo already does and
which cannot break if a mode bit is lost to a patch application or a non-POSIX
checkout:
scripts/package-release.sh -> bash scripts/ci/check-binary-composition.sh
.github/workflows/release.yml -> bash scripts/ci/append-vt-notes.sh
With the 100755 modes from the previous commit, both sides are now correct, and
either alone would have been sufficient.
tests/test_script_exec_bit_contract.sh pins the class so it cannot recur: any
tracked .sh whose COMMITTED mode is non-executable must not appear as the first
word of a command in workflows, scripts, test-infrastructure or the Makefiles.
The committed mode is the thing that matters and the thing no local run can
check -- the working copy having the bit is exactly why this shipped.
Verified in both directions: passes on this tree, and fails on the exact defect
when the mode and the call site are reverted together. It joins backslash
continuations before analysing, because its own first draft reported a false
positive on
... && bash \
test-infrastructure/vm/vm-run-tests.sh --soak
where the interpreter sits on the preceding line. A contract that cries wolf
teaches people to ignore contracts, so that had to be right before it could be
useful.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The pre-release ML false-positive tolerance (single-engine Microsoft "!ml"
verdicts downgradable with Defender endpoint evidence, #1340) is reverted by
owner decision: cbm does not ship binaries carrying a VirusTotal detection,
demonstrably false or not. A "trojan" badge on a release asset is a
reputation cost the project is not willing to price in, however good the
accompanying evidence.
The gate returns to its original form: any detection, by any engine, on any
artifact, on any version blocks the release. The endpoint verification tool
and the evidence side-channel are removed with it; the notes renderer keeps
its extracted-script form but only ever states a verified "0 detections".
False positives are resolved upstream instead: verify the bytes on a real
Defender endpoint, submit a Microsoft false-positive report for the exact
hashes, wait for the detection to clear, then RE-RUN the failed verify job --
which does not rebuild, so the cleared hashes are the shipped hashes.
tests/test_vt_gate_zero_tolerance_contract.sh pins the decision: clean
passes; 1 malicious and 1 suspicious each block across stable, -rc., -pre
and -alpha versions; plus a tripwire for the specific reverted evidence
mechanism returning. Loosening this gate again has to consciously delete
that contract.
The skip_tests dispatch input and the script-extracted notes step survive
the revert -- both are orthogonal to gating policy.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Release run 30464288732 was blocked by the VirusTotal gate: three linux-amd64
binaries flagged 1/62 by Microsoft's Wacatac.B!ml -- fully stripped binaries
(0 symbols, verified on the exact artifacts), the state that scanned clean in
the two previous cycles. Meanwhile a real Defender endpoint (engine
1.1.26060.3008, signatures 1.455.410.0 updated the same day, RTP on) scans
the identical bytes clean. Four cycles of evidence now say the same thing:
this verdict is an unstable ML decision boundary, not a property of the code,
and no build-side lever moves it durably -- stripping, downloader removal and
metadata changes each "worked" only until a later build flipped it back.
So stop treating the flag as buildable-away and verify it honestly instead:
check-virustotal.sh may downgrade BLOCKED to TOLERATED only when ALL hold:
- pre-release version (-rc./-pre/-alpha/-beta); stable releases never
- every failing file flagged by exactly ONE engine
- that engine is Microsoft and the verdict ends in "!ml" (never a
signature name)
- hash-pinned Defender ENDPOINT evidence is attached to the draft release
(defender-endpoint-verification.txt) proving Microsoft's shipping
product, signature-updated at scan time, reports the exact bytes clean
av-endpoint-verify.sh (new) produces that evidence: downloads the draft
assets, scans them on the local Windows VM endpoint, refuses to attest if
RTP is off or Defender itself detects, uploads the hash-pinned result.
The gate prints the exact command when evidence is missing; re-running the
failed verify job does not rebuild, so the bytes stay fixed.
append-vt-notes.sh (new, extracted from inline YAML per venue-parity) then
renders the release-notes table honestly: a tolerated file reads "1/62 ML
false positive, endpoint-verified clean", never "0 detections".
tests/test_vt_gate_tolerance_contract.sh pins all nine decision directions
against a stubbed VT API and release store -- clean pass, stable-never,
missing/stale/DETECTED evidence, signature-named verdict, non-Microsoft
engine, multi-engine -- so the tolerance provably fails closed.
Also: release.yml gains skip_tests for re-releases of an already test-green
tree (build/smoke/soak/verify always run; lint failures still gate via
!cancelled() && !failure()).
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Windows shipped a PAIR: a small permanent launcher (codebase-memory-mcp.exe)
plus the real product binary (codebase-memory-mcp.payload.exe). The launcher
existed for exactly one reason — a running .exe cannot replace its own image
on Windows, so an in-process self-update needs a second resident binary to do
the swap.
That stub is statically indistinguishable from a dropper: a small, unsigned,
zero-prevalence PE whose whole job is verify-and-execute another binary.
Defender's ML scored it Trojan:Win32/Wacatac.B!ml and blocked the v0.9.1-rc.1
release at the VirusTotal gate. It is not fixable in our code on x64 —
bcrypt-free, stripped, VERSIONINFO'd, minimal-resource and even
resource-FREE builds on CI's own MSYS2 CLANG64 toolchain were all flagged,
while the product binary scans clean on every platform.
So remove the stub and move self-update OUT of the process into install.ps1,
which runs while cbm is NOT running: Windows' image lock only blocks a
process from replacing ITSELF. now prints the exact PowerShell
command (with the Unblock-File hint for Mark-of-the-Web); install.ps1 is
idempotent, so re-running it IS the update — it stops the daemon, renames the
running binary aside (the one mutation Windows permits on a running image),
publishes the new one, and sweeps retired copies.
Windows now matches Linux and macOS: ONE binary per platform.
* packaging, install.ps1, npm and PyPI wrappers all carry a single binary
* the launcher/payload ABI contract and ~2500 lines of stub state machinery
are deleted
* every daemon start, CLI call and hook fire loses a process spawn, a named
pipe handshake and an stdio relay
* test_windows_bundle_contract.sh is rewritten as an INVERTED contract: it
now asserts no shipped surface can reintroduce a launcher/payload pair,
and that install.ps1 retires the running binary before publishing
Verified: VirusTotal 0/67 on the packaged binary and 0/58 on install.ps1 (no
certificate involved); macOS and Linux full suites green; Windows guards all
green including the new update-handoff contract; npm 10/10; PyPI 3/3.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
A version containing a prerelease suffix (v0.9.1-rc.1) now publishes as an
RC on every channel instead of masquerading as stable:
- GitHub release is created with prerelease: true (never marked Latest)
- npm publishes under the `next` dist-tag, so plain `npm install` keeps
serving the last stable; testers opt in with codebase-memory-mcp@next
- the MCP Registry sync is skipped (the registry has no channel concept)
- PyPI needs no change: PEP 440 already treats rc versions as pre-releases
that pip only installs with --pre
Stable releases (no dash) are byte-identical in behavior.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
v4's PASSING smoke legs already ran 12-14 minutes against the 15-min
timeout; v5 lost 13 green-path jobs to that wall (GitHub reports job
timeouts as 'cancelled', zero real failures). The unified smoke
legitimately runs more than the pre-unification one — artifact mode,
real download/update phases, a Phase 15 that actually executes.
Budget doctrine (see _soak.yml, violated twice there historically):
the timeout sits above the worst case with headroom.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The version preflight PASSED (both images printed their version) and
then the job died in the cleanup trap: Windows keeps the just-exited
image's file busy for ~100ms (section teardown / first-touch AV), so
'rm -rf' in the EXIT trap raced it — 'Device or resource busy', exit 1
on a job whose actual work was green (observed on windows-11-arm ui in
dry run 30232946513). Temp-dir cleanup on an ephemeral runner is
best-effort with one retry, never a gate.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The manifest content (leg, slice, list hash) is fully determined at
slicing time, so write it up front: a red run's manifest is exactly as
load-bearing for the cross-shard union proof as a green one's.
With that, a missing manifest at upload time can only mean the job
died before the harness started (contract step / build failure) — that
failure is already the job's red, so the upload warns instead of
stacking a second error on top (the recurring 'No files were found
with the provided path: build/c/test-logs/shard-manifest.txt'
annotation). The shard-completeness job still gates the union.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Dry run 30225685743 failed at the ensure-defender step on every
Windows job: the hosted images policy-lock real-time protection OFF
(WinDefend starts, Set-MpPreference accepts, RTP stays off —
untamperable from the job).
Revert to the pre-gate runner behavior: no ensure-defender step in
runner jobs, the release Defender scan is on-demand and fail-soft
again. The local VM preflight KEEPS the fail-closed Defender-ON gate
(same canonical script) — AV-interaction coverage is a deliberate
local superset where the platform allows it.
The venue-parity contract now enforces the INVERSE for runners
(ensure-defender must not gate hosted jobs) and still requires the VM
preflight gate; VM README documents the verified posture delta.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Closes the closable local-vs-remote venue gaps so that, outside arch
legs / YAML glue / release plumbing, every local red is a remote red
and vice versa on shared legs:
- scripts/package-release.sh: THE canonical archive step (names +
five-file Windows bundle layout defined once); _build.yml's eight
inline archive blocks become calls to it.
- scripts/ci/smoke-artifact.sh: the artifact-flow smoke lane — build,
package, extract, then the canonical wrapper in artifact mode; wired
as run.sh smoke-artifact (compose service), win.sh smoke-artifact,
and directly runnable on macOS. Archive-layout bugs now surface
locally instead of in a release dry run.
- glibc-floor leg (Dockerfile.glibc22 + compose + run.sh): portable
binary smokes on ubuntu-22.04/glibc 2.35; the dynamic binary must
refuse there (2.38+ floor by design).
- Defender-ON parity (user directive): scripts/ci/ensure-defender.ps1
enables + VERIFIES real-time protection, fail-closed; runs in every
Windows CI job (_test x2, _soak x3, _smoke, pr.yml) AND in the VM
preflight; _smoke.yml's scan engine-failure soft-skip becomes a red
gate. Expected cost: slower Windows jobs (AV scanning during
install/build/test I/O); the next dry run proves the runner side.
- Contracts: launcher-bundle five-file check retargeted onto
package-release.sh + per-archive canonical-call association;
venue-parity contract requires the new lanes, counts one
ensure-defender step per Windows job, adds --help probes for the new
entries. Extended contract fails on the pre-change tree (verified:
20 violations on HEAD).
- VM README: ephemerality/Defender posture documented — utmctl has no
snapshot verb, so per-run revert stays a manual qcow2 option; the
sweep preflight remains the standing mechanism.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The staged venue-unification bundle: _smoke.yml onto the shared
wrappers with an extracted-artifact input (CBM_SMOKE_ARTIFACT_DIR),
soak-legs.sh as the one canonical soak entry (quick + query-leak legs
across all venues), protected per-user TEMP roots shared across
venues, clean-disk preflights before every VM/Docker run, win.sh and
run.sh routed through the canonical scripts, soak.yml retired, and
the venue-parity contract (whitelist walker: workflows may provision
or call canonical entries, nothing else).
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The windows-11-arm leg's first execution (broad matrix, release dry
run) died before any test ran: the trap-UBSan SANITIZE value passed
unquoted through the workflow ternary, so -fsanitize-trap=undefined
arrived as its own word and make consumed the leading -f as its
makefile flag ('No rule to make target sanitize-trap=undefined').
- The workflow quotes the ternary value so test.sh receives one
argument.
- test.sh forwards make arguments through an ARRAY instead of a
re-split string, so a VAR=VAL whose value contains spaces survives
every boundary (empty-array expansion kept bash-3.2-safe for the
macOS runners under set -u).
Verified: argument-parse proof (argc=1 with the four-flag value) and a
full macOS suite run through the changed plumbing, 6776 passed,
0 failed — identical totals.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Flip shard_suites on for all three callers of the reusable test
workflow. This commit is deliberately the only topology change:
reverting it alone restores the previous single-job-per-leg CI shape
while keeping the (inert without it) sharding machinery and the
cache-warm pipeline.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
PR iteration was bounded by single-job suite runs (~40-50 min critical
path) and by every first-of-ref build starting cold. Same suites, same
sanitizers, same gates — the schedule, the cache reuse, and a new
runtime coverage proof change.
- run-tests-parallel.sh accepts CBM_TEST_SHARD="i/N" and runs a
deterministic slice of --list-suites; unset selects everything, so the
path is inert outside CI. Known-heavy suites are dealt to shards in
weight order first (naive modulo stacked the two slowest, store_arch
and daemon_runtime, onto one job), the parallel wave and the serial
tail are sliced separately so every shard keeps its own quiet tail,
and the per-shard union guard proves each job ran exactly its slice.
- shard-completeness job: per-shard guards cannot see a mis-plumbed
CBM_TEST_SHARD (two jobs running the same slice passes every local
check while a slice runs nowhere). Every leg now uploads a manifest
(leg, i/N, sha of the full suite list, its slice) and one aggregation
job re-proves per leg that the shards agreed on the list, indices are
exactly 1..N, and the union of slices IS the list. Runs on unsharded
topologies too, where each leg's single manifest must cover the list.
- workflow topology (shard_suites input, off by default = byte-identical
to the pre-shard matrix): ubuntu legs run 3 shards, Windows 2 (every
extra Windows shard re-pays ~5 min of MSYS2 setup), macOS stays at 1
(not the critical path; mac runner concurrency ceilings are the
tightest). The Windows test job also pre-creates build\c with a
protected DACL so build products inherit it — workspace drive roots
grant Authenticated Users Modify by inheritance, which the activation
transaction's source-directory policy correctly refuses.
- ccache: a final ref-less restore key lets a fresh PR start from the
newest cache GitHub's scoping permits, and a nightly build-only
cache-warm workflow keeps main-scoped caches at most a day stale —
without it the fallback had no warm source, since nothing built in
main's scope. The strictly-per-ref policy this replaces cost ~8-12
minutes on every first-of-ref build for no safety gain:
CCACHE_COMPILERCHECK=content makes a stale or foreign cache able to
miss but never to return wrong output. Sharded legs restore-all/
save-one (every shard builds the identical objects, so shard 1's
cache carries the full set) — save volume stays flat and warm caches
stop being evicted by per-shard duplicates.
- suite timeouts: daemon_runtime joins the slow tier. It measures ~610s
solo on arm64 under ASan (10-round loop, no hang), so the 900s default
under a loaded 4-job CI runner was a slowness kill masquerading as a
hang detector.
Validated locally: 3-shard union over the real suite list is complete
with no duplicates, one top-heavy suite per shard, and the summed
3-shard totals reproduce the unsharded run exactly (6776 passed,
0 failed, 4 skipped).
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Close the cross-platform sanitizer gaps that were leaving real
concurrency and undefined-behavior bugs uncaught, and fix a data race
the first widened run surfaced.
- tsan: the data-race gate ran three suites (mem, slab_alloc, parallel)
over no real threaded production code. It now covers every threaded
surface that runs clean and stable under TSan: the parallel-extraction
worker pool (parallel, worker_pool, pipeline), the filesystem watcher,
the embedded HTTP server (httpd), diagnostics sampling, the MCP server
and mutation guard, subprocess supervision, and the runnable
daemon-coordination paths (daemon, daemon_application). daemon_runtime
(deadlocks under TSan+fork), daemon_ipc and daemon_frontend
(test-harness synchronization, not production) are excluded with the
reasons recorded in the Makefile.
- tsan: fixed a genuine data race the widened gate immediately found —
cbm_lsp_max_walk_depth's lazy cache was read and written by parallel
LSP-extraction workers without synchronization. A data race is
undefined behavior even when every worker computes the same value, so
the cache slot is now a relaxed atomic: a plain load on the hot path,
and a first-touch double-compute simply stores the same value.
- tsan(ci + local): the test-tsan job now runs on Linux amd64, Linux
arm64, AND native ARM64 macOS (the threading code is shared, so a race
is usually caught on all three, but scheduler differences let each
surface one the others miss). The local ladder gained `run.sh tsan`
and `tsan-amd64` plus the matching compose services. TSan's shadow
memory aborts under modern high-entropy ASLR, so the containers run
under `setarch -R` with an unconfined seccomp profile (the personality
syscall is otherwise blocked) and the CI Linux legs lower
vm.mmap_rnd_bits first; amd64 TSan cannot run under x86_64-on-ARM
translation and is a real-hardware/CI gate only (documented in
run.sh).
- ubsan(win/arm64): native ARM64 Windows had no sanitizer at all —
AddressSanitizer ships no aarch64-w64-windows-gnu runtime. UBSan in
trap mode (-fsanitize-trap=undefined) needs no runtime library, so it
instruments natively and turns undefined behavior into an
illegal-instruction trap; -fstack-protector-strong adds stack-smash
coverage the heap tools miss. The GitHub windows-11-arm leg switches
from unsanitized to this, and vm/win.sh gains trap-ubsan-build /
trap-ubsan-test for local iteration (reproduce under the emulated
x86_64 UBSan to see which check fired). The whole codebase builds and
runs clean under it.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Daemon lifecycle and Windows correctness, verified on a real Windows 11
ARM64 VM through the maintained test-infrastructure/vm drivers, plus the
macOS and Linux arm64 suites and the container lint gate.
Daemon lifecycle:
- daemon start/stop/status subcommands. `daemon start` launches a
PERMANENT daemon (spawn shape is byte-exact argv; survives idle
periods and session ends) and reports an already-active daemon
instead of failing. Permanence is honored at every stop latch:
last-committed-client disconnect, host initial-client window,
coordinator release, and application final-session close — a
permanent daemon also keeps admitting new sessions after its last
one closes.
- daemon stop refuses while sessions are active and lists the blocking
peers (pid/role) that must finish first; an idle daemon drains
through the activation-shutdown machinery with the ACK ordered after
connection interrupts. A second stop is idempotent. The wire ops are
no-cohort first-frame requests with peer fingerprint authentication,
so stop/status never conflict with an exact-build admission gate.
- One-shot CLI commands now execute through the daemon (index workers
keep their local supervised path). A cold CLI run that had to spawn a
temporary daemon prints a hint that `daemon start` removes the
per-command startup tax; a warm daemon is recycled silently.
- Hooks are connect-only fail-open: with no daemon present the hook
emits a visible, rate-limited notice (Claude-dialect systemMessage
plus stderr for other dialects) and always exits 0 — augmentation is
never allowed to block the caller's tool use.
- Version skew: a newer-build client automatically drains an
older-build permanent daemon (strict semantic-version triples only;
dev builds never auto-drain) and the build-conflict message names
`cbm daemon stop` as the manual escape hatch.
Windows IPC/runtime (real-VM verified):
- ipc(win): persistent pending overlapped ConnectNamedPipe. The accept
path used to destroy its listening pipe instance on every 20 ms poll
timeout; a client attaching in the teardown window was severed or left
on an orphaned pipe object whose HELLO no server handle could ever
read, absorbing the connect until the client's own timeout expired.
The pending connect now survives poll timeouts and nothing is
destroyed while a client could be attaching.
- ipc(win): drain-before-close for final responses. Closing a named-pipe
server handle can discard a just-sent response before the peer reads
it (POSIX stream sockets never lose buffered data on close). A bounded
cbm_daemon_ipc_connection_drain (read-until-peer-EOF; no-op on POSIX,
immediate on interrupted connections) now precedes close in
runtime_worker_finish and runtime_reject_inline, so hello-conflict,
capacity and disconnect acknowledgements reliably reach the peer.
- runtime: CLOSE_INTENT wire frame. A Windows named-pipe client has no
transport half-close, so close_begin now announces departure with an
explicit frame (ordered after APPLICATION_CANCEL, before the local
interrupt); the server releases the client's admission on receipt
instead of waiting for the handle to close. Admission-drop timing is
now identical to POSIX shutdown() semantics on every platform.
- runtime(win): client close cancellation. close_begin serializes with
request publication under the send lock, best-effort sends the active
token's APPLICATION_CANCEL frame, then interrupts local I/O; the
server cancels MCP/subprocess work promptly. Contract tests accept
both correct outcomes (interrupted transport or decoded CANCELLED).
- runtime: activation acknowledgement ordering. The activation ACK is
the requester's license to act on "snapshotted and draining", so every
connection interrupt is now initiated before the ACK is sent; a
session could previously get one more request serviced after the
requester observed the ACK.
- service(win): deadline-bounded private-file prepare. The conflict-log
prepare retry loop (100 x Sleep(2), which rounds up to the ~16 ms
timer granularity) burned ~1.6 s against permanently obstructed paths,
stalling hello rejections past the client's timeout. The retry budget
is now a 250 ms deadline; transient share collisions still retry.
- subprocess(win): cmd.exe /C payload encoder quotes metacharacters
correctly (root cause of the git-on-Windows failure cluster).
- watcher: SHA-256 buffer sizing (CBM_SZ_64 -> CBM_SZ_128) and a native
Windows stop/unwatch cancellation test with exact-image verification.
- httpd: send_all writes in bounded 64 KiB slices. A single giant
nonblocking send() on Windows is absorbed wholesale into AFD kernel
buffering regardless of SO_SNDBUF, so send deadlines and interrupts
could never engage against a slow peer (and the full payload was
pinned in nonpaged pool). Slicing restores a deterministic
backpressure point; a test hook pins SO_SNDBUF for the deadline and
interrupt tests.
- ui/http: shutdown lifecycle — interrupt checks, response-wide send
deadline, explicit connection states, refusal to free a server while
a listener-owned connection is active.
Windows long-path support:
- Central path-aware wide conversion (canonicalize via GetFullPathNameW
and prepend the extended-length prefix for absolute paths >=240) at
the compat chokepoints (cbm_fopen/compat_fs/mkstemp/mkdtemp), sqlite
store opens, and the daemon build-fingerprint/log paths. Deep managed
installs (a 64-hex generation directory routinely exceeds MAX_PATH)
now index, stage and activate correctly.
- activation transaction: its own file APIs and the component-walking
ancestry validators now operate in the extended-length namespace;
the launcher path is canonicalized (and prefixed when deep) once at
entry so every downstream exact-string comparison stays
form-consistent.
- Executable self-resolution uses the wide APIs (GetModuleFileNameW,
GetFileAttributesW) so non-ASCII install paths survive argv[0]
resolution.
Windows launcher install/uninstall transaction:
- FileRenameInfoEx names are NUL-terminated in an over-allocated
buffer. FileNameLength governs per the contract, but filter drivers
read FileName as NUL-terminated and appended adjacent heap bytes to
created names — a flaky, garbage-suffixed rename target. Both the CLI
and the launcher rename helpers are fixed.
- Uninstall retires state via rename-aside (.cbm ->
.cbm-retired-v1-<tag>-<pid>) with the retired tag shortened to 16 hex
chars so the bare rename target stays under the FileRenameInfoEx
NT-conversion ceiling at guard depths; 64 bits still uniquely
identify the generation.
- When the running launcher's mapped generation backings pin .cbm
against rename, the backings are relocated to activation-<pid>-N
.retired tombstones beside the install (a mapped image may be renamed,
never deleted; the launcher's liveness-guarded sweep reclaims stale
tombstones). Every relocation is recorded, and a FAILED uninstall
reverses the moves after restoring .cbm — via MoveFileExW with
extended-length paths on both arguments, since the deep generation
target is beyond the handle-based rename's bare-path reach — so a
restored install keeps its generation backings and stays runnable.
- After a committed uninstall the retired tree's backings are relocated
out so the tree is shallow enough for the detached cleanup's rd, and
the cleanup's working directory strips the extended-length prefix
(CreateProcessW lpCurrentDirectory silently ignores prefixed paths).
- Files created under Administrators-default-owner directories
(CopyFileW destinations, CREATE_NEW tombstones, probe directories)
are explicitly owner-stamped so the exact-owner validators hold on
runner images; guard fixtures stamp hand-built trees the same way.
Diagnostics, tests and infra:
- diagnostics: discovery is now an always-delivered JSON control record
(new cbm_log_control) that survives CBM_LOG_LEVEL suppression and
paths containing spaces; placement honors $TMPDIR with /tmp fallback
via a diagnostics-local helper; the soak parser reads the JSON record;
documented in docs/CONFIGURATION.md. Red-first coverage for suppressed
log levels, TMPDIR-with-spaces, and native Windows output-contract
assertions.
- tests(win): daemon_ipc/daemon_frontend fixtures now build endpoint
parents with production-shaped ancestry (LocalAppData on Windows, via
th_secure_runtime_parent_new) — the runtime ancestry validation
correctly refuses temp roots whose ancestors grant mutation rights to
Authenticated Users (C:/msys64/tmp, GitHub-runner work dirs) — and
drive the documented startup-owner publication flow before reading
generation-bound endpoint addresses. This turns the 26 Windows
failures previously visible in CI's full-test job green without
weakening any validation.
- tests(win): the launcher guard covers the full permanent-launcher
contract including failed-uninstall restore and immediate reinstall
after uninstall; new daemon lifecycle and reworked hook-augment
guards run the start/recycle/stop flow end to end.
- tests: CBM_SKIP_PERF is now actually consumed by the test runner
(it was set by CI but never read, so perf suites ran everywhere);
four throughput/bench suites are classified as perf, the heavy
store_arch suite moved to the slow-timeout tier, and two
wall-clock-sensitive assertions were rewritten as invariant checks
with coarse hang-detector backstops.
- build/test infra: build-dir safety contract, UI dev-proxy security
contract, soak daemon-recovery contract, path-safety helper, the
Windows VM worktree-sync contract wired into scripts/test.sh, and
vm/win.sh guards building its clean embedded-UI product in an
isolated BUILD_DIR so it cannot clobber the incremental test build.
provision-windows.sh now installs Node.js for the guards UI build.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Combines the ccache steps with this branch's Windows hardening: the
protected temp root and profile staging keep their roles, the cache
steps slot alongside them, and the strictly per-ref keys apply
unchanged. The daemon branch's own CI legs warm from the second push
on.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
ccache with CCACHE_COMPILERCHECK=content everywhere: every cache entry
is keyed on the CONTENT of the compiler binary plus the fully
preprocessed translation unit, so a hit is provably the identical
compilation - a stale, foreign, or corrupted cache can only MISS, never
return wrong output. No CCACHE_BASEDIR and no path rewriting: debug
info and sanitizer report paths stay exact.
Locally scripts/env.sh routes compilers through ccache's masquerade
directories when present (opt-out CBM_NO_CCACHE=1) - $CC keeps its
plain name, so verify_compiler, make, and link lines are untouched.
CI caches are strictly per-ref by policy on top of GitHub's own branch
scoping: keys embed github.ref, so no base-branch fallback - a new PR
builds cold once and only its own pushes warm it. Cached: the four
test jobs and the pr-smoke matrix. Release builds in _build.yml stay
deliberately uncached.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Completes the handoff task for the Windows harness: the native test job
ran with the MSYS-shared /tmp and the runner's inherited
LocalAppData\Temp ACLs, which the daemon's strict private-filesystem
checks reject by design. The test-windows job now creates a per-user
root under the profile with an owner-stamped, protected current-SID
DACL and routes TEMP/TMP (native form) and TMPDIR (POSIX form) through
it before scripts/test.sh runs.
The guards script already hardened its root DACL but ownership is never
inherited on Windows: bundle copies created by the admin-group runner
token can come out Administrators-owned, and the launcher's exe policy
requires the exact current user as owner. Stamp the current SID on the
staged bundle dir and both executables explicitly.
Job topology, gates, and triggers are unchanged; the new step is a
sub-second pwsh block in the existing test-windows job.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
github/issue-labeler compiles patterns with JavaScript RegExp, which
rejects PCRE-only inline (?i) groups with SyntaxError: Invalid group, so
the workflow failed before evaluating any rule and no issue ever got an
area label (#764).
Replace the third-party action with first-party actions/github-script:
the same config rules are compiled with the i flag applied centrally
(case-insensitivity preserved), labels stay additive-only via addLabels,
and a pattern that fails to compile now fails the run loudly instead of
silently no-oping. Strip (?i) from all 7 config patterns and document
the line format the workflow parses.
Verified locally: all 7 rules compile and match expected labels across
sample issue texts; the old pattern reproduces the SyntaxError.
Closes#764
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The Defender step treated any non-zero MpCmdRun exit as 'flagged binary!'
and failed the job. But MpCmdRun -Scan returns 2 for a real detection and
0 for clean; any other non-zero means the scan engine itself could not run
(e.g. hr=0x800106ba: the Defender antimalware service is unavailable on the
runner), which is a transient runner-side flake, not a detection. This
false-failed the v0.9.0 release on its first attempt. Only exit 2 now hard-
blocks; an engine failure emits a warning and continues.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The persistent windows-11-arm smoke Phase 12a failure was a 404, not a network
error: it requested codebase-memory-mcp-windows-amd64.zip on the arm64 leg. Cause:
DL_ARCH came from `uname -m`, which on windows-11-arm is an emulated x86_64 MSYS2
uname reporting "x86_64" -> wrong (amd64) archive -> 404 (server has arm64). Phase
14 worked because the binary's own detect_arch() is native. Prefer SMOKE_ARCH
(passed from the smoke workflow's matrix.arch) over uname; fall back to uname for
local runs. This is the real cause the earlier curl/proxy/ipv4 attempts masked --
the 404 was swallowed by 2>/dev/null until #905 surfaced it.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Root cause of the windows-11-arm smoke Phase 12a failure: the smoke `curl` was
curl-8.20.0 from the base MSYS2 (msys) repo, which on an ARM64 host runs as an
x86_64 binary under emulation, and its networking could not reach the local
http.server -- while every NATIVE ARM64 process did (python's http.server bound
and served, and the product binary's own downloader completed Phase 14's update
from the same 127.0.0.1:18080). Install the native mingw-w64-clang-<arch>-curl so
the smoke curl is native (aarch64 on windows-11-arm), matching python and the
product binary. It takes PATH precedence over the base msys curl in the CLANG*
environments. amd64 gets the native x86_64 curl (already worked, now explicit).
Last red legs in the release dry run.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
curl (already installed) failed instantly in smoke Phase 12a on windows-11-arm
while the server was up and the binary's own downloader reached it -- the classic
localhost->::1 (IPv6) resolution: msys2 curl tried IPv6 but python's http.server
was IPv4-only. Bind the server to 127.0.0.1 and use a 127.0.0.1 SMOKE_DOWNLOAD_URL
so both ends are explicit IPv4. amd64 windows is unaffected (already reached via
IPv4). These were the last red legs in the release dry run.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The smoke-windows job's msys2 install list omitted curl, relying on an ambient
curl that resolves on CLANG64 (amd64) but not on CLANGARM64 (windows-11-arm).
Phase 12a's `curl` download failed there even though the server was up -- the
binary's own downloader (Phase 14) succeeded against the same server. Install
curl explicitly so both arches use the msys2 curl in the same network namespace
as the local http.server. These were the last two red legs in the release dry
run (everything else -- test, build incl build-windows-arm64, all smoke, and
soak incl soak-quick-windows-arm64 -- is green).
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Two release-blockers surfaced once #882 made every smoke leg a required gate
(continue-on-error previously masked them):
- smoke-unix ran the *dynamic* linux binary on ubuntu-22.04 / 22.04-arm, where
it cannot start (it links glibc 2.38+) -> Phase 1 died. Older-glibc coverage
is the -portable (static) binary's job and is green via the smoke-linux-portable
broad legs, so drop the impossible dynamic broad legs (keep macOS).
- the ClamAV scan died on every linux leg: apt auto-starts the clamav-freshclam
daemon which holds the freshclam log lock, so the manual freshclam failed with
"Failed to lock the log file ... Resource temporarily unavailable". Stop the
daemon first.
Also add native ARM64 Windows as a first-class release target -- the one platform
we test (windows-11-arm) but never shipped:
- build-windows-arm64 job (CLANGARM64) producing codebase-memory-mcp[-ui]-windows-arm64.zip
- smoke-windows is now arch-aware: windows-11-arm smokes the NATIVE arm64 binary
instead of the x86_64 binary under emulation
- a native soak-quick-windows-arm64 leg (no sanitizer -- ASan is unavailable on
native ARM64 Windows)
- arm64 detection in install.ps1
release.yml already publishes windows-arm64 (asset filter + merge-multiple) and
build_update_url/detect_arch already resolve it; npm/pip/install.sh compute arm64
already. scoop/winget/chocolatey per-version manifests are updated at release time.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The dry-run smoke Phase 14 runs the binary's real `update` command, which on
linux downloads the fully-static "-portable" asset (build_update_url in
src/cli/cli.c appends -portable on linux; _build.yml's build-linux-portable job
ships codebase-memory-mcp[-ui]-linux-<arch>-portable.tar.gz). The smoke artifact
server only served the non-portable name, so `update` 404'd and the config-
refresh assertion failed on every linux smoke leg. The mismatch (the smoke half
of the -portable delivery change) was masked until every smoke leg became a
required gate.
Mirror the tarball under the -portable name on linux (standard and ui) so the
served names match build_update_url; checksums.txt globs *.tar.gz and picks up
the new files. Verified the fixed logic serves the exact name the binary
requests for both variants; the gating smoke Phase 14 is the regression guard.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Closes#423, #20. A non-ASCII repo path passed to
`codebase-memory-mcp cli index_repository "<json>"` was mangled on Windows, so the
command failed with "repo_path is required" instead of indexing the real directory.
POSIX is unaffected (argv is already UTF-8). Two boundaries had to be fixed:
1. argv read: main() took only the narrow `int main(int argc, char **argv)`, so the
CRT handed it argv in the active ANSI code page. On Windows main() now rebuilds
argv from the wide command line (GetCommandLineW + CommandLineToArgvW) and
converts each element to UTF-8 (cbm_wide_to_utf8).
2. worker spawn: the index supervisor (and the UI index thread) spawned the worker
via CreateProcessA, which re-interpreted the UTF-8 command line through the ANSI
code page -- re-mangling the path at the parent->worker boundary even after (1).
Both spawns now build a wide command line (cbm_utf8_to_wide) and use CreateProcessW.
Without (2) the default (supervisor-enabled) path stayed broken while only the
in-process path worked, so this is the complete end-to-end fix, not just argv.
Promotes tests/windows/test_cli_non_ascii_arg.py from an opt-in known-red to a green
Windows guard (scripts/test-windows.ps1 + the test-windows-guards CI job). The guard
now exercises the DEFAULT supervised path (it drops the suite's CBM_INDEX_SUPERVISOR=0
so it actually crosses the worker-spawn boundary). Verified on native Windows: RED
before the fix, GREEN after (a non-ASCII repo indexes, nodes>0). Also adds -lshell32
explicitly (main.c now uses CommandLineToArgvW) and refreshes RED_TEST_ANALYSIS.md.
Signed-off-by: Flipper <jacobphilipp@ymail.com>
PR CI ran tests but never built the binary or smoke-tested it, so built-binary
regressions only surfaced at the release dry run — e.g. the Windows
CreateProcess argv-quoting bug (worker got mangled JSON) slipped all the way to
the release gate.
Add a path-gated light smoke: on PRs that touch product code (src/, internal/,
Makefile.cbm, build.sh, smoke-test.sh, env.sh), build the PRODUCTION binary and
run scripts/smoke-test.sh on ubuntu-latest + macos-14 + windows-latest (native,
reliable — not the flaky broad/emulated legs). Every leg gates. Docs/CI/test PRs
skip it and stay fast (gh pr diff --name-only filter). The full broad/emulated
smoke stays in the release dry run.
SMOKE_DOWNLOAD_URL is intentionally unset so smoke-test.sh's download/update
phases self-skip; the core index/search/trace phases (which catch the
CreateProcess class) still run. ci-ok now treats skipped as OK so PRs that skip
smoke still pass.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Two defects let the release dry run report green despite real smoke failures:
1. Matrix collapse — the smoke matrices used {variant:[standard,ui],
include:$OS_LIST}. GitHub's include-merge overwrites the variant combos
last-wins with each OS entry, so only the LAST OS ran (both variants) and the
earlier ones were silently dropped. In broad mode windows smoke only ran
windows-11-arm (emulated x86_64), never windows-latest (native); unix only
macos-15; portable only ubuntu-22.04-arm. Rebuild each matrix as an explicit
os x variant cartesian so every leg actually runs.
2. Optional escape hatch — the broad legs carried optional:true +
continue-on-error: matrix.optional, so a failing leg didn't fail _smoke.yml
and didn't gate the dry run or a release. Removed all optional flags + the
continue-on-error lines; every product-smoke leg now gates.
smoke-packages stays continue-on-error (external Glama directory image, not the
shipped product). fail-fast:false kept so all legs still run for a full picture.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The release dry run must be all-green on every platform — drop the
optional/continue-on-error flags from the broad matrix legs (ubuntu-22.04,
ubuntu-22.04-arm, macos-15, windows-2025) so a failure on any of them gates
the run. Only affects broad_platforms=true (the dry run); the core PR-CI
matrix is unchanged.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
libgit2 is licensed GPLv2-with-linking-exception, and this project is
deliberately GPL-free (see scripts/license-policy.json). libgit2 was only
an OPTIONAL, faster git-history code path guarded by HAVE_LIBGIT2 and
auto-detected via pkg-config; it always shipped with a popen("git log ...")
fallback that release binaries already used (they were built without
libgit2). Making that popen fallback the SOLE git-history implementation
means there is NO change to shipped behavior.
Changes:
- pass_githistory.c: collapse the HAVE_LIBGIT2 #ifdef so only the popen
parse_git_log remains; drop the now-unused <git2.h>/<time.h> includes.
- cbm.c / cbm.h / main.c / subprocess.c / index_supervisor.h: remove the
libgit2 mimalloc allocator bind and its >=1.7.0 version guards; keep the
tree-sitter + sqlite3 binds; correct the comments.
- Makefile.cbm: remove the optional-libgit2 pkg-config detection block,
the REQUIRE_LIBGIT2 error gate, and all LIBGIT2 CFLAGS/LIBS/FLAGS.
- .github/workflows/_test.yml: drop the REQUIRE_LIBGIT2 test-matrix leg
plus its libgit2-dev/pkg-config apt install (lower CI cost, no new
gating, no trigger change).
- flake.nix: drop libgit2/pkg-config from the dev shell (now-dead dep).
HAVE_LIBGIT2 no longer exists anywhere in the tree.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Martin Vogel