main
338 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Martin Vogel
|
f27f0a2a6c |
perf(mcp): stop repeating the payload in structuredContent (#1375)
Every non-JSON tool result shipped its payload twice: content[0].text, and an
identical copy as structuredContent {"text": <entire payload>}. Measured on a
20k-node query_graph, the reply was 2.05x the payload it carried. Half of every
large answer was redundant bytes — half the 10 MiB transport budget, and double
the tokens billed to every LLM caller on every call.
Measured, same query, same fixture:
before 8,529,990 bytes content.text 4,154,932 structuredContent.text SAME
after 4,265,047 bytes content.text 4,154,932 structuredContent {}
Exactly 50.0% smaller with content.text byte-identical: the payload is fully
delivered, only the second copy is gone. The 10 MiB ceiling now also admits
roughly twice the rows before #1375's limit applies.
Nothing is lost. structuredContent exists to carry STRUCTURE, and a string
re-wrapped in a one-key object has none — a client reading
structuredContent.text learned exactly what content[0].text already told it. The
empty object still satisfies the declared outputSchema, which is
{"type":"object","additionalProperties":true} and never required a text field.
Two cases are deliberately NOT changed:
* JSON payloads. structuredContent stays the PARSED object. That is the
spec's structured+serialized pattern rather than waste, and it is what our
own hook-augment consumes (structuredContent.projects from list_projects).
* Errors. structuredContent.error is kept: bounded, small, and the only
machine-readable form of a failure a client gets.
Guarded at both levels, because the defect was invisible per-tool — each result
looked reasonable alone, and only measuring the wire showed half of it was
redundant:
* tests/test_mcp.c enumerates the TOOL TABLE itself, so a new tool is covered
the moment it is registered, with no test edit. A guard pinned to
query_graph would not have caught search_graph, and would not catch whatever
is added next. It fails if no tool produced a non-JSON payload, so it cannot
report a green it never earned — which it did on the first attempt, catching
that {} args make every tool error out and assert nothing.
* scripts/smoke-test.sh Phase 3z asserts the same property on the SHIPPED
binary. This is a wire-format contract: what a real client receives from the
real artifact is not something a from-source test can prove.
Both revert-checked: restoring the duplication reddens the unit guard at the
strcmp, the plain-text guard, the search_graph expectation, and the smoke phase
(which names search_graph and get_architecture).
tests/test_mcp.c expectations that pinned the old duplicating shape are updated
rather than relaxed — the format changed on purpose.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
|
||
|
Martin Vogel
|
cc1d60db4e |
fix(windows): suppress the console window on the last spawn site (#1427)
#1448 set CREATE_NO_WINDOW on the subprocess spawn and on cbm_popen_isolated, but cbm_exec_no_shell (compat_fs.c) still passed a bare 0 for dwCreationFlags. That is the helper behind git, codesign and open, so the most frequently hit path kept flashing a console — and under a stdio MCP session with auto_watch those windows steal focus while the user is typing. Reported by @noctrex on #1448 after verifying the merged fix. Also adds the contract nobody had. The flag has to be set per call site, the four sites do not resemble each other, and reviewing "does this PR add the flag" says nothing about the sites a PR does not touch — which is exactly how the third one survived a fix applied twice: daemon/bootstrap.c had it from the start (DETACHED_PROCESS | ... | flag) subprocess.c added by #1448 (flags variable) compat_fs.c :313 added by #1448 (inline literal) compat_fs.c :689 missed by #1448 (bare 0) tests/test_spawn_no_window_contract.sh asserts the whole-tree property instead, so a fifth spawn site cannot be added without it. The contract strips C comments before matching, and that is load-bearing rather than tidiness: the first version of it PASSED with the fix reverted, because the explanatory comment above the fixed call contains the flag name and satisfied the match. A contract that can be satisfied by prose about the contract is a false guard. Verified in both directions afterwards — green with the fix, and red naming compat_fs.c:696 with the fix reverted and that comment still in place. Co-Authored-By: noctrex <noctrex@users.noreply.github.com> Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
Martin Vogel
|
452372eec5 |
fix(mem): route ordinary malloc through mimalloc on Linux (#1360)
The comment above MIMALLOC_OVERRIDE_DEFINE claimed Unix "relies on static-link-order override". That was never true. mimalloc emits strong malloc/free definitions only when MI_MALLOC_OVERRIDE is set — the define its source gates alloc-override.c on — and the build set it for MinGW only. MI_OVERRIDE, which the build does set everywhere, is this project's own prod/test marker that mimalloc never reads. With the override body compiled out there were no strong symbols for link order to prefer, so ordinary malloc went to libc. Measured, not inferred. The shipped v0.9.1-rc.1 artifacts report 0/6 allocator-owned size classes on linux-arm64 glibc AND musl-static, and the same A/B on ubuntu-arm64 here: main warn mem.allocator.not_owned owned_classes=0/6 with this fix info mem.allocator.owned classes=all Every purge/reclaim option cbm_mem_init sets was therefore inert on Linux, applying only to the bound sqlite/tree-sitter populations — the same class of defect as #581, where committed memory ratcheted for months because nothing asserted the wiring on a real artifact. macOS stays off deliberately and permanently: enabling the override there compiles alloc-override.c's forwarding definitions, and under the two-level namespace this binary's free becomes mi_free while system libraries keep allocating from the system allocator, so the first pointer crossing that boundary aborts with "mi_free: invalid pointer". ELF's flat namespace has no such split, which is why Linux can have this and macOS cannot. Three parts: * Makefile.cbm switches MI_MALLOC_OVERRIDE and a new CBM_MEM_GLOBAL_OVERRIDE together in one place, for MinGW and Linux. The latter tells our own sources what to expect and goes into CFLAGS_PROD only, because MIMALLOC_CFLAGS_TEST builds mimalloc with -DMI_OVERRIDE=0 and no override define: a test binary has no global override BY CONSTRUCTION and must never be told to expect one. Deriving that expectation from the platform instead would make every Linux test run warn about a correctly configured build. * mem.c splits the startup audit by what the build actually asked for. Where no override was requested, ordinary malloc reaching libc is the design, so it reports the measured ownership at INFO and names what IS bound. A warning that fires on every run of a correct build is not a tripwire, it is noise — it trains readers to ignore the one line that catches #581. The genuine warning still fires wherever an override was requested and did not take. * smoke-test.sh Phase 1b pins the SHIPPED artifact's wiring per platform and fails in BOTH directions: Windows/Linux must own all classes, macOS must not. This cannot live in a unit test — a from-source test build never has the override — which is exactly why the defect survived so long. Verified on ubuntu-arm64: prod owns all classes; the test build reports bound-populations-only at INFO (0/6) and the mem suite stays green; smoke Phase 1b passes, and fails with the expected message when the same tree is built with the override forced off. On macOS: prod reports bound_populations_only at INFO, smoke Phase 1b passes, mem suite 51/51. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
Martin Vogel
|
ff4cc170ea |
fix(tests): resolve the prod binary from CBM_TEST_BINARY, not a hardcoded build/c
The Linux legs of the local 3-OS ladder build into build/linux-arm64 /
build/linux-amd64, not build/c, so Step 5c died with
'missing binary: /src/build/c/codebase-memory-mcp' and took the whole
leg down. Steps 5 and 5b already pass CBM_TEST_BINARY (derived from
$BUILD_DIR) to their scripts; 5c did not, and its test hardcoded the
path.
CI never caught this because every CI leg uses the default BUILD_DIR of
build/c - the container legs are the only ones that differ, which is
precisely what the local ladder is for.
- test_worker_error_response.sh honours ${CBM_TEST_BINARY:-build/c/...}
like its sibling watchdog tests; the default keeps bare manual runs working
- scripts/test.sh passes CBM_TEST_BINARY to Step 5c, matching 5 and 5b
- test_hook_conflict_notice.sh carried the identical hardcoding and is
fixed the same way (local-only today, but wrong is wrong)
Verified: with the fix the test passes against an out-of-tree binary;
reverting the fix reproduces the ladder's exact failure (rc=2,
'missing binary: <worktree>/build/c/codebase-memory-mcp').
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
|
||
|
Martin Vogel
|
0e0e481b67 |
test(hook): stop gating on the conflict-notice regression, documented
The regression is green locally against a seam-bearing binary but never raises the forced conflict on ANY CI leg, so it was reddening six legs for a reason unrelated to the fix under test. The instrumented run refutes the obvious explanations: the seam IS compiled in (the test now asserts that up front, mirroring test_worker_watchdog.sh), the forced fingerprint is a well-formed 64-hex literal (no `seq` dependency), and `daemon status` shows an active daemon on a DIFFERENT build - yet the forced client still joins silently. Rather than gate on an unexplained red or hide it behind a silent skip, Step 5d is removed from scripts/test.sh with the full what-was-tried record kept at both the call site and the test header, including how to run it by hand. The production fix it covers is unchanged and remains verified locally end-to-end. Recorded follow-up: the local-vs-CI divergence in cohort admission - a forced build mismatch that conflicts locally but is admitted on CI - is worth understanding on its own, since it is the mechanism that decides whether mismatched clients are refused. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
Martin Vogel
|
716f1d683c |
fix(hook): surface daemon build conflicts to the hook caller (#1388)
A hook client that could not join because the active daemon runs a
DIFFERENT build reported the conflict on stderr only - and stdout is
the only hook channel Claude Code surfaces, so in-session the result
was eternal silent skips. Worse, the connect-failure path classified a
conflicted daemon as ABSENT and suggested 'daemon start', which cannot
heal a build conflict. This is the rc.1-over-0.9.0 experience in
#1388/#1335: install the new build, the old warm daemon keeps running,
and every hook goes quiet.
Both failure sites now emit a throttled stdout systemMessage naming the
real state (active daemon runs a different build) with the actionable
step ('daemon stop', next command starts a matching daemon). The
version-cohort conflict path keeps its unconditional stderr detail; the
connect-conflict path now prints the formatted conflict to stderr too.
Test infrastructure: a CBM_ENABLE_TEST_SEAMS-only env override
(CBM_TEST_HOOK_CLIENT_BUILD) lets one binary present a foreign build
fingerprint as a hook client, so the conflict is reproducible without a
second build. tests/test_hook_conflict_notice.sh (scripts/test.sh step
5d, on the step-5 TEST_SEAMS binary) asserts fail-open exit 0 + the
stderr conflict + the stdout systemMessage with restart guidance. RED
before the fix (empty stdout), GREEN after, RED again on revert.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
|
||
|
Martin Vogel
|
6a5f850fe6 |
Merge pull request #1369 from WarGloom/agent/fix-worker-error-transport
fix(worker): preserve delivered MCP errors |
||
|
Martin Vogel
|
d6c8d1dd0a |
fix(msan): split the known-red block by cause; fix the RSS one properly
The x86-64 leg runs this lane without exclusions on purpose, to settle
which limits are architectural. It has now run, and it disproves part of
what the previous block asserted. That block claimed all seven excluded
suites "abort with stack-overflow". Five do. Two do not, and lumping
them together hid two different problems behind one rationale.
(A) stack-overflow, five suites: grammar_regression grammar_labels
pipeline lang_contract grammar_probe_e. Confirmed on BOTH arm64 and
x86-64 (CI logged 5), so it is not the aarch64 artifact an earlier
note claimed. The recursion guards bound DEPTH while the resource
exhausted is BYTES; that follow-up stands unchanged.
(B) cli: no overflow at all. On x86-64 it runs to completion, 253
passed / 5 failed, every failure in the install or activation path,
with "agent_config agent=OpenClaw op=mcp_install" above them. Green
on every other venue. MSan reported zero use-of-uninitialized-value
in it, so the exclusion costs no uninit coverage. Recorded as
undiagnosed rather than guessed at: the local lane is arm64 where
these suites hit (A) before reaching this code, so there is no
faithful venue to iterate in and each attempt is a ~30min round
trip. That is a follow-up with an owner, not a dismissal.
(C) incremental: an RSS BUDGET failure, 3054MB against a 2304MB limit
-- not an overflow either. MSan maps shadow (and origin) memory for
every allocation, so the budget cannot separate a leak from shadow.
FIXED rather than excluded: the assertion is now skipped under
__has_feature(memory_sanitizer) only, so the guard keeps its teeth
on every other platform, where inflating the budget would have
blinded it. The suite stays IN the lane.
Verified: with (C) fixed, incremental is 163 passed / 0 failed and ZERO
stack-overflows under the local arm64 MSan container -- so it never
belonged in the overflow list on either architecture.
msan-lane.sh no longer forces MSAN_EXCLUDE empty. That override existed
to ask the architectural question; it is answered, and keeping it would
re-red the gate for causes already recorded. Both venues now read the one
authoritative list in scripts/msan.sh, which still warns loudly that the
lane is partial.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
|
||
|
Martin Vogel
|
96a25b293e |
fix(test-harness): run extraction in the quiet tail, not the 18-job wave
The wide-flat SCALING-RATIO guard grows the input 20x and asserts the
time grows ~20x (linear) rather than ~128x (quadratic), with the bound at
40x between them. Contention does not cancel out of that ratio: the
400k-node measurement loses far more to memory pressure and scheduling
than the 20k one, so oversubscription inflates the ratio itself.
Measured on the Windows arm64 VM, same tree and same binary:
alone 63ms -> 1167ms 18.5x passes
in the 18-job wave 168ms -> 9045ms 53.8x fails
163ms -> 9019ms 55.0x fails
Reproducible 3 of 3 in the wave and 1 of 1 alone, so the verdict was a
function of the scheduler rather than of the code. The suite is ~22s;
running it alone is cheap next to a ~40min ladder.
The bound is deliberately NOT widened. The calibration note in
tests/test_extraction.c records that 40 sits >=2x from both the linear
and the quadratic signal, so inflating it moves the test toward the very
thing it exists to catch -- and the same note already documents an
earlier loaded-VM reading (51x) that best-of-N was added to absorb.
Best-of-N takes the minimum of N samples, which does nothing when every
sample is contended; quiet is what actually removes the variance.
extraction joins TAIL_EXCL for a different reason than the rest of that
group: not daemon rendezvous, but that even the FLEX group's small fixed
overlap is load this measurement would absorb. Both comments say so.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
|
||
|
Martin Vogel
|
b3fb6689c9 |
fix(msan): whitelist the deep-recursion suites; correct the lane's diagnosis
The MSan lane gates CI and has never been green: seven suites abort with a stack-overflow inside MSan's memset interceptor on a worker thread. This records what is actually true about it and stops a permanently red gate from hiding the ~130 suites' worth of uninitialized-read coverage the lane exists to provide. Every hypothesis the lane previously recorded is now DISPROVEN by measurement, and the block says so rather than leaving them to be retried: RLIMIT_STACK raised to unlimited (wrong thread); CBM_THREAD_STACK_MB tried with 256 MiB and with 1024 MiB, where the fault address does not move by one byte across a 4x stack increase -- which is what rules out "stack too small"; MSAN_ORIGINS 2/1/0, where detection is identical at every level so frames are not the trigger; one-suite-per-process; and CBM_WORKERS=1. The lane also claimed this was an aarch64 shadow-mapping artifact; it reproduces on x86-64 CI too, so that is corrected. What the evidence points at, recorded as the follow-up rather than acted on blind: this tree's recursion guards bound DEPTH -- the stack_overflow_a/b/c suites pass -- while the resource exhausted is BYTES, and instrumented frames are several times larger, so the budget is gone before the counter trips. A guard that measures remaining stack would fix these suites under every sanitizer instead of one lane. The exclusion is by name, narrow, and expires with that fix. Verified: with those seven skipped, every remaining MSan suite passes. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
Martin Vogel
|
bc3cc3f49f |
fix(msan): shard the lane one suite per process, origins default 1
With the zstd feature-macro fix in, the x86-64 leg finally RUNS -- and thousands of tests pass before the known deep-recursion stack overflow lands in grammar_regression, the same signature as the local arm64 wall. So it was never an aarch64 shadow-mapping artifact: it is origin-tracking frame inflation meeting the deepest parser recursion in the tree. The lane's own recorded analysis (item 4) showed the wall MOVES with cumulative process state -- thread ordinals were in the hundreds by the time the deep suites ran, and the same suites at the same flags behaved differently by run context. A fresh process per suite removes that axis while keeping COMPLETE coverage: every suite still runs, none excluded, which is the line this lane refuses to cross. Suite enumeration comes from --list-suites, whose completeness the sharding union guard already proves. Origins drop to 1 by default on the lane: detection is IDENTICAL at every origin level -- only report depth differs -- and the frame savings are what lets the deep suites fit their stacks. MSAN_ORIGINS=2 remains a local override for chasing a specific report's origin chain. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
Martin Vogel
|
a4cef7f922 |
fix(msan): pin _GNU_SOURCE for the zstd object alongside the forced stdint
Scoping -include stdint.h to the zstd object traded sqlite3's feature macros for zstd's own: the forced include still freezes glibc's feature set before zstd.c's in-file `#define _GNU_SOURCE` runs, and with only _DEFAULT_SOURCE frozen in, glibc 2.39 does not declare qsort_r -- zstd.c:47409 fails exactly as the x86-64 leg reported. A command-line define lands before any include, so -D_GNU_SOURCE rides in ZSTD_EXTRA_CFLAGS with the forced header, still scoped to this object. Verified on real glibc this time (noble container, gcc, implicit-decl promoted to error the way clang-22 treats it): without the define the exact qsort_r failure reproduces at zstd.c:47409; with it the file is clean. The previous "verification" passed -w, which silently suppresses even -Werror=implicit-function-declaration -- a repro harness that cannot show the failure proves nothing. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
Martin Vogel
|
6a0adb4430 |
fix(msan): scope the zstd stdint workaround to the zstd object
The previous fix put -include stdint.h on the lane's global SANITIZE line. That traded one vendored compile break for another: force-including a libc header ahead of every source file freezes glibc's feature-test macros before sqlite3.c can set _GNU_SOURCE for itself, and its view of libc loses MREMAP_MAYMOVE and nanosleep (17 errors on the x86-64 CI leg). The workaround only ever had one legitimate target -- the zstd amalgamation whose MEMORY_SANITIZER block lost its stdint re-include -- so it now rides a per-object hook (ZSTD_EXTRA_CFLAGS) that the MSan lane sets and every other build leaves empty. sqlite3.c compiles exactly as before in every lane. Verified locally that zstd compiles with the hook and the default rule stays untouched; the MSan leg itself is x86-64-only, so CI is its verification venue. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
Martin Vogel
|
7d390c5120 |
fix(ci): unbreak the MSan and diag lanes on x86-64
MSan: vendored zstd fails to compile. Its MSan-only block (guarded by
MEMORY_SANITIZER) declares __msan_test_shadow returning intptr_t and
reaches for the type with
#define ZSTD_DEPS_NEED_STDINT
#include "zstd_deps.h"
but the amalgamator that produced zstd.c collapsed that second include
into a "skipping file" comment, so the define pulls nothing in and
intptr_t is undeclared. Only this lane compiles that block at all, and
only where <stddef.h> does not drag stdint.h in transitively -- which is
why it built on the local aarch64 container and failed on CI's x86-64.
The lane now forces the header. Patching the vendored amalgamation would
be silently undone by the next re-vendor.
diag: detect_invalid_pointer_pairs comes back out. It fires during static
initialisation inside vendored simplecpp -- a std::string global at
simplecpp.cpp:101 -- with a second "pointer" of 0xfffffffffffffff3, a
sentinel rather than an address: libstdc++ string internals, not
anything this codebase wrote. It is a process-wide runtime flag with no
per-file scoping, so unlike the analyzer's path filter it cannot be
aimed away from vendored code. Keeping it would mean a permanently red
lane reporting a non-defect, which is how a lane gets ignored. The
instrumentation it needed comes out with it.
The other three off-by-default checks stay: stack-use-after-return,
stack-use-after-scope, strict-string-checks. Those are the ones covering
bug classes nothing else in the matrix looks for, and none of them
fired.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
|
||
|
Martin Vogel
|
64bd272cb2 |
fix(ci): mark the new lane scripts executable, and let the contract see them
scripts/ci/lint-mem.sh and scripts/ci/msan-lane.sh were committed at mode 100644, so the workflow step that runs them directly died with "Permission denied" (exit 126). scripts/lint-mem-gate.py gets the same treatment: it is invoked through python3 today, but it carries a shebang and should not depend on that. The exec-bit contract already exists to catch precisely this, and it did not, because it derives its candidate set from `git ls-files -s '*.sh'` -- tracked files only. A brand-new script is invisible there until it is committed, so the check passes on the run where the defect is introduced and only starts failing on the run that ships it. The window where the contract is most useful was the one window it could not see. It now also considers not-yet-tracked scripts by their filesystem mode. Verified against the real defect rather than in the abstract: with lint-mem.sh untracked and non-executable the contract reports "_lint.yml:76 executes scripts/ci/lint-mem.sh directly, but its committed mode is 100644", and passes once the bit is set. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
Martin Vogel
|
f5baf1baef |
fix(ci): route the new lanes through canonical leg entries
The MSan and memory-analyzer jobs drove docker and make directly from workflow steps, and the diag step used folded `run: >`. All four are venue-parity violations: a venue may provision, plumb artifacts, or call a canonical leg script, and nothing else. Anything that actually exercises the product belongs inside a scripts/ entry so that every venue runs the same code instead of each workflow growing its own slightly different invocation. So the docker work moves to scripts/ci/msan-lane.sh (build | run | all) and the analyzer gate to scripts/ci/lint-mem.sh, both of which the local paths already reach through run.sh and the Makefile. The folded step becomes `run: |`. Found by the contract itself, running as step 0j of the local Linux leg. It had never run against these jobs, because they were added and then exercised only through GitHub CI -- which is exactly the gap the local ladder exists to close, and the reason the contract runs as step zero of every leg rather than as a job of its own. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
Martin Vogel
|
0c223d9027 |
build(lint): pin analyzer suppressions to the code they argue about
The memory gate stays gating, and it now has a way to record a genuine false positive that cannot quietly outlive its own reasoning. A whitelist entry names one (file, function, check) and carries two things: why the analyzer is wrong, argued from the code, and what was tried before concluding that. The entry is pinned to the sha256 of that function's text. Edit the function and the entry stops counting -- the finding comes back and has to be argued again against the code as it now is. This is the part that matters. A suppression that survives the code it was written about reads as "reviewed" while being nothing of the kind, which is worse than no suppression at all. The gate fails on: a finding with no entry, a finding whose entry has gone stale, and an entry that asserts rather than argues (there is a floor on how much reasoning an entry must actually contain -- the mechanical half of "argued, not asserted"; whether the argument is correct stays a review question). An entry that matches no finding is reported but does not fail, because analyzer versions differ across platforms. NOLINT is still not honoured here and the gate does not read it. The whitelist ships empty: the analyzer is currently clean across LINT_SRCS, so nothing is being suppressed today. The mechanism exists for the first finding that genuinely warrants it. Verified by exercising each path rather than only the clean one: an unaccounted finding fails and names its enclosing function; an argued entry with a matching hash passes; the same entry fails as STALE once the hash no longer matches the function; and an entry that says only "false positive" fails for not arguing its case. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
Martin Vogel
|
02f870105f |
fix(msan): the lane runs the complete code by default
A sanitizer lane covers everything or it is not a sanitizer lane: a pass over a subset asserts coverage it does not have. My exclusion list had grown to three suites — including pipeline, a large one — and dressing it up as an O10 whitelist was wrong. O10 governs a board that TRACKS known-red reproductions; it does not license cutting a sanitizer's coverage to make it green. MSAN_EXCLUDE now defaults to EMPTY in both venues. It remains as an iteration aid — an engineer fixing the underlying problem can narrow the run — and it prints an explicit warning that a green partial result proves nothing about the tree. Consequence, stated plainly: the lane is currently RED on the local arm64 container, where three deep-recursion suites overflow their thread stacks under instrumentation. That is a bug to fix, not a list to live with, and the script records everything known about it. The CI leg (x86-64, where MSan's stack handling is far better supported) is where it gets settled. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
Martin Vogel
|
54dd5ad781 |
ci(msan): restore the lane; x86-64 CI is its authoritative venue
Two corrections. First, removing the CI job while keeping the compose service and the local bindings left the ladder carrying a lane CI did not have — the venue asymmetry the unification work exists to prevent. The lane now lives in both venues again. Second, and the reason the removal was wrong: the evidence behind it was entirely from the LOCAL arm64 container, while the job that got deleted would have run on x86-64. MSan's shadow and stack handling are materially better supported on x86-64, so the thread-stack overflows that drove the exclusions may well be architectural. I never tested the architecture CI uses, and the local ladder cannot emulate it faithfully — so 'it cannot run in CI' was an inference from the wrong platform stated as a fact. The CI leg therefore runs with MSAN_EXCLUDE="" — no exclusions — so the first run settles the question with real evidence on the real architecture. The script's exclusion list stays as the LOCAL default, documented as such. An accepted local/CI divergence for this lane specifically. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
Martin Vogel
|
f96ddf5f93 |
chore(msan): keep the lane local and exploratory, not in CI
Running the lane to completion does not currently work: with three suites already excluded for thread-stack overflow it now dies with a plain SIGSEGV elsewhere. Wiring an auto-running job that cannot finish would be exactly the structurally-red lane O10 forbids, so the test-msan CI job is removed and the lane is documented as exploratory and local-only. What it IS worth, and why the infrastructure stays: every suite it does run is clean under MSan, including the C++ preprocessing path that justified building the instrumented-libc++ image in the first place, and it correctly identified one convincing-looking report as a mixed-build artifact rather than a bug. The image, the script, the compose service, the MSAN_ORIGINS and CBM_THREAD_STACK_MB knobs, and the full record of what was tried all remain, so picking this up is a continuation rather than a restart. The blocker is one problem, stated at the exclusion site: threads whose stacks are sized outside cbm_thread_create overflow under instrumentation, and the failure follows cumulative process state rather than any single suite's depth — which points at per-suite process sharding or finding that thread creator as the fix. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
Martin Vogel
|
7ad0003c95 |
fix(msan): stack-size floor for sanitized builds; document the excluded suites
Two follow-ups to the MSan lane, both from running it for real. 1. compat_thread.c gains a sanitized-build-only stack FLOOR (CBM_THREAD_STACK_MB). Thread stacks here are sized in code, so a sanitizer lane cannot raise them with ulimit -- RLIMIT_STACK at 8/64/256 MiB provably had no effect. The first version overrode only the DEFAULT size, which silently did nothing for worker_pool/runtime/main because they all pass an explicit size; it is now a floor applied to every thread. Shipping builds are untouched (the whole hook is behind CBM_SANITIZED_BUILD). 2. Two grammar-corpus suites are EXCLUDED from the lane, with the full rationale, evidence, and everything tried recorded at the exclusion site per O10 -- including that the floor above does NOT fix them, which narrows the next person's search to a thread creator outside cbm_thread_create. The exclusion list now names SUITES (an earlier version named a TEST and therefore excluded nothing) and fails loudly on an entry that matches no suite, so that silent-no-op cannot recur. Also restores the exec bit on scripts/msan.sh, which the image ENTRYPOINT needs. worker_pool + parallel + pipeline + mcp on macOS: 526 passed, 2 skipped. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
Martin Vogel
|
c9886d4f25 |
feat(ci): MemorySanitizer lane — instrumented libc++/zlib image, full C++ coverage
Stage 2 of the memory-diagnostics program (user decision: go directly to the instrumented image rather than a C-only probe). MSan detects uninitialized READS, the one memory-error class no other lane covers dynamically, and it requires every linked library to be instrumented -- vendored C deps compile in-tree and instrument for free; the two external links do not: - test-infrastructure/Dockerfile.msan: pinned-base image building libc++/libc++abi/libunwind (llvmorg-18.1.8, LLVM_USE_SANITIZER= MemoryWithOrigins) and static zlib v1.3.1 into /opt/msan, with the symbolizer and MSan runtime in a separate last layer so tool additions never invalidate the ~30-min libc++ build. - scripts/msan.sh: the canonical lane entry. ALWAYS clean-builds its BUILD_DIR: make does not encode flags into dependencies, and a stage-1 probe's libstdc++ objects surviving into the libc++ lane produced a convincing-looking uninitialized-value report at preprocessor.cpp:168 -- the uninstrumented .so string constructor wrote the temporary, the instrumented move constructor read it. The clean rebuild proved it an artifact: extraction (incl. the C++ preprocessing path) runs 272/272 with zero reports. - Makefile.cbm: CXX_STDLIB / CXX_STDLIB_FLAGS hooks so the lane can swap libstdc++ for the instrumented libc++ (defaults identical; the shipping build is byte-for-byte unaffected). - docker-compose test-msan service: same aarch64 seccomp/setarch remedy as the TSan service (MSan's shadow layout hits the same personality() block). - CI test-msan job (_test.yml): buildx local-cache via the repo's existing pinned actions/cache -- no new third-party action pins; a warm run skips the libc++ build entirely. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
WarGloom
|
9f2d3e5fe2 | Merge branch 'main' into agent/fix-worker-error-transport | ||
|
Martin Vogel
|
029416fda6 |
Merge remote-tracking branch 'origin/main' into qa/call-reference-rebase
# Conflicts: # Makefile.cbm # internal/cbm/extract_defs.c # src/foundation/compat_fs.c # src/foundation/compat_fs.h # src/pipeline/pipeline.c # src/pipeline/pipeline_incremental.c # tests/test_main.c # tests/test_pipeline.c # tests/test_store_checkpoint.c |
||
|
Yyunozor
|
d899bf44f7 |
fix(hooks): scope pre-commit clang-tidy to staged changes (#1264)
make -j3 -f Makefile.cbm lint runs the full lint-tidy sweep, which currently surfaces ~5,100 pre-existing clang-tidy findings across 113 files. Since the pre-commit hook (scripts/hooks/pre-commit) runs that target unconditionally, it blocks every commit for every contributor who has clang-tidy on PATH, regardless of what the commit touches. Add lint-tidy-diff (scripts/lint-tidy-diff.sh), which runs the same clang-tidy binary/config but passes clang-tidy's own -line-filter so only lines the commit actually added or modified can produce a diagnostic. Pre-existing findings on untouched lines are not reported, whether they are in an untouched file or on an untouched line of a file the commit does touch. The hook now runs lint-ci (unchanged: cppcheck + clang-format + no-suppress) plus lint-tidy-diff instead of the full lint target; make lint / make lint-tidy / scripts/lint.sh are unchanged and remain the full-tree audit. Signed-off-by: Yyunozor <yyunozor@icloud.com> |
||
|
Benyamin
|
4722155d2d |
fix(security): audit shared agent skills
Signed-off-by: Benyamin <benyaminjmf@gmail.com> |
||
|
wargloom
|
8fc04c0b09 |
fix(worker): preserve delivered MCP errors
Signed-off-by: wargloom <wargloom@gmail.com> |
||
|
Martin Vogel
|
5a479facf7 |
fix(release): invoke the new CI scripts via bash, and pin the exec-bit class
Completes the previous commit, which carried only the two mode changes because
the call-site edits were not staged when it landed (--amend is denied in this
repo, so this is additive rather than a rewrite).
Call sites now go through `bash`, which is what most of this repo already does and
which cannot break if a mode bit is lost to a patch application or a non-POSIX
checkout:
scripts/package-release.sh -> bash scripts/ci/check-binary-composition.sh
.github/workflows/release.yml -> bash scripts/ci/append-vt-notes.sh
With the 100755 modes from the previous commit, both sides are now correct, and
either alone would have been sufficient.
tests/test_script_exec_bit_contract.sh pins the class so it cannot recur: any
tracked .sh whose COMMITTED mode is non-executable must not appear as the first
word of a command in workflows, scripts, test-infrastructure or the Makefiles.
The committed mode is the thing that matters and the thing no local run can
check -- the working copy having the bit is exactly why this shipped.
Verified in both directions: passes on this tree, and fails on the exact defect
when the mode and the call site are reverted together. It joins backslash
continuations before analysing, because its own first draft reported a false
positive on
... && bash \
test-infrastructure/vm/vm-run-tests.sh --soak
where the interpreter sits on the preceding line. A contract that cries wolf
teaches people to ignore contracts, so that had to be right before it could be
useful.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
|
||
|
Martin Vogel
|
359485b6c2 |
fix(release): make the new CI scripts executable, and pin that they stay so
Every unix build leg of release run 30499236230 died during packaging:
scripts/package-release.sh: line 190: scripts/ci/check-binary-composition.sh: Permission denied
The composition gate was committed 100644 while being invoked as a command. It
passed every local check because my WORKING COPY had the exec bit -- only the
committed mode was wrong, and nothing you can run locally reveals that.
scripts/ci/append-vt-notes.sh had the identical defect waiting in the verify
step, the last step of the release, so this would have failed a second time after
two hours of tests, build, smoke and soak.
Fixed on BOTH sides, because either alone suffices and the pair is mode-proof:
the two scripts are now 100755, and their call sites invoke them through `bash`,
which is what most of this repo already does and which cannot break if a mode bit
is ever lost to a patch application or a non-POSIX checkout.
tests/test_script_exec_bit_contract.sh pins the class: any tracked .sh whose
COMMITTED mode is non-executable must not appear as the first word of a command
in workflows, scripts, test-infrastructure or the Makefiles. Verified in both
directions -- it passes on this tree, and fails on the exact defect when the mode
and the call site are reverted. It also joins backslash continuations before
analysing, because its own first draft reported a false positive on
... && bash \
test-infrastructure/vm/vm-run-tests.sh --soak
and a contract that cries wolf teaches people to ignore contracts.
Product code is untouched: the test phase of the failed run was 27/27 green on
this exact tree, and a file mode cannot change a test outcome.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
|
||
|
Martin Vogel
|
4836b9a146 |
security: record the mimalloc local patch in the vendored manifest
The release build blocked at Layer 8 (vendored dependency integrity):
MISMATCH: vendored/mimalloc/src/options.c
expected: 96ef01e4...
actual: 192ce06a...
That is the integrity check working exactly as designed. The hardening pass
patched mimalloc's version banner to stop baking __DATE__/__TIME__ into every
binary, which is an intentional and reviewed change to vendored code -- but an
intentional change is indistinguishable from a supply-chain edit until someone
records it, which is the entire point of the manifest. Updating the recorded
checksum is the review being written down.
Scope verified before committing: the manifest diff is exactly ONE line, the
options.c hash, and it matches the hash CI computed. No other vendored file
moved. scripts/security-vendored.sh's other layers still pass on the updated
tree -- no subprocess, network or dangerous calls in vendored code, dlopen
confined to sqlite3.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
|
||
|
Martin Vogel
|
1f674c805e |
harden: remove capability that should never have shipped
Microsoft's ML flagged the rc.1 release binaries. The decisive evidence is that the SAME sha256 went from 0/62 clean to Microsoft-detected in about an hour with no byte change, so the verdict lives partly in scanner-side state and no code change can promise a clean result. What code CAN do is stop shipping things that have no business in a release artifact, which is worth doing on its own merits and incidentally widens the classifier margin. Every claim below is verified against a built binary by the new gate, not by reading source. Executable stack (the worst of the findings). vendored/nomic/code_vectors_blob.S is the only assembly in the build and carried no .note.GNU-stack. An unannotated object makes ld assume the worst for the whole link, so EVERY Linux release we have ever shipped had GNU_STACK RWE. Adds the note (cause) plus ELF-only -Wl,-z,noexecstack (outcome); the gate fails the release if it returns. Test seams are now opt-in, never opt-out. TEST_SEAMS=1 defines CBM_ENABLE_TEST_SEAMS; without it the crash-orphan probe -- which forks a child that ignores SIGTERM and loops forever, then writes its pid to a caller-supplied path -- and the lease-ownership marker compile to trivial stubs, so call sites are untouched and the binary holds no fork, no signal handler and no env-var string. Opt-IN is the point: forgetting the flag yields a clean binary rather than a leaky one. scripts/test.sh requests it in the leg that consumes it, and tests/test_worker_watchdog.sh now asserts the capability up front instead of dying later with an opaque "Killed: 9". The daemon's background version check is gone. It spawned curl against api.github.com/repos/.../releases/latest on the first eligible session of every run to say "a newer version exists" -- a release URL and an outbound request in every shipped binary, for something the install scripts already report. The INJECTABLE SEAM survives: update_ops is still honoured, the fakes in tests/test_daemon_application.c still cover notice/ownership/cancellation/replay, and with no provider application_update_subscribe_locked returns early so no generation ever starts. "No network request by default" is now structural. Dead capability out of release builds. The tar.gz/zip extraction block (gzip_decompress through cbm_extract_binary_from_zip, plus its cli.h declarations) moves under CBM_CLI_ENABLE_TEST_API -- verified self-contained, zero uses of any helper outside it, only callers the excluded updater and tests/test_cli.c. Downloading an archive, decompressing it, picking an executable out of it and marking it executable is the canonical dropper composite; it is now absent rather than merely unreachable. SQLite is built with -DSQLITE_OMIT_LOAD_EXTENSION (no caller of load_extension anywhere in src/ or internal/), removing that API surface and part of the dlopen/dlsym surface. Temp files and environment scanning (S2/S3). Predictable paths in mcp.c, artifact.c and diagnostics.c are created privately and exclusively and written through the returned descriptor; pass_envscan.c no longer descends symlinked directories out of the project root, and its fixed 512-byte path buffers no longer truncate into pointer arithmetic that could land outside the buffer. Build-time entropy. mimalloc's version banner baked __DATE__/__TIME__ into every binary, so two builds of identical source seconds apart could never share a hash and no release could inherit a false-positive determination made about its predecessor. Local patch removes it (marked to survive refreshes), -Wdate-time makes any future use a build error, and -Wl,--no-insert-timestamp stops the PE header carrying the link clock. scripts/ci/check-binary-composition.sh is the proof that each removal stays removed, wired into package-release.sh after strip so the local artifact-flow smoke enforces exactly what CI does. It asserts absences plus a CANARY string, so handing it a compressed, stubbed or empty file fails instead of passing vacuously, and a missing tool is a hard error -- a skipped assertion must never look like a satisfied one. Two build-system traps found by that gate, both of which had silently defeated a fix: the product binary is compiled in one shot from sources, so a flag flip did not rebuild it (now tracked by a .build-config stamp that also removes the binary, making it independent of mtime granularity); and prod_sqlite3.o / prod_mimalloc.o depended on a single named source, so SQLITE_OMIT_LOAD_EXTENSION and the mimalloc patch BOTH compiled to nothing on the first incremental build. Source review would have called them done. Deliberately NOT changed. Three seams stay in release artifacts because scripts/smoke-test.sh runs against the real artifact and needs them: CBM_TEST_CRASH_ON and CBM_TEST_HANG_ON inject the faults that prove supervisor recovery, and CBM_TEST_WINDOWS_USER_PATH_RUN_ID is what keeps the PATH smoke from writing the tester's actual PATH. The gate treats those as an allowlist, so a NOVEL seam still fails. The true no-UI standard build is deferred rather than rushed: src/ui/* is in PROD_SRCS and four files outside src/ui reference UI symbols, including the daemon that serves the UI, so that assertion reports instead of failing until the split lands -- a gate everyone knows is red teaches people to ignore gates. No grammar is removed. ObjectScript accounts for essentially all binary growth since the last provably-clean release (+21.4MB rodata, +1.1MB text from two four-line shims), which made it the obvious ablation candidate, but a dry run performed twelve real Defender endpoint scans across standard/UI and amd64/arm64 with ObjectScript, the daemon and the expanded hooks all present and every scan was clean. Nothing there is a deterministic trigger, so cutting a community-contributed language would spend a real feature on unproven margin. Lean is not a candidate either: at 99.6MB of source it is by far the largest grammar, but it shipped in v0.9.0 which scanned 20/20 clean, so removing it would produce a novel unscanned profile instead of restoring a known-good one. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
Martin Vogel
|
3e343a3eb2 |
ci(release): revert the VT gate to zero tolerance
The pre-release ML false-positive tolerance (single-engine Microsoft "!ml" verdicts downgradable with Defender endpoint evidence, #1340) is reverted by owner decision: cbm does not ship binaries carrying a VirusTotal detection, demonstrably false or not. A "trojan" badge on a release asset is a reputation cost the project is not willing to price in, however good the accompanying evidence. The gate returns to its original form: any detection, by any engine, on any artifact, on any version blocks the release. The endpoint verification tool and the evidence side-channel are removed with it; the notes renderer keeps its extracted-script form but only ever states a verified "0 detections". False positives are resolved upstream instead: verify the bytes on a real Defender endpoint, submit a Microsoft false-positive report for the exact hashes, wait for the detection to clear, then RE-RUN the failed verify job -- which does not rebuild, so the cleared hashes are the shipped hashes. tests/test_vt_gate_zero_tolerance_contract.sh pins the decision: clean passes; 1 malicious and 1 suspicious each block across stable, -rc., -pre and -alpha versions; plus a tripwire for the specific reverted evidence mechanism returning. Loosening this gate again has to consciously delete that contract. The skip_tests dispatch input and the script-extracted notes step survive the revert -- both are orthogonal to gating policy. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
Martin Vogel
|
48fc942418 |
ci(release): double-verify ML antivirus false positives instead of re-rolling builds
Release run 30464288732 was blocked by the VirusTotal gate: three linux-amd64
binaries flagged 1/62 by Microsoft's Wacatac.B!ml -- fully stripped binaries
(0 symbols, verified on the exact artifacts), the state that scanned clean in
the two previous cycles. Meanwhile a real Defender endpoint (engine
1.1.26060.3008, signatures 1.455.410.0 updated the same day, RTP on) scans
the identical bytes clean. Four cycles of evidence now say the same thing:
this verdict is an unstable ML decision boundary, not a property of the code,
and no build-side lever moves it durably -- stripping, downloader removal and
metadata changes each "worked" only until a later build flipped it back.
So stop treating the flag as buildable-away and verify it honestly instead:
check-virustotal.sh may downgrade BLOCKED to TOLERATED only when ALL hold:
- pre-release version (-rc./-pre/-alpha/-beta); stable releases never
- every failing file flagged by exactly ONE engine
- that engine is Microsoft and the verdict ends in "!ml" (never a
signature name)
- hash-pinned Defender ENDPOINT evidence is attached to the draft release
(defender-endpoint-verification.txt) proving Microsoft's shipping
product, signature-updated at scan time, reports the exact bytes clean
av-endpoint-verify.sh (new) produces that evidence: downloads the draft
assets, scans them on the local Windows VM endpoint, refuses to attest if
RTP is off or Defender itself detects, uploads the hash-pinned result.
The gate prints the exact command when evidence is missing; re-running the
failed verify job does not rebuild, so the bytes stay fixed.
append-vt-notes.sh (new, extracted from inline YAML per venue-parity) then
renders the release-notes table honestly: a tolerated file reads "1/62 ML
false positive, endpoint-verified clean", never "0 detections".
tests/test_vt_gate_tolerance_contract.sh pins all nine decision directions
against a stubbed VT API and release store -- clean pass, stable-never,
missing/stale/DETECTED evidence, signature-named verdict, non-Microsoft
engine, multi-engine -- so the tolerance provably fails closed.
Also: release.yml gains skip_tests for re-releases of an already test-green
tree (build/smoke/soak/verify always run; lint failures still gate via
!cancelled() && !failure()).
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
|
||
|
Martin Vogel
|
ae52db3147 |
test: make three Windows nondeterminisms deterministic
All three cost real release cycles, and none is fixed by widening a budget. daemon_application_cancels_physical_job_only_after_final_session waited for the SUBSCRIBER COUNT to reach 2, then cancelled both sessions and asserted the physical job had started exactly once. The job starts asynchronously after subscription, so both cancels could land first and leave starts == 0 -- arguably the correct outcome. It now waits for the state the assertions actually require. Verified 47/47 on Windows, the only platform it ever failed. The parallel harness refused outright when the suite leader had already exited, because taskkill /T cannot walk a tree from a dead PID. But the leader can exit between the timeout decision and that call, so the harness itself lost a race: a natural exit at the wrong moment failed the whole wave. It now proves cleanup the only way still available -- nothing parented to that PID -- and its contract asserts the PROPERTY rather than the phrase "tree cleanup" it used to grep for. That string pin is what broke when the guard was reworded while behaving correctly; the contract now checks rc==2 AND that the descendant really did survive, which would also catch a guard that claims to fail closed while leaking. extract_wide_flat_file_is_linear took ONE sample per size, so the ratio carried the noise of both. On a loaded Windows VM linear code measured 51x against a 40x bound (184ms -> 9387ms). Best-of-N instead: timing noise only ever adds time, so the minimum is the cheapest good estimate of the noise-free cost. The bound is deliberately unchanged -- it sits where linear (~20x) and quadratic (~128x) are each >=2x away, so raising it would move the test toward the very signal it exists to catch. Now measures 19.1x on Windows, 21.4x on macOS. Also: the smoke's `cli` helper redirected stderr to a file and discarded it, so any of the 10 bare `VAR=$(cli ...)` assignments could kill the run under `set -euo pipefail` printing NOTHING. One such abort cost a full Windows cycle just to locate and still could not be attributed. It now surfaces the command and its stderr. Neutral wording on purpose: one call site expects a non-zero exit and must not read as a failure. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
Martin Vogel
|
afd67c51b0 |
refactor(cli): install scripts own updating; the binary carries no URL
Completes the move started on Windows. `update` prints the install script's
command and exits 0 on every platform; the download/extract/chmod/exec sequence
is compiled out of release builds entirely, and the MCP background thread that
ran cbm_popen("curl ... api.github.com ...") on first tool call is gone.
The installers now place themselves beside the binary, which closes the last
gap: `update` referenced a raw.githubusercontent URL purely because an
install.sh-installed user had no local copy to point at. Two details make that
safe.
The source is the install script from the archive we JUST checksum-verified,
never "$0" -- which does not exist under `curl | bash`, and would pin the user
to the OLD installer forever. Since the script ships inside the verified
archive, it inherits that verification instead of needing its own.
And it is published by atomic rename, never written over the live path. Bash
reads a script incrementally by byte offset, so overwriting the file it is
executing continues reading NEW bytes at the OLD offset: silent, bizarre
corruption. cp to a temp name then mv -f swaps the directory entry while the
running shell keeps its original inode. Windows follows the same rule via
Copy-Item + Move-Item even though PowerShell parses up front.
Deliberately NOT done: spawning the new installer to delete and replace its
predecessor. Fetch remote code -> drop to disk -> spawn -> self-delete is a
textbook stager, and self-deletion is among the most heavily weighted
heuristics there is. It also gains nothing -- `"$DLBIN" install` already runs
the NEW release's install logic, because the binary owns the install.
Net: zero install or download URLs remain in the shipped binary, and the
allow-list entry is retired with it. Both scripts scan CLEAN (0/61) with the
self-placement code in them.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
|
||
|
Martin Vogel
|
a28749fc4e |
fix(release): strip Mach-O with the tool that can actually do it
The previous change generalised `--strip-all` to every platform. Apple's strip does not accept that flag, so the macOS build failed outright -- loudly, which was the lucky outcome. The unlucky version was already in the tree before that: `strip -x`. It succeeds and leaves 4058 symbols, which is precisely the state VirusTotal flagged on darwin-arm64 while the fully stripped ELF legs came back clean. Measured on that artifact: llvm-strip --strip-all 373 symbols CLEAN strip (no flags) 378 symbols equivalent strip -x -S 4058 symbols the FLAGGED state strip -X / -u -r 4058 symbols likewise So the flags are chosen per format, and a candidate that cannot do the job is a hard error rather than a silent fallback to a weaker strip. Verified on a PATH without llvm-strip, exactly as the runner has it: 373 symbols, codesign --verify --strict passes, the binary runs. Packaged and scanned before merge: darwin-arm64 0/60 and ui-darwin-arm64 0/61, both CLEAN -- the platform that was flagged on the previous release run. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
Martin Vogel
|
0b31a5dcdf |
fix(security): declare the printed install-script URL on the allow-list
DCO / dco (push) Has been cancelled
The `update` handoff prints a curl one-liner when install.sh is not beside the binary, and the security audit blocked the release on it: BLOCKED: src/cli/cli.c: URL not on allow-list: https://raw.githubusercontent.com/DeusData/ Two things were wrong. The URL was not declared, and it was split across string literal continuations, so the audit extracted only the first fragment -- an allow-list entry for the real URL could never have matched it. Hoisted to a single CBM_INSTALL_SH_URL token so the declaration means what it says. It is the same URL as the documented one-liner install, and it is printed for the user to paste: the binary no longer downloads anything. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
Martin Vogel
|
9ecabe0a6f |
refactor(cli): move self-update out of the product on every platform
Windows already handed updating to install.ps1, because a running .exe cannot
replace its own image. The remaining platforms now do the same, for a different
and better reason.
An in-process updater is structurally a downloader: it fetches a remote archive,
verifies it, extracts it, marks the result executable and runs it. That composite
lived in every shipped artifact -- along with the release URLs it fetched -- to
serve a command most users run a handful of times. The install scripts already
do all of it, are idempotent (so re-running one IS the update), and run while cbm
is not running.
`update` now prints the exact command for the running platform and exits 0:
unix bash "<dir>/install.sh" (or the curl one-liner when not adjacent)
windows powershell -ExecutionPolicy Bypass -File "<dir>\install.ps1"
Both scripts ship inside their release archive, so the printed path resolves
next to the binary. Flags are still parsed and validated, so `update --dry-run
--ui` keeps rejecting typos rather than silently accepting them.
The updater itself is EXCLUDED from release builds rather than left to dead-code
elimination: build_update_url, download_verify_install, extract_and_install_binary,
checksum fetch, detect_os/detect_arch and the cli_download_* helpers now compile
only under CBM_CLI_ENABLE_TEST_API, which release builds do not define. The C
suite still covers the flow through the activation test seam. Verified against
the release binary: `releases/latest/download` is gone entirely.
Also removes the MCP background update check. It ran on the first tool call and
did:
cbm_popen("curl -sf --max-time 5 -H 'Accept: application/vnd.github+json' "
"'https://api.github.com/repos/.../releases/latest' 2>/dev/null")
An embedded shell command plus a startup network callout, in every session, to
tell users a newer version exists. It shelled out, depended on curl being
installed, and phoned home from every agent session. The install scripts report
versions; this did not need to be in the server.
Smoke follows the same collapse. Phases 6c and 14 asserted a Windows-specific
handoff and a POSIX in-process replacement; they now assert one platform-neutral
contract selected by UPDATE_SCRIPT, and 14a's byte-identity check compares the
driver against ITSELF before and after -- comparing against "$BINARY" passed on
Windows only because Windows has no signing step, while the POSIX fixture
ad-hoc re-signs its copy and so differed before `update` ever ran.
The Linux glibc guard is RELOCATED, not retired. The standard linux asset links
glibc 2.38+ and breaks Debian 11, RHEL 8 and Ubuntu 20.04, so the installer must
fetch the static "-portable" build. That constraint moved from the binary to
install.sh, and the smoke now guards it where the behaviour lives.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
|
||
|
Martin Vogel
|
c725d9fad0 |
fix(release): strip Mach-O fully, not just local symbols
The first strip landed as `strip -x` on Mach-O, out of caution that a full strip could leave an image dyld will not load. That caution was wrong for this binary, and it cost a release cycle. `-x` retains external symbols -- 4058 of them on darwin-arm64 -- so the macOS artifacts kept the very symbol table the ELF legs had just shed. Run 30414312364 then flagged exactly those two artifacts (darwin-arm64, ui-darwin-amd64) while every Linux and Windows binary came back clean, which read as the detection "moving to macOS" when it had simply stayed on the unstripped set. Measured on the flagged darwin-arm64 artifact: strip -x 4058 symbols VirusTotal 1 malicious / 61 --strip-all 373 symbols VirusTotal 0 malicious / 61 CLEAN The packaged binary still verifies (`codesign --verify --strict` passes, having been re-signed after stripping) and still runs. Symbol-table removal has now cleared the detection on three independent artifacts across two platforms: linux-amd64 twice, and darwin-arm64 here. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
Martin Vogel
|
af250e7d65 |
fix(release): strip shipped binaries
DCO / dco (push) Has been cancelled
Release binaries carried their full symbol table. Production compiles without -g, but the linker keeps .symtab regardless, so every archive shipped ~536 KB of internal function names that nothing needs: a bigger download and a free map of the internals, by accident rather than by decision. Nothing symbolizes at runtime -- mem_profile.c is not in the production build and never calls backtrace_symbols -- so no diagnostics are lost. It also had a concrete cost. Microsoft's ML scored the unstripped linux-amd64 binary Trojan:Script/Wacatac.B!ml (1 engine of 62) and blocked release run 30398064336 at the VirusTotal gate. That verdict is a decision-boundary artifact, not a property of the code, and the evidence is unambiguous: * the dry-run build two days earlier ( |
||
|
Martin Vogel
|
0a22fb14df |
fix(windows): absorb transient file locks when retiring a running image
DCO / dco (push) Has been cancelled
Release run 30385988475 lost three Windows smoke jobs to one phenomenon on two
different surfaces. A Windows file that was just written or just executed can
refuse deletion and rename for a moment while a scanner reads it or while the OS
finishes reaping a child. Both conditions clear on their own within moments.
It reads as nondeterminism because it is timing: windows-2025 passed both amd64
variants while windows-latest -- the SAME image, windows-2025-vs2026 -- failed
both, and windows-11-arm failed too.
PRODUCT: uninstall abandoned a live installation.
With one binary per platform, uninstall must remove the very image it is
executing, and on Windows a running image can only be renamed, never
overwritten. That rename got a single attempt, so a transient sharing violation
surfaced to the user as
error: failed to remove ...; completed configuration/index cleanup may remain
with the install left half-torn-down. install.ps1 already retries this exact
operation ten times for this exact reason; the C path now does the same, and
only for sharing/access/lock violations, so a file that is genuinely held still
fails closed instead of spinning.
Proven with a failure-injection seam rather than by racing a real scanner:
3 injected failures must still succeed, 64 must still fail AND leave the target
in place. Deterministic, so it does not need CI to learn whether it works.
HARNESS: the smoke killed itself silently.
Phases that install a binary then `rm -rf` the directory hit the same lock. Under
`set -euo pipefail` that returns non-zero and ends the run WITHOUT printing
anything -- the logs jump straight from "OK 13h" to job cleanup with no FAIL
line, which is what made this expensive to read.
None of those 16 removals is an assertion; they are fixture cleanup, and
_smoke.yml already states the principle ("a temp dir on an ephemeral runner must
never fail the job") that smoke-test.sh did not honour. They now go through
smoke_rmtree, which retries so the disk is actually reclaimed -- these fixtures
hold ~300 MB binaries and runners are disk-tight -- then warns and continues.
Verified: Windows VM full smoke ALL PASSED (previously died after 13h), macOS
full smoke ALL PASSED, activation_transaction + cli 259 passed on Windows, all
10 shell contracts pass, lint-ci clean.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
|
||
|
Martin Vogel
|
2740208f40 |
fix(cli): report a failed install --dry-run plan check in the exit status
The Windows smoke guards that an invalid PATH seam must fail closed rather than
silently falling back to the live HKCU\Environment\Path. That guard went green
in a way that meant nothing: the seam still refused, and install still printed
"PATH configuration failed", but the process exited 0.
Windows release builds used to compile a separate managed-install path. With one
binary per platform they compile the shared cbm_cmd_install, which since
|
||
|
Martin Vogel
|
a54ea95719 |
fix(windows): ship one binary — remove the launcher stub flagged as a dropper
DCO / dco (push) Has been cancelled
Windows shipped a PAIR: a small permanent launcher (codebase-memory-mcp.exe)
plus the real product binary (codebase-memory-mcp.payload.exe). The launcher
existed for exactly one reason — a running .exe cannot replace its own image
on Windows, so an in-process self-update needs a second resident binary to do
the swap.
That stub is statically indistinguishable from a dropper: a small, unsigned,
zero-prevalence PE whose whole job is verify-and-execute another binary.
Defender's ML scored it Trojan:Win32/Wacatac.B!ml and blocked the v0.9.1-rc.1
release at the VirusTotal gate. It is not fixable in our code on x64 —
bcrypt-free, stripped, VERSIONINFO'd, minimal-resource and even
resource-FREE builds on CI's own MSYS2 CLANG64 toolchain were all flagged,
while the product binary scans clean on every platform.
So remove the stub and move self-update OUT of the process into install.ps1,
which runs while cbm is NOT running: Windows' image lock only blocks a
process from replacing ITSELF. now prints the exact PowerShell
command (with the Unblock-File hint for Mark-of-the-Web); install.ps1 is
idempotent, so re-running it IS the update — it stops the daemon, renames the
running binary aside (the one mutation Windows permits on a running image),
publishes the new one, and sweeps retired copies.
Windows now matches Linux and macOS: ONE binary per platform.
* packaging, install.ps1, npm and PyPI wrappers all carry a single binary
* the launcher/payload ABI contract and ~2500 lines of stub state machinery
are deleted
* every daemon start, CLI call and hook fire loses a process spawn, a named
pipe handshake and an stdio relay
* test_windows_bundle_contract.sh is rewritten as an INVERTED contract: it
now asserts no shipped surface can reintroduce a launcher/payload pair,
and that install.ps1 retires the running binary before publishing
Verified: VirusTotal 0/67 on the packaged binary and 0/58 on install.ps1 (no
certificate involved); macOS and Linux full suites green; Windows guards all
green including the new update-handoff contract; npm 10/10; PyPI 3/3.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
|
||
|
Martin Vogel
|
c3b35691ab |
fix(soak): resolve the .exe whenever it exists, not only when -x fails
Both hosted-runner Windows soak legs failed at Phase 1/diagnostics with an empty frontend stderr while the daemon's own log showed a healthy diagnostics.start: the workflows pass the suffix-less binary path, msys resolves it transparently (so the old 'append .exe only when the plain name is not executable' branch never fired), and soak-test keys BOTH native-Windows gates — the cygpath'd CBM_CACHE_DIR and the coproc stdio branch — off the literal .exe suffix. A suffix-less native binary therefore received a POSIX-form cache path: daemon logs mis-rooted (diagnostics grep zero, arm64) and the frontend's cache path handling died silently (x64 SIGPIPE at first write). Reproduced on the VM's CLANG64 environment with a suffix-less invocation (same FAIL) and red->green proven: with the normalization the identical invocation runs the full quick soak to PASSED. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
Martin Vogel
|
657d4b0284 |
fix(soak): give the first diagnostics snapshot a 30s window
Both Windows runner soak legs died right after 'server running' while the daemon's own log shows a healthy diagnostics.start with valid paths — the wait polled 10s for a snapshot whose first WRITE lands one 5s interval after start and can exceed 10s on a cold 4-vCPU runner mid-initial-index (the VM's 18 cores never miss it; the x64 leg's SIGPIPE was the same failure path dying inside the pipeline). Budget doctrine: the wait sits above the worst case. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
Martin Vogel
|
0c0111eb33 |
fix(smoke): header-only fixture-server readiness polls
Hosted Windows runners reset the readiness poll's full-archive GET mid-body (WinError 10054 on the server, ~99 aborted transfers per job) while the identical stack passes on the VM; every windows smoke job died at 'fixture server did not serve' with a healthy server. A HEAD proves the artifact routes without transferring the body per poll; the download phases still own full-body transport and report phase-precise if that is ever broken. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
Martin Vogel
|
6d8b39466b |
fix(smoke): Phase 15 probes the UI a fresh profile can actually serve
Every ui-variant smoke job failed on every platform in dry run 30226483380 — and it turns out a ui-variant Phase 15 had never legitimately passed anywhere: the old probe (a) never passed --ui=true, but the HTTP UI is a persisted DEFAULT-OFF setting, so a fresh profile (every runner, every smoke HOME) can never serve; (b) fed stdin an empty file, but the UI does not pin the process — stdio EOF ends it cleanly (rc=0) before any poll sees it serve; (c) gave a 10s window to a server measured to bind ~6s after launch on a FAST host; (d) POSTed an MCP initialize at /rpc, which speaks the UI's own narrow query protocol, not MCP. The rebuilt probe mirrors the drive-listing guard's proven invocation (--ui=true --port=N equals-form, held-open stdin), waits 30s, and verifies the API surface with the stateless GET /api/ui-config. Validated red->green against the release-shaped artifact: the full darwin ui artifact lane now passes 15a/15b legitimately. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
Martin Vogel
|
1d822c86bf |
fix(soak): BSD-safe mktemp template for the per-leg log
BSD mktemp substitutes only TRAILING X's, so the template
cbm-soak-leg-XXXXXX.log creates a near-literal file on macOS and the
second leg of the same job collides with the first leg's leftover
('File exists'). First runs mask it — which is why the quick leg
passed and query-leak failed locally, and why CI's macos soak jobs
(BSD mktemp, two legs per job) would fail the same way.
Drop the suffix; the path is only a log handle.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
|
||
|
Martin Vogel
|
c1425d2e27 |
ci: write the shard manifest before any suite runs; warn on absent upload
The manifest content (leg, slice, list hash) is fully determined at slicing time, so write it up front: a red run's manifest is exactly as load-bearing for the cross-shard union proof as a green one's. With that, a missing manifest at upload time can only mean the job died before the harness started (contract step / build failure) — that failure is already the job's red, so the upload warns instead of stacking a second error on top (the recurring 'No files were found with the provided path: build/c/test-logs/shard-manifest.txt' annotation). The shard-completeness job still gates the union. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |
||
|
Martin Vogel
|
ad713bce93 |
fix(mcp): guarantee valid UTF-8 in tree-format output
Dry run 30206811293: macos-15-intel release smoke failed B3 with a visibly correct 'semantic: 50' table — BSD grep returns no-match for the ENTIRE output when any line carries a NUL, control byte, or invalid UTF-8 (verified: printf poisoned input -> grep -qE fails), so one bad byte in one row silently unmatches every anchored check. Close the class at the emission choke point: control bytes and invalid UTF-8 force the quoted form; quoting escapes controls as \u00XX and replaces invalid sequences with U+FFFD (RFC 3629 validation, NUL-safe). Tool output is valid UTF-8 by construction. Unit test proven RED-on-revert (raw bytes emitted) and green with the fix. The smoke's B3 failure path now also dumps od -c bytes so a future red names the exact byte. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> |