The local Windows leg caught this within hours of merging #1532: on a machine
that is not domain-joined, USERDOMAIN is the literal string WORKGROUP, and
`WORKGROUP\test` is not a resolvable principal —
WARN: build-dir DACL stamp (pre-wave) failed (user=WORKGROUP\test ...):
WORKGROUP\test: No mapping between account names and security IDs was done.
so the grant fails outright and the tree keeps its inherited
`Authenticated Users:(M)`. The guard then correctly reports that cross-account
mutation is still possible, and the whole leg stops. Since most Windows
machines are not domain-joined, that is the common case, not the exotic one.
#1532 was fixing a real bug in the other direction (a bare name resolves
against the machine first, so a host named like its user grants to an empty
principal). Both failures are name RESOLUTION failures, so this stops resolving
names: the account is identified by SID, which is unambiguous everywhere. The
SYSTEM and Administrators grants in these same commands have always used that
form — the current user was the odd one out. Name lookup survives only as a
fallback where PowerShell is unavailable.
Applied to all three scripts carrying the stamp (run-tests-parallel, soak-test,
memlab), since they share the failure and would otherwise drift apart.
Co-Authored-By: Kiborgik <Kiborgik@users.noreply.github.com>
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The Windows DACL stamps in run-tests-parallel.sh, soak-test.sh and
memlab.sh pass the current user to icacls by bare name. Git Bash resolves
whoami to coreutils, which prints an unqualified name, and icacls resolves
an unqualified name against the machine before the user. On a host whose
name equals the user's (COMPUTERNAME=BUILD with user build) the grant
lands on an empty principal:
BUILD\:(OI)(CI)(F)
Combined with the /inheritance:r on the same invocation, the directory
ends up granting no usable account. run-tests-parallel.sh then cannot
write its own log directory and the wave dies with:
build/c/test-logs/results.txt: Permission denied
FAIL: test-runner --list-suites exited nonzero
Prefix the name with USERDOMAIN when it is set, which makes the account
unambiguous, and fall back to the bare name otherwise. SYSTEM and
Administrators in the same invocations already use SID form and are
unaffected.
Signed-off-by: PETRO YAKOVYSHYN <petroyakovyshyn@gmail.com>
Both Windows runner soak legs died right after 'server running' while
the daemon's own log shows a healthy diagnostics.start with valid
paths — the wait polled 10s for a snapshot whose first WRITE lands
one 5s interval after start and can exceed 10s on a cold 4-vCPU
runner mid-initial-index (the VM's 18 cores never miss it; the x64
leg's SIGPIPE was the same failure path dying inside the pipeline).
Budget doctrine: the wait sits above the worst case.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The staged venue-unification bundle: _smoke.yml onto the shared
wrappers with an extracted-artifact input (CBM_SMOKE_ARTIFACT_DIR),
soak-legs.sh as the one canonical soak entry (quick + query-leak legs
across all venues), protected per-user TEMP roots shared across
venues, clean-disk preflights before every VM/Docker run, win.sh and
run.sh routed through the canonical scripts, soak.yml retired, and
the venue-parity contract (whitelist walker: workflows may provision
or call canonical entries, nothing else).
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The native Windows soak could never start its server, so that gate had
never actually measured anything. The root cause was the harness, not the
daemon: an MSYS filesystem FIFO feeding a native Windows process delivers
empty stdin (reproduced 5/5), while a Bash anonymous coprocess carries the
full JSON-RPC initialize exchange to the same protected payload, and the
failing daemon log showed an orderly client disconnect rather than a
daemon-side eviction. The soak now uses a named coprocess on native
Windows only, duplicating its endpoints to stable fd3/fd4 and closing the
originals so closing fd3 still delivers EOF; POSIX keeps the FIFO path.
The coproc syntax sits inside eval because macOS system Bash 3.2 must
still parse this file even though only MSYS2 Bash 5 executes that branch.
With transport repaired the next exact failure surfaced: native Windows
Python cannot open an MSYS /c/... diagnostics path. Both consumers now
pipe the file through stdin, the pattern already established elsewhere in
the repo, and the soak recovery contract forbids reintroducing a direct
native-Python open.
Also on the release path:
- The smoke fixture server no longer fsyncs before atomically publishing
its port. This is ephemeral readiness signalling, not crash-durable
state, and the macOS Intel runner failed inside that durability sync.
When the contract does fail it now reports the observable state --
waited, exit status, port file, staged temp files, interpreter, startup
log -- because the previous verdict named nothing on the one runner we
cannot reproduce locally.
- The POSIX publication test hook no longer compiles into Windows builds,
fixing an exact -Werror unused-variable failure. The setter keeps a
parameter-consuming Windows stub because it is public API.
- The Wine leg assembles the real release layout (payload plus canonical
launcher) and version-checks both, running the launcher through cmd so
it has a Windows-visible parent. The unsupported Wine soak is removed:
Wine stays a fast compile/package/version check, and native Windows is
authoritative for daemon, locking, ACL and process-lifetime semantics.
- run.sh soak-windows routes to the native Windows VM, which validates the
payload, builds the protected per-user temp root, stamps ACLs, and
refuses success without a completion summary.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The fuzz harness swallowed the target's exit status through '|| true', so
a SIGSEGV counted as a pass; the status now propagates (a planted crash
yields 139) and payloads derive from a logged, replayable seed, with a
missing python3 failing the gate instead of generating zero mutations.
The Windows guard runner classified unknown exit codes as skips; the
contract is now explicit (0 green, 1 red, 2 precondition-skip, anything
else a failure) and an all-skip run fails as verifying nothing.
The smoke suite gains crash-class detection on the phase-9b tolerance
paths and phase 11 kill handling (rc >= 128 or a missing jsonrpc banner
fails), a free-port pick plus readiness poll for the UI phase, and a
phase-15b failure that actually exits nonzero.
The soak reader resynchronizes on late JSON-RPC responses by draining to
the matching id, and the analysis fails when fewer than 60 percent of
snapshot attempts produced rows — a vacuous analysis previously passed.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Fixes surfaced while bringing the shared-coordination-daemon branch
green across macOS, the Linux containers, and the real Windows-ARM64 VM,
plus a new stability guard that caught one of them.
- daemon(win): the CLI teardown transition latched a failed release as
permanent, but a Windows participant-state release must briefly
try-hold the shared startup/legacy gates and legitimately collides
with a concurrent one-shot's teardown; the release is retriable by
contract (it always retains the transition), so parallel one-shot
commands no longer report "CLI coordination cleanup failed" despite
succeeding. Found by the new stability guard's churn section.
- cli/main(win): install/update/uninstall and the Augment hook-script
removal derive the managed launcher's identity from its plain
drive-form path (cli_windows_plain_utf8), not the \\?\ extended-length
form, so agent-config ownership matches on Windows and uninstall no
longer leaves the MCP entry or owned hook scripts behind.
- security: the `daemon start --open` browser launch is now shell-free
(ShellExecuteW on Windows, cbm_exec_no_shell elsewhere) instead of
system(); the CORS origin check spells out its two literal loopback
URLs so the static URL audit sees a complete value. Both clear the
Layer-1 allow-list audit.
- coordination: version-cohort lock retries now sleep a per-process
jittered interval — fixed-period retries can phase-lock two
participants so one starves. The activation-quiesce test's
observation window now covers worst-case candidate staging (three
tamper-defense hashes over a ~1 GB sanitizer binary exceed the old
30 s budget on container I/O).
- smoke/soak: the agent-config smoke matches Windows config paths
through their escaped-backslash quoted form; the soak parser skips the
update-available banner before the JSON summary and gained a one-shot
CLI admission-churn phase inside the RSS/FD leak window. The build-dir
safety fixture creates a real symlink/junction (MSYS2 ln -s otherwise
deep-copies) so its traversal-refusal contract is genuinely exercised,
and the symlinked-agent-roots test demotes its planted links to an
unprivileged owner when run as root so the refusal is really tested.
- tests(win): a new daemon-stability guard exercises the parameter
surface, hook fail-open with its rate-limited notice, start-twice and
occupied-port handling, busy-stop refusal against a live MCP session,
kill -9 crash recovery, and sequential/parallel one-shot churn.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Daemon lifecycle and Windows correctness, verified on a real Windows 11
ARM64 VM through the maintained test-infrastructure/vm drivers, plus the
macOS and Linux arm64 suites and the container lint gate.
Daemon lifecycle:
- daemon start/stop/status subcommands. `daemon start` launches a
PERMANENT daemon (spawn shape is byte-exact argv; survives idle
periods and session ends) and reports an already-active daemon
instead of failing. Permanence is honored at every stop latch:
last-committed-client disconnect, host initial-client window,
coordinator release, and application final-session close — a
permanent daemon also keeps admitting new sessions after its last
one closes.
- daemon stop refuses while sessions are active and lists the blocking
peers (pid/role) that must finish first; an idle daemon drains
through the activation-shutdown machinery with the ACK ordered after
connection interrupts. A second stop is idempotent. The wire ops are
no-cohort first-frame requests with peer fingerprint authentication,
so stop/status never conflict with an exact-build admission gate.
- One-shot CLI commands now execute through the daemon (index workers
keep their local supervised path). A cold CLI run that had to spawn a
temporary daemon prints a hint that `daemon start` removes the
per-command startup tax; a warm daemon is recycled silently.
- Hooks are connect-only fail-open: with no daemon present the hook
emits a visible, rate-limited notice (Claude-dialect systemMessage
plus stderr for other dialects) and always exits 0 — augmentation is
never allowed to block the caller's tool use.
- Version skew: a newer-build client automatically drains an
older-build permanent daemon (strict semantic-version triples only;
dev builds never auto-drain) and the build-conflict message names
`cbm daemon stop` as the manual escape hatch.
Windows IPC/runtime (real-VM verified):
- ipc(win): persistent pending overlapped ConnectNamedPipe. The accept
path used to destroy its listening pipe instance on every 20 ms poll
timeout; a client attaching in the teardown window was severed or left
on an orphaned pipe object whose HELLO no server handle could ever
read, absorbing the connect until the client's own timeout expired.
The pending connect now survives poll timeouts and nothing is
destroyed while a client could be attaching.
- ipc(win): drain-before-close for final responses. Closing a named-pipe
server handle can discard a just-sent response before the peer reads
it (POSIX stream sockets never lose buffered data on close). A bounded
cbm_daemon_ipc_connection_drain (read-until-peer-EOF; no-op on POSIX,
immediate on interrupted connections) now precedes close in
runtime_worker_finish and runtime_reject_inline, so hello-conflict,
capacity and disconnect acknowledgements reliably reach the peer.
- runtime: CLOSE_INTENT wire frame. A Windows named-pipe client has no
transport half-close, so close_begin now announces departure with an
explicit frame (ordered after APPLICATION_CANCEL, before the local
interrupt); the server releases the client's admission on receipt
instead of waiting for the handle to close. Admission-drop timing is
now identical to POSIX shutdown() semantics on every platform.
- runtime(win): client close cancellation. close_begin serializes with
request publication under the send lock, best-effort sends the active
token's APPLICATION_CANCEL frame, then interrupts local I/O; the
server cancels MCP/subprocess work promptly. Contract tests accept
both correct outcomes (interrupted transport or decoded CANCELLED).
- runtime: activation acknowledgement ordering. The activation ACK is
the requester's license to act on "snapshotted and draining", so every
connection interrupt is now initiated before the ACK is sent; a
session could previously get one more request serviced after the
requester observed the ACK.
- service(win): deadline-bounded private-file prepare. The conflict-log
prepare retry loop (100 x Sleep(2), which rounds up to the ~16 ms
timer granularity) burned ~1.6 s against permanently obstructed paths,
stalling hello rejections past the client's timeout. The retry budget
is now a 250 ms deadline; transient share collisions still retry.
- subprocess(win): cmd.exe /C payload encoder quotes metacharacters
correctly (root cause of the git-on-Windows failure cluster).
- watcher: SHA-256 buffer sizing (CBM_SZ_64 -> CBM_SZ_128) and a native
Windows stop/unwatch cancellation test with exact-image verification.
- httpd: send_all writes in bounded 64 KiB slices. A single giant
nonblocking send() on Windows is absorbed wholesale into AFD kernel
buffering regardless of SO_SNDBUF, so send deadlines and interrupts
could never engage against a slow peer (and the full payload was
pinned in nonpaged pool). Slicing restores a deterministic
backpressure point; a test hook pins SO_SNDBUF for the deadline and
interrupt tests.
- ui/http: shutdown lifecycle — interrupt checks, response-wide send
deadline, explicit connection states, refusal to free a server while
a listener-owned connection is active.
Windows long-path support:
- Central path-aware wide conversion (canonicalize via GetFullPathNameW
and prepend the extended-length prefix for absolute paths >=240) at
the compat chokepoints (cbm_fopen/compat_fs/mkstemp/mkdtemp), sqlite
store opens, and the daemon build-fingerprint/log paths. Deep managed
installs (a 64-hex generation directory routinely exceeds MAX_PATH)
now index, stage and activate correctly.
- activation transaction: its own file APIs and the component-walking
ancestry validators now operate in the extended-length namespace;
the launcher path is canonicalized (and prefixed when deep) once at
entry so every downstream exact-string comparison stays
form-consistent.
- Executable self-resolution uses the wide APIs (GetModuleFileNameW,
GetFileAttributesW) so non-ASCII install paths survive argv[0]
resolution.
Windows launcher install/uninstall transaction:
- FileRenameInfoEx names are NUL-terminated in an over-allocated
buffer. FileNameLength governs per the contract, but filter drivers
read FileName as NUL-terminated and appended adjacent heap bytes to
created names — a flaky, garbage-suffixed rename target. Both the CLI
and the launcher rename helpers are fixed.
- Uninstall retires state via rename-aside (.cbm ->
.cbm-retired-v1-<tag>-<pid>) with the retired tag shortened to 16 hex
chars so the bare rename target stays under the FileRenameInfoEx
NT-conversion ceiling at guard depths; 64 bits still uniquely
identify the generation.
- When the running launcher's mapped generation backings pin .cbm
against rename, the backings are relocated to activation-<pid>-N
.retired tombstones beside the install (a mapped image may be renamed,
never deleted; the launcher's liveness-guarded sweep reclaims stale
tombstones). Every relocation is recorded, and a FAILED uninstall
reverses the moves after restoring .cbm — via MoveFileExW with
extended-length paths on both arguments, since the deep generation
target is beyond the handle-based rename's bare-path reach — so a
restored install keeps its generation backings and stays runnable.
- After a committed uninstall the retired tree's backings are relocated
out so the tree is shallow enough for the detached cleanup's rd, and
the cleanup's working directory strips the extended-length prefix
(CreateProcessW lpCurrentDirectory silently ignores prefixed paths).
- Files created under Administrators-default-owner directories
(CopyFileW destinations, CREATE_NEW tombstones, probe directories)
are explicitly owner-stamped so the exact-owner validators hold on
runner images; guard fixtures stamp hand-built trees the same way.
Diagnostics, tests and infra:
- diagnostics: discovery is now an always-delivered JSON control record
(new cbm_log_control) that survives CBM_LOG_LEVEL suppression and
paths containing spaces; placement honors $TMPDIR with /tmp fallback
via a diagnostics-local helper; the soak parser reads the JSON record;
documented in docs/CONFIGURATION.md. Red-first coverage for suppressed
log levels, TMPDIR-with-spaces, and native Windows output-contract
assertions.
- tests(win): daemon_ipc/daemon_frontend fixtures now build endpoint
parents with production-shaped ancestry (LocalAppData on Windows, via
th_secure_runtime_parent_new) — the runtime ancestry validation
correctly refuses temp roots whose ancestors grant mutation rights to
Authenticated Users (C:/msys64/tmp, GitHub-runner work dirs) — and
drive the documented startup-owner publication flow before reading
generation-bound endpoint addresses. This turns the 26 Windows
failures previously visible in CI's full-test job green without
weakening any validation.
- tests(win): the launcher guard covers the full permanent-launcher
contract including failed-uninstall restore and immediate reinstall
after uninstall; new daemon lifecycle and reworked hook-augment
guards run the start/recycle/stop flow end to end.
- tests: CBM_SKIP_PERF is now actually consumed by the test runner
(it was set by CI but never read, so perf suites ran everywhere);
four throughput/bench suites are classified as perf, the heavy
store_arch suite moved to the slow-timeout tier, and two
wall-clock-sensitive assertions were rewritten as invariant checks
with coarse hang-detector backstops.
- build/test infra: build-dir safety contract, UI dev-proxy security
contract, soak daemon-recovery contract, path-safety helper, the
Windows VM worktree-sync contract wired into scripts/test.sh, and
vm/win.sh guards building its clean embedded-UI product in an
isolated BUILD_DIR so it cannot clobber the incremental test build.
provision-windows.sh now installs Node.js for the guards UI build.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
On NixOS (and other non-FHS systems) /bin/bash does not exist, so
scripts with an absolute shebang fail to run. Switch the remaining
holdouts to /usr/bin/env bash: eleven scripts/*.sh,
test-infrastructure/run.sh, and the three Claude Code hook scripts
emitted by src/cli/cli.c (gate, session reminder, subagent reminder).
Distilled from PR #674, with parser-test coverage preserved: the
infra_parse_shell* fixtures in tests/test_pipeline.c intentionally keep
#!/bin/bash so absolute-path shebang extraction stays covered, and
tests/repro fixtures are untouched.
Also replace the GitHub-PAT-shaped fixture string flagged in the #674
thread with an obviously fake placeholder (ghp_FAKE...) that still
matches the ghp_ + 36-alnum secret detector.
Co-authored-by: Sandro Jäckel <sandro.jaeckel@gmail.com>
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
#581 (query-path memory growth) does not reproduce in our soak: a 10-min,
200-file query-only run shows RSS flat-to-shrinking and committed plateauing
(mimalloc already purges on free, purge_delay=0). Rather than ship speculative
mitigations for a leak we cannot measure, this adds the observability to find it
where it actually occurs, plus a continuous guard.
- diag: the diagnostics writer now appends a PERSISTENT NDJSON memory trajectory
(one sample / 5s: rss, committed, peak fields, page_faults, fd, queries), kept
on exit (rotates to a .1 generation past 8 MB) so users can send the time
series post-mortem. The previous latest-snapshot file was overwritten every 5s
and deleted on stop.
- ci: the soak now also runs a read-only query-leak leg (CBM_SOAK_MODE query-leak,
no reindex/mutate) on every platform including Windows, so a regression that
introduces a query-path leak is caught. soak-test.sh RESULTS_DIR is now
env-overridable so both legs keep separate artifacts.
- docs: README Troubleshooting and Diagnostics section + a bug-report field
explain the no-telemetry stance and how to capture and share the trajectory.
Build clean; unit suite 5714/0; trajectory verified to persist post-exit.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
- _soak.yml: timeout-minutes was 30 (soak-quick) / 45 (asan) while nightly passes
duration_minutes=240 — every 'nightly 4h soak' was silently KILLED at 30 min and
never ran multi-hour. Raise to 300/60 so the soak can actually complete.
- soak-test.sh: add CBM_SOAK_MODE=query-leak (default unchanged). It indexes once
then hammers read-only tools (search_graph/query_graph/trace_path/
get_code_snippet/search_code) with NO reindex/mutation — so index_repository's
cbm_mem_collect never runs to sweep the query-only leak #581 implicates. The
existing RSS ceiling/slope/ratio checks become the #581 detector.
- soak.yml: workflow_dispatch (duration_minutes, mode) + push to qa/soak-** ;
builds the prod binary and runs the soak with timeout = duration + 60, on
ubuntu + macos. Pushing a qa/soak-* branch starts a real multi-hour run.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
- Add CBM_SKIP_PERF=1 env var to skip incremental/perf test suite
- CI and Docker test targets skip perf by default (run.sh perf for manual)
- Convert all perf assertions to warnings (log timing, never block)
- Fix store.h anonymous enum in struct (GCC rejects, clang accepts)
- Fix test_store_search.c mkstemp on non-template path
- Add ca-certificates to Docker test image for git HTTPS
- Add cbm_gmtime_r shim in compat.h (Windows gmtime_s wrapper)
- Fix compat.c missing constants.h include (Windows build)
- Fix platform.c _environ redeclaration on mingw
- Rename trace_call_path -> trace_path in smoke/soak/fuzz scripts
- Phase 14: copy binary with .exe suffix on Windows (was creating
non-.exe copy that uninstall didn't remove)
- Phase 14e: check both .exe and non-.exe paths
- Soak latency: exclude index_repository from max query latency
(indexing 377 files is legitimately slow on Windows/MSYS2)
MinGW python3 is a native Windows binary that doesn't understand
POSIX paths like /tmp/foo.json. Piping file content through cat
(which runs in MSYS2 bash and handles path translation) to python's
stdin avoids the issue entirely.
Fixes Windows smoke test 8a and soak baseline collection.
Investigation: macOS arm64 grew 1MB (43→44MB) but slope was 1123
KB/hr. Linux amd64 grew 3MB (25→28MB) but slope was only 424 KB/hr.
The linear regression fits noise on short runs — small timing
fluctuations produce wildly different slopes on nearly flat data.
For 10-min quick soak: use RSS ceiling (200MB) + ratio (3.0x) +
FD drift + idle CPU. Slope is still reported but not enforced.
For 30+ min runs: slope enforcement at 500 KB/hr (enough samples
for reliable regression).
- Test project: 377 files (80 Python + 40 Go + 40 TSX + configs)
instead of 5 files. RSS baseline now ~47MB (real workload).
- Snapshots every 10s (was 30s) — 7+ data points in 1-min run.
- Reindex every 2min compressed (was 5min) — more cycles per run.
- heap_committed fallback: use RSS when mimalloc reports 0.
- Windows soak: added to quick soak matrix (MSYS2 + python3 + git).
- ASan soak: builds with -fsanitize=address (was building release).
- Collect snapshot with single python3 call (was 6 separate calls).
Martin Vogel