Martin Vogel
0fe0f453be
Every lane here was either asserting coverage it did not have, or was reachable only by bypassing the entry point that is supposed to define the ladder. TSan: no exclusions any more. The three suites the Makefile documented as excluded are back. daemon_ipc and daemon_frontend no longer reproduce the harness race and thread leaks they were excluded for. daemon_runtime did not deadlock as the comment claimed -- it reported a real production data race on the log sink, fixed separately. Excluding a suite from a sanitizer lane hides exactly the class of bug the lane exists to find, so the comment block now records what was actually true rather than what was assumed. TSAN_OPTIONS gains report_thread_leaks=0. This disables the thread- HYGIENE check only; race detection is untouched. Several daemon fixtures fork after the process has gone multi-threaded, and in the forked child TSan sees the parent's already-finished threads as never-joined even where the fixture joins them. It fires on macOS and not Linux, i.e. it tracks fork semantics rather than anything about this code. The alternative was dropping whole suites, which costs real race coverage; this costs none. UBSan: tre.o no longer builds with -fno-sanitize=alignment. Alignment was switched off for a vendored regex engine that ships in the product, which is where the check is least redundant, not most. LSan on macOS: new test-lsan target and test-lsan-macos CI leg. LSan is on by default under ASan on Linux, so the Linux legs have always had leak coverage. On macOS it is off by default and Apple's clang refuses to enable it outright, so that platform had none at all. Apple's refusal is not a darwin limitation -- upstream LLVM supports LSan on darwin/ arm64. The lane is the ordinary ASan suite built with Homebrew LLVM and run with detect_leaks=1; it runs the full suite clean and was checked to still catch a deliberately leaked allocation. MSan: reachable from the local ladder. The image and compose service existed but run.sh had no leg, so the only way in was to drive docker compose by hand -- which means it was not part of the ladder in any meaningful sense. The image also moves to clang 22, matching the diag and analyzer lanes instead of sitting four majors behind on noble's default. The leg documents the aarch64 shadow-mapping failure so a local arm64 stack overflow in the grammar suites is not mistaken for a code defect; the GitHub leg runs x86-64, which is the mapping that matters. Off-by-default ASan checks: the diagnostic lane, and its CI twin, now run detect_stack_use_after_return, detect_stack_use_after_scope, detect_invalid_pointer_pairs (with the -fsanitize=pointer-compare, pointer-subtract instrumentation it requires) and strict_string_checks. Running ASan is not the same as running all of it, and these four cover bug classes nothing in the matrix was looking for. They stay on the diagnostic lane rather than the gating ones until they have a clean history there; promoting them is a separate deliberate step. Verified: macOS TSan 940 passed / 3 skipped / 0 races over the full suite set; the macOS leak lane 7375 passed / 4 skipped / 0 leaks, with LeakSanitizer confirmed armed under that exact toolchain and option set by checking it still reports a deliberately leaked allocation. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
test-infrastructure/ — the local venues
The local half of the 3-OS CI ladder. Every leg here runs the SAME canonical
scripts CI runs (scripts/test.sh, build.sh, smoke-local.sh,
soak-legs.sh, vm-smoke.sh — see scripts/README.md); this directory only
provisions containers and the Windows VM. The venue-parity contract
(tests/test_venue_parity_contract.sh, Step 0j of every test leg) fails the
build if a venue grows its own harness logic — here or in the workflows.
Entry points
| venue | entry | notes |
|---|---|---|
| Linux (arm64 + amd64) + cross-compile | ./run.sh <leg> (--help for the leg list) |
Colima only — Docker Desktop is broken on this machine. A clean-disk preflight (scripts/ci/preflight-docker.sh) runs before every leg: GitHub runners are ephemeral with a known-free 14 GB disk, so the long-lived VM is swept back to that shape first. |
| Real Windows (UTM ARM64 VM) | vm/win.sh <command> (win.sh help) |
Same preflight idea (scripts/ci/clean-test-residue.ps1), plus CI's protected per-user TEMP root. Wine (run.sh windows) is a compile check only — never a substitute for the VM. |
| macOS | run the canonical scripts natively | scripts/test.sh, scripts/build.sh, scripts/smoke-local.sh … |
The ladder (before any push)
scripts/test.sh— macOS native (or--suites …while iterating)./test-infrastructure/run.sh full— Linux arm64 test/build/TSan/smoke/portable + mingw cross-compile (+amd64,soak-linuxas needed)vm/win.sh test-par·guards·smoke-install·soak— the real-Windows legs
Infra unavailable = a run blocker to escalate — never a silent bypass.
Fidelity guarantees (what makes local predictive of CI)
- Same scripts, same sequence, same flags — platform specifics (e.g. the CLANGARM64 trap-UBSan default) live inside the canonical entries, applied identically everywhere.
- Same environment shape — protected TEMP roots, sandboxed smoke, clean builds with the content-verified ccache (a hit is byte-identical to a cold compile; caches only accelerate, never change results).
- Same starting disk — preflights sweep residue and BLOCK below the runner's 14 GB floor (a quietly full disk fails inside install paths and masquerades as a product bug).
- Known, deliberate residuals — runner physics (4 vCPU vs local cores, shared tenancy), Defender ON in the VM vs OFF on GitHub runners, no TSan on Windows anywhere, no ASan runtime on native ARM64 Windows (trap-UBSan + PageHeap stand in).
See vm/README.md for VM provisioning and day-to-day VM mechanics.