The 8e gate compared uname -s to the bare literal MINGW64_NT, which
never matches the real value (MINGW64_NT-10.0-...), so the POSIX
branch silently ran on Windows and looked for the extensionless shim
that #929 no longer installs. Branch by platform prefix: Windows
expects the .cmd shim and the absence of the legacy extensionless
twin; POSIX unchanged. Content checks (never blocks, delegates to
hook-augment) now run on both.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
cbm_writer_open used a raw ANSI-CP fopen for the hand-rolled SQLite
writer - the one file-creating call on the dump chain without a
UTF-8-to-wide conversion. On a non-ASCII Windows profile
(C:\Users\Kovacs Janos) extract and resolve succeeded but the dump
failed with pipeline.err phase=dump: fopen could not create graph.db
under the profile-derived cache dir while cbm_mkdir_p (already wide)
had happily created the directory around it. Routed through cbm_fopen,
matching #700 (pass readers) and #973 (canonicalization); POSIX is
byte-transparent.
New Windows guard test_non_ascii_cache_dump.py indexes an ASCII repo
into a non-ASCII CBM_CACHE_DIR and requires a query_graph readback
with Function count > 0 - proving the DB was written to and reopened
from the non-ASCII cache, not merely that no error surfaced. The
existing non-ASCII guard only varied the repo path against an ASCII
cache, which is why this class went unseen.
The USERPROFILE-sourced env encoding question (Layer 2 - _environ is
ANSI-CP, not UTF-8) is deliberately not bundled; it needs verification
on a real Windows box and will be filed separately if the guard alone
does not settle it.
Closes#996
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
SIGABRT (free of a non-libmalloc pointer in ts_stack_delete) on the
second index_repository in one process, whenever a sequential run is
followed by a parallel one. The sequential path parses on the CALLING
thread and left its TLS parser alive; the next parallel run switches
the global tree-sitter allocator to the slab (cbm_slab_install), the
calling thread participates in extraction, reuses the stale parser,
and its per-file teardown frees mimalloc-epoch memory through
slab_free's foreign-pointer fallback -> plain free() -> libmalloc
abort. The v0.9.0 supervisor masks this on the default MCP path (fresh
worker process per index); the in-process path and every embedded
consumer died. Root-caused with allocator-epoch instrumentation: the
victim parser is created at epoch 0 in cbm_pipeline_pass_definitions
and deleted at epoch 1 in extract_worker.
Two boundary fixes:
- run_sequential_pipeline destroys the calling thread's parser at run
end (also bounds its memory like the parallel workers already do);
- cbm_parallel_extract destroys the calling thread's parser BEFORE
cbm_slab_install, so no parser ever crosses the allocator switch.
Guards: prod-tier smoke invariant inv_second_index_inprocess
(sequential-then-parallel double index over stdio, RED with signal 6
on the unfixed binary — ASan builds mask the allocator seam, so the
unit tier cannot catch this class) + suite-tier fork-isolated
index_second_inprocess_run_survives_issue773 pinning the in-process
double-index contract. Repro 5/5 clean after the fix (was 5/5 abort).
Closes#773
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Completes the compact-output pass across the remaining query tools
(consumers are LLM agents; every response byte is context tokens):
- trace_path: callees[N]{qn,hop}/callers[N]{qn,hop} tables (risk/test/
args columns per flags); the per-hop JSON key envelope was 84% of the
payload. 3,589B -> 1,634B on a representative trace.
- query_graph: rows[N]{cols}: table (columns double as the header);
format:"json" restores columns/rows arrays.
- search_code compact mode: results/raw/dirs tables; drops the
field (duplicate of the qn tail). 2,488B -> 1,135B for 17 hits.
- search_code full mode: per-hit source capped at a 60-line window
anchored on the first match, with source_start/source_truncated
markers — uncapped whole-symbol dumps ran to 142KB per response; the
complete symbol stays one get_code_snippet call away.
- get_architecture: default (no aspects) is now a summary (languages,
packages, entry_points + always-on totals/label/type counts) with an
aspects_hint — the old default rendered EVERYTHING including the full
file_tree (94KB -> 4.5KB -> 2.8KB with TOON tables); all sections
emit as TOON tables, aspects:[...] and ["all"] keep full access
(52KB vs 94KB even for the explicit full dump).
- get_code_snippet: drops the property-blob enrichment (41% of the
response; signature/docstring are literally in the source; fp/sp/bt
never belonged there). Metrics remain reachable via search_graph
fields=[...].
- list_projects: per-project 12-field git block (mostly null) replaced
by the branch name; name/root/nodes/edges/size kept.
- search_graph description: in/out documented as TOTAL all-edge degree,
NOT caller/callee counts (agent evals mis-read it as callers).
- format:"json" escape hatch on every converted tool.
Guards: tool_output_byte_budgets pins absolute byte ceilings on default
search/trace outputs (a property blob sneaking back into row emission
blows them immediately); snippet_enriched_properties inverted to pin
the no-spill contract. Suite 5,984 green; smoke updated to the TOON
contract end-to-end.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Tool responses land in an LLM agent's context window, where the legacy
per-node JSON objects were dominated by data no agent can use: the
similarity/semantic pipeline intermediates fp (~450B minhash hex), bt
(body-token bag) and sp (structural profile) plus 14 complexity metrics
were grafted onto every result by enrich_node_properties — ~64% of every
node emission, ~370 tokens per search hit.
search_graph now defaults to TOON (Token-Oriented Object Notation,
toonformat.dev): scalars as 'key: value' lines and results as a
'results[N]{qn,label,file,lines,in,out}:' header + one row per hit.
Published benchmarks show equal-or-better LLM retrieval accuracy at
40-60% fewer tokens than JSON; rows now also carry line ranges, which
the legacy output lacked.
- fields:[...] opts into per-node property columns (complexity,
signature, docstring, ...); fp/sp/bt are blocklisted even there
- format:"json" restores the legacy verbose objects byte-identically;
include_connected forces it (nested neighbor lists)
- BM25 and semantic modes emit the same table shape (rank/score column)
- semantic-only calls no longer run the unfiltered regex search that
silently prepended up to 200 enriched nodes (102.8KB -> 4.2KB)
- default limit 200 -> 50 (BM25 100 -> 50): cheap first page, page via
has_more/offset as before
- new emitter module src/mcp/compact_out.{c,h} (string builder + TOON
scalar/table emission with spec quoting rules)
Measured on a real index (14-hit regex search): 20,793B -> 1,327B
(-94%); BM25 19,667B -> 5,081B; semantic 102,837B -> 4,181B. A typical
discovery session drops ~50.9K -> ~4.7K tokens combined with the
follow-up slices.
Guards: tool_search_graph_toon_never_leaks_internal_fields (sentinel
fp/sp/bt never emitted, even when requested), rewritten
tool_search_graph_includes_node_properties pins default-lean/fields-
opt-in/json-escape-hatch; suite 5983 passed, smoke green.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Perl LSP landed in #461; scan its implementation against PerlNavigator
(MIT, the leading OSS Perl language server) for defensive
copy-detection, same rationale as the phpactor entry. Verified clean:
no verbatim string/comment overlap.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Check 8d still locked the matcher to the pre-#963 'Grep|Glob' and failed
the pr-smoke leg on all platforms. The matcher now includes Read (the
augmenter injects the coverage note when a not-fully-indexed file is
read; structurally non-blocking, so the issue-#362 gate hazard cannot
recur). The guard now pins the exact new matcher and still rejects
Search/catch-all creep.
Refs #963
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
install.ps1 kept detecting arch=amd64 on windows-11-arm because it runs under x64
emulation, where neither $env:PROCESSOR_ARCHITECTURE nor .NET OSArchitecture (on
Windows PowerShell 5.1) reports the real Arm64. Add a CBM_ARCH env override to
install.ps1 (wins over auto-detect; also a genuine escape hatch for emulated
invocations) and have smoke Phase 13 pass the authoritative DL_ARCH. Deterministic
-- no reliance on in-process detection under emulation. Same emulated-arch class
as #907 (uname) and #908.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The persistent windows-11-arm smoke Phase 12a failure was a 404, not a network
error: it requested codebase-memory-mcp-windows-amd64.zip on the arm64 leg. Cause:
DL_ARCH came from `uname -m`, which on windows-11-arm is an emulated x86_64 MSYS2
uname reporting "x86_64" -> wrong (amd64) archive -> 404 (server has arm64). Phase
14 worked because the binary's own detect_arch() is native. Prefer SMOKE_ARCH
(passed from the smoke workflow's matrix.arch) over uname; fall back to uname for
local runs. This is the real cause the earlier curl/proxy/ipv4 attempts masked --
the 404 was swallowed by 2>/dev/null until #905 surfaced it.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Phase 12a swallowed curl's stderr (2>/dev/null), so the persistent windows-arm64
"curl download failed" was undiagnosable. Two changes: (1) add --noproxy '*' so
curl never routes the local 127.0.0.1 test server through a proxy env var -- a
strong candidate since the app's own WinHTTP downloader reaches the server in
Phase 14 while only msys2 curl fails, instantly; (2) print curl's stderr on
failure so the actual reason is visible if it persists. Harmless on all
platforms (the smoke server is always local).
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Closes#423, #20. A non-ASCII repo path passed to
`codebase-memory-mcp cli index_repository "<json>"` was mangled on Windows, so the
command failed with "repo_path is required" instead of indexing the real directory.
POSIX is unaffected (argv is already UTF-8). Two boundaries had to be fixed:
1. argv read: main() took only the narrow `int main(int argc, char **argv)`, so the
CRT handed it argv in the active ANSI code page. On Windows main() now rebuilds
argv from the wide command line (GetCommandLineW + CommandLineToArgvW) and
converts each element to UTF-8 (cbm_wide_to_utf8).
2. worker spawn: the index supervisor (and the UI index thread) spawned the worker
via CreateProcessA, which re-interpreted the UTF-8 command line through the ANSI
code page -- re-mangling the path at the parent->worker boundary even after (1).
Both spawns now build a wide command line (cbm_utf8_to_wide) and use CreateProcessW.
Without (2) the default (supervisor-enabled) path stayed broken while only the
in-process path worked, so this is the complete end-to-end fix, not just argv.
Promotes tests/windows/test_cli_non_ascii_arg.py from an opt-in known-red to a green
Windows guard (scripts/test-windows.ps1 + the test-windows-guards CI job). The guard
now exercises the DEFAULT supervised path (it drops the suite's CBM_INDEX_SUPERVISOR=0
so it actually crosses the worker-spawn boundary). Verified on native Windows: RED
before the fix, GREEN after (a non-ASCII repo indexes, nodes>0). Also adds -lshell32
explicitly (main.c now uses CommandLineToArgvW) and refreshes RED_TEST_ANALYSIS.md.
Signed-off-by: Flipper <jacobphilipp@ymail.com>
libgit2 is licensed GPLv2-with-linking-exception, and this project is
deliberately GPL-free (see scripts/license-policy.json). libgit2 was only
an OPTIONAL, faster git-history code path guarded by HAVE_LIBGIT2 and
auto-detected via pkg-config; it always shipped with a popen("git log ...")
fallback that release binaries already used (they were built without
libgit2). Making that popen fallback the SOLE git-history implementation
means there is NO change to shipped behavior.
Changes:
- pass_githistory.c: collapse the HAVE_LIBGIT2 #ifdef so only the popen
parse_git_log remains; drop the now-unused <git2.h>/<time.h> includes.
- cbm.c / cbm.h / main.c / subprocess.c / index_supervisor.h: remove the
libgit2 mimalloc allocator bind and its >=1.7.0 version guards; keep the
tree-sitter + sqlite3 binds; correct the comments.
- Makefile.cbm: remove the optional-libgit2 pkg-config detection block,
the REQUIRE_LIBGIT2 error gate, and all LIBGIT2 CFLAGS/LIBS/FLAGS.
- .github/workflows/_test.yml: drop the REQUIRE_LIBGIT2 test-matrix leg
plus its libgit2-dev/pkg-config apt install (lower CI cost, no new
gating, no trigger change).
- flake.nix: drop libgit2/pkg-config from the dev shell (now-dead dep).
HAVE_LIBGIT2 no longer exists anywhere in the tree.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Distilled from #789 (safe features only). Adds backend dead-code classification
(status + in_calls in the layout JSON), a GET /api/repo-info endpoint for GitHub
deep-links, and the frontend dead-code filters, node code preview, and deep-links.
The render-cap revert was dropped (kept 2000 for DEFAULT_MAX_NODES/HARD_MAX_NODES
and GRAPH_RENDER_NODE_LIMIT); the sidebar regex-search refactor was omitted.
Security fixes over the original: repo-info strips credentials from any returned
remote_url; the legitimate https blob-URL construction is allow-listed so the
static gate passes; libgit2 is not re-initialized/shutdown per request (reuses the
process-wide init from cbm_alloc_init); deep-link path segments are URL-encoded.
Co-authored-by: Andy Zehady <azehady@ciroos.ai>
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The index supervisor gate (cbm_index_supervisor_should_wrap) wrapped ANY
embedder of cbm_mcp_handle_tool: it resolved the CURRENT executable and
re-invoked it as '<self> cli --index-worker index_repository ...'. In a
test binary 'self' is the test runner, which ignores those args and
re-runs its suites instead, producing recursive spawn chains (observed
11-minute hangs, and multiplied kernel VM-map/memory load during the
2026-07-04 host panics). The gate is now opt-in: only a process that
called cbm_index_supervisor_mark_host() — the real binary's main(),
before any subcommand dispatch — may spawn workers. Embedders index
in-process by default; the test mains additionally pin
CBM_INDEX_SUPERVISOR=0 as belt-and-suspenders. Guarded by
index_supervisor_gate_requires_marked_host_issue845 (fork+alarm harness,
new spawn-count test hook; red on the old gate, green without the env
kill switch on the new one).
A supervised worker whose supervisor died also kept indexing as an
orphan: the parent-death watchdog only ran in MCP-server mode. run_cli
now starts the same watchdog (detached; on ppid change it only writes
to stderr and _exit(0)s, so it is safe outside server mode) when it
detects --index-worker. Guarded by tests/test_worker_watchdog.sh
(scripts/test.sh Step 5b): a CBM_TEST_HANG_ON busy-spin pins the worker
mid-index, the wrapper parent is killed, and the worker must self-exit
within 15s — red without the wiring, green with it.
Fixes#845.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Rebased onto current main and reworked in response to review. Three of the four
Windows reds were fixed upstream since the branch was cut at b075f05, so they are
now green regression guards; the fourth stays a genuine known-red.
- test_non_ascii_path.py (#636/#357): green guard - fixed by #700 (per-pass
readers now route through cbm_fopen -> _wfopen). Re-verified green on main.
- test_hook_augment.py (#618): green guard - fixed by #619 (cbm_is_walkable_abs_path
accepts drive-letter X:/ cwd). Re-verified green on main.
- test_ui_drive_listing.py (#548): rewritten. The fix exposes drives via a new
roots[] field, not the dirs[] array the old test asserted (which would stay red
against fixed code). Now asserts every fixed drive is in roots and browsable.
Re-verified green on main (drives C:/D:/E:).
- test_cli_non_ascii_arg.py (#423/#20): unchanged - main() is still narrow-argv
with no wide command line, so this remains genuinely red (the keeper).
- scripts/test-windows.ps1: split green guards (gate CI) from opt-in known-reds;
add -GuardsOnly; run indexing in-process (CBM_INDEX_SUPERVISOR=0) so a guard
reflects the path/hook/drive fix under test, not the index-worker spawn path.
- .github/workflows/_test.yml: new test-windows-guards job builds the product+UI
binary (scripts/build.sh --with-ui) and runs the guards with -GuardsOnly so
#700/#619/#548 stay enforced on Windows CI.
- RED_TEST_ANALYSIS.md: refreshed to record the landed fixes and current status.
Signed-off-by: Flipper <jacobphilipp@ymail.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Extends the Windows red-test suite with two more reproduced, Windows-specific
failures. No production fixes.
- windows_hook_augment_emits_context (integration, #618): the PreToolUse
Grep/Glob augmenter `hook-augment` emits empty stdout for every payload on
Windows. src/cli/hook_augment.c gates on POSIX-style absolute paths
(cwd[0] == '/' and the walk-up loop's dir[0] == '/'), which a Windows
drive-letter cwd never satisfies, so the graph augmentation never fires. A
control search_graph confirms the symbol is indexed.
- windows_ui_picker_reaches_all_drives (integration, #548): the UI directory
picker's GET /api/browse?path=/ returns no entries and never enumerates
logical drives, so drives other than the system drive (D:\, E:\) cannot be
selected. handle_browse in src/ui/http_server.c uses opendir without a
GetLogicalDriveStrings root case. Needs a UI build (cbm-with-ui) and >1 drive;
otherwise it reports a precondition (exit 2).
Also records additional ruled-out seed areas (get_code_snippet sanitizes
non-UTF-8 to U+FFFD #530.3; stdio handshake/flush works #513/#530.1/#635; mapped
subst-drive indexing keeps the DB #227/#367) and cross-platform items left out
of this Windows-only PR (#530.2 nested gitignore, #530.5 .git/info/exclude,
#530.4 libgit2 build, #581 memory soak).
Signed-off-by: Flipper <jacobphilipp@ymail.com>
Adds Windows-only red tests and analysis for native Windows failures found
during a Windows red-test campaign. This change contains no production fixes.
- windows_non_ascii_repo_path_preserves_definitions (integration): byte-identical
TypeScript fixtures indexed under non-ASCII parent paths (Latin-1 accents,
Cyrillic, CJK, Greek) extract zero definitions and only File/Folder nodes
(5 nodes / 4 edges) versus the ASCII baseline (12 nodes / 20 edges / 5
definitions). The pipeline source readers open files with fopen() on a UTF-8
path, which the Windows CRT interprets in the active ANSI code page; directory
discovery already uses the wide API, so files are listed but never parsed.
- windows_cli_non_ascii_repo_path_is_honored (integration): the documented
`cli index_repository` entrypoint rejects a non-ASCII repo_path because main()
does not read the wide command line, so argv arrives in the ANSI code page.
Both reproduce at the product surface (real MCP process, real stdio, real SQLite
DB), are deterministic, and pass on Linux/macOS. A PowerShell runner builds the
binary and runs the suite; standard-library Python only. See
tests/windows/RED_TEST_ANALYSIS.md for environment, commands, ruled-out seed
areas, and suspected fix locations.
Signed-off-by: Flipper <jacobphilipp@ymail.com>
- format the include trailing comments and wrap the over-long reap log line
(clang-format-20 violations flagged by lint / lint)
- allowlist subprocess.c's fork with justification (security-static): the
supervisor primitive forks and execs immediately; no code runs in the
forked image
- move bin into the fork-child block alongside argv (cppcheck variableScope
consistency)
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
On NixOS (and other non-FHS systems) /bin/bash does not exist, so
scripts with an absolute shebang fail to run. Switch the remaining
holdouts to /usr/bin/env bash: eleven scripts/*.sh,
test-infrastructure/run.sh, and the three Claude Code hook scripts
emitted by src/cli/cli.c (gate, session reminder, subagent reminder).
Distilled from PR #674, with parser-test coverage preserved: the
infra_parse_shell* fixtures in tests/test_pipeline.c intentionally keep
#!/bin/bash so absolute-path shebang extraction stays covered, and
tests/repro fixtures are untouched.
Also replace the GitHub-PAT-shaped fixture string flagged in the #674
thread with an obviously fake placeholder (ghp_FAKE...) that still
matches the ghp_ + 36-alnum secret detector.
Co-authored-by: Sandro Jäckel <sandro.jaeckel@gmail.com>
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Completes the crash/hang supervisor. A file that makes the indexer make NO
progress (an external-scanner infinite loop, not a crash) is now contained the
same way a crash is: the supervisor's no-progress quiet-timeout kills the worker,
classifies it as a hang, and the recovery loop quarantines the marker-pinned file
and re-runs until a clean run indexes the good files.
- src/mcp/index_supervisor.c: worker_quiet_timeout_ms() defaults to a generous
15 min NO-PROGRESS timeout (reset by the worker's periodic log heartbeat), with
the CBM_INDEX_WORKER_TIMEOUT_S override. (--progress deliberately NOT added: it
installs a replace-mode sink + carriage-return updates that would remove the
newline heartbeat the quiet-timeout relies on.)
- Quarantine entries now carry the phase ("rel\tphase"): a fault signal -> "crash",
a no-progress kill -> "hang". internal/cbm/cbm.c parses it (bare path tolerated,
defaults crash) and exposes cbm_index_quarantine_phase(); the extract loops
(pass_parallel.c, pass_definitions.c) report the real phase + reason in skipped[].
- src/mcp/mcp.c: recovery loop logs the outcome, and a best-effort-PARTIAL terminal
spawn — when the loop cannot converge but files were quarantined, one final
quarantine-only run yields a partial index (good files indexed, known-bad ones
skipped) + a high-severity index.supervisor.partial log, instead of a hard error.
- scripts/smoke-invariants.sh: inv_hanger_skipped_cli (hang twin of the crasher
guard — baseline rc=124 vacuity check; supervised -> status indexed + hanger
phase=hang + good file indexed). Hardened run_bounded to force-kill (SIGKILL) a
SIGTERM-catching busy-spin and to not block on the fallback fifo open — the hang
guard is the first invariant to actually hang and exposed this latent harness bug.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Stage 3b contained a crash to the worker but returned a failure response. Now the
supervisor pins the exact crashing file, quarantines it, and re-indexes the
healthy files: index_repository returns status="indexed" with the crasher listed
in skipped[] (phase="crash") and every other file indexed normally.
On a crash/hang the supervisor re-runs the worker single-threaded with a per-file
marker (CBM_INDEX_MARKER_FILE) so the marker names the EXACT crasher; that file is
appended to a quarantine list (CBM_INDEX_QUARANTINE_FILE) and the worker is
re-spawned until a clean run indexes the good files. Bounded by
CBM_INDEX_MAX_RESTARTS (default 100); each crash quarantines one new file, so the
loop makes monotonic progress.
- internal/cbm/cbm.c/.h: per-file marker + a lazily-loaded crash-quarantine set,
with a HARD guard at the top of cbm_extract_file (the one choke point every
extract pass funnels through) that short-circuits a quarantined file to an empty
result BEFORE the marker/parser — so no re-extracting pass (sequential
pass_calls/usages/semantic on a cache miss) can crash on it again.
- src/pipeline/pipeline.c: effective_worker_count() honours CBM_INDEX_SINGLE_THREAD
(=1 worker, sequential path) for exact single-threaded attribution.
- src/pipeline/pass_parallel.c, pass_definitions.c: report a quarantined file as a
phase="crash" skip via the Stage-2 skip plumbing so it surfaces in skipped[].
- src/mcp/index_supervisor.{h,c}: cbm_index_spawn_worker gains single_thread /
marker / quarantine knobs, passed to the child as inherited env around the
sequential spawn.
- src/mcp/mcp.c: index_run_supervised recovery loop (attribute -> quarantine ->
re-run until clean; else a contained-failure response).
- scripts/smoke-invariants.sh: inv_crasher_skipped_cli upgrades the 3b guard to the
skip-continue contract (baseline crash escapes; supervised -> status indexed +
crasher phase=crash + good file indexed + good file NOT skipped).
In-run quarantine only (not persisted across separate index runs); content-hash
persistence + re-admit is a follow-up.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
tools/list is cursor-paginated (MCP_TOOLS_PAGE_SIZE=8): the server returns a
page of tools plus result.nextCursor, and a compliant client follows the cursor
to retrieve the rest. inv_tools_list only read the first page and expected all
14 tools in one response, so it failed with "got 8, expected 14" even though
the server is correct. Follow nextCursor across pages and assert the union.
No server change — the pagination behavior was already correct.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
A single pathological file that hard-crashes the native indexer (SIGSEGV / stack
overflow / abort / tree-sitter assertion) previously took down the whole MCP
server or CLI (issue #668). index_repository now runs in a worker subprocess: the
parent reaps it, and a crash is contained — the parent survives and reports it
instead of dying.
- src/mcp/index_supervisor.{h,c}: worker-role state + cbm_index_spawn_worker(),
which spawns `<self> cli --index-worker index_repository <json> --response-out
<tmp>` via the subprocess primitive, reaps, and classifies. fork+exec only;
recursion is prevented by the --index-worker argv flag, never an ambient env
var (which could misfire and silently run unprotected in-process).
- src/mcp/mcp.c: a supervisor gate at the top of handle_index_repository wraps the
run unless this process is the worker or CBM_INDEX_SUPERVISOR=0 (kill switch).
Clean exit passes the worker's response through; a crash/hang returns a
contained-failure response; a spawn failure degrades to the in-process path.
- src/main.c: parse --index-worker / --response-out; the worker writes its result
to the response file for the parent to read back.
- src/ui/http_server.c: its existing index spawn now passes --index-worker so it
does not re-supervise (one isolation layer, no redundant process nesting).
- src/foundation/subprocess.c: the forked child redirects output with open()+dup2()
instead of freopen() — async-signal-safe, no malloc between fork and exec, since
the parent may be multithreaded (server threads + mimalloc/sqlite/libgit2 state).
- internal/cbm/cbm.c: CBM_TEST_CRASH_ON / CBM_TEST_HANG_ON deterministic fault
injectors (test-only, env-gated) so the guard is honest — green iff a real fault
is genuinely contained, not a fixture that may stop faulting.
- scripts/smoke-invariants.sh: inv_crasher_contained_cli asserts the crash escapes
as a signal without the supervisor (rc>=128) and is contained + reported with it
(rc<128, outcome=crash).
Precise skip-and-continue (quarantine the culprit file, index the rest) is layered
on in a follow-up; this change contains the crash and keeps the process alive.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Companion to the raw-JSON deprecation warning committed on this branch (adds the
smoke coverage that was meant to land with it).
Migrate scripts/smoke-test.sh from raw-JSON `cli <tool> '{...}'` invocations to
FLAG form for every tool (index_repository / search_graph / trace_path /
get_graph_schema / query_graph incl. all Cypher checks / get_architecture /
search_code / delete_project), with identical results verified against a prior
run; flag form passes the Cypher query as one argv token, so the old
JSON-escaping helper is removed. Add dedicated guards for an integer flag
(--limit / --depth), a bare boolean (--exclude-entry-points), a repeated array
flag (--semantic-query -> JSON array), piped stdin, --args-file, and per-tool
--help (RC 0 + shows a flag; unknown tool -> RC!=0), plus the deprecation guard
(raw JSON warns on stderr; flag form does not). Non-CLI phases (incl. Phase 10
binary security) unchanged. Full smoke: ALL PASSED.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
verify_compiler() in scripts/env.sh ran `file` on the clang path and
grepped for arm64/universal. Nix's clang is a bash wrapper script, so
`file` reports "ASCII text executable" and the check falsely concluded
"x86_64 only", aborting before any compile. Replace it with a capability
probe: compile + link a trivial program for the target -arch and trust
the result — correct for wrapper scripts (Nix, ccache) and cross-compilers.
Make arch targeting explicit and toolchain-agnostic: export
ARCHFLAGS="-arch <arch>" from env.sh and fold it into CC/CXX in
Makefile.cbm via `override`, so it reaches every compile and link
(including the vendored objects) and survives a command-line CC=
override. This replaces the `arch -<arch> make` prefix, which cannot
work with Nix's fixed-target clang. Drop the now-redundant ARCH_PREFIX
from build/test/lint/repro.
Refs #705.
Signed-off-by: Kris Williams <115474+kriswill@users.noreply.github.com>
The "tampered arm64 binary is SIGKILLed (137)" premise is empirically false on
current macOS CI runners for an ad-hoc-signed CLI binary -- the binary has no
CS_KILL/hardened-runtime flag, so a tampered code page is not killed: it
executes the garbage and crashes with SIGILL (exit 132, run 28365724001), not
137. (remove-signature and corrupt-blob both ad-hoc re-sign on exec and run to
exit 0.) So no runtime exit code is a deterministic guard here, and
"tamper -> crash" is near-tautological (zeroed code crashes regardless of
signing).
Assert the real, deterministic integrity invariant instead: `codesign
--verify` REJECTS a tampered copy (the CodeDirectory page hashes no longer
match the modified code), while the untampered binary verifies cleanly (10a).
It is a pure userspace hash check -- no tampered code is executed. The copy is
separate, so the original binary stays intact for the 10e re-sign test.
Refs: github.com/garrytan/gstack#997, github.com/nodejs/node#40827
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Refines the previous 10c change on this branch. Garbling only the signature
blob still ran (exit=0 on dry-run 28360363173): since macOS 11 a binary with
a missing/invalid signature is ad-hoc re-signed on exec by newer macOS and
RUNS, so neither remove-signature nor a corrupt blob triggers the kill.
The reliable "tampered binary is SIGKILLed (137)" trigger is tampering the
SIGNED CODE while leaving the valid signature attached: the kernel validates
each executed page against the intact CodeDirectory hash, finds the mismatch,
and kills the process before user code runs. Zero the entry-point
instructions (LC_MAIN entryoff, extracted dynamically) plus a span of early
__text on a SEPARATE copy, leaving the Mach-O header + load commands intact so
it still parses. The original binary is untouched, so the later 10e re-sign
step stays valid. x86_64 keeps remove-signature (code signing is not enforced
there).
Refs: github.com/garrytan/gstack#997, github.com/nodejs/node#40827
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Smoke test 10c (Phase 10 binary security E2E) verifies that an arm64 binary
with an invalid code signature is SIGKILLed (exit 137). It did
`codesign --remove-signature` then ran the binary, expecting the kernel to
kill it. But since macOS 11, a binary with NO LC_CODE_SIGNATURE is ad-hoc
re-signed on exec by newer macOS and RUNS (exit 0) -- so the test went flaky
and then consistently red on updated CI runner images (dry-runs 28350650225
and 28354735368 both failed only here; every other job, including the full
cross-platform test matrix, passed).
Corrupt the signature blob in place instead, leaving the LC_CODE_SIGNATURE
load command intact: AMFI then sees "signed but invalid" and rejects the
binary before any user code runs (deterministic 137). Only the signature
blob is garbled (not the code), so the later 10e re-sign step stays valid --
it replaces the blob and the code is untouched. x86_64 keeps
remove-signature (code signing is not enforced there).
Refs: github.com/garrytan/gstack#997, github.com/nodejs/node#40827
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
#581 (query-path memory growth) does not reproduce in our soak: a 10-min,
200-file query-only run shows RSS flat-to-shrinking and committed plateauing
(mimalloc already purges on free, purge_delay=0). Rather than ship speculative
mitigations for a leak we cannot measure, this adds the observability to find it
where it actually occurs, plus a continuous guard.
- diag: the diagnostics writer now appends a PERSISTENT NDJSON memory trajectory
(one sample / 5s: rss, committed, peak fields, page_faults, fd, queries), kept
on exit (rotates to a .1 generation past 8 MB) so users can send the time
series post-mortem. The previous latest-snapshot file was overwritten every 5s
and deleted on stop.
- ci: the soak now also runs a read-only query-leak leg (CBM_SOAK_MODE query-leak,
no reindex/mutate) on every platform including Windows, so a regression that
introduces a query-path leak is caught. soak-test.sh RESULTS_DIR is now
env-overridable so both legs keep separate artifacts.
- docs: README Troubleshooting and Diagnostics section + a bug-report field
explain the no-telemetry stance and how to capture and share the trajectory.
Build clean; unit suite 5714/0; trajectory verified to persist post-exit.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The multi-platform smoke run flagged index-cli + index-status as empty/not-ready
even though the binary indexed fine (nodes>0, status ready) — the MCP tool result
wraps its payload as a JSON STRING with escaped quotes (\"nodes\":N), but the
checks grepped for unescaped "nodes":N. Strip backslashes/quotes before matching
and tolerate the nodes= log form. (Binary was healthy; this was a script-parsing
gap the wide-matrix smoke surfaced.)
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
- scripts/smoke-invariants.sh: 30-check battery against the PROD binary —
--version/--help, MCP initialize handshake with stdin OPEN (#513), tools/list
(all 14), EVERY tool invocable with valid JSON-RPC + no crash, index→non-empty
graph, malformed-input resilience (bad JSON / empty / huge line / binary /
non-UTF8 / missing path), clean EOF exit, shared-lib resolution, install
dry-run. Bounded waits (read -t / timeout), no sleep loops; msys2-safe.
- .github/workflows/smoke.yml: runs it on the WIDEST runner matrix — ubuntu
22.04+24.04 (x64+arm64; 22.04 = older glibc / AlmaLinux class), macos
14/15/15-intel, windows 2022/2025 + windows-11-arm (experimental). A FAIL on
any platform is a binary a user would receive. workflow_dispatch + qa/smoke-**.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Martin Vogel