harden: remove capability that should never have shipped
Microsoft's ML flagged the rc.1 release binaries. The decisive evidence is that the SAME sha256 went from 0/62 clean to Microsoft-detected in about an hour with no byte change, so the verdict lives partly in scanner-side state and no code change can promise a clean result. What code CAN do is stop shipping things that have no business in a release artifact, which is worth doing on its own merits and incidentally widens the classifier margin. Every claim below is verified against a built binary by the new gate, not by reading source. Executable stack (the worst of the findings). vendored/nomic/code_vectors_blob.S is the only assembly in the build and carried no .note.GNU-stack. An unannotated object makes ld assume the worst for the whole link, so EVERY Linux release we have ever shipped had GNU_STACK RWE. Adds the note (cause) plus ELF-only -Wl,-z,noexecstack (outcome); the gate fails the release if it returns. Test seams are now opt-in, never opt-out. TEST_SEAMS=1 defines CBM_ENABLE_TEST_SEAMS; without it the crash-orphan probe -- which forks a child that ignores SIGTERM and loops forever, then writes its pid to a caller-supplied path -- and the lease-ownership marker compile to trivial stubs, so call sites are untouched and the binary holds no fork, no signal handler and no env-var string. Opt-IN is the point: forgetting the flag yields a clean binary rather than a leaky one. scripts/test.sh requests it in the leg that consumes it, and tests/test_worker_watchdog.sh now asserts the capability up front instead of dying later with an opaque "Killed: 9". The daemon's background version check is gone. It spawned curl against api.github.com/repos/.../releases/latest on the first eligible session of every run to say "a newer version exists" -- a release URL and an outbound request in every shipped binary, for something the install scripts already report. The INJECTABLE SEAM survives: update_ops is still honoured, the fakes in tests/test_daemon_application.c still cover notice/ownership/cancellation/replay, and with no provider application_update_subscribe_locked returns early so no generation ever starts. "No network request by default" is now structural. Dead capability out of release builds. The tar.gz/zip extraction block (gzip_decompress through cbm_extract_binary_from_zip, plus its cli.h declarations) moves under CBM_CLI_ENABLE_TEST_API -- verified self-contained, zero uses of any helper outside it, only callers the excluded updater and tests/test_cli.c. Downloading an archive, decompressing it, picking an executable out of it and marking it executable is the canonical dropper composite; it is now absent rather than merely unreachable. SQLite is built with -DSQLITE_OMIT_LOAD_EXTENSION (no caller of load_extension anywhere in src/ or internal/), removing that API surface and part of the dlopen/dlsym surface. Temp files and environment scanning (S2/S3). Predictable paths in mcp.c, artifact.c and diagnostics.c are created privately and exclusively and written through the returned descriptor; pass_envscan.c no longer descends symlinked directories out of the project root, and its fixed 512-byte path buffers no longer truncate into pointer arithmetic that could land outside the buffer. Build-time entropy. mimalloc's version banner baked __DATE__/__TIME__ into every binary, so two builds of identical source seconds apart could never share a hash and no release could inherit a false-positive determination made about its predecessor. Local patch removes it (marked to survive refreshes), -Wdate-time makes any future use a build error, and -Wl,--no-insert-timestamp stops the PE header carrying the link clock. scripts/ci/check-binary-composition.sh is the proof that each removal stays removed, wired into package-release.sh after strip so the local artifact-flow smoke enforces exactly what CI does. It asserts absences plus a CANARY string, so handing it a compressed, stubbed or empty file fails instead of passing vacuously, and a missing tool is a hard error -- a skipped assertion must never look like a satisfied one. Two build-system traps found by that gate, both of which had silently defeated a fix: the product binary is compiled in one shot from sources, so a flag flip did not rebuild it (now tracked by a .build-config stamp that also removes the binary, making it independent of mtime granularity); and prod_sqlite3.o / prod_mimalloc.o depended on a single named source, so SQLITE_OMIT_LOAD_EXTENSION and the mimalloc patch BOTH compiled to nothing on the first incremental build. Source review would have called them done. Deliberately NOT changed. Three seams stay in release artifacts because scripts/smoke-test.sh runs against the real artifact and needs them: CBM_TEST_CRASH_ON and CBM_TEST_HANG_ON inject the faults that prove supervisor recovery, and CBM_TEST_WINDOWS_USER_PATH_RUN_ID is what keeps the PATH smoke from writing the tester's actual PATH. The gate treats those as an allowlist, so a NOVEL seam still fails. The true no-UI standard build is deferred rather than rushed: src/ui/* is in PROD_SRCS and four files outside src/ui reference UI symbols, including the daemon that serves the UI, so that assertion reports instead of failing until the split lands -- a gate everyone knows is red teaches people to ignore gates. No grammar is removed. ObjectScript accounts for essentially all binary growth since the last provably-clean release (+21.4MB rodata, +1.1MB text from two four-line shims), which made it the obvious ablation candidate, but a dry run performed twelve real Defender endpoint scans across standard/UI and amd64/arm64 with ObjectScript, the daemon and the expanded hooks all present and every scan was clean. Nothing there is a deterministic trigger, so cutting a community-contributed language would spend a real feature on unproven margin. Lean is not a candidate either: at 99.6MB of source it is by far the largest grammar, but it shipped in v0.9.0 which scanned 20/20 clean, so removing it would produce a novel unscanned profile instead of restoring a known-good one. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
This commit is contained in:
+81
-12
@@ -43,8 +43,12 @@ GCC_ONLY_FLAGS := -Wno-format-truncation -Wno-unused-result \
|
||||
-Wno-stringop-truncation -Wno-alloc-size-larger-than
|
||||
endif
|
||||
|
||||
# -Wdate-time: with -Werror, any use of __DATE__/__TIME__/__TIMESTAMP__ fails the
|
||||
# build. Build-time entropy is how one identical source tree produced a new hash
|
||||
# on every compile (see the local patch in vendored/mimalloc/src/options.c) —
|
||||
# once removed, it must not creep back through a new call site.
|
||||
CFLAGS_COMMON = -std=c11 -D_DEFAULT_SOURCE -D_GNU_SOURCE -Wall -Wextra -Werror \
|
||||
-Wno-unused-parameter -Wno-sign-compare \
|
||||
-Wno-unused-parameter -Wno-sign-compare -Wdate-time \
|
||||
$(GCC_ONLY_FLAGS) \
|
||||
-Isrc -Ivendored -Ivendored/sqlite3 \
|
||||
-Ivendored/mimalloc/include \
|
||||
@@ -54,11 +58,24 @@ CXXFLAGS_COMMON = -std=c++14 -Wall -Wextra -Werror \
|
||||
-Wno-unused-parameter \
|
||||
-I$(CBM_DIR) -I$(TS_INCLUDE)
|
||||
|
||||
# Test seams are OPT-IN, never opt-out. Some suites drive behaviour that only a
|
||||
# test should be able to ask for (fork an orphan the watchdog must reap, publish
|
||||
# a lease-ownership marker). That code has no production caller and reads exactly
|
||||
# like malware to a generic classifier, so it must not be in a shipped binary.
|
||||
# Opt-IN means the failure mode of forgetting the flag is a CLEAN binary rather
|
||||
# than a leaky one — the opposite choice would make every future release depend
|
||||
# on someone remembering. scripts/test.sh passes TEST_SEAMS=1 for the suites that
|
||||
# need it; the test-runner always has them.
|
||||
TEST_SEAM_DEFINE :=
|
||||
ifeq ($(TEST_SEAMS),1)
|
||||
TEST_SEAM_DEFINE := -DCBM_ENABLE_TEST_SEAMS=1
|
||||
endif
|
||||
|
||||
# Production flags (CFLAGS_EXTRA allows CI to inject -DCBM_VERSION)
|
||||
# CBM_BIND_TS_ALLOCATOR=1: bind the tree-sitter runtime to mimalloc (#424). Only
|
||||
# the prod build uses mimalloc (MI_OVERRIDE=1); the test build is CRT+ASan, where
|
||||
# binding would create an alloc/free mismatch, so the guard is prod-only.
|
||||
CFLAGS_PROD = $(CFLAGS_COMMON) -O2 -DCBM_BIND_TS_ALLOCATOR=1 $(CFLAGS_EXTRA)
|
||||
CFLAGS_PROD = $(CFLAGS_COMMON) -O2 -DCBM_BIND_TS_ALLOCATOR=1 $(TEST_SEAM_DEFINE) $(CFLAGS_EXTRA)
|
||||
CXXFLAGS_PROD = $(CXXFLAGS_COMMON) -O2
|
||||
|
||||
# Test flags: debug + sanitizers (override SANITIZE= to disable on Windows)
|
||||
@@ -66,7 +83,7 @@ SANITIZE = -fsanitize=address,undefined -fno-omit-frame-pointer
|
||||
EDITOR_TEST_DEFINES = -DCBM_JSON_LIKE_ENABLE_TEST_API=1 \
|
||||
-DCBM_TOML_EDIT_ENABLE_TEST_API=1 -DCBM_YAML_ENABLE_TEST_API=1 \
|
||||
-DCBM_TEXT_EDIT_ENABLE_TEST_API=1 -DCBM_CLI_ENABLE_TEST_API=1 \
|
||||
-DCBM_DIAGNOSTICS_ENABLE_TEST_API=1
|
||||
-DCBM_DIAGNOSTICS_ENABLE_TEST_API=1 -DCBM_ENABLE_TEST_SEAMS=1
|
||||
# The build system is the single source of truth for "is this binary
|
||||
# instrumented": compiler-specific probes (__SANITIZE_ADDRESS__) miss
|
||||
# clang's feature-check spelling and every non-ASan sanitizer, so the
|
||||
@@ -120,7 +137,11 @@ ifeq ($(IS_LINUX),yes)
|
||||
MIMALLOC_WRAP_FLAGS_POSIX := $(foreach sym,$(MIMALLOC_WRAP_SYMS_POSIX),-Wl,--wrap=$(sym))
|
||||
endif
|
||||
ifeq ($(IS_MINGW),yes)
|
||||
WIN32_LIBS := -lws2_32 -lpsapi -lshell32 -ladvapi32 -Wl,--allow-multiple-definition -Wl,--stack,8388608 -static
|
||||
# --no-insert-timestamp: the PE header otherwise carries the link wall clock, so
|
||||
# two Windows builds of identical source are never byte-identical and every
|
||||
# release is a brand-new file to a reputation system. Same motivation as the
|
||||
# mimalloc __DATE__ patch; ELF and Mach-O have no equivalent field to clear.
|
||||
WIN32_LIBS := -lws2_32 -lpsapi -lshell32 -ladvapi32 -Wl,--allow-multiple-definition -Wl,--stack,8388608 -Wl,--no-insert-timestamp -static
|
||||
endif
|
||||
|
||||
# STATIC=1 produces a fully static binary (for Alpine/musl portable builds)
|
||||
@@ -128,14 +149,25 @@ ifeq ($(STATIC),1)
|
||||
STATIC_FLAGS := -static
|
||||
endif
|
||||
|
||||
# W^X: demand a non-executable stack on ELF. This is belt to the braces of the
|
||||
# .note.GNU-stack annotation in vendored/nomic/code_vectors_blob.S — that note
|
||||
# fixes the CAUSE (an unannotated object makes ld assume the worst for the whole
|
||||
# link), this flag fixes the OUTCOME, and the composition gate proves it on the
|
||||
# shipped artifact. ELF-only: Apple's ld rejects -z noexecstack outright and it
|
||||
# is meaningless for PE, so it is gated rather than made "common".
|
||||
ELF_HARDENING_FLAGS :=
|
||||
ifeq ($(IS_LINUX),yes)
|
||||
ELF_HARDENING_FLAGS := -Wl,-z,noexecstack
|
||||
endif
|
||||
|
||||
# The POSIX wrap shim exists only so the profiler can observe allocations. It
|
||||
# must never reach a sanitized build: the Linux/macOS test builds are CRT+ASan,
|
||||
# and redirecting malloc into mimalloc underneath ASan's own interception mixes
|
||||
# two allocators on the same pointers. Windows keeps its wrap flags everywhere,
|
||||
# because there the shim is what makes mimalloc own the allocations at all.
|
||||
LDFLAGS = -lm -lstdc++ -lpthread -lz $(WIN32_LIBS) $(STATIC_FLAGS) $(MIMALLOC_WRAP_FLAGS)
|
||||
LDFLAGS_TEST = -lm -lstdc++ -lpthread -lz $(SANITIZE) $(WIN32_LIBS) $(MIMALLOC_WRAP_FLAGS)
|
||||
LDFLAGS_TSAN = -lm -lstdc++ -lpthread -lz $(TSAN_SANITIZE) $(WIN32_LIBS) $(MIMALLOC_WRAP_FLAGS)
|
||||
LDFLAGS = -lm -lstdc++ -lpthread -lz $(WIN32_LIBS) $(STATIC_FLAGS) $(MIMALLOC_WRAP_FLAGS) $(ELF_HARDENING_FLAGS)
|
||||
LDFLAGS_TEST = -lm -lstdc++ -lpthread -lz $(SANITIZE) $(WIN32_LIBS) $(MIMALLOC_WRAP_FLAGS) $(ELF_HARDENING_FLAGS)
|
||||
LDFLAGS_TSAN = -lm -lstdc++ -lpthread -lz $(TSAN_SANITIZE) $(WIN32_LIBS) $(MIMALLOC_WRAP_FLAGS) $(ELF_HARDENING_FLAGS)
|
||||
|
||||
# ── Source files ─────────────────────────────────────────────────
|
||||
|
||||
@@ -358,9 +390,20 @@ MIMALLOC_CFLAGS_TEST = -std=c11 -g -O1 -w \
|
||||
# sqlite3 (vendored amalgamation — compiled ourselves for ASan instrumentation)
|
||||
# SQLITE_ENABLE_FTS5: enables the FTS5 full-text search extension used by the
|
||||
# BM25 search path in search_graph (see nodes_fts virtual table in store.c).
|
||||
#
|
||||
# SQLITE_OMIT_LOAD_EXTENSION: no cbm code calls sqlite3_load_extension or
|
||||
# sqlite3_enable_load_extension, and we never want a graph DB to be able to pull
|
||||
# a shared library into the process. Loading is off by default at runtime, but
|
||||
# "disabled by default" is a setting while "not compiled in" is a property —
|
||||
# this compiles the machinery out, removing the API surface and part of the
|
||||
# dlopen/dlsym import surface with it. The composition gate asserts the symbols
|
||||
# are absent from every release artifact.
|
||||
SQLITE3_OMIT_FLAGS = -DSQLITE_OMIT_LOAD_EXTENSION
|
||||
SQLITE3_SRC = vendored/sqlite3/sqlite3.c
|
||||
SQLITE3_CFLAGS = -std=c11 -O2 -w -DSQLITE_DQS=0 -DSQLITE_THREADSAFE=1 -DSQLITE_ENABLE_FTS5
|
||||
SQLITE3_CFLAGS_TEST = -std=c11 -g -O1 -w -DSQLITE_DQS=0 -DSQLITE_THREADSAFE=1 -DSQLITE_ENABLE_FTS5
|
||||
SQLITE3_CFLAGS = -std=c11 -O2 -w -DSQLITE_DQS=0 -DSQLITE_THREADSAFE=1 -DSQLITE_ENABLE_FTS5 \
|
||||
$(SQLITE3_OMIT_FLAGS)
|
||||
SQLITE3_CFLAGS_TEST = -std=c11 -g -O1 -w -DSQLITE_DQS=0 -DSQLITE_THREADSAFE=1 -DSQLITE_ENABLE_FTS5 \
|
||||
$(SQLITE3_OMIT_FLAGS)
|
||||
|
||||
# TRE regex (vendored, Windows only — POSIX uses system <regex.h>)
|
||||
TRE_SRC = vendored/tre/tre_all.c
|
||||
@@ -647,7 +690,14 @@ $(BUILD_DIR)/mimalloc.o: $(MIMALLOC_SRC) | $(BUILD_DIR)
|
||||
$(BUILD_DIR)/tsan_mimalloc.o: $(MIMALLOC_SRC) | $(BUILD_DIR)
|
||||
$(CC) $(MIMALLOC_CFLAGS_TEST) $(TSAN_SANITIZE) -c -o $@ $<
|
||||
|
||||
$(BUILD_DIR)/prod_mimalloc.o: $(MIMALLOC_SRC) | $(BUILD_DIR)
|
||||
# static.c is an AMALGAMATION: it #includes options.c and friends, none of which
|
||||
# make can see through the single named prerequisite, and the compile flags live
|
||||
# in this Makefile. Both gaps bit for real — the __DATE__ removal in options.c and
|
||||
# SQLITE_OMIT_LOAD_EXTENSION below each silently did nothing on an incremental
|
||||
# build because the object was considered up to date. Depend on the included
|
||||
# sources AND on this Makefile so a flag change rebuilds too.
|
||||
$(BUILD_DIR)/prod_mimalloc.o: $(MIMALLOC_SRC) $(wildcard vendored/mimalloc/src/*.c) \
|
||||
$(wildcard vendored/mimalloc/include/*.h) Makefile.cbm | $(BUILD_DIR)
|
||||
$(CC) $(MIMALLOC_CFLAGS) -c -o $@ $<
|
||||
|
||||
# sqlite3 object files (vendored amalgamation)
|
||||
@@ -661,7 +711,7 @@ $(BUILD_DIR)/sqlite3.o: $(SQLITE3_SRC) | $(BUILD_DIR)
|
||||
$(BUILD_DIR)/tsan_sqlite3.o: $(SQLITE3_SRC) | $(BUILD_DIR)
|
||||
$(CC) $(SQLITE3_CFLAGS_TEST) $(TSAN_SANITIZE) -c -o $@ $<
|
||||
|
||||
$(BUILD_DIR)/prod_sqlite3.o: $(SQLITE3_SRC) | $(BUILD_DIR)
|
||||
$(BUILD_DIR)/prod_sqlite3.o: $(SQLITE3_SRC) Makefile.cbm | $(BUILD_DIR)
|
||||
$(CC) $(SQLITE3_CFLAGS) -c -o $@ $<
|
||||
|
||||
# TRE regex (only compiled on Windows — POSIX uses system <regex.h>)
|
||||
@@ -835,7 +885,26 @@ OBJS_VENDORED_PROD = $(MIMALLOC_OBJ_PROD) $(SQLITE3_OBJ_PROD) $(TRE_OBJ_PROD) $(
|
||||
|
||||
MAIN_SRC = src/main.c
|
||||
|
||||
$(BUILD_DIR)/codebase-memory-mcp: $(MAIN_SRC) $(PROD_SRCS) $(EXTRACTION_SRCS) $(AC_LZ4_SRCS) $(ZSTD_SRCS) $(SQLITE_WRITER_SRC) $(OBJS_VENDORED_PROD) | $(BUILD_DIR)
|
||||
# Rebuild when the build CONFIGURATION changes, not only when sources do. The
|
||||
# product binary is compiled in ONE shot from sources, so after a TEST_SEAMS or
|
||||
# version flip make finds the binary newer than every source and skips the recipe
|
||||
# entirely, handing back a binary built with the PREVIOUS configuration. The
|
||||
# dangerous direction is a release build silently keeping test seams, which is a
|
||||
# property no reviewer can see by reading the diff. The stamp's CONTENT is the
|
||||
# configuration and it is rewritten only when that content changes, so ordinary
|
||||
# incremental builds are unaffected. (Single-quoted: the signature may contain
|
||||
# the escaped quotes of -DCBM_VERSION, which are literal inside '', but must not
|
||||
# contain a literal single quote.)
|
||||
BUILD_CONFIG_SIG := TEST_SEAMS=$(TEST_SEAMS)|CFLAGS_EXTRA=$(CFLAGS_EXTRA)
|
||||
.PHONY: build-config-check
|
||||
build-config-check:
|
||||
$(BUILD_DIR)/.build-config: build-config-check | $(BUILD_DIR)
|
||||
@printf '%s\n' '$(BUILD_CONFIG_SIG)' > $@.tmp
|
||||
@if cmp -s $@.tmp $@; then rm -f $@.tmp; \
|
||||
else mv -f $@.tmp $@; rm -f $(BUILD_DIR)/codebase-memory-mcp; \
|
||||
echo "=== build config changed -> full rebuild: $(BUILD_CONFIG_SIG) ==="; fi
|
||||
|
||||
$(BUILD_DIR)/codebase-memory-mcp: $(MAIN_SRC) $(PROD_SRCS) $(EXTRACTION_SRCS) $(AC_LZ4_SRCS) $(ZSTD_SRCS) $(SQLITE_WRITER_SRC) $(OBJS_VENDORED_PROD) $(BUILD_DIR)/.build-config | $(BUILD_DIR)
|
||||
$(CC) $(CFLAGS_PROD) -o $@ \
|
||||
$(MAIN_SRC) $(PROD_SRCS) \
|
||||
$(EXTRACTION_SRCS) $(AC_LZ4_SRCS) $(ZSTD_SRCS) $(SQLITE_WRITER_SRC) \
|
||||
|
||||
@@ -0,0 +1,404 @@
|
||||
#!/usr/bin/env bash
|
||||
# check-binary-composition.sh — release gate: assert what must NOT be inside a
|
||||
# shipped artifact.
|
||||
#
|
||||
# Microsoft Defender's ML classifier flagged the v0.9.1-rc.1 binaries. The
|
||||
# hardening pass that followed *removed capability* — an executable stack, an
|
||||
# in-process updater, test-only environment seams, the embedded HTTP/UI server
|
||||
# and its process enumerator in the standard build. Each of those regresses
|
||||
# invisibly: one restored #include, one Makefile source list edit, one revived
|
||||
# call site, and nothing else in CI notices — the binary just quietly gets its
|
||||
# malware-shaped surface back and the next release gets flagged again.
|
||||
#
|
||||
# This script is the proof that each removal stayed removed. It asserts only
|
||||
# NEGATIVE properties (needle absent), plus one canary string we know ships,
|
||||
# because an absence check aimed at the wrong file — a compressed artifact, a
|
||||
# stub, a truncated download — would otherwise pass vacuously and read green.
|
||||
# A missing tool is a hard error for the same reason: a skipped assertion must
|
||||
# never look like a satisfied one.
|
||||
#
|
||||
# Usage: scripts/ci/check-binary-composition.sh [--variant=auto|standard|ui] <binary-or-dir>...
|
||||
# Directories are scanned recursively; format (ELF / Mach-O / PE) is detected
|
||||
# per file from its magic bytes and each assertion runs where it is
|
||||
# meaningful. Exit 0 = every assertion passed, 1 = at least one failed,
|
||||
# 2 = usage error, missing tool, or nothing checkable was found (a vacuous
|
||||
# run is a failure, not a pass).
|
||||
set -euo pipefail
|
||||
|
||||
case "${1:-}" in
|
||||
-h | --help)
|
||||
sed -n '2,25p' "$0" | sed 's/^# \{0,1\}//'
|
||||
exit 0
|
||||
;;
|
||||
esac
|
||||
|
||||
# ── Needles ─────────────────────────────────────────────────────────
|
||||
# Every needle below was grep-verified against the tree it comes from: it
|
||||
# exists ONLY in the code the assertion is about, so a hit means that code is
|
||||
# linked in, not that some unrelated string happens to look similar.
|
||||
|
||||
# S6: worker/Windows test seams read these env vars. A release binary that
|
||||
# still honours them lets any process on the box steer the indexer's child
|
||||
# processes and file placement — and CBM_TEST_* in a shipped binary is exactly
|
||||
# the "debug/injection hooks" shape AV heuristics score on.
|
||||
SEAM_NEEDLES=(
|
||||
'CBM_TEST_WORKER_DESCENDANT_PID_FILE'
|
||||
'CBM_TEST_WINDOWS_USER_PATH_RUN_ID'
|
||||
)
|
||||
|
||||
# The in-process updater (download-and-replace-own-binary) is the single most
|
||||
# malware-shaped behaviour we ever shipped; updates now run from install.sh
|
||||
# out-of-process. These four needles cover both halves of what was removed:
|
||||
# the download base URL, the checksum URL built on top of it, the GitHub API
|
||||
# release query of the daemon's background version check, and that request's
|
||||
# Accept header (which survives even if the URL is ever assembled at runtime).
|
||||
UPDATER_NEEDLES=(
|
||||
'releases/latest/download'
|
||||
'api.github.com/repos'
|
||||
'releases/latest'
|
||||
'Accept: application/vnd.github+json'
|
||||
)
|
||||
|
||||
# SQLite loadable extensions = arbitrary code execution through a database
|
||||
# file. The amalgamation is built with SQLITE_OMIT_LOAD_EXTENSION; if that ever
|
||||
# drops out, dlopen()/LoadLibrary() re-enters the store layer.
|
||||
# NOTE: release binaries are stripped, so the two API symbol names only appear
|
||||
# when the surface is *exported*. The dlopen error text is the needle that
|
||||
# still works on a stripped build — it is compiled in only when the feature is.
|
||||
SQLITE_LOADEXT_NEEDLES=(
|
||||
'sqlite3_load_extension'
|
||||
'sqlite3_enable_load_extension'
|
||||
'unable to open shared library ['
|
||||
)
|
||||
|
||||
# The standard artifact must not contain the UI's HTTP server at all: an
|
||||
# unauthenticated localhost listener plus a process enumerator is precisely the
|
||||
# behaviour pair a classifier reads as a backdoor. Needles are split across
|
||||
# both UI translation units and are independent of each other, so a refactor of
|
||||
# any single one cannot silently disarm the assertion:
|
||||
# 'HTTP/1.1 %d %s' httpd.c response-status writer
|
||||
# 'Request Header Fields Too Large' httpd.c status-text table
|
||||
# '/api/ui-config', '/api/processes' http_server.c route dispatch
|
||||
# 'ps -eo …' / '[c]odebase-memory-mcp' the popen() process enumerator's
|
||||
# shell pipeline, both halves
|
||||
# Verified unique to src/ui/ across src/, vendored/ and internal/ — the only
|
||||
# other occurrences in the repo are tests/test_httpd.c and scripts/, neither of
|
||||
# which is ever linked into a release binary.
|
||||
UI_HTTP_NEEDLES=(
|
||||
'HTTP/1.1 %d %s'
|
||||
'Request Header Fields Too Large'
|
||||
'/api/ui-config'
|
||||
'/api/processes'
|
||||
'ps -eo pid,pcpu,rss,etime,comm'
|
||||
'[c]odebase-memory-mcp'
|
||||
)
|
||||
|
||||
# Canary: proves the needle scan can actually see this file's strings. Without
|
||||
# it, handing the gate a gzip, a stub or a 0-byte file would pass every
|
||||
# absence assertion. 168+ occurrences in a real artifact, 0 in anything else.
|
||||
CANARY_NEEDLE='codebase-memory-mcp'
|
||||
|
||||
# ── Args ────────────────────────────────────────────────────────────
|
||||
VARIANT=auto
|
||||
TARGETS=()
|
||||
for arg in "$@"; do
|
||||
case "$arg" in
|
||||
--variant=*) VARIANT="${arg#--variant=}" ;;
|
||||
-*)
|
||||
echo "FAIL: unknown flag $arg (see --help)" >&2
|
||||
exit 2
|
||||
;;
|
||||
*) TARGETS+=("$arg") ;;
|
||||
esac
|
||||
done
|
||||
case "$VARIANT" in
|
||||
auto | standard | ui) ;;
|
||||
*)
|
||||
echo "FAIL: --variant must be auto, standard or ui (got '$VARIANT')" >&2
|
||||
exit 2
|
||||
;;
|
||||
esac
|
||||
if [ "${#TARGETS[@]}" -eq 0 ]; then
|
||||
echo "FAIL: no binaries or directories given (see --help)" >&2
|
||||
exit 2
|
||||
fi
|
||||
|
||||
# ── ELF program-header reader ───────────────────────────────────────
|
||||
# Resolved lazily: a Windows/macOS-only run must not fail for want of readelf,
|
||||
# but an ELF in scope with no reader available must fail loudly (exit 2) rather
|
||||
# than skip the executable-stack assertion.
|
||||
ELF_READER=''
|
||||
ELF_READER_KIND=''
|
||||
resolve_elf_reader() {
|
||||
[ -n "$ELF_READER" ] && return 0
|
||||
for cand in readelf llvm-readelf /opt/homebrew/opt/llvm/bin/llvm-readelf \
|
||||
/usr/local/opt/llvm/bin/llvm-readelf; do
|
||||
if command -v "$cand" >/dev/null 2>&1; then
|
||||
ELF_READER="$cand"
|
||||
ELF_READER_KIND=readelf
|
||||
return 0
|
||||
fi
|
||||
done
|
||||
for cand in objdump llvm-objdump /opt/homebrew/opt/llvm/bin/llvm-objdump; do
|
||||
if command -v "$cand" >/dev/null 2>&1; then
|
||||
ELF_READER="$cand"
|
||||
ELF_READER_KIND=objdump
|
||||
return 0
|
||||
fi
|
||||
done
|
||||
return 1
|
||||
}
|
||||
|
||||
# Echoes the GNU_STACK flag field ("RWE", "RW", "rwx", "rw-"), empty if the
|
||||
# header is absent or unparseable.
|
||||
gnu_stack_flags() {
|
||||
case "$ELF_READER_KIND" in
|
||||
readelf)
|
||||
"$ELF_READER" -lW "$1" 2>/dev/null |
|
||||
awk '/GNU_STACK/ { print $(NF - 1); exit }'
|
||||
;;
|
||||
objdump)
|
||||
# objdump -p prints " STACK off ..." and the flags on the next line.
|
||||
"$ELF_READER" -p "$1" 2>/dev/null |
|
||||
awk '/STACK off/ { getline
|
||||
for (i = 1; i <= NF; i++)
|
||||
if ($i == "flags") { print $(i + 1); exit } }'
|
||||
;;
|
||||
esac
|
||||
}
|
||||
|
||||
# ── Reporting ───────────────────────────────────────────────────────
|
||||
# PASS and FAIL both go to stdout so the per-assertion sequence stays in order
|
||||
# in a CI log (stderr would interleave nondeterministically); only the final
|
||||
# verdict is echoed to stderr, which is what a failed step's tail shows.
|
||||
pass_count=0
|
||||
fail_count=0
|
||||
FAILURES=()
|
||||
|
||||
report() { # verdict, assertion-id, file-token, message
|
||||
printf '%-4s %-22s %s: %s\n' "$1" "$2" "$3" "$4"
|
||||
if [ "$1" = FAIL ]; then
|
||||
fail_count=$((fail_count + 1))
|
||||
FAILURES+=("$2 $3: $4")
|
||||
else
|
||||
pass_count=$((pass_count + 1))
|
||||
fi
|
||||
}
|
||||
|
||||
assert_absent() { # file, token, assertion-id, needle
|
||||
if LC_ALL=C grep -a -q -F -e "$4" "$1"; then
|
||||
report FAIL "$3" "$2" "'$4' is PRESENT (must not ship)"
|
||||
else
|
||||
report PASS "$3" "$2" "'$4' absent"
|
||||
fi
|
||||
}
|
||||
|
||||
assert_present() { # file, token, assertion-id, needle, why-pass, why-fail
|
||||
if LC_ALL=C grep -a -q -F -e "$4" "$1"; then
|
||||
report PASS "$3" "$2" "'$4' present ($5)"
|
||||
else
|
||||
report FAIL "$3" "$2" "'$4' is MISSING — $6"
|
||||
fi
|
||||
}
|
||||
|
||||
# ── Per-file checks ─────────────────────────────────────────────────
|
||||
detect_format() { # file → elf | macho | pe | other
|
||||
magic=$(LC_ALL=C od -An -N4 -tx1 "$1" 2>/dev/null | tr -d ' \n')
|
||||
case "$magic" in
|
||||
7f454c46) echo elf ;;
|
||||
cffaedfe | cefaedfe | feedface | feedfacf | cafebabe | bebafeca) echo macho ;;
|
||||
4d5a*) echo pe ;;
|
||||
*) echo other ;;
|
||||
esac
|
||||
}
|
||||
|
||||
checked_files=0
|
||||
skipped_files=0
|
||||
|
||||
check_file() {
|
||||
file="$1"
|
||||
# Two path components: both variants ship a binary literally named
|
||||
# "codebase-memory-mcp", so the parent directory is what tells them apart.
|
||||
token=$(printf '%s' "$file" | awk -F/ '{ if (NF > 1) print $(NF - 1) "/" $NF; else print $NF }')
|
||||
fmt=$(detect_format "$file")
|
||||
if [ "$fmt" = other ]; then
|
||||
printf 'skip %-22s %s: not an ELF/Mach-O/PE binary\n' '-' "$token"
|
||||
skipped_files=$((skipped_files + 1))
|
||||
return 0
|
||||
fi
|
||||
|
||||
is_ui=0
|
||||
case "$VARIANT" in
|
||||
ui) is_ui=1 ;;
|
||||
standard) is_ui=0 ;;
|
||||
auto)
|
||||
# The UI archive is codebase-memory-mcp-ui-<os>-<arch>, but the binary
|
||||
# inside it is just "codebase-memory-mcp" — so the whole path decides,
|
||||
# not the basename. Printed below so a misclassified path is visible in
|
||||
# the log instead of silently disarming the UI assertion.
|
||||
case "$file" in
|
||||
*-ui | *-ui.* | *-ui-* | *-ui/*) is_ui=1 ;;
|
||||
esac
|
||||
;;
|
||||
esac
|
||||
|
||||
variant_label=standard
|
||||
[ "$is_ui" -eq 1 ] && variant_label='UI (A5-no-ui-http not applicable)'
|
||||
printf '\n── %s [%s, %s] ──\n' "$token" "$fmt" "$variant_label"
|
||||
checked_files=$((checked_files + 1))
|
||||
|
||||
# A0 — anti-vacuity canary; every assertion below is an absence check.
|
||||
assert_present "$file" "$token" A0-canary "$CANARY_NEEDLE" \
|
||||
'the needle scan can read this file' \
|
||||
'this is not one of our artifacts, or its strings are unreadable (packed/compressed/truncated) — every absence assertion below would pass vacuously'
|
||||
|
||||
# A1 — executable stack (ELF only). Every Linux artifact of v0.9.1-rc.1
|
||||
# shipped GNU_STACK RWE: a writable+executable stack, which no modern
|
||||
# binary has and which any classifier weighs heavily.
|
||||
if [ "$fmt" = elf ]; then
|
||||
if ! resolve_elf_reader; then
|
||||
echo "FAIL: no readelf/llvm-readelf/objdump available; cannot assert" \
|
||||
"the non-executable stack for $token — refusing to skip it" >&2
|
||||
exit 2
|
||||
fi
|
||||
flags=$(gnu_stack_flags "$file")
|
||||
# readelf spells the flags "RWE"/"RW", objdump "rwx"/"rw-" — match both.
|
||||
case "$flags" in
|
||||
'')
|
||||
report FAIL A1-noexec-stack "$token" \
|
||||
"no GNU_STACK program header found ($ELF_READER) — without PT_GNU_STACK the loader may fall back to READ_IMPLIES_EXEC"
|
||||
;;
|
||||
*E* | *X* | *e* | *x*)
|
||||
report FAIL A1-noexec-stack "$token" \
|
||||
"GNU_STACK is $flags — the stack is EXECUTABLE (link with -z noexecstack; check .S/asm objects for a missing .note.GNU-stack)"
|
||||
;;
|
||||
*)
|
||||
report PASS A1-noexec-stack "$token" "GNU_STACK is $flags (no execute bit)"
|
||||
;;
|
||||
esac
|
||||
else
|
||||
printf 'n/a %-22s %s: executable-stack check is ELF-only\n' A1-noexec-stack "$token"
|
||||
fi
|
||||
|
||||
# A2 — test-only seams.
|
||||
for needle in "${SEAM_NEEDLES[@]}"; do
|
||||
assert_absent "$file" "$token" A2-no-test-seams "$needle"
|
||||
done
|
||||
# Sweep for seams nobody thought to pin: a new CBM_TEST_* env var added to
|
||||
# production code lands here on its first release, not on the next audit.
|
||||
# Two seams remain in release artifacts BY DECISION, so the wildcard is an
|
||||
# allowlist rather than a blanket ban: scripts/smoke-test.sh runs against the
|
||||
# real release artifact, and these are what let it do so honestly —
|
||||
# CRASH_ON/HANG_ON inject the faults that prove supervisor recovery, and
|
||||
# WINDOWS_USER_PATH_RUN_ID is what stops the PATH smoke from writing the
|
||||
# tester's actual PATH. Deleting them would trade genuine release-artifact
|
||||
# coverage for a cosmetic win. Anything NOT on this list is a novel seam and
|
||||
# fails, which is the property that matters: the list can only shrink by
|
||||
# decision, never grow by accident.
|
||||
seam_allowed='CBM_TEST_CRASH_ON CBM_TEST_HANG_ON CBM_TEST_WINDOWS_USER_PATH_RUN_ID'
|
||||
seam_unexpected=''
|
||||
for found in $(LC_ALL=C grep -a -o -E 'CBM_TEST_[A-Za-z0-9_]+' "$file" 2>/dev/null |
|
||||
sort -u || true); do
|
||||
case " $seam_allowed " in
|
||||
*" $found "*) ;;
|
||||
*) seam_unexpected="$seam_unexpected $found" ;;
|
||||
esac
|
||||
done
|
||||
if [ -n "$seam_unexpected" ]; then
|
||||
report FAIL A2-no-test-seams "$token" \
|
||||
"unexpected CBM_TEST_* seam(s):$seam_unexpected (allowlist: $seam_allowed)"
|
||||
else
|
||||
report PASS A2-no-test-seams "$token" 'no CBM_TEST_* seams beyond the smoke allowlist'
|
||||
fi
|
||||
|
||||
# A3 — updater/release URLs.
|
||||
for needle in "${UPDATER_NEEDLES[@]}"; do
|
||||
assert_absent "$file" "$token" A3-no-updater-urls "$needle"
|
||||
done
|
||||
|
||||
# A4 — SQLite loadable-extension surface.
|
||||
for needle in "${SQLITE_LOADEXT_NEEDLES[@]}"; do
|
||||
assert_absent "$file" "$token" A4-no-sqlite-loadext "$needle"
|
||||
done
|
||||
# Positive control for A4: the three needles above are all *absent* on a
|
||||
# stripped build even when the feature is compiled in, so on its own that
|
||||
# assertion can pass vacuously. SQLite's own compile-option table names
|
||||
# every OMIT_* it was built with — when that table is present (it ships
|
||||
# unless SQLITE_OMIT_COMPILEOPTION_DIAGS is set) it proves the omission
|
||||
# positively rather than by absence.
|
||||
if LC_ALL=C grep -a -q -F -e 'sqlite_compileoption_get' "$file"; then
|
||||
assert_present "$file" "$token" A4-no-sqlite-loadext 'OMIT_LOAD_EXTENSION' \
|
||||
'sqlite reports SQLITE_OMIT_LOAD_EXTENSION' \
|
||||
'sqlite was NOT built with SQLITE_OMIT_LOAD_EXTENSION, so the dlopen()/LoadLibrary() extension loader is linked in'
|
||||
else
|
||||
printf 'n/a %-22s %s: %s\n' A4-no-sqlite-loadext "$token" \
|
||||
"sqlite compile-option table absent; A4 rests on absence needles only"
|
||||
fi
|
||||
|
||||
# A5 — UI/HTTP subsystem, standard artifacts only.
|
||||
#
|
||||
# NOT YET ENFORCED. src/ui/{config,http_server,layout3d,httpd,embedded_stub}.c
|
||||
# are still in PROD_SRCS, and the standard build merely substitutes empty
|
||||
# embedded assets — so the HTTP server and its popen("ps … | grep") process
|
||||
# enumerator are linked into artifacts that can never serve a UI. Excluding
|
||||
# them is a real refactor: four files outside src/ui (src/main.c,
|
||||
# src/daemon/host.c, src/daemon/application.c, src/mcp/index_supervisor.c)
|
||||
# reference UI symbols, including the daemon that serves the UI.
|
||||
#
|
||||
# Until that lands, this assertion reports rather than fails. A gate everyone
|
||||
# knows is red teaches people to ignore gates, and a known-failing check has
|
||||
# no more enforcement value than this note — but the needles are kept live and
|
||||
# exercised so that the day the split lands, flipping CBM_CHECK_UI_ABSENT=1
|
||||
# (then making it the default) is a one-line change and not a rewrite.
|
||||
if [ "$is_ui" -eq 1 ]; then
|
||||
printf 'n/a %-22s %s: UI artifact, HTTP server ships here by design\n' \
|
||||
A5-no-ui-http "$token"
|
||||
elif [ "${CBM_CHECK_UI_ABSENT:-0}" = "1" ]; then
|
||||
for needle in "${UI_HTTP_NEEDLES[@]}"; do
|
||||
assert_absent "$file" "$token" A5-no-ui-http "$needle"
|
||||
done
|
||||
else
|
||||
ui_hits=0
|
||||
for needle in "${UI_HTTP_NEEDLES[@]}"; do
|
||||
if LC_ALL=C grep -a -q -F -e "$needle" "$file"; then
|
||||
ui_hits=$((ui_hits + 1))
|
||||
fi
|
||||
done
|
||||
printf 'INFO %-22s %s: %d/%d UI/HTTP needles present (no-UI split pending; set CBM_CHECK_UI_ABSENT=1 to enforce)\n' \
|
||||
A5-no-ui-http "$token" "$ui_hits" "${#UI_HTTP_NEEDLES[@]}"
|
||||
fi
|
||||
}
|
||||
|
||||
# ── Walk the targets ────────────────────────────────────────────────
|
||||
for target in "${TARGETS[@]}"; do
|
||||
if [ -d "$target" ]; then
|
||||
# "! -type d" rather than "-type f": a symlinked or otherwise unusual
|
||||
# artifact must produce a visible skip line, never vanish from the run.
|
||||
# sort keeps the report order stable across platforms (find order is not).
|
||||
while IFS= read -r f; do
|
||||
[ -n "$f" ] && check_file "$f"
|
||||
done <<EOF
|
||||
$(find "$target" ! -type d | LC_ALL=C sort)
|
||||
EOF
|
||||
elif [ -f "$target" ]; then
|
||||
check_file "$target"
|
||||
else
|
||||
echo "FAIL: $target is neither a file nor a directory" >&2
|
||||
exit 2
|
||||
fi
|
||||
done
|
||||
|
||||
printf '\n'
|
||||
if [ "$checked_files" -eq 0 ]; then
|
||||
echo "FAIL: no ELF/Mach-O/PE binary found in the given targets ($skipped_files file(s) skipped)" >&2
|
||||
echo " a gate that checked nothing is not a green gate" >&2
|
||||
exit 2
|
||||
fi
|
||||
if [ "$fail_count" -ne 0 ]; then
|
||||
echo "BINARY COMPOSITION GATE FAILED: $fail_count assertion(s) over $checked_files binary/binaries" >&2
|
||||
for f in "${FAILURES[@]}"; do
|
||||
echo " - $f" >&2
|
||||
done
|
||||
exit 1
|
||||
fi
|
||||
echo "BINARY COMPOSITION OK: $pass_count assertion(s) passed over $checked_files binary/binaries"
|
||||
@@ -169,6 +169,11 @@ if [ "$GOOS" = "windows" ]; then
|
||||
trap 'rm -rf "$PACK_DIR"' EXIT
|
||||
cp "$PAYLOAD" "$PACK_DIR/codebase-memory-mcp.exe"
|
||||
strip_release_binary "$PACK_DIR/codebase-memory-mcp.exe" || exit 2
|
||||
# Gate the artifact AFTER strip: strip is the last byte-changing step, so
|
||||
# this inspects exactly what goes into the archive. Runs here rather than in
|
||||
# a workflow step so the local artifact-flow smoke enforces the same thing.
|
||||
scripts/ci/check-binary-composition.sh --variant="$VARIANT" \
|
||||
"$PACK_DIR/codebase-memory-mcp.exe" || exit 2
|
||||
cp LICENSE install.ps1 "$PACK_DIR/"
|
||||
scripts/gen-third-party-notices.sh "$PACK_DIR/THIRD_PARTY_NOTICES.md"
|
||||
(
|
||||
@@ -182,6 +187,8 @@ else
|
||||
[ -f "$BUILD_DIR/codebase-memory-mcp" ] ||
|
||||
{ echo "package-release: build first; missing $BUILD_DIR/codebase-memory-mcp" >&2; exit 2; }
|
||||
strip_release_binary "$BUILD_DIR/codebase-memory-mcp" || exit 2
|
||||
scripts/ci/check-binary-composition.sh --variant="$VARIANT" \
|
||||
"$BUILD_DIR/codebase-memory-mcp" || exit 2
|
||||
cp LICENSE install.sh "$BUILD_DIR/"
|
||||
scripts/gen-third-party-notices.sh "$BUILD_DIR/THIRD_PARTY_NOTICES.md"
|
||||
tar -czf "$OUT_DIR/$NAME.tar.gz" -C "$BUILD_DIR" \
|
||||
|
||||
+7
-1
@@ -259,8 +259,14 @@ fi
|
||||
|
||||
# Step 5: Parent-death watchdog regression (#406/#407). Builds the prod stdio
|
||||
# binary and verifies it self-exits when its launching parent is killed.
|
||||
#
|
||||
# TEST_SEAMS=1: the worker-mode leg below needs the crash-orphan probe, which is
|
||||
# compiled out of ordinary builds (it forks a SIGTERM-ignoring child — see
|
||||
# src/main.c). Requesting it HERE, in the leg that consumes it, is what keeps
|
||||
# release artifacts free of it; scripts/ci/check-binary-composition.sh proves
|
||||
# they stay that way.
|
||||
echo "=== Step 5: parent-death watchdog regression (#406/#407) ==="
|
||||
make -j"$NPROC" -f Makefile.cbm cbm ${MAKE_ARGS[@]+"${MAKE_ARGS[@]}"}
|
||||
make -j"$NPROC" -f Makefile.cbm cbm TEST_SEAMS=1 ${MAKE_ARGS[@]+"${MAKE_ARGS[@]}"}
|
||||
WATCHDOG_BINARY="$ROOT/$BUILD_DIR/codebase-memory-mcp"
|
||||
CBM_TEST_BINARY="$WATCHDOG_BINARY" bash "$ROOT/tests/test_parent_watchdog.sh"
|
||||
|
||||
|
||||
+17
-1
@@ -5947,7 +5947,21 @@ static int cli_ensure_windows_user_path(const char *bin_dir, bool dry_run) {
|
||||
|
||||
#endif
|
||||
|
||||
/* ── Tar.gz extraction ────────────────────────────────────────── */
|
||||
/* ── Tar.gz / zip extraction (TEST-ONLY) ──────────────────────────
|
||||
*
|
||||
* The only callers of this block are the in-process updater — already excluded
|
||||
* from release builds — and tests/test_cli.c. The DEFINITIONS were nevertheless
|
||||
* unguarded, so every shipped binary carried a complete archive extractor with
|
||||
* no way to reach it: the translation unit is compiled and linked whole, with no
|
||||
* LTO or function-section garbage collection to drop it.
|
||||
*
|
||||
* "Download an archive, decompress it in memory, pick an executable out of it,
|
||||
* write it to disk and mark it executable" is the canonical dropper composite.
|
||||
* We do not do that in production, and now we cannot: the capability is not in
|
||||
* the artifact rather than merely unreachable within it. Verified by
|
||||
* scripts/ci/check-binary-composition.sh.
|
||||
*/
|
||||
#ifdef CBM_CLI_ENABLE_TEST_API
|
||||
|
||||
/* Decompress gzip data into a malloc'd buffer. Returns NULL on failure.
|
||||
* *out_total receives the decompressed size. Caller must free the result. */
|
||||
@@ -6217,6 +6231,8 @@ unsigned char *cbm_extract_binary_from_zip(const unsigned char *data, int data_l
|
||||
return NULL;
|
||||
}
|
||||
|
||||
#endif /* CBM_CLI_ENABLE_TEST_API — tar.gz / zip extraction */
|
||||
|
||||
/* ── Index management ─────────────────────────────────────────── */
|
||||
|
||||
static const char *get_cache_dir(const char *home_dir) {
|
||||
|
||||
+7
-1
@@ -336,7 +336,11 @@ int cbm_ensure_path(const char *bin_dir, const char *rc_file, bool dry_run);
|
||||
/* Get the Codex CLI instructions content. */
|
||||
const char *cbm_get_codex_instructions(void);
|
||||
|
||||
/* ── Tar.gz extraction ────────────────────────────────────────── */
|
||||
/* ── Tar.gz / zip extraction (TEST-ONLY) ──────────────────────────
|
||||
* Reachable only from the excluded in-process updater and tests/test_cli.c.
|
||||
* Declared and defined under the test guard so the capability is absent from
|
||||
* release artifacts rather than present-but-unreachable. */
|
||||
#ifdef CBM_CLI_ENABLE_TEST_API
|
||||
|
||||
/* Extract a binary named "codebase-memory-mcp*" from a tar.gz buffer.
|
||||
* Returns malloc'd binary content and sets *out_len.
|
||||
@@ -348,6 +352,8 @@ unsigned char *cbm_extract_binary_from_targz(const unsigned char *data, int data
|
||||
* Returns NULL on error. Caller must free. */
|
||||
unsigned char *cbm_extract_binary_from_zip(const unsigned char *data, int data_len, int *out_len);
|
||||
|
||||
#endif /* CBM_CLI_ENABLE_TEST_API */
|
||||
|
||||
/* ── Index management ─────────────────────────────────────────── */
|
||||
|
||||
/* List .db files in the cache directory (~/.cache/codebase-memory-mcp/).
|
||||
|
||||
+28
-148
@@ -55,15 +55,25 @@ enum {
|
||||
APPLICATION_MARKER_MAX_BYTES = 64 * 1024 * 1024,
|
||||
APPLICATION_MAX_SUSPECTS = 65536,
|
||||
APPLICATION_UPDATE_POLL_US = 10000,
|
||||
APPLICATION_UPDATE_TIMEOUT_MS = 7000,
|
||||
APPLICATION_BACKGROUND_REAP_MS = 10000,
|
||||
APPLICATION_UPDATE_VERSION_CAP = 128,
|
||||
APPLICATION_UPDATE_NOTICE_CAP = 1024,
|
||||
APPLICATION_UPDATE_RESPONSE_MAX = 1024 * 1024,
|
||||
};
|
||||
|
||||
#define APPLICATION_UPDATE_URL \
|
||||
"https://api.github.com/repos/DeusData/codebase-memory-mcp/releases/latest"
|
||||
/* There is deliberately NO production update-check provider. The daemon used to
|
||||
* spawn `curl` against the GitHub releases API on the first eligible session of
|
||||
* every run, purely to print "a newer version exists". That put a release URL
|
||||
* and an outbound request into every shipped binary, and made a developer tool
|
||||
* phone home from every agent session, to deliver something the install scripts
|
||||
* and package managers already report.
|
||||
*
|
||||
* The SEAM below survives: `update_ops` remains injectable, and the notice,
|
||||
* ownership, cancellation and generation-replay logic is still exercised by the
|
||||
* fakes in tests/test_daemon_application.c. With no provider installed the whole
|
||||
* machinery simply never starts a generation (see
|
||||
* application_update_subscribe_locked), so a build that ships no provider makes
|
||||
* no network request by default -- the property, not just the absence of a call.
|
||||
*/
|
||||
|
||||
typedef struct cbm_daemon_application_watch cbm_daemon_application_watch_t;
|
||||
typedef struct cbm_daemon_application_session cbm_daemon_application_session_t;
|
||||
@@ -183,13 +193,6 @@ struct cbm_daemon_application {
|
||||
bool permanent;
|
||||
};
|
||||
|
||||
typedef struct {
|
||||
cbm_subprocess_t *process;
|
||||
char output_path[APPLICATION_PATH_CAP];
|
||||
char latest_version[APPLICATION_UPDATE_VERSION_CAP];
|
||||
bool terminal;
|
||||
} application_update_worker_t;
|
||||
|
||||
static void application_job_unsubscribe_locked(cbm_daemon_application_job_t *job);
|
||||
static void application_watch_job_unsubscribe_session_locked(
|
||||
cbm_daemon_application_session_t *session);
|
||||
@@ -630,132 +633,6 @@ static bool application_unique_recovery_file(char out[APPLICATION_PATH_CAP], con
|
||||
return true;
|
||||
}
|
||||
|
||||
static int application_update_worker_start_default(
|
||||
void *context, cbm_daemon_application_update_worker_t *worker_out) {
|
||||
(void)context;
|
||||
if (!worker_out) {
|
||||
return -1;
|
||||
}
|
||||
*worker_out = NULL;
|
||||
application_update_worker_t *worker = calloc(1, sizeof(*worker));
|
||||
if (!worker || !application_unique_recovery_file(worker->output_path, "update")) {
|
||||
free(worker);
|
||||
return -1;
|
||||
}
|
||||
const char *argv[] = {
|
||||
"curl",
|
||||
"-sf",
|
||||
"--max-time",
|
||||
"5",
|
||||
"--max-filesize",
|
||||
"1048576",
|
||||
"-H",
|
||||
"Accept: application/vnd.github+json",
|
||||
APPLICATION_UPDATE_URL,
|
||||
NULL,
|
||||
};
|
||||
cbm_proc_opts_t options = {
|
||||
.bin = "curl",
|
||||
.argv = argv,
|
||||
.log_file = worker->output_path,
|
||||
.quiet_timeout_ms = APPLICATION_UPDATE_TIMEOUT_MS,
|
||||
.cancel_grace_ms = CBM_SUBPROCESS_DEFAULT_CANCEL_GRACE_MS,
|
||||
.delete_log_on_exit = false,
|
||||
};
|
||||
if (cbm_subprocess_spawn(&options, &worker->process) != 0) {
|
||||
(void)cbm_unlink(worker->output_path);
|
||||
free(worker);
|
||||
return -1;
|
||||
}
|
||||
*worker_out = worker;
|
||||
return 0;
|
||||
}
|
||||
|
||||
static void application_update_worker_read_version(application_update_worker_t *worker) {
|
||||
int64_t size = cbm_file_size(worker->output_path);
|
||||
if (size <= 0 || size > APPLICATION_UPDATE_RESPONSE_MAX) {
|
||||
return;
|
||||
}
|
||||
FILE *file = cbm_fopen(worker->output_path, "rb");
|
||||
if (!file) {
|
||||
return;
|
||||
}
|
||||
char *bytes = malloc((size_t)size);
|
||||
size_t read = bytes ? fread(bytes, 1, (size_t)size, file) : 0;
|
||||
(void)fclose(file);
|
||||
if (read != (size_t)size) {
|
||||
free(bytes);
|
||||
return;
|
||||
}
|
||||
yyjson_doc *document = yyjson_read(bytes, read, 0);
|
||||
yyjson_val *root = document ? yyjson_doc_get_root(document) : NULL;
|
||||
yyjson_val *tag = yyjson_is_obj(root) ? yyjson_obj_get(root, "tag_name") : NULL;
|
||||
const char *version = yyjson_is_str(tag) ? yyjson_get_str(tag) : NULL;
|
||||
if (version && version[0] && strlen(version) < sizeof(worker->latest_version)) {
|
||||
bool valid = true;
|
||||
for (const unsigned char *cursor = (const unsigned char *)version; *cursor; cursor++) {
|
||||
if (!(isalnum(*cursor) || *cursor == '.' || *cursor == '-' || *cursor == '_' ||
|
||||
*cursor == '+')) {
|
||||
valid = false;
|
||||
break;
|
||||
}
|
||||
}
|
||||
if (valid) {
|
||||
(void)snprintf(worker->latest_version, sizeof(worker->latest_version), "%s", version);
|
||||
}
|
||||
}
|
||||
yyjson_doc_free(document);
|
||||
free(bytes);
|
||||
}
|
||||
|
||||
static cbm_daemon_application_update_poll_t application_update_worker_poll_default(
|
||||
void *context, cbm_daemon_application_update_worker_t handle, const char **latest_version_out) {
|
||||
(void)context;
|
||||
if (latest_version_out) {
|
||||
*latest_version_out = NULL;
|
||||
}
|
||||
application_update_worker_t *worker = handle;
|
||||
if (!worker || !worker->process || !latest_version_out) {
|
||||
return CBM_DAEMON_APPLICATION_UPDATE_POLL_ERROR;
|
||||
}
|
||||
if (!worker->terminal) {
|
||||
cbm_proc_result_t result;
|
||||
cbm_proc_poll_t status = cbm_subprocess_poll(worker->process, &result);
|
||||
if (status == CBM_PROC_POLL_RUNNING) {
|
||||
return CBM_DAEMON_APPLICATION_UPDATE_POLL_RUNNING;
|
||||
}
|
||||
if (status != CBM_PROC_POLL_TERMINAL) {
|
||||
return CBM_DAEMON_APPLICATION_UPDATE_POLL_ERROR;
|
||||
}
|
||||
worker->terminal = true;
|
||||
if (result.outcome == CBM_PROC_CLEAN && result.exit_code == 0 && result.tree_quiesced &&
|
||||
!result.supervision_failed && !result.cancellation_requested) {
|
||||
application_update_worker_read_version(worker);
|
||||
}
|
||||
}
|
||||
*latest_version_out = worker->latest_version[0] ? worker->latest_version : NULL;
|
||||
return CBM_DAEMON_APPLICATION_UPDATE_POLL_TERMINAL;
|
||||
}
|
||||
|
||||
static bool application_update_worker_cancel_default(
|
||||
void *context, cbm_daemon_application_update_worker_t handle) {
|
||||
(void)context;
|
||||
application_update_worker_t *worker = handle;
|
||||
return worker && worker->process && cbm_subprocess_request_cancel(worker->process);
|
||||
}
|
||||
|
||||
static void application_update_worker_destroy_default(
|
||||
void *context, cbm_daemon_application_update_worker_t handle) {
|
||||
(void)context;
|
||||
application_update_worker_t *worker = handle;
|
||||
if (!worker) {
|
||||
return;
|
||||
}
|
||||
cbm_subprocess_destroy(worker->process);
|
||||
(void)cbm_unlink(worker->output_path);
|
||||
free(worker);
|
||||
}
|
||||
|
||||
static bool application_recovery_files_create(char marker_path[APPLICATION_PATH_CAP],
|
||||
char quarantine_path[APPLICATION_PATH_CAP]) {
|
||||
if (!application_unique_recovery_file(marker_path, "marker")) {
|
||||
@@ -1939,6 +1816,13 @@ static void *application_update_thread(void *opaque) {
|
||||
|
||||
static void application_update_subscribe_locked(cbm_daemon_application_session_t *session) {
|
||||
cbm_daemon_application_t *application = session->application;
|
||||
/* No provider, no generation. This is what makes "the daemon performs no
|
||||
* network request by default" a structural property rather than a promise:
|
||||
* with update_ops empty nothing is ever started, so no session can observe
|
||||
* a generation, become its owner, or wait on it. */
|
||||
if (!application->update_ops.start) {
|
||||
return;
|
||||
}
|
||||
if (application->update_generation_started) {
|
||||
if (application->update_thread_started && !application->update_thread_done &&
|
||||
!application->update_cancel_requested && !session->update_owner) {
|
||||
@@ -2894,17 +2778,13 @@ cbm_daemon_application_t *cbm_daemon_application_new(
|
||||
free(application);
|
||||
return NULL;
|
||||
}
|
||||
if (!application->update_ops.start) {
|
||||
application->update_ops = (cbm_daemon_application_update_ops_t){
|
||||
.context = NULL,
|
||||
.start = application_update_worker_start_default,
|
||||
.poll = application_update_worker_poll_default,
|
||||
.cancel = application_update_worker_cancel_default,
|
||||
.destroy = application_update_worker_destroy_default,
|
||||
};
|
||||
}
|
||||
if (!application->update_ops.poll || !application->update_ops.cancel ||
|
||||
!application->update_ops.destroy) {
|
||||
/* No provider means no update checking, which is the production default now
|
||||
* that the curl-based GitHub check is gone. An INCOMPLETE provider is still
|
||||
* a programming error: a caller that supplies `start` must supply the whole
|
||||
* quartet, or the thread would start work it cannot poll, cancel or free. */
|
||||
if (application->update_ops.start &&
|
||||
(!application->update_ops.poll || !application->update_ops.cancel ||
|
||||
!application->update_ops.destroy)) {
|
||||
cbm_mutex_destroy(&application->mutex);
|
||||
free(application);
|
||||
return NULL;
|
||||
|
||||
@@ -29,7 +29,10 @@
|
||||
|
||||
#ifdef _WIN32
|
||||
#include "foundation/win_utf8.h"
|
||||
#include <fcntl.h> /* _O_* for the exclusive stats-file create */
|
||||
#include <io.h> /* _wopen / _close */
|
||||
#include <process.h>
|
||||
#include <sys/stat.h> /* _S_IREAD / _S_IWRITE */
|
||||
#include <windows.h>
|
||||
#define getpid _getpid
|
||||
#else
|
||||
@@ -136,6 +139,52 @@ static void diag_stats_write(const char *text, void *arg) {
|
||||
}
|
||||
}
|
||||
|
||||
/* Create the stats file at its fixed, discoverable path without ever writing
|
||||
* through something another process planted there. Exclusive creation IS the
|
||||
* guarantee, so the predictable name stays safe: the unlink drops a stale file
|
||||
* from an earlier run with this pid (and, if a local attacker pre-created a
|
||||
* symlink, removes the link itself — never its target), and the O_EXCL create
|
||||
* that follows fails closed if anything reappears at the path in between. The
|
||||
* write therefore lands on a file this process just created, or not at all.
|
||||
* O_NOFOLLOW is belt-and-braces for the same window on POSIX. Mode 0600: the
|
||||
* snapshot describes this process's heap layout, so it is owner-only. */
|
||||
static FILE *diag_open_private_stats_file(const char *path) {
|
||||
(void)cbm_unlink(path);
|
||||
#ifdef _WIN32
|
||||
/* _wopen mirrors cbm_mkstemp's Windows contract — the ANSI CRT interprets
|
||||
* the UTF-8 bytes of a non-ASCII %TEMP% in the local codepage and fails. */
|
||||
wchar_t *wide = cbm_path_to_wide(path);
|
||||
if (!wide) {
|
||||
return NULL;
|
||||
}
|
||||
int descriptor = _wopen(wide, _O_WRONLY | _O_CREAT | _O_EXCL | _O_BINARY | _O_NOINHERIT,
|
||||
_S_IREAD | _S_IWRITE);
|
||||
free(wide);
|
||||
if (descriptor < 0) {
|
||||
return NULL;
|
||||
}
|
||||
FILE *sink = _fdopen(descriptor, "wb");
|
||||
if (!sink) {
|
||||
(void)_close(descriptor);
|
||||
}
|
||||
return sink;
|
||||
#else
|
||||
int flags = O_WRONLY | O_CREAT | O_EXCL | O_CLOEXEC;
|
||||
#ifdef O_NOFOLLOW
|
||||
flags |= O_NOFOLLOW;
|
||||
#endif
|
||||
int descriptor = open(path, flags, 0600);
|
||||
if (descriptor < 0) {
|
||||
return NULL;
|
||||
}
|
||||
FILE *sink = fdopen(descriptor, "wb");
|
||||
if (!sink) {
|
||||
(void)close(descriptor);
|
||||
}
|
||||
return sink;
|
||||
#endif
|
||||
}
|
||||
|
||||
static void diag_write_allocator_stats(void) {
|
||||
char flag[CBM_SZ_16];
|
||||
if (cbm_safe_getenv("CBM_MEM_STATS", flag, sizeof(flag), NULL) == NULL || flag[0] != '1') {
|
||||
@@ -145,13 +194,14 @@ static void diag_write_allocator_stats(void) {
|
||||
/* Deliberately NOT inside the diagnostics directory: that tree is
|
||||
* owner-private with anchored (openat-based) writes on POSIX, which a plain
|
||||
* path-based open does not satisfy. This file is a developer diagnostic, so
|
||||
* a predictable temp path keeps it working identically on every platform. */
|
||||
* a predictable temp path keeps it working identically on every platform —
|
||||
* diag_open_private_stats_file is what makes that path safe to write. */
|
||||
int written =
|
||||
snprintf(path, sizeof(path), "%s/cbm-allocator-stats-%d.txt", cbm_tmpdir(), (int)getpid());
|
||||
if (written <= 0 || (size_t)written >= sizeof(path)) {
|
||||
return;
|
||||
}
|
||||
FILE *sink = cbm_fopen(path, "wb");
|
||||
FILE *sink = diag_open_private_stats_file(path);
|
||||
if (!sink) {
|
||||
return;
|
||||
}
|
||||
|
||||
+39
-6
@@ -201,11 +201,22 @@ static void main_project_lock_release_fully(cbm_project_lock_lease_t **lease) {
|
||||
}
|
||||
}
|
||||
|
||||
/* Test-only ownership proof for the real-binary POSIX smoke. The environment
|
||||
* variable is otherwise inert, and only a supervised physical worker may
|
||||
* publish it. Publication occurs after the native project lease is acquired,
|
||||
* so a marker from the worker also proves that its polling supervisor did not
|
||||
* retain the same exclusive lease. */
|
||||
/* Test-only ownership proof consumed by the POSIX worker-lease contract tests.
|
||||
* The environment variable is otherwise inert, and only a supervised physical
|
||||
* worker may publish it. Publication occurs after the native project lease is
|
||||
* acquired, so a marker from the worker also proves that its polling supervisor
|
||||
* did not retain the same exclusive lease.
|
||||
*
|
||||
* COMPILED OUT of ordinary builds alongside the watchdog probe above. This one
|
||||
* is benign in isolation (an O_EXCL|O_NOFOLLOW PID file), but it is still
|
||||
* test-only code reachable through a caller-supplied path in a shipped binary,
|
||||
* and its consumers all build with TEST_SEAMS=1. The two seams smoke genuinely
|
||||
* needs against real release artifacts (CBM_TEST_CRASH_ON / CBM_TEST_HANG_ON
|
||||
* fault injection, and CBM_TEST_WINDOWS_USER_PATH_RUN_ID, which is what keeps
|
||||
* the PATH smoke from touching the real user PATH) deliberately REMAIN: smoke's
|
||||
* whole value is exercising the artifact we ship, and removing them would trade
|
||||
* release-artifact coverage for a cosmetic win. */
|
||||
#ifdef CBM_ENABLE_TEST_SEAMS
|
||||
static bool main_test_worker_project_lock_marker(const main_local_cli_mutation_t *mutation) {
|
||||
#ifdef _WIN32
|
||||
(void)mutation;
|
||||
@@ -238,6 +249,12 @@ static bool main_test_worker_project_lock_marker(const main_local_cli_mutation_t
|
||||
return close(marker) == 0 && written;
|
||||
#endif
|
||||
}
|
||||
#else
|
||||
static bool main_test_worker_project_lock_marker(const main_local_cli_mutation_t *mutation) {
|
||||
(void)mutation;
|
||||
return true;
|
||||
}
|
||||
#endif
|
||||
|
||||
static bool main_local_cli_mutation_begin(void *context, const char *project) {
|
||||
main_local_cli_mutation_t *mutation = context;
|
||||
@@ -390,7 +407,18 @@ static bool worker_prepare_process_group(void) {
|
||||
|
||||
/* Test-only crash-orphan probe used by tests/test_worker_watchdog.sh. It is
|
||||
* created before the watchdog thread so fork never occurs in a multithreaded
|
||||
* worker, and inherits the worker's isolated process group. */
|
||||
* worker, and inherits the worker's isolated process group.
|
||||
*
|
||||
* COMPILED OUT of ordinary builds (see TEST_SEAMS in Makefile.cbm). "Fork a
|
||||
* child that ignores SIGTERM and loops forever, then write its PID to a path
|
||||
* the caller chose" is a fine test probe and an appalling thing to find in a
|
||||
* shipped executable — it is precisely the shape a generic malware classifier
|
||||
* is built to notice, and it has no production caller. Seams are OPT-IN so the
|
||||
* failure mode of forgetting the flag is a clean binary, not a leaky one; the
|
||||
* suites that need it build with TEST_SEAMS=1, and
|
||||
* scripts/ci/check-binary-composition.sh fails the release if the marker
|
||||
* string ever reappears in an artifact. */
|
||||
#ifdef CBM_ENABLE_TEST_SEAMS
|
||||
static bool worker_start_watchdog_test_descendant(void) {
|
||||
char pid_path[CBM_SZ_4K] = {0};
|
||||
if (!cbm_safe_getenv("CBM_TEST_WORKER_DESCENDANT_PID_FILE", pid_path, sizeof(pid_path), NULL) ||
|
||||
@@ -429,6 +457,11 @@ static bool worker_start_watchdog_test_descendant(void) {
|
||||
}
|
||||
return written;
|
||||
}
|
||||
#else
|
||||
static bool worker_start_watchdog_test_descendant(void) {
|
||||
return true;
|
||||
}
|
||||
#endif
|
||||
|
||||
static bool worker_start_parent_watchdog(pid_t initial_ppid) {
|
||||
static parent_watchdog_config_t worker_config;
|
||||
|
||||
+137
-27
@@ -74,10 +74,16 @@ enum {
|
||||
#include <process.h>
|
||||
#include <windows.h>
|
||||
#define getpid _getpid
|
||||
/* Write through the descriptor cbm_mkstemp returned rather than reopening its
|
||||
* path — see search_scratch_open. Mirrors config_toml_edit.c's toml_fdopen. */
|
||||
#define mcp_fdopen _fdopen
|
||||
#define mcp_close _close
|
||||
#else
|
||||
#include <unistd.h>
|
||||
#include <poll.h>
|
||||
#include <fcntl.h>
|
||||
#define mcp_fdopen fdopen
|
||||
#define mcp_close close
|
||||
#endif
|
||||
#include <yyjson/yyjson.h>
|
||||
#include <ctype.h>
|
||||
@@ -8972,10 +8978,14 @@ static void classify_all_grep_hits(grep_match_t *gm, int gm_count, cbm_store_t *
|
||||
* normalized on Windows first), so prefiltering can only skip files whose
|
||||
* hits would be dropped anyway — results-preserving by construction.
|
||||
* *out_written receives the number of records written (0 = the filter
|
||||
* excluded every indexed file). */
|
||||
* excluded every indexed file).
|
||||
*
|
||||
* `fl` is the caller's already-open binary stream on the descriptor cbm_mkstemp
|
||||
* created inside the private scratch directory; this function never opens or
|
||||
* closes it, so the list is never reachable through a predictable pathname. */
|
||||
static bool write_scoped_filelist(cbm_mcp_server_t *srv, const char *project, const char *root_path,
|
||||
const char *filelist, bool has_path_filter,
|
||||
cbm_regex_t *path_regex, int *out_written) {
|
||||
FILE *fl, bool has_path_filter, cbm_regex_t *path_regex,
|
||||
int *out_written) {
|
||||
*out_written = 0;
|
||||
cbm_store_t *pre_store = resolve_store(srv, project);
|
||||
if (!pre_store) {
|
||||
@@ -8987,7 +8997,6 @@ static bool write_scoped_filelist(cbm_mcp_server_t *srv, const char *project, co
|
||||
indexed_count == 0) {
|
||||
return false;
|
||||
}
|
||||
FILE *fl = fopen(filelist, "wb");
|
||||
bool ok = false;
|
||||
int written = 0;
|
||||
if (fl) {
|
||||
@@ -9026,7 +9035,8 @@ static bool write_scoped_filelist(cbm_mcp_server_t *srv, const char *project, co
|
||||
#endif
|
||||
written++;
|
||||
}
|
||||
(void)fclose(fl);
|
||||
/* The stream stays open — the caller owns it and closes it (flushing
|
||||
* these records to disk) before the grep subprocess reads the list. */
|
||||
ok = true;
|
||||
}
|
||||
for (int fi = 0; fi < indexed_count; fi++) {
|
||||
@@ -9107,15 +9117,115 @@ static bool validate_search_args(const char *root_path, const char *file_pattern
|
||||
return true;
|
||||
}
|
||||
|
||||
/* Write pattern to a temp file for grep -f. Returns true on success. */
|
||||
static bool write_pattern_file(char *tmpfile, int tmpfile_sz, const char *pattern) {
|
||||
snprintf(tmpfile, tmpfile_sz, "%s/cbm_search_%d.pat", cbm_tmpdir(), (int)getpid());
|
||||
FILE *tf = fopen(tmpfile, "w");
|
||||
if (!tf) {
|
||||
/* Private scratch for one search_code scan: the grep -f pattern file and the
|
||||
* scoped file list.
|
||||
*
|
||||
* Both used to be fixed, guessable paths derived from the pid —
|
||||
* "<tmp>/cbm_search_<pid>.pat" and its ".files" companion — opened with a plain
|
||||
* fopen. Another local user could pre-plant a symlink at either name and
|
||||
* redirect the write; two searches in the same process could also collide on
|
||||
* them. Now both live inside a directory created by cbm_mkdtemp (0700 on POSIX,
|
||||
* an explicit owner-only DACL on Windows) under an unguessable XXXXXX suffix,
|
||||
* and each file is created by cbm_mkstemp — O_CREAT|O_EXCL at mode 0600, so the
|
||||
* create fails rather than following anything already at the name. Every write
|
||||
* goes through the descriptor cbm_mkstemp returned; neither path is ever
|
||||
* reopened by name.
|
||||
*
|
||||
* Sizing: cbm_mkdtemp copies its expanded result back into `dir`, and its own
|
||||
* internal buffer is CBM_SZ_512, so `dir` must be at least that big to receive
|
||||
* it. The two file paths are `dir` plus a short basename. */
|
||||
typedef struct {
|
||||
char dir[CBM_SZ_512];
|
||||
char pattern_path[CBM_SZ_1K];
|
||||
char filelist_path[CBM_SZ_1K];
|
||||
FILE *filelist; /* held open for write_scoped_filelist; closed by the caller */
|
||||
} search_scratch_t;
|
||||
|
||||
/* Create <scratch>/<basename>-XXXXXX exclusively and return a stream on the
|
||||
* descriptor. On failure `path_out` is emptied so cleanup skips it. */
|
||||
static FILE *search_scratch_file(const char *dir, const char *basename, char *path_out,
|
||||
size_t path_sz) {
|
||||
path_out[0] = '\0';
|
||||
int written = snprintf(path_out, path_sz, "%s/%s-XXXXXX", dir, basename);
|
||||
if (written <= 0 || (size_t)written >= path_sz) {
|
||||
path_out[0] = '\0';
|
||||
return NULL;
|
||||
}
|
||||
int descriptor = cbm_mkstemp(path_out);
|
||||
if (descriptor < 0) {
|
||||
path_out[0] = '\0';
|
||||
return NULL;
|
||||
}
|
||||
/* Binary mode: the file list uses an explicit per-platform record separator
|
||||
* (NUL for xargs -0, newline for PowerShell) that CRLF translation would
|
||||
* corrupt — the same reason the previous code opened it "wb". */
|
||||
FILE *stream = mcp_fdopen(descriptor, "wb");
|
||||
if (!stream) {
|
||||
(void)mcp_close(descriptor);
|
||||
(void)cbm_unlink(path_out);
|
||||
path_out[0] = '\0';
|
||||
}
|
||||
return stream;
|
||||
}
|
||||
|
||||
/* Anchored cleanup: removes both scratch files and the private directory. Safe
|
||||
* to call more than once and on any partially-initialised scratch, so every
|
||||
* exit from handle_search_code can call it unconditionally. rmdir succeeding is
|
||||
* itself the proof nothing was left inside. */
|
||||
static void search_scratch_close(search_scratch_t *scratch) {
|
||||
if (scratch->filelist) {
|
||||
(void)fclose(scratch->filelist);
|
||||
scratch->filelist = NULL;
|
||||
}
|
||||
if (scratch->pattern_path[0] != '\0') {
|
||||
(void)cbm_unlink(scratch->pattern_path);
|
||||
scratch->pattern_path[0] = '\0';
|
||||
}
|
||||
if (scratch->filelist_path[0] != '\0') {
|
||||
(void)cbm_unlink(scratch->filelist_path);
|
||||
scratch->filelist_path[0] = '\0';
|
||||
}
|
||||
if (scratch->dir[0] != '\0') {
|
||||
(void)cbm_rmdir(scratch->dir);
|
||||
scratch->dir[0] = '\0';
|
||||
}
|
||||
}
|
||||
|
||||
/* Open the scratch directory, write `pattern` to the grep -f file, and leave the
|
||||
* file list open for write_scoped_filelist. Returns true on success; on failure
|
||||
* everything already created is removed before returning. */
|
||||
static bool search_scratch_open(search_scratch_t *scratch, const char *pattern) {
|
||||
scratch->dir[0] = '\0';
|
||||
scratch->pattern_path[0] = '\0';
|
||||
scratch->filelist_path[0] = '\0';
|
||||
scratch->filelist = NULL;
|
||||
|
||||
int written =
|
||||
snprintf(scratch->dir, sizeof(scratch->dir), "%s/cbm-search-XXXXXX", cbm_tmpdir());
|
||||
if (written <= 0 || (size_t)written >= sizeof(scratch->dir) || !cbm_mkdtemp(scratch->dir)) {
|
||||
scratch->dir[0] = '\0';
|
||||
return false;
|
||||
}
|
||||
|
||||
FILE *pattern_file = search_scratch_file(scratch->dir, "pat", scratch->pattern_path,
|
||||
sizeof(scratch->pattern_path));
|
||||
if (!pattern_file) {
|
||||
search_scratch_close(scratch);
|
||||
return false;
|
||||
}
|
||||
bool ok = fprintf(pattern_file, "%s\n", pattern) >= 0;
|
||||
ok = fclose(pattern_file) == 0 && ok;
|
||||
if (!ok) {
|
||||
search_scratch_close(scratch);
|
||||
return false;
|
||||
}
|
||||
|
||||
scratch->filelist = search_scratch_file(scratch->dir, "files", scratch->filelist_path,
|
||||
sizeof(scratch->filelist_path));
|
||||
if (!scratch->filelist) {
|
||||
search_scratch_close(scratch);
|
||||
return false;
|
||||
}
|
||||
(void)fprintf(tf, "%s\n", pattern);
|
||||
(void)fclose(tf);
|
||||
return true;
|
||||
}
|
||||
|
||||
@@ -9245,8 +9355,8 @@ static char *handle_search_code(cbm_mcp_server_t *srv, const char *args) {
|
||||
}
|
||||
|
||||
/* ── Phase 1: Grep scan ──────────────────────────────────── */
|
||||
char tmpfile[CBM_SZ_256];
|
||||
if (!write_pattern_file(tmpfile, sizeof(tmpfile), pattern)) {
|
||||
search_scratch_t scratch;
|
||||
if (!search_scratch_open(&scratch, pattern)) {
|
||||
char errmsg[CBM_SZ_256];
|
||||
snprintf(errmsg, sizeof(errmsg), "search failed: cannot create temp file (%s)",
|
||||
strerror(errno));
|
||||
@@ -9256,6 +9366,8 @@ static char *handle_search_code(cbm_mcp_server_t *srv, const char *args) {
|
||||
free(file_pattern);
|
||||
return cbm_mcp_text_result(errmsg, true);
|
||||
}
|
||||
const char *tmpfile = scratch.pattern_path;
|
||||
const char *filelist = scratch.filelist_path;
|
||||
|
||||
/* No grep-level match limit — let grep find all matches, then dedup and
|
||||
* cap in our code. The -m flag caused results from large vendored files
|
||||
@@ -9267,13 +9379,16 @@ static char *handle_search_code(cbm_mcp_server_t *srv, const char *args) {
|
||||
* Query the graph for distinct file paths, write them to a temp file,
|
||||
* then use xargs to pass them to grep. Falls back to recursive grep if
|
||||
* no indexed files found (project not fully indexed). */
|
||||
char filelist[CBM_SZ_256];
|
||||
snprintf(filelist, sizeof(filelist), "%s.files", tmpfile);
|
||||
bool scoped = false;
|
||||
int scoped_written = 0;
|
||||
|
||||
scoped = write_scoped_filelist(srv, project, root_path, filelist, has_path_filter,
|
||||
scoped = write_scoped_filelist(srv, project, root_path, scratch.filelist, has_path_filter,
|
||||
has_path_filter ? &path_regex : NULL, &scoped_written);
|
||||
/* Close before grep runs: this is what flushes the records the helper wrote
|
||||
* through the descriptor. Clearing the field hands ownership to
|
||||
* search_scratch_close, which still unlinks the file itself. */
|
||||
(void)fclose(scratch.filelist);
|
||||
scratch.filelist = NULL;
|
||||
|
||||
/* Collect grep matches into array */
|
||||
int gm_count = 0;
|
||||
@@ -9284,8 +9399,7 @@ static char *handle_search_code(cbm_mcp_server_t *srv, const char *args) {
|
||||
* platform-dependent (GNU execs grep once with no operands, BSD
|
||||
* skips), and the post-grep filter would drop every hit anyway. */
|
||||
gm = malloc(sizeof(grep_match_t)); /* empty set; freed below */
|
||||
cbm_unlink(tmpfile);
|
||||
cbm_unlink(filelist);
|
||||
search_scratch_close(&scratch);
|
||||
} else {
|
||||
char cmd[CBM_SZ_4K];
|
||||
build_grep_cmd(cmd, sizeof(cmd), use_regex, scoped, file_pattern, tmpfile, filelist,
|
||||
@@ -9293,10 +9407,7 @@ static char *handle_search_code(cbm_mcp_server_t *srv, const char *args) {
|
||||
|
||||
FILE *fp = cbm_popen(cmd, "r");
|
||||
if (!fp) {
|
||||
cbm_unlink(tmpfile);
|
||||
if (scoped) {
|
||||
cbm_unlink(filelist);
|
||||
}
|
||||
search_scratch_close(&scratch);
|
||||
free(root_path);
|
||||
free(pattern);
|
||||
free(project);
|
||||
@@ -9307,10 +9418,9 @@ static char *handle_search_code(cbm_mcp_server_t *srv, const char *args) {
|
||||
gm = collect_grep_matches(fp, root_path, strlen(root_path), has_path_filter, &path_regex,
|
||||
grep_limit, &gm_count);
|
||||
cbm_pclose(fp);
|
||||
cbm_unlink(tmpfile);
|
||||
if (scoped) {
|
||||
cbm_unlink(filelist);
|
||||
}
|
||||
/* Both scratch files and the private directory go here — unlike the old
|
||||
* code, the file list is removed even when the scan was not scoped. */
|
||||
search_scratch_close(&scratch);
|
||||
}
|
||||
|
||||
/* ── Phase 2+3: Block expansion + graph ranking ──────────── */
|
||||
|
||||
+70
-13
@@ -434,6 +434,60 @@ static const char *DROP_INDEXES_SQL = "DROP INDEX IF EXISTS idx_nodes_label;"
|
||||
|
||||
/* ── Export helpers ───────────────────────────────────────────────── */
|
||||
|
||||
/* Owner-private scratch directory holding the snapshot copy.
|
||||
*
|
||||
* VACUUM INTO refuses to write a destination that already exists, so the
|
||||
* destination file cannot be pre-created with exclusive semantics the way
|
||||
* cbm_mkstemp would — sqlite has to be the one that creates it. Containing it in
|
||||
* a directory only this user can enter buys the same protection: cbm_mkdtemp
|
||||
* creates with 0700 on POSIX and an explicit owner-only DACL on Windows, and the
|
||||
* XXXXXX suffix makes the path unguessable. The old fixed
|
||||
* "<tmp>/cbm_artifact_tmp.db" was vulnerable on both counts — another local user
|
||||
* could pre-plant a symlink there to redirect the copy, and two concurrent
|
||||
* exports collided on the one name. */
|
||||
typedef struct {
|
||||
char dir[CBM_SZ_512]; /* cbm_mkdtemp copies its result back into this buffer */
|
||||
char db[CBM_SZ_4K];
|
||||
} artifact_snapshot_tmp_t;
|
||||
|
||||
static bool artifact_snapshot_tmp_open(artifact_snapshot_tmp_t *tmp) {
|
||||
tmp->dir[0] = '\0';
|
||||
tmp->db[0] = '\0';
|
||||
int written = snprintf(tmp->dir, sizeof(tmp->dir), "%s/cbm-artifact-XXXXXX", cbm_tmpdir());
|
||||
if (written <= 0 || (size_t)written >= sizeof(tmp->dir) || !cbm_mkdtemp(tmp->dir)) {
|
||||
tmp->dir[0] = '\0';
|
||||
return false;
|
||||
}
|
||||
written = snprintf(tmp->db, sizeof(tmp->db), "%s/snapshot.db", tmp->dir);
|
||||
if (written <= 0 || (size_t)written >= sizeof(tmp->db)) {
|
||||
(void)cbm_rmdir(tmp->dir);
|
||||
tmp->dir[0] = '\0';
|
||||
return false;
|
||||
}
|
||||
return true;
|
||||
}
|
||||
|
||||
/* Anchored cleanup — every exit from prepare_snapshot_db runs this, so neither
|
||||
* the copy nor its WAL/SHM sidecars outlive the export. Removing the directory
|
||||
* last doubles as the check that nothing was left inside it: rmdir only
|
||||
* succeeds once it is empty. */
|
||||
static void artifact_snapshot_tmp_close(artifact_snapshot_tmp_t *tmp) {
|
||||
if (tmp->dir[0] == '\0') {
|
||||
return;
|
||||
}
|
||||
static const char *const suffixes[] = {"-wal", "-shm"};
|
||||
for (size_t i = 0; i < sizeof(suffixes) / sizeof(suffixes[0]); i++) {
|
||||
char sidecar[CBM_SZ_4K];
|
||||
int written = snprintf(sidecar, sizeof(sidecar), "%s%s", tmp->db, suffixes[i]);
|
||||
if (written > 0 && (size_t)written < sizeof(sidecar)) {
|
||||
(void)cbm_unlink(sidecar);
|
||||
}
|
||||
}
|
||||
(void)cbm_unlink(tmp->db);
|
||||
(void)cbm_rmdir(tmp->dir);
|
||||
tmp->dir[0] = '\0';
|
||||
}
|
||||
|
||||
/* Prepare a stripped DB copy for best-quality export.
|
||||
* VACUUM INTO → (optionally) drop indexes → VACUUM. Returns malloc'd buffer
|
||||
* or NULL. VACUUM INTO runs on BOTH quality levels: it is the consistent
|
||||
@@ -441,9 +495,15 @@ static const char *DROP_INDEXES_SQL = "DROP INDEX IF EXISTS idx_nodes_label;"
|
||||
* committed transactions still in the -wal and can be mid-checkpoint torn
|
||||
* (#895). Only the index-stripping is BEST-only. */
|
||||
static char *prepare_snapshot_db(const char *db_path, size_t *out_size, bool strip_indexes) {
|
||||
char tmp_path[CBM_SZ_4K];
|
||||
snprintf(tmp_path, sizeof(tmp_path), "%s/cbm_artifact_tmp.db", cbm_tmpdir());
|
||||
cbm_unlink(tmp_path);
|
||||
artifact_snapshot_tmp_t tmp;
|
||||
if (!artifact_snapshot_tmp_open(&tmp)) {
|
||||
artifact_export_fail("prepare_snapshot_dir", cbm_tmpdir(), "private_tmpdir_failed", errno);
|
||||
return NULL;
|
||||
}
|
||||
/* Fresh private directory ⇒ the destination is absent by construction, which
|
||||
* is exactly what VACUUM INTO requires. The old unlink-the-stale-file step
|
||||
* is gone with the fixed name it existed to clear. */
|
||||
const char *tmp_path = tmp.db;
|
||||
|
||||
/* VACUUM INTO: clean compacted copy. Use raw sqlite3 to bypass store authorizer
|
||||
* (which blocks ATTACH, used internally by VACUUM INTO). */
|
||||
@@ -452,6 +512,7 @@ static char *prepare_snapshot_db(const char *db_path, size_t *out_size, bool str
|
||||
const char *err = raw_db ? sqlite3_errmsg(raw_db) : "sqlite_open";
|
||||
artifact_export_fail("open_source_db", db_path, err, 0);
|
||||
sqlite3_close(raw_db);
|
||||
artifact_snapshot_tmp_close(&tmp);
|
||||
return NULL;
|
||||
}
|
||||
|
||||
@@ -464,7 +525,7 @@ static char *prepare_snapshot_db(const char *db_path, size_t *out_size, bool str
|
||||
if (vrc != SQLITE_OK) {
|
||||
artifact_export_fail("vacuum_into", tmp_path, errmsg ? errmsg : sqlite3_errstr(vrc), 0);
|
||||
sqlite3_free(errmsg);
|
||||
cbm_unlink(tmp_path);
|
||||
artifact_snapshot_tmp_close(&tmp);
|
||||
return NULL;
|
||||
}
|
||||
|
||||
@@ -478,19 +539,15 @@ static char *prepare_snapshot_db(const char *db_path, size_t *out_size, bool str
|
||||
}
|
||||
}
|
||||
|
||||
/* Reopened by path rather than held on a descriptor across VACUUM INTO,
|
||||
* because sqlite owns the create. The private directory is what makes that
|
||||
* safe: an attacker who cannot enter it cannot swap the file underneath. */
|
||||
char *data = read_file_alloc(tmp_path, out_size);
|
||||
if (!data || *out_size == 0) {
|
||||
artifact_export_fail("read_stripped_db", tmp_path, "empty_or_unreadable", errno);
|
||||
}
|
||||
cbm_unlink(tmp_path);
|
||||
|
||||
/* Clean up WAL/SHM from temp */
|
||||
char wal[CBM_SZ_4K];
|
||||
char shm[CBM_SZ_4K];
|
||||
snprintf(wal, sizeof(wal), "%s-wal", tmp_path);
|
||||
snprintf(shm, sizeof(shm), "%s-shm", tmp_path);
|
||||
cbm_unlink(wal);
|
||||
cbm_unlink(shm);
|
||||
/* Removes the copy, its WAL/SHM sidecars, and the private directory. */
|
||||
artifact_snapshot_tmp_close(&tmp);
|
||||
return data;
|
||||
}
|
||||
|
||||
|
||||
+213
-21
@@ -27,11 +27,24 @@ enum {
|
||||
#include <ctype.h>
|
||||
#include "foundation/compat_fs.h"
|
||||
#include "foundation/compat_regex.h"
|
||||
#include "foundation/platform.h" /* cbm_normalize_path_sep */
|
||||
#include <stdio.h>
|
||||
#include <stdlib.h>
|
||||
#include <string.h>
|
||||
#include <sys/stat.h>
|
||||
|
||||
#ifdef _WIN32
|
||||
#include "foundation/win_utf8.h" /* cbm_path_to_wide */
|
||||
#include <windows.h>
|
||||
/* Canonical paths on both sides come from the same resolver, so casing already
|
||||
* agrees; comparing case-insensitively removes the doubt at no cost. */
|
||||
#define envscan_path_ncmp _strnicmp
|
||||
#else
|
||||
#include <fcntl.h>
|
||||
#include <unistd.h>
|
||||
#define envscan_path_ncmp strncmp
|
||||
#endif
|
||||
|
||||
/* ── Regex patterns (compiled lazily) ──────────────────────────── */
|
||||
|
||||
static cbm_regex_t dockerfile_re; /* ENV|ARG KEY=VALUE or KEY VALUE */
|
||||
@@ -311,12 +324,145 @@ static int scan_line(const char *line, file_type_t ft, char *key_out, size_t key
|
||||
}
|
||||
}
|
||||
|
||||
/* ── Containment ───────────────────────────────────────────────── */
|
||||
|
||||
/* Join dir + name, rejecting rather than truncating.
|
||||
*
|
||||
* Silent truncation here was not merely a lost path. The root-relative suffix
|
||||
* used to be derived as `full_path + strlen(root_path)`, so once the real root
|
||||
* was longer than this fixed buffer that pointer landed past the end of the
|
||||
* array — out-of-bounds arithmetic on every entry below it. Refusing the path is
|
||||
* what keeps envscan_relative_suffix in bounds by construction. */
|
||||
static bool envscan_join_path(char *out, size_t out_sz, const char *dir, const char *name) {
|
||||
if (out_sz == 0) {
|
||||
return false;
|
||||
}
|
||||
int written = snprintf(out, out_sz, "%s/%s", dir, name);
|
||||
if (written <= 0 || (size_t)written >= out_sz) {
|
||||
out[0] = '\0';
|
||||
cbm_log_warn("envscan.path_too_long", "dir", dir, "name", name);
|
||||
return false;
|
||||
}
|
||||
return true;
|
||||
}
|
||||
|
||||
/* Root-relative suffix of a path built beneath root_path.
|
||||
*
|
||||
* The prefix comparison is not a formality: it succeeding is precisely what
|
||||
* proves `path` holds at least root_len bytes, which is what makes
|
||||
* `path + root_len` a valid pointer. Never compute this suffix from an
|
||||
* unverified strlen. Returns NULL when `path` is not under the root. */
|
||||
static const char *envscan_relative_suffix(const char *path, const char *root_path,
|
||||
size_t root_len) {
|
||||
if (strncmp(path, root_path, root_len) != 0) {
|
||||
return NULL;
|
||||
}
|
||||
const char *rel = path + root_len;
|
||||
while (*rel == '/') {
|
||||
rel++;
|
||||
}
|
||||
return rel;
|
||||
}
|
||||
|
||||
/* Stat WITHOUT following links, so a repository-controlled symlink cannot lead
|
||||
* the walk outside the project. The previous stat() resolved the link, reported
|
||||
* the target's S_ISDIR, and the walk happily descended wherever it pointed.
|
||||
* Returns 0 on success, CBM_NOT_FOUND to skip the entry.
|
||||
* Mirrors pass_pkgmap.c's pkgmap_safe_stat and discover.c's safe_stat. */
|
||||
static int envscan_safe_stat(const char *abs_path, struct stat *st) {
|
||||
#ifdef _WIN32
|
||||
/* Windows has no lstat; a junction or directory symlink surfaces as a
|
||||
* reparse point, which is the same escape hatch, so screen the attribute.
|
||||
* The wide stat also keeps non-ASCII paths off the ANSI CRT. */
|
||||
wchar_t *wide = cbm_path_to_wide(abs_path);
|
||||
if (!wide) {
|
||||
return CBM_NOT_FOUND;
|
||||
}
|
||||
DWORD attributes = GetFileAttributesW(wide);
|
||||
if (attributes != INVALID_FILE_ATTRIBUTES && (attributes & FILE_ATTRIBUTE_REPARSE_POINT)) {
|
||||
free(wide);
|
||||
return CBM_NOT_FOUND;
|
||||
}
|
||||
struct _stat64 wst;
|
||||
int ret = _wstat64(wide, &wst);
|
||||
free(wide);
|
||||
if (ret != 0) {
|
||||
return CBM_NOT_FOUND;
|
||||
}
|
||||
st->st_mode = wst.st_mode;
|
||||
st->st_size = wst.st_size;
|
||||
st->st_mtime = wst.st_mtime;
|
||||
return 0;
|
||||
#else
|
||||
if (lstat(abs_path, st) != 0) {
|
||||
return CBM_NOT_FOUND;
|
||||
}
|
||||
if (S_ISLNK(st->st_mode)) {
|
||||
return CBM_NOT_FOUND;
|
||||
}
|
||||
return 0;
|
||||
#endif
|
||||
}
|
||||
|
||||
/* True when `candidate` still resolves to somewhere under `canonical_root`.
|
||||
*
|
||||
* Belt to envscan_safe_stat's braces. The link screen refuses the symlink
|
||||
* itself; this catches escapes a mode check cannot see — a resolved path that
|
||||
* leaves the root for any other reason. `canonical_root` is resolved once per
|
||||
* scan because cbm_canonical_path costs a syscall per component. */
|
||||
static bool envscan_within_root(const char *candidate, const char *canonical_root) {
|
||||
char resolved[CBM_SZ_4K];
|
||||
if (!cbm_canonical_path(candidate, resolved, sizeof(resolved))) {
|
||||
return false;
|
||||
}
|
||||
#ifdef _WIN32
|
||||
cbm_normalize_path_sep(resolved);
|
||||
#endif
|
||||
size_t root_len = strlen(canonical_root);
|
||||
if (root_len == 0 || envscan_path_ncmp(resolved, canonical_root, root_len) != 0) {
|
||||
return false;
|
||||
}
|
||||
/* Demand a real component boundary, so root "/repo" does not admit
|
||||
* "/repo-elsewhere". */
|
||||
return canonical_root[root_len - 1] == '/' || resolved[root_len] == '\0' ||
|
||||
resolved[root_len] == '/';
|
||||
}
|
||||
|
||||
/* ── Public API ────────────────────────────────────────────────── */
|
||||
|
||||
/* Open a candidate config file without following a symlink where the platform
|
||||
* allows it. Mode "r" is preserved from the previous cbm_fopen call so the
|
||||
* per-platform newline handling the line loop already copes with is unchanged. */
|
||||
static FILE *envscan_open_file(const char *full_path) {
|
||||
#ifdef _WIN32
|
||||
/* No O_NOFOLLOW equivalent here; reparse points were screened by
|
||||
* envscan_safe_stat. cbm_fopen is required for UTF-8 → _wfopen mapping. */
|
||||
return cbm_fopen(full_path, "r");
|
||||
#else
|
||||
/* O_NOFOLLOW closes the window between the lstat above and this open: if the
|
||||
* entry turned into a symlink in between, the open fails rather than reading
|
||||
* through it. A raw open is correct on POSIX — paths are bytes there, and the
|
||||
* cbm_fopen rule exists for Windows' wide-API mapping. */
|
||||
int flags = O_RDONLY | O_CLOEXEC;
|
||||
#ifdef O_NOFOLLOW
|
||||
flags |= O_NOFOLLOW;
|
||||
#endif
|
||||
int descriptor = open(full_path, flags);
|
||||
if (descriptor < 0) {
|
||||
return NULL;
|
||||
}
|
||||
FILE *stream = fdopen(descriptor, "r");
|
||||
if (!stream) {
|
||||
(void)close(descriptor);
|
||||
}
|
||||
return stream;
|
||||
#endif
|
||||
}
|
||||
|
||||
/* Scan a single file for env URL bindings. Returns number of bindings added. */
|
||||
static int scan_env_file(const char *full_path, const char *rel, file_type_t ft,
|
||||
cbm_env_binding_t *out, int max_out) {
|
||||
FILE *f = cbm_fopen(full_path, "r");
|
||||
FILE *f = envscan_open_file(full_path);
|
||||
if (!f) {
|
||||
return 0;
|
||||
}
|
||||
@@ -360,29 +506,52 @@ static int scan_env_file(const char *full_path, const char *rel, file_type_t ft,
|
||||
return count;
|
||||
}
|
||||
|
||||
/* Process a single directory entry for env scanning. Returns bindings added. */
|
||||
/* Process a single directory entry for env scanning. Returns bindings added.
|
||||
* `root_len` is strlen of the root the walk started from, verified against
|
||||
* path_stack[0] by the caller; `canonical_root` is that root resolved once. */
|
||||
static int process_env_entry(cbm_dirent_t *ent, const char *dir_path, const char *root_path,
|
||||
cbm_env_binding_t *out, int max_out, char path_stack[][CBM_SZ_512],
|
||||
int *stack_top, char **excluded_dirs, int excluded_count) {
|
||||
size_t root_len, const char *canonical_root, cbm_env_binding_t *out,
|
||||
int max_out, char path_stack[][CBM_SZ_512], int *stack_top,
|
||||
char **excluded_dirs, int excluded_count) {
|
||||
char full_path[CBM_SZ_512];
|
||||
snprintf(full_path, sizeof(full_path), "%s/%s", dir_path, ent->name);
|
||||
const char *rel = full_path + strlen(root_path);
|
||||
while (*rel == '/') {
|
||||
rel++;
|
||||
if (!envscan_join_path(full_path, sizeof(full_path), dir_path, ent->name)) {
|
||||
return 0;
|
||||
}
|
||||
const char *rel = envscan_relative_suffix(full_path, root_path, root_len);
|
||||
if (!rel) {
|
||||
return 0;
|
||||
}
|
||||
|
||||
struct stat st;
|
||||
if (stat(full_path, &st) != 0) {
|
||||
if (envscan_safe_stat(full_path, &st) != 0) {
|
||||
return 0;
|
||||
}
|
||||
if (S_ISDIR(st.st_mode)) {
|
||||
if (!is_ignored_dir(ent->name) &&
|
||||
!cbm_pipeline_relpath_is_excluded(rel, excluded_dirs, excluded_count) &&
|
||||
*stack_top < CBM_SZ_256) {
|
||||
strncpy(path_stack[*stack_top], full_path, sizeof(path_stack[0]) - 1);
|
||||
path_stack[*stack_top][sizeof(path_stack[0]) - SKIP_ONE] = '\0';
|
||||
(*stack_top)++;
|
||||
if (is_ignored_dir(ent->name) ||
|
||||
cbm_pipeline_relpath_is_excluded(rel, excluded_dirs, excluded_count) ||
|
||||
*stack_top >= CBM_SZ_256) {
|
||||
return 0;
|
||||
}
|
||||
if (!envscan_within_root(full_path, canonical_root)) {
|
||||
cbm_log_warn("envscan.dir_outside_root", "path", full_path, "root", canonical_root);
|
||||
return 0;
|
||||
}
|
||||
/* Checked copy. The stack rows are the same width as full_path, so this
|
||||
* cannot truncate today — but check anyway rather than lean on that
|
||||
* coincidence: a truncated row is exactly what reintroduced the
|
||||
* out-of-bounds suffix arithmetic. */
|
||||
int written = snprintf(path_stack[*stack_top], sizeof(path_stack[0]), "%s", full_path);
|
||||
if (written <= 0 || (size_t)written >= sizeof(path_stack[0])) {
|
||||
cbm_log_warn("envscan.path_too_long", "dir", dir_path, "name", ent->name);
|
||||
return 0;
|
||||
}
|
||||
(*stack_top)++;
|
||||
return 0;
|
||||
}
|
||||
/* Only regular files are config files. Without this, the lstat above still
|
||||
* lets a FIFO through, and opening one blocks the pass until a writer
|
||||
* appears. */
|
||||
if (!S_ISREG(st.st_mode)) {
|
||||
return 0;
|
||||
}
|
||||
if (is_secret_file(ent->name)) {
|
||||
@@ -405,14 +574,36 @@ int cbm_scan_project_env_urls_excluded(const char *root_path, cbm_env_binding_t
|
||||
int count = 0;
|
||||
char path_stack[CBM_SZ_256][CBM_SZ_512];
|
||||
int stack_top = SKIP_ONE;
|
||||
strncpy(path_stack[0], root_path, sizeof(path_stack[0]) - 1);
|
||||
path_stack[0][sizeof(path_stack[0]) - SKIP_ONE] = '\0';
|
||||
/* Reject an over-long root instead of truncating it in. The old strncpy left
|
||||
* path_stack[0] SHORTER than root_path, and every suffix pointer derived from
|
||||
* strlen(root_path) was then past the end of its buffer. Establishing
|
||||
* root_len from the copy that actually fit is what removes that class. */
|
||||
int written = snprintf(path_stack[0], sizeof(path_stack[0]), "%s", root_path);
|
||||
if (written <= 0 || (size_t)written >= sizeof(path_stack[0])) {
|
||||
cbm_log_warn("envscan.root_too_long", "root", root_path);
|
||||
return 0;
|
||||
}
|
||||
size_t root_len = (size_t)written;
|
||||
|
||||
/* Resolved once per scan: every directory the walk wants to descend is
|
||||
* checked against this, and resolving costs a syscall per component. */
|
||||
char canonical_root[CBM_SZ_4K];
|
||||
if (!cbm_canonical_path(root_path, canonical_root, sizeof(canonical_root))) {
|
||||
/* Unresolvable root — nothing to walk. Previously opendir simply failed
|
||||
* on the first iteration, so the result is the same: zero bindings. */
|
||||
return 0;
|
||||
}
|
||||
#ifdef _WIN32
|
||||
cbm_normalize_path_sep(canonical_root);
|
||||
#endif
|
||||
|
||||
while (stack_top > 0 && count < max_out) {
|
||||
stack_top--;
|
||||
char dir_path[CBM_SZ_512];
|
||||
strncpy(dir_path, path_stack[stack_top], sizeof(dir_path) - SKIP_ONE);
|
||||
dir_path[sizeof(dir_path) - SKIP_ONE] = '\0';
|
||||
int copied = snprintf(dir_path, sizeof(dir_path), "%s", path_stack[stack_top]);
|
||||
if (copied <= 0 || (size_t)copied >= sizeof(dir_path)) {
|
||||
continue;
|
||||
}
|
||||
|
||||
cbm_dir_t *d = cbm_opendir(dir_path);
|
||||
if (!d) {
|
||||
@@ -420,8 +611,9 @@ int cbm_scan_project_env_urls_excluded(const char *root_path, cbm_env_binding_t
|
||||
}
|
||||
cbm_dirent_t *ent;
|
||||
while ((ent = cbm_readdir(d)) && count < max_out) {
|
||||
count += process_env_entry(ent, dir_path, root_path, out + count, max_out - count,
|
||||
path_stack, &stack_top, excluded_dirs, excluded_count);
|
||||
count += process_env_entry(ent, dir_path, root_path, root_len, canonical_root,
|
||||
out + count, max_out - count, path_stack, &stack_top,
|
||||
excluded_dirs, excluded_count);
|
||||
}
|
||||
cbm_closedir(d);
|
||||
}
|
||||
|
||||
@@ -36,6 +36,20 @@ if [[ ! -x "${BINARY}" ]]; then
|
||||
exit 2
|
||||
fi
|
||||
|
||||
# PRECONDITION, not a skip: this test drives the crash-orphan probe, which is
|
||||
# compiled out of ordinary builds (it forks a SIGTERM-ignoring child, which must
|
||||
# never ship — see src/main.c and TEST_SEAMS in Makefile.cbm). Against a
|
||||
# seam-less binary the probe silently no-ops and the test dies with an opaque
|
||||
# "Killed: 9" many lines later. Assert the capability up front and say exactly
|
||||
# how to get it. Failing (not skipping) is deliberate: a skip here would hide
|
||||
# the loss of watchdog coverage entirely.
|
||||
if ! LC_ALL=C grep -a -q -F 'CBM_TEST_WORKER_DESCENDANT_PID_FILE' "${BINARY}"; then
|
||||
echo "binary lacks the crash-orphan test seam: ${BINARY}" >&2
|
||||
echo " rebuild it with: make -f Makefile.cbm cbm TEST_SEAMS=1" >&2
|
||||
echo " or run this test through scripts/test.sh, which does that for you." >&2
|
||||
exit 2
|
||||
fi
|
||||
|
||||
if command -v shasum >/dev/null 2>&1; then
|
||||
BUILD_FINGERPRINT="$(shasum -a 256 "${BINARY}" | awk '{print $1}')"
|
||||
elif command -v sha256sum >/dev/null 2>&1; then
|
||||
|
||||
@@ -221,7 +221,16 @@ mi_decl_export void mi_options_print_out(mi_output_fun* out, void* arg) mi_attr_
|
||||
const int vermajor = MI_MALLOC_VERSION/10000;
|
||||
const int verminor = (MI_MALLOC_VERSION%10000)/100;
|
||||
const int verpatch = (MI_MALLOC_VERSION%100);
|
||||
_mi_fprintf(out, arg, "v%i.%i.%i%s%s (built on %s, %s)\n", vermajor, verminor, verpatch,
|
||||
/* CBM LOCAL PATCH (keep across mimalloc refreshes): upstream appends
|
||||
* "(built on __DATE__, __TIME__)" here. mimalloc is linked into every cbm
|
||||
* binary, so those two macros made every build byte-unique — two compilations
|
||||
* of identical source seconds apart could never produce the same hash. That
|
||||
* defeats reproducible builds, and it also means a released artifact can never
|
||||
* inherit an antivirus false-positive determination made about its
|
||||
* predecessor, because every rebuild is a brand-new file to a reputation
|
||||
* system. The banner keeps the version; the wall clock is not diagnostics.
|
||||
* -Wdate-time in CFLAGS_COMMON makes any new use a build failure. */
|
||||
_mi_fprintf(out, arg, "v%i.%i.%i%s%s\n", vermajor, verminor, verpatch,
|
||||
#if defined(MI_CMAKE_BUILD_TYPE)
|
||||
", " mi_stringify(MI_CMAKE_BUILD_TYPE)
|
||||
#else
|
||||
@@ -233,7 +242,7 @@ mi_decl_export void mi_options_print_out(mi_output_fun* out, void* arg) mi_attr_
|
||||
#else
|
||||
""
|
||||
#endif
|
||||
, __DATE__, __TIME__);
|
||||
);
|
||||
|
||||
// show options
|
||||
for (int i = 0; i < _mi_option_last; i++) {
|
||||
|
||||
@@ -30,6 +30,17 @@ PRETRAINED_VECTOR_BLOB_LEN:
|
||||
.long PRETRAINED_VECTOR_BLOB_END - PRETRAINED_VECTOR_BLOB
|
||||
|
||||
#else
|
||||
/* WHY: an ELF object that carries no .note.GNU-stack tells the linker
|
||||
* nothing about its stack requirement, and GNU ld then assumes the WORST
|
||||
* for the whole link — every Linux release binary shipped GNU_STACK RWE
|
||||
* because of this one omission. This is the only assembly source in the
|
||||
* build, so it alone decided that property. The note must stay even though
|
||||
* a blob of constant data obviously never executes: absence is the signal,
|
||||
* not the contents. -Wl,-z,noexecstack in the link flags enforces the
|
||||
* outcome, and scripts/ci/check-binary-composition.sh fails the release if
|
||||
* an executable stack ever comes back. */
|
||||
.section .note.GNU-stack,"",@progbits
|
||||
|
||||
.section .rodata,"a",@progbits
|
||||
.globl PRETRAINED_VECTOR_BLOB
|
||||
.globl PRETRAINED_VECTOR_BLOB_LEN
|
||||
|
||||
Reference in New Issue
Block a user