Commit Graph

1078 Commits

Author SHA1 Message Date
Martin Vogel 7ba84e3754 feat(sql): first-class Table/View nodes with FROM/JOIN lineage
CREATE TABLE / CREATE VIEW / CREATE MATERIALIZED VIEW now produce Table
and View nodes (previously generic Variable), CREATE PROCEDURE produces
a Function, and schema-qualified DDL names (schema.table) are named by
the table identifier instead of the schema. A view's FROM/JOIN relations
are emitted as usages and resolve into view -> table USAGE lineage
edges.

Relations join the cross-file name registry so lineage can resolve, with
two structural safeguards:

- Registry membership is defined once by cbm_label_is_registry_symbol
  (helpers.c); the full, parallel and incremental seed sites all call
  it, ending the KEEP-IN-SYNC copies the old label lists required.
- The default cbm_registry_resolve vetoes relation-labeled results:
  common table names (users, orders, config) collide with code
  identifiers in every language, and no CALLS/USAGE/READS/WRITES/THROWS/
  handler/decorator consumer may bind them. The SQL lineage path opts in
  through the new cbm_registry_resolve_lineage.

Table/View also join the registry-only per-file LSP surface labels so a
table rename invalidates dependent SQL files on incremental (no stale
lineage edges), rank with the type tier in BM25 search, and appear in
the architecture boundary/package/cluster queries via the pinned
CBM_SQL_RELATION_LABELS fragment.

Tests: extraction trio (labels, lineage usages, schema-qualified names),
grammar golden + probe updates, relation-label contract pin, and two
pipeline tests: cross-language isolation (binding: fails without the
veto) and incremental table-rename stale-lineage.

Closes #574.

Co-authored-by: alexisperinger-ux <alexis.peringer@iss-stoxx.com>
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
2026-08-20 00:12:28 +02:00
Martin Vogel 98c1f8c3e1 test: gate-chain contract pins the explicit-result conditions
The contract required the bare '!cancelled() && !failure()' idiom on
build/smoke/soak — the exact form the v0.10.7 incident proved fail-open
(failure() does not cover a CANCELLED needed job). It now requires each
gate's explicit accepted results, the sanctioned skip_tests clause on
build, and the preflight version guard wired before lint.

Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
2026-08-18 22:30:28 +02:00
Martin Vogel afebf0092b Merge pull request #1326 from Enferlain/fix/1287-persisted-coverage-summary
fix(index): preserve persisted coverage summaries
2026-08-18 18:57:53 +02:00
Martin Vogel 39bd040505 Merge pull request #1699 from DeusData/fix/lf-endings-distill
fix(build): LF contract for extensionless git hooks + a line-ending guard
2026-08-18 16:36:50 +02:00
Martin Vogel 00beec9397 Merge pull request #1608 from ertankucukoglu/fix/search-code-cancellation
fix(mcp): bound and cancel Windows code search
2026-08-18 16:36:40 +02:00
Martin Vogel fe4396f906 Merge branch 'main' into fix/1287-persisted-coverage-summary 2026-08-18 11:21:46 +02:00
Martin Vogel 32633bab9a fix(build): LF contract for extensionless git hooks + a line-ending guard
The *.sh eol=lf rule landed via #1314, but the git hooks
(scripts/git-hooks/commit-msg, scripts/hooks/pre-commit) are extensionless
and were still at the mercy of core.autocrlf. Add their explicit entries,
plus the contract test from #1272 wired as scripts/test.sh Step 0t: every
shell entrypoint (*.sh + both hook directories) must carry an eol=lf
attribute, with a matched-zero-files guard so a broken glob can never pass
vacuously.

Distilled from #1272: the diagnosis (CRLF checkouts breaking shebangs under
WSL/MSYS), the hook-file coverage, and the guard design are @xumian520's;
verified RED without the *.sh rule (105 uncovered entrypoints) and green
with it (107 files).

Co-Authored-By: xumian520 <126989134+xumian520@users.noreply.github.com>
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
2026-08-18 11:21:36 +02:00
Martin Vogel 58ef9f19ac fix(cypher): expansion materializes every matched row — the cap falsified aggregates
Second mechanism of #1196, exposed by the reporter's v0.10.6 retest: every
relationship-expansion site capped its output buffer at bind_cap*10, so edges
past the cap were silently dropped BEFORE WHERE and aggregation ever ran.
count() then reported the scanned prefix as if it were a fact — 9,360 of
13,691 DEFINES field-measured at --max-rows 1000 — and a LABEL on the source
did not protect you (the label workaround only ever fixed source
enumeration, which #1323 already made exact). max_rows is an output-row
limit per the public header; projection already enforces it.

All five capped sites now share one growable append (geometric growth,
size_t sizing): the per-hop expansion, its fixed/variable-length helpers and
process_edges, the bound-terminal driver, and the cross-join outer buffer.
Only allocation failure stops materialisation; match_count stays truthful
either way, so the #627 OPTIONAL contracts hold (a saturated buffer can no
longer exist, and the fallback rows share the same append). The #601
deadline still bounds pathological time, and hop caps keep bounding depth —
this removes only the silent row-dropping.

Regression test: 2 labeled sources x 30 edges with max_rows=2 — the old cap
returned count=20; ground truth 60 now holds, and the list form returns
exactly max_rows rows. Proven RED before and RED again on revert; cypher
181/181 and mcp suites green, including every OPTIONAL/#627 semantics test.

Fixes #1196 (together with #1323, which fixed the unlabeled source scan).

Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
2026-08-18 11:19:32 +02:00
Martin Vogel 07cac7d6b3 Merge pull request #1371 from Joseph-MingEn/fix/py-aliased-from-import-calls
fix(python): resolve aliased from-import CALLS to real def
2026-08-18 11:19:19 +02:00
Martin Vogel 847da2fff2 fix: restore test closer lost in the merge resolution
The conflict's shared tail (#endif + closing brace) was emitted once for
two adjacent Windows tests; re-add the closer so both tests compile.

Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
2026-08-18 11:18:43 +02:00
Martin Vogel e24ce80bdc Merge origin/main into fix/search-code-cancellation
Trivial insertion-point collision in tests/test_mcp.c: both this branch and
main (#1704's UTF-8 pin) add a Windows search_code test at the same spot.
Both tests kept.
2026-08-18 11:08:17 +02:00
Martin Vogel 9de6831d2c Merge pull request #1695 from astandrik/codex/fix-1691-test-runtime-isolation
fix(test): isolate harness daemon runtimes
2026-08-18 11:00:03 +02:00
Martin Vogel 4dd099679d fix(windows): pin the search_code PowerShell pipe to UTF-8
PowerShell 5.1 encodes stdout for a native-process pipe in the console
OEM codepage, so raw search content containing characters the inherited
CP cannot carry (Cyrillic under CP437/850, ...) reached
collect_grep_matches as '?' — and whether it degraded depended entirely
on which console the server happened to inherit. That surfaced as the
intermittent test_mcp raw-Русский mojibake on the windows CI leg and
means real Windows users in a default console get '?' for all
non-ASCII search_code raw content.

Every generated command now pins [Console]::OutputEncoding to UTF-8, so
the pipe is codepage-independent by construction. The read side needs
no pin: Select-String decodes BOM-less UTF-8 via .NET StreamReader
defaults. A Windows-side builder test asserts all five command variants
carry the prelude.

Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
2026-08-18 09:31:59 +02:00
Martin Vogel 773bb037dc fix(test): the stdin gate's #1359 guards survive a registry with no zero-argument tool
#1181 gave list_projects pagination parameters, retiring the last
empty-properties schema — and the two #1359 regression tests leaned on
list_projects as their live zero-argument example, so main went red the
moment the merge train composed (the PR was green on its July base, which
predated these tests).

The gate's schema→decision core is split behind a CBM_CLI_ENABLE_TEST_API
seam, so the zero-argument branch stays pinned directly (empty properties,
absent properties, populated properties) regardless of what the registry
ships; list_projects now asserts its NEW truth (piped args accepted, TTY
still refused); and the schema↔gate parity sweep keeps running over every
tool without the impossible >=1 zero-argument floor. Production behavior
is unchanged — this is the tests catching up with an intended schema
change, plus a seam so they never again depend on a shipped example.

Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
2026-08-18 07:15:09 +02:00
Martin Vogel 9a547b3399 Merge pull request #1253 from anfedoro/agent/fix-search-code-utf8
fix: preserve UTF-8 in search_code source
2026-08-18 00:14:32 +02:00
Martin Vogel 78d1b2f0a6 Merge pull request #1203 from ibaldr89/fix/extensionless-shebang-language
fix(discover): detect extensionless scripts by shebang
2026-08-18 00:14:17 +02:00
Martin Vogel b61949355b Merge branch 'main' into fix/1287-persisted-coverage-summary 2026-08-18 00:12:02 +02:00
Martin Vogel 3dab881b66 Merge pull request #1181 from tmonestudio/codex/fix-list-projects-coverage-scope
fix(mcp): restore scalable project discovery
2026-08-18 00:08:28 +02:00
Martin Vogel ce1f273efe Merge pull request #1467 from PR9000/fix/freebsd-platform-support
fix(daemon): add native FreeBSD process-image identity and /home log-path alias
2026-08-18 00:08:05 +02:00
Martin Vogel 415a64a2f9 Merge pull request #1263 from astandrik/fix/index-mode-capability-rebuild
fix: rebuild index when mode adds capabilities
2026-08-18 00:07:51 +02:00
Martin Vogel 89f0cd43c0 Merge pull request #1647 from rudi193-cmd/fix/725-cross-language-suffix-match
fix(registry): drop suffix_match CALLS across language boundaries
2026-08-18 00:05:33 +02:00
Martin Vogel 22d93dd12b Merge pull request #1323 from Enferlain/fix/1196-unlabeled-candidate-limit
fix(cypher): scan all unlabeled query candidates
2026-08-18 00:05:18 +02:00
Martin Vogel 2397d58da6 Merge pull request #1325 from Enferlain/fix/1284-list-valued-fields
fix(search): preserve compound requested fields
2026-08-18 00:04:55 +02:00
Martin Vogel de4ec9b5e9 Merge pull request #1319 from JJordan0C/fix/search-graph-semantic-only-results
fix(mcp): isolate semantic-only JSON search
2026-08-18 00:04:36 +02:00
Martin Vogel 33a3f0322a Merge pull request #1308 from Yyunozor/fix/issue-1294-is-test-tests-dir
fix(extract): converge Function/Method is_test with the tests/ path filter
2026-08-18 00:02:28 +02:00
astandrik 0eb22f0205 fix(test): isolate harness daemon runtimes
Signed-off-by: astandrik <astandrik@yandex-team.ru>
2026-08-17 22:07:11 +03:00
astandrik d024f41e7f fix(pipeline): preserve coverage and artifact ordering
Keep caller-requested discovery scope while rebuilding changed weaker-mode requests at the stronger stored coverage. Export persistent artifacts only after the replacement database generation is published.

Signed-off-by: astandrik <astandrik@yandex-team.ru>
2026-08-17 13:03:40 +03:00
Ertan 506151f6c1 fix(mcp): propagate cancellation to Windows code search
Signed-off-by: Ertan <ertan.kucukoglu@gmail.com>
2026-08-17 08:49:50 +03:00
Martin Vogel 93e93087b4 Merge pull request #1683 from DeusData/fix/install-cluster
fix(install): Hermes YAML constructs, goose required name, annotated MCP entry repair (#1631, #1675, #1630)
2026-08-17 07:10:29 +02:00
Martin Vogel 20ad3e5b89 fix(cli): forward CBM_RUNTIME_DIR in the generated Codex configuration (#1664)
Codex sanitizes stdio MCP subprocess environments to the names listed in
env_vars. Since #1645 CBM_RUNTIME_DIR relocates the daemon rendezvous, so a
Codex subprocess that does not receive it looks for the daemon in the DEFAULT
location and never finds it — the same silent client/daemon split
CBM_CACHE_DIR caused in #1562. Both names decide WHICH daemon a process talks
to and are now forwarded unconditionally (forward-if-present semantics);
behavioural knobs (log level, workers, budgets) deliberately stay
unforwarded — that broader list remains #1664's open enhancement question.

Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
2026-08-17 05:40:38 +02:00
Martin Vogel 276664ebff fix(cli): compare our binary path separator-insensitively in MCP ownership (#1582)
gotspatel's live opencode.json stores our entry with backslashes
(`C:\...\codebase-memory-mcp.exe`) while the installer compares its own path
with forward slashes — the same file on disk, refused over the separator
spelling, so op=mcp_install failed on a correctly-installed machine (and on
Windows the dead-path probe rightly reported the binary PRESENT, which turned
the mismatch into a hard refusal).

Ownership comparison now treats `\` and `/` as equal everywhere and folds
case on Windows only, where the filesystem is case-insensitive; POSIX
byte-exactness otherwise holds. The annotated entry that names this binary is
recognised as already satisfied and preserved byte-for-byte.

Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
2026-08-17 05:12:08 +02:00
Martin Vogel 4cd84422ac fix(yaml): accept a UTF-8 BOM as a document prologue (#1656)
PowerShell 5.1's `Set-Content -Encoding UTF8` writes a BOM, so real
Windows-authored Hermes configs start with EF BB BF — and both edit ops
failed content-independently (the reporter's 26-byte reproduction is their
23-byte file plus exactly this BOM; reproduced RED on macOS with the same
bytes, so the platform was never the variable).

The document read now validates past a leading BOM and yaml_doc_init treats
it as a prologue: the first line's structure starts after it, the key lookup
still sees our own section when the BOM immediately precedes it (guarded by a
dedicated no-duplicate-section test), and every edit splices interior ranges,
so the BOM survives writes byte-for-byte. Non-document inputs — keys, entry
blocks, identity scalars — keep the strict no-BOM rule.

Also makes the moved-entry cli test fixture platform-correct: the Windows
dead-path probe can only prove a fixed-drive path absent, so the Windows
branch uses one; a POSIX-shaped path is refused there by design.

Fixes #1656.

Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
2026-08-17 04:54:14 +02:00
Martin Vogel 33d1ecea1b fix(cli): install-entry compatibility repairs for goose and annotated MCP entries
Two install failures with the same root theme — the entry we write is a
compatibility contract with the agent's parser, and both sides of that
contract needed repair:

goose (#1675): ExtensionConfig::Stdio declares `name` as a required serde
field with no default, and goose's loader silently drops entries that fail to
deserialize — install reported success and the extension was invisible. The
goose block now carries `name: codebase-memory-mcp`; the non-goose YAML
schema stays name-free. A CBM_CLI_ENABLE_TEST_API seam asserts the exact
block bytes per schema.

annotated MCP entries (#1630, the deferred field-merge): dbd20eaa recognised
an entry the client annotated ("enabled": true beside our command/type) but
could only leave it untouched, because replacing the whole entry would drop
the client's keys. config_json_like gains
cbm_json_like_replace_field_raw_if_unchanged — splice ONE member's value,
preserving every other byte (comments, ordering, client keys) — and the
upsert flow uses it on the two AUTHORIZED repair channels only:

- a relocating update (the entry names the previous managed binary), and
- the existing Windows dead-path probe, which previously fell back to a
  wholesale rewrite and lost the annotations.

POSIX keeps its doctrine unchanged: a config-supplied path is never trusted,
so a moved-looking entry without that authority is preserved byte-for-byte
and install fails loudly (cli_editor_mcp_preserves_unrecorded_posix_absolute_
entries_without_probe holds). All repair/refusal paths are covered by tests
proven RED on the unfixed flow.

Fixes #1675.
Fixes #1630.

Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
2026-08-17 04:23:50 +02:00
Martin Vogel b54998d242 fix(yaml): accept the #1631 constructs and repair prior owned entries
Four legal-YAML constructs from the reporters' real Hermes configs made
`install` fail permanently (any one of them aborted mcp_install and/or
pre_llm_hook_install):

- exact empty flow collections as values (`plugins: []`, `tool_choice: {}`)
  — now validated key-only in both the mapping-body and sequence document
  scans, mirroring the #1673 empty-mapping exception; non-empty flow
  collections stay rejected.
- block sequences at the same indent as their mapping key (column-0 `- item`)
  — item lines directly after a value-less key are structure, not malformed
  keys, in the root walker, the key matcher, and the sequence mapping-range
  walker.
- double-quoted scalars continued across lines with a trailing `\` — the doc
  loader now precomputes per-line continuation flags; continuation lines are
  value bytes every structural walker skips, and a document ending inside an
  open continuation stays an error.
- mid-word quote characters in plain scalars (`LET'S`) — quotes are scalar
  indicators only at a node start (range start, after `:`, after `-`),
  exactly like the #1639 anchor/alias rule; real quoted values keep their
  protection.

Byte-identity alone also froze users on canonicals older releases wrote:
galaxy's entry had `command:` unquoted, and the goose block gained `name:`
(#1675), so the existing entry was declared FOREIGN forever. An entry under
our key now repairs when it parses as a known prior shape (single command
line, or the pre-name goose block) with a codebase-memory-mcp[.exe] command
basename; anything else stays FOREIGN and the file untouched.

End-to-end: both reporters' full configs (iandol 15.6 KB, galaxy 15.2 KB) now
install with zero agent_config errors, every original line byte-preserved,
and the goose upgrade path rewrites the old block in place. Each construct
carries a distilled regression test proven RED on the unfixed editor.

Fixes #1631.

Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
2026-08-17 04:23:50 +02:00
Martin Vogel 41d240accf Merge pull request #1681 from DeusData/feat/scaling-probe
perf(lsp): eliminate the cross-LSP O(n²) — shared Java registry, own-file overlay, complexity guard (#1669)
2026-08-17 02:05:47 +02:00
Martin Vogel e87b42ceba Merge pull request #1680 from pcristin/fix/goose-empty-flow-mapping
fix(yaml): accept empty flow mappings in mapping bodies
2026-08-17 01:37:52 +02:00
Martin Vogel 51770de0b6 perf(ts): memoize expression-type evaluation per node
ts_eval_expr_type and ts_signature_for_call are mutually recursive: resolving
a call evaluates its argument expressions once per lookup path (method
dispatch + namespace fallback), and in tsc-compiled spread files the first
argument is itself the next nested Object.assign(...) call — the same subtree
re-evaluates once per enclosing level, 2^n total. The TS suite's
objectSpreadRepeatedComplexity.js (3.6 KB, 5 nodes) measured 20.4 s; with the
memo its eval cost is zero within measurement noise of a one-file control,
and the microsoft/TypeScript corpus drops 37.5 -> ~24 s warm (nodes
byte-identical, edges within the known scheduler jitter).

Expression types are position-pure within a file pass (one node = one scope
path; the per-file walk is single-threaded and deterministic), so one eval
per node is the correct semantics, not a cache trade-off. The memo is a
per-file, arena-backed, linear-probe table keyed on TSNode.id. Results
produced under a depth-cap or budget bail are never stored: both bail sites
bump a degradation counter, and a store only happens when the subtree
completed clean — a degraded UNKNOWN can therefore never shadow a later full
evaluation.

The regression guard asserts work, not wall-clock: the nested-Object.assign
shape must complete without exhausting the deterministic eval budget, read
back through a new CBM_ENABLE_TEST_SEAMS accessor pair. The seam lives in the
lsp_all unity object, so GRAMMAR_CFLAGS_TEST/TSAN now carry the seams define
(test artifacts always have seams; prod never does). Verified RED without the
memo (budget exhausted, suite 47.8 s) and green with it (3.2 s).

Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
2026-08-17 00:16:16 +02:00
Martin Vogel ec16764fee fix(test): convert forbidden SKIP()s in the complexity suite to policy forms
The no-skips lint gate (scripts/check-no-test-skips.sh) rightly rejected the
throughput-report test's two SKIP() calls:

- CBM_SKIP_PERF=1 is deliberate operator configuration, not a hidden
  environment failure: reporting is off by request, so the test PASSes with a
  stderr note instead of skipping.
- an uncreatable report dir IS an environment failure and now FAILs with the
  remedy in the message (set CBM_COMPLEXITY_REPORT_DIR), per the policy text.

Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
2026-08-17 00:16:15 +02:00
Martin Vogel e700b6215f test(complexity): deterministic O(n^2) guard on work-counter ratios
Finding #1669 took an 11-corpus A/B across two release binaries. This
suite makes that bug class fail a unit test in seconds, on every
platform, from tiny corpora.

Method: build k and 2k REPLICATED module copies, run the full in-process
pipeline on both, assert counter RATIOS. Independent copies mean every
extensive quantity — nodes, edges, Σ per-file registry defs — must grow
linearly (ratio ~2). A files x corpus coupling makes per-file work itself
grow with k and lands at ratio ~4. Ratios expose the exponent regardless
of absolute scale, so 60-120 files suffice.

Verdicts are pure functions of (code, input): gates ride ONLY on
deterministic work counters and data-product counts, never on wall time.
Throughput (nodes/s, edges/s) is information-only, written to
private/benchmarks/complexity-<ts>.json (local, gitignored; skipped
under CBM_SKIP_PERF where rates are meaningless).

Two corpus shapes, both needed:

- Independent modules (java/py/go/ts templates): catches cross-module
  contamination and dedup breakage. The #1669 bug is GREEN here — fully
  closed modules filter perfectly, which is exactly why it survived.
- The growing shared package (bigpkg): one Java package whose file count
  scales with k — the real-repo shape (files concentrate in large
  packages). The JVM namespace filter branch makes per-file work track
  package size, so this corpus is the honest #1669 reproducer:
  ratio 4.00 RED on the pre-fix tree, 4.00 RED for a module-scoped
  overlay, 2.00 GREEN for the own-file overlay. It discriminated the
  correct fix design before the fix was written.

Both legs of every pair exceed MIN_FILES_FOR_PARALLEL(50): below it the
sequential path runs, which builds no shared registries and would be the
wrong code path to gate (its per-file cost is bounded by the 50-file
ceiling).

Recorded but deliberately NOT gated, with reasons at the case:
tail_candidates and fallback_rows are legitimately superlinear under
replication until those scans are bounded, and measured ~1 ns/unit.

Every ratio gate carries a non-vacuousness floor on the base counter so
broken counter wiring fails loudly instead of green-washing
(cbm_pxc_count_perfile_defs feeds the overlay path into the same
counter the fallback path already used; wired for Java, extend to the
TS overlay when touching ts_lsp).

Dynamic coverage: languages iterate CBM_LANG_COUNT; embedded templates
cover the LSP-hybrid languages, and tests/fixtures/complexity/<lang>/
dirs are auto-discovered so a new language joins the guard by dropping
fixtures. Uncovered languages are listed in the report with the reason.

The local report additionally carries per-language node/edge counts with
ratios and a per-pass elapsed_ms table per run (captured via a TEE log
sink during the in-process pipeline runs) — trend data for humans, still
never a gate.

Refs #1669.

Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
2026-08-16 18:43:44 +02:00
pcristin f277034fe5 fix(yaml): accept empty flow mappings in mapping bodies
Signed-off-by: pcristin <xxxokzxxx@protonmail.com>
2026-08-16 12:28:54 +00:00
Martin Vogel f6e3af4325 feat(diagnostics): make superlinear passes visible from an ordinary run
Finding the 0.10.x indexing regression took an 11-corpus A/B across two
release binaries, a subset-scaling series, and a per-pass exponent fit.
None of that should have been necessary: the pass that carried it,
cross-file LSP, was already timed and already logged its def count. The
numbers were there, just never normalised into something a reader could
judge.

Three additions, all shipped without a flag except the detailed curve:

- parallel.resolve.cross_lsp_cost — cross-LSP cost NORMALISED per file,
  next to defs_total. Wall time cannot separate "big repo" from
  "superlinear pass"; us_per_file can. Measured on one Java tree:

      files=3710   defs=102845  us_per_file=35129   us_per_file_per_kdef=341
      files=14833  defs=339866  us_per_file=106722  us_per_file_per_kdef=314

  Per-file cost tripled while per-kdef stayed flat — the fingerprint of
  work proportional to the whole corpus (files x defs). One grep on two
  differently sized repos now answers what previously took a two-binary
  bench.

- parallel.resolve.scan_cost — candidates visited per tail-match lookup,
  plus fallback_rows, which the code has counted since #1085 but exposed
  only to a test. On the same tree the tail scan reached 242M candidate
  visits (n^2.04), worth seeing even though it proved cheap in wall time.

- cbm_scale_probe (foundation/profile.h) — samples cumulative elapsed at
  1/8, 1/4, 1/2 and 1 of a pass's items and fits k in T ~ n^k, warning
  once k reaches 1.35 in shipped builds. Wired into parallel_extract and
  parallel_resolve.

The probe's limits are documented rather than oversold: it catches growth
WITHIN a run, and would NOT have caught this bug, whose per-item cost is
constant-but-large within any single run (it reported 1.26 while the
cross-corpus exponent was 1.86). That is exactly why us_per_item and
us_per_file are emitted alongside it.

Tests are deterministic by construction: the exponent fit is a pure
function fed synthetic points, and the checkpoint bookkeeping is asserted
directly. A test that proved the detector by generating a real quadratic
workload would be asserting on the scheduler.

No product behaviour changes; diagnostics only.

Refs #1669.

Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
2026-08-16 13:12:25 +02:00
Martin Vogel 3491a8e83b fix(mem): restore mimalloc's Linux arena-commit default (#1654)
Since #1360 routed ordinary malloc/new through mimalloc on Linux, the
arena policy governs every allocation in the process rather than just the
bound sqlite/tree_sitter populations. cbm sets arena_eager_commit=0, so
mimalloc commits sub-ranges with mprotect(PROT_READ|PROT_WRITE) over a
PROT_NONE reservation, and each partial commit SPLITS the reserved VMA.

Measured on the Go corpus, Linux arm64, shipped binaries:

  v0.9.0   10 mappings, at ANY worker count
  v0.10.5  ~22k mappings, peak; the count tracks CONCURRENCY
           (999 at 1 worker, 8460 at 4, 11965 at 18)

Two consequences, both of which #1654 reported from a 96-CPU/376 GB host:
the mmap/mprotect churn serialises on the kernel's per-process mmap_lock,
and the VMA count climbs toward vm.max_map_count, after which mmap fails
for ANY size -- so mimalloc reported it could not allocate 10 KB while
`free -g` still showed 246 GB available.

mimalloc's own default for this option is 2, meaning "eager-commit arenas
only on an OS that overcommits (i.e. linux)", precisely because commit is
free there until pages are touched. Overriding it to 0 opted Linux out of
the default written for Linux. Restore it on Linux only; every other
platform keeps the lazy setting, where commit is NOT free and the
upfront-memory reason still holds (Windows especially, #581).

Measured effect, same corpus and host, baseline build vs this build:

  mappings  22450 -> 17312  (-23%)
  wall       92.4s -> 92.6s (unchanged)
  peak RSS  19.14 -> 19.22 GB (unchanged)

This is a partial mitigation, not a cure: the remaining ~17k mappings are
individual 64 KB-3 MB extraction buffers, each taking its own mmap (the
worker reserves ~40 GB of address space for ~19 GB of RSS). Pooling those
is the durable fix and is deliberately left out of this change.

Guard: mem_arena_eager_commit_follows_platform_commit_cost pins the
platform split so the Linux default cannot be silently opted out again.

Reproduction and controlled 2x2 (only vm.max_map_count varied) are
recorded on #1654.

Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
2026-08-16 03:23:05 +02:00
Martin Vogel 63f0a6c0e7 test(parse-coverage): free the extraction results in the #1610 tests
LeakSanitizer on CI caught all five new tests leaking their CBMFileResult:

    Indirect leak of 24 byte(s) ... ts_tree_new
      cbm_extract_file_ex cbm.c:1256
      do_extract test_parse_coverage.c:39
      test_dockerfile_missing_final_newline_not_flagged_issue1610:272
    SUMMARY: AddressSanitizer: 706504 byte(s) leaked in 189 allocation(s)

Every pre-existing test in this suite calls cbm_free_result before PASS; the new
ones did not. The local run could not have found it - LeakSanitizer reports
"detect_leaks is not supported on this platform" on macOS arm64, so this class
of defect is CI-only here.

Each test now captures what it asserts, frees, and only then decides, so the
early-FAIL paths do not leak either. The cross-grammar loop prints its
diagnostic before freeing so the failure message keeps naming the grammar.

While correcting the guard, a first attempt left ASSERT_TRUE(flagged ||
has_ranges || true) in real_error_before_eof_still_flagged - always true, and it
would have silently disarmed the guard that stops the EOF suppression from being
over-broad. Removed. The guard is re-proven binding: forcing
cbm_is_eof_terminator_miss to return true makes EIGHT tests fail, including both
guards, and restoring it returns the suite to green.

parse_coverage 14 passed.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
2026-08-15 17:22:59 +02:00
Martin Vogel 8669ba8f9e fix(extract): a missing final newline is not a partial parse
A file that does not end with a newline leaves the grammar's mandatory line
terminator MISSING. cbm_collect_error_regions counted that node, so the file was
reported parse_partial with the last line as its error range.

It is not a miss. The node is ZERO-WIDTH and sits at EOF: the parser consumed no
source for it, so by construction nothing was dropped - no construct can live in
a zero-byte span - and every real instruction above it parsed normally. Proven
by dumping the tree: the reporter's two-line Dockerfile yields
(source_file (from_instruction ...) (entrypoint_instruction ...) (MISSING "\n"))
with both instructions intact and the MISSING node spanning bytes 73-73.

It was never Dockerfile-specific. Stripping the trailing newline from the 156
linkable grammar fixtures flips 13 of them to has_error, and SIX produce regions:
dockerfile, tcl, fish, gomod, hyprlang - and makefile, which is a genuinely
different case (its ERROR has WIDTH; the recipe really is lost).

Worse, the ones that stayed silent did so for no principled reason. ini, fsharp,
beancount, requirements, gitignore, sshconfig and kconfig omit the same
terminator, but theirs is a HIDDEN node and hidden nodes are invisible to
ts_node_child(). Whether a user was told their file was partially parsed came
down to whether that grammar's author declared the terminator visible.

The cost was not cosmetic: a phantom parse_partial writes a "<project>::missed"
shadow row, and until #1609 that row made the project fail cross-repo validation
as BOTH source and target. A single absent byte could remove an entire
repository from cross-repo intelligence with no error shown anywhere.

The suppression is deliberately narrow - zero-width AND at EOF. A MISSING or
ERROR node with width still counts even at EOF, and anything before EOF is
untouched. Both callers pass the raw root, so one source_len is correct for
both; verified rather than assumed, since root is bound once and never
reassigned.

Reported by @vitaliy-shatskiy, who could not share the original file and instead
rebuilt the property from scratch with a byte-exact script - an editor would
have silently re-added the newline and hidden it. Their isolation matrix ruled
out BOM, CRLF vs LF, exec-form vs shell-form and file length before we looked at
it once.

Reproduce-first, revert-checked: the Dockerfile and cross-grammar tests fail on
the previous tree and pass with the fix; forcing the new predicate to return
false brings the identical REDs back. Two guards pin the boundary and hold in
both directions - a width-bearing failure at EOF (makefile) and a real
mid-file ERROR in a file that ALSO lacks its final newline (built from
C_IFDEF_SPLIT, the fixture this suite already proves is flagged).

parse_coverage 14, extraction 276, language 217, infrascan 3,
grammar_regression 1 - 511 passed, 0 failed.

Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
2026-08-15 16:09:04 +02:00
Martin Vogel 3162f09173 Merge pull request #1638 from astandrik/codex/fix-1633-codex-hook-preflight-diagnostics
fix(cli): explain Codex hook preflight refusals
2026-08-15 15:44:03 +02:00
Martin Vogel b416d30c7c Merge pull request #1653 from DeusData/fix/update-names-missing-installer
fix(cli): only name an installer that is actually there
2026-08-15 15:43:57 +02:00
Martin Vogel d58962c010 fix(cli): only name an installer that is actually there
`update` hands off to install.sh (install.ps1 on Windows) and prints the command
to run. It built that command from cbm_detect_self_path - the BINARY's directory
- and treated "I resolved my own location" as "the installer is beside me".

Those are different questions. install.sh is placed beside the binary by
install.sh itself, but a binary that was moved, packaged by a distro, or built
from source has no installer next to it. We printed the path anyway:

    bash "/home/<user>/.local/bin/install.sh"
    /usr/bin/bash: /home/<user>/.local/bin/install.sh: No such file or directory

Reported on discussion #1560 (#1632) by a user who was already three releases
deep in install trouble and had just been told, by us, to run a file that does
not exist.

`update` exists to tell someone how to proceed. Ending the interaction on a
command that cannot run is the one outcome it must not produce - and the
fallback text was already there and already correct, naming install.sh as
shipping in the release archive without asserting a path.

The probe goes through cbm_path_info_utf8 so a non-ASCII install directory
resolves on Windows, and rejects a DIRECTORY of that name, because `bash <dir>`
is not a command either. A symlink still counts: it is reported rather than
followed, and the shell runs it perfectly well.

The Windows branch gets the same treatment; it had the identical assumption
about install.ps1.

Reproduce-first and revert-checked: with the probe forced to return true - the
old behaviour - the new test fails with "a directory with no installer must not
be named as one", and passes once it is restored. cli: 276 passed.

Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
2026-08-15 10:43:15 +02:00
Martin Vogel 6a701b5d09 Merge pull request #1652 from DeusData/fix/cross-repo-shadow-row
fix(cross-repo): stop a `::missed` shadow row making a project unresolvable
2026-08-15 10:21:39 +02:00
astandrik d0351fd99a test(cli): cover v0.10.2 Codex hook upgrades
Signed-off-by: astandrik <astandrik@yandex-team.ru>
2026-08-15 10:37:24 +03:00
astandrik 00e0cf38d6 fix(cli): harden hook diagnostic contract
Signed-off-by: astandrik <astandrik@yandex-team.ru>
2026-08-15 10:37:24 +03:00