A file that does not end with a newline leaves the grammar's mandatory line
terminator MISSING. cbm_collect_error_regions counted that node, so the file was
reported parse_partial with the last line as its error range.
It is not a miss. The node is ZERO-WIDTH and sits at EOF: the parser consumed no
source for it, so by construction nothing was dropped - no construct can live in
a zero-byte span - and every real instruction above it parsed normally. Proven
by dumping the tree: the reporter's two-line Dockerfile yields
(source_file (from_instruction ...) (entrypoint_instruction ...) (MISSING "\n"))
with both instructions intact and the MISSING node spanning bytes 73-73.
It was never Dockerfile-specific. Stripping the trailing newline from the 156
linkable grammar fixtures flips 13 of them to has_error, and SIX produce regions:
dockerfile, tcl, fish, gomod, hyprlang - and makefile, which is a genuinely
different case (its ERROR has WIDTH; the recipe really is lost).
Worse, the ones that stayed silent did so for no principled reason. ini, fsharp,
beancount, requirements, gitignore, sshconfig and kconfig omit the same
terminator, but theirs is a HIDDEN node and hidden nodes are invisible to
ts_node_child(). Whether a user was told their file was partially parsed came
down to whether that grammar's author declared the terminator visible.
The cost was not cosmetic: a phantom parse_partial writes a "<project>::missed"
shadow row, and until #1609 that row made the project fail cross-repo validation
as BOTH source and target. A single absent byte could remove an entire
repository from cross-repo intelligence with no error shown anywhere.
The suppression is deliberately narrow - zero-width AND at EOF. A MISSING or
ERROR node with width still counts even at EOF, and anything before EOF is
untouched. Both callers pass the raw root, so one source_len is correct for
both; verified rather than assumed, since root is bound once and never
reassigned.
Reported by @vitaliy-shatskiy, who could not share the original file and instead
rebuilt the property from scratch with a byte-exact script - an editor would
have silently re-added the newline and hidden it. Their isolation matrix ruled
out BOM, CRLF vs LF, exec-form vs shell-form and file length before we looked at
it once.
Reproduce-first, revert-checked: the Dockerfile and cross-grammar tests fail on
the previous tree and pass with the fix; forcing the new predicate to return
false brings the identical REDs back. Two guards pin the boundary and hold in
both directions - a width-bearing failure at EOF (makefile) and a real
mid-file ERROR in a file that ALSO lacks its final newline (built from
C_IFDEF_SPLIT, the fixture this suite already proves is flagged).
parse_coverage 14, extraction 276, language 217, infrascan 3,
grammar_regression 1 - 511 passed, 0 failed.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Two loose ends from the bounds work.
The decompression side already took size_t so a >2 GiB capacity could not wrap
through int, but compression still took int for both the source length and the
destination capacity, and the artifact export cast a size_t database size down
to reach it. A database past 2 GiB would have handed the encoder a negative
length. Both lengths and the bound helper are size_t now, and the function
returns int64_t like its decompressing counterpart.
The discovery walk now prunes the cache directory by absolute path. A custom
CBM_CACHE_DIR may sit inside a repository — tests do it routinely — and walking
into it pulls every other project's graph database into this project's file
list. This is the narrow form of a concern that was briefly implemented as
refusing any root that contained the cache; refusing a whole root was too blunt,
and not walking the cache is what the concern actually asks for.
The test fails without the prune: a .go file planted under the cache is
otherwise discovered, and "cache" is not in the built-in skip list, so the
assertion is not vacuous.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
A pass over places where a bound was either miscalculated or missing.
UI response builders (src/ui/http_server.c) mixed the clamping helper
http_appendf with raw cursor indexing. Route the per-line separators and
quotes through the helper too, so the cursor cannot leave the buffer once
output saturates it, and terminate explicitly afterwards — the helper pins
the cursor and then writes nothing, which would otherwise leave the "%s"
reply with no terminator in range. The log buffer was budgeted at
LOG_LINE_MAX + 10 per line while JSON escaping can double every byte, so
size it for the escaped worst case. Job status entries now escape their path
and error fields, which were interpolated raw.
Cypher (src/cypher/cypher.c): the WHERE grammar descends once per nested '('
and once per NOT, so parse depth followed the query text rather than
anything bounded. Track depth on parser_t and refuse past 256 levels with a
parse error. Bounding parse depth also bounds the resulting tree, so the
recursive evaluator inherits the limit.
Gitignore (src/discover/gitignore.c): '**' retries the remainder at every
position and consecutive groups multiply, so match cost is exponential in
the number of groups. Thread a step budget through the matcher and give up
past 20000 steps, reporting no-match so a pathological pattern fails to
ignore rather than ignoring the wrong files. glob_match keeps its name so
recursion_whitelist.h still describes the functions that recurse; the new
non-recursive wrapper that seeds the budget is glob_match_bounded.
Call extraction (internal/cbm/extract_calls.c): extract_fp_callee recursed
once per applied argument, so stack use followed the parse-tree depth of the
indexed file. Rewritten as a left-spine loop. Reassigning node/nk before
continuing leaves the fall-through cases where the recursive form left them,
so behaviour is unchanged and the extraction suite is untouched.
CALLS props (src/pipeline/pass_calls.c): `cap - pos - PAIR_LEN` wraps once
pos reaches cap - PAIR_LEN and stops bounding the copy below. Guard
additively, matching the closing write in the same function.
Shell-bound paths (src/foundation/str_util.*): the three git shell-out sites
each wrap a repo path in cmd.exe-compatible double quotes, where %VAR%,
!VAR! and ^ remain active. Two carried private copies of the check that
rejects those three on Windows and the third used the bare validator.
Promote the stricter form to cbm_validate_shell_path_arg and route all three
through it, so the copies cannot drift apart again.
Tests cover the buffer bounds, the parse-depth refusal, matcher termination
and the validator. The bounds tests run in a forked child so a violation
surfaces as a signal instead of silently.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The memory-diagnostics report's priority-4 lane (path-sensitive clang-analyzer,
memory checks only) run over all 111 production files. 21 findings triaged;
the real ones, all cold-path (none can explain #581's per-query residual):
LEAKS
- mcp get_architecture: scope_path leaked on the missing-store early return
(REQUIRE_STORE frees only `project`); allocate after the gate.
- pass_definitions: cancellation mid-extraction leaked the pass-owned result
cache including already-extracted entries; mirror the end-of-pass cleanup.
- store package-boundary scan: the row-scan abort path freed the node arrays
but not the boundary accumulators or their duplicated package strings.
- cbm quarantine set: a duplicate path line leaked the replaced value (and a
fresh key copy -- the table borrows key pointers); a partial strdup failure
leaked the surviving half. Reuse the stored key for duplicates.
- pass_githistory: unchecked malloc/strdup -- an OOM dereferenced NULL and a
failed strdup leaked the index cell. Allocate before claiming the slot.
NULL/UB
- cli config subcommand: NULL argv with nonzero argc slipped the guard (the
inner `argv &&` shielded only the help comparison) into argv[0].
- store bfs_multi: a negative max_results broke out before any row was
written, then freed fields of an unwritten negative-index slot. Clamp.
- pass_calls emit_http_async_edge: the service-pattern call sites pass a NULL
target behind a hand-duplicated URL predicate; a drift between the copies
turned target->id into a null deref. The callee is now total.
- sqlite_writer: both leaf-array OOM paths left leaf_count stale with a NULL
array, walking pb_finalize_* into leaves[0]; consistent empty state routes
them to the existing root=0 failure return.
HARDENED (invariants true but invisible to path-sensitive analysis)
- Leiden CSR + aggregate arrays, SCC adjacency: calloc + endpoint guards, so
a future degree/collection miscount degrades benignly instead of UB.
- SCC cycle fill: the ncyc==0 no-slot invariant made local.
RECORDED FALSE POSITIVES (no code change)
- yaml sequence starts (loop bound == alloc bound), cypher agg arrays (same
count both sides), mcp read_message ch (assigned by fgetc each iteration),
pkgmap clean buffer, mcp csize (Tarjan: ncomp>=1 when nverts>=1), vendored
verstable x2.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
GATE + LANES (user decision: runner cost accepted)
- make lint-mem (local triage) and lint-mem-ci (gating: vendored-filtered,
any remaining finding fails). The gate is green because every false
positive above was restructured for provability -- calloc'd fill-cursor
arrays, explicit Tarjan invariant, zeroed buffer tails, min-1-element
allocations -- never suppressed.
- make diag: pinned newest-LLVM ASan/UBSan lane with straighter stacks.
- CI: lint-mem job (_lint.yml) and test-diag job (_test.yml), both on the
pinned LLVM 22 apt toolchain. Cost disclosure: roughly +25-40 min and
+25-60 min (ccache-warm) per push respectively.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
c_lsp_process_file carried __attribute__((no_sanitize("address"))) over the
entire C LSP orchestration, excluding every direct memory access in it from
ASan -- placed for a stack-pointer-to-registry lifetime bug whose witness test
also had its assertion discarded ((void)find_resolved), so nothing could ever
prove the bug gone. Flagged by the memory-diagnostics standards report
(private/C_MEMORY_DIAGNOSTICS_STANDARDS_REPORT.md, priority 2) as the single
highest-value ASan coverage gap in the repository.
This branch's C LSP registry rework resolved the underlying lifetime: with the
suppression removed, the full c_lsp suite passes under ASan+UBSan (760 tests),
including the formerly-failing template field type resolution, whose test now
asserts the resolved call instead of discarding it.
Honest limit: the original bug predates the branch and its fix is the branch's
broad registry rework, not an isolatable commit -- so the revert-proof is
main-vs-branch (main: suppression + discarded assertion; branch: neither),
not a single-change revert-check.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The deep-nesting torture tests (stack_overflow_a/b) went from 0-1s per test on
main to 39-119s on this branch -- the 900s suite budget killed them on every
venue except the M4 (three GitHub CI platforms and the local Linux leg, each
dying mid-suite at a DIFFERENT test, which is what pointed at shared machinery
rather than any one language).
Sampling the child process found two quadratic layers, both branch-added:
1. recompute_state iterated the WHOLE scope stack on every code-bearing node
to rebuild the walk-state flags -- O(depth) per node, and a deep descent
pushes a frame per level, so deep trees paid O(n x depth). Each frame now
saves the complete walk-state tuple it displaces and pop restores it
verbatim: push and pop are O(1) and kind-agnostic, and the per-node
recompute is gone entirely. The CALL frame's effect is applied by
push_call_scope after the caller fills the invocation triple, preserving
the old ordering exactly.
2. is_reference_node fetched ts_node_parent for EVERY identifier in EVERY
language to serve a Puppet/Vimscript sigil-wrapper check -- the language
gate sat inside the condition, after the fetch. ts_node_parent descends
from the root (O(depth)), so all languages paid O(depth) per identifier.
The gate now precedes the fetch; only Puppet/Vimscript files pay it.
macOS, both suites together: 543s -> 54s. Per test: ts_cyclic 119s -> 6s,
python_deep 105s -> 6s, go_deep 39s -> 8s, php_deep 45s -> 8s. The residual
6-8s vs main's 0-1s is the branch's larger legitimate per-node work; the
remaining Python attribute-site parent walk is a bounded follow-up, recorded.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Maintainer decision (option B) closing the Kotlin property-reference repro's
platform divergence at its root instead of tie-breaking the fallback.
`holder::handler` (parsed as navigation by the vendored grammar) produced a
USAGE edge whose target was chosen by the name-only registry fallback: with a
same-named property (Holder.handler) and function (Functions.handler) in the
project, the winner was registration order -- readdir order -- so Windows
(lexicographic NTFS) borrowed the FUNCTION while macOS happened to pick the
property. The graph's answer must not depend on directory enumeration.
The occurrence is now claimed by exact, receiver-typed resolution. Five links,
each of which was missing:
1. extract side: a Kotlin navigation member read is a semantic-reference
candidate at the member occurrence, so an LSP row can claim it. With no row
the join finds nothing -- but a candidate no longer falls back to the
name-only registry guess, which is the fail-closed direction this PR is
built on.
2. kotlin cross resolution: a property READ with a proven receiver emits a
CALL_REFERENCE row against the property. The property is not a callable
target, so the join can only produce USAGE, never a fabricated
CALL_REFERENCE. Value reads only; calls stay with the invocation machinery.
3. receiver typing across files: kotlin_resolve_class_name composed
<this module>.<name> for an unimported cross-file type, which can never
name a type defined in another file. A per-call unique-short-name map
(built from the project defs, ambiguous names fail closed) resolves the
annotation to the real registered QN. Hash lookup, no scans.
4. cross registry fields: pxc_map_label dropped Variable defs entirely, so no
cross registry ever saw a property. They now flow through (every language's
registrar filters by explicit label, so only Kotlin consumes them) and the
Kotlin registrar attaches them as fields of their receiver type,
hash-bucketed -- a per-type scan would be the registry-tail-scan quadratic
pattern.
5. def side: class-body variables now record their declaring class
(parent_class) -- previously only methods did. The QN stays module-level,
so this is additive metadata: the only structural parent_class consumer is
Method-gated (DEFINES_METHOD), verified in pass_definitions/pass_parallel/
pipeline_incremental.
Emission targets each field's REAL def QN carried through the field map:
kotlin class properties are minted with module-level QNs (proj.Holder.handler,
not proj.Holder.Holder.handler), so the composed form would name a node that
does not exist and the join would silently drop the edge.
The repro now proves the property edge on every platform for the same reason,
not by racing readdir.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Two gaps in the callable-reference path, found while attributing a
Windows-only failure of the Kotlin property-reference repro:
1. kt_callable_reference_has_ambiguous_parent treated every unlisted parent
kind as ambiguous, including function_body -- so a single-expression body
(fun f() = ::handler) never resolved its reference at all and fell to the
name-only registry fallback. A single-expression body is an unconditional
context: the expression IS the value, no branch selects among candidates.
2. The typed-receiver branch (Type::member, value::member) only consulted
kotlin_lookup_method, so a reference to a PROPERTY emitted nothing even
when the receiver type provably has that member. It now emits a row against
the property QN; the property is not a callable target, so the downstream
join can only produce USAGE, never a fabricated CALL_REFERENCE.
Note: this does NOT close the property-reference repro on Windows. That
fixture's holder::handler parses as navigation_expression under the vendored
grammar, whose member occurrence is not a semantic-reference candidate -- the
edge is decided by the name-only registry fallback, whose winner among
same-named symbols is registration order (= readdir order, platform-
dependent). That divergence is a design question recorded separately.
repro_reference_precision + kotlin_lsp on macOS: 170 passed.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Maintainer decision closing the last open RED from PR item 2.
`callback = handler` contains a genuine value occurrence of handler, and its
exact callable identity is proven -- the alias binding below could not be
created otherwise. The policy of this PR is proven exact value =>
CALL_REFERENCE, so that occurrence now emits a resolved reference row instead
of remaining an ordinary USAGE. Bare identifier RHS only, and the usage-carrier
rule in extract_usages.c is extended in lockstep, Python-gated -- a candidate
the LSP never matches would change nothing, but the asymmetry would be a trap.
The strategy distinction mirrors the argument path: a source name that is
itself a local alias claims lsp_callable_alias, so the local-shadow guard
admits exactly what it admits for arguments and refuses everything else.
In the graph the two occurrences inside `argument` -- the RHS and the alias
use in accept(callback) -- surface as ONE CALL_REFERENCE edge and zero USAGE
edges, because edge dedup is by (src, tgt, type) and collapsing multiple sites
between the same pair is its existing, intended behavior. That is precisely
what the fixture always asserted, so it needed no relaxation, only the comment
recording the decision.
Also: GCC rejects repro_call_argument_matrix_b's _Static_assert comparing
enumerators of two different anonymous enums (-Werror=enum-compare); clang
accepts it, so only the Linux leg saw it. Cast both sides to int.
repro_reference_precision + call_reference_contract: 100 passed, 0 failed.
This closes all five item-2 REDs.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
make lint-ci was red on this branch before any of the rebase work.
cppcheck reported all four halves of two ULARGE_INTEGER values in
cbm_path_info_utf8 as assigned-but-never-read. The code is correct -- it is a
union, and .QuadPart reads exactly what .LowPart/.HighPart wrote -- but cppcheck
does not model that aliasing, so it cannot see the read.
Composing the two 64-bit values arithmetically says the same thing without the
union, so the checker needs no exception. That is the repository's stated
preference: refactor first, adjust the rule second, suppress only as a rare
justified exception -- and a suppression here would have to be re-justified by
every future reader.
Also applies clang-format to the lines this rebase touched in pipeline.c and
extract_usages.c, plus one pre-existing violation in pipeline_incremental.c.
Formatting only; the file set is limited to what LINT_SRCS/LINT_HDRS actually
covers, so no unrelated whole-file reflow rides along.
make -f Makefile.cbm lint-ci: passes.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
PR item 2. Four of its five REDs; the fifth is a policy question left open.
THREE SHADOW REDS -- a rebound name kept looking exactly bound.
An imported `callback` that Python has already rebound must lose exact callable
proof, or the graph reports a reference to a callable the name no longer refers
to. Three binders were invisible:
(callback := 0) module-level walrus
[(callback := 0) for _ in (0,)] walrus inside a comprehension
type callback = int PEP 695 type alias
The first two because a module-level expression statement is routed to call
resolution and never reaches the binding scanner at all, so no binder inside one
was ever seen. The third because the scanner had no branch for
type_alias_statement, and its generic recursion only descends -- an identifier
alone binds nothing.
The comprehension case needed care in the other direction: the whole node was
skipped, correctly, because its iteration variables are private to it. But an
assignment expression binds in the CONTAINING scope (PEP 572), which is exactly
what separates it from every other binder a comprehension can hold. It is now
scanned for walrus targets only.
PARENTHESISED CALLABLE ARGUMENT -- `accept((handler))` produced no reference.
The semantic row was recorded on the outermost parenthesised argument while the
usage carrier stopped at the parentheses and was never marked a callable-value
candidate at all, so the occurrence-exact join had nothing to join. A bare
identifier now climbs to the same wrapper python_direct_callable_attribute_site
already returns for the bound-method form, so `accept((handler))` and
`accept((service.handler))` agree on one occurrence instead of disagreeing.
That climb uses the cursor the caller already holds and checks the parent kind
before doing anything. Walking up with ts_node_parent instead costs one slow
fallback per identifier in the file, which the linearity guard in
test_extraction.c rejects -- correctly, and it caught exactly that here.
The fixture's expected span was also wrong: "((handler))" is the argument LIST,
one byte wider on each side than either party, and per argument it would collide
for any call with more than one argument. It now names "(handler)" -- the
occurrence both sides actually use. Production was genuinely broken; the fixture
was additionally wrong about where.
LOCAL CALLABLE ALIAS -- a proven alias was refused by the shadow guard.
`callback = handler` makes callback a local, so the usage is a local shadow, and
a shadowed usage may only claim a semantic reference through the alias strategy.
Python emitted lsp_callable_value_reference for every callable argument, so the
one case the guard exists to admit could never satisfy it.
A lexical binding that does not simply name the module symbol of the same
spelling is an alias introduced in this body, and only that now claims the alias
strategy. A module function referenced by its own name keeps the strategy it has
always had, so the guard still refuses everything it refused before.
STILL OPEN: the same fixture asserts no ordinary USAGE from `argument`, and the
right-hand side of `callback = handler` is a second genuine occurrence of
handler as a value. Whether that should become a CALL_REFERENCE (this PR's own
proven-value policy, making the count 2) or emit nothing is a graph-content
decision, not a defect, and is left for the maintainer.
repro_reference_precision, call_reference_contract, extraction: 371 passed,
1 failed (that fixture).
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
r.HandleFunc(base+"/login", handler) fell through extract_positional_url
with no branch for binary_expression, so first_string_arg stayed NULL. A
real BFF with 47 such registrations (all base+"literal") produced only 9
Route nodes. Recover the literal suffix from the right-hand operand of a
"+" concatenation; a right side that is not itself a literal is left
unresolved rather than guessed.
Fixes#1249
Signed-off-by: Amir Fathi <amirfathi.me@gmail.com>
extract_defs.c grew a file-static is_namespace_scope_kind() that restated the
C++/CUDA case from the shared cbm_is_namespace_scope_kind() and added Nix, then
took over the def-side call site. Restating the shared predicate dropped its
TypeScript case (`internal_module`), so TS namespace members lost the namespace
segment of their qualified name on the def side while extract_unified.c and the
enclosing-scope walk in helpers.c -- both still on the shared predicate --
continued to qualify them.
The two halves then disagreed about the same symbol, and `MyNS.inner()` stopped
resolving: the call site looked for `.MyNS.inner` and the definition had been
minted without the `MyNS` segment.
Fixed by delegating to the shared predicate instead of restating part of it, so
the wrapper adds only the case the shared predicate cannot express (Nix needs
the node, because the decision depends on the binding's value rather than its
kind string). One source of truth for kind -> namespace-scope.
This is the same class of defect as a hand-maintained second copy of a list:
the copy was correct when written and silently wrong the moment the original
grew a case.
Fixes six failures, all of which pass before the def-side call site changed and
none of which are in the same file as the change:
ts_lsp tslsp_nocrash_namespace
tslsp_nested_namespace_preserves_exact_callers
parallel 2 x TS_EXACT_LOCAL_CLASS edge-count parity
matrix_new_constructs 2 x app_module
ts_lsp, parallel and matrix_new_constructs: 434 passed, 0 failed.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
`nix_var_types` has declared `binding` since the language was added, but no Nix
case existed in extract_var_names and no path reached a binding in the first
place, so the Variable count for Nix was unconditionally zero.
Two things were in the way. extract_variables iterates only the file root's
DIRECT children, and a Nix file's root child is its header lambda
(`{ pkgs, lib, ... }:`), so the generic loop saw nothing — the same shape as the
definition bug. And extract_var_names had no Nix branch, so a binding could not
have been named even if reached.
Scope follows the rule the other languages already use rather than inventing one.
extract_variables mints FILE scope and never locals: a C++ declaration inside a
function body is not a Variable. For Nix, file scope is the `let` bindings and
the returned attrset — a let binding is file scope in the same sense a C++
file-static is, and the attrset is the exported surface. A binding in a deeper
attrset is not.
That bound is load-bearing. Every Nix binding's parent is a binding_set at any
depth, so admitting them all would mint a node per `enable = true` in a NixOS
module's settings tree — the per-leaf flood the Helm values.yaml case in this
same function already exists to avoid.
Bindings whose value is a lambda or an attribute set are skipped: the first is
already minted as a Function by the def walk, the second is a scope
(is_namespace_scope_kind), so minting either here would double-count it.
Names follow the attrpath convention from the previous commit — leaf as the name,
whole path in the qualified name, so `services.nginx.enable = true` is name
`enable`, QN proj.file.services.nginx.enable. push_var_def grows a _qn variant
for that; the plain form delegates to it and is unchanged for every other
language.
Extraction and pipeline suites: 494 passed, 0 failed.
Signed-off-by: Jason Bowman <jason@json64.dev>
A Nix binding's name is a PATH, and the resolver took only its first segment
(`child_by_field_name(attrpath, "attr")`). Three defects followed from that one
modelling gap:
- `setA = { dup = …; }` and `setB = { dup = …; }` both minted the qualified
name proj.file.dup. The second definition, and every CALLS edge sourced from
it, was silently discarded at write.
- `a.b.fn = …` minted a definition named `a`, colliding with every other
binding whose path began `a`.
- `"kebab-case" = …` minted a name with the quote characters in it, so every
consumer keying on the name had to know to re-quote.
Adopts the convention already used for C++ namespaces — name is the leaf
segment, qualified_name is enclosing scope plus leaf, so `ns::serialize` is name
`serialize` and QN proj.file.ns.serialize. For Nix that means:
- the resolver returns the LAST attrpath segment;
- leading segments become scope, so `a.b.fn = …` and the nested spelling
`a = { b = { fn = …; }; }` produce the same QN, as they must — one is sugar
for the other;
- a binding whose value is an attribute set is a scope, mirroring
is_namespace_scope_kind's treatment of a C++ namespace. `let` bindings are
deliberately NOT scopes: they are lexical, and C++ does not qualify by block
scope either;
- a quoted segment is unquoted, and an interpolated one (`"${x}" = …`) mints
nothing at all, since it has no statically knowable name — the same call the
Makefile dot-prefix guard makes.
The two scope sources compose: `setA = { a.b.fn = …; }` is
proj.file.setA.a.b.fn.
Definition QNs and call-scope QNs are computed by two different functions —
extract_defs.c's compute_class_qn and extract_unified.c's own. If they disagree
by one segment the CALLS edge names a source node that was never minted and is
dropped at write, with no error and no failing extraction assertion. Rather than
implement the rule twice, the Nix computation lives in helpers.c and both call
it, which makes that class of drift impossible instead of merely unlikely. Note
also that the def-side enclosing-QN gate is language-gated while the call-side is
not, so widening one without the other would have produced exactly that mismatch.
Signed-off-by: Jason Bowman <jason@json64.dev>
A Nix library or module file's root expression is normally itself a function —
`{ pkgs, lib, ... }: <body>` — the near-universal header for the ecosystem.
That node matches nix_func_types, so walk_defs handed it to extract_func_def,
which resolved no name for it (the Nix resolver requires the function's parent
to be a `binding`; the root lambda's parent is the file root, `source_code`) and
minted nothing. walk_defs then hit `if (!descend_into_func) continue;` and
abandoned the whole subtree, so no binding below the header was ever visited.
The effect was total for the dominant file shape, and invisible for the rest:
a file opening with a bare `let` or attrset was walked normally, so the language
looked supported. Measured on a 320-file Nix repository, 253 of whose files
carry a function header:
before after
Function 48 1900
CALLS 56 1443
files w/defs 8 258
Adding CBM_LANG_NIX to descend_into_func is sufficient. Descending does not
over-mint from curried lambdas: in `f = a: b: ...` the inner `b:` has a
function_expression parent, resolves no name, and stays out. Checked clean for
over-minting on map/foldl' arguments, `with` bodies, `inherit`, and
parenthesized lambdas; rec attrsets, nested attrsets, `let` inside a function
body, and factory-returned attrsets all mint correctly.
Call attribution is unaffected. CALLS scope is computed by a separate traversal
(push_boundary_scopes -> compute_func_qn) that calls the same shared
cbm_resolve_func_name this path uses, so definition names and call-scope names
agree by construction, and a scope is pushed only when that name resolves
non-NULL. The edge-count rise is previously-dropped edges being written, not
re-attributed ones: before the fix those edges named a source node that did not
exist.
The change is gated on ctx->language, so no other language's walk is affected.
Signed-off-by: Jason Bowman <jason@json64.dev>
The hosted windows-11-arm shards crash at process load (STATUS_ILLEGAL_INSTRUCTION,
pass=0, secs=0) while main passes them, and removing the TLS detach callback did
not help -- so the remaining candidate is this layer, which touches every
allocation on Windows through the interposer. It declares a _Thread_local guard
read on every malloc, and thread-local storage in a static MinGW aarch64 link is
exactly the fragile case; nothing in it needs to ship.
The profiler did its job: it produced the attribution that identified #581's
mechanism, and that analysis is recorded. Sources stay in the tree, unbuilt, so
a future investigation can re-enable them deliberately.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The ARM64 fallback captured a return address inside the capture helper, so
every allocation collapsed onto the handful of hook addresses and the site
column named the profiler instead of the caller. Each hook now supplies its own
return address. The census also reports allocation volume, which separates
'the profiler sees nothing' from 'it sees traffic that is all freed'.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
SQLite and tree-sitter are bound directly to mi_malloc, so they never pass the
malloc interposer and the profile could not see them — which excluded the two
subsystems that allocate the most per request.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Preload the shared Tree-sitter array helpers before the pinned Haskell scanner so optimized builds do not retain a stale contents pointer across realloc. Keep a compile-time signature guard to prevent regression.
Fixes#1231
Signed-off-by: wargloom <wargloom@gmail.com>
Close the cross-platform sanitizer gaps that were leaving real
concurrency and undefined-behavior bugs uncaught, and fix a data race
the first widened run surfaced.
- tsan: the data-race gate ran three suites (mem, slab_alloc, parallel)
over no real threaded production code. It now covers every threaded
surface that runs clean and stable under TSan: the parallel-extraction
worker pool (parallel, worker_pool, pipeline), the filesystem watcher,
the embedded HTTP server (httpd), diagnostics sampling, the MCP server
and mutation guard, subprocess supervision, and the runnable
daemon-coordination paths (daemon, daemon_application). daemon_runtime
(deadlocks under TSan+fork), daemon_ipc and daemon_frontend
(test-harness synchronization, not production) are excluded with the
reasons recorded in the Makefile.
- tsan: fixed a genuine data race the widened gate immediately found —
cbm_lsp_max_walk_depth's lazy cache was read and written by parallel
LSP-extraction workers without synchronization. A data race is
undefined behavior even when every worker computes the same value, so
the cache slot is now a relaxed atomic: a plain load on the hot path,
and a first-touch double-compute simply stores the same value.
- tsan(ci + local): the test-tsan job now runs on Linux amd64, Linux
arm64, AND native ARM64 macOS (the threading code is shared, so a race
is usually caught on all three, but scheduler differences let each
surface one the others miss). The local ladder gained `run.sh tsan`
and `tsan-amd64` plus the matching compose services. TSan's shadow
memory aborts under modern high-entropy ASLR, so the containers run
under `setarch -R` with an unconfined seccomp profile (the personality
syscall is otherwise blocked) and the CI Linux legs lower
vm.mmap_rnd_bits first; amd64 TSan cannot run under x86_64-on-ARM
translation and is a real-hardware/CI gate only (documented in
run.sh).
- ubsan(win/arm64): native ARM64 Windows had no sanitizer at all —
AddressSanitizer ships no aarch64-w64-windows-gnu runtime. UBSan in
trap mode (-fsanitize-trap=undefined) needs no runtime library, so it
instruments natively and turns undefined behavior into an
illegal-instruction trap; -fstack-protector-strong adds stack-smash
coverage the heap tools miss. The GitHub windows-11-arm leg switches
from unsanitized to this, and vm/win.sh gains trap-ubsan-build /
trap-ubsan-test for local iteration (reproduce under the emulated
x86_64 UBSan to see which check fired). The whole codebase builds and
runs clean under it.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Client-side URL extraction only recognized static string literals; any
template literal was silently skipped, so parameterized endpoints never
produced HTTP_CALLS edges or Route nodes and cross-repo route matching
missed them (the server side already normalizes path params to {}).
New cbm_template_string_text() flattens a template_string node: string
fragments verbatim, each ${...} substitution becomes {}. Wired into:
- extract_positional_url / extract_string_value (call-arg URLs)
- handle_string_refs (URL-shaped refs from const/return positions)
- handle_string_constants (module-level const lookups)
`/api/v1/things/${id}` now yields __route__ANY__/api/v1/things/{} and
the enclosing function gets the HTTP_CALLS edge, joining the canonical
placeholder shape of server-side routes.
Signed-off-by: Charles Queiroz <fcqueiroz@liquibase.com>
CodeQL cpp/comparison-with-wider-type (4 high-severity alerts, #69-#72) flagged
two loops in the vendored ObjectScript scanner (objectscript_common.h, in both
the _routine and _udl grammar copies) where a uint8_t counter is compared
against an int length: the reverse_marker scan and the html_marker_buffer
reversal. Both lengths are hard-bounded by MARKER_BUFFER_MAX_LEN (30), so the
uint8_t counter can never wrap and the flagged infinite loop is unreachable in
practice — but widening the counters to int removes the pattern and clears the
gate. Provably behavior-neutral (int holds every uint8_t value; length <= 30).
Recorded as a local modification in the vendored grammar MANIFEST so it is
re-applied on the next re-vendor.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
A function-like macro invocation whose argument is a type token — e.g.
ALLOC(int, n) — makes tree-sitter's C/C++ grammar emit an ERROR node (it parses
the type in expression position), which cbm_collect_error_regions recorded as a
parse_partial coverage gap. But the macro is #defined in the same file and the
call sits inside an already-extracted function body, so nothing is actually
missing from the graph — it's a benign call the grammar can't parse without the
preprocessor (#1071, systematic across allocation-macro-style codebases).
Subtract an error region only when it both (a) contains an invocation of a
file-defined function-like macro and (b) is fully enclosed by an extracted
Function/Method body. Condition (b) keeps a TOP-LEVEL macro invocation that
expands to a definition still flagged (#949: the generated def isn't in the
original span), and fails safe — a benign top-level call stays flagged rather
than a real gap being hidden.
Tests: a type-arg macro call inside a function no longer reports parse_partial;
a real in-body syntax error still does; #949/#946 preserved.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Comments are NAMED tree-sitter nodes, so the prev-sibling walk in
extract_decorators() stopped at one — silently dropping every decorator ABOVE
an interleaved comment:
@Post('login') <-- dropped
@HttpCode(HttpStatus.OK) <-- dropped
// throttled per IP and account
@Throttle({ ... }) <-- kept
async login(...)
The route then vanished from decorator/route queries, so documenting a
decorator made the endpoint disappear from the graph. Real-world impact: on a
NestJS backend, 1 of 95 HTTP endpoints was missing — the one whose throttle
policy carried an explanatory comment.
Comments are now transparent to the walk, the same way anonymous tokens
(e.g. TS `export`) already were. Reuses the existing is_comment_node() helper.
Signed-off-by: KolisCode <jhohantma@gmail.com>
Martin Vogel