perf(synthesis): provably-empty pass gates + prefilters — render/expo/rn/mybatis stop scanning repos they can't match; iface memo

Store-arc round 2 (#1388 follow-up). The synthesis pool barrier on dubbo
carried ~1.4s of passes that provably could not emit an edge for the
project: reactRenderEdges fanned out over every class before checking for
a render method (now: one indexed name lookup bounds candidates — not a
language gate, Java Litho-style render+setState still matches);
expo/rn cross-platform pairing streamed every method row without the
languages their edges require (now registry-gated: expo needs swift AND
kotlin file-languages, rn needs a JS-family caller for isBridge);
mybatis built its full java-method index before discovering there were no
mapper-XML methods (now collects the XML side first). ifaceEdges — real
work — stops re-fetching a hub interface's methods once per implementer
and skips supertype-less classes before any per-class lookup.

dubbo warm wall 8.49-8.79 → 8.14-8.24s (n=3/arm, caffeinated); barrier
784→435ms; the full removed pass work lands on low-core envelopes where
synthesis runs sequentially. Dumps byte-identical: dubbo old-vs-new,
pooled-vs-sequential, kernel-vs-wasm (441,270 rows) + excalidraw JSX-live
control (89,903 rows, 46 react-render edges reproduced). Suite 2,689 ×2
with CODEGRAPH_KERNEL_EXPECT=1.

Also ships the diagnostics that located the round (zero cost when off):
CODEGRAPH_RESOLVE_PROFILE=2 attributes per-ref time to resolveOne's
strategies (stage:*) and the name-matcher's sub-matchers (nm:*);
CODEGRAPH_SYNTH_TIMINGS now prints the store worker's decode-vs-SQL
split. Killed by measurement, recorded in the PR: import-failure negative
cache (both-outcome names exist — static imports resolve via
instance-method on jvm-miss), jvm-miss early return (1,939 later-strategy
edges), jsxEdges language gate (Java generics text produces jsx edges),
and §4d buffer→bind on Spring repos (extract() hook forces the decoded
path — kernel=0 bundles measured).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
Colby McHenry
2026-07-20 21:14:55 -05:00
parent 27c3c55436
commit c7dfd84246
5 changed files with 179 additions and 23 deletions
+1
View File
@@ -24,6 +24,7 @@ and adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
- Indexing very large projects on multi-core machines got faster again: the parallel-resolution workers now periodically refresh their read-only database connections, which lets database housekeeping advance instead of silently building up a backlog behind long-lived readers — a backlog that was taxing the indexer's own writes. Graphs remain byte-for-byte identical; the win is largest at Linux-kernel scale on many-core machines.
- Indexing on macOS now uses the machine's real memory headroom when sizing its parallel-resolution workers. macOS deliberately keeps RAM filled with reclaimable cache, so the previous free-memory reading came back tiny (~1GB on an otherwise idle machine) and silently halved the worker pool — a medium Java project's fresh index ran about 1520% slower than the hardware allowed. Graphs remain byte-for-byte identical; the same fix also lets a memory-driven analysis cache engage fully on macOS for large C codebases.
- Fresh indexing got a sizeable across-the-board speedup: during the initial build, the database's secondary lookup indexes are set aside and rebuilt once after parsing instead of being maintained row by row — the same proven trick the later linking phase already used, now applied to the whole parse lane — and the reference-resolution loop likewise stops maintaining lookup indexes it never reads, rebuilding them at the end when almost nothing is left in the table. A medium Java project's parse phase runs about 58% faster and its full fresh index about 19% faster end-to-end; a Linux-kernel-scale index that took ~15 minutes on an 8-core machine now completes in about 11, with the resolution phase alone dropping by a third. Graphs remain byte-for-byte identical, and incremental syncs are unaffected.
- The dynamic-dispatch analysis at the end of indexing now skips passes that provably can't produce anything for the project at hand: React re-render bridging when no class has a `render` method, React Native and Expo cross-platform pairing when the required languages aren't present, and MyBatis mapper linking when there's no mapper XML. Previously each of these scanned the whole graph before coming up empty — on a 4,000-file Java project that was about 0.9 seconds of wasted analysis per fresh index. The interface-implementation bridging pass also got cheaper on real work: it no longer re-fetches a hub interface's method list once per implementer, and classes that extend or implement nothing are skipped before any per-class lookups. Graphs remain byte-for-byte identical.
- Indexing large C and C++ codebases spends much less time in the function-pointer dispatch analysis (the pass that connects handler tables like a command table or an ops struct to their call sites): each source file is now read and prepared once instead of four times, files that can't contribute any dispatch wiring are skipped outright in the later linking steps, and on platforms with the native engine the per-file scanning itself now runs natively too. On a Linux-kernel-scale tree the pass runs about a third faster end-to-end, with graphs byte-for-byte identical; platforms without a native binary keep the same results on the previous path.
### Fixes
+31
View File
@@ -41,6 +41,15 @@ const port = parentPort;
let db: SqliteDatabase | null = null;
let queries: QueryBuilder | null = null;
// CODEGRAPH_SYNTH_TIMINGS: split the writer lane's busy time into its two
// halves — kernel-buffer decode+finalize (JS-object materialization, the §4d
// buffer→bind candidate) vs the SQL bundle store — printed once at close.
const STORE_TIMINGS = !!process.env.CODEGRAPH_SYNTH_TIMINGS;
let decodeNs = 0n;
let storeNs = 0n;
let bundleCount = 0;
let kernelBundleCount = 0;
type InMessage =
| { type: 'open'; dbPath: string; fastInit: boolean }
| { type: 'bundle'; bundle: StoreBundle | KernelStoreBundle }
@@ -88,6 +97,23 @@ port.on('message', (msg: InMessage) => {
}
case 'bundle': {
if (!queries) throw new Error('store-worker: bundle before open');
if (STORE_TIMINGS) {
bundleCount++;
const t0 = process.hrtime.bigint();
let bundle: StoreBundle;
if ('kernel' in msg.bundle) {
kernelBundleCount++;
bundle = decodeKernelBundle(msg.bundle);
} else {
bundle = msg.bundle;
}
const t1 = process.hrtime.bigint();
queries.storeFileBundle(bundle);
decodeNs += t1 - t0;
storeNs += process.hrtime.bigint() - t1;
port.postMessage({ type: 'ack' });
break;
}
const bundle = 'kernel' in msg.bundle ? decodeKernelBundle(msg.bundle) : msg.bundle;
queries.storeFileBundle(bundle);
port.postMessage({ type: 'ack' });
@@ -98,6 +124,11 @@ port.on('message', (msg: InMessage) => {
break;
}
case 'close': {
if (STORE_TIMINGS && bundleCount > 0) {
console.error(
`[store-timing] bundles=${bundleCount} (kernel=${kernelBundleCount}) decode=${(Number(decodeNs / 1_000_000n) / 1000).toFixed(2)}s store=${(Number(storeNs / 1_000_000n) / 1000).toFixed(2)}s`
);
}
try {
db?.close();
} catch {
+54 -7
View File
@@ -401,8 +401,23 @@ async function reactRenderEdges(queries: QueryBuilder, ctx: ResolutionContext, o
let scanned255 = 0;
const edges: Edge[] = [];
const seen = new Set<string>();
// A class can only emit here if it CONTAINS a method named `render` — so one
// indexed name lookup bounds the candidate set up front, and the class scan
// below skips everything else before its per-class edge/node queries. On a
// repo with few/no render methods (any non-React codebase) this collapses
// the pass from every-class fan-out to ~zero DB work, with identical output:
// the skipped classes fail the same `render` check today, just after paying
// for their children. (Not a language gate: `render` + `this.setState(` in
// Java — e.g. Litho — legitimately matches today and still does.)
const renderOwners = new Set<string>();
for (const n of ctx.getNodesByName('render')) {
if (n.kind !== 'method') continue;
for (const e of queries.getIncomingEdges(n.id, ['contains'])) renderOwners.add(e.source);
}
if (renderOwners.size === 0) return edges;
for (const cls of queries.iterateNodesByKind('class')) {
if ((++scanned255 & 63) === 0) await onYield();
if (!renderOwners.has(cls.id)) continue;
const children = queries.getOutgoingEdges(cls.id, ['contains'])
.map((e) => queries.getNodeById(e.target))
.filter((n): n is Node => !!n && n.kind === 'method');
@@ -1029,11 +1044,20 @@ async function interfaceOverrideEdges(queries: QueryBuilder, onYield: MaybeYield
let scanned255 = 0;
const edges: Edge[] = [];
const seen = new Set<string>();
const methodsOf = (classId: string): Node[] =>
queries
// Memoized: a popular base interface's method list is otherwise re-fetched
// once per implementer (dubbo-style hub interfaces have hundreds), and the
// memo only ever serves reads. Same rows, same order — byte-identical.
const methodsMemo = new Map<string, Node[]>();
const methodsOf = (classId: string): Node[] => {
const hit = methodsMemo.get(classId);
if (hit) return hit;
const methods = queries
.getOutgoingEdges(classId, ['contains'])
.map((e) => queries.getNodeById(e.target))
.filter((n): n is Node => !!n && n.kind === 'method');
methodsMemo.set(classId, methods);
return methods;
};
// Concrete-side kinds vary by language: `class` covers Java / Kotlin /
// C# / TS / Swift-classes / Scala-classes; `struct` covers Swift value
// types that conform to protocols. Iterate both.
@@ -1041,9 +1065,14 @@ async function interfaceOverrideEdges(queries: QueryBuilder, onYield: MaybeYield
for (const kind of concreteKinds) {
for (const cls of queries.iterateNodesByKind(kind)) {
if ((++scanned255 & 63) === 0) await onYield();
// A class can only emit here if it HAS a supertype edge — check that
// (one edge query) before materializing its methods: most classes in a
// typical graph extend/implement nothing and skip in one hop.
const sups = queries.getOutgoingEdges(cls.id, ['implements', 'extends']);
if (sups.length === 0) continue;
const implMethods = methodsOf(cls.id).filter((n) => IFACE_OVERRIDE_LANGS.has(n.language));
if (implMethods.length === 0) continue;
for (const sup of queries.getOutgoingEdges(cls.id, ['implements', 'extends'])) {
for (const sup of sups) {
const base = queries.getNodeById(sup.target);
if (!base || !IFACE_OVERRIDE_LANGS.has(base.language) || base.id === cls.id) continue;
// Group impl methods by name to handle OVERLOADS: an interface `list()` and
@@ -1783,6 +1812,18 @@ async function mybatisJavaXmlEdges(queries: QueryBuilder, onYield: MaybeYield):
let scanned255 = 0;
const edges: Edge[] = [];
const seen = new Set<string>();
// Collect the XML side FIRST (mapper `<select id=…>` statements extracted as
// xml-language method nodes): if the project has none — every Java+XML repo
// that doesn't use MyBatis — return before paying the full java-method
// stream below. Same rowid stream order as matching inline, so the edge
// output is byte-identical when mappers do exist.
const xmlMethods: Node[] = [];
for (const m of queries.iterateNodesByKind('method')) {
if ((++scanned255 & 63) === 0) await onYield();
if (m.language === 'xml') xmlMethods.push(m);
}
if (xmlMethods.length === 0) return edges;
// Index Java methods by `<ClassName>::<methodName>` for O(1) lookup.
const javaIndex = new Map<string, Node[]>();
for (const m of queries.iterateNodesByKind('method')) {
@@ -1797,9 +1838,8 @@ async function mybatisJavaXmlEdges(queries: QueryBuilder, onYield: MaybeYield):
if (arr) arr.push(m); else javaIndex.set(key, [m]);
}
for (const xml of queries.iterateNodesByKind('method')) {
for (const xml of xmlMethods) {
if ((++scanned255 & 63) === 0) await onYield();
if (xml.language !== 'xml') continue;
// Qualified name: `<namespace>::<id>`. Extract the simple class name.
const colonIdx = xml.qualifiedName.lastIndexOf('::');
if (colonIdx < 0) continue;
@@ -3511,8 +3551,15 @@ export const SYNTH_PASSES: SynthPassDef[] = [
{ name: 'goGrpcEdges', gate: (has) => has('go'), run: (q, _c, y) => goGrpcStubImplEdges(q, y) },
{ name: 'rnEventEdgesList', gate: (has) => has(...JS_FAMILY), run: (_q, c, y) => rnEventEdges(c, y) },
{ name: 'fabricNativeEdges', gate: ALWAYS, run: (_q, c, y) => fabricNativeImplEdges(c, y) },
{ name: 'expoXPlatEdges', gate: ALWAYS, run: (q, _c, y) => expoCrossPlatformEdges(q, y) },
{ name: 'rnXPlatEdges', gate: ALWAYS, run: (q, _c, y) => rnCrossPlatformEdges(q, y) },
// Expo module nodes (`expo-module:` ids) are emitted only from .swift/.kt
// files, and a pair needs BOTH platforms — so without both languages the
// pass's only collection loop is provably empty (it was streaming every
// method row on pure-Java repos to find nothing).
{ name: 'expoXPlatEdges', gate: (has) => has('swift') && has('kotlin'), run: (q, _c, y) => expoCrossPlatformEdges(q, y) },
// An RN cross-platform edge requires a JS-language caller on the native
// method (`isBridge`) — no JS-family files means no JS-language nodes, so
// the result is provably empty.
{ name: 'rnXPlatEdges', gate: (has) => has(...JS_FAMILY), run: (q, _c, y) => rnCrossPlatformEdges(q, y) },
{
name: 'mybatisEdges',
gate: (has) => has('java', 'kotlin') && has('xml'),
+46 -8
View File
@@ -16,7 +16,7 @@ import {
FrameworkResolver,
ImportMapping,
} from './types';
import { matchReference, matchFunctionRef, matchDottedCallChain, matchScopedCallChain, matchMethodCall, sameLanguageFamily, crossesKnownFamily } from './name-matcher';
import { matchReference, matchFunctionRef, matchDottedCallChain, matchScopedCallChain, matchMethodCall, sameLanguageFamily, crossesKnownFamily, dumpNameMatcherProfile } from './name-matcher';
import { resolveViaImport, resolveJvmImport, extractImportMappings, extractReExports, loadCppIncludeDirs, isPhpIncludePathRef, isCobolCopybookRef, isNixPathImportRef, clearImportResolverMemos } from './import-resolver';
import { ResolverPool, minRefsForPool } from './resolver-pool';
import { detectFrameworks } from './frameworks';
@@ -805,12 +805,14 @@ export class ReferenceResolver {
ref.language === 'arkts' && ref.referenceName.startsWith('.')
? ref.referenceName.slice(1)
: ref.referenceName;
if (
!isNixPathImportRef(ref) &&
!this.hasAnyPossibleMatch(existenceName) &&
!this.matchesAnyImport(ref) &&
!this.frameworks.some((f) => f.claimsReference?.(ref.referenceName))
) {
const tPre = this.profileStages ? process.hrtime.bigint() : 0n;
const preFilterPass =
isNixPathImportRef(ref) ||
this.hasAnyPossibleMatch(existenceName) ||
this.matchesAnyImport(ref) ||
this.frameworks.some((f) => f.claimsReference?.(ref.referenceName));
if (this.profileStages) this.stageAdd('preFilter', ref, preFilterPass, tPre);
if (!preFilterPass) {
return null;
}
@@ -838,7 +840,9 @@ export class ReferenceResolver {
// JVM FQN imports skip framework/name-matcher: `import com.example.Bar`
// resolves directly through the qualifiedName index, which is unambiguous
// even when several `Bar` classes exist in different packages.
const tJvm = this.profileStages ? process.hrtime.bigint() : 0n;
const jvmImport = resolveJvmImport(ref, this.context);
if (this.profileStages) this.stageAdd('jvmImport', ref, !!jvmImport, tJvm);
if (jvmImport) return jvmImport;
// Razor/Blazor: a markup or `@code` type ref resolves through the file's
@@ -858,16 +862,25 @@ export class ReferenceResolver {
// JS → native `calls`) — `gateFrameworkLanguage` only drops a type/import
// edge between two KNOWN families (see its doc), never a `calls` bridge or
// a config↔code edge.
const tFw = this.profileStages ? process.hrtime.bigint() : 0n;
let fwEarly: ResolvedRef | null = null;
for (const framework of this.frameworks) {
const result = this.gateFrameworkLanguage(framework.resolve(ref, this.context), ref);
if (result) {
if (result.confidence >= 0.9) return result; // High confidence, return immediately
if (result.confidence >= 0.9) {
fwEarly = result; // High confidence, return immediately (below)
break;
}
candidates.push(result);
}
}
if (this.profileStages) this.stageAdd('frameworks', ref, fwEarly !== null, tFw);
if (fwEarly) return fwEarly;
// Strategy 2: Try import-based resolution
const tImp = this.profileStages ? process.hrtime.bigint() : 0n;
const importResult = this.gateLanguage(resolveViaImport(ref, this.context), ref);
if (this.profileStages) this.stageAdd('viaImport', ref, !!importResult, tImp);
if (importResult) {
if (importResult.confidence >= 0.9) return importResult;
candidates.push(importResult);
@@ -892,7 +905,9 @@ export class ReferenceResolver {
}
// Strategy 3: Try name matching
const tName = this.profileStages ? process.hrtime.bigint() : 0n;
let nameResult = this.gateLanguage(matchReference(ref, this.context), ref);
if (this.profileStages) this.stageAdd('nameMatch', ref, !!nameResult, tName);
// Nix has no ambient cross-file namespace — a callee binds lexically
// (same file) or through explicit import/callPackage wiring (the import
// path above). A cross-file name match is wrong by construction: every
@@ -1278,6 +1293,27 @@ export class ReferenceResolver {
private resolveProfile: Map<string, { n: number; ns: bigint }> | null =
process.env.CODEGRAPH_RESOLVE_PROFILE ? new Map() : null;
/**
* CODEGRAPH_RESOLVE_PROFILE=2 additionally attributes time to the
* STRATEGIES inside resolveOne (`stage:<name>|<refKind>|hit/miss` rows in
* the same histogram) i.e. WHICH machinery a failing class of refs pays
* for, not just that it fails. =1 keeps the per-outcome rows only.
*/
private profileStages: boolean = process.env.CODEGRAPH_RESOLVE_PROFILE === '2';
private stageAdd(stage: string, ref: UnresolvedRef, hit: boolean, t0: bigint): void {
if (!this.resolveProfile) return;
const dt = process.hrtime.bigint() - t0;
const key = `stage:${stage}|${ref.referenceKind}|${hit ? 'hit' : 'miss'}`;
const slot = this.resolveProfile.get(key);
if (slot) {
slot.n++;
slot.ns += dt;
} else {
this.resolveProfile.set(key, { n: 1, ns: dt });
}
}
private resolveOneTimed(ref: UnresolvedRef): ResolvedRef | null {
if (!this.resolveProfile) return this.resolveOne(ref);
const t0 = process.hrtime.bigint();
@@ -1305,6 +1341,8 @@ export class ReferenceResolver {
`[resolve-profile] ${label} ${r.k}: n=${r.n} total=${(r.ms / 1000).toFixed(1)}s avg=${((r.ms * 1000) / Math.max(1, r.n)).toFixed(0)}µs`
);
}
// =2 only: this thread's matchReference sub-stage table rides along.
dumpNameMatcherProfile(label);
}
resolveListForAdmission(refs: UnresolvedReference[]): {
+47 -8
View File
@@ -2047,6 +2047,45 @@ export function matchFuzzy(
/** ArkUI attribute-helper decorators a `.attr(...)` chain may resolve to. */
const ARKUI_ATTRIBUTE_DECORATORS = new Set(['Extend', 'Styles', 'AnimatableExtend', 'Builder']);
/**
* CODEGRAPH_RESOLVE_PROFILE=2 sub-stage attribution for matchReference's
* strategy pipeline (`nm:<stage>|<refKind>|hit/miss`). Module-global because
* the matcher is a free function; each thread (main + every pool worker) has
* its own module instance, and dumpNameMatcherProfile is invoked from
* ReferenceResolver.dumpResolveProfile so worker tables surface too.
*/
const NM_PROFILE: Map<string, { n: number; ns: bigint }> | null =
process.env.CODEGRAPH_RESOLVE_PROFILE === '2' ? new Map() : null;
function nmTimed(stage: string, ref: UnresolvedRef, fn: () => ResolvedRef | null): ResolvedRef | null {
if (!NM_PROFILE) return fn();
const t0 = process.hrtime.bigint();
const r = fn();
const dt = process.hrtime.bigint() - t0;
const key = `nm:${stage}|${ref.referenceKind}|${r ? 'hit' : 'miss'}`;
const slot = NM_PROFILE.get(key);
if (slot) {
slot.n++;
slot.ns += dt;
} else {
NM_PROFILE.set(key, { n: 1, ns: dt });
}
return r;
}
/** Dump this thread's matchReference sub-stage table to stderr (no-op unless =2). */
export function dumpNameMatcherProfile(label: string): void {
if (!NM_PROFILE || NM_PROFILE.size === 0) return;
const rows = [...NM_PROFILE.entries()]
.map(([k, v]) => ({ k, n: v.n, ms: Number(v.ns / 1_000_000n) }))
.sort((a, b) => b.ms - a.ms);
for (const r of rows) {
console.error(
`[resolve-profile] ${label} ${r.k}: n=${r.n} total=${(r.ms / 1000).toFixed(1)}s avg=${((r.ms * 1000) / Math.max(1, r.n)).toFixed(0)}µs`
);
}
}
export function matchReference(
ref: UnresolvedRef,
context: ResolutionContext
@@ -2119,18 +2158,18 @@ export function matchReference(
let result: ResolvedRef | null;
// 0. File path match (e.g., "snippets/drawer-menu.liquid" → file node)
result = matchByFilePath(ref, context);
result = nmTimed('filePath', ref, () => matchByFilePath(ref, context));
if (result) return result;
// 1. Qualified name match (highest confidence)
result = matchByQualifiedName(ref, context);
result = nmTimed('qualifiedName', ref, () => matchByQualifiedName(ref, context));
if (result) return result;
// 1b. C++ chained call whose receiver is another call — `Foo::instance().bar()`
// encoded as `Foo::instance().bar` by the extractor (#645). Resolve the
// receiver's type from what the inner call returns, then the method on it.
if (ref.language === 'cpp' || ref.language === 'c') {
result = matchCppCallChain(ref, context);
result = nmTimed('cppChain', ref, () => matchCppCallChain(ref, context));
if (result) return result;
}
@@ -2139,7 +2178,7 @@ export function matchReference(
// type is the factory's `self` (PHP `: self`/`: static`, Rust `-> Self`) or
// concrete return type.
if (ref.language === 'php' || ref.language === 'rust') {
result = matchScopedCallChain(ref, context);
result = nmTimed('scopedChain', ref, () => matchScopedCallChain(ref, context));
if (result) return result;
}
@@ -2161,20 +2200,20 @@ export function matchReference(
ref.language === 'objc' ||
ref.language === 'pascal'
) {
result = matchDottedCallChain(ref, context);
result = nmTimed('dottedChain', ref, () => matchDottedCallChain(ref, context));
if (result) return result;
}
// 2. Method call pattern
result = matchMethodCall(ref, context);
result = nmTimed('methodCall', ref, () => matchMethodCall(ref, context));
if (result) return result;
// 3. Exact name match
result = matchByExactName(ref, context);
result = nmTimed('exactName', ref, () => matchByExactName(ref, context));
if (result) return result;
// 4. Fuzzy match (lowest confidence)
result = matchFuzzy(ref, context);
result = nmTimed('fuzzy', ref, () => matchFuzzy(ref, context));
if (result) return result;
return null;