fix/mutation-commit-receipt
9 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Andrey Kumanyaev
|
cb6b21ed97 |
Recognise migration paths with native Windows separators
The indexer hands coverage extractors an OS-native relative path on purpose: graphRelKey keeps backslashes on Windows so an incremental re-index's evict lookup matches the keys the cold walk wrote. IsMigrationPath matched '/'-separated segments only, so on Windows no nested migration file was recognised — migration ingest never ran, the canonical table / column / migration nodes were never emitted, and unreferenced_tables reported zero because nothing had a provider. Fold separators before matching. |
||
|
Andrey Kumanyaev
|
1bf4994ed1 |
feat(sql): ingest live-DB/migration schema as table+column nodes by default
The `gortex db schema` deliverable was inert by default: applyMigration- Extraction only emitted KindTable+KindMigration (no columns), and it ran only when the `sql` coverage domain was enabled — which defaults off and actively strips SQL nodes, so a written-out schema produced nothing. - ExtractCreateTablesWithColumns parses CREATE TABLE column lists (paren/quote-aware: VARCHAR(255), NUMERIC(10,2) stay intact; table-level constraints skipped) so tables now ingest with canonical col:: KindColumn nodes linked by EdgeMemberOf. - Migration/DDL extraction is decoupled from the noisy code-side `sql` gate: it is high-signal (CREATE TABLE in a migration file or a generated dump is unambiguous) so it always runs, and its nodes carry Meta[origin]=migration to survive stripSQLArtifacts; only the code-side string-literal SQL stays gated. - A `gortex db schema` dump is recognised by its header marker (IsGeneratedSchema) wherever it is saved and ingested with the real dialect (GeneratedSchemaDialect), not just under migrations/. |
||
|
Andrey Kumanyaev
|
2dc3631872 |
feat(cli): live database DSN connector for schema ingestion (gortex db schema --postgres)
Schema only entered the graph from SQL/ORM source files; a live database was opaque. 'gortex db schema --postgres <dsn>' connects via pgx, reads tables / columns / primary keys / foreign keys from information_schema (IntrospectPostgres), and renders standard CREATE TABLE + FK DDL (LiveSchema.ToDDL). Writing that DDL into a tracked repo feeds it through Gortex's existing, well-tested SQL extractor, so the live schema becomes db::postgres:: table/column nodes (with reference edges) identical to the ones migrations produce — cross-referencing the live database with the code that queries it, no separate ingestion path. The DDL generation is a pure, deterministic, round-trip-tested function; only the connection itself needs the database. Adds the pure-Go github.com/jackc/pgx/v5 driver. |
||
|
Andrey Kumanyaev
|
a3f5101ff9 |
refactor: mcp.Server.graph + analysis/etc. take graph.Store, not *graph.Graph
Mechanical interface-widening across the codebase so the daemon
can run on different storage backends (memory, ladybug, sqlite,
duckdb). Every public function that previously took
*graph.Graph as a parameter now takes graph.Store (the
interface *graph.Graph already implements).
What changed:
- internal/mcp/Server.graph: *graph.Graph -> graph.Store
- 55 files across 18 packages: parameter signatures only
- 3 struct fields where the parameter-type change cascaded
(wiki.Inputs.Graph, wiki.Generator.graph, docs.Deps.Graph,
dataflow.Engine.g, skills.Generator.graph)
- 2 in-package functions in internal/graph: ClassifyZeroEdge,
CaveatForZeroEdge
No behavioural change: every method called on a parameter is on
the graph.Store interface, *graph.Graph satisfies graph.Store,
and every existing caller continues to work because Store is
strictly more permissive than *graph.Graph.
What this unlocks: the daemon can now construct a Server with
any graph.Store implementation (store_ladybug, store_sqlite,
store_duckdb), not just the in-memory *graph.Graph. The capability
interfaces (PageRanker, CommunityDetector, ComponentFinder,
KCorer, SymbolSearcher, VectorSearcher) auto-engage via the
existing type assertions in handleAnalyze*. Cmd/gortex/server.go
backend selector flag lands in the next commit.
Driven via 4 parallel agents per leaf package (audit/search,
dataflow/query/exporter, wiki/semantic/contracts/resolver,
releases/blame/cochange/coverage/docs/server/skills/sql) plus
hand-edits for the cross-cutting bits.
|
||
|
Andrey Kumanyaev
|
e55c4e5f38 |
parser/languages, sql, mcp/analyze: KindString-anchored sql_rebuild + log_events + error_msg registry
Extends the KindString registry beyond metric/error_msg/route to also capture SQL queries (context="sql") and log messages (context="log_message") emitted by Go callers. Every SQL call now seeds a KindString shadow alongside the existing KindTable/EdgeQueries pair; every log call seeds one alongside the KindEvent emit. The shadows are gated by their owning domain (stripSQLArtifacts and stripObservabilityArtifacts respectively). Three new downstream consumers built on the registry: - analyze kind=sql_rebuild — sql.RebuildTablesFromStringRegistry walks every KindString sql node and rederives the KindTable/KindColumn/ EdgeQueries/EdgeReadsCol/EdgeWritesCol layer with zero source re-parse. Idempotent. Use after enabling the sql gate on an existing index or recovering the SQL layer after a snapshot round-trip drops Meta. - analyze kind=log_events — aggregates KindString log_message by literal value with per-entry severity and emitter list. Richer than the canonical KindEvent grouping when one literal collapses across multiple canonicalised event nodes. - analyze kind=error_surface — each thrower row now carries an error_msgs field (the error_msg literals emitted by it via EdgeEmits), joining the "what error types propagate" view with "what literal messages originate here" in one row. Also: - analysis: single-seed shortcut in fillImpactFromReach — skips the multi-seed merge/seen-map/re-sort overhead since precomputed tier slices are already unique and sorted by BuildIndex. ~1.8x faster on the 1000-caller fan-in benchmark (165µs vs 297µs, 5x fewer allocs). - analysis: fast-path perf test reworked to gate on relative speedup vs the live walk (>= 1.3x) plus a loose absolute backstop (15ms), removing the CI-flaky 3ms wall-time ceiling. 1000 iters with warm-up for stable measurement. |
||
|
Andrey Kumanyaev
|
7eff4819c2 |
parser, sql, graph, mcp: dbt / SQLMesh model + column extraction
Specialized extractor (internal/parser/languages/dbt.go) dispatched into
by the .sql and .yml host extractors via content sniff:
- dbt SQL models — Jinja ref/source/config markers or a models/-tree
path; emits a KindTable model node, KindColumn nodes from the SELECT
projection, and EdgeDependsOn lineage per ref()/source()
- SQLMesh SQL models — leading MODEL(...) block; parses block
properties, the columns(...) property (or SELECT projection
fallback), and FROM/JOIN lineage
- dbt schema YAML — schema.yml-style files fingerprinted by content;
extracts models/seeds/snapshots/sources with columns, descriptions,
data types, and tests
Node IDs are deterministic and name-derived (dbt::model::<name>,
dbt::source::<s>.<t>, sqlmesh::model::<qualified>) so lineage edges
resolve through node identity with no resolver pass. EdgeDependsOn now
also carries dbt/SQLMesh model lineage alongside infra dependencies.
sql.ProjectionColumns: new exported helper returning a query's
alias-aware output column names, reused by both model paths.
analyze kind=dbt_models: surfaces every dbt/SQLMesh model/seed/snapshot/
source with column count and lineage fan-in/out; framework/type/
materialized/name filters, GCX output.
|
||
|
Andrey Kumanyaev
|
24f088e423 |
blame, coverage, parser, semantic, sql: produce edges and nodes the schema declared but extractors never emitted
A graph-builder audit turned up 22 cases where node and edge
kinds were declared but no producer ever populated them — agents
asking "who authored X", "which tests cover Y", or "what columns
does this query write" got empty results, and several signals
the spec promised cross-language only existed in the Go extractor.
Schema fixes
- blame.EnrichGraph emits KindTeam person nodes (ID
team::<email>, meta.kind=person, repo-scoped via RepoPrefix)
plus EdgeAuthored edges alongside the existing meta.last_authored
stamp.
- coverage.EnrichGraph inverts each EdgeTests pointing at a
covered subject and emits EdgeCoveredBy carrying
meta.coverage_pct; 0%-covered subjects are skipped so the
relation reflects actual coverage rather than test existence.
- parser.ParseTree exposes HasParseErrors / CountParseErrors and
indexer.stampParseErrors stamps meta.has_parse_errors /
meta.parse_errors on the KindFile node so index_health can rank
broken sources.
Go fidelity
- golang.go switches value-side selectors and bare-ident value
uses from EdgeReferences to EdgeReads (LHS-of-assign already
emitted EdgeWrites). Return-statement reads are captured so
the "every value use is a read" rule holds end-to-end.
- go_function_shape.emitGoClosureCaptures walks each func_literal
for free variables and emits EdgeCaptures with meta.name,
honoring closure parameters, range-clause loop vars, type-
switch bindings, and var/const decls as scopes that suppress
the capture.
Function shape generalized
- New ts_/py_/rust_/java_function_shape.go helpers emit
KindParam, EdgeParamOf, EdgeTypedAs, EdgeReturns,
KindGenericParam + EdgeMemberOf for TypeScript, Python, Rust,
and Java. Each ships a type-canonicalizer that strips idiomatic
wrappers (Promise/Optional/Result/Mono/List/Box/Vec/Awaited/
PEP-604 unions, etc.) and skips primitives so the emitted
edges land on real type nodes.
Async spawns
- TS: await_expression and Promise.all/allSettled/race/then.
- Python: await and asyncio.{gather, create_task,
ensure_future, run, wait, wait_for, shield}.
- Rust: await_expression and tokio::spawn / spawn_blocking,
async_std::task::spawn, smol::spawn.
- Kotlin: launch / async / runBlocking / withContext /
coroutineScope + .await(). Lambdas are walked because the
Kotlin extractor doesn't materialise them as graph nodes.
- C#: await_expression and Task.Run / Task.Factory.StartNew,
ThreadPool.QueueUserWorkItem, Parallel.{ForEach, For, Invoke}.
LSP overhaul
- protocol.go gains CallHierarchy / TypeHierarchy capabilities
and item types; provider.go wires prepareCallHierarchy +
outgoingCalls / incomingCalls and prepareTypeHierarchy +
supertypes / subtypes.
- enrichCallHierarchy promotes text_matched / ast_inferred call
edges to lsp_resolved and adds calls the AST extractor missed
(typically cross-file).
- enrichTypeHierarchy emits EdgeExtends / EdgeImplements for
every indexed type or interface — the biggest non-Go win
because AST extraction can't follow `extends X` / `implements I`
across files.
- Hover / references / implementation resolve to the actual
identifier column instead of col=0. The old default empty-
resulted every method declaration in indented contexts: jdtls,
omnisharp, kotlin-language-server, and rust-analyzer all
require the position to land on the identifier itself.
Column-level SQL
- sql.ExtractColumns and sql.ColumnNodeID extract column refs
from INSERT col-lists, UPDATE SET assignments, and single-
table SELECT projections. Multi-table SELECTs (any JOIN)
suppress column emission because v1 can't disambiguate which
table each column belongs to without a real SQL parser.
- go_sql.go threads ColumnRef alongside TableRef and emits
KindColumn nodes plus EdgeReadsCol / EdgeWritesCol so
"which functions read this column" works alongside the
table-level EdgeQueries.
|
||
|
Andrey Kumanyaev
|
b6b24294c8 |
sql, indexer: extract CREATE TABLE from migrations and join to query-string tables
After SQL v1 modeled tables referenced by query strings, the
natural complement was a ground-truth source: the migration
files that declare those tables. This adds a second extraction
path that picks up CREATE TABLE statements from `.sql` files
under migration directories, emits KindMigration nodes for the
files plus KindTable nodes for the schemas they declare, and
shares the canonical db::<dialect>::<schema>.<table> ID with
the query-string extractor — so a migration creating `users`
and a Go query reading from `users` resolve to the same graph
node, exactly the join the spec called for.
- sql: ExtractCreateTables runs a DDL-aware regex that handles
every modifier shape — CREATE [GLOBAL/LOCAL] TEMPORARY |
TEMP | UNLOGGED TABLE [IF NOT EXISTS] — plus all four
identifier-quoting styles (ANSI, MySQL, T-SQL, bare). Schema-
qualified names retain their schema. ALTER and DROP
statements deliberately don't extract: a migration that
drops a table doesn't *provide* it, and per-migration
ordering for alter-as-delta is out of v1 scope.
- sql: IsMigrationPath recognises .sql files under directories
named migrate or migrations (case-insensitive). Matches
Rails (db/migrate/), golang-migrate (migrations/), and most
ORM generators. The path-only detection is fast and
repository-walk-free.
- sql: MigrationNodeID returns `migration::<path>` synthetic
IDs in the same shape as db:: tables and module:: deps. The
path component lets agents walk back to the originating file
in one hop.
- indexer: applyMigrationExtraction runs inside the existing
applyCoverageDomains hook when the SQL gate is enabled.
Detects migration paths, runs ExtractCreateTables, emits
KindMigration + per-table KindTable + EdgeProvides edges.
Tables share the canonical ID with query-string-derived
tables — graph.AddNode dedupes on ID, so the migration's
declaration and the runtime query call site naturally merge
into one node with two incoming edge kinds (provides from
the migration, queries from the caller).
- indexer: stripSQLArtifacts extended to drop KindMigration
nodes and EdgeProvides edges originating from them when the
SQL gate is off. Same endpoint-aware shape as the existing
KindTable / EdgeQueries strip.
- tests: 4 new cases in sql — CREATE TABLE basic + all
modifier variants + dedup-at-same-schema-table +
ALTER/DROP-don't-extract — plus path-detection and ID
helpers. End-to-end on a synthetic fixture (db/migrate/
init.sql declaring two tables, pkg/db.go querying one of
them) produces a 5-node graph: migration node, two table
nodes, one query call site; with provides edges from
migration → tables and a queries edge from caller →
matching table. Race-enabled package tests pass.
v1 scope notes documented in code: dialect on migration
tables is generic (the .sql file alone doesn't reveal the
runtime dialect — agents can join through the file path or
surrounding go.mod when it matters); migration ordering and
dependencies between migrations aren't modeled; EdgeConsumes
from a migration to the FK-target table it REFERENCES isn't
yet emitted, only the declaring EdgeProvides is captured.
|
||
|
Andrey Kumanyaev
|
3e37702450 |
sql, parser, indexer: extract table refs from string-literal SQL queries
An agent asking "which services touch the users table", "did this
PR add a write to a high-traffic table", or "find every emitter
of TRUNCATE in pkg/admin" had no graph answer — string-literal
SQL was opaque to every query tool. This adds a regex-based
extractor that emits one KindTable node per distinct table and
EdgeQueries edges from each call site, so blast-radius and
ownership questions about database tables become single-call
graph queries.
- sql: new package. ExtractTables runs six regex passes (FROM,
JOIN, INSERT INTO, UPDATE, DELETE FROM, TRUNCATE [TABLE])
over a query string and dedupes by (op, schema, table). The
regex strips the four shapes of SQL identifier quoting (ANSI
"x", MySQL `x`, T-SQL [x], bare). splitSchemaTable splits
schema-qualified names while keeping the immediate parent
when a database segment is also present (db.schema.table →
schema=schema, table=table; the database segment is dropped
because it's rarely useful for graph queries).
- sql: maskDeleteFromForFromPattern is a pre-pass that
substitutes FROM with a sentinel inside DELETE FROM clauses
so the bare FROM regex doesn't double-match the same table
as both a select and a delete. Go's regexp doesn't support
negative lookbehind, so masking is the cleanest workaround.
Sentinel is non-keyword (__GFOX_FROM__) so the regex's
character class ignores it.
- parser: new go_sql.go. goSQLExecMethods enumerates Go SQL-
driver method names from database/sql (Query, Exec, QueryRow,
Prepare, *Context variants), sqlx (Get, Select, NamedExec,
NamedQuery, MustExec), and pgx. The shape-driven heuristic
catches most Go SQL libraries without per-library plumbing
— name collisions outside SQL contexts (cache.Get,
search.Query) are the false-positive surface that justifies
the gate staying default-off.
- parser: detectGoSQLCall walks the call's argument_list for
the first string-literal arg, runs sql.ExtractTables, and
returns the table refs. emitGoSQLEvents deduplicates
KindTable nodes within a file (cross-file dedup happens
automatically since graph.AddNode is idempotent on the
canonical db::<dialect>::<schema>.<table> ID) and emits one
EdgeQueries per call site with op and method on the meta.
- indexer: stripSQLArtifacts drops KindTable nodes and
EdgeQueries edges when the sql coverage domain is gated off.
Endpoint-aware so leftover edges to stripped table nodes are
pruned. Mirrors the strip passes for flags / configs /
observability — same dispatch and same justification (the
domain ships default-off because string-literal extraction
has a known false-positive rate).
- tests: 11 cases in sql (basic select, joins, the full insert/
update/delete/truncate set, all four quoting styles, schema-
qualified identifiers, dedup at the same-op same-table key,
mixed-op CTEs, empty + no-table queries, helper coverage)
plus 5 cases in parser/languages (extraction across multiple
SQL operations, dynamic-query skip, non-SQL-method
cache-shape negative case, schema-qualified meta, dedup
across call sites). End-to-end on a synthetic fixture with
SELECT + DELETE FROM + INSERT INTO produces three correctly
typed table nodes and three queries edges. Race-enabled
package tests pass.
v1 trade-offs documented in code: dynamic SQL (string
concatenation, query builders) is invisible; SQL keywords
used as identifiers misclassify; column-level resolution
(EdgeReadsCol / EdgeWritesCol) is deferred to a follow-up;
dialect is generic for all extractions until per-driver
inference lands; migration files / ORM models / sqlc codegen
as schema sources are not yet wired in.
|