Compare commits

...

43 Commits

Author SHA1 Message Date
Marius van Niekerk 1f81c0f572 test(storage): scope incremental lookups after rebase
Current main requires incremental session lookup to include the agent because one path can belong to multiple provider namespaces. Preserve the existing test intent by selecting the seeded Claude and Codex sessions explicitly.
2026-08-17 14:40:04 -04:00
Marius van Niekerk 00c9e9b458 fix(storage): preserve delegated usage through Bun
The usage cutover must select complete Claude snapshots before applying model and session filters, then credit the surviving tokens and fees to the original session. Otherwise delegated transcripts change dashboard totals, session counts, activity reports, and web-search costs when the shared Bun path replaces backend-specific SQL.\n\nKeep operational DuckDB costing aligned with that contract and remove renderer-only tests whose implementation no longer has a production caller.
2026-08-17 14:40:04 -04:00
Marius van Niekerk 42010cacb6 test(storage): model stamped PostgreSQL metadata values
Stamped convergence now validates the metadata value rather than only key existence. Make the schema probe return the stored compatibility value so fail-closed and DDL-skip tests continue to exercise the stamped path.
2026-08-17 14:34:42 -04:00
Marius van Niekerk afc2b9adb2 fix(storage): keep shared reads and pricing portable
Bun SelectQuery.Clone drops a transaction connection, which deadlocks composite reads on a single-connection DuckDB store. Rebind cloned session queries to the guarded view handle.

Bun also renders zero-valued default-tagged fields as DEFAULT, but existing PostgreSQL pricing-band columns have no physical defaults. Bind every canonical numeric price explicitly while retaining bounded shared batches across all adapters.
2026-08-17 14:34:42 -04:00
Marius van Niekerk 78bfaeb6f6 test(storage): follow canonical trigger ownership
Foundation now gives Bun-owned identity triggers non-colliding names so legacy startup DDL cannot overwrite them. Keep the later stamped-drift regression pointed at that canonical trigger while preserving its fail-without-repair assertion.
2026-08-17 14:34:42 -04:00
Marius van Niekerk a35654a7be perf(storage): stream content analytics rows
Signal and trend requests can span an entire archive, so retaining every matching transcript body makes peak memory grow with archive size. Reduce narrow content projections as database rows arrive and bound each query to a small session batch without limiting the final totals.\n\nSignal drill-down still ranks the complete candidate set, then loads full message content only for the requested examples. Trend session filtering keeps date bounds at message time so timestamp fallback behavior does not change.
2026-08-17 14:34:42 -04:00
Marius van Niekerk 615688b100 fix(storage): preserve Bun composite read snapshots
Remote snapshot adapters can replay a read callback after the underlying generation changes. Composite session and message reads must therefore discard the first attempt and publish only the accepted retry, including count, hydration, and timing data.\n\nThe PostgreSQL usage cutover also left an unreachable query renderer behind and dropped a required schema-validation return. Remove only renderer paths with no production caller, retain the activity-report row and pricing contracts, and restore the schema prerequisite.
2026-08-17 14:34:42 -04:00
Marius van Niekerk 588b4d4418 fix(storage): preserve Bun analytics parity
The shared analytics and usage paths must retain backend ordering, optional-schema, replay, and activity-window contracts while keeping remote reads bounded. Review exposed edge cases where the first unified implementation could mix rejected snapshots, drift across dialects, or substitute the wrong timestamps.

Keep only genuinely dialect-specific timestamp expressions at the adapter boundary and preserve the established empty-result and nil-safety contracts so the next search/vector cutover starts from a verified common store.
2026-08-17 14:34:42 -04:00
Marius van Niekerk 90d7658597 refactor(storage): unify Bun analytics queries
Analytics, trends, activity reports, and recent edits are common serving behavior, but independent backend implementations made filters, timestamp handling, and aggregation semantics drift across engines.\n\nRoute the full family through one replay-safe Bun snapshot and shared reducers, leaving backend wrappers with only their operational usage helpers. Remove the private SQL-builder tests and concrete scanners so future analytics changes have one owner and one literal cross-backend contract.
2026-08-17 14:34:42 -04:00
Marius van Niekerk 2377af3175 fix(storage): preserve Bun consistency invariants
Unified reads and writes must stay inside one backend snapshot and one atomic publication boundary, including retry, optional-schema, pricing, and mirror identity edge cases. Without those guarantees the common Bun path could combine generations, leak replay state, or partially publish adapter metadata.

Keep filtering and time windows in SQL, use database time for portable insight ordering, and make the remaining adapter-specific capabilities explicit so the next cutover slices can delete concrete paths safely.
2026-08-17 14:18:27 -04:00
Marius van Niekerk efe269a140 refactor(storage): unify Bun usage queries
Pricing and usage are common storage behavior, but three independent implementations made exact money arithmetic, catalog fallback, windowing, and transaction semantics vulnerable to backend drift.\n\nRoute all usage reads and pricing state through BunStore, preserve engine-neutral cost reducers, and make pricing-row plus band replacement atomic so every adapter observes the same catalog snapshot.
2026-08-17 14:18:27 -04:00
Marius van Niekerk 9d7b2f1314 fix(storage): resolve Bun review invariants
Shared storage callbacks and mirror replication must remain correct under replay, read-only operation, stale replicated identities, and partial metadata failures. These cases sit at adapter boundaries where a nominally unified query can otherwise publish rejected state or make a valid mirror unreadable.

Preserve read-only Recall validation, make composite results attempt-local, reconcile mirror-owned pin IDs, and publish DuckDB generation metadata atomically with an explicit compatibility requirement. Keep the implementation plan aligned so later Bun cutover work retains those constraints.
2026-08-17 14:14:19 -04:00
Marius van Niekerk 95930a365a refactor(storage): unify Bun Recall operations
Recall should follow the same guarded handle and capability policy as the other common Store families. Concrete remote stubs split method ownership and made read-only rejection depend on each wrapper.

Route canonical entry reads, inserts, and query events through BunStore while retaining SQLite-only FTS, vector, import, and eval behavior behind an explicit capability.
2026-08-17 14:14:18 -04:00
Marius van Niekerk 6118033218 fix(storage): preserve Bun mutation invariants
Unified Bun mutations must retain each adapter’s operational guarantees and expose result counts only after atomic success. Quack also needs a replacement identity that cannot collide when descriptive metadata repeats.

Keep SQLite FTS cleanup and PostgreSQL database-clock revisions behind adapter hooks, and stamp disposable DuckDB mirrors with a schema-v11 opaque generation.
2026-08-17 14:14:18 -04:00
Marius van Niekerk 589b500605 refactor(storage): unify Bun session mutations
Session rename and trash behavior must preserve the same user-tombstone, alias-exclusion, and atomicity rules regardless of the serving engine. Separate implementations made PostgreSQL ownership timestamps, SQLite watcher baselines, and DuckDB read-only policy easy to drift.

Route the family through operation-scoped Bun transactions, retain only adapter-owned operational touches, and reject unsupported writes before opening a backend write guard.
2026-08-17 14:14:18 -04:00
Marius van Niekerk 6697d195db fix(storage): harden Bun consistency and capabilities
Composite reads and curation writes must remain coherent across engine lifecycle changes, canonical ID transitions, and restricted PostgreSQL roles. Without explicit adapter guarantees, a unified method could still mix DuckDB generations, hydrate an entire archive, collide replicated pin IDs, or lose the established read-only sentinel.\n\nMake snapshot semantics mandatory, keep the one-time cutover fence fixed, bound candidate hydration by selected projects, and separate generated versus mirrored curation identities. Probe PostgreSQL insight insertion and deletion independently while preserving permission failures as ErrReadOnly.
2026-08-17 14:14:18 -04:00
Marius van Niekerk 6a1b8fd701 refactor(storage): unify Bun insight operations
Insight persistence and reads are dashboard behavior regardless of serving engine, but concrete implementations duplicated filtering, timestamp normalization, and write policy. That drift made future schema work depend on three method families instead of the canonical Bun model.\n\nRoute insights through the operation-scoped common store, keep DuckDB read-only before SQL, and require PostgreSQL's existing insight capability probe before permitting writes.
2026-08-17 14:14:18 -04:00
Marius van Niekerk e621ddaef9 refactor(storage): unify Bun curation operations
Stars and pins are dashboard curation regardless of the serving engine, but concrete implementations encoded different message identities and read-only behavior. Keeping them separate made PostgreSQL and DuckDB semantics drift from the canonical session/ordinal model.

Route the family through operation-scoped Bun writes, preserve target-generated pin identities during replicated upserts, and keep DuckDB rejection ahead of SQL while allowing PostgreSQL curation despite its public remote-mode status.
2026-08-17 14:14:18 -04:00
Marius van Niekerk 4c93f8188e fix(storage): close Bun cutover review gaps
The long-lived canonical cutover must stay ahead of main-line data versions and treat an installed replacement as final even when its caller is canceled after the atomic swap. Otherwise a successful resync can be reported stale or reopened by an older writer.

Keep composite inventory, governance, and candidate reads on one backend snapshot with bounded hydration, and compare SQLite activity as instants so mixed-offset archives preserve the cross-backend contract.
2026-08-17 14:14:18 -04:00
Marius van Niekerk 8921e6ee09 refactor(storage): unify project data reads
Inventory, governance rules, and worktree candidates must observe the same canonical session provenance and source-scoped mappings on every serving backend. Separate scanners made cross-archive isolation and snapshot selection depend on the adapter in use.\n\nRun these reads once through BunStore, preserve the existing pure identity and governance reducers, and prove the literal contract against SQLite, DuckDB, and PostgreSQL before deleting the concrete implementations.
2026-08-17 14:14:18 -04:00
Marius van Niekerk 1a03108be2 fix(storage): complete canonical Bun cutover
The shared-store cutover must preserve archive chronology, parser bookkeeping, and source provenance while retiring backend-specific identity readers. Without these invariants, an upgraded archive could be reopened by an older binary or leave a required resync incomplete across restarts.\n\nMake the cutover version floor and pending-resync state atomic, keep canonical identity replacement and artifact writes provenance-complete, and restrict adapter-specific hydration to SQLite operational fields.
2026-08-17 14:14:17 -04:00
Marius van Niekerk 0ca7675e11 refactor(storage): unify project identity reads
Project identity must resolve from the same source-scoped rows on every serving backend before inventory and governance can move into the common store. Centralize observation selection and archive-scope aggregation in BunStore so SQLite, PostgreSQL, and DuckDB derive identical project identities without a backend-specific scanner.
2026-08-17 14:14:17 -04:00
Marius van Niekerk 34cd6a0747 fix(storage): harden unified Bun invariants
The shared-store cutover cannot trust a compatibility stamp unless canonical parents, indexes, triggers, and backend column contracts still match. Fail closed on stamped drift and advance the archive data version so older binaries cannot write through the one-time identity cutover.

Keep core reads bounded and chronological across large transcripts and mixed SQLite timestamps, while preserving PostgreSQL's native update marker and serialized convergence semantics.
2026-08-17 14:14:17 -04:00
Marius van Niekerk 7f1632bf7b refactor(storage): unify core Bun reads
Core storage reads had drifted across SQLite, PostgreSQL, and DuckDB, making lifecycle and schema parity difficult to preserve. Centralize canonical session, message, timing, cursor, and metadata reads in BunStore so adapters only own guarded handles and engine lifecycle.

Keep result hydration bounded, preserve case-sensitive partial-ID behavior portably, and expose source-file metadata only through full-session reads.
2026-08-17 14:14:17 -04:00
Marius van Niekerk cf03ad76f3 test(scripts): tolerate mapping trigger generations
The pre-journal archive fixture can be replayed before or after the top-layer legacy-trigger removal. Treat both legacy and source-scoped trigger names as optional so the fixture removes whichever schema generation exists instead of failing during setup.
2026-08-17 14:14:16 -04:00
Marius van Niekerk fda824c0df test(scripts): align pre-journal archive fixture
The screenshot compatibility fixture started from a current foundation database but removed only the legacy mapping journals. Leaving source-scoped journal triggers and the common-schema stamp described an impossible historical archive and made the test depend on stale schema state.
2026-08-17 14:14:16 -04:00
Marius van Niekerk d4970c4d6a fix(storage): bound stamped PostgreSQL validation
A compatibility stamp attests row invariants at convergence time. Rechecking foreign-key and duplicate-row invariants on every stamped open rejects legacy ownerless rows that replication intentionally adopts and turns startup into an unbounded data scan.

Keep stamped PostgreSQL validation structural while retaining the complete row checks before the one-time stamp is written.
2026-08-17 14:14:16 -04:00
Marius van Niekerk 3099beb627 fix(storage): reject unformatted Quack arguments
The Quack Bun resolver receives fully formatted SQL, so silently discarding direct driver arguments can execute a different query than the caller intended. Reject argument-bearing calls before forwarding while preserving Bun's zero-argument path and surfacing QueryRow failures through a recoverable row.
2026-08-17 14:14:16 -04:00
Marius van Niekerk afb02fd8f7 fix(storage): fail closed on stamped schema drift
Stamped SQLite archives reinstalled source identity triggers and reran table-wide invariants on every open, masking drift and making startup scale with archive size. Validate non-conflicting source-owned triggers and canonical indexes after the one-time convergence stamp, while leaving row invariants in the convergence transaction.

Keep the shared timestamp boundary compatible with SQLite's empty sentinel and persist canonical UTC RFC3339Nano text instead of driver-native time values.
2026-08-17 14:14:16 -04:00
Marius van Niekerk a92a5ff95e fix(storage): constrain pricing schema convergence
The common schema upgrade treated every underscore-prefixed model as internal metadata and accepted legacy text timestamp columns after stamping. That could delete valid private model pricing and let read-only or push startup use a schema that no longer matches the canonical timestamp contract.

Limit migration to the three reserved metadata keys and reject non-native PostgreSQL timestamp columns. Unstamped writable schemas can still converge under the advisory lock, while stamped or read-only drift now fails closed.
2026-08-17 14:14:16 -04:00
Marius van Niekerk ff1f92b302 fix(storage): tighten Bun schema boundaries
Canonical pricing rows must carry real timestamps, workflow staging must remain valid before publication, and stamped PostgreSQL schemas must be validated under the migration lock. Keeping these invariants in the foundation prevents later stack layers from inheriting dialect-specific repair paths.

Move SQLite pricing refresh state into dedicated metadata, normalize PostgreSQL pricing columns in the existing convergence transaction, and require explicit DuckDB mapping timestamps. Trigger removal remains owned by the final cutover in PR #1347 and kata vzty.
2026-08-17 14:14:16 -04:00
Marius van Niekerk f86a37692d docs(storage): align Bun cutover contracts
The unified storage plan overstated fresh SQLite constraints, treated pricing metadata as timestamps, and split write ownership across stack layers. Those contradictions obscured which guarantees the foundation can enforce and where the final write cutover belongs.\n\nRecord one SQLite physical relationship matrix, real pricing timestamps with separate refresh metadata, workflow-owned provenance completeness, lock-bound PostgreSQL validation, and unsupported downgrade recovery. Assign trigger removal to the final stack layer after application writes are centralized.
2026-08-17 14:14:15 -04:00
Marius van Niekerk d7011548cc fix(storage): harden canonical cutover
A stamped common schema must not replay stale identity inputs, accept empty session provenance, or preserve PostgreSQL constraints that reject canonical ordinal writes. Make the cutover fail closed while keeping SQLite upgrades additive and DuckDB rebuild-only.\n\nRoute local identity, snapshot, mapping, resync, and backfill behavior through the source-scoped tables so the legacy SQLite inputs become inert after their one-time migration.
2026-08-17 14:14:15 -04:00
Marius van Niekerk 7637cdf9bf refactor(storage): converge Bun schemas
A shared Bun store can only replace the three query paths once every backend exposes the same durable row contract without risking persistent data. Converge SQLite and PostgreSQL transactionally in place, stamp source provenance before new archive writes, and rebuild DuckDB from the canonical registry at schema version 10.

Keep shipped SQLite row-id aliases intact and let DuckDB omit only foreign-key DDL that conflicts with atomic mirror replacement; canonical ordinal relationships and explicit replacement ordering preserve the common behavior.
2026-08-17 14:14:15 -04:00
Marius van Niekerk a589e26764 fix(storage): close Bun schema and lifecycle gaps
The unified store cannot cut queries over safely while canonical schemas omit deletion and dedup semantics, row conversion can silently erase malformed timestamps, or guarded backends disagree about cursor and recovery state. Make those contracts explicit before schema convergence depends on them.\n\nPreserve SQLite's shipped row-ID relationships as physical aliases while defining ordinal-based common keys, and keep DuckDB's only constraint difference to its unsupported cascade syntax with explicit child-first mirror deletion.
2026-08-17 14:14:15 -04:00
Marius van Niekerk 864948b05d refactor(storage): add guarded Bun backends
The shared Bun store must not retain raw pools across SQLite archive reopens or DuckDB mirror replacements, and PostgreSQL's coarse remote-mode flag cannot describe its narrower curation capabilities.

Bind Bun execution to each adapter's existing lifetime guard, authorize writes by operation family, and route Quack SELECTs through its query table function. Keep handle ownership with the native pools so the cutover does not introduce a second close lifecycle.
2026-08-17 14:14:15 -04:00
Marius van Niekerk d5bd76629f refactor(storage): define canonical Bun models
The storage cutover needs one durable row contract before adapter migration can replace three independent query paths. Define the common serving tables, source-scoped identity shapes, UTC timestamp boundary, and lossless session/message conversions so later slices can converge schema and queries against a single Bun model registry.\n\nKeep SQLite's shipped message row ID as an accepted physical alias while making session/ordinal the generated logical key, avoiding a destructive archive rebuild.
2026-08-17 14:14:15 -04:00
Marius van Niekerk b77e41f8d1 docs(storage): define canonical key provenance
A unified schema still needs explicit compatibility rules for SQLite's shipped message rowid and a non-empty provenance source for parser-created sessions. Fix the logical message key, preserve the archive's physical alias without rebuilding it, and require archive/database identity to flow through migration and batch writes.
2026-08-17 14:14:15 -04:00
Marius van Niekerk 3e27ad9062 fix(storage): harden DuckDB dialect identity
DuckDB must remain distinguishable from both current and future built-in Bun dialects, and mirror IDs must stay source-assigned even when a canonical model carries an autoincrement tag for another engine. Prove the generated catalog has neither a default nor a sequence before later schema work depends on this invariant.
2026-08-17 14:14:15 -04:00
Marius van Niekerk 556cd39380 fix(storage): resolve unified Bun review gaps
The cutover plan needs executable decisions for identity ownership, replication conflicts, pricing state, search hit identity, Quack coverage, and performance enforcement before later slices can safely remove backend-specific paths. Record those boundaries and make the DuckDB execution suite portable to platforms without a Windows ARM64 driver binary.
2026-08-17 14:14:15 -04:00
Marius van Niekerk 18f53f6ce0 feat(storage): add Bun DuckDB dialect
DuckDB needs a first-class Bun identity so the unified store can generate portable schema and queries without inheriting SQLite sequence or type behavior. Keep the advertised feature set conservative and tied to real driver execution, including rollback, conflict, returning, CTE, and update-from behavior.
2026-08-17 14:14:14 -04:00
Marius van Niekerk 1d188215e3 docs: plan unified Bun storage
The storage cutover spans schema, lifecycle, query, search, replication, and migration boundaries that must land in a dependency-safe order.

Record executable TDD slices and verification gates so the three stores converge without weakening archive safety, PostgreSQL write capabilities, DuckDB rebuild semantics, or performance checks.
2026-08-17 14:14:14 -04:00
Marius van Niekerk 83869ce55c docs: design unified Bun storage
The three database stores currently duplicate most query and scanning behavior, so schema parity depends on maintaining independent implementations. Record the approved cutover to one Bun-backed store and one canonical model registry before the large refactor begins.\n\nThe design preserves SQLite as the archive, PostgreSQL as an in-place-migrated sync target, and DuckDB as a rebuildable mirror. It also fixes the allowed backend seams to lifecycle, operational metadata, and small FTS/vector capabilities, avoiding a permanent legacy path during the transition.
2026-08-17 14:14:14 -04:00
213 changed files with 24222 additions and 29936 deletions
+5 -1
View File
@@ -574,7 +574,11 @@ func TestExportSessionsUpgradeRequiresBackgroundEvidenceBackfill(t *testing.T) {
require.NoError(t, database.Close())
raw, err := sql.Open("sqlite3", dbPath)
require.NoError(t, err)
_, err = raw.Exec(`DROP TABLE session_project_identity_snapshots`)
_, err = raw.Exec(`
DROP TABLE session_project_identity_snapshots;
DELETE FROM source_session_project_identity_snapshots
WHERE source_session_id = 'legacy'
`)
require.NoError(t, err)
require.NoError(t, raw.Close())
File diff suppressed because it is too large Load Diff
@@ -0,0 +1,466 @@
# Unified Bun Storage Design
## Goal
Replace the independent SQLite, PostgreSQL, and DuckDB `db.Store` query paths
with one shared Uptrace Bun-backed implementation. Define the durable schema
once, preserve the existing role of each database, and limit engine-specific
code to connection lifecycle, operational metadata, and small full-text/vector
search capabilities.
## Current Problem
AgentsView currently exposes one `db.Store` contract but implements it three
times. The local SQLite archive, PostgreSQL read store, and DuckDB read store
repeat query construction, row scanning, filter handling, analytics, usage, and
curation behavior. A small shared query-dialect helper reduces some filter
drift, but most behavior still lives in backend-specific packages.
This structure makes every storage-visible feature a three-path change. It also
allows the physical schemas and result conversions to diverge even when the
engines support practically identical tables and SQL.
## Scope
This refactor will:
1. Add Uptrace Bun as the common schema, query, execution, transaction, and row
scanning layer.
1. Define one canonical model registry for durable cross-backend data.
1. Implement a dedicated Bun dialect for DuckDB.
1. Replace the three common `db.Store` implementations with one shared store.
1. Route parser ingestion, PostgreSQL push, and DuckDB mirror population through
the canonical Bun models and transaction/upsert helpers.
1. Migrate existing SQLite and PostgreSQL databases forward in place.
1. Rebuild and atomically replace DuckDB mirrors at a new schema version.
1. Retain small backend-specific full-text and vector implementations.
The backend roles do not change in this work:
- SQLite remains the persistent writable archive and system of record.
- PostgreSQL remains a synchronized remote store with its current read and
limited curation/write capabilities.
- DuckDB remains a disposable read mirror. A local push writes it; serve and
Quack paths remain read-only.
## Architecture
`internal/db` remains the domain package and owner of the `db.Store` contract.
It gains the canonical persistence models and one Bun-backed common store.
The existing concrete types become thin compositions around that common store:
- SQLite `DB` retains writer/reader pool ownership, write serialization,
checkpointing, reopening, draining, archive maintenance, and resync state.
- `postgres.Store` retains PostgreSQL connection policy, sync coordination,
schema/search-path setup, and PostgreSQL search capabilities.
- `duckdb.Store` retains mirror file identity, replacement watching, local or
Quack transport, and DuckDB search capabilities.
These wrappers do not implement duplicate common store queries. They supply a
backend adapter to the common store and expose only responsibilities that are
genuinely backend-specific.
### Backend Adapter
The backend adapter provides:
- engine identity and Bun dialect;
- guarded access to the current Bun read and write handles;
- read-only/write capability policy;
- transaction entry points;
- engine-specific search capabilities; and
- close, reopen, and replacement lifecycle behavior.
Shared methods execute inside adapter callbacks. This prevents a query from
retaining a SQLite or DuckDB handle after the wrapper has swapped or retired it.
PostgreSQL uses the same interface with a stable pool.
### DuckDB Bun Dialect
DuckDB receives a first-class internal implementation of Bun's `schema.Dialect`
contract based on `schema.BaseDialect`. It must not identify itself as SQLite or
inherit SQLite-only sequence/type behavior.
The dialect owns:
- the exact Bun feature flags supported by the pinned DuckDB driver;
- identifier quoting;
- string, JSON, byte, boolean, and UTC timestamp literals;
- Go-to-DuckDB type mapping;
- primary-key and generated-ID DDL behavior;
- default schema/catalog behavior; and
- table metadata normalization required by canonical schema generation.
DuckDB catalog DDL contains only defaults that Quack can attach. The dialect
omits dynamic timestamp defaults such as `current_timestamp`; every DuckDB
writer supplies those timestamp values explicitly instead.
Focused execution tests against a real temporary DuckDB database determine the
feature set. Unsupported operations fail explicitly rather than falling back to
SQLite behavior.
## Canonical Schema
One Bun model registry defines every durable entity shared between the serving
backends, including sessions, messages, usage events, tool data, pricing,
project identity, worktree mappings, curation, insights, and the related join or
metadata tables needed to serve those features.
DuckDB receives the complete common column set even when a mirror does not
populate a feature. This keeps common queries and scans structurally identical.
Physical representations may differ only where the engines require equivalent
syntax or affinity:
- SQLite stores booleans with integer affinity; PostgreSQL and DuckDB use native
booleans.
- Identity/sequence syntax and integer widths follow the engine while preserving
the same logical key.
- SQLite may retain text timestamp affinity in shipped tables; PostgreSQL and
DuckDB use native timestamp types. Canonical scanners normalize all values
to UTC domain values.
- Pricing and pricing-band `updated_at` values are always timestamps. Pricing
refresh versions, attempt markers, and other arbitrary state live in a
dedicated SQLite `pricing_metadata` extension rather than sentinel pricing
rows.
- JSON uses a portable representation unless a narrowly scoped engine feature
requires a native type.
The following remain documented extensions rather than canonical data tables:
- SQLite archive, parser, skip, watcher, and resync bookkeeping;
- SQLite pricing refresh metadata;
- PostgreSQL push/source bookkeeping;
- DuckDB `sync_metadata` and mirror provenance;
- FTS tables, generated search columns, and search indexes; and
- vector generation tables and vector indexes.
The common table definitions, columns, relationships, and ordinary-index
semantics do not otherwise fork by engine. DuckDB is the narrow constraint
syntax exception: its mutable mirror tables omit foreign-key DDL because DuckDB
rejects parent updates and same-transaction child-first replacement even when
cascading actions are absent. The read-only mirror writer enforces the same
relationships through atomic whole-session replacement and explicit child-first
deletion. SQLite and PostgreSQL generate `ON DELETE CASCADE` from the shared
relationship metadata.
The convergence uses this ownership matrix:
| Area | Canonical serving representation | Adapter-owned extension |
| -------------------------------------- | ---------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------- |
| Sessions | Domain fields plus `source_archive_id` and `source_database_generation` provenance | SQLite parser cursors/linearity, PostgreSQL owner and remote-curation baselines, DuckDB push fingerprints |
| Messages and dependent rows | `(session_id, ordinal)` is the logical message key; source row IDs remain data columns where present | FTS/vector derived rows and generation state |
| Identity and mappings | `source_archives` and the three `source_*` identity/mapping tables, including source archive and generation keys | SQLite change journals and publication revisions; PostgreSQL publication-scope ownership tables |
| Usage, pricing, curation, and insights | One common column set and logical key per registered Bun model | SQLite pricing refresh metadata, provider import cursors, and backend capability probes |
When a pricing table is absent on a compatible read-only target or contains no
rows, embedded rates form the canonical base catalogue. Custom rates overlay
that base and an explicitly supplied effective catalogue overlays both; SQLite,
PostgreSQL, and DuckDB expose identical pricing provenance for this state.
Generated curation targets retain their own pin IDs on logical-key conflicts.
Mirrors instead preserve positive source-assigned pin IDs and transactionally
reconcile a reused ID away from any stale logical owner before the current pin
adopts it. Preserve-mode reconciliation is never used for generated-ID targets,
and a failed mirrored batch restores both the stale owner and prior logical pin.
Canonical generated message DDL uses `(session_id, ordinal)` as its composite
primary key and keeps `id` as an optional source row identifier. The shipped
SQLite archive is the one physical compatibility alias: it retains
`id INTEGER PRIMARY KEY` and its existing unique `(session_id, ordinal)`
constraint. Schema validation accepts that pair as logically equivalent and
never rebuilds the persistent table. PostgreSQL already uses the composite key;
DuckDB adopts it on the version-10 rebuild. No common query or relationship may
depend on SQLite rowid identity.
The same rule applies to dependent rows. Canonical tool calls use
`(session_id, message_ordinal, call_index)`, tool-result events extend that key
with `event_index`, and pins relate to messages through `(session_id, ordinal)`.
The shipped SQLite `tool_calls.message_id` and `pinned_messages.message_id`
columns remain non-null physical aliases because removing them would require
destructive table rebuilds. The SQLite adapter resolves those aliases from the
inserted message inside the same transaction; shared queries and every other
backend use only the canonical ordinal keys. PostgreSQL's prior
`pinned_messages.message_id` column is a nullable data alias after convergence;
the canonical `(session_id, ordinal)` key is authoritative there. Migration
tests retain existing tool calls and pins and prove new writes satisfy the
accepted representation on each persistent backend.
SQLite archives cannot gain table-level foreign keys through additive
`ALTER TABLE`, and the normal fresh `Open` path creates the same persistent
table shape before Bun fills missing common objects. SQLite therefore has one
accepted physical relationship matrix rather than separate fresh and upgraded
catalogs:
- shipped `messages.id` plus unique `(session_id, ordinal)` represents the
canonical message key;
- `tool_calls.message_id` retains message-delete cascade behavior while the
backfilled/triggered `message_ordinal` and canonical unique index represent
the logical tool key;
- `pinned_messages.message_id` retains message-delete cascade behavior while the
canonical ordinal index represents the logical pin key; and
- shipped tool-result rows retain their session cascade, while every
replacement/delete transaction removes result events before calls/messages.
PostgreSQL schemas receive the full canonical foreign-key metadata. SQLite
compatibility checks require the single matrix above and validate its data
invariants; no persistent table is rebuilt to make catalog text match. Tests
inspect the catalog created by the normal `Open` path, not an isolated Bun-only
schema constructor.
Canonical foreign-key metadata preserves session/message deletion behavior for
PostgreSQL and isolated generated-schema fixtures; SQLite preserves the same
observable deletion behavior through its accepted physical matrix. Canonical
usage dedup indexes use portable `CASE` expressions so empty keys may repeat
while non-empty keys remain unique on all engines; shipped SQLite/PostgreSQL
partial indexes are accepted as physically equivalent. Mirror-only source IDs,
including cursor-usage and secret-finding IDs, are nullable data rather than
generated canonical keys. Conversion of a non-empty timestamp that is not one of
the proven persistent forms returns an error and aborts the enclosing write
instead of silently producing `NULL` or a zero time.
Existing SQLite `project_identity_observations`,
`session_project_identity_snapshots`, and `worktree_project_mappings` are
one-time migration inputs. The convergence transaction backfills the canonical
`source_*` tables with the archive ID/generation before shared reads switch; the
same cutover redirects subsequent writes, so the old tables are not a runtime
fallback or dual-write path. PostgreSQL and DuckDB already use the canonical
source-scoped identity shape.
Opaque project selector keys use the aggregate identity scope derived from the
complete canonical `source_archives` set on every backend. SQLite does not keep
its former local-archive-only scope after cutover. Unresolved or ambiguous
response-scoped selector keys may change whenever canonical `source_archives`
membership changes through archive addition, retirement, or replacement.
Resolved repository identity keys remain stable. Callers must treat selector
keys as response-scoped opaque identifiers, not durable aliases; no dual key or
compatibility lookup remains.
`sessions.source_archive_id` and `sessions.source_database_generation` are
required publication provenance. Normal SQLite session writers read the stable
archive ID and database ID under the guarded handle and stamp both values in the
same transaction. The artifact-import coordinator may persist an incomplete row
during its crash-recoverable staging phase; governed reads and mirror
publication exclude that row until the coordinator completes provenance. Common
schema validation checks the columns, keys, and portable row invariants, not
this workflow state. The convergence migration still backfills both columns on
legacy sessions in the same transaction as the source-scoped identity tables.
## Schema Creation and Migration
Canonical Bun models generate fresh-database DDL and provide the expected schema
used by compatibility checks. Bun automatic reset or destructive schema diffing
is not used on persistent databases.
Existing shipped migrations remain immutable. This change adds one forward
schema-convergence migration and follows these rules:
- SQLite receives only additive or otherwise non-destructive changes. The
archive is never dropped, truncated, or recreated, and existing sessions are
preserved.
- PostgreSQL applies the corresponding changes transactionally in place and
retains existing synchronized data.
- The three existing pricing metadata rows (`_fallback_version`,
`_litellm_last_attempt`, and `_pricing_storage_version`) are moved once into
`pricing_metadata` and removed from `model_pricing`. Other
underscore-prefixed model patterns remain pricing data. Runtime reads use
only the new table; there is no dual read or write path.
- The migration leaves no permanent dual-schema read or write path. Once the
transaction succeeds, only the canonical model is used.
- DuckDB does not migrate in place. Its `SchemaVersion` is incremented; a push
builds a new canonical mirror, validates it, and swaps it atomically after
confirming the target is an AgentsView mirror.
Where an existing physical type is a valid engine-specific representation of the
canonical logical type, compatibility validation treats it as an explicit alias
rather than rewriting stored data without benefit.
The SQLite compatibility stamp is checked inside the convergence transaction. An
unstamped archive copies the three legacy identity inputs exactly once, installs
the publication machinery owned by this stack layer, validates invariants, and
stamps the same commit. A stamped archive validates and fails closed on drift;
it never replays legacy inputs or attempts an implicit repair. PostgreSQL always
acquires its advisory transaction lock before relying on the stamp, rechecks it
under that lock, and validates a stamped schema before returning. PostgreSQL
read compatibility and the push fast path require native timestamp types for
both pricing `updated_at` columns; writable push converges the shipped text
columns, while read-only serve and stamped drift fail closed. SQLite's writer
transaction/busy timeout and PostgreSQL's advisory lock serialize concurrent
openers.
Fail-closed validation checks the exact compatibility-stamp value, every column
selected by a canonical Bun model, logical-key uniqueness, registered parent
rows and indexes, SQLite's canonical trigger definitions, and PostgreSQL's
nullable `pinned_messages.message_id` compatibility constraint. Validation
executes no repair DDL; an unstamped migration establishes the same invariants
before writing the stamp.
Downgrading a database after this cutover is unsupported. Older released
binaries cannot recognize the new stamp and no trigger, shim, or dual-schema
path attempts to police them. Before upgrading a persistent archive, operators
who need a downgrade path must keep a pre-upgrade backup and restore that backup
before running the older binary. Read-only open on the new binary requires the
canonical source-scoped tables and reports an incompatible schema rather than
silently falling back.
## Query and Write Flow
Each backend opener configures its native `database/sql` pool and wraps the
current handle with Bun. Direct `database/sql` use is limited to opening,
driver-specific connection setup, pool configuration, handle draining, and
handle replacement. Application queries, schema operations, and transactions
flow through Bun.
PostgreSQL probes insight insertion and deletion independently because roles may
grant only one privilege. Those results authorize distinct write operations; if
permissions or transaction read-only state change after probing, the adapter
maps SQLSTATE `25006` and `42501` back to `db.ErrReadOnly` while retaining the
driver error in the chain.
`BunBackend.ConsistentView` is mandatory for composite reads. SQLite uses one
read transaction, PostgreSQL uses a repeatable-read transaction, and a local
DuckDB serving mirror holds one immutable guarded handle. A mutable direct
DuckDB handle uses one transaction. Quack cannot carry a remote transaction
across separate `query()` requests, so its adapter reads an opaque mirror
generation before and after the callback and retries the complete callback when
a server-side mirror replacement changes that token; repeated instability
returns an error rather than a mixed-generation result. Consistent-view
callbacks can therefore replay and must stage results until the guarded call
returns successfully. There is no non-snapshot fallback for adapters.
Common store methods use Bun models and query builders. Complex CTEs and
aggregates may use parameterized Bun raw fragments, but query composition,
literal formatting, execution, transactions, and model scanning remain under
Bun.
Those fragments use only the portable SQL subset exercised by all three
backends. UTC parsing, calendar bucketing, percentiles, regex normalization, and
JSON interpretation move to shared Go reducers whenever the engines do not share
semantics. The one non-search rendering exception is chronological filtering of
SQLite's shipped text timestamps: the SQLite adapter supplies `julianday`
expressions while PostgreSQL and DuckDB compare native timestamps. The shared
method still owns the query and result contract and does not branch on backend
identity. Any further non-search expression difference requires a design update.
Parser ingestion writes canonical rows through the SQLite adapter. PostgreSQL
push and DuckDB mirror population consume the same row models and common batch
helpers. The PostgreSQL adapter supplies the narrow replication conflict policy
that protects `owner_marker`, exclusions, aliases, and target-owned rename/trash
baselines; these operational fields never participate in generic replacement.
DuckDB keeps fingerprint gating and replaces a whole session in one transaction:
it deletes every dependent message, usage, tool, finding, and curation row
before inserting the canonical replacement, and propagates hard deletes before
advancing mirror metadata.
Quack uses the SQL generated by the DuckDB Bun dialect. The DuckDB adapter sends
that SQL through the attached catalog's `query()` transport and passes returned
rows to Bun's model scanner. Quack transport details do not leak into shared
store methods.
## Full-Text and Vector Capabilities
Full-text and vector search are the allowed backend-specific query areas.
Capabilities accept canonical filters and return canonical identifiers,
ordinals, stable tool/result coordinates, unit ranges, subordinate state,
scores, or ranks. The common store resolves lexical message hits to semantic
units, hydrates results, and owns final ordering.
SQLite may continue to use FTS5 and sqlite-vec, PostgreSQL may use its native
full-text facilities and pgvector, and DuckDB may use its supported regex/FTS
and vector facilities. These implementations may differ in syntax and index
maintenance, but their observable filters, result identity, and ordering remain
aligned as closely as engine semantics permit.
Engine switches, placeholder builders, and duplicate scan functions are not
allowed in common store methods. A difference that cannot fit the small search
capability boundary requires an explicit design update rather than an ad hoc
backend branch.
## Errors and Atomicity
Errors include operation and backend context while preserving established
sentinel behavior, including `db.ErrReadOnly`, `db.ErrSemanticUnavailable`, and
not-found handling based on `sql.ErrNoRows`.
Unsupported dialect features return a direct capability error. The shared store
does not retry a query through a legacy implementation or another dialect.
All multi-row writes and migrations are transactional. A model, extension, or
metadata failure rolls back the transaction. DuckDB replacement remains
validate-then-swap, so a failed build cannot replace a valid mirror. SQLite and
PostgreSQL schema versions advance only in the transaction that installs the
complete migration.
## Testing
Implementation follows vertical test-driven slices: add a failing observable
contract test, implement the common Bun behavior, verify it on the participating
backends, and then remove the superseded backend-specific method.
A reusable store-contract suite seeds hand-written canonical rows and asserts
literal outcomes for filtering, pagination, ordering, analytics, usage,
curation, and mutations. It runs against real temporary SQLite and DuckDB
databases locally and against PostgreSQL under the existing `pgtest` setup.
DuckDB dialect tests cover owned behavior only:
- canonical DDL/type generation;
- literal encoding;
- declared feature support;
- transactions and conflict handling; and
- representative Bun queries executed against a real temporary DuckDB file.
Migration tests cover a fresh database and the previous shipped schema with
existing data. SQLite and PostgreSQL tests assert that data survives and the
canonical schema becomes usable. DuckDB tests assert rebuild, validation, and
atomic replacement rather than in-place migration.
Existing lifecycle tests continue to protect SQLite pool reopening/draining,
DuckDB mirror swaps, and Quack reattachment. FTS/vector parity tests use shared
fixtures and literal expected result identities while allowing capability-
specific setup.
The Quack integration suite also runs representative Bun-generated parameterized
reads and model scans through a real `query()` attachment, including quotes,
binary/JSON values, stale-attachment retry, and credential redaction. A
resolver-only unit test is not sufficient evidence for the remote path.
Performance-sensitive list, search, usage, and analytics paths retain benchmark
coverage. Completion requires focused tests, the full Go suite, PostgreSQL
integration tests, DuckDB-tagged tests, formatting, vetting, and the
repository's backend benchmark gate where supported.
Deletion of the old query paths is verified by the compiler, focused source
searches during handoff, and passing replacement behavior tests. No test asserts
that deleted files or symbols remain absent.
## Cutover and Completion Criteria
The cutover is atomic at the code level. There is no runtime flag, compatibility
wrapper, or dual old/new store implementation.
The refactor is complete when:
1. All common `db.Store` behavior is implemented once through Bun.
1. SQLite, PostgreSQL, and DuckDB use the canonical model registry.
1. DuckDB uses the dedicated Bun dialect for local and Quack queries.
1. Existing SQLite and PostgreSQL data upgrades in place.
1. DuckDB mirrors rebuild and swap under the new schema version.
1. Backend-specific code is limited to lifecycle, operational metadata, sync
transport, and small FTS/vector capabilities.
1. Duplicate PostgreSQL and DuckDB common query/scanning implementations are
removed.
1. Contract, migration, lifecycle, integration, and performance checks pass.
## Non-goals
- Making DuckDB a system of record or accepting remote DuckDB writes.
- Changing SQLite, PostgreSQL, or DuckDB configuration and CLI behavior.
- Making all three backends equally writable.
- Replacing the underlying database drivers.
- Adding a permanent compatibility adapter for the old store implementations.
- Unifying engine internals that do not represent shared durable domain data.
- Moving SQLite Recall extraction, evidence reconciliation, FTS, or vector
generation into the common schema. Common Store entry points use the SQLite
Bun handle when Recall is available; other adapters reject them before SQL.
+8
View File
@@ -26,6 +26,7 @@ require (
github.com/testcontainers/testcontainers-go v0.43.0
github.com/testcontainers/testcontainers-go/modules/postgres v0.43.0
github.com/tidwall/gjson v1.19.0
github.com/uptrace/bun v1.2.18
go.kenn.io/docbank v0.11.0
go.kenn.io/kit v0.13.1
golang.org/x/mod v0.40.0
@@ -81,6 +82,7 @@ require (
github.com/jackc/pgpassfile v1.0.0 // indirect
github.com/jackc/pgservicefile v0.0.0-20240606120523-5a60cdf6a761 // indirect
github.com/jackc/puddle/v2 v2.2.2 // indirect
github.com/jinzhu/inflection v1.0.0 // indirect
github.com/klauspost/cpuid/v2 v2.3.0 // indirect
github.com/klauspost/crc32 v1.3.0 // indirect
github.com/lufia/plan9stats v0.0.0-20211012122336-39d0f177ccd0 // indirect
@@ -112,6 +114,7 @@ require (
github.com/prometheus/common v0.67.5 // indirect
github.com/prometheus/otlptranslator v1.0.0 // indirect
github.com/prometheus/procfs v0.20.1 // indirect
github.com/puzpuzpuz/xsync/v3 v3.5.1 // indirect
github.com/remyoudompheng/bigfft v0.0.0-20230129092748-24d4a6f8daec // indirect
github.com/rs/xid v1.6.0 // indirect
github.com/segmentio/asm v1.1.3 // indirect
@@ -122,6 +125,11 @@ require (
github.com/tinylib/msgp v1.6.4 // indirect
github.com/tklauser/go-sysconf v0.3.16 // indirect
github.com/tklauser/numcpus v0.11.0 // indirect
github.com/tmthrgd/go-hex v0.0.0-20190904060850-447a3041c3bc // indirect
github.com/uptrace/bun/dialect/pgdialect v1.2.18 // indirect
github.com/uptrace/bun/dialect/sqlitedialect v1.2.18 // indirect
github.com/vmihailenco/msgpack/v5 v5.4.1 // indirect
github.com/vmihailenco/tagparser/v2 v2.0.0 // indirect
github.com/yosida95/uritemplate/v3 v3.0.2 // indirect
github.com/yusufpapurcu/wmi v1.2.4 // indirect
github.com/zeebo/xxh3 v1.1.0 // indirect
+16
View File
@@ -127,6 +127,8 @@ github.com/jackc/pgx/v5 v5.10.0 h1:VhSvgU2jSli8o3AqIEOTJr7rZwAEUVo4E4XhR94Zfr0=
github.com/jackc/pgx/v5 v5.10.0/go.mod h1:mal1tBGAFfLHvZzaYh77YS/eC6IX9OWbRV1QIIM0Jn4=
github.com/jackc/puddle/v2 v2.2.2 h1:PR8nw+E/1w0GLuRFSmiioY6UooMp6KJv0/61nB7icHo=
github.com/jackc/puddle/v2 v2.2.2/go.mod h1:vriiEXHvEE654aYKXXjOvZM39qJ0q+azkZFrfEOc3H4=
github.com/jinzhu/inflection v1.0.0 h1:K317FqzuhWc8YvSVlFMCCUb36O/S9MCKRDI7QkRKD/E=
github.com/jinzhu/inflection v1.0.0/go.mod h1:h+uFLlag+Qp1Va5pdKtLDYj+kHp5pxUVkryuEj+Srlc=
github.com/klauspost/asmfmt v1.3.2 h1:4Ri7ox3EwapiOjCki+hw14RyKk201CN4rzyCJRFLpK4=
github.com/klauspost/asmfmt v1.3.2/go.mod h1:AG8TuvYojzulgDAMCnYn50l/5QV3Bs/tp6j0HLHbNSE=
github.com/klauspost/compress v1.19.0 h1:sXLILfc9jV2QYWkzFOPWStmcUVH2RHEB1JCdY2oVvCQ=
@@ -222,6 +224,8 @@ github.com/prometheus/otlptranslator v1.0.0 h1:s0LJW/iN9dkIH+EnhiD3BlkkP5QVIUVEo
github.com/prometheus/otlptranslator v1.0.0/go.mod h1:vRYWnXvI6aWGpsdY/mOT/cbeVRBlPWtBNDb7kGR3uKM=
github.com/prometheus/procfs v0.20.1 h1:XwbrGOIplXW/AU3YhIhLODXMJYyC1isLFfYCsTEycfc=
github.com/prometheus/procfs v0.20.1/go.mod h1:o9EMBZGRyvDrSPH1RqdxhojkuXstoe4UlK79eF5TGGo=
github.com/puzpuzpuz/xsync/v3 v3.5.1 h1:GJYJZwO6IdxN/IKbneznS6yPkVC+c3zyY/j19c++5Fg=
github.com/puzpuzpuz/xsync/v3 v3.5.1/go.mod h1:VjzYrABPabuM4KyBh1Ftq6u8nhwY5tBPKP9jpmh0nnA=
github.com/remyoudompheng/bigfft v0.0.0-20230129092748-24d4a6f8daec h1:W09IVJc94icq4NjY3clb7Lk8O1qJ8BdBEF8z0ibU0rE=
github.com/remyoudompheng/bigfft v0.0.0-20230129092748-24d4a6f8daec/go.mod h1:qqbHyh8v60DhA7CoWK5oRCqLrMHRGoxYCSS9EjAz6Eo=
github.com/rogpeppe/go-internal v1.15.0 h1:D0RCU5rMAp+SpgkiNdrjfJ+LX4J1M32V2NeCY7EJ6hc=
@@ -271,6 +275,18 @@ github.com/tklauser/go-sysconf v0.3.16 h1:frioLaCQSsF5Cy1jgRBrzr6t502KIIwQ0MArYI
github.com/tklauser/go-sysconf v0.3.16/go.mod h1:/qNL9xxDhc7tx3HSRsLWNnuzbVfh3e7gh/BmM179nYI=
github.com/tklauser/numcpus v0.11.0 h1:nSTwhKH5e1dMNsCdVBukSZrURJRoHbSEQjdEbY+9RXw=
github.com/tklauser/numcpus v0.11.0/go.mod h1:z+LwcLq54uWZTX0u/bGobaV34u6V7KNlTZejzM6/3MQ=
github.com/tmthrgd/go-hex v0.0.0-20190904060850-447a3041c3bc h1:9lRDQMhESg+zvGYmW5DyG0UqvY96Bu5QYsTLvCHdrgo=
github.com/tmthrgd/go-hex v0.0.0-20190904060850-447a3041c3bc/go.mod h1:bciPuU6GHm1iF1pBvUfxfsH0Wmnc2VbpgvbI9ZWuIRs=
github.com/uptrace/bun v1.2.18 h1:3HnRcMfS6OBPMG1eSOzlbFJ/X/AyMEJb7rMxE6VQvDU=
github.com/uptrace/bun v1.2.18/go.mod h1:wNltaKJk4JtOt4SG5I5zmA7v0/Mzjh1+/S906Rayd3Y=
github.com/uptrace/bun/dialect/pgdialect v1.2.18 h1:IZ6nM2+OYrL8lkEAy7UkSEZvoa3vluTAUlZfPtlRB2k=
github.com/uptrace/bun/dialect/pgdialect v1.2.18/go.mod h1:Tqdf4QP1okrGYpXfodXvCOK6Ob1OOTwSaoAzCgBB3IU=
github.com/uptrace/bun/dialect/sqlitedialect v1.2.18 h1:Z33SY/U++XK9uGWqS4h8OZVxfCXguIG+sU9cYq2PGFQ=
github.com/uptrace/bun/dialect/sqlitedialect v1.2.18/go.mod h1:1MVOS/Ncy4FZbkJcgUFH6OqYoQinYNjkEwsmNQEXz2A=
github.com/vmihailenco/msgpack/v5 v5.4.1 h1:cQriyiUvjTwOHg8QZaPihLWeRAAVoCpE00IUPn0Bjt8=
github.com/vmihailenco/msgpack/v5 v5.4.1/go.mod h1:GaZTsDaehaPpQVyxrf5mtQlH+pc21PIudVV/E3rRQok=
github.com/vmihailenco/tagparser/v2 v2.0.0 h1:y09buUbR+b5aycVFQs/g70pqKVZNBmxwAhO7/IwNM9g=
github.com/vmihailenco/tagparser/v2 v2.0.0/go.mod h1:Wri+At7QHww0WTrCBeu4J6bNtoV6mEfg5OIWRZA9qds=
github.com/x448/float16 v0.8.4 h1:qLwI1I70+NjRFUR3zs1JPUCgaCXSh3SW62uAKT1mSBM=
github.com/x448/float16 v0.8.4/go.mod h1:14CWIYCyZA/cWjXOioeEpHeN/83MdbZDRQHoFcYsOfg=
github.com/yosida95/uritemplate/v3 v3.0.2 h1:Ed3Oyj9yrmi9087+NczuL5BwkIc4wvTb5zIM+UJPGz4=
-147
View File
@@ -1,12 +1,5 @@
package db
import (
"context"
"database/sql"
"fmt"
"time"
)
// SessionActivityBucket holds message counts for one time interval.
type SessionActivityBucket struct {
StartTime string `json:"start_time"`
@@ -70,143 +63,3 @@ func abs64(x int64) int64 {
}
return x
}
// GetSessionActivity returns time-bucketed message counts for a
// session. Only visible messages are counted (system and
// prefix-detected injected messages excluded).
func (d *DB) GetSessionActivity(
ctx context.Context, sessionID string,
) (*SessionActivityResponse, error) {
return getSessionActivitySQLite(d, ctx, sessionID)
}
func getSessionActivitySQLite(
d *DB, ctx context.Context, sessionID string,
) (*SessionActivityResponse, error) {
// Count all messages, including system (for TotalMessages field).
var total int
err := d.getReader().QueryRowContext(ctx,
`SELECT COUNT(*) FROM messages WHERE session_id = ?`,
sessionID,
).Scan(&total)
if err != nil {
return nil, fmt.Errorf("counting messages: %w", err)
}
// Visible-message filter: exclude persisted system messages and
// prefix-detected injected user messages.
visibleFilter := "m.is_system = 0 AND " + SystemPrefixSQL("m.content", "m.role")
// Get min and max timestamps from visible messages with valid timestamps.
// Use julianday() for sub-second precision — strftime('%s') truncates.
tsFilter := "m.timestamp IS NOT NULL AND m.timestamp != '' AND julianday(m.timestamp) IS NOT NULL"
var minEpoch, maxEpoch sql.NullFloat64
err = d.getReader().QueryRowContext(ctx, fmt.Sprintf(`
SELECT
MIN((julianday(m.timestamp) - 2440587.5) * 86400.0),
MAX((julianday(m.timestamp) - 2440587.5) * 86400.0)
FROM messages m
WHERE m.session_id = ?
AND %s
AND %s`,
visibleFilter, tsFilter,
), sessionID).Scan(&minEpoch, &maxEpoch)
if err != nil {
return nil, fmt.Errorf("querying timestamp range: %w", err)
}
// If no timestamps, return empty buckets with total count.
if !minEpoch.Valid || !maxEpoch.Valid {
return &SessionActivityResponse{
Buckets: []SessionActivityBucket{},
TotalMessages: total,
}, nil
}
// Use floor of min epoch as anchor so bucket boundaries
// align to whole seconds. Compute duration from the exact
// float values to preserve sub-second precision.
epochMin := int64(minEpoch.Float64)
durationSec := int64(maxEpoch.Float64 - minEpoch.Float64)
interval := SnapInterval(durationSec)
// Query: group visible messages into buckets using float epoch
// for sub-second precision. The anchor is truncated to whole
// seconds so bucket boundaries align cleanly.
rows, err := d.getReader().QueryContext(ctx, fmt.Sprintf(`
SELECT
CAST(((julianday(m.timestamp) - 2440587.5) * 86400.0 - ?) / ? AS INTEGER) AS bucket_idx,
SUM(CASE WHEN m.role = 'user' THEN 1 ELSE 0 END) AS user_count,
SUM(CASE WHEN m.role = 'assistant' THEN 1 ELSE 0 END) AS asst_count,
MIN(m.ordinal) AS first_ordinal
FROM messages m
WHERE m.session_id = ?
AND %s
AND %s
GROUP BY bucket_idx
ORDER BY bucket_idx`,
visibleFilter, tsFilter,
), epochMin, interval, sessionID)
if err != nil {
return nil, fmt.Errorf("querying activity buckets: %w", err)
}
defer rows.Close()
type bucketRow struct {
idx int
userCount int
asstCount int
firstOrdinal int
}
var populated []bucketRow
maxIdx := 0
for rows.Next() {
var br bucketRow
if err := rows.Scan(
&br.idx, &br.userCount, &br.asstCount, &br.firstOrdinal,
); err != nil {
return nil, fmt.Errorf("scanning bucket row: %w", err)
}
populated = append(populated, br)
if br.idx > maxIdx {
maxIdx = br.idx
}
}
if err := rows.Err(); err != nil {
return nil, fmt.Errorf("iterating bucket rows: %w", err)
}
// Build the full bucket array including empty gaps.
bucketCount := maxIdx + 1
buckets := make([]SessionActivityBucket, bucketCount)
// Precompute a lookup from bucket index to populated row.
popMap := make(map[int]bucketRow, len(populated))
for _, br := range populated {
popMap[br.idx] = br
}
for i := range buckets {
startSec := epochMin + int64(i)*interval
endSec := startSec + interval
startTime := time.Unix(startSec, 0).UTC()
endTime := time.Unix(endSec, 0).UTC()
bucket := SessionActivityBucket{
StartTime: startTime.Format(time.RFC3339),
EndTime: endTime.Format(time.RFC3339),
}
if br, ok := popMap[i]; ok {
bucket.UserCount = br.userCount
bucket.AssistantCount = br.asstCount
ord := br.firstOrdinal
bucket.FirstOrdinal = &ord
}
buckets[i] = bucket
}
return &SessionActivityResponse{
Buckets: buckets,
IntervalSeconds: interval,
TotalMessages: total,
}, nil
}
+6 -172
View File
@@ -9,6 +9,7 @@ import (
"sort"
"time"
"github.com/uptrace/bun"
"go.kenn.io/agentsview/internal/activity"
"go.kenn.io/agentsview/internal/export"
"go.kenn.io/agentsview/internal/money"
@@ -34,111 +35,6 @@ func activityReportRangeBoundsUTC(q activity.Query) (string, string) {
q.RangeEnd.UTC().Format(boundLayout)
}
// GetActivityReport assembles a concurrency- and usage-oriented report
// for the resolved range `q`. Sessions and activity are fetched from the
// filtered candidate set. Usage loads candidate rows plus only the
// cross-session Claude peers needed for complete-snapshot selection, keeping
// the resulting streams consistent without materializing the whole window.
//
// The filter `f` is honored as-is: callers that want one-shot or
// automated sessions included must pass them through with the
// corresponding exclusions disabled. Subagent and fork sessions are
// always counted so the cost totals match GetDailyUsage, which never
// filters by relationship_type. Fork sessions hold only their own
// rewound-branch messages (the parsers partition entries across
// branches), so counting them adds no duplicate activity; any usage
// rows that do recur across sessions collapse in the aggregator's
// dedup, the same guarantee GetDailyUsage relies on.
func (db *DB) GetActivityReport(
ctx context.Context, f AnalyticsFilter, q activity.Query,
) (activity.Report, error) {
artifacts, err := db.BuildActivityReportArtifacts(ctx, f, q, nil)
if err != nil {
return activity.Report{}, err
}
artifacts.Report.BySession = artifacts.Sessions
artifacts.Report.SessionsTotal = len(artifacts.Sessions)
return artifacts.Report, nil
}
func (db *DB) BuildActivityReportArtifacts(
ctx context.Context,
f AnalyticsFilter,
q activity.Query,
onProgress activity.ProgressFunc,
) (activity.CandidateArtifacts, error) {
reportProgress(onProgress, activity.Progress{Phase: activity.ProgressLoadingSessions})
f.IncludeSubagents = true
f.IncludeForks = true
rangeStartUTC, rangeEndUTC := activityReportRangeBoundsUTC(q)
lowerBound := paddedUTCBound(q.RangeStart.UTC().Format(time.RFC3339), -14)
upperBound := paddedUTCBound(q.RangeEnd.UTC().Format(time.RFC3339), 14)
sessions, ids, err := db.activityReportSessions(
ctx, f, rangeStartUTC, rangeEndUTC)
if err != nil {
return activity.CandidateArtifacts{}, err
}
reportProgress(onProgress, activity.Progress{
Phase: activity.ProgressLoadingUsage, SessionsTotal: len(sessions),
})
usage, pricing, err := db.activityReportUsage(ctx, ids, lowerBound, upperBound, q)
if err != nil {
return activity.CandidateArtifacts{}, err
}
rowsProcessed := int64(0)
source := db.activityReportCandidateSource(ids, q)
artifacts, err := activity.BuildCandidateArtifactsFromSourceWithSurvivorUsage(ctx, activity.Params{
RangeStart: q.RangeStart,
RangeEnd: q.RangeEnd,
Loc: q.Loc,
EffectiveEnd: q.EffectiveEnd,
Partial: q.Partial,
GapCapSeconds: q.GapCapSeconds,
Bucket: q.Bucket,
}, sessions, func(
ctx context.Context, yield func(activity.IntervalCandidate) error,
) error {
reportProgress(onProgress, activity.Progress{
Phase: activity.ProgressScanningActivity, SessionsTotal: len(sessions),
})
return source(ctx, func(candidate activity.IntervalCandidate) error {
rowsProcessed++
reportProgress(onProgress, activity.Progress{
Phase: activity.ProgressScanningActivity,
SessionsTotal: len(sessions), RowsProcessed: rowsProcessed,
})
return yield(candidate)
})
}, usage)
if err != nil {
return activity.CandidateArtifacts{}, fmt.Errorf("aggregating activity report: %w", err)
}
reportProgress(onProgress, activity.Progress{
Phase: activity.ProgressFinalizing, SessionsTotal: len(sessions),
SessionsProcessed: len(sessions), RowsProcessed: rowsProcessed,
})
artifacts.Report.SchemaVersion = export.ActivityReportSchemaVersion
artifacts.Report.Pricing = pricing
projects, err := db.BuildProjectIdentityMap(ctx,
activityReportProjectLabels(sessions))
if err != nil {
return activity.CandidateArtifacts{}, err
}
artifacts.Report.BySession = artifacts.Sessions
activity.SanitizeProjectLabels(&artifacts.Report, projects)
artifacts.Sessions = artifacts.Report.BySession
artifacts.Report.BySession = []activity.SessionRow{}
artifacts.Report.Projects = export.ProjectMapForWire(projects)
reportProgress(onProgress, activity.Progress{
Phase: activity.ProgressDone, SessionsTotal: len(sessions),
SessionsProcessed: len(sessions), RowsProcessed: rowsProcessed,
})
return artifacts, nil
}
func reportProgress(callback activity.ProgressFunc, progress activity.Progress) {
if callback != nil {
callback(progress)
@@ -160,7 +56,7 @@ func (db *DB) GetSessionUsageRows(
if len(ids) == 0 {
return nil, nil
}
pricing, err := db.loadPricingMap(ctx)
pricing, err := db.LoadPricingMap(ctx)
if err != nil {
return nil, fmt.Errorf("loading pricing: %w", err)
}
@@ -388,26 +284,6 @@ func activityReportProjectLabels(
return sortedSetKeys(set)
}
// activityReportSessions returns the candidate sessions whose window
// overlaps the exact range [rangeStartUTC, rangeEndUTC), plus their
// IDs. The ID set defines the scope for the activity and usage fetches.
// NULLIF guards the empty-string timestamp fallbacks SQLite stores so a
// session with an empty ended_at but a valid started_at still falls back
// correctly, matching the activity-expression convention elsewhere.
//
// The effective-end fallback for a session with no ended_at uses its
// latest message timestamp before started_at, so a still-open or
// partially-parsed session that began before the range but has messages
// inside it is not dropped. COALESCE short-circuits, so the correlated
// MAX subquery runs only for the rare sessions missing an ended_at.
func (db *DB) activityReportSessions(
ctx context.Context, f AnalyticsFilter, rangeStartUTC, rangeEndUTC string,
) ([]activity.SessionMeta, []string, error) {
return db.activityReportSessionsFrom(
ctx, db.getReader(), f, rangeStartUTC, rangeEndUTC,
)
}
func (db *DB) activityReportSessionsFrom(
ctx context.Context,
q sessionExportQuerier,
@@ -672,48 +548,6 @@ func (db *DB) ActivityReportCandidateSource(
return db.activityReportCandidateSource(ids, q)
}
// activityReportUsage selects complete snapshots across the padded range,
// then keeps rows attributed to the candidate sessions. Rows are ordered on
// parsed instants so mixed RFC3339 representations remain chronological.
func (db *DB) activityReportUsage(
ctx context.Context, ids []string, lowerBound, upperBound string, q activity.Query,
) ([]activity.UsageRow, *export.PricingBlock, error) {
return db.activityReportUsageFrom(
ctx, db.getReader(), ids, lowerBound, upperBound, q,
)
}
func (db *DB) activityReportUsageFrom(
ctx context.Context,
source sessionExportQuerier,
ids []string,
lowerBound, upperBound string,
q activity.Query,
) ([]activity.UsageRow, *export.PricingBlock, error) {
candidates, rateResolver, err := db.loadActivityReportUsageCandidatesFrom(
ctx, source, ids, lowerBound, upperBound, false,
)
if err != nil {
return nil, nil, err
}
sortActivityReportUsageCandidates(candidates)
baseRows := make([]activity.UsageRow, len(candidates))
for i, candidate := range candidates {
row := candidate.row
_, row.OutputTokens, _, _, _ = dailyUsageRowTokens(candidate.scan)
row.WebSearchRequests = usageRowWebSearchRequests(
candidate.scan.usageSource, candidate.scan.tokenJSON)
baseRows[i] = row
}
mask, attribution, webSearchRequests :=
activity.UsageSurvivorSelectionForSessions(
q.RangeStart, q.RangeEnd, q.EffectiveEnd, baseRows, ids,
)
return materializeActivityReportUsageCandidates(
candidates, mask, attribution, webSearchRequests, rateResolver,
)
}
// activityReportUsageCandidate retains the scanned source fields until the
// survivor set is known. Pricing provenance is recorded only for survivors in
// the ordinary activity report, while reporting export can materialize every
@@ -726,9 +560,9 @@ type activityReportUsageCandidate struct {
ordinal int64
}
func (db *DB) loadActivityReportUsageCandidatesFrom(
func (db *BunStore) loadActivityReportUsageCandidatesFrom(
ctx context.Context,
source sessionExportQuerier,
source bun.IDB,
ids []string,
lowerBound, upperBound string,
restrictToIDs bool,
@@ -900,9 +734,9 @@ func sortActivityReportUsageCandidates(
// activityReportUsageCandidatesFrom returns normalized padded-range rows
// without sorting or applying a survivor mask. Reporting export merges these
// rows with standalone candidates before imposing either operation.
func (db *DB) activityReportUsageCandidatesFrom(
func (db *BunStore) activityReportUsageCandidatesFrom(
ctx context.Context,
source sessionExportQuerier,
source bun.IDB,
ids []string,
lowerBound, upperBound string,
includeWebSearch bool,
+9 -3
View File
@@ -11,6 +11,7 @@ import (
sqlite3 "github.com/mattn/go-sqlite3"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
"github.com/uptrace/bun"
"go.kenn.io/agentsview/internal/activity"
"go.kenn.io/agentsview/internal/export"
@@ -507,9 +508,14 @@ func TestLoadActivityReportUsageCandidatesBoundsFilteredWorkingSet(t *testing.T)
})
}
candidates, _, err := d.loadActivityReportUsageCandidatesFrom(
ctx, d.getReader(), []string{"candidate"},
"2026-06-15T10:00:00Z", "2026-06-17T10:00:00Z", false)
var candidates []activityReportUsageCandidate
err := d.consistentView(ctx, func(store bun.IDB) error {
var loadErr error
candidates, _, loadErr = d.loadActivityReportUsageCandidatesFrom(
ctx, store, []string{"candidate"},
"2026-06-15T10:00:00Z", "2026-06-17T10:00:00Z", false)
return loadErr
})
require.NoError(t, err)
require.Len(t, candidates, 2,
"the working set contains the candidate and its Claude peer only")
+25 -2723
View File
File diff suppressed because it is too large Load Diff
+5
View File
@@ -72,6 +72,11 @@ func (db *DB) applyArtifactImportedSession(
"artifact imported session write does not match provenance",
)
}
identity, err := db.localArchiveIdentity(ctx)
if err != nil {
return result, err
}
stampSessionArchiveIdentity(&write.Session, identity)
write = sanitizeSessionBatchWrite(write)
db.mu.Lock()
+273
View File
@@ -0,0 +1,273 @@
package db
import (
"context"
"fmt"
"sort"
"time"
"github.com/uptrace/bun"
"go.kenn.io/agentsview/internal/activity"
"go.kenn.io/agentsview/internal/export"
)
func (s *BunStore) GetActivityReport(
ctx context.Context, f AnalyticsFilter, q activity.Query,
) (activity.Report, error) {
artifacts, err := s.BuildActivityReportArtifacts(ctx, f, q, nil)
if err != nil {
return activity.Report{}, err
}
artifacts.Report.BySession = artifacts.Sessions
artifacts.Report.SessionsTotal = len(artifacts.Sessions)
return artifacts.Report, nil
}
func (s *BunStore) BuildActivityReportArtifacts(
ctx context.Context,
f AnalyticsFilter,
q activity.Query,
onProgress activity.ProgressFunc,
) (activity.CandidateArtifacts, error) {
reportProgress(onProgress, activity.Progress{Phase: activity.ProgressLoadingSessions})
f.IncludeSubagents = true
f.IncludeForks = true
lowerBound := paddedUTCBound(q.RangeStart.UTC().Format(time.RFC3339), -14)
upperBound := paddedUTCBound(q.RangeEnd.UTC().Format(time.RFC3339), 14)
var artifacts activity.CandidateArtifacts
err := s.consistentView(ctx, func(store bun.IDB) error {
candidateSessions, err := s.bunAnalyticsSessionsFrom(ctx, store, f, false)
if err != nil {
return err
}
candidateMessages, err := bunAnalyticsMessagesFrom(
ctx, store, bunAnalyticsSessionIDs(candidateSessions),
)
if err != nil {
return err
}
messagesBySession := bunAnalyticsMessagesBySession(candidateMessages)
var sessions []activity.SessionMeta
var ids []string
for _, row := range candidateSessions {
start := bunAnalyticsSessionTime(row)
end := start
if row.EndedAt != nil {
end = row.EndedAt.UTC()
} else {
for _, message := range messagesBySession[row.ID] {
if message.Timestamp != nil && message.Timestamp.After(end) {
end = message.Timestamp.UTC()
}
}
}
if end.Before(q.RangeStart.UTC()) || !start.Before(q.RangeEnd.UTC()) {
continue
}
title := row.ID
for _, candidate := range []*string{
row.DisplayName, row.SessionName, new(row.Project),
} {
if candidate != nil && *candidate != "" {
title = *candidate
break
}
}
sessions = append(sessions, activity.SessionMeta{
SessionID: row.ID, Title: title, Project: row.Project,
Agent: row.Agent, Machine: row.Machine,
StartedAt: bunAnalyticsTimeString(row.StartedAt),
EndedAt: bunAnalyticsTimeString(row.EndedAt),
IsAutomated: row.IsAutomated,
})
ids = append(ids, row.ID)
}
sort.Slice(sessions, func(i, j int) bool {
return sessions[i].SessionID < sessions[j].SessionID
})
ids = ids[:0]
allowed := make(map[string]struct{}, len(sessions))
for _, session := range sessions {
ids = append(ids, session.SessionID)
allowed[session.SessionID] = struct{}{}
}
events := make([]activity.ActivityEvent, 0, len(candidateMessages))
for _, message := range candidateMessages {
if _, ok := allowed[message.SessionID]; !ok || message.Timestamp == nil {
continue
}
events = append(events, activity.ActivityEvent{
SessionID: message.SessionID, Ordinal: message.Ordinal,
Role: message.Role, Timestamp: bunAnalyticsTimeString(message.Timestamp),
Model: message.Model,
})
}
sort.Slice(events, func(i, j int) bool {
if events[i].SessionID != events[j].SessionID {
return events[i].SessionID < events[j].SessionID
}
return events[i].Ordinal < events[j].Ordinal
})
reportProgress(onProgress, activity.Progress{
Phase: activity.ProgressLoadingUsage, SessionsTotal: len(sessions),
})
usage, pricing, err := s.bunActivityReportUsageFrom(
ctx, store, ids, lowerBound, upperBound, q,
)
if err != nil {
return err
}
candidates := activity.PairActivityEvents(
events, q.RangeStart, q.EffectiveEnd,
time.Duration(q.GapCapSeconds)*time.Second,
)
rowsProcessed := int64(0)
built, err := activity.BuildCandidateArtifactsFromSourceWithSurvivorUsage(
ctx,
activity.Params{
RangeStart: q.RangeStart,
RangeEnd: q.RangeEnd,
Loc: q.Loc,
EffectiveEnd: q.EffectiveEnd,
Partial: q.Partial,
GapCapSeconds: q.GapCapSeconds,
Bucket: q.Bucket,
},
sessions,
func(
ctx context.Context,
yield func(activity.IntervalCandidate) error,
) error {
reportProgress(onProgress, activity.Progress{
Phase: activity.ProgressScanningActivity,
SessionsTotal: len(sessions),
})
for _, candidate := range candidates {
if err := ctx.Err(); err != nil {
return err
}
rowsProcessed++
reportProgress(onProgress, activity.Progress{
Phase: activity.ProgressScanningActivity,
SessionsTotal: len(sessions),
RowsProcessed: rowsProcessed,
})
if err := yield(candidate); err != nil {
return err
}
}
return nil
},
usage,
)
if err != nil {
return fmt.Errorf("aggregating Bun activity report: %w", err)
}
reportProgress(onProgress, activity.Progress{
Phase: activity.ProgressFinalizing,
SessionsTotal: len(sessions),
SessionsProcessed: len(sessions),
RowsProcessed: rowsProcessed,
})
built.Report.SchemaVersion = export.ActivityReportSchemaVersion
built.Report.Pricing = pricing
projects, err := buildBunProjectIdentityMapFrom(
ctx, store, activityReportProjectLabels(sessions),
)
if err != nil {
return err
}
built.Report.BySession = built.Sessions
activity.SanitizeProjectLabels(&built.Report, projects)
built.Sessions = built.Report.BySession
built.Report.BySession = []activity.SessionRow{}
built.Report.Projects = export.ProjectMapForWire(projects)
reportProgress(onProgress, activity.Progress{
Phase: activity.ProgressDone,
SessionsTotal: len(sessions),
SessionsProcessed: len(sessions),
RowsProcessed: rowsProcessed,
})
artifacts = built
return nil
})
return artifacts, err
}
func (s *BunStore) bunActivityReportUsageFrom(
ctx context.Context,
store bun.IDB,
ids []string,
lowerBound, upperBound string,
q activity.Query,
) ([]activity.UsageRow, *export.PricingBlock, error) {
pricing, err := s.loadPricingMapFrom(ctx, store)
if err != nil {
return nil, nil, fmt.Errorf("loading activity-report pricing: %w", err)
}
rateResolver := export.NewPricingResolver(pricing)
if len(ids) == 0 {
block, blockErr := rateResolver.BuildBlock()
return []activity.UsageRow{}, &block, blockErr
}
loc := q.Loc
if loc == nil {
loc = time.UTC
}
projections, err := s.loadBunUsageProjections(ctx, store, UsageFilter{
From: q.RangeStart.In(loc).Format("2006-01-02"),
To: q.RangeEnd.In(loc).Format("2006-01-02"),
Timezone: q.Timezone,
}, false, "")
if err != nil {
return nil, nil, err
}
lower, lowerErr := parseTimestamp(lowerBound)
upper, upperErr := parseTimestamp(upperBound)
var candidates []activityReportUsageCandidate
for _, projection := range projections {
row := usageProjectionToDailyRow(projection)
parsed, parseErr := parseTimestamp(row.ts)
if parseErr == nil {
if lowerErr == nil && parsed.Before(lower) {
continue
}
if upperErr == nil && parsed.After(upper) {
continue
}
}
ordinal := int64(-1)
if row.messageOrdinal.Valid {
ordinal = row.messageOrdinal.Int64
}
candidates = append(candidates, activityReportUsageCandidate{
scan: row, ts: parsed, validTS: parseErr == nil, ordinal: ordinal,
row: activity.UsageRow{
SessionID: row.sessionID, Model: row.model, Timestamp: row.ts,
Project: row.project, Machine: row.machine, MessageOrdinal: ordinal,
UsageSource: row.usageSource, Agent: row.agent,
ClaudeMessageID: row.claudeMessageID,
ClaudeRequestID: row.claudeRequestID, SourceUUID: row.sourceUUID,
UsageDedupKey: row.usageDedupKey,
},
})
}
sortActivityReportUsageCandidates(candidates)
baseRows := make([]activity.UsageRow, len(candidates))
for i, candidate := range candidates {
row := candidate.row
_, row.OutputTokens, _, _, _ = dailyUsageRowTokens(candidate.scan)
row.WebSearchRequests = usageRowWebSearchRequests(
candidate.scan.usageSource, candidate.scan.tokenJSON)
baseRows[i] = row
}
mask, attribution, webSearchRequests :=
activity.UsageSurvivorSelectionForSessions(
q.RangeStart, q.RangeEnd, q.EffectiveEnd, baseRows, ids,
)
return materializeActivityReportUsageCandidates(
candidates, mask, attribution, webSearchRequests, rateResolver,
)
}
File diff suppressed because it is too large Load Diff
+76
View File
@@ -0,0 +1,76 @@
package db
import (
"fmt"
"strings"
"testing"
"github.com/stretchr/testify/require"
)
var (
benchmarkTrendsTermsResult TrendsTermsResponse
benchmarkSignalsResult SignalsAnalyticsResponse
)
func BenchmarkBunContentAnalyticsStreaming(b *testing.B) {
database := testDB(b)
const (
sessionCount = 128
messagesPerSession = 8
)
started := "2026-08-04T12:00:00Z"
content := strings.Repeat("this is broken again seam ", 256)
messages := make([]Message, 0, sessionCount*messagesPerSession)
for sessionIndex := range sessionCount {
id := fmt.Sprintf("content-bench-%03d", sessionIndex)
require.NoError(b, database.UpsertSession(Session{
ID: id, Project: "content-bench", Machine: "host", Agent: "codex",
CreatedAt: started, StartedAt: &started,
MessageCount: messagesPerSession, UserMessageCount: messagesPerSession,
}))
for ordinal := range messagesPerSession {
messages = append(messages, Message{
SessionID: id, Ordinal: ordinal, Role: "user", Content: content,
ContentLength: len(content), Timestamp: started,
})
}
}
require.NoError(b, database.InsertMessages(messages))
_, err := database.getWriter().Exec(
"UPDATE sessions SET quality_signal_version = ?",
CurrentQualitySignalVersion,
)
require.NoError(b, err)
terms, err := ParseTrendTerms([]string{"seam"})
require.NoError(b, err)
filter := AnalyticsFilter{
Project: "content-bench", From: "2026-08-04", To: "2026-08-04",
Timezone: "UTC",
}
bytesPerOperation := int64(len(content) * len(messages))
b.Run("trends", func(b *testing.B) {
b.ReportAllocs()
b.SetBytes(bytesPerOperation)
for range b.N {
result, err := database.GetTrendsTerms(b.Context(), filter, terms, "day")
if err != nil {
b.Fatal(err)
}
benchmarkTrendsTermsResult = result
}
})
b.Run("signals", func(b *testing.B) {
b.ReportAllocs()
b.SetBytes(bytesPerOperation)
for range b.N {
result, err := database.GetAnalyticsSignals(b.Context(), filter)
if err != nil {
b.Fatal(err)
}
benchmarkSignalsResult = result
}
})
}
@@ -0,0 +1,292 @@
package db
import (
"context"
"database/sql"
"fmt"
"strings"
"testing"
"time"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
"github.com/uptrace/bun"
)
type replayingAnalyticsBackend struct {
first, second bun.IDB
attempts int
}
func (*replayingAnalyticsBackend) Name() string { return "replaying-analytics" }
func (*replayingAnalyticsBackend) ReadOnly() bool { return true }
func (*replayingAnalyticsBackend) Capabilities() BackendCapabilities {
return BackendCapabilities{}
}
func (*replayingAnalyticsBackend) SessionQueryDialect() QueryDialect {
return SQLiteBunSessionQueryDialect()
}
func (*replayingAnalyticsBackend) SessionVersion(
context.Context, bun.IDB, string,
) (int, int64, error) {
return 0, 0, sql.ErrNoRows
}
func (b *replayingAnalyticsBackend) View(
_ context.Context, callback func(bun.IDB) error,
) error {
return callback(b.second)
}
func (b *replayingAnalyticsBackend) ConsistentView(
_ context.Context, callback func(bun.IDB) error,
) error {
b.attempts++
if err := callback(b.first); err != nil {
return err
}
b.attempts++
return callback(b.second)
}
func (*replayingAnalyticsBackend) Update(
context.Context, func(bun.IDB) error,
) error {
return ErrReadOnly
}
func TestBunAnalyticsTerminationUsesActivityWindows(t *testing.T) {
database := testDB(t)
now := time.Now().UTC()
for _, row := range []struct {
id, status string
age time.Duration
}{
{id: "analytics-active-clean", status: "clean", age: 5 * time.Minute},
{id: "analytics-active-flagged", status: "truncated", age: 5 * time.Minute},
{id: "analytics-stale", status: "tool_call_pending", age: 30 * time.Minute},
{id: "analytics-unclean", status: "truncated", age: 2 * time.Hour},
} {
ended := now.Add(-row.age).Format(time.RFC3339Nano)
status := row.status
require.NoError(t, database.UpsertSession(Session{
ID: row.id, Project: "termination", Machine: "host", Agent: "codex",
CreatedAt: ended, StartedAt: &ended, EndedAt: &ended,
MessageCount: 1, UserMessageCount: 1, TerminationStatus: &status,
}))
require.NoError(t, database.InsertMessages([]Message{{
SessionID: row.id, Ordinal: 0, Role: "assistant",
Content: "done", ContentLength: 4, Timestamp: ended,
}}))
}
for _, test := range []struct {
filter string
want int
}{
{filter: "active", want: 2},
{filter: "stale", want: 1},
{filter: "unclean", want: 1},
} {
t.Run(test.filter, func(t *testing.T) {
summary, err := database.GetAnalyticsSummary(t.Context(), AnalyticsFilter{
Project: "termination", Termination: test.filter,
})
require.NoError(t, err)
assert.Equal(t, test.want, summary.TotalSessions)
})
}
}
func TestBunAnalyticsToolsFallsBackToSessionTime(t *testing.T) {
database := testDB(t)
started := "2026-08-04T10:00:00Z"
file := "fallback.go"
require.NoError(t, database.UpsertSession(Session{
ID: "analytics-tool-fallback", Project: "fallback", Machine: "host",
Agent: "codex", CreatedAt: started, StartedAt: &started,
MessageCount: 1, UserMessageCount: 1,
}))
require.NoError(t, database.InsertMessages([]Message{{
SessionID: "analytics-tool-fallback", Ordinal: 0, Role: "assistant",
Content: "edit", ContentLength: 4, HasToolUse: true,
ToolCalls: []ToolCall{{
ToolName: "Edit", Category: "Edit", FilePath: file,
}},
}}))
hour := 10
result, err := database.GetAnalyticsTools(t.Context(), AnalyticsFilter{
Project: "fallback", Timezone: "UTC", Hour: &hour,
})
require.NoError(t, err)
assert.Equal(t, 1, result.TotalCalls)
}
func TestBunAnalyticsSummaryReportsOnlyPresentRequestedModels(t *testing.T) {
database := testDB(t)
started := "2026-08-04T11:00:00Z"
require.NoError(t, database.UpsertSession(Session{
ID: "analytics-model-list", Project: "models", Machine: "host",
Agent: "codex", CreatedAt: started, StartedAt: &started,
MessageCount: 1, UserMessageCount: 1,
}))
require.NoError(t, database.InsertMessages([]Message{{
SessionID: "analytics-model-list", Ordinal: 0, Role: "assistant",
Content: "model", ContentLength: 5, Timestamp: started, Model: "model-a",
}}))
result, err := database.GetAnalyticsSummary(t.Context(), AnalyticsFilter{
Project: "models", Model: "model-a,missing-model",
})
require.NoError(t, err)
assert.Equal(t, []string{"model-a"}, result.Models)
}
func TestBunAnalyticsProjectsPublishesOnlyAcceptedReplay(t *testing.T) {
first := testDB(t)
second := testDB(t)
for database, project := range map[*DB]string{first: "rejected", second: "accepted"} {
started := "2026-08-04T12:00:00Z"
id := "analytics-" + project
require.NoError(t, database.UpsertSession(Session{
ID: id, Project: project, Machine: "host", Agent: "codex",
CreatedAt: started, StartedAt: &started,
MessageCount: 1, UserMessageCount: 1,
}))
require.NoError(t, database.InsertMessages([]Message{{
SessionID: id, Ordinal: 0, Role: "assistant",
Content: project, ContentLength: len(project), Timestamp: started,
}}))
}
backend := &replayingAnalyticsBackend{
first: first.bunReader, second: second.bunReader,
}
result, err := NewBunStore(backend).GetAnalyticsProjects(
t.Context(), AnalyticsFilter{},
)
require.NoError(t, err)
assert.Equal(t, 2, backend.attempts)
require.Len(t, result.Projects, 1)
assert.Equal(t, "accepted", result.Projects[0].Name)
}
func TestBunRecentEditsHydratesOnlyRequestedGroups(t *testing.T) {
database := testDB(t)
started := "2026-08-04T13:00:00Z"
require.NoError(t, database.UpsertSession(Session{
ID: "bounded-edits", Project: "edits", Machine: "host", Agent: "codex",
CreatedAt: started, StartedAt: &started,
MessageCount: 1, UserMessageCount: 1,
}))
calls := make([]ToolCall, 30)
for index := range calls {
file := fmt.Sprintf("file-%02d.go", index)
calls[index] = ToolCall{
CallIndex: index, ToolName: "Edit", Category: "Edit", FilePath: file,
}
}
require.NoError(t, database.InsertMessages([]Message{{
SessionID: "bounded-edits", Ordinal: 0, Role: "assistant",
Content: "edits", ContentLength: 5, Timestamp: started,
HasToolUse: true, ToolCalls: calls,
}}))
hook := new(countingQueryHook)
store := NewBunStore(&sessionContractBackend{
store: database.bunReader.WithQueryHook(hook),
})
result, err := store.RecentEdits(t.Context(), RecentEditsParams{
Project: "edits", Limit: 1, MaxEditsPerFile: 2,
})
require.NoError(t, err)
require.Len(t, result.Files, 1)
assert.True(t, result.HasMore)
assert.Equal(t, 1, hook.selects)
require.Len(t, hook.queries, 1)
assert.NotContains(t, hook.queries[0], "input_json")
assert.NotContains(t, hook.queries[0], "result_content")
assert.NotContains(t, hook.queries[0], "messages.content")
}
func TestBunContentAnalyticsStreamsAcrossSessionBatches(t *testing.T) {
database := testDB(t)
started := "2026-08-04T12:00:00Z"
messages := make([]Message, 0, 33)
for index := range 33 {
id := fmt.Sprintf("content-stream-%02d", index)
require.NoError(t, database.UpsertSession(Session{
ID: id, Project: "content-stream", Machine: "host", Agent: "codex",
CreatedAt: started, StartedAt: &started,
MessageCount: 1, UserMessageCount: 1,
}))
content := "this is broken again seam"
messages = append(messages, Message{
SessionID: id, Ordinal: 0, Role: "user", Content: content,
ContentLength: len(content), Timestamp: started,
})
}
require.NoError(t, database.InsertMessages(messages))
_, err := database.getWriter().Exec(
"UPDATE sessions SET quality_signal_version = ?",
CurrentQualitySignalVersion,
)
require.NoError(t, err)
hook := new(countingQueryHook)
store := NewBunStore(&sessionContractBackend{
store: database.bunReader.WithQueryHook(hook),
})
terms, err := ParseTrendTerms([]string{"seam"})
require.NoError(t, err)
trends, err := store.GetTrendsTerms(t.Context(), AnalyticsFilter{
Project: "content-stream", From: "2026-08-04", To: "2026-08-04",
Timezone: "UTC",
}, terms, "day")
require.NoError(t, err)
assert.Equal(t, 33, trends.MessageCount)
require.Len(t, trends.Series, 1)
assert.Equal(t, 33, trends.Series[0].Total)
trendContentQueries := bunContentSelects(hook.queries)
require.Len(t, trendContentQueries, 2)
for _, query := range trendContentQueries {
assert.NotContains(t, query, "has_thinking")
assert.NotContains(t, query, "has_tool_use")
assert.NotContains(t, query, "content_length")
assert.NotContains(t, query, "output_tokens")
assert.NotContains(t, query, "is_sidechain")
}
hook.queries = nil
signalResult, err := store.GetAnalyticsSignals(t.Context(), AnalyticsFilter{
Project: "content-stream", From: "2026-08-04", To: "2026-08-04",
Timezone: "UTC",
})
require.NoError(t, err)
assert.Equal(t, 33, signalResult.QualityHealth.Totals.FrustrationMarkerCount)
assert.Equal(t, 33,
signalResult.QualityHealth.SessionsWithSignal.FrustrationMarkerCount)
signalContentQueries := bunContentSelects(hook.queries)
require.Len(t, signalContentQueries, 2)
for _, query := range signalContentQueries {
assert.NotContains(t, query, "ordinal")
assert.NotContains(t, query, "model")
assert.NotContains(t, query, "timestamp")
assert.NotContains(t, query, "has_tool_use")
}
}
func bunContentSelects(queries []string) []string {
var contentQueries []string
for _, query := range queries {
normalized := strings.ToLower(query)
if strings.Contains(normalized, `from "messages"`) &&
strings.Contains(normalized, `"content"`) {
contentQueries = append(contentQueries, normalized)
}
}
return contentQueries
}
+222
View File
@@ -0,0 +1,222 @@
package db
import (
"context"
"database/sql"
"fmt"
"github.com/uptrace/bun"
"go.kenn.io/agentsview/internal/db/bunmodel"
)
// BunBackend owns one engine's guarded Bun handle lifecycle. ConsistentView
// callbacks may be replayed by adapters that cannot hold one remote snapshot;
// callbacks must stage results and publish them only after ConsistentView
// returns successfully.
type BunBackend interface {
Name() string
ReadOnly() bool
Capabilities() BackendCapabilities
SessionQueryDialect() QueryDialect
SessionVersion(context.Context, bun.IDB, string) (int, int64, error)
View(context.Context, func(bun.IDB) error) error
ConsistentView(context.Context, func(bun.IDB) error) error
Update(context.Context, func(bun.IDB) error) error
}
type bunSessionFullHydrator interface {
HydrateSessionFull(context.Context, bun.IDB, *Session) error
}
type bunCurationSessionLocker interface {
LockCurationSession(context.Context, bun.IDB, string) error
}
// BunTablePresenceProbe is implemented by adapters whose compatible read-only
// schemas may omit optional canonical tables.
type BunTablePresenceProbe interface {
BunTableExists(context.Context, bun.IDB, string) (bool, error)
}
// WriteOperation identifies a separately authorized family of mutations.
type WriteOperation uint8
const (
WriteArchive WriteOperation = iota
WriteCuration
WriteInsight // insertion/generation
WriteInsightDelete
WriteSessionManagement
WriteRecall
)
// BackendCapabilities describes features that cannot be inferred from a
// store's coarse public ReadOnly value.
type BackendCapabilities struct {
Recall bool
Writes map[WriteOperation]bool
SessionMutations SessionMutationAdapter
}
// SessionMutationAdapter owns engine-specific timestamp expressions and
// transaction-scoped side effects around the canonical session mutations.
type SessionMutationAdapter interface {
ApplyTouch(*bun.UpdateQuery, bunmodel.Timestamp)
ApplySoftDelete(*bun.UpdateQuery, bunmodel.Timestamp)
AfterRestore(context.Context, bun.Tx, string) error
BeforeDelete(context.Context, bun.Tx, []string) error
}
// AllowsWrite reports whether an operation family is authorized.
func (c BackendCapabilities) AllowsWrite(operation WriteOperation) bool {
return c.Writes[operation]
}
type sqliteBunBackend struct {
store *DB
}
var _ BunBackend = (*sqliteBunBackend)(nil)
func (*sqliteBunBackend) Name() string { return "sqlite" }
func (b *sqliteBunBackend) ReadOnly() bool { return b.store.readOnly }
func (b *sqliteBunBackend) Capabilities() BackendCapabilities {
if b.store.readOnly {
return BackendCapabilities{Recall: true}
}
return BackendCapabilities{
Recall: true,
SessionMutations: sqliteSessionMutationAdapter{},
Writes: map[WriteOperation]bool{
WriteArchive: true,
WriteCuration: true,
WriteInsight: true,
WriteInsightDelete: true,
WriteSessionManagement: true,
WriteRecall: true,
},
}
}
func (*sqliteBunBackend) BunTableExists(
ctx context.Context, store bun.IDB, table string,
) (bool, error) {
var exists bool
err := store.NewRaw(`SELECT EXISTS (
SELECT 1 FROM sqlite_master WHERE type = 'table' AND name = ?
)`, table).Scan(ctx, &exists)
return exists, err
}
type sqliteSessionMutationAdapter struct{}
func (sqliteSessionMutationAdapter) ApplyTouch(
query *bun.UpdateQuery, now bunmodel.Timestamp,
) {
query.Set("local_modified_at = ?", now)
}
func (adapter sqliteSessionMutationAdapter) ApplySoftDelete(
query *bun.UpdateQuery, now bunmodel.Timestamp,
) {
query.Set("deleted_at = ?", now)
adapter.ApplyTouch(query, now)
}
func (sqliteSessionMutationAdapter) AfterRestore(
ctx context.Context, tx bun.Tx, id string,
) error {
if _, err := tx.NewDelete().Table("local_session_source_baselines").
Where("session_id = ?", id).Exec(ctx); err != nil {
return fmt.Errorf("clearing restored session %s source baseline: %w", id, err)
}
if _, err := tx.NewDelete().Table("provider_freshness").
Where(`file_path = (
SELECT file_path FROM sessions WHERE id = ?
)`, id).Exec(ctx); err != nil {
return fmt.Errorf("clearing restored session %s provider freshness: %w", id, err)
}
return nil
}
func (sqliteSessionMutationAdapter) BeforeDelete(
_ context.Context, tx bun.Tx, ids []string,
) error {
for _, id := range ids {
if err := deleteSessionMessagesTx(tx.Tx, id); err != nil {
return fmt.Errorf("pre-deleting session %s messages: %w", id, err)
}
}
return nil
}
func (*sqliteBunBackend) SessionQueryDialect() QueryDialect {
return SQLiteBunSessionQueryDialect()
}
func (*sqliteBunBackend) SessionVersion(
ctx context.Context, store bun.IDB, id string,
) (int, int64, error) {
return FileSessionVersion(ctx, store, id)
}
func (*sqliteBunBackend) HydrateSessionFull(
ctx context.Context, store bun.IDB, session *Session,
) error {
var operational struct {
NextOrdinal int `bun:"next_ordinal"`
LastEntryUUID *string `bun:"last_entry_uuid"`
ClaudeLinearParse *bool `bun:"claude_linear_parse"`
LastWriteIncremental bool `bun:"last_write_incremental"`
}
if err := store.NewSelect().Table("sessions").
Column("next_ordinal", "last_entry_uuid", "claude_linear_parse", "last_write_incremental").
Where("id = ?", session.ID).Scan(ctx, &operational); err != nil {
return err
}
session.NextOrdinal = operational.NextOrdinal
session.LastEntryUUID = operational.LastEntryUUID
session.ClaudeLinearParse = operational.ClaudeLinearParse
session.LastWriteIncremental = operational.LastWriteIncremental
return nil
}
func (b *sqliteBunBackend) View(
_ context.Context, fn func(bun.IDB) error,
) error {
b.store.connMu.RLock()
defer b.store.connMu.RUnlock()
return fn(b.store.bunReader)
}
func (b *sqliteBunBackend) ConsistentView(
ctx context.Context, fn func(bun.IDB) error,
) error {
b.store.connMu.RLock()
defer b.store.connMu.RUnlock()
return b.store.bunReader.RunInTx(
ctx, &sql.TxOptions{ReadOnly: true},
func(_ context.Context, tx bun.Tx) error { return fn(tx) },
)
}
func (b *sqliteBunBackend) Update(
_ context.Context, fn func(bun.IDB) error,
) error {
b.store.mu.Lock()
defer b.store.mu.Unlock()
b.store.connMu.RLock()
defer b.store.connMu.RUnlock()
if b.store.readOnly {
return ErrReadOnly
}
if b.store.bunWriter == nil {
if b.store.writerClosed.Load() {
return ErrWriterClosed
}
return ErrReadOnly
}
return fn(b.store.bunWriter)
}
+103
View File
@@ -0,0 +1,103 @@
package db
import (
"context"
"database/sql"
"testing"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
"github.com/uptrace/bun"
)
type recordingBunBackend struct {
readOnly bool
capabilities BackendCapabilities
insideGuard bool
viewCalls int
consistentViewCalls int
updateCalls int
}
func (*recordingBunBackend) Name() string { return "recording" }
func (b *recordingBunBackend) ReadOnly() bool { return b.readOnly }
func (b *recordingBunBackend) Capabilities() BackendCapabilities {
return b.capabilities
}
func (*recordingBunBackend) SessionQueryDialect() QueryDialect {
return PortableBunSessionQueryDialect()
}
func (*recordingBunBackend) SessionVersion(
context.Context, bun.IDB, string,
) (int, int64, error) {
return 0, 0, sql.ErrNoRows
}
func (b *recordingBunBackend) View(
_ context.Context, fn func(bun.IDB) error,
) error {
b.viewCalls++
b.insideGuard = true
defer func() { b.insideGuard = false }()
return fn(nil)
}
func (b *recordingBunBackend) ConsistentView(
_ context.Context, fn func(bun.IDB) error,
) error {
b.consistentViewCalls++
b.insideGuard = true
defer func() { b.insideGuard = false }()
return fn(nil)
}
func (b *recordingBunBackend) Update(
_ context.Context, fn func(bun.IDB) error,
) error {
b.updateCalls++
b.insideGuard = true
defer func() { b.insideGuard = false }()
return fn(nil)
}
func TestBunStoreViewRunsCallbackInsideBackendGuard(t *testing.T) {
backend := &recordingBunBackend{}
store := NewBunStore(backend)
err := store.view(t.Context(), func(bun.IDB) error {
assert.True(t, backend.insideGuard)
return nil
})
require.NoError(t, err)
assert.Equal(t, 1, backend.viewCalls)
assert.False(t, backend.insideGuard)
}
func TestBunStoreUpdateUsesOperationCapability(t *testing.T) {
backend := &recordingBunBackend{
readOnly: true,
capabilities: BackendCapabilities{
Writes: map[WriteOperation]bool{WriteCuration: true},
},
}
store := NewBunStore(backend)
err := store.update(t.Context(), WriteArchive, func(bun.IDB) error {
require.Fail(t, "unauthorized archive callback ran")
return nil
})
assert.ErrorIs(t, err, ErrReadOnly)
assert.Equal(t, 0, backend.updateCalls)
err = store.update(t.Context(), WriteCuration, func(bun.IDB) error {
assert.True(t, backend.insideGuard)
return nil
})
require.NoError(t, err)
assert.Equal(t, 1, backend.updateCalls)
}
+349
View File
@@ -0,0 +1,349 @@
package db
import (
"context"
"database/sql"
"fmt"
"time"
"github.com/uptrace/bun"
"go.kenn.io/agentsview/internal/db/bunmodel"
)
const bunCurationBatchSize = 500
// PinRowIDPolicy declares whether replicated pin identities belong to the
// target curation store or to a source-assigned read mirror.
type PinRowIDPolicy uint8
const (
GeneratePinRowIDs PinRowIDPolicy = iota
PreservePinRowIDs
)
type bunPinnedMessageReadRow struct {
ID int64 `bun:"id"`
SessionID string `bun:"session_id"`
MessageID int64 `bun:"message_id"`
Ordinal int `bun:"ordinal"`
Note *string `bun:"note"`
CreatedAt bunmodel.Timestamp `bun:"created_at"`
Content *string `bun:"content"`
Role *string `bun:"role"`
SessionProject *string `bun:"session_project"`
SessionAgent *string `bun:"session_agent"`
SessionDisplayName *string `bun:"session_display_name"`
SessionFirstMessage *string `bun:"session_first_message"`
}
// StarSession marks an existing session as starred through the common
// operation-scoped curation writer. Repeating the operation is idempotent.
func (s *BunStore) StarSession(sessionID string) (bool, error) {
ctx := context.Background()
createdAt := bunmodel.NewTimestamp(time.Now())
starred := false
err := s.update(ctx, WriteCuration, func(store bun.IDB) error {
return store.RunInTx(ctx, nil, func(ctx context.Context, tx bun.Tx) error {
result, err := tx.NewRaw(`
INSERT INTO starred_sessions (session_id, created_at)
SELECT ?, ? WHERE EXISTS (
SELECT 1 FROM sessions WHERE id = ?
)
ON CONFLICT (session_id) DO NOTHING`,
sessionID, createdAt, sessionID,
).Exec(ctx)
if err != nil {
return fmt.Errorf("starring session %s: %w", sessionID, err)
}
inserted, err := result.RowsAffected()
if err != nil {
return fmt.Errorf("counting starred session %s: %w", sessionID, err)
}
if inserted > 0 {
starred = true
return nil
}
exists, err := tx.NewSelect().Table("sessions").
Where("id = ?", sessionID).Exists(ctx)
if err != nil {
return fmt.Errorf("checking session %s: %w", sessionID, err)
}
starred = exists
return nil
})
})
if err != nil {
return false, err
}
return starred, nil
}
// UnstarSession removes a session star.
func (s *BunStore) UnstarSession(sessionID string) error {
ctx := context.Background()
return s.update(ctx, WriteCuration, func(store bun.IDB) error {
if _, err := store.NewDelete().Model((*bunmodel.StarredSession)(nil)).
Where("session_id = ?", sessionID).Exec(ctx); err != nil {
return fmt.Errorf("unstarring session %s: %w", sessionID, err)
}
return nil
})
}
// ListStarredSessionIDs returns starred session IDs newest first.
func (s *BunStore) ListStarredSessionIDs(
ctx context.Context,
) ([]string, error) {
var rows []struct {
SessionID string `bun:"session_id"`
}
err := s.view(ctx, func(store bun.IDB) error {
return store.NewSelect().Table("starred_sessions").Column("session_id").
OrderExpr("created_at DESC").OrderExpr("session_id DESC").
Scan(ctx, &rows)
})
if err != nil {
return nil, fmt.Errorf("listing starred sessions: %w", err)
}
ids := make([]string, len(rows))
for i, row := range rows {
ids[i] = row.SessionID
}
return ids, nil
}
// BulkStarSessions stars each existing session atomically. Missing IDs are
// ignored so stale client-side curation cannot abort the batch.
func (s *BunStore) BulkStarSessions(sessionIDs []string) error {
ctx := context.Background()
createdAt := bunmodel.NewTimestamp(time.Now())
return s.update(ctx, WriteCuration, func(store bun.IDB) error {
if len(sessionIDs) == 0 {
return nil
}
return store.RunInTx(ctx, nil, func(ctx context.Context, tx bun.Tx) error {
for _, sessionID := range sessionIDs {
if _, err := tx.NewRaw(`
INSERT INTO starred_sessions (session_id, created_at)
SELECT ?, ? WHERE EXISTS (
SELECT 1 FROM sessions WHERE id = ?
)
ON CONFLICT (session_id) DO NOTHING`,
sessionID, createdAt, sessionID,
).Exec(ctx); err != nil {
return fmt.Errorf("starring session %s: %w", sessionID, err)
}
}
return nil
})
})
}
// PinMessage creates or updates a pin after resolving the public message ID to
// the canonical (session_id, ordinal) key. Generated IDs come from RETURNING on
// every writable engine.
func (s *BunStore) PinMessage(
sessionID string, messageID int64, note *string,
) (int64, error) {
ctx := context.Background()
createdAt := bunmodel.NewTimestamp(time.Now())
var pinID int64
err := s.update(ctx, WriteCuration, func(store bun.IDB) error {
return store.RunInTx(ctx, nil, func(ctx context.Context, tx bun.Tx) error {
if err := s.lockCurationSession(ctx, tx, sessionID); err != nil {
return err
}
err := tx.NewRaw(`
INSERT INTO pinned_messages (
session_id, message_id, ordinal, source_uuid, note, created_at
)
SELECT ?, COALESCE(m.id, CAST(m.ordinal AS BIGINT)),
m.ordinal, m.source_uuid, ?, ?
FROM messages AS m
WHERE m.session_id = ?
AND (m.id = ? OR (m.id IS NULL AND m.ordinal = ?))
ON CONFLICT (session_id, ordinal) DO UPDATE SET
message_id = excluded.message_id,
source_uuid = excluded.source_uuid,
note = excluded.note
RETURNING id`,
sessionID, note, createdAt, sessionID, messageID, messageID,
).Scan(ctx, &pinID)
if err == sql.ErrNoRows {
pinID = 0
return nil
}
if err != nil {
return fmt.Errorf("pinning message: %w", err)
}
return nil
})
})
if err != nil {
return 0, err
}
return pinID, nil
}
// UnpinMessage removes the canonical pin addressed by its public message ID.
func (s *BunStore) UnpinMessage(sessionID string, messageID int64) error {
ctx := context.Background()
return s.update(ctx, WriteCuration, func(store bun.IDB) error {
return store.RunInTx(ctx, nil, func(ctx context.Context, tx bun.Tx) error {
if err := s.lockCurationSession(ctx, tx, sessionID); err != nil {
return err
}
if _, err := tx.NewRaw(`
DELETE FROM pinned_messages
WHERE session_id = ?
AND ordinal IN (
SELECT ordinal FROM messages
WHERE session_id = ?
AND COALESCE(id, CAST(ordinal AS BIGINT)) = ?
)`, sessionID, sessionID, messageID).Exec(ctx); err != nil {
return fmt.Errorf("unpinning message: %w", err)
}
return nil
})
})
}
func (s *BunStore) lockCurationSession(
ctx context.Context, store bun.IDB, sessionID string,
) error {
locker, ok := s.backend.(bunCurationSessionLocker)
if !ok {
return nil
}
return locker.LockCurationSession(ctx, store, sessionID)
}
// ListPinnedMessages returns session pins or the visible archive-wide pin page.
func (s *BunStore) ListPinnedMessages(
ctx context.Context, sessionID string, project string,
) ([]PinnedMessage, error) {
var rows []bunPinnedMessageReadRow
err := s.view(ctx, func(store bun.IDB) error {
query := store.NewSelect().TableExpr("pinned_messages AS pin").
ColumnExpr("pin.id AS id").
ColumnExpr("pin.session_id AS session_id").
ColumnExpr("COALESCE(message.id, CAST(pin.ordinal AS BIGINT)) AS message_id").
ColumnExpr("pin.ordinal AS ordinal").
ColumnExpr("pin.note AS note").
ColumnExpr("pin.created_at AS created_at").
Join("LEFT JOIN messages AS message").
JoinOn("message.session_id = pin.session_id").
JoinOn("message.ordinal = pin.ordinal")
if sessionID != "" {
return query.Where("pin.session_id = ?", sessionID).
OrderExpr("pin.created_at DESC").OrderExpr("pin.id DESC").
Scan(ctx, &rows)
}
query = query.
ColumnExpr("message.content AS content").
ColumnExpr("message.role AS role").
ColumnExpr("session.project AS session_project").
ColumnExpr("session.agent AS session_agent").
ColumnExpr("COALESCE(session.display_name, session.session_name) AS session_display_name").
ColumnExpr("session.first_message AS session_first_message").
Join("JOIN sessions AS session").
JoinOn("session.id = pin.session_id").
JoinOn("session.deleted_at IS NULL")
if project != "" {
query = query.Where("session.project = ?", project)
}
return query.OrderExpr("pin.created_at DESC").OrderExpr("pin.id DESC").
Limit(500).Scan(ctx, &rows)
})
if err != nil {
return nil, fmt.Errorf("listing pinned messages: %w", err)
}
var pins []PinnedMessage
for _, row := range rows {
pins = append(pins, PinnedMessage{
ID: row.ID, SessionID: row.SessionID, MessageID: row.MessageID,
Ordinal: row.Ordinal, Note: row.Note,
CreatedAt: formatBunCurationTime(row.CreatedAt.Time),
Content: row.Content,
Role: row.Role,
SessionProject: row.SessionProject,
SessionAgent: row.SessionAgent,
SessionDisplayName: row.SessionDisplayName,
SessionFirstMessage: row.SessionFirstMessage,
})
}
return pins, nil
}
func formatBunCurationTime(value time.Time) string {
if value.IsZero() {
return ""
}
return value.UTC().Format(time.RFC3339Nano)
}
// UpsertStarredSessionRows writes canonical replicated star rows atomically.
func UpsertStarredSessionRows(
ctx context.Context, store bun.IDB, rows []bunmodel.StarredSession,
) error {
if len(rows) == 0 {
return nil
}
return store.RunInTx(ctx, nil, func(ctx context.Context, tx bun.Tx) error {
for start := 0; start < len(rows); start += bunCurationBatchSize {
end := min(start+bunCurationBatchSize, len(rows))
batch := rows[start:end]
if _, err := tx.NewInsert().Model(&batch).
On("CONFLICT (session_id) DO UPDATE").
Set("created_at = EXCLUDED.created_at").Exec(ctx); err != nil {
return fmt.Errorf("upserting starred session rows: %w", err)
}
}
return nil
})
}
// UpsertPinnedMessageRows writes canonical replicated pin rows atomically.
// Generated target IDs stay stable on logical-key conflicts. Mirror rows adopt
// source IDs after removing any stale logical row that still owns a reused ID.
func UpsertPinnedMessageRows(
ctx context.Context, store bun.IDB, rows []bunmodel.PinnedMessage,
idPolicy PinRowIDPolicy,
) error {
if len(rows) == 0 {
return nil
}
return store.RunInTx(ctx, nil, func(ctx context.Context, tx bun.Tx) error {
for _, row := range rows {
query := tx.NewInsert().Model(&row)
switch idPolicy {
case GeneratePinRowIDs:
query = query.ExcludeColumn("id")
case PreservePinRowIDs:
if row.ID == 0 {
return fmt.Errorf("preserving replicated pin id: source id is zero")
}
if _, err := tx.NewDelete().Table("pinned_messages").
Where("id = ?", row.ID).
Where("(session_id != ? OR ordinal != ?)", row.SessionID, row.Ordinal).
Exec(ctx); err != nil {
return fmt.Errorf("reconciling preserved pin id %d: %w", row.ID, err)
}
default:
return fmt.Errorf("upserting pinned message rows: unknown id policy %d", idPolicy)
}
query = query.On("CONFLICT (session_id, ordinal) DO UPDATE")
if idPolicy == PreservePinRowIDs {
query = query.Set("id = EXCLUDED.id")
}
if _, err := query.
Set("message_id = EXCLUDED.message_id").
Set("source_uuid = EXCLUDED.source_uuid").
Set("note = EXCLUDED.note").
Set("created_at = EXCLUDED.created_at").Exec(ctx); err != nil {
return fmt.Errorf("upserting pinned message rows: %w", err)
}
}
return nil
})
}
+373
View File
@@ -0,0 +1,373 @@
package db
import (
"context"
"database/sql"
"testing"
"time"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
"github.com/uptrace/bun"
"github.com/uptrace/bun/dialect/sqlitedialect"
"go.kenn.io/agentsview/internal/db/bunmodel"
)
type writableCurationTestBackend struct {
store bun.IDB
}
func (*writableCurationTestBackend) Name() string { return "curation-test" }
func (*writableCurationTestBackend) ReadOnly() bool { return false }
func (*writableCurationTestBackend) Capabilities() BackendCapabilities {
return BackendCapabilities{Writes: map[WriteOperation]bool{
WriteCuration: true,
WriteInsight: true,
WriteInsightDelete: true,
}}
}
func (*writableCurationTestBackend) SessionQueryDialect() QueryDialect {
return SQLiteBunSessionQueryDialect()
}
func (*writableCurationTestBackend) SessionVersion(
context.Context, bun.IDB, string,
) (int, int64, error) {
return 0, 0, sql.ErrNoRows
}
func (b *writableCurationTestBackend) View(
_ context.Context, fn func(bun.IDB) error,
) error {
return fn(b.store)
}
func (b *writableCurationTestBackend) ConsistentView(
_ context.Context, fn func(bun.IDB) error,
) error {
return fn(b.store)
}
func (b *writableCurationTestBackend) Update(
_ context.Context, fn func(bun.IDB) error,
) error {
return fn(b.store)
}
func TestBunCurationWritesSupplyCanonicalCreationTime(t *testing.T) {
raw, err := sql.Open("sqlite3", ":memory:")
require.NoError(t, err)
raw.SetMaxOpenConns(1)
t.Cleanup(func() { require.NoError(t, raw.Close()) })
store := bun.NewDB(raw, sqlitedialect.New())
require.NoError(t, CreateCommonSchema(t.Context(), store))
_, err = store.NewInsert().Model(&bunmodel.SourceArchive{
SourceArchiveID: "writer-archive", SourceArchiveSalt: "writer-salt",
}).Exec(t.Context())
require.NoError(t, err)
created := bunmodel.NewTimestamp(time.Date(2026, 8, 3, 12, 0, 0, 0, time.UTC))
_, err = store.NewInsert().Model(&bunmodel.Session{
ID: "writer-session", Project: "writer", Machine: "host", Agent: "codex",
CreatedAt: created, SourceArchiveID: "writer-archive",
SourceDatabaseGeneration: "writer-generation",
}).Exec(t.Context())
require.NoError(t, err)
messageID := int64(81)
_, err = store.NewInsert().Model(&bunmodel.Message{
ID: &messageID, SessionID: "writer-session", Ordinal: 1,
Role: "assistant", Content: "canonical writer",
}).Exec(t.Context())
require.NoError(t, err)
common := NewBunStore(&writableCurationTestBackend{store: store})
starred, err := common.StarSession("writer-session")
require.NoError(t, err)
assert.True(t, starred)
pinID, err := common.PinMessage("writer-session", messageID, nil)
require.NoError(t, err)
assert.Positive(t, pinID)
}
func TestBunInsightWriteSuppliesCanonicalCreationTime(t *testing.T) {
raw, err := sql.Open("sqlite3", ":memory:")
require.NoError(t, err)
raw.SetMaxOpenConns(1)
t.Cleanup(func() { require.NoError(t, raw.Close()) })
store := bun.NewDB(raw, sqlitedialect.New())
require.NoError(t, CreateCommonSchema(t.Context(), store))
common := NewBunStore(&writableCurationTestBackend{store: store})
id, err := common.InsertInsight(Insight{
Type: "daily_activity", DateFrom: "2026-08-03", DateTo: "2026-08-03",
Agent: "codex", Content: "canonical insight writer",
})
require.NoError(t, err)
assert.Positive(t, id)
got, err := common.GetInsight(t.Context(), id)
require.NoError(t, err)
require.NotNil(t, got)
assert.NotEmpty(t, got.CreatedAt)
}
func TestBunCachedInsightOrdersMixedSQLiteTimestampsChronologically(t *testing.T) {
raw, err := sql.Open("sqlite3", ":memory:")
require.NoError(t, err)
raw.SetMaxOpenConns(1)
t.Cleanup(func() { require.NoError(t, raw.Close()) })
store := bun.NewDB(raw, sqlitedialect.New())
require.NoError(t, CreateCommonSchema(t.Context(), store))
common := NewBunStore(&writableCurationTestBackend{store: store})
oldTimestamp, err := bunmodel.ParseTimestamp(
time.Now().UTC().Format("2006-01-02") + "T00:00:00Z",
)
require.NoError(t, err)
old := bunmodel.Insight{
Type: "llm_canned", DateFrom: "2026-08-03", DateTo: "2026-08-03",
Agent: "codex", Content: "old", CacheKey: "mixed-timestamp-cache",
CreatedAt: oldTimestamp,
}
_, err = store.NewInsert().Model(&old).Exec(t.Context())
require.NoError(t, err)
newID, err := common.InsertInsight(Insight{
Type: "llm_canned", DateFrom: "2026-08-03", DateTo: "2026-08-03",
Agent: "codex", Content: "new", CacheKey: "mixed-timestamp-cache",
})
require.NoError(t, err)
got, err := common.GetCachedInsight(t.Context(), "mixed-timestamp-cache")
require.NoError(t, err)
require.NotNil(t, got)
assert.Equal(t, newID, got.ID)
assert.Equal(t, "new", got.Content)
}
func TestUpsertPinnedMessageRowsGeneratesTargetIDOnSourceCollision(t *testing.T) {
database := testDB(t)
for _, sessionID := range []string{"target-pin", "replicated-pin", "generated-pin"} {
insertSession(t, database, sessionID, "curation")
insertMessages(t, database, asstMsg(sessionID, 0, sessionID))
}
targetMessages, err := database.GetMessages(
t.Context(), "target-pin", 0, 10, true,
)
require.NoError(t, err)
require.Len(t, targetMessages, 1)
targetPinID, err := database.PinMessage("target-pin", targetMessages[0].ID, nil)
require.NoError(t, err)
assert.Positive(t, targetPinID)
replicatedMessages, err := database.GetMessages(
t.Context(), "replicated-pin", 0, 10, true,
)
require.NoError(t, err)
require.Len(t, replicatedMessages, 1)
replicatedMessageID := replicatedMessages[0].ID
require.NoError(t, UpsertPinnedMessageRows(
t.Context(), database.bunWriter, []bunmodel.PinnedMessage{{
ID: targetPinID, SessionID: "replicated-pin",
MessageID: &replicatedMessageID, Ordinal: replicatedMessages[0].Ordinal,
CreatedAt: bunmodel.NewTimestamp(
time.Date(2026, 8, 3, 13, 0, 0, 0, time.UTC),
),
}}, GeneratePinRowIDs,
))
replicatedPins, err := database.ListPinnedMessages(
t.Context(), "replicated-pin", "",
)
require.NoError(t, err)
require.Len(t, replicatedPins, 1)
assert.NotEqual(t, targetPinID, replicatedPins[0].ID)
generatedMessages, err := database.GetMessages(
t.Context(), "generated-pin", 0, 10, true,
)
require.NoError(t, err)
require.Len(t, generatedMessages, 1)
generatedPinID, err := database.PinMessage(
"generated-pin", generatedMessages[0].ID, nil,
)
require.NoError(t, err)
assert.NotEqual(t, targetPinID, generatedPinID)
assert.NotEqual(t, replicatedPins[0].ID, generatedPinID)
}
func TestBunCurationRowUpsertsRefreshCanonicalKeys(t *testing.T) {
raw, err := sql.Open("sqlite3", ":memory:")
require.NoError(t, err)
raw.SetMaxOpenConns(1)
t.Cleanup(func() { require.NoError(t, raw.Close()) })
store := bun.NewDB(raw, sqlitedialect.New())
require.NoError(t, CreateCommonSchema(t.Context(), store))
first := bunmodel.NewTimestamp(time.Date(2026, 8, 3, 12, 0, 0, 0, time.UTC))
second := bunmodel.NewTimestamp(time.Date(2026, 8, 3, 13, 0, 0, 0, time.UTC))
archive := bunmodel.SourceArchive{
SourceArchiveID: "curation-upsert-archive", SourceArchiveSalt: "salt",
}
_, err = store.NewInsert().Model(&archive).Exec(t.Context())
require.NoError(t, err)
const rootOldID = "curation-upsert-old"
const rootNewID = "curation-upsert-new"
for _, sessionID := range []string{rootOldID, rootNewID} {
session := bunmodel.Session{
ID: sessionID, Project: "curation", Machine: "host", Agent: "codex",
CreatedAt: first, SourceArchiveID: archive.SourceArchiveID,
SourceDatabaseGeneration: "curation-upsert-generation",
}
_, err = store.NewInsert().Model(&session).Exec(t.Context())
require.NoError(t, err)
}
messageID := int64(802)
message := bunmodel.Message{
ID: &messageID, SessionID: rootNewID, Ordinal: 1,
Role: "assistant", Content: "curation", Timestamp: &first,
}
_, err = store.NewInsert().Model(&message).Exec(t.Context())
require.NoError(t, err)
require.NoError(t, UpsertStarredSessionRows(t.Context(), store,
[]bunmodel.StarredSession{
{SessionID: rootOldID, CreatedAt: first},
{SessionID: rootNewID, CreatedAt: first},
},
))
require.NoError(t, UpsertStarredSessionRows(t.Context(), store,
[]bunmodel.StarredSession{{SessionID: rootNewID, CreatedAt: second}},
))
var stars []bunmodel.StarredSession
require.NoError(t, store.NewSelect().Model(&stars).
OrderExpr("session_id ASC").Scan(t.Context()))
require.Len(t, stars, 2)
assert.Equal(t, rootNewID, stars[0].SessionID)
assert.Equal(t, second.Time, stars[0].CreatedAt.Time)
initialNote := "initial replicated pin"
require.NoError(t, UpsertPinnedMessageRows(t.Context(), store,
[]bunmodel.PinnedMessage{{
ID: 3001, SessionID: rootNewID, MessageID: &messageID,
Ordinal: 1, SourceUUID: "source-one", Note: &initialNote,
CreatedAt: first,
}}, GeneratePinRowIDs,
))
var initialPin bunmodel.PinnedMessage
require.NoError(t, store.NewSelect().Model(&initialPin).
Where("session_id = ?", rootNewID).
Where("ordinal = 1").Scan(t.Context()))
assert.Positive(t, initialPin.ID)
assert.NotEqual(t, int64(3001), initialPin.ID,
"the target generates new replicated pin identities")
updatedNote := "updated replicated pin"
require.NoError(t, UpsertPinnedMessageRows(t.Context(), store,
[]bunmodel.PinnedMessage{{
ID: 9999, SessionID: rootNewID, MessageID: &messageID,
Ordinal: 1, SourceUUID: "source-two", Note: &updatedNote,
CreatedAt: second,
}}, GeneratePinRowIDs,
))
var pin bunmodel.PinnedMessage
require.NoError(t, store.NewSelect().Model(&pin).
Where("session_id = ?", rootNewID).
Where("ordinal = 1").Scan(t.Context()))
assert.Equal(t, initialPin.ID, pin.ID,
"the target's generated pin identity stays stable on conflict")
assert.Equal(t, "source-two", pin.SourceUUID)
require.NotNil(t, pin.Note)
assert.Equal(t, updatedNote, *pin.Note)
assert.Equal(t, second.Time, pin.CreatedAt.Time)
}
func TestUpsertPinnedMessageRowsPreservesSourceIDForMirrorPolicy(t *testing.T) {
raw, err := sql.Open("sqlite3", ":memory:")
require.NoError(t, err)
raw.SetMaxOpenConns(1)
t.Cleanup(func() { require.NoError(t, raw.Close()) })
store := bun.NewDB(raw, sqlitedialect.New())
require.NoError(t, CreateCommonSchema(t.Context(), store))
_, err = store.NewInsert().Model(&bunmodel.SourceArchive{
SourceArchiveID: "mirror-archive", SourceArchiveSalt: "mirror-salt",
}).Exec(t.Context())
require.NoError(t, err)
created := bunmodel.NewTimestamp(time.Date(2026, 8, 3, 12, 0, 0, 0, time.UTC))
for _, sessionID := range []string{"mirror-session", "stale-session"} {
_, err = store.NewInsert().Model(&bunmodel.Session{
ID: sessionID, Project: "mirror", Machine: "host", Agent: "codex",
CreatedAt: created, SourceArchiveID: "mirror-archive",
SourceDatabaseGeneration: "mirror-generation",
}).Exec(t.Context())
require.NoError(t, err)
}
messageID := int64(501)
staleMessageID := int64(502)
for _, message := range []bunmodel.Message{
{ID: &messageID, SessionID: "mirror-session", Ordinal: 1,
Role: "assistant", Content: "mirror pin"},
{ID: &staleMessageID, SessionID: "stale-session", Ordinal: 1,
Role: "assistant", Content: "stale pin"},
} {
_, err = store.NewInsert().Model(&message).Exec(t.Context())
require.NoError(t, err)
}
require.NoError(t, UpsertPinnedMessageRows(
t.Context(), store, []bunmodel.PinnedMessage{{
ID: 7001, SessionID: "mirror-session", MessageID: &messageID,
Ordinal: 1, CreatedAt: created,
}}, PreservePinRowIDs,
))
require.NoError(t, UpsertPinnedMessageRows(
t.Context(), store, []bunmodel.PinnedMessage{{
ID: 7002, SessionID: "stale-session", MessageID: &staleMessageID,
Ordinal: 1, CreatedAt: created,
}}, PreservePinRowIDs,
))
require.NoError(t, UpsertPinnedMessageRows(
t.Context(), store, []bunmodel.PinnedMessage{{
ID: 7002, SessionID: "mirror-session", MessageID: &messageID,
Ordinal: 1, CreatedAt: created,
}}, PreservePinRowIDs,
))
var pin bunmodel.PinnedMessage
require.NoError(t, store.NewSelect().Model(&pin).
Where("session_id = ?", "mirror-session").Scan(t.Context()))
assert.Equal(t, int64(7002), pin.ID,
"a replacement mirror row adopts the current source identity")
staleCount, err := store.NewSelect().Table("pinned_messages").
Where("session_id = ?", "stale-session").Count(t.Context())
require.NoError(t, err)
assert.Zero(t, staleCount,
"the stale mirror row no longer owns the reused source identity")
}
func TestUpsertPinnedMessageRowsRollsBackGeneratedBatch(t *testing.T) {
database := testDB(t)
insertSession(t, database, "rollback-pin", "curation")
insertMessages(t, database, asstMsg("rollback-pin", 0, "valid pin"))
messages, err := database.GetMessages(
t.Context(), "rollback-pin", 0, 10, true,
)
require.NoError(t, err)
require.Len(t, messages, 1)
validMessageID := messages[0].ID
missingMessageID := int64(999_999)
created := bunmodel.NewTimestamp(time.Date(2026, 8, 3, 14, 0, 0, 0, time.UTC))
err = UpsertPinnedMessageRows(
t.Context(), database.bunWriter, []bunmodel.PinnedMessage{
{
ID: 8001, SessionID: "rollback-pin", MessageID: &validMessageID,
Ordinal: 0, CreatedAt: created,
},
{
ID: 8002, SessionID: "missing-session", MessageID: &missingMessageID,
Ordinal: 0, CreatedAt: created,
},
}, GeneratePinRowIDs,
)
require.Error(t, err)
pins, readErr := database.ListPinnedMessages(t.Context(), "rollback-pin", "")
require.NoError(t, readErr)
assert.Empty(t, pins)
}
+794
View File
@@ -0,0 +1,794 @@
package db
import (
"context"
"fmt"
"slices"
"sort"
"strings"
"time"
"github.com/uptrace/bun"
"go.kenn.io/agentsview/internal/db/bunmodel"
"go.kenn.io/agentsview/internal/export"
)
const bunDataListBatchSize = 500
// ListProjectIdentityObservations returns canonical source-scoped identity
// observations ordered by archive and raw project identity.
func (s *BunStore) ListProjectIdentityObservations(
ctx context.Context,
labels []string,
) ([]export.ProjectIdentityObservation, error) {
if ctx == nil {
ctx = context.Background()
}
var observations []export.ProjectIdentityObservation
err := s.view(ctx, func(store bun.IDB) error {
var err error
observations, err = listBunProjectIdentityObservations(ctx, store, labels)
return err
})
if err != nil {
return nil, err
}
return observations, nil
}
func listBunProjectIdentityObservations(
ctx context.Context,
store bun.IDB,
labels []string,
) ([]export.ProjectIdentityObservation, error) {
if labels != nil && len(labels) == 0 {
return []export.ProjectIdentityObservation{}, nil
}
if labels == nil {
return listBunProjectIdentityObservationChunk(ctx, store, nil)
}
sortedLabels := slices.Clone(labels)
slices.Sort(sortedLabels)
sortedLabels = slices.Compact(sortedLabels)
observations := make([]export.ProjectIdentityObservation, 0)
for start := 0; start < len(sortedLabels); start += bunDataListBatchSize {
end := min(start+bunDataListBatchSize, len(sortedLabels))
rows, err := listBunProjectIdentityObservationChunk(
ctx, store, sortedLabels[start:end],
)
if err != nil {
return nil, err
}
observations = append(observations, rows...)
}
sortBunProjectIdentityObservations(observations)
return observations, nil
}
func listBunProjectIdentityObservationChunk(
ctx context.Context,
store bun.IDB,
labels []string,
) ([]export.ProjectIdentityObservation, error) {
var rows []bunmodel.SourceProjectIdentityObservation
query := store.NewSelect().Model(&rows)
if len(labels) > 0 {
query = query.Where("project IN (?)", bun.List(labels))
}
if err := query.OrderExpr("source_archive_id ASC").
OrderExpr("project ASC").OrderExpr("machine ASC").
OrderExpr("root_path ASC").OrderExpr("git_remote ASC").
Scan(ctx); err != nil {
return nil, fmt.Errorf("listing project identity observations: %w", err)
}
observations := make([]export.ProjectIdentityObservation, 0, len(rows))
for _, row := range rows {
observations = append(observations, projectIdentityObservationFromBunRow(row))
}
return observations, nil
}
func projectIdentityObservationFromBunRow(
row bunmodel.SourceProjectIdentityObservation,
) export.ProjectIdentityObservation {
return export.ProjectIdentityObservation{
SourceArchiveID: row.SourceArchiveID, SourceArchiveSalt: row.SourceArchiveSalt,
Project: row.Project, Machine: row.Machine, RootPath: row.RootPath,
GitRemote: row.GitRemote, GitRemoteName: row.GitRemoteName,
RepositoryPath: row.RepositoryPath, WorktreeName: row.WorktreeName,
WorktreeRootPath: row.WorktreeRootPath,
WorktreeRelationship: export.WorktreeRelationship(row.WorktreeRelationship),
CheckoutState: export.CheckoutState(row.CheckoutState), GitBranch: row.GitBranch,
RemoteResolution: export.ProjectResolution(row.RemoteResolution),
RemoteCandidateCount: row.RemoteCandidateCount,
ObservedAt: row.ObservedAt.UTC(), NormalizedRemote: row.NormalizedRemote,
KeySource: row.KeySource, Key: row.Key,
}
}
func sortBunProjectIdentityObservations(
observations []export.ProjectIdentityObservation,
) {
sort.SliceStable(observations, func(i, j int) bool {
left, right := observations[i], observations[j]
if left.SourceArchiveID != right.SourceArchiveID {
return left.SourceArchiveID < right.SourceArchiveID
}
if left.Project != right.Project {
return left.Project < right.Project
}
if left.Machine != right.Machine {
return left.Machine < right.Machine
}
if left.RootPath != right.RootPath {
return left.RootPath < right.RootPath
}
return left.GitRemote < right.GitRemote
})
}
// BuildProjectIdentityMap resolves raw labels from canonical source-scoped
// observations and the complete set of contributing archive identities.
func (s *BunStore) BuildProjectIdentityMap(
ctx context.Context,
labels []string,
) (map[string]export.ProjectMapEntry, error) {
if labels != nil && len(labels) == 0 {
return map[string]export.ProjectMapEntry{}, nil
}
if ctx == nil {
ctx = context.Background()
}
var projects map[string]export.ProjectMapEntry
err := s.view(ctx, func(store bun.IDB) error {
var err error
projects, err = buildBunProjectIdentityMapFrom(ctx, store, labels)
return err
})
if err != nil {
return nil, err
}
return projects, nil
}
func buildBunProjectIdentityMapFrom(
ctx context.Context, store bun.IDB, labels []string,
) (map[string]export.ProjectMapEntry, error) {
if labels != nil && len(labels) == 0 {
return map[string]export.ProjectMapEntry{}, nil
}
observations, err := listBunProjectIdentityObservations(ctx, store, labels)
if err != nil {
return nil, err
}
scope, err := bunSourceArchiveIdentityScope(ctx, store, observations)
if err != nil {
return nil, err
}
return export.BuildProjectsMapWithScope(labels, observations, scope), nil
}
func bunSourceArchiveIdentityScope(
ctx context.Context,
store bun.IDB,
observations []export.ProjectIdentityObservation,
) (export.IdentityScope, error) {
var rows []bunmodel.SourceArchive
if err := store.NewSelect().Model(&rows).
OrderExpr("source_archive_id ASC").Scan(ctx); err != nil {
return export.IdentityScope{}, fmt.Errorf("listing source archives: %w", err)
}
scopes := make([]export.IdentityScope, 0, len(rows))
for _, row := range rows {
scopes = append(scopes, export.IdentityScope{
ArchiveID: row.SourceArchiveID, ArchiveSalt: row.SourceArchiveSalt,
})
}
switch len(scopes) {
case 0:
return bunObservationIdentityScope(observations), nil
case 1:
return scopes[0], nil
default:
return export.AggregateIdentityScope(scopes), nil
}
}
func bunObservationIdentityScope(
observations []export.ProjectIdentityObservation,
) export.IdentityScope {
unique := make(map[string]export.IdentityScope)
for _, observation := range observations {
scope := export.IdentityScope{
ArchiveID: strings.TrimSpace(observation.SourceArchiveID),
ArchiveSalt: strings.TrimSpace(observation.SourceArchiveSalt),
}
if scope.ArchiveID == "" || scope.ArchiveSalt == "" {
continue
}
unique[scope.ArchiveID+"\x00"+scope.ArchiveSalt] = scope
}
if len(unique) == 0 {
return export.LegacySharedStoreIdentityScope()
}
scopes := make([]export.IdentityScope, 0, len(unique))
for _, scope := range unique {
scopes = append(scopes, scope)
}
if len(scopes) == 1 {
return scopes[0]
}
return export.AggregateIdentityScope(scopes)
}
type bunProjectInventoryAggregateRow struct {
Project string `bun:"project"`
Sessions int `bun:"sessions"`
Machines int `bun:"machines"`
Agents int `bun:"agents"`
DistinctCwds int `bun:"distinct_cwds"`
FirstActivity *bunmodel.Timestamp `bun:"first_activity"`
LastActivity *bunmodel.Timestamp `bun:"last_activity"`
}
type bunGovernanceSessionRow struct {
ID string `bun:"id"`
Machine string `bun:"machine"`
Project string `bun:"project"`
Cwd string `bun:"cwd"`
FilePath string `bun:"file_path"`
SourceArchiveID string `bun:"source_archive_id"`
}
type bunCandidateSessionRow struct {
ID string `bun:"id"`
Project string `bun:"project"`
Machine string `bun:"machine"`
Cwd string `bun:"cwd"`
SourceArchiveID string `bun:"source_archive_id"`
SourceDatabaseGeneration string `bun:"source_database_generation"`
}
type bunProjectMappingRow struct {
archiveID string
mapping WorktreeProjectMapping
}
// GetProjectInventory aggregates the canonical visible session and mapping
// rows through the same Bun handle on every backend.
func (s *BunStore) GetProjectInventory(ctx context.Context) (ProjectInventory, error) {
if ctx == nil {
ctx = context.Background()
}
var inventory ProjectInventory
err := s.consistentView(ctx, func(store bun.IDB) error {
agg, err := listBunProjectInventoryAggregates(
ctx, store, s.backend.SessionQueryDialect(),
)
if err != nil {
return err
}
rawProjects := make([]string, 0, len(agg))
for project := range agg {
rawProjects = append(rawProjects, project)
}
projects, _, err := buildBunProjectIdentityMap(ctx, store, rawProjects)
if err != nil {
return err
}
visibleArchives, err := listBunVisibleArchiveIDs(ctx, store)
if err != nil {
return err
}
mappingRows, err := listBunProjectMappings(
ctx, store, nil, visibleArchives,
)
if err != nil {
return err
}
governanceRows, err := listBunGovernanceSessions(ctx, store, nil)
if err != nil {
return err
}
archiveMappings := bunArchiveMappings(mappingRows)
eval := EvaluateGovernedSessions(
archiveMappings,
bunMappingEvaluationRows(governanceRows),
)
mappings := make([]WorktreeProjectMapping, len(mappingRows))
for i, row := range mappingRows {
mappings[i] = row.mapping
}
rows, totalSessions := buildProjectInventoryRows(agg, rawProjects, projects)
annotateProjectInventoryRows(rows, mappings, eval, projects)
inventory = ProjectInventory{
Projects: rows, TotalProjects: len(rows), TotalSessions: totalSessions,
GovernedSessions: eval.GovernedSessions,
}
return nil
})
if err != nil {
return ProjectInventory{}, err
}
return inventory, nil
}
// ListProjectRules returns every canonical rule for one machine and evaluates
// it only against sessions from the same source archive.
func (s *BunStore) ListProjectRules(
ctx context.Context, machine string,
) (ProjectRules, error) {
if ctx == nil {
ctx = context.Background()
}
machine = strings.TrimSpace(machine)
var result ProjectRules
err := s.consistentView(ctx, func(store bun.IDB) error {
machines, err := listBunProjectRuleMachines(ctx, store)
if err != nil {
return err
}
mappingRows, err := listBunProjectMappings(ctx, store, &machine, nil)
if err != nil {
return err
}
governanceRows, err := listBunGovernanceSessions(ctx, store, &machine)
if err != nil {
return err
}
eval := EvaluateGovernedSessions(
bunArchiveMappings(mappingRows),
bunMappingEvaluationRows(governanceRows),
)
rules := make([]ProjectRule, len(mappingRows))
for i, row := range mappingRows {
rules[i] = ProjectRule{
WorktreeProjectMapping: row.mapping,
SourceArchiveID: row.archiveID,
GovernedSessions: eval.SessionsByRule[GovernedRuleKey{
SourceArchiveID: row.archiveID,
Machine: row.mapping.Machine,
PathPrefix: row.mapping.PathPrefix,
}],
}
}
result = ProjectRules{
Machine: machine, Machines: machines, Rules: rules,
}
return nil
})
if err != nil {
return ProjectRules{}, err
}
return result, nil
}
// ListArchiveWorktreeCandidates builds canonical project candidates from
// visible sessions, exact-generation snapshots, and identity observations.
func (s *BunStore) ListArchiveWorktreeCandidates(
ctx context.Context,
request ArchiveWorktreeCandidateRequest,
) ([]WorktreeReclassificationCandidate, error) {
if strings.TrimSpace(request.ProjectKey) == "" {
return nil, fmt.Errorf("project_key is required")
}
if ctx == nil {
ctx = context.Background()
}
var candidates []WorktreeReclassificationCandidate
err := s.consistentView(ctx, func(store bun.IDB) error {
labels, err := listBunCandidateProjectLabels(ctx, store)
if err != nil {
return err
}
projects, _, err := buildBunProjectIdentityMap(
ctx, store, sortedSetKeys(labels),
)
if err != nil {
return err
}
selectedProjects := SelectWorktreeCandidateProjects(request, labels, projects)
if len(selectedProjects) == 0 {
candidates = []WorktreeReclassificationCandidate{}
return nil
}
selected, err := listBunCandidateSessions(
ctx, store, sortedSetKeys(selectedProjects),
)
if err != nil {
return err
}
selectedIDs := make([]string, len(selected))
for i, session := range selected {
selectedIDs[i] = session.ID
}
snapshots, err := listBunProjectIdentitySnapshots(ctx, store, selectedIDs)
if err != nil {
return err
}
observations, err := listBunProjectIdentityObservations(
ctx, store, sortedSetKeys(selectedProjects),
)
if err != nil {
return err
}
details := make([]WorktreeCandidateSession, 0, len(selected))
for _, session := range selected {
detail := WorktreeCandidateSession{
ID: session.ID, Project: session.Project,
Machine: session.Machine, Cwd: session.Cwd,
}
if snapshot, ok := snapshots[bunSnapshotKey{
archiveID: session.SourceArchiveID,
generation: session.SourceDatabaseGeneration,
sessionID: session.ID,
}]; ok {
detail.Snapshot = projectIdentitySnapshotFromBunRow(snapshot)
detail.HasSnapshot = true
}
details = append(details, detail)
}
candidates = BuildWorktreeCandidates(details, observations)
return nil
})
if err != nil {
return nil, err
}
return candidates, nil
}
func listBunProjectInventoryAggregates(
ctx context.Context, store bun.IDB, dialect QueryDialect,
) (map[string]projectInventoryAgg, error) {
startedOrder := dialect.timestampExpr("started_at")
activityOrder := "COALESCE(" + dialect.timestampExpr("ended_at") + ", " +
dialect.timestampExpr("started_at") + ")"
startedRaw := bunNullableTimestamp("started_at")
activityRaw := "COALESCE(" + bunNullableTimestamp("ended_at") + ", " +
bunNullableTimestamp("started_at") + ")"
query := fmt.Sprintf(`
WITH visible AS (
SELECT id, project, machine, agent,
CASE WHEN cwd IS NOT NULL AND cwd != ''
THEN replace(cwd, '\', '/') END AS normalized_cwd,
%s AS started_order, %s AS started_raw,
%s AS activity_order, %s AS activity_raw
FROM sessions
WHERE deleted_at IS NULL
), ranked AS (
SELECT *,
ROW_NUMBER() OVER (
PARTITION BY project
ORDER BY (started_order IS NULL) ASC,
started_order ASC, id ASC
) AS first_rank,
ROW_NUMBER() OVER (
PARTITION BY project
ORDER BY (activity_order IS NULL) ASC,
activity_order DESC, id DESC
) AS last_rank
FROM visible
)
SELECT project, COUNT(*) AS sessions,
COUNT(DISTINCT machine) AS machines,
COUNT(DISTINCT agent) AS agents,
COUNT(DISTINCT normalized_cwd) AS distinct_cwds,
MAX(CASE WHEN first_rank = 1 THEN started_raw END) AS first_activity,
MAX(CASE WHEN last_rank = 1 THEN activity_raw END) AS last_activity
FROM ranked
GROUP BY project
ORDER BY project`, startedOrder, startedRaw, activityOrder, activityRaw)
var rows []bunProjectInventoryAggregateRow
if err := store.NewRaw(query).Scan(ctx, &rows); err != nil {
return nil, fmt.Errorf("aggregating canonical project inventory: %w", err)
}
result := make(map[string]projectInventoryAgg, len(rows))
for _, row := range rows {
agg := projectInventoryAgg{
sessions: row.Sessions, machines: row.Machines,
agents: row.Agents, distinctCwds: row.DistinctCwds,
}
if row.FirstActivity != nil && !row.FirstActivity.IsZero() {
value := row.FirstActivity.UTC()
agg.first = &value
}
if row.LastActivity != nil && !row.LastActivity.IsZero() {
value := row.LastActivity.UTC()
agg.last = &value
}
result[row.Project] = agg
}
return result, nil
}
func listBunVisibleArchiveIDs(
ctx context.Context, store bun.IDB,
) (map[string]struct{}, error) {
var rows []struct {
SourceArchiveID string `bun:"source_archive_id"`
}
if err := store.NewSelect().Table("sessions").Column("source_archive_id").
Where("deleted_at IS NULL").Where("source_archive_id != ''").
Group("source_archive_id").OrderExpr("source_archive_id ASC").
Scan(ctx, &rows); err != nil {
return nil, fmt.Errorf("listing canonical visible source archives: %w", err)
}
result := make(map[string]struct{}, len(rows))
for _, row := range rows {
result[row.SourceArchiveID] = struct{}{}
}
return result, nil
}
func listBunProjectMappings(
ctx context.Context,
store bun.IDB,
machine *string,
visibleArchives map[string]struct{},
) ([]bunProjectMappingRow, error) {
if visibleArchives != nil && len(visibleArchives) == 0 {
return []bunProjectMappingRow{}, nil
}
archiveIDs := sortedSetKeys(visibleArchives)
var rows []bunmodel.SourceWorktreeProjectMapping
load := func(chunk []string) error {
var batch []bunmodel.SourceWorktreeProjectMapping
query := store.NewSelect().Model(&batch)
if machine != nil {
query = query.Where("machine = ?", *machine)
}
if visibleArchives != nil {
query = query.Where("source_archive_id IN (?)", bun.List(chunk))
}
if err := query.Scan(ctx); err != nil {
return fmt.Errorf("listing canonical worktree mappings: %w", err)
}
rows = append(rows, batch...)
return nil
}
if visibleArchives == nil {
if err := load(nil); err != nil {
return nil, err
}
} else {
for start := 0; start < len(archiveIDs); start += bunDataListBatchSize {
end := min(start+bunDataListBatchSize, len(archiveIDs))
if err := load(archiveIDs[start:end]); err != nil {
return nil, err
}
}
}
sort.SliceStable(rows, func(i, j int) bool {
if machine != nil {
if rows[i].PathPrefix != rows[j].PathPrefix {
return rows[i].PathPrefix < rows[j].PathPrefix
}
return rows[i].SourceArchiveID < rows[j].SourceArchiveID
}
if rows[i].SourceArchiveID != rows[j].SourceArchiveID {
return rows[i].SourceArchiveID < rows[j].SourceArchiveID
}
if rows[i].Machine != rows[j].Machine {
return rows[i].Machine < rows[j].Machine
}
return rows[i].PathPrefix < rows[j].PathPrefix
})
result := make([]bunProjectMappingRow, len(rows))
for i, row := range rows {
result[i] = bunProjectMappingRow{
archiveID: row.SourceArchiveID,
mapping: WorktreeProjectMapping{
ID: row.ID, Machine: row.Machine, PathPrefix: row.PathPrefix,
Layout: row.Layout, Project: row.Project,
OriginalProject: row.OriginalProject, Enabled: row.Enabled,
CreatedAt: formatBunDataTime(row.CreatedAt.Time),
UpdatedAt: formatBunDataTime(row.UpdatedAt.Time),
},
}
}
return result, nil
}
func listBunProjectRuleMachines(
ctx context.Context, store bun.IDB,
) ([]string, error) {
var rows []struct {
Machine string `bun:"machine"`
}
if err := store.NewRaw(`
SELECT machine FROM sessions
WHERE deleted_at IS NULL AND machine != ''
UNION
SELECT machine FROM source_worktree_project_mappings
WHERE machine != ''
ORDER BY machine`).Scan(ctx, &rows); err != nil {
return nil, fmt.Errorf("listing canonical project rule machines: %w", err)
}
machines := make([]string, len(rows))
for i, row := range rows {
machines[i] = row.Machine
}
return machines, nil
}
func listBunGovernanceSessions(
ctx context.Context, store bun.IDB, machine *string,
) ([]bunGovernanceSessionRow, error) {
var rows []bunGovernanceSessionRow
query := store.NewSelect().TableExpr("sessions AS session").
ColumnExpr("session.id AS id").
ColumnExpr("session.machine AS machine").
ColumnExpr("session.project AS project").
ColumnExpr("session.cwd AS cwd").
ColumnExpr("COALESCE(session.file_path, '') AS file_path").
ColumnExpr("session.source_archive_id AS source_archive_id").
Where("session.deleted_at IS NULL").
Where("session.source_archive_id != ''").
Where(`EXISTS (
SELECT 1 FROM source_worktree_project_mappings AS mapping
WHERE mapping.source_archive_id = session.source_archive_id
AND mapping.machine = session.machine
AND mapping.enabled = TRUE
)`)
if machine != nil {
query = query.Where("session.machine = ?", *machine)
}
if err := query.OrderExpr("session.id ASC").Scan(ctx, &rows); err != nil {
return nil, fmt.Errorf("listing canonical governance sessions: %w", err)
}
return rows, nil
}
func bunMappingEvaluationRows(
rows []bunGovernanceSessionRow,
) []MappingEvaluationRow {
result := make([]MappingEvaluationRow, len(rows))
for i, row := range rows {
result[i] = MappingEvaluationRow{
SessionID: row.ID, Machine: row.Machine, Project: row.Project,
Cwd: row.Cwd, FilePath: row.FilePath,
SourceArchiveID: row.SourceArchiveID,
}
}
return result
}
func listBunCandidateProjectLabels(
ctx context.Context, store bun.IDB,
) (map[string]struct{}, error) {
var rows []struct {
Project string `bun:"project"`
}
if err := store.NewSelect().Table("sessions").Column("project").
Where("deleted_at IS NULL").Group("project").OrderExpr("project ASC").
Scan(ctx, &rows); err != nil {
return nil, fmt.Errorf("listing canonical candidate projects: %w", err)
}
labels := make(map[string]struct{}, len(rows))
for _, row := range rows {
labels[row.Project] = struct{}{}
}
return labels, nil
}
func listBunCandidateSessions(
ctx context.Context, store bun.IDB, projects []string,
) ([]bunCandidateSessionRow, error) {
rows := make([]bunCandidateSessionRow, 0)
for start := 0; start < len(projects); start += bunDataListBatchSize {
end := min(start+bunDataListBatchSize, len(projects))
var batch []bunCandidateSessionRow
if err := store.NewSelect().Table("sessions").Column(
"id", "project", "machine", "cwd", "source_archive_id",
"source_database_generation",
).Where("deleted_at IS NULL").
Where("project IN (?)", bun.List(projects[start:end])).
OrderExpr("id ASC").Scan(ctx, &batch); err != nil {
return nil, fmt.Errorf("listing canonical candidate sessions: %w", err)
}
rows = append(rows, batch...)
}
sort.SliceStable(rows, func(i, j int) bool { return rows[i].ID < rows[j].ID })
return rows, nil
}
func formatBunDataTime(value time.Time) string {
if value.IsZero() {
return ""
}
return value.UTC().Format(time.RFC3339Nano)
}
func bunArchiveMappings(rows []bunProjectMappingRow) []ArchiveMappings {
byArchive := make(map[string][]WorktreeProjectMapping)
order := make([]string, 0)
for _, row := range rows {
if _, ok := byArchive[row.archiveID]; !ok {
order = append(order, row.archiveID)
}
byArchive[row.archiveID] = append(byArchive[row.archiveID], row.mapping)
}
result := make([]ArchiveMappings, len(order))
for i, archiveID := range order {
result[i] = ArchiveMappings{
SourceArchiveID: archiveID, Mappings: byArchive[archiveID],
}
}
return result
}
func buildBunProjectIdentityMap(
ctx context.Context,
store bun.IDB,
labels []string,
) (map[string]export.ProjectMapEntry, []export.ProjectIdentityObservation, error) {
observations, err := listBunProjectIdentityObservations(ctx, store, labels)
if err != nil {
return nil, nil, err
}
scope, err := bunSourceArchiveIdentityScope(ctx, store, observations)
if err != nil {
return nil, nil, err
}
return export.BuildProjectsMapWithScope(labels, observations, scope), observations, nil
}
type bunSnapshotKey struct {
archiveID string
generation string
sessionID string
}
func listBunProjectIdentitySnapshots(
ctx context.Context,
store bun.IDB,
sessionIDs []string,
) (map[bunSnapshotKey]bunmodel.SourceSessionProjectIdentitySnapshot, error) {
result := make(map[bunSnapshotKey]bunmodel.SourceSessionProjectIdentitySnapshot)
for start := 0; start < len(sessionIDs); start += bunDataListBatchSize {
end := min(start+bunDataListBatchSize, len(sessionIDs))
var rows []bunmodel.SourceSessionProjectIdentitySnapshot
if err := store.NewSelect().
TableExpr("source_session_project_identity_snapshots AS snapshot").
ColumnExpr("snapshot.*").
Join("JOIN sessions AS session").
JoinOn("session.source_archive_id = snapshot.source_archive_id").
JoinOn("session.source_database_generation = snapshot.source_database_generation").
JoinOn("session.id = snapshot.source_session_id").
Where("session.deleted_at IS NULL").
Where("session.id IN (?)", bun.List(sessionIDs[start:end])).
OrderExpr("snapshot.source_archive_id ASC").
OrderExpr("snapshot.source_database_generation ASC").
OrderExpr("snapshot.source_session_id ASC").Scan(ctx, &rows); err != nil {
return nil, fmt.Errorf("listing canonical project identity snapshots: %w", err)
}
for _, row := range rows {
result[bunSnapshotKey{
archiveID: row.SourceArchiveID,
generation: row.SourceDatabaseGeneration,
sessionID: row.SourceSessionID,
}] = row
}
}
return result, nil
}
func projectIdentitySnapshotFromBunRow(
row bunmodel.SourceSessionProjectIdentitySnapshot,
) export.ProjectIdentityObservation {
return export.ProjectIdentityObservation{
SourceArchiveID: row.SourceArchiveID, Project: row.Project,
Machine: row.Machine, RootPath: row.RootPath, GitRemote: row.GitRemote,
GitRemoteName: row.GitRemoteName, RepositoryPath: row.RepositoryPath,
WorktreeName: row.WorktreeName, WorktreeRootPath: row.WorktreeRootPath,
WorktreeRelationship: export.WorktreeRelationship(row.WorktreeRelationship),
CheckoutState: export.CheckoutState(row.CheckoutState), GitBranch: row.GitBranch,
RemoteResolution: export.ProjectResolution(row.RemoteResolution),
RemoteCandidateCount: row.RemoteCandidateCount, ObservedAt: row.ObservedAt.UTC(),
NormalizedRemote: row.NormalizedRemote, KeySource: row.KeySource, Key: row.Key,
}
}
+190
View File
@@ -0,0 +1,190 @@
package db
import (
"context"
"database/sql"
"fmt"
"strings"
"testing"
"time"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
"github.com/uptrace/bun"
"github.com/uptrace/bun/dialect/sqlitedialect"
"go.kenn.io/agentsview/internal/db/bunmodel"
)
type candidateQueryHook struct {
queries []string
}
func (*candidateQueryHook) BeforeQuery(
ctx context.Context, _ *bun.QueryEvent,
) context.Context {
return ctx
}
func (h *candidateQueryHook) AfterQuery(
_ context.Context, event *bun.QueryEvent,
) {
if event.Operation() == "SELECT" {
h.queries = append(h.queries, event.Query)
}
}
func TestBunProjectRuleSessionsHydrateOnlyEnabledArchiveMachineScopes(t *testing.T) {
raw, err := sql.Open("sqlite3", ":memory:")
require.NoError(t, err)
raw.SetMaxOpenConns(1)
t.Cleanup(func() { require.NoError(t, raw.Close()) })
store := bun.NewDB(raw, sqlitedialect.New())
require.NoError(t, CreateCommonSchema(t.Context(), store))
for _, archive := range []bunmodel.SourceArchive{
{SourceArchiveID: "archive-a", SourceArchiveSalt: "salt-a"},
{SourceArchiveID: "archive-b", SourceArchiveSalt: "salt-b"},
} {
_, err = store.NewInsert().Model(&archive).Exec(t.Context())
require.NoError(t, err)
}
now := bunmodel.NewTimestamp(time.Date(2026, 8, 3, 12, 0, 0, 0, time.UTC))
_, err = store.NewInsert().Model(&bunmodel.SourceWorktreeProjectMapping{
SourceArchiveID: "archive-a", Machine: "wanted-machine",
PathPrefix: "/wanted", Layout: WorktreeMappingLayoutExplicit,
Project: "wanted", Enabled: true, CreatedAt: now, UpdatedAt: now,
}).Exec(t.Context())
require.NoError(t, err)
sessions := []bunmodel.Session{
{
ID: "wanted", Project: "wanted", Machine: "wanted-machine",
Agent: "codex", Cwd: "/wanted/repo", CreatedAt: now,
SourceArchiveID: "archive-a", SourceDatabaseGeneration: "generation-a",
},
{
ID: "wrong-archive", Project: "other", Machine: "wanted-machine",
Agent: "codex", Cwd: "/wanted/repo", CreatedAt: now,
SourceArchiveID: "archive-b", SourceDatabaseGeneration: "generation-b",
},
}
for i := range 500 {
sessions = append(sessions, bunmodel.Session{
ID: fmt.Sprintf("unrelated-%03d", i), Project: "unrelated",
Machine: "unrelated-machine", Agent: "codex", Cwd: "/elsewhere",
CreatedAt: now, SourceArchiveID: "archive-a",
SourceDatabaseGeneration: "generation-a",
})
}
_, err = store.NewInsert().Model(&sessions).Exec(t.Context())
require.NoError(t, err)
machine := "wanted-machine"
rows, err := listBunGovernanceSessions(t.Context(), store, &machine)
require.NoError(t, err)
require.Len(t, rows, 1)
assert.Equal(t, "wanted", rows[0].ID)
assert.Equal(t, "archive-a", rows[0].SourceArchiveID)
}
func TestBunWorktreeCandidatesHydrateOnlySelectedProjects(t *testing.T) {
raw, err := sql.Open("sqlite3", ":memory:")
require.NoError(t, err)
raw.SetMaxOpenConns(1)
t.Cleanup(func() { require.NoError(t, raw.Close()) })
store := bun.NewDB(raw, sqlitedialect.New())
require.NoError(t, CreateCommonSchema(t.Context(), store))
_, err = store.NewInsert().Model(&bunmodel.SourceArchive{
SourceArchiveID: "candidate-archive", SourceArchiveSalt: "candidate-salt",
}).Exec(t.Context())
require.NoError(t, err)
created := bunmodel.NewTimestamp(time.Date(2026, 8, 3, 12, 0, 0, 0, time.UTC))
sessions := []bunmodel.Session{{
ID: "selected-session", Project: "selected-project", Machine: "selected-host",
Agent: "codex", Cwd: "/workspace/selected", CreatedAt: created,
SourceArchiveID: "candidate-archive", SourceDatabaseGeneration: "candidate-generation",
}}
for i := range 500 {
sessions = append(sessions, bunmodel.Session{
ID: fmt.Sprintf("unrelated-session-%03d", i), Project: "unrelated-project",
Machine: "unrelated-host", Agent: "codex", Cwd: "/workspace/unrelated",
CreatedAt: created, SourceArchiveID: "candidate-archive",
SourceDatabaseGeneration: "candidate-generation",
})
}
_, err = store.NewInsert().Model(&sessions).Exec(t.Context())
require.NoError(t, err)
base := NewBunStore(&sessionContractBackend{store: store})
projects, err := base.BuildProjectIdentityMap(
t.Context(), []string{"selected-project", "unrelated-project"},
)
require.NoError(t, err)
hook := new(candidateQueryHook)
common := NewBunStore(&sessionContractBackend{store: store.WithQueryHook(hook)})
candidates, err := common.ListArchiveWorktreeCandidates(
t.Context(), ArchiveWorktreeCandidateRequest{
ProjectLabel: "selected-project",
ProjectKey: projects["selected-project"].ProjectKey,
},
)
require.NoError(t, err)
require.Len(t, candidates, 1)
assert.Equal(t, 1, candidates[0].ContributingSessions)
assert.Equal(t, "selected-session", candidates[0].Examples[0].SessionID)
selectedSessionQueries := 0
for _, query := range hook.queries {
normalized := strings.ToLower(query)
if !strings.Contains(normalized, `from "sessions"`) ||
!strings.Contains(normalized, `"id"`) {
continue
}
selectedSessionQueries++
assert.Contains(t, normalized, "project in",
"session hydration must be constrained by the selected project")
}
assert.Equal(t, 1, selectedSessionQueries,
"candidate reads should hydrate selected session details once")
}
func TestSQLiteConsistentViewKeepsOneReadSnapshot(t *testing.T) {
database := testDB(t)
backend := &sqliteBunBackend{store: database}
archiveID, err := database.GetArchiveID(t.Context())
require.NoError(t, err)
databaseID, err := database.GetDatabaseID(t.Context())
require.NoError(t, err)
inserted := make(chan error, 1)
err = backend.ConsistentView(t.Context(), func(store bun.IDB) error {
var before int
if err := store.NewSelect().Table("sessions").ColumnExpr("COUNT(*)").
Scan(t.Context(), &before); err != nil {
return err
}
go func() {
_, insertErr := database.getWriter().ExecContext(t.Context(), `
INSERT INTO sessions (
id, project, machine, agent, created_at,
source_archive_id, source_database_generation
) VALUES ('concurrent', 'p', 'm', 'a',
'2026-08-03T12:00:00Z', ?, ?)`,
archiveID, databaseID,
)
inserted <- insertErr
}()
if insertErr := <-inserted; insertErr != nil {
return fmt.Errorf("committing concurrent SQLite insert: %w", insertErr)
}
var after int
if err := store.NewSelect().Table("sessions").ColumnExpr("COUNT(*)").
Scan(t.Context(), &after); err != nil {
return err
}
assert.Equal(t, before, after)
return nil
})
require.NoError(t, err)
}
+161
View File
@@ -0,0 +1,161 @@
package db
import (
"context"
"database/sql"
"fmt"
"strings"
"github.com/uptrace/bun"
"go.kenn.io/agentsview/internal/db/bunmodel"
)
// InsertInsight stores a dashboard insight through the operation-scoped common
// writer and returns the engine-generated canonical ID.
func (s *BunStore) InsertInsight(insight Insight) (int64, error) {
ctx := context.Background()
row := insightToBunRow(insight)
var id int64
err := s.update(ctx, WriteInsight, func(store bun.IDB) error {
return store.RunInTx(ctx, nil, func(ctx context.Context, tx bun.Tx) error {
if err := tx.NewInsert().Model(&row).
ExcludeColumn("id").Value("created_at", "current_timestamp").
Returning("id").
Scan(ctx, &id); err != nil {
return fmt.Errorf("inserting insight: %w", err)
}
return nil
})
})
if err != nil {
return 0, err
}
return id, nil
}
// DeleteInsight removes a dashboard insight by canonical ID.
func (s *BunStore) DeleteInsight(id int64) error {
ctx := context.Background()
return s.update(ctx, WriteInsightDelete, func(store bun.IDB) error {
if _, err := store.NewDelete().Model((*bunmodel.Insight)(nil)).
Where("id = ?", id).Exec(ctx); err != nil {
return fmt.Errorf("deleting insight %d: %w", id, err)
}
return nil
})
}
// ListInsights returns the newest canonical insights matching the filter.
func (s *BunStore) ListInsights(
ctx context.Context, filter InsightFilter,
) ([]Insight, error) {
var rows []bunmodel.Insight
err := s.view(ctx, func(store bun.IDB) error {
query := applyBunInsightFilter(store.NewSelect().Model(&rows), filter)
return query.OrderExpr(s.bunInsightTimeOrder() + " DESC").OrderExpr("id DESC").
Limit(maxInsights).Scan(ctx)
})
if err != nil {
return nil, fmt.Errorf("querying insights: %w", err)
}
insights := make([]Insight, len(rows))
for i, row := range rows {
insights[i] = insightFromBunRow(row)
}
return insights, nil
}
// GetInsight returns one insight by ID or nil when it does not exist.
func (s *BunStore) GetInsight(
ctx context.Context, id int64,
) (*Insight, error) {
var row bunmodel.Insight
err := s.view(ctx, func(store bun.IDB) error {
return store.NewSelect().Model(&row).Where("id = ?", id).Scan(ctx)
})
if err == sql.ErrNoRows {
return nil, nil
}
if err != nil {
return nil, fmt.Errorf("getting insight %d: %w", id, err)
}
insight := insightFromBunRow(row)
return &insight, nil
}
// GetCachedInsight returns the newest insight for a non-empty cache key.
func (s *BunStore) GetCachedInsight(
ctx context.Context, cacheKey string,
) (*Insight, error) {
if strings.TrimSpace(cacheKey) == "" {
return nil, nil
}
var row bunmodel.Insight
err := s.view(ctx, func(store bun.IDB) error {
return store.NewSelect().Model(&row).Where("cache_key = ?", cacheKey).
OrderExpr(s.bunInsightTimeOrder() + " DESC").OrderExpr("id DESC").
Limit(1).Scan(ctx)
})
if err == sql.ErrNoRows {
return nil, nil
}
if err != nil {
return nil, fmt.Errorf("getting cached insight: %w", err)
}
insight := insightFromBunRow(row)
return &insight, nil
}
func (s *BunStore) bunInsightTimeOrder() string {
dialect := s.backend.SessionQueryDialect()
if dialect.timestampOrderExpr != nil {
return dialect.timestampOrderExpr("created_at")
}
return "created_at"
}
func applyBunInsightFilter(
query *bun.SelectQuery, filter InsightFilter,
) *bun.SelectQuery {
if filter.Type != "" {
query = query.Where("type = ?", filter.Type)
}
if filter.GlobalOnly {
query = query.Where("project IS NULL")
} else if filter.Project != "" {
query = query.Where("project = ?", filter.Project)
}
if filter.DateFrom != "" {
query = query.Where("date_from >= ?", filter.DateFrom)
}
if filter.DateTo != "" {
query = query.Where("date_to <= ?", filter.DateTo)
}
return query
}
func insightToBunRow(insight Insight) bunmodel.Insight {
return bunmodel.Insight{
ID: insight.ID, Type: insight.Type, DateFrom: insight.DateFrom,
DateTo: insight.DateTo, Project: insight.Project, Agent: insight.Agent,
Model: insight.Model, Prompt: insight.Prompt, Content: insight.Content,
Kind: insight.Kind, SchemaVersion: insight.SchemaVersion,
TemplateID: insight.TemplateID, TemplateVersion: insight.TemplateVersion,
AggregateHash: insight.AggregateHash, CacheKey: insight.CacheKey,
CacheStatus: insight.CacheStatus, ProvenanceJSON: insight.ProvenanceJSON,
StructuredJSON: insight.StructuredJSON,
}
}
func insightFromBunRow(row bunmodel.Insight) Insight {
return Insight{
ID: row.ID, Type: row.Type, DateFrom: row.DateFrom, DateTo: row.DateTo,
Project: row.Project, Agent: row.Agent, Model: row.Model, Prompt: row.Prompt,
Content: row.Content, Kind: row.Kind, SchemaVersion: row.SchemaVersion,
TemplateID: row.TemplateID, TemplateVersion: row.TemplateVersion,
AggregateHash: row.AggregateHash, CacheKey: row.CacheKey,
CacheStatus: row.CacheStatus, ProvenanceJSON: row.ProvenanceJSON,
StructuredJSON: row.StructuredJSON,
CreatedAt: formatBunCurationTime(row.CreatedAt.Time),
}
}
+611
View File
@@ -0,0 +1,611 @@
package db
import (
"context"
"database/sql"
"fmt"
"slices"
"sort"
"time"
"github.com/uptrace/bun"
"go.kenn.io/agentsview/internal/db/bunmodel"
)
// GetMessages returns ordinal-paginated canonical messages with tool data.
func (s *BunStore) GetMessages(
ctx context.Context, sessionID string, from, limit int, asc bool,
) ([]Message, error) {
if limit <= 0 || limit > MaxMessageLimit {
limit = DefaultMessageLimit
}
var pendingMessages []Message
err := s.consistentView(ctx, func(store bun.IDB) error {
query := store.NewSelect().Model((*bunmodel.Message)(nil)).
Where("session_id = ?", sessionID)
if asc {
query = query.Where("ordinal >= ?", from).OrderExpr("ordinal ASC")
} else {
query = query.Where("ordinal <= ?", from).OrderExpr("ordinal DESC")
}
rows, err := scanBunMessages(ctx, query.Limit(limit))
if err != nil {
return fmt.Errorf("querying messages: %w", err)
}
if err := attachBunToolData(ctx, store, rows); err != nil {
return err
}
pendingMessages = rows
return nil
})
if err != nil {
return nil, err
}
return pendingMessages, nil
}
// GetMessagesWindow implements linear and around-anchor retrieval once for all
// Bun backends. The anchor is retained even when its role is filtered out.
func (s *BunStore) GetMessagesWindow(
ctx context.Context, sessionID string, window MessageWindow,
) ([]Message, error) {
if window.Around == nil {
from := 0
if window.From != nil {
from = *window.From
}
if len(window.Roles) == 0 {
return s.GetMessages(ctx, sessionID, from, window.Limit, window.Asc)
}
if window.Limit <= 0 || window.Limit > MaxMessageLimit {
window.Limit = DefaultMessageLimit
}
var pendingMessages []Message
err := s.consistentView(ctx, func(store bun.IDB) error {
query := store.NewSelect().Model((*bunmodel.Message)(nil)).
Where("session_id = ?", sessionID).
Where("role IN (?)", bun.List(window.Roles))
if window.Asc {
query = query.Where("ordinal >= ?", from).OrderExpr("ordinal ASC")
} else {
query = query.Where("ordinal <= ?", from).OrderExpr("ordinal DESC")
}
rows, err := scanBunMessages(ctx, query.Limit(window.Limit))
if err != nil {
return fmt.Errorf("querying role-filtered messages: %w", err)
}
if err := attachBunToolData(ctx, store, rows); err != nil {
return err
}
pendingMessages = rows
return nil
})
if err != nil {
return nil, err
}
return pendingMessages, nil
}
anchor := *window.Around
var pendingMessages []Message
err := s.consistentView(ctx, func(store bun.IDB) error {
queryPart := func(operator, order string, limit int, roles bool) ([]Message, error) {
if limit <= 0 {
return []Message{}, nil
}
query := store.NewSelect().Model((*bunmodel.Message)(nil)).
Where("session_id = ?", sessionID).
Where("ordinal "+operator+" ?", anchor).
OrderExpr("ordinal " + order).Limit(limit)
if roles && len(window.Roles) > 0 {
query = query.Where("role IN (?)", bun.List(window.Roles))
}
return scanBunMessages(ctx, query)
}
before, err := queryPart("<", "DESC", window.Before, true)
if err != nil {
return fmt.Errorf("querying before-window messages: %w", err)
}
slices.Reverse(before)
anchorRows, err := queryPart("=", "ASC", 1, false)
if err != nil {
return fmt.Errorf("querying anchor message: %w", err)
}
after, err := queryPart(">", "ASC", window.After, true)
if err != nil {
return fmt.Errorf("querying after-window messages: %w", err)
}
messages := make([]Message, 0, len(before)+len(anchorRows)+len(after))
messages = append(messages, before...)
messages = append(messages, anchorRows...)
messages = append(messages, after...)
if err := attachBunToolData(ctx, store, messages); err != nil {
return err
}
pendingMessages = messages
return nil
})
if err != nil {
return nil, err
}
return pendingMessages, nil
}
// GetAllMessages returns all canonical messages in ordinal order.
func (s *BunStore) GetAllMessages(
ctx context.Context, sessionID string,
) ([]Message, error) {
pendingMessages := []Message{}
err := s.consistentView(ctx, func(store bun.IDB) error {
rows, err := scanBunMessages(ctx, store.NewSelect().
Model((*bunmodel.Message)(nil)).Where("session_id = ?", sessionID).
OrderExpr("ordinal ASC"))
if err != nil {
return fmt.Errorf("querying all messages: %w", err)
}
if err := attachBunToolData(ctx, store, rows); err != nil {
return err
}
pendingMessages = rows
return nil
})
if err != nil {
return nil, err
}
return pendingMessages, nil
}
func scanBunMessages(ctx context.Context, query *bun.SelectQuery) ([]Message, error) {
var rows []bunmodel.Message
if err := query.Scan(ctx, &rows); err != nil {
return nil, err
}
messages := make([]Message, 0, len(rows))
for _, row := range rows {
message := messageFromBunRow(row)
if row.ID == nil {
message.ID = int64(message.Ordinal)
}
messages = append(messages, message)
}
return messages, nil
}
func attachBunToolData(
ctx context.Context, store bun.IDB, messages []Message,
) error {
if len(messages) == 0 {
return nil
}
messageIndex := make(map[int]int, len(messages))
ordinals := make([]int, 0, len(messages))
for i, message := range messages {
messageIndex[message.Ordinal] = i
ordinals = append(ordinals, message.Ordinal)
}
const hydrationBatchSize = 500
calls := make([]bunmodel.ToolCall, 0)
for start := 0; start < len(ordinals); start += hydrationBatchSize {
end := min(start+hydrationBatchSize, len(ordinals))
var batch []bunmodel.ToolCall
if err := store.NewSelect().Model(&batch).
Where("session_id = ?", messages[0].SessionID).
Where("message_ordinal IN (?)", bun.List(ordinals[start:end])).
OrderExpr("message_ordinal ASC").OrderExpr("call_index ASC").
Scan(ctx); err != nil {
return fmt.Errorf("querying tool calls: %w", err)
}
calls = append(calls, batch...)
}
for _, row := range calls {
messagePosition, ok := messageIndex[row.MessageOrdinal]
if !ok || row.CallIndex < 0 {
continue
}
for len(messages[messagePosition].ToolCalls) <= row.CallIndex {
messages[messagePosition].ToolCalls = append(
messages[messagePosition].ToolCalls, ToolCall{},
)
}
messages[messagePosition].ToolCalls[row.CallIndex] = toolCallFromBunRow(row)
}
events := make([]bunmodel.ToolResultEvent, 0)
for start := 0; start < len(ordinals); start += hydrationBatchSize {
end := min(start+hydrationBatchSize, len(ordinals))
var batch []bunmodel.ToolResultEvent
if err := store.NewSelect().Model(&batch).
Where("session_id = ?", messages[0].SessionID).
Where("tool_call_message_ordinal IN (?)", bun.List(ordinals[start:end])).
OrderExpr("tool_call_message_ordinal ASC").
OrderExpr("call_index ASC").OrderExpr("event_index ASC").
Scan(ctx); err != nil {
return fmt.Errorf("querying tool result events: %w", err)
}
events = append(events, batch...)
}
for _, row := range events {
messagePosition, ok := messageIndex[row.ToolCallMessageOrdinal]
if !ok || row.CallIndex < 0 ||
row.CallIndex >= len(messages[messagePosition].ToolCalls) {
continue
}
messages[messagePosition].ToolCalls[row.CallIndex].ResultEvents = append(
messages[messagePosition].ToolCalls[row.CallIndex].ResultEvents,
toolResultEventFromBunRow(row),
)
}
return nil
}
type bunTimingSessionRow struct {
ID string `bun:"id"`
StartedAt *bunmodel.Timestamp `bun:"started_at"`
EndedAt *bunmodel.Timestamp `bun:"ended_at"`
}
type bunTimingMessageRow struct {
Ordinal int `bun:"ordinal"`
Timestamp *bunmodel.Timestamp `bun:"timestamp"`
HasToolUse bool `bun:"has_tool_use"`
}
type bunTimingCallRow struct {
MessageOrdinal int `bun:"message_ordinal"`
CallIndex int `bun:"call_index"`
ToolUseID string `bun:"tool_use_id"`
ToolName string `bun:"tool_name"`
Category string `bun:"category"`
SkillName *string `bun:"skill_name"`
SubagentSessionID *string `bun:"subagent_session_id"`
InputJSON *string `bun:"input_json"`
}
type bunTimingEventRow struct {
ToolCallMessageOrdinal int `bun:"tool_call_message_ordinal"`
CallIndex int `bun:"call_index"`
Source string `bun:"source"`
Status string `bun:"status"`
Timestamp *bunmodel.Timestamp `bun:"timestamp"`
EventIndex int `bun:"event_index"`
}
func toolCallFromBunRow(row bunmodel.ToolCall) ToolCall {
call := ToolCall{
SessionID: row.SessionID, MessageOrdinal: row.MessageOrdinal,
ToolName: row.ToolName, Category: row.Category, ToolUseID: row.ToolUseID,
CallIndex: row.CallIndex,
}
if row.MessageID != nil {
call.MessageID = *row.MessageID
}
if row.InputJSON != nil {
call.InputJSON = *row.InputJSON
}
if row.SkillName != nil {
call.SkillName = *row.SkillName
}
if row.ResultContentLength != nil {
call.ResultContentLength = *row.ResultContentLength
}
if row.ResultContent != nil {
call.ResultContent = *row.ResultContent
}
if row.SubagentSessionID != nil {
call.SubagentSessionID = *row.SubagentSessionID
}
if row.FilePath != nil {
call.FilePath = *row.FilePath
}
return call
}
func toolResultEventFromBunRow(row bunmodel.ToolResultEvent) ToolResultEvent {
event := ToolResultEvent{
Source: row.Source, Status: row.Status, Content: row.Content,
ContentLength: row.ContentLength, EventIndex: row.EventIndex,
Timestamp: requiredTimestampFromBunRowPtr(row.Timestamp),
}
if row.ToolUseID != nil {
event.ToolUseID = *row.ToolUseID
}
if row.AgentID != nil {
event.AgentID = *row.AgentID
}
if row.SubagentSessionID != nil {
event.SubagentSessionID = *row.SubagentSessionID
}
return event
}
// GetResumeModelCounts counts non-synthetic assistant models.
func (s *BunStore) GetResumeModelCounts(
ctx context.Context, sessionID string,
) ([]ModelCount, error) {
var counts []ModelCount
err := s.view(ctx, func(store bun.IDB) error {
return store.NewSelect().Table("messages").
Column("model").ColumnExpr("COUNT(*) AS count").
Where("session_id = ?", sessionID).Where("role = ?", "assistant").
Where("model <> ''").Where("model <> ?", "<synthetic>").
Group("model").OrderExpr("model ASC").Scan(ctx, &counts)
})
if err != nil {
return nil, fmt.Errorf("querying resume model counts: %w", err)
}
return counts, nil
}
// GetSessionActivity reduces canonical timestamps in Go so bucket semantics do
// not depend on an engine's date functions.
func (s *BunStore) GetSessionActivity(
ctx context.Context, sessionID string,
) (*SessionActivityResponse, error) {
type activityMessage struct {
Ordinal int `bun:"ordinal"`
Role string `bun:"role"`
Content string `bun:"content"`
IsSystem bool `bun:"is_system"`
Timestamp any `bun:"timestamp"`
}
var rows []activityMessage
err := s.view(ctx, func(store bun.IDB) error {
return store.NewSelect().Table("messages").
Column("ordinal", "role", "content", "is_system", "timestamp").
Where("session_id = ?", sessionID).
OrderExpr("ordinal ASC").Scan(ctx, &rows)
})
if err != nil {
return nil, fmt.Errorf("querying activity messages: %w", err)
}
type visibleMessage struct {
message activityMessage
at time.Time
}
visible := make([]visibleMessage, 0, len(rows))
var minTime, maxTime time.Time
for _, row := range rows {
if row.IsSystem || IsSystemPrefixed(row.Content, row.Role) {
continue
}
at, ok := bunActivityTime(row.Timestamp)
if !ok {
continue
}
if len(visible) == 0 || at.Before(minTime) {
minTime = at
}
if len(visible) == 0 || at.After(maxTime) {
maxTime = at
}
visible = append(visible, visibleMessage{message: row, at: at})
}
if len(visible) == 0 {
return &SessionActivityResponse{
Buckets: []SessionActivityBucket{}, TotalMessages: len(rows),
}, nil
}
anchor := minTime.Unix()
interval := SnapInterval(maxTime.Unix() - minTime.Unix())
type bucketValue struct {
user, assistant int
first *int
}
populated := make(map[int]bucketValue)
maxIndex := 0
for _, item := range visible {
index := int((item.at.Unix() - anchor) / interval)
value := populated[index]
switch item.message.Role {
case "user":
value.user++
case "assistant":
value.assistant++
}
if value.first == nil || item.message.Ordinal < *value.first {
ordinal := item.message.Ordinal
value.first = &ordinal
}
populated[index] = value
maxIndex = max(maxIndex, index)
}
buckets := make([]SessionActivityBucket, maxIndex+1)
for index := range buckets {
start := time.Unix(anchor+int64(index)*interval, 0).UTC()
value := populated[index]
buckets[index] = SessionActivityBucket{
StartTime: start.Format(time.RFC3339),
EndTime: start.Add(time.Duration(interval) * time.Second).Format(time.RFC3339),
UserCount: value.user, AssistantCount: value.assistant,
FirstOrdinal: value.first,
}
}
return &SessionActivityResponse{
Buckets: buckets, IntervalSeconds: interval, TotalMessages: len(rows),
}, nil
}
func bunActivityTime(value any) (time.Time, bool) {
switch value := value.(type) {
case nil:
return time.Time{}, false
case time.Time:
return value.UTC(), !value.IsZero()
case string:
parsed, err := bunmodel.ParseTimestamp(value)
return parsed.Time, err == nil && !parsed.IsZero()
case []byte:
return bunActivityTime(string(value))
default:
return time.Time{}, false
}
}
// GetSessionTiming assembles timing from canonical rows in Go.
func (s *BunStore) GetSessionTiming(
ctx context.Context, sessionID string,
) (*SessionTiming, error) {
now := time.Now().UTC()
type timingRows struct {
sessionRow bunTimingSessionRow
messages []bunTimingMessageRow
calls []bunTimingCallRow
events []bunTimingEventRow
subagents map[string]bunTimingSessionRow
}
var pending timingRows
err := s.consistentView(ctx, func(store bun.IDB) error {
attempt := timingRows{subagents: make(map[string]bunTimingSessionRow)}
if err := store.NewSelect().Table("sessions").
Column("id", "started_at", "ended_at").Where("id = ?", sessionID).
Where("deleted_at IS NULL").Scan(ctx, &attempt.sessionRow); err != nil {
return err
}
if err := store.NewSelect().Table("messages").
Column("ordinal", "timestamp", "has_tool_use").
Where("session_id = ?", sessionID).
OrderExpr("ordinal ASC").Scan(ctx, &attempt.messages); err != nil {
return err
}
if err := store.NewSelect().Table("tool_calls").
Column(
"message_ordinal", "call_index", "tool_use_id", "tool_name",
"category", "skill_name", "subagent_session_id", "input_json",
).
Where("session_id = ?", sessionID).
OrderExpr("message_ordinal ASC").OrderExpr("call_index ASC").
Scan(ctx, &attempt.calls); err != nil {
return err
}
subagentIDs := make([]string, 0)
for _, call := range attempt.calls {
if call.SubagentSessionID != nil && *call.SubagentSessionID != "" {
subagentIDs = append(subagentIDs, *call.SubagentSessionID)
}
}
if len(subagentIDs) > 0 {
var rows []bunTimingSessionRow
if err := store.NewSelect().Table("sessions").
Column("id", "started_at", "ended_at").
Where("id IN (?)", bun.List(subagentIDs)).Scan(ctx, &rows); err != nil {
return fmt.Errorf("querying timing subagents: %w", err)
}
for _, row := range rows {
attempt.subagents[row.ID] = row
}
}
if err := store.NewSelect().Table("tool_result_events").
Column(
"tool_call_message_ordinal", "call_index", "source", "status",
"timestamp", "event_index",
).
Where("session_id = ?", sessionID).
OrderExpr("tool_call_message_ordinal ASC").OrderExpr("call_index ASC").
OrderExpr("event_index ASC").Scan(ctx, &attempt.events); err != nil {
return err
}
pending = attempt
return nil
})
if err == sql.ErrNoRows {
return nil, nil
}
if err != nil {
return nil, fmt.Errorf("querying session timing: %w", err)
}
sessionRow := pending.sessionRow
messages := pending.messages
calls := pending.calls
events := pending.events
subagents := pending.subagents
session := Session{
ID: sessionRow.ID, StartedAt: timestampFromBunRow(sessionRow.StartedAt),
EndedAt: timestampFromBunRow(sessionRow.EndedAt),
}
turnRows := make([]TurnRow, 0, len(messages))
for index, message := range messages {
turn := TurnRow{
MessageID: int64(message.Ordinal), Ordinal: int64(message.Ordinal),
Timestamp: requiredTimestampFromBunRowPtr(message.Timestamp),
HasToolUse: message.HasToolUse,
}
if message.HasToolUse {
end := ""
if index+1 < len(messages) {
end = requiredTimestampFromBunRowPtr(messages[index+1].Timestamp)
} else if session.EndedAt != nil {
end = *session.EndedAt
}
if duration, ok := bunTimingMillis(turn.Timestamp, end); ok {
turn.DurationMs = &duration
}
}
turnRows = append(turnRows, turn)
}
eventsByCall := make(map[[2]int][]bunTimingEventRow)
for _, event := range events {
key := [2]int{event.ToolCallMessageOrdinal, event.CallIndex}
eventsByCall[key] = append(eventsByCall[key], event)
}
callRows := make([]CallRow, 0, len(calls))
for _, call := range calls {
row := CallRow{
MessageID: int64(call.MessageOrdinal), ToolUseID: call.ToolUseID,
ToolName: call.ToolName, Category: call.Category,
SkillName: call.SkillName, SubagentSessionID: call.SubagentSessionID,
}
if call.InputJSON != nil {
row.InputJSON = *call.InputJSON
}
usedSubagentTiming := false
if call.SubagentSessionID != nil {
if subagent, ok := subagents[*call.SubagentSessionID]; ok {
start := requiredTimestampFromBunRowPtr(subagent.StartedAt)
end := requiredTimestampFromBunRowPtr(subagent.EndedAt)
if end == "" {
end = now.Format(time.RFC3339)
}
if duration, valid := bunTimingMillis(start, end); valid {
row.DurationMs = &duration
usedSubagentTiming = true
}
}
}
if !usedSubagentTiming {
var started, completed string
for _, event := range eventsByCall[[2]int{call.MessageOrdinal, call.CallIndex}] {
if event.Source != "tool_execution" || event.Timestamp == nil {
continue
}
switch event.Status {
case "started":
if started == "" {
started = requiredTimestampFromBunRowPtr(event.Timestamp)
}
case "completed", "errored":
completed = requiredTimestampFromBunRowPtr(event.Timestamp)
}
}
if duration, valid := bunTimingMillis(started, completed); valid {
row.DurationMs = &duration
row.CompletedAt = completed
}
}
callRows = append(callRows, row)
}
sort.Slice(callRows, func(i, j int) bool {
return callRows[i].MessageID < callRows[j].MessageID
})
return AssembleTiming(&session, turnRows, callRows, now), nil
}
func bunTimingMillis(start, end string) (int64, bool) {
startTime, err := time.Parse(time.RFC3339Nano, start)
if err != nil {
return 0, false
}
endTime, err := time.Parse(time.RFC3339Nano, end)
if err != nil || endTime.Before(startTime) {
return 0, false
}
return endTime.Sub(startTime).Milliseconds(), true
}
+317
View File
@@ -0,0 +1,317 @@
package db
import (
"context"
"database/sql"
"encoding/json"
"strings"
"testing"
"time"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
"github.com/uptrace/bun"
"github.com/uptrace/bun/dialect/sqlitedialect"
"go.kenn.io/agentsview/internal/db/bunmodel"
)
func TestBunStoreMessageCompositeReadsPublishOnlyAcceptedReplayAttempt(
t *testing.T,
) {
first := testDB(t)
second := testDB(t)
const sessionID = "replayed-messages"
seed := func(database *DB, label string) {
t.Helper()
require.NoError(t, database.UpsertSession(Session{
ID: sessionID, Project: "replaying-reads", Machine: "host", Agent: "codex",
MessageCount: 1, UserMessageCount: 1,
}))
require.NoError(t, database.InsertMessages([]Message{{
SessionID: sessionID, Ordinal: 5, Role: "assistant",
Content: label + " message", ContentLength: len(label) + len(" message"),
HasToolUse: true,
ToolCalls: []ToolCall{{
ToolName: label + " tool", Category: "Read", ToolUseID: label + "-call",
ResultEvents: []ToolResultEvent{{
ToolUseID: label + "-call", Source: "tool_execution",
Status: label + " status", Content: label + " result",
}},
}},
}}))
}
seed(first, "rejected")
seed(second, "accepted")
store := NewBunStore(&replayingReadBackend{
first: first.bunReader, second: second.bunReader,
})
anchor := 5
reads := []struct {
name string
read func(context.Context) ([]Message, error)
}{
{
name: "messages",
read: func(ctx context.Context) ([]Message, error) {
return store.GetMessages(ctx, sessionID, 0, 10, true)
},
},
{
name: "message window",
read: func(ctx context.Context) ([]Message, error) {
return store.GetMessagesWindow(ctx, sessionID, MessageWindow{
Around: &anchor,
})
},
},
{
name: "all messages",
read: func(ctx context.Context) ([]Message, error) {
return store.GetAllMessages(ctx, sessionID)
},
},
}
for _, test := range reads {
t.Run(test.name, func(t *testing.T) {
messages, err := test.read(t.Context())
require.NoError(t, err)
require.Len(t, messages, 1)
assert.Equal(t, "accepted message", messages[0].Content)
require.Len(t, messages[0].ToolCalls, 1)
assert.Equal(t, "accepted tool", messages[0].ToolCalls[0].ToolName)
require.NotEmpty(t, messages[0].ToolCalls[0].ResultEvents)
for _, event := range messages[0].ToolCalls[0].ResultEvents {
assert.Equal(t, "accepted status", event.Status)
}
})
}
}
func TestBunStoreSessionTimingPublishesOnlyAcceptedReplayAttempt(t *testing.T) {
first := testDB(t)
second := testDB(t)
const sessionID = "replayed-timing"
seed := func(database *DB, label, started, ended, messageAt string) {
t.Helper()
require.NoError(t, database.UpsertSession(Session{
ID: sessionID, Project: "replaying-reads", Machine: "host", Agent: "codex",
StartedAt: &started, EndedAt: &ended, MessageCount: 1,
}))
require.NoError(t, database.InsertMessages([]Message{{
SessionID: sessionID, Ordinal: 0, Role: "assistant",
Content: label, ContentLength: len(label), Timestamp: messageAt,
HasToolUse: true,
ToolCalls: []ToolCall{{
ToolName: label + " tool", Category: "Read", ToolUseID: label + "-call",
}},
}}))
}
seed(first, "rejected", "2026-08-01T10:00:00Z", "2026-08-01T10:10:00Z",
"2026-08-01T10:01:00Z")
seed(second, "accepted", "2026-08-01T12:00:00Z", "2026-08-01T12:03:00Z",
"2026-08-01T12:01:00Z")
timing, err := NewBunStore(&replayingReadBackend{
first: first.bunReader, second: second.bunReader,
}).GetSessionTiming(t.Context(), sessionID)
require.NoError(t, err)
require.NotNil(t, timing)
require.Len(t, timing.Turns, 1)
require.Len(t, timing.Turns[0].Calls, 1)
assert.Equal(t, "accepted tool", timing.Turns[0].Calls[0].ToolName)
require.NotNil(t, timing.Turns[0].Calls[0].DurationMs)
assert.Equal(t, int64(120_000), *timing.Turns[0].Calls[0].DurationMs)
}
func TestBunStoreGetSessionTimingUsesOneGuardForSubagentHydration(t *testing.T) {
raw, err := sql.Open("sqlite3", ":memory:")
require.NoError(t, err)
raw.SetMaxOpenConns(1)
t.Cleanup(func() { require.NoError(t, raw.Close()) })
store := bun.NewDB(raw, sqlitedialect.New())
require.NoError(t, CreateCommonSchema(t.Context(), store))
_, err = store.NewInsert().Model(&bunmodel.SourceArchive{
SourceArchiveID: "archive", SourceArchiveSalt: "salt",
}).Exec(t.Context())
require.NoError(t, err)
timestamp := func(hour, minute int) *bunmodel.Timestamp {
value := bunmodel.NewTimestamp(
time.Date(2026, 8, 2, hour, minute, 0, 0, time.UTC),
)
return &value
}
parentID := "timing-parent"
rows := []bunmodel.Session{
{
ID: parentID, Project: "alpha", Machine: "host", Agent: "codex",
StartedAt: timestamp(10, 0), EndedAt: timestamp(10, 10),
CreatedAt: *timestamp(10, 0), SourceArchiveID: "archive",
SourceDatabaseGeneration: "generation",
},
{
ID: "timing-child", Project: "alpha", Machine: "host", Agent: "codex",
StartedAt: timestamp(10, 2), EndedAt: timestamp(10, 4),
CreatedAt: *timestamp(10, 2), ParentSessionID: &parentID,
RelationshipType: "subagent", SourceArchiveID: "archive",
SourceDatabaseGeneration: "generation",
},
}
for index := range rows {
_, err = store.NewInsert().Model(&rows[index]).Exec(t.Context())
require.NoError(t, err)
}
messageID := int64(1)
_, err = store.NewInsert().Model(&bunmodel.Message{
ID: &messageID, SessionID: parentID, Ordinal: 0, Role: "assistant",
Content: "delegate", ContentLength: 8, Timestamp: timestamp(10, 1),
HasToolUse: true, TokenUsage: json.RawMessage(`{}`),
}).Exec(t.Context())
require.NoError(t, err)
toolID := int64(1)
subagentID := "timing-child"
_, err = store.NewInsert().Model(&bunmodel.ToolCall{
ID: &toolID, MessageID: &messageID, SessionID: parentID,
MessageOrdinal: 0, ToolName: "Task", Category: "Task",
ToolUseID: "call-child", SubagentSessionID: &subagentID,
}).Exec(t.Context())
require.NoError(t, err)
hook := new(countingQueryHook)
backend := &sessionContractBackend{store: store.WithQueryHook(hook)}
common := NewBunStore(backend)
timing, err := common.GetSessionTiming(t.Context(), parentID)
require.NoError(t, err)
require.NotNil(t, timing)
require.Len(t, timing.Turns, 1)
require.Len(t, timing.Turns[0].Calls, 1)
require.NotNil(t, timing.Turns[0].Calls[0].DurationMs)
assert.Equal(t, int64(120000), *timing.Turns[0].Calls[0].DurationMs)
assert.Equal(t, 1, backend.viewCalls)
var checkedMessages, checkedCalls, checkedEvents bool
for _, query := range hook.queries {
lower := strings.ToLower(query)
switch {
case strings.Contains(lower, `from "messages"`):
checkedMessages = true
assert.NotContains(t, lower, `"content"`)
assert.NotContains(t, lower, `"thinking_text"`)
assert.NotContains(t, lower, `"token_usage"`)
case strings.Contains(lower, `from "tool_calls"`):
checkedCalls = true
assert.NotContains(t, lower, `"result_content"`)
case strings.Contains(lower, `from "tool_result_events"`):
checkedEvents = true
assert.NotContains(t, lower, `"content"`)
}
}
assert.True(t, checkedMessages)
assert.True(t, checkedCalls)
assert.True(t, checkedEvents)
}
func TestBunStoreGetAllMessagesBatchesToolHydration(t *testing.T) {
raw, err := sql.Open("sqlite3", ":memory:")
require.NoError(t, err)
raw.SetMaxOpenConns(1)
t.Cleanup(func() { require.NoError(t, raw.Close()) })
store := bun.NewDB(raw, sqlitedialect.New())
require.NoError(t, CreateCommonSchema(t.Context(), store))
_, err = store.NewInsert().Model(&bunmodel.SourceArchive{
SourceArchiveID: "archive", SourceArchiveSalt: "salt",
}).Exec(t.Context())
require.NoError(t, err)
_, err = store.NewInsert().Model(&bunmodel.Session{
ID: "large-tool-hydration", Project: "alpha", Machine: "host", Agent: "codex",
CreatedAt: bunmodel.NewTimestamp(time.Date(2026, 8, 2, 10, 0, 0, 0, time.UTC)),
SourceArchiveID: "archive", SourceDatabaseGeneration: "generation",
}).Exec(t.Context())
require.NoError(t, err)
_, err = raw.ExecContext(t.Context(), `
WITH RECURSIVE ordinals(value) AS (
SELECT 0 UNION ALL SELECT value + 1 FROM ordinals WHERE value < 1000
)
INSERT INTO messages (
session_id, ordinal, role, content, token_usage
)
SELECT 'large-tool-hydration', value, 'assistant', '', '{}'
FROM ordinals`)
require.NoError(t, err)
hook := new(countingQueryHook)
common := NewBunStore(&sessionContractBackend{store: store.WithQueryHook(hook)})
messages, err := common.GetAllMessages(t.Context(), "large-tool-hydration")
require.NoError(t, err)
require.Len(t, messages, 1001)
assert.Equal(t, 7, hook.selects,
"one message query plus three bounded call and event batches")
}
func TestBunStoreGetSessionTimingFallsBackToEventsForMissingSubagent(
t *testing.T,
) {
raw, err := sql.Open("sqlite3", ":memory:")
require.NoError(t, err)
raw.SetMaxOpenConns(1)
t.Cleanup(func() { require.NoError(t, raw.Close()) })
store := bun.NewDB(raw, sqlitedialect.New())
require.NoError(t, CreateCommonSchema(t.Context(), store))
_, err = store.NewInsert().Model(&bunmodel.SourceArchive{
SourceArchiveID: "archive", SourceArchiveSalt: "salt",
}).Exec(t.Context())
require.NoError(t, err)
parse := func(value string) *bunmodel.Timestamp {
parsed, parseErr := bunmodel.ParseTimestamp(value)
require.NoError(t, parseErr)
return &parsed
}
_, err = store.NewInsert().Model(&bunmodel.Session{
ID: "missing-subagent-parent", Project: "alpha", Machine: "host", Agent: "codex",
StartedAt: parse("2026-08-02T10:00:00Z"),
EndedAt: parse("2026-08-02T10:02:00Z"),
CreatedAt: *parse("2026-08-02T10:00:00Z"),
SourceArchiveID: "archive", SourceDatabaseGeneration: "generation",
}).Exec(t.Context())
require.NoError(t, err)
messageID := int64(1)
_, err = store.NewInsert().Model(&bunmodel.Message{
ID: &messageID, SessionID: "missing-subagent-parent", Ordinal: 0,
Role: "assistant", Content: "delegate", ContentLength: 8,
Timestamp: parse("2026-08-02T10:01:00Z"), HasToolUse: true,
TokenUsage: json.RawMessage(`{}`),
}).Exec(t.Context())
require.NoError(t, err)
missingSubagent := "missing-child"
_, err = store.NewInsert().Model(&bunmodel.ToolCall{
SessionID: "missing-subagent-parent", MessageOrdinal: 0,
ToolName: "Task", Category: "Task", ToolUseID: "call-missing",
SubagentSessionID: &missingSubagent,
}).Exec(t.Context())
require.NoError(t, err)
for index, event := range []struct {
status, at string
}{
{status: "started", at: "2026-08-02T10:01:05Z"},
{status: "completed", at: "2026-08-02T10:01:20Z"},
} {
_, err = store.NewInsert().Model(&bunmodel.ToolResultEvent{
SessionID: "missing-subagent-parent", ToolCallMessageOrdinal: 0,
CallIndex: 0, Source: "tool_execution", Status: event.status,
Timestamp: parse(event.at), EventIndex: index,
}).Exec(t.Context())
require.NoError(t, err)
}
timing, err := NewBunStore(&sessionContractBackend{store: store}).
GetSessionTiming(t.Context(), "missing-subagent-parent")
require.NoError(t, err)
require.NotNil(t, timing)
require.Len(t, timing.Turns, 1)
require.Len(t, timing.Turns[0].Calls, 1)
require.NotNil(t, timing.Turns[0].Calls[0].DurationMs)
assert.Equal(t, int64(15000), *timing.Turns[0].Calls[0].DurationMs)
require.NotNil(t, timing.Turns[0].DurationMs)
assert.Equal(t, int64(20000), *timing.Turns[0].DurationMs)
}
+359
View File
@@ -0,0 +1,359 @@
package db
import (
"context"
"database/sql"
"fmt"
"slices"
"time"
"github.com/uptrace/bun"
"go.kenn.io/agentsview/internal/db/bunmodel"
)
const bunMutationBatchSize = 400
// RenameSession sets or clears the user-owned display name of an active
// session.
func (s *BunStore) RenameSession(id string, displayName *string) error {
ctx := context.Background()
err := s.update(ctx, WriteSessionManagement, func(store bun.IDB) error {
query := store.NewUpdate().Model((*bunmodel.Session)(nil)).
Set("display_name = ?", displayName).
Where("id = ?", id).
Where("deleted_at IS NULL")
s.backend.Capabilities().SessionMutations.ApplyTouch(
query, bunmodel.NewTimestamp(time.Now().UTC()),
)
if _, err := query.Exec(ctx); err != nil {
return fmt.Errorf("renaming session %s: %w", id, err)
}
return nil
})
return err
}
// SoftDeleteSession moves an active session to user trash. A recoverable
// source-missing tombstone becomes user trash so a later source return cannot
// revive a session the user explicitly removed.
func (s *BunStore) SoftDeleteSession(id string) error {
ctx := context.Background()
err := s.update(ctx, WriteSessionManagement, func(store bun.IDB) error {
now := bunmodel.NewTimestamp(time.Now().UTC())
query := store.NewUpdate().Model((*bunmodel.Session)(nil)).
Set("deletion_cause = NULL").
Where("id = ?", id).
Where("(deleted_at IS NULL OR deletion_cause = ?)", deletionCauseSourceMissing)
s.backend.Capabilities().SessionMutations.ApplySoftDelete(query, now)
if _, err := query.Exec(ctx); err != nil {
return fmt.Errorf("soft deleting session %s: %w", id, err)
}
return nil
})
return err
}
// SoftDeleteSessions moves multiple active or source-missing sessions to user
// trash in one transaction and returns the number changed.
func (s *BunStore) SoftDeleteSessions(ids []string) (int, error) {
ctx := context.Background()
total := 0
err := s.update(ctx, WriteSessionManagement, func(store bun.IDB) error {
if len(ids) == 0 {
return nil
}
transactionTotal := 0
err := store.RunInTx(ctx, nil, func(ctx context.Context, tx bun.Tx) error {
now := bunmodel.NewTimestamp(time.Now().UTC())
for start := 0; start < len(ids); start += bunMutationBatchSize {
end := min(start+bunMutationBatchSize, len(ids))
query := tx.NewUpdate().Model((*bunmodel.Session)(nil)).
Set("deletion_cause = NULL").
Where("id IN (?)", bun.List(ids[start:end])).
Where("(deleted_at IS NULL OR deletion_cause = ?)", deletionCauseSourceMissing)
s.backend.Capabilities().SessionMutations.ApplySoftDelete(query, now)
result, err := query.Exec(ctx)
if err != nil {
return fmt.Errorf("soft deleting sessions: %w", err)
}
count, err := result.RowsAffected()
if err != nil {
return fmt.Errorf("counting soft deleted sessions: %w", err)
}
transactionTotal += int(count)
}
return nil
})
if err == nil {
total = transactionTotal
}
return err
})
if err != nil {
return 0, err
}
return total, nil
}
// RestoreSession restores one user-trashed session and invalidates source
// freshness so changes made while it was trashed are parsed. Source-missing
// tombstones remain protected for watcher reconciliation.
func (s *BunStore) RestoreSession(id string) (int64, error) {
ctx := context.Background()
var restored int64
err := s.update(ctx, WriteSessionManagement, func(store bun.IDB) error {
var transactionRestored int64
err := store.RunInTx(ctx, nil, func(ctx context.Context, tx bun.Tx) error {
query := tx.NewUpdate().Model((*bunmodel.Session)(nil)).
Set("deleted_at = NULL").
Set("deletion_cause = NULL").
Set("data_version = ?", max(CurrentDataVersion()-1, 0)).
Where("id = ?", id).
Where("deleted_at IS NOT NULL").
Where("deletion_cause IS NULL")
s.backend.Capabilities().SessionMutations.ApplyTouch(
query, bunmodel.NewTimestamp(time.Now().UTC()),
)
result, err := query.Exec(ctx)
if err != nil {
return fmt.Errorf("restoring session %s: %w", id, err)
}
transactionRestored, err = result.RowsAffected()
if err != nil {
return fmt.Errorf("counting restored session %s: %w", id, err)
}
if transactionRestored > 0 {
return s.backend.Capabilities().SessionMutations.AfterRestore(ctx, tx, id)
}
return nil
})
if err == nil {
restored = transactionRestored
}
return err
})
if err != nil {
return 0, err
}
return restored, nil
}
// ListTrashedSessions returns user-trashed sessions newest first. Recoverable
// source-missing tombstones are not user trash.
func (s *BunStore) ListTrashedSessions(ctx context.Context) ([]Session, error) {
sessions := []Session{}
err := s.view(ctx, func(store bun.IDB) error {
var rows []bunmodel.Session
query := store.NewSelect().Model(&rows).
Where("deleted_at IS NOT NULL").
Where("deletion_cause IS NULL")
orderExpr := "deleted_at"
if timestampOrder := s.backend.SessionQueryDialect().timestampOrderExpr; timestampOrder != nil {
orderExpr = timestampOrder("deleted_at")
}
if err := query.OrderExpr(orderExpr + " DESC").
OrderExpr("id ASC").Limit(500).Scan(ctx); err != nil {
return fmt.Errorf("listing trashed sessions: %w", err)
}
sessions = make([]Session, 0, len(rows))
for _, row := range rows {
sessions = append(sessions, visibleSessionFromBunRow(row))
}
return nil
})
if err != nil {
return nil, err
}
return sessions, nil
}
// DeleteSessionIfTrashed permanently deletes one user-trashed session and
// records its canonical and alias identities as exclusions.
func (s *BunStore) DeleteSessionIfTrashed(id string) (int64, error) {
ctx := context.Background()
var deleted int64
err := s.update(ctx, WriteSessionManagement, func(store bun.IDB) error {
var transactionDeleted int64
err := store.RunInTx(ctx, nil, func(ctx context.Context, tx bun.Tx) error {
result, err := tx.NewUpdate().Model((*bunmodel.Session)(nil)).
Set("deleted_at = deleted_at").
Where("id = ?", id).
Where("deleted_at IS NOT NULL").
Where("deletion_cause IS NULL").Exec(ctx)
if err != nil {
return fmt.Errorf("locking trashed session %s: %w", id, err)
}
locked, err := result.RowsAffected()
if err != nil {
return fmt.Errorf("counting locked trashed session %s: %w", id, err)
}
if locked == 0 {
return nil
}
rows, err := selectUserTrashRows(ctx, tx, "id = ?", id)
if err != nil {
return fmt.Errorf("loading trashed session %s: %w", id, err)
}
if err := s.permanentlyDeleteSessionRows(ctx, tx, rows); err != nil {
return fmt.Errorf("permanently deleting trashed session %s: %w", id, err)
}
transactionDeleted = int64(len(rows))
return nil
})
if err == nil {
deleted = transactionDeleted
}
return err
})
if err != nil {
return 0, err
}
return deleted, nil
}
// EmptyTrash permanently deletes every user-trashed session and records its
// canonical and alias identities as exclusions.
func (s *BunStore) EmptyTrash() (int, error) {
ctx := context.Background()
deleted := 0
err := s.update(ctx, WriteSessionManagement, func(store bun.IDB) error {
transactionDeleted := 0
err := store.RunInTx(ctx, nil, func(ctx context.Context, tx bun.Tx) error {
if _, err := tx.NewUpdate().Model((*bunmodel.Session)(nil)).
Set("deleted_at = deleted_at").
Where("deleted_at IS NOT NULL").
Where("deletion_cause IS NULL").Exec(ctx); err != nil {
return fmt.Errorf("locking trashed sessions: %w", err)
}
rows, err := selectUserTrashRows(ctx, tx, "1 = 1")
if err != nil {
return fmt.Errorf("loading trashed sessions: %w", err)
}
if err := s.permanentlyDeleteSessionRows(ctx, tx, rows); err != nil {
return fmt.Errorf("emptying trash: %w", err)
}
transactionDeleted = len(rows)
return nil
})
if err == nil {
deleted = transactionDeleted
}
return err
})
if err != nil {
return 0, err
}
return deleted, nil
}
func selectUserTrashRows(
ctx context.Context, store bun.IDB, where string, args ...any,
) ([]bunmodel.Session, error) {
var rows []bunmodel.Session
err := store.NewSelect().Model(&rows).
Column("id", "agent", "file_path").
Where(where, args...).
Where("deleted_at IS NOT NULL").
Where("deletion_cause IS NULL").
Scan(ctx)
return rows, err
}
func (s *BunStore) permanentlyDeleteSessionRows(
ctx context.Context, store bun.Tx, rows []bunmodel.Session,
) error {
if len(rows) == 0 {
return nil
}
selectedIDs := make([]string, 0, len(rows))
for _, row := range rows {
selectedIDs = append(selectedIDs, row.ID)
}
excludedIDs, err := connectedSessionAliasIDs(ctx, store, selectedIDs)
if err != nil {
return fmt.Errorf("loading session aliases: %w", err)
}
for _, row := range rows {
filePath := sql.NullString{}
if row.FilePath != nil {
filePath = sql.NullString{String: *row.FilePath, Valid: true}
}
if aliasID := vibeFallbackAliasID(row.ID, row.Agent, filePath); aliasID != "" {
excludedIDs = appendUniqueString(excludedIDs, aliasID)
}
}
now := bunmodel.NewTimestamp(time.Now().UTC())
for start := 0; start < len(excludedIDs); start += bunMutationBatchSize {
end := min(start+bunMutationBatchSize, len(excludedIDs))
exclusions := make([]bunmodel.ExcludedSession, 0, end-start)
for _, id := range excludedIDs[start:end] {
exclusions = append(exclusions, bunmodel.ExcludedSession{ID: id, CreatedAt: now})
}
if _, err := store.NewInsert().Model(&exclusions).
On("CONFLICT (id) DO NOTHING").Exec(ctx); err != nil {
return fmt.Errorf("recording excluded session ids: %w", err)
}
}
if err := s.backend.Capabilities().SessionMutations.BeforeDelete(
ctx, store, excludedIDs,
); err != nil {
return err
}
for start := 0; start < len(excludedIDs); start += bunMutationBatchSize {
end := min(start+bunMutationBatchSize, len(excludedIDs))
if _, err := store.NewDelete().Model((*bunmodel.Session)(nil)).
Where("id IN (?)", bun.List(excludedIDs[start:end])).Exec(ctx); err != nil {
return fmt.Errorf("deleting excluded session rows: %w", err)
}
}
return nil
}
func connectedSessionAliasIDs(
ctx context.Context, store bun.IDB, initial []string,
) ([]string, error) {
all := make([]string, 0, len(initial))
seen := make(map[string]struct{}, len(initial))
frontier := make([]string, 0, len(initial))
for _, id := range initial {
if _, ok := seen[id]; ok || id == "" {
continue
}
seen[id] = struct{}{}
all = append(all, id)
frontier = append(frontier, id)
}
for len(frontier) > 0 {
next := []string{}
for start := 0; start < len(frontier); start += bunMutationBatchSize {
end := min(start+bunMutationBatchSize, len(frontier))
var aliases []bunmodel.SessionAlias
if err := store.NewSelect().Model(&aliases).
Where("session_id IN (?) OR alias_id IN (?)",
bun.List(frontier[start:end]), bun.List(frontier[start:end])).
Scan(ctx); err != nil {
return nil, err
}
for _, alias := range aliases {
for _, id := range []string{alias.SessionID, alias.AliasID} {
if _, ok := seen[id]; ok || id == "" {
continue
}
seen[id] = struct{}{}
all = append(all, id)
next = append(next, id)
}
}
}
frontier = next
}
return all, nil
}
func appendUniqueString(values []string, value string) []string {
if slices.Contains(values, value) {
return values
}
return append(values, value)
}
+181
View File
@@ -0,0 +1,181 @@
package db
import (
"database/sql"
"fmt"
"testing"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
)
func TestSoftDeleteSessionsReturnsZeroWhenTransactionRollsBack(t *testing.T) {
database := testDB(t)
ids := make([]string, 401)
for index := range ids {
ids[index] = fmt.Sprintf("rollback-soft-delete-%03d", index)
insertSession(t, database, ids[index], "rollback")
}
require.NoError(t, database.Update(func(tx *sql.Tx) error {
_, err := tx.Exec(`
CREATE TRIGGER fail_second_soft_delete_batch
BEFORE UPDATE OF deleted_at ON sessions
WHEN NEW.id = 'rollback-soft-delete-400'
BEGIN
SELECT RAISE(ABORT, 'forced second batch failure');
END`)
return err
}))
count, err := database.SoftDeleteSessions(ids)
require.Error(t, err)
assert.Zero(t, count)
first, readErr := database.GetSessionFull(t.Context(), ids[0])
require.NoError(t, readErr)
require.NotNil(t, first)
assert.Nil(t, first.DeletedAt)
}
func TestRestoreSessionReturnsZeroWhenTransactionRollsBack(t *testing.T) {
database := testDB(t)
insertSession(t, database, "rollback-restore", "rollback")
require.NoError(t, database.SoftDeleteSession("rollback-restore"))
require.NoError(t, database.Update(func(tx *sql.Tx) error {
if _, err := tx.Exec(`
INSERT INTO local_session_source_baselines
(session_id, machine, agent, file_path)
VALUES ('rollback-restore', 'host', 'codex', '/tmp/rollback')`); err != nil {
return err
}
_, err := tx.Exec(`
CREATE TRIGGER fail_restore_baseline_delete
BEFORE DELETE ON local_session_source_baselines
BEGIN
SELECT RAISE(ABORT, 'forced restore failure');
END`)
return err
}))
restored, err := database.RestoreSession("rollback-restore")
require.Error(t, err)
assert.Zero(t, restored)
session, readErr := database.GetSessionFull(t.Context(), "rollback-restore")
require.NoError(t, readErr)
require.NotNil(t, session)
assert.NotNil(t, session.DeletedAt)
}
func TestDeleteSessionIfTrashedReturnsZeroWhenCommitFails(t *testing.T) {
database := testDB(t)
insertSession(t, database, "rollback-delete", "rollback")
require.NoError(t, database.SoftDeleteSession("rollback-delete"))
installDeferredSessionDeleteFailure(t, database)
deleted, err := database.DeleteSessionIfTrashed("rollback-delete")
require.Error(t, err)
assert.Zero(t, deleted)
session, readErr := database.GetSessionFull(t.Context(), "rollback-delete")
require.NoError(t, readErr)
require.NotNil(t, session)
assert.NotNil(t, session.DeletedAt)
assert.False(t, database.IsSessionExcluded("rollback-delete"))
}
func TestEmptyTrashReturnsZeroWhenCommitFails(t *testing.T) {
database := testDB(t)
for _, id := range []string{"rollback-empty-a", "rollback-empty-b"} {
insertSession(t, database, id, "rollback")
require.NoError(t, database.SoftDeleteSession(id))
}
installDeferredSessionDeleteFailure(t, database)
deleted, err := database.EmptyTrash()
require.Error(t, err)
assert.Zero(t, deleted)
for _, id := range []string{"rollback-empty-a", "rollback-empty-b"} {
session, readErr := database.GetSessionFull(t.Context(), id)
require.NoError(t, readErr)
require.NotNil(t, session)
assert.NotNil(t, session.DeletedAt)
assert.False(t, database.IsSessionExcluded(id))
}
}
func TestDeleteSessionIfTrashedPreDeletesSQLiteFTSContent(t *testing.T) {
database := testDB(t)
insertSession(t, database, "fts-delete", "rollback")
insertMessages(t, database, asstMsg("fts-delete", 0, "large searchable transcript"))
require.NoError(t, database.SoftDeleteSession("fts-delete"))
require.NoError(t, database.Update(func(tx *sql.Tx) error {
_, err := tx.Exec(`
CREATE TRIGGER require_bulk_fts_delete
BEFORE DELETE ON messages
WHEN EXISTS (
SELECT 1 FROM sqlite_schema
WHERE type = 'trigger' AND name = 'messages_ad'
)
BEGIN
SELECT RAISE(ABORT, 'messages_ad still active');
END`)
return err
}))
deleted, err := database.DeleteSessionIfTrashed("fts-delete")
require.NoError(t, err)
assert.EqualValues(t, 1, deleted)
session, readErr := database.GetSessionFull(t.Context(), "fts-delete")
require.NoError(t, readErr)
assert.Nil(t, session)
}
func installDeferredSessionDeleteFailure(t *testing.T, database *DB) {
t.Helper()
require.NoError(t, database.Update(func(tx *sql.Tx) error {
for _, statement := range []string{
`CREATE TABLE mutation_commit_parent (id INTEGER PRIMARY KEY)`,
`CREATE TABLE mutation_commit_child (
parent_id INTEGER NOT NULL,
FOREIGN KEY (parent_id) REFERENCES mutation_commit_parent(id)
DEFERRABLE INITIALLY DEFERRED
)`,
`CREATE TRIGGER fail_session_delete_commit
AFTER DELETE ON sessions
BEGIN
INSERT INTO mutation_commit_child(parent_id) VALUES (1);
END`,
} {
if _, err := tx.Exec(statement); err != nil {
return err
}
}
return nil
}))
}
func TestBunStoreSessionMutationsRejectBeforeBackendUpdate(t *testing.T) {
backend := &recordingBunBackend{}
store := NewBunStore(backend)
name := "forbidden"
require.ErrorIs(t, store.RenameSession("session", &name), ErrReadOnly)
require.ErrorIs(t, store.SoftDeleteSession("session"), ErrReadOnly)
count, err := store.SoftDeleteSessions([]string{"session"})
assert.Zero(t, count)
require.ErrorIs(t, err, ErrReadOnly)
restored, err := store.RestoreSession("session")
assert.Zero(t, restored)
require.ErrorIs(t, err, ErrReadOnly)
deleted, err := store.DeleteSessionIfTrashed("session")
assert.Zero(t, deleted)
require.ErrorIs(t, err, ErrReadOnly)
emptied, err := store.EmptyTrash()
assert.Zero(t, emptied)
require.ErrorIs(t, err, ErrReadOnly)
assert.Zero(t, backend.updateCalls)
}
+308
View File
@@ -0,0 +1,308 @@
package db
import (
"context"
"database/sql"
"fmt"
"io"
"strings"
"github.com/uptrace/bun"
corerecall "go.kenn.io/agentsview/internal/recall"
)
// bunRecallCapability contains the SQLite-only Recall paths that depend on
// local FTS/vector state, evidence reconciliation, or archive-only tables.
// Public Store ownership and capability policy remain on BunStore.
type bunRecallCapability interface {
QueryRecallEntries(context.Context, RecallQuery) (RecallPage, error)
ImportAcceptedRecallEntriesJSONLWithOptions(
context.Context, io.Reader, RecallImportOptions,
) (RecallImportResult, error)
IngestEvalTrajectory(
context.Context, EvalTrajectoryIngest,
) (EvalTrajectoryIngestResult, error)
}
func (b *sqliteBunBackend) QueryRecallEntries(
ctx context.Context, query RecallQuery,
) (RecallPage, error) {
return b.store.queryRecallEntries(ctx, query)
}
func (b *sqliteBunBackend) ImportAcceptedRecallEntriesJSONLWithOptions(
ctx context.Context, reader io.Reader, options RecallImportOptions,
) (RecallImportResult, error) {
return b.store.importAcceptedRecallEntriesJSONLWithOptions(ctx, reader, options)
}
func (b *sqliteBunBackend) IngestEvalTrajectory(
ctx context.Context, input EvalTrajectoryIngest,
) (EvalTrajectoryIngestResult, error) {
return b.store.ingestEvalTrajectory(ctx, input)
}
func (s *BunStore) recallCapability() (bunRecallCapability, error) {
if !s.backend.Capabilities().Recall {
return nil, ErrReadOnly
}
capability, ok := s.backend.(bunRecallCapability)
if !ok {
return nil, fmt.Errorf(
"backend %s advertises Recall without a Recall capability",
s.backend.Name(),
)
}
return capability, nil
}
// GetRecallEntry returns one Recall entry and its evidence through the guarded
// Bun read handle.
func (s *BunStore) GetRecallEntry(
ctx context.Context, id string,
) (*RecallEntry, error) {
if !s.backend.Capabilities().Recall {
return nil, ErrReadOnly
}
var staged *RecallEntry
err := s.consistentView(ctx, func(store bun.IDB) error {
var entry RecallEntry
err := store.NewRaw(
"SELECT "+recallBaseCols+" FROM recall_entries WHERE id = ?", id,
).Scan(ctx, &entry)
if err == sql.ErrNoRows {
staged = nil
return nil
}
if err != nil {
return fmt.Errorf("getting recall entry %s: %w", id, err)
}
evidence, err := listRecallEvidenceBun(ctx, store, []string{id})
if err != nil {
return err
}
entry.Evidence = evidence[id]
staged = &entry
return nil
})
if err != nil {
return nil, err
}
return staged, nil
}
// ListRecallEntries returns canonical Recall rows and evidence through one
// coherent Bun view.
func (s *BunStore) ListRecallEntries(
ctx context.Context, query RecallQuery,
) ([]RecallEntry, error) {
if !s.backend.Capabilities().Recall {
return nil, ErrReadOnly
}
if err := ValidateRecallQuery(query); err != nil {
return nil, err
}
query = NormalizeRecallQuery(query)
where, args := buildRecallEntryWhere(query, false)
limit := recallLimit(query.Limit)
if query.ProbeNext {
limit++
}
args = append(args, limit)
entries := []RecallEntry{}
err := s.consistentView(ctx, func(store bun.IDB) error {
attemptEntries := []RecallEntry{}
if err := store.NewRaw(
"SELECT "+recallBaseCols+" FROM recall_entries WHERE "+where+
" ORDER BY updated_at DESC, id ASC LIMIT ?", args...,
).Scan(ctx, &attemptEntries); err != nil {
return fmt.Errorf("querying entries: %w", err)
}
if len(attemptEntries) == 0 {
entries = attemptEntries
return nil
}
evidence, err := listRecallEvidenceBun(ctx, store, recallIDs(attemptEntries))
if err != nil {
return err
}
for index := range attemptEntries {
attemptEntries[index].Evidence = evidence[attemptEntries[index].ID]
}
entries = attemptEntries
return nil
})
if err != nil {
return nil, err
}
if len(entries) == 0 {
return nil, nil
}
return entries, nil
}
// QueryRecallEntries dispatches to SQLite's local FTS/vector capability after
// the common Store has enforced Recall availability.
func (s *BunStore) QueryRecallEntries(
ctx context.Context, query RecallQuery,
) (RecallPage, error) {
capability, err := s.recallCapability()
if err != nil {
return RecallPage{}, err
}
return capability.QueryRecallEntries(ctx, query)
}
// InsertRecallEntry writes one entry and its evidence through the guarded Bun
// transaction.
func (s *BunStore) InsertRecallEntry(entry RecallEntry) (string, error) {
ctx := context.Background()
var id string
err := s.update(ctx, WriteRecall, func(store bun.IDB) error {
if err := normalizeRecallEntryReviewState(&entry); err != nil {
return err
}
if entry.ID == "" {
return fmt.Errorf("recall entry id is required")
}
if entry.Status == "" {
entry.Status = corerecall.StatusAccepted
}
if err := store.RunInTx(ctx, nil, func(ctx context.Context, tx bun.Tx) error {
return insertRecallEntryTx(ctx, tx, entry)
}); err != nil {
return fmt.Errorf("inserting recall entry: %w", err)
}
id = entry.ID
return nil
})
return id, err
}
// RecordRecallQueryEvent inserts a completed query snapshot and exposures
// atomically through Bun.
func (s *BunStore) RecordRecallQueryEvent(
ctx context.Context, event RecallQueryEvent,
) (string, error) {
if ctx == nil {
ctx = context.Background()
}
var id string
err := s.update(ctx, WriteRecall, func(store bun.IDB) error {
event.QueryID = strings.TrimSpace(event.QueryID)
event.Surface = strings.TrimSpace(event.Surface)
event.ScorePolicyVersion = strings.TrimSpace(event.ScorePolicyVersion)
if event.Surface == "" {
return fmt.Errorf("recall query surface is required")
}
if event.FiltersJSON == "" {
event.FiltersJSON = "{}"
}
if event.ScorePolicyVersion == "" {
event.ScorePolicyVersion = RecallLexicalScorePolicyVersion
}
if event.QueryID == "" {
generated, err := newUUIDv4()
if err != nil {
return fmt.Errorf("generating recall query id: %w", err)
}
event.QueryID = generated
}
if err := store.RunInTx(ctx, nil, func(ctx context.Context, tx bun.Tx) error {
if _, err := tx.ExecContext(ctx, `
INSERT INTO recall_query_events (
id, query_text, surface, filters_json, trusted_only,
score_policy_version, result_count, packed_count,
top_score, miss_reason
) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?)`,
event.QueryID, event.Query, event.Surface, event.FiltersJSON,
event.TrustedOnly, event.ScorePolicyVersion, event.ResultCount,
event.PackedCount, event.TopScore, event.MissReason,
); err != nil {
return fmt.Errorf("inserting recall query event: %w", err)
}
for start := 0; start < len(event.Exposures); start += recallExposureInsertBatchSize {
end := min(start+recallExposureInsertBatchSize, len(event.Exposures))
batch := event.Exposures[start:end]
if err := insertRecallQueryExposureBatch(
ctx, tx, event.QueryID, batch,
); err != nil {
return fmt.Errorf(
"inserting recall query exposure ranks %d through %d: %w",
batch[0].Rank, batch[len(batch)-1].Rank, err,
)
}
}
return nil
}); err != nil {
return fmt.Errorf("recording recall query event: %w", err)
}
id = event.QueryID
return nil
})
return id, err
}
// ImportAcceptedRecallEntriesJSONL imports accepted probe results through the
// SQLite Recall capability.
func (s *BunStore) ImportAcceptedRecallEntriesJSONL(
ctx context.Context, reader io.Reader,
) (RecallImportResult, error) {
return s.ImportAcceptedRecallEntriesJSONLWithOptions(
ctx, reader, RecallImportOptions{},
)
}
func (s *BunStore) ImportAcceptedRecallEntriesJSONLWithOptions(
ctx context.Context, reader io.Reader, options RecallImportOptions,
) (RecallImportResult, error) {
if !options.DryRun && !s.backend.Capabilities().AllowsWrite(WriteRecall) {
return RecallImportResult{}, ErrReadOnly
}
capability, err := s.recallCapability()
if err != nil {
return RecallImportResult{}, err
}
return capability.ImportAcceptedRecallEntriesJSONLWithOptions(
ctx, reader, options,
)
}
// IngestEvalTrajectory routes FTS-indexed eval ingestion through the local
// Recall capability after the common write-policy check.
func (s *BunStore) IngestEvalTrajectory(
ctx context.Context, input EvalTrajectoryIngest,
) (EvalTrajectoryIngestResult, error) {
if !s.backend.Capabilities().AllowsWrite(WriteRecall) {
return EvalTrajectoryIngestResult{}, ErrReadOnly
}
capability, err := s.recallCapability()
if err != nil {
return EvalTrajectoryIngestResult{}, err
}
return capability.IngestEvalTrajectory(ctx, input)
}
func listRecallEvidenceBun(
ctx context.Context, store bun.IDB, ids []string,
) (map[string][]RecallEvidence, error) {
result := make(map[string][]RecallEvidence, len(ids))
for start := 0; start < len(ids); start += bunMutationBatchSize {
end := min(start+bunMutationBatchSize, len(ids))
var evidence []RecallEvidence
if err := store.NewRaw(`
SELECT id, entry_id, session_id, message_start_ordinal,
message_end_ordinal, message_start_source_uuid,
message_end_source_uuid, content_digest, tool_use_id, snippet
FROM recall_evidence
WHERE entry_id IN (?)
ORDER BY entry_id ASC, id ASC`, bun.List(ids[start:end])).
Scan(ctx, &evidence); err != nil {
return nil, fmt.Errorf("querying recall evidence: %w", err)
}
for _, row := range evidence {
result[row.EntryID] = append(result[row.EntryID], row)
}
}
return result, nil
}
+116
View File
@@ -0,0 +1,116 @@
package db
import (
"context"
"database/sql"
"testing"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
"github.com/uptrace/bun"
)
type replayingRecallBackend struct {
first bun.IDB
second bun.IDB
}
func (*replayingRecallBackend) Name() string { return "replaying-recall" }
func (*replayingRecallBackend) ReadOnly() bool { return true }
func (*replayingRecallBackend) Capabilities() BackendCapabilities {
return BackendCapabilities{Recall: true}
}
func (*replayingRecallBackend) SessionQueryDialect() QueryDialect {
return PortableBunSessionQueryDialect()
}
func (*replayingRecallBackend) SessionVersion(
context.Context, bun.IDB, string,
) (int, int64, error) {
return 0, 0, sql.ErrNoRows
}
func (b *replayingRecallBackend) View(
_ context.Context, fn func(bun.IDB) error,
) error {
return fn(b.second)
}
func (b *replayingRecallBackend) ConsistentView(
_ context.Context, fn func(bun.IDB) error,
) error {
if err := fn(b.first); err != nil {
return err
}
return fn(b.second)
}
func (*replayingRecallBackend) Update(
context.Context, func(bun.IDB) error,
) error {
return ErrReadOnly
}
func TestGetRecallEntryPublishesOnlyAcceptedReplayAttempt(t *testing.T) {
first := testDB(t)
second := testDB(t)
require.NoError(t, first.UpsertSession(Session{
ID: "replayed-source", Project: "recall", Machine: "host", Agent: "codex",
}))
_, err := first.InsertRecallEntry(RecallEntry{
ID: "replayed-entry", Type: "fact", Scope: "global",
Title: "Rejected first attempt", Body: "This row disappears before retry.",
SourceSessionID: "replayed-source", Transferable: true, ProvenanceOK: true,
})
require.NoError(t, err)
store := NewBunStore(&replayingRecallBackend{
first: first.bunReader, second: second.bunReader,
})
entry, err := store.GetRecallEntry(t.Context(), "replayed-entry")
require.NoError(t, err)
assert.Nil(t, entry)
}
func TestListRecallEntriesPublishesOnlyAcceptedReplayAttempt(t *testing.T) {
first := testDB(t)
second := testDB(t)
for _, database := range []*DB{first, second} {
require.NoError(t, database.UpsertSession(Session{
ID: "replayed-list-source", Project: "recall", Machine: "host", Agent: "codex",
}))
}
_, err := first.InsertRecallEntry(RecallEntry{
ID: "replayed-list", Type: "fact", Scope: "global",
Title: "Rejected list title", Body: "Rejected list body.",
SourceSessionID: "replayed-list-source", Transferable: true, ProvenanceOK: true,
Evidence: []RecallEvidence{{
SessionID: "replayed-list-source", MessageStartOrdinal: 1,
MessageEndOrdinal: 1, Snippet: "rejected evidence",
}},
})
require.NoError(t, err)
_, err = second.InsertRecallEntry(RecallEntry{
ID: "replayed-list", Type: "fact", Scope: "global",
Title: "Accepted list title", Body: "Accepted list body.",
SourceSessionID: "replayed-list-source", Transferable: true, ProvenanceOK: true,
Evidence: []RecallEvidence{{
SessionID: "replayed-list-source", MessageStartOrdinal: 2,
MessageEndOrdinal: 2, Snippet: "accepted evidence",
}},
})
require.NoError(t, err)
store := NewBunStore(&replayingRecallBackend{
first: first.bunReader, second: second.bunReader,
})
entries, err := store.ListRecallEntries(t.Context(), RecallQuery{})
require.NoError(t, err)
require.Len(t, entries, 1)
assert.Equal(t, "Accepted list title", entries[0].Title)
require.Len(t, entries[0].Evidence, 1)
assert.Equal(t, "accepted evidence", entries[0].Evidence[0].Snippet)
}
+156
View File
@@ -0,0 +1,156 @@
package db
import (
"context"
"fmt"
"strings"
"github.com/uptrace/bun"
"go.kenn.io/agentsview/internal/db/bunmodel"
)
type bunRecentEditProjection struct {
Project string `bun:"project"`
FilePath string `bun:"file_path"`
EditCount int `bun:"edit_count"`
LastEditedAt *bunmodel.Timestamp `bun:"last_edited_at"`
LastSessionID string `bun:"last_session_id"`
SessionID string `bun:"session_id"`
Ordinal int `bun:"ordinal"`
ToolUseID string `bun:"tool_use_id"`
CallIndex int `bun:"call_index"`
ToolName string `bun:"tool_name"`
Category string `bun:"category"`
Timestamp *bunmodel.Timestamp `bun:"timestamp"`
}
func (s *BunStore) RecentEdits(
ctx context.Context, params RecentEditsParams,
) (RecentEditsResult, error) {
params = NormalizeRecentEditsParams(params)
var result RecentEditsResult
err := s.consistentView(ctx, func(store bun.IDB) error {
attempt, err := s.recentEditsFrom(ctx, store, params)
if err != nil {
return err
}
result = attempt
return nil
})
return result, err
}
func (s *BunStore) recentEditsFrom(
ctx context.Context, store bun.IDB, params RecentEditsParams,
) (RecentEditsResult, error) {
timestampExpr := "m.timestamp"
sortExpr := timestampExpr
if s.backend.SessionQueryDialect().timestampOrderExpr != nil {
timestampExpr = "NULLIF(m.timestamp, '')"
sortExpr = "julianday(" + timestampExpr + ")"
}
predicates := []string{
"s.deleted_at IS NULL",
"tc.category IN ('Edit', 'Write')",
"tc.file_path IS NOT NULL",
"TRIM(tc.file_path) != ''",
}
var args []any
if params.Project != "" {
predicates = append(predicates, "s.project = ?")
args = append(args, params.Project)
}
if params.Search != "" {
predicates = append(predicates,
"LOWER(tc.file_path) LIKE ? ESCAPE '\\'")
args = append(args,
"%"+EscapeLikePattern(strings.ToLower(params.Search))+"%")
}
query := fmt.Sprintf(`
WITH edit_rows AS (
SELECT s.project, tc.file_path,
COUNT(*) OVER (
PARTITION BY s.project, tc.file_path
) AS edit_count,
s.id AS session_id,
tc.message_ordinal AS ordinal,
tc.tool_use_id, tc.call_index, tc.tool_name, tc.category,
%[1]s AS timestamp,
%[2]s AS edit_sort,
ROW_NUMBER() OVER (
PARTITION BY s.project, tc.file_path
ORDER BY %[2]s DESC NULLS LAST, s.id DESC,
tc.message_ordinal DESC, tc.call_index DESC
) AS edit_rank
FROM tool_calls AS tc
JOIN sessions AS s ON s.id = tc.session_id
LEFT JOIN messages AS m
ON m.session_id = tc.session_id
AND m.ordinal = tc.message_ordinal
WHERE %[3]s
),
paged_files AS (
SELECT project, file_path, edit_count,
timestamp AS last_edited_at,
session_id AS last_session_id,
ordinal AS last_ordinal,
call_index AS last_call_index, edit_sort
FROM edit_rows
WHERE edit_rank = 1
ORDER BY edit_sort DESC NULLS LAST, session_id DESC,
ordinal DESC, call_index DESC, file_path DESC
LIMIT ? OFFSET ?
)
SELECT page.project, page.file_path, page.edit_count,
page.last_edited_at, page.last_session_id,
edit.session_id, edit.ordinal, edit.tool_use_id,
edit.call_index, edit.tool_name, edit.category, edit.timestamp
FROM paged_files AS page
JOIN edit_rows AS edit
ON edit.project = page.project AND edit.file_path = page.file_path
WHERE edit.edit_rank <= ?
ORDER BY page.edit_sort DESC NULLS LAST, page.last_session_id DESC,
page.last_ordinal DESC, page.last_call_index DESC,
page.file_path DESC, edit.edit_rank ASC`,
timestampExpr, sortExpr, strings.Join(predicates, " AND "),
)
args = append(args, params.Limit+1, params.Offset, params.MaxEditsPerFile)
var rows []bunRecentEditProjection
if err := store.NewRaw(query, args...).Scan(ctx, &rows); err != nil {
return RecentEditsResult{}, fmt.Errorf("querying Bun recent edits: %w", err)
}
return buildBunRecentEditPage(rows, params), nil
}
func buildBunRecentEditPage(
rows []bunRecentEditProjection, params RecentEditsParams,
) RecentEditsResult {
files := []RecentEditFile{}
indices := make(map[string]int)
for _, row := range rows {
key := row.Project + "\x00" + row.FilePath
index, ok := indices[key]
if !ok {
files = append(files, RecentEditFile{
Project: row.Project, FilePath: row.FilePath,
EditCount: row.EditCount,
LastEditedAt: bunAnalyticsTimeString(row.LastEditedAt),
LastSessionID: row.LastSessionID, Edits: []RecentEdit{},
EditsTruncated: row.EditCount > params.MaxEditsPerFile,
})
index = len(files) - 1
indices[key] = index
}
files[index].Edits = append(files[index].Edits, RecentEdit{
SessionID: row.SessionID, Ordinal: row.Ordinal,
ToolUseID: row.ToolUseID, CallIndex: row.CallIndex,
ToolName: row.ToolName, Category: row.Category,
Timestamp: bunAnalyticsTimeString(row.Timestamp),
})
}
hasMore := len(files) > params.Limit
if hasMore {
files = files[:params.Limit]
}
return RecentEditsResult{Files: files, HasMore: hasMore}
}
+320
View File
@@ -0,0 +1,320 @@
package db
import (
"encoding/json"
"fmt"
"time"
"go.kenn.io/agentsview/internal/db/bunmodel"
)
func timestampToBunRow(value string) (*bunmodel.Timestamp, error) {
if value == "" {
return nil, nil
}
timestamp, err := bunmodel.ParseTimestamp(value)
if err != nil {
return nil, err
}
return &timestamp, nil
}
func timestampPtrToBunRow(value *string) (*bunmodel.Timestamp, error) {
if value == nil {
return nil, nil
}
return timestampToBunRow(*value)
}
func timestampFromBunRow(value *bunmodel.Timestamp) *string {
if value == nil || value.IsZero() {
return nil
}
formatted := value.Time.UTC().Format(time.RFC3339Nano)
return &formatted
}
func requiredTimestampToBunRow(value string) (bunmodel.Timestamp, error) {
timestamp, err := timestampToBunRow(value)
if err != nil {
return bunmodel.Timestamp{}, err
}
if timestamp == nil {
return bunmodel.Timestamp{}, fmt.Errorf("timestamp is empty")
}
return *timestamp, nil
}
func requiredTimestampFromBunRow(value bunmodel.Timestamp) string {
if value.IsZero() {
return ""
}
return value.Time.UTC().Format(time.RFC3339Nano)
}
func sessionToBunRow(session Session) (bunmodel.Session, error) {
row := bunmodel.Session{
ID: session.ID,
Project: session.Project,
Machine: session.Machine,
Agent: session.Agent,
AgentLabel: session.AgentLabel,
Entrypoint: session.Entrypoint,
SessionKind: session.SessionKind,
FirstMessage: session.FirstMessage,
DisplayName: session.DisplayName,
SessionName: session.SessionName,
MessageCount: session.MessageCount,
UserMessageCount: session.UserMessageCount,
ParentSessionID: session.ParentSessionID,
ParserParentSessionID: session.ParserParentSessionID,
RelationshipType: session.RelationshipType,
TotalOutputTokens: session.TotalOutputTokens,
PeakContextTokens: session.PeakContextTokens,
HasTotalOutputTokens: session.HasTotalOutputTokens,
HasPeakContextTokens: session.HasPeakContextTokens,
IsAutomated: session.IsAutomated,
ToolFailureSignalCount: session.ToolFailureSignalCount,
ToolRetryCount: session.ToolRetryCount,
EditChurnCount: session.EditChurnCount,
ConsecutiveFailureMax: session.ConsecutiveFailureMax,
Outcome: session.Outcome,
OutcomeConfidence: session.OutcomeConfidence,
EndedWithRole: session.EndedWithRole,
FinalFailureStreak: session.FinalFailureStreak,
CompactionCount: session.CompactionCount,
MidTaskCompactionCount: session.MidTaskCompactionCount,
ContextPressureMax: session.ContextPressureMax,
HealthScore: session.HealthScore,
HealthGrade: session.HealthGrade,
HasToolCalls: session.HasToolCalls,
HasContextData: session.HasContextData,
SecretLeakCount: session.SecretLeakCount,
SecretsRulesVersion: session.SecretsRulesVersion,
QualitySignalVersion: session.QualitySignalVersion,
ShortPromptCount: session.ShortPromptCount,
UnstructuredStart: session.UnstructuredStart,
MissingSuccessCriteriaCount: session.MissingSuccessCriteriaCount,
MissingVerificationCount: session.MissingVerificationCount,
DuplicatePromptCount: session.DuplicatePromptCount,
NoCodeContextCount: session.NoCodeContextCount,
RunawayToolLoopCount: session.RunawayToolLoopCount,
DataVersion: session.DataVersion,
Cwd: session.Cwd,
GitBranch: session.GitBranch,
SourceSessionID: session.SourceSessionID,
SourceVersion: session.SourceVersion,
TranscriptFidelity: session.TranscriptFidelity,
ParserMalformedLines: session.ParserMalformedLines,
IsTruncated: session.IsTruncated,
DeletionCause: session.DeletionCause,
TerminationStatus: session.TerminationStatus,
FilePath: session.FilePath,
FileSize: session.FileSize,
FileMtime: session.FileMtime,
FileInode: session.FileInode,
FileDevice: session.FileDevice,
FileHash: session.FileHash,
TranscriptRevision: "0",
SourceArchiveID: session.SourceArchiveID,
SourceDatabaseGeneration: session.SourceDatabaseGeneration,
}
if session.TranscriptRevision != nil {
row.TranscriptRevision = *session.TranscriptRevision
}
optionalTimestamps := []struct {
name string
value *string
dest **bunmodel.Timestamp
}{
{"started_at", session.StartedAt, &row.StartedAt},
{"ended_at", session.EndedAt, &row.EndedAt},
{"signals_pending_since", session.SignalsPendingSince, &row.SignalsPendingSince},
{"deleted_at", session.DeletedAt, &row.DeletedAt},
{"local_modified_at", session.LocalModifiedAt, &row.LocalModifiedAt},
}
for _, field := range optionalTimestamps {
value, err := timestampPtrToBunRow(field.value)
if err != nil {
return bunmodel.Session{}, fmt.Errorf(
"session %q %s: %w", session.ID, field.name, err,
)
}
*field.dest = value
}
createdAt, err := requiredTimestampToBunRow(session.CreatedAt)
if err != nil {
return bunmodel.Session{}, fmt.Errorf(
"session %q created_at: %w", session.ID, err,
)
}
row.CreatedAt = createdAt
return row, nil
}
func sessionFromBunRow(row bunmodel.Session) Session {
return Session{
ID: row.ID,
Project: row.Project,
Machine: row.Machine,
Agent: row.Agent,
AgentLabel: row.AgentLabel,
Entrypoint: row.Entrypoint,
SessionKind: row.SessionKind,
FirstMessage: row.FirstMessage,
DisplayName: row.DisplayName,
SessionName: row.SessionName,
StartedAt: timestampFromBunRow(row.StartedAt),
EndedAt: timestampFromBunRow(row.EndedAt),
MessageCount: row.MessageCount,
UserMessageCount: row.UserMessageCount,
ParentSessionID: row.ParentSessionID,
ParserParentSessionID: row.ParserParentSessionID,
RelationshipType: row.RelationshipType,
TotalOutputTokens: row.TotalOutputTokens,
PeakContextTokens: row.PeakContextTokens,
HasTotalOutputTokens: row.HasTotalOutputTokens,
HasPeakContextTokens: row.HasPeakContextTokens,
IsAutomated: row.IsAutomated,
ToolFailureSignalCount: row.ToolFailureSignalCount,
ToolRetryCount: row.ToolRetryCount,
EditChurnCount: row.EditChurnCount,
ConsecutiveFailureMax: row.ConsecutiveFailureMax,
Outcome: row.Outcome,
OutcomeConfidence: row.OutcomeConfidence,
EndedWithRole: row.EndedWithRole,
FinalFailureStreak: row.FinalFailureStreak,
SignalsPendingSince: timestampFromBunRow(row.SignalsPendingSince),
CompactionCount: row.CompactionCount,
MidTaskCompactionCount: row.MidTaskCompactionCount,
ContextPressureMax: row.ContextPressureMax,
HealthScore: row.HealthScore,
HealthGrade: row.HealthGrade,
HasToolCalls: row.HasToolCalls,
HasContextData: row.HasContextData,
SecretLeakCount: row.SecretLeakCount,
SecretsRulesVersion: row.SecretsRulesVersion,
QualitySignalVersion: row.QualitySignalVersion,
ShortPromptCount: row.ShortPromptCount,
UnstructuredStart: row.UnstructuredStart,
MissingSuccessCriteriaCount: row.MissingSuccessCriteriaCount,
MissingVerificationCount: row.MissingVerificationCount,
DuplicatePromptCount: row.DuplicatePromptCount,
NoCodeContextCount: row.NoCodeContextCount,
RunawayToolLoopCount: row.RunawayToolLoopCount,
DataVersion: row.DataVersion,
Cwd: row.Cwd,
GitBranch: row.GitBranch,
SourceSessionID: row.SourceSessionID,
SourceVersion: row.SourceVersion,
TranscriptFidelity: row.TranscriptFidelity,
ParserMalformedLines: row.ParserMalformedLines,
IsTruncated: row.IsTruncated,
DeletedAt: timestampFromBunRow(row.DeletedAt),
DeletionCause: row.DeletionCause,
TerminationStatus: row.TerminationStatus,
FilePath: row.FilePath,
FileSize: row.FileSize,
FileMtime: row.FileMtime,
FileInode: row.FileInode,
FileDevice: row.FileDevice,
FileHash: row.FileHash,
LocalModifiedAt: timestampFromBunRow(row.LocalModifiedAt),
TranscriptRevision: &row.TranscriptRevision,
CreatedAt: requiredTimestampFromBunRow(row.CreatedAt),
SourceArchiveID: row.SourceArchiveID,
SourceDatabaseGeneration: row.SourceDatabaseGeneration,
}
}
func messageToBunRow(message Message) (bunmodel.Message, error) {
var id *int64
if message.ID != 0 {
id = &message.ID
}
timestamp, err := timestampToBunRow(message.Timestamp)
if err != nil {
return bunmodel.Message{}, fmt.Errorf(
"message %q ordinal %d timestamp: %w",
message.SessionID, message.Ordinal, err,
)
}
return bunmodel.Message{
ID: id,
SessionID: message.SessionID,
Ordinal: message.Ordinal,
Role: message.Role,
Content: message.Content,
ThinkingText: message.ThinkingText,
Timestamp: timestamp,
HasThinking: message.HasThinking,
HasToolUse: message.HasToolUse,
ContentLength: message.ContentLength,
IsSystem: message.IsSystem,
Model: message.Model,
TokenUsage: append(json.RawMessage(nil), message.TokenUsage...),
ContextTokens: message.ContextTokens,
OutputTokens: message.OutputTokens,
HasContextTokens: message.HasContextTokens,
HasOutputTokens: message.HasOutputTokens,
ClaudeMessageID: message.ClaudeMessageID,
ClaudeRequestID: message.ClaudeRequestID,
SourceType: message.SourceType,
SourceSubtype: message.SourceSubtype,
PromptSource: message.PromptSource,
SourceUUID: message.SourceUUID,
SourceParentUUID: message.SourceParentUUID,
IsSidechain: message.IsSidechain,
IsCompactBoundary: message.IsCompactBoundary,
}, nil
}
func messageFromBunRow(row bunmodel.Message) Message {
var id int64
if row.ID != nil {
id = *row.ID
}
return Message{
ID: id,
SessionID: row.SessionID,
Ordinal: row.Ordinal,
Role: row.Role,
Content: row.Content,
ThinkingText: row.ThinkingText,
Timestamp: requiredTimestampFromBunRowPtr(row.Timestamp),
HasThinking: row.HasThinking,
HasToolUse: row.HasToolUse,
ContentLength: row.ContentLength,
IsSystem: row.IsSystem,
Model: row.Model,
TokenUsage: append(json.RawMessage(nil), row.TokenUsage...),
ContextTokens: row.ContextTokens,
OutputTokens: row.OutputTokens,
HasContextTokens: row.HasContextTokens,
HasOutputTokens: row.HasOutputTokens,
ClaudeMessageID: row.ClaudeMessageID,
ClaudeRequestID: row.ClaudeRequestID,
SourceType: row.SourceType,
SourceSubtype: row.SourceSubtype,
PromptSource: row.PromptSource,
SourceUUID: row.SourceUUID,
SourceParentUUID: row.SourceParentUUID,
IsSidechain: row.IsSidechain,
IsCompactBoundary: row.IsCompactBoundary,
}
}
func requiredTimestampFromBunRowPtr(value *bunmodel.Timestamp) string {
formatted := timestampFromBunRow(value)
if formatted == nil {
return ""
}
return *formatted
}
+176
View File
@@ -0,0 +1,176 @@
package db
import (
"encoding/json"
"testing"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
)
func TestBunRowSessionRoundTripPreservesCanonicalFields(t *testing.T) {
first := "first"
display := "display"
sessionName := "source name"
started := "2026-08-02T12:00:00Z"
ended := "2026-08-02T12:30:00Z"
parent := "parent"
parserParent := "parser-parent"
pending := "2026-08-02T12:29:00Z"
pressure := 0.82
healthScore := 91
healthGrade := "A"
deleted := "2026-08-03T09:00:00Z"
deletionCause := "source_missing"
termination := "clean"
filePath := "/fixture/session.jsonl"
fileSize := int64(4096)
fileMtime := int64(1_754_131_800)
fileInode := int64(22)
fileDevice := int64(7)
fileHash := "sha256:fixture"
localModified := "2026-08-02T12:31:00Z"
revision := "revision-4"
want := Session{
ID: "session-1", Project: "project", Machine: "machine", Agent: "claude",
AgentLabel: "Claude", Entrypoint: "cli", SessionKind: "interactive",
FirstMessage: &first, DisplayName: &display, SessionName: &sessionName,
StartedAt: &started, EndedAt: &ended, MessageCount: 12, UserMessageCount: 5,
ParentSessionID: &parent, ParserParentSessionID: &parserParent,
RelationshipType: "subagent", TotalOutputTokens: 900, PeakContextTokens: 1800,
HasTotalOutputTokens: true, HasPeakContextTokens: true, IsAutomated: true,
ToolFailureSignalCount: 2, ToolRetryCount: 3, EditChurnCount: 4,
ConsecutiveFailureMax: 5, Outcome: "success", OutcomeConfidence: "high",
EndedWithRole: "assistant", FinalFailureStreak: 1, SignalsPendingSince: &pending,
CompactionCount: 6, MidTaskCompactionCount: 2, ContextPressureMax: &pressure,
HealthScore: &healthScore, HealthGrade: &healthGrade, HasToolCalls: true,
HasContextData: true, SecretLeakCount: 1, SecretsRulesVersion: "rules-3",
QualitySignalVersion: 3, ShortPromptCount: 2, UnstructuredStart: true,
MissingSuccessCriteriaCount: 1, MissingVerificationCount: 2,
DuplicatePromptCount: 3, NoCodeContextCount: 4, RunawayToolLoopCount: 5,
DataVersion: 8, Cwd: "/fixture/project", GitBranch: "feature/test",
SourceSessionID: "source-session", SourceVersion: "1.2",
TranscriptFidelity: "exact", ParserMalformedLines: 2, IsTruncated: true,
DeletedAt: &deleted, DeletionCause: &deletionCause, TerminationStatus: &termination,
FilePath: &filePath, FileSize: &fileSize, FileMtime: &fileMtime,
FileInode: &fileInode, FileDevice: &fileDevice, FileHash: &fileHash,
LocalModifiedAt: &localModified, TranscriptRevision: &revision,
CreatedAt: "2026-08-02T11:59:00Z",
SourceArchiveID: "archive-1", SourceDatabaseGeneration: "database-7",
}
row, err := sessionToBunRow(want)
require.NoError(t, err)
got := sessionFromBunRow(row)
assert.Equal(t, want, got)
}
func TestBunRowSessionConversionRejectsMalformedOptionalTimestamp(t *testing.T) {
malformed := "not-a-timestamp"
_, err := sessionToBunRow(Session{
ID: "session-1", CreatedAt: "2026-08-02T11:59:00Z",
StartedAt: &malformed,
})
require.Error(t, err)
assert.Contains(t, err.Error(), "started_at")
assert.Contains(t, err.Error(), malformed)
}
func TestBunRowSessionConversionRejectsMalformedRequiredTimestamp(t *testing.T) {
_, err := sessionToBunRow(Session{
ID: "session-1", CreatedAt: "not-a-created-at",
})
require.Error(t, err)
assert.Contains(t, err.Error(), "created_at")
assert.Contains(t, err.Error(), "not-a-created-at")
}
func TestBunRowMessageRoundTripPreservesJSONAndOptionalID(t *testing.T) {
for name, id := range map[string]int64{"source id": 41, "missing source id": 0} {
t.Run(name, func(t *testing.T) {
want := Message{
ID: id, SessionID: "session-1", Ordinal: 7, Role: "assistant",
Content: "answer", ThinkingText: "reasoning",
Timestamp: "2026-08-02T12:30:00Z", HasThinking: true,
HasToolUse: true, ContentLength: 6, Model: "model-1",
TokenUsage: json.RawMessage(`{"input_tokens":12,"output_tokens":7}`),
ContextTokens: 1200, OutputTokens: 7, HasContextTokens: true,
HasOutputTokens: true, ClaudeMessageID: "message-id",
ClaudeRequestID: "request-id", IsSystem: true, SourceType: "event",
SourceSubtype: "assistant", PromptSource: "user", SourceUUID: "uuid-1",
SourceParentUUID: "uuid-0", IsSidechain: true, IsCompactBoundary: true,
}
row, err := messageToBunRow(want)
require.NoError(t, err)
if id == 0 {
assert.Nil(t, row.ID)
} else {
require.NotNil(t, row.ID)
assert.Equal(t, id, *row.ID)
}
assert.Equal(t, want, messageFromBunRow(row))
})
}
}
func TestBunRowMessageConversionRejectsMalformedTimestamp(t *testing.T) {
_, err := messageToBunRow(Message{
SessionID: "session-1", Ordinal: 7, Timestamp: "malformed",
})
require.Error(t, err)
assert.Contains(t, err.Error(), "timestamp")
assert.Contains(t, err.Error(), "malformed")
}
func TestBunRowPriorSQLiteMessageKeyAliasRemainsReadable(t *testing.T) {
d := testDB(t)
rows, err := d.getReader().Query(`PRAGMA table_info(messages)`)
require.NoError(t, err)
defer rows.Close()
primaryKeys := make(map[string]int)
for rows.Next() {
var cid, notNull, primaryKey int
var name, columnType string
var defaultValue any
require.NoError(t, rows.Scan(
&cid, &name, &columnType, &notNull, &defaultValue, &primaryKey,
))
if primaryKey != 0 {
primaryKeys[name] = primaryKey
}
}
require.NoError(t, rows.Err())
assert.Equal(t, map[string]int{"id": 1}, primaryKeys)
indexRows, err := d.getReader().Query(`PRAGMA index_list(messages)`)
require.NoError(t, err)
defer indexRows.Close()
foundCompositeAlias := false
for indexRows.Next() {
var seq, unique, partial int
var name, origin string
require.NoError(t, indexRows.Scan(&seq, &name, &unique, &origin, &partial))
if unique == 0 {
continue
}
columnRows, queryErr := d.getReader().Query(
`SELECT name FROM pragma_index_info(?) ORDER BY seqno`, name,
)
require.NoError(t, queryErr)
var columns []string
for columnRows.Next() {
var column string
require.NoError(t, columnRows.Scan(&column))
columns = append(columns, column)
}
require.NoError(t, columnRows.Err())
require.NoError(t, columnRows.Close())
if assert.ObjectsAreEqual([]string{"session_id", "ordinal"}, columns) {
foundCompositeAlias = true
}
}
require.NoError(t, indexRows.Err())
assert.True(t, foundCompositeAlias)
}
+862
View File
@@ -0,0 +1,862 @@
package db
import (
"context"
"database/sql"
"fmt"
"slices"
"strings"
"github.com/uptrace/bun"
"github.com/uptrace/bun/dialect"
"go.kenn.io/agentsview/internal/db/bunmodel"
)
const duckDBPrivateDialectName dialect.Name = -1
// CommonSchemaCompatibilityMetadataKey stamps completion of the canonical
// common-schema convergence transaction on persistent backends.
const CommonSchemaCompatibilityMetadataKey = "bun_common_schema_v1"
var sqliteCommonSchemaColumnMigrations = []schemaColumnMigration{
{
"sessions", "source_archive_id",
"ALTER TABLE sessions ADD COLUMN source_archive_id TEXT NOT NULL DEFAULT ''",
},
{
"sessions", "source_database_generation",
"ALTER TABLE sessions ADD COLUMN source_database_generation TEXT NOT NULL DEFAULT ''",
},
{
"tool_calls", "message_ordinal",
"ALTER TABLE tool_calls ADD COLUMN message_ordinal INTEGER",
},
{
"pinned_messages", "source_uuid",
"ALTER TABLE pinned_messages ADD COLUMN source_uuid TEXT NOT NULL DEFAULT ''",
},
{
"source_worktree_project_mappings", "id",
"ALTER TABLE source_worktree_project_mappings ADD COLUMN id INTEGER NOT NULL DEFAULT 0",
},
{
"source_worktree_project_mappings", "created_at",
"ALTER TABLE source_worktree_project_mappings ADD COLUMN created_at TEXT NOT NULL DEFAULT ''",
},
}
// CreateCommonSchema creates the canonical serving tables and ordinary indexes
// in registry order. Adapter-owned operational, FTS, and vector schema remains
// outside this function.
func CreateCommonSchema(ctx context.Context, db bun.IDB) error {
includeForeignKeys := db.Dialect().Name() != duckDBPrivateDialectName
for _, table := range bunmodel.CommonTables() {
create := db.NewCreateTable().Model(table.Model).IfNotExists()
if includeForeignKeys {
for _, foreignKey := range table.ForeignKeys {
create.ForeignKey(bunmodel.ForeignKeyDefinition(foreignKey, true))
}
}
if _, err := create.Exec(ctx); err != nil {
return fmt.Errorf("creating common table %s: %w", table.Name, err)
}
for _, index := range table.Indexes {
createIndex := db.NewCreateIndex().Model(table.Model).
Index(index.Name).IfNotExists()
if index.Unique {
createIndex.Unique()
}
for _, column := range index.Columns {
createIndex.Column(column)
}
for _, expression := range index.Expressions {
createIndex.ColumnExpr(expression)
}
if _, err := createIndex.Exec(ctx); err != nil {
return fmt.Errorf(
"creating common index %s on %s: %w",
index.Name, table.Name, err,
)
}
}
}
return nil
}
// CheckCommonSchema verifies that every canonical table exposes every model
// column required by shared Bun reads. Engine-specific compatibility checks
// additionally validate physical keys and operational extensions.
func CheckCommonSchema(ctx context.Context, db bun.IDB) error {
if err := checkCommonSchemaColumns(ctx, db); err != nil {
return err
}
return checkCommonSchemaRows(ctx, db)
}
// CheckCommonSchemaStructure validates the canonical table projections without
// rescanning row invariants already attested by a compatibility stamp.
func CheckCommonSchemaStructure(ctx context.Context, db bun.IDB) error {
return checkCommonSchemaColumns(ctx, db)
}
func checkCommonSchemaColumns(ctx context.Context, db bun.IDB) error {
for _, table := range bunmodel.CommonTables() {
columns := bunmodel.ModelColumns(table.Model)
rows, err := db.NewSelect().Table(table.Name).Column(columns...).Limit(0).
Rows(ctx)
if err != nil {
return fmt.Errorf("checking common table %s: %w", table.Name, err)
}
if err := rows.Close(); err != nil {
return fmt.Errorf("closing common table %s check: %w", table.Name, err)
}
}
return nil
}
func checkCommonSchemaRows(ctx context.Context, db bun.IDB) error {
checks := []struct {
name string
query string
}{
{
"tool call message ordinal",
`SELECT EXISTS (
SELECT 1 FROM tool_calls WHERE message_ordinal IS NULL
)`,
},
{
"message logical key",
`SELECT EXISTS (
SELECT 1 FROM messages
GROUP BY session_id, ordinal HAVING COUNT(*) > 1
)`,
},
{
"tool call logical key",
`SELECT EXISTS (
SELECT 1 FROM tool_calls
GROUP BY session_id, message_ordinal, call_index
HAVING COUNT(*) > 1
)`,
},
{
"pin logical key",
`SELECT EXISTS (
SELECT 1 FROM pinned_messages
GROUP BY session_id, ordinal HAVING COUNT(*) > 1
)`,
},
}
for _, check := range checks {
var invalid bool
if err := db.NewRaw(check.query).Scan(ctx, &invalid); err != nil {
return fmt.Errorf("checking common %s: %w", check.name, err)
}
if invalid {
return fmt.Errorf("common %s is invalid", check.name)
}
}
for _, table := range bunmodel.CommonTables() {
for _, foreignKey := range table.ForeignKeys {
join := make([]string, len(foreignKey.Columns))
for index := range foreignKey.Columns {
join[index] = "child." + quoteCommonIdentifier(foreignKey.Columns[index]) +
" = parent." + quoteCommonIdentifier(foreignKey.ReferencedColumns[index])
}
query := "SELECT EXISTS (SELECT 1 FROM " + quoteCommonIdentifier(table.Name) +
" AS child LEFT JOIN " + quoteCommonIdentifier(foreignKey.ReferencedTable) +
" AS parent ON " + strings.Join(join, " AND ") +
" WHERE parent." + quoteCommonIdentifier(foreignKey.ReferencedColumns[0]) +
" IS NULL)"
var invalid bool
if err := db.NewRaw(query).Scan(ctx, &invalid); err != nil {
return fmt.Errorf("checking common %s canonical parent: %w", table.Name, err)
}
if invalid {
return fmt.Errorf(
"common %s canonical parent %s is missing",
table.Name, foreignKey.ReferencedTable,
)
}
}
}
return nil
}
func quoteCommonIdentifier(identifier string) string {
return `"` + strings.ReplaceAll(identifier, `"`, `""`) + `"`
}
func (db *DB) convergeSQLiteCommonSchemaLocked(
ctx context.Context, beforeStamp func() error,
) error {
tx, err := db.bunWriter.BeginTx(ctx, nil)
if err != nil {
return fmt.Errorf("starting common SQLite schema migration: %w", err)
}
defer func() { _ = tx.Rollback() }()
complete, err := sqliteCommonSchemaStamped(ctx, tx)
if err != nil {
return err
}
if complete {
if err := checkCommonSchemaColumns(ctx, tx); err != nil {
return fmt.Errorf("validating stamped common SQLite schema: %w", err)
}
if err := checkSQLiteCanonicalSchemaObjects(ctx, tx); err != nil {
return fmt.Errorf("validating stamped common SQLite schema: %w", err)
}
if err := tx.Commit(); err != nil {
return fmt.Errorf("closing common SQLite schema validation: %w", err)
}
return nil
}
if err := applyColumnMigrations(
sqliteCommonSchemaColumnMigrations,
func(query string, args ...any) rowScanner {
return tx.QueryRowContext(ctx, query, args...)
},
func(query string, args ...any) (sql.Result, error) {
return tx.ExecContext(ctx, query, args...)
},
); err != nil {
return err
}
if _, err := tx.ExecContext(ctx, `
UPDATE tool_calls
SET message_ordinal = (
SELECT ordinal FROM messages WHERE messages.id = tool_calls.message_id
)
WHERE message_ordinal IS NULL`); err != nil {
return fmt.Errorf("backfilling tool call message ordinals: %w", err)
}
if _, err := tx.ExecContext(ctx, sqliteToolCallOrdinalTriggerDDL); err != nil {
return fmt.Errorf("installing tool call message ordinal trigger: %w", err)
}
if err := convergeSQLitePricingMetadata(ctx, tx); err != nil {
return err
}
if err := CreateCommonSchema(ctx, tx); err != nil {
return err
}
if _, err := tx.ExecContext(ctx, `
UPDATE source_worktree_project_mappings
SET created_at = updated_at
WHERE created_at = ''`); err != nil {
return fmt.Errorf("backfilling source worktree mapping creation times: %w", err)
}
databaseGeneration, err := sqliteMetadataValue(
ctx, tx, archiveMetadataDatabaseIDKey,
)
if err != nil {
return err
}
archiveID, err := sqliteMetadataValue(ctx, tx, archiveMetadataArchiveIDKey)
if err != nil {
return err
}
archiveSalt, err := sqliteMetadataValue(ctx, tx, archiveMetadataArchiveSaltKey)
if err != nil {
return err
}
if databaseGeneration == "" || archiveID == "" || archiveSalt == "" {
return fmt.Errorf("common SQLite schema migration requires archive identity")
}
if _, err := tx.NewInsert().Model(&bunmodel.SourceArchive{
SourceArchiveID: archiveID, SourceArchiveSalt: archiveSalt,
}).On("CONFLICT (source_archive_id) DO UPDATE").
Set("source_archive_salt = EXCLUDED.source_archive_salt").Exec(ctx); err != nil {
return fmt.Errorf("recording common source archive: %w", err)
}
if _, err := tx.ExecContext(ctx, `
UPDATE sessions
SET source_archive_id = ?, source_database_generation = ?
WHERE source_archive_id = '' OR source_database_generation = ''`,
archiveID, databaseGeneration,
); err != nil {
return fmt.Errorf("backfilling session source provenance: %w", err)
}
if err := backfillSQLiteCommonIdentity(
ctx, tx, archiveID, archiveSalt, databaseGeneration,
); err != nil {
return err
}
if err := installSQLiteCanonicalIdentityTriggers(ctx, tx); err != nil {
return err
}
if err := CheckCommonSchema(ctx, tx); err != nil {
return err
}
if err := checkSQLiteCanonicalSchemaObjects(ctx, tx); err != nil {
return err
}
if beforeStamp != nil {
if err := beforeStamp(); err != nil {
return err
}
}
if _, err := tx.ExecContext(ctx, `
INSERT INTO archive_metadata (key, value)
VALUES (?, '1')
ON CONFLICT(key) DO UPDATE SET
value = excluded.value,
updated_at = strftime('%Y-%m-%dT%H:%M:%fZ','now')`,
CommonSchemaCompatibilityMetadataKey,
); err != nil {
return fmt.Errorf("stamping common SQLite schema: %w", err)
}
if err := tx.Commit(); err != nil {
return fmt.Errorf("committing common SQLite schema migration: %w", err)
}
return nil
}
func convergeSQLitePricingMetadata(ctx context.Context, db bun.IDB) error {
if _, err := db.ExecContext(ctx, `
CREATE TABLE IF NOT EXISTS pricing_metadata (
key TEXT PRIMARY KEY,
value TEXT NOT NULL,
updated_at TEXT NOT NULL
DEFAULT (strftime('%Y-%m-%dT%H:%M:%fZ','now'))
);
INSERT INTO pricing_metadata (key, value)
SELECT model_pattern, updated_at
FROM model_pricing
WHERE model_pattern IN (
'_fallback_version',
'_litellm_last_attempt',
'_pricing_storage_version'
)
ON CONFLICT(key) DO UPDATE SET
value = excluded.value,
updated_at = strftime('%Y-%m-%dT%H:%M:%fZ','now');
DELETE FROM model_pricing
WHERE model_pattern IN (
'_fallback_version',
'_litellm_last_attempt',
'_pricing_storage_version'
);
`); err != nil {
return fmt.Errorf("converging SQLite pricing metadata: %w", err)
}
return nil
}
const sqliteCanonicalIdentityTriggerDDL = `
DROP TRIGGER IF EXISTS trg_source_project_identity_observations_revision_insert;
DROP TRIGGER IF EXISTS trg_source_project_identity_observations_revision_update;
DROP TRIGGER IF EXISTS trg_source_project_identity_observations_revision_delete;
DROP TRIGGER IF EXISTS trg_source_session_project_identity_snapshots_revision_insert;
DROP TRIGGER IF EXISTS trg_source_session_project_identity_snapshots_revision_update;
DROP TRIGGER IF EXISTS trg_source_session_project_identity_snapshots_revision_delete;
DROP TRIGGER IF EXISTS trg_sessions_create_source_project_identity_snapshot;
DROP TRIGGER IF EXISTS trg_sessions_delete_source_project_identity_snapshot;
DROP TRIGGER IF EXISTS trg_source_worktree_project_mappings_revision_insert;
DROP TRIGGER IF EXISTS trg_source_worktree_project_mappings_revision_update;
DROP TRIGGER IF EXISTS trg_source_worktree_project_mappings_revision_delete;
CREATE TRIGGER trg_source_project_identity_observations_revision_insert
AFTER INSERT ON source_project_identity_observations BEGIN
INSERT INTO archive_metadata (key, value)
VALUES ('project_identity_publication_revision', '1')
ON CONFLICT(key) DO UPDATE SET
value = CAST(CAST(value AS INTEGER) + 1 AS TEXT),
updated_at = strftime('%Y-%m-%dT%H:%M:%fZ','now');
INSERT INTO project_identity_observation_changes (
project, machine, root_path, git_remote, revision, deleted
) VALUES (
NEW.project, NEW.machine, NEW.root_path, NEW.git_remote,
(SELECT CAST(value AS INTEGER) FROM archive_metadata
WHERE key = 'project_identity_publication_revision'), 0
) ON CONFLICT(project, machine, root_path, git_remote) DO UPDATE SET
revision = excluded.revision, deleted = 0;
END;
CREATE TRIGGER trg_source_project_identity_observations_revision_update
AFTER UPDATE ON source_project_identity_observations BEGIN
INSERT INTO archive_metadata (key, value)
VALUES ('project_identity_publication_revision', '1')
ON CONFLICT(key) DO UPDATE SET
value = CAST(CAST(value AS INTEGER) + 1 AS TEXT),
updated_at = strftime('%Y-%m-%dT%H:%M:%fZ','now');
INSERT INTO project_identity_observation_changes (
project, machine, root_path, git_remote, revision, deleted
) VALUES (
OLD.project, OLD.machine, OLD.root_path, OLD.git_remote,
(SELECT CAST(value AS INTEGER) FROM archive_metadata
WHERE key = 'project_identity_publication_revision'), 1
) ON CONFLICT(project, machine, root_path, git_remote) DO UPDATE SET
revision = excluded.revision, deleted = 1;
INSERT INTO project_identity_observation_changes (
project, machine, root_path, git_remote, revision, deleted
) VALUES (
NEW.project, NEW.machine, NEW.root_path, NEW.git_remote,
(SELECT CAST(value AS INTEGER) FROM archive_metadata
WHERE key = 'project_identity_publication_revision'), 0
) ON CONFLICT(project, machine, root_path, git_remote) DO UPDATE SET
revision = excluded.revision, deleted = 0;
END;
CREATE TRIGGER trg_source_project_identity_observations_revision_delete
AFTER DELETE ON source_project_identity_observations BEGIN
INSERT INTO archive_metadata (key, value)
VALUES ('project_identity_publication_revision', '1')
ON CONFLICT(key) DO UPDATE SET
value = CAST(CAST(value AS INTEGER) + 1 AS TEXT),
updated_at = strftime('%Y-%m-%dT%H:%M:%fZ','now');
INSERT INTO project_identity_observation_changes (
project, machine, root_path, git_remote, revision, deleted
) VALUES (
OLD.project, OLD.machine, OLD.root_path, OLD.git_remote,
(SELECT CAST(value AS INTEGER) FROM archive_metadata
WHERE key = 'project_identity_publication_revision'), 1
) ON CONFLICT(project, machine, root_path, git_remote) DO UPDATE SET
revision = excluded.revision, deleted = 1;
END;
CREATE TRIGGER trg_source_session_project_identity_snapshots_revision_insert
AFTER INSERT ON source_session_project_identity_snapshots BEGIN
INSERT INTO archive_metadata (key, value)
VALUES ('project_identity_publication_revision', '1')
ON CONFLICT(key) DO UPDATE SET
value = CAST(CAST(value AS INTEGER) + 1 AS TEXT),
updated_at = strftime('%Y-%m-%dT%H:%M:%fZ','now');
INSERT INTO session_project_identity_snapshot_changes (
session_id, project, revision, deleted
) VALUES (
NEW.source_session_id, NEW.project,
(SELECT CAST(value AS INTEGER) FROM archive_metadata
WHERE key = 'project_identity_publication_revision'), 0
) ON CONFLICT(session_id, project) DO UPDATE SET
revision = excluded.revision, deleted = 0;
END;
CREATE TRIGGER trg_source_session_project_identity_snapshots_revision_update
AFTER UPDATE ON source_session_project_identity_snapshots BEGIN
INSERT INTO archive_metadata (key, value)
VALUES ('project_identity_publication_revision', '1')
ON CONFLICT(key) DO UPDATE SET
value = CAST(CAST(value AS INTEGER) + 1 AS TEXT),
updated_at = strftime('%Y-%m-%dT%H:%M:%fZ','now');
INSERT INTO session_project_identity_snapshot_changes (
session_id, project, revision, deleted
) VALUES (
OLD.source_session_id, OLD.project,
(SELECT CAST(value AS INTEGER) FROM archive_metadata
WHERE key = 'project_identity_publication_revision'), 1
) ON CONFLICT(session_id, project) DO UPDATE SET
revision = excluded.revision, deleted = 1;
INSERT INTO session_project_identity_snapshot_changes (
session_id, project, revision, deleted
) VALUES (
NEW.source_session_id, NEW.project,
(SELECT CAST(value AS INTEGER) FROM archive_metadata
WHERE key = 'project_identity_publication_revision'), 0
) ON CONFLICT(session_id, project) DO UPDATE SET
revision = excluded.revision, deleted = 0;
END;
CREATE TRIGGER trg_source_session_project_identity_snapshots_revision_delete
AFTER DELETE ON source_session_project_identity_snapshots BEGIN
INSERT INTO archive_metadata (key, value)
VALUES ('project_identity_publication_revision', '1')
ON CONFLICT(key) DO UPDATE SET
value = CAST(CAST(value AS INTEGER) + 1 AS TEXT),
updated_at = strftime('%Y-%m-%dT%H:%M:%fZ','now');
INSERT INTO session_project_identity_snapshot_changes (
session_id, project, revision, deleted
) VALUES (
OLD.source_session_id, OLD.project,
(SELECT CAST(value AS INTEGER) FROM archive_metadata
WHERE key = 'project_identity_publication_revision'), 1
) ON CONFLICT(session_id, project) DO UPDATE SET
revision = excluded.revision, deleted = 1;
END;
CREATE TRIGGER trg_sessions_create_source_project_identity_snapshot
AFTER INSERT ON sessions
WHEN NEW.source_archive_id <> '' AND NEW.source_database_generation <> '' BEGIN
INSERT INTO source_session_project_identity_snapshots (
source_archive_id, source_database_generation, source_session_id,
project, machine, root_path, worktree_relationship,
checkout_state, git_branch, remote_resolution, observed_at
) VALUES (
NEW.source_archive_id, NEW.source_database_generation, NEW.id,
NEW.project, NEW.machine, NEW.cwd, 'unknown',
CASE WHEN NEW.git_branch <> '' THEN 'branch' ELSE 'unknown' END,
NEW.git_branch, 'unknown', strftime('%Y-%m-%dT%H:%M:%fZ','now')
) ON CONFLICT(
source_archive_id, source_database_generation, source_session_id
) DO NOTHING;
END;
CREATE TRIGGER trg_sessions_delete_source_project_identity_snapshot
AFTER DELETE ON sessions BEGIN
DELETE FROM source_session_project_identity_snapshots
WHERE source_archive_id = OLD.source_archive_id
AND source_database_generation = OLD.source_database_generation
AND source_session_id = OLD.id;
END;
CREATE TRIGGER trg_source_worktree_project_mappings_revision_insert
AFTER INSERT ON source_worktree_project_mappings BEGIN
INSERT INTO archive_metadata (key, value)
VALUES ('worktree_mapping_publication_revision', '1')
ON CONFLICT(key) DO UPDATE SET
value = CAST(CAST(value AS INTEGER) + 1 AS TEXT),
updated_at = strftime('%Y-%m-%dT%H:%M:%fZ','now');
INSERT INTO worktree_project_mapping_changes
(machine, path_prefix, revision, deleted)
VALUES (NEW.machine, NEW.path_prefix,
(SELECT CAST(value AS INTEGER) FROM archive_metadata
WHERE key = 'worktree_mapping_publication_revision'), 0)
ON CONFLICT(machine, path_prefix) DO UPDATE SET
revision = excluded.revision, deleted = 0;
END;
CREATE TRIGGER trg_source_worktree_project_mappings_revision_update
AFTER UPDATE ON source_worktree_project_mappings BEGIN
INSERT INTO archive_metadata (key, value)
VALUES ('worktree_mapping_publication_revision', '1')
ON CONFLICT(key) DO UPDATE SET
value = CAST(CAST(value AS INTEGER) + 1 AS TEXT),
updated_at = strftime('%Y-%m-%dT%H:%M:%fZ','now');
INSERT INTO worktree_project_mapping_changes
(machine, path_prefix, revision, deleted)
VALUES (OLD.machine, OLD.path_prefix,
(SELECT CAST(value AS INTEGER) FROM archive_metadata
WHERE key = 'worktree_mapping_publication_revision'), 1)
ON CONFLICT(machine, path_prefix) DO UPDATE SET
revision = excluded.revision, deleted = 1;
INSERT INTO worktree_project_mapping_changes
(machine, path_prefix, revision, deleted)
VALUES (NEW.machine, NEW.path_prefix,
(SELECT CAST(value AS INTEGER) FROM archive_metadata
WHERE key = 'worktree_mapping_publication_revision'), 0)
ON CONFLICT(machine, path_prefix) DO UPDATE SET
revision = excluded.revision, deleted = 0;
END;
CREATE TRIGGER trg_source_worktree_project_mappings_revision_delete
AFTER DELETE ON source_worktree_project_mappings BEGIN
INSERT INTO archive_metadata (key, value)
VALUES ('worktree_mapping_publication_revision', '1')
ON CONFLICT(key) DO UPDATE SET
value = CAST(CAST(value AS INTEGER) + 1 AS TEXT),
updated_at = strftime('%Y-%m-%dT%H:%M:%fZ','now');
INSERT INTO worktree_project_mapping_changes
(machine, path_prefix, revision, deleted)
VALUES (OLD.machine, OLD.path_prefix,
(SELECT CAST(value AS INTEGER) FROM archive_metadata
WHERE key = 'worktree_mapping_publication_revision'), 1)
ON CONFLICT(machine, path_prefix) DO UPDATE SET
revision = excluded.revision, deleted = 1;
END;`
var sqliteCanonicalIdentityTriggerNames = []string{
"trg_source_project_identity_observations_revision_insert",
"trg_source_project_identity_observations_revision_update",
"trg_source_project_identity_observations_revision_delete",
"trg_source_session_project_identity_snapshots_revision_insert",
"trg_source_session_project_identity_snapshots_revision_update",
"trg_source_session_project_identity_snapshots_revision_delete",
"trg_sessions_create_source_project_identity_snapshot",
"trg_sessions_delete_source_project_identity_snapshot",
"trg_source_worktree_project_mappings_revision_insert",
"trg_source_worktree_project_mappings_revision_update",
"trg_source_worktree_project_mappings_revision_delete",
}
const sqliteToolCallOrdinalTriggerDefinition = `
CREATE TRIGGER tool_calls_fill_message_ordinal
AFTER INSERT ON tool_calls
WHEN NEW.message_ordinal IS NULL
BEGIN
UPDATE tool_calls
SET message_ordinal = (
SELECT ordinal FROM messages WHERE messages.id = NEW.message_id
)
WHERE id = NEW.id;
END`
const sqliteToolCallOrdinalTriggerDDL = `
DROP TRIGGER IF EXISTS tool_calls_fill_message_ordinal;
` + sqliteToolCallOrdinalTriggerDefinition
func installSQLiteCanonicalIdentityTriggers(ctx context.Context, db bun.IDB) error {
if _, err := db.ExecContext(ctx, sqliteCanonicalIdentityTriggerDDL); err != nil {
return fmt.Errorf("installing canonical SQLite identity triggers: %w", err)
}
return nil
}
func checkSQLiteCanonicalSchemaObjects(ctx context.Context, db bun.IDB) error {
if err := checkSQLiteCanonicalIdentityTriggers(ctx, db); err != nil {
return err
}
if err := checkSQLiteToolCallOrdinalTrigger(ctx, db); err != nil {
return err
}
return checkSQLiteCanonicalIndexes(ctx, db)
}
func checkSQLiteCanonicalIdentityTriggers(ctx context.Context, db bun.IDB) error {
for _, name := range sqliteCanonicalIdentityTriggerNames {
var got string
if err := db.NewRaw(`
SELECT sql FROM sqlite_schema
WHERE type = 'trigger' AND name = ?`, name,
).Scan(ctx, &got); err != nil {
return fmt.Errorf("checking canonical SQLite trigger %s: %w", name, err)
}
want, err := sqliteCanonicalIdentityTriggerSQL(name)
if err != nil {
return err
}
if normalizeSQLiteSchemaSQL(got) != normalizeSQLiteSchemaSQL(want) {
return fmt.Errorf("canonical SQLite trigger %s has drifted", name)
}
}
return nil
}
func checkSQLiteToolCallOrdinalTrigger(ctx context.Context, db bun.IDB) error {
var got string
if err := db.NewRaw(`
SELECT sql FROM sqlite_schema
WHERE type = 'trigger' AND name = 'tool_calls_fill_message_ordinal'`,
).Scan(ctx, &got); err != nil {
return fmt.Errorf("checking canonical SQLite trigger tool_calls_fill_message_ordinal: %w", err)
}
if normalizeSQLiteSchemaSQL(got) !=
normalizeSQLiteSchemaSQL(sqliteToolCallOrdinalTriggerDefinition) {
return fmt.Errorf("canonical SQLite trigger tool_calls_fill_message_ordinal has drifted")
}
return nil
}
func sqliteCanonicalIdentityTriggerSQL(name string) (string, error) {
startMarker := "CREATE TRIGGER " + name
start := strings.Index(sqliteCanonicalIdentityTriggerDDL, startMarker)
if start < 0 {
return "", fmt.Errorf("canonical SQLite trigger %s is not defined", name)
}
rest := sqliteCanonicalIdentityTriggerDDL[start:]
end := strings.Index(rest, "\nEND;")
if end < 0 {
return "", fmt.Errorf("canonical SQLite trigger %s is incomplete", name)
}
return rest[:end+len("\nEND;")], nil
}
func checkSQLiteCanonicalIndexes(ctx context.Context, db bun.IDB) error {
for _, table := range bunmodel.CommonTables() {
for _, index := range table.Indexes {
if err := checkSQLiteCanonicalIndex(ctx, db, table, index); err != nil {
return err
}
}
}
return nil
}
func checkSQLiteCanonicalIndex(
ctx context.Context, db bun.IDB, table bunmodel.Table, index bunmodel.Index,
) error {
var unique, partial bool
if err := db.NewRaw(`
SELECT "unique", partial FROM pragma_index_list(?) WHERE name = ?`,
table.Name, index.Name,
).Scan(ctx, &unique, &partial); err != nil {
return fmt.Errorf("checking canonical SQLite index %s: %w", index.Name, err)
}
if unique != index.Unique {
return fmt.Errorf("canonical SQLite index %s has drifted", index.Name)
}
var indexColumns []struct {
Name sql.NullString `bun:"name"`
}
if err := db.NewRaw(`
SELECT name FROM pragma_index_info(?) ORDER BY seqno`, index.Name,
).Scan(ctx, &indexColumns); err != nil {
return fmt.Errorf("reading canonical SQLite index %s: %w", index.Name, err)
}
columns := make([]string, len(indexColumns))
for i := range indexColumns {
columns[i] = indexColumns[i].Name.String
}
if len(index.Expressions) == 0 {
if !slices.Equal(columns, index.Columns) {
return fmt.Errorf("canonical SQLite index %s has drifted", index.Name)
}
return nil
}
logicalColumns := sqliteExpressionIndexColumns(index.Expressions)
if partial && slices.Equal(columns, logicalColumns) {
var definition string
if err := db.NewRaw(`
SELECT sql FROM sqlite_schema
WHERE type = 'index' AND name = ?`, index.Name,
).Scan(ctx, &definition); err != nil {
return fmt.Errorf("reading canonical SQLite index %s definition: %w", index.Name, err)
}
if strings.Contains(normalizeSQLiteSchemaSQL(definition), " where dedup_key != ''") {
return nil
}
}
return fmt.Errorf("canonical SQLite index %s has drifted", index.Name)
}
func sqliteExpressionIndexColumns(expressions []string) []string {
columns := make([]string, 0, len(expressions))
for _, expression := range expressions {
normalized := strings.TrimSpace(strings.Trim(expression, "()"))
thenAt := strings.Index(normalized, " THEN ")
endAt := strings.LastIndex(normalized, " END")
if thenAt < 0 || endAt < thenAt {
return nil
}
columns = append(columns, normalized[thenAt+len(" THEN "):endAt])
}
return columns
}
func normalizeSQLiteSchemaSQL(query string) string {
normalized := strings.ToLower(strings.Join(strings.Fields(query), " "))
normalized = strings.Replace(normalized, " if not exists ", " ", 1)
return strings.TrimSuffix(normalized, ";")
}
func sqliteCommonSchemaStamped(ctx context.Context, db bun.IDB) (bool, error) {
var value string
err := db.NewSelect().Table("archive_metadata").Column("value").
Where("key = ?", CommonSchemaCompatibilityMetadataKey).Scan(ctx, &value)
if err == sql.ErrNoRows {
return false, nil
}
if err != nil {
return false, fmt.Errorf("probing common SQLite schema stamp: %w", err)
}
if value != "1" {
return false, fmt.Errorf("common SQLite schema stamp has invalid value %q", value)
}
return true, nil
}
func sqliteMetadataValue(
ctx context.Context, db bun.IDB, key string,
) (string, error) {
var value string
if err := db.NewSelect().Table("archive_metadata").Column("value").
Where("key = ?", key).Scan(ctx, &value); err != nil {
return "", fmt.Errorf("reading SQLite archive metadata %s: %w", key, err)
}
return value, nil
}
func backfillSQLiteCommonIdentity(
ctx context.Context,
tx bun.Tx,
archiveID string,
archiveSalt string,
databaseGeneration string,
) error {
if _, err := tx.ExecContext(ctx, `
INSERT INTO source_project_identity_observations (
source_archive_id, source_archive_salt,
project, machine, root_path, git_remote, git_remote_name,
repository_path, worktree_name, worktree_root_path,
worktree_relationship, checkout_state, git_branch,
remote_resolution, remote_candidate_count, observed_at,
normalized_remote, key_source, key
)
SELECT ?, ?, project, machine, root_path, git_remote, git_remote_name,
repository_path, worktree_name, worktree_root_path,
worktree_relationship, checkout_state, git_branch,
remote_resolution, remote_candidate_count, observed_at,
normalized_remote, key_source, key
FROM project_identity_observations
WHERE true
ON CONFLICT(source_archive_id, project, machine, root_path, git_remote)
DO UPDATE SET
source_archive_salt = excluded.source_archive_salt,
git_remote_name = excluded.git_remote_name,
repository_path = excluded.repository_path,
worktree_name = excluded.worktree_name,
worktree_root_path = excluded.worktree_root_path,
worktree_relationship = excluded.worktree_relationship,
checkout_state = excluded.checkout_state,
git_branch = excluded.git_branch,
remote_resolution = excluded.remote_resolution,
remote_candidate_count = excluded.remote_candidate_count,
observed_at = excluded.observed_at,
normalized_remote = excluded.normalized_remote,
key_source = excluded.key_source,
key = excluded.key`, archiveID, archiveSalt,
); err != nil {
return fmt.Errorf("backfilling source project identities: %w", err)
}
if _, err := tx.ExecContext(ctx, `
INSERT INTO source_session_project_identity_snapshots (
source_archive_id, source_database_generation, source_session_id,
project, machine, root_path, git_remote, git_remote_name,
repository_path, worktree_name, worktree_root_path,
worktree_relationship, checkout_state, git_branch,
remote_resolution, remote_candidate_count, observed_at,
normalized_remote, key_source, key
)
SELECT ?, ?, session_id, project, machine, root_path, git_remote,
git_remote_name, repository_path, worktree_name, worktree_root_path,
worktree_relationship, checkout_state, git_branch,
remote_resolution, remote_candidate_count, observed_at,
normalized_remote, key_source, key
FROM session_project_identity_snapshots
WHERE true
ON CONFLICT(source_archive_id, source_database_generation, source_session_id)
DO UPDATE SET
project = excluded.project,
machine = excluded.machine,
root_path = excluded.root_path,
git_remote = excluded.git_remote,
git_remote_name = excluded.git_remote_name,
repository_path = excluded.repository_path,
worktree_name = excluded.worktree_name,
worktree_root_path = excluded.worktree_root_path,
worktree_relationship = excluded.worktree_relationship,
checkout_state = excluded.checkout_state,
git_branch = excluded.git_branch,
remote_resolution = excluded.remote_resolution,
remote_candidate_count = excluded.remote_candidate_count,
observed_at = excluded.observed_at,
normalized_remote = excluded.normalized_remote,
key_source = excluded.key_source,
key = excluded.key`, archiveID, databaseGeneration,
); err != nil {
return fmt.Errorf("backfilling source session identities: %w", err)
}
if _, err := tx.ExecContext(ctx, `
INSERT INTO source_worktree_project_mappings (
id, source_archive_id, machine, path_prefix, layout,
project, original_project, enabled, created_at, updated_at
)
SELECT id, ?, machine, path_prefix, layout,
project, original_project, enabled, created_at, updated_at
FROM worktree_project_mappings
WHERE true
ON CONFLICT(source_archive_id, machine, path_prefix) DO UPDATE SET
layout = excluded.layout,
project = excluded.project,
original_project = excluded.original_project,
enabled = excluded.enabled,
updated_at = excluded.updated_at`, archiveID,
); err != nil {
return fmt.Errorf("backfilling source worktree mappings: %w", err)
}
return nil
}
+309
View File
@@ -0,0 +1,309 @@
package db
import (
"database/sql"
"path/filepath"
"testing"
"time"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
"github.com/uptrace/bun"
"github.com/uptrace/bun/dialect/sqlitedialect"
"go.kenn.io/agentsview/internal/db/bunmodel"
)
func TestBunSchemaCreatesAndChecksCanonicalSQLiteSchema(t *testing.T) {
raw, err := sql.Open("sqlite3", ":memory:")
require.NoError(t, err)
raw.SetMaxOpenConns(1)
t.Cleanup(func() { require.NoError(t, raw.Close()) })
_, err = raw.ExecContext(t.Context(), `PRAGMA foreign_keys = ON`)
require.NoError(t, err)
store := bun.NewDB(raw, sqlitedialect.New())
require.NoError(t, CreateCommonSchema(t.Context(), store))
require.NoError(t, CheckCommonSchema(t.Context(), store))
_, err = store.NewInsert().Model(&bunmodel.SourceArchive{
SourceArchiveID: "archive-1", SourceArchiveSalt: "salt-1",
}).Exec(t.Context())
require.NoError(t, err)
want := bunmodel.Session{
ID: "canonical-session", Project: "project", Machine: "machine",
Agent: "agent", CreatedAt: bunmodel.NewTimestamp(
time.Date(2026, 8, 2, 12, 0, 0, 0, time.UTC),
),
SourceArchiveID: "archive-1",
SourceDatabaseGeneration: "generation-1",
}
_, err = store.NewInsert().Model(&want).Exec(t.Context())
require.NoError(t, err)
var got bunmodel.Session
require.NoError(t, store.NewSelect().Model(&got).
Where("id = ?", want.ID).Scan(t.Context()))
assert.Equal(t, want.ID, got.ID)
assert.Equal(t, want.SourceArchiveID, got.SourceArchiveID)
assert.Equal(t, want.SourceDatabaseGeneration, got.SourceDatabaseGeneration)
}
func TestBunSchemaNormalSQLiteOpenAcceptsCanonicalSession(t *testing.T) {
database := testDB(t)
archiveID, err := database.GetArchiveID(t.Context())
require.NoError(t, err)
generation, err := database.GetDatabaseID(t.Context())
require.NoError(t, err)
want := bunmodel.Session{
ID: "normal-open-session", Project: "project", Machine: "machine",
Agent: "agent", CreatedAt: bunmodel.NewTimestamp(
time.Date(2026, 8, 2, 12, 0, 0, 0, time.UTC),
),
SourceArchiveID: archiveID,
SourceDatabaseGeneration: generation,
}
err = database.update(t.Context(), WriteArchive, func(store bun.IDB) error {
_, insertErr := store.NewInsert().Model(&want).Exec(t.Context())
return insertErr
})
require.NoError(t, err)
var got bunmodel.Session
err = database.view(t.Context(), func(store bun.IDB) error {
return store.NewSelect().Model(&got).
Where("id = ?", want.ID).Scan(t.Context())
})
require.NoError(t, err)
assert.Equal(t, want.ID, got.ID)
assert.Equal(t, archiveID, got.SourceArchiveID)
assert.Equal(t, generation, got.SourceDatabaseGeneration)
}
func TestBunSchemaNormalSQLiteOpenUsesAcceptedRelationshipMatrix(t *testing.T) {
database := testDB(t)
type foreignKey struct {
Parent string
From string
To string
OnDelete string
}
wantByTable := map[string][]foreignKey{
"messages": {{
Parent: "sessions", From: "session_id", To: "id", OnDelete: "CASCADE",
}},
"tool_calls": {
{Parent: "messages", From: "message_id", To: "id", OnDelete: "CASCADE"},
{Parent: "sessions", From: "session_id", To: "id", OnDelete: "CASCADE"},
},
"pinned_messages": {
{Parent: "messages", From: "message_id", To: "id", OnDelete: "CASCADE"},
{Parent: "sessions", From: "session_id", To: "id", OnDelete: "CASCADE"},
},
"tool_result_events": {{
Parent: "sessions", From: "session_id", To: "id", OnDelete: "CASCADE",
}},
}
for table, want := range wantByTable {
rows, err := database.getReader().QueryContext(t.Context(), `
SELECT "table", "from", "to", on_delete
FROM pragma_foreign_key_list(?)`, table)
require.NoError(t, err, table)
var got []foreignKey
for rows.Next() {
var item foreignKey
require.NoError(t, rows.Scan(
&item.Parent, &item.From, &item.To, &item.OnDelete,
), table)
got = append(got, item)
}
require.NoError(t, rows.Err(), table)
require.NoError(t, rows.Close(), table)
assert.ElementsMatch(t, want, got, table)
}
}
func TestBunSchemaRawSQLiteToolCallInsertResolvesMessageOrdinal(t *testing.T) {
database := testDB(t)
insertSession(t, database, "raw-tool-session", "project")
insertMessages(t, database, userMsg("raw-tool-session", 5, "hello"))
var messageID int64
require.NoError(t, database.getReader().QueryRowContext(t.Context(), `
SELECT id FROM messages
WHERE session_id = 'raw-tool-session' AND ordinal = 5`,
).Scan(&messageID))
_, err := database.getWriter().ExecContext(t.Context(), `
INSERT INTO tool_calls (
message_id, session_id, tool_name, category, call_index
) VALUES (?, 'raw-tool-session', 'Read', 'Read', 0)`, messageID)
require.NoError(t, err)
var ordinal int
require.NoError(t, database.getReader().QueryRowContext(t.Context(), `
SELECT message_ordinal FROM tool_calls
WHERE session_id = 'raw-tool-session' AND call_index = 0`,
).Scan(&ordinal))
assert.Equal(t, 5, ordinal)
}
func TestBunSchemaSessionBatchStampsArchiveProvenance(t *testing.T) {
database := testDB(t)
createdAt := "2026-08-02T12:00:00Z"
_, err := database.WriteSessionBatchAtomic([]SessionBatchWrite{{
Session: Session{
ID: "stamped-session", Project: "project", Machine: "machine",
Agent: "agent", CreatedAt: createdAt,
},
Messages: []Message{{
SessionID: "stamped-session", Ordinal: 0, Role: "user",
Content: "hello", Timestamp: createdAt,
}},
ReplaceMessages: true,
}})
require.NoError(t, err)
var archiveID, generation string
require.NoError(t, database.getReader().QueryRowContext(t.Context(), `
SELECT source_archive_id, source_database_generation
FROM sessions WHERE id = 'stamped-session'`,
).Scan(&archiveID, &generation))
assert.NotEmpty(t, archiveID)
assert.NotEmpty(t, generation)
}
func TestBunSchemaStampedReopenRejectsTriggerDriftWithoutRepair(t *testing.T) {
path := filepath.Join(t.TempDir(), "archive.db")
database, err := Open(path)
require.NoError(t, err)
require.NoError(t, database.Close())
raw, err := sql.Open("sqlite3", path)
require.NoError(t, err)
_, err = raw.ExecContext(t.Context(), `
DROP TRIGGER IF EXISTS trg_sessions_delete_source_project_identity_snapshot;
CREATE TRIGGER trg_sessions_delete_source_project_identity_snapshot
AFTER DELETE ON sessions BEGIN
SELECT 'drifted trigger';
END;`)
require.NoError(t, err)
require.NoError(t, raw.Close())
reopened, err := Open(path)
if reopened != nil {
require.NoError(t, reopened.Close())
}
require.Error(t, err)
raw, err = sql.Open("sqlite3", path)
require.NoError(t, err)
t.Cleanup(func() { require.NoError(t, raw.Close()) })
var triggerSQL string
require.NoError(t, raw.QueryRowContext(t.Context(), `
SELECT sql FROM sqlite_schema
WHERE type = 'trigger'
AND name = 'trg_sessions_delete_source_project_identity_snapshot'`,
).Scan(&triggerSQL))
assert.Contains(t, triggerSQL, "drifted trigger")
}
func TestBunSchemaStampedReopenRejectsIndexDriftWithoutRepair(t *testing.T) {
path := filepath.Join(t.TempDir(), "archive.db")
database, err := Open(path)
require.NoError(t, err)
require.NoError(t, database.Close())
raw, err := sql.Open("sqlite3", path)
require.NoError(t, err)
_, err = raw.ExecContext(t.Context(), `
DROP INDEX idx_sessions_project;
CREATE INDEX idx_sessions_project ON sessions(machine);`)
require.NoError(t, err)
require.NoError(t, raw.Close())
reopened, err := Open(path)
if reopened != nil {
require.NoError(t, reopened.Close())
}
require.Error(t, err)
raw, err = sql.Open("sqlite3", path)
require.NoError(t, err)
t.Cleanup(func() { require.NoError(t, raw.Close()) })
var indexedColumn string
require.NoError(t, raw.QueryRowContext(t.Context(), `
SELECT name FROM pragma_index_info('idx_sessions_project')
ORDER BY seqno LIMIT 1`,
).Scan(&indexedColumn))
assert.Equal(t, "machine", indexedColumn)
}
func TestBunSchemaStampedReopenSkipsRowInvariantScans(t *testing.T) {
path := filepath.Join(t.TempDir(), "archive.db")
database, err := Open(path)
require.NoError(t, err)
archiveID, err := database.GetArchiveID(t.Context())
require.NoError(t, err)
databaseID, err := database.GetDatabaseID(t.Context())
require.NoError(t, err)
_, err = database.getWriter().ExecContext(t.Context(), `
INSERT INTO sessions (
id, project, machine, agent, created_at,
source_archive_id, source_database_generation
) VALUES (?, 'project', 'machine', 'agent', ?, ?, ?);
INSERT INTO messages (session_id, ordinal, role, content)
VALUES ('invalid-tool-session', 0, 'assistant', 'done');
INSERT INTO tool_calls (
message_id, session_id, message_ordinal,
tool_name, category, call_index
) SELECT id, session_id, ordinal, 'Read', 'Read', 0
FROM messages WHERE session_id = 'invalid-tool-session';
UPDATE tool_calls SET message_ordinal = NULL
WHERE session_id = 'invalid-tool-session';`,
"invalid-tool-session", "2026-08-02T12:00:00Z", archiveID, databaseID,
)
require.NoError(t, err)
require.NoError(t, database.Close())
reopened, err := Open(path)
require.NoError(t, err)
t.Cleanup(func() { require.NoError(t, reopened.Close()) })
var invalidRows int
require.NoError(t, reopened.getReader().QueryRowContext(t.Context(), `
SELECT count(*) FROM tool_calls WHERE message_ordinal IS NULL`,
).Scan(&invalidRows))
assert.Equal(t, 1, invalidRows)
}
func TestCheckCommonSchemaRejectsMissingCanonicalParent(t *testing.T) {
raw, err := sql.Open("sqlite3", ":memory:")
require.NoError(t, err)
raw.SetMaxOpenConns(1)
t.Cleanup(func() { require.NoError(t, raw.Close()) })
store := bun.NewDB(raw, sqlitedialect.New())
require.NoError(t, CreateCommonSchema(t.Context(), store))
_, err = store.NewInsert().Model(&bunmodel.SourceArchive{
SourceArchiveID: "archive", SourceArchiveSalt: "salt",
}).Exec(t.Context())
require.NoError(t, err)
_, err = store.NewInsert().Model(&bunmodel.Session{
ID: "parent-check", Project: "project", Machine: "machine", Agent: "agent",
CreatedAt: bunmodel.NewTimestamp(
time.Date(2026, 8, 2, 12, 0, 0, 0, time.UTC),
),
SourceArchiveID: "archive",
SourceDatabaseGeneration: "generation",
}).Exec(t.Context())
require.NoError(t, err)
_, err = store.NewInsert().Model(&bunmodel.ToolCall{
SessionID: "parent-check", MessageOrdinal: 7,
ToolName: "Read", Category: "Read",
}).Exec(t.Context())
require.NoError(t, err)
err = CheckCommonSchema(t.Context(), store)
require.Error(t, err)
assert.Contains(t, err.Error(), "tool_calls canonical parent")
}
+940
View File
@@ -0,0 +1,940 @@
package db
import (
"context"
"crypto/hmac"
"crypto/sha256"
"database/sql"
"encoding/base64"
"encoding/json"
"fmt"
"strings"
"github.com/uptrace/bun"
"go.kenn.io/agentsview/internal/db/bunmodel"
)
var bunSessionQueryDialect = QueryDialect{
name: "bun",
placeholderStyle: placeholderQuestion,
trueLiteral: "TRUE",
falseLiteral: "FALSE",
dateStartExpr: func(q func(string) string) string {
return "COALESCE(" + bunNullableTimestamp(q("started_at")) +
", " + q("created_at") + ")"
},
dateEndExpr: func(q func(string) string) string {
outerID := q("id")
if outerID == "id" {
outerID = "session.id"
}
return "COALESCE(" + bunNullableTimestamp(q("ended_at")) +
", (SELECT MAX(" + bunNullableTimestamp("m.timestamp") +
") FROM messages m WHERE m.session_id = " + outerID +
" AND " + bunNullableTimestamp("m.timestamp") + " IS NOT NULL), " +
bunNullableTimestamp(q("started_at")) + ", " + q("created_at") + ")"
},
dateParam: func(ph string) string { return ph },
activityParam: func(ph string) string { return ph },
cursorActivityExpr: "COALESCE(" + bunNullableTimestamp("ended_at") + ", " +
bunNullableTimestamp("started_at") + ", created_at)",
cursorParam: func(ph string) string { return ph },
castCursor: func(ph string, _ valueKind) string { return ph },
portableEmptyTimestamp: true,
terminationExpr: "COALESCE(" + bunNullableTimestamp("ended_at") + ", " +
bunNullableTimestamp("started_at") + ", created_at)",
terminationKind: timestampText,
caseInsensitiveLike: "LIKE",
caseInsensitiveLikeEsc: `ESCAPE '\'`,
regexPredicate: func(col, ph string) string {
return col + " LIKE " + ph
},
sidebarChildRelationships: []string{"subagent", "fork"},
canonicalChildRelationships: []string{"subagent", "fork", "continuation"},
}
// PortableBunSessionQueryDialect returns the Bun-placeholder session dialect
// used by engines with native timestamp comparison semantics.
func PortableBunSessionQueryDialect() QueryDialect {
return bunSessionQueryDialect
}
// SQLiteBunSessionQueryDialect preserves chronological comparisons for the
// shipped SQLite text timestamp representation while retaining Bun's portable
// question-mark placeholders and common non-time predicates.
func SQLiteBunSessionQueryDialect() QueryDialect {
dialect := bunSessionQueryDialect
sqlite := SQLiteQueryDialect()
dialect.dateStartExpr = sqlite.dateStartExpr
dialect.dateEndExpr = func(q func(string) string) string {
outerID := q("id")
if outerID == "id" {
outerID = "session.id"
}
return "julianday(COALESCE(NULLIF(" + q("ended_at") +
", ''), (SELECT m.timestamp FROM messages m" +
" WHERE m.session_id = " + outerID +
" AND m.timestamp != '' ORDER BY julianday(m.timestamp)" +
" DESC, m.timestamp DESC LIMIT 1), NULLIF(" + q("started_at") +
", ''), " + q("created_at") + "))"
}
dialect.dateParam = sqlite.dateParam
dialect.timestampOrderExpr = func(column string) string {
return "julianday(NULLIF(" + column + ", ''))"
}
dialect.castCursor = func(placeholder string, kind valueKind) string {
if kind == kindTimestamp {
return "julianday(" + placeholder + ")"
}
return placeholder
}
dialect.terminationExpr = sqlite.terminationExpr
dialect.terminationKind = sqlite.terminationKind
return dialect
}
func bunNullableTimestamp(column string) string {
return "CASE WHEN CAST(" + column + " AS VARCHAR) = '' THEN NULL ELSE " +
column + " END"
}
// EncodeCursor returns a base64-encoded, HMAC-signed cursor string.
func (s *BunStore) EncodeCursor(cursor SessionCursor) string {
payload, _ := json.Marshal(cursor)
s.cursorMu.RLock()
mac := hmac.New(sha256.New, s.cursorSecret)
s.cursorMu.RUnlock()
_, _ = mac.Write(payload)
return base64.RawURLEncoding.EncodeToString(payload) + "." +
base64.RawURLEncoding.EncodeToString(mac.Sum(nil))
}
// DecodeCursor verifies and decodes a session cursor. Unsigned legacy cursors
// remain readable, but their cached total is discarded.
func (s *BunStore) DecodeCursor(value string) (SessionCursor, error) {
parts := strings.Split(value, ".")
if len(parts) == 1 {
payload, err := base64.RawURLEncoding.DecodeString(parts[0])
if err != nil {
return SessionCursor{}, fmt.Errorf("%w: %v", ErrInvalidCursor, err)
}
var cursor SessionCursor
if err := json.Unmarshal(payload, &cursor); err != nil {
return SessionCursor{}, fmt.Errorf("%w: %v", ErrInvalidCursor, err)
}
cursor.Total = 0
return cursor, nil
}
if len(parts) != 2 {
return SessionCursor{}, fmt.Errorf("%w: invalid format", ErrInvalidCursor)
}
payload, err := base64.RawURLEncoding.DecodeString(parts[0])
if err != nil {
return SessionCursor{}, fmt.Errorf("%w: invalid payload: %v", ErrInvalidCursor, err)
}
signature, err := base64.RawURLEncoding.DecodeString(parts[1])
if err != nil {
return SessionCursor{}, fmt.Errorf("%w: invalid signature encoding: %v", ErrInvalidCursor, err)
}
s.cursorMu.RLock()
mac := hmac.New(sha256.New, s.cursorSecret)
s.cursorMu.RUnlock()
_, _ = mac.Write(payload)
if !hmac.Equal(signature, mac.Sum(nil)) {
return SessionCursor{}, fmt.Errorf("%w: signature mismatch", ErrInvalidCursor)
}
var cursor SessionCursor
if err := json.Unmarshal(payload, &cursor); err != nil {
return SessionCursor{}, fmt.Errorf("%w: invalid json: %v", ErrInvalidCursor, err)
}
return cursor, nil
}
// ListSessions returns a cursor-paginated list from canonical session rows.
// Count and page scans share one backend guard so handle replacement cannot
// split one logical read across mirror generations.
func (s *BunStore) ListSessions(
ctx context.Context, filter SessionFilter,
) (SessionPage, error) {
if filter.Limit <= 0 || filter.Limit > MaxSessionLimit {
filter.Limit = DefaultSessionLimit
}
resolvedSort := ResolveSort(filter)
var cursor SessionCursor
if filter.Cursor != "" {
var err error
cursor, err = s.DecodeCursor(filter.Cursor)
if err != nil {
return SessionPage{}, err
}
}
var pendingPage SessionPage
err := s.consistentView(ctx, func(store bun.IDB) error {
total := cursor.Total
dialect := s.backend.SessionQueryDialect()
where, args := BuildSessionFilterSQL(filter, dialect)
base := store.NewSelect().Model((*bunmodel.Session)(nil))
base = applyBunWhere(base, where, args)
if total <= 0 {
count, err := base.Clone().Conn(store).Count(ctx)
if err != nil {
return fmt.Errorf("counting sessions: %w", err)
}
total = count
}
query := base.Clone().Conn(store)
if filter.Cursor != "" {
values, err := CursorPredicateValues(cursor, resolvedSort)
if err != nil {
return err
}
builder := NewQueryBuilder(dialect, 0)
query = query.Where(
builder.CursorPredicate(resolvedSort, filter, values, cursor.ID),
builder.Args()...,
)
}
orderBuilder := NewQueryBuilder(dialect, 0)
order := strings.TrimPrefix(
orderBuilder.OrderByClause(resolvedSort, filter), "ORDER BY ",
)
var rows []bunmodel.Session
if len(orderBuilder.Args()) == 0 {
query = query.OrderExpr(order)
} else {
query = query.OrderExpr(order, orderBuilder.Args()...)
}
if err := query.Limit(filter.Limit+1).Scan(ctx, &rows); err != nil {
return fmt.Errorf("querying sessions: %w", err)
}
page := SessionPage{Total: total}
page.Sessions = make([]Session, 0, min(len(rows), filter.Limit))
for _, row := range rows[:min(len(rows), filter.Limit)] {
page.Sessions = append(page.Sessions, baseSessionFromBunRow(row))
}
if len(rows) > filter.Limit {
last := page.Sessions[len(page.Sessions)-1]
page.NextCursor = s.EncodeCursor(
NextSessionCursor(&last, resolvedSort, total, filter),
)
}
pendingPage = page
return nil
})
if err != nil {
return SessionPage{}, err
}
return pendingPage, nil
}
func visibleSessionFromBunRow(row bunmodel.Session) Session {
session := sessionFromBunRow(row)
if session.DisplayName == nil {
session.DisplayName = session.SessionName
}
return session
}
func baseSessionFromBunRow(row bunmodel.Session) Session {
session := visibleSessionFromBunRow(row)
session.SessionName = nil
session.FilePath = nil
session.FileSize = nil
session.FileMtime = nil
session.FileInode = nil
session.FileDevice = nil
session.FileHash = nil
session.LocalModifiedAt = nil
return session
}
// GetSession returns a visible session and excludes trashed rows.
func (s *BunStore) GetSession(ctx context.Context, id string) (*Session, error) {
return s.getSession(ctx, id, false)
}
// GetSessionFull includes trashed rows and canonical file metadata.
func (s *BunStore) GetSessionFull(ctx context.Context, id string) (*Session, error) {
return s.getSession(ctx, id, true)
}
func (s *BunStore) getSession(
ctx context.Context, id string, includeDeleted bool,
) (*Session, error) {
var session *Session
err := s.view(ctx, func(store bun.IDB) error {
var err error
session, err = s.getSessionFrom(ctx, store, id, includeDeleted)
return err
})
if err != nil {
return nil, fmt.Errorf("getting session %s: %w", id, err)
}
return session, nil
}
func (s *BunStore) getSessionFrom(
ctx context.Context, store bun.IDB, id string, includeDeleted bool,
) (*Session, error) {
var row bunmodel.Session
query := store.NewSelect().Model(&row).Where("id = ?", id)
if !includeDeleted {
query = query.Where("deleted_at IS NULL")
}
if err := query.Scan(ctx); err != nil {
if err == sql.ErrNoRows {
return nil, nil
}
return nil, err
}
if includeDeleted {
session := visibleSessionFromBunRow(row)
if hydrator, ok := s.backend.(bunSessionFullHydrator); ok {
if err := hydrator.HydrateSessionFull(ctx, store, &session); err != nil {
return nil, err
}
}
return &session, nil
}
session := baseSessionFromBunRow(row)
return &session, nil
}
const partialSessionCandidateBatchSize = 64
// FindSessionIDsByPartial performs literal, case-sensitive substring matching.
func (s *BunStore) FindSessionIDsByPartial(
ctx context.Context, partial string, limit int,
) ([]string, error) {
if partial == "" {
return nil, nil
}
if limit <= 0 {
limit = 5
}
ids := make([]string, 0, min(limit, partialSessionCandidateBatchSize))
err := s.view(ctx, func(store bun.IDB) error {
activityExpr := bunSessionActivityExpr("session")
pattern := "%" + EscapeLikePattern(partial) + "%"
var cursorActivity any
var cursorID string
hasCursor := false
for len(ids) < limit {
var candidates []struct {
ID string `bun:"id"`
Activity any `bun:"activity"`
}
query := store.NewSelect().TableExpr("sessions AS session").
ColumnExpr("session.id").
ColumnExpr(activityExpr+" AS activity").
Where("session.id LIKE ? ESCAPE '\\'", pattern).
Where("session.deleted_at IS NULL")
if hasCursor {
query = query.Where(
"("+activityExpr+" < ? OR ("+activityExpr+
" = ? AND session.id < ?))",
cursorActivity, cursorActivity, cursorID,
)
}
if err := query.OrderExpr(activityExpr+" DESC").
OrderExpr("session.id DESC").
Limit(partialSessionCandidateBatchSize).
Scan(ctx, &candidates); err != nil {
return err
}
for _, candidate := range candidates {
if strings.Contains(candidate.ID, partial) {
ids = append(ids, candidate.ID)
if len(ids) == limit {
return nil
}
}
}
if len(candidates) < partialSessionCandidateBatchSize {
return nil
}
last := candidates[len(candidates)-1]
cursorActivity = last.Activity
cursorID = last.ID
hasCursor = true
}
return nil
})
if err != nil {
return nil, fmt.Errorf("finding sessions by partial id %q: %w", partial, err)
}
return ids, nil
}
// GetChildSessions returns non-deleted children ordered by start time.
func (s *BunStore) GetChildSessions(
ctx context.Context, parentID string,
) ([]Session, error) {
var rows []bunmodel.Session
err := s.view(ctx, func(store bun.IDB) error {
return store.NewSelect().Model(&rows).
Where("parent_session_id = ?", parentID).
Where("deleted_at IS NULL").
OrderExpr("COALESCE(" + bunNullableTimestamp("started_at") +
", created_at) ASC").
OrderExpr("id ASC").Scan(ctx)
})
if err != nil {
return nil, fmt.Errorf("querying child sessions for %s: %w", parentID, err)
}
sessions := make([]Session, 0, len(rows))
for _, row := range rows {
sessions = append(sessions, baseSessionFromBunRow(row))
}
return sessions, nil
}
// GetSidebarSessionIndex returns canonical roots and their descendants.
func (s *BunStore) GetSidebarSessionIndex(
ctx context.Context, filter SessionFilter,
) (SidebarSessionIndex, error) {
filter.IncludeChildren = true
filter.IncludeOrphans = true
var pendingIndex SidebarSessionIndex
err := s.consistentView(ctx, func(store bun.IDB) error {
var index SidebarSessionIndex
var err error
if filter.Limit > 0 || filter.Cursor != "" || filter.Starred {
index, err = s.getSidebarSessionIndexPage(
ctx, store, filter, s.backend.SessionQueryDialect(),
)
} else {
index, err = s.getSidebarSessionIndexAll(
ctx, store, filter, s.backend.SessionQueryDialect(),
)
}
if err != nil {
return err
}
pendingIndex = index
return nil
})
if err != nil {
return SidebarSessionIndex{}, err
}
return pendingIndex, nil
}
func sidebarRootFilter(filter SessionFilter, dialect QueryDialect) (string, []any) {
rootFilter := filter
rootFilter.IncludeChildren = false
rootFilter.Cursor = ""
rootFilter.Starred = false
where, args := BuildSessionBaseFilterSQL(rootFilter, dialect)
where += " AND " + BuildCanonicalRootWhere(
dialect, "session", filter.IncludeOrphans,
)
return where, args
}
func bunSessionActivityExpr(alias string) string {
qualify := func(column string) string {
if alias == "" {
return column
}
return alias + "." + column
}
return "COALESCE(" + bunNullableTimestamp(qualify("ended_at")) + ", " +
bunNullableTimestamp(qualify("started_at")) + ", " +
qualify("created_at") + ")"
}
func bunSessionActivityOrderExpr(alias string, dialect QueryDialect) string {
return dialect.timestampExpr(bunSessionActivityExpr(alias))
}
func sidebarRootTreeSQL(
rootWhere, childAutomationWhere string, starred bool,
) string {
return `WITH RECURSIVE root_candidates(id) AS (
SELECT session.id
FROM sessions AS session
WHERE ` + rootWhere + `
),
tree(root_id, id) AS (
SELECT id, id FROM root_candidates
UNION
SELECT t.root_id, child.id
FROM sessions AS child
JOIN tree AS t ON child.parent_session_id = t.id
WHERE child.message_count > 0
AND child.deleted_at IS NULL` + childAutomationWhere + `
)` + sidebarStarredRootCTE(starred)
}
func sidebarStarredRootCTE(enabled bool) string {
if !enabled {
return ""
}
return `,
eligible_roots(id) AS (
SELECT DISTINCT tree.root_id
FROM tree
JOIN starred_sessions AS starred ON starred.session_id = tree.id
)`
}
func sidebarStarredRootJoin(enabled bool) string {
if !enabled {
return ""
}
return "JOIN eligible_roots AS eligible ON eligible.id = t.root_id"
}
func sidebarChildAutomationWhere(filter SessionFilter, dialect QueryDialect) string {
predicate := automationScopePredicate(filter, dialect, "child")
if predicate == "" {
return ""
}
return " AND " + predicate
}
func (s *BunStore) getSidebarSessionIndexAll(
ctx context.Context, store bun.IDB, filter SessionFilter, dialect QueryDialect,
) (SidebarSessionIndex, error) {
rootWhere, rootArgs := sidebarRootFilter(filter, dialect)
var total int
if err := store.NewRaw(
"SELECT COUNT(*) FROM sessions AS session WHERE "+rootWhere,
rootArgs...,
).Scan(ctx, &total); err != nil {
return SidebarSessionIndex{}, fmt.Errorf("counting sidebar roots: %w", err)
}
where, args := BuildSessionFilterSQL(filter, dialect)
var rows []bunmodel.Session
query := store.NewSelect().Model(&rows)
query = applyBunWhere(query, where, args)
if err := query.
OrderExpr(bunSessionActivityOrderExpr("", dialect) + " DESC").
OrderExpr("id DESC").Scan(ctx); err != nil {
return SidebarSessionIndex{}, fmt.Errorf(
"querying sidebar session index: %w", err,
)
}
return SidebarSessionIndex{
Sessions: sidebarRowsFromBun(rows),
Total: total,
}, nil
}
func (s *BunStore) getSidebarSessionIndexPage(
ctx context.Context, store bun.IDB, filter SessionFilter, dialect QueryDialect,
) (SidebarSessionIndex, error) {
if filter.Limit <= 0 || filter.Limit > MaxSessionLimit {
filter.Limit = DefaultSessionLimit
}
rootWhere, rootArgs := sidebarRootFilter(filter, dialect)
childAutomationWhere := sidebarChildAutomationWhere(filter, dialect)
treeSQL := sidebarRootTreeSQL(
rootWhere, childAutomationWhere, filter.Starred,
)
var cursor SessionCursor
total := 0
if filter.Cursor != "" {
var err error
cursor, err = s.DecodeCursor(filter.Cursor)
if err != nil {
return SidebarSessionIndex{}, err
}
total = cursor.Total
}
if total <= 0 {
countSQL := "SELECT COUNT(*) FROM root_candidates"
if filter.Starred {
countSQL = "SELECT COUNT(*) FROM eligible_roots"
}
if err := store.NewRaw(treeSQL+" "+countSQL, rootArgs...).
Scan(ctx, &total); err != nil {
return SidebarSessionIndex{}, fmt.Errorf(
"counting sidebar roots: %w", err,
)
}
}
type rootRow struct {
ID string `bun:"id"`
Activity bunmodel.Timestamp `bun:"activity"`
}
rootActivityJoin := sidebarStarredRootJoin(filter.Starred)
rootActivityExpr := bunSessionActivityExpr("session")
rootActivityOrderExpr := bunSessionActivityOrderExpr("session", dialect)
rootPageSQL := treeSQL + `,
ranked_root_activity(id, activity, activity_order, activity_rank) AS (
SELECT t.root_id AS id,
` + rootActivityExpr + ` AS activity,
` + rootActivityOrderExpr + ` AS activity_order,
ROW_NUMBER() OVER (
PARTITION BY t.root_id
ORDER BY ` + rootActivityOrderExpr + ` DESC, session.id DESC
) AS activity_rank
FROM tree AS t
` + rootActivityJoin + `
JOIN sessions AS session ON session.id = t.id
),
root_activity(id, activity, activity_order) AS (
SELECT id, activity, activity_order
FROM ranked_root_activity
WHERE activity_rank = 1
)
SELECT id, activity
FROM root_activity`
rootPageArgs := append([]any(nil), rootArgs...)
if filter.Cursor != "" {
activity, err := bunmodel.ParseTimestamp(cursor.EndedAt)
if err != nil {
return SidebarSessionIndex{}, fmt.Errorf(
"%w: invalid sidebar activity: %v", ErrInvalidCursor, err,
)
}
activityParam := dialect.dateParam("?")
rootPageSQL += `
WHERE activity_order < ` + activityParam + `
OR (activity_order = ` + activityParam + ` AND id < ?)`
rootPageArgs = append(
rootPageArgs, activity.Time, activity.Time, cursor.ID,
)
}
rootPageSQL += `
ORDER BY activity_order DESC, id DESC
LIMIT ?`
rootPageArgs = append(rootPageArgs, filter.Limit+1)
var roots []rootRow
if err := store.NewRaw(rootPageSQL, rootPageArgs...).Scan(ctx, &roots); err != nil {
return SidebarSessionIndex{}, fmt.Errorf(
"querying sidebar root page: %w", err,
)
}
index := SidebarSessionIndex{
Sessions: []SidebarSessionIndexRow{},
Total: total,
}
if len(roots) == 0 {
return index, nil
}
selected := roots
if len(roots) > filter.Limit {
selected = roots[:filter.Limit]
last := selected[len(selected)-1]
index.NextCursor = s.EncodeCursor(SessionCursor{
EndedAt: requiredTimestampFromBunRow(last.Activity),
ID: last.ID,
Total: total,
})
}
valueRows := make([]string, len(selected))
treeArgs := make([]any, 0, len(selected)*2)
for i, root := range selected {
valueRows[i] = "(?, ?)"
treeArgs = append(treeArgs, root.ID, i)
}
treePageSQL := `WITH RECURSIVE root_page(id, ord) AS (
VALUES ` + strings.Join(valueRows, ", ") + `
),
tree(id, ord) AS (
SELECT id, ord FROM root_page
UNION
SELECT child.id, tree.ord
FROM sessions AS child
JOIN tree ON child.parent_session_id = tree.id
WHERE child.message_count > 0
AND child.deleted_at IS NULL` + childAutomationWhere + `
),
ranked_tree(id, ord) AS (
SELECT id, MIN(ord) AS ord
FROM tree
GROUP BY id
)
SELECT ` + bunModelColumns("session", (*bunmodel.Session)(nil)) + `
FROM sessions AS session
JOIN ranked_tree AS ranked ON session.id = ranked.id
ORDER BY ranked.ord ASC, ` +
bunSessionActivityOrderExpr("session", dialect) + ` DESC,
session.id DESC`
var rows []bunmodel.Session
if err := store.NewRaw(treePageSQL, treeArgs...).Scan(ctx, &rows); err != nil {
return SidebarSessionIndex{}, fmt.Errorf(
"querying sidebar tree page: %w", err,
)
}
index.Sessions = sidebarRowsFromBun(rows)
return index, nil
}
func bunModelColumns(alias string, model any) string {
columns := bunmodel.ModelColumns(model)
for i, column := range columns {
columns[i] = alias + `."` + column + `"`
}
return strings.Join(columns, ", ")
}
func sidebarRowsFromBun(rows []bunmodel.Session) []SidebarSessionIndexRow {
out := make([]SidebarSessionIndexRow, 0, len(rows))
for _, row := range rows {
out = append(out, sidebarRowFromBun(row))
}
return out
}
func applyBunWhere(
query *bun.SelectQuery, where string, args []any,
) *bun.SelectQuery {
if len(args) == 0 {
return query.Where(where)
}
return query.Where(where, args...)
}
func sidebarRowFromBun(row bunmodel.Session) SidebarSessionIndexRow {
displayName := row.DisplayName
if displayName == nil {
displayName = row.SessionName
}
return SidebarSessionIndexRow{
ID: row.ID, ParentSessionID: row.ParentSessionID,
RelationshipType: row.RelationshipType, Project: row.Project,
Machine: row.Machine, Agent: row.Agent, AgentLabel: row.AgentLabel,
Entrypoint: row.Entrypoint, SessionKind: row.SessionKind,
DisplayName: displayName, StartedAt: timestampFromBunRow(row.StartedAt),
EndedAt: timestampFromBunRow(row.EndedAt),
CreatedAt: requiredTimestampFromBunRow(row.CreatedAt),
TerminationStatus: row.TerminationStatus, MessageCount: row.MessageCount,
UserMessageCount: row.UserMessageCount,
TranscriptRevision: &row.TranscriptRevision,
IsAutomated: row.IsAutomated,
IsTeammate: row.FirstMessage != nil && strings.Contains(*row.FirstMessage, "<teammate-message"),
}
}
// GetSessionVersion returns the canonical message count and source marker.
func (s *BunStore) GetSessionVersion(id string) (int, int64, bool) {
var count int
var marker int64
err := s.view(context.Background(), func(store bun.IDB) error {
var versionErr error
count, marker, versionErr = s.backend.SessionVersion(
context.Background(), store, id,
)
return versionErr
})
if err != nil {
return 0, 0, false
}
return count, marker, true
}
// FileSessionVersion returns the portable file-fingerprint marker used by the
// SQLite archive and DuckDB mirror.
func FileSessionVersion(
ctx context.Context, store bun.IDB, id string,
) (int, int64, error) {
var row struct {
MessageCount int `bun:"message_count"`
FileMtime *int64 `bun:"file_mtime"`
FileHash *string `bun:"file_hash"`
LocalModifiedAt *bunmodel.Timestamp `bun:"local_modified_at"`
}
err := store.NewSelect().Table("sessions").
Column("message_count", "file_mtime", "file_hash", "local_modified_at").
Where("id = ?", id).Scan(ctx, &row)
if err != nil {
return 0, 0, err
}
mtime := int64(0)
if row.FileMtime != nil {
mtime = *row.FileMtime
}
hash := ""
if row.FileHash != nil {
hash = *row.FileHash
}
modified := requiredTimestampFromBunRowPtr(row.LocalModifiedAt)
return row.MessageCount, SessionVersionMarker(
fmt.Sprintf("%d", mtime), hash, modified,
), nil
}
func applyBunRootVisibility(
query *bun.SelectQuery, excludeOneShot, excludeAutomated bool,
) *bun.SelectQuery {
query = query.Where("message_count > 0").
Where("relationship_type NOT IN (?)", bun.List([]string{"subagent", "fork"})).
Where("deleted_at IS NULL")
if excludeOneShot {
if excludeAutomated {
query = query.Where("user_message_count > 1")
} else {
query = query.WhereGroup(" AND ", func(q *bun.SelectQuery) *bun.SelectQuery {
return q.Where("user_message_count > 1").WhereOr("is_automated = ?", true)
})
}
}
if excludeAutomated {
query = query.Where("is_automated = ?", false)
}
return query
}
// GetStats returns aggregate metadata for visible root sessions.
func (s *BunStore) GetStats(
ctx context.Context, excludeOneShot, excludeAutomated bool,
) (Stats, error) {
var row struct {
SessionCount int `bun:"session_count"`
MessageCount int `bun:"message_count"`
ProjectCount int `bun:"project_count"`
MachineCount int `bun:"machine_count"`
EarliestSession *bunmodel.Timestamp `bun:"earliest_session"`
}
err := s.view(ctx, func(store bun.IDB) error {
query := store.NewSelect().Table("sessions").
ColumnExpr("COUNT(*) AS session_count").
ColumnExpr("CAST(COALESCE(SUM(message_count), 0) AS BIGINT) AS message_count").
ColumnExpr("COUNT(DISTINCT project) AS project_count").
ColumnExpr("COUNT(DISTINCT machine) AS machine_count").
ColumnExpr("MIN(COALESCE(" + bunNullableTimestamp("started_at") +
", created_at)) AS earliest_session")
return applyBunRootVisibility(query, excludeOneShot, excludeAutomated).
Scan(ctx, &row)
})
if err != nil {
return Stats{}, fmt.Errorf("fetching stats: %w", err)
}
return Stats{
SessionCount: row.SessionCount, MessageCount: row.MessageCount,
ProjectCount: row.ProjectCount, MachineCount: row.MachineCount,
EarliestSession: timestampFromBunRow(row.EarliestSession),
}, nil
}
// GetProjects returns visible root-session counts by project.
func (s *BunStore) GetProjects(
ctx context.Context, excludeOneShot, excludeAutomated bool,
) ([]ProjectInfo, error) {
var rows []struct {
Name string `bun:"name"`
SessionCount int `bun:"session_count"`
}
err := s.view(ctx, func(store bun.IDB) error {
query := store.NewSelect().Table("sessions").
ColumnExpr("project AS name").
ColumnExpr("COUNT(*) AS session_count")
return applyBunRootVisibility(query, excludeOneShot, excludeAutomated).
Group("project").OrderExpr("project ASC").Scan(ctx, &rows)
})
if err != nil {
return nil, fmt.Errorf("querying projects: %w", err)
}
projects := make([]ProjectInfo, 0, len(rows))
for _, row := range rows {
projects = append(projects, ProjectInfo{
Name: row.Name, SessionCount: row.SessionCount,
})
}
return projects, nil
}
// GetActiveProjectLabels returns all project labels on non-deleted sessions.
func (s *BunStore) GetActiveProjectLabels(ctx context.Context) ([]string, error) {
var labels []string
err := s.view(ctx, func(store bun.IDB) error {
return store.NewSelect().Table("sessions").Distinct().Column("project").
Where("deleted_at IS NULL").OrderExpr("project ASC").Scan(ctx, &labels)
})
if err != nil {
return nil, fmt.Errorf("querying active project labels: %w", err)
}
return labels, nil
}
// GetAgents returns visible root-session counts by agent.
func (s *BunStore) GetAgents(
ctx context.Context, excludeOneShot, excludeAutomated bool,
) ([]AgentInfo, error) {
var rows []struct {
Name string `bun:"name"`
SessionCount int `bun:"session_count"`
}
err := s.view(ctx, func(store bun.IDB) error {
query := store.NewSelect().Table("sessions").
ColumnExpr("agent AS name").ColumnExpr("COUNT(*) AS session_count").
Where("agent <> ''")
return applyBunRootVisibility(query, excludeOneShot, excludeAutomated).
Group("agent").OrderExpr("agent ASC").Scan(ctx, &rows)
})
if err != nil {
return nil, fmt.Errorf("querying agents: %w", err)
}
agents := make([]AgentInfo, 0, len(rows))
for _, row := range rows {
agents = append(agents, AgentInfo{
Name: row.Name, SessionCount: row.SessionCount,
})
}
return agents, nil
}
// GetMachines returns distinct machine names under the requested visibility.
func (s *BunStore) GetMachines(
ctx context.Context, excludeOneShot, excludeAutomated bool,
) ([]string, error) {
var machines []string
err := s.view(ctx, func(store bun.IDB) error {
query := store.NewSelect().Table("sessions").Distinct().Column("machine").
Where("deleted_at IS NULL")
if excludeOneShot {
if excludeAutomated {
query = query.Where("user_message_count > 1")
} else {
query = query.WhereGroup(" AND ", func(q *bun.SelectQuery) *bun.SelectQuery {
return q.Where("user_message_count > 1").WhereOr("is_automated = ?", true)
})
}
}
if excludeAutomated {
query = query.Where("is_automated = ?", false)
}
return query.OrderExpr("machine ASC").Scan(ctx, &machines)
})
if err != nil {
return nil, fmt.Errorf("querying machines: %w", err)
}
return machines, nil
}
// GetBranches returns distinct project/branch pairs for visible roots.
func (s *BunStore) GetBranches(
ctx context.Context, excludeOneShot, excludeAutomated bool,
) ([]BranchInfo, error) {
var rows []struct {
Project string `bun:"project"`
Branch string `bun:"git_branch"`
}
err := s.view(ctx, func(store bun.IDB) error {
query := store.NewSelect().Table("sessions").Distinct().
Column("project", "git_branch")
return applyBunRootVisibility(query, excludeOneShot, excludeAutomated).
OrderExpr("project ASC").OrderExpr("git_branch ASC").Scan(ctx, &rows)
})
if err != nil {
return nil, fmt.Errorf("querying branches: %w", err)
}
branches := make([]BranchInfo, 0, len(rows))
for _, row := range rows {
branches = append(branches, BranchInfo{
Project: row.Project, Branch: row.Branch,
Token: EncodeBranchFilterToken(row.Project, row.Branch),
})
}
return branches, nil
}
+451
View File
@@ -0,0 +1,451 @@
package db
import (
"context"
"database/sql"
"fmt"
"strings"
"testing"
"time"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
"github.com/uptrace/bun"
"github.com/uptrace/bun/dialect/sqlitedialect"
"go.kenn.io/agentsview/internal/db/bunmodel"
)
type sessionContractBackend struct {
store bun.IDB
viewCalls int
}
type replayingReadBackend struct {
first, second bun.IDB
}
func (*replayingReadBackend) Name() string { return "replaying-read" }
func (*replayingReadBackend) ReadOnly() bool { return true }
func (*replayingReadBackend) Capabilities() BackendCapabilities {
return BackendCapabilities{}
}
func (*replayingReadBackend) SessionQueryDialect() QueryDialect {
return SQLiteBunSessionQueryDialect()
}
func (*replayingReadBackend) SessionVersion(
context.Context, bun.IDB, string,
) (int, int64, error) {
return 0, 0, sql.ErrNoRows
}
func (b *replayingReadBackend) View(
_ context.Context, fn func(bun.IDB) error,
) error {
return fn(b.first)
}
func (b *replayingReadBackend) ConsistentView(
_ context.Context, fn func(bun.IDB) error,
) error {
if err := fn(b.first); err != nil {
return err
}
return fn(b.second)
}
func (*replayingReadBackend) Update(
context.Context, func(bun.IDB) error,
) error {
return ErrReadOnly
}
func (*sessionContractBackend) Name() string { return "session-contract" }
func (*sessionContractBackend) ReadOnly() bool { return true }
func (*sessionContractBackend) Capabilities() BackendCapabilities {
return BackendCapabilities{}
}
func (*sessionContractBackend) SessionQueryDialect() QueryDialect {
return SQLiteBunSessionQueryDialect()
}
func (*sessionContractBackend) SessionVersion(
ctx context.Context, store bun.IDB, id string,
) (int, int64, error) {
return FileSessionVersion(ctx, store, id)
}
func (b *sessionContractBackend) View(
_ context.Context, fn func(bun.IDB) error,
) error {
b.viewCalls++
return fn(b.store)
}
func (b *sessionContractBackend) ConsistentView(
_ context.Context, fn func(bun.IDB) error,
) error {
b.viewCalls++
return fn(b.store)
}
func (*sessionContractBackend) Update(
context.Context, func(bun.IDB) error,
) error {
return ErrReadOnly
}
type countingQueryHook struct {
selects int
queries []string
}
func (h *countingQueryHook) BeforeQuery(
ctx context.Context, _ *bun.QueryEvent,
) context.Context {
return ctx
}
func (h *countingQueryHook) AfterQuery(
_ context.Context, event *bun.QueryEvent,
) {
if event.Operation() == "SELECT" {
h.selects++
h.queries = append(h.queries, event.Query)
}
}
func TestBunStoreFindSessionIDsByPartialUsesBoundedKeysetBatches(t *testing.T) {
raw, err := sql.Open("sqlite3", ":memory:")
require.NoError(t, err)
raw.SetMaxOpenConns(1)
t.Cleanup(func() { require.NoError(t, raw.Close()) })
store := bun.NewDB(raw, sqlitedialect.New())
require.NoError(t, CreateCommonSchema(t.Context(), store))
_, err = store.NewInsert().Model(&bunmodel.SourceArchive{
SourceArchiveID: "archive", SourceArchiveSalt: "salt",
}).Exec(t.Context())
require.NoError(t, err)
newer := bunmodel.NewTimestamp(
time.Date(2026, 8, 3, 12, 0, 0, 0, time.UTC),
)
older := bunmodel.NewTimestamp(
time.Date(2026, 8, 2, 12, 0, 0, 0, time.UTC),
)
rows := make([]bunmodel.Session, 0, 65)
for i := range 64 {
rows = append(rows, bunmodel.Session{
ID: fmt.Sprintf("MATCH-%03d", i), Project: "alpha",
Machine: "host", Agent: "codex", CreatedAt: newer,
SourceArchiveID: "archive", SourceDatabaseGeneration: "generation",
})
}
rows = append(rows, bunmodel.Session{
ID: "match-valid", Project: "alpha", Machine: "host", Agent: "codex",
CreatedAt: older, SourceArchiveID: "archive",
SourceDatabaseGeneration: "generation",
})
_, err = store.NewInsert().Model(&rows).Exec(t.Context())
require.NoError(t, err)
hook := new(countingQueryHook)
backend := &sessionContractBackend{store: store.WithQueryHook(hook)}
common := NewBunStore(backend)
ids, err := common.FindSessionIDsByPartial(t.Context(), "match", 1)
require.NoError(t, err)
assert.Equal(t, []string{"match-valid"}, ids)
assert.Equal(t, 1, backend.viewCalls)
assert.Equal(t, 2, hook.selects)
require.Len(t, hook.queries, 2)
for _, query := range hook.queries {
assert.True(t, strings.Contains(query, "LIMIT 64"), query)
}
}
func TestBunStoreListSessionsKeepsQueriesAndResultsBounded(t *testing.T) {
for _, matchingRows := range []int{2, 50} {
t.Run(fmt.Sprintf("matching_rows_%d", matchingRows), func(t *testing.T) {
raw, err := sql.Open("sqlite3", ":memory:")
require.NoError(t, err)
raw.SetMaxOpenConns(1)
t.Cleanup(func() { require.NoError(t, raw.Close()) })
store := bun.NewDB(raw, sqlitedialect.New())
require.NoError(t, CreateCommonSchema(t.Context(), store))
_, err = store.NewInsert().Model(&bunmodel.SourceArchive{
SourceArchiveID: "archive", SourceArchiveSalt: "salt",
}).Exec(t.Context())
require.NoError(t, err)
createdAt := bunmodel.NewTimestamp(
time.Date(2026, 8, 2, 12, 0, 0, 0, time.UTC),
)
rows := make([]bunmodel.Session, 0, matchingRows+1)
rows = append(rows, bunmodel.Session{
ID: "wanted", Project: "alpha", Machine: "host", Agent: "codex",
MessageCount: 2, UserMessageCount: 2, CreatedAt: createdAt,
SourceArchiveID: "archive", SourceDatabaseGeneration: "generation",
})
for i := 1; i < matchingRows; i++ {
rows = append(rows, bunmodel.Session{
ID: fmt.Sprintf("extra-%03d", i), Project: "alpha",
Machine: "host", Agent: "codex", MessageCount: 2,
UserMessageCount: 2, CreatedAt: createdAt,
SourceArchiveID: "archive", SourceDatabaseGeneration: "generation",
})
}
rows = append(rows, bunmodel.Session{
ID: "other-project", Project: "beta", Machine: "host", Agent: "codex",
MessageCount: 2, UserMessageCount: 2, CreatedAt: createdAt,
SourceArchiveID: "archive", SourceDatabaseGeneration: "generation",
})
_, err = store.NewInsert().Model(&rows).Exec(t.Context())
require.NoError(t, err)
hook := new(countingQueryHook)
backend := &sessionContractBackend{store: store.WithQueryHook(hook)}
common := NewBunStore(backend)
page, err := common.ListSessions(t.Context(), SessionFilter{
Project: "alpha", Limit: 1,
})
require.NoError(t, err)
require.Len(t, page.Sessions, 1)
assert.Equal(t, "wanted", page.Sessions[0].ID)
assert.Equal(t, matchingRows, page.Total)
assert.NotEmpty(t, page.NextCursor)
assert.Equal(t, 1, backend.viewCalls)
assert.Equal(t, 2, hook.selects, "count plus bounded page query")
})
}
}
func TestBunStoreSessionCompositeReadsPublishOnlyAcceptedReplayAttempt(
t *testing.T,
) {
first := testDB(t)
second := testDB(t)
seed := func(database *DB, rootIDs []string, childID string) {
t.Helper()
for _, id := range rootIDs {
require.NoError(t, database.UpsertSession(Session{
ID: id, Project: "replaying-reads", Machine: "host", Agent: "codex",
MessageCount: 1, UserMessageCount: 1,
}))
}
parentID := rootIDs[0]
require.NoError(t, database.UpsertSession(Session{
ID: childID, Project: "replaying-reads", Machine: "host", Agent: "codex",
MessageCount: 1, UserMessageCount: 1, ParentSessionID: &parentID,
RelationshipType: "subagent",
}))
}
seed(first, []string{"rejected-root-a", "rejected-root-b"}, "rejected-child")
seed(second, []string{"accepted-root"}, "accepted-child")
store := NewBunStore(&replayingReadBackend{
first: first.bunReader, second: second.bunReader,
})
t.Run("list sessions", func(t *testing.T) {
page, err := store.ListSessions(t.Context(), SessionFilter{
Project: "replaying-reads", Limit: 10,
})
require.NoError(t, err)
assert.Equal(t, 1, page.Total)
require.Len(t, page.Sessions, 1)
assert.Equal(t, "accepted-root", page.Sessions[0].ID)
})
t.Run("sidebar index", func(t *testing.T) {
index, err := store.GetSidebarSessionIndex(t.Context(), SessionFilter{
Project: "replaying-reads",
})
require.NoError(t, err)
assert.Equal(t, 1, index.Total)
require.Len(t, index.Sessions, 2)
assert.ElementsMatch(t, []string{"accepted-root", "accepted-child"}, []string{
index.Sessions[0].ID, index.Sessions[1].ID,
})
})
}
func TestBunStoreListSessionsUsesChronologicalSQLiteActivity(t *testing.T) {
raw, err := sql.Open("sqlite3", ":memory:")
require.NoError(t, err)
raw.SetMaxOpenConns(1)
t.Cleanup(func() { require.NoError(t, raw.Close()) })
store := bun.NewDB(raw, sqlitedialect.New())
require.NoError(t, CreateCommonSchema(t.Context(), store))
_, err = store.NewInsert().Model(&bunmodel.SourceArchive{
SourceArchiveID: "archive", SourceArchiveSalt: "salt",
}).Exec(t.Context())
require.NoError(t, err)
_, err = raw.ExecContext(t.Context(), `
INSERT INTO sessions (
id, project, machine, agent, message_count, created_at,
started_at, ended_at, source_archive_id, source_database_generation
) VALUES
('fractional-activity', 'time', 'host', 'codex', 2,
'2024-01-01T00:00:00Z', '2024-01-01T00:00:00Z', NULL,
'archive', 'generation'),
('offset-before-cutoff', 'time', 'host', 'codex', 1,
'2024-01-01T00:00:00Z', '2024-01-01T00:00:00Z',
'2024-01-01T01:00:00+01:00', 'archive', 'generation');
INSERT INTO messages (session_id, ordinal, role, content, timestamp, token_usage)
VALUES
('fractional-activity', 0, 'assistant', '',
'2024-01-01T00:00:01Z', '{}'),
('fractional-activity', 1, 'assistant', '',
'2024-01-01T00:00:01.500Z', '{}')`)
require.NoError(t, err)
page, err := NewBunStore(&sessionContractBackend{store: store}).ListSessions(
t.Context(), SessionFilter{
Project: "time", ActiveSince: "2024-01-01T00:00:01.250Z", Limit: 10,
},
)
require.NoError(t, err)
require.Len(t, page.Sessions, 1)
assert.Equal(t, "fractional-activity", page.Sessions[0].ID)
}
func TestBunStoreListSessionsPaginatesSQLiteTimestampsChronologically(t *testing.T) {
raw, err := sql.Open("sqlite3", ":memory:")
require.NoError(t, err)
raw.SetMaxOpenConns(1)
t.Cleanup(func() { require.NoError(t, raw.Close()) })
store := bun.NewDB(raw, sqlitedialect.New())
require.NoError(t, CreateCommonSchema(t.Context(), store))
_, err = store.NewInsert().Model(&bunmodel.SourceArchive{
SourceArchiveID: "archive", SourceArchiveSalt: "salt",
}).Exec(t.Context())
require.NoError(t, err)
_, err = raw.ExecContext(t.Context(), `
INSERT INTO sessions (
id, project, machine, agent, message_count, created_at,
ended_at, source_archive_id, source_database_generation
) VALUES
('chronological-new', 'time', 'host', 'codex', 1,
'2024-01-01T00:30:00Z', '2024-01-01T00:30:00Z',
'archive', 'generation'),
('lexical-new', 'time', 'host', 'codex', 1,
'2024-01-01T01:00:00+01:00', '2024-01-01T01:00:00+01:00',
'archive', 'generation')`)
require.NoError(t, err)
common := NewBunStore(&sessionContractBackend{store: store})
first, err := common.ListSessions(t.Context(), SessionFilter{
Project: "time", Limit: 1,
})
require.NoError(t, err)
require.Len(t, first.Sessions, 1)
assert.Equal(t, "chronological-new", first.Sessions[0].ID)
assert.NotEmpty(t, first.NextCursor)
second, err := common.ListSessions(t.Context(), SessionFilter{
Project: "time", Limit: 1, Cursor: first.NextCursor,
})
require.NoError(t, err)
require.Len(t, second.Sessions, 1)
assert.Equal(t, "lexical-new", second.Sessions[0].ID)
assert.Empty(t, second.NextCursor)
}
func TestBunStoreTerminationFilterUsesSQLiteInstants(t *testing.T) {
raw, err := sql.Open("sqlite3", ":memory:")
require.NoError(t, err)
raw.SetMaxOpenConns(1)
t.Cleanup(func() { require.NoError(t, raw.Close()) })
store := bun.NewDB(raw, sqlitedialect.New())
require.NoError(t, CreateCommonSchema(t.Context(), store))
_, err = store.NewInsert().Model(&bunmodel.SourceArchive{
SourceArchiveID: "archive", SourceArchiveSalt: "salt",
}).Exec(t.Context())
require.NoError(t, err)
now := time.Now().UTC()
offset := time.FixedZone("plus-three", 3*60*60)
activeText := now.Add(-5 * time.Minute).In(offset).Format(time.RFC3339Nano)
staleText := now.Add(-30 * time.Minute).In(offset).Format(time.RFC3339Nano)
uncleanText := now.Add(-2 * time.Hour).In(offset).Format(time.RFC3339Nano)
_, err = raw.ExecContext(t.Context(), `
INSERT INTO sessions (
id, project, machine, agent, message_count, created_at, ended_at,
termination_status, source_archive_id, source_database_generation
) VALUES
('offset-active', 'time', 'host', 'codex', 1, ?, ?, 'tool_call_pending',
'archive', 'generation'),
('offset-stale', 'time', 'host', 'codex', 1, ?, ?, 'tool_call_pending',
'archive', 'generation'),
('offset-unclean', 'time', 'host', 'codex', 1, ?, ?, 'tool_call_pending',
'archive', 'generation')`,
activeText, activeText, staleText, staleText, uncleanText, uncleanText,
)
require.NoError(t, err)
common := NewBunStore(&sessionContractBackend{store: store})
for _, test := range []struct {
termination string
wantID string
}{
{termination: "active", wantID: "offset-active"},
{termination: "stale", wantID: "offset-stale"},
{termination: "unclean", wantID: "offset-unclean"},
} {
t.Run(test.termination, func(t *testing.T) {
page, err := common.ListSessions(t.Context(), SessionFilter{
Project: "time", Termination: test.termination, Limit: 10,
})
require.NoError(t, err)
require.Len(t, page.Sessions, 1)
assert.Equal(t, test.wantID, page.Sessions[0].ID)
})
}
}
func TestBunStoreSidebarPaginatesSQLiteActivityChronologically(t *testing.T) {
raw, err := sql.Open("sqlite3", ":memory:")
require.NoError(t, err)
raw.SetMaxOpenConns(1)
t.Cleanup(func() { require.NoError(t, raw.Close()) })
store := bun.NewDB(raw, sqlitedialect.New())
require.NoError(t, CreateCommonSchema(t.Context(), store))
_, err = store.NewInsert().Model(&bunmodel.SourceArchive{
SourceArchiveID: "archive", SourceArchiveSalt: "salt",
}).Exec(t.Context())
require.NoError(t, err)
_, err = raw.ExecContext(t.Context(), `
INSERT INTO sessions (
id, project, machine, agent, message_count, created_at, ended_at,
source_archive_id, source_database_generation
) VALUES
('chronological-new', 'time', 'host', 'codex', 1,
'2024-01-01T00:30:00Z', '2024-01-01T00:30:00Z',
'archive', 'generation'),
('lexical-new', 'time', 'host', 'codex', 1,
'2024-01-01T01:00:00+01:00', '2024-01-01T01:00:00+01:00',
'archive', 'generation')`)
require.NoError(t, err)
common := NewBunStore(&sessionContractBackend{store: store})
first, err := common.GetSidebarSessionIndex(t.Context(), SessionFilter{
Project: "time", Limit: 1,
})
require.NoError(t, err)
require.Len(t, first.Sessions, 1)
assert.Equal(t, "chronological-new", first.Sessions[0].ID)
assert.NotEmpty(t, first.NextCursor)
second, err := common.GetSidebarSessionIndex(t.Context(), SessionFilter{
Project: "time", Limit: 1, Cursor: first.NextCursor,
})
require.NoError(t, err)
require.Len(t, second.Sessions, 1)
assert.Equal(t, "lexical-new", second.Sessions[0].ID)
assert.Empty(t, second.NextCursor)
}
+97
View File
@@ -0,0 +1,97 @@
package db
import (
"context"
"sync"
"github.com/uptrace/bun"
"go.kenn.io/agentsview/internal/config"
"go.kenn.io/agentsview/internal/export"
)
type pricingState struct {
custom map[string]config.CustomModelRate
effective map[string]export.ModelRates
emptyCatalog map[string]export.ModelRates
}
// BunStore is the common query runtime embedded by every concrete store.
type BunStore struct {
backend BunBackend
cursorMu sync.RWMutex
cursorSecret []byte
pricingMu sync.RWMutex
pricing pricingState
}
// NewBunStore creates a shared store over one guarded backend.
func NewBunStore(backend BunBackend) *BunStore {
return &BunStore{
backend: backend,
pricing: pricingState{emptyCatalog: fallbackRateMap()},
}
}
// SetCursorSecret updates the shared cursor signing key.
func (s *BunStore) SetCursorSecret(secret []byte) {
s.cursorMu.Lock()
defer s.cursorMu.Unlock()
s.cursorSecret = append([]byte(nil), secret...)
}
// SetCustomPricing updates the shared custom pricing overrides.
func (s *BunStore) SetCustomPricing(pricing map[string]config.CustomModelRate) {
s.pricingMu.Lock()
defer s.pricingMu.Unlock()
s.pricing.custom = pricing
s.pricing.effective = nil
}
// SetEffectivePricing updates the shared effective pricing catalogue.
func (s *BunStore) SetEffectivePricing(pricing map[string]export.ModelRates) {
s.pricingMu.Lock()
defer s.pricingMu.Unlock()
s.pricing.custom = nil
s.pricing.effective = cloneModelRates(pricing)
}
// SetEmptyCatalogPricing updates the shared empty-catalog fallback rates.
func (s *BunStore) SetEmptyCatalogPricing(pricing map[string]export.ModelRates) {
s.pricingMu.Lock()
defer s.pricingMu.Unlock()
s.pricing.emptyCatalog = cloneModelRates(pricing)
}
func cloneModelRates(pricing map[string]export.ModelRates) map[string]export.ModelRates {
clone := make(map[string]export.ModelRates, len(pricing))
for model, rates := range pricing {
rates.Bands = append([]export.PricingBand(nil), rates.Bands...)
clone[model] = rates
}
return clone
}
func (s *BunStore) view(
ctx context.Context, fn func(bun.IDB) error,
) error {
return s.backend.View(ctx, fn)
}
func (s *BunStore) consistentView(
ctx context.Context, fn func(bun.IDB) error,
) error {
return s.backend.ConsistentView(ctx, fn)
}
func (s *BunStore) update(
ctx context.Context,
operation WriteOperation,
fn func(bun.IDB) error,
) error {
if !s.backend.Capabilities().AllowsWrite(operation) {
return ErrReadOnly
}
return s.backend.Update(ctx, fn)
}
@@ -0,0 +1,281 @@
package db_test
import (
"database/sql"
"path/filepath"
"testing"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
"go.kenn.io/agentsview/internal/db"
"go.kenn.io/agentsview/internal/storetest"
)
func TestBunStoreCoreContract(t *testing.T) {
storetest.RunCoreContract(t, storetest.Backend{
Name: "sqlite",
Open: func(t *testing.T) storetest.CoreStore {
database, err := db.Open(filepath.Join(t.TempDir(), "core-contract.db"))
require.NoError(t, err)
t.Cleanup(func() { require.NoError(t, database.Close()) })
archiveID, err := database.GetArchiveID(t.Context())
require.NoError(t, err)
generation, err := database.GetDatabaseID(t.Context())
require.NoError(t, err)
require.NoError(t, database.Update(func(tx *sql.Tx) error {
return storetest.InsertSQLiteCoreFixture(
t.Context(), tx, archiveID, generation,
)
}))
return database.BunStore
},
Seed: func(*testing.T, storetest.CoreStore) storetest.Fixture {
return storetest.CoreFixture()
},
})
}
func TestBunStoreIdentityContract(t *testing.T) {
storetest.RunIdentityContract(t, storetest.IdentityBackend{
Name: "sqlite",
Open: func(t *testing.T) (storetest.IdentityStore, storetest.IdentityFixture) {
database, err := db.Open(filepath.Join(t.TempDir(), "identity-contract.db"))
require.NoError(t, err)
t.Cleanup(func() { require.NoError(t, database.Close()) })
archiveID, err := database.GetArchiveID(t.Context())
require.NoError(t, err)
archiveSalt, err := database.GetArchiveSalt(t.Context())
require.NoError(t, err)
var fixture storetest.IdentityFixture
require.NoError(t, database.Update(func(tx *sql.Tx) error {
var insertErr error
fixture, insertErr = storetest.InsertSQLiteIdentityFixture(
t.Context(), tx, archiveID, archiveSalt,
)
return insertErr
}))
return database, fixture
},
})
}
func TestBunStoreDataContract(t *testing.T) {
storetest.RunDataContract(t, storetest.DataBackend{
Name: "sqlite",
Open: func(t *testing.T) (storetest.DataStore, storetest.IdentityFixture) {
database, err := db.Open(filepath.Join(t.TempDir(), "data-contract.db"))
require.NoError(t, err)
t.Cleanup(func() { require.NoError(t, database.Close()) })
archiveID, err := database.GetArchiveID(t.Context())
require.NoError(t, err)
archiveSalt, err := database.GetArchiveSalt(t.Context())
require.NoError(t, err)
var fixture storetest.IdentityFixture
require.NoError(t, database.Update(func(tx *sql.Tx) error {
var insertErr error
fixture, insertErr = storetest.InsertSQLiteIdentityFixture(
t.Context(), tx, archiveID, archiveSalt,
)
return insertErr
}))
return database.BunStore, fixture
},
})
}
func TestBunStoreCurationContract(t *testing.T) {
storetest.RunCurationContract(t, storetest.CurationBackend{
Name: "sqlite", Writable: true,
Open: func(t *testing.T) (storetest.CurationStore, storetest.CurationFixture) {
database, err := db.Open(filepath.Join(t.TempDir(), "curation-contract.db"))
require.NoError(t, err)
t.Cleanup(func() { require.NoError(t, database.Close()) })
archiveID, err := database.GetArchiveID(t.Context())
require.NoError(t, err)
generation, err := database.GetDatabaseID(t.Context())
require.NoError(t, err)
var fixture storetest.CurationFixture
require.NoError(t, database.Update(func(tx *sql.Tx) error {
var insertErr error
fixture, insertErr = storetest.InsertSQLiteCurationFixture(
t.Context(), tx, archiveID, generation,
)
return insertErr
}))
return database.BunStore, fixture
},
})
}
func TestBunStoreInsightContract(t *testing.T) {
storetest.RunInsightContract(t, storetest.InsightBackend{
Name: "sqlite", Writable: true,
Open: func(t *testing.T) (storetest.InsightStore, storetest.InsightFixture) {
database, err := db.Open(filepath.Join(t.TempDir(), "insight-contract.db"))
require.NoError(t, err)
t.Cleanup(func() { require.NoError(t, database.Close()) })
var fixture storetest.InsightFixture
require.NoError(t, database.Update(func(tx *sql.Tx) error {
var insertErr error
fixture, insertErr = storetest.InsertSQLiteInsightFixture(t.Context(), tx)
return insertErr
}))
return database.BunStore, fixture
},
})
}
func TestBunStoreMutationContract(t *testing.T) {
storetest.RunMutationContract(t, storetest.MutationBackend{
Name: "sqlite", Writable: true,
Open: func(t *testing.T, extraTrashRows int) storetest.MutationHarness {
database, err := db.Open(filepath.Join(t.TempDir(), "mutation-contract.db"))
require.NoError(t, err)
t.Cleanup(func() { require.NoError(t, database.Close()) })
archiveID, err := database.GetArchiveID(t.Context())
require.NoError(t, err)
generation, err := database.GetDatabaseID(t.Context())
require.NoError(t, err)
var fixture storetest.MutationFixture
require.NoError(t, database.Update(func(tx *sql.Tx) error {
var insertErr error
fixture, insertErr = storetest.InsertSQLiteMutationFixture(
t.Context(), tx, archiveID, generation, extraTrashRows,
)
return insertErr
}))
return storetest.MutationHarness{
Store: database.BunStore,
Rows: fixture,
IsExcluded: func(t *testing.T, id string) bool {
t.Helper()
return database.IsSessionExcluded(id)
},
RestoreBaselinePresent: func(t *testing.T, id string) bool {
t.Helper()
var count int
require.NoError(t, database.Reader().QueryRowContext(t.Context(), `
SELECT count(*) FROM local_session_source_baselines
WHERE session_id = ?`, id).Scan(&count))
return count > 0
},
}
},
})
}
func TestBunStoreRecallContract(t *testing.T) {
storetest.RunRecallContract(t, storetest.RecallBackend{
Name: "sqlite", Readable: true, Writable: true,
Open: func(t *testing.T) storetest.RecallStore {
database, err := db.Open(filepath.Join(t.TempDir(), "recall-contract.db"))
require.NoError(t, err)
t.Cleanup(func() { require.NoError(t, database.Close()) })
require.NoError(t, database.UpsertSession(db.Session{
ID: "bun-recall-source", Project: "recall-contract",
Machine: "contract-machine", Agent: "codex",
}))
return database.BunStore
},
})
}
func TestBunStoreReadOnlyRecallContract(t *testing.T) {
storetest.RunRecallContract(t, storetest.RecallBackend{
Name: "sqlite-read-only", Readable: true,
Open: func(t *testing.T) storetest.RecallStore {
path := filepath.Join(t.TempDir(), "read-only-recall-contract.db")
database, err := db.Open(path)
require.NoError(t, err)
require.NoError(t, database.UpsertSession(db.Session{
ID: "bun-recall-source", Project: "recall-contract",
Machine: "contract-machine", Agent: "codex",
}))
_, err = database.InsertRecallEntry(db.RecallEntry{
ID: "bun-recall-entry", Type: "fact", Scope: "project",
Title: "Canonical Recall entry",
Body: "Recall reads remain available from a read-only archive.",
Project: "recall-contract", SourceSessionID: "bun-recall-source",
Transferable: true, ProvenanceOK: true,
})
require.NoError(t, err)
require.NoError(t, database.Close())
readOnly, err := db.OpenReadOnly(path)
require.NoError(t, err)
t.Cleanup(func() { require.NoError(t, readOnly.Close()) })
return readOnly.BunStore
},
})
}
func TestBunStoreUsageContract(t *testing.T) {
storetest.RunUsageContract(t, storetest.UsageBackend{
Name: "sqlite",
Open: func(t *testing.T) storetest.UsageStore {
database, err := db.Open(filepath.Join(t.TempDir(), "usage-contract.db"))
require.NoError(t, err)
t.Cleanup(func() { require.NoError(t, database.Close()) })
archiveID, err := database.GetArchiveID(t.Context())
require.NoError(t, err)
generation, err := database.GetDatabaseID(t.Context())
require.NoError(t, err)
require.NoError(t, database.Update(func(tx *sql.Tx) error {
return storetest.InsertSQLiteUsageFixture(
t.Context(), tx, archiveID, generation,
)
}))
return database.BunStore
},
})
}
func TestBunStoreReadOnlyUsageAllowsMissingOptionalTables(t *testing.T) {
path := filepath.Join(t.TempDir(), "read-only-optional-usage.db")
database, err := db.Open(path)
require.NoError(t, err)
archiveID, err := database.GetArchiveID(t.Context())
require.NoError(t, err)
generation, err := database.GetDatabaseID(t.Context())
require.NoError(t, err)
require.NoError(t, database.Update(func(tx *sql.Tx) error {
if err := storetest.InsertSQLiteUsageFixture(
t.Context(), tx, archiveID, generation,
); err != nil {
return err
}
_, err := tx.ExecContext(t.Context(), `DROP TABLE cursor_usage_events`)
return err
}))
require.NoError(t, database.Close())
readOnly, err := db.OpenReadOnly(path)
require.NoError(t, err)
t.Cleanup(func() { require.NoError(t, readOnly.Close()) })
result, err := readOnly.GetDailyUsage(t.Context(), db.UsageFilter{
From: "2026-08-02", To: "2026-08-02", Timezone: "UTC",
})
require.NoError(t, err)
require.Len(t, result.Daily, 1)
assert.Equal(t, 25, result.Daily[0].InputTokens)
assert.Equal(t, 6, result.Daily[0].OutputTokens)
}
func TestBunStoreAnalyticsContract(t *testing.T) {
storetest.RunAnalyticsContract(t, storetest.AnalyticsBackend{
Name: "sqlite",
Open: func(t *testing.T) storetest.AnalyticsStore {
database, err := db.Open(filepath.Join(t.TempDir(), "analytics-contract.db"))
require.NoError(t, err)
t.Cleanup(func() { require.NoError(t, database.Close()) })
archiveID, err := database.GetArchiveID(t.Context())
require.NoError(t, err)
generation, err := database.GetDatabaseID(t.Context())
require.NoError(t, err)
require.NoError(t, storetest.InsertBunAnalyticsFixture(
t.Context(), database.BunWriterForTest(), archiveID, generation,
))
return database.BunStore
},
})
}
+9
View File
@@ -0,0 +1,9 @@
package db
import "github.com/uptrace/bun"
// BunWriterForTest exposes the guarded SQLite test store to external
// cross-backend contracts without adding a production-only accessor.
func (db *DB) BunWriterForTest() bun.IDB {
return db.bunWriter
}
+190
View File
@@ -0,0 +1,190 @@
package db
import (
"context"
"database/sql"
"fmt"
"strings"
"github.com/uptrace/bun"
"go.kenn.io/agentsview/internal/db/bunmodel"
)
func (s *BunStore) GetTrendsTerms(
ctx context.Context,
f AnalyticsFilter,
terms []TrendTermInput,
granularity string,
) (TrendsTermsResponse, error) {
if granularity == "" {
granularity = "week"
}
var result TrendsTermsResponse
err := s.consistentView(ctx, func(store bun.IDB) error {
sessionFilter := f
sessionFilter.From = ""
sessionFilter.To = ""
sessionFilter.DayOfWeek = nil
sessionFilter.Hour = nil
sessionFilter.Model = ""
sessions, err := s.bunAnalyticsSessionsFrom(
ctx, store, sessionFilter, false,
)
if err != nil {
return err
}
result, err = buildBunTrendsTerms(
sessions, f, terms, granularity,
func(consume func(bunmodel.Message) error) error {
return streamBunTrendMessages(
ctx, store, bunAnalyticsSessionIDs(sessions), consume,
)
},
)
return err
})
return result, err
}
func streamBunTrendMessages(
ctx context.Context, store bun.IDB, sessionIDs []string,
consume func(bunmodel.Message) error,
) error {
return queryChunkedSize(
sessionIDs, bunAnalyticsContentSessionBatchSize,
func(chunk []string) error {
rows, err := store.NewSelect().Table("messages").
Column("session_id", "ordinal", "role", "model", "is_system", "content").
ColumnExpr("CAST(timestamp AS VARCHAR) AS timestamp").
Where("session_id IN (?)", bun.List(chunk)).
OrderExpr("session_id ASC, ordinal ASC").Rows(ctx)
if err != nil {
return err
}
for rows.Next() {
var message bunmodel.Message
var timestamp sql.NullString
if err := rows.Scan(
&message.SessionID, &message.Ordinal, &message.Role, &message.Model,
&message.IsSystem, &message.Content, &timestamp,
); err != nil {
_ = rows.Close()
return fmt.Errorf("scanning Bun trend message: %w", err)
}
message.Timestamp = parseBunAnalyticsTimestamp(timestamp)
if err := consume(message); err != nil {
_ = rows.Close()
return err
}
}
if err := rows.Err(); err != nil {
_ = rows.Close()
return fmt.Errorf("iterating Bun trend messages: %w", err)
}
if err := rows.Close(); err != nil {
return fmt.Errorf("closing Bun trend messages: %w", err)
}
return nil
},
)
}
func buildBunTrendsTerms(
sessions []bunmodel.Session,
f AnalyticsFilter,
terms []TrendTermInput,
granularity string,
stream func(func(bunmodel.Message) error) error,
) (TrendsTermsResponse, error) {
buckets := TrendBucketRange(f.From, f.To, granularity)
bucketIndex := trendBucketIndex(buckets)
counts := make([][]int, len(terms))
for i := range counts {
counts[i] = make([]int, len(buckets))
}
messageCounts := make([]int, len(buckets))
sessionMap := make(map[string]bunmodel.Session, len(sessions))
for _, session := range sessions {
sessionMap[session.ID] = session
}
process := func(sessionID, content string, localTime bunmodel.Timestamp) {
date := localTime.Time.In(f.location()).Format("2006-01-02")
if !inDateRange(date, f.From, f.To) {
return
}
bucketDate := trendBucketDate(localTime.Time, f.location(), granularity)
bucket, ok := bucketIndex[bucketDate]
if !ok {
return
}
messageCounts[bucket]++
for i, term := range terms {
counts[i][bucket] += countTrendOccurrences(content, term)
}
_ = sessionID
}
filter := f.messageScopeFilter()
modelFiltering := strings.TrimSpace(f.Model) != ""
if modelFiltering {
reducer := NewScopeReducer(filter, func(message ScopedMessage) {
if !message.HasLocalTime {
return
}
process(message.SessionID, message.Content,
bunmodel.NewTimestamp(message.LocalTime))
})
if err := stream(func(message bunmodel.Message) error {
if message.Role != "user" && message.Role != "assistant" {
return nil
}
if message.IsSystem || IsSystemPrefixed(message.Content, message.Role) {
return nil
}
session, ok := sessionMap[message.SessionID]
if !ok {
return nil
}
local := bunAnalyticsSessionTime(session).In(f.location())
if message.Timestamp != nil {
local = message.Timestamp.In(f.location())
}
return reducer.Push(MessageInput{
SessionID: message.SessionID, Ordinal: message.Ordinal,
Role: message.Role, Model: message.Model, IsSystem: message.IsSystem,
Timestamp: bunAnalyticsTimeString(message.Timestamp),
LocalTime: local, HasLocalTime: true, Content: message.Content,
})
}); err != nil {
return TrendsTermsResponse{}, err
}
} else {
if err := stream(func(message bunmodel.Message) error {
if message.Role != "user" && message.Role != "assistant" {
return nil
}
if message.IsSystem || IsSystemPrefixed(message.Content, message.Role) {
return nil
}
session, ok := sessionMap[message.SessionID]
if !ok {
return nil
}
local := bunAnalyticsSessionTime(session).In(f.location())
if message.Timestamp != nil {
local = message.Timestamp.In(f.location())
}
if !filter.MatchesDayHour(local, true) {
return nil
}
process(message.SessionID, message.Content, bunmodel.NewTimestamp(local))
return nil
}); err != nil {
return TrendsTermsResponse{}, err
}
}
return BuildTrendsTermsResponse(
f.From, f.To, granularity, buckets, terms, counts, messageCounts,
), nil
}
File diff suppressed because it is too large Load Diff
+246
View File
@@ -0,0 +1,246 @@
package db
import (
"context"
"database/sql"
"fmt"
"testing"
"time"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
"github.com/uptrace/bun"
"go.kenn.io/agentsview/internal/db/bunmodel"
"go.kenn.io/agentsview/internal/money"
)
type alternatingUsageBackend struct {
first, second bun.IDB
views int
attempts int
}
type optionalUsageBackend struct {
*sessionContractBackend
missing map[string]bool
}
func (b *optionalUsageBackend) BunTableExists(
_ context.Context, _ bun.IDB, table string,
) (bool, error) {
return !b.missing[table], nil
}
func (*alternatingUsageBackend) Name() string { return "alternating-usage" }
func (*alternatingUsageBackend) ReadOnly() bool { return true }
func (*alternatingUsageBackend) Capabilities() BackendCapabilities {
return BackendCapabilities{}
}
func (*alternatingUsageBackend) SessionQueryDialect() QueryDialect {
return SQLiteBunSessionQueryDialect()
}
func (*alternatingUsageBackend) SessionVersion(
context.Context, bun.IDB, string,
) (int, int64, error) {
return 0, 0, sql.ErrNoRows
}
func (b *alternatingUsageBackend) View(
_ context.Context, fn func(bun.IDB) error,
) error {
return fn(b.second)
}
func (b *alternatingUsageBackend) ConsistentView(
_ context.Context, fn func(bun.IDB) error,
) error {
b.views++
b.attempts++
if err := fn(b.first); err != nil {
return err
}
b.attempts++
return fn(b.second)
}
func (*alternatingUsageBackend) Update(
context.Context, func(bun.IDB) error,
) error {
return ErrReadOnly
}
func TestGetDailyUsageKeepsPricingRowsAndIdentityInOneView(t *testing.T) {
first := testDB(t)
second := testDB(t)
seedUsageSnapshot := func(database *DB, inputTokens int, inputRate int64) {
t.Helper()
startedAt := "2026-08-03T12:00:00Z"
require.NoError(t, database.UpsertSession(Session{
ID: "snapshot-usage", Project: "snapshot", Machine: "host", Agent: "codex",
StartedAt: &startedAt, MessageCount: 1, UserMessageCount: 1,
}))
require.NoError(t, database.InsertMessages([]Message{{
SessionID: "snapshot-usage", Ordinal: 0, Role: "assistant",
Content: "usage", ContentLength: 5, Timestamp: startedAt,
Model: "snapshot-model", TokenUsage: fmt.Appendf(
nil, `{"input_tokens":%d}`, inputTokens,
),
}}))
require.NoError(t, database.UpsertModelPricing([]ModelPricing{{
ModelPattern: "snapshot-model",
InputPerMTok: money.Money{Microdollars: inputRate},
}}))
}
seedUsageSnapshot(first, 1, 1_000_000)
seedUsageSnapshot(second, 100, 100_000_000)
backend := &alternatingUsageBackend{
first: first.bunReader, second: second.bunReader,
}
store := NewBunStore(backend)
result, err := store.GetDailyUsage(t.Context(), UsageFilter{
From: "2026-08-03", To: "2026-08-03", Timezone: "UTC",
})
require.NoError(t, err)
assert.Equal(t, 100, result.Totals.InputTokens)
assert.Equal(t, int64(10_000), result.Totals.TotalCost.Microdollars)
assert.Equal(t, 1, backend.views)
assert.Equal(t, 2, backend.attempts)
}
func TestLoadPricingMapAllowsMissingOptionalBandsTable(t *testing.T) {
database := testDB(t)
require.NoError(t, database.UpsertModelPricing([]ModelPricing{{
ModelPattern: "base-only", InputPerMTok: money.Money{Microdollars: 7},
}}))
_, err := database.getWriter().ExecContext(
t.Context(), "DROP TABLE model_pricing_bands",
)
require.NoError(t, err)
backend := &optionalUsageBackend{
sessionContractBackend: &sessionContractBackend{store: database.bunReader},
missing: map[string]bool{"model_pricing_bands": true},
}
rows, err := NewBunStore(backend).LoadPricingMap(t.Context())
require.NoError(t, err)
var found bool
for _, row := range rows {
if row.ModelPattern == "base-only" {
found = true
assert.Equal(t, int64(7), row.Rates.InputPerMTok.Microdollars)
assert.Empty(t, row.Rates.Bands)
}
}
assert.True(t, found)
}
func TestAppendBunUsageTerminationFilterUsesProvidedReference(t *testing.T) {
database := testDB(t)
reference := time.Date(2030, 1, 2, 12, 0, 0, 0, time.UTC)
for _, row := range []struct {
id, status string
age time.Duration
}{
{id: "active", age: 5 * time.Minute},
{id: "stale", status: "tool_call_pending", age: 30 * time.Minute},
{id: "unclean", status: "truncated", age: 2 * time.Hour},
{id: "stale-clean", status: "clean", age: 30 * time.Minute},
} {
ended := reference.Add(-row.age).Format(time.RFC3339Nano)
status := row.status
require.NoError(t, database.UpsertSession(Session{
ID: row.id, Project: "clock", Machine: "host", Agent: "codex",
CreatedAt: ended, StartedAt: &ended, EndedAt: &ended,
MessageCount: 1, TerminationStatus: &status,
}))
}
var ids []string
query := database.bunReader.NewSelect().TableExpr("sessions AS s").Column("s.id")
query = appendBunUsageTerminationFilter(
query, "active,stale,unclean", SQLiteBunSessionQueryDialect(), reference,
)
require.NoError(t, query.OrderExpr("s.id ASC").Scan(t.Context(), &ids))
assert.Equal(t, []string{"active", "stale", "unclean"}, ids)
}
func TestUpsertModelPricingRowsReplacesBandsAtomically(t *testing.T) {
database := testDB(t)
ctx := t.Context()
base := bunmodel.ModelPricing{
ModelPattern: "atomic-model", InputMicrodollarsPerMTok: 1,
OutputMicrodollarsPerMTok: 2,
UpdatedAt: mustBunTimestamp(t, "2026-08-03T12:00:00Z"),
}
require.NoError(t, UpsertModelPricingRows(
ctx, database.bunWriter,
[]bunmodel.ModelPricing{base},
[]bunmodel.ModelPricingBand{
{ModelPattern: "atomic-model", AboveInputTokens: 100,
InputMicrodollarsPerMTok: 3,
UpdatedAt: mustBunTimestamp(t, "2026-08-03T12:00:00Z")},
{ModelPattern: "atomic-model", AboveInputTokens: 200,
InputMicrodollarsPerMTok: 4,
UpdatedAt: mustBunTimestamp(t, "2026-08-03T12:00:00Z")},
},
))
require.NoError(t, UpsertModelPricingRows(
ctx, database.bunWriter,
[]bunmodel.ModelPricing{{
ModelPattern: "atomic-model", InputMicrodollarsPerMTok: 5,
OutputMicrodollarsPerMTok: 6,
UpdatedAt: mustBunTimestamp(t, "2026-08-03T13:00:00Z"),
}},
[]bunmodel.ModelPricingBand{{
ModelPattern: "atomic-model", AboveInputTokens: 300,
InputMicrodollarsPerMTok: 7,
UpdatedAt: mustBunTimestamp(t, "2026-08-03T13:00:00Z"),
}},
))
stored, err := database.GetModelPricing("atomic-model")
require.NoError(t, err)
require.NotNil(t, stored)
assert.Equal(t, int64(5), stored.InputPerMTok.Microdollars)
require.Len(t, stored.Bands, 1)
assert.Equal(t, 300, stored.Bands[0].AboveInputTokens)
assert.Equal(t, int64(7), stored.Bands[0].InputPerMTok.Microdollars)
err = UpsertModelPricingRows(
ctx, database.bunWriter,
[]bunmodel.ModelPricing{{
ModelPattern: "atomic-model", InputMicrodollarsPerMTok: 99,
OutputMicrodollarsPerMTok: 100,
UpdatedAt: mustBunTimestamp(t, "2026-08-03T14:00:00Z"),
}},
[]bunmodel.ModelPricingBand{
{ModelPattern: "atomic-model", AboveInputTokens: 400,
InputMicrodollarsPerMTok: 8,
UpdatedAt: mustBunTimestamp(t, "2026-08-03T14:00:00Z")},
{ModelPattern: "atomic-model", AboveInputTokens: 400,
InputMicrodollarsPerMTok: 9,
UpdatedAt: mustBunTimestamp(t, "2026-08-03T14:00:00Z")},
},
)
require.Error(t, err)
stored, err = database.GetModelPricing("atomic-model")
require.NoError(t, err)
require.NotNil(t, stored)
assert.Equal(t, int64(5), stored.InputPerMTok.Microdollars,
"base update must roll back with band replacement")
require.Len(t, stored.Bands, 1)
assert.Equal(t, 300, stored.Bands[0].AboveInputTokens,
"deleted bands must roll back with the failed insert")
}
func mustBunTimestamp(t *testing.T, value string) bunmodel.Timestamp {
t.Helper()
timestamp, err := bunmodel.ParseTimestamp(value)
require.NoError(t, err)
return timestamp
}
+80
View File
@@ -0,0 +1,80 @@
package bunmodel
import "github.com/uptrace/bun"
type StarredSession struct {
bun.BaseModel `bun:"table:starred_sessions"`
SessionID string `bun:"session_id,pk"`
CreatedAt Timestamp `bun:"created_at,type:TIMESTAMPTZ,notnull"`
}
type PinnedMessage struct {
bun.BaseModel `bun:"table:pinned_messages"`
ID int64 `bun:"id,pk,autoincrement"`
SessionID string `bun:"session_id,notnull"`
MessageID *int64 `bun:"message_id,nullzero"`
Ordinal int `bun:"ordinal,notnull"`
SourceUUID string `bun:"source_uuid,notnull,default:''"`
Note *string `bun:"note,nullzero"`
CreatedAt Timestamp `bun:"created_at,type:TIMESTAMPTZ,notnull"`
}
type ExcludedSession struct {
bun.BaseModel `bun:"table:excluded_sessions"`
ID string `bun:"id,pk"`
CreatedAt Timestamp `bun:"created_at,type:TIMESTAMPTZ,notnull"`
}
type SessionAlias struct {
bun.BaseModel `bun:"table:session_aliases"`
SessionID string `bun:"session_id,pk"`
AliasID string `bun:"alias_id,pk"`
CreatedAt Timestamp `bun:"created_at,type:TIMESTAMPTZ,notnull"`
}
type Insight struct {
bun.BaseModel `bun:"table:insights"`
ID int64 `bun:"id,pk,autoincrement"`
Type string `bun:"type,notnull"`
DateFrom string `bun:"date_from,notnull"`
DateTo string `bun:"date_to,notnull"`
Project *string `bun:"project,nullzero"`
Agent string `bun:"agent,notnull"`
Model *string `bun:"model,nullzero"`
Prompt *string `bun:"prompt,nullzero"`
Content string `bun:"content,notnull"`
Kind string `bun:"kind,notnull,default:''"`
SchemaVersion string `bun:"schema_version,notnull,default:''"`
TemplateID string `bun:"template_id,notnull,default:''"`
TemplateVersion string `bun:"template_version,notnull,default:''"`
AggregateHash string `bun:"aggregate_hash,notnull,default:''"`
CacheKey string `bun:"cache_key,notnull,default:''"`
CacheStatus string `bun:"cache_status,notnull,default:''"`
ProvenanceJSON string `bun:"provenance_json,notnull,default:''"`
StructuredJSON string `bun:"structured_json,notnull,default:''"`
CreatedAt Timestamp `bun:"created_at,type:TIMESTAMPTZ,notnull"`
}
type SecretFinding struct {
bun.BaseModel `bun:"table:secret_findings"`
ID *int64 `bun:"id,nullzero"`
SessionID string `bun:"session_id,notnull"`
RuleName string `bun:"rule_name,notnull"`
Confidence string `bun:"confidence,notnull"`
LocationKind string `bun:"location_kind,notnull"`
MessageOrdinal int `bun:"message_ordinal,notnull"`
CallIndex *int `bun:"call_index,nullzero"`
EventIndex *int `bun:"event_index,nullzero"`
MatchStart int `bun:"match_start,notnull"`
MatchEnd int `bun:"match_end,notnull"`
MatchIndex int `bun:"match_index,notnull"`
RedactedMatch string `bun:"redacted_match,notnull"`
RulesVersion string `bun:"rules_version,notnull"`
CreatedAt Timestamp `bun:"created_at,type:TIMESTAMPTZ,notnull"`
}
@@ -0,0 +1,180 @@
//go:build !(windows && arm64)
package bunmodel
import (
"database/sql"
"testing"
"time"
_ "github.com/duckdb/duckdb-go/v2"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
"github.com/uptrace/bun"
"go.kenn.io/agentsview/internal/duckdb/bundialect"
)
func TestCommonTablesGeneratedSchemaExecutesInDuckDB(t *testing.T) {
raw, err := sql.Open("duckdb", "")
require.NoError(t, err)
raw.SetMaxOpenConns(1)
t.Cleanup(func() { require.NoError(t, raw.Close()) })
store := bun.NewDB(raw, bundialect.New())
for _, table := range CommonTables() {
ddl := registeredCreateTable(store, table, false).String()
for _, column := range criticalTableColumns[table.Name] {
assert.Contains(t, ddl, `"`+column+`"`, "%s.%s", table.Name, column)
}
_, err := registeredCreateTable(store, table, false).Exec(t.Context())
require.NoError(t, err, table.Name)
require.NoError(t, createRegisteredIndexes(t.Context(), store, table))
}
}
func TestCommonTablesDuckDBOmitsDynamicTimestampDefaults(t *testing.T) {
raw, err := sql.Open("duckdb", "")
require.NoError(t, err)
raw.SetMaxOpenConns(1)
t.Cleanup(func() { require.NoError(t, raw.Close()) })
store := bun.NewDB(raw, bundialect.New())
_, err = store.NewCreateTable().
Model((*SourceWorktreeProjectMapping)(nil)).Exec(t.Context())
require.NoError(t, err)
for _, column := range []string{"created_at", "updated_at"} {
var defaultExpression *string
err := raw.QueryRowContext(t.Context(), `
SELECT column_default
FROM information_schema.columns
WHERE table_schema = 'main'
AND table_name = 'source_worktree_project_mappings'
AND column_name = ?`, column).Scan(&defaultExpression)
require.NoError(t, err, column)
assert.Nil(t, defaultExpression, column)
}
want := time.Date(2026, 8, 5, 14, 30, 0, 123_000_000, time.UTC)
_, err = raw.ExecContext(t.Context(), `
INSERT INTO source_worktree_project_mappings (
source_archive_id, machine, path_prefix, created_at, updated_at
) VALUES ('archive', 'machine', '/repo', ?, ?)`, want, want)
require.NoError(t, err)
var createdAt, updatedAt time.Time
require.NoError(t, raw.QueryRowContext(t.Context(), `
SELECT created_at, updated_at
FROM source_worktree_project_mappings
WHERE source_archive_id = 'archive'`,
).Scan(&createdAt, &updatedAt))
assert.Equal(t, want, createdAt)
assert.Equal(t, want, updatedAt)
}
func TestCommonTablesDuckDBAllowsSourceOptionalIDsAndDeduplicatesNonEmptyKeys(t *testing.T) {
raw, err := sql.Open("duckdb", "")
require.NoError(t, err)
raw.SetMaxOpenConns(1)
t.Cleanup(func() { require.NoError(t, raw.Close()) })
store := bun.NewDB(raw, bundialect.New())
for _, table := range CommonTables() {
_, err := registeredCreateTable(store, table, false).Exec(t.Context())
require.NoError(t, err, table.Name)
require.NoError(t, createRegisteredIndexes(t.Context(), store, table))
}
_, err = raw.ExecContext(t.Context(), `
INSERT INTO cursor_usage_events (occurred_at, model, dedup_key)
VALUES
('2026-08-02 12:00:00', 'model', ''),
('2026-08-02 12:01:00', 'model', '');
INSERT INTO secret_findings (
session_id, rule_name, confidence, location_kind,
message_ordinal, match_start, match_end, match_index,
redacted_match, rules_version, created_at
) VALUES (
'session-1', 'rule', 'high', 'message', 1,
0, 4, 0, '[REDACTED]', 'v1', '2026-08-02 12:02:00'
);
`)
require.NoError(t, err, "DuckDB mirror IDs are source-optional")
_, err = raw.ExecContext(t.Context(), `
INSERT INTO cursor_usage_events (occurred_at, model, dedup_key)
VALUES ('2026-08-02 12:03:00', 'model', 'same')`)
require.NoError(t, err)
_, err = raw.ExecContext(t.Context(), `
INSERT INTO cursor_usage_events (occurred_at, model, dedup_key)
VALUES ('2026-08-02 12:04:00', 'model', 'same')`)
require.Error(t, err)
_, err = raw.ExecContext(t.Context(), `
INSERT INTO source_archives VALUES ('archive-1', 'salt-1');
INSERT INTO sessions (
id, project, machine, agent, created_at,
source_archive_id, source_database_generation
) VALUES (
'session-1', 'project', 'machine', 'agent',
'2026-08-02 12:00:00', 'archive-1', 'generation-1'
);
INSERT INTO secret_findings (
session_id, rule_name, confidence, location_kind,
message_ordinal, match_start, match_end, match_index,
redacted_match, rules_version, created_at
) VALUES (
'session-1', 'rule', 'high', 'message', 1,
0, 4, 0, '[REDACTED]', 'v1', '2026-08-02 12:02:00'
);
`)
require.NoError(t, err)
}
func TestCommonTablesDuckDBPreservesNativeBooleansJSONAndOptionalMessageID(t *testing.T) {
raw, err := sql.Open("duckdb", "")
require.NoError(t, err)
raw.SetMaxOpenConns(1)
t.Cleanup(func() { require.NoError(t, raw.Close()) })
store := bun.NewDB(raw, bundialect.New())
_, err = store.NewCreateTable().Model((*Message)(nil)).Exec(t.Context())
require.NoError(t, err)
_, err = raw.ExecContext(t.Context(), `
INSERT INTO messages (
session_id, ordinal, role, content, timestamp,
has_thinking, has_tool_use, is_system, token_usage
) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?)`,
"session-1", 8, "assistant", "done",
"2026-08-02 16:30:00", true, false, true,
`{"output_tokens":7}`,
)
require.NoError(t, err)
var got Message
require.NoError(t, store.NewSelect().Model(&got).
Where("session_id = ? AND ordinal = ?", "session-1", 8).
Scan(t.Context()))
assert.Nil(t, got.ID)
assert.True(t, got.HasThinking)
assert.False(t, got.HasToolUse)
assert.True(t, got.IsSystem)
assert.JSONEq(t, `{"output_tokens":7}`, string(got.TokenUsage))
}
func TestCommonTablesDuckDBRejectsNonTimestampPricingValues(t *testing.T) {
raw, err := sql.Open("duckdb", "")
require.NoError(t, err)
raw.SetMaxOpenConns(1)
t.Cleanup(func() { require.NoError(t, raw.Close()) })
store := bun.NewDB(raw, bundialect.New())
for _, table := range CommonTables() {
_, err := registeredCreateTable(store, table, false).Exec(t.Context())
require.NoError(t, err, table.Name)
}
_, err = raw.ExecContext(t.Context(), `
INSERT INTO model_pricing (
model_pattern, input_microdollars_per_mtok,
output_microdollars_per_mtok, updated_at
) VALUES ('__pricing_seed_version__', 0, 0, '2')`)
require.Error(t, err)
}
+74
View File
@@ -0,0 +1,74 @@
package bunmodel
import "github.com/uptrace/bun"
type SourceArchive struct {
bun.BaseModel `bun:"table:source_archives"`
SourceArchiveID string `bun:"source_archive_id,pk"`
SourceArchiveSalt string `bun:"source_archive_salt,notnull"`
}
type SourceProjectIdentityObservation struct {
bun.BaseModel `bun:"table:source_project_identity_observations"`
SourceArchiveID string `bun:"source_archive_id,pk"`
SourceArchiveSalt string `bun:"source_archive_salt,notnull,default:''"`
Project string `bun:"project,pk"`
Machine string `bun:"machine,pk"`
RootPath string `bun:"root_path,pk"`
GitRemote string `bun:"git_remote,pk"`
GitRemoteName string `bun:"git_remote_name,notnull,default:''"`
RepositoryPath string `bun:"repository_path,notnull,default:''"`
WorktreeName string `bun:"worktree_name,notnull,default:''"`
WorktreeRootPath string `bun:"worktree_root_path,notnull,default:''"`
WorktreeRelationship string `bun:"worktree_relationship,notnull,default:'unknown'"`
CheckoutState string `bun:"checkout_state,notnull,default:'unknown'"`
GitBranch string `bun:"git_branch,notnull,default:''"`
RemoteResolution string `bun:"remote_resolution,notnull,default:'unknown'"`
RemoteCandidateCount int `bun:"remote_candidate_count,notnull,default:0"`
ObservedAt Timestamp `bun:"observed_at,type:TIMESTAMPTZ,notnull"`
NormalizedRemote string `bun:"normalized_remote,notnull,default:''"`
KeySource string `bun:"key_source,notnull,default:''"`
Key string `bun:"key,notnull,default:''"`
}
type SourceSessionProjectIdentitySnapshot struct {
bun.BaseModel `bun:"table:source_session_project_identity_snapshots"`
SourceArchiveID string `bun:"source_archive_id,pk"`
SourceDatabaseGeneration string `bun:"source_database_generation,pk"`
SourceSessionID string `bun:"source_session_id,pk"`
Project string `bun:"project,notnull"`
Machine string `bun:"machine,notnull"`
RootPath string `bun:"root_path,notnull,default:''"`
GitRemote string `bun:"git_remote,notnull,default:''"`
GitRemoteName string `bun:"git_remote_name,notnull,default:''"`
RepositoryPath string `bun:"repository_path,notnull,default:''"`
WorktreeName string `bun:"worktree_name,notnull,default:''"`
WorktreeRootPath string `bun:"worktree_root_path,notnull,default:''"`
WorktreeRelationship string `bun:"worktree_relationship,notnull,default:'unknown'"`
CheckoutState string `bun:"checkout_state,notnull,default:'unknown'"`
GitBranch string `bun:"git_branch,notnull,default:''"`
RemoteResolution string `bun:"remote_resolution,notnull,default:'unknown'"`
RemoteCandidateCount int `bun:"remote_candidate_count,notnull,default:0"`
ObservedAt Timestamp `bun:"observed_at,type:TIMESTAMPTZ,notnull"`
NormalizedRemote string `bun:"normalized_remote,notnull,default:''"`
KeySource string `bun:"key_source,notnull,default:''"`
Key string `bun:"key,notnull,default:''"`
}
type SourceWorktreeProjectMapping struct {
bun.BaseModel `bun:"table:source_worktree_project_mappings"`
ID int64 `bun:"id,notnull,default:0"`
SourceArchiveID string `bun:"source_archive_id,pk"`
Machine string `bun:"machine,pk"`
PathPrefix string `bun:"path_prefix,pk"`
Layout string `bun:"layout,notnull,default:'explicit'"`
Project string `bun:"project,notnull,default:''"`
OriginalProject string `bun:"original_project,notnull,default:''"`
Enabled bool `bun:"enabled,notnull,default:true"`
CreatedAt Timestamp `bun:"created_at,type:TIMESTAMPTZ,notnull,default:CURRENT_TIMESTAMP"`
UpdatedAt Timestamp `bun:"updated_at,type:TIMESTAMPTZ,notnull,default:CURRENT_TIMESTAMP"`
}
+80
View File
@@ -0,0 +1,80 @@
package bunmodel
import (
"encoding/json"
"github.com/uptrace/bun"
)
// Message uses the portable logical key (session_id, ordinal). ID is retained
// only as an optional source-row identifier for the shipped SQLite archive.
type Message struct {
bun.BaseModel `bun:"table:messages"`
ID *int64 `bun:"id,nullzero"`
SessionID string `bun:"session_id,pk"`
Ordinal int `bun:"ordinal,pk"`
Role string `bun:"role,notnull"`
Content string `bun:"content,notnull"`
ThinkingText string `bun:"thinking_text,notnull,default:''"`
Timestamp *Timestamp `bun:"timestamp,type:TIMESTAMPTZ,nullzero"`
HasThinking bool `bun:"has_thinking,notnull,default:false"`
HasToolUse bool `bun:"has_tool_use,notnull,default:false"`
ContentLength int `bun:"content_length,notnull,default:0"`
IsSystem bool `bun:"is_system,notnull,default:false"`
Model string `bun:"model,notnull,default:''"`
TokenUsage json.RawMessage `bun:"token_usage,type:TEXT,notnull,default:''"`
ContextTokens int `bun:"context_tokens,notnull,default:0"`
OutputTokens int `bun:"output_tokens,notnull,default:0"`
HasContextTokens bool `bun:"has_context_tokens,notnull,default:false"`
HasOutputTokens bool `bun:"has_output_tokens,notnull,default:false"`
ClaudeMessageID string `bun:"claude_message_id,notnull,default:''"`
ClaudeRequestID string `bun:"claude_request_id,notnull,default:''"`
SourceType string `bun:"source_type,notnull,default:''"`
SourceSubtype string `bun:"source_subtype,notnull,default:''"`
PromptSource string `bun:"prompt_source,notnull,default:''"`
SourceUUID string `bun:"source_uuid,notnull,default:''"`
SourceParentUUID string `bun:"source_parent_uuid,notnull,default:''"`
IsSidechain bool `bun:"is_sidechain,notnull,default:false"`
IsCompactBoundary bool `bun:"is_compact_boundary,notnull,default:false"`
}
// ToolCall is located by the canonical message key and call index. The source
// row and SQLite message IDs are optional data, not relationships.
type ToolCall struct {
bun.BaseModel `bun:"table:tool_calls"`
ID *int64 `bun:"id,nullzero"`
MessageID *int64 `bun:"message_id,nullzero"`
SessionID string `bun:"session_id,pk"`
MessageOrdinal int `bun:"message_ordinal,pk"`
ToolName string `bun:"tool_name,notnull"`
Category string `bun:"category,notnull"`
CallIndex int `bun:"call_index,pk,default:0"`
ToolUseID string `bun:"tool_use_id,notnull,default:''"`
InputJSON *string `bun:"input_json,nullzero"`
SkillName *string `bun:"skill_name,nullzero"`
ResultContentLength *int `bun:"result_content_length,nullzero"`
ResultContent *string `bun:"result_content,nullzero"`
SubagentSessionID *string `bun:"subagent_session_id,nullzero"`
FilePath *string `bun:"file_path,nullzero"`
}
// ToolResultEvent is one chronological output update for a tool call.
type ToolResultEvent struct {
bun.BaseModel `bun:"table:tool_result_events"`
ID *int64 `bun:"id,nullzero"`
SessionID string `bun:"session_id,pk"`
ToolCallMessageOrdinal int `bun:"tool_call_message_ordinal,pk"`
CallIndex int `bun:"call_index,pk,default:0"`
ToolUseID *string `bun:"tool_use_id,nullzero"`
AgentID *string `bun:"agent_id,nullzero"`
SubagentSessionID *string `bun:"subagent_session_id,nullzero"`
Source string `bun:"source,notnull"`
Status string `bun:"status,notnull"`
Content string `bun:"content,notnull"`
ContentLength int `bun:"content_length,notnull,default:0"`
Timestamp *Timestamp `bun:"timestamp,type:TIMESTAMPTZ,nullzero"`
EventIndex int `bun:"event_index,pk,default:0"`
}
+105
View File
@@ -0,0 +1,105 @@
// Package bunmodel defines the persistence-only models shared by the storage
// adapters. Parser state, adapter bookkeeping, and API presentation fields do
// not belong here.
package bunmodel
import (
"database/sql/driver"
"errors"
"fmt"
"time"
"github.com/uptrace/bun"
)
// ErrUnsupportedTimestamp identifies a non-empty persistent timestamp that
// cannot be represented by the canonical schema.
var ErrUnsupportedTimestamp = errors.New("unsupported timestamp")
// Timestamp is the canonical persistence timestamp. It accepts the native
// time values returned by PostgreSQL and DuckDB as well as the text forms
// already stored by SQLite, and normalizes every non-null value to UTC.
type Timestamp struct {
time.Time
}
var timestampLayouts = []string{
time.RFC3339Nano,
"2006-01-02 15:04:05.999999999Z07:00",
"2006-01-02 15:04:05Z07:00",
"2006-01-02 15:04:05.999999999",
"2006-01-02 15:04:05",
}
// NewTimestamp returns a canonical UTC timestamp.
func NewTimestamp(value time.Time) Timestamp {
return Timestamp{Time: value.UTC()}
}
// ParseTimestamp parses a supported persistent timestamp representation.
func ParseTimestamp(value string) (Timestamp, error) {
for _, layout := range timestampLayouts {
parsed, err := time.Parse(layout, value)
if err == nil {
return NewTimestamp(parsed), nil
}
}
return Timestamp{}, fmt.Errorf("%w %q", ErrUnsupportedTimestamp, value)
}
// Scan implements sql.Scanner.
func (t *Timestamp) Scan(src any) error {
switch value := src.(type) {
case nil:
t.Time = time.Time{}
return nil
case time.Time:
t.Time = value.UTC()
return nil
case string:
if value == "" {
t.Time = time.Time{}
return nil
}
parsed, err := ParseTimestamp(value)
if err != nil {
return err
}
*t = parsed
return nil
case []byte:
return t.Scan(string(value))
default:
return fmt.Errorf("scanning timestamp from %T", src)
}
}
// Value implements driver.Valuer.
func (t Timestamp) Value() (driver.Value, error) {
return t.UTC().Format(time.RFC3339Nano), nil
}
// ModelPricing is one model-pattern pricing row.
type ModelPricing struct {
bun.BaseModel `bun:"table:model_pricing"`
ModelPattern string `bun:"model_pattern,pk"`
InputMicrodollarsPerMTok int64 `bun:"input_microdollars_per_mtok,notnull,default:0"`
OutputMicrodollarsPerMTok int64 `bun:"output_microdollars_per_mtok,notnull,default:0"`
CacheCreationMicrodollarsPerMTok int64 `bun:"cache_creation_microdollars_per_mtok,notnull,default:0"`
CacheReadMicrodollarsPerMTok int64 `bun:"cache_read_microdollars_per_mtok,notnull,default:0"`
UpdatedAt Timestamp `bun:"updated_at,type:TIMESTAMPTZ,notnull"`
}
// ModelPricingBand overrides pricing above one input-token threshold.
type ModelPricingBand struct {
bun.BaseModel `bun:"table:model_pricing_bands"`
ModelPattern string `bun:"model_pattern,pk"`
AboveInputTokens int64 `bun:"above_input_tokens,pk"`
InputMicrodollarsPerMTok int64 `bun:"input_microdollars_per_mtok,notnull,default:0"`
OutputMicrodollarsPerMTok int64 `bun:"output_microdollars_per_mtok,notnull,default:0"`
CacheCreationMicrodollarsPerMTok int64 `bun:"cache_creation_microdollars_per_mtok,notnull,default:0"`
CacheReadMicrodollarsPerMTok int64 `bun:"cache_read_microdollars_per_mtok,notnull,default:0"`
UpdatedAt Timestamp `bun:"updated_at,type:TIMESTAMPTZ,notnull"`
}
+191
View File
@@ -0,0 +1,191 @@
package bunmodel
import (
"fmt"
"reflect"
"slices"
"strings"
"github.com/uptrace/bun/schema"
)
// Index is one portable ordinary index owned by the common schema.
type Index struct {
Name string
Columns []string
Expressions []string
Unique bool
}
// ForeignKey is one canonical relationship. DuckDB cannot use foreign keys on
// mutable mirror tables because it rejects parent updates and cascading
// actions; its mirror writer preserves the relationship explicitly.
type ForeignKey struct {
Columns []string
ReferencedTable string
ReferencedColumns []string
OnDeleteCascade bool
}
// Table registers one canonical model and its ordinary indexes.
type Table struct {
Name string
Model any
Indexes []Index
ForeignKeys []ForeignKey
}
// ForeignKeyDefinition renders the portable Bun ForeignKey clause. Adapters
// that cannot use foreign-key DDL on mutable tables omit the clause and enforce
// the registered relationship through their atomic writer.
func ForeignKeyDefinition(foreignKey ForeignKey, includeCascade bool) string {
definition := fmt.Sprintf(
"%s REFERENCES %s %s",
quotedColumns(foreignKey.Columns),
quotedIdentifier(foreignKey.ReferencedTable),
quotedColumns(foreignKey.ReferencedColumns),
)
if includeCascade && foreignKey.OnDeleteCascade {
definition += " ON DELETE CASCADE"
}
return definition
}
func quotedColumns(columns []string) string {
quoted := make([]string, len(columns))
for i, column := range columns {
quoted[i] = quotedIdentifier(column)
}
return "(" + strings.Join(quoted, ", ") + ")"
}
func quotedIdentifier(identifier string) string {
return `"` + strings.ReplaceAll(identifier, `"`, `""`) + `"`
}
var commonTables = []Table{
{Name: "source_archives", Model: (*SourceArchive)(nil)},
{Name: "sessions", Model: (*Session)(nil), ForeignKeys: []ForeignKey{
{Columns: []string{"source_archive_id"}, ReferencedTable: "source_archives", ReferencedColumns: []string{"source_archive_id"}},
}, Indexes: []Index{
{Name: "idx_sessions_ended", Columns: []string{"ended_at", "id"}},
{Name: "idx_sessions_project", Columns: []string{"project"}},
{Name: "idx_sessions_machine", Columns: []string{"machine"}},
{Name: "idx_sessions_parent", Columns: []string{"parent_session_id"}},
{Name: "idx_sessions_started", Columns: []string{"started_at"}},
{Name: "idx_sessions_agent", Columns: []string{"agent"}},
{Name: "idx_sessions_termination_status", Columns: []string{"termination_status"}},
}},
{Name: "messages", Model: (*Message)(nil), ForeignKeys: []ForeignKey{
{Columns: []string{"session_id"}, ReferencedTable: "sessions", ReferencedColumns: []string{"id"}, OnDeleteCascade: true},
}, Indexes: []Index{
{Name: "idx_messages_session_role", Columns: []string{"session_id", "role"}},
{Name: "idx_messages_timestamp", Columns: []string{"timestamp"}},
}},
{Name: "usage_events", Model: (*UsageEvent)(nil), ForeignKeys: []ForeignKey{
{Columns: []string{"session_id"}, ReferencedTable: "sessions", ReferencedColumns: []string{"id"}, OnDeleteCascade: true},
}, Indexes: []Index{
{Name: "idx_usage_events_session", Columns: []string{"session_id"}},
{Name: "idx_usage_events_occurred", Columns: []string{"occurred_at"}},
{Name: "idx_usage_events_dedup", Expressions: []string{
"(CASE WHEN dedup_key <> '' THEN session_id END)",
"(CASE WHEN dedup_key <> '' THEN source END)",
"(CASE WHEN dedup_key <> '' THEN dedup_key END)",
}, Unique: true},
}},
{Name: "cursor_usage_events", Model: (*CursorUsageEvent)(nil), Indexes: []Index{
{Name: "idx_cursor_usage_events_dedup", Expressions: []string{
"(CASE WHEN dedup_key <> '' THEN dedup_key END)",
}, Unique: true},
{Name: "idx_cursor_usage_events_occurred", Columns: []string{"occurred_at"}},
{Name: "idx_cursor_usage_events_model", Columns: []string{"model"}},
}},
{Name: "model_pricing", Model: (*ModelPricing)(nil)},
{Name: "model_pricing_bands", Model: (*ModelPricingBand)(nil), ForeignKeys: []ForeignKey{
{Columns: []string{"model_pattern"}, ReferencedTable: "model_pricing", ReferencedColumns: []string{"model_pattern"}, OnDeleteCascade: true},
}},
{Name: "source_project_identity_observations", Model: (*SourceProjectIdentityObservation)(nil), ForeignKeys: []ForeignKey{
{Columns: []string{"source_archive_id"}, ReferencedTable: "source_archives", ReferencedColumns: []string{"source_archive_id"}, OnDeleteCascade: true},
}, Indexes: []Index{
{Name: "idx_source_project_identity_observations_project", Columns: []string{"project"}},
}},
{Name: "source_session_project_identity_snapshots", Model: (*SourceSessionProjectIdentitySnapshot)(nil), ForeignKeys: []ForeignKey{
{Columns: []string{"source_archive_id"}, ReferencedTable: "source_archives", ReferencedColumns: []string{"source_archive_id"}, OnDeleteCascade: true},
}, Indexes: []Index{
{Name: "idx_source_session_project_identity_snapshots_project", Columns: []string{"source_archive_id", "project"}},
}},
{Name: "source_worktree_project_mappings", Model: (*SourceWorktreeProjectMapping)(nil), ForeignKeys: []ForeignKey{
{Columns: []string{"source_archive_id"}, ReferencedTable: "source_archives", ReferencedColumns: []string{"source_archive_id"}, OnDeleteCascade: true},
}},
{Name: "tool_calls", Model: (*ToolCall)(nil), ForeignKeys: []ForeignKey{
{Columns: []string{"session_id", "message_ordinal"}, ReferencedTable: "messages", ReferencedColumns: []string{"session_id", "ordinal"}, OnDeleteCascade: true},
}, Indexes: []Index{
{Name: "idx_tool_calls_dedup", Columns: []string{"session_id", "message_ordinal", "call_index"}, Unique: true},
{Name: "idx_tool_calls_session", Columns: []string{"session_id"}},
{Name: "idx_tool_calls_category", Columns: []string{"category"}},
{Name: "idx_tool_calls_file_path", Columns: []string{"file_path"}},
}},
{Name: "tool_result_events", Model: (*ToolResultEvent)(nil), ForeignKeys: []ForeignKey{
{Columns: []string{"session_id", "tool_call_message_ordinal", "call_index"}, ReferencedTable: "tool_calls", ReferencedColumns: []string{"session_id", "message_ordinal", "call_index"}, OnDeleteCascade: true},
}, Indexes: []Index{
{Name: "idx_tool_result_events_session", Columns: []string{"session_id"}},
{Name: "idx_tool_result_events_dedup", Columns: []string{"session_id", "tool_call_message_ordinal", "call_index", "event_index"}, Unique: true},
}},
{Name: "secret_findings", Model: (*SecretFinding)(nil), ForeignKeys: []ForeignKey{
{Columns: []string{"session_id"}, ReferencedTable: "sessions", ReferencedColumns: []string{"id"}, OnDeleteCascade: true},
}, Indexes: []Index{
{Name: "idx_secret_findings_session", Columns: []string{"session_id"}},
{Name: "idx_secret_findings_rule", Columns: []string{"rule_name"}},
}},
{Name: "starred_sessions", Model: (*StarredSession)(nil), ForeignKeys: []ForeignKey{
{Columns: []string{"session_id"}, ReferencedTable: "sessions", ReferencedColumns: []string{"id"}, OnDeleteCascade: true},
}},
{Name: "excluded_sessions", Model: (*ExcludedSession)(nil)},
{Name: "session_aliases", Model: (*SessionAlias)(nil), ForeignKeys: []ForeignKey{
{Columns: []string{"session_id"}, ReferencedTable: "sessions", ReferencedColumns: []string{"id"}, OnDeleteCascade: true},
}},
{Name: "pinned_messages", Model: (*PinnedMessage)(nil), ForeignKeys: []ForeignKey{
{Columns: []string{"session_id", "ordinal"}, ReferencedTable: "messages", ReferencedColumns: []string{"session_id", "ordinal"}, OnDeleteCascade: true},
}, Indexes: []Index{
{Name: "idx_pinned_session", Columns: []string{"session_id"}},
{Name: "idx_pinned_ordinal", Columns: []string{"session_id", "ordinal"}, Unique: true},
{Name: "idx_pinned_created", Columns: []string{"created_at"}},
}},
{Name: "insights", Model: (*Insight)(nil), Indexes: []Index{
{Name: "idx_insights_lookup", Columns: []string{"type", "date_from", "date_to", "project"}},
{Name: "idx_insights_cache", Columns: []string{"cache_key", "created_at"}},
}},
}
var modelRegistry = schema.NewTables(schema.NewNopQueryGen().Dialect())
// CommonTables returns a copy so adapters cannot mutate the canonical
// registry or one another's index definitions.
func CommonTables() []Table {
tables := make([]Table, len(commonTables))
for i := range commonTables {
tables[i] = commonTables[i]
tables[i].Indexes = slices.Clone(commonTables[i].Indexes)
for j := range tables[i].Indexes {
tables[i].Indexes[j].Columns = slices.Clone(tables[i].Indexes[j].Columns)
tables[i].Indexes[j].Expressions = slices.Clone(tables[i].Indexes[j].Expressions)
}
tables[i].ForeignKeys = slices.Clone(commonTables[i].ForeignKeys)
for j := range tables[i].ForeignKeys {
tables[i].ForeignKeys[j].Columns = slices.Clone(tables[i].ForeignKeys[j].Columns)
tables[i].ForeignKeys[j].ReferencedColumns = slices.Clone(tables[i].ForeignKeys[j].ReferencedColumns)
}
}
return tables
}
// ModelColumns returns sorted Bun column names for a canonical model.
func ModelColumns(model any) []string {
table := modelRegistry.Get(reflect.TypeOf(model))
columns := make([]string, 0, len(table.Fields))
for _, field := range table.Fields {
columns = append(columns, field.Name)
}
slices.Sort(columns)
return columns
}
+363
View File
@@ -0,0 +1,363 @@
package bunmodel
import (
"context"
"database/sql"
"fmt"
"sort"
"strings"
"testing"
"time"
_ "github.com/mattn/go-sqlite3"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
"github.com/uptrace/bun"
"github.com/uptrace/bun/dialect/pgdialect"
"github.com/uptrace/bun/dialect/sqlitedialect"
"github.com/uptrace/bun/schema"
)
func registeredCreateTable(
db *bun.DB, table Table, includeForeignKeys bool,
) *bun.CreateTableQuery {
query := db.NewCreateTable().Model(table.Model).IfNotExists()
if includeForeignKeys {
for _, foreignKey := range table.ForeignKeys {
query.ForeignKey(ForeignKeyDefinition(foreignKey, true))
}
}
return query
}
func createRegisteredIndexes(
ctx context.Context, db *bun.DB, table Table,
) error {
for _, index := range table.Indexes {
query := db.NewCreateIndex().Model(table.Model).
Index(index.Name).IfNotExists()
if index.Unique {
query.Unique()
}
for _, column := range index.Columns {
query.Column(column)
}
for _, expression := range index.Expressions {
query.ColumnExpr(expression)
}
if _, err := query.Exec(ctx); err != nil {
return fmt.Errorf("creating index %s: %w", index.Name, err)
}
}
return nil
}
var criticalTableColumns = map[string][]string{
"sessions": {
"id", "project", "agent", "started_at", "created_at",
"source_archive_id", "source_database_generation",
},
"messages": {
"id", "session_id", "ordinal", "token_usage", "timestamp",
},
"usage_events": {"session_id", "source", "occurred_at"},
"cursor_usage_events": {"occurred_at", "model", "is_headless"},
"tool_calls": {"session_id", "message_ordinal", "call_index"},
"tool_result_events": {"session_id", "tool_call_message_ordinal", "event_index"},
"secret_findings": {"session_id", "message_ordinal", "redacted_match"},
"model_pricing": {"model_pattern", "input_microdollars_per_mtok", "updated_at"},
"model_pricing_bands": {"model_pattern", "above_input_tokens", "updated_at"},
"starred_sessions": {"session_id", "created_at"},
"pinned_messages": {"session_id", "ordinal", "source_uuid"},
"excluded_sessions": {"id", "created_at"},
"session_aliases": {"session_id", "alias_id"},
"insights": {"type", "content", "created_at"},
"source_archives": {"source_archive_id", "source_archive_salt"},
"source_project_identity_observations": {
"source_archive_id", "project", "observed_at", "key",
},
"source_session_project_identity_snapshots": {
"source_archive_id", "source_database_generation",
"source_session_id", "observed_at",
},
"source_worktree_project_mappings": {
"source_archive_id", "machine", "path_prefix", "enabled",
},
}
func TestCommonTablesContainCanonicalServingSchema(t *testing.T) {
want := []string{
"cursor_usage_events",
"excluded_sessions",
"insights",
"messages",
"model_pricing",
"model_pricing_bands",
"pinned_messages",
"secret_findings",
"session_aliases",
"sessions",
"source_archives",
"source_project_identity_observations",
"source_session_project_identity_snapshots",
"source_worktree_project_mappings",
"starred_sessions",
"tool_calls",
"tool_result_events",
"usage_events",
}
got := make([]string, 0, len(CommonTables()))
for _, table := range CommonTables() {
got = append(got, table.Name)
}
sort.Strings(got)
assert.Equal(t, want, got)
assert.Subset(t, ModelColumns((*Session)(nil)), []string{
"agent", "created_at", "deleted_at", "ended_at", "id", "machine",
"message_count", "project", "source_archive_id",
"source_database_generation", "started_at", "transcript_revision",
})
}
func TestCommonTablesGenerateCanonicalMessageCompositeKey(t *testing.T) {
for name, dialect := range map[string]schema.Dialect{
"postgresql": pgdialect.New(),
"sqlite": sqlitedialect.New(),
} {
t.Run(name, func(t *testing.T) {
raw, err := sql.Open("sqlite3", ":memory:")
require.NoError(t, err)
t.Cleanup(func() { require.NoError(t, raw.Close()) })
store := bun.NewDB(raw, dialect)
ddl := store.NewCreateTable().Model((*Message)(nil)).String()
assert.Contains(t, ddl, `PRIMARY KEY ("session_id", "ordinal")`)
assert.NotContains(t, ddl, `"id" INTEGER NOT NULL PRIMARY KEY`)
assert.NotContains(t, ddl, `"id" BIGINT NOT NULL PRIMARY KEY`)
})
}
}
func TestCommonTablesDeclareMessageRelationshipsAndDedupConstraints(t *testing.T) {
tables := make(map[string]Table)
for _, table := range CommonTables() {
tables[table.Name] = table
}
assert.Contains(t, tables["messages"].ForeignKeys, ForeignKey{
Columns: []string{"session_id"},
ReferencedTable: "sessions",
ReferencedColumns: []string{"id"},
OnDeleteCascade: true,
})
assert.Contains(t, tables["tool_calls"].ForeignKeys, ForeignKey{
Columns: []string{"session_id", "message_ordinal"},
ReferencedTable: "messages",
ReferencedColumns: []string{"session_id", "ordinal"},
OnDeleteCascade: true,
})
assert.Contains(t, tables["pinned_messages"].ForeignKeys, ForeignKey{
Columns: []string{"session_id", "ordinal"},
ReferencedTable: "messages",
ReferencedColumns: []string{"session_id", "ordinal"},
OnDeleteCascade: true,
})
assert.Contains(t, tables["usage_events"].Indexes, Index{
Name: "idx_usage_events_dedup",
Expressions: []string{
"(CASE WHEN dedup_key <> '' THEN session_id END)",
"(CASE WHEN dedup_key <> '' THEN source END)",
"(CASE WHEN dedup_key <> '' THEN dedup_key END)",
},
Unique: true,
})
assert.Contains(t, tables["cursor_usage_events"].Indexes, Index{
Name: "idx_cursor_usage_events_dedup",
Expressions: []string{
"(CASE WHEN dedup_key <> '' THEN dedup_key END)",
},
Unique: true,
})
}
func TestCommonTablesGeneratedSchemaExecutesInSQLite(t *testing.T) {
raw, err := sql.Open("sqlite3", ":memory:")
require.NoError(t, err)
raw.SetMaxOpenConns(1)
t.Cleanup(func() { require.NoError(t, raw.Close()) })
store := bun.NewDB(raw, sqlitedialect.New())
for _, table := range CommonTables() {
ddl := registeredCreateTable(store, table, true).String()
for _, column := range criticalTableColumns[table.Name] {
assert.Contains(t, ddl, `"`+column+`"`, "%s.%s", table.Name, column)
}
_, err := registeredCreateTable(store, table, true).Exec(t.Context())
require.NoError(t, err, table.Name)
require.NoError(t, createRegisteredIndexes(t.Context(), store, table))
}
}
func TestCommonTablesGeneratedSQLiteSchemaEnforcesCascadeAndDedupBehavior(t *testing.T) {
raw, err := sql.Open("sqlite3", ":memory:")
require.NoError(t, err)
raw.SetMaxOpenConns(1)
t.Cleanup(func() { require.NoError(t, raw.Close()) })
_, err = raw.Exec("PRAGMA foreign_keys = ON")
require.NoError(t, err)
store := bun.NewDB(raw, sqlitedialect.New())
for _, table := range CommonTables() {
_, err := registeredCreateTable(store, table, true).Exec(t.Context())
require.NoError(t, err, table.Name)
require.NoError(t, createRegisteredIndexes(t.Context(), store, table))
}
_, err = raw.ExecContext(t.Context(), `
INSERT INTO source_archives VALUES ('archive-1', 'salt-1');
INSERT INTO sessions (
id, project, machine, agent, created_at,
source_archive_id, source_database_generation
) VALUES (
'session-1', 'project', 'machine', 'agent',
'2026-08-02T12:00:00Z', 'archive-1', 'generation-1'
);
INSERT INTO messages (session_id, ordinal, role, content)
VALUES ('session-1', 4, 'assistant', 'done');
INSERT INTO tool_calls (
session_id, message_ordinal, tool_name, category, call_index
) VALUES ('session-1', 4, 'Read', 'Read', 0);
INSERT INTO pinned_messages (session_id, ordinal, created_at)
VALUES ('session-1', 4, '2026-08-02T12:01:00Z');
INSERT INTO usage_events (session_id, source, model, dedup_key)
VALUES
('session-1', 'parser', 'model', ''),
('session-1', 'parser', 'model', '');
`)
require.NoError(t, err)
_, err = raw.ExecContext(t.Context(), `
INSERT INTO usage_events (session_id, source, model, dedup_key)
VALUES ('session-1', 'parser', 'model', 'same')`)
require.NoError(t, err)
_, err = raw.ExecContext(t.Context(), `
INSERT INTO usage_events (session_id, source, model, dedup_key)
VALUES ('session-1', 'parser', 'model', 'same')`)
require.Error(t, err)
_, err = raw.ExecContext(t.Context(), `DELETE FROM sessions WHERE id = 'session-1'`)
require.NoError(t, err)
for _, table := range []string{"messages", "tool_calls", "pinned_messages", "usage_events"} {
var count int
require.NoError(t, raw.QueryRowContext(
t.Context(), "SELECT count(*) FROM "+table,
).Scan(&count))
assert.Zero(t, count, table)
}
}
func TestCommonTablesGeneratedSchemaRendersForPostgreSQL(t *testing.T) {
raw, err := sql.Open("sqlite3", ":memory:")
require.NoError(t, err)
t.Cleanup(func() { require.NoError(t, raw.Close()) })
store := bun.NewDB(raw, pgdialect.New())
for _, table := range CommonTables() {
ddl := registeredCreateTable(store, table, true).String()
assert.True(t, strings.HasPrefix(ddl, "CREATE TABLE IF NOT EXISTS "))
for _, column := range criticalTableColumns[table.Name] {
assert.Contains(t, ddl, `"`+column+`"`, "%s.%s", table.Name, column)
}
}
}
func TestBunRowTimestampScannerNormalizesSupportedInputsToUTC(t *testing.T) {
want := time.Date(2026, 8, 2, 16, 30, 0, 123_456_000, time.UTC)
tests := map[string]any{
"native time": time.Date(
2026, 8, 2, 12, 30, 0, 123_456_000,
time.FixedZone("EDT", -4*60*60),
),
"RFC3339 text": "2026-08-02T12:30:00.123456-04:00",
"SQLite text": "2026-08-02 12:30:00.123456-04:00",
"SQLite bytes": []byte("2026-08-02 16:30:00.123456"),
"millisecond Z text": "2026-08-02T16:30:00.123456Z",
}
for name, input := range tests {
t.Run(name, func(t *testing.T) {
var got Timestamp
require.NoError(t, got.Scan(input))
assert.Equal(t, want, got.Time)
value, err := got.Value()
require.NoError(t, err)
assert.Equal(t, "2026-08-02T16:30:00.123456Z", value)
})
}
}
func TestBunRowTimestampScannerAcceptsSQLiteEmptySentinel(t *testing.T) {
for name, input := range map[string]any{
"text": "",
"bytes": []byte(""),
} {
t.Run(name, func(t *testing.T) {
var got Timestamp
require.NoError(t, got.Scan(input))
assert.True(t, got.IsZero())
})
}
}
func TestBunRowTimestampValuePersistsRFC3339NanoText(t *testing.T) {
value := NewTimestamp(time.Date(
2026, 8, 2, 12, 30, 0, 123_456_789,
time.FixedZone("EDT", -4*60*60),
))
got, err := value.Value()
require.NoError(t, err)
assert.Equal(t, "2026-08-02T16:30:00.123456789Z", got)
}
func TestBunRowTimestampScannerPreservesDatabaseNull(t *testing.T) {
var got Timestamp
require.NoError(t, got.Scan(nil))
assert.True(t, got.IsZero())
}
func TestBunRowSQLiteAliasesPreserveBooleansJSONAndOptionalMessageID(t *testing.T) {
raw, err := sql.Open("sqlite3", ":memory:")
require.NoError(t, err)
raw.SetMaxOpenConns(1)
t.Cleanup(func() { require.NoError(t, raw.Close()) })
store := bun.NewDB(raw, sqlitedialect.New())
_, err = store.NewCreateTable().Model((*Message)(nil)).Exec(t.Context())
require.NoError(t, err)
_, err = raw.ExecContext(t.Context(), `
INSERT INTO messages (
session_id, ordinal, role, content, timestamp,
has_thinking, has_tool_use, is_system, token_usage
) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?)`,
"session-1", 4, "assistant", "done",
"2026-08-02 12:30:00-04:00", 1, 0, 1,
`{"input_tokens":12}`,
)
require.NoError(t, err)
var got Message
require.NoError(t, store.NewSelect().Model(&got).
Where("session_id = ? AND ordinal = ?", "session-1", 4).
Scan(t.Context()))
assert.Nil(t, got.ID)
assert.True(t, got.HasThinking)
assert.False(t, got.HasToolUse)
assert.True(t, got.IsSystem)
assert.JSONEq(t, `{"input_tokens":12}`, string(got.TokenUsage))
require.NotNil(t, got.Timestamp)
assert.Equal(t, time.Date(2026, 8, 2, 16, 30, 0, 0, time.UTC), got.Timestamp.Time)
}
+83
View File
@@ -0,0 +1,83 @@
package bunmodel
import "github.com/uptrace/bun"
// Session is the common durable session row. The source provenance fields are
// required; adapters stamp them before writes reach the shared store.
type Session struct {
bun.BaseModel `bun:"table:sessions"`
ID string `bun:"id,pk"`
Project string `bun:"project,notnull"`
Machine string `bun:"machine,notnull"`
Agent string `bun:"agent,notnull"`
AgentLabel string `bun:"agent_label,notnull,default:''"`
Entrypoint string `bun:"entrypoint,notnull,default:''"`
SessionKind string `bun:"session_kind,notnull,default:''"`
FirstMessage *string `bun:"first_message,nullzero"`
DisplayName *string `bun:"display_name,nullzero"`
SessionName *string `bun:"session_name,nullzero"`
StartedAt *Timestamp `bun:"started_at,type:TIMESTAMPTZ,nullzero"`
EndedAt *Timestamp `bun:"ended_at,type:TIMESTAMPTZ,nullzero"`
MessageCount int `bun:"message_count,notnull,default:0"`
UserMessageCount int `bun:"user_message_count,notnull,default:0"`
ParentSessionID *string `bun:"parent_session_id,nullzero"`
ParserParentSessionID *string `bun:"parser_parent_session_id,nullzero"`
RelationshipType string `bun:"relationship_type,notnull,default:''"`
TotalOutputTokens int `bun:"total_output_tokens,notnull,default:0"`
PeakContextTokens int `bun:"peak_context_tokens,notnull,default:0"`
HasTotalOutputTokens bool `bun:"has_total_output_tokens,notnull,default:false"`
HasPeakContextTokens bool `bun:"has_peak_context_tokens,notnull,default:false"`
IsAutomated bool `bun:"is_automated,notnull,default:false"`
ToolFailureSignalCount int `bun:"tool_failure_signal_count,notnull,default:0"`
ToolRetryCount int `bun:"tool_retry_count,notnull,default:0"`
EditChurnCount int `bun:"edit_churn_count,notnull,default:0"`
ConsecutiveFailureMax int `bun:"consecutive_failure_max,notnull,default:0"`
Outcome string `bun:"outcome,notnull,default:'unknown'"`
OutcomeConfidence string `bun:"outcome_confidence,notnull,default:'low'"`
EndedWithRole string `bun:"ended_with_role,notnull,default:''"`
FinalFailureStreak int `bun:"final_failure_streak,notnull,default:0"`
SignalsPendingSince *Timestamp `bun:"signals_pending_since,type:TIMESTAMPTZ,nullzero"`
CompactionCount int `bun:"compaction_count,notnull,default:0"`
MidTaskCompactionCount int `bun:"mid_task_compaction_count,notnull,default:0"`
ContextPressureMax *float64 `bun:"context_pressure_max,nullzero"`
HealthScore *int `bun:"health_score,nullzero"`
HealthGrade *string `bun:"health_grade,nullzero"`
HasToolCalls bool `bun:"has_tool_calls,notnull,default:false"`
HasContextData bool `bun:"has_context_data,notnull,default:false"`
SecretLeakCount int `bun:"secret_leak_count,notnull,default:0"`
SecretsRulesVersion string `bun:"secrets_rules_version,notnull,default:''"`
QualitySignalVersion int `bun:"quality_signal_version,notnull,default:0"`
ShortPromptCount int `bun:"short_prompt_count,notnull,default:0"`
UnstructuredStart bool `bun:"unstructured_start,notnull,default:false"`
MissingSuccessCriteriaCount int `bun:"missing_success_criteria_count,notnull,default:0"`
MissingVerificationCount int `bun:"missing_verification_count,notnull,default:0"`
DuplicatePromptCount int `bun:"duplicate_prompt_count,notnull,default:0"`
NoCodeContextCount int `bun:"no_code_context_count,notnull,default:0"`
RunawayToolLoopCount int `bun:"runaway_tool_loop_count,notnull,default:0"`
DataVersion int `bun:"data_version,notnull,default:0"`
Cwd string `bun:"cwd,notnull,default:''"`
GitBranch string `bun:"git_branch,notnull,default:''"`
SourceSessionID string `bun:"source_session_id,notnull,default:''"`
SourceVersion string `bun:"source_version,notnull,default:''"`
TranscriptFidelity string `bun:"transcript_fidelity,notnull,default:''"`
ParserMalformedLines int `bun:"parser_malformed_lines,notnull,default:0"`
IsTruncated bool `bun:"is_truncated,notnull,default:false"`
DeletedAt *Timestamp `bun:"deleted_at,type:TIMESTAMPTZ,nullzero"`
DeletionCause *string `bun:"deletion_cause,nullzero"`
TerminationStatus *string `bun:"termination_status,nullzero"`
FilePath *string `bun:"file_path,nullzero"`
FileSize *int64 `bun:"file_size,nullzero"`
FileMtime *int64 `bun:"file_mtime,nullzero"`
FileInode *int64 `bun:"file_inode,nullzero"`
FileDevice *int64 `bun:"file_device,nullzero"`
FileHash *string `bun:"file_hash,nullzero"`
LocalModifiedAt *Timestamp `bun:"local_modified_at,type:TIMESTAMPTZ,nullzero"`
TranscriptRevision string `bun:"transcript_revision,notnull,default:'0'"`
CreatedAt Timestamp `bun:"created_at,type:TIMESTAMPTZ,notnull"`
SourceArchiveID string `bun:"source_archive_id,notnull,default:''"`
SourceDatabaseGeneration string `bun:"source_database_generation,notnull,default:''"`
}
+44
View File
@@ -0,0 +1,44 @@
package bunmodel
import "github.com/uptrace/bun"
// UsageEvent stores session- or message-level token accounting.
type UsageEvent struct {
bun.BaseModel `bun:"table:usage_events"`
ID int64 `bun:"id,pk,autoincrement"`
SessionID string `bun:"session_id,notnull"`
MessageOrdinal *int `bun:"message_ordinal,nullzero"`
Source string `bun:"source,notnull"`
Model string `bun:"model,notnull"`
InputTokens int `bun:"input_tokens,notnull,default:0"`
OutputTokens int `bun:"output_tokens,notnull,default:0"`
CacheCreationInputTokens int `bun:"cache_creation_input_tokens,notnull,default:0"`
CacheReadInputTokens int `bun:"cache_read_input_tokens,notnull,default:0"`
ReasoningTokens int `bun:"reasoning_tokens,notnull,default:0"`
CostMicrodollars *int64 `bun:"cost_microdollars,nullzero"`
CostStatus string `bun:"cost_status,notnull,default:''"`
CostSource string `bun:"cost_source,notnull,default:''"`
OccurredAt *Timestamp `bun:"occurred_at,type:TIMESTAMPTZ,nullzero"`
DedupKey string `bun:"dedup_key,notnull,default:''"`
}
// CursorUsageEvent stores authoritative Cursor admin usage data.
type CursorUsageEvent struct {
bun.BaseModel `bun:"table:cursor_usage_events"`
ID *int64 `bun:"id,nullzero"`
OccurredAt Timestamp `bun:"occurred_at,type:TIMESTAMPTZ,notnull"`
Model string `bun:"model,notnull"`
Kind string `bun:"kind,notnull,default:''"`
InputTokens int `bun:"input_tokens,notnull,default:0"`
OutputTokens int `bun:"output_tokens,notnull,default:0"`
CacheWriteTokens int `bun:"cache_write_tokens,notnull,default:0"`
CacheReadTokens int `bun:"cache_read_tokens,notnull,default:0"`
ChargedMicrodollars int64 `bun:"charged_microdollars,notnull,default:0"`
CursorTokenFeeMicrodollars int64 `bun:"cursor_token_fee_microdollars,notnull,default:0"`
UserID string `bun:"user_id,notnull,default:''"`
UserEmail string `bun:"user_email,notnull,default:''"`
IsHeadless bool `bun:"is_headless,notnull,default:false"`
DedupKey string `bun:"dedup_key,notnull,default:''"`
}
+61 -179
View File
@@ -17,8 +17,9 @@ import (
"sync/atomic"
"time"
"go.kenn.io/agentsview/internal/config"
"go.kenn.io/agentsview/internal/export"
"github.com/uptrace/bun"
"github.com/uptrace/bun/dialect/sqlitedialect"
"go.kenn.io/agentsview/internal/parser"
)
@@ -384,7 +385,9 @@ CREATE INDEX IF NOT EXISTS idx_provider_freshness_updated_at
// (80: Kimi Code tool-step usage reparse. Protocol-1.4 transcripts can persist
// tool.result before step.end, so existing Kimi and Kimi Work rows may omit
// per-message usage for tool-calling steps. Re-parsing attaches the trailing
// step usage to the assistant tool-call message.)
// step usage to the assistant tool-call message. The same release performs the
// canonical Bun schema cutover, so this gate also prevents an older binary from
// reopening the archive after its compatibility stamp is committed.)
// (81: Pi-family flat cache-write usage reparse. Existing Pi and OMP rows can
// persist cache creation under cacheWrite, which older parses ignored.
// Re-parsing restores those tokens to per-message usage and computed cost.)
@@ -604,12 +607,18 @@ END;
// concurrent HTTP handler goroutines can safely read while
// Reopen/CloseConnections swap the underlying *sql.DB.
type DB struct {
path string
writer atomic.Pointer[sql.DB]
reader atomic.Pointer[sql.DB]
mu sync.Mutex // serializes writes
connMu sync.RWMutex
retired []*sql.DB // old pools kept open for in-flight reads
*BunStore
path string
writer atomic.Pointer[sql.DB]
reader atomic.Pointer[sql.DB]
// bunReader and bunWriter are swapped with the raw pools under connMu.
// Bun does not own or close those pools.
bunReader *bun.DB
bunWriter *bun.DB
mu sync.Mutex // serializes writes
connMu sync.RWMutex
retired []*sql.DB // old pools kept open for in-flight reads
// undrainedPools holds closed pools whose connections had not drained
// when CloseWriter or CloseConnections gave up. They must drain before
// a later close reports success, or write ownership could be released
@@ -623,13 +632,6 @@ type DB struct {
writerClosed atomic.Bool
dataStale atomic.Bool // set by Open when user_version < dataVersion
cursorMu sync.RWMutex
cursorSecret []byte
customPricing map[string]config.CustomModelRate
effectivePricing map[string]export.ModelRates
emptyCatalogPricing map[string]export.ModelRates
checkpointMu sync.Mutex
checkpointStop chan struct{}
checkpointDone chan struct{}
@@ -871,43 +873,6 @@ func (db *DB) requireWritable() error {
return nil
}
func (db *DB) SetCustomPricing(p map[string]config.CustomModelRate) {
db.customPricing = p
db.effectivePricing = nil
}
// SetEffectivePricing installs in-memory pricing rows with explicit provenance
// sources for read-only fallback paths that cannot seed model_pricing.
func (db *DB) SetEffectivePricing(
p map[string]export.ModelRates,
) {
db.customPricing = nil
db.effectivePricing = make(map[string]export.ModelRates, len(p))
for model, rates := range p {
rates.Bands = append([]export.PricingBand(nil), rates.Bands...)
db.effectivePricing[model] = rates
}
}
// SetEmptyCatalogPricing installs in-memory rates that are used only when the
// query source loading pricing sees no stored catalog rows.
func (db *DB) SetEmptyCatalogPricing(
p map[string]export.ModelRates,
) {
db.emptyCatalogPricing = make(map[string]export.ModelRates, len(p))
for model, rates := range p {
rates.Bands = append([]export.PricingBand(nil), rates.Bands...)
db.emptyCatalogPricing[model] = rates
}
}
// SetCursorSecret updates the secret key used for cursor signing.
func (db *DB) SetCursorSecret(secret []byte) {
db.cursorMu.Lock()
defer db.cursorMu.Unlock()
db.cursorSecret = append([]byte(nil), secret...)
}
// makeDSN builds a SQLite connection string with shared pragmas.
//
// Both branches emit a file: URI. mattn/go-sqlite3 forwards the `_`-prefixed
@@ -974,10 +939,6 @@ func Open(path string) (*DB, error) {
return nil, err
}
if err := d.migrateColumns(); err != nil {
d.Close()
return nil, fmt.Errorf("migrating columns: %w", err)
}
if _, err := d.GetOrCreateDatabaseID(context.Background()); err != nil {
d.Close()
return nil, fmt.Errorf("initializing database id: %w", err)
@@ -990,11 +951,14 @@ func Open(path string) (*DB, error) {
d.Close()
return nil, fmt.Errorf("initializing archive salt: %w", err)
}
if err := d.migrateColumns(); err != nil {
d.Close()
return nil, fmt.Errorf("migrating columns: %w", err)
}
if err := d.EnsureProjectIdentityBackfillQueued(context.Background()); err != nil {
d.Close()
return nil, fmt.Errorf("queueing project identity backfill: %w", err)
}
if dataStale || schemaRepairNeeded {
d.dataStale.Store(true)
log.Printf(
@@ -1037,112 +1001,6 @@ CREATE TABLE IF NOT EXISTS session_project_identity_snapshot_changes (
);
CREATE INDEX IF NOT EXISTS idx_session_project_identity_snapshot_changes_revision
ON session_project_identity_snapshot_changes(revision);
DROP TRIGGER IF EXISTS trg_project_identity_observations_revision_insert;
DROP TRIGGER IF EXISTS trg_project_identity_observations_revision_update;
DROP TRIGGER IF EXISTS trg_project_identity_observations_revision_delete;
DROP TRIGGER IF EXISTS trg_session_project_identity_snapshots_revision_insert;
DROP TRIGGER IF EXISTS trg_session_project_identity_snapshots_revision_update;
DROP TRIGGER IF EXISTS trg_session_project_identity_snapshots_revision_delete;
CREATE TRIGGER IF NOT EXISTS trg_project_identity_observations_revision_insert
AFTER INSERT ON project_identity_observations BEGIN
INSERT INTO archive_metadata (key, value) VALUES ('project_identity_publication_revision', '1')
ON CONFLICT(key) DO UPDATE SET value = CAST(CAST(value AS INTEGER) + 1 AS TEXT),
updated_at = strftime('%Y-%m-%dT%H:%M:%fZ','now');
INSERT INTO project_identity_observation_changes (
project, machine, root_path, git_remote, revision, deleted
) VALUES (
NEW.project, NEW.machine, NEW.root_path, NEW.git_remote,
(SELECT CAST(value AS INTEGER) FROM archive_metadata
WHERE key = 'project_identity_publication_revision'), 0
) ON CONFLICT(project, machine, root_path, git_remote) DO UPDATE SET
revision = excluded.revision, deleted = 0;
END;
CREATE TRIGGER IF NOT EXISTS trg_project_identity_observations_revision_update
AFTER UPDATE ON project_identity_observations BEGIN
INSERT INTO archive_metadata (key, value) VALUES ('project_identity_publication_revision', '1')
ON CONFLICT(key) DO UPDATE SET value = CAST(CAST(value AS INTEGER) + 1 AS TEXT),
updated_at = strftime('%Y-%m-%dT%H:%M:%fZ','now');
INSERT INTO project_identity_observation_changes (
project, machine, root_path, git_remote, revision, deleted
) VALUES (
OLD.project, OLD.machine, OLD.root_path, OLD.git_remote,
(SELECT CAST(value AS INTEGER) FROM archive_metadata
WHERE key = 'project_identity_publication_revision'), 1
) ON CONFLICT(project, machine, root_path, git_remote) DO UPDATE SET
revision = excluded.revision, deleted = 1;
INSERT INTO project_identity_observation_changes (
project, machine, root_path, git_remote, revision, deleted
) VALUES (
NEW.project, NEW.machine, NEW.root_path, NEW.git_remote,
(SELECT CAST(value AS INTEGER) FROM archive_metadata
WHERE key = 'project_identity_publication_revision'), 0
) ON CONFLICT(project, machine, root_path, git_remote) DO UPDATE SET
revision = excluded.revision, deleted = 0;
END;
CREATE TRIGGER IF NOT EXISTS trg_project_identity_observations_revision_delete
AFTER DELETE ON project_identity_observations BEGIN
INSERT INTO archive_metadata (key, value) VALUES ('project_identity_publication_revision', '1')
ON CONFLICT(key) DO UPDATE SET value = CAST(CAST(value AS INTEGER) + 1 AS TEXT),
updated_at = strftime('%Y-%m-%dT%H:%M:%fZ','now');
INSERT INTO project_identity_observation_changes (
project, machine, root_path, git_remote, revision, deleted
) VALUES (
OLD.project, OLD.machine, OLD.root_path, OLD.git_remote,
(SELECT CAST(value AS INTEGER) FROM archive_metadata
WHERE key = 'project_identity_publication_revision'), 1
) ON CONFLICT(project, machine, root_path, git_remote) DO UPDATE SET
revision = excluded.revision, deleted = 1;
END;
CREATE TRIGGER IF NOT EXISTS trg_session_project_identity_snapshots_revision_insert
AFTER INSERT ON session_project_identity_snapshots BEGIN
INSERT INTO archive_metadata (key, value) VALUES ('project_identity_publication_revision', '1')
ON CONFLICT(key) DO UPDATE SET value = CAST(CAST(value AS INTEGER) + 1 AS TEXT),
updated_at = strftime('%Y-%m-%dT%H:%M:%fZ','now');
INSERT INTO session_project_identity_snapshot_changes (
session_id, project, revision, deleted
) VALUES (
NEW.session_id, NEW.project,
(SELECT CAST(value AS INTEGER) FROM archive_metadata
WHERE key = 'project_identity_publication_revision'), 0
) ON CONFLICT(session_id, project) DO UPDATE SET
revision = excluded.revision, deleted = 0;
END;
CREATE TRIGGER IF NOT EXISTS trg_session_project_identity_snapshots_revision_update
AFTER UPDATE ON session_project_identity_snapshots BEGIN
INSERT INTO archive_metadata (key, value) VALUES ('project_identity_publication_revision', '1')
ON CONFLICT(key) DO UPDATE SET value = CAST(CAST(value AS INTEGER) + 1 AS TEXT),
updated_at = strftime('%Y-%m-%dT%H:%M:%fZ','now');
INSERT INTO session_project_identity_snapshot_changes (
session_id, project, revision, deleted
) VALUES (
OLD.session_id, OLD.project,
(SELECT CAST(value AS INTEGER) FROM archive_metadata
WHERE key = 'project_identity_publication_revision'), 1
) ON CONFLICT(session_id, project) DO UPDATE SET
revision = excluded.revision, deleted = 1;
INSERT INTO session_project_identity_snapshot_changes (
session_id, project, revision, deleted
) VALUES (
NEW.session_id, NEW.project,
(SELECT CAST(value AS INTEGER) FROM archive_metadata
WHERE key = 'project_identity_publication_revision'), 0
) ON CONFLICT(session_id, project) DO UPDATE SET
revision = excluded.revision, deleted = 0;
END;
CREATE TRIGGER IF NOT EXISTS trg_session_project_identity_snapshots_revision_delete
AFTER DELETE ON session_project_identity_snapshots BEGIN
INSERT INTO archive_metadata (key, value) VALUES ('project_identity_publication_revision', '1')
ON CONFLICT(key) DO UPDATE SET value = CAST(CAST(value AS INTEGER) + 1 AS TEXT),
updated_at = strftime('%Y-%m-%dT%H:%M:%fZ','now');
INSERT INTO session_project_identity_snapshot_changes (
session_id, project, revision, deleted
) VALUES (
OLD.session_id, OLD.project,
(SELECT CAST(value AS INTEGER) FROM archive_metadata
WHERE key = 'project_identity_publication_revision'), 1
) ON CONFLICT(session_id, project) DO UPDATE SET
revision = excluded.revision, deleted = 1;
END;
`
const projectIdentitySnapshotInvariantSchemaSQL = `
@@ -1440,13 +1298,16 @@ func OpenReadOnly(path string) (*DB, error) {
db := &DB{path: path, readOnly: true}
db.reader.Store(reader)
db.cursorSecret = make([]byte, 32)
if _, err := rand.Read(db.cursorSecret); err != nil {
db.bunReader = bun.NewDB(reader, sqlitedialect.New())
db.BunStore = NewBunStore(&sqliteBunBackend{store: db})
cursorSecret := make([]byte, 32)
if _, err := rand.Read(cursorSecret); err != nil {
reader.Close()
return nil, fmt.Errorf(
"generating cursor secret: %w", err,
)
}
db.SetCursorSecret(cursorSecret)
return db, nil
}
@@ -1469,6 +1330,11 @@ var readOnlyRequiredTables = []string{
"pg_sync_state",
"model_pricing",
"model_pricing_bands",
"pricing_metadata",
"source_archives",
"source_project_identity_observations",
"source_session_project_identity_snapshots",
"source_worktree_project_mappings",
"secret_findings",
"recall_entries",
"recall_evidence",
@@ -1513,6 +1379,13 @@ func readOnlyRequiredSchema() (map[string][]string, error) {
)
return
}
store := bun.NewDB(conn, sqlitedialect.New())
if err := CreateCommonSchema(context.Background(), store); err != nil {
readOnlyRequiredSchemaErr = fmt.Errorf(
"loading common schema probe: %w", err,
)
return
}
schema, err := tableColumns(conn, readOnlyRequiredTables)
if err != nil {
readOnlyRequiredSchemaErr = err
@@ -2650,11 +2523,10 @@ func (db *DB) migrateColumns() error {
"creating project identity metadata: %w", err,
)
}
if _, err := w.Exec(projectIdentityRevisionSchemaSQL); err != nil {
return fmt.Errorf("creating project identity revision triggers: %w", err)
}
if _, err := w.Exec(projectIdentitySnapshotInvariantSchemaSQL); err != nil {
return fmt.Errorf("creating project identity snapshot trigger: %w", err)
if err := db.convergeSQLiteCommonSchemaLocked(
context.Background(), nil,
); err != nil {
return err
}
if err := db.scrubProjectIdentityGitRemoteCredentialsLocked(w); err != nil {
return err
@@ -2669,7 +2541,6 @@ func (db *DB) migrateColumns() error {
if err := requeueInvalidArtifactPublicationsLocked(w); err != nil {
return err
}
runRepair, err := db.shouldRunTokenCoverageRepairLocked(w)
if err != nil {
return err
@@ -3022,13 +2893,16 @@ func (db *DB) scrubProjectIdentityGitRemoteCredentialsLocked(
}
defer func() { _ = tx.Rollback() }()
if _, err := tx.Exec(`
DELETE FROM project_identity_observations
DELETE FROM source_project_identity_observations
WHERE git_remote = ''
AND EXISTS (
SELECT 1 FROM project_identity_observations remote
WHERE remote.project = project_identity_observations.project
AND remote.machine = project_identity_observations.machine
AND remote.root_path = project_identity_observations.root_path
SELECT 1 FROM source_project_identity_observations remote
WHERE remote.source_archive_id =
source_project_identity_observations.source_archive_id
AND remote.project = source_project_identity_observations.project
AND remote.machine = source_project_identity_observations.machine
AND remote.root_path =
source_project_identity_observations.root_path
AND remote.git_remote != ''
)`); err != nil {
return fmt.Errorf("removing stale project identity root fallbacks: %w", err)
@@ -3755,15 +3629,19 @@ func openAndInit(path string, schemaRepairNeeded bool) (*DB, error) {
db := &DB{path: path}
db.writer.Store(writer)
db.reader.Store(reader)
db.bunWriter = bun.NewDB(writer, sqlitedialect.New())
db.bunReader = bun.NewDB(reader, sqlitedialect.New())
db.BunStore = NewBunStore(&sqliteBunBackend{store: db})
db.cursorSecret = make([]byte, 32)
if _, err := rand.Read(db.cursorSecret); err != nil {
cursorSecret := make([]byte, 32)
if _, err := rand.Read(cursorSecret); err != nil {
writer.Close()
reader.Close()
return nil, fmt.Errorf(
"generating cursor secret: %w", err,
)
}
db.SetCursorSecret(cursorSecret)
if schemaRepairNeeded {
db.mu.Lock()
err = repairLegacySchemaBeforeInit(db.getWriter())
@@ -4375,6 +4253,8 @@ func (db *DB) reopenLocked() error {
retired := append([]*sql.DB(nil), db.retired...)
oldWriter := db.writer.Swap(writer)
oldReader := db.reader.Swap(reader)
db.bunWriter = bun.NewDB(writer, sqlitedialect.New())
db.bunReader = bun.NewDB(reader, sqlitedialect.New())
// Reopen fully restores the writer pool, so clear any writer-closed barrier
// a prior CloseWriter set. Without this a resync swap that ran behind the
// worker write barrier would reopen the pool yet keep rejecting writes.
@@ -4434,6 +4314,7 @@ func (db *DB) CloseWriter() error {
defer db.mu.Unlock()
db.connMu.Lock()
old := db.writer.Swap(nil)
db.bunWriter = nil
db.writerClosed.Store(true)
pending := db.undrainedPools
db.undrainedPools = nil
@@ -4494,6 +4375,7 @@ func (db *DB) ReopenWriter() error {
db.connMu.Lock()
old := db.writer.Swap(writer)
db.bunWriter = bun.NewDB(writer, sqlitedialect.New())
db.writerClosed.Store(false)
db.connMu.Unlock()
+151 -65
View File
@@ -23,6 +23,8 @@ import (
"github.com/google/go-cmp/cmp"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
"github.com/uptrace/bun"
"go.kenn.io/agentsview/internal/db/bunmodel"
"go.kenn.io/agentsview/internal/export"
"go.kenn.io/agentsview/internal/money"
)
@@ -815,11 +817,13 @@ func TestMigration_ResultContentColumn(t *testing.T) {
SELECT id, message_id, session_id, tool_name,
category, tool_use_id, input_json,
skill_name, result_content_length,
subagent_session_id
subagent_session_id, file_path, call_index,
message_ordinal
FROM tool_calls;
DROP TABLE tool_calls;
ALTER TABLE tool_calls_old RENAME TO tool_calls;
`)
DELETE FROM archive_metadata WHERE key = ?;
`, CommonSchemaCompatibilityMetadataKey)
requireNoError(t, err, "drop result_content column")
// Verify column is gone and tool_calls row exists.
@@ -978,7 +982,8 @@ func TestMigration_ToolResultEventsTable(t *testing.T) {
_, err = conn.Exec(fmt.Sprintf(`
DROP TABLE tool_result_events;
PRAGMA user_version = %d;
`, legacyVersion))
DELETE FROM archive_metadata WHERE key = ?;
`, legacyVersion), CommonSchemaCompatibilityMetadataKey)
requireNoError(t, err, "drop tool_result_events")
var count int
@@ -3039,6 +3044,10 @@ func TestSessionFileInfo(t *testing.T) {
func TestGetSessionFull(t *testing.T) {
d := testDB(t)
ctx := context.Background()
archiveID, err := d.GetArchiveID(ctx)
require.NoError(t, err)
databaseGeneration, err := d.GetDatabaseID(ctx)
require.NoError(t, err)
t.Run("AllMetadata", func(t *testing.T) {
insertSession(t, d, "full-1", "proj", func(s *Session) {
@@ -3056,22 +3065,24 @@ func TestGetSessionFull(t *testing.T) {
requireNoError(t, err, "GetSessionFull")
require.NotNil(t, got, "expected non-nil session")
want := &Session{
ID: "full-1",
Project: "proj",
MessageCount: 5,
FilePath: new("/tmp/session.jsonl"),
FileSize: new(int64(2048)),
FileMtime: new(int64(1700000000)),
FileHash: new("abc123"),
TranscriptRevision: new("0"),
FirstMessage: new("hello"),
StartedAt: new(tsZero),
EndedAt: new(tsHour1),
Machine: defaultMachine,
Agent: defaultAgent,
Outcome: "unknown",
OutcomeConfidence: "low",
CreatedAt: got.CreatedAt,
ID: "full-1",
Project: "proj",
MessageCount: 5,
FilePath: new("/tmp/session.jsonl"),
FileSize: new(int64(2048)),
FileMtime: new(int64(1700000000)),
FileHash: new("abc123"),
TranscriptRevision: new("0"),
FirstMessage: new("hello"),
StartedAt: new(tsZero),
EndedAt: new(tsHour1),
Machine: defaultMachine,
Agent: defaultAgent,
Outcome: "unknown",
OutcomeConfidence: "low",
CreatedAt: got.CreatedAt,
SourceArchiveID: archiveID,
SourceDatabaseGeneration: databaseGeneration,
}
if diff := cmp.Diff(want, got); diff != "" {
t.Errorf("GetSessionFull mismatch (-want +got):\n%s", diff)
@@ -3087,15 +3098,17 @@ func TestGetSessionFull(t *testing.T) {
requireNoError(t, err, "GetSessionFull")
require.NotNil(t, got, "expected non-nil session")
want := &Session{
ID: "full-2",
Project: "proj",
MessageCount: 1,
TranscriptRevision: new("0"),
Machine: defaultMachine,
Agent: defaultAgent,
Outcome: "unknown",
OutcomeConfidence: "low",
CreatedAt: got.CreatedAt,
ID: "full-2",
Project: "proj",
MessageCount: 1,
TranscriptRevision: new("0"),
Machine: defaultMachine,
Agent: defaultAgent,
Outcome: "unknown",
OutcomeConfidence: "low",
CreatedAt: got.CreatedAt,
SourceArchiveID: archiveID,
SourceDatabaseGeneration: databaseGeneration,
}
if diff := cmp.Diff(want, got); diff != "" {
t.Errorf("GetSessionFull mismatch (-want +got):\n%s", diff)
@@ -3613,7 +3626,7 @@ func TestToolCallNewColumns(t *testing.T) {
insertSession(t, d, "s1", "proj")
insertMessages(t, d, Message{
SessionID: "s1",
Ordinal: 0,
Ordinal: 4,
Role: "assistant",
Content: "[Read: main.go]",
ContentLength: 15,
@@ -3630,10 +3643,11 @@ func TestToolCallNewColumns(t *testing.T) {
var toolUseID, inputJSON sql.NullString
var resultLen sql.NullInt64
var messageOrdinal int
err := d.Reader().QueryRow(`
SELECT tool_use_id, input_json, result_content_length
SELECT tool_use_id, input_json, result_content_length, message_ordinal
FROM tool_calls WHERE session_id = 's1'
`).Scan(&toolUseID, &inputJSON, &resultLen)
`).Scan(&toolUseID, &inputJSON, &resultLen, &messageOrdinal)
requireNoError(t, err, "query tool_calls")
require.True(t, toolUseID.Valid, "tool_use_id valid")
assert.Equal(t, "toolu_abc", toolUseID.String, "tool_use_id")
@@ -3641,6 +3655,7 @@ func TestToolCallNewColumns(t *testing.T) {
assert.Equal(t, `{"file_path":"main.go"}`, inputJSON.String, "input_json")
require.True(t, resultLen.Valid, "result_content_length valid")
assert.Equal(t, int64(500), resultLen.Int64, "result_content_length")
assert.Equal(t, 4, messageOrdinal, "message_ordinal")
}
func TestToolCallSkillName(t *testing.T) {
@@ -4194,6 +4209,61 @@ func TestReopen(t *testing.T) {
requireSessionExists(t, d, "s2")
}
func TestSQLiteBunReopenUsesCurrentGuardedHandle(t *testing.T) {
d := testDB(t)
insertSession(t, d, "bun-reopen", "before")
readProject := func() (string, error) {
var project string
err := d.view(t.Context(), func(store bun.IDB) error {
return store.NewSelect().Model((*bunmodel.Session)(nil)).
Column("project").Where("id = ?", "bun-reopen").
Scan(t.Context(), &project)
})
return project, err
}
project, err := readProject()
require.NoError(t, err)
assert.Equal(t, "before", project)
_, err = d.getWriter().Exec(
`UPDATE sessions SET project = ? WHERE id = ?`, "after", "bun-reopen",
)
require.NoError(t, err)
require.NoError(t, d.Reopen())
project, err = readProject()
require.NoError(t, err)
assert.Equal(t, "after", project)
}
func TestSQLiteBunViewKeepsReopenBehindInFlightCallback(t *testing.T) {
d := testDB(t)
started := make(chan struct{})
release := make(chan struct{})
viewDone := make(chan error, 1)
go func() {
viewDone <- d.view(t.Context(), func(bun.IDB) error {
close(started)
<-release
return nil
})
}()
<-started
reopenDone := make(chan error, 1)
go func() { reopenDone <- d.Reopen() }()
select {
case err := <-reopenDone:
close(release)
require.Failf(t, "Reopen returned during guarded Bun view", "error: %v", err)
case <-time.After(50 * time.Millisecond):
}
close(release)
require.NoError(t, <-viewDone)
require.NoError(t, <-reopenDone)
}
func TestReopenAfterSwap(t *testing.T) {
dir := t.TempDir()
@@ -4867,7 +4937,7 @@ func TestCopyInsightsFrom(t *testing.T) {
func TestCopyModelPricingFrom(t *testing.T) {
dir := t.TempDir()
// Source DB with pricing rows and a sentinel meta row.
// Source DB with pricing rows and separate refresh metadata.
srcPath := filepath.Join(dir, "src.db")
srcDB := testDBAtPath(t, srcPath, "src")
require.NoError(t, srcDB.UpsertModelPricing([]ModelPricing{
@@ -5635,6 +5705,8 @@ func TestCopyOrphanedDataFrom_LegacyNoIsSystem(t *testing.T) {
content_length
FROM messages`)
requireNoError(t, err, "copy to messages_new")
_, err = raw.Exec("DROP TRIGGER IF EXISTS tool_calls_fill_message_ordinal")
requireNoError(t, err, "drop tool call ordinal trigger")
_, err = raw.Exec("DROP TABLE messages")
requireNoError(t, err, "drop messages")
_, err = raw.Exec(
@@ -7278,8 +7350,13 @@ func TestOpenRepairsLegacyCurrentSchemaTokenCoverageOnce(t *testing.T) {
`INSERT INTO sessions (
id, project, machine, agent, message_count,
total_output_tokens, peak_context_tokens,
has_total_output_tokens, has_peak_context_tokens
) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?)`,
has_total_output_tokens, has_peak_context_tokens,
source_archive_id, source_database_generation
)
SELECT ?, ?, ?, ?, ?, ?, ?, ?, ?, archive.value, generation.value
FROM archive_metadata archive
JOIN archive_metadata generation ON generation.key = 'database_id'
WHERE archive.key = 'archive_id'`,
"current", "proj", "local", "claude", 1,
0, 0, false, false,
)
@@ -7560,11 +7637,10 @@ func TestGetSessionForIncremental(t *testing.T) {
)
requireNoError(t, err, "UpdateSessionIncremental legacy")
got, err := d.GetSessionFull(context.Background(), info.ID)
requireNoError(t, err, "GetSessionFull legacy")
require.NotNil(t, got, "legacy session missing after incremental")
assert.Equal(t, 3, reflectedIntField(got, "NextOrdinal"), "NextOrdinal")
assert.Equal(t, "entry-3", reflectedStringField(got, "LastEntryUUID"), "LastEntryUUID")
got, ok := d.GetSessionForIncremental(path, "claude")
require.True(t, ok, "legacy session missing after incremental")
assert.Equal(t, 3, got.NextOrdinal, "NextOrdinal")
assert.Equal(t, "entry-3", got.LastEntryUUID, "LastEntryUUID")
assert.True(t, got.HasTotalOutputTokens, "stored HasTotalOutputTokens = false, want true")
assert.True(t, got.HasPeakContextTokens, "stored HasPeakContextTokens = false, want true")
})
@@ -7704,8 +7780,12 @@ func TestUpdateSessionIncremental(t *testing.T) {
assert.Equal(t, ended, *got.EndedAt, "EndedAt")
require.NotNil(t, got.FileSize, "FileSize nil")
assert.Equal(t, int64(2048), *got.FileSize, "FileSize")
assert.Equal(t, 9, reflectedIntField(got, "NextOrdinal"), "NextOrdinal")
assert.Equal(t, "uuid-9", reflectedStringField(got, "LastEntryUUID"), "LastEntryUUID")
incremental, ok := d.GetSessionForIncremental(
"/tmp/sessions/update.jsonl", "codex",
)
require.True(t, ok, "session missing from incremental lookup")
assert.Equal(t, 9, incremental.NextOrdinal, "NextOrdinal")
assert.Equal(t, "uuid-9", incremental.LastEntryUUID, "LastEntryUUID")
assert.Equal(t, 500, got.TotalOutputTokens, "TotalOutputTokens")
assert.Equal(t, 1600, got.PeakContextTokens, "PeakContextTokens")
assert.True(t, got.HasTotalOutputTokens, "HasTotalOutputTokens = false, want true")
@@ -7771,6 +7851,21 @@ func TestUpdateSessionIncrementalTerminationStatus(t *testing.T) {
}
}
func parseDiffSessionSnapshot(t *testing.T, d *DB, id string) *Session {
t.Helper()
sessions, err := d.ListSessionsModifiedBetween(
t.Context(), "", "", nil, nil,
)
require.NoError(t, err)
for i := range sessions {
if sessions[i].ID == id {
return &sessions[i]
}
}
require.FailNow(t, "session missing from parse-diff snapshot", "id=%s", id)
return nil
}
// TestLastWriteIncrementalMarker pins the parse-diff detection signal:
// a fresh full write (UpsertSession) leaves last_write_incremental
// false, an incremental append (WriteSessionIncremental) sets it true,
@@ -7797,9 +7892,7 @@ func TestLastWriteIncrementalMarker(t *testing.T) {
}
requireNoError(t, d.UpsertSession(base), "initial full upsert")
got, err := d.GetSessionFull(context.Background(), "inc-marker")
requireNoError(t, err, "get after full upsert")
require.NotNil(t, got, "session after full upsert")
got := parseDiffSessionSnapshot(t, d, "inc-marker")
assert.False(t, got.LastWriteIncremental,
"full write path must leave last_write_incremental false")
@@ -7815,9 +7908,7 @@ func TestLastWriteIncrementalMarker(t *testing.T) {
},
), "incremental write")
got, err = d.GetSessionFull(context.Background(), "inc-marker")
requireNoError(t, err, "get after incremental write")
require.NotNil(t, got, "session after incremental write")
got = parseDiffSessionSnapshot(t, d, "inc-marker")
assert.True(t, got.LastWriteIncremental,
"incremental append must set last_write_incremental true")
@@ -7828,9 +7919,7 @@ func TestLastWriteIncrementalMarker(t *testing.T) {
// benign skew as real drift after any routine append-only full-parse
// sync (Claude/Codex take ReplaceMessages=false).
requireNoError(t, d.UpsertSession(base), "second full upsert")
got, err = d.GetSessionFull(context.Background(), "inc-marker")
requireNoError(t, err, "get after second full upsert")
require.NotNil(t, got, "session after second full upsert")
got = parseDiffSessionSnapshot(t, d, "inc-marker")
assert.True(t, got.LastWriteIncremental,
"a bare session upsert must preserve the marker (messages not re-normalized)")
@@ -7840,9 +7929,7 @@ func TestLastWriteIncrementalMarker(t *testing.T) {
"inc-marker",
[]Message{asstMsg("inc-marker", 1, "renormalized reply")},
), "full message replace")
got, err = d.GetSessionFull(context.Background(), "inc-marker")
requireNoError(t, err, "get after full message replace")
require.NotNil(t, got, "session after full message replace")
got = parseDiffSessionSnapshot(t, d, "inc-marker")
assert.False(t, got.LastWriteIncremental,
"a full message re-normalization must clear last_write_incremental")
}
@@ -7875,23 +7962,19 @@ func TestBatchWriteIncrementalMarkerReplaceMode(t *testing.T) {
IncrementalSessionUpdate{MsgCount: 2, UserMsgCount: 1, NextOrdinal: 2},
), "incremental write")
got, err := d.GetSessionFull(context.Background(), "batch-marker")
requireNoError(t, err, "get after incremental write")
require.NotNil(t, got, "session after incremental write")
got := parseDiffSessionSnapshot(t, d, "batch-marker")
require.True(t, got.LastWriteIncremental, "marker set by incremental write")
// Append-only full-parse batch (ReplaceMessages=false): must preserve.
appendOnly := []Message{userMsg("batch-marker", 0, "hello"), asstMsg("batch-marker", 2, "next")}
_, err = d.WriteSessionBatch([]SessionBatchWrite{{
_, err := d.WriteSessionBatch([]SessionBatchWrite{{
Session: base,
Messages: appendOnly,
DataVersion: CurrentDataVersion(),
ReplaceMessages: false,
}})
requireNoError(t, err, "append-only batch write")
got, err = d.GetSessionFull(context.Background(), "batch-marker")
requireNoError(t, err, "get after append-only batch")
require.NotNil(t, got, "session after append-only batch")
got = parseDiffSessionSnapshot(t, d, "batch-marker")
assert.True(t, got.LastWriteIncremental,
"append-only batch (ReplaceMessages=false) must preserve the marker")
@@ -7903,9 +7986,7 @@ func TestBatchWriteIncrementalMarkerReplaceMode(t *testing.T) {
ReplaceMessages: true,
}})
requireNoError(t, err, "full-replace batch write")
got, err = d.GetSessionFull(context.Background(), "batch-marker")
requireNoError(t, err, "get after full-replace batch")
require.NotNil(t, got, "session after full-replace batch")
got = parseDiffSessionSnapshot(t, d, "batch-marker")
assert.False(t, got.LastWriteIncremental,
"full-replace batch (ReplaceMessages=true) must clear the marker")
}
@@ -8699,11 +8780,16 @@ func TestCopySessionMetadataScrubsProjectIdentityGitRemoteCredentials(t *testing
oldDB, err := Open(oldPath)
requireNoError(t, err, "open old")
_, err = oldDB.rawWriter().Exec(`
INSERT INTO project_identity_observations (
INSERT INTO source_project_identity_observations (
source_archive_id, source_archive_salt,
project, machine, root_path, git_remote, git_remote_name,
worktree_name, worktree_root_path, observed_at,
normalized_remote, key_source, key
) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)`,
)
SELECT archive.value, salt.value, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?
FROM archive_metadata archive
JOIN archive_metadata salt ON salt.key = 'archive_salt'
WHERE archive.key = 'archive_id'`,
"alpha", "laptop", root,
"https://"+"user:token@"+"github.com/acme/alpha.git", "origin",
"", "", "2026-05-01T00:00:00Z",
+1 -1
View File
@@ -570,7 +570,7 @@ func (db *DB) PinnedMessagesBySession(
rows, err := db.getReader().QueryContext(ctx,
"SELECT "+pinnedBaseCols+
" FROM pinned_messages WHERE session_id IN ("+ph+")"+
" ORDER BY session_id, created_at DESC",
" ORDER BY session_id, created_at DESC, id DESC",
args...,
)
if err != nil {
+36
View File
@@ -235,3 +235,39 @@ func TestBatchedFindingsAndPinsMatchPerSession(t *testing.T) {
_, hasEmptyPins := pins["fp-empty"]
assert.False(t, hasEmptyPins, "pin map omits sessions without pins")
}
func TestPinnedMessagesBySessionMatchesCanonicalTimeAndTieOrder(t *testing.T) {
database := testDB(t)
insertSession(t, database, "pin-parity", "alpha")
insertMessages(t, database,
asstMsg("pin-parity", 0, "first"),
asstMsg("pin-parity", 1, "second"),
)
messages, err := database.GetMessages(
t.Context(), "pin-parity", 0, 10, true,
)
require.NoError(t, err)
require.Len(t, messages, 2)
for _, message := range messages {
_, err := database.PinMessage("pin-parity", message.ID, nil)
require.NoError(t, err)
}
_, err = database.getWriter().ExecContext(t.Context(), `
UPDATE pinned_messages
SET created_at = '2026-08-03T13:00:00.000Z'
WHERE session_id = 'pin-parity'`)
require.NoError(t, err)
want, err := database.ListPinnedMessages(t.Context(), "pin-parity", "")
require.NoError(t, err)
require.Len(t, want, 2)
assert.Greater(t, want[0].ID, want[1].ID)
assert.Equal(t, "2026-08-03T13:00:00Z", want[0].CreatedAt)
assert.Equal(t, "2026-08-03T13:00:00Z", want[1].CreatedAt)
bySession, err := database.PinnedMessagesBySession(
t.Context(), []string{"pin-parity"},
)
require.NoError(t, err)
assert.Equal(t, want, bySession["pin-parity"])
}
-84
View File
@@ -2,7 +2,6 @@ package db
import (
"context"
"fmt"
"path/filepath"
"strings"
"testing"
@@ -230,86 +229,3 @@ func TestEvaluateGovernedSessionsMatchesApplyEvaluator(t *testing.T) {
assert.Equal(t, 5, evaluation.GovernedSessions,
"in-fresh, in-samelabel, win-style, sibling-ref, empty-sibling are governed")
}
// TestGovernedEvaluationTouchesOnlyRuleMachines is a cardinality-scaling
// regression: the candidate-row fetch that feeds governed evaluation must be
// bounded by the number of machines carrying an ENABLED worktree mapping,
// not by total archive size and not merely by having some mapping (enabled
// or disabled). It seeds one enabled mapping on machine "ws" with 3
// sessions, a DISABLED mapping on "disabled-host" with 200 sessions, and 200
// more unrelated sessions on a machine with no mapping at all, then drives
// the real production filtering step (governedCandidateMachines, fed by
// ListAllWorktreeProjectMappings) into the candidate-row loader and asserts
// it returns exactly the 3 "ws" rows.
func TestGovernedEvaluationTouchesOnlyRuleMachines(t *testing.T) {
d := testDB(t)
ctx := context.Background()
_, err := d.CreateWorktreeProjectMapping(ctx, WorktreeProjectMapping{
Machine: "ws", PathPrefix: "/work/repo", Project: "alpha", Enabled: true,
})
require.NoError(t, err, "create enabled mapping")
_, err = d.CreateWorktreeProjectMapping(ctx, WorktreeProjectMapping{
Machine: "disabled-host", PathPrefix: "/work/other", Project: "beta",
Enabled: false,
})
require.NoError(t, err, "create disabled mapping")
wantIDs := make([]string, 0, 3)
for i := range 3 {
id := fmt.Sprintf("ws-%d", i)
wantIDs = append(wantIDs, id)
insertSession(t, d, id, "misc", func(s *Session) {
s.Machine = "ws"
s.Cwd = fmt.Sprintf("/work/repo/sub-%d", i)
})
}
// 200 sessions on a machine whose only mapping is disabled: these must
// not enter the candidate set even though the machine has a mapping.
for i := range 200 {
insertSession(t, d, fmt.Sprintf("disabled-%04d", i), "misc", func(s *Session) {
s.Machine = "disabled-host"
})
}
for i := range 200 {
insertSession(t, d, fmt.Sprintf("norule-%04d", i), "misc", func(s *Session) {
s.Machine = "no-rule-host"
})
}
mappings, err := d.ListAllWorktreeProjectMappings(ctx)
require.NoError(t, err, "list mappings")
machines := governedCandidateMachines(mappings)
rows, err := d.projectInventoryCandidateRows(ctx, "archive-1", machines)
require.NoError(t, err, "load candidate rows")
require.Len(t, rows, 3,
"fetch must be bounded by machines with an ENABLED rule, "+
"not by mapped machines or archive size")
gotIDs := make([]string, 0, len(rows))
for _, row := range rows {
assert.Equal(t, "ws", row.Machine)
assert.Equal(t, "archive-1", row.SourceArchiveID)
gotIDs = append(gotIDs, row.SessionID)
}
assert.ElementsMatch(t, wantIDs, gotIDs)
}
// TestGovernedCandidateMachinesExcludesDisabledMappings is a narrow unit
// test on the enabled-filtering step in isolation: a machine whose only
// mapping is disabled must not appear in the candidate machine set, even
// though it has a mapping row.
func TestGovernedCandidateMachinesExcludesDisabledMappings(t *testing.T) {
mappings := []WorktreeProjectMapping{
{Machine: "ws", PathPrefix: "/work/repo", Project: "alpha", Enabled: true},
{Machine: "disabled-host", PathPrefix: "/work/other", Project: "beta",
Enabled: false},
}
machines := governedCandidateMachines(mappings)
assert.Equal(t, map[string]struct{}{"ws": {}}, machines,
"only machines with an enabled mapping belong in the candidate set")
}
-166
View File
@@ -2,9 +2,7 @@ package db
import (
"context"
"database/sql"
"fmt"
"strings"
)
// Insight represents a row in the insights table.
@@ -39,162 +37,8 @@ type InsightFilter struct {
DateTo string // YYYY-MM-DD, "" = no filter
}
const insightBaseCols = `id, type, date_from, date_to,
project, agent, model, prompt, content,
kind, schema_version, template_id, template_version,
aggregate_hash, cache_key, cache_status,
provenance_json, structured_json, created_at`
func scanInsightRow(rs rowScanner) (Insight, error) {
var s Insight
err := rs.Scan(
&s.ID, &s.Type, &s.DateFrom, &s.DateTo,
&s.Project, &s.Agent,
&s.Model, &s.Prompt, &s.Content,
&s.Kind, &s.SchemaVersion, &s.TemplateID,
&s.TemplateVersion, &s.AggregateHash, &s.CacheKey,
&s.CacheStatus, &s.ProvenanceJSON, &s.StructuredJSON,
&s.CreatedAt,
)
return s, err
}
func buildInsightFilter(
f InsightFilter,
) (string, []any) {
var preds []string
var args []any
if f.Type != "" {
preds = append(preds, "type = ?")
args = append(args, f.Type)
}
if f.GlobalOnly {
preds = append(preds, "project IS NULL")
} else if f.Project != "" {
preds = append(preds, "project = ?")
args = append(args, f.Project)
}
if f.DateFrom != "" {
preds = append(preds, "date_from >= ?")
args = append(args, f.DateFrom)
}
if f.DateTo != "" {
preds = append(preds, "date_to <= ?")
args = append(args, f.DateTo)
}
if len(preds) == 0 {
return "1=1", nil
}
return strings.Join(preds, " AND "), args
}
// InsertInsight inserts an insight and returns its ID.
func (db *DB) InsertInsight(s Insight) (int64, error) {
db.mu.Lock()
defer db.mu.Unlock()
res, err := db.getWriter().Exec(`
INSERT INTO insights (
type, date_from, date_to, project,
agent, model, prompt, content,
kind, schema_version, template_id,
template_version, aggregate_hash, cache_key,
cache_status, provenance_json, structured_json
) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)`,
s.Type, s.DateFrom, s.DateTo, s.Project,
s.Agent, s.Model, s.Prompt, s.Content,
s.Kind, s.SchemaVersion, s.TemplateID,
s.TemplateVersion, s.AggregateHash, s.CacheKey,
s.CacheStatus, s.ProvenanceJSON, s.StructuredJSON,
)
if err != nil {
return 0, fmt.Errorf("inserting insight: %w", err)
}
return res.LastInsertId()
}
// GetCachedInsight returns the newest insight saved with cacheKey.
// Returns nil, nil if no cache entry exists.
func (db *DB) GetCachedInsight(
ctx context.Context, cacheKey string,
) (*Insight, error) {
if strings.TrimSpace(cacheKey) == "" {
return nil, nil
}
row := db.getReader().QueryRowContext(
ctx,
"SELECT "+insightBaseCols+
" FROM insights WHERE cache_key = ?"+
" ORDER BY created_at DESC, id DESC LIMIT 1",
cacheKey,
)
s, err := scanInsightRow(row)
if err == sql.ErrNoRows {
return nil, nil
}
if err != nil {
return nil, fmt.Errorf(
"getting cached insight: %w", err,
)
}
return &s, nil
}
const maxInsights = 500
// ListInsights returns insights matching the filter,
// ordered by created_at DESC, capped at 500 rows.
func (db *DB) ListInsights(
ctx context.Context, f InsightFilter,
) ([]Insight, error) {
where, args := buildInsightFilter(f)
query := "SELECT " + insightBaseCols +
" FROM insights WHERE " + where +
" ORDER BY created_at DESC, id DESC" +
" LIMIT " + fmt.Sprintf("%d", maxInsights)
rows, err := db.getReader().QueryContext(ctx, query, args...)
if err != nil {
return nil, fmt.Errorf("querying insights: %w", err)
}
defer rows.Close()
var insights []Insight
for rows.Next() {
s, err := scanInsightRow(rows)
if err != nil {
return nil, fmt.Errorf("scanning insight: %w", err)
}
insights = append(insights, s)
}
return insights, rows.Err()
}
// GetInsight returns a single insight by ID.
// Returns nil, nil if not found.
func (db *DB) GetInsight(
ctx context.Context, id int64,
) (*Insight, error) {
row := db.getReader().QueryRowContext(
ctx,
"SELECT "+insightBaseCols+
" FROM insights WHERE id = ?",
id,
)
s, err := scanInsightRow(row)
if err == sql.ErrNoRows {
return nil, nil
}
if err != nil {
return nil, fmt.Errorf(
"getting insight %d: %w", id, err,
)
}
return &s, nil
}
// CopyInsightsFrom copies all insights from the database at
// sourcePath into this database using ATTACH/DETACH.
func (db *DB) CopyInsightsFrom(sourcePath string) error {
@@ -266,13 +110,3 @@ func (db *DB) CopyInsightsFrom(sourcePath string) error {
}
return nil
}
// DeleteInsight removes an insight by ID.
func (db *DB) DeleteInsight(id int64) error {
db.mu.Lock()
defer db.mu.Unlock()
_, err := db.getWriter().Exec(
"DELETE FROM insights WHERE id = ?", id,
)
return err
}
+411
View File
@@ -3,14 +3,425 @@ package db
import (
"context"
"database/sql"
"errors"
"fmt"
"path/filepath"
"strings"
"testing"
"time"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
"go.kenn.io/agentsview/internal/export"
)
func priorCommonSQLiteSchema() string {
return strings.NewReplacer(
" source_archive_id TEXT NOT NULL DEFAULT '',\n"+
" source_database_generation TEXT NOT NULL DEFAULT '',\n", "",
" call_index INTEGER,\n message_ordinal INTEGER\n",
" call_index INTEGER\n",
" ordinal INTEGER NOT NULL,\n"+
" source_uuid TEXT NOT NULL DEFAULT '',\n",
" ordinal INTEGER NOT NULL,\n",
"CREATE TABLE IF NOT EXISTS pricing_metadata (\n"+
" key TEXT PRIMARY KEY,\n"+
" value TEXT NOT NULL,\n"+
" updated_at TEXT NOT NULL\n"+
" DEFAULT (strftime('%Y-%m-%dT%H:%M:%fZ','now'))\n"+
");\n\n",
"",
).Replace(schemaSQL)
}
const priorCommonSQLiteRows = `
INSERT INTO sessions (
id, project, machine, agent, first_message, message_count, created_at
) VALUES (
'common-legacy-session', 'legacy-project', 'legacy-machine', 'claude',
'named legacy session', 1, '2026-08-01T12:00:00Z'
);
INSERT INTO messages (
id, session_id, ordinal, role, content, has_tool_use, content_length
) VALUES (
41, 'common-legacy-session', 7, 'assistant', 'legacy answer', 1, 13
);
INSERT INTO tool_calls (
id, message_id, session_id, tool_name, category, call_index
) VALUES (
51, 41, 'common-legacy-session', 'Read', 'Read', 0
);
INSERT INTO pinned_messages (
id, session_id, message_id, ordinal, note, created_at
) VALUES (
61, 'common-legacy-session', 41, 7, 'keep this pin',
'2026-08-01T12:01:00Z'
);
INSERT INTO project_identity_observations (
session_id, project, machine, root_path, git_remote, observed_at,
normalized_remote, key_source, key
) VALUES (
'common-legacy-session', 'legacy-project', 'legacy-machine',
'/work/legacy', 'https://example.invalid/legacy.git',
'2026-08-01T12:02:00Z', 'example.invalid/legacy', 'git', 'legacy-key'
);
INSERT OR REPLACE INTO session_project_identity_snapshots (
session_id, project, machine, root_path, git_remote, observed_at,
normalized_remote, key_source, key
) VALUES (
'common-legacy-session', 'legacy-project', 'legacy-machine',
'/work/legacy', 'https://example.invalid/legacy.git',
'2026-08-01T12:02:00Z', 'example.invalid/legacy', 'git', 'legacy-key'
);
INSERT INTO worktree_project_mappings (
id, machine, path_prefix, layout, project, enabled
) VALUES (
71, 'legacy-machine', '/work/legacy', 'explicit', 'legacy-project', 1
);
INSERT INTO model_pricing (
model_pattern, input_microdollars_per_mtok,
output_microdollars_per_mtok, updated_at
) VALUES
('_fallback_version', 0, 0, 'legacy-v42'),
('_private-model', 1250000, 2500000, '2026-08-05T12:00:00Z');`
func createPriorCommonSQLiteArchive(t *testing.T, path string) {
t.Helper()
conn, err := sql.Open("sqlite3", makeDSN(path, false))
require.NoError(t, err)
conn.SetMaxOpenConns(1)
_, err = conn.Exec(priorCommonSQLiteSchema())
require.NoError(t, err)
_, err = conn.Exec(priorCommonSQLiteRows)
require.NoError(t, err)
_, err = conn.Exec(fmt.Sprintf("PRAGMA user_version = %d", dataVersion))
require.NoError(t, err)
require.NoError(t, conn.Close())
}
func TestLegacySchemaCommonConvergenceRetainsRowsAndBackfillsProvenance(t *testing.T) {
path := filepath.Join(t.TempDir(), "legacy-common.db")
createPriorCommonSQLiteArchive(t, path)
database, err := Open(path)
require.NoError(t, err)
defer database.Close()
session := requireSessionExists(t, database, "common-legacy-session")
assert.Equal(t, "named legacy session", *session.FirstMessage)
var archiveID, databaseGeneration string
require.NoError(t, database.getReader().QueryRowContext(t.Context(), `
SELECT source_archive_id, source_database_generation
FROM sessions WHERE id = 'common-legacy-session'`,
).Scan(&archiveID, &databaseGeneration))
assert.NotEmpty(t, archiveID)
assert.NotEmpty(t, databaseGeneration)
var toolOrdinal, pinOrdinal int
var pinNote, pinSourceUUID string
require.NoError(t, database.getReader().QueryRowContext(t.Context(), `
SELECT message_ordinal FROM tool_calls WHERE id = 51`,
).Scan(&toolOrdinal))
assert.Equal(t, 7, toolOrdinal)
require.NoError(t, database.getReader().QueryRowContext(t.Context(), `
SELECT ordinal, note, source_uuid FROM pinned_messages WHERE id = 61`,
).Scan(&pinOrdinal, &pinNote, &pinSourceUUID))
assert.Equal(t, 7, pinOrdinal)
assert.Equal(t, "keep this pin", pinNote)
assert.Empty(t, pinSourceUUID)
var observationArchive, observationSalt string
require.NoError(t, database.getReader().QueryRowContext(t.Context(), `
SELECT source_archive_id, source_archive_salt
FROM source_project_identity_observations
WHERE project = 'legacy-project'`,
).Scan(&observationArchive, &observationSalt))
assert.Equal(t, archiveID, observationArchive)
assert.NotEmpty(t, observationSalt)
var snapshotArchive, snapshotGeneration string
require.NoError(t, database.getReader().QueryRowContext(t.Context(), `
SELECT source_archive_id, source_database_generation
FROM source_session_project_identity_snapshots
WHERE source_session_id = 'common-legacy-session'`,
).Scan(&snapshotArchive, &snapshotGeneration))
assert.Equal(t, archiveID, snapshotArchive)
assert.Equal(t, databaseGeneration, snapshotGeneration)
var mappingArchive string
require.NoError(t, database.getReader().QueryRowContext(t.Context(), `
SELECT source_archive_id FROM source_worktree_project_mappings
WHERE machine = 'legacy-machine' AND path_prefix = '/work/legacy'`,
).Scan(&mappingArchive))
assert.Equal(t, archiveID, mappingArchive)
var stamp string
require.NoError(t, database.getReader().QueryRowContext(t.Context(), `
SELECT value FROM archive_metadata WHERE key = ?`,
CommonSchemaCompatibilityMetadataKey,
).Scan(&stamp))
assert.Equal(t, "1", stamp)
var pricingMetadataValue, pricingMetadataUpdatedAt string
require.NoError(t, database.getReader().QueryRowContext(t.Context(), `
SELECT value, updated_at FROM pricing_metadata
WHERE key = '_fallback_version'`,
).Scan(&pricingMetadataValue, &pricingMetadataUpdatedAt))
assert.Equal(t, "legacy-v42", pricingMetadataValue)
_, err = time.Parse(time.RFC3339Nano, pricingMetadataUpdatedAt)
require.NoError(t, err)
var pricingSentinelCount int
require.NoError(t, database.getReader().QueryRowContext(t.Context(), `
SELECT count(*) FROM model_pricing
WHERE model_pattern = '_fallback_version'`,
).Scan(&pricingSentinelCount))
assert.Zero(t, pricingSentinelCount)
var privateInput, privateOutput int64
require.NoError(t, database.getReader().QueryRowContext(t.Context(), `
SELECT input_microdollars_per_mtok, output_microdollars_per_mtok
FROM model_pricing WHERE model_pattern = '_private-model'`,
).Scan(&privateInput, &privateOutput))
assert.Equal(t, int64(1250000), privateInput)
assert.Equal(t, int64(2500000), privateOutput)
}
func TestLegacySchemaCommonCutoverWritesCanonicalRowsAndDoesNotReplay(t *testing.T) {
path := filepath.Join(t.TempDir(), "legacy-common-cutover.db")
createPriorCommonSQLiteArchive(t, path)
database, err := Open(path)
require.NoError(t, err)
observation := export.ProjectIdentityObservation{
SessionID: "common-legacy-session", Project: "runtime-project",
Machine: "legacy-machine", RootPath: "/work/runtime",
GitRemote: "https://example.invalid/runtime.git",
ObservedAt: time.Date(2026, 8, 2, 13, 0, 0, 0, time.UTC),
}
require.NoError(t, database.UpsertProjectIdentityObservation(
t.Context(), observation,
))
_, err = database.CreateWorktreeProjectMapping(t.Context(), WorktreeProjectMapping{
Machine: "legacy-machine", PathPrefix: "/work/runtime",
Layout: WorktreeMappingLayoutExplicit, Project: "runtime-project",
Enabled: true,
})
require.NoError(t, err)
for table, want := range map[string]int{
"project_identity_observations": 1,
"session_project_identity_snapshots": 1,
"worktree_project_mappings": 1,
"source_project_identity_observations": 2,
"source_session_project_identity_snapshots": 1,
"source_worktree_project_mappings": 2,
} {
var count int
require.NoError(t, database.getReader().QueryRowContext(t.Context(),
"SELECT count(*) FROM "+table,
).Scan(&count))
assert.Equal(t, want, count, table)
}
require.NoError(t, database.Close())
conn, err := sql.Open("sqlite3", makeDSN(path, false))
require.NoError(t, err)
conn.SetMaxOpenConns(1)
_, err = conn.ExecContext(t.Context(), `
UPDATE project_identity_observations
SET key = 'stale-legacy'
WHERE project = 'legacy-project';
UPDATE source_project_identity_observations
SET key = 'canonical-after-cutover'
WHERE project = 'legacy-project';
UPDATE worktree_project_mappings
SET project = 'stale-legacy'
WHERE path_prefix = '/work/legacy';
UPDATE source_worktree_project_mappings
SET project = 'canonical-after-cutover'
WHERE path_prefix = '/work/legacy';
`)
require.NoError(t, err)
require.NoError(t, conn.Close())
database, err = Open(path)
require.NoError(t, err)
defer database.Close()
var identityKey, mappingProject string
require.NoError(t, database.getReader().QueryRowContext(t.Context(), `
SELECT key FROM source_project_identity_observations
WHERE project = 'legacy-project'`,
).Scan(&identityKey))
assert.Equal(t, "canonical-after-cutover", identityKey)
require.NoError(t, database.getReader().QueryRowContext(t.Context(), `
SELECT project FROM source_worktree_project_mappings
WHERE path_prefix = '/work/legacy'`,
).Scan(&mappingProject))
assert.Equal(t, "canonical-after-cutover", mappingProject)
}
func TestLegacySchemaCommonConvergenceAddsCompleteMappingShape(t *testing.T) {
path := filepath.Join(t.TempDir(), "legacy-common-mapping-shape.db")
createPriorCommonSQLiteArchive(t, path)
conn, err := sql.Open("sqlite3", makeDSN(path, false))
require.NoError(t, err)
conn.SetMaxOpenConns(1)
_, err = conn.ExecContext(t.Context(), `
CREATE TABLE source_worktree_project_mappings (
source_archive_id TEXT NOT NULL,
machine TEXT NOT NULL,
path_prefix TEXT NOT NULL,
layout TEXT NOT NULL DEFAULT 'explicit',
project TEXT NOT NULL DEFAULT '',
original_project TEXT NOT NULL DEFAULT '',
enabled INTEGER NOT NULL DEFAULT 1,
updated_at TEXT NOT NULL DEFAULT (strftime('%Y-%m-%dT%H:%M:%fZ','now')),
PRIMARY KEY (source_archive_id, machine, path_prefix)
);
INSERT INTO worktree_project_mappings (
id, machine, path_prefix, layout, project, enabled
) VALUES (
72, 'legacy-machine', '/work/legacy-second',
'explicit', 'legacy-project', 1
)`)
require.NoError(t, err)
require.NoError(t, conn.Close())
database, err := Open(path)
require.NoError(t, err)
require.NoError(t, database.Close())
conn, err = sql.Open("sqlite3", makeDSN(path, true))
require.NoError(t, err)
defer conn.Close()
for _, column := range []string{"id", "created_at"} {
var count int
require.NoError(t, conn.QueryRowContext(t.Context(), `
SELECT count(*) FROM pragma_table_info('source_worktree_project_mappings')
WHERE name = ?`, column).Scan(&count))
assert.Equal(t, 1, count, column)
}
rows, err := conn.QueryContext(t.Context(), `
SELECT id, path_prefix
FROM source_worktree_project_mappings
WHERE machine = 'legacy-machine'
ORDER BY id`)
require.NoError(t, err)
defer rows.Close()
type mappingIdentity struct {
id int64
prefix string
}
var mappings []mappingIdentity
for rows.Next() {
var mapping mappingIdentity
require.NoError(t, rows.Scan(&mapping.id, &mapping.prefix))
mappings = append(mappings, mapping)
}
require.NoError(t, rows.Err())
assert.Equal(t, []mappingIdentity{
{id: 71, prefix: "/work/legacy"},
{id: 72, prefix: "/work/legacy-second"},
}, mappings)
}
func TestLegacySchemaStampedCommonSchemaRejectsTriggerDriftWithoutRepair(
t *testing.T,
) {
path := filepath.Join(t.TempDir(), "stamped-common-trigger-drift.db")
database, err := Open(path)
require.NoError(t, err)
require.NoError(t, database.Close())
const triggerName = "trg_source_project_identity_observations_revision_insert"
conn, err := sql.Open("sqlite3", makeDSN(path, false))
require.NoError(t, err)
conn.SetMaxOpenConns(1)
_, err = conn.ExecContext(t.Context(), `
DROP TRIGGER `+triggerName+`;
CREATE TRIGGER `+triggerName+`
AFTER INSERT ON source_project_identity_observations BEGIN
SELECT 1;
END`)
require.NoError(t, err)
var driftedSQL string
require.NoError(t, conn.QueryRowContext(t.Context(), `
SELECT sql FROM sqlite_master WHERE type = 'trigger' AND name = ?`,
triggerName,
).Scan(&driftedSQL))
require.NoError(t, conn.Close())
database, err = Open(path)
require.Error(t, err)
assert.Nil(t, database)
assert.Contains(t, err.Error(), "canonical SQLite trigger")
conn, err = sql.Open("sqlite3", makeDSN(path, true))
require.NoError(t, err)
defer conn.Close()
var afterSQL string
require.NoError(t, conn.QueryRowContext(t.Context(), `
SELECT sql FROM sqlite_master WHERE type = 'trigger' AND name = ?`,
triggerName,
).Scan(&afterSQL))
assert.Equal(t, driftedSQL, afterSQL)
}
func TestLegacySchemaCommonConvergenceRollsBackDDLDataAndStamp(t *testing.T) {
path := filepath.Join(t.TempDir(), "legacy-common-rollback.db")
createPriorCommonSQLiteArchive(t, path)
database, err := openAndInit(path, false)
require.NoError(t, err)
_, err = database.GetOrCreateDatabaseID(t.Context())
require.NoError(t, err)
_, err = database.GetOrCreateArchiveID(t.Context())
require.NoError(t, err)
_, err = database.GetOrCreateArchiveSalt(t.Context())
require.NoError(t, err)
injected := errors.New("injected common convergence failure")
database.mu.Lock()
err = database.convergeSQLiteCommonSchemaLocked(t.Context(), func() error {
return injected
})
database.mu.Unlock()
require.ErrorIs(t, err, injected)
require.NoError(t, database.Close())
conn, err := sql.Open("sqlite3", makeDSN(path, false))
require.NoError(t, err)
conn.SetMaxOpenConns(1)
defer conn.Close()
for table, column := range map[string]string{
"sessions": "source_archive_id",
"tool_calls": "message_ordinal",
"pinned_messages": "source_uuid",
} {
var count int
require.NoError(t, conn.QueryRow(`
SELECT count(*) FROM pragma_table_info(?) WHERE name = ?`,
table, column,
).Scan(&count))
assert.Zero(t, count, "%s.%s", table, column)
}
var sourceTableCount, stampCount int
require.NoError(t, conn.QueryRow(`
SELECT count(*) FROM sqlite_master
WHERE type = 'table' AND name = 'source_archives'`,
).Scan(&sourceTableCount))
assert.Zero(t, sourceTableCount)
require.NoError(t, conn.QueryRow(`
SELECT count(*) FROM archive_metadata WHERE key = ?`,
CommonSchemaCompatibilityMetadataKey,
).Scan(&stampCount))
assert.Zero(t, stampCount)
var sessionCount int
require.NoError(t, conn.QueryRow(`
SELECT count(*) FROM sessions WHERE id = 'common-legacy-session'`,
).Scan(&sessionCount))
assert.Equal(t, 1, sessionCount)
}
const legacyMessagesAndToolCallsSchema = `
CREATE TABLE messages (
id INTEGER PRIMARY KEY,
+2 -2
View File
@@ -1210,8 +1210,8 @@ func TestLinkSubagentSessionsForSessionsPlanIsBatchBounded(t *testing.T) {
assert.NotContains(t, plan, "SCAN tc",
"scoped linking must not scan tool_calls: per-event cost "+
"has to track the batch, not the archive's edges\n"+plan)
assert.Contains(t, plan, "idx_tool_calls_session",
"the spawner-side branch must seek the session_id index\n"+
assert.Regexp(t, `idx_tool_calls_(session|dedup)`, plan,
"the spawner-side branch must seek a session-prefixed index\n"+
plan)
assert.Contains(t, plan, "idx_tool_calls_subagent",
"the child-side branch must seek the subagent partial "+
+7 -258
View File
@@ -8,7 +8,6 @@ import (
"errors"
"fmt"
"log"
"slices"
"strings"
"time"
"unicode"
@@ -52,7 +51,7 @@ const (
// 999-variable limit so binaries built against older SQLite
// versions still work.
messageInsertRowsPerStmt = 38 // 26 params per row
toolCallInsertRowsPerStmt = 83 // 12 params per row (999/12 = 83)
toolCallInsertRowsPerStmt = 76 // 13 params per row (999/13 = 76)
toolResultEventInsertRowsPerStmt = 80 // 12 params per row
)
@@ -61,6 +60,7 @@ const (
type ToolCall struct {
MessageID int64 `json:"-"`
SessionID string `json:"-"`
MessageOrdinal int `json:"-"`
ToolName string `json:"tool_name"`
Category string `json:"category"`
ToolUseID string `json:"tool_use_id,omitempty"`
@@ -144,49 +144,6 @@ func (m Message) TokenPresence() (bool, bool) {
)
}
// GetMessages returns paginated messages for a session.
// from: starting ordinal (inclusive)
// limit: max messages to return
// asc: true for ascending ordinal order, false for descending
func (db *DB) GetMessages(
ctx context.Context,
sessionID string, from, limit int, asc bool,
) ([]Message, error) {
if limit <= 0 || limit > MaxMessageLimit {
limit = DefaultMessageLimit
}
dir := "ASC"
op := ">="
if !asc {
dir = "DESC"
op = "<="
}
query := fmt.Sprintf(`
SELECT %s
FROM messages
WHERE session_id = ? AND ordinal %s ?
ORDER BY ordinal %s
LIMIT ?`, selectMessageCols, op, dir)
rows, err := db.getReader().QueryContext(
ctx, query, sessionID, from, limit,
)
if err != nil {
return nil, fmt.Errorf("querying messages: %w", err)
}
defer rows.Close()
msgs, err := scanMessages(rows)
if err != nil {
return nil, err
}
if err := db.attachToolCalls(ctx, msgs); err != nil {
return nil, err
}
return msgs, nil
}
// MessageWindow parameterises GetMessagesWindow. Exactly one retrieval
// mode: Around non-nil = symmetric window; otherwise linear from/limit.
type MessageWindow struct {
@@ -199,207 +156,6 @@ type MessageWindow struct {
Roles []string // empty = all roles
}
// GetMessagesWindow returns messages for a session using either linear
// pagination (mirroring GetMessages, optionally role-filtered) or a
// symmetric window centered on an ordinal (Around/Before/After). Around
// mode always includes the anchor row even when its own role is excluded
// by Roles; the before/after counts are taken after applying the role
// filter, so they count role-matching messages rather than raw ordinal
// distance from the anchor.
func (db *DB) GetMessagesWindow(
ctx context.Context, sessionID string, w MessageWindow,
) ([]Message, error) {
if w.Around != nil {
return db.getMessagesAroundAnchor(ctx, sessionID, w)
}
from := 0
if w.From != nil {
from = *w.From
}
if len(w.Roles) == 0 {
return db.GetMessages(ctx, sessionID, from, w.Limit, w.Asc)
}
return db.getMessagesLinearRoleFiltered(
ctx, sessionID, from, w.Limit, w.Asc, w.Roles,
)
}
// getMessagesLinearRoleFiltered is GetMessages plus an "AND role IN (...)"
// predicate, used when MessageWindow.Roles is non-empty.
func (db *DB) getMessagesLinearRoleFiltered(
ctx context.Context,
sessionID string, from, limit int, asc bool, roles []string,
) ([]Message, error) {
if limit <= 0 || limit > MaxMessageLimit {
limit = DefaultMessageLimit
}
dir := "ASC"
op := ">="
if !asc {
dir = "DESC"
op = "<="
}
roleClause, roleArgs := roleFilterClause(roles)
query := fmt.Sprintf(`
SELECT %s
FROM messages
WHERE session_id = ? AND ordinal %s ?%s
ORDER BY ordinal %s
LIMIT ?`, selectMessageCols, op, roleClause, dir)
args := append([]any{sessionID, from}, roleArgs...)
args = append(args, limit)
rows, err := db.getReader().QueryContext(ctx, query, args...)
if err != nil {
return nil, fmt.Errorf("querying role-filtered messages: %w", err)
}
defer rows.Close()
msgs, err := scanMessages(rows)
if err != nil {
return nil, err
}
if err := db.attachToolCalls(ctx, msgs); err != nil {
return nil, err
}
return msgs, nil
}
// getMessagesAroundAnchor implements MessageWindow's Around mode: three
// queries (before/anchor/after) merged into one ascending slice. The
// anchor query has no role predicate so the anchor row is always present;
// before/after apply the role filter (when set) before taking Before/After
// rows, so the counts reflect role-matching messages, not raw ordinals.
func (db *DB) getMessagesAroundAnchor(
ctx context.Context, sessionID string, w MessageWindow,
) ([]Message, error) {
anchor := *w.Around
beforeLimit := max(w.Before, 0)
afterLimit := max(w.After, 0)
roleClause, roleArgs := roleFilterClause(w.Roles)
beforeQuery := fmt.Sprintf(`
SELECT %s FROM messages
WHERE session_id = ? AND ordinal < ?%s
ORDER BY ordinal DESC LIMIT ?`, selectMessageCols, roleClause)
beforeArgs := append([]any{sessionID, anchor}, roleArgs...)
beforeArgs = append(beforeArgs, beforeLimit)
before, err := db.queryMessageRows(ctx, beforeQuery, beforeArgs...)
if err != nil {
return nil, fmt.Errorf("querying before-window messages: %w", err)
}
slices.Reverse(before)
anchorQuery := fmt.Sprintf(`
SELECT %s FROM messages WHERE session_id = ? AND ordinal = ?`,
selectMessageCols)
anchorMsgs, err := db.queryMessageRows(ctx, anchorQuery, sessionID, anchor)
if err != nil {
return nil, fmt.Errorf("querying anchor message: %w", err)
}
afterQuery := fmt.Sprintf(`
SELECT %s FROM messages
WHERE session_id = ? AND ordinal > ?%s
ORDER BY ordinal ASC LIMIT ?`, selectMessageCols, roleClause)
afterArgs := append([]any{sessionID, anchor}, roleArgs...)
afterArgs = append(afterArgs, afterLimit)
after, err := db.queryMessageRows(ctx, afterQuery, afterArgs...)
if err != nil {
return nil, fmt.Errorf("querying after-window messages: %w", err)
}
msgs := make([]Message, 0, len(before)+len(anchorMsgs)+len(after))
msgs = append(msgs, before...)
msgs = append(msgs, anchorMsgs...)
msgs = append(msgs, after...)
if err := db.attachToolCalls(ctx, msgs); err != nil {
return nil, err
}
return msgs, nil
}
// queryMessageRows runs query and scans the resulting message rows,
// without attaching tool calls (callers batch that across the merged set).
func (db *DB) queryMessageRows(
ctx context.Context, query string, args ...any,
) ([]Message, error) {
rows, err := db.getReader().QueryContext(ctx, query, args...)
if err != nil {
return nil, err
}
defer rows.Close()
return scanMessages(rows)
}
// roleFilterClause returns an "AND role IN (...)" clause and its bind
// args for the given roles, or ("", nil) when roles is empty.
func roleFilterClause(roles []string) (string, []any) {
if len(roles) == 0 {
return "", nil
}
placeholders := make([]string, len(roles))
args := make([]any, len(roles))
for i, r := range roles {
placeholders[i] = "?"
args[i] = r
}
return " AND role IN (" + strings.Join(placeholders, ",") + ")", args
}
// GetAllMessages returns all messages for a session ordered by ordinal.
func (db *DB) GetAllMessages(
ctx context.Context, sessionID string,
) ([]Message, error) {
rows, err := db.getReader().QueryContext(ctx, fmt.Sprintf(`
SELECT %s
FROM messages
WHERE session_id = ?
ORDER BY ordinal ASC`, selectMessageCols), sessionID)
if err != nil {
return nil, fmt.Errorf("querying all messages: %w", err)
}
defer rows.Close()
msgs, err := scanMessages(rows)
if err != nil {
return nil, err
}
if err := db.attachToolCalls(ctx, msgs); err != nil {
return nil, err
}
return msgs, nil
}
func (db *DB) GetResumeModelCounts(
ctx context.Context, sessionID string,
) ([]ModelCount, error) {
rows, err := db.getReader().QueryContext(ctx, `
SELECT model, COUNT(*)
FROM messages
WHERE session_id = ?
AND role = 'assistant'
AND model != ''
AND model != '<synthetic>'
GROUP BY model`,
sessionID,
)
if err != nil {
return nil, fmt.Errorf("querying resume model counts: %w", err)
}
defer rows.Close()
var counts []ModelCount
for rows.Next() {
var count ModelCount
if err := rows.Scan(&count.Model, &count.Count); err != nil {
return nil, fmt.Errorf("scanning resume model count: %w", err)
}
counts = append(counts, count)
}
if err := rows.Err(); err != nil {
return nil, fmt.Errorf("iterating resume model counts: %w", err)
}
return counts, nil
}
// EmbeddableUnit is one embedding document: a single embeddable user
// message, or a run of contiguous embeddable assistant messages.
type EmbeddableUnit struct {
@@ -796,10 +552,10 @@ func multiRowPlaceholders(rows, cols int) string {
func insertToolCallsChunkTx(
tx *sql.Tx, calls []ToolCall,
) error {
args := make([]any, 0, len(calls)*12)
args := make([]any, 0, len(calls)*13)
for _, tc := range calls {
args = append(args,
tc.MessageID, tc.SessionID,
tc.MessageID, tc.SessionID, tc.MessageOrdinal,
tc.ToolName, tc.Category,
nilIfEmpty(tc.ToolUseID),
nilIfEmpty(tc.InputJSON),
@@ -813,11 +569,11 @@ func insertToolCallsChunkTx(
}
query := `
INSERT INTO tool_calls
(message_id, session_id, tool_name, category,
(message_id, session_id, message_ordinal, tool_name, category,
tool_use_id, input_json, skill_name,
result_content_length, result_content, subagent_session_id,
file_path, call_index)
VALUES ` + multiRowPlaceholders(len(calls), 12)
VALUES ` + multiRowPlaceholders(len(calls), 13)
if _, err := tx.Exec(query, args...); err != nil {
return fmt.Errorf(
"inserting tool_calls batch (%d rows): %w",
@@ -1794,14 +1550,6 @@ func restoreLegacyPinByRankTx(
return nil
}
// attachToolCalls loads tool_calls for the given messages
// and attaches them to each message's ToolCalls field.
func (db *DB) attachToolCalls(
ctx context.Context, msgs []Message,
) error {
return attachToolCallsWithQuerier(ctx, db.getReader(), msgs)
}
type messageRowsQuerier interface {
QueryContext(
ctx context.Context, query string, args ...any,
@@ -2638,6 +2386,7 @@ func resolveToolCalls(
calls = append(calls, ToolCall{
MessageID: ids[i],
SessionID: m.SessionID,
MessageOrdinal: m.Ordinal,
ToolName: tc.ToolName,
Category: tc.Category,
ToolUseID: tc.ToolUseID,
+8 -7
View File
@@ -818,25 +818,25 @@ func TestToolCallFilePathCallIndexRoundTrip(t *testing.T) {
ctx := context.Background()
insertSession(t, d, "sess-1", "proj")
insertMessages(t, d, userMsg("sess-1", 0, "hello"))
insertMessages(t, d, userMsg("sess-1", 7, "hello"))
// Fetch the message id assigned by the DB.
var msgID int64
require.NoError(t, d.getReader().QueryRowContext(ctx,
`SELECT id FROM messages WHERE session_id = 'sess-1' AND ordinal = 0`,
`SELECT id FROM messages WHERE session_id = 'sess-1' AND ordinal = 7`,
).Scan(&msgID))
tx, err := d.getWriter().Begin()
require.NoError(t, err, "begin tx")
err = insertToolCallsChunkTx(tx, []ToolCall{
{
MessageID: msgID, SessionID: "sess-1",
MessageID: msgID, SessionID: "sess-1", MessageOrdinal: 7,
ToolName: "Edit", Category: "Edit",
ToolUseID: "tu1", InputJSON: `{"file_path":"/a/b.go"}`,
FilePath: "/a/b.go", CallIndex: 0,
},
{
MessageID: msgID, SessionID: "sess-1",
MessageID: msgID, SessionID: "sess-1", MessageOrdinal: 7,
ToolName: "Write", Category: "Write",
ToolUseID: "tu2", InputJSON: `{"file":"/c/d.go"}`,
FilePath: "/c/d.go", CallIndex: 1,
@@ -846,10 +846,11 @@ func TestToolCallFilePathCallIndexRoundTrip(t *testing.T) {
require.NoError(t, tx.Commit(), "commit")
var fp string
var ci int
var ordinal, ci int
require.NoError(t, d.getReader().QueryRowContext(ctx,
`SELECT file_path, call_index FROM tool_calls
WHERE tool_use_id = 'tu2'`).Scan(&fp, &ci))
`SELECT file_path, message_ordinal, call_index FROM tool_calls
WHERE tool_use_id = 'tu2'`).Scan(&fp, &ordinal, &ci))
assert.Equal(t, "/c/d.go", fp)
assert.Equal(t, 7, ordinal)
assert.Equal(t, 1, ci)
}
+385 -136
View File
@@ -206,6 +206,11 @@ func (d *DB) CopyOrphanedDataFromExcluding(
if err := copySessionDataForIDs(ctx, tx, "_orphaned_ids"); err != nil {
return 0, fmt.Errorf("copying orphaned data: %w", err)
}
if err := stampCopiedSessionProvenance(
ctx, tx, "_orphaned_ids",
); err != nil {
return 0, fmt.Errorf("stamping orphan provenance: %w", err)
}
sourceVersion := copiedSourceDataVersion(ctx, tx)
if err := removeGeneratedIdentitySnapshotsWithoutSource(
ctx, tx, "_orphaned_ids", sourceVersion,
@@ -310,6 +315,9 @@ func (d *DB) CopyTrashedDataFrom(sourcePath string) (int, error) {
if err := copySessionDataForIDs(ctx, tx, "_trashed_ids"); err != nil {
return 0, fmt.Errorf("copying trashed data: %w", err)
}
if err := stampCopiedSessionProvenance(ctx, tx, "_trashed_ids"); err != nil {
return 0, fmt.Errorf("stamping trashed provenance: %w", err)
}
sourceVersion := copiedSourceDataVersion(ctx, tx)
if err := removeGeneratedIdentitySnapshotsWithoutSource(
ctx, tx, "_trashed_ids", sourceVersion,
@@ -1134,6 +1142,12 @@ func (d *DB) CopySessionMetadataFrom(
return fmt.Errorf("begin metadata tx: %w", err)
}
defer func() { _ = tx.Rollback() }()
var previousArchiveID string
if err := tx.QueryRowContext(ctx, `
SELECT value FROM main.archive_metadata WHERE key = 'archive_id'`,
).Scan(&previousArchiveID); err != nil {
return fmt.Errorf("reading destination archive identity: %w", err)
}
// Copy user-managed metadata from the quiesced old DB. User-owned
// deleted_at is copied for all rows. Recoverable source_missing state is
@@ -1443,6 +1457,7 @@ func (d *DB) CopySessionMetadataFrom(
FROM old_db.archive_metadata
WHERE key NOT IN (
'database_id',
'bun_common_schema_resync_pending_v1',
'project_identity_publication_revision',
'session_deletion_publication_revision',
'worktree_mapping_publication_revision'
@@ -1454,6 +1469,32 @@ func (d *DB) CopySessionMetadataFrom(
updated_at = excluded.updated_at`); err != nil {
return fmt.Errorf("copying archive metadata: %w", err)
}
if _, err := tx.ExecContext(ctx, `
INSERT INTO main.source_archives (
source_archive_id, source_archive_salt
)
SELECT archive.value, salt.value
FROM main.archive_metadata archive
JOIN main.archive_metadata salt
ON salt.key = 'archive_salt'
WHERE archive.key = 'archive_id'
ON CONFLICT(source_archive_id) DO UPDATE SET
source_archive_salt = excluded.source_archive_salt`); err != nil {
return fmt.Errorf("recording copied source archive: %w", err)
}
var copiedArchiveID string
if err := tx.QueryRowContext(ctx, `
SELECT value FROM main.archive_metadata WHERE key = 'archive_id'`,
).Scan(&copiedArchiveID); err != nil {
return fmt.Errorf("reading copied archive identity: %w", err)
}
if copiedArchiveID != previousArchiveID {
if err := rekeyLocalArchiveRows(
ctx, tx, previousArchiveID, copiedArchiveID,
); err != nil {
return err
}
}
}
// The session_deletion_changes journal is deliberately NOT copied from
@@ -1462,95 +1503,15 @@ func (d *DB) CopySessionMetadataFrom(
// journal continuity across the swap has no consumer. The fresh
// database's journal starts over with its own counter.
if oldDBHasTable(ctx, tx, "project_identity_observations") {
identityColumn := func(name, fallback string) string {
if oldDBHasColumn(ctx, tx, "project_identity_observations", name) {
return name
}
return fallback
}
if _, err := tx.ExecContext(ctx, `
INSERT INTO main.project_identity_observations (
source_archive_id, source_archive_salt,
project, machine, root_path, git_remote, git_remote_name,
repository_path, worktree_name, worktree_root_path,
worktree_relationship, checkout_state, git_branch,
remote_resolution, remote_candidate_count, observed_at,
normalized_remote, key_source, key
)
SELECT `+identityColumn("source_archive_id", "''")+`,
`+identityColumn("source_archive_salt", "''")+`,
project, machine, root_path, git_remote, git_remote_name,
`+identityColumn("repository_path", "''")+`,
worktree_name, worktree_root_path,
`+identityColumn("worktree_relationship", "'unknown'")+`,
`+identityColumn("checkout_state", "'unknown'")+`,
`+identityColumn("git_branch", "''")+`,
`+identityColumn("remote_resolution", "'unknown'")+`,
`+identityColumn("remote_candidate_count", "0")+`, observed_at,
normalized_remote, key_source, key
FROM old_db.project_identity_observations
WHERE true
ON CONFLICT(project, machine, root_path, git_remote) DO NOTHING`); err != nil {
return fmt.Errorf("copying project identity observations: %w", err)
}
if _, err := tx.ExecContext(ctx, `
DELETE FROM main.project_identity_observations
WHERE git_remote = ''
AND EXISTS (
SELECT 1
FROM main.project_identity_observations remote
WHERE remote.project = main.project_identity_observations.project
AND remote.machine = main.project_identity_observations.machine
AND remote.root_path = main.project_identity_observations.root_path
AND remote.git_remote != ''
)`); err != nil {
return fmt.Errorf(
"removing stale project identity root fallbacks: %w", err)
}
if err := scrubProjectIdentityGitRemoteCredentialsTx(ctx, tx); err != nil {
return err
}
if err := copyProjectIdentityObservationsFromAttached(ctx, tx); err != nil {
return err
}
sourceVersion := copiedSourceDataVersion(ctx, tx)
if sourceVersion >= projectIdentitySourceSnapshotDataVersion &&
oldDBHasTable(ctx, tx, "session_project_identity_snapshots") {
if _, err := tx.ExecContext(ctx, `
INSERT INTO main.session_project_identity_snapshots (
session_id, project, machine, root_path, git_remote,
git_remote_name, repository_path, worktree_name,
worktree_root_path, worktree_relationship, checkout_state,
git_branch, remote_resolution, remote_candidate_count,
observed_at, normalized_remote, key_source, key
)
SELECT session_id, project, machine, root_path, git_remote,
git_remote_name, repository_path, worktree_name,
worktree_root_path, worktree_relationship, checkout_state,
git_branch, remote_resolution, remote_candidate_count,
observed_at, normalized_remote, key_source, key
FROM old_db.session_project_identity_snapshots
WHERE session_id IN (SELECT id FROM main.sessions)
ON CONFLICT(session_id) DO UPDATE SET
project = excluded.project,
machine = excluded.machine,
root_path = excluded.root_path,
git_remote = excluded.git_remote,
git_remote_name = excluded.git_remote_name,
repository_path = excluded.repository_path,
worktree_name = excluded.worktree_name,
worktree_root_path = excluded.worktree_root_path,
worktree_relationship = excluded.worktree_relationship,
checkout_state = excluded.checkout_state,
git_branch = excluded.git_branch,
remote_resolution = excluded.remote_resolution,
remote_candidate_count = excluded.remote_candidate_count,
observed_at = excluded.observed_at,
normalized_remote = excluded.normalized_remote,
key_source = excluded.key_source,
key = excluded.key`); err != nil {
return fmt.Errorf("copying session project identity snapshots: %w", err)
}
if err := copySessionProjectIdentitySnapshotsFromAttached(
ctx, tx, sourceVersion,
); err != nil {
return err
}
if oldDBHasTable(ctx, tx, "sessions") {
@@ -1601,58 +1562,304 @@ func (d *DB) CopySessionMetadataFrom(
}
}
// Copy persistent worktree project mappings. Omit id so
// primary-key values from old_db cannot shadow existing
// destination rows. ResyncAll may pre-copy mappings into
// the temp DB before parsing, so the final metadata copy
// reconciles the table to the quiesced source state.
if oldDBHasTable(ctx, tx, "worktree_project_mappings") {
layoutSelect := "'" + WorktreeMappingLayoutExplicit + "'"
if oldDBHasColumn(ctx, tx, "worktree_project_mappings", "layout") {
layoutSelect = "layout"
}
originalProjectSelect := "''"
if oldDBHasColumn(ctx, tx, "worktree_project_mappings", "original_project") {
originalProjectSelect = "original_project"
}
if _, err := tx.ExecContext(ctx, `
DELETE FROM main.worktree_project_mappings
WHERE NOT EXISTS (
SELECT 1
FROM old_db.worktree_project_mappings old_m
WHERE old_m.machine = main.worktree_project_mappings.machine
AND replace(old_m.path_prefix, char(92), '/') =
main.worktree_project_mappings.path_prefix
)`); err != nil {
return fmt.Errorf("reconciling worktree project mappings: %w", err)
}
if _, err := tx.ExecContext(ctx, `
INSERT INTO main.worktree_project_mappings
(machine, path_prefix, layout, project, original_project,
enabled, created_at, updated_at)
SELECT machine, replace(path_prefix, char(92), '/'),
`+layoutSelect+`, project,
`+originalProjectSelect+`, enabled, created_at, updated_at
FROM old_db.worktree_project_mappings
WHERE true
ON CONFLICT(machine, path_prefix) DO UPDATE SET
layout = excluded.layout,
project = excluded.project,
original_project = CASE
WHEN worktree_project_mappings.original_project = ''
THEN excluded.original_project
ELSE worktree_project_mappings.original_project
END,
enabled = excluded.enabled,
created_at = excluded.created_at,
updated_at = excluded.updated_at`); err != nil {
return fmt.Errorf("copying worktree project mappings: %w", err)
}
if err := reconcileWorktreeProjectMappingsFromAttached(ctx, tx); err != nil {
return err
}
return tx.Commit()
}
func rekeyLocalArchiveRows(
ctx context.Context,
tx *sql.Tx,
previousArchiveID string,
copiedArchiveID string,
) error {
for _, table := range []string{
"source_project_identity_observations",
"source_session_project_identity_snapshots",
"source_worktree_project_mappings",
"sessions",
} {
if _, err := tx.ExecContext(ctx, `UPDATE main.`+table+`
SET source_archive_id = ? WHERE source_archive_id = ?`,
copiedArchiveID, previousArchiveID,
); err != nil {
return fmt.Errorf("rekeying copied archive table %s: %w", table, err)
}
}
if _, err := tx.ExecContext(ctx, `
DELETE FROM main.source_archives WHERE source_archive_id = ?`,
previousArchiveID,
); err != nil {
return fmt.Errorf("retiring replaced source archive: %w", err)
}
return nil
}
func copyProjectIdentityObservationsFromAttached(
ctx context.Context,
tx *sql.Tx,
) error {
sourceTable := ""
identityColumn := func(name, fallback string) string {
return fallback
}
if oldDBHasTable(ctx, tx, "source_project_identity_observations") {
sourceTable = "old_db.source_project_identity_observations"
identityColumn = func(name, _ string) string { return "old." + name }
} else if oldDBHasTable(ctx, tx, "project_identity_observations") {
sourceTable = "old_db.project_identity_observations"
identityColumn = func(name, fallback string) string {
if oldDBHasColumn(ctx, tx, "project_identity_observations", name) {
return "old." + name
}
return fallback
}
}
if sourceTable == "" {
return nil
}
if _, err := tx.ExecContext(ctx, `
INSERT INTO main.source_project_identity_observations (
source_archive_id, source_archive_salt,
project, machine, root_path, git_remote, git_remote_name,
repository_path, worktree_name, worktree_root_path,
worktree_relationship, checkout_state, git_branch,
remote_resolution, remote_candidate_count, observed_at,
normalized_remote, key_source, key
)
SELECT archive.value, salt.value,
old.project, old.machine, old.root_path, old.git_remote,
old.git_remote_name,
`+identityColumn("repository_path", "''")+`,
old.worktree_name, old.worktree_root_path,
`+identityColumn("worktree_relationship", "'unknown'")+`,
`+identityColumn("checkout_state", "'unknown'")+`,
`+identityColumn("git_branch", "''")+`,
`+identityColumn("remote_resolution", "'unknown'")+`,
`+identityColumn("remote_candidate_count", "0")+`,
old.observed_at, old.normalized_remote, old.key_source, old.key
FROM `+sourceTable+` old
JOIN main.archive_metadata archive ON archive.key = 'archive_id'
JOIN main.archive_metadata salt ON salt.key = 'archive_salt'
WHERE true
ON CONFLICT(
source_archive_id, project, machine, root_path, git_remote
) DO NOTHING`); err != nil {
return fmt.Errorf("copying project identity observations: %w", err)
}
if _, err := tx.ExecContext(ctx, `
DELETE FROM main.source_project_identity_observations
WHERE source_archive_id = (
SELECT value FROM main.archive_metadata WHERE key = 'archive_id'
)
AND git_remote = ''
AND EXISTS (
SELECT 1
FROM main.source_project_identity_observations remote
WHERE remote.source_archive_id =
main.source_project_identity_observations.source_archive_id
AND remote.project =
main.source_project_identity_observations.project
AND remote.machine =
main.source_project_identity_observations.machine
AND remote.root_path =
main.source_project_identity_observations.root_path
AND remote.git_remote != ''
)`); err != nil {
return fmt.Errorf(
"removing stale project identity root fallbacks: %w", err)
}
return scrubProjectIdentityGitRemoteCredentialsTx(ctx, tx)
}
func copySessionProjectIdentitySnapshotsFromAttached(
ctx context.Context,
tx *sql.Tx,
sourceVersion int,
) error {
if sourceVersion < projectIdentitySourceSnapshotDataVersion {
return nil
}
var source string
if oldDBHasTable(ctx, tx, "source_session_project_identity_snapshots") {
source = `
SELECT current.source_archive_id,
current.source_database_generation, old.source_session_id,
old.project, old.machine, old.root_path, old.git_remote,
old.git_remote_name, old.repository_path, old.worktree_name,
old.worktree_root_path, old.worktree_relationship,
old.checkout_state, old.git_branch, old.remote_resolution,
old.remote_candidate_count, old.observed_at,
old.normalized_remote, old.key_source, old.key
FROM old_db.source_session_project_identity_snapshots old
JOIN old_db.sessions previous
ON previous.id = old.source_session_id
AND previous.source_archive_id = old.source_archive_id
AND previous.source_database_generation =
old.source_database_generation
JOIN main.sessions current ON current.id = old.source_session_id`
} else if oldDBHasTable(ctx, tx, "session_project_identity_snapshots") {
source = `
SELECT current.source_archive_id,
current.source_database_generation, old.session_id,
old.project, old.machine, old.root_path, old.git_remote,
old.git_remote_name, old.repository_path, old.worktree_name,
old.worktree_root_path, old.worktree_relationship,
old.checkout_state, old.git_branch, old.remote_resolution,
old.remote_candidate_count, old.observed_at,
old.normalized_remote, old.key_source, old.key
FROM old_db.session_project_identity_snapshots old
JOIN main.sessions current ON current.id = old.session_id`
} else {
return nil
}
if _, err := tx.ExecContext(ctx, `
INSERT INTO main.source_session_project_identity_snapshots (
source_archive_id, source_database_generation, source_session_id,
project, machine, root_path, git_remote, git_remote_name,
repository_path, worktree_name, worktree_root_path,
worktree_relationship, checkout_state, git_branch,
remote_resolution, remote_candidate_count, observed_at,
normalized_remote, key_source, key
)`+source+`
ON CONFLICT(
source_archive_id, source_database_generation, source_session_id
) DO UPDATE SET
project = excluded.project,
machine = excluded.machine,
root_path = excluded.root_path,
git_remote = excluded.git_remote,
git_remote_name = excluded.git_remote_name,
repository_path = excluded.repository_path,
worktree_name = excluded.worktree_name,
worktree_root_path = excluded.worktree_root_path,
worktree_relationship = excluded.worktree_relationship,
checkout_state = excluded.checkout_state,
git_branch = excluded.git_branch,
remote_resolution = excluded.remote_resolution,
remote_candidate_count = excluded.remote_candidate_count,
observed_at = excluded.observed_at,
normalized_remote = excluded.normalized_remote,
key_source = excluded.key_source,
key = excluded.key`); err != nil {
return fmt.Errorf("copying session project identity snapshots: %w", err)
}
return nil
}
func reconcileWorktreeProjectMappingsFromAttached(
ctx context.Context,
tx *sql.Tx,
) error {
return copyWorktreeProjectMappingsFromAttached(ctx, tx, true)
}
func copyWorktreeProjectMappingsFromAttached(
ctx context.Context,
tx *sql.Tx,
deleteMissing bool,
) error {
sourceTable := ""
sourceFilter := "true"
layoutSelect := "'" + WorktreeMappingLayoutExplicit + "'"
originalProjectSelect := "''"
if oldDBHasTable(ctx, tx, "source_worktree_project_mappings") {
sourceTable = "old_db.source_worktree_project_mappings"
sourceFilter = `old.source_archive_id = (
SELECT value FROM old_db.archive_metadata WHERE key = 'archive_id'
)`
layoutSelect = "old.layout"
originalProjectSelect = "old.original_project"
} else if oldDBHasTable(ctx, tx, "worktree_project_mappings") {
sourceTable = "old_db.worktree_project_mappings"
if oldDBHasColumn(ctx, tx, "worktree_project_mappings", "layout") {
layoutSelect = "old.layout"
}
if oldDBHasColumn(ctx, tx, "worktree_project_mappings", "original_project") {
originalProjectSelect = "old.original_project"
}
}
if sourceTable == "" {
return nil
}
conflictUpdate := `
original_project = CASE
WHEN source_worktree_project_mappings.original_project = ''
THEN excluded.original_project
ELSE source_worktree_project_mappings.original_project
END`
if deleteMissing {
if _, err := tx.ExecContext(ctx, `
DELETE FROM main.source_worktree_project_mappings
WHERE source_archive_id = (
SELECT value FROM main.archive_metadata WHERE key = 'archive_id'
)
AND NOT EXISTS (
SELECT 1 FROM `+sourceTable+` old
WHERE `+sourceFilter+`
AND old.machine =
main.source_worktree_project_mappings.machine
AND replace(old.path_prefix, char(92), '/') =
main.source_worktree_project_mappings.path_prefix
)`); err != nil {
return fmt.Errorf("reconciling worktree project mappings: %w", err)
}
conflictUpdate = `
layout = excluded.layout,
project = excluded.project,
original_project = CASE
WHEN source_worktree_project_mappings.original_project = ''
THEN excluded.original_project
ELSE source_worktree_project_mappings.original_project
END,
enabled = excluded.enabled,
created_at = excluded.created_at,
updated_at = excluded.updated_at`
}
if _, err := tx.ExecContext(ctx, `
WITH source_rows AS (
SELECT old.machine,
replace(old.path_prefix, char(92), '/') AS path_prefix,
`+layoutSelect+` AS layout, old.project,
`+originalProjectSelect+` AS original_project,
old.enabled, old.created_at, old.updated_at
FROM `+sourceTable+` old
WHERE `+sourceFilter+`
), id_base AS (
SELECT COALESCE(MAX(id), 0) AS max_id
FROM main.source_worktree_project_mappings
WHERE source_archive_id = (
SELECT value FROM main.archive_metadata WHERE key = 'archive_id'
)
)
INSERT INTO main.source_worktree_project_mappings (
id, source_archive_id, machine, path_prefix, layout, project,
original_project, enabled, created_at, updated_at
)
SELECT id_base.max_id + ROW_NUMBER() OVER (
ORDER BY source_rows.machine, source_rows.path_prefix
),
archive.value, source_rows.machine, source_rows.path_prefix,
source_rows.layout, source_rows.project,
source_rows.original_project, source_rows.enabled,
source_rows.created_at, source_rows.updated_at
FROM source_rows
CROSS JOIN id_base
JOIN main.archive_metadata archive ON archive.key = 'archive_id'
WHERE true
ON CONFLICT(source_archive_id, machine, path_prefix) DO UPDATE SET`+
conflictUpdate); err != nil {
return fmt.Errorf("copying worktree project mappings: %w", err)
}
return nil
}
// oldDBHasTable checks if a table exists in old_db.
// Must be called within a connection that has old_db attached.
func oldDBHasTable(
@@ -2044,6 +2251,26 @@ func copySessionDataForIDs(
return nil
}
func stampCopiedSessionProvenance(
ctx context.Context,
tx *sql.Tx,
tempIDsTable string,
) error {
if _, err := tx.ExecContext(ctx, `
UPDATE main.sessions
SET source_archive_id = (
SELECT value FROM main.archive_metadata WHERE key = 'archive_id'
),
source_database_generation = (
SELECT value FROM main.archive_metadata WHERE key = 'database_id'
)
WHERE id IN (SELECT id FROM `+tempIDsTable+`)
AND (source_archive_id = '' OR source_database_generation = '')`); err != nil {
return fmt.Errorf("stamping copied session provenance: %w", err)
}
return nil
}
// removeGeneratedIdentitySnapshotsWithoutSource removes placeholder snapshots
// created by the session-insert trigger for the current copy batch. Sources
// predating parser-source snapshots cannot provide trustworthy replacements,
@@ -2059,16 +2286,38 @@ func removeGeneratedIdentitySnapshotsWithoutSource(
) error {
missingSourceSnapshot := "true"
if sourceVersion >= projectIdentitySourceSnapshotDataVersion &&
oldDBHasTable(ctx, tx, "source_session_project_identity_snapshots") {
missingSourceSnapshot = `NOT EXISTS (
SELECT 1
FROM old_db.source_session_project_identity_snapshots old_snapshot
JOIN old_db.sessions old_session
ON old_session.id = old_snapshot.source_session_id
AND old_session.source_archive_id = old_snapshot.source_archive_id
AND old_session.source_database_generation =
old_snapshot.source_database_generation
WHERE old_snapshot.source_session_id =
source_session_project_identity_snapshots.source_session_id
)`
} else if sourceVersion >= projectIdentitySourceSnapshotDataVersion &&
oldDBHasTable(ctx, tx, "session_project_identity_snapshots") {
missingSourceSnapshot = `NOT EXISTS (
SELECT 1 FROM old_db.session_project_identity_snapshots old_snapshot
WHERE old_snapshot.session_id =
session_project_identity_snapshots.session_id
source_session_project_identity_snapshots.source_session_id
)`
}
if _, err := tx.ExecContext(ctx, `
DELETE FROM main.session_project_identity_snapshots
WHERE session_id IN (SELECT id FROM `+tempIDsTable+`)
DELETE FROM main.source_session_project_identity_snapshots
WHERE EXISTS (
SELECT 1 FROM main.sessions current
WHERE current.id IN (SELECT id FROM `+tempIDsTable+`)
AND current.source_archive_id =
main.source_session_project_identity_snapshots.source_archive_id
AND current.source_database_generation =
main.source_session_project_identity_snapshots.source_database_generation
AND current.id =
main.source_session_project_identity_snapshots.source_session_id
)
AND `+missingSourceSnapshot); err != nil {
return fmt.Errorf("removing generated identity snapshots: %w", err)
}
+7 -122
View File
@@ -2,8 +2,9 @@ package db
import (
"context"
"database/sql"
"fmt"
"go.kenn.io/agentsview/internal/db/bunmodel"
)
// PinnedMessage represents a row in the pinned_messages table.
@@ -28,133 +29,17 @@ const pinnedBaseCols = `id, session_id, message_id, ordinal, note, created_at`
func scanPinnedRow(rs rowScanner) (PinnedMessage, error) {
var p PinnedMessage
var createdAt bunmodel.Timestamp
err := rs.Scan(
&p.ID, &p.SessionID, &p.MessageID,
&p.Ordinal, &p.Note, &p.CreatedAt,
&p.Ordinal, &p.Note, &createdAt,
)
if err == nil {
p.CreatedAt = requiredTimestampFromBunRow(createdAt)
}
return p, err
}
func scanPinnedRowWithContent(rs rowScanner) (PinnedMessage, error) {
var p PinnedMessage
err := rs.Scan(
&p.ID, &p.SessionID, &p.MessageID,
&p.Ordinal, &p.Note, &p.CreatedAt,
&p.Content, &p.Role,
&p.SessionProject, &p.SessionAgent, &p.SessionDisplayName,
&p.SessionFirstMessage,
)
return p, err
}
// PinMessage creates a pin for a message. If the message is
// already pinned, the note is updated. The message must belong to
// the specified session (enforced via INSERT ... SELECT).
func (db *DB) PinMessage(
sessionID string, messageID int64, note *string,
) (int64, error) {
db.mu.Lock()
defer db.mu.Unlock()
// Use INSERT ... SELECT to enforce session-message ownership
// and read ordinal from the messages table (not the client).
// RowsAffected is not checked because SQLite may report 0 on
// an idempotent upsert (same note value). Instead we rely on
// the subsequent SELECT to detect a missing pin.
if _, err := db.getWriter().Exec(
`INSERT INTO pinned_messages (session_id, message_id, ordinal, note)
SELECT ?, m.id, m.ordinal, ?
FROM messages m
WHERE m.id = ? AND m.session_id = ?
ON CONFLICT(session_id, message_id) DO UPDATE SET note = excluded.note`,
sessionID, note, messageID, sessionID,
); err != nil {
return 0, fmt.Errorf("pinning message: %w", err)
}
// Retrieve the actual row ID (LastInsertId is unreliable on
// upsert in SQLite). If no row exists the message did not
// belong to the session (the INSERT ... SELECT matched nothing).
var id int64
err := db.getWriter().QueryRow(
"SELECT id FROM pinned_messages WHERE session_id = ? AND message_id = ?",
sessionID, messageID,
).Scan(&id)
if err != nil {
if err == sql.ErrNoRows {
return 0, nil
}
return 0, fmt.Errorf("retrieving pin id: %w", err)
}
return id, nil
}
// UnpinMessage removes a pin.
func (db *DB) UnpinMessage(sessionID string, messageID int64) error {
db.mu.Lock()
defer db.mu.Unlock()
_, err := db.getWriter().Exec(
"DELETE FROM pinned_messages WHERE session_id = ? AND message_id = ?",
sessionID, messageID,
)
return err
}
// ListPinnedMessages returns all pins, optionally filtered by session or project.
// Pass empty sessionID for all pins across all sessions.
// When listing all pins, message content and role are included.
// project is only applied when sessionID is empty.
func (db *DB) ListPinnedMessages(
ctx context.Context, sessionID string, project string,
) ([]PinnedMessage, error) {
var query string
var args []any
if sessionID != "" {
query = "SELECT " + pinnedBaseCols +
" FROM pinned_messages WHERE session_id = ?" +
" ORDER BY created_at DESC"
args = []any{sessionID}
} else {
// Join sessions to exclude trashed sessions and include
// session metadata (project, agent, display_name) so the
// frontend doesn't need a separate lookup.
query = `SELECT p.id, p.session_id, p.message_id, p.ordinal,
p.note, p.created_at, m.content, m.role,
s.project, s.agent, COALESCE(s.display_name, s.session_name), s.first_message
FROM pinned_messages p
JOIN sessions s ON p.session_id = s.id AND s.deleted_at IS NULL
LEFT JOIN messages m ON p.message_id = m.id`
if project != "" {
query += " WHERE s.project = ?"
args = []any{project}
}
query += " ORDER BY p.created_at DESC LIMIT 500"
}
rows, err := db.getReader().QueryContext(ctx, query, args...)
if err != nil {
return nil, fmt.Errorf("listing pinned messages: %w", err)
}
defer rows.Close()
var pins []PinnedMessage
withContent := sessionID == ""
for rows.Next() {
var p PinnedMessage
var scanErr error
if withContent {
p, scanErr = scanPinnedRowWithContent(rows)
} else {
p, scanErr = scanPinnedRow(rows)
}
if scanErr != nil {
return nil, fmt.Errorf("scanning pinned message: %w", scanErr)
}
pins = append(pins, p)
}
return pins, rows.Err()
}
// PinCurationEntry is one pinned message's full curation-relevant identity:
// the state a curation fingerprint needs to detect not just a note-only
// edit (PinMessage on an already-pinned message updates the note in place,
+9 -2
View File
@@ -4,6 +4,9 @@ import (
"crypto/rand"
"database/sql"
"fmt"
"github.com/uptrace/bun"
"github.com/uptrace/bun/dialect/sqlitedialect"
)
// OpenPreparedTestDB opens a private test database file that has already been
@@ -30,15 +33,19 @@ func OpenPreparedTestDB(path string) (*DB, error) {
db := &DB{path: path}
db.writer.Store(writer)
db.reader.Store(reader)
db.bunWriter = bun.NewDB(writer, sqlitedialect.New())
db.bunReader = bun.NewDB(reader, sqlitedialect.New())
db.BunStore = NewBunStore(&sqliteBunBackend{store: db})
db.cursorSecret = make([]byte, 32)
if _, err := rand.Read(db.cursorSecret); err != nil {
cursorSecret := make([]byte, 32)
if _, err := rand.Read(cursorSecret); err != nil {
writer.Close()
reader.Close()
return nil, fmt.Errorf(
"generating prepared test cursor secret: %w", err,
)
}
db.SetCursorSecret(cursorSecret)
db.startWALCheckpointLoop()
return db, nil
+20 -21
View File
@@ -310,13 +310,12 @@ func (db *DB) DeleteModelPricing(patterns []string) error {
return nil
}
// GetPricingMeta reads a metadata value stored as a sentinel
// row in model_pricing. Returns "" if not found.
// GetPricingMeta reads SQLite-local pricing refresh state. Returns "" if not
// found.
func (db *DB) GetPricingMeta(key string) (string, error) {
var val string
err := db.getReader().QueryRow(
`SELECT updated_at FROM model_pricing
WHERE model_pattern = ?`, key,
`SELECT value FROM pricing_metadata WHERE key = ?`, key,
).Scan(&val)
if err == sql.ErrNoRows {
return "", nil
@@ -329,17 +328,15 @@ func (db *DB) GetPricingMeta(key string) (string, error) {
return val, nil
}
// SetPricingMeta stores a metadata value as a sentinel row
// in model_pricing with zero pricing fields.
// SetPricingMeta stores SQLite-local pricing refresh state separately from
// model rates.
func (db *DB) SetPricingMeta(key, value string) error {
_, err := db.getWriter().Exec(
`INSERT INTO model_pricing
(model_pattern, input_microdollars_per_mtok, output_microdollars_per_mtok,
cache_creation_microdollars_per_mtok, cache_read_microdollars_per_mtok,
updated_at)
VALUES (?, 0, 0, 0, 0, ?)
ON CONFLICT(model_pattern) DO UPDATE SET
updated_at = excluded.updated_at`,
`INSERT INTO pricing_metadata (key, value)
VALUES (?, ?)
ON CONFLICT(key) DO UPDATE SET
value = excluded.value,
updated_at = strftime('%Y-%m-%dT%H:%M:%fZ','now')`,
key, value,
)
if err != nil {
@@ -350,9 +347,8 @@ func (db *DB) SetPricingMeta(key, value string) error {
return nil
}
// CopyModelPricingFrom copies every model_pricing row (including
// sentinel metadata rows such as the fallback-version and
// refresh-attempt markers) from the database file at sourcePath.
// CopyModelPricingFrom copies model pricing, bands, and SQLite-local pricing
// refresh metadata from the database file at sourcePath.
// Called during a resync so the rebuilt DB keeps pricing across the
// swap; without it every usage cost reads as zero until the next
// daemon startup re-seeds the table.
@@ -410,6 +406,12 @@ func (db *DB) CopyModelPricingFrom(sourcePath string) error {
); err != nil {
return fmt.Errorf("copying model pricing bands: %w", err)
}
if _, err := tx.ExecContext(ctx, `
INSERT OR REPLACE INTO pricing_metadata (key, value, updated_at)
SELECT key, value, updated_at FROM old_db.pricing_metadata`,
); err != nil {
return fmt.Errorf("copying pricing metadata: %w", err)
}
if err := tx.Commit(); err != nil {
return fmt.Errorf("committing model pricing copy: %w", err)
}
@@ -470,14 +472,11 @@ func (db *DB) InsertMissingModelPricing(
// GetModelPricing returns pricing for an exact model match.
// Returns nil, nil if not found.
// HasModelPricingRows reports whether any non-meta pricing rows are
// stored, using the same meta-row exclusion as pricing map loads.
// HasModelPricingRows reports whether any pricing rows are stored.
func (db *DB) HasModelPricingRows(ctx context.Context) (bool, error) {
var exists bool
err := db.getReader().QueryRowContext(ctx,
`SELECT EXISTS(
SELECT 1 FROM model_pricing
WHERE model_pattern NOT LIKE '\_%' ESCAPE '\')`,
`SELECT EXISTS(SELECT 1 FROM model_pricing)`,
).Scan(&exists)
if err != nil {
return false, fmt.Errorf("checking pricing rows: %w", err)
+1 -2
View File
@@ -14,8 +14,7 @@ type pricingQuerier interface {
QueryContext(context.Context, string, ...any) (*sql.Rows, error)
}
// ListModelPricing returns every pricing row, including sentinel
// metadata rows (for example `_fallback_version`).
// ListModelPricing returns every model pricing row.
func (db *DB) ListModelPricing(
ctx context.Context,
) ([]ModelPricing, error) {
+35 -35
View File
@@ -3,6 +3,7 @@ package db
import (
"context"
"testing"
"time"
"go.kenn.io/agentsview/internal/config"
"go.kenn.io/agentsview/internal/export"
@@ -70,7 +71,10 @@ func TestUpsertModelPricingPricingBandsReplacesCompleteSet(t *testing.T) {
assert.Equal(t, 200_000, got.Bands[0].AboveInputTokens)
assert.Equal(t, 272_000, got.Bands[1].AboveInputTokens)
assert.NotEmpty(t, got.Bands[0].UpdatedAt)
require.NoError(t, d.SetPricingMeta("banded-model", "2000-01-01T00:00:00Z"))
_, err = d.getWriter().Exec(`
UPDATE model_pricing SET updated_at = ? WHERE model_pattern = ?`,
"2000-01-01T00:00:00Z", "banded-model")
require.NoError(t, err)
updated := initial
updated.Bands = []PricingBand{{
@@ -105,7 +109,7 @@ func TestFilterChangedModelPricingDetectsPricingBandOnlyChange(t *testing.T) {
Bands: []PricingBand{{
AboveInputTokens: 200_000,
InputPerMTok: money.MustParseDollars("2"),
UpdatedAt: "old",
UpdatedAt: "2026-08-05T12:00:00Z",
}},
}}
desired := []ModelPricing{{
@@ -114,7 +118,7 @@ func TestFilterChangedModelPricingDetectsPricingBandOnlyChange(t *testing.T) {
Bands: []PricingBand{{
AboveInputTokens: 200_000,
InputPerMTok: money.MustParseDollars("3"),
UpdatedAt: "new",
UpdatedAt: "2026-08-05T12:01:00Z",
}},
}}
@@ -191,21 +195,13 @@ func TestUpsertModelPricingOverwrites(t *testing.T) {
func TestFilterChangedModelPricingIgnoresUpdatedAtOnlyDifferences(t *testing.T) {
existing := []ModelPricing{
{
ModelPattern: "_fallback_version",
InputPerMTok: money.MustParseDollars("0"),
OutputPerMTok: money.MustParseDollars("0"),
CacheCreationPerMTok: money.MustParseDollars("0"),
CacheReadPerMTok: money.MustParseDollars("0"),
UpdatedAt: "v1",
},
{
ModelPattern: "same-model",
InputPerMTok: money.MustParseDollars("1"),
OutputPerMTok: money.MustParseDollars("2"),
CacheCreationPerMTok: money.MustParseDollars("3"),
CacheReadPerMTok: money.MustParseDollars("4"),
UpdatedAt: "old",
UpdatedAt: "2026-08-05T12:00:00Z",
},
{
ModelPattern: "changed-model",
@@ -213,25 +209,17 @@ func TestFilterChangedModelPricingIgnoresUpdatedAtOnlyDifferences(t *testing.T)
OutputPerMTok: money.MustParseDollars("2"),
CacheCreationPerMTok: money.MustParseDollars("3"),
CacheReadPerMTok: money.MustParseDollars("4"),
UpdatedAt: "old",
UpdatedAt: "2026-08-05T12:00:00Z",
},
}
desired := []ModelPricing{
{
ModelPattern: "_fallback_version",
InputPerMTok: money.MustParseDollars("0"),
OutputPerMTok: money.MustParseDollars("0"),
CacheCreationPerMTok: money.MustParseDollars("0"),
CacheReadPerMTok: money.MustParseDollars("0"),
UpdatedAt: "v2",
},
{
ModelPattern: "same-model",
InputPerMTok: money.MustParseDollars("1"),
OutputPerMTok: money.MustParseDollars("2"),
CacheCreationPerMTok: money.MustParseDollars("3"),
CacheReadPerMTok: money.MustParseDollars("4"),
UpdatedAt: "new",
UpdatedAt: "2026-08-05T12:01:00Z",
},
{
ModelPattern: "changed-model",
@@ -239,7 +227,7 @@ func TestFilterChangedModelPricingIgnoresUpdatedAtOnlyDifferences(t *testing.T)
OutputPerMTok: money.MustParseDollars("9"),
CacheCreationPerMTok: money.MustParseDollars("3"),
CacheReadPerMTok: money.MustParseDollars("4"),
UpdatedAt: "new",
UpdatedAt: "2026-08-05T12:01:00Z",
},
{
ModelPattern: "missing-model",
@@ -247,17 +235,17 @@ func TestFilterChangedModelPricingIgnoresUpdatedAtOnlyDifferences(t *testing.T)
OutputPerMTok: money.MustParseDollars("6"),
CacheCreationPerMTok: money.MustParseDollars("7"),
CacheReadPerMTok: money.MustParseDollars("8"),
UpdatedAt: "new",
UpdatedAt: "2026-08-05T12:01:00Z",
},
}
gotSummary, gotRows := FilterChangedModelPricing(existing, desired)
assert.Equal(t, PricingChangeSummary{
Total: 4,
Total: 3,
Missing: 1,
Changed: 1,
Unchanged: 2,
Unchanged: 1,
}, gotSummary)
require.Len(t, gotRows, 2)
assert.Equal(t, "changed-model", gotRows[0].ModelPattern)
@@ -288,13 +276,25 @@ func TestPricingMeta(t *testing.T) {
require.NoError(t, err, "GetPricingMeta v2")
require.Equal(t, "v2", got)
// Sentinel row does not interfere with model lookups.
var storedValue, metadataUpdatedAt string
require.NoError(t, d.getReader().QueryRow(`
SELECT value, updated_at FROM pricing_metadata
WHERE key = '_fallback_version'`,
).Scan(&storedValue, &metadataUpdatedAt))
assert.Equal(t, "v2", storedValue)
_, err = time.Parse(time.RFC3339Nano, metadataUpdatedAt)
require.NoError(t, err, "pricing metadata updated_at must be a timestamp")
var sentinelCount int
require.NoError(t, d.getReader().QueryRow(`
SELECT count(*) FROM model_pricing
WHERE model_pattern = '_fallback_version'`,
).Scan(&sentinelCount))
assert.Zero(t, sentinelCount, "metadata must not create fake pricing rows")
p, err := d.GetModelPricing("_fallback_version")
require.NoError(t, err, "GetModelPricing sentinel")
if p != nil {
assert.Zero(t, p.InputPerMTok,
"sentinel should have zero pricing, got %+v", p)
}
require.NoError(t, err, "GetModelPricing metadata key")
assert.Nil(t, p)
}
func TestGetModelPricingNotFound(t *testing.T) {
@@ -390,7 +390,7 @@ func TestLoadPricingMapKeepsCustomSourceWhenRatesMatchFallback(t *testing.T) {
},
})
rows, err := d.loadPricingMap(ctx)
rows, err := d.LoadPricingMap(ctx)
require.NoError(t, err, "loadPricingMap")
resolver := export.NewPricingResolver(rows)
lookup := resolver.Lookup("gpt-5.5")
@@ -401,7 +401,7 @@ func TestLoadPricingMapKeepsCustomSourceWhenRatesMatchFallback(t *testing.T) {
block, err := resolver.BuildBlock()
require.NoError(t, err)
assert.Equal(t, "custom", block.Source)
assert.Equal(t, "custom+embedded", block.Source)
assert.Equal(t, 1, block.CustomOverrideCount)
}
@@ -460,7 +460,7 @@ func TestLoadPricingMapTreatsBandOnlyFallbackMismatchAsFetched(t *testing.T) {
CacheReadPerMTok: fallback.CacheReadPerMTok,
}}))
rows, err := d.loadPricingMap(context.Background())
rows, err := d.LoadPricingMap(context.Background())
require.NoError(t, err)
lookup := export.NewPricingResolver(rows).Lookup("gpt-5.5")
require.True(t, lookup.OK)
+197 -105
View File
@@ -109,7 +109,7 @@ func (db *DB) LoadProjectIdentityPublicationDelta(
o.remote_resolution, o.remote_candidate_count, o.observed_at,
o.normalized_remote, o.key_source, o.key
FROM project_identity_observation_changes c
JOIN project_identity_observations o
JOIN source_project_identity_observations o
ON o.project = c.project AND o.machine = c.machine
AND o.root_path = c.root_path AND o.git_remote = c.git_remote
`+where+` AND c.deleted = 0
@@ -177,16 +177,19 @@ func (db *DB) LoadProjectIdentityPublicationDelta(
projects, excludeProjects,
)
rows, err = db.getReader().QueryContext(ctx, `
SELECT s.session_id, s.project, s.machine, s.root_path, s.git_remote,
SELECT s.source_session_id, s.project, s.machine, s.root_path, s.git_remote,
s.git_remote_name, s.repository_path, s.worktree_name,
s.worktree_root_path, s.worktree_relationship, s.checkout_state,
s.git_branch, s.remote_resolution, s.remote_candidate_count,
s.observed_at, s.normalized_remote, s.key_source, s.key
FROM session_project_identity_snapshot_changes c
JOIN session_project_identity_snapshots s
ON s.session_id = c.session_id AND s.project = c.project
JOIN source_session_project_identity_snapshots s
ON s.source_session_id = c.session_id AND s.project = c.project
JOIN sessions owner
ON owner.id = s.session_id AND owner.deleted_at IS NULL
ON owner.id = s.source_session_id
AND owner.source_archive_id = s.source_archive_id
AND owner.source_database_generation = s.source_database_generation
AND owner.deleted_at IS NULL
`+snapshotWhere+` AND c.deleted = 0
AND (TRIM(s.key_source) != '' OR TRIM(s.worktree_root_path) != '')
ORDER BY c.session_id, c.project`, snapshotArgs...)
@@ -363,6 +366,13 @@ func (db *DB) CopyArchiveIdentityFrom(sourcePath string) error {
return fmt.Errorf("beginning archive identity copy: %w", err)
}
defer func() { _ = tx.Rollback() }()
var previousArchiveID string
if err := tx.QueryRowContext(ctx, `
SELECT value FROM archive_metadata WHERE key = ?`,
archiveMetadataArchiveIDKey,
).Scan(&previousArchiveID); err != nil {
return fmt.Errorf("reading prior archive identity: %w", err)
}
for _, key := range []string{
archiveMetadataArchiveIDKey,
archiveMetadataArchiveSaltKey,
@@ -380,6 +390,24 @@ func (db *DB) CopyArchiveIdentityFrom(sourcePath string) error {
return fmt.Errorf("copying archive identity %s: %w", key, err)
}
}
if _, err := tx.ExecContext(ctx, `
INSERT INTO source_archives (source_archive_id, source_archive_salt)
VALUES (?, ?)
ON CONFLICT(source_archive_id) DO UPDATE SET
source_archive_salt = excluded.source_archive_salt`,
metadata[archiveMetadataArchiveIDKey].value,
metadata[archiveMetadataArchiveSaltKey].value,
); err != nil {
return fmt.Errorf("recording copied source archive identity: %w", err)
}
if previousArchiveID != metadata[archiveMetadataArchiveIDKey].value {
if err := rekeyLocalArchiveRows(
ctx, tx, previousArchiveID,
metadata[archiveMetadataArchiveIDKey].value,
); err != nil {
return err
}
}
if err := tx.Commit(); err != nil {
return fmt.Errorf("committing archive identity copy: %w", err)
}
@@ -657,19 +685,53 @@ func (db *DB) SetArchiveIdentityForTest(ctx context.Context, id, salt string) er
}
db.mu.Lock()
defer db.mu.Unlock()
for key, value := range map[string]string{
archiveMetadataArchiveIDKey: id, archiveMetadataArchiveSaltKey: salt,
tx, err := db.getWriter().BeginTx(ctx, nil)
if err != nil {
return fmt.Errorf("beginning archive identity repair: %w", err)
}
defer func() { _ = tx.Rollback() }()
var previousArchiveID string
if err := tx.QueryRowContext(ctx, `
SELECT value FROM archive_metadata WHERE key = ?`,
archiveMetadataArchiveIDKey,
).Scan(&previousArchiveID); err != nil {
return fmt.Errorf("reading prior archive identity: %w", err)
}
for _, item := range []struct {
key string
value string
}{
{archiveMetadataArchiveIDKey, id},
{archiveMetadataArchiveSaltKey, salt},
} {
if _, err := db.getWriter().ExecContext(ctx, `
if _, err := tx.ExecContext(ctx, `
INSERT INTO archive_metadata (key, value)
VALUES (?, ?)
ON CONFLICT(key) DO UPDATE SET value = excluded.value,
updated_at = strftime('%Y-%m-%dT%H:%M:%fZ','now')`,
key, value,
item.key, item.value,
); err != nil {
return fmt.Errorf("setting archive identity: %w", err)
}
}
if _, err := tx.ExecContext(ctx, `
INSERT INTO source_archives (source_archive_id, source_archive_salt)
VALUES (?, ?)
ON CONFLICT(source_archive_id) DO UPDATE SET
source_archive_salt = excluded.source_archive_salt`, id, salt,
); err != nil {
return fmt.Errorf("setting source archive identity: %w", err)
}
if previousArchiveID != id {
if err := rekeyLocalArchiveRows(
ctx, tx, previousArchiveID, id,
); err != nil {
return err
}
}
if err := tx.Commit(); err != nil {
return fmt.Errorf("committing archive identity repair: %w", err)
}
return nil
}
@@ -714,6 +776,16 @@ func (db *DB) upsertProjectIdentityObservationWithSnapshotProject(
if err != nil {
return err
}
identity, err := db.localArchiveIdentity(ctx)
if err != nil {
return err
}
archiveSalt, err := db.GetArchiveSalt(ctx)
if err != nil {
return err
}
obs.SourceArchiveID = identity.SourceArchiveID
obs.SourceArchiveSalt = archiveSalt
db.mu.Lock()
defer db.mu.Unlock()
@@ -803,6 +875,17 @@ func (db *DB) upsertSessionWithProjectIdentity(
)
}
obs = normalized
identity, err := db.localArchiveIdentity(context.Background())
if err != nil {
return sessionUpsertResult{}, err
}
stampSessionArchiveIdentity(&s, identity)
archiveSalt, err := db.GetArchiveSalt(context.Background())
if err != nil {
return sessionUpsertResult{}, err
}
obs.SourceArchiveID = identity.SourceArchiveID
obs.SourceArchiveSalt = archiveSalt
db.mu.Lock()
defer db.mu.Unlock()
tx, err := db.getWriter().Begin()
@@ -905,8 +988,8 @@ func writeSessionProjectIdentitySnapshotExec(
return nil
}
if _, err := exec.ExecContext(ctx, `
DELETE FROM session_project_identity_snapshots
WHERE session_id = ?`, sessionID); err != nil {
DELETE FROM source_session_project_identity_snapshots
WHERE source_session_id = ?`, sessionID); err != nil {
return fmt.Errorf("deleting session project identity snapshot: %w", err)
}
return nil
@@ -956,8 +1039,10 @@ func (db *DB) RestoreSessionProjectsFromIdentitySnapshots(
rows, err := tx.QueryContext(ctx, `
SELECT s.id, s.project, snap.project
FROM sessions s
JOIN session_project_identity_snapshots snap
ON snap.session_id = s.id
JOIN source_session_project_identity_snapshots snap
ON snap.source_session_id = s.id
AND snap.source_archive_id = s.source_archive_id
AND snap.source_database_generation = s.source_database_generation
WHERE snap.project != ''
AND snap.remote_resolution = 'resolved'
AND snap.git_remote != ''
@@ -998,14 +1083,18 @@ func (db *DB) RestoreSessionProjectsFromIdentitySnapshots(
UPDATE sessions
SET project = (
SELECT snap.project
FROM session_project_identity_snapshots snap
WHERE snap.session_id = sessions.id
FROM source_session_project_identity_snapshots snap
WHERE snap.source_session_id = sessions.id
AND snap.source_archive_id = sessions.source_archive_id
AND snap.source_database_generation = sessions.source_database_generation
)
WHERE sessions.deleted_at IS NULL
AND EXISTS (
SELECT 1
FROM session_project_identity_snapshots snap
WHERE snap.session_id = sessions.id
FROM source_session_project_identity_snapshots snap
WHERE snap.source_session_id = sessions.id
AND snap.source_archive_id = sessions.source_archive_id
AND snap.source_database_generation = sessions.source_database_generation
AND snap.project != ''
AND snap.remote_resolution = 'resolved'
AND snap.git_remote != ''
@@ -1050,8 +1139,8 @@ func reconcileSessionProjectIdentityAggregatesTx(
var machine, rootPath, gitRemote string
err := tx.QueryRowContext(ctx, `
SELECT machine, root_path, git_remote
FROM session_project_identity_snapshots
WHERE session_id = ?`, sessionID,
FROM source_session_project_identity_snapshots
WHERE source_session_id = ?`, sessionID,
).Scan(&machine, &rootPath, &gitRemote)
if errors.Is(err, sql.ErrNoRows) {
return nil
@@ -1072,7 +1161,7 @@ func reconcileSessionProjectIdentityAggregatesTx(
seen[project] = struct{}{}
if _, err := tx.ExecContext(ctx, `
DELETE FROM project_identity_observations
DELETE FROM source_project_identity_observations
WHERE project = ? AND machine = ?
AND root_path = ? AND git_remote = ?`,
project, machine, rootPath, gitRemote,
@@ -1083,7 +1172,7 @@ func reconcileSessionProjectIdentityAggregatesTx(
}
_, err := tx.ExecContext(ctx, `
INSERT INTO project_identity_observations (
INSERT INTO source_project_identity_observations (
source_archive_id, source_archive_salt, project, machine,
root_path, git_remote, git_remote_name, repository_path,
worktree_name, worktree_root_path, worktree_relationship,
@@ -1091,7 +1180,7 @@ func reconcileSessionProjectIdentityAggregatesTx(
remote_candidate_count, observed_at, normalized_remote,
key_source, key
)
SELECT '', '', ?, snap.machine,
SELECT snap.source_archive_id, archive.source_archive_salt, ?, snap.machine,
snap.root_path, snap.git_remote, snap.git_remote_name,
snap.repository_path, snap.worktree_name,
snap.worktree_root_path, snap.worktree_relationship,
@@ -1099,16 +1188,20 @@ func reconcileSessionProjectIdentityAggregatesTx(
snap.remote_resolution, snap.remote_candidate_count,
snap.observed_at, snap.normalized_remote,
snap.key_source, snap.key
FROM session_project_identity_snapshots snap
INDEXED BY idx_session_project_identity_snapshots_evidence
FROM source_session_project_identity_snapshots snap
JOIN source_archives archive
ON archive.source_archive_id = snap.source_archive_id
WHERE snap.machine = ? AND snap.root_path = ?
AND snap.git_remote = ?
AND EXISTS (
SELECT 1 FROM sessions s
WHERE s.id = snap.session_id AND s.deleted_at IS NULL
WHERE s.id = snap.source_session_id
AND s.source_archive_id = snap.source_archive_id
AND s.source_database_generation = snap.source_database_generation
AND s.deleted_at IS NULL
AND s.machine = ? AND s.project = ?
)
ORDER BY snap.observed_at DESC, snap.session_id
ORDER BY snap.observed_at DESC, snap.source_session_id
LIMIT 1`,
project, machine, rootPath, gitRemote, machine, project,
)
@@ -1131,9 +1224,12 @@ func upsertSessionProjectIdentitySnapshotExec(
if obs.SessionID == "" {
return nil
}
var sessionExists int
if err := queryRow(ctx, `SELECT 1 FROM sessions WHERE id = ?`,
obs.SessionID).Scan(&sessionExists); err != nil {
var sourceArchiveID, sourceDatabaseGeneration string
if err := queryRow(ctx, `
SELECT source_archive_id, source_database_generation
FROM sessions WHERE id = ?`, obs.SessionID).Scan(
&sourceArchiveID, &sourceDatabaseGeneration,
); err != nil {
if errors.Is(err, sql.ErrNoRows) {
return nil
}
@@ -1144,8 +1240,12 @@ func upsertSessionProjectIdentitySnapshotExec(
var existingProject string
err := queryRow(ctx, `
SELECT remote_resolution, key, project
FROM session_project_identity_snapshots
WHERE session_id = ?`, obs.SessionID).Scan(
FROM source_session_project_identity_snapshots
WHERE source_archive_id = ?
AND source_database_generation = ?
AND source_session_id = ?`,
sourceArchiveID, sourceDatabaseGeneration, obs.SessionID,
).Scan(
&existing, &existingKey, &existingProject,
)
if err == nil {
@@ -1156,9 +1256,14 @@ func upsertSessionProjectIdentitySnapshotExec(
if preserveExisting {
if allowProjectCorrection && existingProject != obs.Project {
if _, err := exec.ExecContext(ctx, `
UPDATE session_project_identity_snapshots
UPDATE source_session_project_identity_snapshots
SET project = ?
WHERE session_id = ?`, obs.Project, obs.SessionID); err != nil {
WHERE source_archive_id = ?
AND source_database_generation = ?
AND source_session_id = ?`,
obs.Project, sourceArchiveID, sourceDatabaseGeneration,
obs.SessionID,
); err != nil {
return fmt.Errorf(
"correcting session project identity snapshot label: %w", err,
)
@@ -1171,14 +1276,17 @@ func upsertSessionProjectIdentitySnapshotExec(
}
_, err = exec.ExecContext(ctx, `
INSERT INTO session_project_identity_snapshots (
session_id, project, machine, root_path, git_remote,
INSERT INTO source_session_project_identity_snapshots (
source_archive_id, source_database_generation, source_session_id,
project, machine, root_path, git_remote,
git_remote_name, repository_path, worktree_name,
worktree_root_path, worktree_relationship, checkout_state,
git_branch, remote_resolution, remote_candidate_count,
observed_at, normalized_remote, key_source, key
) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
ON CONFLICT(session_id) DO UPDATE SET
) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
ON CONFLICT(
source_archive_id, source_database_generation, source_session_id
) DO UPDATE SET
project = excluded.project,
machine = excluded.machine,
root_path = excluded.root_path,
@@ -1196,7 +1304,8 @@ func upsertSessionProjectIdentitySnapshotExec(
normalized_remote = excluded.normalized_remote,
key_source = excluded.key_source,
key = excluded.key`,
obs.SessionID, obs.Project, obs.Machine, obs.RootPath,
sourceArchiveID, sourceDatabaseGeneration, obs.SessionID,
obs.Project, obs.Machine, obs.RootPath,
obs.GitRemote, obs.GitRemoteName, obs.RepositoryPath,
obs.WorktreeName, obs.WorktreeRootPath, obs.WorktreeRelationship,
obs.CheckoutState, obs.GitBranch, obs.RemoteResolution,
@@ -1232,14 +1341,34 @@ func upsertProjectIdentityObservationExecExcludingRemote(
obs export.ProjectIdentityObservation,
excludeRemote string,
) error {
if obs.SourceArchiveID == "" && obs.SessionID != "" {
if err := queryRow(ctx, `
SELECT source_archive_id FROM sessions WHERE id = ?`,
obs.SessionID,
).Scan(&obs.SourceArchiveID); err != nil {
return fmt.Errorf("reading identity observation archive: %w", err)
}
}
if obs.SourceArchiveSalt == "" && obs.SourceArchiveID != "" {
if err := queryRow(ctx, `
SELECT source_archive_salt FROM source_archives
WHERE source_archive_id = ?`, obs.SourceArchiveID,
).Scan(&obs.SourceArchiveSalt); err != nil {
return fmt.Errorf("reading identity observation archive salt: %w", err)
}
}
if obs.SourceArchiveID == "" || obs.SourceArchiveSalt == "" {
return fmt.Errorf("project identity observation archive identity is required")
}
if obs.GitRemote == "" && obs.RemoteResolution != export.ProjectResolutionAmbiguous {
var exists int
query := `
SELECT 1 FROM project_identity_observations
WHERE project = ? AND machine = ? AND root_path = ?
SELECT 1 FROM source_project_identity_observations
WHERE source_archive_id = ?
AND project = ? AND machine = ? AND root_path = ?
AND (git_remote != '' OR remote_resolution = ?)`
args := []any{
obs.Project, obs.Machine, obs.RootPath,
obs.SourceArchiveID, obs.Project, obs.Machine, obs.RootPath,
export.ProjectResolutionAmbiguous,
}
if excludeRemote != "" {
@@ -1258,17 +1387,18 @@ func upsertProjectIdentityObservationExecExcludingRemote(
return fmt.Errorf("checking project identity remote observation: %w", err)
}
} else if _, err := exec.ExecContext(ctx, `
DELETE FROM project_identity_observations
WHERE project = ? AND machine = ? AND root_path = ?
DELETE FROM source_project_identity_observations
WHERE source_archive_id = ?
AND project = ? AND machine = ? AND root_path = ?
AND git_remote = '' AND remote_resolution != ?`,
obs.Project, obs.Machine, obs.RootPath,
obs.SourceArchiveID, obs.Project, obs.Machine, obs.RootPath,
export.ProjectResolutionAmbiguous,
); err != nil {
return fmt.Errorf("removing stale project identity root fallback: %w", err)
}
_, err := exec.ExecContext(ctx, `
INSERT INTO project_identity_observations (
INSERT INTO source_project_identity_observations (
source_archive_id, source_archive_salt,
project, machine, root_path, git_remote, git_remote_name,
repository_path, worktree_name, worktree_root_path,
@@ -1276,8 +1406,9 @@ func upsertProjectIdentityObservationExecExcludingRemote(
remote_resolution, remote_candidate_count, observed_at,
normalized_remote, key_source, key
) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
ON CONFLICT(project, machine, root_path, git_remote) DO UPDATE SET
source_archive_id = excluded.source_archive_id,
ON CONFLICT(
source_archive_id, project, machine, root_path, git_remote
) DO UPDATE SET
source_archive_salt = excluded.source_archive_salt,
git_remote_name = excluded.git_remote_name,
repository_path = excluded.repository_path,
@@ -1378,7 +1509,7 @@ func scrubProjectIdentityGitRemoteCredentialsTx(
worktree_relationship, checkout_state, git_branch,
remote_resolution, remote_candidate_count, observed_at,
normalized_remote, key_source, key
FROM project_identity_observations
FROM source_project_identity_observations
WHERE git_remote != ''`)
if err != nil {
return fmt.Errorf("listing project identity remotes for scrub: %w", err)
@@ -1448,7 +1579,7 @@ func scrubProjectIdentityGitRemoteCredentialsTx(
return fmt.Errorf("scrubbing project identity remote: %w", err)
}
if _, err := tx.ExecContext(ctx, `
DELETE FROM project_identity_observations
DELETE FROM source_project_identity_observations
WHERE project = ? AND machine = ? AND root_path = ?
AND git_remote = ?`,
scrub.obs.Project, scrub.obs.Machine, scrub.obs.RootPath,
@@ -1460,23 +1591,9 @@ func scrubProjectIdentityGitRemoteCredentialsTx(
return nil
}
// ListProjectIdentityObservations returns the aggregate identity
// observations for the given raw project labels, or every stored
// observation when labels is nil. Rows are ordered by (project, machine,
// root_path, git_remote). Label lists of any size are supported: labels
// are sorted, deduplicated, and split into maxSQLVars-sized chunks so the
// IN list never exceeds SQLite's bind-variable limit. Because project is
// the leading ORDER BY key, the chunks partition the sorted label list
// into disjoint ranges, and SQLite's default BINARY collation matches
// Go's byte-wise string order, concatenating per-chunk results preserves
// the single-query global ordering.
func (db *DB) ListProjectIdentityObservations(
ctx context.Context,
labels []string,
) ([]export.ProjectIdentityObservation, error) {
return db.listProjectIdentityObservationsFrom(ctx, db.getReader(), labels)
}
// listProjectIdentityObservationsFrom retains the transaction-scoped identity
// projection used by export snapshots. Serving reads use BunStore's canonical
// source-scoped implementation.
func (db *DB) listProjectIdentityObservationsFrom(
ctx context.Context,
q sessionExportQuerier,
@@ -1523,7 +1640,7 @@ func listProjectIdentityObservationsChunk(
worktree_relationship, checkout_state, git_branch,
remote_resolution, remote_candidate_count, observed_at,
normalized_remote, key_source, key
FROM project_identity_observations`
FROM source_project_identity_observations`
args := make([]any, 0, len(labels))
if len(labels) > 0 {
placeholders := make([]string, 0, len(labels))
@@ -1613,13 +1730,13 @@ func (db *DB) listSessionProjectIdentitySnapshotsFrom(
args[i] = id
}
rows, err := q.QueryContext(ctx, `
SELECT session_id, project, machine, root_path, git_remote,
SELECT source_session_id, project, machine, root_path, git_remote,
git_remote_name, repository_path, worktree_name,
worktree_root_path, worktree_relationship, checkout_state,
git_branch, remote_resolution, remote_candidate_count,
observed_at, normalized_remote, key_source, key
FROM session_project_identity_snapshots
WHERE session_id IN (`+strings.Join(placeholders, ",")+`)`, args...)
FROM source_session_project_identity_snapshots
WHERE source_session_id IN (`+strings.Join(placeholders, ",")+`)`, args...)
if err != nil {
return nil, fmt.Errorf("listing session project identity snapshots: %w", err)
}
@@ -1658,13 +1775,13 @@ func (db *DB) ListSessionProjectIdentitySnapshots(
ctx context.Context,
) ([]export.ProjectIdentityObservation, error) {
rows, err := db.getReader().QueryContext(ctx, `
SELECT session_id, project, machine, root_path, git_remote,
SELECT source_session_id, project, machine, root_path, git_remote,
git_remote_name, repository_path, worktree_name,
worktree_root_path, worktree_relationship, checkout_state,
git_branch, remote_resolution, remote_candidate_count,
observed_at, normalized_remote, key_source, key
FROM session_project_identity_snapshots
ORDER BY session_id`)
FROM source_session_project_identity_snapshots
ORDER BY source_session_id`)
if err != nil {
return nil, fmt.Errorf("listing all session project identity snapshots: %w", err)
}
@@ -1738,20 +1855,22 @@ func (db *DB) ListPublishableSessionProjectIdentitySnapshots(
}
appendSet("owner.project", projects, false)
appendSet("owner.project", excludeProjects, true)
appendSet("snap.session_id", ids, false)
appendSet("snap.source_session_id", ids, false)
rows, err := db.getReader().QueryContext(ctx, `
SELECT snap.session_id, snap.project, snap.machine, snap.root_path,
SELECT snap.source_session_id, snap.project, snap.machine, snap.root_path,
snap.git_remote, snap.git_remote_name, snap.repository_path,
snap.worktree_name, snap.worktree_root_path,
snap.worktree_relationship, snap.checkout_state,
snap.git_branch, snap.remote_resolution,
snap.remote_candidate_count, snap.observed_at,
snap.normalized_remote, snap.key_source, snap.key
FROM session_project_identity_snapshots snap
JOIN sessions owner ON owner.id = snap.session_id
FROM source_session_project_identity_snapshots snap
JOIN sessions owner ON owner.id = snap.source_session_id
AND owner.source_archive_id = snap.source_archive_id
AND owner.source_database_generation = snap.source_database_generation
WHERE `+strings.Join(predicates, " AND ")+`
ORDER BY snap.session_id`, args...)
ORDER BY snap.source_session_id`, args...)
if err != nil {
return nil, fmt.Errorf(
"listing publishable session project identity snapshots: %w",
@@ -1811,33 +1930,6 @@ func (db *DB) ListPublishableSessionProjectIdentitySnapshots(
return out, err
}
func (db *DB) BuildProjectIdentityMap(
ctx context.Context,
labels []string,
) (map[string]export.ProjectMapEntry, error) {
if ctx == nil {
ctx = context.Background()
}
if labels != nil && len(labels) == 0 {
return map[string]export.ProjectMapEntry{}, nil
}
observations, err := db.ListProjectIdentityObservations(ctx, labels)
if err != nil {
return nil, err
}
archiveID, err := db.GetArchiveID(ctx)
if err != nil {
return nil, err
}
archiveSalt, err := db.GetArchiveSalt(ctx)
if err != nil {
return nil, err
}
return export.BuildProjectsMapWithScope(labels, observations, export.IdentityScope{
ArchiveID: archiveID, ArchiveSalt: archiveSalt,
}), nil
}
func newUUIDv4() (string, error) {
var b [16]byte
if _, err := rand.Read(b[:]); err != nil {
+13 -6
View File
@@ -58,8 +58,11 @@ func ensureProjectIdentityBackfillQueuedTx(
SELECT EXISTS(
SELECT 1 FROM sessions s
WHERE NOT EXISTS (
SELECT 1 FROM session_project_identity_snapshots p
WHERE p.session_id = s.id
SELECT 1 FROM source_session_project_identity_snapshots p
WHERE p.source_archive_id = s.source_archive_id
AND p.source_database_generation =
s.source_database_generation
AND p.source_session_id = s.id
)
)`).Scan(&missing); err != nil {
return fmt.Errorf("checking project identity backfill candidates: %w", err)
@@ -199,8 +202,10 @@ func (db *DB) countMissingProjectIdentitySnapshots(ctx context.Context) (int, er
if err := db.getReader().QueryRowContext(ctx, `
SELECT count(*) FROM sessions s
WHERE NOT EXISTS (
SELECT 1 FROM session_project_identity_snapshots p
WHERE p.session_id = s.id
SELECT 1 FROM source_session_project_identity_snapshots p
WHERE p.source_archive_id = s.source_archive_id
AND p.source_database_generation = s.source_database_generation
AND p.source_session_id = s.id
)`).Scan(&missing); err != nil {
return 0, fmt.Errorf("counting missing project identity snapshots: %w", err)
}
@@ -226,8 +231,10 @@ func (db *DB) ProjectIdentityBackfillCandidatesAfter(
s.started_at
FROM sessions s
WHERE s.id > ? AND NOT EXISTS (
SELECT 1 FROM session_project_identity_snapshots p
WHERE p.session_id = s.id
SELECT 1 FROM source_session_project_identity_snapshots p
WHERE p.source_archive_id = s.source_archive_id
AND p.source_database_generation = s.source_database_generation
AND p.source_session_id = s.id
)
ORDER BY s.id
LIMIT ?`, afterID, projectIdentityBackfillBatchSize)
@@ -22,7 +22,8 @@ func TestEnsureProjectIdentityBackfillRequeuesUnverifiedCompletedGap(
Agent: "codex",
}))
_, err := d.getWriter().ExecContext(ctx,
`DELETE FROM session_project_identity_snapshots WHERE session_id = ?`,
`DELETE FROM source_session_project_identity_snapshots
WHERE source_session_id = ?`,
"missing-snapshot")
require.NoError(t, err)
_, err = d.getWriter().ExecContext(ctx, `
@@ -117,8 +118,8 @@ func TestResyncOrphanCopyLeavesLegacySnapshotGapEligibleForBackfill(
ObservedAt: time.Date(2026, 7, 1, 0, 0, 0, 0, time.UTC),
}))
_, err = source.getWriter().ExecContext(ctx, `
DELETE FROM session_project_identity_snapshots
WHERE session_id IN ('live', 'legacy-orphan')`)
DELETE FROM source_session_project_identity_snapshots
WHERE source_session_id IN ('live', 'legacy-orphan')`)
require.NoError(t, err)
require.NoError(t, source.Close())
@@ -228,8 +229,8 @@ func TestResyncTrashedCopyLeavesLegacySnapshotGapEligibleForBackfill(
UPDATE sessions
SET deleted_at = strftime('%Y-%m-%dT%H:%M:%fZ','now')
WHERE id = 'legacy-trashed';
DELETE FROM session_project_identity_snapshots
WHERE session_id = 'legacy-trashed'`)
DELETE FROM source_session_project_identity_snapshots
WHERE source_session_id = 'legacy-trashed'`)
require.NoError(t, err)
require.NoError(t, source.Close())
@@ -261,7 +262,7 @@ func TestProjectIdentityBackfillBatchUsesKeysetAndAdvancesAtomically(
}))
}
_, err := d.getWriter().ExecContext(ctx,
`DELETE FROM session_project_identity_snapshots`)
`DELETE FROM source_session_project_identity_snapshots`)
require.NoError(t, err)
require.NoError(t, d.EnsureProjectIdentityBackfillQueued(ctx))
require.NoError(t, d.StartProjectIdentityBackfill(ctx))
@@ -304,7 +305,8 @@ func TestProjectIdentityBackfillPersistsUnknownSnapshotForEmptyProject(
ID: "unresolved", Machine: "local", Agent: "antigravity-cli",
}))
_, err := d.getWriter().ExecContext(ctx,
`DELETE FROM session_project_identity_snapshots WHERE session_id = ?`,
`DELETE FROM source_session_project_identity_snapshots
WHERE source_session_id = ?`,
"unresolved")
require.NoError(t, err)
require.NoError(t, d.EnsureProjectIdentityBackfillQueued(ctx))
+64 -18
View File
@@ -18,7 +18,9 @@ import (
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
"github.com/uptrace/bun"
"go.kenn.io/agentsview/internal/db/bunmodel"
"go.kenn.io/agentsview/internal/export"
)
@@ -250,8 +252,10 @@ func TestLoadProjectIdentityPublicationDeltaReturnsRowsAndTombstones(
},
))
_, err = d.rawWriter().ExecContext(ctx, `
DELETE FROM project_identity_observations
WHERE project = ? AND machine = ? AND root_path = ? AND git_remote = ?`,
DELETE FROM source_project_identity_observations
WHERE source_archive_id = (
SELECT value FROM archive_metadata WHERE key = 'archive_id'
) AND project = ? AND machine = ? AND root_path = ? AND git_remote = ?`,
"beta", "local", "/workspace/beta", "https://example.com/beta.git",
)
require.NoError(t, err)
@@ -316,6 +320,16 @@ func TestCopyArchiveIdentityFromPreservesLogicalArchiveAndNewGeneration(
databaseID, err := target.GetDatabaseID(ctx)
require.NoError(t, err)
assert.Equal(t, "new-generation", databaseID)
var sourceArchives []bunmodel.SourceArchive
require.NoError(t, target.view(ctx, func(store bun.IDB) error {
return store.NewSelect().Model(&sourceArchives).
OrderExpr("source_archive_id ASC").Scan(ctx)
}))
assert.Equal(t, []bunmodel.SourceArchive{{
SourceArchiveID: "stable-archive", SourceArchiveSalt: strings.Repeat("a", 64),
}}, sourceArchives,
"copying the logical archive identity must retire the temporary identity")
}
func TestProjectObservationArchiveSaltIsCreatedAndStable(t *testing.T) {
@@ -378,12 +392,16 @@ func TestProjectIdentityObservationRoundTripsRepositoryContext(t *testing.T) {
}
require.NoError(t, d.UpsertProjectIdentityObservation(ctx, want))
wantArchiveID, err := d.GetArchiveID(ctx)
require.NoError(t, err)
wantArchiveSalt, err := d.GetArchiveSalt(ctx)
require.NoError(t, err)
got, err := d.ListProjectIdentityObservations(ctx, []string{"app"})
require.NoError(t, err)
require.Len(t, got, 1)
assert.Equal(t, want.RepositoryPath, got[0].RepositoryPath)
assert.Equal(t, want.SourceArchiveID, got[0].SourceArchiveID)
assert.Equal(t, want.SourceArchiveSalt, got[0].SourceArchiveSalt)
assert.Equal(t, wantArchiveID, got[0].SourceArchiveID)
assert.Equal(t, wantArchiveSalt, got[0].SourceArchiveSalt)
assert.Equal(t, want.WorktreeRelationship, got[0].WorktreeRelationship)
assert.Equal(t, want.CheckoutState, got[0].CheckoutState)
assert.Equal(t, want.RemoteResolution, got[0].RemoteResolution)
@@ -511,6 +529,9 @@ func TestProjectObservationMigrationStripsStoredGitRemoteCredentials(t *testing.
_, err = d.rawWriter().Exec(`DELETE FROM stats WHERE key = ?`,
projectIdentityRemoteScrubCompletedKey)
require.NoError(t, err)
_, err = d.rawWriter().Exec(`DELETE FROM archive_metadata WHERE key = ?`,
CommonSchemaCompatibilityMetadataKey)
require.NoError(t, err)
require.NoError(t, d.Close())
reopened, err := Open(path)
@@ -583,21 +604,41 @@ func TestListProjectIdentityObservationsChunksLargeLabelLists(t *testing.T) {
// preserve the single-query (project, machine, ...) ordering.
const labelCount = maxSQLVars + 50
labels := make([]string, 0, 2*labelCount)
var sb strings.Builder
sb.WriteString(`INSERT INTO project_identity_observations
(project, machine, root_path, observed_at) VALUES `)
args := make([]any, 0, labelCount)
for i := range labelCount {
label := fmt.Sprintf("chunked-project-%04d", i)
labels = append(labels, label, label)
if i > 0 {
sb.WriteString(",")
}
sb.WriteString("(?, 'host.example', '/srv/app', '2025-06-02T10:00:00Z')")
args = append(args, label)
}
_, err := d.getWriter().Exec(sb.String(), args...)
require.NoError(t, err, "seed chunked observations")
for start := 0; start < labelCount; start += 400 {
end := min(start+400, labelCount)
var sb strings.Builder
sb.WriteString(`INSERT INTO source_project_identity_observations (
source_archive_id, source_archive_salt,
project, machine, root_path, git_remote, git_remote_name,
repository_path, worktree_name, worktree_root_path,
worktree_relationship, checkout_state, git_branch,
remote_resolution, remote_candidate_count, observed_at,
normalized_remote, key_source, key
)
SELECT archive.value, salt.value, labels.project,
'host.example', '/srv/app', '', '', '', '', '',
'unknown', 'unknown', '', 'unknown', 0,
'2025-06-02T10:00:00Z', '', '', ''
FROM (`)
args := make([]any, 0, end-start)
for i := start; i < end; i++ {
if i == start {
sb.WriteString("SELECT ? AS project")
} else {
sb.WriteString(" UNION ALL SELECT ?")
}
args = append(args, fmt.Sprintf("chunked-project-%04d", i))
}
sb.WriteString(`) labels
JOIN archive_metadata archive ON archive.key = 'archive_id'
JOIN archive_metadata salt ON salt.key = 'archive_salt'`)
_, err := d.getWriter().Exec(sb.String(), args...)
require.NoError(t, err, "seed chunked observations")
}
// Reverse the (duplicated) label list to prove the lookup sorts it
// before partitioning into chunks.
@@ -1359,11 +1400,16 @@ func TestProjectObservationScrubDowngradesUnusableRemoteToFallback(t *testing.T)
ctx := context.Background()
root := filepath.Join(t.TempDir(), "repo")
_, err := d.getWriter().ExecContext(ctx, `
INSERT INTO project_identity_observations (
INSERT INTO source_project_identity_observations (
source_archive_id, source_archive_salt,
project, machine, root_path, git_remote, git_remote_name,
worktree_name, worktree_root_path, observed_at,
normalized_remote, key_source, key
) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)`,
)
SELECT archive.value, salt.value, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?
FROM archive_metadata archive
JOIN archive_metadata salt ON salt.key = 'archive_salt'
WHERE archive.key = 'archive_id'`,
"app", "laptop", root, "file:///tmp/app.git", "origin",
"", "", "2026-07-03T12:00:00Z", "", "", "",
)
@@ -1707,7 +1753,7 @@ func projectObservationRowCount(t *testing.T, d *DB) int {
t.Helper()
var n int
require.NoError(t, d.getReader().QueryRow(
`SELECT COUNT(*) FROM project_identity_observations`,
`SELECT COUNT(*) FROM source_project_identity_observations`,
).Scan(&n))
return n
}
-209
View File
@@ -1,11 +1,7 @@
package db
import (
"context"
"database/sql"
"fmt"
"sort"
"strings"
"time"
"go.kenn.io/agentsview/internal/export"
@@ -46,108 +42,6 @@ type projectInventoryAgg struct {
last *time.Time
}
// GetProjectInventory aggregates every visible session into a per-project
// inventory: session/machine/agent/cwd counts and activity bounds, plus
// worktree-mapping-rule attribution (which enabled rules target each
// project, and whether it was ever recorded as a rule's original_project).
func (db *DB) GetProjectInventory(ctx context.Context) (ProjectInventory, error) {
if ctx == nil {
ctx = context.Background()
}
agg, err := db.projectInventoryAggregate(ctx)
if err != nil {
return ProjectInventory{}, err
}
rawProjects := make([]string, 0, len(agg))
for project := range agg {
rawProjects = append(rawProjects, project)
}
mappings, eval, err := db.projectInventoryGovernance(ctx)
if err != nil {
return ProjectInventory{}, err
}
projects, err := db.BuildProjectIdentityMap(ctx, rawProjects)
if err != nil {
return ProjectInventory{}, err
}
rows, totalSessions := buildProjectInventoryRows(agg, rawProjects, projects)
annotateProjectInventoryRows(rows, mappings, eval, projects)
return ProjectInventory{
Projects: rows,
TotalProjects: len(rows),
TotalSessions: totalSessions,
GovernedSessions: eval.GovernedSessions,
}, nil
}
// projectInventoryAggregate runs the one-pass aggregation over visible
// sessions, grouped by raw (unsanitized) project label. cwd distinctness
// normalizes backslashes to forward slashes so a Windows-style cwd and its
// POSIX-style equivalent collapse to one entry; this is the dominant
// cross-platform duplicate case and is expressible in all three storage
// dialects.
func (db *DB) projectInventoryAggregate(
ctx context.Context,
) (map[string]projectInventoryAgg, error) {
rows, err := db.getReader().QueryContext(ctx, projectInventoryAggregateQuery())
if err != nil {
return nil, fmt.Errorf("aggregating project inventory: %w", err)
}
defer rows.Close()
out := map[string]projectInventoryAgg{}
for rows.Next() {
var project string
var agg projectInventoryAgg
var first, last sql.NullString
if err := rows.Scan(
&project, &agg.sessions, &agg.machines, &agg.agents,
&agg.distinctCwds, &first, &last,
); err != nil {
return nil, fmt.Errorf("scanning project inventory row: %w", err)
}
if first.Valid && first.String != "" {
if t, err := parseTimestamp(first.String); err == nil {
agg.first = &t
}
}
if last.Valid && last.String != "" {
if t, err := parseTimestamp(last.String); err == nil {
agg.last = &t
}
}
out[project] = agg
}
if err := rows.Err(); err != nil {
return nil, fmt.Errorf("iterating project inventory rows: %w", err)
}
return out, nil
}
// projectInventoryAggregateQuery returns the one-pass aggregation SQL
// grouped by raw project label. Factored out so tests can EXPLAIN QUERY
// PLAN it directly and assert it stays a single sessions scan. Timestamps
// are stored as TEXT and legacy rows may hold empty strings instead of
// NULL; NULLIF keeps such rows from corrupting MIN (an empty string sorts
// before every real timestamp).
func projectInventoryAggregateQuery() string {
return `
SELECT project,
COUNT(*),
COUNT(DISTINCT machine),
COUNT(DISTINCT agent),
COUNT(DISTINCT CASE WHEN cwd IS NOT NULL AND cwd != ''
THEN replace(cwd, '\', '/') END),
MIN(NULLIF(started_at, '')),
MAX(COALESCE(NULLIF(ended_at, ''), NULLIF(started_at, '')))
FROM sessions
WHERE deleted_at IS NULL
GROUP BY project
ORDER BY project`
}
// buildProjectInventoryRows groups raw project labels by opaque project key.
// Display-label sanitization is presentation-only: distinct absolute-path
// projects may both display as empty without losing either row or key.
@@ -196,109 +90,6 @@ func buildProjectInventoryRows(
return rowList, totalSessions
}
// projectInventoryGovernance loads every worktree mapping (enabled and
// disabled) plus the candidate session rows for machines with at least one
// enabled mapping, then runs the shared governed-session evaluator.
func (db *DB) projectInventoryGovernance(
ctx context.Context,
) ([]WorktreeProjectMapping, GovernedEvaluation, error) {
mappings, err := db.ListAllWorktreeProjectMappings(ctx)
if err != nil {
return nil, GovernedEvaluation{}, fmt.Errorf(
"listing worktree mappings for project inventory: %w", err)
}
archiveID, err := db.GetArchiveID(ctx)
if err != nil {
return nil, GovernedEvaluation{}, fmt.Errorf(
"resolving archive id for project inventory: %w", err)
}
machines := governedCandidateMachines(mappings)
candidates, err := db.projectInventoryCandidateRows(ctx, archiveID, machines)
if err != nil {
return nil, GovernedEvaluation{}, err
}
eval := EvaluateGovernedSessions(
[]ArchiveMappings{{SourceArchiveID: archiveID, Mappings: mappings}},
candidates,
)
return mappings, eval, nil
}
// governedCandidateMachines returns the set of machines carrying at least
// one enabled worktree mapping. Only these machines' sessions are fetched
// as governed-evaluation candidates; a machine whose mappings are all
// disabled (or that has no mapping at all) contributes no candidate rows,
// regardless of its session count.
func governedCandidateMachines(
mappings []WorktreeProjectMapping,
) map[string]struct{} {
machines := map[string]struct{}{}
for _, m := range mappings {
if m.Enabled {
machines[m.Machine] = struct{}{}
}
}
return machines
}
// projectInventoryCandidateRows returns the prefiltered session rows the
// evaluator needs: every visible session on a machine with at least one
// enabled worktree mapping. SourceArchiveID is set to the local archive ID
// for every row, matching the ArchiveMappings entry built by the caller.
func (db *DB) projectInventoryCandidateRows(
ctx context.Context, archiveID string, machines map[string]struct{},
) ([]MappingEvaluationRow, error) {
if len(machines) == 0 {
return nil, nil
}
machineList := sortedSetKeys(machines)
query, args := projectInventoryCandidateQuery(machineList)
rows, err := db.getReader().QueryContext(ctx, query, args...)
if err != nil {
return nil, fmt.Errorf(
"querying project inventory candidate sessions: %w", err)
}
defer rows.Close()
var out []MappingEvaluationRow
for rows.Next() {
var row MappingEvaluationRow
if err := rows.Scan(
&row.SessionID, &row.Machine, &row.Project, &row.Cwd, &row.FilePath,
); err != nil {
return nil, fmt.Errorf(
"scanning project inventory candidate session: %w", err)
}
row.SourceArchiveID = archiveID
out = append(out, row)
}
if err := rows.Err(); err != nil {
return nil, fmt.Errorf(
"iterating project inventory candidate sessions: %w", err)
}
return out, nil
}
// projectInventoryCandidateQuery returns the candidate-row SQL and its bind
// args for the given machine list. Factored out so tests can EXPLAIN QUERY
// PLAN it directly and assert it uses the sessions machine index.
func projectInventoryCandidateQuery(machineList []string) (string, []any) {
placeholders := make([]string, len(machineList))
args := make([]any, len(machineList))
for i, m := range machineList {
placeholders[i] = "?"
args[i] = m
}
query := `
SELECT id, machine, project, cwd, COALESCE(file_path, '')
FROM sessions
WHERE deleted_at IS NULL
AND machine IN (` + strings.Join(placeholders, ",") + `)`
return query, args
}
// annotateProjectInventoryRows sets EnabledRulesTargeting and
// RecordedAsOriginal on rows in place, keyed by opaque project identity.
//
-51
View File
@@ -2,9 +2,7 @@ package db
import (
"context"
"database/sql"
"sort"
"strings"
"testing"
"time"
@@ -276,52 +274,3 @@ func TestGetProjectInventoryManyDistinctProjects(t *testing.T) {
assert.Equal(t, projectCount, inv.TotalSessions)
assert.Len(t, inv.Projects, projectCount)
}
func TestProjectInventorySingleAggregationPass(t *testing.T) {
t.Run("aggregation query is a single sessions scan", func(t *testing.T) {
d := testDB(t)
rows, err := d.getReader().Query(
"EXPLAIN QUERY PLAN " + projectInventoryAggregateQuery(),
)
require.NoError(t, err)
defer rows.Close()
details := explainQueryPlanDetails(t, rows)
sessionsScans := 0
for _, detail := range details {
if strings.Contains(detail, "sessions") {
sessionsScans++
}
}
assert.Equal(t, 1, sessionsScans,
"aggregation must scan sessions exactly once, got plan: %s",
strings.Join(details, "; "))
})
t.Run("candidate query uses the sessions machine index", func(t *testing.T) {
d := testDB(t)
query, args := projectInventoryCandidateQuery([]string{"ws"})
rows, err := d.getReader().Query("EXPLAIN QUERY PLAN "+query, args...)
require.NoError(t, err)
defer rows.Close()
details := explainQueryPlanDetails(t, rows)
assert.Contains(t, strings.Join(details, "\n"), "idx_sessions_machine",
"candidate fetch must use the sessions machine index, not a full scan")
})
}
// explainQueryPlanDetails scans the detail column of an EXPLAIN QUERY PLAN
// result set.
func explainQueryPlanDetails(t *testing.T, rows *sql.Rows) []string {
t.Helper()
var details []string
for rows.Next() {
var id, parent, notused int
var detail string
require.NoError(t, rows.Scan(&id, &parent, &notused, &detail))
details = append(details, detail)
}
require.NoError(t, rows.Err())
return details
}
-91
View File
@@ -1,11 +1,5 @@
package db
import (
"context"
"fmt"
"strings"
)
// ProjectRule is one worktree mapping rule plus its provenance and current
// governed-session count. Rule identity across archives is
// (SourceArchiveID, Machine, PathPrefix), not the embedded
@@ -27,88 +21,3 @@ type ProjectRules struct {
Machines []string `json:"machines"`
Rules []ProjectRule `json:"rules"`
}
// ListProjectRules lists every worktree mapping rule for machine, enabled
// and disabled, each annotated with its current governed-session count.
// Disabled rules always report zero governed sessions since only enabled
// rules enter the evaluator. The machine list retains the typeahead
// contract of worktreeMappingsResponse.Machines: every machine with a live
// session, unioned with every machine that has a stored mapping, regardless
// of the machine argument.
func (db *DB) ListProjectRules(ctx context.Context, machine string) (ProjectRules, error) {
if ctx == nil {
ctx = context.Background()
}
machine = strings.TrimSpace(machine)
machines, err := db.ListWorktreeProjectMappingMachines(ctx)
if err != nil {
return ProjectRules{}, fmt.Errorf("listing project rule machines: %w", err)
}
mappings, err := db.ListWorktreeProjectMappings(ctx, machine)
if err != nil {
return ProjectRules{}, fmt.Errorf("listing project rules: %w", err)
}
archiveID, err := db.GetArchiveID(ctx)
if err != nil {
return ProjectRules{}, fmt.Errorf("resolving archive id for project rules: %w", err)
}
sessionsByRule, err := db.projectRulesGovernedCounts(ctx, archiveID, machine, mappings)
if err != nil {
return ProjectRules{}, err
}
rules := make([]ProjectRule, len(mappings))
for i, m := range mappings {
rules[i] = ProjectRule{
WorktreeProjectMapping: m,
SourceArchiveID: archiveID,
GovernedSessions: sessionsByRule[GovernedRuleKey{
SourceArchiveID: archiveID,
Machine: m.Machine,
PathPrefix: m.PathPrefix,
}],
}
}
return ProjectRules{
Machine: machine,
Machines: machines,
Rules: rules,
}, nil
}
// projectRulesGovernedCounts evaluates the machine's enabled rules against
// its visible sessions and returns the resulting per-rule governed counts.
// It reuses projectInventoryCandidateRows (project_inventory.go) for the
// candidate-row assembly rather than duplicating that query.
func (db *DB) projectRulesGovernedCounts(
ctx context.Context, archiveID, machine string, mappings []WorktreeProjectMapping,
) (map[GovernedRuleKey]int, error) {
hasEnabled := false
for _, m := range mappings {
if m.Enabled {
hasEnabled = true
break
}
}
if !hasEnabled {
return nil, nil
}
candidates, err := db.projectInventoryCandidateRows(
ctx, archiveID, map[string]struct{}{machine: {}},
)
if err != nil {
return nil, err
}
eval := EvaluateGovernedSessions(
[]ArchiveMappings{{SourceArchiveID: archiveID, Mappings: mappings}},
candidates,
)
return eval.SessionsByRule, nil
}
+17 -1
View File
@@ -43,7 +43,15 @@ type QueryDialect struct {
castCursor func(string, valueKind) string
// emptyStringIsNull is true for backends (SQLite) that store unset
// timestamps as empty strings rather than SQL NULL.
emptyStringIsNull bool
emptyStringIsNull bool
// portableEmptyTimestamp uses a CASE/CAST expression accepted by all Bun
// backends so shared queries can read shipped SQLite empty strings while
// retaining native timestamp values on PostgreSQL and DuckDB.
portableEmptyTimestamp bool
// timestampOrderExpr renders one timestamp sort operand. SQLite uses
// julianday so ordering and cursor predicates compare instants rather than
// the archive's mixed textual representations.
timestampOrderExpr func(string) string
terminationExpr string
terminationKind timestampKind
caseInsensitiveLike string
@@ -325,6 +333,12 @@ func duckCastCursor(ph string, kind valueKind) string {
// SQLite stores empty strings for missing timestamps; other backends use real
// NULLs, so the column reference passes through unchanged.
func (d QueryDialect) timestampExpr(col string) string {
if d.timestampOrderExpr != nil {
return d.timestampOrderExpr(col)
}
if d.portableEmptyTimestamp {
return "CASE WHEN CAST(" + col + " AS VARCHAR) = '' THEN NULL ELSE " + col + " END"
}
if d.emptyStringIsNull {
return "NULLIF(" + col + ", '')"
}
@@ -873,6 +887,8 @@ func terminationPredicate(
func (b *QueryBuilder) terminationParam(t time.Time) string {
switch b.dialect.terminationKind {
case timestampText:
return b.dialect.activityParam(b.Add(t.Format(time.RFC3339Nano)))
case timestampUnixSeconds:
return b.Add(t.Unix())
case timestampCast:
+10
View File
@@ -11,6 +11,8 @@ import (
_ "github.com/mattn/go-sqlite3"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
"github.com/uptrace/bun"
"github.com/uptrace/bun/dialect/sqlitedialect"
"go.kenn.io/agentsview/internal/money"
)
@@ -303,6 +305,10 @@ func TestReadOnlySchemaCompatibilityRejectsMissingReadColumn(t *testing.T) {
{"pg sync state", "pg_sync_state", "value"},
{"model pricing", "model_pricing", "updated_at"},
{"pricing band", "model_pricing_bands", "input_microdollars_per_mtok"},
{"source archive", "source_archives", "source_archive_salt"},
{"source identity", "source_project_identity_observations", "key"},
{"source snapshot", "source_session_project_identity_snapshots", "key"},
{"source worktree mapping", "source_worktree_project_mappings", "updated_at"},
{"secret finding", "secret_findings", "rules_version"},
{"recall entry", "recall_entries", "uncertainty"},
{"recall evidence", "recall_evidence", "snippet"},
@@ -359,6 +365,8 @@ func TestReadOnlyRequiredSchemaDerivedFromSchemaDDL(t *testing.T) {
t.Cleanup(func() { require.NoError(t, conn.Close()) })
_, err = conn.Exec(schemaSQL)
require.NoError(t, err)
store := bun.NewDB(conn, sqlitedialect.New())
require.NoError(t, CreateCommonSchema(t.Context(), store))
want := make(map[string][]string, len(readOnlyRequiredTables))
for _, table := range readOnlyRequiredTables {
@@ -397,6 +405,8 @@ func openReadOnlySchemaProbe(t *testing.T) *sql.DB {
t.Cleanup(func() { require.NoError(t, conn.Close()) })
_, err = conn.Exec(schemaSQL)
require.NoError(t, err)
store := bun.NewDB(conn, sqlitedialect.New())
require.NoError(t, CreateCommonSchema(t.Context(), store))
return conn
}
+11 -96
View File
@@ -171,36 +171,6 @@ func scanRecallEvidenceRow(rs rowScanner) (RecallEvidence, error) {
return e, err
}
func (db *DB) InsertRecallEntry(m RecallEntry) (string, error) {
if err := normalizeRecallEntryReviewState(&m); err != nil {
return "", err
}
db.mu.Lock()
defer db.mu.Unlock()
if m.ID == "" {
return "", fmt.Errorf("recall entry id is required")
}
if m.Status == "" {
m.Status = corerecall.StatusAccepted
}
tx, err := db.getWriter().Begin()
if err != nil {
return "", fmt.Errorf("begin recall insert: %w", err)
}
defer func() { _ = tx.Rollback() }()
if err := insertRecallEntryTx(tx, m); err != nil {
return "", err
}
if err := tx.Commit(); err != nil {
return "", fmt.Errorf("commit recall insert: %w", err)
}
return m.ID, nil
}
// CopyRecallEntriesFrom copies entries and their evidence from a source database
// into this database. A full resync rebuilds the DB from source files, which
// never contain entries, so without this copy every accepted entry is
@@ -558,7 +528,7 @@ func supersedeRecallEntryTx(
return err
}
if err := insertRecallEntryTx(tx, replacement); err != nil {
if err := insertRecallEntryTx(ctx, tx, replacement); err != nil {
return err
}
result, err := tx.ExecContext(ctx, `
@@ -606,14 +576,20 @@ func requireActiveRecallSupersessionTarget(
return nil
}
func insertRecallEntryTx(tx *sql.Tx, m RecallEntry) error {
type recallExecer interface {
ExecContext(context.Context, string, ...any) (sql.Result, error)
}
func insertRecallEntryTx(
ctx context.Context, tx recallExecer, m RecallEntry,
) error {
if err := normalizeRecallEntryReviewState(&m); err != nil {
return err
}
if err := validateRecallEvidenceOwnership(m); err != nil {
return err
}
_, err := tx.Exec(`
_, err := tx.ExecContext(ctx, `
INSERT INTO recall_entries (
id, type, scope, status, review_state, title, body, trigger,
confidence, uncertainty, project, cwd, git_branch, agent,
@@ -632,7 +608,7 @@ func insertRecallEntryTx(tx *sql.Tx, m RecallEntry) error {
}
for _, e := range m.Evidence {
_, err = tx.Exec(`
_, err = tx.ExecContext(ctx, `
INSERT INTO recall_evidence (
entry_id, session_id, message_start_ordinal,
message_end_ordinal, message_start_source_uuid,
@@ -678,67 +654,6 @@ func normalizeRecallEntryReviewState(m *RecallEntry) error {
return nil
}
func (db *DB) GetRecallEntry(ctx context.Context, id string) (*RecallEntry, error) {
row := db.getReader().QueryRowContext(
ctx,
"SELECT "+recallBaseCols+" FROM recall_entries WHERE id = ?",
id,
)
m, err := scanRecallEntryRow(row)
if err == sql.ErrNoRows {
return nil, nil
}
if err != nil {
return nil, fmt.Errorf("getting recall entry %s: %w", id, err)
}
evidence, err := db.listRecallEvidence(ctx, []string{id})
if err != nil {
return nil, err
}
m.Evidence = evidence[id]
return &m, nil
}
func (db *DB) ListRecallEntries(
ctx context.Context, q RecallQuery,
) ([]RecallEntry, error) {
if err := ValidateRecallQuery(q); err != nil {
return nil, err
}
q = NormalizeRecallQuery(q)
where, args := buildRecallEntryWhere(q, false)
limit := recallLimit(q.Limit)
if q.ProbeNext {
limit++
}
query := "SELECT " + recallBaseCols +
" FROM recall_entries WHERE " + where +
" ORDER BY updated_at DESC, id ASC LIMIT ?"
args = append(args, limit)
rows, err := db.getReader().QueryContext(ctx, query, args...)
if err != nil {
return nil, fmt.Errorf("querying entries: %w", err)
}
defer rows.Close()
entries, err := scanRecallEntryRows(rows)
if err != nil {
return nil, err
}
if len(entries) == 0 {
return nil, nil
}
evidence, err := db.listRecallEvidence(ctx, recallIDs(entries))
if err != nil {
return nil, err
}
for i := range entries {
entries[i].Evidence = evidence[entries[i].ID]
}
return entries, nil
}
// ListServedRecallSourceRuns returns the extraction/import source runs that
// currently contribute entries to the accepted Recall corpus.
func (db *DB) ListServedRecallSourceRuns(ctx context.Context) ([]string, error) {
@@ -1131,7 +1046,7 @@ func (db *DB) listRecallEvidenceLikeCandidates(
return scanRecallEntryRowsWithEvidence(ctx, db, rows)
}
func (db *DB) QueryRecallEntries(
func (db *DB) queryRecallEntries(
ctx context.Context, q RecallQuery,
) (RecallPage, error) {
if err := ValidateRecallQuery(q); err != nil {
+6 -1
View File
@@ -66,7 +66,7 @@ type EvalTrajectoryIngestResult struct {
// creates a distinct eval corpus instead of silently retaining stale rows. It
// mirrors the /import write path — a placeholder session satisfies the
// source_session_id FK, then each chunk is inserted only if absent.
func (db *DB) IngestEvalTrajectory(
func (db *DB) ingestEvalTrajectory(
ctx context.Context, in EvalTrajectoryIngest,
) (EvalTrajectoryIngestResult, error) {
in = normalizeEvalTrajectoryIngest(in)
@@ -290,6 +290,11 @@ func newEvalTrajectorySession(
func (db *DB) ingestEvalTrajectoryChunks(
ctx context.Context, session Session, entries []RecallEntry,
) (int, error) {
identity, err := db.localArchiveIdentity(ctx)
if err != nil {
return 0, err
}
stampSessionArchiveIdentity(&session, identity)
db.mu.Lock()
defer db.mu.Unlock()
tx, err := db.getWriter().BeginTx(ctx, nil)
+15 -3
View File
@@ -1397,7 +1397,7 @@ func insertExtractedRecallEntriesTx(
"checking extracted entry %s: %w", entry.ID, err,
)
}
if err := insertRecallEntryTx(tx, entry); err != nil {
if err := insertRecallEntryTx(ctx, tx, entry); err != nil {
return 0, err
}
inserted++
@@ -1563,10 +1563,10 @@ func verifyExtractSessionGuardTx(
case !nullableStringEqual(revision, u.TranscriptRevision):
return extractDriftErrorf(
"session %s transcript revision changed", u.SessionID)
case !nullableStringEqual(localModified, u.LocalModifiedAt):
case !nullableTimestampEqual(localModified, u.LocalModifiedAt):
return extractDriftErrorf(
"session %s was written to during distillation", u.SessionID)
case !nullableStringEqual(endedAt, u.EndedAt):
case !nullableTimestampEqual(endedAt, u.EndedAt):
// A bare session-row update can reopen or re-date a session
// without moving any other guarded field; eligibility treats
// ended_at as state, so the commit must too.
@@ -1596,6 +1596,18 @@ func nullableStringEqual(stored sql.NullString, expected *string) bool {
return stored.String == *expected
}
func nullableTimestampEqual(stored sql.NullString, expected *string) bool {
if nullableStringEqual(stored, expected) {
return true
}
if !stored.Valid || expected == nil {
return false
}
storedTime, storedOK := ParseStoredTimestamp(stored.String)
expectedTime, expectedOK := ParseStoredTimestamp(*expected)
return storedOK && expectedOK && storedTime.Equal(expectedTime)
}
// bindExtractedEvidenceTx stamps host-derived provenance onto each entry's
// evidence rows: the content digest and stable endpoint UUIDs the evidence
// reconciler later re-verifies. Ranges are bound once and shared, since every
+13 -11
View File
@@ -76,15 +76,7 @@ type probeRecallEvidence struct {
Snippets []string `json:"snippets"`
}
func (db *DB) ImportAcceptedRecallEntriesJSONL(
ctx context.Context, r io.Reader,
) (RecallImportResult, error) {
return db.ImportAcceptedRecallEntriesJSONLWithOptions(
ctx, r, RecallImportOptions{},
)
}
func (db *DB) ImportAcceptedRecallEntriesJSONLWithOptions(
func (db *DB) importAcceptedRecallEntriesJSONLWithOptions(
ctx context.Context, r io.Reader, opts RecallImportOptions,
) (RecallImportResult, error) {
var result RecallImportResult
@@ -384,6 +376,14 @@ func (db *DB) importAcceptedRecallEntry(
if recall.Status == "" {
recall.Status = corerecall.StatusAccepted
}
var identity ArchiveIdentity
if !opts.RequireExistingSessions {
var err error
identity, err = db.localArchiveIdentity(ctx)
if err != nil {
return false, err
}
}
db.mu.Lock()
defer db.mu.Unlock()
@@ -430,7 +430,7 @@ func (db *DB) importAcceptedRecallEntry(
if !opts.RequireExistingSessions {
recall.ProvenanceOK = false
if err := ensureRecallImportSessionTx(ctx, tx, item); err != nil {
if err := ensureRecallImportSessionTx(ctx, tx, item, identity); err != nil {
return false, fmt.Errorf("preparing source session: %w", err)
}
}
@@ -447,7 +447,7 @@ func (db *DB) importAcceptedRecallEntry(
); err != nil {
return false, err
}
} else if err := insertRecallEntryTx(tx, recall); err != nil {
} else if err := insertRecallEntryTx(ctx, tx, recall); err != nil {
return false, err
}
@@ -733,8 +733,10 @@ func ensureRecallImportSessionTx(
ctx context.Context,
tx *sql.Tx,
m probeAcceptedRecallEntry,
identity ArchiveIdentity,
) error {
session := recallImportPlaceholderSession(m)
stampSessionArchiveIdentity(&session, identity)
if err := validateRecallImportPlaceholderSessionStateWithQueryer(
ctx, tx, session.ID,
); err != nil {
+1 -79
View File
@@ -47,87 +47,9 @@ type RecallQueryExposure struct {
Packed bool `json:"packed"`
}
// RecordRecallQueryEvent inserts an event and all exposures atomically. An
// empty query ID receives a cryptographically random UUID.
func (db *DB) RecordRecallQueryEvent(
ctx context.Context,
event RecallQueryEvent,
) (string, error) {
if err := db.requireWritable(); err != nil {
return "", err
}
if ctx == nil {
ctx = context.Background()
}
event.QueryID = strings.TrimSpace(event.QueryID)
event.Surface = strings.TrimSpace(event.Surface)
event.ScorePolicyVersion = strings.TrimSpace(event.ScorePolicyVersion)
if event.Surface == "" {
return "", fmt.Errorf("recall query surface is required")
}
if event.FiltersJSON == "" {
event.FiltersJSON = "{}"
}
if event.ScorePolicyVersion == "" {
event.ScorePolicyVersion = RecallLexicalScorePolicyVersion
}
if event.QueryID == "" {
id, err := newUUIDv4()
if err != nil {
return "", fmt.Errorf("generating recall query id: %w", err)
}
event.QueryID = id
}
db.mu.Lock()
defer db.mu.Unlock()
tx, err := db.getWriter().BeginTx(ctx, nil)
if err != nil {
return "", fmt.Errorf("beginning recall query event: %w", err)
}
defer func() { _ = tx.Rollback() }()
if _, err := tx.ExecContext(ctx, `
INSERT INTO recall_query_events (
id, query_text, surface, filters_json, trusted_only,
score_policy_version, result_count, packed_count,
top_score, miss_reason
) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?)`,
event.QueryID,
event.Query,
event.Surface,
event.FiltersJSON,
event.TrustedOnly,
event.ScorePolicyVersion,
event.ResultCount,
event.PackedCount,
event.TopScore,
event.MissReason,
); err != nil {
return "", fmt.Errorf("inserting recall query event: %w", err)
}
for start := 0; start < len(event.Exposures); start += recallExposureInsertBatchSize {
end := min(start+recallExposureInsertBatchSize, len(event.Exposures))
batch := event.Exposures[start:end]
if err := insertRecallQueryExposureBatch(
ctx, tx, event.QueryID, batch,
); err != nil {
return "", fmt.Errorf(
"inserting recall query exposure ranks %d through %d: %w",
batch[0].Rank,
batch[len(batch)-1].Rank,
err,
)
}
}
if err := tx.Commit(); err != nil {
return "", fmt.Errorf("committing recall query event: %w", err)
}
return event.QueryID, nil
}
func insertRecallQueryExposureBatch(
ctx context.Context,
tx *sql.Tx,
tx recallExecer,
queryID string,
exposures []RecallQueryExposure,
) error {
+3 -70
View File
@@ -1,7 +1,6 @@
package db
import (
"context"
"database/sql"
"fmt"
"strings"
@@ -63,75 +62,9 @@ func NormalizeRecentEditsParams(p RecentEditsParams) RecentEditsParams {
// RecentEdits returns files ordered by most-recent edit across all sessions,
// grouped by (project, file_path), with up to MaxEditsPerFile recent edits
// inlined per file. Trashed sessions are excluded.
func (db *DB) RecentEdits(
ctx context.Context, p RecentEditsParams,
) (RecentEditsResult, error) {
p = NormalizeRecentEditsParams(p)
projectClause := ""
if p.Project != "" {
projectClause = "AND s.project = ?"
}
searchClause := ""
if p.Search != "" {
searchClause = `AND tc.file_path LIKE ? ESCAPE '\'`
}
query := `
WITH ranked AS (
SELECT s.project AS project, tc.file_path AS file_path,
tc.session_id AS session_id, tc.tool_name AS tool_name,
tc.category AS category, tc.tool_use_id AS tool_use_id,
tc.call_index AS call_index, m.ordinal AS ordinal,
m.timestamp AS timestamp,
ROW_NUMBER() OVER (
PARTITION BY s.project, tc.file_path
ORDER BY m.timestamp DESC NULLS LAST, tc.session_id DESC,
m.ordinal DESC, tc.call_index DESC) AS rn,
COUNT(*) OVER (PARTITION BY s.project, tc.file_path) AS edit_count
FROM tool_calls tc
JOIN messages m ON m.id = tc.message_id
JOIN sessions s ON s.id = tc.session_id
WHERE tc.category IN ('Edit','Write')
AND tc.file_path IS NOT NULL AND tc.file_path <> ''
AND s.deleted_at IS NULL
` + projectClause + `
` + searchClause + `
),
file_page AS (
SELECT project, file_path, edit_count,
timestamp AS last_edited_at, session_id AS last_session_id,
ordinal AS last_ordinal, call_index AS last_call_index
FROM ranked
WHERE rn = 1
ORDER BY last_edited_at DESC NULLS LAST, last_session_id DESC,
last_ordinal DESC, last_call_index DESC, file_path DESC
LIMIT ? OFFSET ?
)
SELECT fp.project, fp.file_path, fp.edit_count, fp.last_edited_at,
fp.last_session_id, r.session_id, r.ordinal, r.tool_use_id,
r.call_index, r.tool_name, r.category, r.timestamp
FROM file_page fp
JOIN ranked r ON r.project = fp.project AND r.file_path = fp.file_path
WHERE r.rn <= ?
ORDER BY fp.last_edited_at DESC NULLS LAST, fp.last_session_id DESC,
fp.last_ordinal DESC, fp.last_call_index DESC, fp.file_path DESC,
r.rn`
// Placeholders bind in text order: project (CTE), search (CTE), LIMIT,
// OFFSET, then K.
qArgs := []any{}
if p.Project != "" {
qArgs = append(qArgs, p.Project)
}
if p.Search != "" {
qArgs = append(qArgs, "%"+EscapeLikePattern(p.Search)+"%")
}
qArgs = append(qArgs, p.Limit+1, p.Offset, p.MaxEditsPerFile)
rows, err := db.getReader().QueryContext(ctx, query, qArgs...)
if err != nil {
return RecentEditsResult{}, fmt.Errorf("querying recent edits: %w", err)
}
defer rows.Close()
return ScanRecentEdits(rows, p)
}
// Placeholders bind in text order: project (CTE), search (CTE), LIMIT,
// OFFSET, then K.
// ScanRecentEdits groups the flat (file × edit) result rows into files,
// preserving row order, and applies has_more and per-file truncation. Shared
+36 -42
View File
@@ -9,6 +9,7 @@ import (
"sort"
"time"
"github.com/uptrace/bun"
"go.kenn.io/agentsview/internal/activity"
"go.kenn.io/agentsview/internal/export"
"go.kenn.io/agentsview/internal/money"
@@ -45,49 +46,42 @@ func (db *DB) ExportReportingDay(
return export.ReportingDay{}, err
}
tx, err := db.getReader().BeginTx(ctx, &sql.TxOptions{ReadOnly: true})
if err != nil {
return export.ReportingDay{}, fmt.Errorf("begin reporting snapshot: %w", err)
}
defer func() {
_ = tx.Rollback()
}()
var day export.ReportingDay
err = db.consistentView(ctx, func(store bun.IDB) error {
// Establish the snapshot before constructing the in-memory document. The
// following implementation stages load all source rows through this tx.
var snapshotMarker int
if err := store.QueryRowContext(
ctx, "SELECT COUNT(*) FROM archive_metadata",
).Scan(&snapshotMarker); err != nil {
return fmt.Errorf("establish reporting snapshot: %w", err)
}
if opts.afterSnapshot != nil {
opts.afterSnapshot()
}
// Establish the snapshot before constructing the in-memory document. The
// following implementation stages load all source rows through this tx.
var snapshotMarker int
if err := tx.QueryRowContext(
ctx, "SELECT COUNT(*) FROM archive_metadata",
).Scan(&snapshotMarker); err != nil {
return export.ReportingDay{}, fmt.Errorf("establish reporting snapshot: %w", err)
}
if opts.afterSnapshot != nil {
opts.afterSnapshot()
}
hours, err := db.reportingHoursFromSnapshot(
ctx, tx, date, hourCount, schemaVersion,
)
if err != nil {
return export.ReportingDay{}, err
}
day, _, err := export.FinalizeReportingDay(export.ReportingDay{
SchemaVersion: schemaVersion,
Date: date.Format("2006-01-02"),
Complete: complete,
Hours: hours,
hours, err := db.reportingHoursFromSnapshot(
ctx, store, date, hourCount, schemaVersion,
)
if err != nil {
return err
}
day, _, err = export.FinalizeReportingDay(export.ReportingDay{
SchemaVersion: schemaVersion,
Date: date.Format("2006-01-02"),
Complete: complete,
Hours: hours,
})
if err != nil {
return fmt.Errorf("finalize reporting date: %w", err)
}
return nil
})
if err != nil {
return export.ReportingDay{}, fmt.Errorf("finalize reporting date: %w", err)
}
if err := tx.Commit(); err != nil {
return export.ReportingDay{}, fmt.Errorf("commit reporting snapshot: %w", err)
}
return day, nil
return day, err
}
func (db *DB) reportingHoursFromSnapshot(
ctx context.Context, tx *sql.Tx, date time.Time, hourCount, schemaVersion int,
ctx context.Context, tx bun.IDB, date time.Time, hourCount, schemaVersion int,
) ([]export.ReportingHour, error) {
hours := make([]export.ReportingHour, hourCount)
if hourCount == 0 {
@@ -234,7 +228,7 @@ func (db *DB) reportingHoursFromSnapshot(
func (db *DB) reportingUsageSessionsFrom(
ctx context.Context,
tx *sql.Tx,
tx bun.IDB,
lowerBound, upperBound string,
) ([]activity.SessionMeta, []string, error) {
usageRows := dailyUsageRowsSQLWithWhere(
@@ -307,7 +301,7 @@ func (db *DB) reportingUsageSessionsFrom(
func (db *DB) reportingStandaloneUsageCandidatesFrom(
ctx context.Context,
tx *sql.Tx,
tx bun.IDB,
query activity.Query,
) ([]activity.UsageRow, error) {
var tableExists int
@@ -647,7 +641,7 @@ func allocateReportingUsageCosts(
func (db *DB) reportingProjectIdentityMapFrom(
ctx context.Context,
tx *sql.Tx,
tx bun.IDB,
labels []string,
) (map[string]export.ProjectMapEntry, error) {
if len(labels) == 0 {
@@ -685,7 +679,7 @@ func (db *DB) reportingProjectIdentityMapFrom(
}
func reportingSessionCreatedAtFrom(
ctx context.Context, tx *sql.Tx, ids []string,
ctx context.Context, tx bun.IDB, ids []string,
) (map[string]time.Time, error) {
out := make(map[string]time.Time, len(ids))
if len(ids) == 0 {
+16 -187
View File
@@ -87,6 +87,8 @@ CREATE TABLE IF NOT EXISTS sessions (
termination_status TEXT,
secret_leak_count INTEGER NOT NULL DEFAULT 0,
secrets_rules_version TEXT NOT NULL DEFAULT '',
source_archive_id TEXT NOT NULL DEFAULT '',
source_database_generation TEXT NOT NULL DEFAULT '',
sync_marker TEXT
);
@@ -320,7 +322,8 @@ CREATE TABLE IF NOT EXISTS tool_calls (
result_content TEXT,
subagent_session_id TEXT,
file_path TEXT,
call_index INTEGER
call_index INTEGER,
message_ordinal INTEGER
);
CREATE INDEX IF NOT EXISTS idx_tool_calls_session
@@ -693,6 +696,7 @@ CREATE TABLE IF NOT EXISTS pinned_messages (
session_id TEXT NOT NULL REFERENCES sessions(id) ON DELETE CASCADE,
message_id INTEGER NOT NULL REFERENCES messages(id) ON DELETE CASCADE,
ordinal INTEGER NOT NULL,
source_uuid TEXT NOT NULL DEFAULT '',
note TEXT,
created_at TEXT NOT NULL DEFAULT (strftime('%Y-%m-%dT%H:%M:%fZ','now')),
UNIQUE(session_id, message_id)
@@ -869,130 +873,10 @@ CREATE TABLE IF NOT EXISTS session_project_identity_snapshot_changes (
CREATE INDEX IF NOT EXISTS idx_session_project_identity_snapshot_changes_revision
ON session_project_identity_snapshot_changes(revision);
DROP TRIGGER IF EXISTS trg_project_identity_observations_revision_insert;
DROP TRIGGER IF EXISTS trg_project_identity_observations_revision_update;
DROP TRIGGER IF EXISTS trg_project_identity_observations_revision_delete;
DROP TRIGGER IF EXISTS trg_session_project_identity_snapshots_revision_insert;
DROP TRIGGER IF EXISTS trg_session_project_identity_snapshots_revision_update;
DROP TRIGGER IF EXISTS trg_session_project_identity_snapshots_revision_delete;
CREATE TRIGGER IF NOT EXISTS trg_project_identity_observations_revision_insert
AFTER INSERT ON project_identity_observations BEGIN
INSERT INTO archive_metadata (key, value) VALUES ('project_identity_publication_revision', '1')
ON CONFLICT(key) DO UPDATE SET value = CAST(CAST(value AS INTEGER) + 1 AS TEXT),
updated_at = strftime('%Y-%m-%dT%H:%M:%fZ','now');
INSERT INTO project_identity_observation_changes (
project, machine, root_path, git_remote, revision, deleted
) VALUES (
NEW.project, NEW.machine, NEW.root_path, NEW.git_remote,
(SELECT CAST(value AS INTEGER) FROM archive_metadata
WHERE key = 'project_identity_publication_revision'), 0
) ON CONFLICT(project, machine, root_path, git_remote) DO UPDATE SET
revision = excluded.revision, deleted = 0;
END;
CREATE TRIGGER IF NOT EXISTS trg_project_identity_observations_revision_update
AFTER UPDATE ON project_identity_observations BEGIN
INSERT INTO archive_metadata (key, value) VALUES ('project_identity_publication_revision', '1')
ON CONFLICT(key) DO UPDATE SET value = CAST(CAST(value AS INTEGER) + 1 AS TEXT),
updated_at = strftime('%Y-%m-%dT%H:%M:%fZ','now');
INSERT INTO project_identity_observation_changes (
project, machine, root_path, git_remote, revision, deleted
) VALUES (
OLD.project, OLD.machine, OLD.root_path, OLD.git_remote,
(SELECT CAST(value AS INTEGER) FROM archive_metadata
WHERE key = 'project_identity_publication_revision'), 1
) ON CONFLICT(project, machine, root_path, git_remote) DO UPDATE SET
revision = excluded.revision, deleted = 1;
INSERT INTO project_identity_observation_changes (
project, machine, root_path, git_remote, revision, deleted
) VALUES (
NEW.project, NEW.machine, NEW.root_path, NEW.git_remote,
(SELECT CAST(value AS INTEGER) FROM archive_metadata
WHERE key = 'project_identity_publication_revision'), 0
) ON CONFLICT(project, machine, root_path, git_remote) DO UPDATE SET
revision = excluded.revision, deleted = 0;
END;
CREATE TRIGGER IF NOT EXISTS trg_project_identity_observations_revision_delete
AFTER DELETE ON project_identity_observations BEGIN
INSERT INTO archive_metadata (key, value) VALUES ('project_identity_publication_revision', '1')
ON CONFLICT(key) DO UPDATE SET value = CAST(CAST(value AS INTEGER) + 1 AS TEXT),
updated_at = strftime('%Y-%m-%dT%H:%M:%fZ','now');
INSERT INTO project_identity_observation_changes (
project, machine, root_path, git_remote, revision, deleted
) VALUES (
OLD.project, OLD.machine, OLD.root_path, OLD.git_remote,
(SELECT CAST(value AS INTEGER) FROM archive_metadata
WHERE key = 'project_identity_publication_revision'), 1
) ON CONFLICT(project, machine, root_path, git_remote) DO UPDATE SET
revision = excluded.revision, deleted = 1;
END;
CREATE TRIGGER IF NOT EXISTS trg_session_project_identity_snapshots_revision_insert
AFTER INSERT ON session_project_identity_snapshots BEGIN
INSERT INTO archive_metadata (key, value) VALUES ('project_identity_publication_revision', '1')
ON CONFLICT(key) DO UPDATE SET value = CAST(CAST(value AS INTEGER) + 1 AS TEXT),
updated_at = strftime('%Y-%m-%dT%H:%M:%fZ','now');
INSERT INTO session_project_identity_snapshot_changes (
session_id, project, revision, deleted
) VALUES (
NEW.session_id, NEW.project,
(SELECT CAST(value AS INTEGER) FROM archive_metadata
WHERE key = 'project_identity_publication_revision'), 0
) ON CONFLICT(session_id, project) DO UPDATE SET
revision = excluded.revision, deleted = 0;
END;
CREATE TRIGGER IF NOT EXISTS trg_session_project_identity_snapshots_revision_update
AFTER UPDATE ON session_project_identity_snapshots BEGIN
INSERT INTO archive_metadata (key, value) VALUES ('project_identity_publication_revision', '1')
ON CONFLICT(key) DO UPDATE SET value = CAST(CAST(value AS INTEGER) + 1 AS TEXT),
updated_at = strftime('%Y-%m-%dT%H:%M:%fZ','now');
INSERT INTO session_project_identity_snapshot_changes (
session_id, project, revision, deleted
) VALUES (
OLD.session_id, OLD.project,
(SELECT CAST(value AS INTEGER) FROM archive_metadata
WHERE key = 'project_identity_publication_revision'), 1
) ON CONFLICT(session_id, project) DO UPDATE SET
revision = excluded.revision, deleted = 1;
INSERT INTO session_project_identity_snapshot_changes (
session_id, project, revision, deleted
) VALUES (
NEW.session_id, NEW.project,
(SELECT CAST(value AS INTEGER) FROM archive_metadata
WHERE key = 'project_identity_publication_revision'), 0
) ON CONFLICT(session_id, project) DO UPDATE SET
revision = excluded.revision, deleted = 0;
END;
CREATE TRIGGER IF NOT EXISTS trg_session_project_identity_snapshots_revision_delete
AFTER DELETE ON session_project_identity_snapshots BEGIN
INSERT INTO archive_metadata (key, value) VALUES ('project_identity_publication_revision', '1')
ON CONFLICT(key) DO UPDATE SET value = CAST(CAST(value AS INTEGER) + 1 AS TEXT),
updated_at = strftime('%Y-%m-%dT%H:%M:%fZ','now');
INSERT INTO session_project_identity_snapshot_changes (
session_id, project, revision, deleted
) VALUES (
OLD.session_id, OLD.project,
(SELECT CAST(value AS INTEGER) FROM archive_metadata
WHERE key = 'project_identity_publication_revision'), 1
) ON CONFLICT(session_id, project) DO UPDATE SET
revision = excluded.revision, deleted = 1;
END;
CREATE TRIGGER IF NOT EXISTS trg_sessions_create_project_identity_snapshot
AFTER INSERT ON sessions BEGIN
INSERT INTO session_project_identity_snapshots (
session_id, project, machine, root_path, worktree_relationship,
checkout_state, git_branch, remote_resolution, observed_at
) VALUES (
NEW.id, NEW.project, NEW.machine, NEW.cwd, 'unknown',
CASE WHEN NEW.git_branch != '' THEN 'branch' ELSE 'unknown' END,
NEW.git_branch, 'unknown', strftime('%Y-%m-%dT%H:%M:%fZ','now')
) ON CONFLICT(session_id) DO NOTHING;
END;
-- Identity journal tables remain migration history, but their legacy-table
-- triggers are not installed here. The atomic common-schema convergence owns
-- the canonical source-scoped trigger definitions and validates them on every
-- stamped reopen without issuing DDL.
-- sync_marker's index and trigger-maintenance DDL live in
-- syncMarkerSchemaSQL (internal/db/db.go), executed post-migration in
@@ -1066,68 +950,6 @@ CREATE TABLE IF NOT EXISTS worktree_project_mapping_changes (
CREATE INDEX IF NOT EXISTS idx_worktree_project_mapping_changes_revision
ON worktree_project_mapping_changes(revision);
DROP TRIGGER IF EXISTS trg_worktree_project_mappings_revision_insert;
DROP TRIGGER IF EXISTS trg_worktree_project_mappings_revision_update;
DROP TRIGGER IF EXISTS trg_worktree_project_mappings_revision_delete;
CREATE TRIGGER IF NOT EXISTS trg_worktree_project_mappings_revision_insert
AFTER INSERT ON worktree_project_mappings
BEGIN
INSERT INTO archive_metadata (key, value)
VALUES ('worktree_mapping_publication_revision', '1')
ON CONFLICT(key) DO UPDATE SET
value = CAST(CAST(value AS INTEGER) + 1 AS TEXT),
updated_at = strftime('%Y-%m-%dT%H:%M:%fZ','now');
INSERT INTO worktree_project_mapping_changes
(machine, path_prefix, revision, deleted)
VALUES (NEW.machine, NEW.path_prefix,
(SELECT CAST(value AS INTEGER) FROM archive_metadata
WHERE key = 'worktree_mapping_publication_revision'), 0)
ON CONFLICT(machine, path_prefix) DO UPDATE SET
revision = excluded.revision, deleted = 0;
END;
CREATE TRIGGER IF NOT EXISTS trg_worktree_project_mappings_revision_update
AFTER UPDATE ON worktree_project_mappings
BEGIN
INSERT INTO archive_metadata (key, value)
VALUES ('worktree_mapping_publication_revision', '1')
ON CONFLICT(key) DO UPDATE SET
value = CAST(CAST(value AS INTEGER) + 1 AS TEXT),
updated_at = strftime('%Y-%m-%dT%H:%M:%fZ','now');
INSERT INTO worktree_project_mapping_changes
(machine, path_prefix, revision, deleted)
VALUES (OLD.machine, OLD.path_prefix,
(SELECT CAST(value AS INTEGER) FROM archive_metadata
WHERE key = 'worktree_mapping_publication_revision'), 1)
ON CONFLICT(machine, path_prefix) DO UPDATE SET
revision = excluded.revision, deleted = 1;
INSERT INTO worktree_project_mapping_changes
(machine, path_prefix, revision, deleted)
VALUES (NEW.machine, NEW.path_prefix,
(SELECT CAST(value AS INTEGER) FROM archive_metadata
WHERE key = 'worktree_mapping_publication_revision'), 0)
ON CONFLICT(machine, path_prefix) DO UPDATE SET
revision = excluded.revision, deleted = 0;
END;
CREATE TRIGGER IF NOT EXISTS trg_worktree_project_mappings_revision_delete
AFTER DELETE ON worktree_project_mappings
BEGIN
INSERT INTO archive_metadata (key, value)
VALUES ('worktree_mapping_publication_revision', '1')
ON CONFLICT(key) DO UPDATE SET
value = CAST(CAST(value AS INTEGER) + 1 AS TEXT),
updated_at = strftime('%Y-%m-%dT%H:%M:%fZ','now');
INSERT INTO worktree_project_mapping_changes
(machine, path_prefix, revision, deleted)
VALUES (OLD.machine, OLD.path_prefix,
(SELECT CAST(value AS INTEGER) FROM archive_metadata
WHERE key = 'worktree_mapping_publication_revision'), 1)
ON CONFLICT(machine, path_prefix) DO UPDATE SET
revision = excluded.revision, deleted = 1;
END;
-- PG sync state: stores watermarks for push sync
CREATE TABLE IF NOT EXISTS pg_sync_state (
key TEXT PRIMARY KEY,
@@ -1172,6 +994,13 @@ CREATE TABLE IF NOT EXISTS model_pricing_bands (
PRIMARY KEY (model_pattern, above_input_tokens)
);
CREATE TABLE IF NOT EXISTS pricing_metadata (
key TEXT PRIMARY KEY,
value TEXT NOT NULL,
updated_at TEXT NOT NULL
DEFAULT (strftime('%Y-%m-%dT%H:%M:%fZ','now'))
);
-- Git aggregation TTL cache: memoizes `git log --numstat` and
-- `gh pr list` results per (repo, author, window) tuple so
-- repeated `agentsview stats` invocations don't re-shell out.
+14
View File
@@ -57,6 +57,13 @@ func (db *DB) WriteSessionBatch(
if len(writes) == 0 {
return result, nil
}
identity, err := db.localArchiveIdentity(context.Background())
if err != nil {
return result, err
}
for i := range writes {
stampSessionArchiveIdentity(&writes[i].Session, identity)
}
db.mu.Lock()
defer db.mu.Unlock()
@@ -139,6 +146,13 @@ func (db *DB) WriteSessionBatchAtomic(
if len(writes) == 0 {
return result, nil
}
identity, err := db.localArchiveIdentity(context.Background())
if err != nil {
return result, err
}
for i := range writes {
stampSessionArchiveIdentity(&writes[i].Session, identity)
}
db.mu.Lock()
defer db.mu.Unlock()
+28 -38
View File
@@ -15,6 +15,7 @@ import (
"strings"
"time"
"github.com/uptrace/bun"
"go.kenn.io/agentsview/internal/activity"
"go.kenn.io/agentsview/internal/export"
"go.kenn.io/agentsview/internal/money"
@@ -189,21 +190,13 @@ func (db *DB) ExportSessionSummaries(
if ctx == nil {
ctx = context.Background()
}
tx, err := db.getReader().BeginTx(ctx, &sql.TxOptions{ReadOnly: true})
if err != nil {
return SessionExportResult{}, fmt.Errorf(
"starting session export snapshot: %w", err)
}
defer func() { _ = tx.Rollback() }()
result, err := db.exportSessionSummariesTx(ctx, tx, opts, true)
if err != nil {
return SessionExportResult{}, err
}
if err := tx.Commit(); err != nil {
return SessionExportResult{}, fmt.Errorf(
"committing session export snapshot: %w", err)
}
return result, nil
var result SessionExportResult
err := db.consistentView(ctx, func(store bun.IDB) error {
var err error
result, err = db.exportSessionSummariesTx(ctx, store, opts, true)
return err
})
return result, err
}
// ExportAllSessionSummaries follows every page inside one read transaction so
@@ -222,36 +215,33 @@ func (db *DB) exportAllSessionSummaries(
if ctx == nil {
ctx = context.Background()
}
tx, err := db.getReader().BeginTx(ctx, &sql.TxOptions{ReadOnly: true})
if err != nil {
return nil, fmt.Errorf("starting complete session export snapshot: %w", err)
}
defer func() { _ = tx.Rollback() }()
pages := []SessionExportResult{}
for {
result, err := db.exportSessionSummariesTx(ctx, tx, opts, false)
if err != nil {
return nil, err
}
pages = append(pages, result)
if afterPage != nil {
if err := afterPage(len(pages)); err != nil {
return nil, err
err := db.consistentView(ctx, func(store bun.IDB) error {
for {
result, err := db.exportSessionSummariesTx(ctx, store, opts, false)
if err != nil {
return err
}
pages = append(pages, result)
if afterPage != nil {
if err := afterPage(len(pages)); err != nil {
return err
}
}
if result.NextCursor == "" {
return nil
}
opts.Cursor = result.NextCursor
}
if result.NextCursor == "" {
break
}
opts.Cursor = result.NextCursor
}
if err := tx.Commit(); err != nil {
return nil, fmt.Errorf("committing complete session export snapshot: %w", err)
})
if err != nil {
return nil, err
}
return pages, nil
}
func (db *DB) exportSessionSummariesTx(
ctx context.Context, tx *sql.Tx, opts SessionExportOptions,
ctx context.Context, tx bun.IDB, opts SessionExportOptions,
cursorIntegrity bool,
) (SessionExportResult, error) {
if opts.Limit <= 0 || opts.Limit > MaxSessionLimit {
@@ -732,7 +722,7 @@ LIMIT ?`
}
func (db *DB) attachSessionExportUsage(
ctx context.Context, q sessionExportQuerier, rows []SessionSummaryRow,
ctx context.Context, q bun.IDB, rows []SessionSummaryRow,
) (*export.PricingBlock, error) {
if len(rows) == 0 {
return nil, nil
+2 -2
View File
@@ -982,7 +982,7 @@ func TestSessionExportCursorPrefixUsesSameSnapshotAsPageQuery(t *testing.T) {
}
where, args := buildSessionExportFilter(SessionFilter{Project: "snapshot"})
tx, err := d.getReader().BeginTx(ctx, &sql.TxOptions{ReadOnly: true})
tx, err := d.bunReader.BeginTx(ctx, &sql.TxOptions{ReadOnly: true})
require.NoError(t, err, "begin read snapshot")
defer func() { require.NoError(t, tx.Rollback(), "rollback read snapshot") }()
@@ -1041,7 +1041,7 @@ func TestSessionExportUsageUsesPageReadSnapshot(t *testing.T) {
HasOutputTokens: true,
})
tx, err := d.getReader().BeginTx(ctx, &sql.TxOptions{ReadOnly: true})
tx, err := d.bunReader.BeginTx(ctx, &sql.TxOptions{ReadOnly: true})
require.NoError(t, err, "begin page snapshot")
defer func() { require.NoError(t, tx.Rollback(), "rollback page snapshot") }()
var messageCount int
+1 -1
View File
@@ -988,7 +988,7 @@ func (db *DB) computeCacheEconomics(
return nil
}
pricing, err := db.loadPricingMap(ctx)
pricing, err := db.LoadPricingMap(ctx)
if err != nil {
return fmt.Errorf("loading pricing: %w", err)
}
+67 -1281
View File
File diff suppressed because it is too large Load Diff
+4 -2
View File
@@ -364,11 +364,13 @@ func intValue(get func(*Session) int) func(*Session, SessionFilter) (string, boo
func recentExpr(b *QueryBuilder, _ SessionFilter) string {
return "COALESCE(" + b.dialect.timestampExpr("ended_at") + ", " +
b.dialect.timestampExpr("started_at") + ", created_at)"
b.dialect.timestampExpr("started_at") + ", " +
b.dialect.timestampExpr("created_at") + ")"
}
func startedExpr(b *QueryBuilder, _ SessionFilter) string {
return "COALESCE(" + b.dialect.timestampExpr("started_at") + ", created_at)"
return "COALESCE(" + b.dialect.timestampExpr("started_at") + ", " +
b.dialect.timestampExpr("created_at") + ")"
}
func plainExpr(col string) func(*QueryBuilder, SessionFilter) string {
+4 -4
View File
@@ -24,7 +24,7 @@ func TestOrderByClause_MultiKey(t *testing.T) {
b := NewQueryBuilder(SQLiteQueryDialect(), 0)
assert.Equal(t,
"ORDER BY message_count ASC, "+
"COALESCE(NULLIF(started_at, ''), created_at) DESC, id DESC",
"COALESCE(NULLIF(started_at, ''), NULLIF(created_at, '')) DESC, id DESC",
b.OrderByClause(rs, SessionFilter{}))
bpg := NewQueryBuilder(PostgresQueryDialect(), 0)
@@ -53,9 +53,9 @@ func TestCursorPredicate_MultiKey(t *testing.T) {
assert.Equal(t,
"((message_count > ?) OR "+
"(message_count = ? AND "+
"COALESCE(NULLIF(started_at, ''), created_at) < ?) OR "+
"COALESCE(NULLIF(started_at, ''), NULLIF(created_at, '')) < ?) OR "+
"(message_count = ? AND "+
"COALESCE(NULLIF(started_at, ''), created_at) = ? AND id < ?))",
"COALESCE(NULLIF(started_at, ''), NULLIF(created_at, '')) = ? AND id < ?))",
gotSQLite)
// Six bound params: one comparison at level 0, two at level 1, three at
// level 2 (the id tie-break being the last).
@@ -79,7 +79,7 @@ func TestCursorPredicate_SingleKeyRecent(t *testing.T) {
rs := resolvedFor(t, "recent:desc")
b := NewQueryBuilder(SQLiteQueryDialect(), 0)
got := b.CursorPredicate(rs, SessionFilter{}, []any{"2024-05-01T00:00:00Z"}, "sid")
activity := "COALESCE(NULLIF(ended_at, ''), NULLIF(started_at, ''), created_at)"
activity := "COALESCE(NULLIF(ended_at, ''), NULLIF(started_at, ''), NULLIF(created_at, ''))"
assert.Equal(t,
"(("+activity+" < ?) OR ("+activity+" = ? AND id < ?))",
got)
-111
View File
@@ -2,81 +2,10 @@ package db
import (
"context"
"database/sql"
"fmt"
"strings"
)
// StarSession marks a session as starred. Uses INSERT...SELECT
// with an EXISTS check so the operation is atomic and avoids FK
// errors if the session is concurrently deleted. Returns false
// if the session does not exist (idempotent for already-starred).
func (db *DB) StarSession(sessionID string) (bool, error) {
db.mu.Lock()
defer db.mu.Unlock()
w := db.getWriter()
res, err := w.Exec(`
INSERT OR IGNORE INTO starred_sessions (session_id)
SELECT ? WHERE EXISTS (SELECT 1 FROM sessions WHERE id = ?)`,
sessionID, sessionID)
if err != nil {
return false, fmt.Errorf("starring session %s: %w", sessionID, err)
}
n, _ := res.RowsAffected()
if n > 0 {
return true, nil // newly starred
}
// Zero rows: either already starred or session doesn't exist.
var exists int
err = w.QueryRow(
"SELECT 1 FROM sessions WHERE id = ?", sessionID,
).Scan(&exists)
if err == sql.ErrNoRows {
return false, nil // session doesn't exist
}
if err != nil {
return false, fmt.Errorf("checking session %s: %w", sessionID, err)
}
return true, nil // already starred
}
// UnstarSession removes a session's star.
func (db *DB) UnstarSession(sessionID string) error {
db.mu.Lock()
defer db.mu.Unlock()
_, err := db.getWriter().Exec(
"DELETE FROM starred_sessions WHERE session_id = ?",
sessionID,
)
if err != nil {
return fmt.Errorf("unstarring session %s: %w", sessionID, err)
}
return nil
}
// ListStarredSessionIDs returns all starred session IDs.
func (db *DB) ListStarredSessionIDs(
ctx context.Context,
) ([]string, error) {
rows, err := db.getReader().QueryContext(ctx,
"SELECT session_id FROM starred_sessions ORDER BY created_at DESC",
)
if err != nil {
return nil, fmt.Errorf("listing starred sessions: %w", err)
}
defer rows.Close()
var ids []string
for rows.Next() {
var id string
if err := rows.Scan(&id); err != nil {
return nil, fmt.Errorf("scanning starred session: %w", err)
}
ids = append(ids, id)
}
return ids, rows.Err()
}
// ListStarredSessionIDsForScope returns starred session IDs restricted to
// the given project scope (see BuildSessionFilterSQL's project/exclude
// semantics), sorted for deterministic output. Cost is bounded by the
@@ -139,43 +68,3 @@ func curationScopeWhere(alias string, projects, excludeProjects []string) (strin
}
return " WHERE " + strings.Join(where, " AND "), args
}
// BulkStarSessions stars multiple sessions in a single transaction.
// Used for migrating localStorage stars to the database.
func (db *DB) BulkStarSessions(sessionIDs []string) error {
if err := db.requireWritable(); err != nil {
return err
}
if len(sessionIDs) == 0 {
return nil
}
db.mu.Lock()
defer db.mu.Unlock()
tx, err := db.getWriter().Begin()
if err != nil {
return fmt.Errorf("beginning transaction: %w", err)
}
defer func() { _ = tx.Rollback() }()
// Use INSERT ... SELECT ... WHERE EXISTS so that stale IDs
// (sessions pruned or deleted from disk) are silently skipped
// instead of causing a foreign key violation that aborts the
// entire migration transaction.
stmt, err := tx.Prepare(`
INSERT OR IGNORE INTO starred_sessions (session_id)
SELECT ? WHERE EXISTS (SELECT 1 FROM sessions WHERE id = ?)`)
if err != nil {
return fmt.Errorf("preparing statement: %w", err)
}
defer stmt.Close()
for _, id := range sessionIDs {
if _, err := stmt.Exec(id, id); err != nil {
return fmt.Errorf("starring session %s: %w", id, err)
}
}
return tx.Commit()
}
-47
View File
@@ -171,50 +171,3 @@ func (db *DB) fileBackedSessionCount(
}
return count, nil
}
// GetStats returns database statistics, counting only root
// sessions with messages (matching the session list filter).
func (db *DB) GetStats(
ctx context.Context,
excludeOneShot, excludeAutomated bool,
) (Stats, error) {
filter := rootSessionFilter
if excludeOneShot {
if !excludeAutomated {
filter += " AND (user_message_count > 1 OR is_automated = 1)"
} else {
filter += " AND user_message_count > 1"
}
}
if excludeAutomated {
filter += " AND is_automated = 0"
}
query := fmt.Sprintf(`
SELECT
(SELECT COUNT(*) FROM sessions
WHERE %s),
(SELECT COALESCE(SUM(message_count), 0)
FROM sessions WHERE %s),
(SELECT COUNT(DISTINCT project) FROM sessions
WHERE %s),
(SELECT COUNT(DISTINCT machine) FROM sessions
WHERE %s),
(SELECT MIN(COALESCE(
NULLIF(started_at, ''), created_at
)) FROM sessions
WHERE %s)`,
filter, filter, filter, filter, filter)
var s Stats
err := db.getReader().QueryRowContext(ctx, query).Scan(
&s.SessionCount,
&s.MessageCount,
&s.ProjectCount,
&s.MachineCount,
&s.EarliestSession,
)
if err != nil {
return Stats{}, fmt.Errorf("fetching stats: %w", err)
}
return s, nil
}
-188
View File
@@ -1,8 +1,6 @@
package db
import (
"context"
"database/sql"
"encoding/json"
"sort"
"strings"
@@ -77,192 +75,6 @@ type CallRow struct {
CompletedAt string
}
// GetSessionTiming computes the per-session timing summary. Returns
// (nil, nil) when the session does not exist (mirrors GetSession's
// contract; the HTTP handler turns this into a 404).
func (db *DB) GetSessionTiming(
ctx context.Context, sessionID string,
) (*SessionTiming, error) {
sess, err := db.GetSession(ctx, sessionID)
if err != nil {
return nil, err
}
if sess == nil {
return nil, nil
}
turnRows, err := db.queryTurnRows(ctx, sessionID)
if err != nil {
return nil, err
}
callRows, err := db.queryCallRows(ctx, sessionID)
if err != nil {
return nil, err
}
return AssembleTiming(sess, turnRows, callRows, time.Now().UTC()), nil
}
func (db *DB) queryTurnRows(
ctx context.Context, sessionID string,
) ([]TurnRow, error) {
rows, err := db.getReader().QueryContext(ctx, `
SELECT
m2.id, m2.ordinal, m2.timestamp, m2.has_tool_use,
CASE
WHEN m2.has_tool_use = 0 THEN NULL
WHEN m2.delta_ms < 0 THEN NULL
ELSE m2.delta_ms
END AS turn_duration_ms
FROM (
SELECT
m.*,
CAST(
ROUND(
(julianday(
COALESCE(
LEAD(m.timestamp) OVER (ORDER BY m.ordinal),
s.ended_at
)
) - julianday(m.timestamp)) * 86400000
) AS INTEGER
) AS delta_ms
FROM messages m
LEFT JOIN sessions s ON s.id = m.session_id
WHERE m.session_id = ?
) m2
ORDER BY m2.ordinal
`, sessionID)
if err != nil {
return nil, err
}
defer rows.Close()
var out []TurnRow
for rows.Next() {
var r TurnRow
var ts sql.NullString
var hasFlag int
var dur sql.NullInt64
if err := rows.Scan(
&r.MessageID, &r.Ordinal, &ts, &hasFlag, &dur,
); err != nil {
return nil, err
}
if ts.Valid {
r.Timestamp = ts.String
}
r.HasToolUse = hasFlag == 1
if dur.Valid {
v := dur.Int64
r.DurationMs = &v
}
out = append(out, r)
}
return out, rows.Err()
}
func (db *DB) queryCallRows(
ctx context.Context, sessionID string,
) ([]CallRow, error) {
now := time.Now().UTC().Format(time.RFC3339)
rows, err := db.getReader().QueryContext(ctx, `
SELECT
tc.message_id,
tc.tool_use_id,
tc.tool_name,
tc.category,
tc.skill_name,
tc.subagent_session_id,
tc.input_json,
(
SELECT tre.timestamp
FROM tool_result_events tre
WHERE tre.session_id = tc.session_id
AND tre.tool_call_message_ordinal = m.ordinal
AND tre.call_index = tc.call_index
AND tre.source = 'tool_execution'
AND tre.status = 'started'
AND NULLIF(tre.timestamp, '') IS NOT NULL
ORDER BY tre.event_index ASC
LIMIT 1
) AS execution_started_at,
(
SELECT tre.timestamp
FROM tool_result_events tre
WHERE tre.session_id = tc.session_id
AND tre.tool_call_message_ordinal = m.ordinal
AND tre.call_index = tc.call_index
AND tre.source = 'tool_execution'
AND tre.status IN ('completed', 'errored')
AND NULLIF(tre.timestamp, '') IS NOT NULL
ORDER BY tre.event_index DESC
LIMIT 1
) AS execution_completed_at,
CASE
WHEN tc.subagent_session_id IS NOT NULL
AND s_sub.started_at IS NOT NULL THEN
CAST(
ROUND(
(julianday(COALESCE(s_sub.ended_at, ?))
- julianday(s_sub.started_at)) * 86400000
) AS INTEGER
)
ELSE NULL
END AS subagent_duration_ms
FROM tool_calls tc
JOIN messages m ON m.id = tc.message_id
LEFT JOIN sessions s_sub
ON s_sub.id = tc.subagent_session_id
WHERE tc.session_id = ?
ORDER BY tc.message_id, tc.id
`, now, sessionID)
if err != nil {
return nil, err
}
defer rows.Close()
var out []CallRow
for rows.Next() {
var r CallRow
var toolUseID, inputJSON sql.NullString
var skill, sub, executionStarted, executionCompleted sql.NullString
var subDur sql.NullInt64
if err := rows.Scan(
&r.MessageID, &toolUseID, &r.ToolName, &r.Category,
&skill, &sub, &inputJSON, &executionStarted, &executionCompleted,
&subDur,
); err != nil {
return nil, err
}
if toolUseID.Valid {
r.ToolUseID = toolUseID.String
}
if skill.Valid {
s := skill.String
r.SkillName = &s
}
if sub.Valid {
s := sub.String
r.SubagentSessionID = &s
}
if inputJSON.Valid {
r.InputJSON = inputJSON.String
}
if subDur.Valid {
v := subDur.Int64
r.DurationMs = &v
} else if executionStarted.Valid && executionCompleted.Valid {
v := millisBetween(executionStarted.String, executionCompleted.String)
if v >= 0 {
r.DurationMs = &v
r.CompletedAt = executionCompleted.String
}
}
out = append(out, r)
}
return out, rows.Err()
}
// AssembleTiming stitches scanned per-turn and per-call rows plus
// session metadata into a SessionTiming. Pure logic — shared by the
// SQLite and PostgreSQL backends. `now` is captured by the caller so
-149
View File
@@ -1,7 +1,6 @@
package db
import (
"context"
"fmt"
"sort"
"strings"
@@ -45,140 +44,6 @@ type TrendsTermsResponse struct {
Series []TrendSeries `json:"series"`
}
func (db *DB) GetTrendsTerms(
ctx context.Context,
f AnalyticsFilter,
terms []TrendTermInput,
granularity string,
) (TrendsTermsResponse, error) {
if granularity == "" {
granularity = "week"
}
loc := f.location()
buckets := TrendBucketRange(f.From, f.To, granularity)
bucketIndex := trendBucketIndex(buckets)
counts := make([][]int, len(terms))
for i := range counts {
counts[i] = make([]int, len(buckets))
}
messageCounts := make([]int, len(buckets))
sessionFilter := f
sessionFilter.From = ""
sessionFilter.To = ""
sessionFilter.DayOfWeek = nil
sessionFilter.Hour = nil
sessionFilter.Model = ""
where, args := sessionFilter.buildWhereWithDate("", false, "s.id")
flt := f.messageScopeFilter()
modelFiltering := len(flt.Models) > 0
query := `SELECT m.session_id, m.ordinal, m.role, m.is_system,
COALESCE(m.model, ''), m.content, COALESCE(m.timestamp, ''),
COALESCE(s.started_at, ''), s.created_at
FROM sessions s
JOIN messages m ON m.session_id = s.id
WHERE ` + where + `
AND m.role IN ('user', 'assistant')
AND m.is_system = 0
AND ` + SystemPrefixSQL("m.content", "m.role") + `
ORDER BY m.session_id, m.ordinal`
rows, err := db.getReader().QueryContext(ctx, query, args...)
if err != nil {
return TrendsTermsResponse{}, fmt.Errorf("querying trends terms: %w", err)
}
defer rows.Close()
type trendRow struct {
sessionID string
role string
isSystem bool
model string
content string
msgTS string
startedAt string
createdAt string
}
processRow := func(row trendRow) {
msgTime, ok := trendMessageLocalTime(row.msgTS, row.startedAt, row.createdAt, loc)
if !ok {
return
}
msgDate := msgTime.Format("2006-01-02")
if !inDateRange(msgDate, f.From, f.To) {
return
}
bucketDate := trendBucketDate(msgTime, loc, granularity)
bucket, ok := bucketIndex[bucketDate]
if !ok {
return
}
messageCounts[bucket]++
for i, term := range terms {
count := countTrendOccurrences(row.content, term)
if count > 0 {
counts[i][bucket] += count
}
}
}
rowStartedAt := make(map[string]string)
rowCreatedAt := make(map[string]string)
emit := func(m ScopedMessage) {
processRow(trendRow{
sessionID: m.SessionID,
role: m.Role,
isSystem: m.IsSystem,
content: m.Content,
msgTS: m.Timestamp,
startedAt: rowStartedAt[m.SessionID],
createdAt: rowCreatedAt[m.SessionID],
})
}
reducer := NewScopeReducer(flt, emit)
for rows.Next() {
var row trendRow
var ordinal int
if err := rows.Scan(
&row.sessionID, &ordinal, &row.role, &row.isSystem,
&row.model, &row.content, &row.msgTS, &row.startedAt,
&row.createdAt,
); err != nil {
return TrendsTermsResponse{}, fmt.Errorf("scanning trends term row: %w", err)
}
if !modelFiltering {
msgTime, ok := trendMessageLocalTime(row.msgTS, row.startedAt, row.createdAt, loc)
if ok && flt.MatchesDayHour(msgTime, true) {
processRow(row)
}
continue
}
rowStartedAt[row.sessionID] = row.startedAt
rowCreatedAt[row.sessionID] = row.createdAt
msgTime, has := trendMessageLocalTime(row.msgTS, row.startedAt, row.createdAt, loc)
if err := reducer.Push(MessageInput{
SessionID: row.sessionID,
Ordinal: ordinal,
Role: row.role,
Model: row.model,
IsSystem: row.isSystem,
Timestamp: row.msgTS,
LocalTime: msgTime,
HasLocalTime: has,
Content: row.content,
}); err != nil {
return TrendsTermsResponse{}, err
}
}
if err := rows.Err(); err != nil {
return TrendsTermsResponse{}, fmt.Errorf("iterating trends term rows: %w", err)
}
return BuildTrendsTermsResponse(
f.From, f.To, granularity, buckets, terms, counts, messageCounts,
), nil
}
func ParseTrendTerms(values []string) ([]TrendTermInput, error) {
terms := make([]TrendTermInput, 0, min(len(values), MaxTrendTerms))
for _, value := range values {
@@ -373,20 +238,6 @@ func mergeCountSpans(spans []matchSpan) int {
return count
}
func trendMessageLocalTime(
messageTS string,
startedAt string,
createdAt string,
loc *time.Location,
) (time.Time, bool) {
for _, ts := range []string{messageTS, startedAt, createdAt} {
if t, ok := localTime(ts, loc); ok {
return t, true
}
}
return time.Time{}, false
}
func trendBucketDate(t time.Time, loc *time.Location, granularity string) string {
return TrendBucketDate(t, loc, granularity)
}
+83 -650
View File
@@ -5,13 +5,14 @@ import (
"database/sql"
"encoding/json"
"fmt"
"slices"
"sort"
"strconv"
"strings"
"time"
"github.com/uptrace/bun"
"go.kenn.io/agentsview/internal/activity"
"go.kenn.io/agentsview/internal/db/bunmodel"
"go.kenn.io/agentsview/internal/export"
"go.kenn.io/agentsview/internal/money"
"go.kenn.io/agentsview/internal/parser"
@@ -257,47 +258,6 @@ func (f UsageFilter) appendUsageSessionFilterClauses(
return where, args
}
// appendUsageMatchingActivityClauses requires the session to have at
// least one row that GetUsageMatchingSessionCount's bounded branch would
// count: an assistant, non-synthetic message (model optional — some
// Copilot assistant messages parse before a model name is known) or a
// usage_events row with a model. Model/ExcludeModel narrow those same
// rows. Seeding the EXISTS subqueries with the matching eligibility
// predicates keeps the unbounded branch's semantics aligned with the
// bounded branch's per-row predicates, so the same filter matches the
// same sessions whether or not a date range is set.
func (f UsageFilter) appendUsageMatchingActivityClauses(
where string, args []any,
) (string, []any) {
var messageArgs []any
messageWhere, messageArgs := f.appendUsageSourceFilterClauses(
usageMatchingMessageSourceEligibility, messageArgs, "m.model",
)
var eventArgs []any
eventWhere, eventArgs := f.appendUsageSourceFilterClauses(
usageEventSourceEligibility, eventArgs, "ue.model",
)
where += `
AND (
EXISTS (
SELECT 1
FROM messages m
WHERE m.session_id = s.id
AND ` + messageWhere + `
)
OR EXISTS (
SELECT 1
FROM usage_events ue
WHERE ue.session_id = s.id
AND ` + eventWhere + `
)
)`
args = append(args, messageArgs...)
args = append(args, eventArgs...)
return where, args
}
func buildUsageTerminationPredSQLite(status string) (string, []any) {
if status == "" || status == "all" {
return "", nil
@@ -377,24 +337,6 @@ const usageMessageSourceEligibility = `
AND m.model != '<synthetic>'`
// usageMatchingMessageEligibility is usageMessageEligibility with the
// token-presence requirement removed and the model-presence requirement
// relaxed to a role check. GetUsageMatchingSessionCount counts sessions
// that have usage-shaped activity even when the agent (e.g. Copilot)
// never records per-message tokens or, for some assistant messages, a
// model name, so it must not gate on m.token_usage or m.model != ” the
// way every token/cost query does; Model/ExcludeModel filters are applied
// separately and still narrow the match when set. Do not reuse this for
// usageRowQuery or its callers — see the usageMessageEligibility doc
// comment above.
const usageMatchingMessageEligibility = `
m.role = 'assistant'
AND m.model != '<synthetic>'
AND s.deleted_at IS NULL`
const usageMatchingMessageSourceEligibility = `
m.role = 'assistant'
AND m.model != '<synthetic>'`
const usageEventEligibility = `
ue.model != ''
AND s.deleted_at IS NULL`
@@ -822,23 +764,6 @@ func dailyUsageRowSelectFromRowsWithMachine(
})
}
// dailyUsageRowSelectFromSnapshotRowsWithMachine reads rows produced by
// snapshotRankedDailyUsageRowsSQL, which already carry the attributed
// session, its metadata, and the partition-wide web-search count (NULL
// for rows that were not ranked, which the scanner parses in Go).
func dailyUsageRowSelectFromSnapshotRowsWithMachine(
rowsSQL string, includeMachine bool,
) string {
return dailyUsageRowSelectFromRowsWithColumns(
rowsSQL, includeMachine, dailyUsageRowColumns{
session: "u.snapshot_attribution_session_id",
webSearch: "u.snapshot_web_search_requests",
project: "u.snapshot_project",
agent: "u.snapshot_agent",
machine: "u.snapshot_machine",
})
}
func dailyUsageRowSelectFromRowsWithColumns(
rowsSQL string, includeMachine bool, cols dailyUsageRowColumns,
) string {
@@ -1003,354 +928,16 @@ func usageBoundedRowsSQL(
return rowsSQL, args
}
// usageMatchingSessionRowsSQLForBounds is usageRowsSQLForBounds's bounded
// branch built from the relaxed usageMatchingMessageEligibility predicates,
// so GetUsageMatchingSessionCount only relaxes the token-usage and
// model-presence requirements and keeps the same per-row
// Model/ExcludeModel filtering as the normal bounded path.
func usageMatchingSessionRowsSQLForBounds(
f UsageFilter, b usageBounds,
) (string, []any) {
return usageBoundedRowsSQL(
f, b,
usageMatchingMessageSourceEligibility, usageMatchingMessageEligibility)
}
func usageRowQuery(f UsageFilter) (string, []any) {
rowsSQL, args := usageRowsSQLForBounds(f, usageBoundsForFilter(f))
query := dailyUsageRowSelectFromRows(rowsSQL)
return query, args
}
func topSessionsUsageRowQuery(f UsageFilter) (string, []any) {
bounds := usageBoundsForFilter(f)
rowsSQL, rowsArgs := usageRowsSQLForBounds(
usageSnapshotInputFilter(f), bounds)
rowsSQL, args := snapshotRankedDailyUsageRowsSQL(
rowsSQL, rowsArgs, f, bounds)
return dailyUsageRowSelectFromSnapshotRowsWithMachine(rowsSQL, false), args
}
func usageSnapshotInputFilter(f UsageFilter) UsageFilter {
return UsageFilter{From: f.From, To: f.To, Timezone: f.Timezone}
}
const dailyCursorUsageRowsSQLTemplate = `
SELECT
'' AS session_id,
NULL AS message_ordinal,
'cursor' AS usage_source,
cu.occurred_at AS ts,
cu.model,
'' AS token_usage,
cu.input_tokens,
cu.output_tokens,
cu.cache_write_tokens AS cache_creation_input_tokens,
cu.cache_read_tokens AS cache_read_input_tokens,
0 AS reasoning_tokens,
cu.charged_microdollars AS cost_microdollars,
'cursor-reported' AS cost_source,
'' AS claude_message_id,
'' AS claude_request_id,
'' AS source_uuid,
cu.dedup_key AS usage_dedup_key,
'' AS project,
'cursor' AS agent,
'' AS machine
FROM cursor_usage_events cu
WHERE %s`
func cursorUsageRowsSQLForBounds(
f UsageFilter, b usageBounds,
) (string, []any, bool) {
termPred, _ := buildUsageTerminationPredSQLite(f.Termination)
// Cursor usage rows carry no project or git branch and bypass the session
// filter, so any filter they cannot satisfy (project, machine, branch)
// must exclude them entirely rather than let them leak into totals.
if len(f.ProjectFilterLabels()) > 0 ||
len(f.ExcludedProjectFilterLabels()) > 0 ||
f.Machine != "" || f.GitBranch != "" || f.MinUserMessages > 0 ||
f.ExcludeOneShot || termPred != "" ||
f.ActiveSince != "" {
return "", nil, false
}
if f.Agent != "" {
vals := strings.Split(f.Agent, ",")
for i := range vals {
vals[i] = strings.TrimSpace(vals[i])
}
if !slices.Contains(vals, "cursor") {
return "", nil, false
}
}
if f.ExcludeAgent != "" {
vals := strings.Split(f.ExcludeAgent, ",")
for i := range vals {
vals[i] = strings.TrimSpace(vals[i])
}
if slices.Contains(vals, "cursor") {
return "", nil, false
}
}
where := "cu.model != ''"
var args []any
scope := normalizeAutomatedScope(f.AutomatedScope, f.ExcludeAutomated)
if pred := automatedScopePredicate(scope, "cu.is_headless"); pred != "" {
where += "\n\tAND " + pred
}
where, args = f.appendUsageSourceFilterClauses(
where, args, "cu.model",
)
where, args = appendUsageColumnBounds(where, "cu.occurred_at", b, args)
rowsSQL := fmt.Sprintf(dailyCursorUsageRowsSQLTemplate, where)
return rowsSQL, args, true
}
func dailyUsageRowsSQLForBounds(
f UsageFilter, b usageBounds, hasCursorTable bool,
) (string, []any) {
sessionRowsSQL, sessionArgs := usageRowsSQLForBounds(
usageSnapshotInputFilter(f), b)
if !hasCursorTable {
return sessionRowsSQL, sessionArgs
}
cursorRowsSQL, cursorArgs, ok := cursorUsageRowsSQLForBounds(f, b)
if !ok {
return sessionRowsSQL, sessionArgs
}
rowsSQL := sessionRowsSQL + "\n\nUNION ALL\n\n" + cursorRowsSQL
args := make([]any, 0, len(sessionArgs)+len(cursorArgs))
args = append(args, sessionArgs...)
args = append(args, cursorArgs...)
return rowsSQL, args
}
func exactUsageUTCWindow(f UsageFilter) usageBounds {
loc := f.location()
var out usageBounds
if f.From != "" {
if from, err := time.ParseInLocation("2006-01-02", f.From, loc); err == nil {
out.from = from.UTC().Format(time.RFC3339Nano)
}
}
if f.To != "" {
if to, err := time.ParseInLocation("2006-01-02", f.To, loc); err == nil {
out.to = to.AddDate(0, 0, 1).UTC().Format(time.RFC3339Nano)
}
}
return out
}
// snapshotRankedDailyUsageRowsSQL wraps rowsSQL so that each Claude request
// (claude_message_id, claude_request_id) contributes one row: the greatest
// output snapshot, attributed to the session that streamed the request
// first, carrying the maximum billed web-search count across its snapshots.
// Rows without complete Claude request identity bypass the ranking.
//
// Only requests that appear more than once are ranked. usage_snapshot_dups
// finds them with an index-only pass over messages, usage_snapshot_ranked
// runs the window functions over just those rows, and every other row
// passes through with itself as attribution and a NULL web-search count that
// the scanner parses from token_usage in Go. Ranking every row through the
// window functions cost two to five times the underlying scan, because
// SQLite sorts and materializes the full-width rows once per window.
//
// rowsArgs are the placeholders of rowsSQL; the returned args carry them in
// position with the ranking's own placeholders. Callers finish with
// dailyUsageRowSelectFromSnapshotRowsWithMachine.
func snapshotRankedDailyUsageRowsSQL(
rowsSQL string, rowsArgs []any, f UsageFilter, b usageBounds,
) (string, []any) {
windowWhere, windowArgs := usageSnapshotWindowWhere(f)
dupsSQL, dupsArgs := usageSnapshotDuplicateRequestsSQL(b)
claudeRowsSQL, claudeArgs := usageSnapshotClaudeMessageRowsSQL(b)
filterWhere := "1=1"
var filterArgs []any
filterWhere, filterArgs = f.appendUsageSourceFilterClauses(
filterWhere, filterArgs, "survivor.model")
filterWhere, filterArgs = f.appendUsageSessionFilterClauses(
filterWhere, filterArgs)
survivorFilter := ""
if filterWhere != "1=1" {
survivorFilter = `
LEFT JOIN sessions s
ON s.id = survivor.snapshot_attribution_session_id
WHERE survivor.snapshot_attribution_session_id = ''
OR (` + filterWhere + `)`
}
outputTokens := fmt.Sprintf(`MIN(MAX(COALESCE(CASE
WHEN json_valid(u.token_usage) THEN CAST(json_extract(
u.token_usage, '$.output_tokens') AS INTEGER)
ELSE agentsview_usage_output_tokens(u.token_usage)
END, 0), 0), %d)`, MaxPlausibleTokens)
webSearchRequests := `MAX(COALESCE(CASE
WHEN json_valid(u.token_usage) THEN CAST(json_extract(
u.token_usage, '$.server_tool_use.web_search_requests'
) AS INTEGER)
ELSE agentsview_usage_web_search_requests(u.token_usage)
END, 0), 0)`
args := make([]any, 0,
len(dupsArgs)+len(claudeArgs)+2*len(windowArgs)+
len(rowsArgs)+len(filterArgs))
args = append(args, dupsArgs...)
args = append(args, claudeArgs...)
args = append(args, windowArgs...)
args = append(args, rowsArgs...)
args = append(args, windowArgs...)
args = append(args, filterArgs...)
return fmt.Sprintf(`
WITH usage_snapshot_dups AS (%[1]s),
usage_snapshot_ranked AS (
SELECT u.session_id, u.message_ordinal,
FIRST_VALUE(u.session_id) OVER attribution
AS snapshot_attribution_session_id,
FIRST_VALUE(u.project) OVER attribution AS snapshot_project,
FIRST_VALUE(u.agent) OVER attribution AS snapshot_agent,
FIRST_VALUE(u.machine) OVER attribution AS snapshot_machine,
ROW_NUMBER() OVER ranking AS snapshot_rank,
MAX(%[5]s) OVER (
ranking ROWS BETWEEN UNBOUNDED PRECEDING
AND UNBOUNDED FOLLOWING
) AS snapshot_web_search_requests
FROM (%[2]s) u
WHERE %[3]s
WINDOW attribution AS (
PARTITION BY u.claude_message_id, u.claude_request_id
ORDER BY julianday(u.ts) IS NULL ASC,
julianday(u.ts) ASC, u.session_id ASC,
COALESCE(u.message_ordinal, -1) ASC,
CASE WHEN julianday(u.ts) IS NULL THEN u.ts ELSE '' END ASC
), ranking AS (
PARTITION BY u.claude_message_id, u.claude_request_id
ORDER BY %[4]s DESC,
julianday(u.ts) IS NULL ASC, julianday(u.ts) DESC,
u.session_id DESC, COALESCE(u.message_ordinal, -1) DESC,
CASE WHEN julianday(u.ts) IS NULL THEN u.ts ELSE '' END DESC
)
),
usage_snapshot_survivors AS (
SELECT u.*,
COALESCE(r.snapshot_attribution_session_id, u.session_id)
AS snapshot_attribution_session_id,
COALESCE(r.snapshot_project, u.project) AS snapshot_project,
COALESCE(r.snapshot_agent, u.agent) AS snapshot_agent,
COALESCE(r.snapshot_machine, u.machine) AS snapshot_machine,
r.snapshot_web_search_requests
FROM (%[6]s) u
LEFT JOIN usage_snapshot_ranked r
ON u.usage_source = 'message'
AND r.session_id = u.session_id
AND r.message_ordinal = u.message_ordinal
WHERE %[3]s
AND (r.snapshot_rank IS NULL OR r.snapshot_rank = 1)
)
SELECT survivor.*
FROM usage_snapshot_survivors survivor%[7]s`,
dupsSQL, claudeRowsSQL, windowWhere, outputTokens,
webSearchRequests, rowsSQL, survivorFilter), args
}
// usageSnapshotWindowWhere restricts rows to the filter's exact UTC window
// so snapshots outside the requested dates neither win a partition nor
// reach the scanner. Rows whose timestamp julianday cannot parse fall back
// to a date-prefix comparison, mirroring the scanner's local-date filter.
func usageSnapshotWindowWhere(f UsageFilter) (string, []any) {
window := exactUsageUTCWindow(f)
where := "1=1"
var args []any
if window.from != "" {
where += `
AND (
julianday(u.ts) >= julianday(?)
OR (julianday(u.ts) IS NULL AND substr(u.ts, 1, 10) >= ?)
)`
args = append(args, window.from, f.From)
}
if window.to != "" {
where += `
AND (
julianday(u.ts) < julianday(?)
OR (julianday(u.ts) IS NULL AND substr(u.ts, 1, 10) <= ?)
)`
args = append(args, window.to, f.To)
}
return where, args
}
// usageSnapshotClaudeIdentity selects the message rows that carry complete
// Claude request identity and could enter the usage row source.
const usageSnapshotClaudeIdentity = usageMessageSourceEligibility + `
AND m.claude_message_id != ''
AND m.claude_request_id != ''`
// usageSnapshotDuplicateRequestsSQL lists the Claude requests that appear on
// more than one eligible message. It over-approximates the ranked set (it
// ignores session eligibility and the fallback session bounds), which only
// sends extra rows through the ranking; the ranking itself applies the exact
// row-source predicates. Bounded filters seek idx_messages_usage_covering
// per timestamp branch so the pass stays proportional to the window;
// unbounded filters read idx_messages_claude_snapshot in partition order.
func usageSnapshotDuplicateRequestsSQL(b usageBounds) (string, []any) {
const key = `m.claude_message_id, m.claude_request_id`
if !b.bounded() {
return `
SELECT ` + key + `
FROM messages m
WHERE ` + usageSnapshotClaudeIdentity + `
GROUP BY ` + key + `
HAVING COUNT(*) > 1`, nil
}
timestampWhere, args := appendUsageColumnBounds(
usageSnapshotClaudeIdentity, "m.timestamp", b, nil)
return `
SELECT claude_message_id, claude_request_id
FROM (
SELECT ` + key + `
FROM messages m
WHERE ` + timestampWhere + `
UNION ALL
SELECT ` + key + `
FROM messages m
WHERE ` + usageSnapshotClaudeIdentity + `
AND m.timestamp IS NULL
UNION ALL
SELECT ` + key + `
FROM messages m
WHERE ` + usageSnapshotClaudeIdentity + `
AND m.timestamp = ''
)
GROUP BY claude_message_id, claude_request_id
HAVING COUNT(*) > 1`, args
}
// usageSnapshotClaudeMessageRowsSQL produces the row-source shape for the
// messages of duplicated Claude requests, using the same eligibility and
// bounds as usageRowsSQLForBounds's message branches so the ranked rows are
// exactly the row source's Claude rows for those requests.
func usageSnapshotClaudeMessageRowsSQL(b usageBounds) (string, []any) {
where := usageMessageEligibility + `
AND m.claude_message_id != ''
AND m.claude_request_id != ''
AND (m.claude_message_id, m.claude_request_id) IN (
SELECT claude_message_id, claude_request_id FROM usage_snapshot_dups
)`
var args []any
if b.bounded() {
timestampWhere, timestampArgs := appendUsageColumnBounds(
"m.timestamp IS NOT NULL AND m.timestamp != ''",
"m.timestamp", b, nil)
fallbackWhere, fallbackArgs := appendUsageColumnBounds(
"NULLIF(m.timestamp, '') IS NULL", "s.started_at", b, nil)
where += `
AND ((` + timestampWhere + `) OR (` + fallbackWhere + `))`
args = append(args, timestampArgs...)
args = append(args, fallbackArgs...)
}
return fmt.Sprintf(dailyUsageMessageRowsSQLTemplate, "messages", where), args
}
func scanUsageRow(rows *sql.Rows) (usageScanRow, error) {
var r usageScanRow
err := rows.Scan(
@@ -1385,10 +972,6 @@ func scanUsageRow(rows *sql.Rows) (usageScanRow, error) {
return r, err
}
func scanDailyUsageRow(rows *sql.Rows) (dailyUsageScanRow, error) {
return scanDailyUsageRowWithMachine(rows, false)
}
func scanDailyUsageRowWithMachine(
rows *sql.Rows, includeMachine bool,
) (dailyUsageScanRow, error) {
@@ -2051,52 +1634,29 @@ func usageDedupTokenForRow(
return usageDedupToken{}, false
}
func (db *DB) loadTopSessionMetadata(
ctx context.Context, sessionIDs []string,
func loadTopSessionMetadataFrom(
ctx context.Context, store bun.IDB, sessionIDs []string,
) (map[string]topSessionMetadata, error) {
out := make(map[string]topSessionMetadata, len(sessionIDs))
if len(sessionIDs) == 0 {
return out, nil
}
placeholders := make([]string, len(sessionIDs))
args := make([]any, len(sessionIDs))
for i, id := range sessionIDs {
placeholders[i] = "?"
args[i] = id
}
query := `
SELECT
id,
COALESCE(NULLIF(COALESCE(display_name, session_name), ''), NULLIF(first_message, ''), NULLIF(project, ''), id) AS display_name,
agent,
project,
COALESCE(started_at, '') AS started_at
FROM sessions
WHERE id IN (` + strings.Join(placeholders, ",") + `)`
rows, err := db.getReader().QueryContext(ctx, query, args...)
if err != nil {
var rows []bunmodel.Session
if err := store.NewSelect().Model(&rows).
Where("id IN (?)", bun.List(sessionIDs)).Scan(ctx); err != nil {
return nil, fmt.Errorf("querying top session metadata: %w", err)
}
defer rows.Close()
for rows.Next() {
var id string
var meta topSessionMetadata
if err := rows.Scan(
&id,
&meta.displayName,
&meta.agent,
&meta.project,
&meta.startedAt,
); err != nil {
return nil, fmt.Errorf("scanning top session metadata: %w", err)
for _, row := range rows {
projection := bunUsageProjection{
SessionID: row.ID, Project: row.Project, Agent: row.Agent,
DisplayName: row.DisplayName, SessionName: row.SessionName,
FirstMessage: row.FirstMessage, SessionStartedAt: row.StartedAt,
}
out[row.ID] = topSessionMetadata{
displayName: usageSessionDisplayName(projection),
agent: row.Agent, project: row.Project,
startedAt: formatUsageTimestamp(row.StartedAt),
}
out[id] = meta
}
if err := rows.Err(); err != nil {
return nil, fmt.Errorf("iterating top session metadata: %w", err)
}
return out, nil
}
@@ -2231,67 +1791,6 @@ func SanitizeDailyUsageProjectLabelsWithCatalog(
}
}
// loadPricingMap reads the model_pricing table into a map for
// in-memory joins. This is much faster than a SQL LEFT JOIN
// on every row of the daily usage scan, since the pricing
// table is tiny and repeated resolver lookups are cached.
func (db *DB) loadPricingMap(
ctx context.Context,
) ([]export.EffectivePricingRow, error) {
return db.loadPricingMapFrom(ctx, db.getReader())
}
func (db *DB) loadPricingMapFrom(
ctx context.Context, q sessionExportQuerier,
) ([]export.EffectivePricingRow, error) {
prices, err := listModelPricingFrom(ctx, q)
if err != nil {
return nil, err
}
fallback := fallbackRateMap()
out := make(map[string]export.ModelRates)
for _, p := range prices {
if strings.HasPrefix(p.ModelPattern, "_") {
continue
}
rates := modelPricingRates(p)
rates.Source = modelPricingSource(p, fallback)
out[p.ModelPattern] = rates
}
if len(out) == 0 {
for model, rates := range db.emptyCatalogPricing {
rates.Bands = append([]export.PricingBand(nil), rates.Bands...)
out[model] = rates
}
}
for model, cp := range db.customPricing {
rates := export.ModelRates{
InputPerMTok: money.Money{
Microdollars: cp.InputMicrodollarsPerMTok,
},
OutputPerMTok: money.Money{
Microdollars: cp.OutputMicrodollarsPerMTok,
},
CacheWritePerMTok: money.Money{
Microdollars: cp.CacheCreationMicrodollarsPerMTok,
},
CacheReadPerMTok: money.Money{
Microdollars: cp.CacheReadMicrodollarsPerMTok,
},
}
rates.Source = customPricingSource()
out[model] = rates
}
for model, rates := range db.effectivePricing {
rates.Bands = append([]export.PricingBand(nil), rates.Bands...)
out[model] = rates
}
return pricingMapRows(out), nil
}
func customPricingSource() export.PricingRowSource {
return export.PricingRowSourceCustom
}
@@ -2423,37 +1922,35 @@ func paddedUTCBound(ts string, hours int) string {
// parses them in Go (faster than SQLite's json_extract per row),
// joins against an in-memory pricing map, and buckets by
// local date.
func (db *DB) GetDailyUsage(
func (s *BunStore) GetDailyUsage(
ctx context.Context, f UsageFilter,
) (DailyUsageResult, error) {
var staged DailyUsageResult
err := s.consistentView(ctx, func(store bun.IDB) error {
var err error
staged, err = s.getDailyUsageFrom(ctx, store, f)
return err
})
return staged, err
}
func (s *BunStore) getDailyUsageFrom(
ctx context.Context, store bun.IDB, f UsageFilter,
) (DailyUsageResult, error) {
loc := f.location()
pricing, err := db.loadPricingMap(ctx)
pricing, err := s.loadPricingMapFrom(ctx, store)
if err != nil {
return DailyUsageResult{},
fmt.Errorf("loading pricing: %w", err)
}
rateResolver := export.NewPricingResolver(pricing)
// Filter on usage timestamp (not only session started_at) so
// long-lived sessions that span date boundaries are included.
// Pad by +/-14h to cover all timezone offsets; the actual
// date filtering happens post-query via localDate.
bounds := usageBoundsForFilter(f)
query, rowsArgs := dailyUsageRowsSQLForBounds(
f, bounds, db.hasCursorUsageTable())
query, args := snapshotRankedDailyUsageRowsSQL(query, rowsArgs, f, bounds)
query = dailyUsageRowSelectFromSnapshotRowsWithMachine(
query, f.Breakdowns)
query += ` ORDER BY u.ts ASC, u.session_id ASC,
COALESCE(u.message_ordinal, -1) ASC`
rows, err := db.getReader().QueryContext(ctx, query, args...)
rows, err := s.loadDailyUsageRowsFrom(ctx, store, f, true, false)
if err != nil {
return DailyUsageResult{},
fmt.Errorf("querying daily usage: %w", err)
}
defer rows.Close()
type bucket struct {
inputTok int
@@ -2485,13 +1982,7 @@ func (db *DB) GetDailyUsage(
// single fallback rate would misreport mixed-model periods.
var totalSavings money.Money
for rows.Next() {
r, scanErr := scanDailyUsageRowWithMachine(rows, f.Breakdowns)
if scanErr != nil {
return DailyUsageResult{},
fmt.Errorf("scanning daily usage row: %w", scanErr)
}
for _, r := range rows {
date := localDate(r.ts, loc)
if f.From != "" && date < f.From {
continue
@@ -2566,10 +2057,6 @@ func (db *DB) GetDailyUsage(
}
sessionCosts[r.sessionID] = sc
}
if err := rows.Err(); err != nil {
return DailyUsageResult{},
fmt.Errorf("iterating daily usage rows: %w", err)
}
sessionIDs := make([]string, 0, len(sessionCosts))
for sessionID := range sessionCosts {
sessionIDs = append(sessionIDs, sessionID)
@@ -2750,8 +2237,8 @@ func (db *DB) GetDailyUsage(
if seenSessions != nil {
sessionCounts = NewUsageSessionCounts(seenSessions)
}
projects, err := db.BuildProjectIdentityMap(ctx,
sortedSetKeys(projectLabels))
projects, err := buildBunProjectIdentityMapFrom(
ctx, store, sortedSetKeys(projectLabels))
if err != nil {
return DailyUsageResult{}, err
}
@@ -2998,7 +2485,8 @@ func (db *DB) GetDailyUsage(
if seenSessions != nil {
sessionCounts = NewUsageSessionCounts(seenSessions)
}
projects, err := db.BuildProjectIdentityMap(ctx, sortedSetKeys(projectLabels))
projects, err := buildBunProjectIdentityMapFrom(
ctx, store, sortedSetKeys(projectLabels))
if err != nil {
return DailyUsageResult{}, err
}
@@ -3086,31 +2574,33 @@ func SortAndLimitTopSessions(
// GetTopSessionsByCost returns sessions ranked by total cost, or by total
// tokens when f.TopSessionsSort is "tokens",
// over the filter range. Default limit 20, max 100.
func (db *DB) GetTopSessionsByCost(
func (s *BunStore) GetTopSessionsByCost(
ctx context.Context, f UsageFilter, limit int,
) ([]TopSessionEntry, error) {
pricing, err := db.loadPricingMap(ctx)
var staged []TopSessionEntry
err := s.consistentView(ctx, func(store bun.IDB) error {
var err error
staged, err = s.getTopSessionsByCostFrom(ctx, store, f, limit)
return err
})
return staged, err
}
func (s *BunStore) getTopSessionsByCostFrom(
ctx context.Context, store bun.IDB, f UsageFilter, limit int,
) ([]TopSessionEntry, error) {
pricing, err := s.loadPricingMapFrom(ctx, store)
if err != nil {
return nil,
fmt.Errorf("loading pricing: %w", err)
}
rateResolver := export.NewPricingResolver(pricing)
query, args := topSessionsUsageRowQuery(f)
// Deterministic order so the dedup "winner" (the session
// that gets credit for a duplicate message.id + request.id
// pair) is stable across runs: earliest timestamp wins,
// then session_id, then message ordinal.
query += ` ORDER BY u.ts ASC, u.session_id ASC,
COALESCE(u.message_ordinal, -1) ASC`
rows, err := db.getReader().QueryContext(ctx, query, args...)
rows, err := s.loadDailyUsageRowsFrom(ctx, store, f, false, false)
if err != nil {
return nil,
fmt.Errorf("querying top sessions: %w", err)
}
defer rows.Close()
loc := f.location()
type sessAccum struct {
@@ -3132,13 +2622,7 @@ func (db *DB) GetTopSessionsByCost(
// totals from GetDailyUsage. Same key and ordering rules.
seen := make(map[usageDedupToken]struct{})
for rows.Next() {
r, err := scanDailyUsageRow(rows)
if err != nil {
return nil,
fmt.Errorf("scanning top sessions row: %w", err)
}
for _, r := range rows {
// Post-query date filter (same as GetDailyUsage).
date := localDate(r.ts, loc)
if f.From != "" && date < f.From {
@@ -3187,10 +2671,6 @@ func (db *DB) GetTopSessionsByCost(
sa.authoritativeCost = &v
}
}
if err := rows.Err(); err != nil {
return nil,
fmt.Errorf("iterating top sessions rows: %w", err)
}
result := make([]TopSessionEntry, 0, len(order))
for _, id := range order {
sa, ok := accum[id]
@@ -3222,7 +2702,7 @@ func (db *DB) GetTopSessionsByCost(
for i := range result {
sessionIDs[i] = result[i].SessionID
}
metadata, err := db.loadTopSessionMetadata(ctx, sessionIDs)
metadata, err := loadTopSessionMetadataFrom(ctx, store, sessionIDs)
if err != nil {
return nil, err
}
@@ -3443,10 +2923,24 @@ func sessionUsageBreakdownLabel(r usageScanRow) string {
// the session does not exist. BreakdownCount is always populated;
// per-row Breakdown entries are built only when includeBreakdown is
// true so callers that need just the totals avoid the row payload.
func (db *DB) GetSessionUsage(
func (s *BunStore) GetSessionUsage(
ctx context.Context, sessionID string, includeBreakdown bool,
) (*SessionUsage, error) {
sess, err := db.GetSession(ctx, sessionID)
var staged *SessionUsage
err := s.consistentView(ctx, func(store bun.IDB) error {
var err error
staged, err = s.getSessionUsageFrom(
ctx, store, sessionID, includeBreakdown,
)
return err
})
return staged, err
}
func (s *BunStore) getSessionUsageFrom(
ctx context.Context, store bun.IDB, sessionID string, includeBreakdown bool,
) (*SessionUsage, error) {
sess, err := s.getSessionFrom(ctx, store, sessionID, false)
if err != nil {
return nil, err
}
@@ -3454,23 +2948,18 @@ func (db *DB) GetSessionUsage(
return nil, nil
}
pricing, err := db.loadPricingMap(ctx)
pricing, err := s.loadPricingMapFrom(ctx, store)
if err != nil {
return nil, fmt.Errorf("loading pricing: %w", err)
}
rateResolver := export.NewPricingResolver(pricing)
query := usageRowSelect() + ` AND u.session_id = ?
ORDER BY u.ts ASC, u.session_id ASC,
COALESCE(u.message_ordinal, -1) ASC,
u.usage_source ASC,
COALESCE(u.usage_dedup_key, '') ASC`
rows, err := db.getReader().QueryContext(ctx, query, sessionID)
rows, err := s.loadSessionUsageRowsFrom(
ctx, store, UsageFilter{}, sessionID,
)
if err != nil {
return nil, fmt.Errorf("querying session usage: %w", err)
}
defer rows.Close()
var cost money.Money
var authoritativeCost *money.Money
var hasComputedCost, hasReportedCost bool
@@ -3481,17 +2970,7 @@ func (db *DB) GetSessionUsage(
breakdown := make([]SessionUsageBreakdownEntry, 0)
breakdownCount := 0
var usageRows []usageScanRow
for rows.Next() {
r, scanErr := scanUsageRow(rows)
if scanErr != nil {
return nil, fmt.Errorf("scanning session usage row: %w", scanErr)
}
usageRows = append(usageRows, r)
}
if err := rows.Err(); err != nil {
return nil, fmt.Errorf("iterating session usage rows: %w", err)
}
usageRows := rows
snapshotRows := make([]activity.UsageRow, len(usageRows))
for i, r := range usageRows {
var outputTokens int
@@ -3518,6 +2997,7 @@ func (db *DB) GetSessionUsage(
activity.ClaudeSnapshotSurvivorSelection(snapshotRows)
deduplicatedOutputTokens := 0
seen := make(map[usageDedupToken]struct{})
for i, r := range usageRows {
if !snapshotMask[i] {
deduplicatedOutputTokens += snapshotRows[i].OutputTokens
@@ -3662,24 +3142,14 @@ func NewUsageSessionCounts(
// Like GetDailyUsage and GetTopSessionsByCost, this query pads
// the UTC bounds by +/-14h and applies a post-query localDate
// filter so timezone-boundary messages are counted correctly.
func (db *DB) GetUsageSessionCounts(
func (s *BunStore) GetUsageSessionCounts(
ctx context.Context, f UsageFilter,
) (UsageSessionCounts, error) {
query, args := topSessionsUsageRowQuery(f)
// Deterministic ordering so the Claude dedup winner — the
// session that "owns" a shared message — is stable across
// runs. Matches GetDailyUsage / GetTopSessionsByCost so all
// three queries agree on which session gets credit.
query += ` ORDER BY u.ts ASC, u.session_id ASC,
COALESCE(u.message_ordinal, -1) ASC`
rows, err := db.getReader().QueryContext(ctx, query, args...)
rows, err := s.loadDailyUsageRows(ctx, f, false, false)
if err != nil {
return UsageSessionCounts{},
fmt.Errorf("querying session counts: %w", err)
}
defer rows.Close()
loc := f.location()
// Track which sessions pass the localDate filter via a
@@ -3698,13 +3168,7 @@ func (db *DB) GetUsageSessionCounts(
// disagree with the deduped token totals.
dedup := make(map[usageDedupToken]struct{})
for rows.Next() {
r, err := scanDailyUsageRow(rows)
if err != nil {
return UsageSessionCounts{},
fmt.Errorf("scanning session counts: %w", err)
}
for _, r := range rows {
// Post-query date filter (same as GetDailyUsage).
date := localDate(r.ts, loc)
if f.From != "" && date < f.From {
@@ -3733,11 +3197,6 @@ func (db *DB) GetUsageSessionCounts(
}
}
}
if err := rows.Err(); err != nil {
return UsageSessionCounts{},
fmt.Errorf("iterating session counts: %w", err)
}
out := UsageSessionCounts{
Total: len(seen),
ByProject: make(map[string]int),
@@ -3758,42 +3217,19 @@ func (db *DB) GetUsageSessionCounts(
// own), the same shape usageRowsSQLForBounds uses, so a session whose
// started_at/ended_at fall outside the window but whose message activity
// falls inside it is still counted.
func (db *DB) GetUsageMatchingSessionCount(
func (s *BunStore) GetUsageMatchingSessionCount(
ctx context.Context, f UsageFilter,
) (int, error) {
bounds := usageBoundsForFilter(f)
if !bounds.bounded() {
where, args := f.appendUsageSessionFilterClauses(usageSessionEligibility, nil)
where, args = f.appendUsageMatchingActivityClauses(where, args)
var count int
err := db.getReader().QueryRowContext(ctx, `
SELECT COUNT(*)
FROM sessions s WHERE `+where, args...).Scan(&count)
if err != nil {
return 0, fmt.Errorf("querying matching usage sessions: %w", err)
}
return count, nil
}
rowsSQL, args := usageMatchingSessionRowsSQLForBounds(f, bounds)
rows, err := db.getReader().QueryContext(
ctx, dailyUsageRowSelectFromRows(rowsSQL), args...)
rows, err := s.loadDailyUsageRows(ctx, f, false, true)
if err != nil {
return 0, fmt.Errorf("querying matching usage sessions: %w", err)
}
defer rows.Close()
loc := f.location()
seen := make(map[string]struct{})
for rows.Next() {
r, err := scanDailyUsageRow(rows)
if err != nil {
return 0, fmt.Errorf("scanning matching usage session: %w", err)
}
for _, r := range rows {
date := localDate(r.ts, loc)
if date == "" {
if usageBoundsForFilter(f).bounded() && date == "" {
continue
}
if f.From != "" && date < f.From {
@@ -3804,8 +3240,5 @@ func (db *DB) GetUsageMatchingSessionCount(
}
seen[r.sessionID] = struct{}{}
}
if err := rows.Err(); err != nil {
return 0, fmt.Errorf("iterating matching usage sessions: %w", err)
}
return len(seen), nil
}
-212
View File
@@ -434,54 +434,6 @@ func TestUsageRowQueryPushesDateBoundsIntoUnion(t *testing.T) {
assert.Equal(t, "2024-07-01T13:59:59Z", args[7])
}
func TestTopSessionsUsageRowQueryUsesNarrowScan(t *testing.T) {
query, args := topSessionsUsageRowQuery(UsageFilter{
From: "2024-06-01",
To: "2024-06-30",
Timezone: "America/New_York",
})
normalized := strings.ToLower(query)
assert.NotContains(t, normalized, "display_name")
assert.NotContains(t, normalized, "first_message")
assert.NotContains(t, normalized, "cost_status")
assert.Contains(t, normalized, "u.cost_source")
assert.NotContains(t, normalized, "user_message_count")
assert.NotContains(t, normalized, "session_activity_at")
assert.NotContains(t, normalized, " as started_at")
assert.NotContains(t, normalized, " as machine")
assert.Contains(t, normalized, "m.timestamp is not null")
assert.Contains(t, normalized, "m.timestamp != ''")
assert.Contains(t, normalized, "ue.occurred_at is not null")
assert.Contains(t, normalized, "nullif(m.timestamp, '') is null")
assert.Contains(t, normalized, "ue.occurred_at is null")
assert.Contains(t, normalized, "m.timestamp >= ?")
assert.Contains(t, normalized, "ue.occurred_at >= ?")
assert.Contains(t, normalized,
"nullif(m.timestamp, '') is null\n\tand s.started_at >= ?")
assert.Contains(t, normalized,
"ue.occurred_at is null\n\tand s.started_at >= ?")
assert.Contains(t, normalized, "m.timestamp <= ?")
assert.Contains(t, normalized, "ue.occurred_at <= ?")
assert.Contains(t, normalized, "julianday(u.ts) >= julianday(?)")
assert.Contains(t, normalized, "julianday(u.ts) < julianday(?)")
padded := []any{"2024-05-31T10:00:00Z", "2024-07-01T13:59:59Z"}
window := []any{
"2024-06-01T04:00:00Z", "2024-06-01",
"2024-07-01T04:00:00Z", "2024-06-30",
}
var want []any
want = append(want, padded...) // duplicate-request pass
want = append(want, padded...) // ranked rows: m.timestamp
want = append(want, padded...) // ranked rows: s.started_at
want = append(want, window...) // ranked rows: exact window
for range 4 { // row source branches
want = append(want, padded...)
}
want = append(want, window...) // survivors: exact window
assert.Equal(t, want, args)
}
func TestUsageEventsReplaceAndList(t *testing.T) {
d := testDB(t)
ctx := context.Background()
@@ -1778,170 +1730,6 @@ func TestGetDailyUsagePrefersTimestampedEqualClaudeSnapshot(t *testing.T) {
assert.Equal(t, 100, result.Totals.OutputTokens)
}
// seedSnapshotTiePair stores one Claude request (msg-tie/req-tie) streamed
// into two sessions with equal output tokens: z-snapshot carries the larger
// input count at zTimestamp and a-snapshot the smaller one at aTimestamp.
func seedSnapshotTiePair(t *testing.T, d *DB, zTimestamp, aTimestamp string) {
t.Helper()
for _, id := range []string{"z-snapshot", "a-snapshot"} {
insertSession(t, d, id, "proj", func(s *Session) {
s.Agent = "claude"
s.StartedAt = new("2026-05-20T10:00:00Z")
})
}
insertMessages(t, d,
Message{
SessionID: "z-snapshot", Ordinal: 0, Role: "assistant",
Timestamp: zTimestamp, Model: "claude-opus-4-6",
TokenUsage: json.RawMessage(
`{"input_tokens":900,"output_tokens":100}`),
OutputTokens: 100, HasOutputTokens: true,
ClaudeMessageID: "msg-tie", ClaudeRequestID: "req-tie",
},
Message{
SessionID: "a-snapshot", Ordinal: 0, Role: "assistant",
Timestamp: aTimestamp, Model: "claude-opus-4-6",
TokenUsage: json.RawMessage(
`{"input_tokens":10,"output_tokens":100}`),
OutputTokens: 100, HasOutputTokens: true,
ClaudeMessageID: "msg-tie", ClaudeRequestID: "req-tie",
},
)
}
// querySnapshotRankedRows runs the snapshot ranking over the real usage row
// source for f and returns the surviving rows as
// (session_id, snapshot_attribution_session_id, token_usage) triples.
func querySnapshotRankedRows(
t *testing.T, d *DB, f UsageFilter,
) [][3]string {
t.Helper()
bounds := usageBoundsForFilter(f)
rowsSQL, rowsArgs := usageRowsSQLForBounds(
usageSnapshotInputFilter(f), bounds)
ranked, args := snapshotRankedDailyUsageRowsSQL(
rowsSQL, rowsArgs, f, bounds)
rows, err := d.getReader().Query(`
SELECT session_id, snapshot_attribution_session_id, token_usage
FROM (`+ranked+`)
ORDER BY session_id`, args...)
require.NoError(t, err)
defer rows.Close()
var out [][3]string
for rows.Next() {
var row [3]string
require.NoError(t, rows.Scan(&row[0], &row[1], &row[2]))
out = append(out, row)
}
require.NoError(t, rows.Err())
return out
}
func TestSnapshotRankedDailyUsageRowsPrefersLatestEqualOutput(t *testing.T) {
d := testDB(t)
seedSnapshotTiePair(t, d,
"2026-05-20T10:31:00Z", "2026-05-20T10:30:00Z")
got := querySnapshotRankedRows(t, d, UsageFilter{})
require.Len(t, got, 1)
assert.Equal(t, "z-snapshot", got[0][0])
assert.Equal(t, "a-snapshot", got[0][1])
assert.JSONEq(t, `{"input_tokens":900,"output_tokens":100}`, got[0][2])
}
func TestSnapshotRankedDailyUsageRowsNormalizesRFC3339Timestamps(t *testing.T) {
tests := []struct {
name string
zTimestamp string
aTimestamp string
wantSession string
wantAttribut string
wantInput int
}{
{
name: "mixed fractional precision",
zTimestamp: "2026-05-20T10:30:00.1Z",
aTimestamp: "2026-05-20T10:30:00Z",
wantSession: "z-snapshot",
wantAttribut: "a-snapshot",
wantInput: 900,
},
{
name: "equivalent offsets use session fallback",
zTimestamp: "2026-05-20T05:30:00-05:00",
aTimestamp: "2026-05-20T10:30:00Z",
wantSession: "z-snapshot",
wantAttribut: "a-snapshot",
wantInput: 900,
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
d := testDB(t)
seedSnapshotTiePair(t, d, tt.zTimestamp, tt.aTimestamp)
got := querySnapshotRankedRows(t, d, UsageFilter{})
require.Len(t, got, 1)
assert.Equal(t, tt.wantSession, got[0][0])
assert.Equal(t, tt.wantAttribut, got[0][1])
assert.JSONEq(t, fmt.Sprintf(
`{"input_tokens":%d,"output_tokens":100}`, tt.wantInput),
got[0][2])
})
}
}
// Only duplicated Claude requests pass through the ranking; every other row
// survives untouched, attributed to its own session, including rows outside
// the window that the ranking must not drag back in.
func TestSnapshotRankedDailyUsageRowsRanksOnlyDuplicatedRequests(t *testing.T) {
d := testDB(t)
seedSnapshotTiePair(t, d,
"2026-05-20T10:31:00Z", "2026-05-20T10:30:00Z")
insertSession(t, d, "solo", "proj", func(s *Session) {
s.Agent = "claude"
s.StartedAt = new("2026-05-20T10:00:00Z")
})
insertMessages(t, d,
Message{
SessionID: "solo", Ordinal: 0, Role: "assistant",
Timestamp: "2026-05-20T10:32:00Z", Model: "claude-opus-4-6",
TokenUsage: json.RawMessage(
`{"input_tokens":5,"output_tokens":7}`),
OutputTokens: 7, HasOutputTokens: true,
ClaudeMessageID: "msg-solo", ClaudeRequestID: "req-solo",
},
Message{
SessionID: "solo", Ordinal: 1, Role: "assistant",
Timestamp: "2026-05-20T10:33:00Z", Model: "claude-opus-4-6",
TokenUsage: json.RawMessage(
`{"input_tokens":6,"output_tokens":8}`),
OutputTokens: 8, HasOutputTokens: true,
},
Message{
SessionID: "solo", Ordinal: 2, Role: "assistant",
Timestamp: "2026-05-21T10:00:00Z", Model: "claude-opus-4-6",
TokenUsage: json.RawMessage(
`{"input_tokens":1,"output_tokens":200}`),
OutputTokens: 200, HasOutputTokens: true,
ClaudeMessageID: "msg-tie", ClaudeRequestID: "req-tie",
},
)
got := querySnapshotRankedRows(t, d, UsageFilter{})
require.Equal(t, [][3]string{
{"solo", "solo", `{"input_tokens":5,"output_tokens":7}`},
{"solo", "solo", `{"input_tokens":6,"output_tokens":8}`},
{"solo", "a-snapshot", `{"input_tokens":1,"output_tokens":200}`},
}, got)
got = querySnapshotRankedRows(t, d, UsageFilter{
From: "2026-05-20", To: "2026-05-20", Timezone: "UTC"})
require.Equal(t, [][3]string{
{"solo", "solo", `{"input_tokens":5,"output_tokens":7}`},
{"solo", "solo", `{"input_tokens":6,"output_tokens":8}`},
{"z-snapshot", "a-snapshot", `{"input_tokens":900,"output_tokens":100}`},
}, got)
}
func TestGetDailyUsage_DedupKeyVariants(t *testing.T) {
d := testDB(t)
require.NoError(t, d.UpsertModelPricing([]ModelPricing{{
-153
View File
@@ -1,10 +1,8 @@
package db
import (
"context"
"crypto/sha256"
"encoding/hex"
"fmt"
"sort"
"strconv"
"strings"
@@ -56,45 +54,6 @@ type worktreeCandidateGroup struct {
sessions []WorktreeCandidateSession
}
// ListArchiveWorktreeCandidates returns the machine/path groups for a
// project selected by (display label, project key) across every visible
// session in the archive, with no Activity date range or filter scoping.
func (db *DB) ListArchiveWorktreeCandidates(
ctx context.Context,
request ArchiveWorktreeCandidateRequest,
) ([]WorktreeReclassificationCandidate, error) {
if strings.TrimSpace(request.ProjectKey) == "" {
return nil, fmt.Errorf("project_key is required")
}
sessions, err := db.archiveWorktreeCandidateSessions(ctx)
if err != nil {
return nil, err
}
labels := make(map[string]struct{})
for _, session := range sessions {
labels[session.project] = struct{}{}
}
projects, err := db.BuildProjectIdentityMap(ctx, sortedSetKeys(labels))
if err != nil {
return nil, err
}
selectedProjects := SelectWorktreeCandidateProjects(
request, labels, projects,
)
if len(selectedProjects) == 0 {
return []WorktreeReclassificationCandidate{}, nil
}
selectedIDs := make([]string, 0, len(sessions))
for _, session := range sessions {
if _, ok := selectedProjects[session.project]; !ok {
continue
}
selectedIDs = append(selectedIDs, session.id)
}
return db.worktreeCandidatesFromSelection(ctx, selectedIDs, selectedProjects)
}
// SelectWorktreeCandidateProjects validates that the requested display label
// identifies the requested opaque project key, then expands the selection to
// raw labels with the same resolved project identity. The display label
@@ -142,66 +101,6 @@ func SelectWorktreeCandidateProjects(
return selected
}
// archiveCandidateSessionRef is the minimal (id, project) pair the
// archive-wide selection query needs; the shared grouping pipeline only
// ever reads a session's ID and project label from the selection step.
type archiveCandidateSessionRef struct {
id, project string
}
// archiveWorktreeCandidateSessions returns every archive-wide visible
// session (deleted_at IS NULL) with no date or relationship-type bound.
// Data inventory counts these same rows, including zero-message sessions,
// so the selected project's session count and its folder groups stay
// reconcilable.
func (db *DB) archiveWorktreeCandidateSessions(
ctx context.Context,
) ([]archiveCandidateSessionRef, error) {
rows, err := db.getReader().QueryContext(ctx, `
SELECT id, project
FROM sessions
WHERE deleted_at IS NULL
ORDER BY id`)
if err != nil {
return nil, fmt.Errorf("querying archive worktree candidate sessions: %w", err)
}
defer rows.Close()
var sessions []archiveCandidateSessionRef
for rows.Next() {
var session archiveCandidateSessionRef
if err := rows.Scan(&session.id, &session.project); err != nil {
return nil, fmt.Errorf("scanning archive worktree candidate session: %w", err)
}
sessions = append(sessions, session)
}
if err := rows.Err(); err != nil {
return nil, fmt.Errorf("iterating archive worktree candidate sessions: %w", err)
}
return sessions, nil
}
// worktreeCandidatesFromSelection runs the shared grouping pipeline
// (snapshot/aggregate/fallback evidence, then deterministic ordering)
// over an already-selected set of session IDs.
// ListArchiveWorktreeCandidates calls it after selecting the archive-wide
// session set.
func (db *DB) worktreeCandidatesFromSelection(
ctx context.Context,
selectedIDs []string,
selectedProjects map[string]struct{},
) ([]WorktreeReclassificationCandidate, error) {
details, err := db.loadWorktreeCandidateSessions(ctx, selectedIDs)
if err != nil {
return nil, err
}
observations, err := db.ListProjectIdentityObservations(
ctx, sortedSetKeys(selectedProjects))
if err != nil {
return nil, err
}
return BuildWorktreeCandidates(details, observations), nil
}
// BuildWorktreeCandidates groups an already-selected set of sessions into
// machine/path worktree reclassification candidates, using snapshot
// evidence first, then compatible aggregate observation evidence, then an
@@ -253,58 +152,6 @@ func BuildWorktreeCandidates(
return result
}
func (db *DB) loadWorktreeCandidateSessions(
ctx context.Context,
ids []string,
) ([]WorktreeCandidateSession, error) {
byID := make(map[string]WorktreeCandidateSession, len(ids))
err := queryChunked(ids, func(chunk []string) error {
placeholders, args := inPlaceholders(chunk)
rows, err := db.getReader().QueryContext(ctx, `
SELECT s.id, s.project, s.machine, s.cwd,
COALESCE(snap.session_id, ''), COALESCE(snap.project, ''),
COALESCE(snap.machine, ''), COALESCE(snap.root_path, ''),
COALESCE(snap.worktree_root_path, ''), COALESCE(snap.key_source, '')
FROM sessions s
LEFT JOIN session_project_identity_snapshots snap
ON snap.session_id = s.id
WHERE s.id IN `+placeholders+` AND s.deleted_at IS NULL
ORDER BY s.id`, args...)
if err != nil {
return fmt.Errorf("querying worktree candidate sessions: %w", err)
}
defer rows.Close()
for rows.Next() {
var session WorktreeCandidateSession
var snapshotSessionID string
if err := rows.Scan(
&session.ID, &session.Project, &session.Machine, &session.Cwd,
&snapshotSessionID, &session.Snapshot.Project,
&session.Snapshot.Machine, &session.Snapshot.RootPath,
&session.Snapshot.WorktreeRootPath, &session.Snapshot.KeySource,
); err != nil {
return fmt.Errorf("scanning worktree candidate session: %w", err)
}
session.HasSnapshot = snapshotSessionID != ""
byID[session.ID] = session
}
if err := rows.Err(); err != nil {
return fmt.Errorf("iterating worktree candidate sessions: %w", err)
}
return nil
})
if err != nil {
return nil, err
}
result := make([]WorktreeCandidateSession, 0, len(byID))
for _, id := range ids {
if session, ok := byID[id]; ok {
result = append(result, session)
}
}
return result, nil
}
func candidateSnapshotRoot(session WorktreeCandidateSession) string {
if !session.HasSnapshot || session.Snapshot.Project != session.Project ||
session.Snapshot.Machine != session.Machine {
+10 -5
View File
@@ -355,7 +355,8 @@ func seedCandidateSession(
func deleteCandidateSnapshot(t *testing.T, d *DB, id string) {
t.Helper()
_, err := d.getWriter().Exec(
`DELETE FROM session_project_identity_snapshots WHERE session_id = ?`, id)
`DELETE FROM source_session_project_identity_snapshots
WHERE source_session_id = ?`, id)
require.NoError(t, err)
}
@@ -365,9 +366,13 @@ func setCandidateSnapshot(
t.Helper()
deleteCandidateSnapshot(t, d, id)
_, err := d.getWriter().Exec(`
INSERT INTO session_project_identity_snapshots (
session_id, project, machine, root_path, worktree_root_path, observed_at
) VALUES (?, ?, ?, ?, ?, ?)`,
id, project, machine, root, worktreeRoot, "2025-06-02T10:00:00Z")
INSERT INTO source_session_project_identity_snapshots (
source_archive_id, source_database_generation, source_session_id,
project, machine, root_path, worktree_root_path, observed_at
)
SELECT source_archive_id, source_database_generation, id,
?, ?, ?, ?, ?
FROM sessions WHERE id = ?`,
project, machine, root, worktreeRoot, "2025-06-02T10:00:00Z", id)
require.NoError(t, err)
}
+8 -3
View File
@@ -78,9 +78,11 @@ func (db *DB) LoadWorktreeMappingPublicationDelta(
SELECT m.id, m.machine, m.path_prefix, m.layout, m.project,
m.original_project, m.enabled, m.created_at, m.updated_at
FROM worktree_project_mapping_changes c
JOIN worktree_project_mappings m
JOIN source_worktree_project_mappings m
ON m.machine = c.machine AND m.path_prefix = c.path_prefix
WHERE c.deleted = 0 AND c.revision > ? AND c.revision <= ?
WHERE m.source_archive_id = (
SELECT value FROM archive_metadata WHERE key = 'archive_id'
) AND c.deleted = 0 AND c.revision > ? AND c.revision <= ?
ORDER BY m.machine, m.path_prefix`,
afterRevision, throughRevision)
if err != nil {
@@ -132,7 +134,10 @@ func (db *DB) ListAllWorktreeProjectMappings(
rows, err := db.getReader().QueryContext(ctx, `
SELECT id, machine, path_prefix, layout, project,
original_project, enabled, created_at, updated_at
FROM worktree_project_mappings
FROM source_worktree_project_mappings
WHERE source_archive_id = (
SELECT value FROM archive_metadata WHERE key = 'archive_id'
)
ORDER BY machine, path_prefix`)
if err != nil {
return nil, fmt.Errorf("listing all worktree mappings: %w", err)
+77 -55
View File
@@ -139,7 +139,7 @@ func worktreePathMatches(prefix string, cwd string) bool {
func scanWorktreeMapping(rows *sql.Rows) (WorktreeProjectMapping, error) {
var m WorktreeProjectMapping
var enabled int
var enabled bool
if err := rows.Scan(
&m.ID,
&m.Machine,
@@ -156,13 +156,13 @@ func scanWorktreeMapping(rows *sql.Rows) (WorktreeProjectMapping, error) {
if m.Layout == "" {
m.Layout = WorktreeMappingLayoutExplicit
}
m.Enabled = enabled != 0
m.Enabled = enabled
return m, nil
}
func scanWorktreeMappingRow(row rowScanner) (WorktreeProjectMapping, error) {
var m WorktreeProjectMapping
var enabled int
var enabled bool
if err := row.Scan(
&m.ID,
&m.Machine,
@@ -179,7 +179,7 @@ func scanWorktreeMappingRow(row rowScanner) (WorktreeProjectMapping, error) {
if m.Layout == "" {
m.Layout = WorktreeMappingLayoutExplicit
}
m.Enabled = enabled != 0
m.Enabled = enabled
return m, nil
}
@@ -190,8 +190,10 @@ func (db *DB) ListWorktreeProjectMappings(
rows, err := db.getReader().QueryContext(ctx, `
SELECT id, machine, path_prefix, layout, project, original_project,
enabled, created_at, updated_at
FROM worktree_project_mappings
WHERE machine = ?
FROM source_worktree_project_mappings
WHERE source_archive_id = (
SELECT value FROM archive_metadata WHERE key = 'archive_id'
) AND machine = ?
ORDER BY path_prefix`, strings.TrimSpace(machine))
if err != nil {
return nil, fmt.Errorf("listing worktree mappings: %w", err)
@@ -232,10 +234,29 @@ func (db *DB) CreateWorktreeProjectMapping(
db.mu.Lock()
defer db.mu.Unlock()
res, err := db.getWriter().ExecContext(ctx, `
INSERT INTO worktree_project_mappings
(machine, path_prefix, layout, project, original_project, enabled)
VALUES (?, ?, ?, ?, ?, ?)`,
tx, err := db.getWriter().BeginTx(ctx, nil)
if err != nil {
return WorktreeProjectMapping{}, fmt.Errorf("beginning worktree mapping create: %w", err)
}
defer func() { _ = tx.Rollback() }()
if err := tx.QueryRowContext(ctx, `
SELECT COALESCE(MAX(id), 0) + 1
FROM source_worktree_project_mappings
WHERE source_archive_id = (
SELECT value FROM archive_metadata WHERE key = 'archive_id'
)`,
).Scan(&normalized.ID); err != nil {
return WorktreeProjectMapping{}, fmt.Errorf("allocating worktree mapping id: %w", err)
}
_, err = tx.ExecContext(ctx, `
INSERT INTO source_worktree_project_mappings
(id, source_archive_id, machine, path_prefix, layout,
project, original_project, enabled, created_at, updated_at)
SELECT ?, value, ?, ?, ?, ?, ?, ?,
strftime('%Y-%m-%dT%H:%M:%fZ','now'),
strftime('%Y-%m-%dT%H:%M:%fZ','now')
FROM archive_metadata WHERE key = 'archive_id'`,
normalized.ID,
normalized.Machine,
normalized.PathPrefix,
normalized.Layout,
@@ -249,7 +270,9 @@ func (db *DB) CreateWorktreeProjectMapping(
}
return WorktreeProjectMapping{}, fmt.Errorf("creating worktree mapping: %w", err)
}
normalized.ID, _ = res.LastInsertId()
if err := tx.Commit(); err != nil {
return WorktreeProjectMapping{}, fmt.Errorf("committing worktree mapping create: %w", err)
}
return db.getWorktreeProjectMappingLocked(ctx, normalized.Machine, normalized.ID)
}
@@ -273,7 +296,7 @@ func (db *DB) UpdateWorktreeProjectMapping(
defer db.mu.Unlock()
res, err := db.getWriter().ExecContext(ctx, `
UPDATE worktree_project_mappings
UPDATE source_worktree_project_mappings
SET path_prefix = ?,
layout = ?,
project = ?,
@@ -283,7 +306,9 @@ func (db *DB) UpdateWorktreeProjectMapping(
END,
enabled = ?,
updated_at = strftime('%Y-%m-%dT%H:%M:%fZ','now')
WHERE id = ? AND machine = ?`,
WHERE source_archive_id = (
SELECT value FROM archive_metadata WHERE key = 'archive_id'
) AND id = ? AND machine = ?`,
normalized.PathPrefix,
normalized.Layout,
normalized.Project,
@@ -314,7 +339,10 @@ func (db *DB) DeleteWorktreeProjectMapping(
defer db.mu.Unlock()
res, err := db.getWriter().ExecContext(ctx,
`DELETE FROM worktree_project_mappings WHERE id = ? AND machine = ?`,
`DELETE FROM source_worktree_project_mappings
WHERE source_archive_id = (
SELECT value FROM archive_metadata WHERE key = 'archive_id'
) AND id = ? AND machine = ?`,
id,
strings.TrimSpace(machine),
)
@@ -336,8 +364,10 @@ func (db *DB) getWorktreeProjectMappingLocked(
row := db.getWriter().QueryRowContext(ctx, `
SELECT id, machine, path_prefix, layout, project, original_project,
enabled, created_at, updated_at
FROM worktree_project_mappings
WHERE id = ? AND machine = ?`,
FROM source_worktree_project_mappings
WHERE source_archive_id = (
SELECT value FROM archive_metadata WHERE key = 'archive_id'
) AND id = ? AND machine = ?`,
id,
machine,
)
@@ -356,8 +386,10 @@ func (db *DB) GetWorktreeProjectMapping(
row := db.getReader().QueryRowContext(ctx, `
SELECT id, machine, path_prefix, layout, project, original_project,
enabled, created_at, updated_at
FROM worktree_project_mappings
WHERE id = ?`, id)
FROM source_worktree_project_mappings
WHERE source_archive_id = (
SELECT value FROM archive_metadata WHERE key = 'archive_id'
) AND id = ?`, id)
return scanWorktreeMappingRow(row)
}
@@ -369,7 +401,10 @@ func (db *DB) ListWorktreeProjectMappingMachines(
rows, err := db.getReader().QueryContext(ctx, `
SELECT machine FROM sessions WHERE deleted_at IS NULL AND machine != ''
UNION
SELECT machine FROM worktree_project_mappings WHERE machine != ''
SELECT machine FROM source_worktree_project_mappings
WHERE source_archive_id = (
SELECT value FROM archive_metadata WHERE key = 'archive_id'
) AND machine != ''
ORDER BY machine`)
if err != nil {
return nil, fmt.Errorf("listing worktree mapping machines: %w", err)
@@ -398,8 +433,10 @@ func (db *DB) ListActiveWorktreeProjectMappingMachines(
) ([]string, error) {
rows, err := db.getReader().QueryContext(ctx, `
SELECT DISTINCT machine
FROM worktree_project_mappings
WHERE enabled = 1 AND machine != ''
FROM source_worktree_project_mappings
WHERE source_archive_id = (
SELECT value FROM archive_metadata WHERE key = 'archive_id'
) AND enabled = 1 AND machine != ''
ORDER BY machine`)
if err != nil {
return nil, fmt.Errorf("listing active worktree mapping machines: %w", err)
@@ -476,32 +513,8 @@ func (db *DB) CopyWorktreeProjectMappingsFrom(sourcePath string) error {
}
defer func() { _ = tx.Rollback() }()
if oldDBHasTable(ctx, tx, "worktree_project_mappings") {
layoutSelect := "'" + WorktreeMappingLayoutExplicit + "'"
if oldDBHasColumn(ctx, tx, "worktree_project_mappings", "layout") {
layoutSelect = "layout"
}
originalProjectSelect := "''"
if oldDBHasColumn(ctx, tx, "worktree_project_mappings", "original_project") {
originalProjectSelect = "original_project"
}
if _, err := tx.ExecContext(ctx, `
INSERT INTO main.worktree_project_mappings
(machine, path_prefix, layout, project, original_project,
enabled, created_at, updated_at)
SELECT machine, replace(path_prefix, char(92), '/'),
`+layoutSelect+`, project,
`+originalProjectSelect+`, enabled, created_at, updated_at
FROM old_db.worktree_project_mappings
WHERE TRUE
ON CONFLICT(machine, path_prefix) DO UPDATE SET
original_project = CASE
WHEN worktree_project_mappings.original_project = ''
THEN excluded.original_project
ELSE worktree_project_mappings.original_project
END`); err != nil {
return fmt.Errorf("copying worktree project mappings: %w", err)
}
if err := copyWorktreeProjectMappingsFromAttached(ctx, tx, false); err != nil {
return err
}
return tx.Commit()
@@ -570,8 +583,10 @@ func (db *DB) activeWorktreeProjectMappings(
rows, err := db.getReader().QueryContext(ctx, `
SELECT id, machine, path_prefix, layout, project, original_project,
enabled, created_at, updated_at
FROM worktree_project_mappings
WHERE machine = ? AND enabled = 1
FROM source_worktree_project_mappings
WHERE source_archive_id = (
SELECT value FROM archive_metadata WHERE key = 'archive_id'
) AND machine = ? AND enabled = 1
ORDER BY length(path_prefix) DESC, path_prefix`,
strings.TrimSpace(machine),
)
@@ -675,8 +690,10 @@ func loadActiveWorktreeMappingsTx(
rows, err := tx.QueryContext(ctx, `
SELECT id, machine, path_prefix, layout, project, original_project,
enabled, created_at, updated_at
FROM worktree_project_mappings
WHERE machine = ? AND enabled = 1
FROM source_worktree_project_mappings
WHERE source_archive_id = (
SELECT value FROM archive_metadata WHERE key = 'archive_id'
) AND machine = ? AND enabled = 1
ORDER BY length(path_prefix) DESC, path_prefix`,
machine,
)
@@ -694,8 +711,10 @@ func loadActiveWorktreeMappingsByMachineTx(
rows, err := tx.QueryContext(ctx, `
SELECT id, machine, path_prefix, layout, project, original_project,
enabled, created_at, updated_at
FROM worktree_project_mappings
WHERE enabled = 1
FROM source_worktree_project_mappings
WHERE source_archive_id = (
SELECT value FROM archive_metadata WHERE key = 'archive_id'
) AND enabled = 1
ORDER BY machine, length(path_prefix) DESC, path_prefix`,
)
if err != nil {
@@ -1168,6 +1187,9 @@ func (db *DB) applyWorktreeProjectMappingsToSessionsByPath(
func isSQLiteUniqueConstraint(err error) bool {
var sqliteErr sqlite3.Error
return errors.As(err, &sqliteErr) &&
sqliteErr.ExtendedCode == sqlite3.ErrConstraintUnique
if !errors.As(err, &sqliteErr) {
return false
}
return sqliteErr.ExtendedCode == sqlite3.ErrConstraintUnique ||
sqliteErr.ExtendedCode == sqlite3.ErrConstraintPrimaryKey
}
+14 -8
View File
@@ -146,10 +146,14 @@ func TestSchemaColumnMigrationAddsWorktreeOriginalProject(t *testing.T) {
migrated, err := Open(path)
require.NoError(t, err, "open and migrate archive")
defer migrated.Close()
mappings, err := migrated.ListWorktreeProjectMappings(ctx, "host-a.example")
require.NoError(t, err, "list migrated mapping")
require.Len(t, mappings, 1)
assert.Empty(t, mappings[0].OriginalProject,
var originalProject string
err = migrated.getReader().QueryRowContext(ctx, `
SELECT original_project FROM worktree_project_mappings
WHERE machine = 'host-a.example'
AND path_prefix = '/srv/worktrees/service'`,
).Scan(&originalProject)
require.NoError(t, err, "read migrated legacy mapping")
assert.Empty(t, originalProject,
"legacy mappings default original project to empty")
}
@@ -1099,8 +1103,8 @@ func TestApplyWorktreeProjectMappingToSessionReconcilesOnlyMovedIdentityKey(
var sourceSnapshots int
require.NoError(t, d.getReader().QueryRowContext(ctx, `
SELECT COUNT(*) FROM session_project_identity_snapshots
WHERE project = ? AND session_id IN (?, ?)`,
SELECT COUNT(*) FROM source_session_project_identity_snapshots
WHERE project = ? AND source_session_id IN (?, ?)`,
"source_project", "retained", "moved",
).Scan(&sourceSnapshots))
assert.Equal(t, 2, sourceSnapshots,
@@ -1300,9 +1304,11 @@ func TestWorktreeProjectMappingsFinalMetadataCopyRefreshesStalePrecopy(
require.NoError(t, srcDB.CloseConnections(), "CloseConnections src")
_, err = dstDB.getWriter().ExecContext(ctx, `
UPDATE worktree_project_mappings
UPDATE source_worktree_project_mappings
SET updated_at = '9999-12-31T23:59:59.999Z'
WHERE machine = ? AND path_prefix = ?`,
WHERE source_archive_id = (
SELECT value FROM archive_metadata WHERE key = 'archive_id'
) AND machine = ? AND path_prefix = ?`,
"laptop",
prefix,
)

Some files were not shown because too many files have changed in this diff Show More