Wes McKinney
43c1e826f8
PostgreSQL-backed agentsview now supports the `semantic` and `hybrid` search modes that previously existed only on SQLite. Embeddings are never computed server-side: `pg push` replicates the local active embedding generation from `vectors.db` into pgvector tables, and `pg serve` (plus every `--pg` direct-read command) answers semantic and hybrid queries from PostgreSQL instead of returning HTTP 501. ## Push path - `pg push` gains a vector phase after sessions/messages: it copies the active generation into per-generation `halfvec(N)` chunk tables with HNSW cosine indexes, plus a shared `vector_documents` mirror so snippets and hit anchoring reuse exact local semantics. Opt out with `--no-vectors` or `push_vectors = false` under `[pg]`. - Generations are keyed by the existing config fingerprint, so machines with identical `[vector.embeddings]` configs share one generation and a serving host searches the union. - Delta state lives in a PG-side `vector_push_state` table written transactionally with the data, so a PG reset is self-healing. The per-session aggregate hash covers full doc row identity, so metadata-only changes re-push. - Doc replacement is park-to-sentinel (park existing rows at negative ordinals, upsert onto the freed slots, delete still-parked rows), because `doc_key` is stable while `UNIQUE(session_id, ordinal)` slots shift. Eviction is scoped to sessions the pusher owns per its owner marker (probed with `FOR UPDATE`) and, for filtered pushes, by local project membership. - Databases without pgvector (CockroachDB, missing extension package, insufficient privilege) degrade gracefully: one notice, vector phase skipped, session/message sync unaffected, semantic search keeps returning 501. ## Read path - `internal/postgres` implements the same `db.VectorSearcher` seam; the RRF merge, unit fusion keys, subordinate penalty, and snippet helpers are exported from `internal/db` and reused, so mode behavior matches SQLite by construction. Invalid-input parity is pinned by a 10-case cross-backend test. - Startup fingerprints the host's encoder config and looks up a matching PG generation; a miss keeps semantic at 501 with an error naming the expected and present fingerprints. - KNN fetches exactly k chunks and sets `hnsw.ef_search = clamp(k, 40, 1000)` per query (with an `iterative_scan` fallback probe past pgvector's ceiling) so large fetches aren't silently capped at the default candidate pool of 40. - Hybrid fuses the vector leg with an ILIKE keyword leg resolved to document units via `vector_documents`. Known limitation: the PG keyword leg is recency-ordered rather than BM25-ranked (documented). ## Maintenance and infra - New `pg vectors list` / `pg vectors drop <id>` commands for inspecting and retiring generations. - The compose test image and CI service container switch to `pgvector/pgvector` images. - Docs: PostgreSQL sections in semantic-search and pg-sync pages, CLI reference entries, and a maintainer-level "PostgreSQL replica" internals section covering the push/read invariants (including the deliberate cross-generation `vector_documents` sharing). ## Where to look - `internal/postgres/vector_push.go` — delta, ownership, and park-to-sentinel replacement (the subtlest code in the PR) - `internal/postgres/vector_search.go` — KNN, ef_search tuning, hydration - `internal/postgres/search_content_hybrid.go` — keyword-leg unit resolution and fusion - `internal/vector/export.go` — the vectors.db export API and its fail-closed version gate 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Wes McKinney <wesm@users.noreply.github.com>
36 lines
951 B
YAML
36 lines
951 B
YAML
# Docker Compose file for integration testing
|
|
# Usage:
|
|
# make postgres-up / make test-postgres / make postgres-down
|
|
# make ssh-up / make test-ssh / make ssh-down
|
|
|
|
services:
|
|
postgres:
|
|
image: pgvector/pgvector:pg16
|
|
environment:
|
|
POSTGRES_USER: agentsview_test
|
|
POSTGRES_PASSWORD: agentsview_test_password
|
|
POSTGRES_DB: agentsview_test
|
|
ports:
|
|
- "5433:5432" # Non-standard port to avoid conflict with local postgres
|
|
healthcheck:
|
|
test: ["CMD-SHELL", "pg_isready -U agentsview_test -d agentsview_test"]
|
|
interval: 2s
|
|
timeout: 5s
|
|
retries: 10
|
|
tmpfs:
|
|
- /var/lib/postgresql/data # Use tmpfs for faster tests, no persistence needed
|
|
|
|
sshd:
|
|
build:
|
|
context: .
|
|
dockerfile: testdata/ssh/Dockerfile
|
|
ports:
|
|
- "2222:22"
|
|
healthcheck:
|
|
test: ["CMD-SHELL", "nc -z localhost 22"]
|
|
interval: 2s
|
|
timeout: 5s
|
|
retries: 10
|
|
tmpfs:
|
|
- /tmp
|