kenn-io--agentsview/docs/internal/background-sync-efficiency.md

Background Sync Efficiency

This document records the runtime and cost-model contracts that keep active session writers from turning into unbounded background work. The implementation lives primarily in internal/sync/watcher.go, internal/parser/codex_cursor.go, and the provider incremental path in internal/sync/engine.go.

Watcher runtime contract

• Production watcher events use a 500 ms first-event batching window. Later events join the pending set without postponing its deadline.
• Watcher callback start times are at least five seconds apart. One worker runs callbacks serially while the fsnotify loop continues draining events and errors, so a long sync cannot block event intake or overlap another sync.
• An idle watcher has no running timer or ticker. The first relevant event creates the next one-shot timer.
• Each pending or in-flight batch retains at most 8,192 unique paths and 2 MiB of path-string bytes. At most one batch is in flight while one more accumulates. Entry count separately bounds map and slice overhead.
• Exceeding either batch limit replaces its individual paths with one explicit full-sync marker. The worker clears event-sensitive freshness caches and force-verifies every discovered file under the same serialization, dispatch-floor, cancellation, and shutdown rules, so overflow bounds memory without losing same-stat source changes.
• Shutdown discards pending paths and waits only for an already-running callback. Normal discovery on the next startup recovers discarded changes.

HTTP mirror journal contract

Persistent HTTP mirrors use the same 8,192-entry and 2 MiB path-byte limits for their crash-recovery journal. Crossing either limit collapses the path set to a full-import marker instead of retaining unbounded path data. Unlike an in-memory watch batch, the marker is durable and remains until database processing and remote skip-cache persistence complete.

For a proven changed source, classification, fingerprinting, parsing, and database writes are independent of unrelated mirror cardinality. An unproven path may widen work only to its owning provider's configured roots. Full archive transfer caused by the half-manifest heuristic does not widen this import scope.

Codex append cursor contract

The Codex provider factory owns one in-memory cursor cache shared by its per-source provider instances. Its lifetime is the sync engine's lifetime; it is not persisted across daemon restarts.

Each cursor is keyed by the cleaned physical path, exact safe byte offset, and inode/device identity where the platform exposes them. The cache is an LRU bounded to 256 entries and 2 MiB of estimated retained data. It contains compact continuation state only: never parsed messages, raw JSON lines, complete prompt bodies, file contents, or open file descriptors.

The database's committed source offset is the cursor commit token. A parse may stage old- and new-offset entries, but only an exact offset from the next database request is eligible. A failed database write therefore retries from the old cursor; an unreachable staged entry is eventually evicted.

Every nonzero resume offset must immediately follow a newline. Incremental parsing commits only complete, valid, newline-terminated JSONL records. Partial records and valid JSON at a newline-less EOF are retried or force a full parse; they are never published as safe cursor boundaries.

Truncation, known file-identity replacement, manual or project refreshes, session_index.jsonl title changes, and records that retroactively update stored messages all fall back to an authoritative full replacement. Safe incremental writes preserve the index-folded mtime and lifecycle-derived termination status alongside message and token aggregates.

Append-only limitation

Cursor correctness assumes that growth is append-only. A same-inode file can grow after bytes inside its already-committed prefix have been rewritten. Size, identity, and boundary checks do not detect that case, and the current full-source fingerprint is not compared with a separately verified stored prefix before incremental parsing. Closing this gap would require rolling hash state or explicit prefix verification and remains deferred.

Cost model and regression evidence

A warm Codex cursor makes continuation-state parsing scale with appended records rather than transcript history. End-to-end append sync is still O(file): the provider's Fingerprint hashes the complete source and the engine's ComputeFileHashPrefix hashes through the newly committed offset.

• BenchmarkCodexIncrementalCursor in internal/parser compares cold prefix reconstruction with the exact warm cursor. It is diagnostic because internal/parser is not in BENCH_GATE_PACKAGES.
• BenchmarkCodexIncrementalSyncReads in internal/sync measures the warm tail between the two remaining linear reads. It is PR-gated because internal/sync is in BENCH_GATE_PACKAGES.

The maintained behavioral gate inventory is in Performance Gates.