Matt Aitken
c2c6e5c705
## Summary Runs created for a Session were triggered without a realtime streams version, so they fell through to the `realtimeStreamsVersion` column default of `v1`. A Session's own `.in` / `.out` channels are always `v2`, so any run-scoped `streams.append()` or `streams.pipe()` call made inside a session run wrote to a different backend than the session it belongs to, and stayed there for the life of the run. The API trigger routes were never affected. They call `determineRealtimeStreamsVersion` with the client's `x-trigger-realtime-streams-version` header and always pass an explicit value, so a current SDK asking for v2 gets it. Only the internal callers that build trigger options by hand were leaning on the column default, which no env var can influence because that path never calls the resolver at all. ## The version resolver Fixing the call site exposed a second problem in `determineRealtimeStreamsVersion`. Its two paths disagreed: an explicit `v2` was checked against the S2 configuration first, but when the caller expressed no preference it returned `REALTIME_STREAMS_DEFAULT_VERSION` verbatim with no check. A deployment that set the default to `v2` without configuring S2 therefore stamped runs `v2`, nothing failed at trigger time, and every later read or write against those runs' streams threw `Realtime streams v2 is required for this run but S2 configuration is missing` for the life of the run. Both paths now resolve through one pure function that takes its configuration rather than reading `env`: ```ts const requested = streamVersion ?? config.defaultVersion; if (requested !== "v2") return "v1"; const hasCredentials = Boolean(config.accessToken) || config.skipAccessTokens; return hasCredentials && Boolean(config.basin) ? "v2" : "v1"; ``` ## The basin requirement `resolveStreamBasin` resolves run, session and organization basins ahead of the global setting, so a deployment that provisions a basin per organization can serve v2 with no global basin at all. Gating purely on the global setting would degrade every run there to `v1`. `determineRealtimeStreamsVersion` therefore takes an optional organization basin, and every caller that holds one passes it, including the session path: ```ts basin: organizationBasinName ?? env.REALTIME_STREAMS_S2_BASIN, ``` This is deliberately the resolved basin and not the `REALTIME_STREAMS_PER_ORG_BASINS_ENABLED` flag. The flag says the feature is on, not that a given organization has been provisioned, and provisioning happens out of band. Keying off the flag would stamp `v2` on runs for unprovisioned organizations, recreating the failure this removes. **This widens behaviour for explicit `v2` requests**, which previously required the global basin: a provisioned organization on a per-org deployment now resolves `v2` where it used to get `v1`. That is intentional, and it makes every path agree. ## Scope Only newly created runs change. A run already stamped `v1` keeps that version for its lifetime by design, since readers resolve the backend from the same column and its existing streams have to stay readable. Scheduled runs reach the same column default through `scheduleEngine.server.ts` and are deliberately left alone: that one is a policy question about `REALTIME_STREAMS_DEFAULT_VERSION` rather than an inconsistency inside a single feature. ## Verification A full-stack e2e boots the real webapp plus Postgres, Redis and s2-lite, creates a Session through the public API so the run comes from the real trigger path, appends records the way `streams.append()` does, and asserts three things at once: the version stamped on the run, that the payload is readable from S2, and that no key exists in Redis. It appends at a realistic record size so the route's body cap and S2's per-record cap are both exercised. Reverting the session-path change flips all three observations, so it fails against the old behaviour rather than passing vacuously. Unit tests cover the resolver matrix, including organization-basin-only and credential-only configurations; two of them fail against the previous resolver. Also verified by hand against a local stack: a real `chat.agent` session run writing 8 records of 250KB through `streams.append()` put 2,049,072 bytes into S2 with no Redis key, while the same agent with the session-path change removed put 2,102,360 bytes into Redis and nothing into S2.
Test containers
Vitest utilities for writing tests against real Postgres, Prisma, Redis and ClickHouse - we don't mock
(see the root CLAUDE.md), we boot containers. Also exposes a duration-weighted shard sequencer for
splitting slow suites across CI shards.
Choosing a fixture
Most tests share one set of containers per vitest worker (booted once, reset between tests) - this is much faster than a container per test. Reach for an isolated variant only when a test needs it.
| Fixture | Postgres | Redis | ClickHouse | Use for |
|---|---|---|---|---|
redisTest |
- | shared | - | redis-only tests |
postgresTest |
shared (clone) | - | - | db-only tests |
containerTest |
shared (clone) | shared | shared | the default - needs all three |
isolatedRedisTest |
- | per-test | - | background redis work (see below) |
containerTestWithIsolatedRedis |
shared (clone) | per-test | shared | background redis work + db/clickhouse |
replicationContainerTest |
per-test | per-test | shared | Postgres→ClickHouse logical replication |
"shared (clone)" = one Postgres per worker with a template database; each test gets a fast CREATE DATABASE ... TEMPLATE clone, so schema isn't re-pushed per test.
The background-work gotcha
If a test spawns work that outlives the test body - a RunEngine, a redis-worker Worker, a
BatchQueue - and that work isn't fully drained before the test ends, you must use an isolated
redis fixture (isolatedRedisTest / containerTestWithIsolatedRedis).
On the shared fixture, the leaked background loop keeps polling the one worker-scoped redis after the
test's clients close, bleeding into the next test. The symptom is an intermittent "Connection is closed" error or a test that hangs until its timeout. FLUSHALL between tests does not fix this -
it clears data, not live connections/loops, so per-test key prefixes won't help either. A plain
db/redis test with no lingering background work is fine on the shared fixtures.
Sharding (./sequencer)
CI splits the slow suites with vitest --shard=i/N. DurationShardingSequencer replaces vitest's
default file-count split with a duration-weighted one: it reads test-timings.json at the repo root
({ "<repo-relative path>": <ms> }) and greedily bin-packs files so each shard does roughly equal
work, not an equal number of files. The packing is deterministic, so every shard computes the same
bins and runs each file exactly once.
Configs opt in via:
import { DurationShardingSequencer } from "@internal/testcontainers/sequencer";
// in defineConfig:
test: {
sequence: {
sequencer: DurationShardingSequencer,
},
}
Adding tests - nothing to do
New test files are discovered by vitest's glob and sharded automatically. A file with no entry in
test-timings.json is given the median duration as a fallback, so it's still placed on exactly one
shard - correctness never depends on the timings being present or current.
What the timings affect is balance. A new heavy test estimated at the median can be under-weighted and land on an already-full shard, making that shard slower. There's headroom between the current makespan and the CI budget to absorb this, so it tolerates drift - but if a shard creeps toward the budget, refresh the timings.
Refreshing test-timings.json
Measure each shard with the JSON reporter and write per-file endTime - startTime (ms), keyed by
repo-relative path, back into test-timings.json. Set GITHUB_ACTIONS=true so suites that
skipIf(CI) are excluded, matching what actually runs on CI:
GITHUB_ACTIONS=true pnpm exec vitest run --reporter=json --outputFile=/tmp/run.json
Stale entries for deleted/renamed files are harmless (they're simply ignored). This is a periodic chore, not a per-PR one.