Eric Allam
f9d57d3bd5
## Summary Adds a second backend for the realtime runs feed (`useRealtimeRun`, `subscribeToRunsWithTag`, `subscribeToBatch`), built to stay healthy when a single busy environment has many subscribers watching many runs at once. It is gated behind a feature flag with the existing backend as the default, so nothing changes for users until it is enabled per environment. ## Design A run change is published once, as a small self-describing record, to a single per-environment channel. Every feed is then a predicate over that one stream rather than owning a channel: - A per-instance router indexes the currently-held feeds by run, tag, and batch. When a run changes it hydrates the affected rows once and serializes them once, then fans the result to every matching feed. One hot shared tag watched by many subscribers costs a single database query and serialize, not one per subscriber. - Feeds that don't match a change are never woken, wake delivery per environment is coalesced on a leading edge (250ms default) so a burst of changes costs one wake, and cold reads coalesce onto a single short-TTL-cached resolve. - An admission gate bounds how many cold ClickHouse resolves run concurrently, so a mass reconnect across many distinct filters queues instead of stampeding the database. - Changes that land while a client is between long-polls are delivered on its next poll instead of waiting for the periodic backstop: each environment buffers its recent change records, subscriptions linger briefly after the last feed closes, and a newly-armed poll replays exactly the connection's gap. - The per-connection replay cursors behind that are shared across instances via Redis (a single timestamp each), so a poll landing on a different instance behind the load balancer still reads the connection's true gap instead of falling back to a cold resolve. Cursor reads have a bounded deadline and degrade to the cold-read path on any Redis trouble. - Tag subscriptions with multiple tags match runs carrying all of the tags, mirroring the existing backend's filter semantics, and live long-polls hold for about 20 seconds to match its cadence. - The per-environment channel supports Redis Cluster sharded pub/sub, so the wake path scales horizontally across shards by environment. - The backend reports its health through OpenTelemetry metrics (delivery lag, poll resolution paths, backstop outcomes, replay and cursor-store activity), with a provisioned Grafana dashboard for local development. Everything is behind the feature flag and tunable via env vars; the existing backend remains the default.
33 lines
784 B
JSON
33 lines
784 B
JSON
{
|
|
"name": "@internal/testcontainers",
|
|
"private": true,
|
|
"version": "0.0.1",
|
|
"main": "./src/index.ts",
|
|
"types": "./src/index.ts",
|
|
"exports": {
|
|
".": "./src/index.ts",
|
|
"./webapp": "./src/webapp.ts",
|
|
"./sequencer": {
|
|
"types": "./src/sequencer.d.cts",
|
|
"default": "./src/sequencer.cjs"
|
|
}
|
|
},
|
|
"dependencies": {
|
|
"@clickhouse/client": "^1.11.1",
|
|
"@opentelemetry/api": "^1.9.1",
|
|
"@trigger.dev/database": "workspace:*",
|
|
"ioredis": "~5.6.0"
|
|
},
|
|
"devDependencies": {
|
|
"@testcontainers/postgresql": "^11.14.0",
|
|
"@testcontainers/redis": "^11.14.0",
|
|
"@trigger.dev/core": "workspace:*",
|
|
"std-env": "^3.9.0",
|
|
"testcontainers": "^11.14.0",
|
|
"tinyexec": "^0.3.0"
|
|
},
|
|
"scripts": {
|
|
"typecheck": "tsc --noEmit"
|
|
}
|
|
}
|