Eric Allam
f9d57d3bd5
## Summary Adds a second backend for the realtime runs feed (`useRealtimeRun`, `subscribeToRunsWithTag`, `subscribeToBatch`), built to stay healthy when a single busy environment has many subscribers watching many runs at once. It is gated behind a feature flag with the existing backend as the default, so nothing changes for users until it is enabled per environment. ## Design A run change is published once, as a small self-describing record, to a single per-environment channel. Every feed is then a predicate over that one stream rather than owning a channel: - A per-instance router indexes the currently-held feeds by run, tag, and batch. When a run changes it hydrates the affected rows once and serializes them once, then fans the result to every matching feed. One hot shared tag watched by many subscribers costs a single database query and serialize, not one per subscriber. - Feeds that don't match a change are never woken, wake delivery per environment is coalesced on a leading edge (250ms default) so a burst of changes costs one wake, and cold reads coalesce onto a single short-TTL-cached resolve. - An admission gate bounds how many cold ClickHouse resolves run concurrently, so a mass reconnect across many distinct filters queues instead of stampeding the database. - Changes that land while a client is between long-polls are delivered on its next poll instead of waiting for the periodic backstop: each environment buffers its recent change records, subscriptions linger briefly after the last feed closes, and a newly-armed poll replays exactly the connection's gap. - The per-connection replay cursors behind that are shared across instances via Redis (a single timestamp each), so a poll landing on a different instance behind the load balancer still reads the connection's true gap instead of falling back to a cold resolve. Cursor reads have a bounded deadline and degrade to the cold-read path on any Redis trouble. - Tag subscriptions with multiple tags match runs carrying all of the tags, mirroring the existing backend's filter semantics, and live long-polls hold for about 20 seconds to match its cadence. - The per-environment channel supports Redis Cluster sharded pub/sub, so the wake path scales horizontally across shards by environment. - The backend reports its health through OpenTelemetry metrics (delivery lag, poll resolution paths, backstop outcomes, replay and cursor-store activity), with a provisioned Grafana dashboard for local development. Everything is behind the feature flag and tunable via env vars; the existing backend remains the default.
33 lines
910 B
JSON
33 lines
910 B
JSON
{
|
|
"name": "supervisor",
|
|
"private": true,
|
|
"version": "0.0.1",
|
|
"main": "dist/index.js",
|
|
"type": "module",
|
|
"scripts": {
|
|
"build": "tsc",
|
|
"dev": "tsx --require dotenv/config --watch src/index.ts || (echo '!! Remember to run: nvm use'; exit 1)",
|
|
"start": "node dist/index.js",
|
|
"test:run": "vitest --no-file-parallelism --run",
|
|
"test:watch": "vitest --no-file-parallelism",
|
|
"typecheck": "tsc --noEmit"
|
|
},
|
|
"dependencies": {
|
|
"@aws-sdk/client-ecr": "^3.839.0",
|
|
"@internal/compute": "workspace:*",
|
|
"@kubernetes/client-node": "^1.0.0",
|
|
"@trigger.dev/core": "workspace:*",
|
|
"dockerode": "^4.0.6",
|
|
"ioredis": "~5.6.0",
|
|
"p-limit": "^6.2.0",
|
|
"prom-client": "^15.1.0",
|
|
"socket.io": "4.7.4",
|
|
"std-env": "^3.8.0",
|
|
"zod": "3.25.76"
|
|
},
|
|
"devDependencies": {
|
|
"@internal/testcontainers": "workspace:*",
|
|
"@types/dockerode": "^3.3.33"
|
|
}
|
|
}
|