## Summary
Adds a second backend for the realtime runs feed (`useRealtimeRun`,
`subscribeToRunsWithTag`, `subscribeToBatch`), built to stay healthy
when a single busy environment has many subscribers watching many runs
at once. It is gated behind a feature flag with the existing backend as
the default, so nothing changes for users until it is enabled per
environment.
## Design
A run change is published once, as a small self-describing record, to a
single per-environment channel. Every feed is then a predicate over that
one stream rather than owning a channel:
- A per-instance router indexes the currently-held feeds by run, tag,
and batch. When a run changes it hydrates the affected rows once and
serializes them once, then fans the result to every matching feed. One
hot shared tag watched by many subscribers costs a single database query
and serialize, not one per subscriber.
- Feeds that don't match a change are never woken, wake delivery per
environment is coalesced on a leading edge (250ms default) so a burst of
changes costs one wake, and cold reads coalesce onto a single
short-TTL-cached resolve.
- An admission gate bounds how many cold ClickHouse resolves run
concurrently, so a mass reconnect across many distinct filters queues
instead of stampeding the database.
- Changes that land while a client is between long-polls are delivered
on its next poll instead of waiting for the periodic backstop: each
environment buffers its recent change records, subscriptions linger
briefly after the last feed closes, and a newly-armed poll replays
exactly the connection's gap.
- The per-connection replay cursors behind that are shared across
instances via Redis (a single timestamp each), so a poll landing on a
different instance behind the load balancer still reads the connection's
true gap instead of falling back to a cold resolve. Cursor reads have a
bounded deadline and degrade to the cold-read path on any Redis trouble.
- Tag subscriptions with multiple tags match runs carrying all of the
tags, mirroring the existing backend's filter semantics, and live
long-polls hold for about 20 seconds to match its cadence.
- The per-environment channel supports Redis Cluster sharded pub/sub, so
the wake path scales horizontally across shards by environment.
- The backend reports its health through OpenTelemetry metrics (delivery
lag, poll resolution paths, backstop outcomes, replay and cursor-store
activity), with a provisioned Grafana dashboard for local development.
Everything is behind the feature flag and tunable via env vars; the
existing backend remains the default.
## Summary
Two papercuts new contributors hit running this repo locally:
1. Fresh clones default to v1 (Redis-only) realtime streams, so Sessions
and `chat.agent` error with `"S2 configuration is missing"`, even though
the `s2` service is already in `docker/docker-compose.yml` and pre-seeds
a `trigger-local` basin. Wire `REALTIME_STREAMS_S2_*` to it in
`.env.example` so the new-contributor flow just works. (Also drop the s2
healthcheck: the image is distroless, so the `wget` check always reports
unhealthy.)
2. Two clones can't both run `pnpm run docker` because ports, project
name, and container names are all hardcoded. Parameterize every host
port as `${VAR:-default}`, drive the project name via
`COMPOSE_PROJECT_NAME` (with a top-level `name:` field as the default),
prefix container names with `${CONTAINER_PREFIX:-}`, and pass
`--env-file .env` so compose reads the same root `.env` the webapp does.
The "Running multiple instances side by side" block in `.env.example`
lists every overridable knob.
Also split the optional services (`electric-shard-1`, `ch-ui`,
`toxiproxy`, `nginx-h2`, `otel-collector`, `prometheus`, `grafana`) into
`docker-compose.extras.yml` behind a new `pnpm run docker:full` script.
The core stack keeps everything the webapp actually needs to boot:
postgres, redis, electric, minio, clickhouse + migrator, s2-lite.
Defaults match every previous hardcoded value, so existing setups keep
working without touching `.env`.
## Test plan
- [x] `pnpm run docker` on a clean clone brings up the core services on
the standard ports under the `triggerdotdev-docker` project name.
- [x] Setting `COMPOSE_PROJECT_NAME=triggerdotdev-docker-alt` + the
`*_HOST_PORT` overrides in `.env` brings up a second stack alongside the
default one with no port or container-name clashes.
- [x] Webapp boots cleanly against the default `.env.example` values;
`/healthcheck` returns 200, no S2 errors.
- [x] s2-lite basin `trigger-local` accepts an append + read via the
same REST endpoints the webapp uses.
- [x] `pnpm run docker:full` brings up the optional services alongside
the core ones in the same project.
Eric Allam