Missing or legacy preferences resolve to the pinned Dark theme (not
system-resolved, so nobody is surprised by light mode) and the default
contrast is 50. Classic remains the flag-off look.
🦋 Changesets PR / Create Release PR (push) Has been cancelled
🚀 Publish Trigger.dev Docker / 📣 Dispatch main image (push) Has been cancelled
🚀 Publish Trigger.dev Docker / typecheck (push) Failing after 2s
🚀 Publish Trigger.dev Docker / units (push) Failing after 2s
🚀 Publish Trigger.dev Docker / publish-webapp (push) Has been skipped
🚀 Publish Trigger.dev Docker / publish-worker-v4 (push) Has been skipped
🚀 Publish Trigger.dev Docker / scan-webapp (push) Has been skipped
🚀 Publish Trigger.dev Docker / scan-supervisor (push) Has been skipped
## Summary
The realtime runs feed hydrates run rows from read replicas, which means
it needs a replica-lag gate to avoid serving a run's previous state
right after a write. Setting
`REALTIME_BACKEND_NATIVE_RUN_READS_FROM_PRIMARY=1` reads those rows from
each run store's primary instead, so there is no lag to gate against: no
probe, no wake delay, no stale-read retries. Off by default, so nothing
changes unless you set it.
## Design
The run stores already decide replica-vs-primary from the *brand* on the
read client they are handed: a branded replica keeps the read on the
owning store's replica, an unbranded writer escalates it to that store's
own primary. So this is a one-line choice at the hydrator, and it stays
correct across topologies. With the run-ops split on, each leg lands on
its own writer and the caller's client is never forwarded across
databases; with the split off, it is the single database's primary.
```ts
const runReader = new RunHydrator({
readClient: runReadsFromPrimary ? prisma : $replica,
runStore,
});
```
The same flag skips constructing the lag estimator, since probing a
replica the feed no longer reads would be measuring the wrong thing.
Independently, `AuroraReplicaLagSource` detected Aurora by letting
`aurora_replica_status()` fail, on the assumption that the app-level
catch made that free. It isn't: an unresolvable function is a query
error the driver reports to the error log on every sample, so a
non-Aurora replica produced a continuous stream of error events while
the estimator quietly fell through to its next candidate. It now
resolves the function with `to_regproc` and memoizes the answer, so the
unparseable call never reaches the wire.
## Summary
The split run-store's id-set read path (`#findRunsByIdSet`, used by the
runs-list hydrate, the realtime hydrator, and engine sweeps) queried the
new store for the entire id set and then probed the legacy store for the
misses. A run's residency is a total function of its id (run-ops ids
live in the new store, every other id in legacy), so each id belongs to
exactly one store. Route each id to its owner and query each store only
for its own ids, in parallel. Same result set, and while a split is
active with most runs still on legacy it removes a wasted new-store
query from every id-set read.
## Change
`#findRunsByIdSet` now partitions the ids by `classifyResidency` and
runs one bounded query per store (skipping an empty side), in parallel,
mirroring `expireRunsBatch` and the single-run `#route`. `finalizeRows`
still applies orderBy/take/skip globally over the merged set.
This drops the id-set path's cross-store fallback, which existed to
prefer the new-store copy when the same id was present in both stores.
That collision cannot arise when each id maps to exactly one store
(nothing writes a legacy-shaped id into the new store), so the fallback
is dead code. The two id-set tests that asserted "new copy wins on
collision" now assert the routing invariant: a legacy-shaped id resolves
to the legacy store and the path never consults the new store.
The open-predicate path (`#findRunsOpen`) is unchanged: an open `where`
has no id to route on, so it still unions both stores and dedupes.
The account page and useSystemThemeSync call normalizeThemePreference in
client code; importing it from dashboardPreferences.server left an empty
module in the browser bundle and crashed /account.
## Default billing alerts + billing limit page UX
- New orgs get default billing alerts: $5, $100, $500, $1000, $2500.
Existing orgs are backfilled by a billing-side data migration (companion
[PR](https://github.com/triggerdotdev/cloud/pull/1657)).
- The billing limit form starts with nothing selected for orgs that
never set a limit — the save button appears once an option is picked.
- The yellow banner now also shows on the billing limits page itself,
asking to configure a limit. Hidden everywhere for members who can't
manage billing.
- Also fixes billing limit alert preview.
Tests
- `apps/webapp/test/billingLimitsRoute.test.ts` — dirty logic for
empty/selected mode
- `apps/webapp/test/billingAlertsDefaults.test.ts` — default values
- `apps/webapp/test/billingAlertsFormat.test.ts` — preview after a limit
change
- Theme preference widens to classic | system | dark | light (default classic,
the unchanged original dark theme)
- System preference resolves to dark/light from the OS setting via a pre-paint
inline script and follows OS changes live
- Dark and Light are the pinned halves of System preference; Dark derives from
Classic with shared cross-mode accent overrides only
- Contrast-tuned light-mode accents (icons, callouts, run-status and shadcn
chart colors) and fixed raw-palette bypasses in components
- Themes override semantic tokens only, keeping room for a future contrast
control
## Summary
Allow each organization member to use the same development branch name
without colliding with another member's environment. Fixes#4320.
## Fix
Development branches now use the existing member-scoped project, slug,
and organization-member key for upserts. Preview branches retain their
project-wide shortcode behavior.
New development branches receive distinct shortcodes while keeping their
readable, member-scoped slugs. Existing branches continue to resolve
through the member-scoped key, so this requires no migration or
backfill.
## Stacked on #4284 — tests only
This PR contains **only the tests** that guard the production fixes in
#4284 (its base). Review #4284 first; this branch adds no production
code.
## What
Caller-driven replica-lag and idempotency guards for every fixed site:
- Each guard **drives the real exported caller** (route loader/action,
presenter `.call()`, service, or engine method) against a **real
Postgres** with the owning replica frozen via the shared
`laggingReplica` testcontainer primitive — never a store-seam
reimplementation.
- For a **fixed** site the guard goes **RED when the production change
is reverted**; for a **tolerated read-view** site it's a caller-driven
**GREEN** proof the miss self-heals (returns null/empty, no mutation,
row live on primary).
- The **global-scope idempotency** guard drives the real dedup + claim
path through a **real `MollifierBuffer` over a Redis testcontainer**
(real SETNX/poll/publish), and covers the cross-DB **andWait** waitpoint
wiring and the **expired/failed clear-and-recreate** reacquire cases.
Run with `vitest --no-file-parallelism` (testcontainers). Verified
GREEN, and revert→RED verified per fixed site.
## What & why
Two related correctness fixes for the run-ops DB split. Under the split,
run-store reads can route to a **lagging read replica**; a just-written
run/waitpoint/batch can then be missed, causing a wrong decision.
**1. Read-your-writes → owning primary.** Surfaced first as an
intermittent `wait.until({ idempotencyKey })` re-wait on retry. Auditing
the run-store read surface found the same class at sibling sites (some
gating mutations or returning spurious 404s, others
tolerable/self-healing). Reads that must observe their own writes now
route to the owning **primary**
(`findRun`/`findWaitpoint`/`findBatchTaskRunByFriendlyId` →
`*OnPrimary`, a primary re-read on a miss, or a retryable 404 where the
SDK polls). Read-view reads stay on the replica. All additive — the
happy path is unchanged.
**2. Global-scope idempotency across the split.** A `global`-scope key
carries no per-run salt, so the same `(env, task, key)` triggered
concurrently from parents resident on **different** run-ops DBs could
dedup-miss on each DB and create a duplicate (the per-DB unique index
can't enforce cross-DB uniqueness). Such triggers (global scope, or
scope-absent, while split is active) are serialized through the existing
Redis idempotency claim, the loser resolves the winner by id across both
DBs, and the claim is reacquired on the expired/failed
clear-and-recreate path. `run`/`attempt` scope embed the run id and
never contend.
## Stacked for review
This is the **base** of a 2-PR stack, split so review is easier:
- **This PR** — production code only (34 files).
- **Stacked tests PR →
https://github.com/triggerdotdev/trigger.dev/pull/4285** — the
caller-driven guards (55 test files) on top of this branch.
## Validation
Local run-ops split, **both 2-DB and 3-DB**, fresh boot on this branch:
SDK canary 64/71 (only the known concurrency/input-streams/s3 failures),
quarantine sweep **0 unexpected** (340 pass / 16 known / 4 local) in
each topology, dashboard e2e 0 failed. No product regressions.
## Summary
Correctness and performance fixes for deployments that split run data
across more than one database. Single-database / self-hosted deployments
are unaffected (they collapse to a single read/write path).
- **Batches list (dashboard):** for some organizations the Batches list
could hide older batches or show them out of order. It now orders and
paginates by creation time (with the id as a stable tiebreak), so every
batch appears exactly once, newest first. The pagination cursor format
changes; older in-flight cursors simply restart from the first page.
- **Reads:** waitpoint and snapshot lookups that are keyed by a single
run now read only the database that holds that run instead of querying
both, removing redundant queries on hot paths (unblock, snapshot reads).
- **Writes:** environment-scoped writes with no owning run (standalone
wait tokens, waitpoint tags, idempotency-key resets) now land in the
same database as that environment's runs, rather than defaulting to the
other one. An idempotency-key reset also falls back to the other
database when it matches nothing, so a reset still clears the key
wherever the run actually lives.
## Notes
Verified end-to-end against multi-database setups: run-keyed reads and
env-scoped writes land on the correct database with no cross-database
writes, and the batches list surfaces every batch in creation order. New
tests cover the batches ordering/reachability and the write-residency
routing.
## Summary
Adds a set of PAT-authenticated management API endpoints so orgs,
projects, members/invites, environment variables, and a few
project/environment settings can be managed programmatically (scripting,
automation) rather than only through the dashboard. Each route is a thin
wrapper over the **existing** service the dashboard already uses, with
the same authorization applied at the route layer - no new business
logic.
## Endpoints
**Organizations**
- `POST /api/v1/orgs` - create an org (`createOrganization`)
- `PATCH /api/v1/orgs/:orgParam` - rename (title)
- `DELETE /api/v1/orgs/:orgParam` - soft-delete
(`DeleteOrganizationService`; keeps the active-subscription guard)
**Members & invites**
- `GET /api/v1/orgs/:orgParam/members` - list members + pending invites
- `DELETE /api/v1/orgs/:orgParam/members/:memberId` - remove a member
(last-member guarded)
- `POST /api/v1/orgs/:orgParam/invites` - invite by email
(`inviteMembers`, sends the invite email)
- `DELETE /api/v1/orgs/:orgParam/invites/:inviteId` - revoke an invite
**Projects**
- `PATCH /api/v1/projects/:projectRef` - rename
(`ProjectSettingsService`)
- `DELETE /api/v1/projects/:projectRef` - soft-delete
(`DeleteProjectService`)
- `PUT /api/v1/projects/:projectRef/default-region` - set the default
region by worker-group name (`SetDefaultRegionService`)
- project GET/list now return `defaultRegion` (worker-group name, or
null when unset)
**Environments**
- `POST /api/v1/projects/:projectRef/:env/pause` and `/resume`
(`PauseEnvironmentService`)
- `POST /api/v1/projects/:projectRef/:env/regenerate-api-key` - rotate
the env secret key (`regenerateApiKey`, RBAC `write:apiKeys`)
- env var create now accepts an optional `isSecret` flag
## Auth & authorization
- All routes authenticate with a **Personal Access Token**
(`Authorization: Bearer tr_pat_...`).
- Org/project routes are built on the PAT route builders in
`apiBuilder.server.ts`: `createLoaderPATApiRoute` (already existed) and
**`createActionPATApiRoute`** (added here - the loader builder had no
mutation counterpart). The builder runs auth, resolves the org/project
role-floor via `context`, and enforces a declarative `authorization`
block using the same RBAC actions the dashboard applies
(`manage:organization` / `read:members` / `manage:members` /
`manage:project`). Handlers keep a membership-scoped query as the floor,
so a non-member gets a 404. This also gives these routes `tenantContext`
user attribution (Sentry) and `ServiceValidationError`-to-status mapping
for free.
- **Membership floor (important).** The OSS RBAC fallback grants a
permissive ability, so `ability.can(...)` can't reject a non-member on
self-hosted. Every handler therefore resolves the target scoped to the
caller's membership (`members: { some: { userId } }`) → 404 for
non-members. `authorization` is the *role* gate; this is the *tenant*
gate. `resolveOrganizationForApiUser`
(`organizationApiAccess.server.ts`) is the org-tier version of the
existing `findProjectByRef` - org-addressed PAT routes are new, so no
such helper existed before.
- Env-tier routes reuse the existing `authorizePatEnvironmentAccess`
(`write:apiKeys`).
### What `createActionPATApiRoute` gives you
A route is pure declaration - the builder handles auth, RBAC,
validation, tracing, and error mapping:
```ts
export const action = createActionPATApiRoute(
{
method: "PUT", // one verb, or ["PATCH", "DELETE"] for multi-verb routes
params: ParamsSchema,
body: SetDefaultRegionRequestBody, // zod-validated
context: async ({ projectRef }) => { // resolve the org for the RBAC role-floor
const project = await prisma.project.findFirst({
where: { externalRef: projectRef, deletedAt: null },
select: { organizationId: true },
});
return project ? { organizationId: project.organizationId } : {};
},
authorization: { action: "manage", resource: () => ({ type: "project" }) },
},
async ({ params, body, authentication, ability }) => {
// auth + authz already enforced. Just do the work.
// `throw new ServiceValidationError("Region not found", 400)` → mapped to that status.
return json({ ok: true });
}
);
```
Handled for you, so handlers stay thin:
- **Method allowlist** - `method` accepts a verb or an array; any other
verb → `405` with an `Allow` header, *before* auth runs:
```ts
const allowedMethods = method ? (Array.isArray(method) ? method :
[method]) : undefined;
if (allowedMethods && !(allowedMethods as
string[]).includes(request.method.toUpperCase())) {
return json({ error: "Method not allowed" }, { status: 405, headers: {
Allow: allowedMethods.join(", ") } });
}
```
- **PAT / user-actor auth** in a single roundtrip → `401` on
missing/invalid/revoked token.
- **RBAC** - `context` computes the caller's role-floor for the target
org/project; `authorization` gates it → `403` with a structured error
body.
- **Sentry attribution** - `tenantContext.enrich({ userId })` so events
from the handler carry the acting user.
- **Typed errors** - a thrown `ServiceValidationError` is mapped to its
`.status` (default 400); anything else → `500`, and expected boundary
errors are logged as `warn` (kept out of Sentry).
- **Validation** - params / query / headers / body are all zod-checked →
`400` with details.
## Notes for reviewers
- Everything wraps an existing service; the intent is API parity for
things that are currently dashboard-only, not new behaviour.
- `createActionPATApiRoute` is new shared infra (the PAT + RBAC mutation
builder that didn't exist). It's self-contained - the loader builder and
existing routes are untouched.
- `@trigger.dev/core` gets one additive field (`defaultRegion` on the
project response, optional/nullable for client-server version skew) -
changeset included, patch.
- `removeTeamMember`'s last-member guard is now atomic (Serializable
transaction via the `$transaction` helper, with retry), so the dashboard
and API both get it server-side. Added a `## Transactions` rule to
`apps/webapp/CLAUDE.md` (always use the `$transaction` helper);
migrating the remaining direct usages is tracked in TRI-11698.
## Open questions
- ~~Is PAT the right auth (vs OAT for automation)?~~ **Resolved: PAT.**
Organization Access Tokens are currently internal-only (used by the
image builder) and not user-accessible, so they can't back this yet.
- Should any of these be gated behind a flag or scope?
- Naming/shape of the routes.
## Summary
When the run-ops split is enabled, the legacy run-ops database client
was always constructed as its own connection pool, even when it points
at the same database as the primary (control-plane) client. On setups
where those two DSNs resolve to the same physical database, this opened
a second, redundant pool and doubled the number of connections used
against that database. This change makes the legacy client reuse the
primary client's pool whenever their DSNs point at the same database,
and only open a separate pool when they genuinely differ.
## Fix
A small `sameDatabaseTarget` comparison (host, port, database name,
user) decides whether the legacy DSN points at the same database as the
primary. When it does, the legacy handle reuses the primary client by
reference, so no second pool is opened. When the DSNs diverge, the
legacy client is built independently as before, so the split still works
once the databases are actually separate.
Two smaller changes ride along:
- An optional per-pool limit for the run-ops read replica, which
connects unpooled and so draws raw backend connections; unset, it falls
back to the existing default and behaviour is unchanged.
- A startup warning about a missing legacy replica URL is now suppressed
when the legacy client shares the primary pool, where it would be
misleading.
## Verification
Booted the webapp end-to-end in three modes and confirmed the pools
opened as expected via the client's own startup logs and live backend
connection counts: split off (single pool), split on with a shared
database (legacy reuses the primary pool, no doubling), and split on
with separate databases (legacy opens its own pool).
Queues that shard work with concurrencyKey get a per-key history tier.
Counter events for CK runs advance a second per-key odometer and carry
both readings on ONE stream entry (cum + ck/ckcum), so per-key
attribution adds no stream volume; the consumer expands the entry into a
base row and a per-key row. A new 10s AggregatingMergeTree tier keyed by
(queue, concurrency_key) holds per-key enqueue/started/ack deltas,
backlog/running maxes, and wait sums. Rows are activity-bound: a
(queue, key, bucket) row exists only when that key had events, so
user-controlled key cardinality cannot inflate the table (benchmarked at
~19 bytes per event with reads under 100ms in the worst shapes).
The per-queue tiers stay exact: their counter and wait aggregations now
consume only base rows, so per-key odometers never merge under one
queue_name and waits are never double counted. A per-queue key limiter
(default 10k) acts as a safety valve; on overflow the per-key row is
dropped while the base row keeps per-queue counts exact. Per-key
odometers use a short TTL, which cumulative counters make loss-free.
The queue detail page gains a top-keys-by-backlog chart, a key table
merging live state (queued, running, oldest wait) with range stats
(started, peak backlog, mean delay), and click-through per-key
drill-down charts. The new queue_metrics_by_key table is also queryable
directly; its delta columns require grouping or pinning BOTH queue and
concurrency_key, enforced by the compile-time merge guard which now
supports compound keys.
CK queues now emit two extra gauge fields from the CK-path Lua scripts:
the number of concurrency keys with queued runs (ZCARD of the ckIndex)
and the head-of-line wait of the most-starved key (now minus the oldest
ckIndex score). Both flow through the existing stream into new
max-aggregated columns on the 10s and 5m tiers, and non-CK scripts keep
the 7-field gauge shape.
The queue detail page grows a concurrency-keys section for queues with
CK activity: charts for backlogged keys and most-starved wait, plus a
live per-key table (queued, running, oldest wait) read from the ckIndex
zset, most starved first. Per-key history is intentionally not stored:
key values are user-controlled, so the per-key dimension stays in Redis
where it is bounded by the live backlog.
The queue simulator gains a tenant-hotspot scenario that stages the CK
gauge columns and a live ckIndex so the per-key table and charts render
with data.
Counter readings for names past the per-env cardinality cap are dropped
instead of merging unrelated odometers under the overflow label (gauges
still flow). The odometer key now shares the stream's shard hash tag so
the INCR plus XADD script stays in one Cluster slot. The counter stream
cap defaults lower when the stream shares the run-queue Redis. The
per-bucket counter boundary undercount is documented on the delta
columns.
Packs the stream sequence with a 1e6 factor (was 1e5) so up to 1M entries per millisecond per shard fit before a seq could spill into the next millisecond's range, far above what a single Redis stream can produce. ms*1e6 stays within UInt64. Also fixes the webapp mapping test that still expected a numeric order_key after the switch to a BigInt-derived string.
## Summary
Run-graph data (runs, batches, waitpoints, and their related tables) can
now live in a database separate from the control plane, with every read
and write routed to the correct database by each run's residency. This
makes reading and writing run data more reliable once the two are split,
and is a no-op for single-database installs.
## Design
- Run-graph table access goes through the run-store router, which
selects the legacy or the new run-ops store per run instead of assuming
one shared client.
- The legacy run-ops client is now independently pointable, so legacy
run data can be served from its own database (and replica) rather than
the control-plane connection.
- Run-graph writes go straight to the run-graph database instead of
being forwarded through the control plane, and replication targets are
split so runs in the new database still replicate to analytics without
under-counting.
- Read-through slots refuse the control-plane client, so a missing
residency fails loudly instead of silently reading the wrong database.
- Migration `20260710120000_drop_remaining_run_graph_seam_foreign_keys`
drops the foreign keys that still crossed the run-graph / control-plane
seam, which is what lets the two live in separate databases.
The split stays off unless explicitly enabled and the two databases are
confirmed physically distinct; startup fails closed otherwise.
Verified by running the full dashboard end-to-end suite against both a
single-database configuration and a three-database configuration
(control plane, the new database, and a physically separate legacy
database), with runs on both residencies. No misrouted reads in either
configuration.
## Summary
v3 (the engine that ran the SDK v3 era, internally
`RunEngineVersion.V1`) is end-of-life. Following the removal of the v3
execution apps
([#4194](https://github.com/triggerdotdev/trigger.dev/pull/4194)) and
the legacy dev websocket
([#4198](https://github.com/triggerdotdev/trigger.dev/pull/4198)), this
removes the remaining v3 execution stack from the server.
Clients still on v3 (an old SDK or CLI that has not upgraded) keep
getting a clear "upgrade to v4" response. Triggers, batch triggers,
reschedules, and deploys that resolve to v3 are rejected with a graceful
4xx pointing at the migration guide, never a 5xx, so a stale client
cannot affect server health. Self-hosted instances still running v3
should stay on the 4.5.x release line until they migrate.
## What is removed
- The MarQS queue and its shared/dev queue consumers.
- The v3 socket.io namespaces (coordinator, provider, shared-queue) and
the v3 run lifecycle services (attempt, checkpoint, and batch-resume).
- The graphile-worker background job system; all live jobs already run
on `@trigger.dev/redis-worker`.
- The `DEPRECATE_V3_ENABLED` flag: v3 is now rejected unconditionally,
so the flag is gone.
- Unused v3 exports from `@trigger.dev/core` (the `v3/zodNamespace`
subpath and the legacy socket message catalogs) and the now-dead MarQS
environment variables.
## What stays
The v4 engine is untouched. The graceful v3 rejection boundary stays,
`determineEngineVersion` still detects a v3 project so it can reject it,
and the batch service plus batch-completion worker stay for current
clients. Live queue concurrency limits and metrics now read from the v4
run engine instead of MarQS, and a brand-new dev environment now
defaults to v4.
## Dependency cleanup
Removes webapp dependencies left unused by this change: `seedrandom` and
`semver` (only the removed v3 code used them) plus a set that was
already dead, their orphaned `@types` packages, and two dead files. Adds
a `knip:deps` script and a `knip.json` config so unused dependencies can
be found the same way going forward.
## Summary
Two threads on the run-ops split path.
Read path: per-item run reads are batched into grouped queries, a
waitpoint's connected-run reads are bounded, and the dedicated-schema
relation hydrators fetch only the requested columns instead of whole
rows. Retrieve also falls back to the other database when a routed read
misses, so a run whose physical residency diverges from its id shape is
still found rather than returning a spurious not-found. Fewer and
lighter queries on the run read path, with no change to results.
Mint-kind flip safety: flipping which database new runs mint to is now a
deterministic wall-clock cutover, for both per-org and global flips. For
a grace window every process resolves the same database, so a flip
cannot route two concurrent triggers that share an idempotency key to
different databases (which would bypass the per-database unique
constraint and create a duplicate run).
Supersedes the earlier #4205 and #4208.
Draft: validation in progress.
## Summary
Under high OTLP ingest volume, the whole decode, transform, and enrich
pipeline runs on the request event loop, so a single CPU core becomes
the ceiling while the rest sit idle. This adds an opt-in worker pool
that moves decode, transform, and LLM-cost enrichment onto worker
threads, keeping the main thread free for I/O. It is off by default
(`OTEL_TRANSFORM_WORKER_POOL_ENABLED`), so behavior is unchanged unless
enabled.
## Design
Workers do decode, filter, convert, and enrich (including LLM pricing
match). The main thread stays the single database reader: it loads the
pricing registry and broadcasts the compiled model rows to the workers
(re-broadcasting on every reload), so workers never touch the database.
The pure transform is extracted into a dependency-light module (no
Prisma/Redis/ClickHouse imports) so it can run inside a worker.
Importantly, the main thread keeps the existing single consolidated
insert path, so ClickHouse insert batching and part count are unchanged.
The parallelism buys CPU headroom, not more insert streams (which would
add merge pressure).
The worker is bundled as a standalone file at build time and ships in
the existing image with no Dockerfile change. In local load testing the
pool sustained roughly 2.6x the throughput of the single-thread path and
kept the main thread responsive under load.
## Problem
Several display/grouping issues in the **Errors** feature, all rooted in
how the ClickHouse error materialized views (`errors_mv_v1`,
`error_occurrences_mv_v1`) read the stored error JSON produced by
`parseError`:
1. **Messageless errors show "Unknown error".** An empty message falls
straight through `coalesce(nullIf(message,''), 'Unknown error')` to the
literal, even though the error's class `name` is available (e.g. an
Effect tagged error `ListMessagesError` with no message).
2. **Unrelated errors collapse into one group.**
`calculateErrorFingerprint` keys on `type : message : stack`, where
`type` is always the union tag (`BUILT_IN_ERROR`, …), `message` is
empty, and the stack isn't read — so every messageless built-in error
(and every string/custom error) hashes to the same constant input → one
fingerprint.
3. **error_type shows the internal tag.** `coalesce(type, name, …)`
always resolves to `type` (always present), so the column shows
`BUILT_IN_ERROR` instead of the real class name.
4. **Stack traces never populate.** The MVs read `error.data.stack`, but
the serializer stores the trace under `stackTrace` — so the column is
always empty.
## Fix
All display changes are `ALTER TABLE … MODIFY QUERY` on the two views
(migration `035`); the fingerprint change is in the webapp.
- **Fingerprint** (`errorFingerprinting.ts`): fall back **message → name
→ raw**. Messageless errors now group by class name (or raw value for
non-Error throws); message-bearing errors are **unchanged**
(short-circuits at `message`), so existing groups don't split — only
currently-messageless errors get their own group going forward.
- **error_message**: same `message → name → raw` fallback before
`'Unknown error'`.
- **error_type**: coalesce `name → code → 'Error'` (drops the reliance
on the union tag). Built-in → class name, internal → `code`,
string/custom → `Error`.
- **stack trace**: read `error.data.stackTrace`. Bounded as before
(serializer caps 50 frames / 1024 chars per line; MV clips to 2000
chars).
## Migration notes
- `MODIFY QUERY` swaps the view query in place (no drop/recreate gap);
Down restores the previous query.
- **Existing rows are left unchanged** — changes apply only to rows
inserted after the migration. No backfill.
## Tests
`errorFingerprinting.test.ts` — 57 pass, incl. new cases for messageless
class names, string/custom raw values, and stability of message-bearing
fingerprints.
Fixes the display-derivation half of TRI-11938 (error_type + stack
trace); relates to TRI-9254 and TRI-9250.
## Summary
The runs page's empty-state check (whether an environment has ever had a
run, which decides between the "getting started" and "no runs match your
filters" states) ran a `findFirst` against the Postgres `TaskRun` table.
This moves it to ClickHouse, the same store the runs list itself reads
from, so the check no longer queries `TaskRun`.
## Design
Only the runs list triggers the check now (via an `includeHasAnyRuns`
flag); the other presenters that reuse `NextRunListPresenter` (API,
schedule detail, waitpoint detail, error group) no longer issue it. When
the list is empty it runs `SELECT 1 FROM task_runs_v2 ... LIMIT 1`
filtered on the full `(organization_id, project_id, environment_id)`
sort-key prefix with a configurable `created_at` lower bound
(`RUN_LIST_HAS_RUNS_LOOKBACK_DAYS`, default 30), so it hits the primary
index and reads minimal granules.
Results are cached in a tiered memory + Redis SWR cache. Only positive
("has runs") results are cached, so an environment with no runs is
always re-checked and its first run shows up immediately.
## Summary
Deleting a project triggered an unbounded database cleanup that scanned
the project's entire run history, so deleting a project with many runs
could be very slow. Project deletion is a soft delete again: run data is
retained and the deletion completes quickly.
## Fix
Project deletion ran a cascade hard-delete whose `BulkActionItem` step
filtered through a relation to `TaskRun` scoped by `projectId`. Prisma
compiles that to an `EXISTS`-join over the project's entire `TaskRun`
set (a large, hot table with no `projectId` index), and it ran on every
project deletion unconditionally.
Removing the cascade-cleanup call restores the prior soft-delete
behaviour: queues are removed, the project is marked deleted, and run
data is retained. The cascade-cleanup service (added in
[#4117](https://github.com/triggerdotdev/trigger.dev/pull/4117)) had no
other callers, so it and its test are deleted.
Container runtimes (cri-o / containerd / podman) can't pull
zstd-compressed layers carried in a Docker v2s2 manifest
(`application/vnd.docker.image.rootfs.diff.tar.zstd`). A deploy built
with an outdated CLI can produce exactly that combination - and today
it's promoted to current and then fails every run at image-pull time.
This extends the pre-promotion image check (#4049) to also inspect the
manifest's layer media types. If any layer uses the unpullable zstd/v2s2
media type, the deploy is rejected at finalize with a clear message to
upgrade the CLI and re-deploy, instead of silently shipping a version
that can't start.
The manifest is already returned by the existing ECR `BatchGetImage`
call, so there's no extra registry request for single-arch images.
Parsing is a lenient Zod schema and **fails open** - a manifest we can't
read never blocks a deploy. Manifest lists / OCI indexes (no top-level
`layers[]`) and OCI zstd (`...tar+zstd`, which runtimes support) pass
unaffected.
Also clarifies in the contributor docs that changesets and
`.server-changes/` notes are user-facing and should be written for
users, not maintainers.
refs TRI-11702
<!-- ccr-slack-attribution -->
_Requested by **Matt Aitken** · [Slack
thread](https://triggerdotdev.slack.com/archives/C032WA2S43F/p1783430373189849?thread_ts=1783430373.189849&cid=C032WA2S43F)_
## ✅ Checklist
- [x] The PR title follows the convention.
- [x] I ran and tested the code works (typecheck of the edited files is
clean; see Testing)
---
## Testing
**Before:** the webapp run-replication test shard failed on nearly every
PR because assertions waited a fixed 1s for rows to replicate from
Postgres → ClickHouse and intermittently checked before the row arrived
under CI load.
**After:** those assertions poll (up to 30s, 250ms interval) until the
rows land, so they pass as soon as replication completes and stop
flaking, without slowing the happy path.
These tests are testcontainers-backed (need Docker + Postgres +
ClickHouse), so the full suite is exercised in CI. Locally I confirmed
the edited `runsReplicationService.part1..part8.test.ts` files
type-check with no new errors.
---
## Changelog
**How:** wrapped the ~21 present-row assertions across
`runsReplicationService.part1..part8.test.ts` in `vi.waitFor`, matching
the existing poll pattern in `part9.test.ts`. Left absence assertions
(expecting 0 rows / no spans) on a fixed settle delay since there is
nothing to poll for. Tests only — no production code changed.
Note: this does NOT touch the `subscribe()` startup race in
`internal-packages/replication/src/client.ts` (a riskier, separate
follow-up).
💯
---
_Generated by [Claude
Code](https://claude.ai/code/session_01KtUdSLKrK17eFVuRYXT6uj)_
---------
Co-authored-by: Claude <noreply@anthropic.com>
## Summary
Adds SDK and API support for run bulk actions. You can now create bulk
cancel or replay actions from `@trigger.dev/sdk` using run IDs or the
same filters as `runs.list()`, then retrieve, list, poll, or abort the
action by its `bulk_` handle.
Tests, docs, changesets added.
## Design
The dashboard bulk action service now accepts structured filters instead
of reading directly from a dashboard request, so the dashboard and API
share the same creation path. Replay actions created through the API are
attributed with the existing `api` trigger source, while
dashboard-created actions keep `dashboard`.
The SDK exposes the new surface under `runs.bulk.*`, including
`targetRegion` for replay region overrides and cursor pagination for
listing bulk actions.
## Filters and runIds
Nuance on filters. If `filter` is provided, it MUST have at least one
key. This is to remove the footgun of passing no filter and selecting
all runs.
```typescript
{ action: "cancel", runIds: ["run_1"] } // valid
{ action: "cancel", runIds: [] } // invalid, min(1)
{ action: "cancel", filter: { status: "FAILED" } } // valid
{ action: "cancel", filter: {} } // invalid
{ action: "cancel", filter: {}, runIds: ["run_1"] } // invalid
```
## Summary
Three fixes to the run-ops database split (the Cloud-only mode where
run-lifecycle rows live on a dedicated Postgres). All are inert in the
default single-database deployment.
The main fix: on the batch trigger paths, a parentless batch's item runs
chose their physical store from a fresh per-org mint-flag read at
processing time, so flipping an org's flag mid-batch could land an item
in a different store than its batch, breaking the `TaskRun.batchId`
foreign key (or silently orphaning the item). The other two harden the
split's safety nets: the schema-parity test now actually compares
columns, and the read fan-out gate now signals when it has been silently
disabled.
## Batch item residency
`RunEngineBatchTriggerService` (api.v2) and the BatchQueue item callback
(api.v3) now anchor each item's id mint on the batch's own friendlyId,
mirroring the already-safe `BatchTriggerV3Service`. Residency is a pure
id-shape check, so an item can no longer diverge from its batch across a
mid-batch flag flip. The pre-failed-run fallback is anchored the same
way (it also sets `batchId`), and the shared mint branch is consolidated
into one helper so every mint path stays in lockstep. No new database
queries; single-database mode is unchanged (a cuid-shaped batch
friendlyId yields a cuid item).
## Schema parity test
The parity test previously read only the dedicated schema and matched
model headers with regexes, so it never compared columns and could not
catch a run-subgraph column that diverged between the two physical
schemas. It now parses both schemas and asserts bidirectional
scalar-column parity (type, nullability, array-ness, default) across the
run-subgraph models, and fails on any field line it can't parse. Scoped
to the run-subgraph models so unrelated control-plane edits don't break
it.
## Read fan-out signal
The split read fan-out gate is decided by the object identity of the NEW
vs control-plane clients. It now warns when both run-ops URLs are set
but the NEW client isn't a distinct instance (fan-out silently off), and
a new test exercises the real topology-into-gate wiring so a future
refactor that aliases the clients can't disable fan-out unnoticed.
## Verification
New unit and glue tests cover all three changes; the DB-backed
residency, store-routing, and topology suites pass against real
Postgres; `typecheck` is clean for both packages.
The engine `triggerTask` suite was a single 2447-line file with 23
`containerTest` cases, each spinning its own Postgres + Redis. vitest
shards by whole file, so all 23 container setups landed on one shard and
dominated its wall-clock. The recorded entry in `test-timings.json`
badly under-counts the real cost (it does not capture the
per-`containerTest` container startup that dominates on CI), so the
duration-sharding sequencer treated the file as light and stacked it,
producing one ~21 minute shard.
Splitting does not reduce the number of container setups; it lets those
23 cases distribute across shards instead of stacking on one. The webapp
unit-test stage is gated by its slowest shard, so this cuts the stage's
wall-clock roughly in half.
## CI timing (before vs after)
Real CI wall-clock of the `Unit Tests: Webapp` shards (`--shard=i/10`).
"Before" is sampled from recent runs on other branches (unsplit file,
from `main`); "after" is this PR.
| Shard | Before (s) | After (s) |
|------:|-----------:|----------:|
| 1 | 250 | 359 |
| 2 | 444 | 411 |
| 3 | 497 | 659 |
| 4 | **1257** | 284 |
| 5 | 545 | 641 |
| 6 | 284 | 644 |
| 7 | 244 | 214 |
| 8 | 340 | 445 |
| 9 | 188 | 395 |
| 10 | 234 | 567 |
| **Slowest shard (gates the stage)** | **~1247s (≈21m)** | **659s
(≈11m)** |
| Sum of all shards | 4283 | 4619 |
Before: shard 4 is the long pole at 1237s / 1247s / 1257s across three
sampled runs (the `triggerTask` file plus whatever else the packer put
with it). After: the six pieces spread across shards, the slowest drops
to 659s. The small rise in summed time is the extra per-file container
startup, paid in parallel across shards, so the gating number still
falls by about 10 minutes.
## Change
Split into six per-concern files that share a `triggerTaskTestHelpers`
module (the `vi.mock` calls stay per-file, since vitest hoists them):
- `triggerTask.test.ts` (3): trigger + concurrencyKey coercion
- `triggerTask.idempotency.test.ts` (4): idempotency + queue resolution
- `triggerTask.debounce.test.ts` (4): retries + debounce validation
- `triggerTask.mollifier.test.ts` (4): mollifier call-site behaviour
- `triggerTask.metadataCache.test.ts` (4): DefaultQueueManager task
metadata cache
- `triggerTask.residency.test.ts` (4): child run residency inheritance
All 23 cases are preserved. The file's `test-timings.json` entry is
split across the new files so bin-packing stays balanced.
While rewriting these files, cleanup was moved to `onTestFinished(() =>
engine.quit())` so an `engine`/`Redis` leaked on a failing assertion no
longer persists on the worker-scoped Redis and cascades into later cases
(`hookTimeout` raised to 60s so the after-cleanup gets the full budget).
Prisma lookups switched from `findUnique` to `findFirst` to match the
repo convention.
Verified: all six files run green locally (23/23), oxlint and oxfmt
clean.
## Summary
On the run-ops database split, a run that waits (`triggerAndWait`,
`batchTriggerAndWait`, `wait.forToken`) could hang forever after its
wait had already completed. The runner reads a resume from
`/snapshots/since` exactly once: if that read returned the resume
snapshot without its completed-waitpoints, the runner logged "executing
without completed waitpoints", advanced its cursor, and never re-read
it, so the awaiting run never continued.
## Root cause
The resume snapshot and its completed-waitpoint rows were written as two
separate commits. This regressed when the split replaced Prisma's atomic
nested `connect` with an FK-free insert (in
[#4163](https://github.com/triggerdotdev/trigger.dev/pull/4163)), and
`/snapshots/since` is served from a read replica. A fetch landing in the
sub-millisecond gap between the two commits, or a multi-reader replica
serving the snapshot from a different point in time than its join rows,
delivered an empty resume. Because the runner consumes each snapshot
once and treats an empty resume as terminal, a single stale read was
fatal and produced a permanent, nondeterministic hang.
## Fixes
- Commit a snapshot and its completed-waitpoint links in one
transaction, restoring the atomicity the split removed.
- Repair the completed-waitpoints from the owning primary when a
multi-reader replica serves the snapshot without its join rows. This
covers single-waitpoint resumes, which carry no
`completedWaitpointOrder` and so were missed by the count-based repair.
- Read the primary in the checkpoint `WAIT_FOR_BATCH` pre-check, so a
batch that already resumed is not re-suspended into a stall.
- Fall back to the primary when a waitpoint token misses both read
replicas, so a token completed immediately after it was minted no longer
returns a spurious 404.
- Route batch-item creation by `batchTaskRunId`, consistent with the
batch-completion count and the row's foreign key.
- Reject control-plane-only relation selects on the dedicated schema
with a clear error instead of an opaque Prisma failure, and stop
`createDateTimeWaitpoint` bypassing residency routing through a caller
transaction.
Verified against the deployed split topology: a resume snapshot and its
completed-waitpoints are now always delivered together, so the runner
can no longer drop a resume.
## Summary
On the run-ops split, NEW-residency runs could hang. Time-based waits
(`wait.for`, `wait.until`, `delay`, waitpoint tokens),
`batchTriggerAndWait`, and attempt starts stalled and never resumed.
Each was a run-ops read or update that hit the wrong database: either
the owning store's read replica when it needed read-your-writes, or the
wrong store entirely because it routed by an id that does not encode
residency.
## Fixes
**Waitpoint resume (the main hang).** The managed resume path reads a
run's completed waitpoints by snapshot id
(`findSnapshotCompletedWaitpointIds`). Snapshot ids are cuids, which
always classify to the legacy store, so a NEW run's join rows (which
live on the new store) were never found. The resumed run saw zero
completed waitpoints and hung. It now fans out across both stores and
merges, like its sibling readers.
**Batch completion.** Batch item completion
(`updateManyBatchTaskRunItems`) routed by the item id, which is also a
cuid, so a NEW batch's items were updated on the wrong store, matched
zero rows, and the batch was treated as already complete (its parent's
`batchTriggerAndWait` then hung). It now routes by the batch id, which
does encode residency, matching the sibling `countBatchTaskRunItems`.
**Read-your-writes on the resume path.** The block-time
pending-waitpoint check (`countPendingWaitpoints`) and the attempt-start
lock check (`findRun` in `startRunAttempt`) both read the owning store's
replica with no read-your-writes guarantee, so a just-committed
waitpoint completion or dequeue lock could be missed under replica lag
and strand the run. Both now read the owning primary.
Each fix ships with a two-database store or engine test that reproduces
the hang and passes with the fix.
## Problem
The run-ops split mints NEW-store run ids as **27-char base62 KSUIDs**.
The supervisor writes the run id into the Kubernetes pod name
(`runner-<id>`), and pod names must be DNS-1123 labels (lowercase
`[a-z0-9-]`) — so uppercase base62 ids make k8s reject the pod (422) and
**those runs never launch** (they loop in `PENDING_EXECUTING` until the
heartbeat-stall handler nacks them, forever). `.toLowerCase()` can't fix
it: base62 has both `A`(10) and `a`(36) as distinct symbols, so folding
collides distinct ids and destroys sort order.
## Fix: change the encoding, not the structure
Mint a **26-char lowercase base32hex** run id:
```
run_<24-char base32hex core><region char><version char>
[ 6-byte ms timestamp ][ 9 CSPRNG bytes ]
```
- **base32hex** (RFC 4648 §7, alphabet `0-9a-v`): lowercase,
order-preserving, DNS-safe; 15 bytes → exactly 24 chars, no padding.
Hand-rolled encode/decode (no new dependency).
- **48-bit ms timestamp** in the leading bytes → plain string sort ==
creation order at millisecond resolution.
- **72 bits CSPRNG** entropy; PK unique constraint is the backstop (no
retry loop).
- **region / version** are raw positional chars (read via one `charAt`
before decoding/routing), version = `"1"`.
DNS-safe from birth and hyphen-free, so **firekeeper is unchanged** —
`runner-<id>-attempt-N` → strip `runner-`, cut at first hyphen still
recovers the exact id incl. region+version.
## Residency discriminator: length → version char
`classifyKind`/`classifyResidency` (`runOpsResidency.ts`) previously
distinguished NEW vs LEGACY by **id length**. That gets ambiguous with a
third format. It now discriminates on the **version char at a fixed
position** (`isRunOpsIdBody`: 26 chars, `[25] === "1"`, base32hex
alphabet) → NEW; everything else → LEGACY. Total, never throws. The
`Residency` (NEW/LEGACY) contract the routing store consumes is
unchanged; the `"ksuid"` `ResidencyKind` label is retained only because
it's the persisted `runOpsMintKsuid` feature-flag value.
## Scope / verification
- Generator + discriminator in `@trigger.dev/core` isomorphic; mint path
+ all id-shape call sites swept (~40 webapp files); changeset added
(`@trigger.dev/core` patch).
- Core unit tests (encode/decode round-trip + property, generator shape,
ms sort-order incl. intra-second, parse partitioned-vs-legacy,
firekeeper round-trip): **24 pass**. `@trigger.dev/core` builds; webapp
typechecks; format/lint clean.
## Open decisions (flagged, not silently chosen)
1. **Backward-compat**: existing 27-char base62 KSUID runs now classify
LEGACY. On test cloud these are the broken/looping runs that never
completed, so this is acceptable — but worth a conscious call before
prod. No transitional length-recognition added (keeps the discriminator
clean).
2. **Storage collation**: the sort guarantee is byte-order — if the
run-ops id column is `TEXT` with default locale collation it's silently
not honored. Confirm whether `COLLATE "C"` / `BYTEA` is needed on the
run-ops schema.
3. **Region sourcing** wiring — see `regionCharForRegion` /
`REGION_CODES`.
---
## ⚠️ Required migration — deploy in lockstep
This PR renames a persisted feature-flag key/value and an env var. These
are **not** changed by the code alone and must be migrated when this
deploys, or affected orgs silently fall back to `cuid` minting (no crash
— `defaultValue: "cuid"`):
1. **Env var** (terraform): `RUN_OPS_MINT_KSUID_ENABLED` →
`RUN_OPS_MINT_ENABLED` (carry the value over).
2. **DB** `organization.featureFlags`: migrate both the key and value
together:
- key `runOpsMintKsuid` → `runOpsMintKind`
- value `"ksuid"` → `"runOpsId"`
Until an org's flag row is migrated, its `runOpsMintKind` lookup misses
and it mints `cuid` (legacy) — so no NEW-store ids for that org until
the data lands.
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
## What
Adds the ability to **automatically migrate the dedicated run-ops
database** (the NEW DB in the run-ops split), matching how every other
database in the system is migrated. Follow-up to the run-ops split
activation.
## Changes
- **Migrate runner** — new
`internal-packages/run-ops-database/scripts/migrate.mjs`, exposed as
`db:migrate:deploy` / `db:migrate:status`. Connects via
`RUN_OPS_DATABASE_URL` (the same var the app uses) and expands `${VAR}`
refs like Prisma's dotenv.
- **Self-host** — `docker/scripts/entrypoint.sh` runs the run-ops
migration on boot when the DB is configured, gated by
`SKIP_RUN_OPS_MIGRATIONS`. Single-DB installs never set the URL, so it's
a clean no-op.
- **Single env-var family** — the run-ops DB is now addressed by one
canonical `RUN_OPS_*` family, connect path and migrations resolving the
identical URL:
- `RUN_OPS_DATABASE_URL` (writer) — replaces `TASK_RUN_DATABASE_URL`
- `RUN_OPS_LEGACY_DATABASE_URL` — replaces
`TASK_RUN_LEGACY_DATABASE_URL`
- `RUN_OPS_DATABASE_READ_REPLICA_URL` — replaces
`TASK_RUN_DATABASE_READ_REPLICA_URL`
- the old `TASK_RUN_*` aliases, the `??` coalesce, the
`runOpsNewDatabaseUrl` indirection, and the migrate-only `directUrl` are
all removed (consumers read `env.RUN_OPS_DATABASE_URL` directly).
`directUrl` was dropped because it was only ever used by `prisma
migrate` (never the app runtime) to bypass a pooler for advisory locks —
premature here since the run-ops connection isn't wired to the app yet.
If a pooler is later introduced for the app, a direct URL can be
reintroduced then.
## Safety
- **Pure rename** — nothing deployed sets any `TASK_RUN_*` var (the
split isn't activated anywhere yet; `.env.example`, docker-compose, and
cloud already use `RUN_OPS_*`), so there is no config migration.
- **Single-DB / self-host** — no new required env var; entrypoint and
migrate are no-ops when `RUN_OPS_DATABASE_URL` is unset.
- **Cloud** — runs migrations as pre-deploy ECS tasks (companion cloud
PR), calling these same `db:migrate:deploy` / `db:migrate:status`
commands.
## Verification
- Live migration against a fresh scratch DB with only
`RUN_OPS_DATABASE_URL` set: both migrations applied, no `P1012`/`P1013`;
`${VAR}` expansion, idempotent re-run, `status`, and no-op skip all
pass.
- Schema parity 4/4; `typecheck --filter webapp` 18/18; affected
split/replication tests 34/34.
## Scope
This delivers automatic migrations only. Enabling the app to *use* the
new DB (setting `RUN_OPS_DATABASE_URL` + `RUN_OPS_SPLIT_ENABLED` on the
service) is a separate activation step.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
## Problem
`LogicalReplicationClient` uses a Redlock leader lock to guarantee a
single active consumer per Postgres logical replication slot. The lock
resource was keyed on the client `name`:
```
logical-replication-client:${this.options.name}
```
A slot permits exactly one consumer, so the lock's job is to serialize
consumers **of a given slot**. Keying it on `name` breaks that whenever
two clients target the same slot with different names — most notably
across a rolling deploy where the client `name` changes but `slotName`
does not. Both acquire *distinct* locks, both consider themselves
leader, and the second to reach `START_REPLICATION` hits `replication
slot "<slot>" is active for PID <n>`. Because that query was
fire-and-forget and its failure was only logged (no retry), the consumer
stopped and replication stalled until the process was restarted.
## Fix
**1. Key the leader lock on `slotName`** — the actual single-consumer
resource:
```
logical-replication-client:${this.options.slotName}
```
Consumers of the same slot now contend on the same lock and hand off
cleanly across restarts/deploys; different slots stay independent.
`name` is kept for logging and the pg `application_name`.
**2. Self-healing resubscribe** (`resubscribeOnFailure`, opt-in) —
instead of logging-and-dying, a client re-subscribes with exponential
backoff after a lost election or a failed `START_REPLICATION`, so a
rolling deploy self-heals: the incoming pod retries until the draining
pod releases the slot, then takes over. Safety:
- `#cleanupAttempt()` unconditionally ends the pg client (freeing the
walsender) and releases the leader lock before rescheduling — retries
never leak connections/locks.
- `shutdown()` sets an intentional-stop latch re-checked after every
`await` in `subscribe()` (and aborts the lock-acquire spin), so a
resubscribe can never race or outlive an intentional shutdown.
- Backoff resets only on genuine stream start, so a permanently stuck
slot backs off to the ceiling and logs loudly rather than tight-looping;
an epoch guard neutralises stale `START_REPLICATION` catches.
Runs- and sessions-replication opt in and use `shutdown()` for all
intentional stops.
**3. Observability** — the admin runs-replication status route probed
the old name-keyed Redis key (would report `leader:false` for every
source after fix#1); now probes the slot-keyed key.
## Tests
`internal-packages/replication/src/client.test.ts` (real Postgres +
Redis containers):
- same-slot/different-name → second client must not double-lead or race
into "slot is active" (the regression)
- a failing `START_REPLICATION` retry loop must not leak connections or
locks
- `shutdown()` during an in-flight `subscribe()` must not leave a zombie
leader
- `subscribe()` after `shutdown()` re-arms `resubscribeOnFailure`
- self-heals once the leader releases the slot
Plus the multi-source wiring test updated to the slot-keyed lock keys.
## Rollout
With the self-healing resubscribe, this ships as a **plain rolling
deploy** — the incoming pods retry across the one-time lock-key
transition and take over once the old pods drain (a brief replication
stall that the durable slot replays on reconnect — no data loss). No
stop-before-start required.
## What
Extends the ClickHouse runs-replication service to fan in from multiple
Postgres sources (the control-plane DB and the run-ops DB) instead of a
single source, plus the admin operations to run and observe it.
- **Multi-source fan-in** (`services/runsReplicationService.server.ts`,
new `runsReplicationInstance.server.ts`,
`runsReplicationGlobal.server.ts`): factors the replication service into
per-source instances and a coordinator so a single ClickHouse target is
fed from more than one Postgres source.
- **Admin ops** (`routes/admin.api.v1.runs-replication.status.ts`,
`admin.api.v1.runs-replication.backfill.ts`,
`v3/services/adminWorker.server.ts`): adds a status endpoint reporting
per-source replication state and updates the backfill entrypoint for the
multi-source shape.
## Why
PR7 of the run-ops split stack, and the final piece: once run state can
live in a separate run-ops DB (earlier PRs), the analytics replication
into ClickHouse has to consume both sources so runs remain queryable
regardless of residency. Behavior-changing for the replication service
internals; the ClickHouse-facing output is unchanged (still one runs
stream), and single-source operation is preserved when the split is not
enabled.
## Tests
New vitest coverage: `runsReplicationInstance.test.ts` (per-source
instance behavior) and `runsReplicationService.part8`/`part9` suites
exercising the multi-source coordinator. Testcontainers-backed
(ClickHouse + Postgres); no mocks.
## Notes
Draft, **stacked on #4118** (`runops/pr06-write-path`). Review that
first; this diff is against it.
Server-change / changeset note to be added at stack-assembly time.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Katia Bulatova