📦 Preview packages (pkg.pr.new) / Build and publish previews (push) Has been cancelled
Making the field non-enumerable only on InsertError left the leak open one
layer down: the client logs the raw ClickHouseError at every insert-error site
before converting it, and the patch assigned rawMessage as a plain enumerable
property, so the offending-row snippet still reached structured logs. The patch
now defines the field non-enumerable itself.
Covers the patch's behaviour with a test that reads the row hint back through
the public parseError export, which also fails loudly if the patch ever stops
being applied.
The untruncated ClickHouse error text is kept only so the recovery path can
read the failing-row hint, but ClickHouse embeds a snippet of the offending
row in its parse errors. It is now a non-enumerable field, so structured
logging and error reporting cannot pick it up while direct reads still work.
The dropped-row count reported zero on tables whose materialized views fold
MV rows into written_rows, which reads as "no data lost" when rows had in fact
been skipped. Reaching an allow_errors insert always means at least one row is
un-ingestable, so the count now reports that floor and flags itself inexact,
and a batch only counts as wholly dropped when ClickHouse's summary says so
exactly. The skip path logs at warn to match, since it always loses a row.
Also drops the two event-repository counters that could never leave zero
(that path skips rows rather than stripping columns, so nothing fed them), and
names the two causes of an allow_errors bail as a bailReason log field instead
of reporting both as a strip-budget hit.
When a single run output, trace span, or payload contained JSON ClickHouse
could not ingest (for example nesting past its depth limit), the whole insert
batch was rejected and those runs and spans silently vanished from the runs
list, traces, and logs.
Recovery is now per-table:
- Runs: follow ClickHouse's failing-row hint to strip just the un-ingestable
JSON column(s) so the run still lands with its status, up to a configurable
limit (RUN_REPLICATION_MAX_POISON_STRIPS_PER_BATCH, default 1). Past the
limit, land the rest with allow_errors and skip the remainder, so recovery
cost stays flat on large flushes instead of re-sending the batch per row.
- Trace events and payloads: land the batch with allow_errors so the good rows
land in one pass and only the un-ingestable rows are skipped.
Reading the failing-row hint needs a patch to @clickhouse/client-common, whose
error parser otherwise discards the row number from the server response.
## Summary
The batches list page orders by `(createdAt DESC, id DESC)`, which is
why [#4361](https://github.com/triggerdotdev/trigger.dev/pull/4361)
added a matching index on `BatchTaskRun`. That index only landed in
`@trigger.dev/database`.
The dedicated run-ops database has its own migration history, so it
never received the index. `BatchListPresenter` reads both databases and
merges, so for environments whose batches live in the dedicated database
the page kept falling back to a scan and in-memory sort, which is the
exact behaviour #4361 set out to fix.
## Fix
Adds the index to the run-ops schema with its own migration. `CREATE
INDEX CONCURRENTLY IF NOT EXISTS`, so it is a no-op where the index
already exists and still records its ledger row.
The second half is the interesting part. Because the two packages own
separate migration histories, a run-graph schema change has to be
authored twice, and nothing made the miss visible: the run-ops status
check truthfully reports "up to date" against its own history, so the
apply step just skips.
`schemaParity.test.ts` compares the physical shape of every model the
run-ops schema declares against its counterpart in
`@trigger.dev/database`: scalar fields with their attributes, plus
`@@index`, `@@unique`, `@@id` and `@@map`. Relation navigation fields
are excluded, since the run-ops schema deliberately drops relations that
would cross a database boundary while keeping the scalar FK column. A
field counts as a relation when its type resolves to a model name, which
keeps enum-typed columns in scope.
Two models are listed as run-ops-only: `CompletedWaitpoint` and
`WaitpointRunConnection`, both explicit FK-free replacements for a
control-plane implicit many-to-many, since an implicit m2m carries a
foreign key that cannot resolve across databases. The test also asserts
that exception list is exhaustive, so a new unpaired model fails rather
than being silently skipped.
Confirmed the guard actually fails: reverting the index turns
`BatchTaskRun` red with the missing `@@index` named in the diff.
## Summary
The batches list page orders by `createdAt DESC, id DESC` filtered by
environment and a created-at window, but the only supporting index on
`BatchTaskRun` was `(runtimeEnvironmentId, id)`. That index can't
satisfy the `createdAt` ordering, so on environments with a large number
of batches the query fell back to a full table scan and in-memory sort,
which could run long enough to hit the statement timeout.
## Fix
Adds `(runtimeEnvironmentId, createdAt DESC, id DESC)` on
`BatchTaskRun`. The query now reads straight from the index in order
with no sort step, returning a page with only a handful of heap fetches
instead of scanning the whole environment slice.
The migration uses `CREATE INDEX CONCURRENTLY IF NOT EXISTS`, so it
takes no table lock and is a no-op if the index already exists.
## Summary
The split run-store's id-set read path (`#findRunsByIdSet`, used by the
runs-list hydrate, the realtime hydrator, and engine sweeps) queried the
new store for the entire id set and then probed the legacy store for the
misses. A run's residency is a total function of its id (run-ops ids
live in the new store, every other id in legacy), so each id belongs to
exactly one store. Route each id to its owner and query each store only
for its own ids, in parallel. Same result set, and while a split is
active with most runs still on legacy it removes a wasted new-store
query from every id-set read.
## Change
`#findRunsByIdSet` now partitions the ids by `classifyResidency` and
runs one bounded query per store (skipping an empty side), in parallel,
mirroring `expireRunsBatch` and the single-run `#route`. `finalizeRows`
still applies orderBy/take/skip globally over the merged set.
This drops the id-set path's cross-store fallback, which existed to
prefer the new-store copy when the same id was present in both stores.
That collision cannot arise when each id maps to exactly one store
(nothing writes a legacy-shaped id into the new store), so the fallback
is dead code. The two id-set tests that asserted "new copy wins on
collision" now assert the routing invariant: a legacy-shaped id resolves
to the legacy store and the path never consults the new store.
The open-predicate path (`#findRunsOpen`) is unchanged: an open `where`
has no id to route on, so it still unions both stores and dedupes.
Fixes TRI-12078
## Summary
Prevents concurrent environment setup requests from creating duplicate
Staging and Preview environments.
## Fix
Adds database-enforced uniqueness for root Staging and Preview
environments.
If two requests race, the losing request loads the environment created
by the winner and continues successfully instead of creating a duplicate
or returning an error.
## Summary
Upgrades the workspace to TypeScript 6.0.3 and applies the compiler,
type, and build configuration changes required to preserve package
layouts and existing runtime behavior, apart from correcting the HTTP
status field used for deployment connection errors.
## Compatibility
- Centralizes TypeScript 6.0.3 through the pnpm workspace catalog.
- Replaces compiler options and module resolution modes that TypeScript
6 no longer accepts.
- Restores explicit Node types where TypeScript 6 no longer includes
them transitively.
- Adds explicit declaration build roots that preserve each package's
existing output layout.
- Patches tsup to stop injecting the removed `baseUrl` option during
declaration builds.
- Uses type-only assertions for stricter typed-array and stream
definitions without changing runtime behavior.
- Reads the EventSource v3 HTTP status from `code`, so deployment
connection errors include it correctly.
- Keeps standalone CLI compatibility fixtures pinned to their existing
TypeScript version and lockfiles.
`turbo run typecheck` and the complete PR test suite are green.
## Stacked on #4284 — tests only
This PR contains **only the tests** that guard the production fixes in
#4284 (its base). Review #4284 first; this branch adds no production
code.
## What
Caller-driven replica-lag and idempotency guards for every fixed site:
- Each guard **drives the real exported caller** (route loader/action,
presenter `.call()`, service, or engine method) against a **real
Postgres** with the owning replica frozen via the shared
`laggingReplica` testcontainer primitive — never a store-seam
reimplementation.
- For a **fixed** site the guard goes **RED when the production change
is reverted**; for a **tolerated read-view** site it's a caller-driven
**GREEN** proof the miss self-heals (returns null/empty, no mutation,
row live on primary).
- The **global-scope idempotency** guard drives the real dedup + claim
path through a **real `MollifierBuffer` over a Redis testcontainer**
(real SETNX/poll/publish), and covers the cross-DB **andWait** waitpoint
wiring and the **expired/failed clear-and-recreate** reacquire cases.
Run with `vitest --no-file-parallelism` (testcontainers). Verified
GREEN, and revert→RED verified per fixed site.
## What & why
Two related correctness fixes for the run-ops DB split. Under the split,
run-store reads can route to a **lagging read replica**; a just-written
run/waitpoint/batch can then be missed, causing a wrong decision.
**1. Read-your-writes → owning primary.** Surfaced first as an
intermittent `wait.until({ idempotencyKey })` re-wait on retry. Auditing
the run-store read surface found the same class at sibling sites (some
gating mutations or returning spurious 404s, others
tolerable/self-healing). Reads that must observe their own writes now
route to the owning **primary**
(`findRun`/`findWaitpoint`/`findBatchTaskRunByFriendlyId` →
`*OnPrimary`, a primary re-read on a miss, or a retryable 404 where the
SDK polls). Read-view reads stay on the replica. All additive — the
happy path is unchanged.
**2. Global-scope idempotency across the split.** A `global`-scope key
carries no per-run salt, so the same `(env, task, key)` triggered
concurrently from parents resident on **different** run-ops DBs could
dedup-miss on each DB and create a duplicate (the per-DB unique index
can't enforce cross-DB uniqueness). Such triggers (global scope, or
scope-absent, while split is active) are serialized through the existing
Redis idempotency claim, the loser resolves the winner by id across both
DBs, and the claim is reacquired on the expired/failed
clear-and-recreate path. `run`/`attempt` scope embed the run id and
never contend.
## Stacked for review
This is the **base** of a 2-PR stack, split so review is easier:
- **This PR** — production code only (34 files).
- **Stacked tests PR →
https://github.com/triggerdotdev/trigger.dev/pull/4285** — the
caller-driven guards (55 test files) on top of this branch.
## Validation
Local run-ops split, **both 2-DB and 3-DB**, fresh boot on this branch:
SDK canary 64/71 (only the known concurrency/input-streams/s3 failures),
quarantine sweep **0 unexpected** (340 pass / 16 known / 4 local) in
each topology, dashboard e2e 0 failed. No product regressions.
## Summary
Correctness and performance fixes for deployments that split run data
across more than one database. Single-database / self-hosted deployments
are unaffected (they collapse to a single read/write path).
- **Batches list (dashboard):** for some organizations the Batches list
could hide older batches or show them out of order. It now orders and
paginates by creation time (with the id as a stable tiebreak), so every
batch appears exactly once, newest first. The pagination cursor format
changes; older in-flight cursors simply restart from the first page.
- **Reads:** waitpoint and snapshot lookups that are keyed by a single
run now read only the database that holds that run instead of querying
both, removing redundant queries on hot paths (unblock, snapshot reads).
- **Writes:** environment-scoped writes with no owning run (standalone
wait tokens, waitpoint tags, idempotency-key resets) now land in the
same database as that environment's runs, rather than defaulting to the
other one. An idempotency-key reset also falls back to the other
database when it matches nothing, so a reset still clears the key
wherever the run actually lives.
## Notes
Verified end-to-end against multi-database setups: run-keyed reads and
env-scoped writes land on the correct database with no cross-database
writes, and the batches list surfaces every batch in creation order. New
tests cover the batches ordering/reachability and the write-residency
routing.
## Summary
Adds experimental Node.js 24 and 26 task runtimes through the
`experimental-node-24` and `experimental-node-26` config values.
Existing runtime defaults and the `node`, `node-22`, and `bun` behavior
remain unchanged. The unprefixed `node-24` and `node-26` config values
remain unavailable until the runtimes are ready for general use.
## Design
Experimental config values normalize to canonical runtime identifiers
before build manifests are created, keeping deployment metadata and
execution behavior consistent. Kubernetes task pods also use the
runtime-default seccomp profile so modern Node.js versions fall back
from io_uring to checkpoint-compatible system calls.
## Summary
`batchTrigger` requests that set a per-item `idempotencyKey` failed with
a 500 when the run-store is split across databases: the per-item
idempotency lookup errored before any run was created. Batches without
per-item keys, single `trigger` idempotency, and batch-level
(`idempotency-key` header) idempotency were unaffected.
## Root cause
`findRunsByIdempotencyKeys` built its `UNION ALL` of per-key
point-lookups with `@trigger.dev/database`'s `Prisma.sql` /
`Prisma.join`, then executed it on whichever store client it was handed.
On the dedicated run-ops store that client is a *separate* generated
Prisma client, and a `Sql` object from a different generated client is
not recognized: the bare `$queryRaw(Prisma.join(...))` form dropped the
query text entirely (`Argument \`query\` is missing`). The
tagged-template form is no better here: joining nested `Prisma.sql`
fragments across the two clients mis-numbers the bound parameters
(`syntax error at or near "$1"`).
## Fix
Build the lookup as a plain parameterized string and run it via
`$queryRawUnsafe` with positional placeholders and bound values, so it
no longer depends on which generated client executes it. The query text
contains only static SQL and integer placeholders; every value
(`runtimeEnvironmentId`, `taskIdentifier`, each key) is bound, so it is
not a raw-interpolation site. Same per-key point-lookup shape as before,
no change on the single-client path.
Verified end-to-end against a bundled build with the run-store split
enabled: before the fix, `batchTrigger` with a per-item key 500s; after,
it returns the runs and dedups correctly across fresh, repeat, and mixed
batches.
## Summary
Updates the internal development, CI, and runtime-image Node version to
24.18.0. SDK compatibility coverage continues to include Node 20, 22,
24, and 26.
The Node type definitions and the package-manager lockfiles now resolve
against Node 24 types.
## Summary
Batch triggers that use per-item idempotency keys could take seconds
instead of milliseconds when the target task had a large run history.
This keeps the idempotency lookup fast regardless of how many runs a
task has accumulated.
## Root cause
The batch path checks which items already have runs by looking up their
idempotency keys with a single `WHERE runtimeEnvironmentId = ? AND
taskIdentifier = ? AND idempotencyKey IN (...)` query. On a very large
`TaskRun` table Postgres underestimates the row count of a specific
`(environment, task)` pair, so once the `IN` list grows past a handful
of keys it stops doing per-key index probes and instead scans every run
for that `(environment, task)` and filters the keys in memory. The cost
is then flat and large regardless of how many keys are being checked,
and a routine `ANALYZE` does not correct the estimate at that table
size.
## Fix
Look each idempotency key up on its own, batched into a `UNION ALL` of
point lookups (chunked, run with bounded concurrency). Each branch is an
equality on all three columns of the unique index, so the planner can
only do a per-key index probe and can never fall back to the range scan.
Same results, same columns, confined to the batch trigger path.
Routine dependency maintenance.
- Pin a few high-fanout transitive deps to current patched versions via
`pnpm.overrides`: `form-data`, `ws`, `undici`, `hono`. Lockfile-only (no
published-package dependency changes); net shrinks via dedup.
- Upgrade `nodemailer` 8 → 9 in `internal-packages/emails` (private
package). The SES transport already uses SESv2 and the
`createTransport`/`sendMail` API is unchanged, so no code changes were
needed. `@types/nodemailer` stays at 8 (no 9.x published yet; types are
compatible).
Verified locally: `pnpm i` clean; `pnpm run typecheck --filter emails`
and `--filter webapp` both pass.
## Summary
Run-graph data (runs, batches, waitpoints, and their related tables) can
now live in a database separate from the control plane, with every read
and write routed to the correct database by each run's residency. This
makes reading and writing run data more reliable once the two are split,
and is a no-op for single-database installs.
## Design
- Run-graph table access goes through the run-store router, which
selects the legacy or the new run-ops store per run instead of assuming
one shared client.
- The legacy run-ops client is now independently pointable, so legacy
run data can be served from its own database (and replica) rather than
the control-plane connection.
- Run-graph writes go straight to the run-graph database instead of
being forwarded through the control plane, and replication targets are
split so runs in the new database still replicate to analytics without
under-counting.
- Read-through slots refuse the control-plane client, so a missing
residency fails loudly instead of silently reading the wrong database.
- Migration `20260710120000_drop_remaining_run_graph_seam_foreign_keys`
drops the foreign keys that still crossed the run-graph / control-plane
seam, which is what lets the two live in separate databases.
The split stays off unless explicitly enabled and the two databases are
confirmed physically distinct; startup fails closed otherwise.
Verified by running the full dashboard end-to-end suite against both a
single-database configuration and a three-database configuration
(control plane, the new database, and a physically separate legacy
database), with runs on both residencies. No misrouted reads in either
configuration.
## Summary
v3 (the engine that ran the SDK v3 era, internally
`RunEngineVersion.V1`) is end-of-life. Following the removal of the v3
execution apps
([#4194](https://github.com/triggerdotdev/trigger.dev/pull/4194)) and
the legacy dev websocket
([#4198](https://github.com/triggerdotdev/trigger.dev/pull/4198)), this
removes the remaining v3 execution stack from the server.
Clients still on v3 (an old SDK or CLI that has not upgraded) keep
getting a clear "upgrade to v4" response. Triggers, batch triggers,
reschedules, and deploys that resolve to v3 are rejected with a graceful
4xx pointing at the migration guide, never a 5xx, so a stale client
cannot affect server health. Self-hosted instances still running v3
should stay on the 4.5.x release line until they migrate.
## What is removed
- The MarQS queue and its shared/dev queue consumers.
- The v3 socket.io namespaces (coordinator, provider, shared-queue) and
the v3 run lifecycle services (attempt, checkpoint, and batch-resume).
- The graphile-worker background job system; all live jobs already run
on `@trigger.dev/redis-worker`.
- The `DEPRECATE_V3_ENABLED` flag: v3 is now rejected unconditionally,
so the flag is gone.
- Unused v3 exports from `@trigger.dev/core` (the `v3/zodNamespace`
subpath and the legacy socket message catalogs) and the now-dead MarQS
environment variables.
## What stays
The v4 engine is untouched. The graceful v3 rejection boundary stays,
`determineEngineVersion` still detects a v3 project so it can reject it,
and the batch service plus batch-completion worker stay for current
clients. Live queue concurrency limits and metrics now read from the v4
run engine instead of MarQS, and a brand-new dev environment now
defaults to v4.
## Dependency cleanup
Removes webapp dependencies left unused by this change: `seedrandom` and
`semver` (only the removed v3 code used them) plus a set that was
already dead, their orphaned `@types` packages, and two dead files. Adds
a `knip:deps` script and a `knip.json` config so unused dependencies can
be found the same way going forward.
## Summary
Two threads on the run-ops split path.
Read path: per-item run reads are batched into grouped queries, a
waitpoint's connected-run reads are bounded, and the dedicated-schema
relation hydrators fetch only the requested columns instead of whole
rows. Retrieve also falls back to the other database when a routed read
misses, so a run whose physical residency diverges from its id shape is
still found rather than returning a spurious not-found. Fewer and
lighter queries on the run read path, with no change to results.
Mint-kind flip safety: flipping which database new runs mint to is now a
deterministic wall-clock cutover, for both per-org and global flips. For
a grace window every process resolves the same database, so a flip
cannot route two concurrent triggers that share an idempotency key to
different databases (which would bypass the per-database unique
constraint and create a duplicate run).
Supersedes the earlier #4205 and #4208.
Draft: validation in progress.
## Summary
Under high OTLP ingest volume, the whole decode, transform, and enrich
pipeline runs on the request event loop, so a single CPU core becomes
the ceiling while the rest sit idle. This adds an opt-in worker pool
that moves decode, transform, and LLM-cost enrichment onto worker
threads, keeping the main thread free for I/O. It is off by default
(`OTEL_TRANSFORM_WORKER_POOL_ENABLED`), so behavior is unchanged unless
enabled.
## Design
Workers do decode, filter, convert, and enrich (including LLM pricing
match). The main thread stays the single database reader: it loads the
pricing registry and broadcasts the compiled model rows to the workers
(re-broadcasting on every reload), so workers never touch the database.
The pure transform is extracted into a dependency-light module (no
Prisma/Redis/ClickHouse imports) so it can run inside a worker.
Importantly, the main thread keeps the existing single consolidated
insert path, so ClickHouse insert batching and part count are unchanged.
The parallelism buys CPU headroom, not more insert streams (which would
add merge pressure).
The worker is bundled as a standalone file at build time and ships in
the existing image with no Dockerfile change. In local load testing the
pool sustained roughly 2.6x the throughput of the single-thread path and
kept the main thread responsive under load.
## Problem
Several display/grouping issues in the **Errors** feature, all rooted in
how the ClickHouse error materialized views (`errors_mv_v1`,
`error_occurrences_mv_v1`) read the stored error JSON produced by
`parseError`:
1. **Messageless errors show "Unknown error".** An empty message falls
straight through `coalesce(nullIf(message,''), 'Unknown error')` to the
literal, even though the error's class `name` is available (e.g. an
Effect tagged error `ListMessagesError` with no message).
2. **Unrelated errors collapse into one group.**
`calculateErrorFingerprint` keys on `type : message : stack`, where
`type` is always the union tag (`BUILT_IN_ERROR`, …), `message` is
empty, and the stack isn't read — so every messageless built-in error
(and every string/custom error) hashes to the same constant input → one
fingerprint.
3. **error_type shows the internal tag.** `coalesce(type, name, …)`
always resolves to `type` (always present), so the column shows
`BUILT_IN_ERROR` instead of the real class name.
4. **Stack traces never populate.** The MVs read `error.data.stack`, but
the serializer stores the trace under `stackTrace` — so the column is
always empty.
## Fix
All display changes are `ALTER TABLE … MODIFY QUERY` on the two views
(migration `035`); the fingerprint change is in the webapp.
- **Fingerprint** (`errorFingerprinting.ts`): fall back **message → name
→ raw**. Messageless errors now group by class name (or raw value for
non-Error throws); message-bearing errors are **unchanged**
(short-circuits at `message`), so existing groups don't split — only
currently-messageless errors get their own group going forward.
- **error_message**: same `message → name → raw` fallback before
`'Unknown error'`.
- **error_type**: coalesce `name → code → 'Error'` (drops the reliance
on the union tag). Built-in → class name, internal → `code`,
string/custom → `Error`.
- **stack trace**: read `error.data.stackTrace`. Bounded as before
(serializer caps 50 frames / 1024 chars per line; MV clips to 2000
chars).
## Migration notes
- `MODIFY QUERY` swaps the view query in place (no drop/recreate gap);
Down restores the previous query.
- **Existing rows are left unchanged** — changes apply only to rows
inserted after the migration. No backfill.
## Tests
`errorFingerprinting.test.ts` — 57 pass, incl. new cases for messageless
class names, string/custom raw values, and stability of message-bearing
fingerprints.
Fixes the display-derivation half of TRI-11938 (error_type + stack
trace); relates to TRI-9254 and TRI-9250.
## Summary
The RBAC and SSO auth plugins can own their own database client, but
they could only read `DATABASE_URL`, so every connection they opened
landed on the primary. The host webapp now resolves writer and
read-replica URLs from its env (the same fallback chain its own Prisma
clients use: control-plane URL first, then the default) and passes them
to the plugins at create time via a shared `PluginDatabaseConfig`, along
with separate connection limits for writes (default 2) and reads
(default 5, tunable via `RBAC_DATABASE_*_CONNECTION_LIMIT` and
`SSO_DATABASE_*_CONNECTION_LIMIT`).
A plugin can then route hot-path reads (per-request auth checks, login
routing) to the read replica and keep only rare mutations on the
primary. With no replica configured, or no plugin installed, nothing
changes: the OSS fallback ignores the new option and keeps reading
through the Prisma clients it is already given.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Creating a Prisma migration now formats its schema first, keeping
migration-related schema edits consistently formatted without adding
work to the repository-wide format command. Run `pnpm run format:prisma`
to format either schema on demand.
## Summary
A Query page (TRQL) query that pulls fields out of a run's `output` with
JSON functions (`JSONExtractString`, `JSONExtractInt`, `JSONHas`, and
the rest of the family) failed with "The first argument of function ...
should be a string containing JSON, illegal type: JSON". Those queries
now work.
## Root cause and fix
`output` is a native ClickHouse `JSON` column, but `JSONExtract*`,
`JSONHas`, `JSONLength`, and `JSONType` all expect a String containing
JSON text. The compiler already swaps in the column's String companion
(`output_text`) when a JSON column is selected or compared, but not
inside function-call arguments, so it emitted `JSONExtractInt(output,
'x')` against the native column.
The fix prints the companion column for the first argument of these
functions when it resolves to a bare JSON field, keeping the table alias
when qualified (so it works in JOINs):
JSONExtractInt(output, 'x') -> JSONExtractInt(output_text, 'x')
JSONExtractArrayRaw(assumeNotNull(output), 'y') ->
JSONExtractArrayRaw(assumeNotNull(output_text), 'y')
It also reaches through value-preserving passthrough wrappers like
`assumeNotNull(...)`, while leaving value-changing wrappers like
`toJSONString(output)` on the native column (that argument is already a
String). The swap is also semantically correct, not just a type fix:
`output_text` is the unwrapped data JSON that the TRQL `output` model
already represents, so field paths line up.
Covered by printer unit tests and a ClickHouse integration test that
runs the whole family (plus the wrapped and `toJSONString` cases)
against a real native-JSON column. Both new cases fail with the exact
"illegal type: JSON" error without the fix.
## Summary
The runs page's empty-state check (whether an environment has ever had a
run, which decides between the "getting started" and "no runs match your
filters" states) ran a `findFirst` against the Postgres `TaskRun` table.
This moves it to ClickHouse, the same store the runs list itself reads
from, so the check no longer queries `TaskRun`.
## Design
Only the runs list triggers the check now (via an `includeHasAnyRuns`
flag); the other presenters that reuse `NextRunListPresenter` (API,
schedule detail, waitpoint detail, error group) no longer issue it. When
the list is empty it runs `SELECT 1 FROM task_runs_v2 ... LIMIT 1`
filtered on the full `(organization_id, project_id, environment_id)`
sort-key prefix with a configurable `created_at` lower bound
(`RUN_LIST_HAS_RUNS_LOOKBACK_DAYS`, default 30), so it hits the primary
index and reads minimal granules.
Results are cached in a tiered memory + Redis SWR cache. Only positive
("has runs") results are cached, so an environment with no runs is
always re-checked and its first run shows up immediately.
## Summary
Adds SDK and API support for run bulk actions. You can now create bulk
cancel or replay actions from `@trigger.dev/sdk` using run IDs or the
same filters as `runs.list()`, then retrieve, list, poll, or abort the
action by its `bulk_` handle.
Tests, docs, changesets added.
## Design
The dashboard bulk action service now accepts structured filters instead
of reading directly from a dashboard request, so the dashboard and API
share the same creation path. Replay actions created through the API are
attributed with the existing `api` trigger source, while
dashboard-created actions keep `dashboard`.
The SDK exposes the new surface under `runs.bulk.*`, including
`targetRegion` for replay region overrides and cursor pagination for
listing bulk actions.
## Filters and runIds
Nuance on filters. If `filter` is provided, it MUST have at least one
key. This is to remove the footgun of passing no filter and selecting
all runs.
```typescript
{ action: "cancel", runIds: ["run_1"] } // valid
{ action: "cancel", runIds: [] } // invalid, min(1)
{ action: "cancel", filter: { status: "FAILED" } } // valid
{ action: "cancel", filter: {} } // invalid
{ action: "cancel", filter: {}, runIds: ["run_1"] } // invalid
```
## Summary
Three fixes to the run-ops database split (the Cloud-only mode where
run-lifecycle rows live on a dedicated Postgres). All are inert in the
default single-database deployment.
The main fix: on the batch trigger paths, a parentless batch's item runs
chose their physical store from a fresh per-org mint-flag read at
processing time, so flipping an org's flag mid-batch could land an item
in a different store than its batch, breaking the `TaskRun.batchId`
foreign key (or silently orphaning the item). The other two harden the
split's safety nets: the schema-parity test now actually compares
columns, and the read fan-out gate now signals when it has been silently
disabled.
## Batch item residency
`RunEngineBatchTriggerService` (api.v2) and the BatchQueue item callback
(api.v3) now anchor each item's id mint on the batch's own friendlyId,
mirroring the already-safe `BatchTriggerV3Service`. Residency is a pure
id-shape check, so an item can no longer diverge from its batch across a
mid-batch flag flip. The pre-failed-run fallback is anchored the same
way (it also sets `batchId`), and the shared mint branch is consolidated
into one helper so every mint path stays in lockstep. No new database
queries; single-database mode is unchanged (a cuid-shaped batch
friendlyId yields a cuid item).
## Schema parity test
The parity test previously read only the dedicated schema and matched
model headers with regexes, so it never compared columns and could not
catch a run-subgraph column that diverged between the two physical
schemas. It now parses both schemas and asserts bidirectional
scalar-column parity (type, nullability, array-ness, default) across the
run-subgraph models, and fails on any field line it can't parse. Scoped
to the run-subgraph models so unrelated control-plane edits don't break
it.
## Read fan-out signal
The split read fan-out gate is decided by the object identity of the NEW
vs control-plane clients. It now warns when both run-ops URLs are set
but the NEW client isn't a distinct instance (fan-out silently off), and
a new test exercises the real topology-into-gate wiring so a future
refactor that aliases the clients can't disable fan-out unnoticed.
## Verification
New unit and glue tests cover all three changes; the DB-backed
residency, store-routing, and topology suites pass against real
Postgres; `typecheck` is clean for both packages.
## Summary
Running `pnpm run db:migrate` locally left `defaultPrices.ts` and
`modelCatalog.ts` in `llm-model-catalog` showing as modified every time,
a ~10k-line diff that only ever changed formatting. This stops the
churn.
## Root cause
The root `db:migrate` script ends in `&& turbo run generate`, which runs
the `generate` script in every package that has one, including this one.
The generator writes its output with `JSON.stringify` (quoted keys, no
trailing commas), but the checked-in copies had been reformatted by
oxfmt (unquoted keys, trailing commas). So the generator output never
matched what was committed, even though the parsed data was identical.
## Fix
Add the two generated files to `.oxfmtrc.json`'s ignore list and commit
the raw generator output, matching how other codegen files in the repo
are already handled (e.g. the tsql grammar). Generation is
deterministic, so `generate` and `format` are both no-ops on a clean
tree now.
No changeset: internal package, dev tooling only, no runtime or public
API change.
## Summary
On the run-ops database split, a run that waits (`triggerAndWait`,
`batchTriggerAndWait`, `wait.forToken`) could hang forever after its
wait had already completed. The runner reads a resume from
`/snapshots/since` exactly once: if that read returned the resume
snapshot without its completed-waitpoints, the runner logged "executing
without completed waitpoints", advanced its cursor, and never re-read
it, so the awaiting run never continued.
## Root cause
The resume snapshot and its completed-waitpoint rows were written as two
separate commits. This regressed when the split replaced Prisma's atomic
nested `connect` with an FK-free insert (in
[#4163](https://github.com/triggerdotdev/trigger.dev/pull/4163)), and
`/snapshots/since` is served from a read replica. A fetch landing in the
sub-millisecond gap between the two commits, or a multi-reader replica
serving the snapshot from a different point in time than its join rows,
delivered an empty resume. Because the runner consumes each snapshot
once and treats an empty resume as terminal, a single stale read was
fatal and produced a permanent, nondeterministic hang.
## Fixes
- Commit a snapshot and its completed-waitpoint links in one
transaction, restoring the atomicity the split removed.
- Repair the completed-waitpoints from the owning primary when a
multi-reader replica serves the snapshot without its join rows. This
covers single-waitpoint resumes, which carry no
`completedWaitpointOrder` and so were missed by the count-based repair.
- Read the primary in the checkpoint `WAIT_FOR_BATCH` pre-check, so a
batch that already resumed is not re-suspended into a stall.
- Fall back to the primary when a waitpoint token misses both read
replicas, so a token completed immediately after it was minted no longer
returns a spurious 404.
- Route batch-item creation by `batchTaskRunId`, consistent with the
batch-completion count and the row's foreign key.
- Reject control-plane-only relation selects on the dedicated schema
with a clear error instead of an opaque Prisma failure, and stop
`createDateTimeWaitpoint` bypassing residency routing through a caller
transaction.
Verified against the deployed split topology: a resume snapshot and its
completed-waitpoints are now always delivered together, so the runner
can no longer drop a resume.
Extend the SSO plugin contract for directory sync and apply membership
effects
from the accounts webhook worker: provision users in mapped groups (role
from
group mapping, else the org default role), deprovision on removal, and
keep a
sticky-removal tombstone so JIT never silently re-adds a removed user.
JIT and
Directory Sync coexist; roles default to Developer (the JIT default-role
picker
has no 'None'). Changing a group's role in the dashboard re-applies it
to that
group's current members immediately. The Directory Sync settings section
(group→role mapping, external-domain + manual-membership policy,
deferred Save)
appears once a domain is verified — independent of SSO — gated by the
hasSso
flag. The settings page polls the whole page while entitled with
override-aware
drafts so in-progress edits are never clobbered.
## Summary
On the run-ops split, NEW-residency runs could hang. Time-based waits
(`wait.for`, `wait.until`, `delay`, waitpoint tokens),
`batchTriggerAndWait`, and attempt starts stalled and never resumed.
Each was a run-ops read or update that hit the wrong database: either
the owning store's read replica when it needed read-your-writes, or the
wrong store entirely because it routed by an id that does not encode
residency.
## Fixes
**Waitpoint resume (the main hang).** The managed resume path reads a
run's completed waitpoints by snapshot id
(`findSnapshotCompletedWaitpointIds`). Snapshot ids are cuids, which
always classify to the legacy store, so a NEW run's join rows (which
live on the new store) were never found. The resumed run saw zero
completed waitpoints and hung. It now fans out across both stores and
merges, like its sibling readers.
**Batch completion.** Batch item completion
(`updateManyBatchTaskRunItems`) routed by the item id, which is also a
cuid, so a NEW batch's items were updated on the wrong store, matched
zero rows, and the batch was treated as already complete (its parent's
`batchTriggerAndWait` then hung). It now routes by the batch id, which
does encode residency, matching the sibling `countBatchTaskRunItems`.
**Read-your-writes on the resume path.** The block-time
pending-waitpoint check (`countPendingWaitpoints`) and the attempt-start
lock check (`findRun` in `startRunAttempt`) both read the owning store's
replica with no read-your-writes guarantee, so a just-committed
waitpoint completion or dequeue lock could be missed under replica lag
and strand the run. Both now read the owning primary.
Each fix ships with a two-database store or engine test that reproduces
the hang and passes with the fix.
## Problem
The run-ops split mints NEW-store run ids as **27-char base62 KSUIDs**.
The supervisor writes the run id into the Kubernetes pod name
(`runner-<id>`), and pod names must be DNS-1123 labels (lowercase
`[a-z0-9-]`) — so uppercase base62 ids make k8s reject the pod (422) and
**those runs never launch** (they loop in `PENDING_EXECUTING` until the
heartbeat-stall handler nacks them, forever). `.toLowerCase()` can't fix
it: base62 has both `A`(10) and `a`(36) as distinct symbols, so folding
collides distinct ids and destroys sort order.
## Fix: change the encoding, not the structure
Mint a **26-char lowercase base32hex** run id:
```
run_<24-char base32hex core><region char><version char>
[ 6-byte ms timestamp ][ 9 CSPRNG bytes ]
```
- **base32hex** (RFC 4648 §7, alphabet `0-9a-v`): lowercase,
order-preserving, DNS-safe; 15 bytes → exactly 24 chars, no padding.
Hand-rolled encode/decode (no new dependency).
- **48-bit ms timestamp** in the leading bytes → plain string sort ==
creation order at millisecond resolution.
- **72 bits CSPRNG** entropy; PK unique constraint is the backstop (no
retry loop).
- **region / version** are raw positional chars (read via one `charAt`
before decoding/routing), version = `"1"`.
DNS-safe from birth and hyphen-free, so **firekeeper is unchanged** —
`runner-<id>-attempt-N` → strip `runner-`, cut at first hyphen still
recovers the exact id incl. region+version.
## Residency discriminator: length → version char
`classifyKind`/`classifyResidency` (`runOpsResidency.ts`) previously
distinguished NEW vs LEGACY by **id length**. That gets ambiguous with a
third format. It now discriminates on the **version char at a fixed
position** (`isRunOpsIdBody`: 26 chars, `[25] === "1"`, base32hex
alphabet) → NEW; everything else → LEGACY. Total, never throws. The
`Residency` (NEW/LEGACY) contract the routing store consumes is
unchanged; the `"ksuid"` `ResidencyKind` label is retained only because
it's the persisted `runOpsMintKsuid` feature-flag value.
## Scope / verification
- Generator + discriminator in `@trigger.dev/core` isomorphic; mint path
+ all id-shape call sites swept (~40 webapp files); changeset added
(`@trigger.dev/core` patch).
- Core unit tests (encode/decode round-trip + property, generator shape,
ms sort-order incl. intra-second, parse partitioned-vs-legacy,
firekeeper round-trip): **24 pass**. `@trigger.dev/core` builds; webapp
typechecks; format/lint clean.
## Open decisions (flagged, not silently chosen)
1. **Backward-compat**: existing 27-char base62 KSUID runs now classify
LEGACY. On test cloud these are the broken/looping runs that never
completed, so this is acceptable — but worth a conscious call before
prod. No transitional length-recognition added (keeps the discriminator
clean).
2. **Storage collation**: the sort guarantee is byte-order — if the
run-ops id column is `TEXT` with default locale collation it's silently
not honored. Confirm whether `COLLATE "C"` / `BYTEA` is needed on the
run-ops schema.
3. **Region sourcing** wiring — see `regionCharForRegion` /
`REGION_CODES`.
---
## ⚠️ Required migration — deploy in lockstep
This PR renames a persisted feature-flag key/value and an env var. These
are **not** changed by the code alone and must be migrated when this
deploys, or affected orgs silently fall back to `cuid` minting (no crash
— `defaultValue: "cuid"`):
1. **Env var** (terraform): `RUN_OPS_MINT_KSUID_ENABLED` →
`RUN_OPS_MINT_ENABLED` (carry the value over).
2. **DB** `organization.featureFlags`: migrate both the key and value
together:
- key `runOpsMintKsuid` → `runOpsMintKind`
- value `"ksuid"` → `"runOpsId"`
Until an org's flag row is migrated, its `runOpsMintKind` lookup misses
and it mints `cuid` (legacy) — so no NEW-store ids for that org until
the data lands.
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
## What
Adds the ability to **automatically migrate the dedicated run-ops
database** (the NEW DB in the run-ops split), matching how every other
database in the system is migrated. Follow-up to the run-ops split
activation.
## Changes
- **Migrate runner** — new
`internal-packages/run-ops-database/scripts/migrate.mjs`, exposed as
`db:migrate:deploy` / `db:migrate:status`. Connects via
`RUN_OPS_DATABASE_URL` (the same var the app uses) and expands `${VAR}`
refs like Prisma's dotenv.
- **Self-host** — `docker/scripts/entrypoint.sh` runs the run-ops
migration on boot when the DB is configured, gated by
`SKIP_RUN_OPS_MIGRATIONS`. Single-DB installs never set the URL, so it's
a clean no-op.
- **Single env-var family** — the run-ops DB is now addressed by one
canonical `RUN_OPS_*` family, connect path and migrations resolving the
identical URL:
- `RUN_OPS_DATABASE_URL` (writer) — replaces `TASK_RUN_DATABASE_URL`
- `RUN_OPS_LEGACY_DATABASE_URL` — replaces
`TASK_RUN_LEGACY_DATABASE_URL`
- `RUN_OPS_DATABASE_READ_REPLICA_URL` — replaces
`TASK_RUN_DATABASE_READ_REPLICA_URL`
- the old `TASK_RUN_*` aliases, the `??` coalesce, the
`runOpsNewDatabaseUrl` indirection, and the migrate-only `directUrl` are
all removed (consumers read `env.RUN_OPS_DATABASE_URL` directly).
`directUrl` was dropped because it was only ever used by `prisma
migrate` (never the app runtime) to bypass a pooler for advisory locks —
premature here since the run-ops connection isn't wired to the app yet.
If a pooler is later introduced for the app, a direct URL can be
reintroduced then.
## Safety
- **Pure rename** — nothing deployed sets any `TASK_RUN_*` var (the
split isn't activated anywhere yet; `.env.example`, docker-compose, and
cloud already use `RUN_OPS_*`), so there is no config migration.
- **Single-DB / self-host** — no new required env var; entrypoint and
migrate are no-ops when `RUN_OPS_DATABASE_URL` is unset.
- **Cloud** — runs migrations as pre-deploy ECS tasks (companion cloud
PR), calling these same `db:migrate:deploy` / `db:migrate:status`
commands.
## Verification
- Live migration against a fresh scratch DB with only
`RUN_OPS_DATABASE_URL` set: both migrations applied, no `P1012`/`P1013`;
`${VAR}` expansion, idempotent re-run, `status`, and no-op skip all
pass.
- Schema parity 4/4; `typecheck --filter webapp` 18/18; affected
split/replication tests 34/34.
## Scope
This delivers automatic migrations only. Enabling the app to *use* the
new DB (setting `RUN_OPS_DATABASE_URL` + `RUN_OPS_SPLIT_ENABLED` on the
service) is a separate activation step.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
## Problem
`LogicalReplicationClient` uses a Redlock leader lock to guarantee a
single active consumer per Postgres logical replication slot. The lock
resource was keyed on the client `name`:
```
logical-replication-client:${this.options.name}
```
A slot permits exactly one consumer, so the lock's job is to serialize
consumers **of a given slot**. Keying it on `name` breaks that whenever
two clients target the same slot with different names — most notably
across a rolling deploy where the client `name` changes but `slotName`
does not. Both acquire *distinct* locks, both consider themselves
leader, and the second to reach `START_REPLICATION` hits `replication
slot "<slot>" is active for PID <n>`. Because that query was
fire-and-forget and its failure was only logged (no retry), the consumer
stopped and replication stalled until the process was restarted.
## Fix
**1. Key the leader lock on `slotName`** — the actual single-consumer
resource:
```
logical-replication-client:${this.options.slotName}
```
Consumers of the same slot now contend on the same lock and hand off
cleanly across restarts/deploys; different slots stay independent.
`name` is kept for logging and the pg `application_name`.
**2. Self-healing resubscribe** (`resubscribeOnFailure`, opt-in) —
instead of logging-and-dying, a client re-subscribes with exponential
backoff after a lost election or a failed `START_REPLICATION`, so a
rolling deploy self-heals: the incoming pod retries until the draining
pod releases the slot, then takes over. Safety:
- `#cleanupAttempt()` unconditionally ends the pg client (freeing the
walsender) and releases the leader lock before rescheduling — retries
never leak connections/locks.
- `shutdown()` sets an intentional-stop latch re-checked after every
`await` in `subscribe()` (and aborts the lock-acquire spin), so a
resubscribe can never race or outlive an intentional shutdown.
- Backoff resets only on genuine stream start, so a permanently stuck
slot backs off to the ceiling and logs loudly rather than tight-looping;
an epoch guard neutralises stale `START_REPLICATION` catches.
Runs- and sessions-replication opt in and use `shutdown()` for all
intentional stops.
**3. Observability** — the admin runs-replication status route probed
the old name-keyed Redis key (would report `leader:false` for every
source after fix#1); now probes the slot-keyed key.
## Tests
`internal-packages/replication/src/client.test.ts` (real Postgres +
Redis containers):
- same-slot/different-name → second client must not double-lead or race
into "slot is active" (the regression)
- a failing `START_REPLICATION` retry loop must not leak connections or
locks
- `shutdown()` during an in-flight `subscribe()` must not leave a zombie
leader
- `subscribe()` after `shutdown()` re-arms `resubscribeOnFailure`
- self-heals once the leader releases the slot
Plus the multi-source wiring test updated to the slot-keyed lock keys.
## Rollout
With the self-healing resubscribe, this ships as a **plain rolling
deploy** — the incoming pods retry across the one-time lock-key
transition and take over once the old pods drain (a brief replication
stall that the durable slot replays on reconnect — no data loss). No
stop-before-start required.
## What
Introduces the run-store routing seam and the run-engine read seams that
let run lifecycle operations be dispatched to either the control-plane
database or a separately-generated run-ops database, depending on where
a run/batch resides.
- **run-store** (`internal-packages/run-store`): adds `runOpsStore.ts`
and substantially expands `PostgresRunStore.ts` so the store can resolve
residency and route reads/writes to the correct backing client.
`types.ts` grows the routing/residency types; `NoopRunStore.ts` is
removed.
- **run-engine** (`internal-packages/run-engine`): adds
`engine/controlPlaneResolver.ts` and routes the per-system read paths
(dequeue, enqueue, waitpoint, checkpoint, run-attempt, ttl, delayed-run,
execution-snapshot, pending-version, debounce, batch) through the
resolver/store instead of talking to a single Prisma client directly.
`engine/errors.ts`, `engine/types.ts`, and `engine/index.ts` are
extended to support injecting the store/resolver.
Three fixes are included on top of the seam work:
- `c6cadd85f` — routes read-your-writes to the owning store's
**writer**, not its lagging replica, so an operation immediately reading
back what it just wrote sees a consistent result.
- `05c912e05` — normalizes run-ops-generation Prisma errors to the
control-plane error class at the store **write boundary**, so
`instanceof` checks and the `P2002` → 422 handling continue to work
across the separately-generated run-ops Prisma client.
- `88d12907f` — resolves NEW-resident batches in
`ApiBatchResultsPresenter` by routing the batch read through the store,
so a dedicated-DB batch resolves instead of returning 404.
The change is heavily test-first: the bulk of the diff is new
unit/integration coverage for the store routing, residency, and each
run-engine system's control-plane resolver path.
## Why
PR4 of the run-ops split stack (PR1–PR3 land the ClickHouse
test-container and earlier plumbing). This PR is the read-path
foundation: it adds the seam and read-routing but leaves the write path
to route through the same seam in a later PR. Behavior-changing where
the three fixes above touch existing read-your-writes /
error-normalization / batch-resolution paths; otherwise additive (new
store module, new resolver, injectable dependencies with existing
single-client behavior preserved when no dedicated store is configured).
## Tests
Extensive new vitest coverage under `run-store/src/*.test.ts` (routing,
residency, dual-schema select, cross-generation error normalization,
read-after-write, idempotency dedup, mixed residency, waitpoint
co-location) and `run-engine/src/engine/**/*.test.ts` (per-system
`controlPlaneResolver` tests, injectability, block-edge residency,
waitpoint read residency, trigger-create routing, lifecycle router).
Testcontainers-backed; no mocks.
## Notes
Draft, **stacked on #4114** (`runops/pr03-clickhouse-tc`). Review that
first; this diff is against it.
Server-change / changeset note to be added at stack-assembly time.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
## What
- Adds `composeTaskRunVersion` to `@internal/clickhouse` (exported from
the package index). It packs a small `originGeneration` epoch into the
top 8 bits of the ReplacingMergeTree version and keeps the producer's
own LSN in the low 56 bits, so `task_runs_v2` rows replicated from more
than one Postgres producer become globally comparable while preserving
in-producer ordering. Single-producer setups never call it and keep
using the raw LSN version.
- Adds unit coverage for the helper in `taskRuns.test.ts` (bit layout,
ordering, epoch precedence, range validation).
- Adds two run-engine tests that exercise the cross-Postgres-version
testcontainer fixture:
- `heteroPostgresFixture.test.ts` — a smoke test asserting byte-identity
and identical `ORDER BY` (under a pinned ICU collation) across two
different Postgres major versions.
- `crossVersionCompat.test.ts` — mirrors the run-engine's real raw-SQL
surfaces and asserts byte-identical, ordering-identical results across
the two versions. Ships with an env-gated block (skipped by default)
that can be pointed at a real dedicated database in CI.
## Why
Third PR in the run-ops split stack. It is purely additive: it
introduces one new exported helper plus tests and changes no runtime
call sites, so on its own it has no runtime behavior change. It lays
down the version-composition primitive and the cross-version
compatibility proof that later PRs in the stack rely on.
## Tests
- New unit tests for `composeTaskRunVersion` in
`internal-packages/clickhouse/src/taskRuns.test.ts` (run against a real
ClickHouse testcontainer).
- New cross-version tests in
`internal-packages/run-engine/src/engine/tests/` running against real
Postgres containers of two different major versions (no mocks). The
env-gated dedicated-database block is skipped unless its URL is set.
## Notes
Draft, **stacked on #4113** (`runops/pr02-db-foundation`). Review that
first; this diff is against it.
Server-change / changeset note to be added at stack-assembly time.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
## What
The **dedicated run-ops database** foundation for the split: a
standalone Prisma package plus the infra to run and migrate it.
- **`internal-packages/run-ops-database`** — a new Prisma package
(`@internal/run-ops-database`) whose schema mirrors the run-execution
tables that will live on the dedicated DB, with its own generated
client, migrations, and migration runner.
- **`prisma/schema.parity.test.ts`** — a parity test that guards the
run-ops schema against drift from the control-plane schema for the
mirrored tables.
- **Docker** — a Postgres 17 service (`docker/Dockerfile.postgres17`,
`docker/docker-compose.yml`) so the dedicated DB is available locally
under the run-ops compose profile.
- **Testcontainers** — hetero fixtures (PG14 legacy + PG17 dedicated) so
later PRs can exercise cross-database behaviour with real containers
rather than mocks.
## Why
This is the **second PR in the run-ops split stack**, stacked on the
core primitives. It stands up the dedicated database and its tooling.
There is **no runtime wiring** into the webapp here — the app does not
read or write this DB yet; that arrives in later PRs. On its own this PR
only adds a package, a docker service, and test fixtures.
## Tests
Schema-parity test for the run-ops schema; hetero testcontainer fixture
smoke test.
## Notes
- Draft, **stacked on #4112** (`runops/pr01-core-residency`). Review
that one first; this diff is against it.
- Server-change / changeset note to be added at stack-assembly time.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Bumps the internal/toolchain Node version to the latest 22.x LTS
(`22.23.1`) and standardises it across the repo. Scope is the **platform
toolchain + the repo's own runtime images** (all `20 → 22` *upgrades*,
off the now-EOL node 20).
### Main changes
- Node `20.20.2 → 22.23.1` across all CI workflows, `.nvmrc`,
`CONTRIBUTING.md`, and the OSS `docker/Dockerfile` (digest-pinned).
- `@types/node → 22.20.0` (root dep + pnpm `overrides`, so the whole
workspace resolves to it); lockfile regenerated.
- `sdk-compat` matrix: adds Node 24 + 26 (keeps 20, still in `engines`).
- **App runtime images → node 22** (were on EOL node 20):
`apps/coordinator` → `node:22.23.1-bookworm-slim`;
`apps/docker-provider` + `apps/kubernetes-provider` → `node:22-alpine`
(reusing the exact digest `apps/supervisor` already runs, so all four
worker images are now identical). Stage aliases renamed off `node-20`.
### Possible issues / test notes
- `@types/node` 22.x can surface new TS errors — typecheck (now on 22)
is the gate.
- **Smoke-test the v3 worker path** — `coordinator` (`crictl`/CRI calls)
and the docker/kubernetes providers (talking to their daemons) now run
on node 22 (alpine/musl for the providers). Upgrade off EOL so low-risk,
but it's deployed runtime code with its own `publish-worker.yml`
pipeline.
## What & why
Two related fixes to how new cloud organizations get onboarded onto the
Free plan.
### 1. Route new cloud orgs through plan selection
New cloud organizations were created already activated, so they skipped
the plan-selection step and went straight to creating projects — which
meant their plan and usage limits were never set up. They're now created
deactivated and routed through plan selection, which activates them once
a plan is chosen. Self-hosted installs have no plan-selection step, so
they're activated immediately on creation and are unaffected.
The `Organization.v3Enabled` field is renamed to `isActivated` to better
describe what it now gates. It's mapped to the existing `v3Enabled`
column, so there's no data migration — only a schema/code rename.
### 2. Allow selecting the Free plan without GitHub verification
Choosing the Free plan no longer requires connecting and verifying a
GitHub account. The plan is applied immediately when selected. This
removes:
- the "Connect to GitHub" dialog and the GitHub-verified badge from the
plan picker
- the account-rejected state
- the now-unreachable GitHub-connect return routes
## Notes
- These changes pair with the corresponding change in the billing
service that applies the Free plan directly; they should be released
together.
## Testing
Verified locally end to end: a new cloud org is routed to plan
selection, the Free plan applies in one click with no GitHub step, the
org is activated, its usage allowance is provisioned, and it lands on
the new-project page.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Once this is merged, oxlint is at a pretty sensible baseline.
**Enable `no-unused-vars`, `typescript/consistent-type-imports`, and
`import/no-duplicates` lint rules**
Turns on three previously-disabled oxlint rules across the monorepo and
fixes all violations:
- **`no-unused-vars`** – enabled as an error with standard ignore
patterns: unused function arguments are ignored by default (`args:
"none"`), variables/caught errors/destructured array elements prefixed
with `_` are allowed, and rest siblings are permitted.
- **`typescript/consistent-type-imports`** – enforced as an error; all
type-only imports now use the `import type` syntax.
- **`import/no-duplicates`** – enforced as an error; duplicate import
statements from the same module have been merged.
The remaining commits clean up the violations found across the codebase:
removing unused variables/imports/type aliases, adding `_` prefixes to
intentionally unused bindings, fixing duplicate imports, and converting
value imports to `import type` where appropriate.
## ✅ Checklist
- [ ] I have followed every step in the [contributing
guide](https://github.com/triggerdotdev/trigger.dev/blob/main/CONTRIBUTING.md)
- [x] The PR title follows the convention.
- [ ] I ran and tested the code works
---
## Testing
`pnpm run typecheck --filter @internal/clickhouse` passes. This only
adds a ClickHouse input-format setting to existing insert calls; the
setting affects type inference for newly-inserted/merged data and is
non-destructive to existing rows.
---
## Changelog
Sets
`input_format_json_infer_array_of_dynamic_from_array_of_different_types
= 1` on every native-JSON insert path:
- `task_runs_v2` (`output`, `error`) — `insertTaskRuns`,
`insertTaskRunsCompactArrays`, and the async-insert variants
- `task_events_v1` / `task_events_v2` (`attributes`)
- `metrics_v1`
- `sessions_v1`
### Why
Our JSON columns contain arrays with mixed element types (e.g.
`[{"key":"value"}, "string", "string"]`). With this setting off — which
is the effective default under `24.12` compatibility — ClickHouse infers
those as deeply nested unnamed `Tuple(JSON, Nullable(String), …)` types.
ClickHouse 26.2 introduced `input_format_binary_max_type_complexity`
(default 1000), and those tuple type trees exceeded the limit, causing
background merges to fail with **Code 117**.
With the setting on (the default since 25.8), mixed-type arrays are
inferred as a single `Array(Dynamic)` — a simpler, flatter type
representation that never approaches the complexity limit, even once the
upstream default limit is restored.
Setting this explicitly at insert time keeps behavior deterministic and
version-controlled, so it does not depend on the server profile or a
future compatibility bump. This is a forward-only change: it only
affects newly inserted/merged data and does not rewrite existing parts.
Our read path re-serializes these columns to strings (`toJSONString` via
the materialized `*_text` columns), so the internal Tuple →
Array(Dynamic) representation change is transparent to the application.
### Companion server-side setting
To also apply this on the ClickHouse side (covers merges and any writes
not going through these code paths), set it on the default user:
```sql
ALTER USER default SETTINGS input_format_json_infer_array_of_dynamic_from_array_of_different_types = 1;
```
---
## Screenshots
_N/A_
💯🤖 Generated with [Claude Code](https://claude.com/claude-code)
https://claude.ai/code/session_01AaChyhestFMBYBWh6bgcCF
---
_Generated by [Claude
Code](https://claude.ai/code/session_01AaChyhestFMBYBWh6bgcCF)_
---------
Co-authored-by: Claude <noreply@anthropic.com>
## Summary
When a scheduled task fires for an organization that is out of
entitlements, the trigger can't proceed. That's an expected outcome, but
it was being logged at error level and surfaced as a failure.
## Fix
The trigger callback now classifies an out-of-entitlements result as its
own error type (`OUT_OF_ENTITLEMENTS`), and the schedule engine logs
both that and the existing queue-limit result as warnings rather than
errors. The run still doesn't fire and the `schedule_execution_failure`
metric still records the outcome (now tagged `out_of_entitlements`), so
nothing about observability or behavior changes beyond the log level.
## Summary
Adds Billing Limits to the webapp.
Customers can set a monthly spend cap. When usage crosses the limit,
billable environments enter a grace period. If the limit is not resolved
before grace expires, new triggers are rejected until the organization
increases or removes the limit.
Eric Allam