A run's physical table is fixed by its id format, not an org's current
runTableV2 flag. An org that was on v2 then flipped the flag off still holds v2
runs (they stay readable, routed by id), so scoping the dedup and hierarchy
reads to "legacy" off the per-org flag would miss those v2 runs: it would
silently drop a v2 run's children and parent on retrieve, and let a duplicate
through idempotency dedup. Gate the scope on whether ANY v2 run can exist in
the deployment (the native realtime master switch) instead. While native is off
no v2 run exists anywhere, so "legacy" is safe and still skips the empty
task_run_v2 query; once native is on, every read covers both tables.
The two realtime v1-streams routes (read/create + append) resolved a
target:"parent"|"root" via a table-bound parentTaskRun/rootTaskRun relation
select, which returns null for a cross-table parent/root in the runTableV2 mixed
window and 404s a target that exists. Select the scalar
parentTaskRunId/rootTaskRunId and resolve the target by id through RunStore
(routes by id format), matching the presenter fixes. The "self" target is
unchanged.
Make the v2 path both correct and performant for turning the flag on:
- findRuns gains an optional tables: "legacy" | "both" scope (mirroring findRun),
threaded through hydrateChildRuns/hydrateParentAndRoot. While an org is not on
v2 its runs only live in TaskRun, so callers pass "legacy" to skip the empty
task_run_v2 query.
- ApiRetrieveRun resolves parent/root and children in parallel (one round-trip
instead of two) and scopes the reads to legacy for non-v2 orgs, so the public
run-retrieve no longer pays an extra both-table query on every call.
- The run-inspector side panel resolves parent/root by id across both tables
(was a table-bound relation select that returned null for a cross-table parent
in the mixed window).
- recover-stuck-runs joins TaskRunExecutionSnapshot against TaskRun UNION
task_run_v2 so a stuck v2 run is found and re-enqueued.
The idempotency-key dedup is a non-id predicate, so RunStore read BOTH run
tables in parallel on every idempotency-keyed trigger, including orgs not cut
over to v2 (whose runs only live in TaskRun, so the task_run_v2 query is always
empty; while native realtime is off that is every org). Add an optional
`tables: "legacy" | "both"` scope to findRun and pass "legacy" from the
idempotency concern when the org is not on v2, keeping the trigger hot path
single-table.
Backfills cross-table tests the audit flagged as missing: findRun legacy-scope
skips task_run_v2, and clearIdempotencyKey fans out across both tables
(byPredicate hits v2; a mixed byFriendlyIds array clears both).
The Electric dual-shape merge was a bridge to let the Electric backend observe
v2 runs during the cutover, but Electric is short-lived and the merge taxed
every tag/batch realtime feed with a second long-poll the moment it deployed.
Gate the v2 run table on the native realtime backend instead (the native client
is table-agnostic and observes v2 runs directly), so a run only routes to
task_run_v2 once its org is on native. Remove the merge module and restore the
single-table Electric proxy.
The cross-table correctness work stays: a v2 run can still have a cross-table
parent or child once an org flips, so the cancelRun cascade, metadata
parent/root routing, the one-time-token claim, and the findRuns guard all still
apply regardless of realtime backend.
- The strengthened findRuns guard threw on GET /api/v1/runs/:runId/spans/:spanId,
which pages child runs with take and no orderBy across both tables. Add a
createdAt order so it takes the bounded cross-table merge (and the 50-row cap
is now deterministic, most recent first) instead of throwing for every org.
- Key the one-time-use-token cross-table claim on the token alone (a reserved
task slot), matching the task-independent oneTimeUseToken unique constraint,
so a multi-task token cannot mint twice across the flip. Stop excluding
triggerAndWait from the token claim. Always resolve a held claim on the
success path (publish, else release) so it cannot leak until its TTL.
The guard added in the previous commit makes that call throw rather than
return a non-deterministic cap; this test asserted the removed cap behavior.
The throw is covered by the guard test alongside the skip/cursor guards.
An unordered take capped each run table independently and concatenated the
two results, so a both-table read could silently drop one table rows once
the other filled the cap. Reject it like the existing skip and cursor guards;
callers that need a bounded cross-table read pass an orderBy for the keyset
merge.
A cuid parent (TaskRun) with a ksuid child (task_run_v2): cancelling the
parent must cascade to the child in the other table. Fails against the old
table-bound childRuns relation, passes with the cross-table findRuns lookup.
Routes that walk the run hierarchy through a Prisma relation only see one
physical table, so during a runTableV2 flag flip (a parent and child on
opposite tables) they silently miss the cross-table run. This closes the
reachable cases:
- cancelRun resolves child runs across both tables, so cancelling a parent
cascades to a child in the other table instead of leaving it executing
and holding concurrency.
- updateMetadata routes metadata.parent/root operations to the scalar
parent/root id, so they reach a parent in the other table instead of
falling back to the child run.
- a one-time-use token with no idempotency key now takes a cross-table
claim for v2 orgs, so two presentations straddling a flip cannot each
mint a run in a different table.
- the Electric shape merge reports up-to-date only when both tables are
caught up, so a multi-chunk initial snapshot no longer drops the rows
that arrive after the first chunk.
Restore the both-table Electric shape merge so tag-list and batch realtime
feeds observe runs in TaskRun and task_run_v2 together, and gate the v2 run
table on the runTableV2 flag alone (drop the native-realtime coupling). New
runs route to task_run_v2 whenever an org has the flag on and stay visible in
realtime on the existing Electric backend.
Single-run feeds route to one table by id format; only tag and batch feeds fan
out to both shapes under one composite continuation.
Completes the Electric-merge removal: a run only routes to task_run_v2 when the deployment has native realtime enabled and the org's realtimeBackend flag is native. Electric shapes are single-table and can't observe a v2 run, so without this gate a v2 run would be realtime-invisible. shouldUseV2RunTable takes the native-realtime master switch as a parameter (kept env-free for unit tests); the trigger mint site and the idempotency pre-gate claim both pass it.
Electric realtime shapes are bound to a single table, so a task_run_v2 run was invisible to realtime subscriptions. The previous approach merged two Electric shapes per tag/batch feed under a composite cursor, which doubled Electric long-poll connections for those feeds. Electric is being retired in favor of the native realtime backend, which is table-agnostic and already observes both run tables, so that merge is throwaway.
Drop the Electric dual-shape merge (revert realtimeClient to its single-table form, remove the merge module) and instead gate runTableV2 on the native backend: a run only routes to task_run_v2 when the deployment has native realtime enabled and the org's realtimeBackend flag is native. This keeps v2 runs realtime-observable without touching Electric, and the gate auto-satisfies once Electric is removed and native is the default. The idempotency pre-gate claim inherits the same gate.
The two-table shape merge could leave one upstream fetch pending without a rejection handler when it aborts the race loser or rethrows from the catch block. Attach a detached no-op catch to both fetches up front so an abandoned fetch can never surface as an unhandled rejection on any path. Also document that a tag/batch subscription opens two upstream Electric connections while an org spans both run tables.
When the two-table realtime shape merge returns as soon as one upstream shape yields, it aborts the other fetch and returns immediately. That promise was left without a rejection handler, so the abort could surface as an unhandled rejection on the server. Attach a no-op catch to the aborted fetch.
The parent/root/child hydration that resolves a run's hierarchy across both run tables looked runs up by id alone. Those pointers are now plain scalars with no foreign-key enforcement, so a stale or malformed pointer could resolve to a run in another environment and leak its metadata through the run and span presenters. Scope every lookup to the run's runtimeEnvironmentId, restoring the same-environment guarantee the table-bound relation select used to provide.
runTableV2 is resolved per organization only, so a global toggle on the admin flags page did nothing. Mark it read-only there to remove the misleading control; per-org control stays on the org dialog.
A run routed to task_run_v2 was invisible to the Electric realtime feed, whose shapes were bound to the TaskRun table, so subscribeToRun, useRealtimeRun, and run polling returned nothing for those runs. Single-run subscriptions now route the shape to the correct table by id format, and the tag and batch feeds run two upstream shapes (TaskRun and task_run_v2) merged under one composite cursor the client round-trips opaquely, so no SDK change is needed.
Concurrent same-key triggers that straddle a runTableV2 flag flip can mint into different physical tables (cuid to TaskRun, ksuid to task_run_v2), whose per-table unique constraints cannot see each other, so neither insert conflicts and two runs share one key. The pre-gate claim now resolves its backend through a claim-only Redis buffer when the mollifier buffer is absent, so it serialises these triggers instead of falling open. v2-cutover orgs are claim-eligible for every idempotency-keyed trigger, including triggerAndWait, debounce, and one-time-use tokens, and the claim-resolved path blocks the parent on the winner's waitpoint.
findRuns rejects a Prisma cursor or a negative take on a both-tables read (neither can span two tables) instead of silently returning a wrong or empty result, and tablesForWhere now routes a plain id or friendlyId equality to the single matching table by id format, not just id:{in} lists. Also documents that the cross-table merge comparator assumes the en_US database collation and the COLLATE C fix needed for other collations.
TaskRunV2 declared implicit many-to-many relations (tags, connectedWaitpoints) whose join tables were never created by any migration and are absent from the database. Nothing reads them (v2 run tags use the scalar runTags array), so they were pure schema-vs-migration drift. Removing them makes the schema match the database with no migration.
## Summary
The logs search page (behind a feature flag) ran ClickHouse out of
memory when browsing back over long time ranges. This keeps it within
bounded memory and fixes a pagination bug that could skip or duplicate
rows at a page boundary.
## Fix
Memory: the list query reads in sort-key order, which opens one read
stream per part in the window, and on object storage those per-part read
buffers dominate peak memory, so it scaled with the number of parts
scanned. Two changes bound it:
- The logs ClickHouse client caps the per-part read buffers via new
env-tunable settings. The object-storage-only setting is opt-in, so it
is never sent to a ClickHouse version that lacks it.
- Recent-first window narrowing: rows come back newest first, so the
presenter probes the most recent window and only widens toward the full
requested range when a page is short. A busy environment fills a page
from a few recent parts instead of scanning the whole range; a quiet one
still returns every row in a couple of cheap reads.
Correctness: the keyset cursor ordered on (triggered_timestamp,
trace_id), which is not unique because the spans of a trace share both,
so rows at a tie could be skipped or duplicated across pages. The cursor
and ORDER BY now include span_id, and the cursor is versioned so stale
cursors reset to the first page.
Guards: the effective page size is capped, and the existing per-query
memory limit lets a pathological wide browse fail with an error instead
of taking the node down.
## ClickHouse 26.2
The memory fix relies on lazy materialization deferring the wide
attributes column to the output rows, which only holds on 26.x. Cloud
already runs 26.2, so this moves the dev stack, testcontainers, and CI
to match. The ClickHouse test suite passes on 26.2.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
When a non-id predicate matches a row in both physical tables, findFirstAcrossTables now returns the v2 copy instead of legacy. Under this PR a run is in exactly one table (createRun routes by id format), so this is a no-op today; it forward-aligns with the later slow legacy to v2 migration, which copies a run into task_run_v2 (the canonical, operated-on copy) before operating. A comment in findRuns marks the matching dedup-by-id work for that migration PR.
A v2 run can reference a legacy parent/root, or have legacy children, when a hierarchy straddles a runTableV2 flip. Prisma relation selects are bound to one table, so the run, span, and API-retrieve presenters returned null parent/root and dropped cross-table children. They now resolve parent/root by id (RunStore routes by id format) and children by a both-table predicate, via a shared hydrateParentAndRoot/hydrateChildRuns helper.
A v2 run DELETE needs the full old row so its ClickHouse soft-delete tombstone carries organization and environment ids; under the default replica identity those are dropped and the tombstone is lost. A migration sets REPLICA IDENTITY FULL on task_run_v2 rather than relying on an out-of-band step, and the replication client now warns when any co-published table that publishes UPDATE/DELETE lacks FULL. Adds a replication test for the v2 DELETE tombstone.
The pre-gate idempotency claim was eligible only when the org was on the mollifier. Concurrent same-key triggers that straddle a runTableV2 flip can mint into different physical tables, whose per-table unique constraints can't see each other, so two runs could share one key. The claim is now also eligible when the org is cut over to the v2 run table, serialising those triggers through Redis.
A single-format id-list narrows findRuns to one physical table, but the ordered+limited path still built the cross-table comparator and threw the time-key guard; it now delegates natively to the one table (Postgres orders within a single table fine). Separately, the in-memory merge comparator ordered strings by code unit while the Postgres keyset continuation orders by the database collation (en_US); switching the comparator to localeCompare makes them agree, so a tied-createdAt boundary spanning both tables no longer skips or duplicates a row.
findRuns now throws when given skip: offset pagination cannot span the two run tables, where each would independently skip N rows from its own result rather than N from the merged result. For an id-list predicate (id in [...]), it now queries only the table whose id format can contain those ids, avoiding a wasted query against an empty task_run_v2 while it is unpopulated during rollout.
## Summary
Adds read methods to `RunStore` (`findRun`, `findRunOrThrow`,
`findRuns`) and routes every Postgres read of `TaskRun` through them,
mirroring how writes already go through the store. Behavior-preserving:
each relocated read keeps its exact query, field selection, and database
client (writer, replica, or transaction). This lets `TaskRun` reads be
retargeted to a different backing store later without touching call
sites.
Stacked on #3981 (the write adapter); that PR is the base of this one.
## Scope
In scope: the run engine, webapp services, presenters, and route
loaders. Three reads that pulled `TaskRun` in through a parent model's
relation `include` (alert delivery, batch results, attempt-dependency
cancellation) are decomposed to fetch the run(s) through the store and
stitch them back, since a relation include would not follow `TaskRun` to
a new table.
Left reading the existing table (out of scope): the legacy MarQS paths,
the legacy trigger idempotency read, and one raw-SQL recovery script
(commented for revisiting at cutover).
## Notes
Reads default to the read replica; callers pass the writer or a
transaction client wherever the original read did, so writer-vs-replica
behavior is unchanged.
Poll for the ClickHouse row with a bounded deadline instead of a fixed sleep, which is flaky under replication lag variance, and stop the replication service in a finally block so a failing assertion cannot leak it into later tests.
Follow-up to #3992, which gated the send runner-side - but only for new
runner images. Existing runners still POST a debug log per line.
When `SEND_RUN_DEBUG_LOGS` is off (default), the route now drops the
request immediately: `skipBodyParsing` skips the body read/parse, a bare
handler returns 204, no wide event. The route stays registered so it
avoids the `No route match` error log; the only per-request log left is
the framework's `logger.debug` trace, suppressed at the default `info`
level. Still counted by request metrics, and 204 is non-retryable so no
retry storm.
Adds a `skipBodyParsing` flag to the internal HTTP server.
Runners were POSTing a debug log to the supervisor for every log line -
one request per line, unbatched and unconditional. The supervisor
already has a `SEND_RUN_DEBUG_LOGS` toggle (off by default) that
discards them on receipt, but the runner fired the request regardless,
so the traffic hit the supervisor either way.
This gates the send at the source. The runner now reads
`TRIGGER_SEND_RUN_DEBUG_LOGS` (off by default, injected by the
supervisor from its existing `SEND_RUN_DEBUG_LOGS` setting) and skips
the POST entirely when disabled. Local log output is unchanged. Dev runs
use a separate path and are unaffected.
## Summary
Adds an environment-scoped HTTP API over the Errors feature, mirroring
the runs API. Task-run failures are grouped by a fingerprint into "error
groups," and this exposes everything you can do with them in the
dashboard:
- `GET /api/v1/errors` lists error groups, with
`filter[taskIdentifier]`, `filter[version]`, `filter[status]`
(`unresolved`/`resolved`/`ignored`), `filter[search]`, a time range, and
cursor pagination.
- `GET /api/v1/errors/{errorId}` retrieves a single group (summary,
lifecycle state, affected versions).
- `POST /api/v1/errors/{errorId}/{resolve,ignore,unresolve}` changes its
state.
- `GET /api/v1/runs?filter[error]={errorId}` lists the runs behind a
group.
Request and response schemas are exported from `@trigger.dev/core/v3` so
the SDK can reuse them, and all endpoints are documented in the API
reference (OpenAPI). `errorId` is the `error_<fingerprint>` friendly id.
## Attribution
State changes record who made them. A plain environment API key has no
user, so `resolvedBy`/`ignoredByUserId` stay null. When the caller uses
an environment JWT obtained by exchanging a personal access token or a
delegated user token at `POST /api/v1/projects/:ref/:env/jwt`, that
exchange now stamps an `act` delegation claim, and the write endpoints
read `act.sub` to attribute the change to the acting user. This is the
first endpoint to consume the `act` claim, so two small pieces of
plumbing ride along: the exchange stamps `act` for personal-access-token
subjects too (it was delegated-token-only), and the public-JWT
bearer-auth path surfaces `act.sub` to the handler.
Built on the delegated-token work in #3997.
## Summary
The CLAUDE.md audit job (`.github/workflows/claude-md-audit.yml`)
frequently hits its 15-turn cap before it finishes reviewing a PR, so
the job fails without posting a verdict. For example, the audit job
failed on [this
run](https://github.com/triggerdotdev/trigger.dev/actions/runs/27837408945/job/82390460772?pr=3990).
This raises `--max-turns` from 15 to 25 to give the review room to
complete, and pins `--model claude-opus-4-8` (the job previously
inherited the action default model).
runsReplicationService co-publishes task_run_v2 alongside TaskRun. It is a column-identical clone, so its WAL rows flow through the same transform into the same ClickHouse table, keeping the mirror complete once orgs cut over to v2 run ids. task_run_v2 needs REPLICA IDENTITY FULL, applied the same out-of-band way as TaskRun, so update and delete events carry the old row.
LogicalReplicationClient gains an optional additionalTables option. These are published alongside the primary table in the same publication, and their WAL events stream through the same data handler. When the publication already exists, missing tables are added via ALTER PUBLICATION ADD TABLE (online, slot-preserving) instead of erroring, so a publication can gain a table without a drop and recreate.
Adds a per-org runTableV2 feature flag, read in memory at the single run-id mint site in the trigger path. When on, the org mints a KSUID id for new runs (routing them to task_run_v2); off, the default, keeps minting legacy ids. The read is a pure lookup on the org featureFlags already loaded at auth, so the trigger path adds no query. RunStore routes purely by id format and never sees this flag.
findRun and findRunOrThrow route by the id/friendlyId in the predicate. A lookup that carries neither (the idempotency-key dedup, or an "are there any runs in this environment" check) previously defaulted to the legacy table and would miss a match that lives in task_run_v2. Such predicates now query both tables in parallel and return the first match, so a reused idempotency key is found wherever its run lives and no duplicate is created.
runsBackfiller paginates on a (createdAt, id) keyset instead of id alone. The ClickHouse runs list restores ClickHouse ranking in memory after hydrating rows by id, since a single SQL order cannot span the two tables.
findRuns now queries both TaskRun and task_run_v2 and merges the two ordered streams into one result. Ordered, limited reads require a time-based key (createdAt) because cuid and ksuid ids do not sort into a shared range, so id alone cannot order the union.
## Summary
Adds a short-lived, delegated token (`tr_uat_...`) that authenticates
against the API as a user without handing out a long-lived personal
access token. You mint one from a PAT, optionally narrow it to a set of
scopes, and give it a lifetime; the API then treats requests as that
user, subject to their role.
`trigger.dev mint-token` is the entry point (it uses your stored PAT):
```bash
UAT=$(trigger.dev mint-token --ttl 3600 --cap read:runs)
```
The token works anywhere a PAT does for user-level endpoints, and can be
exchanged for an environment JWT at `POST
/api/v1/projects/:ref/:env/jwt` to reach environment-scoped data (the
same exchange a PAT supports).
## How it works
A user-actor token is a short-lived JWT verified by a new first-class
`authenticateUserActor` method on the RBAC plugin. Self-hosters get a
built-in fallback; role-aware enforcement comes from the plugin.
Effective permissions are the intersection of the user's role and the
token's optional scope cap, so a token is only ever narrower than the
user, never broader.
Minting is restricted to personal access tokens (a token can't mint
another one, and an environment key can't mint one). Tokens default to a
1 hour lifetime (max 365 days). When exchanged for an environment JWT,
the user is stamped on it for attribution and the scope cap is carried
through.
Select the TaskRun or task_run_v2 table per operation from the run id's
format (KSUID routes to v2, anything else to legacy) via a runModel helper,
so a run is read and written in its own table. Batch and predicate-keyed
operations span both tables. Behavior-preserving for legacy runs.
Production drops the foreign keys on and referencing the run tables, but the
test harness builds via prisma db push, which recreates them from the schema
relations. Drop them after the push so test databases match production and a
run can live in either run table.
Give TaskRunV2 the same relation surface as TaskRun (belongs-to plus child
collections, with child relations sharing the existing scalar fields) so run
reads through the store can include relations regardless of table. No DB
foreign keys: stripped in production migrations and in the test harness.
Dan Sutton