- updateMetadata cross-table test: wrap the body in try/finally so stopFlushing
always runs and the flush loop cannot bleed into later tests on a failure path.
- cancelling cross-table cancel-cascade test: poll for the child CANCELED status
with a deadline instead of a fixed 1s sleep, to de-flake it under slow CI.
- recover-stuck-runs: constrain each UNION branch by id = ANY(runIds) so the
recovery query scans only candidate rows instead of unioning both full tables
before the join.
Reverts 525e36366 (graceful legacy-cursor handling). The legacy bare-id cursor
only matters for a backfill in flight across the single deploy that changed the
cursor format, and that backfill is internal and admin-triggered, so the
one-time transition is handled operationally: do not run a backfill during the
rollout deploy. Keeping decodeBackfillCursor throwing surfaces genuine cursor
corruption loudly instead of silently restarting the window. The old keyset was
id-ordered and the new one createdAt-ordered, so a stale cursor cannot be safely
resumed anyway, only restarted.
The backfill cursor format changed from a bare run id to a composite
<createdAt>_<id>, and decodeBackfillCursor threw on anything without the
separator. A backfill in flight across that change hands the new decoder an old
bare-id cursor, so it would throw on every batch. Treat an unparsable cursor (a
legacy bare id, or corrupt) as "no cursor" and restart the window: re-backfill
is idempotent (ClickHouse ReplacingMergeTree keyed by run id), so the in-flight
job self-recovers instead of failing. Logs a warning. Adds cursor round-trip and
legacy-format tests.
Code-review follow-ups on the v2 hardening:
- runTableV2Status no longer starts its background poller under vitest
(NODE_ENV=test). The module is imported by the mint/read sites, so the
import-time poll plus setInterval was firing live DB queries against the test
database and leaking a timer, and the async refresh could race tests that
drive the cached status directly. Tests exercise the gates by mutating the
cached state, so the poller only gets in the way.
- The publication-readiness probe now filters pg_publication_tables on
schemaname = "public", so a same-named table in another published schema
cannot satisfy the check.
#1 publication-readiness interlock: a v2 run minted before task_run_v2 is in the
ClickHouse replication publication is permanently absent from ClickHouse
(Postgres only decodes changes for transactions that begin after ALTER
PUBLICATION ADD TABLE, which the replication leader runs at its own startup, not
via a migration), and the run list/metrics/tags are ClickHouse-only. Add a
cached, periodically-refreshed status (runTableV2Status.server.ts) and gate
minting (triggerTask, triggerFailedTask) through canMintV2Run = org cut over AND
table published. Minting fails safe to legacy until the publication carries the
table and self-heals once it does, removing the manual pg_publication_tables
enable step.
#2 native-rollback read scope: cross-table read scoping (idempotency dedup,
ApiRetrieveRun hierarchy) keyed on the native master switch alone, so disabling
native realtime after v2 runs exist re-scoped reads to legacy and hid existing
v2 runs (an idempotency dedup miss means duplicate execution). Scope on
v2RunsMayExist (native on OR task_run_v2 has rows) instead; it is monotonic, so
the read scope cannot regress once v2 runs exist.
triggerFailedTask minted the pre-failed run with a cuid (RunId.generate), so it
landed in legacy TaskRun even for an org cut over to v2. Trigger-time failures
(queue limits, validation, payload errors) are common for some orgs, and these
runs frequently carry a parentTaskRunId / resumeParentOnCompletion / batch, so
each one created an ongoing cross-table edge (a v2 parent or batch with a legacy
failed child) on the failure path, not just the transient mixed window. Gate the
id mint on shouldUseV2RunTable like triggerTask: the main call() path uses the
request org flags; the degraded callWithoutTraceEvents() path loads them by org
id and falls back to a legacy id only if the org cannot be resolved.
Prisma emits foreign keys as REFERENCES "public"."TaskRun" (schema-qualified) in
every generated migration in this repo, but the guard only matched the bare
"TaskRun"/"task_run_v2", so it was effectively a no-op against real migrate-dev
output: it would not catch a regenerated _v2_fkey, nor the implicit m2m
_WaitpointRunConnections_A_fkey / _TaskRunToTaskRunTag_A_fkey (the former would
break trigger-and-wait for v2 runs). Match both bare and schema-qualified forms,
with fixtures pinning both so the qualified form cannot regress undetected.
Make task_run_v2 a faithful (non-degraded) clone of TaskRun and make the v2
cutover safe to actually enable:
- Add the INCLUDE (id) WITH (fillfactor=90) covering clause to task_run_v2's
(runtimeEnvironmentId, createdAt DESC) index so the dashboard run-list query
keeps index-only scans, matching TaskRun. Columns were already identical; v2
now has every index TaskRun has (plus a (createdAt, id) keyset index it needs
for the cross-table cursor merge).
- Reject enabling runTableV2 for an org unless realtimeBackend is "native"
(validateFeatureFlagInvariants), wired into both admin feature-flag write
routes. A v2 run minted while the org is still on Electric is realtime
invisible, so block the bad combination at write time, not just at read time.
- Add a guard test that fails CI if a generated migration re-adds an incoming
foreign key to TaskRun (after the decoupling drop) or adds one referencing
task_run_v2: the Prisma drift footgun that would re-couple the tables or break
cross-table run creation.
State that no v2 run exists "until native is enabled" (minting requires it)
rather than the absolute "while native is off", which is not true after a
deployment-wide native rollback. Comment-only.
A run's physical table is fixed by its id format, not an org's current
runTableV2 flag. An org that was on v2 then flipped the flag off still holds v2
runs (they stay readable, routed by id), so scoping the dedup and hierarchy
reads to "legacy" off the per-org flag would miss those v2 runs: it would
silently drop a v2 run's children and parent on retrieve, and let a duplicate
through idempotency dedup. Gate the scope on whether ANY v2 run can exist in
the deployment (the native realtime master switch) instead. While native is off
no v2 run exists anywhere, so "legacy" is safe and still skips the empty
task_run_v2 query; once native is on, every read covers both tables.
The two realtime v1-streams routes (read/create + append) resolved a
target:"parent"|"root" via a table-bound parentTaskRun/rootTaskRun relation
select, which returns null for a cross-table parent/root in the runTableV2 mixed
window and 404s a target that exists. Select the scalar
parentTaskRunId/rootTaskRunId and resolve the target by id through RunStore
(routes by id format), matching the presenter fixes. The "self" target is
unchanged.
Make the v2 path both correct and performant for turning the flag on:
- findRuns gains an optional tables: "legacy" | "both" scope (mirroring findRun),
threaded through hydrateChildRuns/hydrateParentAndRoot. While an org is not on
v2 its runs only live in TaskRun, so callers pass "legacy" to skip the empty
task_run_v2 query.
- ApiRetrieveRun resolves parent/root and children in parallel (one round-trip
instead of two) and scopes the reads to legacy for non-v2 orgs, so the public
run-retrieve no longer pays an extra both-table query on every call.
- The run-inspector side panel resolves parent/root by id across both tables
(was a table-bound relation select that returned null for a cross-table parent
in the mixed window).
- recover-stuck-runs joins TaskRunExecutionSnapshot against TaskRun UNION
task_run_v2 so a stuck v2 run is found and re-enqueued.
The idempotency-key dedup is a non-id predicate, so RunStore read BOTH run
tables in parallel on every idempotency-keyed trigger, including orgs not cut
over to v2 (whose runs only live in TaskRun, so the task_run_v2 query is always
empty; while native realtime is off that is every org). Add an optional
`tables: "legacy" | "both"` scope to findRun and pass "legacy" from the
idempotency concern when the org is not on v2, keeping the trigger hot path
single-table.
Backfills cross-table tests the audit flagged as missing: findRun legacy-scope
skips task_run_v2, and clearIdempotencyKey fans out across both tables
(byPredicate hits v2; a mixed byFriendlyIds array clears both).
The Electric dual-shape merge was a bridge to let the Electric backend observe
v2 runs during the cutover, but Electric is short-lived and the merge taxed
every tag/batch realtime feed with a second long-poll the moment it deployed.
Gate the v2 run table on the native realtime backend instead (the native client
is table-agnostic and observes v2 runs directly), so a run only routes to
task_run_v2 once its org is on native. Remove the merge module and restore the
single-table Electric proxy.
The cross-table correctness work stays: a v2 run can still have a cross-table
parent or child once an org flips, so the cancelRun cascade, metadata
parent/root routing, the one-time-token claim, and the findRuns guard all still
apply regardless of realtime backend.
- The strengthened findRuns guard threw on GET /api/v1/runs/:runId/spans/:spanId,
which pages child runs with take and no orderBy across both tables. Add a
createdAt order so it takes the bounded cross-table merge (and the 50-row cap
is now deterministic, most recent first) instead of throwing for every org.
- Key the one-time-use-token cross-table claim on the token alone (a reserved
task slot), matching the task-independent oneTimeUseToken unique constraint,
so a multi-task token cannot mint twice across the flip. Stop excluding
triggerAndWait from the token claim. Always resolve a held claim on the
success path (publish, else release) so it cannot leak until its TTL.
Routes that walk the run hierarchy through a Prisma relation only see one
physical table, so during a runTableV2 flag flip (a parent and child on
opposite tables) they silently miss the cross-table run. This closes the
reachable cases:
- cancelRun resolves child runs across both tables, so cancelling a parent
cascades to a child in the other table instead of leaving it executing
and holding concurrency.
- updateMetadata routes metadata.parent/root operations to the scalar
parent/root id, so they reach a parent in the other table instead of
falling back to the child run.
- a one-time-use token with no idempotency key now takes a cross-table
claim for v2 orgs, so two presentations straddling a flip cannot each
mint a run in a different table.
- the Electric shape merge reports up-to-date only when both tables are
caught up, so a multi-chunk initial snapshot no longer drops the rows
that arrive after the first chunk.
Restore the both-table Electric shape merge so tag-list and batch realtime
feeds observe runs in TaskRun and task_run_v2 together, and gate the v2 run
table on the runTableV2 flag alone (drop the native-realtime coupling). New
runs route to task_run_v2 whenever an org has the flag on and stay visible in
realtime on the existing Electric backend.
Single-run feeds route to one table by id format; only tag and batch feeds fan
out to both shapes under one composite continuation.
Completes the Electric-merge removal: a run only routes to task_run_v2 when the deployment has native realtime enabled and the org's realtimeBackend flag is native. Electric shapes are single-table and can't observe a v2 run, so without this gate a v2 run would be realtime-invisible. shouldUseV2RunTable takes the native-realtime master switch as a parameter (kept env-free for unit tests); the trigger mint site and the idempotency pre-gate claim both pass it.
Electric realtime shapes are bound to a single table, so a task_run_v2 run was invisible to realtime subscriptions. The previous approach merged two Electric shapes per tag/batch feed under a composite cursor, which doubled Electric long-poll connections for those feeds. Electric is being retired in favor of the native realtime backend, which is table-agnostic and already observes both run tables, so that merge is throwaway.
Drop the Electric dual-shape merge (revert realtimeClient to its single-table form, remove the merge module) and instead gate runTableV2 on the native backend: a run only routes to task_run_v2 when the deployment has native realtime enabled and the org's realtimeBackend flag is native. This keeps v2 runs realtime-observable without touching Electric, and the gate auto-satisfies once Electric is removed and native is the default. The idempotency pre-gate claim inherits the same gate.
The two-table shape merge could leave one upstream fetch pending without a rejection handler when it aborts the race loser or rethrows from the catch block. Attach a detached no-op catch to both fetches up front so an abandoned fetch can never surface as an unhandled rejection on any path. Also document that a tag/batch subscription opens two upstream Electric connections while an org spans both run tables.
When the two-table realtime shape merge returns as soon as one upstream shape yields, it aborts the other fetch and returns immediately. That promise was left without a rejection handler, so the abort could surface as an unhandled rejection on the server. Attach a no-op catch to the aborted fetch.
The parent/root/child hydration that resolves a run's hierarchy across both run tables looked runs up by id alone. Those pointers are now plain scalars with no foreign-key enforcement, so a stale or malformed pointer could resolve to a run in another environment and leak its metadata through the run and span presenters. Scope every lookup to the run's runtimeEnvironmentId, restoring the same-environment guarantee the table-bound relation select used to provide.
runTableV2 is resolved per organization only, so a global toggle on the admin flags page did nothing. Mark it read-only there to remove the misleading control; per-org control stays on the org dialog.
A run routed to task_run_v2 was invisible to the Electric realtime feed, whose shapes were bound to the TaskRun table, so subscribeToRun, useRealtimeRun, and run polling returned nothing for those runs. Single-run subscriptions now route the shape to the correct table by id format, and the tag and batch feeds run two upstream shapes (TaskRun and task_run_v2) merged under one composite cursor the client round-trips opaquely, so no SDK change is needed.
Concurrent same-key triggers that straddle a runTableV2 flag flip can mint into different physical tables (cuid to TaskRun, ksuid to task_run_v2), whose per-table unique constraints cannot see each other, so neither insert conflicts and two runs share one key. The pre-gate claim now resolves its backend through a claim-only Redis buffer when the mollifier buffer is absent, so it serialises these triggers instead of falling open. v2-cutover orgs are claim-eligible for every idempotency-keyed trigger, including triggerAndWait, debounce, and one-time-use tokens, and the claim-resolved path blocks the parent on the winner's waitpoint.
## Summary
The logs search page (behind a feature flag) ran ClickHouse out of
memory when browsing back over long time ranges. This keeps it within
bounded memory and fixes a pagination bug that could skip or duplicate
rows at a page boundary.
## Fix
Memory: the list query reads in sort-key order, which opens one read
stream per part in the window, and on object storage those per-part read
buffers dominate peak memory, so it scaled with the number of parts
scanned. Two changes bound it:
- The logs ClickHouse client caps the per-part read buffers via new
env-tunable settings. The object-storage-only setting is opt-in, so it
is never sent to a ClickHouse version that lacks it.
- Recent-first window narrowing: rows come back newest first, so the
presenter probes the most recent window and only widens toward the full
requested range when a page is short. A busy environment fills a page
from a few recent parts instead of scanning the whole range; a quiet one
still returns every row in a couple of cheap reads.
Correctness: the keyset cursor ordered on (triggered_timestamp,
trace_id), which is not unique because the spans of a trace share both,
so rows at a tie could be skipped or duplicated across pages. The cursor
and ORDER BY now include span_id, and the cursor is versioned so stale
cursors reset to the first page.
Guards: the effective page size is capped, and the existing per-query
memory limit lets a pathological wide browse fail with an error instead
of taking the node down.
## ClickHouse 26.2
The memory fix relies on lazy materialization deferring the wide
attributes column to the output rows, which only holds on 26.x. Cloud
already runs 26.2, so this moves the dev stack, testcontainers, and CI
to match. The ClickHouse test suite passes on 26.2.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
A v2 run can reference a legacy parent/root, or have legacy children, when a hierarchy straddles a runTableV2 flip. Prisma relation selects are bound to one table, so the run, span, and API-retrieve presenters returned null parent/root and dropped cross-table children. They now resolve parent/root by id (RunStore routes by id format) and children by a both-table predicate, via a shared hydrateParentAndRoot/hydrateChildRuns helper.
A v2 run DELETE needs the full old row so its ClickHouse soft-delete tombstone carries organization and environment ids; under the default replica identity those are dropped and the tombstone is lost. A migration sets REPLICA IDENTITY FULL on task_run_v2 rather than relying on an out-of-band step, and the replication client now warns when any co-published table that publishes UPDATE/DELETE lacks FULL. Adds a replication test for the v2 DELETE tombstone.
The pre-gate idempotency claim was eligible only when the org was on the mollifier. Concurrent same-key triggers that straddle a runTableV2 flip can mint into different physical tables, whose per-table unique constraints can't see each other, so two runs could share one key. The claim is now also eligible when the org is cut over to the v2 run table, serialising those triggers through Redis.
## Summary
Adds read methods to `RunStore` (`findRun`, `findRunOrThrow`,
`findRuns`) and routes every Postgres read of `TaskRun` through them,
mirroring how writes already go through the store. Behavior-preserving:
each relocated read keeps its exact query, field selection, and database
client (writer, replica, or transaction). This lets `TaskRun` reads be
retargeted to a different backing store later without touching call
sites.
Stacked on #3981 (the write adapter); that PR is the base of this one.
## Scope
In scope: the run engine, webapp services, presenters, and route
loaders. Three reads that pulled `TaskRun` in through a parent model's
relation `include` (alert delivery, batch results, attempt-dependency
cancellation) are decomposed to fetch the run(s) through the store and
stitch them back, since a relation include would not follow `TaskRun` to
a new table.
Left reading the existing table (out of scope): the legacy MarQS paths,
the legacy trigger idempotency read, and one raw-SQL recovery script
(commented for revisiting at cutover).
## Notes
Reads default to the read replica; callers pass the writer or a
transaction client wherever the original read did, so writer-vs-replica
behavior is unchanged.
Poll for the ClickHouse row with a bounded deadline instead of a fixed sleep, which is flaky under replication lag variance, and stop the replication service in a finally block so a failing assertion cannot leak it into later tests.
## Summary
Adds an environment-scoped HTTP API over the Errors feature, mirroring
the runs API. Task-run failures are grouped by a fingerprint into "error
groups," and this exposes everything you can do with them in the
dashboard:
- `GET /api/v1/errors` lists error groups, with
`filter[taskIdentifier]`, `filter[version]`, `filter[status]`
(`unresolved`/`resolved`/`ignored`), `filter[search]`, a time range, and
cursor pagination.
- `GET /api/v1/errors/{errorId}` retrieves a single group (summary,
lifecycle state, affected versions).
- `POST /api/v1/errors/{errorId}/{resolve,ignore,unresolve}` changes its
state.
- `GET /api/v1/runs?filter[error]={errorId}` lists the runs behind a
group.
Request and response schemas are exported from `@trigger.dev/core/v3` so
the SDK can reuse them, and all endpoints are documented in the API
reference (OpenAPI). `errorId` is the `error_<fingerprint>` friendly id.
## Attribution
State changes record who made them. A plain environment API key has no
user, so `resolvedBy`/`ignoredByUserId` stay null. When the caller uses
an environment JWT obtained by exchanging a personal access token or a
delegated user token at `POST /api/v1/projects/:ref/:env/jwt`, that
exchange now stamps an `act` delegation claim, and the write endpoints
read `act.sub` to attribute the change to the acting user. This is the
first endpoint to consume the `act` claim, so two small pieces of
plumbing ride along: the exchange stamps `act` for personal-access-token
subjects too (it was delegated-token-only), and the public-JWT
bearer-auth path surfaces `act.sub` to the handler.
Built on the delegated-token work in #3997.
runsReplicationService co-publishes task_run_v2 alongside TaskRun. It is a column-identical clone, so its WAL rows flow through the same transform into the same ClickHouse table, keeping the mirror complete once orgs cut over to v2 run ids. task_run_v2 needs REPLICA IDENTITY FULL, applied the same out-of-band way as TaskRun, so update and delete events carry the old row.
Adds a per-org runTableV2 feature flag, read in memory at the single run-id mint site in the trigger path. When on, the org mints a KSUID id for new runs (routing them to task_run_v2); off, the default, keeps minting legacy ids. The read is a pure lookup on the org featureFlags already loaded at auth, so the trigger path adds no query. RunStore routes purely by id format and never sees this flag.
runsBackfiller paginates on a (createdAt, id) keyset instead of id alone. The ClickHouse runs list restores ClickHouse ranking in memory after hydrating rows by id, since a single SQL order cannot span the two tables.
## Summary
Adds a short-lived, delegated token (`tr_uat_...`) that authenticates
against the API as a user without handing out a long-lived personal
access token. You mint one from a PAT, optionally narrow it to a set of
scopes, and give it a lifetime; the API then treats requests as that
user, subject to their role.
`trigger.dev mint-token` is the entry point (it uses your stored PAT):
```bash
UAT=$(trigger.dev mint-token --ttl 3600 --cap read:runs)
```
The token works anywhere a PAT does for user-level endpoints, and can be
exchanged for an environment JWT at `POST
/api/v1/projects/:ref/:env/jwt` to reach environment-scoped data (the
same exchange a PAT supports).
## How it works
A user-actor token is a short-lived JWT verified by a new first-class
`authenticateUserActor` method on the RBAC plugin. Self-hosters get a
built-in fallback; role-aware enforcement comes from the plugin.
Effective permissions are the intersection of the user's role and the
token's optional scope cap, so a token is only ever narrower than the
user, never broader.
Minting is restricted to personal access tokens (a token can't mint
another one, and an environment key can't mint one). Tokens default to a
1 hour lifetime (max 365 days). When exchanged for an environment JWT,
the user is stamped on it for attribution and the scope cap is carried
through.
## Summary
Replaces the multi-select popover task type filter on the Tasks page
with a single-select segmented control: **All** plus icon-only
**Agent**, **Standard**, and **Scheduled** segments. Each segment has a
tooltip showing its label and a number-key shortcut (0-3), and the
search field no longer autofocuses so the shortcuts work on page load.
## ✅ Checklist
- [x] I have followed every step in the [contributing
guide](https://github.com/triggerdotdev/trigger.dev/blob/main/CONTRIBUTING.md)
- [x] The PR title follows the convention.
- [x] I ran and tested the code works
## Summary
The Personal Access Tokens page now shows each token's maximum role in a
new column, so you can see at a glance what a token is capped to. The
column only appears when an RBAC plugin is installed, and shows "-" for
tokens with no cap. Its header tooltip reuses the same explanation shown
in the create-token panel.
Importing the service pulls the cancel chain, which eagerly initializes the
concurrency tracker singleton and requires REDIS_HOST/REDIS_PORT at import
time, so the suite cannot load in the unit-test shards without stacking
mocks. The decompose it covered is exercised by the analogous batch-results
container test and confirmed by review, so drop this one rather than mock
the tracker and cancel chain.
The new container tests import the service and presenter, which pull the
db.server singleton in through their base classes. Mock it so the tests do
not try to connect to the env database when none is reachable (the CI unit
shards), matching the existing webapp container-test pattern. The tests use
the injected testcontainer prisma for all reads.
Decompose the three reads that pulled TaskRun in through a parent model's
relation include (alert, batch results, attempt dependencies): query the
parent without the include, hydrate the run(s) via RunStore in a single
batched read, and stitch them back. Preserves field selection, ordering,
null handling and the query client. Adds container-backed tests for the
batch-results and cancel-dependencies paths.
Relocate the route and loader TaskRun reads to the RunStore read methods,
preserving the exact client per site, including the replica-resolve then
writer-recheck realtime paths. Behavior-preserving.
Dan Sutton