## Summary
Adds Billing Limits to the webapp.
Customers can set a monthly spend cap. When usage crosses the limit,
billable environments enter a grace period. If the limit is not resolved
before grace expires, new triggers are rejected until the organization
increases or removes the limit.
## Summary
The in-dashboard agent's datastore now runs migrations over a direct
(non-pooler) connection. A transaction-mode pooler can't run the
migrator (no advisory locks, no multi-statement DDL), so when the
agent's database sits behind a pooler the migration step needs a
separate direct connection.
The application keeps connecting over the pooled
`DASHBOARD_AGENT_DATABASE_URL`. Only the migration entry points changed
(`drizzle.config.ts`, `migrate.mjs`, `migrate-status.mjs`); the runtime
client is untouched.
## Connection resolution (migrations)
```
DASHBOARD_AGENT_DIRECT_URL direct agent connection (used for migrations)
DASHBOARD_AGENT_DATABASE_URL pooled agent connection (preserves current behavior)
DIRECT_URL main direct connection (single-database fallback)
DATABASE_URL last resort
```
Mirrors the existing `DATABASE_URL` / `DIRECT_URL` split. Fully
backward-compatible: with nothing new set, resolution is identical to
before. The agent-specific vars take precedence over the main
`DIRECT_URL`, so a separate agent database is never migrated against the
wrong one. When the agent falls back to the main single database,
migrations now prefer its direct connection.
Closes this feature request:
[https://triggerdev.featurebase.app/p/isolated-dev-sessions-for-multiple-local-trigger-dev-instances](https://triggerdev.featurebase.app/p/isolated-dev-sessions-for-multiple-local-trigger-dev-instances)
### Feature notes:
- CLI `trigger dev` works as before
- `trigger dev --branch my-branch` to create a new branch and run
against it.
- `trigger dev archive --branch my-branch` to archive (or in webapp).
- New webapp page to manage and archive dev branches, currently feature
flagged.
### Implementation details:
- No changes to data model, no backfill. `isBranchableEnvironment`
column is ignored for dev branches, we use `parentEnvironmentId IS NULL`
instead.
- `x-trigger-branch` overloaded for preview and dev branches
- New `TRIGGER_DEV_BRANCH` env var available locally.
`TRIGGER_PREVIEW_BRANCH` overloaded for child runs.
- Lots of new glue code to sanitise the branch checks.
### Rollout
- Deploy webapp/API changes (all backwards compatible)
- Manual tests on some orgs
- Deploy docs, release CLI, flip feature flag for webapp feature
### NB
- `api.v1.projects.$projectRef.environments.ts` will return
`isBranchableEnvironment: true` for all dev environments.
### Prerequisites
- [x] Typecheck will not pass until we make a new release of
`@trigger.dev/platform` and bump it here
## Summary
The dashboard agent's database migrations could be silently skipped when
its database is shared with another Drizzle application, leaving the
`trigger_dashboard_agent` schema uncreated and a later migration failing
with `schema "trigger_dashboard_agent" does not exist`.
## Root cause
Drizzle's migrator decides what to run by reading the most recent row
from its journal table by `created_at`, and skipping any migration dated
at or before it. The dashboard-agent runner used Drizzle's default
journal table, `drizzle.__drizzle_migrations`, which every Drizzle app
shares by default. When the database is shared, another app's journal
row dated between two of our migrations makes the migrator treat the
earlier one (the `CREATE SCHEMA`) as already applied and run a later one
against a schema that was never created.
## Fix
- Track the dashboard-agent migrations in a dedicated journal table
(`drizzle.__dashboard_agent_migrations`), in both the deploy runner
(`migrate.mjs`) and the drizzle-kit config, so its history is
independent of any other Drizzle app sharing the database. The table
stays in the `drizzle` schema so the first migration's `CREATE SCHEMA
"trigger_dashboard_agent"` does not collide with it.
- Make the first two migrations idempotent (`CREATE SCHEMA/TABLE/INDEX
IF NOT EXISTS`) so databases that already tracked them under the old
journal table re-run cleanly after the rename instead of erroring on the
bare `CREATE SCHEMA`.
Verified against a Postgres seeded to reproduce the skip: the old
default-table path fails as above, the dedicated-table path creates the
schema and all tables, and re-running on an already-migrated database is
a clean no-op.
## Summary
The `runqueue.workerQueue.length` gauge only reported a worker queue's
depth while runs were being dequeued from it. When dequeues stop, the
metric goes stale or missing, so a queue that has backed up because
nothing is draining it can't be alerted on. This adds a small observer
that refreshes the observed set of worker queues from the
`WorkerInstanceGroup` records on an interval, so every active worker
queue (and its scheduled split variant) keeps reporting its length
regardless of dequeue activity.
The observer is off by default and enabled per service via
`RUN_ENGINE_WORKER_QUEUE_OBSERVER_ENABLED`, reads from the read replica,
and skips a configurable set of cloud providers
(`RUN_ENGINE_WORKER_QUEUE_OBSERVER_EXCLUDED_CLOUD_PROVIDERS`, default
`digitalocean`). When enabled it is the source of truth for the observed
set, so the per-dequeue registration is skipped on that instance, and it
groups by worker queue so the per-instance duplicates collapse to the
true depth.
Also removes the unused `GET`/`POST /api/v1/workers` endpoints. Their
only consumer was a CLI command group that is no longer registered.
## Verification
Verified end to end against a local stack: the gauge reports each worker
queue's length with no dequeues happening, excludes the configured
providers, includes hidden groups, and the removed endpoints return as
if they never existed. Added a run-engine test
(`workerQueueObservation.test.ts`).
## Summary
Adds an in-dashboard AI agent: a chat panel, reachable from any
environment
page, that answers questions about your runs, errors, tasks, and
analytics,
diagnoses why a run failed, charts your data, reads your connected
repo's
source, and answers product and how-to questions. It is gated behind the
`hasDashboardAgentAccess` feature flag (global or per-org, default off),
so
this PR ships disabled: the launcher is hidden unless the flag is
enabled.
## Design
The agent runs as a standalone `chat.agent` Trigger task in its own
internal
package, with no access to the webapp database, Prisma, or ClickHouse.
It reads
the user's data over the public API, acting as the user via a
short-lived
delegated user-actor token minted server-side each turn (never in the
browser),
building on
[#3997](https://github.com/triggerdotdev/trigger.dev/pull/3997). The
error and analytics tools use
[#4005](https://github.com/triggerdotdev/trigger.dev/pull/4005)
and the TRQL query API.
The first turn of a new chat streams from a warm webapp route (Head
Start) while
the durable agent boots in parallel. Structured answers (a run-failure
diagnosis
card, a live chart) render through a small typed view catalog rather
than
arbitrary markup. A knowledge lane forwards product and how-to questions
to the
support assistant.
Conversation history lives in a separate Drizzle-backed store on its own
Postgres schema, kept as a display read-model so it can never corrupt
the
agent's model context.
The SDK changes add an `apiClient` option to
`chat.createStartSessionAction` and
`chat.headStart`, and keep the Head Start tool-approval tail intact
across a
custom `prepareMessages` hook so prompt caching and Head Start compose.
## Summary
Fixes the fair-queue weighted environment shuffle, which biased
environment ordering whenever fair-queue biases are enabled (the default
configuration).
## Root cause
`#weightedShuffle` in `fairQueueSelectionStrategy.ts` computed the total
weight once and drew its random pivot against that full-set total on
every iteration, but never decremented the total as items were removed
from the working set. After the first pick, the pivot frequently
overshot the sum of the remaining items, so the inner selection loop ran
off the end and clamped to the last remaining element. The result
systematically over-selected whichever environment sat at the tail of
the set.
The first slot stayed fair (the full total is correct on the first
draw), but later positions were ordered by environment iteration order
rather than by the intended concurrency-limit and available-capacity
weighting. For four equal-weight environments, the final position landed
on one env ~9% of the time and another ~42%, instead of ~25% each.
The two sibling selection paths (`#weightedRandomQueueOrder` and
`#selectTopEnvs`) already decrement the total before splicing; this
brings the env shuffle in line with them.
## Fix
```ts
result.push(items[index].envId);
totalWeight -= items[index].weight;
items.splice(index, 1);
```
Adds a regression test that runs the weighted shuffle over equal-weight
envs with biases enabled and asserts each env lands in every position
roughly uniformly. It fails on the old code (tail position ~37%) and
passes with the fix.
Reported in #4001.
## Summary
Sessions started from the agent Test playground were tagged with a
`"playground"` tag that rendered in the Sessions table's Tags column.
They are now flagged with a real `Session.isTest` boolean (mirroring
`TaskRun.isTest`) and surfaced as a dedicated **Test** column with a
check icon, to the left of Tags, on both the Sessions page and the Agent
landing page, plus a matching **Test** property on the session detail
page. This mirrors how Standard and Scheduled task runs already indicate
test runs.
## Design
`isTest` is a new `Session` column (Postgres) replicated into ClickHouse
`sessions_v1` alongside the existing fields. The Sessions list reads
`isTest` from Postgres for display (ClickHouse only supplies the ordered
session IDs), so the column renders correctly without a ClickHouse
backfill.
The playground action now sets `isTest: true` on session create instead
of writing the `"playground"` tag. The triggered run still carries
`playground:true` in its own tags (unchanged). A migration backfills
existing sessions, setting `isTest = true` and stripping the
now-redundant `"playground"` tag where it is present, so the list and
detail views render consistently without read-time tag filtering.
## Summary
The logs search page (behind a feature flag) ran ClickHouse out of
memory when browsing back over long time ranges. This keeps it within
bounded memory and fixes a pagination bug that could skip or duplicate
rows at a page boundary.
## Fix
Memory: the list query reads in sort-key order, which opens one read
stream per part in the window, and on object storage those per-part read
buffers dominate peak memory, so it scaled with the number of parts
scanned. Two changes bound it:
- The logs ClickHouse client caps the per-part read buffers via new
env-tunable settings. The object-storage-only setting is opt-in, so it
is never sent to a ClickHouse version that lacks it.
- Recent-first window narrowing: rows come back newest first, so the
presenter probes the most recent window and only widens toward the full
requested range when a page is short. A busy environment fills a page
from a few recent parts instead of scanning the whole range; a quiet one
still returns every row in a couple of cheap reads.
Correctness: the keyset cursor ordered on (triggered_timestamp,
trace_id), which is not unique because the spans of a trace share both,
so rows at a tie could be skipped or duplicated across pages. The cursor
and ORDER BY now include span_id, and the cursor is versioned so stale
cursors reset to the first page.
Guards: the effective page size is capped, and the existing per-query
memory limit lets a pathological wide browse fail with an error instead
of taking the node down.
## ClickHouse 26.2
The memory fix relies on lazy materialization deferring the wide
attributes column to the output rows, which only holds on 26.x. Cloud
already runs 26.2, so this moves the dev stack, testcontainers, and CI
to match. The ClickHouse test suite passes on 26.2.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
## Summary
Adds read methods to `RunStore` (`findRun`, `findRunOrThrow`,
`findRuns`) and routes every Postgres read of `TaskRun` through them,
mirroring how writes already go through the store. Behavior-preserving:
each relocated read keeps its exact query, field selection, and database
client (writer, replica, or transaction). This lets `TaskRun` reads be
retargeted to a different backing store later without touching call
sites.
Stacked on #3981 (the write adapter); that PR is the base of this one.
## Scope
In scope: the run engine, webapp services, presenters, and route
loaders. Three reads that pulled `TaskRun` in through a parent model's
relation `include` (alert delivery, batch results, attempt-dependency
cancellation) are decomposed to fetch the run(s) through the store and
stitch them back, since a relation include would not follow `TaskRun` to
a new table.
Left reading the existing table (out of scope): the legacy MarQS paths,
the legacy trigger idempotency read, and one raw-SQL recovery script
(commented for revisiting at cutover).
## Notes
Reads default to the read replica; callers pass the writer or a
transaction client wherever the original read did, so writer-vs-replica
behavior is unchanged.
## Summary
Adds an environment-scoped HTTP API over the Errors feature, mirroring
the runs API. Task-run failures are grouped by a fingerprint into "error
groups," and this exposes everything you can do with them in the
dashboard:
- `GET /api/v1/errors` lists error groups, with
`filter[taskIdentifier]`, `filter[version]`, `filter[status]`
(`unresolved`/`resolved`/`ignored`), `filter[search]`, a time range, and
cursor pagination.
- `GET /api/v1/errors/{errorId}` retrieves a single group (summary,
lifecycle state, affected versions).
- `POST /api/v1/errors/{errorId}/{resolve,ignore,unresolve}` changes its
state.
- `GET /api/v1/runs?filter[error]={errorId}` lists the runs behind a
group.
Request and response schemas are exported from `@trigger.dev/core/v3` so
the SDK can reuse them, and all endpoints are documented in the API
reference (OpenAPI). `errorId` is the `error_<fingerprint>` friendly id.
## Attribution
State changes record who made them. A plain environment API key has no
user, so `resolvedBy`/`ignoredByUserId` stay null. When the caller uses
an environment JWT obtained by exchanging a personal access token or a
delegated user token at `POST /api/v1/projects/:ref/:env/jwt`, that
exchange now stamps an `act` delegation claim, and the write endpoints
read `act.sub` to attribute the change to the acting user. This is the
first endpoint to consume the `act` claim, so two small pieces of
plumbing ride along: the exchange stamps `act` for personal-access-token
subjects too (it was delegated-token-only), and the public-JWT
bearer-auth path surfaces `act.sub` to the handler.
Built on the delegated-token work in #3997.
## Summary
Adds a short-lived, delegated token (`tr_uat_...`) that authenticates
against the API as a user without handing out a long-lived personal
access token. You mint one from a PAT, optionally narrow it to a set of
scopes, and give it a lifetime; the API then treats requests as that
user, subject to their role.
`trigger.dev mint-token` is the entry point (it uses your stored PAT):
```bash
UAT=$(trigger.dev mint-token --ttl 3600 --cap read:runs)
```
The token works anywhere a PAT does for user-level endpoints, and can be
exchanged for an environment JWT at `POST
/api/v1/projects/:ref/:env/jwt` to reach environment-scoped data (the
same exchange a PAT supports).
## How it works
A user-actor token is a short-lived JWT verified by a new first-class
`authenticateUserActor` method on the RBAC plugin. Self-hosters get a
built-in fallback; role-aware enforcement comes from the plugin.
Effective permissions are the intersection of the user's role and the
token's optional scope cap, so a token is only ever narrower than the
user, never broader.
Minting is restricted to personal access tokens (a token can't mint
another one, and an environment key can't mint one). Tokens default to a
1 hour lifetime (max 365 days). When exchanged for an environment JWT,
the user is stamped on it for attribution and the scope cap is carried
through.
Replicates `TaskRun.planType` into the `task_runs_v2` ClickHouse table
so run analytics can group by plan type.
Adds a `plan_type` column (goose migration `033`,
`LowCardinality(String)`), the replication insert mapping, and the
matching schema/column/type entries - same shape as the recent `region`
addition. Write-once at trigger, so it just rides along on existing
replicated rows. Internal analytics only; not exposed in the Query API.
Adds an opt-in mechanism to route a configurable percentage of
organizations onto the compute (MicroVM) backing of their region at
trigger time, without changing their stored region settings.
Routing is gated by three global feature flags -
`computeMigrationEnabled`, `computeMigrationFreePercentage`,
`computeMigrationPaidPercentage` - plus a per-org
`computeMigrationEnabled` override that wins in both directions. A
region's compute backing is resolved from a new
`WorkerInstanceGroup.region` column: a container group and its MicroVM
group share one geo `region`, so the migration swaps the resolved worker
queue to the backing group's queue. Orgs are bucketed deterministically
by id, so ramping a percentage down keeps a strict subset rather than
reshuffling, and a region with no compute backing is never touched.
Everything is off by default - behaviour is unchanged unless the flags
are set.
The flags and the worker-region groups are read on the trigger hot path
from in-memory snapshots rather than the database: a small
`createReloadingRegistry` helper loads each at startup and refreshes
them on an interval, so no per-trigger query is added and a percentage
or kill-switch change propagates within the reload interval. A cold
replica whose snapshot hasn't loaded yet reads as not-migrated (the
container path) and self-corrects on the next load - the same cold-start
contract as the datastore / LLM-pricing registries, with a
`reloading_registry_loaded` metric so a never-loaded registry is
alertable.
The same migration decision is consulted at deploy-time template
creation so a migrated org gets a compute template built ahead of its
first run. This runs in shadow mode (best-effort, never fails the
deploy) by default, or - when the `computeMigrationRequireTemplate` flag
is on - in required mode, built synchronously at deploy so the first run
never builds on-demand and template errors surface at deploy time.
So operators keep "which runs ran where" while customers only see
geography: the run's actual worker queue is stored raw, and the geo
region is stamped separately on `TaskRun.region` (and a new ClickHouse
`region` column) at trigger time. Read surfaces - the dashboard, the
API, and the Query/Logs page - show the geo region, falling back to the
worker queue for runs written before the column existed.
Minor follow-ups left out of scope: the percentage flags render as text
inputs on the admin flags page (the catalog UI has no numeric control
type yet), and `createReloadingRegistry` could later gain pub/sub for
sub-second cross-replica propagation if the reload interval proves too
slow.
## Summary
The Models page is now split into two tabs. **Your models** shows the
models your project has actually used in the selected time range, with
usage charts (cost over time, tokens over time, calls by model), a
per-model table of calls / cost / avg TTFC / avg tokens-per-sec, and
calls/tokens trend sparklines. **Model library** is the full catalog,
reordered from alphabetical to a relevance-based provider order
(Anthropic, OpenAI, Google, then the rest), newest models first within
each provider, with a "New" badge on models released in the last 7 days.
One time-range selector drives the whole Your models tab, so the charts,
the table, and the sparklines all share the same window. Opening a model
shows its own metrics with an independent range picker and a "View in AI
metrics" link that opens the AI metrics dashboard filtered to that
model. The active tab is kept in the URL so it survives a refresh and is
shareable.
## Prompt caching & cost accuracy
Both the Your models tab and the AI metrics dashboard now surface
prompt-cache usage: a cache-savings column plus per-model cached-tokens
and cache-hit-rate views, and a caching section on the dashboard (hit
rate, cached tokens, estimated savings, and hit rate by model).
Building this surfaced a cost bug. `input_tokens` is the total prompt
count and already includes cache-read and cache-creation tokens, but the
cost pipeline charged the full input at the input price and then added a
separate cache line, so cached tokens were billed twice (and on
Anthropic, cache reads were never discounted because their price is
keyed differently). The input price now applies only to the non-cached
remainder, with cache prices resolved across the provider-specific keys,
so LLM cost and the cache hit-rate metric are accurate. Hit rate is
computed as cached reads over total input.
## Notes
Also fixes React "invalid DOM property" console warnings from the
provider icons (the Llama and DeepSeek SVGs used raw `fill-rule` /
`clip-rule` / `clip-path` attributes), which this page surfaces by
rendering more provider icons.
## Screenshots
**Your models tab:** usage charts and a per-model table with
calls/tokens trend sparklines.
<img width="2560" height="1267" alt="1-your-models-tab"
src="https://github.com/user-attachments/assets/859bd24f-9047-4828-8bbb-83e5882846d6"
/>
**Model library:** provider-relevance ordering with a "New" badge on
models released in the last 7 days.
<img width="2560" height="1267" alt="2-model-library-tab"
src="https://github.com/user-attachments/assets/46dd54b9-80f9-4922-ade9-5935b08dfebc"
/>
**Model detail, Metrics tab:** per-model range picker and a "View in AI
metrics" link.
<img width="2560" height="1267" alt="3-model-detail-metrics"
src="https://github.com/user-attachments/assets/0f65d9d0-6142-4918-93f0-110bb277101a"
/>
**View in AI metrics:** the dashboard deep-linked and filtered to the
selected model.
<img width="2560" height="1267" alt="4-ai-metrics-filtered"
src="https://github.com/user-attachments/assets/821f256c-e305-493c-98c7-eafaf2f57f83"
/>
## What
`buildJwtAbility` — the decoder for public-token scope strings
(`read:tags:…`, `read:runs:run_abc`, `admin`, …) — now lives in
`@trigger.dev/plugins` as the single source of truth.
`@trigger.dev/rbac` re-exports it, so the built-in fallback and any auth
plugin interpret a token identically.
Scope strings are split on only the first **two** colons
(`action:type:id`), so a resource id that itself contains colons — e.g.
a tag like `user:123` — is matched in full rather than truncated to its
first segment. (The fallback already did this; this makes it the one
shared implementation.)
`@trigger.dev/plugins` is now **private (unpublished)** and gains a
`@triggerdotdev/source` export condition, so consumers bundle it from
source per-commit like `@trigger.dev/core` instead of resolving a
published version — no cross-version coordination.
## Why
Two hand-maintained copies of the scope grammar drift, and the
difference silently changes what a token grants. One shared decoder
removes that class of bug.
## Notes
- No changeset: `@trigger.dev/plugins` is now private and
`@trigger.dev/rbac` is internal — neither is published.
- Unit coverage for the colon-id path lives in
`internal-packages/rbac/src/ability.test.ts` (now exercising the shared
function).
Fixes an intermittent `Test timed out in 30000ms` in the
`streamBatchItems` suite. Not a logic hang — the 30s budget covers
container setup, and each case boots its own per-test Redis container +
a full `RunEngine`, so under CI Docker contention a cold boot can cross
30s (which is why the failure moved between tests).
- New `containerTestWithIsolatedRedisNoClickhouse` fixture (Postgres
clone + per-test Redis, no ClickHouse) — this suite never uses
ClickHouse, but the old fixture's auto `resetClickhouse` forced a
ClickHouse boot + migration onto the cold-start test.
- Raised `testTimeout` 30s → 120s, matching the run-engine convention
for this footprint.
## Summary
Adds a second backend for the realtime runs feed (`useRealtimeRun`,
`subscribeToRunsWithTag`, `subscribeToBatch`), built to stay healthy
when a single busy environment has many subscribers watching many runs
at once. It is gated behind a feature flag with the existing backend as
the default, so nothing changes for users until it is enabled per
environment.
## Design
A run change is published once, as a small self-describing record, to a
single per-environment channel. Every feed is then a predicate over that
one stream rather than owning a channel:
- A per-instance router indexes the currently-held feeds by run, tag,
and batch. When a run changes it hydrates the affected rows once and
serializes them once, then fans the result to every matching feed. One
hot shared tag watched by many subscribers costs a single database query
and serialize, not one per subscriber.
- Feeds that don't match a change are never woken, wake delivery per
environment is coalesced on a leading edge (250ms default) so a burst of
changes costs one wake, and cold reads coalesce onto a single
short-TTL-cached resolve.
- An admission gate bounds how many cold ClickHouse resolves run
concurrently, so a mass reconnect across many distinct filters queues
instead of stampeding the database.
- Changes that land while a client is between long-polls are delivered
on its next poll instead of waiting for the periodic backstop: each
environment buffers its recent change records, subscriptions linger
briefly after the last feed closes, and a newly-armed poll replays
exactly the connection's gap.
- The per-connection replay cursors behind that are shared across
instances via Redis (a single timestamp each), so a poll landing on a
different instance behind the load balancer still reads the connection's
true gap instead of falling back to a cold resolve. Cursor reads have a
bounded deadline and degrade to the cold-read path on any Redis trouble.
- Tag subscriptions with multiple tags match runs carrying all of the
tags, mirroring the existing backend's filter semantics, and live
long-polls hold for about 20 seconds to match its cadence.
- The per-environment channel supports Redis Cluster sharded pub/sub, so
the wake path scales horizontally across shards by environment.
- The backend reports its health through OpenTelemetry metrics (delivery
lag, poll resolution paths, backstop outcomes, replay and cursor-store
activity), with a provisioned Grafana dashboard for local development.
Everything is behind the feature flag and tunable via env vars; the
existing backend remains the default.
## Summary
When `RUN_ENGINE_READ_REPLICA_SNAPSHOTS_SINCE_ENABLED` is on,
`RunEngine.getSnapshotsSince` reads from the read replica. During write
spikes the replica can briefly lag, so the snapshot id a runner just
learned from the writer isn't visible there yet: the lookup threw, the
worker route returned a 500, and the runner waited for its next poll —
turning sub-second snapshot notifications into poll-interval latency
exactly when things are busiest. This PR makes the flag safe to enable:
a replica miss of the since snapshot gets one jittered retry on the
replica (most lag windows are shorter than the ~50–200ms wait, so the
writer is never touched), then falls back to the primary, observed via a
new `run_engine.snapshots_since.replica_miss` counter with an `outcome`
attribute (`replica_retry` vs `primary`). Only genuine misses — absent
on the primary too — remain errors.
## Design
- `getExecutionSnapshotsSince` now throws a typed
`ExecutionSnapshotNotFoundError` so the engine can distinguish the
expected lag miss from real failures. The message string is unchanged
and the error never leaves the engine.
- The recovery path only engages when the flag is on, a distinct replica
client is configured, and no transaction client was passed. With the
flag off, the path is behaviorally identical to before.
- Retry delay bounds are configurable
(`RUN_ENGINE_SNAPSHOTS_SINCE_REPLICA_RETRY_MIN_MS`/`MAX_MS`, default
50/200; `MAX_MS=0` skips the replica retry and goes straight to the
primary).
- The warn log fires only when the primary serves the read (the writer
spill is the operationally interesting event); replica-retry recoveries
are counted but quiet. A permanently-missing snapshot id stays an
error-level failure with a `failedDuring` field, so lag metrics aren't
polluted by bogus ids.
- Stale-tail lag (replica has the since snapshot but not newer rows)
deliberately still returns the replica's view; the next poll catches up.
- The since-snapshot anchor lookup is now scoped to the polled run
(`where: { id, runId }`), so a snapshot id from a different run raises
not-found instead of silently anchoring a too-wide window of the run's
snapshots.
## Test plan
All vitest + testcontainers, no mocks. A new `schemaOnlyPrisma` fixture
(migrated-but-empty clone database) simulates a replica that hasn't
caught up, and a real in-memory OTel meter pins the counter semantics
per outcome.
- [x] Replica catches up during the jittered retry window → served by
the replica, `outcome=replica_retry` = 1, primary never consulted
- [x] Replica permanently missing the since snapshot → served by the
primary, `outcome=primary` = 1
- [x] Snapshot missing on both replica and primary → null, counter = 0
- [x] Replica has the since snapshot but lags by one → the replica's
view is served, no fallback (verified discriminating power: the test
fails if reads secretly hit the primary)
- [x] Flag off with a replica configured → primary serves the read
- [x] Transaction client provided → bypasses the replica entirely
- [x] Since snapshot belonging to a different run → null
- [x] Existing getSnapshotsSince + waitpoints suites green; run-engine,
testcontainers, and webapp typechecks pass
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
## Summary
- `internal-packages/rbac/src/index.ts` — in `LazyController.load()`'s
catch block, throw an Error when `process.env.REQUIRE_PLUGINS === "1"`
instead of silently falling back. The throw is captured into the lazy
controller's init promise, so it surfaces on the first method call.
- `apps/webapp/app/routes/healthcheck.tsx` — `await
rbac.isUsingPlugin()` after the DB ping. With `REQUIRE_PLUGINS=1` and a
failed plugin load, the throw surfaces here and the healthcheck returns
500 → readiness probe fails → rollout is rolled back. Noop for
self-hosters.
- `.server-changes/require-plugins-fail-fast.md` — server-changes entry.
- `internal-packages/rbac/src/require-plugins.test.ts` — 4 unit tests
covering loader branching: unset → fallback, `=1` → throw,
`forceFallback: true` wins, only exactly `"1"` enforces.
- `internal-packages/testcontainers/src/webapp.ts` — adds
`requirePlugins?: boolean` to `StartWebappOptions`. Implies
`forceRbacFallback: false`.
- `apps/webapp/test/healthcheck-require-plugins.e2e.test.ts` — e2e
closes the loop: spawns a real webapp, hits `/healthcheck` via HTTP,
asserts 500 with `REQUIRE_PLUGINS=1` and 200 without.
## Motivation
Today the RBAC plugin loader catches any plugin-load failure (missing
module, broken transitive dep, init throw) and silently returns the
default fallback implementation. This is the correct behaviour for
self-hosters who don't ship the plugin — but it's dangerous in
deployments where the plugin is expected to load: an
accidentally-missing or broken plugin would silently disable
enforcement.
`REQUIRE_PLUGINS=1` makes the loader fail loudly in those deployments.
The variable name is intentionally plural and generic — future plugin
contracts (audit logs, SSO) can read the same flag without renaming.
Closes
[TRI-9852](https://linear.app/triggerdotdev/issue/TRI-9852/require-plugins1-fail-fast-for-required-plugin-loads).
## Test plan
- [x] `pnpm run test --filter @trigger.dev/rbac` — 38/38 tests pass,
including the 4 new loader tests
- [x] `pnpm run typecheck --filter webapp` — passes
- [x] `pnpm run typecheck --filter @trigger.dev/rbac --filter
@internal/testcontainers` — passes
- [x] e2e test added (`healthcheck-require-plugins.e2e.test.ts`) — CI
runs it via `e2e-webapp.yml`. Couldn't run locally (no Docker daemon
up); CI has Docker provisioned.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Add an optional network_labels field to the internal compute client's
create and restore request schemas and forward per-VM endpoint labels on
both paths, so a restored VM keeps the same labels as a freshly-booted
one. Mirrors the label the Kubernetes workload manager already sets on
the run pod.
---------
Co-authored-by: nicktrn <55853254+nicktrn@users.noreply.github.com>
## Problem
`ClickHouseRunsRepository.listRunIds` / `listRuns` order results by the
composite key `(created_at, run_id)`, but the cursor predicate cut on
`run_id` **alone**:
```ts
.where("run_id < {runId: String}", { runId: cursor })
.orderBy("created_at DESC, run_id DESC")
```
This is only sound when `run_id` lexicographic order matches
`created_at` order. `run_id`s are cuids — only coarsely time-sortable —
so when a burst of runs is created within a sub-second window, the two
orders can diverge. When they do, the next-page predicate (`run_id <
cursor`, where `cursor` is the *last* page element = the smallest
`created_at`, not necessarily the smallest `run_id`):
- **re-includes** rows already returned on a previous page (duplicates),
and
- **skips** rows it should have returned (silent data loss).
For bulk **replay** this caused runs to be replayed more than once
(replay has no idempotency guard). For the dashboard and the `runs.list`
API it could silently repeat or skip runs at page boundaries.
## Fix
Make the cursor predicate match the composite ordering:
- Cursors now encode the full `(created_at, run_id)` key as an **opaque
URL-safe base64 token**
(`base64url({"c":<createdAtMs>,"r":"<runId>"})`), and the query cuts on
the matching tuple — `(created_at, run_id) < (…)` forward / `> (…)`
backward.
- The `ORDER BY` is unchanged, so the query stays aligned with the
table's primary key — no performance regression (the tuple range
predicate is actually more index-friendly than `run_id <` alone).
- Cursors are **server-issued opaque tokens** (the SDK only echoes
`pagination.next` / `pagination.previous` back), so this needs **no
client/SDK update**. Legacy cursors were the bare internal `run_id`;
they're detected by decode failure (a cuid isn't a valid base64-wrapped
JSON payload) and fall back to the old `run_id`-only predicate, so
in-flight cursors keep working and drain naturally. New cursors also no
longer expose a bare internal run id.
- `listRunIds` is now the single cursor-aware list primitive: it returns
`{ runIds, pagination: { nextCursor, previousCursor } }`, and `listRuns`
builds on it (one place constructs cursors). Bulk actions consume the
same method and advance by `pagination.nextCursor`, finishing when it's
`null`.
- `getTaskRunsQueryBuilder` now also selects
`toUnixTimestamp64Milli(created_at) AS created_at_ms`, using a dedicated
`TaskRunListQueryResult` schema. The shared `TaskRunV2QueryResult` stays
`run_id`-only so the run-engine pending-version lookup
(`getPendingVersionIdsQueryBuilder`, which selects only `run_id`)
doesn't fail validation on a column it doesn't query.
## Tests
New `runsRepositoryCursor.test.ts` (testcontainer-backed, real
Postgres→ClickHouse replication):
- **forward** pagination returns every run exactly once when `run_id`
order is the reverse of `created_at` order (reproduces the
duplicate/skip bug — fails on `main`; this
walk-until-`nextCursor`-null-and-assert-complete is exactly the bulk
action's iteration),
- **backward** pagination round-trips to the previous page across a
boundary,
- **legacy** bare-`run_id` cursor still uses the old predicate
(backwards compatibility).
The existing `runsRepository` suites (part1–4) still pass; `part4`'s
`count new runs with listRunIds` test was updated for the new `{ runIds,
pagination }` return shape, and the `clickhouse` `taskRuns`
query-builder snapshots were regenerated for the added `created_at_ms`
column.
## Notes
- Separate, pre-existing issue (out of scope, not introduced here):
`listRuns`' backward display-slicing (`rows.slice(1, size+1)` when
`hasMore`) has an off-by-one that can return a straddled page. Tracked
separately.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Two small hygiene tweaks to **dev-only** images:
- `docker/Dockerfile.postgres`: add `--no-install-recommends` to the
partman install (leaner image, skips unneeded recommended packages).
- `internal-packages/clickhouse/Dockerfile`: run the migration helper as
a non-root user.
Both are local-dev images (the `pnpm run docker` stack) - no impact on
the published webapp image, prod, or self-hosting.
Speeds up and de-flakes the unit-test suite: testcontainers booted once
per vitest worker (per-test isolation kept only where a test runs
background redis work that outlives it), a duration-weighted shard
sequencer so each shard does roughly equal work, the slowest suites
split, two genuine flakes fixed (`streamBatchItems` shared-redis leak;
run-engine waits that relied on fixed sleeps), and transient DockerHub
pulls retried.
**Timings (CI, per-shard wall):** worst unit-test shard ~771s → ~294s;
packages/webapp shards ~250-270s, most internal ~190-240s. All 25 shards
green.
A shard breaks down as ~70s fixed setup (install / image-pull /
generate) + ~70s cold `^build` + the actual container tests. So the
remaining cost is mostly the tests themselves plus that fixed setup.
**Next (separate, timings):**
- **typecheck (~6m24s)** — the slowest check overall; bound by
full-graph `tsc`, not the TS version (a TS6 branch is still ~6m17s). The
real lever is **tsgo** (the Go compiler).
- Possible later: turbo CI caching could trim the ~70s cold build on
*warm* runs, but it's conditional (cold runs rebuild anyway) and doesn't
touch setup or test time — secondary.
`cli-v3` e2e and `sdk-compat` are path-gated (don't run on test-infra
changes) and already comfortably fast.
## Summary
Adds a trace export to the run page. From the new **Export trace** menu
you can
copy a run's full trace to the clipboard as Markdown (for pasting into
an AI
assistant) or download it as a flat Log, a Markdown table, or JSON
Lines.
Internal engine-debug events are filtered out by default, and errors are
surfaced inline with their message.
## Design
The export streams events from the store to the gzipped response one at
a time
and never materialises the span tree, so a trace of any size exports
with
bounded memory and without stalling the server. Output is flat and
chronological: each line carries its own `spanId ← parentSpanId`, so the
hierarchy is reconstructable without nesting. Formats share a single
streaming
pipeline and are pluggable via `?format=log|jsonl|markdown`, so adding a
format
is an isolated change.
## Screenshots
**Export menu**
<img width="370" height="252" alt="trace-export-menu"
src="https://github.com/user-attachments/assets/3d10304a-8c49-4606-b15d-2859b137419f"
/>
**In context**
<img width="2400" height="1802" alt="trace-export-run-page"
src="https://github.com/user-attachments/assets/46c80b30-303b-47c6-9ace-a2fb06f6cb61"
/>
A unit-test shard recently failed on a timing race rather than a real
regression - a run-engine waitpoint test sleeps 1250ms waiting on a
1000ms timeout that's processed by a ~1000ms worker poll, so on a
CPU-starved shard the margin evaporates and the whole matrix goes red.
Because `fail-fast` defaults on, that one flake cancels the sibling
shards, and the only recovery is re-running the entire matrix "just to
be sure" - which is itself slow.
This is the low-risk first pass at that pain:
- `fail-fast: false` on the webapp and internal shard matrices, so one
flaky shard no longer cancels its siblings. "Re-run failed jobs" now
re-runs just the failed shard instead of the whole matrix.
- CI-scoped `retry: process.env.CI ? 2 : 0` on the timing-sensitive
packages (`run-engine`, `redis-worker`, `schedule-engine`). Flakes
self-heal in CI; local runs stay at `retry: 0` so they still surface in
dev. A stopgap until the timing tests are made deterministic.
- `fetch-depth: 1` on the unit-test checkouts - they don't use git
history, so the full clone was wasted setup time across ~20 jobs.
- Reconcile the pre-pull image tags with what testcontainers actually
pulls (`redis:7-alpine` -> `redis:7.2`, `ryuk:0.11.0` -> `ryuk:0.14.0`)
and add `minio/minio:latest` to the webapp pre-pull. Otherwise those
images pull unauthenticated at test time and risk Docker Hub rate-limit
flakes (worst on fork PRs, where the authenticated pre-pull is skipped
entirely).
Deeper follow-ups - bigger runners, turbo remote cache, runtime-weighted
sharding, and the real root-cause fix (container reuse / template-DB
isolation + deterministic timing tests) - are tracked under TRI-10484.
Hardens the webapp Docker image and adds a CVE scan of each published
image.
- Base image `bullseye-slim` → `bookworm-slim` (Debian 12), pinned by
digest. Adds `apt-get upgrade` + `--no-install-recommends` + apt-cache
cleanup across the build stages so OS packages are patched at build
time.
- Moves the `react-email` CLI to `devDependencies` in
`internal-packages/emails` — only the `email dev` preview script uses
it; the runtime render path is `@react-email/render` +
`@react-email/components`. This also drops the bundled `esbuild` binary
from the production image.
- Bumps `goose` v3.26.0 → v3.27.1 and its Go builder image 1.23 → 1.26.
- Adds a reusable Trivy image-scan workflow wired into `publish.yml`, so
every published image (main builds and releases) is scanned for
OS-package CVEs right after it's pushed to GHCR. Report-only (writes to
the run summary), runs alongside the worker publishes so it never blocks
a deploy.
Verified locally: the image builds clean on the new base, and
`@react-email/render` carries no `esbuild` dependency so email rendering
is unaffected.
## Summary
The Prisma CLI was missing from production builds of the webapp image,
so anything that shells out to `prisma` at startup failed. The container
entrypoint and the standalone migration step both run `prisma migrate
deploy` / `prisma migrate status`, and those broke with `Command
"prisma" not found`.
## Fix
`prisma` was a `devDependency` of `@trigger.dev/database`. It had only
been landing in the pruned `--prod` install as a side effect of pnpm
auto-installing it as a peer of `@prisma/client`. A recent dependency
change shifted peer resolution so prisma stopped being materialized into
the production tree, and the CLI disappeared from the image.
Moving `prisma` into `dependencies` of `@trigger.dev/database` makes the
CLI an explicit part of production installs. It lands in the webapp
image only: the separately deployed supervisor, coordinator, and
provider images don't reach the database package in their production
trees (`core` only `devDepends` on it, so it isn't transitive), so
they're unaffected.
Verified against a locally built production image: `pnpm --filter
@trigger.dev/database exec prisma --version` now resolves the CLI and
the schema engine instead of failing.
## Summary
Adds a nullable `pricing_unit` column to the LLM model registry's
`llm_models` table, recording how each model is billed ("tokens",
"characters", "images", "minutes", "requests", "free", "not_findable").
It lets pricing-coverage reporting exclude models that aren't priced
per-token (image/video/audio models currently drag the "% priced" number
down even though they can never carry a per-token price), and lays the
groundwork for non-token pricing.
The default model catalog is entirely per-token, so `seed` and
`syncLlmCatalog` set `pricing_unit="tokens"` on those rows. The admin
LLM model form (create + edit) and the admin API get a pricing-unit
selector so admin-curated models can set it; existing rows can stay
unset.
Auto-discovered models get their unit from the model-registry pipeline,
which lands separately.
---------
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
## ✅ Checklist
- [x] I have followed every step in the [contributing
guide](https://github.com/triggerdotdev/trigger.dev/blob/main/CONTRIBUTING.md)
- [x] The PR title follows the convention.
- [x] I ran and tested the code works
---
## Testing
Upgraded the email packages in `internal-packages/emails`:
| Package | Before | After |
| --- | --- | --- |
| `@react-email/components` | `0.0.16` | `1.0.12` |
| `@react-email/render` | `^0.0.12` | `^2.0.8` |
| `react-email` (CLI) | `^2.1.1` | `^6.5.0` |
| `react-dom` | _(missing)_ | `^18.2.0` (now a required peer of render)
|
**Breaking change handled:** `render()` is now async (`Promise<string>`)
in `@react-email/render` v1+. Added `await` in the `aws-ses`, `smtp` and
`null` transports. `EmailClient` and the webapp callers were already
async and needed no changes.
Verification:
- `pnpm run typecheck --filter emails` ✅
- `pnpm run typecheck --filter webapp` (consumer of the `emails`
package) ✅
- **Before/after render comparison**: rendered all 11 templates
(magic-link, invite, welcome, alert-attempt/run/error-group,
deployment-failure/success, mfa-enabled/disabled, bulk-action-complete)
to HTML with both the old and new packages and compared them visually +
via HTML diff. Output is visually identical. The only HTML changes come
from upstream improvements: `<Body>` now wraps content in a
`<table>`/`<td>` for better email-client compatibility, an
`x-apple-disable-message-reformatting` meta tag was added, and CSS
shorthand (e.g. `margin`) is now also emitted as longhand. No visual
regressions; the `CodeBlock`/dracula theme, buttons, and row/column
layouts all render correctly.
No changeset or `.server-changes/` entry is added: `emails` is a private
internal package (not under `packages/`), and there is no user-facing
behavior change.
---
## Changelog
Upgrade `react-email` and `@react-email/{components,render}` in
`internal-packages/emails` to their latest versions and adapt the mail
transports to the now-async `render()` API.
---
## Screenshots
Rendered email templates before vs after the upgrade (visually
identical):
**Before** (`@react-email/components@0.0.16`, `render@0.0.12`)
Brings the OpenTelemetry packages up to the latest coherent release
across the webapp and the published packages (`@trigger.dev/core`, the
CLI, `@trigger.dev/sdk`) plus
`internal-packages/{tracing,testcontainers}`:
- `@opentelemetry/sdk-node` 0.218.0
- `@opentelemetry/core` 2.7.1
- `@opentelemetry/host-metrics` 0.38.3
We were already on the otel 2.x line, so this is a same-major minor move
- the versions are pinned to `@opentelemetry/sdk-node@0.218.0`'s own
declared dependency set so the experimental (0.2xx) and stable (2.x)
packages stay coherent (mixing them is the usual otel breakage).
**One code change:** otel 0.215 made `forceFlush()` a required method on
`LogRecordExporter`, so `ExternalLogRecordExporterWrapper` (core's
tracing SDK) gains a `forceFlush()` that delegates to the underlying
exporter.
**Notable upgrades along the way:** OTLP exporters can take a custom
HTTP agent (connection pooling/keepAlive on the export path), HTTP
request headers are captured at span creation, and core hot-path perf
improvements in 2.6.1/2.7. `host-metrics` 0.37→0.38 is a clean upgrade.
Patch changeset added for the three published packages. References
projects are intentionally untouched.
Verified: `@trigger.dev/core` / CLI / `@trigger.dev/sdk` build, webapp +
`@internal/tracing` typecheck - all green.
Adds wide-event observability for the supervisor: one flat-keyed JSON
line per dequeue iteration, workload-server route, and run socket
lifecycle event. Events carry `trace_id` sourced from the inbound W3C
traceparent plus `meta.run_id` and related identifiers, so they join
across services by run.
The outbound warm-start POST also forwards the inbound traceparent so
the upstream receiver continues the same trace instead of minting a new
one.
Off by default behind `TRIGGER_WIDE_EVENTS_ENABLED`. With the flag off,
no events are emitted, no ALS state is allocated, and the outbound
warm-start request is unchanged — every call site was audited to confirm
the off path is byte-identical to current behavior.
Dequeue-path phase timings recorded under `phase.<name>.duration_ms`:
`restore`, `warm_start`, `workload_create`. A `path_taken` extra
distinguishes `restore` / `warm_start` / `cold_create` /
`skipped_no_image`.
Refs TRI-9480.
## Summary
The Runs list now updates live without requiring a page refresh. Status
changes and other run fields are updated in place while runs are
executing.
When new runs matching the current filters are created, a "New runs
created" refresh button appears above the list.
Root runs now show a live child-run status breakdown directly in the
status tooltip.
### List live update
- Visible runs update in place while they are still running.
- A "New runs created" refresh button appears when new matching runs are
detected.
- Polling stops when all visible runs have finished and a refresh button
is already shown.
- Polling pauses when the browser tab is not visible.
- Runs list status updates and new-run detection share a single
runs/live polling path.
### Child-status tooltip
- Root run tooltips now display a breakdown of child run statuses.
- Child statuses are loaded when the tooltip opens (after a 400ms hover
delay).
- The tooltip stays up to date while child runs are still changing
state.
- Handles cases where child runs continue running after their parent run
has completed, or have not yet been created.
### Supporting changes
- Added hidden-tab awareness to polling.
- Added safeguards around polling inputs (`runIds` deduping and limits).
## Test plan
- [x] pnpm run typecheck --filter webapp passes
- [x] cd apps/webapp && pnpm run test
./test/presenters/mapRunToLiveFields.test.ts --run passes
- [x] cd apps/webapp && pnpm run test
./test/runsRepository.part2.test.ts --run -t "hasNewRuns" passes
### Manual smoke:
- [x] Active runs update without a page refresh.
- [x] A new matching run shows the refresh banner and the banner actions
work as expected.
- [x] Root run tooltips show live child-status updates and stop polling
once child runs settle.
---------
Co-authored-by: Ekaterina Bulatova <kathiekiwi@Ekaterinas-MacBook-Pro.local>
## Summary
The replay side of the mollifier:
- `DrainerHandler`: reads buffered snapshots and replays them through
`engine.trigger` to materialise PG rows.
- `RunEngine.createCancelledRun`: new public method the handler uses to
write CANCELED rows directly from snapshots (bypass queue + waitpoint,
emit `runCancelled`). Tolerates the cjson empty-table tags edge case
found during validation.
- Drainer fairness: org → env rotation so a heavy env doesn't starve
light ones in the same org.
- Stale-entry sweep + telemetry + alertable gauge so a stuck/offline
drainer surfaces in alerts.
Both the drainer and sweep default-off; nothing fires unless flagged on
(`TRIGGER_MOLLIFIER_DRAINER_ENABLED`,
`TRIGGER_MOLLIFIER_STALE_SWEEP_ENABLED`).
Stacked on the trigger-time decisions PR.
## Test plan
- [x] \`pnpm run typecheck --filter webapp\` passes
- [x] \`pnpm run test --filter webapp
test/mollifierDrainerHandler.test.ts\` passes
- [x] \`pnpm run test --filter webapp test/mollifierStaleSweep.test.ts\`
passes
- [x] \`pnpm run test --filter @internal/run-engine
src/engine/tests/createCancelledRun.test.ts\` passes
- [x] \`pnpm run test --filter @trigger.dev/redis-worker
packages/redis-worker/src/mollifier/drainer.test.ts\` passes
---
## Ship-gate follow-up fix
**Drainer writes SYSTEM_FAILURE on max-attempts exhaustion.** Adds an
`onTerminalFailure` callback on `MollifierDrainerOptions` so the
customer's run lands a SYSTEM_FAILURE PG row even when the drainer
exhausts `MAX_ATTEMPTS` on a retryable PG error (previously
`buffer.fail()` was called with no row written → silent data loss). The
callback runs before `buffer.fail()` on every terminal path
(non-retryable AND max-attempts-exhausted), and re-throwing a retryable
error from the callback causes the drainer to requeue rather than fail.
Bumps `@trigger.dev/redis-worker` to a **minor** changeset (additive
option + new exported types). Includes 5 unit tests covering both
terminal causes plus the requeue-on-retryable-callback-failure path and
no-callback back-compat.
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
## Summary
Drops the `TaskRun_status_runtimeEnvironmentId_createdAt_id_idx` index
from the `TaskRun` table. After #3742 gated the legacy
`WAITING_FOR_DEPLOY` drain to V1-engine workers only, this index sees
zero scans on both writer and reader replicas. Removing it cuts index
maintenance on every `TaskRun` INSERT/UPDATE.
## Why
The index existed to support `WHERE status = X AND runtimeEnvironmentId
= Y` queries from `ExecuteTasksWaitingForDeployService`, which is
V1-only and no longer triggered on V2 deployments. A code grep across
`apps/webapp` and `internal-packages/run-engine` confirmed no V2
production query uses this access pattern — every other `status:` filter
on `TaskRun` is paired with `id`/`friendlyId`/`parentSpanId` and uses a
different index.
Dropping it also unlocks HOT updates on the dequeue path. The dequeue
`UPDATE` modifies `status` (`QUEUED` -> `DEQUEUED`), and `status` is the
leading column of this index — its presence blocked HOT eligibility for
every `TaskRun` UPDATE. With the index gone, dequeue UPDATEs can become
HOT, reducing WAL bytes and removing the B-tree page contention on this
index's right-edge leaves.
Uses `DROP INDEX CONCURRENTLY` to avoid blocking writes during the drop.
## Sequencing
Should only ship once #3742 has soaked long enough to confirm the index
is genuinely cold (24h+ of zero scans on `pg_stat_user_indexes`).
## Summary
When a background worker registers, the engine resolves runs that were
queued before the worker was ready (status `PENDING_VERSION`). That
lookup used to scan a Postgres status index on `TaskRun`. Move it to
ClickHouse: query candidate run ids from `task_runs_v2`, then refetch
the actual rows from Postgres by primary key with a `status =
'PENDING_VERSION'` guard for idempotency.
## Design
The lookup is a pluggable interface on the run engine
(`PendingVersionRunIdLookup`). The webapp wires a ClickHouse-backed
implementation through the org-scoped `clickhouseFactory` using a new
`"engine"` client type, configured by `RUN_ENGINE_CLICKHOUSE_*` env
vars. The URL falls back to `CLICKHOUSE_URL` when unset, so self-hosted
deployments don't need new config to keep working.
When the lookup returns no candidates, one bounded retry is scheduled
~5s later to cover ClickHouse replication lag against `task_runs_v2`.
The Postgres status guard on both the candidate refetch and the inner
`updateMany` prevents double-promotion when a retry races with a
concurrent deploy.
Tests cover three existing PENDING_VERSION cases via a small
Postgres-backed test adapter; new ClickHouse-backed integration tests
will follow.
## Summary
Drops the unused composite Postgres index
`TaskRun_scheduleId_createdAt_idx`. The schedule list view reads from
ClickHouse, so this index served no Prisma query while still being
maintained on every `TaskRun` INSERT/UPDATE. Removing it reduces write
amplification on the primary database.
Sibling to the prior drop of `TaskRun_scheduleId_idx` and the earlier
removal of the `TaskRun.scheduleId` foreign key — all stemming from
migrating schedule-aware reads to ClickHouse.
## Verification
- Sampled `pg_stat_user_indexes` for `TaskRun` over multiple hours —
zero scans against this index.
- Grepped the codebase for any Prisma query filtering
`TaskRun.scheduleId` — none found. All schedule-aware listing routes
through `clickhouseRunsRepository`.
Added `OrganizationDataStore` which allows orgs to have data stored in
specific separate services.
For now this is just used for ClickHouse. When using ClickHouse we get a
client for the factory and pass in the org id.
Particular care has to be made with two hot-insert paths:
1. RunReplicationService
2. OTLPExporter
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: Claude <noreply@anthropic.com>
(`OBJECT_STORE_BASE_URL`) and a named protocol provider
(`OBJECT_STORE_DEFAULT_PROTOCOL=s3`), chat.agent session snapshot writes
landed in the named provider but reads fell through to the default — so
the recovery boot couldn't find the snapshot it had just written.
After a mid-stream cancel, the missing snapshot triggered a fallback
replay path that dropped the user's follow-up message, leaving the chat
stuck in `submitted` indefinitely.
Fix:
- New `/api/v1/sessions/:id/snapshot-url` route handles PUT + GET
symmetrically — both prefix unprefixed keys with
`OBJECT_STORE_DEFAULT_PROTOCOL` so they always round-trip through the
same store.
- `Session.chatSnapshotStoragePath` persists the resolved URI on first
write so future protocol changes don't strand existing snapshots.
Reads prefer the stored URI and fall back to the computed default for
pre-column sessions.
- SDK calls `createChatSnapshotUploadUrl` / `getChatSnapshotUrl`; the
generic v1/v2 packets endpoints are unchanged.
## Test plan
- [x] Configure local with two providers (R2 default + MinIO `s3` named)
and `OBJECT_STORE_DEFAULT_PROTOCOL=s3`.
- [x] Reproduce hang: send a message, cancel mid-stream, send another —
without the fix it hangs in `submitted`; with the fix it streams.
- [x] Snapshot lands in the `s3`-protocol bucket and
`Session.chatSnapshotStoragePath` is set after first write.
- [x] SDK unit tests pass; webapp typecheck passes.
Env-var lookups via `GET
/api/v1/projects/:projectRef/envvars/:slug/:name` run a Prisma
`findMany` on `EnvironmentVariableValue` filtered by `environmentId` +
`isSecret`. The only existing indexes are the primary key and a unique
on `(variableId, environmentId)`, so `environmentId` is never the
leading column — the planner falls back to a Parallel Seq Scan over the
whole table to find what is, in practice, a handful of rows per
environment.
Two changes:
- Add a btree index on `EnvironmentVariableValue(environmentId)` so the
planner switches to an index scan. The composite `(variableId,
environmentId)` unique stays in place; the new index is purely additive.
- Route the `findMany` inside `getEnvironmentWithRedactedSecrets`
through the read replica via a new `replicaClient` constructor param on
the repository (defaulting to `$replica`, mirroring how `prismaClient`
defaults to `prisma`). Writes and read-after-write methods stay on the
primary.
## Test plan
- [ ] `pnpm run typecheck --filter webapp`
- [ ] Confirm `EXPLAIN` plan flips from Parallel Seq Scan to an index
scan
- [ ] Existing env-var route tests still pass
## Summary
Two papercuts new contributors hit running this repo locally:
1. Fresh clones default to v1 (Redis-only) realtime streams, so Sessions
and `chat.agent` error with `"S2 configuration is missing"`, even though
the `s2` service is already in `docker/docker-compose.yml` and pre-seeds
a `trigger-local` basin. Wire `REALTIME_STREAMS_S2_*` to it in
`.env.example` so the new-contributor flow just works. (Also drop the s2
healthcheck: the image is distroless, so the `wget` check always reports
unhealthy.)
2. Two clones can't both run `pnpm run docker` because ports, project
name, and container names are all hardcoded. Parameterize every host
port as `${VAR:-default}`, drive the project name via
`COMPOSE_PROJECT_NAME` (with a top-level `name:` field as the default),
prefix container names with `${CONTAINER_PREFIX:-}`, and pass
`--env-file .env` so compose reads the same root `.env` the webapp does.
The "Running multiple instances side by side" block in `.env.example`
lists every overridable knob.
Also split the optional services (`electric-shard-1`, `ch-ui`,
`toxiproxy`, `nginx-h2`, `otel-collector`, `prometheus`, `grafana`) into
`docker-compose.extras.yml` behind a new `pnpm run docker:full` script.
The core stack keeps everything the webapp actually needs to boot:
postgres, redis, electric, minio, clickhouse + migrator, s2-lite.
Defaults match every previous hardcoded value, so existing setups keep
working without touching `.env`.
## Test plan
- [x] `pnpm run docker` on a clean clone brings up the core services on
the standard ports under the `triggerdotdev-docker` project name.
- [x] Setting `COMPOSE_PROJECT_NAME=triggerdotdev-docker-alt` + the
`*_HOST_PORT` overrides in `.env` brings up a second stack alongside the
default one with no port or container-name clashes.
- [x] Webapp boots cleanly against the default `.env.example` values;
`/healthcheck` returns 200, no S2 errors.
- [x] s2-lite basin `trigger-local` accepts an append + read via the
same REST endpoints the webapp uses.
- [x] `pnpm run docker:full` brings up the optional services alongside
the core ones in the same project.
## Summary
Codify two rules for ClickHouse migration authors that came out of the
029/030 ordering incident on the TRI-9367 test cloud deploy:
1. **Number files to `max(existing) + 1`, never slot in below the
latest.** Goose runs in strict mode in the cloud deploy pipeline and
refuses to apply a missing version below the current version — slotting
a file in below an already-applied number blocks the next deploy.
2. **DDL must be idempotent** (`ADD COLUMN IF NOT EXISTS`, `DROP COLUMN
IF EXISTS`, `CREATE TABLE IF NOT EXISTS`, etc.) so a retry or
out-of-order apply (`goose up --allow-missing` for local recovery,
manual fixups) is a no-op rather than an error.
## Where the rules live
- `internal-packages/clickhouse/CLAUDE.md` — full rules + example for
migration authors (and AI agents writing migrations).
- `.claude/REVIEW.md` — added a 🔴 finding under "What makes a 🔴
Important finding" so PR reviewers flag either fault as blocking.
The existing migration files are left untouched; the idempotency
requirement applies going forward.
## Test plan
- [ ] Next ClickHouse migration PR uses `IF NOT EXISTS` / `IF EXISTS`
forms
- [ ] No new migration files numbered below an already-applied version
on test/prod
## Summary
Renumber `029_add_task_kind_to_task_runs_v2.sql` →
`031_add_task_kind_to_task_runs_v2.sql` to fix a deploy-blocking
out-of-order migration, and make the DDL idempotent with `ADD COLUMN IF
NOT EXISTS` / `DROP COLUMN IF EXISTS`.
## Root cause
- Migration `030_create_sessions_v1.sql` landed on main on 2026-04-28
(PR #3417) and was applied to test cloud ClickHouse on a subsequent
deploy. Current goose version on test ClickHouse: **30**.
- Migration `029_add_task_kind_to_task_runs_v2.sql` was authored later
on 2026-05-10 as part of the Sessions primitive PR series (`be1a6cf8`).
- The next test cloud deploy failed because goose strict-mode refused to
apply a missing version *before* the current version:
```
goose run: error: found 1 missing migrations before current version 30:
version 29: 029_add_task_kind_to_task_runs_v2.sql
```
## Fix
1. **Rename to `031_*`** (next available number after 030). Goose now
treats it as a new migration after 030 and applies it cleanly on
test/prod where the column does not yet exist.
2. **Make the DDL idempotent** (`ADD COLUMN IF NOT EXISTS`). The
original 029 may have been applied in environments that ran goose with
`--allow-missing` (e.g. some local dev databases) — those would have the
column already, and the rename causes goose to see 031 as new and
re-attempt the ADD. Idempotent DDL keeps that path safe. The `Down`
mirrors with `DROP COLUMN IF EXISTS`.
## Test plan
- [ ] Test cloud deploy (after this lands) successfully runs the
ClickHouse migration step
- [ ] `task_kind` column shows up on `trigger_dev.task_runs_v2`
post-migration
- [ ] Local environments that had previously applied 029 do not error on
the next `goose up`
Adds Sessions, a durable, run-aware stream primitive that scopes
session.in / session.out records to a session (not a single run).
Records survive run boundaries; reconnect-from-last-event-id is built in.
Server foundation:
- New /realtime/v1/sessions/:session/:io/append + /records routes
- sessionRunManager + sessionsRepository + clickhouseSessionsRepository
- mintRunToken for short-lived per-session tokens
- s2Append retry-with-backoff + undici cause diagnostics
- /api/v[12]/packets/* exempt from customer rate limits
- BackgroundWorker schema gains taskKind enum (TASK, AGENT, SCHEDULED)
- TaskRun.taskKind column + clickhouse 029_add_task_kind_to_task_runs_v2
Core types:
- new sessionStreams, inputStreams, realtimeStreams packages in @trigger.dev/core
- session-streams-api / realtime-streams-api surface
Sessions dashboard UI (the primitive's own viewer):
- /sessions index + detail routes
- SessionsTable, SessionFilters, SessionStatus, CloseSessionDialog
- AGENT/SCHEDULED filter in RunFilters + TaskTriggerSource
Includes the sessions-primitive changeset.
## Summary
Queues that use concurrency keys can no longer bypass the per-queue
length cap, and the "Queued | Running" columns in the dashboard now show
the true total across all CK variants instead of 0.
The cap and the dashboard both relied on `ZCARD` of the base queue key,
but CK-keyed runs live under `<base>:ck:<variant>` keys. Any queue that
used concurrency keys read 0 — letting a single CK variant grow
unbounded past the user's configured cap.
## Fix
Two per-base-queue counters are maintained inside the CK Lua scripts:
`<base>:lengthCounter` and `<base>:runningCounter`. Non-CK
enqueue/dequeue paths are untouched.
Counters are lazy-initialized the first time a CK enqueue (or nack)
lands on a queue: the Lua script sums `ZCARD` across the variants
tracked by `ckIndex`, sets the counter, then `INCR`s. Pre-existing CK
backlog on already-populated queues is captured automatically — no batch
migration required.
`INCR`/`DECR` is gated on `ZADD`/`SADD` returning 1 (a new entry vs an
idempotent no-op), so duplicate enqueues or re-dequeues don't inflate
the counter.
The counter is `SET` with a 24-hour TTL on init. `INCR`/`DECR` do not
extend the TTL, so the counter expires daily and the next CK operation
re-seeds it from `ckIndex`. This bounds any drift that accumulates
during the rolling-deploy overlap window — where old (un-Tracked) and
new (Tracked) webapp instances briefly coexist — to ≤24 hours, with no
admin sweep or background reconciler needed.
Read paths pipeline `ZCARD`/`SCARD` on the base key + `GET` on the
counter and sum. A missing counter is treated as 0, so pure non-CK
queues see the same answer as before.
The counter-aware scripts ship alongside the originals with a `Tracked`
suffix for rolling-deploy safety; a follow-up PR will drop the originals
once this has rolled out.
## Test plan
- [ ] `pnpm run test --filter @internal/run-engine` — 116 tests pass,
including a new `ckCounters.test.ts` covering lazy init from
pre-existing backlog, churn, floor-at-zero, the non-CK regression case,
mixed CK + non-CK on the same base queue, idempotent re-enqueue
(ZADD-already-exists), 24h TTL on the counter, and nack re-seeding after
counter expiry.
- [ ] Verified end-to-end against a live local environment:
- Triggered 24 CK enqueues across 4 variants → `lengthCounter=16`,
`runningCounter=8`, dashboard showed Queued=16 / Running=8 for the CK
queue.
- Set the env queue cap to 16, triggered 12 more enqueues → 8 succeeded,
4 rejected with `QueueSizeLimitExceededError`.
- Deleted the counter on a queue with 31 messages already sitting in CK
variants, triggered one more enqueue → counter materialized to 31 from
the `ckIndex` sum, then INCR'd.
Katia Bulatova