The API key policy methods are optional on the plugin-facing controller
contract, so `Pick` over it yields optional members that these call sites
would have to guard. Both already receive the LazyController singleton,
which has substituted its fail-closed defaults, so point them at
HostRbacController and keep the call sites guard-free.
The API-key policy methods are capability extensions, so declare them
optional on RoleBaseAccessController and normalize the surface in
LazyController rather than requiring every implementation to carry them.
An authorization extension is compiled against whichever core commit its
base image ships, so a required additive method forces both sides to move
together and turns rolling an extension back into a build failure instead
of a graceful degradation.
Absence fails closed: no preset catalogue, and prepareApiKeyPolicy refuses
outright rather than defaulting to full access, so an extension below this
contract cannot mint a credential. Keys already issued are unaffected —
they authorize from the scopes persisted on their row.
loader.create() now returns HostRbacController, the total surface, so host
callers neither guard nor invent their own absent-extension default.
A number passed to `expirationTime` is a Unix timestamp in seconds, not
milliseconds as the JSDoc claimed. Following the old docs produced a token
that effectively never expired.
Also fail loudly when an additional API key reaches a local self-signing
fallback. Those keys are not the environment's JWT signing material, so the
token would never verify. Every endpoint that returns a public access token
sets `x-trigger-jwt`, so this is unreachable today.
## Summary
Additional environment API keys can use SDK APIs that require public
access tokens. The SDK detects the additional-key format and asks the
Trigger.dev server to mint scoped tokens instead of attempting to sign
them locally.
Root environment keys retain their existing local-signing behavior.
Trigger and batch clients also prefer server-issued tokens returned in
response headers while preserving compatibility with older servers.
## Deployment notes
This package update is safe to publish before servers expose additional
key creation. Existing root keys continue to use the current path, while
an additional key used with an older server fails with an actionable
upgrade error.
## Summary
The runs list on a task's page now updates live, matching the main Runs
page. Run rows update their status, duration, and cost in place as runs
progress, and a "N new runs" button appears in the header when newer
runs come in so you can pull them into the list without a manual
refresh. This applies to both standard and scheduled task pages.
## Design
It reuses the Runs page's polling hook. A task page scopes its runs by
the task in the URL path rather than a `tasks` query filter, so the hook
now takes an optional task slug and scopes new-run detection to it. The
"new runs" button sits in the header, outside the deferred runs table,
so the count is lifted to the page and the click action is passed
through a ref. That keeps the table streaming on first load instead of
blocking the header on the runs query.
When newer runs come in, a `1 new run` button appears in the task page
header, to the left of the time filter. Clicking it pulls the new runs
into the list.
Adds a gitignore rule for `**/docs/superpowers/` so locally-generated
planning and design scratch docs under that path aren't committed;
preventive only, no-op for existing tree.
The SSO & Directory Sync settings page decided access by comparing the
organization's plan code against the literal string `"enterprise"`. The
webapp now reads a `hasSso` entitlement from plan limits.
## Changes
- **`settings.sso` route** — `planAllowsSso` reads `limits.hasSso`
rather than the plan code; the loader and the action gate on a shared
`getSsoEntitlement` helper.
- **`platform.v3.server`** — new `getSsoEntitlement(orgId)` returning
`entitled | not_entitled | unknown`, behind a new SWR cache namespace
(60s fresh / 120s stale, memory + Redis). This replaces an uncached
billing round-trip that previously ran on every settings load, so the
page gets cheaper than it was.
- **`directorySyncEffects`** — the entitlement is now checked before
applying membership effects, per organization and memoised across a
batch.
- **`@trigger.dev/platform` 1.2.0 → 1.3.0** — required, see below.
## Behaviour worth reviewing
**Revocation now stops SCIM.** Previously the plan check existed only on
the settings page, so an org that lost access kept receiving
directory-sync pushes indefinitely; only the config UI froze. Provision
*and* deprovision are gated, so a revoked entitlement can't remove
members either.
**An unreadable entitlement throws instead of skipping.** Effects are
idempotent and the worker retries, so retrying is lossless where
dropping would silently lose a directory change. It's raised at `warn`
level so a transient billing blip doesn't page anyone.
**The login path is deliberately untouched.** A hard entitlement check
there turns a billing outage into a login outage. Consequence: an org
that loses the entitlement keeps its existing SSO logins working until
the connection is removed. Gating sign-in is a separate decision.
**Self-hosted is unaffected.** With no billing service configured the
helper returns `entitled`, leaving plugin presence and the kill switch
as the only gates — a self-hoster who installed the plugin isn't locked
out of it.
## The dependency bump is load-bearing
The `Limits` schema is a plain `z.object`, so it *strips* unknown keys.
On 1.2.0 the `hasSso` field was silently discarded during parsing and
read as `undefined` no matter what billing sent — a structural accessor
would not have helped. Verified against both builds:
```
1.2.0 → parsed: true | hasSso survives: false
1.3.0 → parsed: true | hasSso survives: true
```
This PR therefore cannot merge before 1.3.0 is published, which it now
is.
## Testing
`apps/webapp/test/directorySyncEffects.server.test.ts` — 7 tests over
the gate: applies when entitled, skips provision and deprovision when
not, throws a warn-level retryable error when unreadable, resolves once
per org across a batch, and gates per org so one unentitled org doesn't
block another.
`pnpm run typecheck --filter webapp` passes (18/18), oxfmt and oxlint
clean.
**Before:** `ip-address` resolved twice in `pnpm-lock.yaml` — `8.1.0`
under `@jsonhero/json-infer-types`, and `10.0.1` under
`express-rate-limit`.
**After:** a single `ip-address@10.2.0` entry, shared by both chains.
**How:** `express-rate-limit@8.2.1` pinned `ip-address` to an exact
version, so the parent itself had to move — `8.5.1` onwards declares a
range instead, and `@modelcontextprotocol/sdk` already allows `^8.2.1`,
so scoping that parent to `^8.6.0` lets `ip-address` resolve on its own.
`@jsonhero/json-infer-types` caps `ip-address` at `^8.1.0` and is
already at its latest published release, so that chain gets a scoped
override instead of a parent bump. `jsbn` and `sprintf-js` drop out of
the tree as a side effect.
Both overrides are parent-scoped, so the `cli-v3` chain is deliberately
untouched: it resolves `@modelcontextprotocol/sdk` 1.25.2, which
declares `express-rate-limit ^7.5.0` and pulls in no `ip-address` at
all.
`pnpm-lock.yaml` regenerated. `package.json` and `pnpm-lock.yaml` are
the only two files changed.
Nothing in the repo imports `ip-address` or `express-rate-limit`
directly. Both chains are transitive under `apps/webapp` —
`@jsonhero/schema-infer` (used by `TestTaskPresenter.server.ts`) and
`@vercel/sdk` — so no published `@trigger.dev/*` package is affected.
---
## Testing
- `pnpm install --lockfile-only` regenerates cleanly, and `pnpm install
--frozen-lockfile --lockfile-only` passes, so the lockfile matches the
manifests.
- Package churn is limited to the intended set: `express-rate-limit`
8.2.1 to 8.6.0, `ip-address` 8.1.0 and 10.0.1 collapsing to 10.2.0, and
`jsbn` / `sprintf-js` removed. No other resolution moved.
- `@jsonhero/json-infer-types` only calls `new Address4()` / `new
Address6()` inside a try/catch to classify strings. Ran that exact logic
against both `8.1.0` and `10.2.0` over 27 inputs (v4, v6, zone IDs,
CIDR, IPv4-mapped, malformed, empty, non-strings): identical results in
all 27. Both are still CJS named exports in `10.2.0`, with the same
`engines` floor.
- Drove the real `inferSchema()` path from `@jsonhero/schema-infer` with
`ip-address` forced to `10.2.0`; it still detects `ipv4` and `ipv6`
formats correctly.
- `express-rate-limit` 8.6.0 keeps the same `express` peer range (`>=
4.11`) and the same node floor as 8.2.1. Its new `debug` dependency
resolves to a version already present in the tree.
- `oxfmt --check` passes on the modified `package.json`.
- Both bumped versions clear the repo's `minimumReleaseAge` window; the
newest `express-rate-limit` (8.6.1) and `ip-address` (10.2.1+) releases
do not yet, which is why this lands on 8.6.0 and 10.2.0.
- Not run here: a full monorepo install, typecheck and test suite. No
TypeScript changed, and neither package leaks types into ours —
`ip-address` is not referenced in `json-infer-types`' or
`schema-infer`'s declaration files — so CI should be the judge of the
wider suite.
---
## Changelog
Routine dependency maintenance, no behaviour change. No changeset or
`.server-changes/` entry: the diff touches only the root `package.json`
and `pnpm-lock.yaml`, not `packages/*`, `integrations/*`, `apps/webapp/`
or `apps/supervisor/`.
Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: nicktrn <55853254+nicktrn@users.noreply.github.com>
<!-- ccr-slack-attribution -->
_Requested via [Slack
thread](https://triggerdotdev.slack.com/archives/C097ZHVKZFA/p1785082528841609)_
`TaskListPresenter` created promises that nothing ever consumed. Two of
the three deferred metrics promises it returned had no reader, no
`await` and no `.catch()`, so when the query behind one of them failed
the rejection had nowhere to go.
## Before / After
**Before**
- `TaskListPresenter.call()` returned four things: `tasks`, `activity`,
`runningStats` and `durations`. Its only caller reads `tasks` and
`runningStats`.
- Every load of the tasks page therefore fired two ClickHouse queries
whose results were thrown away.
- If either of those two queries failed, the resulting promise rejection
was unhandled — nothing was awaiting it and nothing had attached an
error handler, so it surfaced as an unhandled rejection at the process
level rather than as an error anyone could attribute to a request.
**After**
- `TaskListPresenter.call()` returns `tasks` and `runningStats` only.
- Two fewer queries run per tasks-page load.
- There is no longer an unconsumed promise that can reject without a
handler. `runningStats` is awaited by its caller, so its failures
continue to be handled the way they always were.
Nothing changes on screen: the tasks page renders `hourlyActivity` and
`runningStates`, and neither of the removed values fed either of those.
## How
The removed values were verified unreferenced before deleting anything:
- `TaskListPresenter` has exactly one caller,
`UnifiedTaskListPresenter`, which reads `taskResult.tasks` and
`taskResult.runningStats` and nothing else.
- No file anywhere in the repo — app code, tests, or type re-exports —
reads an `activity` or `durations` field off the presenter's result.
- `UnifiedTaskListPresenter` builds its own
`unifiedTaskListHourlyActivity` query for the 24h chart the page
actually renders, which is what made the presenter's separate 7-day
daily activity data redundant.
- `getDailyTaskActivity` and `getAverageDurations` on
`ClickHouseEnvironmentMetricsRepository` had no callers other than the
two lines being deleted, so they and their now-orphaned helpers and
types were removed too.
Changes:
- `apps/webapp/app/presenters/v3/TaskListPresenter.server.ts` — drop the
`activity` and `durations` fields (both from the main return and from
the no-current-worker early return) and the two repository calls behind
them. Drop the unreferenced `TaskActivity` type alias. The "don't await
this" comment on the remaining `runningStats` promise now spells out
that the caller has to consume it.
- `apps/webapp/app/services/environmentMetricsRepository.server.ts` —
remove `getDailyTaskActivity` and `getAverageDurations` from the
`EnvironmentMetricsRepository` interface and its ClickHouse
implementation, along with `fillInDailyTaskActivity` and the
`DailyTaskActivity` / `AverageDurations` types.
`getCurrentRunningStats` is the control that shows the diagnosis is
right. It throws on query failure in exactly the same way as the two
removed methods — `if (queryError) throw queryError` — but it never
produced an unhandled rejection, because `UnifiedTaskListPresenter`
passes its promise into a `Promise.all(...).then(...)` chain that the
route then awaits. Same failure mode, opposite outcome, and the only
difference is whether anything consumes the promise.
Follow-ups, not in this PR:
- `AgentListPresenter` returns three sparkline promises in the same
shape and they look similarly unconsumed. Left alone here to keep this
change reviewable.
- With these two callers gone, the `getTaskActivity` and
`getAverageDurations` query builders in `@internal/clickhouse` have no
remaining callers in this repo. Whether to remove them is a separate
call for someone who owns that package.
## ✅ Checklist
- [x] I have followed every step in the [contributing
guide](https://github.com/triggerdotdev/trigger.dev/blob/main/CONTRIBUTING.md)
- [x] The PR title follows the convention.
- [x] I ran and tested the code works
---
## Testing
- `pnpm run typecheck --filter webapp` — passes. This is the meaningful
check here: it proves nothing still references the removed fields,
methods or types.
- `pnpm run format` and `pnpm run lint:fix` — clean, no changes
produced.
- No test file referenced the removed symbols, so no test needed
updating.
---
## Changelog
Server-only change, so this carries a `.server-changes/` note rather
than a changeset:
`.server-changes/task-list-remove-unused-metrics-queries.md`.
> The tasks page no longer runs two queries whose results were never
displayed, cutting wasted work on every page load and removing a source
of hidden server errors
---
## Screenshots
_No visual change — the removed data was never rendered._
Co-authored-by: Claude <noreply@anthropic.com>
## ✅ Checklist
- [x] I have followed every step in the [contributing
guide](https://github.com/triggerdotdev/trigger.dev/blob/main/CONTRIBUTING.md)
- [x] The PR title follows the convention.
- [ ] I ran and tested the code works — n/a, this PR only deletes two
workflow files
---
## Summary
**Before:** two scheduled workflows in this repo posted Dependabot
digests to Slack — a critical-alert check every morning at 08:00 UTC,
and a summary of all open alerts on Mondays at 08:00 UTC.
**After:** neither runs. This reporting is handled centrally now, so the
two in-repo workflows were duplicating it.
**How:** deletes `.github/workflows/dependabot-critical-alerts.yml` and
`.github/workflows/dependabot-weekly-summary.yml`. Both were
self-contained — inline shell, no shared scripts or composite actions —
so nothing else in `.github/` referenced them.
Dependabot itself is unchanged: `.github/dependabot.yml`, alerts, and
version updates all keep working. This removes only the two Slack
notifiers.
The `ENABLE_DEPENDABOT_ALERTS` repository variable existed only to
switch these two workflows off. Nothing else reads it, so it can be
removed from the repository settings if it's set.
---
## Testing
No runtime code changes — this PR only removes two scheduled workflow
files. Verified that nothing else in the repo references either
filename, either workflow name, or the `ENABLE_DEPENDABOT_ALERTS`
variable.
---
## Changelog
Removed the two in-repo scheduled workflows that posted Dependabot
digests to Slack.
Co-authored-by: Claude <noreply@anthropic.com>
🦋 Changesets PR / Create Release PR (push) Has been cancelled
🚀 Publish Trigger.dev Docker / 📣 Dispatch main image (push) Has been cancelled
🚀 Publish Trigger.dev Docker / typecheck (push) Failing after 2s
🚀 Publish Trigger.dev Docker / units (push) Failing after 2s
🚀 Publish Trigger.dev Docker / publish-webapp (push) Has been skipped
🚀 Publish Trigger.dev Docker / publish-worker-v4 (push) Has been skipped
🚀 Publish Trigger.dev Docker / scan-webapp (push) Has been skipped
🚀 Publish Trigger.dev Docker / scan-supervisor (push) Has been skipped
## Summary
The realtime runs feed hydrates run rows from read replicas, which means
it needs a replica-lag gate to avoid serving a run's previous state
right after a write. Setting
`REALTIME_BACKEND_NATIVE_RUN_READS_FROM_PRIMARY=1` reads those rows from
each run store's primary instead, so there is no lag to gate against: no
probe, no wake delay, no stale-read retries. Off by default, so nothing
changes unless you set it.
## Design
The run stores already decide replica-vs-primary from the *brand* on the
read client they are handed: a branded replica keeps the read on the
owning store's replica, an unbranded writer escalates it to that store's
own primary. So this is a one-line choice at the hydrator, and it stays
correct across topologies. With the run-ops split on, each leg lands on
its own writer and the caller's client is never forwarded across
databases; with the split off, it is the single database's primary.
```ts
const runReader = new RunHydrator({
readClient: runReadsFromPrimary ? prisma : $replica,
runStore,
});
```
The same flag skips constructing the lag estimator, since probing a
replica the feed no longer reads would be measuring the wrong thing.
Independently, `AuroraReplicaLagSource` detected Aurora by letting
`aurora_replica_status()` fail, on the assumption that the app-level
catch made that free. It isn't: an unresolvable function is a query
error the driver reports to the error log on every sample, so a
non-Aurora replica produced a continuous stream of error events while
the estimator quietly fell through to its next candidate. It now
resolves the function with `to_regproc` and memoizes the answer, so the
unparseable call never reaches the wire.
## Summary
When a `chat.agent` (or `chat.createSession`) turn's model stream fails
mid-response (e.g. a transport timeout like `UND_ERR_BODY_TIMEOUT`), the
assistant output that already streamed was dropped: `onTurnComplete`
fired with `responseMessage: undefined`, and the manual loop's
`turn.complete()` rethrew without keeping the partial. Apps that
register `hydrateMessages` are hit hardest, since boot-time tail-replay
recovery is off by design.
This preserves the streamed-so-far assistant output while still
reporting the turn as errored, so persistence and recovery keep it.
## Scope of behavior change
Only the **error path** changes. Successful turns are unaffected: the
same chunks stream to the client in the same order, and
backpressure/cancel behave as before. Everything here is a correctness
improvement on a turn that hit a source-stream failure.
## What it does
Follow-up to #4304 (`chat.pipeAndCapture`), extending the same
partial-recovery to the two loops that lacked it:
- **`chat.agent`**: taps the response stream (via a `TransformStream`,
so pass-through backpressure and cancel are preserved) to buffer chunks,
and on a source-stream failure reconstructs the partial (preferring the
`onFinish` message). It's surfaced on the error-path `onTurnComplete`
(`responseMessage`, `rawResponseMessage`, `uiMessages`, `newUIMessages`,
`newMessages`) and committed to the accumulator so the next turn and the
reboot snapshot keep it.
- **`chat.createSession` / `turn.complete()`**: the reconstructed
partial is accumulated (so `turn.uiMessages` reflects it and the caller
can persist after catching) before `turn.complete()` rethrows.
`onBeforeTurnComplete` stays skipped on the error path (it hands out a
writer for a stream that has already broken).
## Correctness properties (each covered by a regression test)
Each test below was confirmed to fail without its fix:
- The recovered partial reaches `onTurnComplete` and the next turn's
accumulated messages.
- An already-committed (possibly enriched) response is not overwritten
if a post-response hook then throws.
- Incomplete tool parts are cleaned from the recovered partial (text
kept), so the UI and model views agree and the next turn isn't poisoned.
- A prior turn's model-only compaction survives an errored turn (append
only the new tail, don't reconvert the full history).
- A reconstructed fragment that reuses an existing message id does not
clobber the complete message.
- Queued `chat.response` data parts are folded into the recovered
partial, matching the success path.
- `newMessages` (model delta) stays symmetric with `newUIMessages`.
## Tests
New `chat-agent-source-stream-error.test.ts` covers the cases above. The
full `@trigger.dev/sdk` unit suite passes and the package build is green
across all supported runtimes (Node 20 to 26, Bun, Deno, Cloudflare
Workers).
Adds an optional `TRIGGER_WARM_START_DISPATCH_URL`. The warm-start
dispatch request uses it when set, otherwise falls back to
`TRIGGER_WARM_START_URL`, so the dispatch target can differ from the
default warm-start URL. No behavior change when unset.
## Summary
Several list endpoints accepted an unbounded page size (`perPage` /
`per_page` / `pageSize`). An unbounded page lets one request pull an
arbitrarily large result set and do a proportional amount of work, which
is a poor default for a shared API.
This clamps the page size to 100 on every list endpoint that was
uncapped, matching the existing cap on `api.v1.runs` and
`api.v1.sessions`. Clamping rather than rejecting keeps existing clients
working: a request for a larger page returns up to 100 items and offset
pagination continues from there.
## Endpoints capped
- `api.v1.schedules` (`perPage`)
- `api.v1.queues` (`perPage`)
- `resources.…versions` (`per_page`)
- `resources.…queues` (`per_page`)
- `admin.api.v1.…engine.report` (`per_page`)
- `admin.api.v1.llm-models` (`pageSize`)
Already capped, left as-is: `api.v1.runs`, `api.v1.sessions`,
`api.v1.deployments`.
## Summary
The batches list page orders by `createdAt DESC, id DESC` filtered by
environment and a created-at window, but the only supporting index on
`BatchTaskRun` was `(runtimeEnvironmentId, id)`. That index can't
satisfy the `createdAt` ordering, so on environments with a large number
of batches the query fell back to a full table scan and in-memory sort,
which could run long enough to hit the statement timeout.
## Fix
Adds `(runtimeEnvironmentId, createdAt DESC, id DESC)` on
`BatchTaskRun`. The query now reads straight from the index in order
with no sort step, returning a page with only a handful of heap fetches
instead of scanning the whole environment slice.
The migration uses `CREATE INDEX CONCURRENTLY IF NOT EXISTS`, so it
takes no table lock and is a no-op if the index already exists.
Adds Prometheus metrics so the supervisor's outbound HTTP calls are
observable - including client-side failures that previously only
surfaced as a log line.
- `supervisor_outbound_request_total{name, method, status, outcome}` -
counts every outbound request. `outcome` separates a transport failure
(`network_error`), an HTTP error response (`http_error`), a response
that failed schema validation (`invalid_response`), and success (`ok`).
- `supervisor_outbound_request_duration_seconds{name, outcome}` -
latency histogram. Leaner labels than the counter (no `status`) to avoid
bucket×label cardinality; buckets match the existing dequeue-latency
histogram since these calls share the same retrying HTTP client and
long-poll envelope.
Coverage:
- The warm-start request (a one-off `fetch`) - instrumented inline; the
response status code is now also included in the failure log (it was
previously dropped).
- All worker API client calls (`SupervisorHttpClient`: dequeue, run
attempt start/complete, heartbeats, snapshots, continue, suspend,
debug-log, connect) - routed through a single instrumented `request()`
helper that reports via an optional `onHttpRequestComplete` callback on
the client, which the supervisor wires into the counter + histogram.
Low cardinality by design: `name` is a **static per-endpoint label**
(e.g. `dequeue`, `start_run_attempt`), never the interpolated URL - so
no run/snapshot IDs land in labels, mirroring the templated `route`
labels on the inbound HTTP server.
Registered on the existing metrics registry, exposed on `/metrics` with
no new wiring. Internal-only change (no package release needed), so the
changelog note is a single `.server-changes` entry.
## Summary
The container entrypoint runs under `set -x`, which echoes every command
to the logs with its variables expanded. Several startup guards
reference full database connection strings, so the DSN (including the
password) was printed to the container logs on every boot. This turns
tracing off around those lines so connection strings are never traced,
while leaving migration behavior and ordinary startup logging unchanged.
## Fix
The leaking lines are the `[ -n "$RUN_OPS_DATABASE_URL" ]` and `[ -n
"$RUN_OPS_LEGACY_DIRECT_URL" ]` guards, and the ClickHouse block (its `[
-n "$CLICKHOUSE_URL" ]` guard plus the lines that build `GOOSE_DBSTRING`
from `CLICKHOUSE_URL`). `set -x` prints each of these with the
credential expanded. Tracing is now disabled around each region and
restored afterward, so non-secret tracing is preserved everywhere else.
The existing legacy-migration subshell already protected its own command
body; this adds the missing protection for the guards and the ClickHouse
block.
```sh
{ set +x; } 2>/dev/null
if [ -n "$RUN_OPS_DATABASE_URL" ]; then
set -x
...
```
## Verification
Built the webapp image and ran it with dummy sentinel connection strings
whose password token is `S3NTINEL_PW_DoNotLog`, then grepped the boot
logs.
Before (unmodified), the token appears in the traced guards:
```
+ [ -n postgresql://user:S3NTINEL_PW_DoNotLog@fake-host:6432/run-ops ]
+ [ -n postgresql://user:S3NTINEL_PW_DoNotLog@fake-host:5432/legacy ]
+ [ -n https://default:S3NTINEL_PW_DoNotLog@fake-host:8443 ]
```
After, `grep S3NTINEL_PW_DoNotLog` on the same run returns nothing, and
the normal "skipping ... migrations" lines still log.
Pins `tar` to `7.5.19` via a root `pnpm.overrides` entry, replacing a
stale range override (`tar@>=7 <7.5.11`) that no longer matched any
installed copy.
The single override collapses all resolved `tar` copies onto one
version:
- `packages/cli-v3` — direct dependency (was 7.5.13)
- `@kubernetes/client-node` (apps/supervisor) — transitive (was 7.5.13)
- `cacache` — transitive (was 6.2.1)
- `giget` — transitive (was 6.2.1)
No source changes; cli-v3's published `^7.5.13` spec already permits
`7.5.19`, so no changeset is needed.
## Summary
The split run-store's id-set read path (`#findRunsByIdSet`, used by the
runs-list hydrate, the realtime hydrator, and engine sweeps) queried the
new store for the entire id set and then probed the legacy store for the
misses. A run's residency is a total function of its id (run-ops ids
live in the new store, every other id in legacy), so each id belongs to
exactly one store. Route each id to its owner and query each store only
for its own ids, in parallel. Same result set, and while a split is
active with most runs still on legacy it removes a wasted new-store
query from every id-set read.
## Change
`#findRunsByIdSet` now partitions the ids by `classifyResidency` and
runs one bounded query per store (skipping an empty side), in parallel,
mirroring `expireRunsBatch` and the single-run `#route`. `finalizeRows`
still applies orderBy/take/skip globally over the merged set.
This drops the id-set path's cross-store fallback, which existed to
prefer the new-store copy when the same id was present in both stores.
That collision cannot arise when each id maps to exactly one store
(nothing writes a legacy-shaped id into the new store), so the fallback
is dead code. The two id-set tests that asserted "new copy wins on
collision" now assert the routing invariant: a legacy-shaped id resolves
to the legacy store and the path never consults the new store.
The open-predicate path (`#findRunsOpen`) is unchanged: an open `where`
has no id to route on, so it still unions both stores and dedupes.
## Default billing alerts + billing limit page UX
- New orgs get default billing alerts: $5, $100, $500, $1000, $2500.
Existing orgs are backfilled by a billing-side data migration (companion
[PR](https://github.com/triggerdotdev/cloud/pull/1657)).
- The billing limit form starts with nothing selected for orgs that
never set a limit — the save button appears once an option is picked.
- The yellow banner now also shows on the billing limits page itself,
asking to configure a limit. Hidden everywhere for members who can't
manage billing.
- Also fixes billing limit alert preview.
Tests
- `apps/webapp/test/billingLimitsRoute.test.ts` — dirty logic for
empty/selected mode
- `apps/webapp/test/billingAlertsDefaults.test.ts` — default values
- `apps/webapp/test/billingAlertsFormat.test.ts` — preview after a limit
change
## Summary
Adds two counters to the native realtime backend so we can see how much
duplicate row serialization the change router does per batch. When a run
changes it can match several held feeds at once (a run subscription plus
one or more tag/list feeds), and today each matching feed serializes
that run's wire value independently. These counters quantify that
fan-out so we can decide whether a shared serialization step is worth
it.
## What they measure
- `realtime_native.emission_run_serializations`: total wire-value
serializations performed across feeds per batch (what the current path
does).
- `realtime_native.emission_distinct_serializations`: distinct (columns,
run) rows those serializations cover (what a serialize-once-per-batch
step would do).
Average feeds-per-run is `run_serializations / distinct_serializations`,
and `1 - distinct / run_serializations` is the serialization work a
shared step could save. Wired through a new optional `onEmissionFanout`
callback on the router. No behavior change.
## Summary
5 improvements, 5 bug fixes.
## Improvements
- Add `node-24` and `node-26` as supported `runtime` options in
`trigger.config.ts`. The `experimental-node-24` and
`experimental-node-26` names are now deprecated aliases and emit a
deprecation warning; switch to `node-24` / `node-26` instead.
([#4337](https://github.com/triggerdotdev/trigger.dev/pull/4337))
```ts
import { defineConfig } from "@trigger.dev/sdk";
export default defineConfig({
runtime: "node-24",
project: "<your-project-ref>",
});
```
- Avoid logging task run environment variable values at debug level
([#4336](https://github.com/triggerdotdev/trigger.dev/pull/4336))
- Custom chat agent loops get two ergonomic wins for owning the turn
loop. ([#4304](https://github.com/triggerdotdev/trigger.dev/pull/4304))
`chat.writeTurnComplete()` now returns the turn boundary's resume
cursors (`lastEventId` for the output stream and `sessionInEventId` for
the input stream), so you can persist them straight from the task
instead of round-tripping them back from the client.
```ts
const { lastEventId, sessionInEventId } = await
chat.writeTurnComplete();
await db.chats.update(chatId, { lastEventId, sessionInEventId });
```
`chat.pipeAndCapture()` no longer throws when a stream is stopped or
fails. It now returns a `PipeAndCaptureResult` whose `message` holds any
partial output captured before the stop or failure, alongside a typed
`status` (`"complete" | "aborted" | "error"`) and, on failure, the
`error`. Read the message off the result:
```ts
const { message, status, error } = await chat.pipeAndCapture(result, {
signal,
});
if (message) conversation.addResponse(message);
if (status === "error") logger.error("turn failed", { error });
```
Note: `pipeAndCapture` previously resolved to `UIMessage | undefined`.
Update call sites to read `.message` from the returned result.
- Suppress a build-time warning that could appear in Vite-based projects
when the optional `@ai-sdk/otel` package is not installed.
([#4188](https://github.com/triggerdotdev/trigger.dev/pull/4188))
## Bug fixes
- Fixes intermittent `trigger dev` run crashes where a run could fail at
boot with a cryptic `Cannot find module .../dev-run-worker.mjs` after a
rebuild had cleaned up the build directory the run was launched against.
Dev runs now retry cleanly instead of hard-crashing when their build
directory is missing, the dev watchdog no longer removes the build tree
of a still-running session, and a run assigned to a worker version that
was superseded by a rebuild now fails fast with a clear message instead
of silently hanging until it times out.
([#4276](https://github.com/triggerdotdev/trigger.dev/pull/4276))
## Server changes
These changes affect the self-hosted Docker image and Trigger.dev Cloud:
- Refreshed the side menu: separate organization and account menus, a
new project switcher, and the menu is now resizable by dragging its
edge. The account Profile page has also been redesigned.
([#4066](https://github.com/triggerdotdev/trigger.dev/pull/4066))
- Allow different organization members to use the same development
branch name without sharing or colliding with each other's branch
environments.
([#4323](https://github.com/triggerdotdev/trigger.dev/pull/4323))
- Limit account settings email input to 254 characters.
([#4330](https://github.com/triggerdotdev/trigger.dev/pull/4330))
- Prevent duplicate Staging and Preview environments when account setup
requests overlap
([#4261](https://github.com/triggerdotdev/trigger.dev/pull/4261))
- Fix the docs link on the empty Prompts page, which pointed to a page
that no longer exists.
([#4247](https://github.com/triggerdotdev/trigger.dev/pull/4247))
<details>
<summary>Raw changeset output</summary>
# Releases
## @trigger.dev/build@4.5.7
### Patch Changes
- Updated dependencies:
- `@trigger.dev/core@4.5.7`
## trigger.dev@4.5.7
### Patch Changes
- Fixes intermittent `trigger dev` run crashes where a run could fail at
boot with a cryptic `Cannot find module .../dev-run-worker.mjs` after a
rebuild had cleaned up the build directory the run was launched against.
Dev runs now retry cleanly instead of hard-crashing when their build
directory is missing, the dev watchdog no longer removes the build tree
of a still-running session, and a run assigned to a worker version that
was superseded by a rebuild now fails fast with a clear message instead
of silently hanging until it times out.
([#4276](https://github.com/triggerdotdev/trigger.dev/pull/4276))
- Add `node-24` and `node-26` as supported `runtime` options in
`trigger.config.ts`. The `experimental-node-24` and
`experimental-node-26` names are now deprecated aliases and emit a
deprecation warning; switch to `node-24` / `node-26` instead.
([#4337](https://github.com/triggerdotdev/trigger.dev/pull/4337))
```ts
import { defineConfig } from "@trigger.dev/sdk";
export default defineConfig({
runtime: "node-24",
project: "<your-project-ref>",
});
```
- Avoid logging task run environment variable values at debug level
([#4336](https://github.com/triggerdotdev/trigger.dev/pull/4336))
- Updated dependencies:
- `@trigger.dev/core@4.5.7`
- `@trigger.dev/build@4.5.7`
- `@trigger.dev/schema-to-json@4.5.7`
## @trigger.dev/core@4.5.7
### Patch Changes
- Add `node-24` and `node-26` as supported `runtime` options in
`trigger.config.ts`. The `experimental-node-24` and
`experimental-node-26` names are now deprecated aliases and emit a
deprecation warning; switch to `node-24` / `node-26` instead.
([#4337](https://github.com/triggerdotdev/trigger.dev/pull/4337))
```ts
import { defineConfig } from "@trigger.dev/sdk";
export default defineConfig({
runtime: "node-24",
project: "<your-project-ref>",
});
```
## @trigger.dev/python@4.5.7
### Patch Changes
- Updated dependencies:
- `@trigger.dev/sdk@4.5.7`
- `@trigger.dev/core@4.5.7`
- `@trigger.dev/build@4.5.7`
## @trigger.dev/react-hooks@4.5.7
### Patch Changes
- Updated dependencies:
- `@trigger.dev/core@4.5.7`
## @trigger.dev/redis-worker@4.5.7
### Patch Changes
- Updated dependencies:
- `@trigger.dev/core@4.5.7`
## @trigger.dev/rsc@4.5.7
### Patch Changes
- Updated dependencies:
- `@trigger.dev/core@4.5.7`
## @trigger.dev/schema-to-json@4.5.7
### Patch Changes
- Updated dependencies:
- `@trigger.dev/core@4.5.7`
## @trigger.dev/sdk@4.5.7
### Patch Changes
- Custom chat agent loops get two ergonomic wins for owning the turn
loop. ([#4304](https://github.com/triggerdotdev/trigger.dev/pull/4304))
`chat.writeTurnComplete()` now returns the turn boundary's resume
cursors (`lastEventId` for the output stream and `sessionInEventId` for
the input stream), so you can persist them straight from the task
instead of round-tripping them back from the client.
```ts
const { lastEventId, sessionInEventId } = await
chat.writeTurnComplete();
await db.chats.update(chatId, { lastEventId, sessionInEventId });
```
`chat.pipeAndCapture()` no longer throws when a stream is stopped or
fails. It now returns a `PipeAndCaptureResult` whose `message` holds any
partial output captured before the stop or failure, alongside a typed
`status` (`"complete" | "aborted" | "error"`) and, on failure, the
`error`. Read the message off the result:
```ts
const { message, status, error } = await chat.pipeAndCapture(result, {
signal,
});
if (message) conversation.addResponse(message);
if (status === "error") logger.error("turn failed", { error });
```
Note: `pipeAndCapture` previously resolved to `UIMessage | undefined`.
Update call sites to read `.message` from the returned result.
- Suppress a build-time warning that could appear in Vite-based projects
when the optional `@ai-sdk/otel` package is not installed.
([#4188](https://github.com/triggerdotdev/trigger.dev/pull/4188))
- Updated dependencies:
- `@trigger.dev/core@4.5.7`
</details>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Wraps the workload_auth_gate_total Counter in the singleton helper (same
pattern as reloadingRegistry.server.ts) so a dev hot reload doesn't
crash with "A metric with the name workload_auth_gate_total has already
been registered". No production behavior change.
## Summary
Adds `node-24` and `node-26` as first-class `runtime` options in
`trigger.config.ts`. Previously these Node versions were only reachable
via the `experimental-node-24` / `experimental-node-26` names.
Those experimental names are now **deprecated aliases**: they still
resolve to `node-24` / `node-26` for backwards compatibility, but
loading a config that uses them prints a deprecation warning pointing at
the new name.
```ts
export default defineConfig({
runtime: "node-24",
project: "<your-project-ref>",
});
```
## Details
- `ConfigRuntime` (the public config schema) now accepts `node-24` and
`node-26` directly; the internal `BuildRuntime` already supported them,
so base images and the deploy path are unchanged.
- `resolveBuildRuntime` passes the new names straight through and keeps
mapping the experimental aliases to their replacements.
- Renamed the runtime helper from `isExperimentalConfigRuntime` to
`isDeprecatedConfigRuntime` and added `deprecatedRuntimeReplacement` so
the CLI can name the replacement in its warning.
- Docs snippet updated to list the new versions and flag the deprecated
names.
Add `.worktrees/` to the repo `.gitignore`.
The pre-push hook runs `oxfmt --check .` and `oxlint .` over the whole
tree, and those tools only read the in-repo ignore files (not a user's
global gitignore). Local git-worktree checkouts placed under
`.worktrees/` therefore got linted/formatted, failing the hook on
unrelated code. Ignoring the directory keeps both tools out of worktree
checkouts. No source changes.
Add a Prometheus gauge `workload_token_enforcement_mode` set to 1 for
the active `WORKLOAD_TOKEN_ENFORCEMENT` value
(`disabled`/`log`/`enforce`), emitted at startup on the shared registry.
The existing mint/verify counters don't distinguish `log` from `enforce`
(the verify outcome is recorded before the reject decision), so
dashboards can't tell which mode a cluster is running. This gauge makes
the active mode queryable at a glance. Supervisor typecheck passes.
Restores the debug panel on the **Tasks** and **Runs** pages, and makes
the data it shows copyable.
Admin/impersonation only — no change for regular users, so there's no
`.server-changes`
<img width="909" height="1420" alt="CleanShot 2026-07-22 at 12 05 27@2x"
src="https://github.com/user-attachments/assets/ce2da167-dc23-422f-83f3-4f4aee9ed32c"
/>
## Summary
The empty-state panel on the Prompts page linked to a docs path that no
longer exists, so the "Prompts docs" button returned a 404. It now
points to the current prompts documentation at /docs/ai/prompts,
matching the link already used in the page header.
## What
The organization side menu previously showed an "Enterprise" badge next
to the SSO & Directory Sync item for any org not on the enterprise plan.
That badge is now removed so the item renders without it.
## Screenshot (before)
<img width="1428" height="649" alt="CleanShot 2026-07-10 at 08 24 41"
src="https://github.com/user-attachments/assets/b9787363-972f-4dd5-bf61-486680f49f4c"
/>
## Summary
Limits user account email addresses to 254 characters in profile
settings and onboarding. Oversized values are rejected before the
uniqueness lookup, and the form fields enforce the same limit in the
browser.
## Fix
Both email update flows use a shared bounded email schema. Basic
validation completes before the uniqueness lookup runs.
## Summary
Allow each organization member to use the same development branch name
without colliding with another member's environment. Fixes#4320.
## Fix
Development branches now use the existing member-scoped project, slug,
and organization-member key for upserts. Preview branches retain their
project-wide shortcode behavior.
New development branches receive distinct shortcodes while keeping their
readable, member-scoped slugs. Existing branches continue to resolve
through the member-scoped key, so this requires no migration or
backfill.
Two small corrections to the organization **Usage** page credits
display.
### 1. Label the credits panel "Credits" (was "Promo credits")
The panel surfaces any credit balance, not only promo-code redemptions,
so "Promo credits" is misleading when the credits come from another
source. Renamed the heading to "Credits".
### 2. Don't show "Included usage" for Enterprise orgs
Enterprise inherits the Pro plan's `includedUsage` value, so the Usage
bar rendered an "Included usage: $50" tier marker for Enterprise
organizations. Enterprise bills against prepaid credits rather than a
per-month included-usage tier, so the marker was misleading. The
`tierLimit` marker is now suppressed for Enterprise (`plan.type ===
"enterprise"`).
Verified with `pnpm run typecheck --filter webapp`.
Redesign of the main side menu: separates Projects and Accounts from the
Organization menu and makes the menu resizable.
**Main changes**
- **Organization & Account menus**: the top-left is now a dedicated
organization menu (Settings, Usage, Billing, Team, SSO, integrations),
with a separate account menu beside it (Profile, PATs, Security,
Logout).
- **Project switcher**: a new Project section above the Environment
selector.
- **Resizable side menu**: drag the right edge to set a custom width
(saved per user), or click the edge to collapse/expand.
- **Environment selector**: reworked to match the Project menu,
including dev-branch handling.
- **Account Profile page**: redesigned into the Security page's
row-and-divider layout.
Preview URL: https://samejr-org-menu-update.triggerlabs.dev/https://github.com/user-attachments/assets/9b199576-6037-4ea6-9bdb-3ee15265b8c2
## Summary
Two ergonomic additions for custom chat-agent loops that own the turn
loop (`chat.customAgent`, `chat.createSession`, and the hand-rolled
primitives).
`chat.writeTurnComplete()` now resolves to `{ lastEventId }`, the resume
cursor for the start of the next turn. A custom loop can persist it
straight from the task instead of round-tripping it back from the client
after the turn ends. The value was already produced internally by the
turn-complete write; the public wrapper simply discarded it.
`chat.pipeAndCapture()` no longer throws when a stream is stopped or
fails. It now resolves to a `PipeAndCaptureResult` carrying any partial
`message` captured before the stop or failure, a typed `status`
(`"complete" | "aborted" | "error"`), and the `error` on failure.
Previously a failed stream threw and the partial was lost, and an abort
was captured only when the AI SDK happened to fire `onFinish` in time.
```ts
const { message, status, error } = await chat.pipeAndCapture(result, { signal });
if (message) conversation.addResponse(message);
if (status === "error") logger.error("turn failed", { error });
const { lastEventId } = await chat.writeTurnComplete();
await db.chats.update(chatId, { lastEventId });
```
## Design
`pipeAndCapture` wraps the pipe in a `try/catch` and classifies the
outcome from the abort signal (a stop drains the source stream cleanly
rather than throwing) versus a thrown error. It also races the
`onFinish` capture against a timeout so a hard stop that prevents
`onFinish` from firing can't hang the caller. This mirrors the capture
path `chat.agent` already uses internally.
The `finishReason` from `onFinish` is surfaced too, since it was already
captured on the built-in path.
The internal `turn.complete()` helper keeps its existing contract: it
still returns `UIMessage | undefined`, still throws on a genuine stream
failure, and still discards output on a full run cancel.
## Breaking change
`chat.pipeAndCapture` previously resolved to `UIMessage | undefined`.
Call sites now read `.message` off the result. This is a young,
low-level API; the docs examples are updated in this PR.
Fixes TRI-12078
## Summary
Prevents concurrent environment setup requests from creating duplicate
Staging and Preview environments.
## Fix
Adds database-enforced uniqueness for root Staging and Preview
environments.
If two requests race, the losing request loads the environment created
by the winner and continues successfully instead of creating a duplicate
or returning an error.
## Summary
Upgrades the workspace to TypeScript 6.0.3 and applies the compiler,
type, and build configuration changes required to preserve package
layouts and existing runtime behavior, apart from correcting the HTTP
status field used for deployment connection errors.
## Compatibility
- Centralizes TypeScript 6.0.3 through the pnpm workspace catalog.
- Replaces compiler options and module resolution modes that TypeScript
6 no longer accepts.
- Restores explicit Node types where TypeScript 6 no longer includes
them transitively.
- Adds explicit declaration build roots that preserve each package's
existing output layout.
- Patches tsup to stop injecting the removed `baseUrl` option during
declaration builds.
- Uses type-only assertions for stricter typed-array and stream
definitions without changing runtime behavior.
- Reads the EventSource v3 HTTP status from `code`, so deployment
connection errors include it correctly.
- Keeps standalone CLI compatibility fixtures pinned to their existing
TypeScript version and lockfiles.
`turbo run typecheck` and the complete PR test suite are green.
## What
The `/promo` page heading rendered with overlapping lines — the two
lines of "Promo codes are for new accounts" collided.
## Why
The page used `Header2` stretched to display sizes (`sm:text-2xl
md:text-3xl lg:text-4xl`), but `Header2` bakes in a fixed `leading-6`
(24px). A 36px font in a 24px line box makes wrapped lines overlap. It
only showed at `sm`+ widths and only on headings that wrap to 2+ lines,
which is why it slipped through — the short single-line headings on the
same page looked fine.
## Fix
Switch both headings to `Header1` — the page-title primitive the sibling
login pages (`login._index`, `login.magic`) already use for exactly this
size. Add `leading-tight` (relative line-height, scales with font size,
and this heading uniquely wraps to two lines) and `pb-4` to match the
login pages' spacing convention.
## Testing
Manually verified the signed-in view (`/promo` while logged in) renders
as two clean, non-overlapping lines across breakpoints. Pure CSS/layout
change — no automated test.
## Summary
5 improvements, 9 bug fixes.
## Breaking changes
- Self-hosted deployments no longer ship shared default credentials;
fresh installs generate their own. If yours still uses a previously
published default, set a unique value before upgrading, or set
`ALLOW_INSECURE_DEFAULT_SECRETS=true` to keep booting while you migrate.
([#4316](https://github.com/triggerdotdev/trigger.dev/pull/4316))
## Improvements
- Require explicit browser approval for CLI and MCP login, with
resilient polling while approval is pending.
([#4316](https://github.com/triggerdotdev/trigger.dev/pull/4316))
- Deployed task telemetry now reports the deployment identifier (e.g.
`deployment_abc123`) in the `worker.id` attribute, instead of an opaque
internal value. Upgrade to get the readable identifier in your own
OpenTelemetry exporters.
([#4316](https://github.com/triggerdotdev/trigger.dev/pull/4316))
- Prevent prototype pollution when applying run metadata operations or
reconstructing nested telemetry attributes, while preserving legitimate
`constructor` and `prototype` fields.
([#4316](https://github.com/triggerdotdev/trigger.dev/pull/4316))
- Add helpers to mint and verify the deployment-scoped token used to
authenticate run controllers to the platform.
([#4316](https://github.com/triggerdotdev/trigger.dev/pull/4316))
## Server changes
These changes affect the self-hosted Docker image and Trigger.dev Cloud:
- Added optional request rate limiting for telemetry ingestion
endpoints.
([#4316](https://github.com/triggerdotdev/trigger.dev/pull/4316))
- Background-worker deployment lookups are now scoped to the
authenticated environment.
([#4316](https://github.com/triggerdotdev/trigger.dev/pull/4316))
- Updating a GitHub App installation from the callback flow is now
scoped to your own organization, so an installation ID belonging to
another organization can no longer be used to refresh that
organization's installation record. The GitHub App installation session
is also now single-use, so completing an installation callback
invalidates its state and it can no longer be replayed.
([#4316](https://github.com/triggerdotdev/trigger.dev/pull/4316))
- Scope schedule and environment-variable writes to the caller's project
and environment
([#4316](https://github.com/triggerdotdev/trigger.dev/pull/4316))
- Reject compute snapshot callbacks that do not match the snapshot
request that created them.
([#4316](https://github.com/triggerdotdev/trigger.dev/pull/4316))
- Require secret-key authentication to initialize the session out
(agent→client) stream, matching the append route.
([#4316](https://github.com/triggerdotdev/trigger.dev/pull/4316))
- Live run and trace subscriptions now validate their identifiers more
strictly and only return data from your own organization.
([#4316](https://github.com/triggerdotdev/trigger.dev/pull/4316))
- Window-function names in the query compiler are now validated against
the allowlist, matching how other function calls are handled.
([#4316](https://github.com/triggerdotdev/trigger.dev/pull/4316))
- Authenticate run controllers to the platform with a signed,
deployment-scoped token.
([#4316](https://github.com/triggerdotdev/trigger.dev/pull/4316))
- Verify that worker actions (starting, completing, and continuing a
run, and reading its snapshots) target a run belonging to the caller's
environment.
([#4316](https://github.com/triggerdotdev/trigger.dev/pull/4316))
<details>
<summary>Raw changeset output</summary>
# Releases
## @trigger.dev/build@4.5.6
### Patch Changes
- Updated dependencies:
- `@trigger.dev/core@4.5.6`
## trigger.dev@4.5.6
### Patch Changes
- Require explicit browser approval for CLI and MCP login, with
resilient polling while approval is pending.
([#4316](https://github.com/triggerdotdev/trigger.dev/pull/4316))
- Deployed task telemetry now reports the deployment identifier (e.g.
`deployment_abc123`) in the `worker.id` attribute, instead of an opaque
internal value. Upgrade to get the readable identifier in your own
OpenTelemetry exporters.
([#4316](https://github.com/triggerdotdev/trigger.dev/pull/4316))
- Updated dependencies:
- `@trigger.dev/core@4.5.6`
- `@trigger.dev/build@4.5.6`
- `@trigger.dev/schema-to-json@4.5.6`
## @trigger.dev/core@4.5.6
### Patch Changes
- Prevent prototype pollution when applying run metadata operations or
reconstructing nested telemetry attributes, while preserving legitimate
`constructor` and `prototype` fields.
([#4316](https://github.com/triggerdotdev/trigger.dev/pull/4316))
- Require explicit browser approval for CLI and MCP login, with
resilient polling while approval is pending.
([#4316](https://github.com/triggerdotdev/trigger.dev/pull/4316))
- Add helpers to mint and verify the deployment-scoped token used to
authenticate run controllers to the platform.
([#4316](https://github.com/triggerdotdev/trigger.dev/pull/4316))
## @trigger.dev/python@4.5.6
### Patch Changes
- Updated dependencies:
- `@trigger.dev/core@4.5.6`
- `@trigger.dev/build@4.5.6`
- `@trigger.dev/sdk@4.5.6`
## @trigger.dev/react-hooks@4.5.6
### Patch Changes
- Updated dependencies:
- `@trigger.dev/core@4.5.6`
## @trigger.dev/redis-worker@4.5.6
### Patch Changes
- Updated dependencies:
- `@trigger.dev/core@4.5.6`
## @trigger.dev/rsc@4.5.6
### Patch Changes
- Updated dependencies:
- `@trigger.dev/core@4.5.6`
## @trigger.dev/schema-to-json@4.5.6
### Patch Changes
- Updated dependencies:
- `@trigger.dev/core@4.5.6`
## @trigger.dev/sdk@4.5.6
### Patch Changes
- Updated dependencies:
- `@trigger.dev/core@4.5.6`
</details>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
## Summary
5 improvements, 5 bug fixes.
## Improvements
- Add experimental Node.js 24 and 26 task runtimes. Set `runtime` to
`experimental-node-24` or `experimental-node-26` in `trigger.config.ts`.
([#4085](https://github.com/triggerdotdev/trigger.dev/pull/4085))
- Add `defaultRegion` to the project GET and list API responses; null
when unset.
([#4146](https://github.com/triggerdotdev/trigger.dev/pull/4146))
## Server changes
These changes affect the self-hosted Docker image and Trigger.dev Cloud:
- Transient internal sync failures are now retried quietly instead of
surfacing as errors.
([#4270](https://github.com/triggerdotdev/trigger.dev/pull/4270))
- Optionally route ClickHouse read traffic to a read replica while
writes stay on the primary. Set `CLICKHOUSE_READER_URL` to move all
reads, or target the busiest paths with `RUNS_LIST_CLICKHOUSE_URL` (runs
list) and `EVENTS_READER_CLICKHOUSE_URL` (traces, spans, logs). All
optional; unset keeps current behavior.
([#4081](https://github.com/triggerdotdev/trigger.dev/pull/4081))
- Remove the deprecated realtime stream write endpoint used by retired
v3 task clients.
([#4250](https://github.com/triggerdotdev/trigger.dev/pull/4250))
- Fix batchTrigger requests that set a per-item idempotency key failing
with an error instead of creating and deduplicating the runs
([#4271](https://github.com/triggerdotdev/trigger.dev/pull/4271))
- Speed up idempotency checks on `batchTrigger` calls that use
idempotency keys. Large batches against a task with a big run history no
longer degrade to multi-second lookups.
([#4255](https://github.com/triggerdotdev/trigger.dev/pull/4255))
- The "Preview branches" usage on the Limits page now counts only
preview branches.
([#4283](https://github.com/triggerdotdev/trigger.dev/pull/4283))
- Avoid opening a redundant database connection pool when the legacy and
primary databases are the same server, preventing connection usage from
doubling.
([#4253](https://github.com/triggerdotdev/trigger.dev/pull/4253))
- Fix pages occasionally loading unstyled or failing to load during a
deploy. The dashboard now reloads automatically to recover.
([#4282](https://github.com/triggerdotdev/trigger.dev/pull/4282))
<details>
<summary>Raw changeset output</summary>
# Releases
## @trigger.dev/build@4.5.5
### Patch Changes
- Updated dependencies:
- `@trigger.dev/core@4.5.5`
## trigger.dev@4.5.5
### Patch Changes
- Add experimental Node.js 24 and 26 task runtimes. Set `runtime` to
`experimental-node-24` or `experimental-node-26` in `trigger.config.ts`.
([#4085](https://github.com/triggerdotdev/trigger.dev/pull/4085))
- Updated dependencies:
- `@trigger.dev/core@4.5.5`
- `@trigger.dev/build@4.5.5`
- `@trigger.dev/schema-to-json@4.5.5`
## @trigger.dev/core@4.5.5
### Patch Changes
- Add experimental Node.js 24 and 26 task runtimes. Set `runtime` to
`experimental-node-24` or `experimental-node-26` in `trigger.config.ts`.
([#4085](https://github.com/triggerdotdev/trigger.dev/pull/4085))
- Add `defaultRegion` to the project GET and list API responses; null
when unset.
([#4146](https://github.com/triggerdotdev/trigger.dev/pull/4146))
## @trigger.dev/python@4.5.5
### Patch Changes
- Updated dependencies:
- `@trigger.dev/core@4.5.5`
- `@trigger.dev/build@4.5.5`
- `@trigger.dev/sdk@4.5.5`
## @trigger.dev/react-hooks@4.5.5
### Patch Changes
- Updated dependencies:
- `@trigger.dev/core@4.5.5`
## @trigger.dev/redis-worker@4.5.5
### Patch Changes
- Updated dependencies:
- `@trigger.dev/core@4.5.5`
## @trigger.dev/rsc@4.5.5
### Patch Changes
- Updated dependencies:
- `@trigger.dev/core@4.5.5`
## @trigger.dev/schema-to-json@4.5.5
### Patch Changes
- Updated dependencies:
- `@trigger.dev/core@4.5.5`
## @trigger.dev/sdk@4.5.5
### Patch Changes
- Updated dependencies:
- `@trigger.dev/core@4.5.5`
</details>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
## What
Run the shared Trivy image scan on every published image through a
single reusable workflow.
## Changes
- Generalise the image-scan workflow (`trivy-image-webapp.yml` ->
`trivy-image.yml`) - it was already parameterised by `image-ref`; only
the run-summary label was image-specific.
- `publish-worker-v4.yml`: expose `version` + `image_repo` as workflow
outputs (single-entry matrix, so unambiguous).
- `publish.yml`: run the shared scan from each publish job
(`scan-webapp`, `scan-supervisor`).
Report-only (writes a table to the run summary), OS packages only
(`vuln-type: os` - library deps stay with Dependabot), never blocks the
publish.
## Stacked on #4284 — tests only
This PR contains **only the tests** that guard the production fixes in
#4284 (its base). Review #4284 first; this branch adds no production
code.
## What
Caller-driven replica-lag and idempotency guards for every fixed site:
- Each guard **drives the real exported caller** (route loader/action,
presenter `.call()`, service, or engine method) against a **real
Postgres** with the owning replica frozen via the shared
`laggingReplica` testcontainer primitive — never a store-seam
reimplementation.
- For a **fixed** site the guard goes **RED when the production change
is reverted**; for a **tolerated read-view** site it's a caller-driven
**GREEN** proof the miss self-heals (returns null/empty, no mutation,
row live on primary).
- The **global-scope idempotency** guard drives the real dedup + claim
path through a **real `MollifierBuffer` over a Redis testcontainer**
(real SETNX/poll/publish), and covers the cross-DB **andWait** waitpoint
wiring and the **expired/failed clear-and-recreate** reacquire cases.
Run with `vitest --no-file-parallelism` (testcontainers). Verified
GREEN, and revert→RED verified per fixed site.
## What & why
Two related correctness fixes for the run-ops DB split. Under the split,
run-store reads can route to a **lagging read replica**; a just-written
run/waitpoint/batch can then be missed, causing a wrong decision.
**1. Read-your-writes → owning primary.** Surfaced first as an
intermittent `wait.until({ idempotencyKey })` re-wait on retry. Auditing
the run-store read surface found the same class at sibling sites (some
gating mutations or returning spurious 404s, others
tolerable/self-healing). Reads that must observe their own writes now
route to the owning **primary**
(`findRun`/`findWaitpoint`/`findBatchTaskRunByFriendlyId` →
`*OnPrimary`, a primary re-read on a miss, or a retryable 404 where the
SDK polls). Read-view reads stay on the replica. All additive — the
happy path is unchanged.
**2. Global-scope idempotency across the split.** A `global`-scope key
carries no per-run salt, so the same `(env, task, key)` triggered
concurrently from parents resident on **different** run-ops DBs could
dedup-miss on each DB and create a duplicate (the per-DB unique index
can't enforce cross-DB uniqueness). Such triggers (global scope, or
scope-absent, while split is active) are serialized through the existing
Redis idempotency claim, the loser resolves the winner by id across both
DBs, and the claim is reacquired on the expired/failed
clear-and-recreate path. `run`/`attempt` scope embed the run id and
never contend.
## Stacked for review
This is the **base** of a 2-PR stack, split so review is easier:
- **This PR** — production code only (34 files).
- **Stacked tests PR →
https://github.com/triggerdotdev/trigger.dev/pull/4285** — the
caller-driven guards (55 test files) on top of this branch.
## Validation
Local run-ops split, **both 2-DB and 3-DB**, fresh boot on this branch:
SDK canary 64/71 (only the known concurrency/input-streams/s3 failures),
quarantine sweep **0 unexpected** (340 pass / 16 known / 4 local) in
each topology, dashboard e2e 0 failed. No product regressions.
## What
- Runs `apps/webapp/scripts/runOpsLegacyGuard.ts --check` as its own PR
job (`runops-guard`), so code that reaches a run-graph table through the
control-plane Prisma client instead of the RunStore fails the build.
- Adds a `trigger-runops` oxlint plugin with two fast, in-editor rules
scoped to `apps/webapp/app`: one for direct `prisma.taskRun`-style
access, one for a control-plane client wired into a read-through slot.
These are the cheap fence; the guard is the type-aware gate.
- Fixes `CancelTaskRunService.callV1`: historical V1 runs are
legacy-resident, so its two finalize writes now go through
`runOpsLegacyPrisma` instead of the control-plane client (they'd miss
the row once legacy is a separate database).
- Regenerates the guard baseline, which had drifted stale (it referenced
files deleted in an earlier PR).
## Why
The guard existed but ran nowhere, so its baseline rotted and a real
residency gap (the V1 cancel writes) sat undetected. Wiring it into CI
turns it into a ratchet against new control-plane run-graph access.
## Verification
Local, against a clean regen: `oxfmt --check`, `oxlint .`, `guard
--check`, and `typecheck --filter webapp` all pass. Remaining baseline
entries are 4 batch-results router reads through type-opaque `as
PrismaReplicaClient` casts (correct at runtime, accepted) + 2 sanctioned
legacy annotations.
## Summary
Correctness and performance fixes for deployments that split run data
across more than one database. Single-database / self-hosted deployments
are unaffected (they collapse to a single read/write path).
- **Batches list (dashboard):** for some organizations the Batches list
could hide older batches or show them out of order. It now orders and
paginates by creation time (with the id as a stable tiebreak), so every
batch appears exactly once, newest first. The pagination cursor format
changes; older in-flight cursors simply restart from the first page.
- **Reads:** waitpoint and snapshot lookups that are keyed by a single
run now read only the database that holds that run instead of querying
both, removing redundant queries on hot paths (unblock, snapshot reads).
- **Writes:** environment-scoped writes with no owning run (standalone
wait tokens, waitpoint tags, idempotency-key resets) now land in the
same database as that environment's runs, rather than defaulting to the
other one. An idempotency-key reset also falls back to the other
database when it matches nothing, so a reset still clears the key
wherever the run actually lives.
## Notes
Verified end-to-end against multi-database setups: run-keyed reads and
env-scoped writes land on the correct database with no cross-database
writes, and the batches list surfaces every batch in creation order. New
tests cover the batches ordering/reachability and the write-residency
routing.
## Problem
The webapp's HTML references content-hashed `/build` assets, and each
running
instance contains exactly one build and returns 404 for asset hashes it
doesn't
have. During a rolling deploy a client can hold HTML from one build
while a
request for one of its assets is served by an instance on a different
build →
missing styles or a failed chunk load.
## What this does
On a `/build` stylesheet/script/chunk load failure, the client does a
**bounded
full-document reload** (at most 2 per 5 minutes, tracked in
`sessionStorage`) so
the page reloads onto a single consistent build. That's the whole
mechanism — no
polling, no `fetch` interception, no blocking overlay, no form
snapshotting.
- `apps/webapp/app/components/StaleAssetRecovery.tsx` — authored as a
typed,
lint-checked function and serialized to an inline script via
`.toString()` (so
the logic is real, reviewable code, not an opaque string), injected
before
`<Links />`, production only.
- Detection: capture-phase `error` listener for
`<link>`/`<script>`/modulepreload
failures under `/build/`, plus an `unhandledrejection` guard for
dynamic-import
failures.
- Guards: once-per-page re-entrancy guard, the bounded reload budget,
and a
`navigator.onLine` check so it never reloads into an offline error page.
- Unit tests in `StaleAssetRecovery.test.ts`.
## Relationship to #4260
Replaces the recovery introduced in #4260 (reverted in #4280) with a
much
smaller, reload-only approach — the previous version intercepted `fetch`
and
could turn a data request into a navigation, and showed a full-screen
overlay on
any asset error; this drops both.
## `/build-version` compatibility shim
`apps/webapp/server.ts` adds a tiny `GET /build-version` endpoint (build
id only,
`no-store`). A previously-deployed client build polls it after an asset
failure
and reloads once it sees a newer build, so those older tabs recover in
one reload
instead of getting stuck. Temporary — safe to remove once older clients
have
cycled out. It deliberately does **not** re-add an `X-Build-Id` response
header.
## Also
Restores the `.server-changes` writing guidance in
`.claude/rules/server-apps.md`
(reverted alongside #4260).
## Self-hosting note
Recovery is most reliable when your load balancer keeps a client on one
instance
for the duration of a deploy (short session stickiness) — the reload
then lands
on a consistent build in one hop.
Removes the deprecated realtime stream write action kept for retired v3
task clients. Supported clients use the targeted stream write routes,
while the existing stream read loader remains unchanged.
Standard `git revert` of #4260.
Its client-side stale-asset recovery is net-negative during normal
deploys:
- The `fetch` interception treats any `?_data=` request (Remix loader
**and** action traffic) as a navigation and, on a build-id mismatch,
`location.assign`es the tab to the fetched URL — an open dashboard tab
can be hard-navigated to a raw data URL during a rolling deploy, losing
unsaved input.
- Any transient `/build` asset error (a network blip, an extension, an
unrelated failed dynamic import) blanks the page behind a full-screen
overlay for ~60s before offering a manual reload.
- It serialized form field values to `sessionStorage` to restore them
across the reload.
This returns the webapp to the pre-#4260 baseline as a fast, low-risk
step.
Follow-ups (separate PRs):
- a minimal reload-only recovery to replace this,
- restore the unrelated `.claude/rules/server-apps.md` docs tidy-up from
#4260 (via cherry-pick),
- a load-balancer stickiness change addressing the root cause.
## Summary
Adds experimental Node.js 24 and 26 task runtimes through the
`experimental-node-24` and `experimental-node-26` config values.
Existing runtime defaults and the `node`, `node-22`, and `bun` behavior
remain unchanged. The unprefixed `node-24` and `node-26` config values
remain unavailable until the runtimes are ready for general use.
## Design
Experimental config values normalize to canonical runtime identifiers
before build manifests are created, keeping deployment metadata and
execution behavior consistent. Kubernetes task pods also use the
runtime-default seccomp profile so modern Node.js versions fall back
from io_uring to checkpoint-compatible system calls.
## Summary
`batchTrigger` requests that set a per-item `idempotencyKey` failed with
a 500 when the run-store is split across databases: the per-item
idempotency lookup errored before any run was created. Batches without
per-item keys, single `trigger` idempotency, and batch-level
(`idempotency-key` header) idempotency were unaffected.
## Root cause
`findRunsByIdempotencyKeys` built its `UNION ALL` of per-key
point-lookups with `@trigger.dev/database`'s `Prisma.sql` /
`Prisma.join`, then executed it on whichever store client it was handed.
On the dedicated run-ops store that client is a *separate* generated
Prisma client, and a `Sql` object from a different generated client is
not recognized: the bare `$queryRaw(Prisma.join(...))` form dropped the
query text entirely (`Argument \`query\` is missing`). The
tagged-template form is no better here: joining nested `Prisma.sql`
fragments across the two clients mis-numbers the bound parameters
(`syntax error at or near "$1"`).
## Fix
Build the lookup as a plain parameterized string and run it via
`$queryRawUnsafe` with positional placeholders and bound values, so it
no longer depends on which generated client executes it. The query text
contains only static SQL and integer placeholders; every value
(`runtimeEnvironmentId`, `taskIdentifier`, each key) is bound, so it is
not a raw-interpolation site. Same per-key point-lookup shape as before,
no change on the single-client path.
Verified end-to-end against a bundled build with the run-store split
enabled: before the fix, `batchTrigger` with a per-item key 500s; after,
it returns the runs and dedups correctly across fresh, repeat, and mixed
batches.
The signup → Attio sync (`attio.server.ts` `#assert`) logged every
non-2xx response at `error` level and threw the same way regardless of
status. Transient upstream failures (5xx/429) are retried by the common
worker and self-heal, so treating them as errors created false alerts
for something that isn't actually a bug.
Now `#assert` splits the two cases:
- **5xx / 429** — Logged at `warn` and thrown with `logLevel: "warn"`,
so they continue to be retried but don't raise error-level alerts. This
reuses the same pattern the worker already honors
(`directorySyncEffects`).
- **4xx** — Unchanged: logged at `error` and thrown, so genuine
integration bugs (schema, permissions, auth, etc.) remain visible.
There is no behavior change to retries or the signup flow. This is a
server-only change.
---------
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
## Summary
Adds optional configuration to send ClickHouse read traffic to a
separate instance (for example a read replica) while writes stay on the
primary `CLICKHOUSE_URL`. This lets operators offload read load (runs
list, traces, logs, queries) from the cluster that handles inserts.
Fully backwards compatible: with nothing new set, every client resolves
to `CLICKHOUSE_URL` exactly as before.
## What it adds
- `CLICKHOUSE_READER_URL` (optional): a single reader endpoint that the
read-only clients fall back to. Read clients resolve `<own URL> ??
CLICKHOUSE_READER_URL ?? CLICKHOUSE_URL`. The task-events client (which
both inserts events and reads traces, spans, and logs) is built as a
reader/writer pair so queries use the reader while inserts stay on
`CLICKHOUSE_URL`.
- `RUNS_LIST_CLICKHOUSE_URL` (optional): a dedicated client for the runs
list (dashboard list, runs list API, live reload, child-status counts),
so the highest-traffic read path can target its own instance.
## Safety
Only read-only clients fall back to the reader: logs, query, admin, runs
list, the pending-version lookup, and the realtime run-id resolver. The
query page is constrained to read-only (the TSQL parser rejects anything
that is not a `SELECT`, and a `readonly` setting is applied). The
task-events client routes inserts to the writer and queries to the
reader per method, so a write can never reach the reader. Pure-write
clients (event inserts, replication) always use `CLICKHOUSE_URL`.
Note: this PR targets a baseline branch rather than `main` so the diff
stays scoped to the read-replica changes. It will be retargeted to
`main` before merge.
---------
Co-authored-by: Eric Allam <eallam@icloud.com>
## Summary
Updates the internal development, CI, and runtime-image Node version to
24.18.0. SDK compatibility coverage continues to include Node 20, 22,
24, and 26.
The Node type definitions and the package-manager lockfiles now resolve
against Node 24 types.
## Summary
Adds a set of PAT-authenticated management API endpoints so orgs,
projects, members/invites, environment variables, and a few
project/environment settings can be managed programmatically (scripting,
automation) rather than only through the dashboard. Each route is a thin
wrapper over the **existing** service the dashboard already uses, with
the same authorization applied at the route layer - no new business
logic.
## Endpoints
**Organizations**
- `POST /api/v1/orgs` - create an org (`createOrganization`)
- `PATCH /api/v1/orgs/:orgParam` - rename (title)
- `DELETE /api/v1/orgs/:orgParam` - soft-delete
(`DeleteOrganizationService`; keeps the active-subscription guard)
**Members & invites**
- `GET /api/v1/orgs/:orgParam/members` - list members + pending invites
- `DELETE /api/v1/orgs/:orgParam/members/:memberId` - remove a member
(last-member guarded)
- `POST /api/v1/orgs/:orgParam/invites` - invite by email
(`inviteMembers`, sends the invite email)
- `DELETE /api/v1/orgs/:orgParam/invites/:inviteId` - revoke an invite
**Projects**
- `PATCH /api/v1/projects/:projectRef` - rename
(`ProjectSettingsService`)
- `DELETE /api/v1/projects/:projectRef` - soft-delete
(`DeleteProjectService`)
- `PUT /api/v1/projects/:projectRef/default-region` - set the default
region by worker-group name (`SetDefaultRegionService`)
- project GET/list now return `defaultRegion` (worker-group name, or
null when unset)
**Environments**
- `POST /api/v1/projects/:projectRef/:env/pause` and `/resume`
(`PauseEnvironmentService`)
- `POST /api/v1/projects/:projectRef/:env/regenerate-api-key` - rotate
the env secret key (`regenerateApiKey`, RBAC `write:apiKeys`)
- env var create now accepts an optional `isSecret` flag
## Auth & authorization
- All routes authenticate with a **Personal Access Token**
(`Authorization: Bearer tr_pat_...`).
- Org/project routes are built on the PAT route builders in
`apiBuilder.server.ts`: `createLoaderPATApiRoute` (already existed) and
**`createActionPATApiRoute`** (added here - the loader builder had no
mutation counterpart). The builder runs auth, resolves the org/project
role-floor via `context`, and enforces a declarative `authorization`
block using the same RBAC actions the dashboard applies
(`manage:organization` / `read:members` / `manage:members` /
`manage:project`). Handlers keep a membership-scoped query as the floor,
so a non-member gets a 404. This also gives these routes `tenantContext`
user attribution (Sentry) and `ServiceValidationError`-to-status mapping
for free.
- **Membership floor (important).** The OSS RBAC fallback grants a
permissive ability, so `ability.can(...)` can't reject a non-member on
self-hosted. Every handler therefore resolves the target scoped to the
caller's membership (`members: { some: { userId } }`) → 404 for
non-members. `authorization` is the *role* gate; this is the *tenant*
gate. `resolveOrganizationForApiUser`
(`organizationApiAccess.server.ts`) is the org-tier version of the
existing `findProjectByRef` - org-addressed PAT routes are new, so no
such helper existed before.
- Env-tier routes reuse the existing `authorizePatEnvironmentAccess`
(`write:apiKeys`).
### What `createActionPATApiRoute` gives you
A route is pure declaration - the builder handles auth, RBAC,
validation, tracing, and error mapping:
```ts
export const action = createActionPATApiRoute(
{
method: "PUT", // one verb, or ["PATCH", "DELETE"] for multi-verb routes
params: ParamsSchema,
body: SetDefaultRegionRequestBody, // zod-validated
context: async ({ projectRef }) => { // resolve the org for the RBAC role-floor
const project = await prisma.project.findFirst({
where: { externalRef: projectRef, deletedAt: null },
select: { organizationId: true },
});
return project ? { organizationId: project.organizationId } : {};
},
authorization: { action: "manage", resource: () => ({ type: "project" }) },
},
async ({ params, body, authentication, ability }) => {
// auth + authz already enforced. Just do the work.
// `throw new ServiceValidationError("Region not found", 400)` → mapped to that status.
return json({ ok: true });
}
);
```
Handled for you, so handlers stay thin:
- **Method allowlist** - `method` accepts a verb or an array; any other
verb → `405` with an `Allow` header, *before* auth runs:
```ts
const allowedMethods = method ? (Array.isArray(method) ? method :
[method]) : undefined;
if (allowedMethods && !(allowedMethods as
string[]).includes(request.method.toUpperCase())) {
return json({ error: "Method not allowed" }, { status: 405, headers: {
Allow: allowedMethods.join(", ") } });
}
```
- **PAT / user-actor auth** in a single roundtrip → `401` on
missing/invalid/revoked token.
- **RBAC** - `context` computes the caller's role-floor for the target
org/project; `authorization` gates it → `403` with a structured error
body.
- **Sentry attribution** - `tenantContext.enrich({ userId })` so events
from the handler carry the acting user.
- **Typed errors** - a thrown `ServiceValidationError` is mapped to its
`.status` (default 400); anything else → `500`, and expected boundary
errors are logged as `warn` (kept out of Sentry).
- **Validation** - params / query / headers / body are all zod-checked →
`400` with details.
## Notes for reviewers
- Everything wraps an existing service; the intent is API parity for
things that are currently dashboard-only, not new behaviour.
- `createActionPATApiRoute` is new shared infra (the PAT + RBAC mutation
builder that didn't exist). It's self-contained - the loader builder and
existing routes are untouched.
- `@trigger.dev/core` gets one additive field (`defaultRegion` on the
project response, optional/nullable for client-server version skew) -
changeset included, patch.
- `removeTeamMember`'s last-member guard is now atomic (Serializable
transaction via the `$transaction` helper, with retry), so the dashboard
and API both get it server-side. Added a `## Transactions` rule to
`apps/webapp/CLAUDE.md` (always use the `$transaction` helper);
migrating the remaining direct usages is tracked in TRI-11698.
## Open questions
- ~~Is PAT the right auth (vs OAT for automation)?~~ **Resolved: PAT.**
Organization Access Tokens are currently internal-only (used by the
image builder) and not user-accessible, so they can't back this yet.
- Should any of these be gated behind a flag or scope?
- Naming/shape of the routes.
## Summary
Batch triggers that use per-item idempotency keys could take seconds
instead of milliseconds when the target task had a large run history.
This keeps the idempotency lookup fast regardless of how many runs a
task has accumulated.
## Root cause
The batch path checks which items already have runs by looking up their
idempotency keys with a single `WHERE runtimeEnvironmentId = ? AND
taskIdentifier = ? AND idempotencyKey IN (...)` query. On a very large
`TaskRun` table Postgres underestimates the row count of a specific
`(environment, task)` pair, so once the `IN` list grows past a handful
of keys it stops doing per-key index probes and instead scans every run
for that `(environment, task)` and filters the keys in memory. The cost
is then flat and large regardless of how many keys are being checked,
and a routine `ANALYZE` does not correct the estimate at that table
size.
## Fix
Look each idempotency key up on its own, batched into a `UNION ALL` of
point lookups (chunked, run with bounded concurrency). Each branch is an
equality on all three columns of the unique index, so the planner can
only do a per-key index probe and can never fall back to the range scan.
Same results, same columns, confined to the batch trigger path.
### Problem
Webapp HTML references content-hashed /build assets, and each Docker
image contains exactly one build with a hard 404 for unknown hashes.
During a rolling deploy, a client holding HTML from the old build may
request old asset hashes from a replica running the new image, causing
missing styles or failed chunk loads.
The page should recover automatically once a compatible build becomes
available, without reload loops or unnecessary interruptions during
normal deployments.
### What changed
- Build changes alone do nothing — no polling, no automatic reloads.
- If a CSS or JavaScript asset fails to load, a recovery overlay is
shown immediately.
- While the server still reports the same build, the client polls for a
newer build using exponential backoff (up to ~60s). As soon as a newer
build is detected, the page reloads automatically.
- If no newer build appears within the timeout, recovery falls back to a
manual Reload action.
- If recovery still fails after the automatic reload, the client stops
retrying and displays a final recovery screen instead of entering a
reload loop.
- Recovery preserves form values and scroll position across the
automatic reload.
Updates the ClickHouse chat agent example page to match the upgraded
example (triggerdotdev/examples#124), which is now a fullstack
generative-UI chat app rather than an agent-only project.
## What changed
- **Overview / tech stack / features** rewritten: Next.js chat app
(`useChat` + `useTriggerChatTransport`, no API route), a
`renderVisualization` tool taking json-render specs rendered with
`@json-render/shadcn` + shadcn charts (Recharts) + mapcn point maps, and
a shared catalog that generates both the system-prompt component
reference and tool-call validation.
- **The agent section** now shows the versioned [AI
Prompt](https://trigger.dev/docs/ai/prompts) pattern (`prompts.define()`
+ `chat.prompt.set()` + `chat.toStreamTextOptions({ registry })`), with
a warning that `experimental_telemetry` comes from the stored prompt —
the docs previously showed a static `system:` string, which silently
ships no LLM observability.
- **New sections** for the shared catalog, the `renderVisualization`
tool, the Next.js chat UI and registry.
- **Relevant code links** updated to the new `src/` layout.
- **Learn more** cards now include Frontend and AI Prompts.
Note: merge after triggerdotdev/examples#124 lands, so the GitHub file
links resolve.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
## Summary
When the run-ops split is enabled, the legacy run-ops database client
was always constructed as its own connection pool, even when it points
at the same database as the primary (control-plane) client. On setups
where those two DSNs resolve to the same physical database, this opened
a second, redundant pool and doubled the number of connections used
against that database. This change makes the legacy client reuse the
primary client's pool whenever their DSNs point at the same database,
and only open a separate pool when they genuinely differ.
## Fix
A small `sameDatabaseTarget` comparison (host, port, database name,
user) decides whether the legacy DSN points at the same database as the
primary. When it does, the legacy handle reuses the primary client by
reference, so no second pool is opened. When the DSNs diverge, the
legacy client is built independently as before, so the split still works
once the databases are actually separate.
Two smaller changes ride along:
- An optional per-pool limit for the run-ops read replica, which
connects unpooled and so draws raw backend connections; unset, it falls
back to the existing default and behaviour is unchanged.
- A startup warning about a missing legacy replica URL is now suppressed
when the legacy client shares the primary pool, where it would be
misleading.
## Verification
Booted the webapp end-to-end in three modes and confirmed the pools
opened as expected via the client's own startup logs and live backend
connection counts: split off (single pool), split on with a shared
database (legacy reuses the primary pool, no doubling), and split on
with separate databases (legacy opens its own pool).
## Summary
2 new features, 11 improvements, 5 bug fixes.
## Breaking changes
- Trigger.dev v3 is no longer supported. For self-hosted deployments,
4.5.0 is the last version we officially support for running v3; stay on
4.5.0 or upgrade to v4. v3 triggers, batch triggers, reschedules, and
deploys now return a clear upgrade message instead of running.
([#4236](https://github.com/triggerdotdev/trigger.dev/pull/4236))
## Improvements
- You can now mark environment variables synced via the `syncEnvVars`
build extension as secrets. Return `{ name, value, isSecret: true }`
from your callback and those variables are stored redacted in the
dashboard, just like manually created secret env vars.
([#4203](https://github.com/triggerdotdev/trigger.dev/pull/4203))
- Remove the legacy `--mcp` and `--mcp-port` options from the `dev`
command. Run the dedicated `trigger mcp` command to start the
Trigger.dev MCP server.
([#4246](https://github.com/triggerdotdev/trigger.dev/pull/4246))
- Removed the unused `ResourceMonitor` export from
`@trigger.dev/core/v3/serverOnly`. It was a server-side logging helper
with no remaining consumers.
([#4244](https://github.com/triggerdotdev/trigger.dev/pull/4244))
- Removed the unused `@trigger.dev/core/v3/zodNamespace` export and the
legacy v3 socket message schemas. These were only used by the
now-retired v3 engine and have no v4 consumers.
([#4236](https://github.com/triggerdotdev/trigger.dev/pull/4236))
## Bug fixes
- Fix a `chat.agent` message-loss race where sending a message right
after an action (such as an undo) could drop the follow-up's response
from the UI until a refresh.
([#4234](https://github.com/triggerdotdev/trigger.dev/pull/4234))
## Server changes
These changes affect the self-hosted Docker image and Trigger.dev Cloud:
- Added `EVENT_REPOSITORY_POSTGRES_WRITES_DISABLED` to skip all
PostgreSQL task-event writes for deployments that store task events in
ClickHouse. Leave it off unless `EVENT_REPOSITORY_DEFAULT_STORE` is
`clickhouse_v2`, otherwise task events are lost.
([#4242](https://github.com/triggerdotdev/trigger.dev/pull/4242))
- Promo credits: a /promo signup landing page, redeeming a promo code
when a new org selects a plan, and showing remaining credits on the
usage page.
([#4138](https://github.com/triggerdotdev/trigger.dev/pull/4138))
- Speed up retrieving a background worker by version. The endpoint no
longer runs a slow lookup that scanned the full task table for large
deployments; it now reuses data it already loads, so the response is the
same but returns much faster.
([#4245](https://github.com/triggerdotdev/trigger.dev/pull/4245))
- Clearer login error when an email address is blocked by the
WHITELISTED_EMAILS setting: the message now explains the address isn't
allowed on this instance instead of the ambiguous "This email is
unauthorized".
([#4220](https://github.com/triggerdotdev/trigger.dev/pull/4220))
- Make the native build server the default in project build settings.
It's now opt-out, stored as a new `disableNativeBuildServer` key. Also
clarifies in the UI that build settings apply to GitHub-triggered and
native build server deployments.
([#3980](https://github.com/triggerdotdev/trigger.dev/pull/3980))
- Optionally process high-volume telemetry ingestion in parallel for
higher throughput under heavy load by setting
`OTEL_TRANSFORM_WORKER_POOL_ENABLED=1`. Off by default.
([#4232](https://github.com/triggerdotdev/trigger.dev/pull/4232))
- Add a `REALTIME_BACKEND_DEFAULT` env var to choose the default
realtime backend (`electric`, `native`, or `shadow`) for environments
whose org has no per-org override. Defaults to `electric`, so existing
behavior is unchanged.
([#4231](https://github.com/triggerdotdev/trigger.dev/pull/4231))
- Clarified on the Regions page that a region only affects where your
runs execute, not where your data is stored. This shows as a tooltip on
the Location column and in the confirmation dialog when you change your
default region.
([#4226](https://github.com/triggerdotdev/trigger.dev/pull/4226))
- Improved the reliability of how run data is read and written.
([#4237](https://github.com/triggerdotdev/trigger.dev/pull/4237))
- Fixed stale login errors: an error from a previous login attempt (for
example a rejected email address) no longer keeps reappearing on the
login page and no longer makes later, successful attempts look like they
failed.
([#4220](https://github.com/triggerdotdev/trigger.dev/pull/4220))
- The Errors page now shows better details for each error. Errors that
don't carry a message — such as errors thrown without a message, or
values thrown that aren't `Error` objects — get a meaningful title
instead of all reading "Unknown error", and are grouped by their name
(or value) rather than collapsed into a single group. The error type now
shows the actual error name, and stack traces now appear where
previously they were missing.
([#4225](https://github.com/triggerdotdev/trigger.dev/pull/4225))
- Return a clear client error when SSO form submissions use an
unsupported content type
([#4238](https://github.com/triggerdotdev/trigger.dev/pull/4238))
- Query page: extracting fields from a run's output with JSON functions
(such as JSONExtractString or JSONExtractInt) no longer fails with an
"illegal type: JSON" error.
([#4221](https://github.com/triggerdotdev/trigger.dev/pull/4221))
<details>
<summary>Raw changeset output</summary>
# Releases
## @trigger.dev/build@4.5.4
### Patch Changes
- You can now mark environment variables synced via the `syncEnvVars`
build extension as secrets. Return `{ name, value, isSecret: true }`
from your callback and those variables are stored redacted in the
dashboard, just like manually created secret env vars.
([#4203](https://github.com/triggerdotdev/trigger.dev/pull/4203))
- Updated dependencies:
- `@trigger.dev/core@4.5.4`
## trigger.dev@4.5.4
### Patch Changes
- Remove the legacy `--mcp` and `--mcp-port` options from the `dev`
command. Run the dedicated `trigger mcp` command to start the
Trigger.dev MCP server.
([#4246](https://github.com/triggerdotdev/trigger.dev/pull/4246))
- Updated dependencies:
- `@trigger.dev/core@4.5.4`
- `@trigger.dev/build@4.5.4`
- `@trigger.dev/schema-to-json@4.5.4`
## @trigger.dev/core@4.5.4
### Patch Changes
- Removed the unused `ResourceMonitor` export from
`@trigger.dev/core/v3/serverOnly`. It was a server-side logging helper
with no remaining consumers.
([#4244](https://github.com/triggerdotdev/trigger.dev/pull/4244))
- Removed the unused `@trigger.dev/core/v3/zodNamespace` export and the
legacy v3 socket message schemas. These were only used by the
now-retired v3 engine and have no v4 consumers.
([#4236](https://github.com/triggerdotdev/trigger.dev/pull/4236))
## @trigger.dev/python@4.5.4
### Patch Changes
- Updated dependencies:
- `@trigger.dev/sdk@4.5.4`
- `@trigger.dev/core@4.5.4`
- `@trigger.dev/build@4.5.4`
## @trigger.dev/react-hooks@4.5.4
### Patch Changes
- Updated dependencies:
- `@trigger.dev/core@4.5.4`
## @trigger.dev/redis-worker@4.5.4
### Patch Changes
- Updated dependencies:
- `@trigger.dev/core@4.5.4`
## @trigger.dev/rsc@4.5.4
### Patch Changes
- Updated dependencies:
- `@trigger.dev/core@4.5.4`
## @trigger.dev/schema-to-json@4.5.4
### Patch Changes
- Updated dependencies:
- `@trigger.dev/core@4.5.4`
## @trigger.dev/sdk@4.5.4
### Patch Changes
- Fix a `chat.agent` message-loss race where sending a message right
after an action (such as an undo) could drop the follow-up's response
from the UI until a refresh.
([#4234](https://github.com/triggerdotdev/trigger.dev/pull/4234))
- Updated dependencies:
- `@trigger.dev/core@4.5.4`
</details>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
The dashboard was redesigned and two pages moved, but the docs still
described the old sidebar:
- **Schedules** no longer has its own sidebar page — schedules are
managed from the **Tasks** page (open a scheduled task to create / view
/ edit / enable-disable / delete them).
- The standalone list-based **Test** page is deprecated — you test a
task from its own **Test** button now.
## Changes
- `tasks/scheduled.mdx`: rewrote the "attaching schedules" and "testing
schedules" sections for the Tasks-based flow, added a "managing
schedules in the dashboard" section, and added explicit callouts noting
both pages moved (so readers — and search — aren't pointed at a page
that no longer exists). Re-shot the four schedule screenshots and fixed
a mislabeled alt text.
- `run-tests.mdx`, `snippets/step-run-test.mdx`,
`guides/examples/sentry-error-tracking.mdx`: replaced "select the Test
page in the sidebar" with the task-first flow plus a callout, and
refreshed `test-dashboard.png`.
TRI-11939
The `ResourceMonitor` server-side logging helper is no longer used. It
periodically logged the webapp process own memory, disk, and CPU usage
behind the `RESOURCE_MONITOR_ENABLED` flag (off by default), and was
also exported from `@trigger.dev/core/v3/serverOnly` with no other
consumers.
This removes the helper, its `@trigger.dev/core` export, the webapp
wiring, and the `RESOURCE_MONITOR_ENABLED` env var. The supervisor has
its own unrelated `ResourceMonitor` class, which is left untouched.
## Summary
Adds `EVENT_REPOSITORY_POSTGRES_WRITES_DISABLED` (default off), which
makes the task-event store skip all PostgreSQL `TaskEvent` writes. It's
for deployments that store task events in ClickHouse
(`EVENT_REPOSITORY_DEFAULT_STORE=clickhouse_v2`) and no longer want the
PostgreSQL copy.
## How it works
The guard sits at the single postgres write boundary,
`TaskEventStore.create` / `createMany`, so it covers every write path
(OTLP ingestion and run-lifecycle events) with one check. Reads are
untouched (`findMany` / trace queries / streaming), so existing
PostgreSQL events remain readable.
Leave it off unless the default store is `clickhouse_v2`, otherwise task
events for any run still routed to PostgreSQL would be dropped.
## What
The `GET
/api/v1/projects/:projectRef/background-workers/:envSlug/:version`
endpoint loaded each file's tasks through the nested `files.tasks`
relation. Prisma resolves that as a separate query:
```sql
SELECT id, slug, "fileId" FROM "BackgroundWorkerTask" WHERE "fileId" IN (...)
```
`BackgroundWorkerTask.fileId` is not indexed — the FK constraint exists,
but Postgres does not auto-create an index for foreign keys — so on a
large table this can only run as a sequential scan, which gets
progressively slower as the table grows and was observed taking minutes
per call in production.
The loader already loads every task for the worker via `tasks: true`,
which uses the indexed `workerId` relation, and those rows already
include `fileId`. This PR groups task slugs by `fileId` in memory from
that already-loaded data and drops the `files.tasks` include entirely.
## Behavior change (latent bug fix)
The response shape is unchanged, but there is a semantic correction for
**source files reused across worker versions** (files are de-duplicated
by `@@unique([projectId, contentHash])`, so one file row can be linked
to many workers).
- **Before:** `file.tasks` came from the `BackgroundWorkerFile.tasks`
relation, i.e. *every* `BackgroundWorkerTask` with that `fileId` —
across all workers sharing the file. So a worker's manifest could list
tasks it doesn't actually have.
- **After:** `file.tasks` is grouped from the queried worker's own
tasks, so it reflects only that worker version's tasks.
Verified on a local DB: 460 files are referenced by tasks from more than
one worker; of 6819 (worker, file) pairs, 6 differ — all one file where
the old union leaked a task slug (`cancellation-test`) into worker
versions that never had it. The new per-worker behavior is the correct
one for a worker-version manifest. (Thanks to the automated review for
flagging this.)
## Analysis
Captured the exact SQL before/after by instrumenting Prisma against real
data (a worker with 62 files):
- **Before:** 5 statements, including the `WHERE "fileId" IN (...)`
scan.
- **After:** 4 statements; the `fileId` query is gone and the other four
are identical.
EXPLAIN of the two access paths:
```
Before WHERE "fileId" IN (...)
Seq Scan on "BackgroundWorkerTask"
Filter: ("fileId" = ANY (...)) -- reads the whole table, scales with table size
After WHERE "workerId" IN (...)
Index Scan using "BackgroundWorkerTask_workerId_slug_key"
Index Cond: ("workerId" = ...) -- bounded by matching rows, scale-independent
```
No new index is required: the `workerId` access path is already covered
by the existing `BackgroundWorkerTask_workerId_slug_key` unique index.
## Testing
- `pnpm run typecheck --filter webapp` passes.
- Query capture + EXPLAIN performed against a local database seeded with
real worker/file/task data.
Routine dependency maintenance.
- Pin a few high-fanout transitive deps to current patched versions via
`pnpm.overrides`: `form-data`, `ws`, `undici`, `hono`. Lockfile-only (no
published-package dependency changes); net shrinks via dedup.
- Upgrade `nodemailer` 8 → 9 in `internal-packages/emails` (private
package). The SES transport already uses SESv2 and the
`createTransport`/`sendMail` API is unchanged, so no code changes were
needed. `@types/nodemailer` stays at 8 (no 9.x published yet; types are
compatible).
Verified locally: `pnpm i` clean; `pnpm run typecheck --filter emails`
and `--filter webapp` both pass.
## Summary
Run-graph data (runs, batches, waitpoints, and their related tables) can
now live in a database separate from the control plane, with every read
and write routed to the correct database by each run's residency. This
makes reading and writing run data more reliable once the two are split,
and is a no-op for single-database installs.
## Design
- Run-graph table access goes through the run-store router, which
selects the legacy or the new run-ops store per run instead of assuming
one shared client.
- The legacy run-ops client is now independently pointable, so legacy
run data can be served from its own database (and replica) rather than
the control-plane connection.
- Run-graph writes go straight to the run-graph database instead of
being forwarded through the control plane, and replication targets are
split so runs in the new database still replicate to analytics without
under-counting.
- Read-through slots refuse the control-plane client, so a missing
residency fails loudly instead of silently reading the wrong database.
- Migration `20260710120000_drop_remaining_run_graph_seam_foreign_keys`
drops the foreign keys that still crossed the run-graph / control-plane
seam, which is what lets the two live in separate databases.
The split stays off unless explicitly enabled and the two databases are
confirmed physically distinct; startup fails closed otherwise.
Verified by running the full dashboard end-to-end suite against both a
single-database configuration and a three-database configuration
(control plane, the new database, and a physically separate legacy
database), with runs on both residencies. No misrouted reads in either
configuration.
## Summary
Refreshes the docs for the v3 sunset: v3 (SDK v3) is end of life, and
4.5.0 is the last version we officially support for running v3.
- The self-hosting overview, plus the docker and kubernetes
version-locking sections, now tell self-hosters on v3 to stay on 4.5.0
or migrate to v4. 4.5.1 and later reject v3 triggers and deploys with an
upgrade message.
- The migration guide's deprecation notice was still written in the
future tense (with dates that have since passed); it now describes v3 as
retired and adds the self-hosted 4.5.0 cutoff. This is the page the
server's upgrade message links to.
- Fixes a stale "v3 project" reference in the CLI overview.
The Mintlify preview will render the callouts for a visual check.
## Summary
v3 (the engine that ran the SDK v3 era, internally
`RunEngineVersion.V1`) is end-of-life. Following the removal of the v3
execution apps
([#4194](https://github.com/triggerdotdev/trigger.dev/pull/4194)) and
the legacy dev websocket
([#4198](https://github.com/triggerdotdev/trigger.dev/pull/4198)), this
removes the remaining v3 execution stack from the server.
Clients still on v3 (an old SDK or CLI that has not upgraded) keep
getting a clear "upgrade to v4" response. Triggers, batch triggers,
reschedules, and deploys that resolve to v3 are rejected with a graceful
4xx pointing at the migration guide, never a 5xx, so a stale client
cannot affect server health. Self-hosted instances still running v3
should stay on the 4.5.x release line until they migrate.
## What is removed
- The MarQS queue and its shared/dev queue consumers.
- The v3 socket.io namespaces (coordinator, provider, shared-queue) and
the v3 run lifecycle services (attempt, checkpoint, and batch-resume).
- The graphile-worker background job system; all live jobs already run
on `@trigger.dev/redis-worker`.
- The `DEPRECATE_V3_ENABLED` flag: v3 is now rejected unconditionally,
so the flag is gone.
- Unused v3 exports from `@trigger.dev/core` (the `v3/zodNamespace`
subpath and the legacy socket message catalogs) and the now-dead MarQS
environment variables.
## What stays
The v4 engine is untouched. The graceful v3 rejection boundary stays,
`determineEngineVersion` still detects a v3 project so it can reject it,
and the batch service plus batch-completion worker stay for current
clients. Live queue concurrency limits and metrics now read from the v4
run engine instead of MarQS, and a brand-new dev environment now
defaults to v4.
## Dependency cleanup
Removes webapp dependencies left unused by this change: `seedrandom` and
`semver` (only the removed v3 code used them) plus a set that was
already dead, their orphaned `@types` packages, and two dead files. Adds
a `knip:deps` script and a `knip.json` config so unused dependencies can
be found the same way going forward.
## Summary
Two threads on the run-ops split path.
Read path: per-item run reads are batched into grouped queries, a
waitpoint's connected-run reads are bounded, and the dedicated-schema
relation hydrators fetch only the requested columns instead of whole
rows. Retrieve also falls back to the other database when a routed read
misses, so a run whose physical residency diverges from its id shape is
still found rather than returning a spurious not-found. Fewer and
lighter queries on the run read path, with no change to results.
Mint-kind flip safety: flipping which database new runs mint to is now a
deterministic wall-clock cutover, for both per-org and global flips. For
a grace window every process resolves the same database, so a flip
cannot route two concurrent triggers that share an idempotency key to
different databases (which would bypass the per-database unique
constraint and create a duplicate run).
Supersedes the earlier #4205 and #4208.
Draft: validation in progress.
Switches the native build server from opt-in to opt-out in project build
settings.
- It's now enabled by default, stored as a new
\`disableNativeBuildServer\` opt-out key so previously-saved
\`useNativeBuildServer: false\` values aren't treated as deliberate
opt-outs.
- The "Use native build server" checkbox is checked by default;
unchecking it persists the opt-out.
- Brief wording: clarifies build settings apply to GitHub-triggered and
native build server deployments, and the native build server hint no
longer says "in the future".
## Summary
Under high OTLP ingest volume, the whole decode, transform, and enrich
pipeline runs on the request event loop, so a single CPU core becomes
the ceiling while the rest sit idle. This adds an opt-in worker pool
that moves decode, transform, and LLM-cost enrichment onto worker
threads, keeping the main thread free for I/O. It is off by default
(`OTEL_TRANSFORM_WORKER_POOL_ENABLED`), so behavior is unchanged unless
enabled.
## Design
Workers do decode, filter, convert, and enrich (including LLM pricing
match). The main thread stays the single database reader: it loads the
pricing registry and broadcasts the compiled model rows to the workers
(re-broadcasting on every reload), so workers never touch the database.
The pure transform is extracted into a dependency-light module (no
Prisma/Redis/ClickHouse imports) so it can run inside a worker.
Importantly, the main thread keeps the existing single consolidated
insert path, so ClickHouse insert batching and part count are unchanged.
The parallelism buys CPU headroom, not more insert streams (which would
add merge pressure).
The worker is bundled as a standalone file at build time and ships in
the existing image with no Dockerfile change. In local load testing the
pool sustained roughly 2.6x the throughput of the single-thread path and
kept the main thread responsive under load.
## Summary
Sending a chat message immediately after an action (for example an undo)
could make the message's response vanish from the UI. The transport
opened a response stream that closed on the *earlier* turn's completion
instead of waiting for the send's own turn. The agent still produced and
persisted the answer, so it reappeared on refresh. Same "disappearing
message" class as
[#4176](https://github.com/triggerdotdev/trigger.dev/pull/4176),
different cause.
## Fix
A send's response stream had no way to tell whether a `turn-complete`
belonged to its turn. `POST /realtime/v1/sessions/:id/in/append` now
returns the appended record's sequence number, and the transport skips
any turn-complete whose `session-in-event-id` (the agent's committed
`.in` cursor) is below that seq, closing only on its own turn. Older
webapps omit the seq, in which case the transport falls back to the
previous behavior, so the SDK and server can ship independently.
Because the fix spans the SDK and the server, both a webapp deploy and
an SDK release are needed for the full effect.
Verified end to end with the ai-chat reference app:
undo-then-immediate-send loses the follow-up's answer before the fix and
streams it inline after, with a revert-the-guard run reproducing the
loss on the same script. Unit tests cover the skip and the no-seq
fallback.
## Summary
The default realtime backend was hardcoded to Electric. This adds a
`REALTIME_BACKEND_DEFAULT` env var (`electric` | `native` | `shadow`,
default `electric`) that chooses the backend for any environment whose
org has no `realtimeBackend` override. Behavior is unchanged unless you
set it; per-org overrides still win.
The default is applied at every point where the per-org flag falls
through: the initial value, the flag lookup default, and the error
fallback.
## Problem
Several display/grouping issues in the **Errors** feature, all rooted in
how the ClickHouse error materialized views (`errors_mv_v1`,
`error_occurrences_mv_v1`) read the stored error JSON produced by
`parseError`:
1. **Messageless errors show "Unknown error".** An empty message falls
straight through `coalesce(nullIf(message,''), 'Unknown error')` to the
literal, even though the error's class `name` is available (e.g. an
Effect tagged error `ListMessagesError` with no message).
2. **Unrelated errors collapse into one group.**
`calculateErrorFingerprint` keys on `type : message : stack`, where
`type` is always the union tag (`BUILT_IN_ERROR`, …), `message` is
empty, and the stack isn't read — so every messageless built-in error
(and every string/custom error) hashes to the same constant input → one
fingerprint.
3. **error_type shows the internal tag.** `coalesce(type, name, …)`
always resolves to `type` (always present), so the column shows
`BUILT_IN_ERROR` instead of the real class name.
4. **Stack traces never populate.** The MVs read `error.data.stack`, but
the serializer stores the trace under `stackTrace` — so the column is
always empty.
## Fix
All display changes are `ALTER TABLE … MODIFY QUERY` on the two views
(migration `035`); the fingerprint change is in the webapp.
- **Fingerprint** (`errorFingerprinting.ts`): fall back **message → name
→ raw**. Messageless errors now group by class name (or raw value for
non-Error throws); message-bearing errors are **unchanged**
(short-circuits at `message`), so existing groups don't split — only
currently-messageless errors get their own group going forward.
- **error_message**: same `message → name → raw` fallback before
`'Unknown error'`.
- **error_type**: coalesce `name → code → 'Error'` (drops the reliance
on the union tag). Built-in → class name, internal → `code`,
string/custom → `Error`.
- **stack trace**: read `error.data.stackTrace`. Bounded as before
(serializer caps 50 frames / 1024 chars per line; MV clips to 2000
chars).
## Migration notes
- `MODIFY QUERY` swaps the view query in place (no drop/recreate gap);
Down restores the previous query.
- **Existing rows are left unchanged** — changes apply only to rows
inserted after the migration. No backfill.
## Tests
`errorFingerprinting.test.ts` — 57 pass, incl. new cases for messageless
class names, string/custom raw values, and stability of message-bearing
fingerprints.
Fixes the display-derivation half of TRI-11938 (error_type + stack
trace); relates to TRI-9254 and TRI-9250.
Applying a directory-sync effect that would demote the org's last Owner
(a
group remap, or a provision) previously threw and 500'd the settings
save. Now
rbac.setUserRole reports code:"last_owner" and applyEffect skips just
that
member (they keep Owner) while the rest of the batch applies.
Adds the machine-readable RoleAssignmentResult.code to the plugin
contract so
callers can tell the last-owner guard apart from a real failure.
## Summary
The RBAC and SSO auth plugins can own their own database client, but
they could only read `DATABASE_URL`, so every connection they opened
landed on the primary. The host webapp now resolves writer and
read-replica URLs from its env (the same fallback chain its own Prisma
clients use: control-plane URL first, then the default) and passes them
to the plugins at create time via a shared `PluginDatabaseConfig`, along
with separate connection limits for writes (default 2) and reads
(default 5, tunable via `RBAC_DATABASE_*_CONNECTION_LIMIT` and
`SSO_DATABASE_*_CONNECTION_LIMIT`).
A plugin can then route hot-path reads (per-request auth checks, login
routing) to the read replica and keep only rare mutations on the
primary. With no replica configured, or no plugin installed, nothing
changes: the OSS fallback ignores the new option and keeps reading
through the Prisma clients it is already given.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
## Summary
This adds an always-visible info tooltip on the Location column and a
note in the "set default region" confirmation dialog making it explicit.
It also removes the obsolete "V4" badge from the Regions page title.
## What & why
Signup promo credits. A new logged-out `/promo?code=<code>` landing page
validates the code and carries it through signup via a cookie. When the
new organization is activated by selecting a plan, the code is redeemed
and its credits are applied; the usage page then shows the remaining
promo credits and their expiry.
## Notes
- The code is redeemed at **plan selection**, not org creation: the
credit grant targets the org's usage allowance, which only exists once a
plan is selected — applying at creation would have nothing to grant
onto. Redemption is best-effort and never blocks plan selection.
- Pairs with the corresponding billing-service change (promo code
validate/apply/credits + grant issuance); the two are released together.
## Testing
Verified locally end to end: `/promo` shows the offer, a new account
carries the code through signup, selecting the Free plan redeems it, and
the usage page shows the remaining credits.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
## What
Adds per-variable secret support to the `syncEnvVars` build extension.
Return `{ name, value, isSecret: true }` and the variable is stored as a
secret (redacted in the dashboard, value non-revealable), just like a
manually created secret env var. Secret and non-secret variables can be
mixed in one callback.
```ts
syncEnvVars(async () => [
{ name: "PUBLIC_API_URL", value: "https://api.example.com" },
{ name: "DATABASE_URL", value: "postgres://...", isSecret: true },
]);
```
## How
Env vars flow through the build pipeline as a flat name→value map, and
the import API's `isSecret` is per-call. So secret vars are carried
through the layer + manifest in parallel `secretEnv` / `secretParentEnv`
maps, and at deploy time they go up in a second `importEnvVars` call
with `isSecret: true` (the plain vars in the first call). The record
form (`{ KEY: "value" }`) is unchanged and stays non-secret.
## Commits
- `feat(core)`: carry secret env vars through the build layer + manifest
schema
- `feat(build)`: partition `isSecret` vars in `syncEnvVars`
- `feat(cli)`: merge secret layers and import them with `isSecret: true`
at deploy
- `test(build)`: cover the partitioning + document `isSecret`
## Testing
- vitest covers the partitioning (secret/non-secret × child/parent) and
that the record form stays non-secret.
- Verified against a local webapp that the deploy's import contract
stores the secret var redacted (`isSecret: true`) and the plain var
visible.
Closes TRI-11099
Creating a Prisma migration now formats its schema first, keeping
migration-related schema edits consistently formatted without adding
work to the repository-wide format command. Run `pnpm run format:prisma`
to format either schema on demand.
## Summary
A Query page (TRQL) query that pulls fields out of a run's `output` with
JSON functions (`JSONExtractString`, `JSONExtractInt`, `JSONHas`, and
the rest of the family) failed with "The first argument of function ...
should be a string containing JSON, illegal type: JSON". Those queries
now work.
## Root cause and fix
`output` is a native ClickHouse `JSON` column, but `JSONExtract*`,
`JSONHas`, `JSONLength`, and `JSONType` all expect a String containing
JSON text. The compiler already swaps in the column's String companion
(`output_text`) when a JSON column is selected or compared, but not
inside function-call arguments, so it emitted `JSONExtractInt(output,
'x')` against the native column.
The fix prints the companion column for the first argument of these
functions when it resolves to a bare JSON field, keeping the table alias
when qualified (so it works in JOINs):
JSONExtractInt(output, 'x') -> JSONExtractInt(output_text, 'x')
JSONExtractArrayRaw(assumeNotNull(output), 'y') ->
JSONExtractArrayRaw(assumeNotNull(output_text), 'y')
It also reaches through value-preserving passthrough wrappers like
`assumeNotNull(...)`, while leaving value-changing wrappers like
`toJSONString(output)` on the native column (that argument is already a
String). The swap is also semantically correct, not just a type fix:
`output_text` is the unwrapped data JSON that the TRQL `output` model
already represents, so field paths line up.
Covered by printer unit tests and a ClickHouse integration test that
runs the whole family (plus the wrapped and `toJSONString` cases)
against a real native-JSON column. Both new cases fail with the exact
"illegal type: JSON" error without the fix.
## ✅ Checklist
- [x] I have followed every step in the [contributing
guide](https://github.com/triggerdotdev/trigger.dev/blob/main/CONTRIBUTING.md)
- [x] The PR title follows the convention.
- [x] I ran and tested the code works
---
## Testing
- Verified the rule emits exactly five errors for un-awaited throws of
the known
async redirect helpers while ignoring awaited throws, returned promises,
and
synchronous `redirect(...)`.
- Verified `--fix` inserts `await` in async functions and produces a
clean
second lint run.
- Verified synchronous functions remain diagnostic-only so autofix
cannot
introduce invalid syntax.
- Ran `pnpm run format`, `pnpm run lint`,
`pnpm run typecheck --filter webapp`, and `git diff --check`.
---
## Changelog
Adds an Oxlint rule that prevents async redirect helpers from being
thrown
without awaiting their `Response`. Existing violations are fixed, the
autofix
is limited to async functions, and the plugin uses an explicit ESM
extension.
---
## Screenshots
See the test-results comment for CLI evidence.
💯
Link to Devin session:
https://app.devin.ai/sessions/e60ad7610773401da3d3040cf1252337
Requested by: @ericallam
---------
Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: Eric Allam <eric@trigger.dev>
- Combine SDK and dashboard bulk actions docs
- Fix API reference pages for bulk actions
- Fix weird rendering on bulk actions page
## Todo
- [ ] not sure about having the SDK+dashboard combined and under "Using
the dashboard"... need to find the right place
## Summary
The limits page didn't spell out that large payloads offload to object
storage automatically, and its single "512KB" note conflated two
different thresholds. This clarifies the behaviour.
On the way in, the SDK uploads any trigger or batch-item payload over
128KB to object storage before sending, so large triggers and batches
don't hit the request body limit (`trigger` / `triggerAndWait` since
4.5.0, `batchTrigger` / `batchTriggerAndWait` since 4.5.2). On
retrieval, payloads and outputs over 512KB are stored in object storage
and returned as a presigned URL from `runs.retrieve`.
## Summary
Magic link login could appear completely broken: submitting your email
on the login page showed a stale "This email is unauthorized" error
instead of the "we've sent you a magic link" confirmation, even when the
address was fine.
This PR reverts
[#4215](https://github.com/triggerdotdev/trigger.dev/pull/4215) (whose
diagnosis and fix turned out to be wrong) and fixes the actual bug,
which was in how login errors are stored and consumed.
## Root cause
Two session bugs compounded on the login page:
- The `/login` loader read the flashed `auth:error` without committing
the session. A Remix flash is only consumed when the session is
committed after the read, so once any attempt flashed an error (for
example an address rejected on an instance with `WHITELISTED_EMAILS`
set), it stayed in the session cookie and reappeared on every later
`/login` visit, making successful attempts look like failures.
- The `/login/magic` action stored its validation and rate limit errors
with `session.set`, which survives every later read and commit, so those
errors stuck permanently.
[#4215](https://github.com/triggerdotdev/trigger.dev/pull/4215) had
instead diagnosed a server-only module leaking into the client bundle
and crashing navigation. Checking the shipped images' client bundles via
their sourcemaps shows `.server` modules were always stubbed out, so
that change fixed nothing and is reverted here.
## Fix
- `/login` reads the flashed error and commits the session when one was
present, so an error renders once and clears. The `redirectTo` branch
now surfaces the error too instead of leaving it in the cookie.
- `/login/magic` flashes its errors instead of `set`ting them.
Verified end-to-end on a live preview environment: a rejected address
shows the error once and a reload clears it; a valid address lands on
the confirmation screen with the address named; GitHub, Google, and SSO
login paths are untouched by this diff.
## Summary
1 improvement, 2 bug fixes.
## Breaking changes
- Removed support for the end-of-life v3 `trigger dev` CLI. Starting a
dev session with an old v3 CLI now returns an upgrade message instead of
connecting - upgrade to the v4 CLI to continue using `trigger dev`.
([#4198](https://github.com/triggerdotdev/trigger.dev/pull/4198))
## Bug fixes
- Fix TS2742 ("inferred type cannot be named") when exporting a
`chat.agent` from a project with declaration emit: `ChatTaskWirePayload`
and `ChatInputChunk` are now declared in the public
`@trigger.dev/sdk/chat` subpath, so inferred agent types emit portable
declarations and the wire types are directly importable.
([#4218](https://github.com/triggerdotdev/trigger.dev/pull/4218))
## Server changes
These changes affect the self-hosted Docker image and Trigger.dev Cloud:
- Reduce primary database load on the runs page by serving its
empty-state check from ClickHouse instead of Postgres.
([#4202](https://github.com/triggerdotdev/trigger.dev/pull/4202))
- Fixed submitting your email on the login page reloading back to an
empty form instead of showing the magic link confirmation screen.
([#4215](https://github.com/triggerdotdev/trigger.dev/pull/4215))
<details>
<summary>Raw changeset output</summary>
# Releases
## @trigger.dev/build@4.5.3
### Patch Changes
- Updated dependencies:
- `@trigger.dev/core@4.5.3`
## trigger.dev@4.5.3
### Patch Changes
- Updated dependencies:
- `@trigger.dev/build@4.5.3`
- `@trigger.dev/core@4.5.3`
- `@trigger.dev/schema-to-json@4.5.3`
## @trigger.dev/python@4.5.3
### Patch Changes
- Updated dependencies:
- `@trigger.dev/sdk@4.5.3`
- `@trigger.dev/build@4.5.3`
- `@trigger.dev/core@4.5.3`
## @trigger.dev/react-hooks@4.5.3
### Patch Changes
- Updated dependencies:
- `@trigger.dev/core@4.5.3`
## @trigger.dev/redis-worker@4.5.3
### Patch Changes
- Updated dependencies:
- `@trigger.dev/core@4.5.3`
## @trigger.dev/rsc@4.5.3
### Patch Changes
- Updated dependencies:
- `@trigger.dev/core@4.5.3`
## @trigger.dev/schema-to-json@4.5.3
### Patch Changes
- Updated dependencies:
- `@trigger.dev/core@4.5.3`
## @trigger.dev/sdk@4.5.3
### Patch Changes
- Fix TS2742 ("inferred type cannot be named") when exporting a
`chat.agent` from a project with declaration emit: `ChatTaskWirePayload`
and `ChatInputChunk` are now declared in the public
`@trigger.dev/sdk/chat` subpath, so inferred agent types emit portable
declarations and the wire types are directly importable.
([#4218](https://github.com/triggerdotdev/trigger.dev/pull/4218))
- Updated dependencies:
- `@trigger.dev/core@4.5.3`
## @trigger.dev/core@4.5.3
</details>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
## Summary
Exporting a `chat.agent` from a project with `declaration: true` failed
with TS2742: the inferred type of the agent references
`ChatTaskWirePayload`, which was declared in an internal module not
reachable through the package exports map, so tsc could only name it via
a file path into `node_modules` and refused to emit. Consumers had to
hand-mirror the wire type and annotate their export.
## Fix
`ChatTaskWirePayload` and `ChatInputChunk` are now declared in
`@trigger.dev/sdk/chat` (a public subpath) and re-exported type-only
from the internal shared module, so every internal import is unchanged
and the browser/server module split is untouched. Declaration emit for
an inferred agent type now produces a portable specifier:
```ts
export declare const chatAgent: Task<"chat-agent", import("@trigger.dev/sdk/chat").ChatTaskWirePayload<MyUIMessage, MyClientData>, unknown>;
```
As a side effect the wire types are now directly importable, which is
what affected users were reconstructing by hand.
## Verification
Reproduced against the built 4.5.2-equivalent package: a consumer
fixture with declaration emit produced `import("<file
path>/ai-shared.js")` in its declaration (the TS2742 trigger); after the
fix the same fixture emits the public specifier with zero diagnostics. A
regression test now builds that consumer simulation in a temp directory
on every test run: it copies the built package into a fake node_modules
(copied, not symlinked, because tsc only applies exports-map naming to
real node_modules paths), compiles the fixture with the TypeScript API,
and asserts no errors, no relative-path imports, and no internal module
references in the emit.
## Summary
Submitting your email on the login page could reload back to an empty
login form instead of showing the "we've sent you a magic link"
confirmation. The magic link email was still sent, so it looked like
nothing happened.
## Root cause
The `/login/magic` route imported a server-only cookie module
(`magicLinkEmailCookie.server.ts`) whose top-level `env.NODE_ENV` read
got bundled into the route's client JS. On the client `env` is
undefined, so the module threw a `TypeError` at module eval, which
aborted Remix's client-side navigation to the confirmation and
hard-reloaded back to `/login`. It only surfaced in production builds
(local dev auto-logs-in, and local prod builds happen to tree-shake the
module out), which is why it slipped through.
## Fix
The email-link strategy already stores the submitted address in the
session (`auth:email`), so the separate cookie was redundant. Deleted
the cookie module and read the address from the session in the loader.
With the module gone, nothing server-only can leak into the client
bundle regardless of tree-shaking.
Verified the confirmation renders with the email address, the SSO
domain-policy redirect (with the email prefilled) still works, and a
production build no longer bundles the module.
Condense the kept rationale comments (logLevel/warn, last-Owner dedup,
role
overwrite) and drop the obvious function-header comments that just
restated
the code. No behavior change.
## Summary
Excludes the non-published plugins workspace from Changesets release
planning so it cannot drive public package version bumps.
## Verification
Ran `pnpm run changeset:version` with temporary changesets for
`@trigger.dev/plugins` and `@trigger.dev/core`; the ignored workspace
produced no release-driver updates, and the public package changeset
versioned normally.
Directory-sync effects are idempotent and the accounts-webhook worker
retries
the whole event, so a single failed attempt (typically a role assignment
losing a serializable race during a backfill burst) is self-healing
rather
than alert-worthy. Tag those thrown errors with logLevel "warn" so the
worker
logs at warn instead of error, keeping them visible for triage without
paging.
## Summary
The runs page's empty-state check (whether an environment has ever had a
run, which decides between the "getting started" and "no runs match your
filters" states) ran a `findFirst` against the Postgres `TaskRun` table.
This moves it to ClickHouse, the same store the runs list itself reads
from, so the check no longer queries `TaskRun`.
## Design
Only the runs list triggers the check now (via an `includeHasAnyRuns`
flag); the other presenters that reuse `NextRunListPresenter` (API,
schedule detail, waitpoint detail, error group) no longer issue it. When
the list is empty it runs `SELECT 1 FROM task_runs_v2 ... LIMIT 1`
filtered on the full `(organization_id, project_id, environment_id)`
sort-key prefix with a configurable `created_at` lower bound
(`RUN_LIST_HAS_RUNS_LOOKBACK_DAYS`, default 30), so it hits the primary
index and reads minimal granules.
Results are cached in a tiered memory + Redis SWR cache. Only positive
("has runs") results are cached, so an environment with no runs is
always re-checked and its first run shows up immediately.
Follow-up to #4194 (v3 execution app + core-helper removal). The v3
(engine V1) is end-of-lifed and enforced off in prod, so this removes a
self-contained slice of the remaining dead v3 code while **keeping every
user-facing deprecation message** - a user still on v3 must still be
told to upgrade.
## Legacy dev websocket
`app/v3/handleWebsockets.server.ts` backs the `/ws` transport used
**only** by the legacy v3 `trigger dev` CLI (v4 dev uses a different
transport). It's now authenticate-then-close with
`V3_DEV_DEPRECATION_MESSAGE`, so an old CLI is still told what to do -
only the legacy `AuthenticatedSocketConnection` / `DevQueueConsumer`
execution behind it (which can no longer run) is removed.
- Deleted `app/v3/authenticatedSocketConnection.server.ts` (its only
consumer).
- `engineDeprecation.server.ts` and the deprecation message constants
are untouched.
## Docs
Deleted the intentionally-legacy "Docker (legacy)" self-hosting page
(`open-source-self-hosting.mdx`) and redirected
`/open-source-self-hosting` (+ the existing
`/v3/open-source-self-hosting` alias) to `/self-hosting/overview`;
repointed the two inbound links. The current `self-hosting/*` docs
already describe the v4 (single supervisor) setup.
## Deliberately out of scope
Despite the branch name, this PR does **not** touch MarQS or the
socket.io coordinator/provider namespaces. Investigation found MarQS is
entangled with **live v2** queue/metrics/concurrency/project-cleanup
code (`runQueue`, `queueSizeLimits`, `taskRunConcurrencyTracker`,
`EnvironmentQueuePresenter`, `registerProjectMetrics`, `deleteProject`),
so it needs a per-file reviewed pass, not a bulk delete. That remainder
stays on TRI-11883.
refs TRI-11883
## Summary
4 improvements, 5 bug fixes.
## Improvements
- Add SDK and API client helpers for run bulk actions.
([#4105](https://github.com/triggerdotdev/trigger.dev/pull/4105))
- Large batch payloads now offload to object storage instead of riding
inline in the trigger request. `batchTrigger` and `batchTriggerAndWait`
(and the by-id and by-task variants) offload any per-item payload over
128KB before sending, the same way single `trigger` and `triggerAndWait`
already do, so a big batch no longer blows past the API body limit.
([#4165](https://github.com/triggerdotdev/trigger.dev/pull/4165))
- Removed internal helpers that were only used by the end-of-life v3
self-hosted compute providers.
([#4194](https://github.com/triggerdotdev/trigger.dev/pull/4194))
- Add an `onEvent` callback to `TriggerChatTransport` /
`useTriggerChatTransport` that emits typed lifecycle events for sends,
stream connects, first chunk, and turn completion. Send-success metrics,
time-to-first-token, and "sent but never answered" watchdogs become a
few lines of client code.
([#4187](https://github.com/triggerdotdev/trigger.dev/pull/4187))
```ts
onEvent: (event) => {
if (event.type === "message-sent") metrics.timing("chat.send_ms",
event.durationMs);
if (event.type === "first-chunk") metrics.timing("chat.ttft_ms",
event.sinceSendMs ?? 0);
},
```
## Bug fixes
- fix(cli): honor the MCP server's `--dev-only` flag
([#4199](https://github.com/triggerdotdev/trigger.dev/pull/4199))
- Fix chat turns that throw (for example from an `onTurnStart` hook)
leaking their message listener, which lost or duplicated messages sent
during later turns.
([#4176](https://github.com/triggerdotdev/trigger.dev/pull/4176))
- Fix `chat.agent` and `chat.createSession` permanently dropping user
messages when several arrived during a single turn: every buffered
message is now dispatched as its own turn instead of only the first.
([#4176](https://github.com/triggerdotdev/trigger.dev/pull/4176))
- Fix chat continuation runs replaying already-answered messages: turns
delivered while the run was suspended now advance the session.in resume
cursor, so a new run picks up exactly where the previous one left off.
([#4176](https://github.com/triggerdotdev/trigger.dev/pull/4176))
- Fix `chat.createSession` swallowing a message sent shortly after
stopping a turn: the turn's message listener now detaches when the
stream settles, so those messages run as the next turn.
([#4176](https://github.com/triggerdotdev/trigger.dev/pull/4176))
<details>
<summary>Raw changeset output</summary>
# Releases
## @trigger.dev/build@4.5.2
### Patch Changes
- Updated dependencies:
- `@trigger.dev/core@4.5.2`
## trigger.dev@4.5.2
### Patch Changes
- fix(cli): honor the MCP server's `--dev-only` flag
([#4199](https://github.com/triggerdotdev/trigger.dev/pull/4199))
- Updated dependencies:
- `@trigger.dev/core@4.5.2`
- `@trigger.dev/build@4.5.2`
- `@trigger.dev/schema-to-json@4.5.2`
## @trigger.dev/core@4.5.2
### Patch Changes
- Add SDK and API client helpers for run bulk actions.
([#4105](https://github.com/triggerdotdev/trigger.dev/pull/4105))
- Large batch payloads now offload to object storage instead of riding
inline in the trigger request. `batchTrigger` and `batchTriggerAndWait`
(and the by-id and by-task variants) offload any per-item payload over
128KB before sending, the same way single `trigger` and `triggerAndWait`
already do, so a big batch no longer blows past the API body limit.
([#4165](https://github.com/triggerdotdev/trigger.dev/pull/4165))
- Removed internal helpers that were only used by the end-of-life v3
self-hosted compute providers.
([#4194](https://github.com/triggerdotdev/trigger.dev/pull/4194))
## @trigger.dev/python@4.5.2
### Patch Changes
- Updated dependencies:
- `@trigger.dev/core@4.5.2`
- `@trigger.dev/sdk@4.5.2`
- `@trigger.dev/build@4.5.2`
## @trigger.dev/react-hooks@4.5.2
### Patch Changes
- Updated dependencies:
- `@trigger.dev/core@4.5.2`
## @trigger.dev/redis-worker@4.5.2
### Patch Changes
- Updated dependencies:
- `@trigger.dev/core@4.5.2`
## @trigger.dev/rsc@4.5.2
### Patch Changes
- Updated dependencies:
- `@trigger.dev/core@4.5.2`
## @trigger.dev/schema-to-json@4.5.2
### Patch Changes
- Updated dependencies:
- `@trigger.dev/core@4.5.2`
## @trigger.dev/sdk@4.5.2
### Patch Changes
- Add SDK and API client helpers for run bulk actions.
([#4105](https://github.com/triggerdotdev/trigger.dev/pull/4105))
- Fix chat turns that throw (for example from an `onTurnStart` hook)
leaking their message listener, which lost or duplicated messages sent
during later turns.
([#4176](https://github.com/triggerdotdev/trigger.dev/pull/4176))
- Fix `chat.agent` and `chat.createSession` permanently dropping user
messages when several arrived during a single turn: every buffered
message is now dispatched as its own turn instead of only the first.
([#4176](https://github.com/triggerdotdev/trigger.dev/pull/4176))
- Fix chat continuation runs replaying already-answered messages: turns
delivered while the run was suspended now advance the session.in resume
cursor, so a new run picks up exactly where the previous one left off.
([#4176](https://github.com/triggerdotdev/trigger.dev/pull/4176))
- Fix `chat.createSession` swallowing a message sent shortly after
stopping a turn: the turn's message listener now detaches when the
stream settles, so those messages run as the next turn.
([#4176](https://github.com/triggerdotdev/trigger.dev/pull/4176))
- Add an `onEvent` callback to `TriggerChatTransport` /
`useTriggerChatTransport` that emits typed lifecycle events for sends,
stream connects, first chunk, and turn completion. Send-success metrics,
time-to-first-token, and "sent but never answered" watchdogs become a
few lines of client code.
([#4187](https://github.com/triggerdotdev/trigger.dev/pull/4187))
```ts
onEvent: (event) => {
if (event.type === "message-sent") metrics.timing("chat.send_ms",
event.durationMs);
if (event.type === "first-chunk") metrics.timing("chat.ttft_ms",
event.sinceSendMs ?? 0);
},
```
- Large batch payloads now offload to object storage instead of riding
inline in the trigger request. `batchTrigger` and `batchTriggerAndWait`
(and the by-id and by-task variants) offload any per-item payload over
128KB before sending, the same way single `trigger` and `triggerAndWait`
already do, so a big batch no longer blows past the API body limit.
([#4165](https://github.com/triggerdotdev/trigger.dev/pull/4165))
- Updated dependencies:
- `@trigger.dev/core@4.5.2`
## @trigger.dev/plugins@4.5.2
### Patch Changes
- Updated dependencies:
- `@trigger.dev/core@4.5.2`
</details>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
## Summary
UI/layout/copy pass over the org **SSO & Directory Sync** settings page
(formerly "Identity & Access"). No logic, gates, flags, or data flow
changed — server-side auth (`manage:sso`), Enterprise entitlement,
action validation, and data loading are all untouched.
- Renamed the nav item, page title, and meta from "Identity & Access" to
"SSO & Directory Sync".
- Added a reusable `SettingsLayout` component system (container,
section, header, row, block, actions) modeled on `/account/security`,
and refactored the SSO page onto it (section titles, dividers, left
title/subtitle + right action rows).
- Tightened all UI copy: concise, active voice, consistent labels, no
em-dashes.
- `Select` primitive: additive `wrap`, `popoverClassName`, and
`placement` props (all default to prior behavior) so role options show a
bright title with a wrapping description, right-aligned popover, and no
horizontal overflow.
- Removed the external-link arrow icon from buttons that open a modal;
kept it only on genuinely external actions (Contact us, Open in new
tab).
- Polished the admin portal link dialog: smaller description, tighter
spacing, `ClipboardField` with a permanent copy button, removed the
redundant Copy link button, and a provider-aware Open label (e.g. "Open
in WorkOS") derived from the link host with a safe fallback.
### SSO page UI
<img width="3568" height="2550" alt="CleanShot 2026-07-08 at 18 52
11@2x"
src="https://github.com/user-attachments/assets/009d2437-7552-4ff0-a457-64744a9fcd88"
/>
### Login with SSO and normal email test (local)
https://github.com/user-attachments/assets/b33a4ce9-c1fa-45c9-bd3c-077cb6fc9473
## Test plan
- [ ] Non-Enterprise org: SSO page shows the upsell state
- [ ] Enterprise org, non-Owner without `manage:sso`: 403
- [ ] Enterprise Owner: verify domains, configure SSO, connect
directory, JIT/default/group role selects, and enforcement toggle all
work
- [ ] Role select popovers: bright title + wrapping description,
right-aligned, no horizontal scroll
- [ ] Admin portal dialog: copy button works, "Open in WorkOS" opens the
portal in a new tab
---------
Co-authored-by: Cursor <cursoragent@cursor.com>
## What
Adds a new Example projects page: **ClickHouse chat agent** — a
`chat.agent()` that answers questions about your data by writing and
running SQL against ClickHouse Cloud via the official Node.js ClickHouse
client.
The page follows the existing example-project format (overview,
features, GitHub repo card, how-it-works with code excerpts, relevant
code links) and is registered in `docs.json` in alphabetical order.
## Note
The GitHub repo card links to
`triggerdotdev/examples/tree/main/clickhouse-chat-agent`, which lands in
a companion examples PR — merge that one first.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
v3 (engine V1) is end-of-lifed and the v3 clusters are gone, so this
removes the dead v3 execution code from the monorepo. It's the first
pass of TRI-11824 - the webapp v3 code paths are deliberately left
untouched and gated for a follow-up.
## Apps
Deletes the three v3-only execution apps and their build wiring:
- `apps/coordinator`, `apps/kubernetes-provider`, `apps/docker-provider`
- `.github/workflows/publish-worker.yml` - it built only those three;
the v4 worker publish is a separate workflow
- Their references in `.changeset/config.json`, `.cursorignore`,
`CHANGESETS.md`, `CONTRIBUTING.md`, `.server-changes/README.md`
- `pnpm-lock.yaml` regenerated to prune the apps and their app-only
dependencies (`socket.io`, `@kubernetes/client-node`, `p-queue`,
`execa`, `prom-client`, `tinyexec`)
## Core
Removes the helpers in `@trigger.dev/core` that only those apps used -
`ProviderShell`, `SimpleLogger`, the `Exec`/process helpers,
`isExecaChildProcess`, `getTextBody`, and `testDockerCheckpoint`. Each
was verified to have no remaining consumers anywhere in the repo.
Kept the helpers still used elsewhere: `ExponentialBackoff` (warm-start
client), `HttpReply`/`getJsonBody` (serverOnly http server),
`SimpleStructuredLogger` (widely used), and
`ZodNamespace`/`ZodSocketConnection` (still referenced by legacy v3
webapp code, hence the follow-up pass).
The `./v3/apps` and `./v3/serverOnly` export subpaths remain - only dead
members were trimmed from their barrels, so no `package.json` exports
changed.
## Verification
`@trigger.dev/core` builds, and `typecheck` passes for core, supervisor,
cli-v3, run-engine, redis-worker, and webapp.
refs TRI-11824
### Problem
The agent playground chat builds its realtime transport baseURL from
apiOrigin, but points it at a same-origin /resources/... dashboard
route. When API_ORIGIN differs from APP_ORIGIN, the in/append POST goes
cross-origin, fails the CORS preflight, and messages never reach the
agent ("Failed to fetch").
It only reproduces where the two origins differ — not locally, where
both default to localhost:3030.
Fixes#4149.
### Fix
Build the base URL from window.location.origin (falling back to
apiOrigin on SSR), so realtime traffic stays same-origin — the same
approach AgentView.tsx already uses.
### Testing
Typecheck passes. The CORS path only manifests when API_ORIGIN !=
APP_ORIGIN, so verify on test-cloud (can't reproduce locally).
## Summary
Deleting a project triggered an unbounded database cleanup that scanned
the project's entire run history, so deleting a project with many runs
could be very slow. Project deletion is a soft delete again: run data is
retained and the deletion completes quickly.
## Fix
Project deletion ran a cascade hard-delete whose `BulkActionItem` step
filtered through a relation to `TaskRun` scoped by `projectId`. Prisma
compiles that to an `EXISTS`-join over the project's entire `TaskRun`
set (a large, hot table with no `projectId` index), and it ran on every
project deletion unconditionally.
Removing the cascade-cleanup call restores the prior soft-delete
behaviour: queues are removed, the project is marked deleted, and run
data is retained. The cascade-cleanup service (added in
[#4117](https://github.com/triggerdotdev/trigger.dev/pull/4117)) had no
other callers, so it and its test are deleted.
## Summary
Realtime streams (AI-agent token streaming and run streams) now default
to v2 for self-hosters, backed by a bundled [s2-lite](https://s2.dev)
service. Self-hosting previously shipped no S2 configuration, so streams
ran on the Redis-backed v1 path and there were no docs for wiring up v2.
Both the Docker Compose stack and the Helm chart now provision s2-lite
with persistent storage and set the stream env vars out of the box.
## What's included
- **Docker Compose**: a persistent `s2` service (s2-lite), a basin init
spec, and the `REALTIME_STREAMS_S2_*` plus
`REALTIME_STREAMS_DEFAULT_VERSION=v2` env on the webapp. `.env.example`
documents the v1 fallback and hosted-S2 options.
- **Helm**: an `s2` StatefulSet, PVC, Service and ConfigMap (runs as the
non-root image user via `fsGroup`), an `s2` values block, and webapp env
wiring with an existing-secret path for hosted S2.
- **Docs**: the `REALTIME_STREAMS_S2_*` and
`REALTIME_STREAMS_DEFAULT_VERSION` vars in the webapp env reference,
plus a "Realtime streams" section in the Docker and Kubernetes
self-hosting guides.
## Notes
- The OSS code default stays `v1`; v2 becomes the default purely through
the self-hosting artifacts, so non-self-host deployments are unaffected.
Disabling s2, or setting the version back to `v1`, cleanly reverts to
Redis-backed v1.
- With v2 enabled, the bundled s2 service is a required dependency for
streaming: if it is down, streams error while the task itself still
runs. That is the intended trade for the better v2 path.
- You can point at a hosted S2 at s2.dev instead of the bundled server.
New `/billing-limits` page covering the full [billing limits
feature](https://trigger.dev/changelog/billing-limits) : the three limit
options (plan / custom / no limit), billing alerts (% of limit or dollar
thresholds), what happens when the limit is reached, the recovery flow,
the soft-limits caveat, and the billing limit marker on the Usage page.
## Summary
posthog-js sent product analytics to PostHog Cloud directly from the
browser. This points `api_host` at a same-origin `/ph` path that
forwards to PostHog Cloud EU server-side, following PostHog's standard
first-party reverse-proxy setup.
## How it works
A resource route forwards each request server-side, splitting by path:
`/ph/static/*` and `/ph/array/*` go to the asset host, everything else
(analytics events, feature flags) goes to the ingest host. It rewrites
the `Host` header, strips the `/ph` prefix, and streams the response
back. Only PostHog's own cookies are forwarded, so the app session
cookie stays first-party. Upstream hosts default to PostHog Cloud EU,
overridable via `POSTHOG_INGEST_HOST` / `POSTHOG_ASSETS_HOST`.
It also sets `cross_subdomain_cookie` so a single PostHog session is
shared across the marketing site and app.
Verified locally: static assets return 200 from the EU asset host, and
analytics events return 200 through the ingest host.
## Summary
`sendMessage` from `useChat` gives no feedback about whether a message
actually reached the backend, and the `fetch` override is wire-level: it
requires knowing endpoint semantics, cannot attribute requests to
messages, and misses the headStart first-turn POST entirely. This adds a
typed `onEvent` observability callback to `TriggerChatTransport` /
`useTriggerChatTransport` so send-success metrics, time-to-first-token,
and "sent but never answered" watchdogs become a few lines of client
code.
## Example
```ts
const transport = useTriggerChatTransport({
task: "my-chat",
accessToken: ({ chatId }) => mintChatAccessToken(chatId),
onEvent: (event) => {
switch (event.type) {
case "message-sent":
// Durably acknowledged by the session's input stream, not just "request accepted".
metrics.increment("chat.message_sent", { source: event.source });
metrics.timing("chat.send_duration_ms", event.durationMs);
break;
case "message-send-failed":
metrics.increment("chat.message_send_failed", { status: event.status });
break;
case "first-chunk":
metrics.timing("chat.ttft_ms", event.sinceSendMs ?? 0);
break;
case "turn-completed":
metrics.timing("chat.turn_duration_ms", event.sinceSendMs ?? 0);
break;
}
},
});
```
## Design
One callback, one discriminated union (`ChatTransportEvent`):
- `message-sent` / `message-send-failed`: terminal send outcomes with
`messageId`, a `source` discriminator (submit, regenerate, steer,
action, stop, head-start), `durationMs`, `bodyBytes`, the append's
idempotency key (`partId`, also stored on the server-side record), and
error + HTTP status on failure. `message-sent` means the append was
durably acknowledged, after any internal token-refresh retries.
- `stream-connected` (with a `resumed` flag and the cursor it connected
from), `first-chunk` (chunk type plus `sinceSendMs` for
time-to-first-token), `turn-completed` (`sinceSendMs` full-turn latency
and the agent's committed input cursor), and `stream-error` follow the
response side, so a send can be paired with the answer that should
follow it. `messageId` on response events is client-side attribution
from the last turn-producing send on that chat.
Emissions sit at the transport's existing choke points, covering every
send path uniformly (including steering and headStart, which the fetch
override cannot observe). Exceptions thrown by the callback are
swallowed: observability can never break the chat. The React hook keeps
the callback live across renders instead of freezing the first-render
closure.
## Verification
Unit tests drive the transport directly with the `fetch` override as the
network stub (send success/failure per source, stream lifecycle, resumed
flag, field enrichment, callback exceptions swallowed). Verified
end-to-end against a realistic metrics setup in the ai-chat reference
app (counters, send-duration and TTFT histograms, and both watchdogs
built purely on these events): a healthy two-turn chat produces exactly
the expected event sequence and TTFT values; an oversized append records
`message_send_failed` with status 413; and killing the worker after a
durable send fires both `sent_but_no_stream` and `sent_but_unanswered`,
reproducing and detecting the "message disappeared" failure mode that
motivated this feature.
## Summary
The run page could show an AI generation cost well above what the
provider actually charged, most visibly for OpenRouter and Vercel AI
Gateway requests where a heavily cache-read prompt was priced at the
full input rate. When the provider reports an exact per-request cost, we
now use that instead of catalog pricing.
## Fix
Gateway and OpenRouter include the exact per-request cost in
`ai.response.providerMetadata` (`openrouter.usage.cost` /
`gateway.cost`). That figure already reflects the cache-read discount
and the real per-provider rate, which the catalog cannot reconstruct:
cache-read counts do not arrive in `gen_ai.usage.*`, and per-model
catalog prices drift from what the provider billed, in either direction.
So provider-reported cost is now preferred, and the catalog is used only
when no provider cost is present.
Fallback routing is covered by the same change: when OpenRouter routes
to a different model, `gen_ai.response.model` already carries the served
model, so the cost follows the served model and the provider's own
figure makes it exact.
`extractProviderCost` now runs on every AI span, so it gets a cheap
`"cost"` substring guard to skip the JSON parse on reasoning-model spans
whose provider metadata carries large reasoning text and no cost field.
Regression tests cover the cache-discount overcharge, fallback
served-model pricing, gateway cost, and the catalog fallback path.
## Summary
`batchTrigger` and `batchTriggerAndWait` (and the by-id and by-task
variants) now offload any per-item payload over 128KB to object storage
before sending, the same way single `trigger`/`triggerAndWait` already
do since
[#3785](https://github.com/triggerdotdev/trigger.dev/pull/3785). A batch
of large items no longer inflates the request body past the API limit.
## Demo
A live local run: `batchTriggerAndWait` of 5 items × 300KB (1.5MB
total). Each item offloads to object storage, so the receiver run rows
hold a 65-byte `application/store` pointer instead of the 300KB body,
and every item round-trips (received == sent).
<img width="1000" height="494" alt="batch large-payload offload demo"
src="https://github.com/user-attachments/assets/77ae3958-97d6-4b5c-ab25-39b217caefbc"
/>
## Design
Both the array and streaming batch paths funnel through
`executeBatchTwoPhase`, so offloading happens once there: each item is
measured, then offloaded through the existing
`conditionallyExportPacket` when it crosses 128KB, with bounded
concurrency so a big batch doesn't fire an unbounded number of presigned
PUTs.
Because items are offloaded before the request, SDK batches arrive as
small `application/store` references, so the server-side inline offload
during item ingest (parallelised in
[#3777](https://github.com/triggerdotdev/trigger.dev/pull/3777)) mostly
no longer fires for them.
Every trigger and item also carries its pre-offload serialised size as
`options.payloadSize`. The trigger span records that value, so an
offloaded payload shows its real size instead of the size of the small
object-store reference (previously the span measured the reference).
Container runtimes (cri-o / containerd / podman) can't pull
zstd-compressed layers carried in a Docker v2s2 manifest
(`application/vnd.docker.image.rootfs.diff.tar.zstd`). A deploy built
with an outdated CLI can produce exactly that combination - and today
it's promoted to current and then fails every run at image-pull time.
This extends the pre-promotion image check (#4049) to also inspect the
manifest's layer media types. If any layer uses the unpullable zstd/v2s2
media type, the deploy is rejected at finalize with a clear message to
upgrade the CLI and re-deploy, instead of silently shipping a version
that can't start.
The manifest is already returned by the existing ECR `BatchGetImage`
call, so there's no extra registry request for single-arch images.
Parsing is a lenient Zod schema and **fails open** - a manifest we can't
read never blocks a deploy. Manifest lists / OCI indexes (no top-level
`layers[]`) and OCI zstd (`...tar+zstd`, which runtimes support) pass
unaffected.
Also clarifies in the contributor docs that changesets and
`.server-changes/` notes are user-facing and should be written for
users, not maintainers.
refs TRI-11702
<!-- ccr-slack-attribution -->
_Requested by **Matt Aitken** · [Slack
thread](https://triggerdotdev.slack.com/archives/C032WA2S43F/p1783430373189849?thread_ts=1783430373.189849&cid=C032WA2S43F)_
## ✅ Checklist
- [x] The PR title follows the convention.
- [x] I ran and tested the code works (typecheck of the edited files is
clean; see Testing)
---
## Testing
**Before:** the webapp run-replication test shard failed on nearly every
PR because assertions waited a fixed 1s for rows to replicate from
Postgres → ClickHouse and intermittently checked before the row arrived
under CI load.
**After:** those assertions poll (up to 30s, 250ms interval) until the
rows land, so they pass as soon as replication completes and stop
flaking, without slowing the happy path.
These tests are testcontainers-backed (need Docker + Postgres +
ClickHouse), so the full suite is exercised in CI. Locally I confirmed
the edited `runsReplicationService.part1..part8.test.ts` files
type-check with no new errors.
---
## Changelog
**How:** wrapped the ~21 present-row assertions across
`runsReplicationService.part1..part8.test.ts` in `vi.waitFor`, matching
the existing poll pattern in `part9.test.ts`. Left absence assertions
(expecting 0 rows / no spans) on a fixed settle delay since there is
nothing to poll for. Tests only — no production code changed.
Note: this does NOT touch the `subscribe()` startup race in
`internal-packages/replication/src/client.ts` (a riskier, separate
follow-up).
💯
---
_Generated by [Claude
Code](https://claude.ai/code/session_01KtUdSLKrK17eFVuRYXT6uj)_
---------
Co-authored-by: Claude <noreply@anthropic.com>
## Summary
Adds SDK and API support for run bulk actions. You can now create bulk
cancel or replay actions from `@trigger.dev/sdk` using run IDs or the
same filters as `runs.list()`, then retrieve, list, poll, or abort the
action by its `bulk_` handle.
Tests, docs, changesets added.
## Design
The dashboard bulk action service now accepts structured filters instead
of reading directly from a dashboard request, so the dashboard and API
share the same creation path. Replay actions created through the API are
attributed with the existing `api` trigger source, while
dashboard-created actions keep `dashboard`.
The SDK exposes the new surface under `runs.bulk.*`, including
`targetRegion` for replay region overrides and cursor pagination for
listing bulk actions.
## Filters and runIds
Nuance on filters. If `filter` is provided, it MUST have at least one
key. This is to remove the footgun of passing no filter and selecting
all runs.
```typescript
{ action: "cancel", runIds: ["run_1"] } // valid
{ action: "cancel", runIds: [] } // invalid, min(1)
{ action: "cancel", filter: { status: "FAILED" } } // valid
{ action: "cancel", filter: {} } // invalid
{ action: "cancel", filter: {}, runIds: ["run_1"] } // invalid
```
## Summary
Three fixes to the run-ops database split (the Cloud-only mode where
run-lifecycle rows live on a dedicated Postgres). All are inert in the
default single-database deployment.
The main fix: on the batch trigger paths, a parentless batch's item runs
chose their physical store from a fresh per-org mint-flag read at
processing time, so flipping an org's flag mid-batch could land an item
in a different store than its batch, breaking the `TaskRun.batchId`
foreign key (or silently orphaning the item). The other two harden the
split's safety nets: the schema-parity test now actually compares
columns, and the read fan-out gate now signals when it has been silently
disabled.
## Batch item residency
`RunEngineBatchTriggerService` (api.v2) and the BatchQueue item callback
(api.v3) now anchor each item's id mint on the batch's own friendlyId,
mirroring the already-safe `BatchTriggerV3Service`. Residency is a pure
id-shape check, so an item can no longer diverge from its batch across a
mid-batch flag flip. The pre-failed-run fallback is anchored the same
way (it also sets `batchId`), and the shared mint branch is consolidated
into one helper so every mint path stays in lockstep. No new database
queries; single-database mode is unchanged (a cuid-shaped batch
friendlyId yields a cuid item).
## Schema parity test
The parity test previously read only the dedicated schema and matched
model headers with regexes, so it never compared columns and could not
catch a run-subgraph column that diverged between the two physical
schemas. It now parses both schemas and asserts bidirectional
scalar-column parity (type, nullability, array-ness, default) across the
run-subgraph models, and fails on any field line it can't parse. Scoped
to the run-subgraph models so unrelated control-plane edits don't break
it.
## Read fan-out signal
The split read fan-out gate is decided by the object identity of the NEW
vs control-plane clients. It now warns when both run-ops URLs are set
but the NEW client isn't a distinct instance (fan-out silently off), and
a new test exercises the real topology-into-gate wiring so a future
refactor that aliases the clients can't disable fan-out unnoticed.
## Verification
New unit and glue tests cover all three changes; the DB-backed
residency, store-routing, and topology suites pass against real
Postgres; `typecheck` is clean for both packages.
## Summary
Sending a message to a chat whose run had ended could make the message
vanish: the continuation run replayed already-answered messages, never
processed the new one, and a page refresh lost it entirely. Chasing that
report surfaced four composing message-loss bugs in the chat session
runtime; this PR fixes all of them, each with a regression test.
## The fixes
1. **Stale resume cursor.** Records delivered while a run was suspended
(the waitpoint path) advanced the SSE resume counter but not the
committed-consume cursor, so the `session-in-event-id` header stamped on
turn-completes went stale by one record per suspended turn. Continuation
boots seed from that header, which is what made them replay
already-processed messages. `session.in.wait()` now advances both
cursors.
2. **Only the first buffered message dispatched.** Messages arriving
during a turn are consumed into a buffer whose end-of-turn pickup
dispatched only the first entry; the buffer was recreated each turn, so
the rest were discarded, and since consuming a record commits the cursor
the loss was permanent. A continuation boot's replay delivers several
records back-to-back, which put the user's new message at index 1 or
later. The buffer now outlives the turn and drains one message per turn
in both `chat.agent` and `chat.createSession` (whose equivalent buffer
was never read at all).
3. **Post-stop window in `chat.createSession`.** The turn's message
listener stayed attached through the stopped turn's post-stream work, so
a message sent shortly after stopping a turn was consumed into the dead
steering queue and lost. The listener now detaches when the stream
settles, matching the `chat.agent` loop.
4. **Handler leak on errored turns.** A turn that threw outside the
streaming section (for example from an `onTurnStart` hook) leaked its
message listener. Previously that silently lost mid-turn messages; with
the loop-level buffer it would have duplicated them instead. The
subscription handle is now detached by the turn's catch/finally, and
`chat.createSession` defensively detaches its prior turn's listener when
user code exits a turn without `complete()`/`done()`.
## Verification
Reproduced end-to-end with the ai-chat reference project before the fix
(message consumed but never answered, two replayed turns, gone on
refresh) and verified after (single clean turn, survives refresh,
turn-complete cursors strictly advancing). Regression tests in
`packages/trigger-sdk/test/pending-message-drain.test.ts` cover all
four, each verified red against the unfixed behavior. A smoke sweep of
the standard chat scenarios (basic send, multi-turn, suspend/resume,
mid-stream refresh, stop, steering, cancel + continue, and the
`createSession` variant) passes on the final branch state.
## Summary
When the platform database is briefly unreachable while a run is
resuming from a wait, the run no longer fails with
`TASK_EXECUTION_ABORTED`. The worker now retries the resume through the
outage instead of aborting on the first blip.
## Root cause
Resuming a run calls the engine's `continue` worker-action endpoint.
That route caught every error and returned a `422`, which the worker's
HTTP client treats as non-retryable. So a transient Prisma
infrastructure error (for example `P1001` "Can't reach database server")
was flattened into a permanent failure: the worker gave up, force-killed
the run process, and completed it with `TASK_EXECUTION_ABORTED`.
## Fix
- The `continue` route now lets infrastructure errors propagate to the
generic 500 handler (message scrubbed, and retryable by the worker's
HTTP client), the same treatment the trigger path already gives them via
`isInfrastructureError`. Genuine validation errors (snapshot mismatch,
invalid state) still return `422`, so a stale retry stays non-retryable.
Resuming is idempotent server-side (guarded by the snapshot id), so
retrying is safe.
- The worker's `continueRunExecution` calls (both the
runner-to-supervisor and supervisor-to-engine hops) retry with a longer,
jittered backoff so they can ride out an outage lasting tens of seconds,
and the jitter keeps a fleet of resuming runs from stampeding the
database the moment it recovers.
Builds on #3960, which scrubbed the leaked message on these routes but
left the status non-retryable.
No changeset: this is a server-side behaviour fix recorded via
`.server-changes`. The `@trigger.dev/core` edits are internal run-engine
worker plumbing, not a public API change.
## Summary
Running `pnpm run db:migrate` locally left `defaultPrices.ts` and
`modelCatalog.ts` in `llm-model-catalog` showing as modified every time,
a ~10k-line diff that only ever changed formatting. This stops the
churn.
## Root cause
The root `db:migrate` script ends in `&& turbo run generate`, which runs
the `generate` script in every package that has one, including this one.
The generator writes its output with `JSON.stringify` (quoted keys, no
trailing commas), but the checked-in copies had been reformatted by
oxfmt (unquoted keys, trailing commas). So the generator output never
matched what was committed, even though the parsed data was identical.
## Fix
Add the two generated files to `.oxfmtrc.json`'s ignore list and commit
the raw generator output, matching how other codegen files in the repo
are already handled (e.g. the tsql grammar). Generation is
deterministic, so `generate` and `format` are both no-ops on a clean
tree now.
No changeset: internal package, dev tooling only, no runtime or public
API change.
## Summary
Depends on #4136.
Docker Compose self-hosting now uses the maintained `latest` image tag
by default instead of the frozen prerelease tag. The version-locking
docs keep pointing production users at explicit versioned tags when they
want pinned upgrades.
The engine `triggerTask` suite was a single 2447-line file with 23
`containerTest` cases, each spinning its own Postgres + Redis. vitest
shards by whole file, so all 23 container setups landed on one shard and
dominated its wall-clock. The recorded entry in `test-timings.json`
badly under-counts the real cost (it does not capture the
per-`containerTest` container startup that dominates on CI), so the
duration-sharding sequencer treated the file as light and stacked it,
producing one ~21 minute shard.
Splitting does not reduce the number of container setups; it lets those
23 cases distribute across shards instead of stacking on one. The webapp
unit-test stage is gated by its slowest shard, so this cuts the stage's
wall-clock roughly in half.
## CI timing (before vs after)
Real CI wall-clock of the `Unit Tests: Webapp` shards (`--shard=i/10`).
"Before" is sampled from recent runs on other branches (unsplit file,
from `main`); "after" is this PR.
| Shard | Before (s) | After (s) |
|------:|-----------:|----------:|
| 1 | 250 | 359 |
| 2 | 444 | 411 |
| 3 | 497 | 659 |
| 4 | **1257** | 284 |
| 5 | 545 | 641 |
| 6 | 284 | 644 |
| 7 | 244 | 214 |
| 8 | 340 | 445 |
| 9 | 188 | 395 |
| 10 | 234 | 567 |
| **Slowest shard (gates the stage)** | **~1247s (≈21m)** | **659s
(≈11m)** |
| Sum of all shards | 4283 | 4619 |
Before: shard 4 is the long pole at 1237s / 1247s / 1257s across three
sampled runs (the `triggerTask` file plus whatever else the packer put
with it). After: the six pieces spread across shards, the slowest drops
to 659s. The small rise in summed time is the extra per-file container
startup, paid in parallel across shards, so the gating number still
falls by about 10 minutes.
## Change
Split into six per-concern files that share a `triggerTaskTestHelpers`
module (the `vi.mock` calls stay per-file, since vitest hoists them):
- `triggerTask.test.ts` (3): trigger + concurrencyKey coercion
- `triggerTask.idempotency.test.ts` (4): idempotency + queue resolution
- `triggerTask.debounce.test.ts` (4): retries + debounce validation
- `triggerTask.mollifier.test.ts` (4): mollifier call-site behaviour
- `triggerTask.metadataCache.test.ts` (4): DefaultQueueManager task
metadata cache
- `triggerTask.residency.test.ts` (4): child run residency inheritance
All 23 cases are preserved. The file's `test-timings.json` entry is
split across the new files so bin-packing stays balanced.
While rewriting these files, cleanup was moved to `onTestFinished(() =>
engine.quit())` so an `engine`/`Redis` leaked on a failing assertion no
longer persists on the worker-scoped Redis and cascades into later cases
(`hookTimeout` raised to 60s so the after-cleanup gets the full budget).
Prisma lookups switched from `findUnique` to `findFirst` to match the
repo convention.
Verified: all six files run green locally (23/23), oxlint and oxfmt
clean.
## Summary
On the run-ops database split, a run that waits (`triggerAndWait`,
`batchTriggerAndWait`, `wait.forToken`) could hang forever after its
wait had already completed. The runner reads a resume from
`/snapshots/since` exactly once: if that read returned the resume
snapshot without its completed-waitpoints, the runner logged "executing
without completed waitpoints", advanced its cursor, and never re-read
it, so the awaiting run never continued.
## Root cause
The resume snapshot and its completed-waitpoint rows were written as two
separate commits. This regressed when the split replaced Prisma's atomic
nested `connect` with an FK-free insert (in
[#4163](https://github.com/triggerdotdev/trigger.dev/pull/4163)), and
`/snapshots/since` is served from a read replica. A fetch landing in the
sub-millisecond gap between the two commits, or a multi-reader replica
serving the snapshot from a different point in time than its join rows,
delivered an empty resume. Because the runner consumes each snapshot
once and treats an empty resume as terminal, a single stale read was
fatal and produced a permanent, nondeterministic hang.
## Fixes
- Commit a snapshot and its completed-waitpoint links in one
transaction, restoring the atomicity the split removed.
- Repair the completed-waitpoints from the owning primary when a
multi-reader replica serves the snapshot without its join rows. This
covers single-waitpoint resumes, which carry no
`completedWaitpointOrder` and so were missed by the count-based repair.
- Read the primary in the checkpoint `WAIT_FOR_BATCH` pre-check, so a
batch that already resumed is not re-suspended into a stall.
- Fall back to the primary when a waitpoint token misses both read
replicas, so a token completed immediately after it was minted no longer
returns a spurious 404.
- Route batch-item creation by `batchTaskRunId`, consistent with the
batch-completion count and the row's foreign key.
- Reject control-plane-only relation selects on the dedicated schema
with a clear error instead of an opaque Prisma failure, and stop
`createDateTimeWaitpoint` bypassing residency routing through a caller
transaction.
Verified against the deployed split topology: a resume snapshot and its
completed-waitpoints are now always delivered together, so the runner
can no longer drop a resume.
Extend the SSO plugin contract for directory sync and apply membership
effects
from the accounts webhook worker: provision users in mapped groups (role
from
group mapping, else the org default role), deprovision on removal, and
keep a
sticky-removal tombstone so JIT never silently re-adds a removed user.
JIT and
Directory Sync coexist; roles default to Developer (the JIT default-role
picker
has no 'None'). Changing a group's role in the dashboard re-applies it
to that
group's current members immediately. The Directory Sync settings section
(group→role mapping, external-domain + manual-membership policy,
deferred Save)
appears once a domain is verified — independent of SSO — gated by the
hasSso
flag. The settings page polls the whole page while entitled with
override-aware
drafts so in-progress edits are never clobbered.
## Summary
On the run-ops split, NEW-residency runs could hang. Time-based waits
(`wait.for`, `wait.until`, `delay`, waitpoint tokens),
`batchTriggerAndWait`, and attempt starts stalled and never resumed.
Each was a run-ops read or update that hit the wrong database: either
the owning store's read replica when it needed read-your-writes, or the
wrong store entirely because it routed by an id that does not encode
residency.
## Fixes
**Waitpoint resume (the main hang).** The managed resume path reads a
run's completed waitpoints by snapshot id
(`findSnapshotCompletedWaitpointIds`). Snapshot ids are cuids, which
always classify to the legacy store, so a NEW run's join rows (which
live on the new store) were never found. The resumed run saw zero
completed waitpoints and hung. It now fans out across both stores and
merges, like its sibling readers.
**Batch completion.** Batch item completion
(`updateManyBatchTaskRunItems`) routed by the item id, which is also a
cuid, so a NEW batch's items were updated on the wrong store, matched
zero rows, and the batch was treated as already complete (its parent's
`batchTriggerAndWait` then hung). It now routes by the batch id, which
does encode residency, matching the sibling `countBatchTaskRunItems`.
**Read-your-writes on the resume path.** The block-time
pending-waitpoint check (`countPendingWaitpoints`) and the attempt-start
lock check (`findRun` in `startRunAttempt`) both read the owning store's
replica with no read-your-writes guarantee, so a just-committed
waitpoint completion or dequeue lock could be missed under replica lag
and strand the run. Both now read the owning primary.
Each fix ships with a two-database store or engine test that reproduces
the hang and passes with the fix.
## Summary
The run-ops runs-replication source now takes its connection URL from
`RUN_REPLICATION_RUN_OPS_DATABASE_URL`, required whenever the run-ops
split is enabled.
The runs replicator speaks the Postgres streaming replication protocol,
which cannot run through a transaction pooler, so it needs its own
direct endpoint separate from the app's `RUN_OPS_DATABASE_URL` (which
may point at a pooler). When the split is on and this is unset, boot
fails via `SplitReplicationMisconfiguredError` rather than silently
falling back to a wrong endpoint.
## What
Adds the missing `COPY scripts/retry-prisma-generate.mjs` to the
supervisor `Containerfile` builder stage, before `RUN pnpm run
generate`.
## Why
The `generate` scripts in `internal-packages/database` and
`internal-packages/run-ops-database` shell out to
`scripts/retry-prisma-generate.mjs`. The supervisor build never copied
that file into the image, so `pnpm run generate` failed:
```
@internal/run-ops-database:generate: Error: Cannot find module '/app/scripts/retry-prisma-generate.mjs'
```
This is the same failure class as #4156 (webapp Dockerfile). The
supervisor `Containerfile` is the **only other** build file that runs
`pnpm run generate` — the coordinator / docker-provider /
kubernetes-provider Containerfiles don't, so this completes the fix.
## Verification
Local `docker build` of the supervisor `Containerfile` builder target —
result appended below once the build completes.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
## Build-blocker hotfix
`publish.yml` on `main` is failing to build the image after #4154
merged:
```
@trigger.dev/database:generate: Error: Cannot find module '/triggerdotdev/scripts/retry-prisma-generate.mjs'
… ERROR: process "/bin/sh -c pnpm run generate" did not complete successfully: exit code: 1
```
## Cause
#4154 (Windows-CI hardening) changed the `generate` scripts of
`@trigger.dev/database` and `@internal/run-ops-database` to call `node
../../scripts/retry-prisma-generate.mjs`. But `docker/Dockerfile`'s
`builder` stage does `COPY docker/scripts ./scripts` (replacing the
scripts dir) and then copies back only the specific root scripts it
needs (`updateVersion.ts`, `bundleSdkDocs.ts`) before `RUN pnpm run
generate` — the new `retry-prisma-generate.mjs` wasn't copied, so `pnpm
run generate` can't find it and the image build fails.
`publish.yml` only runs on push to `main` (not on PRs), so #4154's PR CI
never built the image and this slipped through.
## Fix
One line — copy the retry script alongside the other root scripts before
the generate step:
```dockerfile
COPY --chown=node:node scripts/retry-prisma-generate.mjs scripts/retry-prisma-generate.mjs
```
## Verification
Built locally with `docker build --target builder` (the stage that runs
`pnpm run generate`) to confirm the generate step now passes — result
appended once it finishes.
No changeset / `.server-changes` — Dockerfile/build-only change, no
package or server-runtime change.
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
## Problem
The run-ops split mints NEW-store run ids as **27-char base62 KSUIDs**.
The supervisor writes the run id into the Kubernetes pod name
(`runner-<id>`), and pod names must be DNS-1123 labels (lowercase
`[a-z0-9-]`) — so uppercase base62 ids make k8s reject the pod (422) and
**those runs never launch** (they loop in `PENDING_EXECUTING` until the
heartbeat-stall handler nacks them, forever). `.toLowerCase()` can't fix
it: base62 has both `A`(10) and `a`(36) as distinct symbols, so folding
collides distinct ids and destroys sort order.
## Fix: change the encoding, not the structure
Mint a **26-char lowercase base32hex** run id:
```
run_<24-char base32hex core><region char><version char>
[ 6-byte ms timestamp ][ 9 CSPRNG bytes ]
```
- **base32hex** (RFC 4648 §7, alphabet `0-9a-v`): lowercase,
order-preserving, DNS-safe; 15 bytes → exactly 24 chars, no padding.
Hand-rolled encode/decode (no new dependency).
- **48-bit ms timestamp** in the leading bytes → plain string sort ==
creation order at millisecond resolution.
- **72 bits CSPRNG** entropy; PK unique constraint is the backstop (no
retry loop).
- **region / version** are raw positional chars (read via one `charAt`
before decoding/routing), version = `"1"`.
DNS-safe from birth and hyphen-free, so **firekeeper is unchanged** —
`runner-<id>-attempt-N` → strip `runner-`, cut at first hyphen still
recovers the exact id incl. region+version.
## Residency discriminator: length → version char
`classifyKind`/`classifyResidency` (`runOpsResidency.ts`) previously
distinguished NEW vs LEGACY by **id length**. That gets ambiguous with a
third format. It now discriminates on the **version char at a fixed
position** (`isRunOpsIdBody`: 26 chars, `[25] === "1"`, base32hex
alphabet) → NEW; everything else → LEGACY. Total, never throws. The
`Residency` (NEW/LEGACY) contract the routing store consumes is
unchanged; the `"ksuid"` `ResidencyKind` label is retained only because
it's the persisted `runOpsMintKsuid` feature-flag value.
## Scope / verification
- Generator + discriminator in `@trigger.dev/core` isomorphic; mint path
+ all id-shape call sites swept (~40 webapp files); changeset added
(`@trigger.dev/core` patch).
- Core unit tests (encode/decode round-trip + property, generator shape,
ms sort-order incl. intra-second, parse partitioned-vs-legacy,
firekeeper round-trip): **24 pass**. `@trigger.dev/core` builds; webapp
typechecks; format/lint clean.
## Open decisions (flagged, not silently chosen)
1. **Backward-compat**: existing 27-char base62 KSUID runs now classify
LEGACY. On test cloud these are the broken/looping runs that never
completed, so this is acceptable — but worth a conscious call before
prod. No transitional length-recognition added (keeps the discriminator
clean).
2. **Storage collation**: the sort guarantee is byte-order — if the
run-ops id column is `TEXT` with default locale collation it's silently
not honored. Confirm whether `COLLATE "C"` / `BYTEA` is needed on the
run-ops schema.
3. **Region sourcing** wiring — see `regionCharForRegion` /
`REGION_CODES`.
---
## ⚠️ Required migration — deploy in lockstep
This PR renames a persisted feature-flag key/value and an env var. These
are **not** changed by the code alone and must be migrated when this
deploys, or affected orgs silently fall back to `cuid` minting (no crash
— `defaultValue: "cuid"`):
1. **Env var** (terraform): `RUN_OPS_MINT_KSUID_ENABLED` →
`RUN_OPS_MINT_ENABLED` (carry the value over).
2. **DB** `organization.featureFlags`: migrate both the key and value
together:
- key `runOpsMintKsuid` → `runOpsMintKind`
- value `"ksuid"` → `"runOpsId"`
Until an org's flag row is migrated, its `runOpsMintKind` lookup misses
and it mints `cuid` (legacy) — so no NEW-store ids for that org until
the data lands.
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
## Summary
The **Run ID** and **Batch ID** filters on the runs list, batches list,
and logs view rejected valid IDs. The input showed an error and the
**Apply** button stayed disabled, so filtering by an affected run or
batch ID from the dashboard was impossible.
The filter validators hard-coded exact friendly-id character lengths.
Friendly IDs come in three generations that all still exist in the data
(`<prefix>_` plus a 21-char nanoid, a 25-char cuid, or a 27-char ksuid),
and the hard-coded lengths never covered all three at once.
## Fix
All the ID filter validators (run, batch, waitpoint, schedule) now share
one helper, `makeFriendlyIdValidator`
(`apps/webapp/app/utils/friendlyId.ts`), which validates by prefix plus
a base62 body of any known generator length (21 / 25 / 27). The cuid and
ksuid lengths are sourced from core so the helper tracks any future
change to those formats. Unit tests assert it accepts the output of the
real id generators and rejects malformed input.
Downstream was already unaffected: run/batch route params and
URL-applied filters use unconstrained validation, so only the manual
filter inputs needed the fix.
## What
Adds the ability to **automatically migrate the dedicated run-ops
database** (the NEW DB in the run-ops split), matching how every other
database in the system is migrated. Follow-up to the run-ops split
activation.
## Changes
- **Migrate runner** — new
`internal-packages/run-ops-database/scripts/migrate.mjs`, exposed as
`db:migrate:deploy` / `db:migrate:status`. Connects via
`RUN_OPS_DATABASE_URL` (the same var the app uses) and expands `${VAR}`
refs like Prisma's dotenv.
- **Self-host** — `docker/scripts/entrypoint.sh` runs the run-ops
migration on boot when the DB is configured, gated by
`SKIP_RUN_OPS_MIGRATIONS`. Single-DB installs never set the URL, so it's
a clean no-op.
- **Single env-var family** — the run-ops DB is now addressed by one
canonical `RUN_OPS_*` family, connect path and migrations resolving the
identical URL:
- `RUN_OPS_DATABASE_URL` (writer) — replaces `TASK_RUN_DATABASE_URL`
- `RUN_OPS_LEGACY_DATABASE_URL` — replaces
`TASK_RUN_LEGACY_DATABASE_URL`
- `RUN_OPS_DATABASE_READ_REPLICA_URL` — replaces
`TASK_RUN_DATABASE_READ_REPLICA_URL`
- the old `TASK_RUN_*` aliases, the `??` coalesce, the
`runOpsNewDatabaseUrl` indirection, and the migrate-only `directUrl` are
all removed (consumers read `env.RUN_OPS_DATABASE_URL` directly).
`directUrl` was dropped because it was only ever used by `prisma
migrate` (never the app runtime) to bypass a pooler for advisory locks —
premature here since the run-ops connection isn't wired to the app yet.
If a pooler is later introduced for the app, a direct URL can be
reintroduced then.
## Safety
- **Pure rename** — nothing deployed sets any `TASK_RUN_*` var (the
split isn't activated anywhere yet; `.env.example`, docker-compose, and
cloud already use `RUN_OPS_*`), so there is no config migration.
- **Single-DB / self-host** — no new required env var; entrypoint and
migrate are no-ops when `RUN_OPS_DATABASE_URL` is unset.
- **Cloud** — runs migrations as pre-deploy ECS tasks (companion cloud
PR), calling these same `db:migrate:deploy` / `db:migrate:status`
commands.
## Verification
- Live migration against a fresh scratch DB with only
`RUN_OPS_DATABASE_URL` set: both migrations applied, no `P1012`/`P1013`;
`${VAR}` expansion, idempotent re-run, `status`, and no-op skip all
pass.
- Schema parity 4/4; `typecheck --filter webapp` 18/18; affected
split/replication tests 34/34.
## Scope
This delivers automatic migrations only. Enabling the app to *use* the
new DB (setting `RUN_OPS_DATABASE_URL` + `RUN_OPS_SPLIT_ENABLED` on the
service) is a separate activation step.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
## Problem
`LogicalReplicationClient` uses a Redlock leader lock to guarantee a
single active consumer per Postgres logical replication slot. The lock
resource was keyed on the client `name`:
```
logical-replication-client:${this.options.name}
```
A slot permits exactly one consumer, so the lock's job is to serialize
consumers **of a given slot**. Keying it on `name` breaks that whenever
two clients target the same slot with different names — most notably
across a rolling deploy where the client `name` changes but `slotName`
does not. Both acquire *distinct* locks, both consider themselves
leader, and the second to reach `START_REPLICATION` hits `replication
slot "<slot>" is active for PID <n>`. Because that query was
fire-and-forget and its failure was only logged (no retry), the consumer
stopped and replication stalled until the process was restarted.
## Fix
**1. Key the leader lock on `slotName`** — the actual single-consumer
resource:
```
logical-replication-client:${this.options.slotName}
```
Consumers of the same slot now contend on the same lock and hand off
cleanly across restarts/deploys; different slots stay independent.
`name` is kept for logging and the pg `application_name`.
**2. Self-healing resubscribe** (`resubscribeOnFailure`, opt-in) —
instead of logging-and-dying, a client re-subscribes with exponential
backoff after a lost election or a failed `START_REPLICATION`, so a
rolling deploy self-heals: the incoming pod retries until the draining
pod releases the slot, then takes over. Safety:
- `#cleanupAttempt()` unconditionally ends the pg client (freeing the
walsender) and releases the leader lock before rescheduling — retries
never leak connections/locks.
- `shutdown()` sets an intentional-stop latch re-checked after every
`await` in `subscribe()` (and aborts the lock-acquire spin), so a
resubscribe can never race or outlive an intentional shutdown.
- Backoff resets only on genuine stream start, so a permanently stuck
slot backs off to the ceiling and logs loudly rather than tight-looping;
an epoch guard neutralises stale `START_REPLICATION` catches.
Runs- and sessions-replication opt in and use `shutdown()` for all
intentional stops.
**3. Observability** — the admin runs-replication status route probed
the old name-keyed Redis key (would report `leader:false` for every
source after fix#1); now probes the slot-keyed key.
## Tests
`internal-packages/replication/src/client.test.ts` (real Postgres +
Redis containers):
- same-slot/different-name → second client must not double-lead or race
into "slot is active" (the regression)
- a failing `START_REPLICATION` retry loop must not leak connections or
locks
- `shutdown()` during an in-flight `subscribe()` must not leave a zombie
leader
- `subscribe()` after `shutdown()` re-arms `resubscribeOnFailure`
- self-heals once the leader releases the slot
Plus the multi-source wiring test updated to the slot-keyed lock keys.
## Rollout
With the self-healing resubscribe, this ships as a **plain rolling
deploy** — the incoming pods retry across the one-time lock-key
transition and take over once the old pods drain (a brief replication
stall that the durable slot replays on reconnect — no data loss). No
stop-before-start required.
## What
Extends the ClickHouse runs-replication service to fan in from multiple
Postgres sources (the control-plane DB and the run-ops DB) instead of a
single source, plus the admin operations to run and observe it.
- **Multi-source fan-in** (`services/runsReplicationService.server.ts`,
new `runsReplicationInstance.server.ts`,
`runsReplicationGlobal.server.ts`): factors the replication service into
per-source instances and a coordinator so a single ClickHouse target is
fed from more than one Postgres source.
- **Admin ops** (`routes/admin.api.v1.runs-replication.status.ts`,
`admin.api.v1.runs-replication.backfill.ts`,
`v3/services/adminWorker.server.ts`): adds a status endpoint reporting
per-source replication state and updates the backfill entrypoint for the
multi-source shape.
## Why
PR7 of the run-ops split stack, and the final piece: once run state can
live in a separate run-ops DB (earlier PRs), the analytics replication
into ClickHouse has to consume both sources so runs remain queryable
regardless of residency. Behavior-changing for the replication service
internals; the ClickHouse-facing output is unchanged (still one runs
stream), and single-source operation is preserved when the split is not
enabled.
## Tests
New vitest coverage: `runsReplicationInstance.test.ts` (per-source
instance behavior) and `runsReplicationService.part8`/`part9` suites
exercising the multi-source coordinator. Testcontainers-backed
(ClickHouse + Postgres); no mocks.
## Notes
Draft, **stacked on #4118** (`runops/pr06-write-path`). Review that
first; this diff is against it.
Server-change / changeset note to be added at stack-assembly time.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
## What
Routes the webapp write path through the run-ops split seam:
trigger/batch minting, idempotency-key resolution, and the run-lifecycle
services now determine residency and dispatch writes to the correct
store.
- **Trigger & batch** (`runEngine/services/triggerTask.server.ts`,
`batchTrigger.server.ts`, `createBatch.server.ts`,
`streamBatchItems.server.ts`, `v3/services/batchTriggerV3.server.ts`):
mint ids with the run-ops-aware minting and route creation/streaming
through the store; batch children inherit the parent's residency.
- **Idempotency** (`runEngine/concerns/idempotencyKeys.server.ts` + new
`idempotencyResidency.server.ts`): idempotency-key lookup/dedup is
residency-aware so a keyed retrigger resolves against the store that
owns the original run.
- **Run lifecycle services** (`createCheckpoint`,
`createTaskRunAttempt`, `enqueueDelayedRun`, `expireEnqueuedRun`,
`finalizeTaskRun`, `resumeBatchRun`, `cancelDevSessionRuns`,
`executeTasksWaitingForDeploy`, `triggerFailedTask`): resolve their
target run through the store rather than a fixed client.
- **Reads that fan out from writes** (`runsRepository` +
`clickhouseRunsRepository`, `BulkActionV2` + batch read-through,
realtime `sessions`/`runReader`, alerts
`deliverAlert`/`performTaskRunAlerts`): route through the read-through
resolver.
- `9535ae63d` — resolves the parent run through an injectable run store
in `TriggerFailedTaskService`.
- `bf8f7c881` — drops the "known-migrated" concept from write-path and
read repos; residency is id-shape only.
- `515b897ea` — self-defaults `resolveWaitpointThroughReadThrough` to
the safe run-ops clients.
## Why
PR6 of the run-ops split stack. This is the write-path counterpart to
the read foundation in the previous PRs: with it in place, both reads
and writes route through the seam. Additive when the split is disabled
(id-shape resolution collapses to the control-plane client);
behavior-changing on the minting, idempotency, and lifecycle paths when
enabled.
## Tests
Large new/expanded vitest suite under `apps/webapp/test/` and colocated
service tests: trigger-task and batch-trigger store routing, residency
inheritance, idempotency dedup residency + legacy-authority, bulk-action
read routing, cancel-dev-session routing, alerts store routing,
runs-repository read-through, realtime session/run-reader read-through
and stream-registration routing, and the waitpoint read-through default.
Testcontainers-backed; no mocks.
## Notes
Draft, **stacked on #4117** (`runops/pr05-webapp-foundation`). Review
that first; this diff is against it.
Server-change / changeset note to be added at stack-assembly time.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
## What
Wires the run-ops split into the webapp: database topology, environment
flags, split-mode gating, and the control-plane resolver/cache layer
that the run-store and run-engine seams from the previous PR plug into.
- **DB topology & env** (`apps/webapp/app/db.server.ts`,
`env.server.ts`, `entry.server.tsx`): adds the run-ops database
clients/topology and the environment variables that configure and gate
the split.
- **runOpsMigration module** (new
`apps/webapp/app/v3/runOpsMigration/`): the webapp-side machinery —
`splitMode.server.ts`, `controlPlaneResolver.server.ts` +
`controlPlaneCache.server.ts`, `readThrough.server.ts`,
`crossSeamGuard.server.ts`, `distinctDbSentinel.server.ts`, id-minting
helpers (`mintBatchFriendlyId`, `runOpsMintKind`,
`resolveInheritedMintKind`), `runOpsCascadeCleanup.server.ts`, the split
read gate, and route/unblock catalogs.
- **Store/engine wiring** (`app/v3/runStore.server.ts`,
`runEngine.server.ts`, `runEngineHandlers.server.ts` + new
`runEngineHandlersShared.server.ts`): points the webapp's store/engine
construction at the resolver, and factors shared handler logic out so
both seams use one path.
- **Read-path touch-ups**: `runtimeEnvironment.server.ts`,
`eventRepository/index.server.ts`, `taskRunHeartbeatFailed.server.ts`,
`engineVersion.server.ts` route their run/environment lookups
read-through the resolver.
- `413a94511` — interlocks split mode against the native realtime
backend so the two aren't enabled in an incompatible combination (see
`.server-changes/run-ops-split-realtime-interlock.md`).
- `dc74c57fd` — drops the earlier "known-migrated" read layer; residency
is determined by id-shape only.
## Why
PR5 of the run-ops split stack. This is the webapp foundation layer: it
stands up the DB topology, flags, and resolver/cache the rest of the
stack depends on, and repoints webapp read paths through the resolver.
Additive when the split is not enabled (existing single-DB behavior
preserved behind flags); behavior-changing on the read-through paths and
the realtime interlock.
## Tests
New vitest coverage across `apps/webapp/test/` and colocated
`*.server.test.ts` files: db topology, split mode, split read gate,
cross-seam guard, mint cutover / flip latency, control-plane cache,
control-plane resolver, distinct-db sentinel, read-through loaders
(route loaders, run-detail loaders, `findEnvironmentFromRun`), and the
run-engine handlers. Testcontainers-backed; no mocks. `pnpm-lock.yaml`
synced for the two new webapp deps.
## Notes
Draft, **stacked on #4116** (`runops/pr04-store-engine`). Review that
first; this diff is against it.
Server-change / changeset note to be added at stack-assembly time.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
## What
Introduces the run-store routing seam and the run-engine read seams that
let run lifecycle operations be dispatched to either the control-plane
database or a separately-generated run-ops database, depending on where
a run/batch resides.
- **run-store** (`internal-packages/run-store`): adds `runOpsStore.ts`
and substantially expands `PostgresRunStore.ts` so the store can resolve
residency and route reads/writes to the correct backing client.
`types.ts` grows the routing/residency types; `NoopRunStore.ts` is
removed.
- **run-engine** (`internal-packages/run-engine`): adds
`engine/controlPlaneResolver.ts` and routes the per-system read paths
(dequeue, enqueue, waitpoint, checkpoint, run-attempt, ttl, delayed-run,
execution-snapshot, pending-version, debounce, batch) through the
resolver/store instead of talking to a single Prisma client directly.
`engine/errors.ts`, `engine/types.ts`, and `engine/index.ts` are
extended to support injecting the store/resolver.
Three fixes are included on top of the seam work:
- `c6cadd85f` — routes read-your-writes to the owning store's
**writer**, not its lagging replica, so an operation immediately reading
back what it just wrote sees a consistent result.
- `05c912e05` — normalizes run-ops-generation Prisma errors to the
control-plane error class at the store **write boundary**, so
`instanceof` checks and the `P2002` → 422 handling continue to work
across the separately-generated run-ops Prisma client.
- `88d12907f` — resolves NEW-resident batches in
`ApiBatchResultsPresenter` by routing the batch read through the store,
so a dedicated-DB batch resolves instead of returning 404.
The change is heavily test-first: the bulk of the diff is new
unit/integration coverage for the store routing, residency, and each
run-engine system's control-plane resolver path.
## Why
PR4 of the run-ops split stack (PR1–PR3 land the ClickHouse
test-container and earlier plumbing). This PR is the read-path
foundation: it adds the seam and read-routing but leaves the write path
to route through the same seam in a later PR. Behavior-changing where
the three fixes above touch existing read-your-writes /
error-normalization / batch-resolution paths; otherwise additive (new
store module, new resolver, injectable dependencies with existing
single-client behavior preserved when no dedicated store is configured).
## Tests
Extensive new vitest coverage under `run-store/src/*.test.ts` (routing,
residency, dual-schema select, cross-generation error normalization,
read-after-write, idempotency dedup, mixed residency, waitpoint
co-location) and `run-engine/src/engine/**/*.test.ts` (per-system
`controlPlaneResolver` tests, injectability, block-edge residency,
waitpoint read residency, trigger-create routing, lifecycle router).
Testcontainers-backed; no mocks.
## Notes
Draft, **stacked on #4114** (`runops/pr03-clickhouse-tc`). Review that
first; this diff is against it.
Server-change / changeset note to be added at stack-assembly time.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Adds a lefthook `pre-push` job that runs the same checks as CI
code-quality (`oxfmt --check .` + `oxlint .`), so formatting/lint
failures are caught locally before they reach a PR. Uses the existing
lefthook setup - no new tooling.
Caveat noted in the config: GitButler uses its own git implementation
and only runs hooks when "Run hooks" is enabled in its per-project
settings; with that off, this protects plain `git push`. Enable that
setting to have it fire on `but push` too.
`db:migrate` ran `prisma migrate deploy` for
`@internal/run-ops-database`, which requires the dedicated run-ops DB
(:5434) that isn't up in a default local — breaking local `db:migrate`
for everyone; excluding it with `--filter=!@internal/run-ops-database`
restores it.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
## Summary
Getting the CLI talking to a local instance meant the browser magic-link
login, which is no good when you're driving things headlessly (an agent,
a container, or just no browser to hand). The seed already prints dev
secret keys for the batch-limit orgs, so it now also mints a personal
access token for the seeded `local@trigger.dev` user and prints a
ready-to-run `export TRIGGER_ACCESS_TOKEN=...` next to them.
Re-seeding stays idempotent: it decrypts and reprints the existing
`local-dev-cli` token rather than piling up a new one on every run.
<!-- GitButler Footer Boundary Top -->
---
This is **part 2 of 2 in a stack** made with GitButler:
- <kbd> 2 </kbd> #4135👈
- <kbd> 1 </kbd> #4137
<!-- GitButler Footer Boundary Bottom -->
---------
Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>
<!-- ccr-slack-attribution -->
_Requested by **Eric Allam** · [Slack
thread](https://triggerdotdev.slack.com/archives/C061L2MHW93/p1783083021892389?thread_ts=1783083021.892389&cid=C061L2MHW93)_
## ✅ Checklist
- [ ] I have followed every step in the [contributing
guide](https://github.com/triggerdotdev/trigger.dev/blob/main/CONTRIBUTING.md)
- [x] The PR title follows the convention.
- [ ] I ran and tested the code works
---
## Testing
N/A — this change only removes two changeset markdown files; there are
no code changes to test.
---
## Changelog
### What changed
Removes two `@trigger.dev/core` changesets that were added for changes
that are not user-facing package changes:
- `.changeset/runops-core-residency.md` — internal run-ops residency
classifier + ksuid mint/decode primitives
- `.changeset/telnet-dev-logs.md` — dev-only
`@trigger.dev/core/v3/telnetLogServer` module
### Why
Per the convention that changesets should only be added when there are
actual user-facing package changes, these two do not qualify. Removing
them keeps the release (currently the automated v4.5.1 PR #4126) from
bumping `@trigger.dev/core` for internal/dev-only additions.
---
_Generated by [Claude
Code](https://claude.ai/code/session_019xCdLwoozZm4Gr5ZHLYiws)_
Co-authored-by: Claude <noreply@anthropic.com>
## Summary
Stable v4 Docker image builds now also publish `v4` and `latest` tags,
giving Docker-based self-hosters a maintained floating tag to use after
a stable release. The Kubernetes guide now uses the current Helm chart
line and current pinned examples, so new installs and upgrades resolve
to the 4.5 chart line instead of the 4.0 line.
## Design
The publish workflows add the floating tags only for stable `v4.x.x`
image tags. Prerelease and `main` builds keep their existing tags.
## What
- Adds `composeTaskRunVersion` to `@internal/clickhouse` (exported from
the package index). It packs a small `originGeneration` epoch into the
top 8 bits of the ReplacingMergeTree version and keeps the producer's
own LSN in the low 56 bits, so `task_runs_v2` rows replicated from more
than one Postgres producer become globally comparable while preserving
in-producer ordering. Single-producer setups never call it and keep
using the raw LSN version.
- Adds unit coverage for the helper in `taskRuns.test.ts` (bit layout,
ordering, epoch precedence, range validation).
- Adds two run-engine tests that exercise the cross-Postgres-version
testcontainer fixture:
- `heteroPostgresFixture.test.ts` — a smoke test asserting byte-identity
and identical `ORDER BY` (under a pinned ICU collation) across two
different Postgres major versions.
- `crossVersionCompat.test.ts` — mirrors the run-engine's real raw-SQL
surfaces and asserts byte-identical, ordering-identical results across
the two versions. Ships with an env-gated block (skipped by default)
that can be pointed at a real dedicated database in CI.
## Why
Third PR in the run-ops split stack. It is purely additive: it
introduces one new exported helper plus tests and changes no runtime
call sites, so on its own it has no runtime behavior change. It lays
down the version-composition primitive and the cross-version
compatibility proof that later PRs in the stack rely on.
## Tests
- New unit tests for `composeTaskRunVersion` in
`internal-packages/clickhouse/src/taskRuns.test.ts` (run against a real
ClickHouse testcontainer).
- New cross-version tests in
`internal-packages/run-engine/src/engine/tests/` running against real
Postgres containers of two different major versions (no mocks). The
env-gated dedicated-database block is skipped unless its URL is set.
## Notes
Draft, **stacked on #4113** (`runops/pr02-db-foundation`). Review that
first; this diff is against it.
Server-change / changeset note to be added at stack-assembly time.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
## Summary
`pnpm run db:seed` failed with `SyntaxError: The requested module
'./app/models/organization.server' does not provide an export named
'createOrganization'`, even though that export exists. Renaming the seed
entry
point from `seed.mts` to `seed.ts` runs it as CommonJS and fixes the
failure.
No seed logic changes.
## Root cause
`seed.mts` is an ES module, but the server modules it imports (`.ts`
files, and
no package declares `"type": "module"`) resolve as CommonJS. tsx
compiles those
to CommonJS using esbuild's getter-based export shape
(`Object.defineProperty(exports, name, { get })`), which Node's
`cjs-module-lexer` does not detect when it links the ESM importer. The
named
exports look absent, so linking throws before any code runs.
The seed only ever needed the `.mts` extension: it has no top-level
`await` and
no `import.meta`. Running it as `.ts` keeps the whole import chain
CommonJS to
CommonJS and never crosses the lexer boundary. Verified the seed runs to
completion after the change.
Possibly caused by Node 22 upgrade?
## What
The **dedicated run-ops database** foundation for the split: a
standalone Prisma package plus the infra to run and migrate it.
- **`internal-packages/run-ops-database`** — a new Prisma package
(`@internal/run-ops-database`) whose schema mirrors the run-execution
tables that will live on the dedicated DB, with its own generated
client, migrations, and migration runner.
- **`prisma/schema.parity.test.ts`** — a parity test that guards the
run-ops schema against drift from the control-plane schema for the
mirrored tables.
- **Docker** — a Postgres 17 service (`docker/Dockerfile.postgres17`,
`docker/docker-compose.yml`) so the dedicated DB is available locally
under the run-ops compose profile.
- **Testcontainers** — hetero fixtures (PG14 legacy + PG17 dedicated) so
later PRs can exercise cross-database behaviour with real containers
rather than mocks.
## Why
This is the **second PR in the run-ops split stack**, stacked on the
core primitives. It stands up the dedicated database and its tooling.
There is **no runtime wiring** into the webapp here — the app does not
read or write this DB yet; that arrives in later PRs. On its own this PR
only adds a package, a docker service, and test fixtures.
## Tests
Schema-parity test for the run-ops schema; hetero testcontainer fixture
smoke test.
## Notes
- Draft, **stacked on #4112** (`runops/pr01-core-residency`). Review
that one first; this diff is against it.
- Server-change / changeset note to be added at stack-assembly time.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
## What
Foundation for the run-ops database split: an isomorphic **id-shape
residency classifier** and the **ksuid mint primitives**, added to
`@trigger.dev/core` under `v3/isomorphic`.
- **`runOpsResidency.ts`** — classifies a run id by its shape: 25-char
cuid → `LEGACY`, 27-char ksuid → `NEW`. Pure and environment-free (safe
on both client and server).
- **`friendlyId.ts`** — ksuid mint primitives and id helpers.
- Both exported via `v3/isomorphic/index.ts`.
## Why
This is the **base of a stacked series** implementing the run-ops DB
split (routing run-execution data to a dedicated database by id-shape).
Later PRs in the series consume this classifier and these primitives to
route reads and writes across the two databases.
On its own this PR is **purely additive** — new isomorphic helpers with
unit tests, no runtime wiring, and no behaviour change to existing code
paths.
## Tests
Unit tests for the classifier (`runOpsResidency.test.ts`) and the id /
mint primitives (`friendlyId.test.ts`).
## Notes
- Draft, stacked on `main`; subsequent PRs in the series build on top of
this one.
- A changeset for `@trigger.dev/core` will be added before this is
marked ready for review.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The dashboard agent deploy runs a matrix leg per environment (staging,
prod). Running them in parallel deployed the same Trigger.dev project
twice at once, and one leg failed to create the background worker while
the other succeeded. `max-parallel: 1` runs the legs sequentially so a
deploy never races itself.
Also aligns the workflow's Node version to the repo standard (`22.23.1`,
per `.nvmrc`).
Verified: re-running the staging leg on its own (no parallel prod)
deployed cleanly.
## Summary
The dashboard agent runs as a `chat.agent` in its own Trigger.dev
project, deployed separately from the app that starts its sessions. This
adds independent, non-disruptive deploys for it: a new workflow deploys
the agent with `--skip-promotion` (so a deploy never becomes the current
version on its own), and the app pins its sessions to a chosen version.
## How it works
`DASHBOARD_AGENT_VERSION` (unset by default) pins agent sessions to a
specific deployed version; when unset, sessions run on the project
environment's current version. The pin is passed on session start and
head start and is forwarded to every continuation run, so a pinned
session stays on its version for its whole life. Cutting over to a new
build becomes a config change (set the version) rather than a redeploy,
and rollback is flipping it back.
The workflow (`dashboard-agent-deploy.yml`) runs a leg per environment
(staging and prod), each gated by its own environment, and triggers on
pushes to `main` that touch the agent or its store (also available via
manual dispatch). Deploy versions are per-environment, so each
environment pins to its own leg's version.
Follow-up to #4120, adding regression coverage for the pause/resume path
in `PauseEnvironmentService`.
Three tests against the real service with testcontainers Postgres (no
mocks), seeded org/project/environment rows, and the same
`AuthenticatedEnvironment` coercion production uses:
1. **resumes a manually paused env** - the actual regression: pause
leaves `pauseSource` null, resume must succeed. Verified this fails with
the exact pre-#4120 symptom (false billing-limit error) when the fix is
reverted locally.
2. **rejects resume of a billing-limit paused env** - the guard still
blocks manual resume while `pauseSource = BILLING_LIMIT`, and the env
stays paused.
3. **manual pause while billing-limit paused is a no-op** - returns
success without overwriting `pauseSource`, so billing-limit converge can
still find and unpause the environment.
Tests-only change, no runtime behavior touched.
Stream dev logs over a local telnet/TCP socket. `trigger dev` mirrors
its terminal output on port 6767 by default (override with
--telnet-logs-port or TRIGGER_DEV_TELNET_LOGS_PORT, 0 disables). webapp,
supervisor, and coordinator each expose an opt-in stream gated on a
per-service *_TELNET_LOGS_PORT env var. New
@trigger.dev/core/v3/telnetLogServer module (localhost-only,
backpressure-safe, plain-text) plus optional static Logger.onLog /
SimpleStructuredLogger.onLog sinks.
Then you (or your agent) can use `nc` to connect and filter out the
stream.
<img width="1103" height="239" alt="image"
src="https://github.com/user-attachments/assets/b4d47efc-8a57-4185-a159-10f2806627ae"
/>
Makes the app error page always render full screen — it previously
inherited the width/offset of whatever container the error boundary was
mounted in (e.g. the centered `max-w-xs` column in the root boundary),
so `min-h-screen` alone couldn't fill the viewport. The root container
now uses `fixed inset-0 z-50` to break out and cover the full screen
regardless of nesting.
Also changes the "Go to homepage" shortcut from `Cmd/Ctrl+G` (which
collides with the browser's native "Find Again") to `Enter`.
## Bug
Manually pausing an environment works, but resuming it always fails
with:
> This environment is paused because your organization reached its
billing limit. Resolve the limit on the billing limits settings page to
resume.
even when no billing limit is in effect. Once paused by a user, an
environment cannot be resumed at all.
## Root cause
A manual pause leaves `RuntimeEnvironment.pauseSource` as `NULL` (only
billing-limit enforcement sets `BILLING_LIMIT`). The resume path in
`PauseEnvironmentService` guards its `updateMany` with:
```ts
NOT: { pauseSource: EnvironmentPauseSource.BILLING_LIMIT }
```
Prisma's `NOT` on a nullable field translates to SQL `!=`, which
excludes `NULL` rows. So the update matches zero rows for every
user-paused environment, and the zero-count branch (meant to catch a
race with billing-limit pausing) returns the misleading billing-limit
error.
Introduced in #3996 (the guard is correct for `BILLING_LIMIT` rows; it
just also swallows `NULL`).
## Fix
Explicitly include `pauseSource: null` rows:
```ts
OR: [
{ pauseSource: null },
{ NOT: { pauseSource: EnvironmentPauseSource.BILLING_LIMIT } },
]
```
Billing-limit-paused environments are still blocked from manual resume,
both by the `getManualPauseEnvironmentResult` guard and by this clause.
## Verification
Reproduced locally: paused an environment via `PauseEnvironmentService`
(DB shows `paused = true`, `pauseSource = NULL`), resume returned the
billing-limit error with `updateMany` matching 0 rows. With the fix,
resume succeeds and the environment unpauses. Billing-paused rows remain
excluded by the same clause.
## Summary
Long task names in the task landing page side menu pushed the **Test**
button off the edge of the panel instead of truncating. The heading now
truncates with an ellipsis so the Test button always stays in view, on
the standard and agent task pages.
## Root cause
The side menu lives in a fixed-width resizable panel with `overflow:
hidden`. Its header row is a grid item, and a grid item's default
`min-width: auto` lets it grow to its content's width. The title
`<span>` uses `truncate` (`white-space: nowrap`), whose min-content is
the full, untruncated name, so the header row expanded past the panel
and the Test button was clipped off the edge.
Adding `min-w-0` to the header container lets it shrink back to the
panel width so the title truncates. The scheduled task page already had
this class; the standard and agent pages did not.
Bumps the internal/toolchain Node version to the latest 22.x LTS
(`22.23.1`) and standardises it across the repo. Scope is the **platform
toolchain + the repo's own runtime images** (all `20 → 22` *upgrades*,
off the now-EOL node 20).
### Main changes
- Node `20.20.2 → 22.23.1` across all CI workflows, `.nvmrc`,
`CONTRIBUTING.md`, and the OSS `docker/Dockerfile` (digest-pinned).
- `@types/node → 22.20.0` (root dep + pnpm `overrides`, so the whole
workspace resolves to it); lockfile regenerated.
- `sdk-compat` matrix: adds Node 24 + 26 (keeps 20, still in `engines`).
- **App runtime images → node 22** (were on EOL node 20):
`apps/coordinator` → `node:22.23.1-bookworm-slim`;
`apps/docker-provider` + `apps/kubernetes-provider` → `node:22-alpine`
(reusing the exact digest `apps/supervisor` already runs, so all four
worker images are now identical). Stage aliases renamed off `node-20`.
### Possible issues / test notes
- `@types/node` 22.x can surface new TS errors — typecheck (now on 22)
is the gate.
- **Smoke-test the v3 worker path** — `coordinator` (`crictl`/CRI calls)
and the docker/kubernetes providers (talking to their daemons) now run
on node 22 (alpine/musl for the providers). Upgrade off EOL so low-risk,
but it's deployed runtime code with its own `publish-worker.yml`
pipeline.
## Summary
Two related timezone bugs in the dashboard.
1. The date/time tooltip could show a UTC offset label that contradicted
the time it displayed. A viewer whose machine clock differs from their
saved timezone (or when a date falls in the other DST phase) would see
something like `Local (UTC +0)` next to a value that isn't at +0.
2. A user's timezone preference silently failed to save whenever their
browser reported a zone like `UTC`, `Etc/UTC`, or `Asia/Kolkata`,
leaving their timestamps stuck in a previously-saved timezone.
## Offset label
The "Local" row formatted its time using the viewer's configured
timezone but computed the `(UTC +n)` label from `new
Date().getTimezoneOffset()`, the browser's offset at the current moment.
Those are two independent sources, so they disagreed when the configured
timezone differed from the machine, and also when the displayed date was
in the opposite DST phase. The label is now derived from the same date
and timezone used to render the row (via `Intl.DateTimeFormat` with
`timeZoneName: "longOffset"`), so it always matches the displayed time.
## Preference persistence
`/resources/timezone` validated the incoming zone against
`Intl.supportedValuesOf("timeZone")`, which lists only canonical zone
ids. Browsers report zones that aren't in that list via
`resolvedOptions().timeZone`, notably `UTC` (and `Etc/UTC`,
`Asia/Kolkata`, `GMT`), so those requests returned 400 and the
preference was never stored. Validation now checks whether the runtime
can resolve the zone at all, which accepts every real zone and still
rejects invalid input.
Added unit tests for both.
## What & why
Two related fixes to how new cloud organizations get onboarded onto the
Free plan.
### 1. Route new cloud orgs through plan selection
New cloud organizations were created already activated, so they skipped
the plan-selection step and went straight to creating projects — which
meant their plan and usage limits were never set up. They're now created
deactivated and routed through plan selection, which activates them once
a plan is chosen. Self-hosted installs have no plan-selection step, so
they're activated immediately on creation and are unaffected.
The `Organization.v3Enabled` field is renamed to `isActivated` to better
describe what it now gates. It's mapped to the existing `v3Enabled`
column, so there's no data migration — only a schema/code rename.
### 2. Allow selecting the Free plan without GitHub verification
Choosing the Free plan no longer requires connecting and verifying a
GitHub account. The plan is applied immediately when selected. This
removes:
- the "Connect to GitHub" dialog and the GitHub-verified badge from the
plan picker
- the account-rejected state
- the now-unreachable GitHub-connect return routes
## Notes
- These changes pair with the corresponding change in the billing
service that applies the Free plan directly; they should be released
together.
## Testing
Verified locally end to end: a new cloud org is routed to plan
selection, the Free plan applies in one click with no GitHub step, the
org is activated, its usage allowance is provisioned, and it lands on
the new-project page.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Once this is merged, oxlint is at a pretty sensible baseline.
**Enable `no-unused-vars`, `typescript/consistent-type-imports`, and
`import/no-duplicates` lint rules**
Turns on three previously-disabled oxlint rules across the monorepo and
fixes all violations:
- **`no-unused-vars`** – enabled as an error with standard ignore
patterns: unused function arguments are ignored by default (`args:
"none"`), variables/caught errors/destructured array elements prefixed
with `_` are allowed, and rest siblings are permitted.
- **`typescript/consistent-type-imports`** – enforced as an error; all
type-only imports now use the `import type` syntax.
- **`import/no-duplicates`** – enforced as an error; duplicate import
statements from the same module have been merged.
The remaining commits clean up the violations found across the codebase:
removing unused variables/imports/type aliases, adding `_` prefixes to
intentionally unused bindings, fixing duplicate imports, and converting
value imports to `import type` where appropriate.
# Trigger.dev v4.5.0
4.5.0 is the GA of the AI Agents platform. Everything built during the
prerelease line (durable agents, Sessions, AI Prompts) is now stable on
the `latest` tag, alongside a set of SDK and runtime improvements.
## AI Agents (`chat.agent`)
Run Vercel AI SDK chat completions as durable Trigger.dev tasks instead
of fragile API routes. A conversation runs as one long-lived task keyed
on `chatId`, so it survives page refreshes, network blips, redeploys,
and crashes, and every turn is a span in the dashboard.
```ts
import { chat } from "@trigger.dev/sdk/ai";
import { streamText, stepCountIs } from "ai";
import { anthropic } from "@ai-sdk/anthropic";
export const myChat = chat.agent({
id: "my-chat",
run: async ({ messages, signal }) => {
return streamText({
...chat.toStreamTextOptions(), // system prompt, compaction, steering, telemetry
model: anthropic("claude-sonnet-4-5"),
messages,
abortSignal: signal,
stopWhen: stepCountIs(15),
});
},
});
```
## Sessions
The durable primitive underneath `chat.agent`, usable on its own: a
run-aware, bidirectional stream channel keyed on a stable `externalId`
whose `.in` / `.out` streams survive run boundaries (suspend, crash,
idle-timeout, redeploy). One Session spans many runs, which makes it a
good fit for agent inboxes and approval flows.
```ts
import { sessions } from "@trigger.dev/sdk";
// Create the session and trigger its first run (idempotent on externalId)
await sessions.start({
type: "inbox",
externalId: userId,
taskIdentifier: "inbox-agent",
});
const session = sessions.open(userId);
await session.in.send({ text: "hello" });
const stream = await session.out.read({ signal: AbortSignal.timeout(30_000) });
for await (const chunk of stream) console.log(chunk); // durable across run swaps
```
## AI Prompts
Define prompt templates as code, versioned on every deploy, and override
the text or model from the dashboard without redeploying
(environment-scoped). Each generation links back to its prompt version
for usage, cost, and latency.
```ts
import { prompts } from "@trigger.dev/sdk";
import { z } from "zod";
export const supportPrompt = prompts.define({
id: "customer-support",
model: "gpt-4o",
variables: z.object({ customerName: z.string(), issue: z.string() }),
content: `You are a support agent for Acme.
Customer: {{customerName}}
Issue: {{issue}}`,
});
// Honors any active dashboard override, else the current deployed version
const resolved = await supportPrompt.resolve({ customerName: "Alice", issue: "Can't log in" });
// resolved.text, resolved.model, resolved.version
```
## `useChat` integration
`useTriggerChatTransport` is a Vercel AI SDK `ChatTransport` that runs
`useChat` over Trigger.dev realtime with no API routes. Text, tool
calls, reasoning, and `data-*` parts stream natively, and it works with
AI SDK v5, v6, and now v7.
## First-turn fast path (`chat.headStart`)
Runs the first turn in your warm server process while the agent boots in
parallel, cutting cold-start time-to-first-chunk roughly in half
(measured ~2.8s to ~1.2s). Available via the new
`@trigger.dev/sdk/chat-server` subpath.
## Human-in-the-loop, stop, and steering
The agent control surface: tool approvals (`needsApproval` +
`addToolApprovalResponse`), client-driven stop-generation, mid-execution
steering (`pendingMessages`), and between-turn context injection
(`chat.inject` / `chat.defer`), all durable across the conversation.
## Agent Skills
`skills.define({ id, path })` bundles a `SKILL.md` folder into your
deploy image. The agent gets a one-line summary up front and loads the
full instructions plus scoped `bash` / `readFile` tools on demand
(progressive disclosure), so a capability is something the model reaches
for rather than a pre-declared typed tool.
## `trigger skills` for coding assistants
`trigger skills` installs version-pinned Trigger.dev skills plus a
bundled docs snapshot into Claude Code, Cursor, GitHub Copilot, and
Codex, so your assistant's Trigger.dev knowledge stays current with your
installed SDK version. `trigger init` now offers to set up the MCP
server and skills too.
## Model library
A new Models page in the dashboard: a catalog of models grouped by
provider with context window, capabilities, and input / output pricing
per 1M tokens, plus a "Your models" tab showing per-model usage, cost,
and cache-hit sparklines from your actual traffic.
## Dev branches
Run multiple local `trigger dev` sessions in parallel (separate git
worktrees or coding agents) without runs colliding, each isolated with
its own dashboard, via `trigger dev --branch <name>`.
## `TriggerClient`
An instantiable client so one process can trigger and read across
projects, environments, and preview branches, each with its own auth and
baseURL, with no shared global state.
```ts
import { TriggerClient } from "@trigger.dev/sdk";
const prod = new TriggerClient({ accessToken: process.env.TRIGGER_PROD_KEY });
const preview = new TriggerClient({
accessToken: process.env.TRIGGER_PREVIEW_KEY,
previewBranch: "signup-flow",
});
await prod.tasks.trigger("send-email", { to: "user@example.com" });
await preview.runs.list({ status: ["COMPLETED"] });
```
## SDK and runtime
- AI SDK 7 support (v5 and v6 still supported), with OpenTelemetry
telemetry auto-wired
- Large trigger-payload offload: trigger payloads at or above 128KB
upload to object storage automatically, using the same auth and baseURL
as the trigger call
- Region support on the runs API: filter runs by region and read each
run's executing region (also on MCP `list_runs`)
- Duplicate task-id detection: `dev` and `deploy` fail with a clear
error instead of silently overwriting
- `envvars.upload` gains an `isSecret` flag to import redacted secret
variables
- Retry hardening: `TASK_MIDDLEWARE_ERROR` now retries under the task's
retry policy
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Removes the release-candidate framing from the docs for the 4.5 GA: the
`@rc` install caveats on the `skills` CLI command, and the AI Agents and
Prompts release-candidate banner (a shared snippet used across the
ai-chat and prompts pages) plus the `>=4.5.0-rc.0` compatibility pin in
the AI reference.
## Summary
Adds the `4.5.0` GA entry to the AI chat agents changelog covering the
`chat.agent` changes in that release: a new `apiClient` option on
`chat.headStart` and `chat.createStartSessionAction` for pointing chat
sessions at a different project or environment, the fix for messaging
chat agents deployed to a preview branch, and the fix for Head Start
handovers when the agent also defines a `prepareMessages` hook.
All four changes are from
[#4018](https://github.com/triggerdotdev/trigger.dev/pull/4018).
Bumps the `@remix-run/*` family in the webapp from 2.17.4 to 2.17.5 to
keep dependencies current.
2.17.5 pulls `@remix-run/router` 1.23.2 → 1.23.3, so the local
route-matching perf patch was rebased onto 1.23.3 (regenerated via `pnpm
patch`). It is functionally identical to the previous one - the only
difference is that 1.23.3 already hoists `decodePath` out of the match
loop upstream, so that hunk is dropped; the per-route-tree branch cache
and the compiled-path cache are unchanged. Also updated the
`@remix-run/dev>tar-fs` override key to track the new dev version.
Verified locally against latest main: `typecheck --filter webapp`
passes, `--frozen-lockfile` is consistent, and the dev server boots and
server-renders pages cleanly (route matching exercised via the patched
router).
## ✅ Checklist
- [ ] I have followed every step in the [contributing
guide](https://github.com/triggerdotdev/trigger.dev/blob/main/CONTRIBUTING.md)
- [x] The PR title follows the convention.
- [ ] I ran and tested the code works
---
## Testing
`pnpm run typecheck --filter @internal/clickhouse` passes. This only
adds a ClickHouse input-format setting to existing insert calls; the
setting affects type inference for newly-inserted/merged data and is
non-destructive to existing rows.
---
## Changelog
Sets
`input_format_json_infer_array_of_dynamic_from_array_of_different_types
= 1` on every native-JSON insert path:
- `task_runs_v2` (`output`, `error`) — `insertTaskRuns`,
`insertTaskRunsCompactArrays`, and the async-insert variants
- `task_events_v1` / `task_events_v2` (`attributes`)
- `metrics_v1`
- `sessions_v1`
### Why
Our JSON columns contain arrays with mixed element types (e.g.
`[{"key":"value"}, "string", "string"]`). With this setting off — which
is the effective default under `24.12` compatibility — ClickHouse infers
those as deeply nested unnamed `Tuple(JSON, Nullable(String), …)` types.
ClickHouse 26.2 introduced `input_format_binary_max_type_complexity`
(default 1000), and those tuple type trees exceeded the limit, causing
background merges to fail with **Code 117**.
With the setting on (the default since 25.8), mixed-type arrays are
inferred as a single `Array(Dynamic)` — a simpler, flatter type
representation that never approaches the complexity limit, even once the
upstream default limit is restored.
Setting this explicitly at insert time keeps behavior deterministic and
version-controlled, so it does not depend on the server profile or a
future compatibility bump. This is a forward-only change: it only
affects newly inserted/merged data and does not rewrite existing parts.
Our read path re-serializes these columns to strings (`toJSONString` via
the materialized `*_text` columns), so the internal Tuple →
Array(Dynamic) representation change is transparent to the application.
### Companion server-side setting
To also apply this on the ClickHouse side (covers merges and any writes
not going through these code paths), set it on the default user:
```sql
ALTER USER default SETTINGS input_format_json_infer_array_of_dynamic_from_array_of_different_types = 1;
```
---
## Screenshots
_N/A_
💯🤖 Generated with [Claude Code](https://claude.com/claude-code)
https://claude.ai/code/session_01AaChyhestFMBYBWh6bgcCF
---
_Generated by [Claude
Code](https://claude.ai/code/session_01AaChyhestFMBYBWh6bgcCF)_
---------
Co-authored-by: Claude <noreply@anthropic.com>
## Summary
Fixes flaky CLI v3 E2E tests by removing fixture-level parallelism.
The suite was using `describe.concurrent`, but each fixture test mutates
its fixture workspace during setup: removing `node_modules`,
renaming/restoring lockfiles, and running package installs. On
Windows/npm this can race or hit file-lock/cache contention, causing
intermittent failures.
## Fix
Run the CLI v3 E2E fixtures serially instead of concurrently.
## Expected impact
This should make the Windows/npm E2E job more stable. The E2E step may
slow down from ~30s to roughly ~60–90s in typical runs, with a
conservative upper bound around ~2 minutes.
Fixes#4046
## Problem
When a single run creates more than 1000 idempotency keys (e.g. a large
batch trigger where each item calls `idempotencyKeys.create()`), the
original key and scope metadata is silently dropped for all but the most
recent 1000 keys.
`idempotencyKeys.create()` returns a plain 64-char hash and stores the
`{ key, scope }` mapping in an in-process catalog keyed by that hash.
That catalog was a fixed-size **LRU capped at 1000 entries**. Once a run
creates more than 1000 keys, the earliest mappings are evicted, so when
the SDK later looks them up to attach `idempotencyKeyOptions` to the
trigger call, it finds nothing and sends `undefined`. The affected runs
then:
- report `ctx.run.idempotencyKey` as the raw hash instead of the
user-provided key
- have no `idempotencyKeyScope`
- show empty `idempotency_key` / `idempotency_key_scope` in the
dashboard and analytics
Deduplication still works (the hash is intact); only the human-readable
metadata is lost, which makes the failure silent and hard to notice.
## Fix
- Replace the LRU catalog with an unbounded in-memory catalog, so every
key created within a run keeps its metadata regardless of how many are
created.
- Clear the catalog at each run boundary via
`resetExecutionEnvironment()` (both dev and managed workers), matching
how every other per-run manager is reset. Deployed workers reuse one
process across many runs (warm starts), so this bounds memory to a
single run's keys instead of accumulating across runs — which is the
reason the size cap existed in the first place.
## Tests
- New public-API test creates 3000 keys and asserts all of them
(including the first) retain their key/scope — this fails on `main` and
passes with the fix.
- New test for the in-memory catalog covers store/retrieve/overwrite,
large-N retention (no eviction), and `clear()`.
- New test asserts the catalog is emptied after a run-boundary reset.
- Replaces the previous LRU catalog + its eviction tests.
Verified: `@trigger.dev/core` and `trigger.dev` both build; all
idempotency tests pass. Changeset added (patch).
The `Exec` helper in `@trigger.dev/core` logs command args at debug
level (and in its output/error metadata). For commands that take a
credential directly on the command line - `--password`, `--token`,
`--secret`, etc. - that value is logged verbatim, so turning on debug
logging can surface secrets in log sinks.
This masks the value of known credential-bearing flags (both `--flag
value` and `--flag=value` forms) before the args are logged. The
executed command is untouched - only the logged copy is redacted. Added
a small unit test for the redaction helper.
Adds a `redis_worker.queue.oldest_message_age` observable gauge (labeled
`worker_name`) and `SimpleQueue.oldestMessageAge()`, reporting the age
of the
oldest overdue message in each queue. Generic queue-stall signal: 0
while a
queue drains healthily, rising only when due work sits undrained
(blocked
dequeue, dead consumer, backpressure) — even when no items are being
processed.
## Summary
Upgrades the dashboard form layer from `@conform-to` 0.9 to 1.x. No
behaviour change is intended; this is the conform API migration only.
conform 1.x peer-depends on `zod` `^3.21 || ^4`, so it runs on the
current zod 3 and is a prerequisite for upgrading the repo to zod 4:
conform 0.9 imports `ZodNativeEnum`, `ZodEffects`, and `ZodPipeline`,
all removed in zod 4, so the webapp cannot build against zod 4 until
conform is on 1.x. Landing this first (on zod 3) keeps the zod 4 PR
focused on zod alone.
## Testing
Did a bunch of local smoke tests and E2E playwright tests, several
rounds of different reviewers, all clean.
Invite acceptance could fail for cloud organizations with many projects
because the whole flow ran inside a single transaction and did too much
work before it completed. In larger orgs, that pushed the transaction
past its timeout and blocked the invite from being accepted.
This PR moves the expensive parts of invite acceptance out of the
transaction, excludes deleted projects from environment setup, fixes
error handling on /invites, and adds regression coverage for the failure
cases.
Fixed trace rendering for child and nested runs in large traces.
Dashboard and trace API responses now load the requested run's trace
subtree instead of depending on the run span appearing in the initial
trace slice.
## Summary
Two robustness fixes in the dashboard's error handling, found while
testing the billing-limit pause/resume flow.
## Toast cookie overflow
Toast messages are flashed into the `__message` session cookie, which
the session store rejects once the serialized cookie passes the
browser's ~4KB limit. Any call site that flashes a raw caught error (a
verbose database or validation message, for example) could turn a toast
into a failed request. `setErrorMessage` / `setSuccessMessage` now clamp
the message length, so a toast can never overflow the cookie. This
protects every toast helper at once.
## Pause/resume reporting
Resuming an environment that is paused by a billing limit is an
expected, user-actionable state, but `PauseEnvironmentService` threw it,
and the service's catch reports every throw at error level. It now
returns that case as a failure result, so callers still surface the
message to the user while genuine errors keep reporting.
## Summary
When a scheduled task fires for an organization that is out of
entitlements, the trigger can't proceed. That's an expected outcome, but
it was being logged at error level and surfaced as a failure.
## Fix
The trigger callback now classifies an out-of-entitlements result as its
own error type (`OUT_OF_ENTITLEMENTS`), and the schedule engine logs
both that and the existing queue-limit result as warnings rather than
errors. The run still doesn't fire and the `schedule_execution_failure`
metric still records the outcome (now tagged `out_of_entitlements`), so
nothing about observability or behavior changes beyond the log level.
## Summary
The billing limits page and the usage-limit banners now require a
dedicated `manage:billing-limits` permission instead of the broader
billing permission. This lets a role be granted control over billing
limits independently of subscription and payment management.
## Details
Both the loader and action of the billing limits settings page check
`manage:billing-limits`. The "Configure billing limit" and "Resolve"
actions in the limit banners (the no-limit-configured, grace, and
rejected states) gate on the same permission. The subscription and
billing pages keep using the existing billing permission.
## Summary
When replaying runs in bulk from a deployed environment, you can now
choose which region the replayed runs run in. The bulk action inspector
shows an "Override region" dropdown that defaults to "Don't override",
which keeps each run in its original region, so replaying a selection
that spans multiple regions doesn't silently re-route anything. Pick a
region and every matched run is replayed there instead.
The dropdown only appears for the replay action in a deployed
environment with more than one region available; cancel actions and
development environments don't show it.
## Design
The selected region is carried through the bulk action as a dedicated
`replayRegion` param, kept separate from the run-list selection filters
so it can't be confused with a region selection filter. When the action
runs, each replay passes it through to the existing region override on
the replay service, which already falls back to each run's original
region when no override is set. "Don't override" is a sentinel value
that the action normalizes away so the service only ever sees a real
region or nothing.
---------
Co-authored-by: Eric Allam <eric@trigger.dev>
## Summary
Improves and unifies the run-activity charts by extracting a shared set
of chart primitives and adopting them on the three task landing pages
(agent, standard, scheduled), with the density and label fixes also
carried over to the dashboard and custom query charts.
Main changes a reviewer should know about:
- **Shared primitives (DRY).** New `ChartCard` (title +
maximize/fullscreen), `ChartSyncContext` (cross-chart hover + zoom
state), `useXAxisTicks` (width-aware tick selection),
`activityTimeAxis`, and `statusColors`, plus a server-side
`activitySeries.server.ts` holding `chooseBucketSeconds`, status
grouping, and the zero-fill helpers. The three task routes and both
presenters were refactored onto these, removing roughly 3x duplicated
tick logic, status-color tables, and bucket-ladder code.
- **Denser bars on short ranges.** Server-side bucketing now uses
`chooseBucketSeconds` (nice-interval ladder, ~72 target, capped at 120
buckets) instead of the hardcoded 1h/6h/1d ladder, so a 5m or 1h range
no longer collapses into a single bar.
- **Width-aware x-axis labels.** Labels are selected to fit the measured
plot width (always first + last, evenly spaced, de-duplicated by
rendered text), stay horizontal, and reflow on panel/window resize.
Y-axis values default to compact form (8K, 1.2M) in
`ChartBar`/`ChartLine`.
- **Synced hover line.** Hovering one of the agent page's three charts
draws a dashed vertical line at the same bucket on the *other* two, and
suppresses it on the hovered chart. It is opt-in via
`ChartSyncProvider`, so single-chart pages are unaffected.
- **Maximize button.** Each chart gets a fullscreen dialog toggle
(reuses the existing dashboard-widget pattern, `v` shortcut while
hovered).
- **Drag-to-zoom on task pages.** Dragging across a task chart sets the
Time/Date filter (`from`/`to` URL params, clearing `period`/`cursor`),
with a From/To tooltip shown during the drag.
- **Custom query charts.** Long categorical x labels (run IDs, task
names) middle-truncate and auto-rotate only when needed, and label
thinning is now width-aware for both bar and line variants. Dashboard
line-chart label density is also width-aware, tuned by a
`TIME_AXIS_LABEL_SPACING_PX` constant.
- **Tests.** 46 new unit tests for the pure logic (bucket selection,
tick spacing, time-axis formatting, zoom range, truncation).
## Intentionally unchanged
- **No click/drag zoom on the dashboard or custom query charts.**
Drag-to-zoom is wired up on the task landing pages only; zooming the
dashboard and custom charts is deliberately deferred to a separate
follow-up PR. A plain click (without a drag) on a task chart is a no-op.
- **The 25 mini activity charts on the Task list (`_index`) page are
untouched.** They are hand-rolled raw-Recharts sparklines kept
deliberately lightweight and do not use these primitives.
- **Other raw-Recharts sparklines are untouched** (the usage sparkline,
errors and prompts pages).
- **No ClickHouse query semantics changed** beyond the bucket-interval
parameter (same filters, same FINAL / `_is_deleted` handling).
- **Webapp-only.** No public package (`packages/*`) changes, so there is
no changeset; the `.server-changes/` entries cover it.
---
## Testing
Added 46 unit tests covering server-side bucket selection, width-aware
tick spacing, time-axis formatting, zoom-range math, and categorical
label truncation (`pnpm --filter webapp run test`), and `pnpm run
typecheck --filter webapp` passes. Manually exercised each task landing
page (agent, standard, scheduled) plus the dashboard and custom query
charts, stepping the Date/Time filter through 5m, 1h, 24h, 7d, and 30d
to confirm dense bars on short ranges, non-overlapping labels that
reflow on resize, the synced hover line across the agent charts, the
maximize button, and drag-to-zoom updating the filter.
---
## Changelog
The activity charts on the task landing pages and the dashboard and
custom query charts now share one set of reusable primitives. X-axis
labels are width-aware so they never overlap and reflow when a panel
resizes, y-axis values are abbreviated (8K, 1.2M), and short time ranges
render dense bars instead of collapsing into a single bar. Hovering any
agent chart mirrors a vertical line on the others, every chart gains a
maximize button, and dragging across a task chart zooms the Time/Date
filter. Long categorical labels such as run IDs and task names
middle-truncate and auto-rotate only when needed.
---
https://github.com/user-attachments/assets/6be09e38-3a0e-4947-b6e3-4839daa2fbe0
## Summary
Adds a single env flag, `DEPRECATE_V3_ENABLED` (default off), that
gracefully winds down the v3 engine (`RunEngineVersion.V1`). While it's
off nothing changes, so self-hosted instances still on v3 keep working.
When it's on:
- Triggers that resolve to v3 are rejected with a clear, actionable
error pointing at the [v4 migration
guide](https://trigger.dev/docs/migrating-from-v3), instead of silently
creating runs that never execute. This covers single triggers, batches,
scheduled fires, replays, and `triggerAndWait`, which all funnel through
one place.
- The legacy `trigger dev` websocket used by v3 CLIs is closed with an
upgrade message (v4 CLIs use a different dev transport).
- The v3 shared-queue consumer refuses to start, so no deployed v3 runs
are dequeued.
- The v3 run-lifecycle background jobs (heartbeat timeout, TTL expiry,
retry, resume batch/dependency, delayed-run enqueue, and scheduled
fires) become no-ops, so abandoned v3 runs stop generating database
load.
This builds on the existing deploy deprecation flag, which already
rejects v3 CLI deploys.
## Design
Enforcement is read through one helper, `isV3Disabled()`. Every gate
combines it with a per-run or per-project engine check (`isV3Disabled()
&& engine === "V1"`), so a v4 run that happens to reach a shared service
behaves exactly as before. v4 (V2) is never affected.
The flag is a hard switch, not a drain: when it's on, in-flight v3 runs
are abandoned in place rather than failed or expired, which is the
intended behaviour for the final shutdown.
## Summary
The `skills` installer command (`npx trigger.dev@latest skills`) ships
in the release candidate but is not yet on the stable release, so
running it with `@latest` fails today. Adds a warning on each docs page
that shows the command, telling users to run it with the `@rc` tag until
it lands in `@latest`.
The commands themselves stay on `@latest`, so nothing needs reverting
once the next stable release ships, just the warnings.
Pages updated:
- `/skills`
- `/mcp-agent-rules`
- `/building-with-ai`
## Summary
Adds Billing Limits to the webapp.
Customers can set a monthly spend cap. When usage crosses the limit,
billable environments enter a grace period. If the limit is not resolved
before grace expires, new triggers are rejected until the organization
increases or removes the limit.
The Tasks page logged a burst of React hydration errors (#421, "this
Suspense boundary received an update before it finished hydrating") on
every load, one per task row for the Running and Activity cells. The
page still worked, but it spammed the console.
The Running and Activity (24h) cells stream in via Remix `defer()` +
`<Suspense>`/`<Await>`, two boundaries per row. A streamed boundary
stays in React's "hydrating" state until its data arrives; if the
backing queries are slow enough that the data is still in flight after
the page loads, a normal background re-render (a server-sent-events
update, a panel layout effect, a revalidation) hits the boundary and
React bails it to client rendering and throws #421. With N rows that is
2N errors. It never reproduced locally because those queries return
instantly there.
Fix: wrap the two cells in `ClientOnly` so they mount after hydration.
The stats still load asynchronously (the task list renders immediately),
but there is no longer an SSR boundary to bail. In the slow-query case
those cells already client-rendered (that was the bail); this just makes
it explicit and silent.
Verified by simulating slow stat queries against a local build: the
errors go from 2-per-row to zero, and the cells render correctly once
the data resolves.
The supervisor image build has been failing since `@trigger.dev/core`
gained
an `ai` peer dependency. `turbo prune` (2.5.4) generates a pruned
lockfile
that references the `ai@6.0.116(zod@3.25.76)` snapshot without including
the
entry itself, which causes `pnpm fetch --frozen-lockfile` to abort.
Bumping to 2.10.0 fixes the pnpm v9 peer dep snapshot pruning. Updated
both
Containerfiles for consistency.
Example failure here:
https://github.com/triggerdotdev/trigger.dev/actions/runs/28225353375/job/83618124564
Broken since:
c06005b3
## Summary
Adds a dedicated, always-on Performance section to the repo PR review
guide (`.claude/REVIEW.md`), so every review weighs new work against
table size, hot paths, and how deep or wide the data it walks can get.
It names the tables to treat as huge (the `TaskRun` family in Postgres,
`task_events_v1`/`v2` in ClickHouse), the hot paths that warrant extra
scrutiny (trigger and batch trigger, dequeue, execution-snapshot writes,
OTEL ingestion, trace and run-list reads), the deep and wide shapes that
turn one run into a large tree or batch, and five named anti-patterns
with severities: per-level re-scans, dropping the partition-pruning
predicate, unbounded `IN` lists, sequential per-level round-trips, and
losing the single-query fast path.
Co-authored-by: Matt Aitken <matt@trigger.dev>
## Summary
The in-dashboard agent's datastore now runs migrations over a direct
(non-pooler) connection. A transaction-mode pooler can't run the
migrator (no advisory locks, no multi-statement DDL), so when the
agent's database sits behind a pooler the migration step needs a
separate direct connection.
The application keeps connecting over the pooled
`DASHBOARD_AGENT_DATABASE_URL`. Only the migration entry points changed
(`drizzle.config.ts`, `migrate.mjs`, `migrate-status.mjs`); the runtime
client is untouched.
## Connection resolution (migrations)
```
DASHBOARD_AGENT_DIRECT_URL direct agent connection (used for migrations)
DASHBOARD_AGENT_DATABASE_URL pooled agent connection (preserves current behavior)
DIRECT_URL main direct connection (single-database fallback)
DATABASE_URL last resort
```
Mirrors the existing `DATABASE_URL` / `DIRECT_URL` split. Fully
backward-compatible: with nothing new set, resolution is identical to
before. The agent-specific vars take precedence over the main
`DIRECT_URL`, so a separate agent database is never migrated against the
wrong one. When the agent falls back to the main single database,
migrations now prefer its direct connection.
<img width="2400" height="1794" alt="chat-ui-closed"
src="https://github.com/user-attachments/assets/35016a72-c6d2-4b6a-8760-b1da5b4a166c"
/>
<img width="2400" height="1794" alt="chat-ui-open"
src="https://github.com/user-attachments/assets/25b3df60-9aa2-40fd-a1ba-1447eeee4c52"
/>
## Summary
The in-dashboard agent was launched from a button pinned to the
bottom-right of every page, which floated over page controls (for
example the run inspector's action bar). It now opens from a compact
"Chat" button on the far right of the page header, and the same button
toggles to "Collapse" while the panel is open. The panel and launcher
are labelled "Chat" in the UI.
The launcher only renders on env-scoped pages where the agent is enabled
(same feature flag gating), so it stays hidden for everyone who doesn't
have it. The existing "Ask AI" support button is untouched; it stays in
place until the agent is turned on by default.
## How it works
`DashboardAgent` (env layout) shares the open/close state through a
small context, and `NavBar` renders a launcher that self-hides whenever
that context is absent. No floating overlay, and the launcher can't
appear on pages where the agent can't open.
## Summary
The in-dashboard agent button was rendered for all admins and
impersonators regardless of the `hasDashboardAgentAccess` flag, so it
appeared even where the agent is disabled (for example, floating over
the run inspector controls). It is now gated by the flag for everyone,
so it stays hidden until the flag is turned on.
## Rollout
Both levers default off, so nothing changes for users until deliberately
enabled:
- **Per-org:** set `hasDashboardAgentAccess` on an org's feature flags
to enable the agent for just that org.
- **All admins:** set `DASHBOARD_AGENT_ADMIN_PREVIEW=1` to give admins
and impersonators an everywhere-preview, independent of the per-org
flag.
Previously admins bypassed the flag unconditionally, which is why the
button showed up before the agent was ready to ship.
Closes this feature request:
[https://triggerdev.featurebase.app/p/isolated-dev-sessions-for-multiple-local-trigger-dev-instances](https://triggerdev.featurebase.app/p/isolated-dev-sessions-for-multiple-local-trigger-dev-instances)
### Feature notes:
- CLI `trigger dev` works as before
- `trigger dev --branch my-branch` to create a new branch and run
against it.
- `trigger dev archive --branch my-branch` to archive (or in webapp).
- New webapp page to manage and archive dev branches, currently feature
flagged.
### Implementation details:
- No changes to data model, no backfill. `isBranchableEnvironment`
column is ignored for dev branches, we use `parentEnvironmentId IS NULL`
instead.
- `x-trigger-branch` overloaded for preview and dev branches
- New `TRIGGER_DEV_BRANCH` env var available locally.
`TRIGGER_PREVIEW_BRANCH` overloaded for child runs.
- Lots of new glue code to sanitise the branch checks.
### Rollout
- Deploy webapp/API changes (all backwards compatible)
- Manual tests on some orgs
- Deploy docs, release CLI, flip feature flag for webapp feature
### NB
- `api.v1.projects.$projectRef.environments.ts` will return
`isBranchableEnvironment: true` for all dev environments.
### Prerequisites
- [x] Typecheck will not pass until we make a new release of
`@trigger.dev/platform` and bump it here
A deployment could be marked deployed and promoted to current without
its image ever landing in the registry. Finalize trusted the CLI: the v1
path never pushed or checked, and the v2/v3 path skips its own push when
the CLI sends `skipPushToRegistry` - which the local-build path always
does. In the happy path the CLI pushes the image itself, so this stayed
latent. But any deviation - `--no-push`/`--load`, a push that lands in a
different registry, or an old CLI - promoted a version whose image can't
be pulled, so every run failed at pull time while the deploy itself
reported success.
This adds a registry existence check after push and before finalize. If
the image isn't there, the deploy fails loudly instead of promoting a
version that can't start. The check is ECR-only (a no-op for other
registries, so self-hosted setups are unaffected) and uses
`BatchGetImage`, which the deploy role already allows. It fails open on
an ambiguous registry error so the check can't itself turn into a deploy
outage. The image reference is the platform-generated value and the
lookup is bound to the configured registry host; the CLI-supplied digest
is validated before use.
Can be turned off with `DEPLOY_IMAGE_VERIFICATION_ENABLED=0` for setups
that push images out of band (e.g. an air-gapped registry the platform
can't reach).
refs TRI-11243
SSO settings page: resolve plan before the role check. A non-Enterprise
org now renders the upsell state for every role instead of showing a
"permission denied" panel to non-Owners for a feature their org can't
use yet. manage:sso is only enforced once the org is actually entitled.
Extracts EMPTY_SSO_STATUS and uses throwPermissionDenied().
Also removes the client-side SSO session fetch guard. It monkeypatched
global window.fetch, which made it the initiator of every request and
obfuscated the real call site on any 4xx/5xx. Session revocation is
still enforced server-side on every authenticated request and surfaces
as a logout redirect on the next navigation/refresh, so the client guard
was UX-only and not worth the cross-cutting cost.
The wait.for() idempotency example passed the idempotency options as a
second argument, but wait.for() takes a single options object — so the
second object was silently ignored and the key never applied. Merged the
fields into the single options object, matching the wait.until()
example.
Supersedes #4021.
Prefixes the dashboard feedback form thread titles with `Web app:` so
support inbox threads coming from the in-app contact form are easy to
tell apart from those submitted on the marketing site, which previously
shared an identical `Contact form:` prefix.
## Summary
The dashboard agent's database migrations could be silently skipped when
its database is shared with another Drizzle application, leaving the
`trigger_dashboard_agent` schema uncreated and a later migration failing
with `schema "trigger_dashboard_agent" does not exist`.
## Root cause
Drizzle's migrator decides what to run by reading the most recent row
from its journal table by `created_at`, and skipping any migration dated
at or before it. The dashboard-agent runner used Drizzle's default
journal table, `drizzle.__drizzle_migrations`, which every Drizzle app
shares by default. When the database is shared, another app's journal
row dated between two of our migrations makes the migrator treat the
earlier one (the `CREATE SCHEMA`) as already applied and run a later one
against a schema that was never created.
## Fix
- Track the dashboard-agent migrations in a dedicated journal table
(`drizzle.__dashboard_agent_migrations`), in both the deploy runner
(`migrate.mjs`) and the drizzle-kit config, so its history is
independent of any other Drizzle app sharing the database. The table
stays in the `drizzle` schema so the first migration's `CREATE SCHEMA
"trigger_dashboard_agent"` does not collide with it.
- Make the first two migrations idempotent (`CREATE SCHEMA/TABLE/INDEX
IF NOT EXISTS`) so databases that already tracked them under the old
journal table re-run cleanly after the rename instead of erroring on the
bare `CREATE SCHEMA`.
Verified against a Postgres seeded to reproduce the skip: the old
default-table path fails as above, the dedicated-table path creates the
schema and all tables, and re-running on an already-migrated database is
a clean no-op.
## Summary
The auto-generated changeset release PR (`changeset-release/main`)
builds its `## Improvements` / `## Bug fixes` summary with
`scripts/enhance-release-pr.mjs`. The script deduplicated summary
entries by PR number, so when a single PR shipped more than one
changeset, only the first entry survived and the rest were silently
dropped from the summary. The dropped entries still appeared in the raw
`<details>` block, which is how the mismatch surfaced (for example in
[#3998](https://github.com/triggerdotdev/trigger.dev/pull/3998), where
one PR's four changesets showed up as a single summary line).
## Fix
Deduplicate on the full entry text rather than the bare PR number. The
entry text embeds the PR link, so:
- the same changeset rendered once per package section still collapses
to one,
- distinct changesets from the same PR are each kept,
- identical descriptions from different PRs stay separate.
Verified against the raw changeset output from
[#3998](https://github.com/triggerdotdev/trigger.dev/pull/3998): that
PR's changesets went from 1 to all 4 in the generated summary.
## Summary
Adds a `RUN_ENGINE_DEQUEUE_DISABLED_WORKER_QUEUES` setting that refuses
worker dequeue requests for the listed worker queues (or base regions),
so their runs stay queued instead of being handed to workers that can't
run them. Blocked dequeues are counted via a
`run_engine.dequeue.blocked` OTel counter (labeled by `worker_queue` and
`region`).
## Summary
The `runqueue.workerQueue.length` gauge only reported a worker queue's
depth while runs were being dequeued from it. When dequeues stop, the
metric goes stale or missing, so a queue that has backed up because
nothing is draining it can't be alerted on. This adds a small observer
that refreshes the observed set of worker queues from the
`WorkerInstanceGroup` records on an interval, so every active worker
queue (and its scheduled split variant) keeps reporting its length
regardless of dequeue activity.
The observer is off by default and enabled per service via
`RUN_ENGINE_WORKER_QUEUE_OBSERVER_ENABLED`, reads from the read replica,
and skips a configurable set of cloud providers
(`RUN_ENGINE_WORKER_QUEUE_OBSERVER_EXCLUDED_CLOUD_PROVIDERS`, default
`digitalocean`). When enabled it is the source of truth for the observed
set, so the per-dequeue registration is skipped on that instance, and it
groups by worker queue so the per-instance duplicates collapse to the
true depth.
Also removes the unused `GET`/`POST /api/v1/workers` endpoints. Their
only consumer was a CLI command group that is no longer registered.
## Verification
Verified end to end against a local stack: the gauge reports each worker
queue's length with no dequeues happening, excludes the configured
providers, includes hidden groups, and the removed endpoints return as
if they never existed. Added a run-engine test
(`workerQueueObservation.test.ts`).
## Summary
Adds an in-dashboard AI agent: a chat panel, reachable from any
environment
page, that answers questions about your runs, errors, tasks, and
analytics,
diagnoses why a run failed, charts your data, reads your connected
repo's
source, and answers product and how-to questions. It is gated behind the
`hasDashboardAgentAccess` feature flag (global or per-org, default off),
so
this PR ships disabled: the launcher is hidden unless the flag is
enabled.
## Design
The agent runs as a standalone `chat.agent` Trigger task in its own
internal
package, with no access to the webapp database, Prisma, or ClickHouse.
It reads
the user's data over the public API, acting as the user via a
short-lived
delegated user-actor token minted server-side each turn (never in the
browser),
building on
[#3997](https://github.com/triggerdotdev/trigger.dev/pull/3997). The
error and analytics tools use
[#4005](https://github.com/triggerdotdev/trigger.dev/pull/4005)
and the TRQL query API.
The first turn of a new chat streams from a warm webapp route (Head
Start) while
the durable agent boots in parallel. Structured answers (a run-failure
diagnosis
card, a live chart) render through a small typed view catalog rather
than
arbitrary markup. A knowledge lane forwards product and how-to questions
to the
support assistant.
Conversation history lives in a separate Drizzle-backed store on its own
Postgres schema, kept as a display read-model so it can never corrupt
the
agent's model context.
The SDK changes add an `apiClient` option to
`chat.createStartSessionAction` and
`chat.headStart`, and keep the Head Start tool-approval tail intact
across a
custom `prepareMessages` hook so prompt caching and Head Start compose.
Adds an in-process backpressure signal that pauses dequeuing when the
Kubernetes cluster is saturated, so work overflows cheaply in the queue
instead of piling up as unschedulable pods. Saturation is read by
scraping the apiserver's total pod-object count
(`apiserver_storage_objects{resource="pods"}`) and applying an
engage/release threshold with hysteresis - a single lightweight
aggregate scrape, not a pod listing.
Backpressure sources are now evaluated independently and OR'd: each
source has its own enable and dry-run flag, and the supervisor engages
if any enabled source trips. This adds the pod-count source alongside
the existing one without changing it, and is extensible to more sources
later. Off by default.
The scrape uses the in-cluster kubeconfig over `https` so TLS verifies
against the cluster CA (the fetch-options helper attaches the CA as an
`https.Agent`, which the global `fetch` ignores - that path silently
dropped the CA). Enabling the pod-count source requires the supervisor's
service account to be granted `get` on the `/metrics` non-resource URL;
that RBAC and the per-deployment env wiring are operator-side and live
elsewhere.
New config (pod-count source):
`TRIGGER_DEQUEUE_BACKPRESSURE_POD_COUNT_ENABLED` (default false),
`_POD_COUNT_DRY_RUN` (default true), `_POD_COUNT_ENGAGE` /
`_POD_COUNT_RELEASE` (hysteresis thresholds), `_POD_COUNT_REFRESH_MS`
(scrape interval, default 5s). The existing source's flags are
unchanged.
Observability: a `supervisor_cluster_pod_count` gauge, and the pod-count
monitor's metrics are namespaced (`supervisor_backpressure_pod_count_*`)
so the existing backpressure metrics keep their names.
## Summary
Fixes the fair-queue weighted environment shuffle, which biased
environment ordering whenever fair-queue biases are enabled (the default
configuration).
## Root cause
`#weightedShuffle` in `fairQueueSelectionStrategy.ts` computed the total
weight once and drew its random pivot against that full-set total on
every iteration, but never decremented the total as items were removed
from the working set. After the first pick, the pivot frequently
overshot the sum of the remaining items, so the inner selection loop ran
off the end and clamped to the last remaining element. The result
systematically over-selected whichever environment sat at the tail of
the set.
The first slot stayed fair (the full total is correct on the first
draw), but later positions were ordered by environment iteration order
rather than by the intended concurrency-limit and available-capacity
weighting. For four equal-weight environments, the final position landed
on one env ~9% of the time and another ~42%, instead of ~25% each.
The two sibling selection paths (`#weightedRandomQueueOrder` and
`#selectTopEnvs`) already decrement the total before splicing; this
brings the env shuffle in line with them.
## Fix
```ts
result.push(items[index].envId);
totalWeight -= items[index].weight;
items.splice(index, 1);
```
Adds a regression test that runs the weighted shuffle over equal-weight
envs with biases enabled and asserts each env lands in every position
roughly uniformly. It fails on the old code (tail position ~37%) and
passes with the fix.
Reported in #4001.
Deploy success/failure wasn't easily observable: compute template
creation only logged, and the terminal deployment statuses (deployed /
failed / timed-out) weren't traced — so deploy health couldn't be seen
without querying the database.
This adds two spans:
- `compute.template.create` around template creation at finalize,
tagging the resolved mode and per-preset outcome. `resolveMode` now
returns its decision (mode + reason) so the span can record why a mode
was chosen.
- `deployment.outcome` via a small shared helper
(`recordDeploymentOutcome`) emitted at every terminal-status write —
finalize (deployed), fail / index-failed / background-worker (failed),
and timeout (timed out) — so deploy success/failure is queryable by
status and reason.
The helper is best-effort (org/project/env enrichment where cheaply
available) and never throws, so telemetry can't break a deploy.
## Summary
Refreshes the SVG artwork for the main task icon and the cached task
variant shown on the run trace span view.
The cached icon (previously a hardcoded blue "T" in a dashed border) now
lives alongside `TaskIcon` in `TaskIcon.tsx` and is drawn with
`currentColor`, so it inherits the `text-tasks` theme color like the
other span icons instead of ignoring it. The standalone
`TaskCachedIcon.tsx` file is removed and its two import sites updated.
## Summary
Sessions started from the agent Test playground were tagged with a
`"playground"` tag that rendered in the Sessions table's Tags column.
They are now flagged with a real `Session.isTest` boolean (mirroring
`TaskRun.isTest`) and surfaced as a dedicated **Test** column with a
check icon, to the left of Tags, on both the Sessions page and the Agent
landing page, plus a matching **Test** property on the session detail
page. This mirrors how Standard and Scheduled task runs already indicate
test runs.
## Design
`isTest` is a new `Session` column (Postgres) replicated into ClickHouse
`sessions_v1` alongside the existing fields. The Sessions list reads
`isTest` from Postgres for display (ClickHouse only supplies the ordered
session IDs), so the column renders correctly without a ClickHouse
backfill.
The playground action now sets `isTest: true` on session create instead
of writing the `"playground"` tag. The triggered run still carries
`playground:true` in its own tags (unchanged). A migration backfills
existing sessions, setting `isTest = true` and stripping the
now-redundant `"playground"` tag where it is present, so the list and
detail views render consistently without read-time tag filtering.
## Summary
Updates the task icons used across the dashboard. `TaskIcon` and its
small variant now use a new burst glyph, and `TasksIcon` adopts the
previous task glyph (the rounded square). Both still render with
`currentColor`, so they inherit text color exactly as before.
Export names are unchanged, so every existing usage (side menu, task and
queue views, run filters) picks up the new artwork with no other code
changes.
## Summary
The logs search page (behind a feature flag) ran ClickHouse out of
memory when browsing back over long time ranges. This keeps it within
bounded memory and fixes a pagination bug that could skip or duplicate
rows at a page boundary.
## Fix
Memory: the list query reads in sort-key order, which opens one read
stream per part in the window, and on object storage those per-part read
buffers dominate peak memory, so it scaled with the number of parts
scanned. Two changes bound it:
- The logs ClickHouse client caps the per-part read buffers via new
env-tunable settings. The object-storage-only setting is opt-in, so it
is never sent to a ClickHouse version that lacks it.
- Recent-first window narrowing: rows come back newest first, so the
presenter probes the most recent window and only widens toward the full
requested range when a page is short. A busy environment fills a page
from a few recent parts instead of scanning the whole range; a quiet one
still returns every row in a couple of cheap reads.
Correctness: the keyset cursor ordered on (triggered_timestamp,
trace_id), which is not unique because the spans of a trace share both,
so rows at a tie could be skipped or duplicated across pages. The cursor
and ORDER BY now include span_id, and the cursor is versioned so stale
cursors reset to the first page.
Guards: the effective page size is capped, and the existing per-query
memory limit lets a pathological wide browse fail with an error instead
of taking the node down.
## ClickHouse 26.2
The memory fix relies on lazy materialization deferring the wide
attributes column to the output rows, which only holds on 26.x. Cloud
already runs 26.2, so this moves the dev stack, testcontainers, and CI
to match. The ClickHouse test suite passes on 26.2.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
## Summary
Adds read methods to `RunStore` (`findRun`, `findRunOrThrow`,
`findRuns`) and routes every Postgres read of `TaskRun` through them,
mirroring how writes already go through the store. Behavior-preserving:
each relocated read keeps its exact query, field selection, and database
client (writer, replica, or transaction). This lets `TaskRun` reads be
retargeted to a different backing store later without touching call
sites.
Stacked on #3981 (the write adapter); that PR is the base of this one.
## Scope
In scope: the run engine, webapp services, presenters, and route
loaders. Three reads that pulled `TaskRun` in through a parent model's
relation `include` (alert delivery, batch results, attempt-dependency
cancellation) are decomposed to fetch the run(s) through the store and
stitch them back, since a relation include would not follow `TaskRun` to
a new table.
Left reading the existing table (out of scope): the legacy MarQS paths,
the legacy trigger idempotency read, and one raw-SQL recovery script
(commented for revisiting at cutover).
## Notes
Reads default to the read replica; callers pass the writer or a
transaction client wherever the original read did, so writer-vs-replica
behavior is unchanged.
Follow-up to #3992, which gated the send runner-side - but only for new
runner images. Existing runners still POST a debug log per line.
When `SEND_RUN_DEBUG_LOGS` is off (default), the route now drops the
request immediately: `skipBodyParsing` skips the body read/parse, a bare
handler returns 204, no wide event. The route stays registered so it
avoids the `No route match` error log; the only per-request log left is
the framework's `logger.debug` trace, suppressed at the default `info`
level. Still counted by request metrics, and 204 is non-retryable so no
retry storm.
Adds a `skipBodyParsing` flag to the internal HTTP server.
Runners were POSTing a debug log to the supervisor for every log line -
one request per line, unbatched and unconditional. The supervisor
already has a `SEND_RUN_DEBUG_LOGS` toggle (off by default) that
discards them on receipt, but the runner fired the request regardless,
so the traffic hit the supervisor either way.
This gates the send at the source. The runner now reads
`TRIGGER_SEND_RUN_DEBUG_LOGS` (off by default, injected by the
supervisor from its existing `SEND_RUN_DEBUG_LOGS` setting) and skips
the POST entirely when disabled. Local log output is unchanged. Dev runs
use a separate path and are unaffected.
## Summary
Adds an environment-scoped HTTP API over the Errors feature, mirroring
the runs API. Task-run failures are grouped by a fingerprint into "error
groups," and this exposes everything you can do with them in the
dashboard:
- `GET /api/v1/errors` lists error groups, with
`filter[taskIdentifier]`, `filter[version]`, `filter[status]`
(`unresolved`/`resolved`/`ignored`), `filter[search]`, a time range, and
cursor pagination.
- `GET /api/v1/errors/{errorId}` retrieves a single group (summary,
lifecycle state, affected versions).
- `POST /api/v1/errors/{errorId}/{resolve,ignore,unresolve}` changes its
state.
- `GET /api/v1/runs?filter[error]={errorId}` lists the runs behind a
group.
Request and response schemas are exported from `@trigger.dev/core/v3` so
the SDK can reuse them, and all endpoints are documented in the API
reference (OpenAPI). `errorId` is the `error_<fingerprint>` friendly id.
## Attribution
State changes record who made them. A plain environment API key has no
user, so `resolvedBy`/`ignoredByUserId` stay null. When the caller uses
an environment JWT obtained by exchanging a personal access token or a
delegated user token at `POST /api/v1/projects/:ref/:env/jwt`, that
exchange now stamps an `act` delegation claim, and the write endpoints
read `act.sub` to attribute the change to the acting user. This is the
first endpoint to consume the `act` claim, so two small pieces of
plumbing ride along: the exchange stamps `act` for personal-access-token
subjects too (it was delegated-token-only), and the public-JWT
bearer-auth path surfaces `act.sub` to the handler.
Built on the delegated-token work in #3997.
## Summary
The CLAUDE.md audit job (`.github/workflows/claude-md-audit.yml`)
frequently hits its 15-turn cap before it finishes reviewing a PR, so
the job fails without posting a verdict. For example, the audit job
failed on [this
run](https://github.com/triggerdotdev/trigger.dev/actions/runs/27837408945/job/82390460772?pr=3990).
This raises `--max-turns` from 15 to 25 to give the review room to
complete, and pins `--model claude-opus-4-8` (the job previously
inherited the action default model).
## Summary
Adds a short-lived, delegated token (`tr_uat_...`) that authenticates
against the API as a user without handing out a long-lived personal
access token. You mint one from a PAT, optionally narrow it to a set of
scopes, and give it a lifetime; the API then treats requests as that
user, subject to their role.
`trigger.dev mint-token` is the entry point (it uses your stored PAT):
```bash
UAT=$(trigger.dev mint-token --ttl 3600 --cap read:runs)
```
The token works anywhere a PAT does for user-level endpoints, and can be
exchanged for an environment JWT at `POST
/api/v1/projects/:ref/:env/jwt` to reach environment-scoped data (the
same exchange a PAT supports).
## How it works
A user-actor token is a short-lived JWT verified by a new first-class
`authenticateUserActor` method on the RBAC plugin. Self-hosters get a
built-in fallback; role-aware enforcement comes from the plugin.
Effective permissions are the intersection of the user's role and the
token's optional scope cap, so a token is only ever narrower than the
user, never broader.
Minting is restricted to personal access tokens (a token can't mint
another one, and an environment key can't mint one). Tokens default to a
1 hour lifetime (max 365 days). When exchanged for an environment JWT,
the user is stamped on it for attribution and the scope cap is carried
through.
## Summary
Replaces the multi-select popover task type filter on the Tasks page
with a single-select segmented control: **All** plus icon-only
**Agent**, **Standard**, and **Scheduled** segments. Each segment has a
tooltip showing its label and a number-key shortcut (0-3), and the
search field no longer autofocuses so the shortcuts work on page load.
## ✅ Checklist
- [x] I have followed every step in the [contributing
guide](https://github.com/triggerdotdev/trigger.dev/blob/main/CONTRIBUTING.md)
- [x] The PR title follows the convention.
- [x] I ran and tested the code works
## Summary
The Personal Access Tokens page now shows each token's maximum role in a
new column, so you can see at a glance what a token is capped to. The
column only appears when an RBAC plugin is installed, and shows "-" for
tokens with no cap. Its header tooltip reuses the same explanation shown
in the create-token panel.
## Summary
Adds the `4.5.0-rc.7` entry to the AI chat changelog, covering the
agent-facing changes in
[v4.5.0-rc.7](https://github.com/triggerdotdev/trigger.dev/releases/tag/v4.5.0-rc.7):
- `chat.headStart` now works with the `chat.customAgent` and
`chat.createSession` backends, not just `chat.agent`
- Opt-in Anthropic system-prompt caching via
`chat.toStreamTextOptions()`
- Three custom-agent-loop fixes: continuation replay, mid-stream
steering, and task-backed tools
- `trigger skills` follow-ups: `trigger-` namespacing, SDK-bundled docs,
and a new cost-savings skill
Generic, non-agent rc.7 items (the CLI uninitialized-project error
message, run-span cost fields) are intentionally left out to keep this
changelog scoped to AI chat agents.
Pushes new organizations and users into the Attio CRM at signup time,
for Customer Success (TRI-10431).
- Orgs → Attio `workspaces`, users → Attio `users`, keyed on Attio's
built-in unique `workspace_id` / `user_id` so writes are idempotent
upserts.
- Runs on the common Redis worker (not inline), so a slow or unavailable
Attio never blocks the signup path; failures retry (3 attempts).
- Hooks: user-created (alongside the existing Loops call) and
org-created (`createOrganization`).
- Gated behind `ATTIO_API_KEY`, no key means the sync is skipped
entirely, so OSS / self-hosted installs are unaffected.
Only creation is covered here (the record "shell"); spend, runs, plan
changes, churn, and role/relationship linking are populated by the
scheduled full sync, tracked separately.
**Deploy note:** requires an Attio API key set as `ATTIO_API_KEY` in the
webapp env, with scopes **Records (read-write)** + **Object
Configuration (read)**, the assert/upsert endpoint reads object config
to resolve the matching attribute. Without the key the sync no-ops.
---------
Co-authored-by: Matt Aitken <matt@mattaitken.com>
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
## ✅ Checklist
- [x] I have followed every step in the [contributing
guide](https://github.com/triggerdotdev/trigger.dev/blob/main/CONTRIBUTING.md)
- [x] The PR title follows the convention.
- [x] I ran and tested the code works
---
## Testing
Ran the webapp locally with the change applied; it compiles and serves.
The edit only swaps the chart card title string from "LLM spend" to "LLM
spend ($)" on the agent landing page.
---
## Changelog
The agent dashboard "LLM spend" chart label now includes the currency
unit, reading "LLM spend ($)".
---
## Screenshots
_[Screenshots]_
💯
A batch of technical-SEO fixes across the docs, all reader-facing
(titles, links, redirects):
- Canonicalize the duplicate CLI command pages: the bare `/cli-dev` and
`/cli-deploy` paths now permanently redirect to their `-commands`
equivalents, and a duplicate navigation entry is removed.
- Give the three pages that all rendered as "Overview" distinct titles
(Building with AI, self-hosting overview, Management API overview), with
sidebar labels unchanged.
- Replace the generic "Learn more" links in the introduction's
build-extension list with descriptive anchor text.
- Switch two http links to https in the Supabase guides, point a
troubleshooting page's help link to Discord, and add missing meta
descriptions to three help and troubleshooting pages.
## Summary
Several dashboard routes performed actions a restricted role should not
be able to do (cancel or replay runs, manage prompt versions, invite and
manage members, manage billing) without any permission check. This adds
role-based permission enforcement to those routes, and disables the
matching UI controls (with a tooltip) when the current role lacks
permission.
Covered actions:
- Runs: cancel and replay (single, bulk create, bulk abort)
- Prompts: create or edit override versions, and promote a version to
current
- Members: invite, resend invite, revoke invite
- Billing: change plan, billing alerts, and the customer portal
## How
Each affected route now goes through the `dashboardLoader` /
`dashboardAction` route builders with an `authorization` block declaring
the required permission (or a per-intent check where one route handles
several intents). Existing tenancy and data-scoping queries are
untouched; this only layers permission checks on top. The UI follows
disable-don't-hide: controls stay visible but disabled with a "You don't
have permission to ..." tooltip.
Two reusable pieces support this: `checkPermissions(ability, checks)`
turns a set of checks into a boolean map a loader returns to the client,
and `PermissionButton` / `PermissionLink` disable the underlying control
and show a tooltip when a permission flag is false.
## Behaviour
No change in the default configuration: permissions are permissive, so
every control stays enabled and every route behaves as before. The
checks only take effect when an RBAC plugin is installed. This also
makes role assignment on invite-accept non-fatal, so a failure there
cannot block joining an org.
Verified with `pnpm run typecheck --filter webapp`; `checkPermissions`
has unit tests.
## Summary
Adds a "Stopping generation" section to the Custom agents page. It
documents how stop works when you drop down from `chat.agent` to
`chat.createSession`: pass `turn.signal` (a combined stop-and-cancel
`AbortSignal`) to `streamText`, and `turn.complete()` cleans up the
aborted partial, accumulates it as its own assistant message, and keeps
the run alive for the next turn. `turn.stopped` distinguishes a user
stop from a full run cancel.
Until now the createSession stop story only existed as scattered fields
in the reference table; the client side (`transport.stopGeneration`) and
the `chat.agent` run-callback signals were documented, but not the
custom-agent turn loop. Steering for these backends is already covered
on the pending messages page, which this page links to.
## Summary
Adds a "Duration and cost while paused" section to the human-in-the-loop
page. It explains that a HITL pause (a no-execute tool waiting on
`addToolOutput`) suspends the run and frees compute, so the human's
thinking time does not count against `maxDuration` (which measures
active CPU time and excludes suspended waitpoint time, the same as
`wait.for`). Customers don't need to raise `maxDuration` or end the run
to support long human waits.
This was a recurring point of confusion: readers assumed the pause holds
the run open and burns the budget. Also updates the how-it-works
pseudocode ("Agent suspends (compute freed)") and links `wait.for` and
`maxDuration` on first mention.
The global feature flags admin page had a few rough edges.
The percentage flags are numeric (`z.coerce.number()`) but rendered as
free-text inputs, so you could type non-numeric values that only failed
validation after submitting - and the error surfaced behind the confirm
dialog. The control-type detection now recognises numbers and renders a
proper number input, with the min/max range as the placeholder so the
type is clear even when the field is unset. The save error also shows
inside the confirm dialog now, not just behind it.
The action buttons were unreachable without zooming out. The admin
layout wrapped each page in a plain block, so `h-full` page content
overran the viewport by the height of the tab bar and got clipped by the
`overflow-hidden` body. Making the layout a flex column bounds each page
to the space below the tabs, so the existing per-page scroll works and
the feature flags page scrolls like the Users/Orgs tabs. Also capped the
confirm dialog's diff list so its footer stays on screen when there are
many changes.
## Summary
Prisma infrastructure failures (P1xxx-class: database unreachable, timed
out, connection dropped, engine init/panic) carry the database hostname
in their `.message`. This captures them centrally for observability and
ensures they never reach API clients verbatim.
## Design
A `$allOperations` client extension on the writer and replica clients
logs infrastructure errors with the originating model and operation,
then rethrows the **original** error unchanged — call sites that branch
on `error.code` (unique-violation idempotency, not-found handling) and
transaction retries keep working. Only infrastructure errors are logged;
routine query/validation errors (P2xxx) are left alone.
`$allOperations` can't see the transaction boundary (`$transaction` is a
client method, not an operation), so infrastructure errors surfacing
from `$transaction()` without a Prisma code — e.g.
`PrismaClientInitializationError` — are logged separately at the
transaction wrapper, where the existing coded-error path would otherwise
miss them.
`clientSafeErrorMessage()` swaps an infrastructure error's message for
`"Internal Server Error"` at the API routes that previously returned
`error.message` raw. Status codes, headers, and every non-infrastructure
message are unchanged.
## Test plan
- [x] P2002 / P2025 rethrow with code intact and are not logged
- [x] Statement errors inside `$transaction` keep their code (retry
logic intact)
- [x] Raw queries wrapped without crashing on the undefined model
- [x] A genuine connectivity failure is logged with model/operation/code
- [x] `clientSafeErrorMessage` obfuscates infra messages, preserves all
others
- [x] `pnpm run typecheck --filter webapp` (12/12)
## Note
Overlaps with #3391 (Prisma 7 migration) on
`apps/webapp/app/db.server.ts` — coordinate rebasing.
Replicates `TaskRun.planType` into the `task_runs_v2` ClickHouse table
so run analytics can group by plan type.
Adds a `plan_type` column (goose migration `033`,
`LowCardinality(String)`), the replication insert mapping, and the
matching schema/column/type entries - same shape as the recent `region`
addition. Write-once at trigger, so it just rides along on existing
replicated rows. Internal analytics only; not exposed in the Query API.
## Summary
7 improvements.
## Improvements
- `@trigger.dev/sdk` now bundles the Trigger.dev agent skills and a
curated snapshot of the docs those skills reference. The skills that
`trigger skills` installs into your coding agent read this content from
node_modules, so the guidance your AI assistant follows is pinned to the
SDK version installed in your project and stays current across upgrades
instead of going stale until the next reinstall.
([#3937](https://github.com/triggerdotdev/trigger.dev/pull/3937))
- Running a CLI command like `dev`, `deploy`, `preview`, or `update`
before initializing a project no longer crashes with a raw `Cannot find
matching package.json` stack trace. The CLI now detects the missing
project and points you to `npx trigger.dev@latest init` instead.
([#3929](https://github.com/triggerdotdev/trigger.dev/pull/3929))
- The agent skills installed by `trigger skills` are now namespaced with
a `trigger-` prefix (e.g. `trigger-authoring-tasks`,
`trigger-getting-started`) so they don't collide with unrelated skills
in your coding agent's skills directory. Adds a `trigger-cost-savings`
skill for auditing and reducing compute spend (right-sizing machines,
`maxDuration`, batching, debounce), and `@trigger.dev/sdk` now bundles
the full Trigger.dev documentation so your agent can read the complete,
version-pinned reference directly from node_modules.
([#3970](https://github.com/triggerdotdev/trigger.dev/pull/3970))
- The run span API response now includes `cachedCost` and
`cacheCreationCost` on the `ai` object, alongside the existing
`inputCost` / `outputCost` / `totalCost`. `inputCost` reflects only the
non-cached input, so these fields let you reconstruct the full cost
breakdown for prompt-cached calls.
([#3958](https://github.com/triggerdotdev/trigger.dev/pull/3958))
- `chat.headStart` now works with the `chat.customAgent` and
`chat.createSession` backends, not only `chat.agent`. The warm step-1
response hands over to your loop the same way it does for a managed
agent. ([#3963](https://github.com/triggerdotdev/trigger.dev/pull/3963))
In a `chat.customAgent` loop, consume the handover on turn 0:
```ts
const conversation = new chat.MessageAccumulator();
const { isFinal, skipped } = await conversation.consumeHandover({
payload });
if (skipped) return; // warm handler aborted, so exit without a turn
if (isFinal) {
await chat.writeTurnComplete(); // step 1 is the response, no streamText
} else {
const result = streamText({ model, messages: conversation.modelMessages,
tools });
// Pass originalMessages so the handed-over tool round merges into the
// step-1 assistant instead of starting a new message.
const response = await chat.pipeAndCapture(result, {
originalMessages: conversation.uiMessages,
});
if (response) await conversation.addResponse(response);
}
```
With `chat.createSession`, the iterator surfaces it as `turn.handover`;
call `turn.complete()` with no argument on a final handover. The
lower-level `chat.waitForHandover()` and `accumulator.applyHandover()`
are also exported for hand-rolled loops.
- Cache your chat agent's system prompt with Anthropic prompt caching.
`chat.toStreamTextOptions()` now emits the system prompt as a cacheable
message when you opt in, so a large, stable system block is billed at
cache-read rates on every turn instead of full price.
([#3952](https://github.com/triggerdotdev/trigger.dev/pull/3952))
```ts
// at the streamText call site (Anthropic sugar)
streamText({
...chat.toStreamTextOptions({ cacheControl: { type: "ephemeral" } }),
messages,
});
// provider-agnostic equivalent
chat.toStreamTextOptions({
systemProviderOptions: { anthropic: { cacheControl: { type: "ephemeral"
} } },
});
// or where the prompt is defined
chat.prompt.set(SYSTEM_PROMPT, {
providerOptions: { anthropic: { cacheControl: { type: "ephemeral" } } },
});
```
Without an option, `system` stays a plain string. Pairs with a
`prepareMessages` cache breakpoint to cache the conversation prefix
across turns too.
- Three fixes for custom agent loops (`chat.customAgent`,
`chat.createSession`, and hand-rolled `MessageAccumulator` loops):
([#3936](https://github.com/triggerdotdev/trigger.dev/pull/3936))
- Continuation runs no longer replay already-answered user messages into
the first turn. The `.in` resume cursor is now seeded before any
listener attaches (the same boot logic `chat.agent` uses), so a chat
that continues after a cancel, crash, or upgrade only sees genuinely new
messages.
- Steering a hand-rolled loop mid-stream no longer wipes the in-flight
assistant response. `chat.pipeAndCapture` now stamps a server-generated
message id on the stream, so a `prepareStep` injection keeps the partial
text instead of replacing the message.
- Task-backed tools (`ai.toolExecute`) now work from custom agent loops:
the parent's session is threaded to the child run, so child tasks can
stream progress into the chat with `chat.stream.writer({ target: "root"
})` instead of failing with "session handle is not initialized".
<details>
<summary>Raw changeset output</summary>
⚠️⚠️⚠️⚠️⚠️⚠️
`main` is currently in **pre mode** so this branch has prereleases
rather than normal releases. If you want to exit prereleases, run
`changeset pre exit` on `main`.
⚠️⚠️⚠️⚠️⚠️⚠️
# Releases
## @trigger.dev/build@4.5.0-rc.7
### Patch Changes
- Updated dependencies:
- `@trigger.dev/core@4.5.0-rc.7`
## trigger.dev@4.5.0-rc.7
### Patch Changes
- `@trigger.dev/sdk` now bundles the Trigger.dev agent skills and a
curated snapshot of the docs those skills reference. The skills that
`trigger skills` installs into your coding agent read this content from
node_modules, so the guidance your AI assistant follows is pinned to the
SDK version installed in your project and stays current across upgrades
instead of going stale until the next reinstall.
([#3937](https://github.com/triggerdotdev/trigger.dev/pull/3937))
- Running a CLI command like `dev`, `deploy`, `preview`, or `update`
before initializing a project no longer crashes with a raw `Cannot find
matching package.json` stack trace. The CLI now detects the missing
project and points you to `npx trigger.dev@latest init` instead.
([#3929](https://github.com/triggerdotdev/trigger.dev/pull/3929))
- The agent skills installed by `trigger skills` are now namespaced with
a `trigger-` prefix (e.g. `trigger-authoring-tasks`,
`trigger-getting-started`) so they don't collide with unrelated skills
in your coding agent's skills directory. Adds a `trigger-cost-savings`
skill for auditing and reducing compute spend (right-sizing machines,
`maxDuration`, batching, debounce), and `@trigger.dev/sdk` now bundles
the full Trigger.dev documentation so your agent can read the complete,
version-pinned reference directly from node_modules.
([#3970](https://github.com/triggerdotdev/trigger.dev/pull/3970))
- Updated dependencies:
- `@trigger.dev/core@4.5.0-rc.7`
- `@trigger.dev/build@4.5.0-rc.7`
- `@trigger.dev/schema-to-json@4.5.0-rc.7`
## @trigger.dev/core@4.5.0-rc.7
### Patch Changes
- The run span API response now includes `cachedCost` and
`cacheCreationCost` on the `ai` object, alongside the existing
`inputCost` / `outputCost` / `totalCost`. `inputCost` reflects only the
non-cached input, so these fields let you reconstruct the full cost
breakdown for prompt-cached calls.
([#3958](https://github.com/triggerdotdev/trigger.dev/pull/3958))
## @trigger.dev/python@4.5.0-rc.7
### Patch Changes
- Updated dependencies:
- `@trigger.dev/sdk@4.5.0-rc.7`
- `@trigger.dev/core@4.5.0-rc.7`
- `@trigger.dev/build@4.5.0-rc.7`
## @trigger.dev/react-hooks@4.5.0-rc.7
### Patch Changes
- Updated dependencies:
- `@trigger.dev/core@4.5.0-rc.7`
## @trigger.dev/redis-worker@4.5.0-rc.7
### Patch Changes
- Updated dependencies:
- `@trigger.dev/core@4.5.0-rc.7`
## @trigger.dev/rsc@4.5.0-rc.7
### Patch Changes
- Updated dependencies:
- `@trigger.dev/core@4.5.0-rc.7`
## @trigger.dev/schema-to-json@4.5.0-rc.7
### Patch Changes
- Updated dependencies:
- `@trigger.dev/core@4.5.0-rc.7`
## @trigger.dev/sdk@4.5.0-rc.7
### Patch Changes
- `@trigger.dev/sdk` now bundles the Trigger.dev agent skills and a
curated snapshot of the docs those skills reference. The skills that
`trigger skills` installs into your coding agent read this content from
node_modules, so the guidance your AI assistant follows is pinned to the
SDK version installed in your project and stays current across upgrades
instead of going stale until the next reinstall.
([#3937](https://github.com/triggerdotdev/trigger.dev/pull/3937))
- `chat.headStart` now works with the `chat.customAgent` and
`chat.createSession` backends, not only `chat.agent`. The warm step-1
response hands over to your loop the same way it does for a managed
agent. ([#3963](https://github.com/triggerdotdev/trigger.dev/pull/3963))
In a `chat.customAgent` loop, consume the handover on turn 0:
```ts
const conversation = new chat.MessageAccumulator();
const { isFinal, skipped } = await conversation.consumeHandover({
payload });
if (skipped) return; // warm handler aborted, so exit without a turn
if (isFinal) {
await chat.writeTurnComplete(); // step 1 is the response, no streamText
} else {
const result = streamText({ model, messages: conversation.modelMessages,
tools });
// Pass originalMessages so the handed-over tool round merges into the
// step-1 assistant instead of starting a new message.
const response = await chat.pipeAndCapture(result, {
originalMessages: conversation.uiMessages,
});
if (response) await conversation.addResponse(response);
}
```
With `chat.createSession`, the iterator surfaces it as `turn.handover`;
call `turn.complete()` with no argument on a final handover. The
lower-level `chat.waitForHandover()` and `accumulator.applyHandover()`
are also exported for hand-rolled loops.
- Add `triggerConfig` support to `chat.headStart()` and
`chat.openSession()`, so the auto-triggered handover-prepare run
inherits tags, queue, machine, and other session trigger options the
same way `chat.createStartSessionAction()` does. The `chat:{chatId}` tag
is prepended automatically.
([#3963](https://github.com/triggerdotdev/trigger.dev/pull/3963))
```ts
export const POST = chat.headStart({
agentId: "my-agent",
triggerConfig: { tags: ["org:acme"], queue: "chat" },
run: async ({ chat }) => streamText({ ...chat.toStreamTextOptions(),
model }),
});
```
Because the session is created once on the first head-start turn and is
idempotent on the chat id, this is the only place to set those options
for a head-start chat's lifetime. `chat.createStartSessionAction()` now
also forwards `maxDuration`, `region`, and `lockToVersion` so both
session entry points stay consistent.
- Cache your chat agent's system prompt with Anthropic prompt caching.
`chat.toStreamTextOptions()` now emits the system prompt as a cacheable
message when you opt in, so a large, stable system block is billed at
cache-read rates on every turn instead of full price.
([#3952](https://github.com/triggerdotdev/trigger.dev/pull/3952))
```ts
// at the streamText call site (Anthropic sugar)
streamText({
...chat.toStreamTextOptions({ cacheControl: { type: "ephemeral" } }),
messages,
});
// provider-agnostic equivalent
chat.toStreamTextOptions({
systemProviderOptions: { anthropic: { cacheControl: { type: "ephemeral"
} } },
});
// or where the prompt is defined
chat.prompt.set(SYSTEM_PROMPT, {
providerOptions: { anthropic: { cacheControl: { type: "ephemeral" } } },
});
```
Without an option, `system` stays a plain string. Pairs with a
`prepareMessages` cache breakpoint to cache the conversation prefix
across turns too.
- Three fixes for custom agent loops (`chat.customAgent`,
`chat.createSession`, and hand-rolled `MessageAccumulator` loops):
([#3936](https://github.com/triggerdotdev/trigger.dev/pull/3936))
- Continuation runs no longer replay already-answered user messages into
the first turn. The `.in` resume cursor is now seeded before any
listener attaches (the same boot logic `chat.agent` uses), so a chat
that continues after a cancel, crash, or upgrade only sees genuinely new
messages.
- Steering a hand-rolled loop mid-stream no longer wipes the in-flight
assistant response. `chat.pipeAndCapture` now stamps a server-generated
message id on the stream, so a `prepareStep` injection keeps the partial
text instead of replacing the message.
- Task-backed tools (`ai.toolExecute`) now work from custom agent loops:
the parent's session is threaded to the child run, so child tasks can
stream progress into the chat with `chat.stream.writer({ target: "root"
})` instead of failing with "session handle is not initialized".
- The agent skills installed by `trigger skills` are now namespaced with
a `trigger-` prefix (e.g. `trigger-authoring-tasks`,
`trigger-getting-started`) so they don't collide with unrelated skills
in your coding agent's skills directory. Adds a `trigger-cost-savings`
skill for auditing and reducing compute spend (right-sizing machines,
`maxDuration`, batching, debounce), and `@trigger.dev/sdk` now bundles
the full Trigger.dev documentation so your agent can read the complete,
version-pinned reference directly from node_modules.
([#3970](https://github.com/triggerdotdev/trigger.dev/pull/3970))
- Updated dependencies:
- `@trigger.dev/core@4.5.0-rc.7`
## @trigger.dev/plugins@4.5.0-rc.7
### Patch Changes
- Updated dependencies:
- `@trigger.dev/core@4.5.0-rc.7`
</details>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Adds an opt-in mechanism to route a configurable percentage of
organizations onto the compute (MicroVM) backing of their region at
trigger time, without changing their stored region settings.
Routing is gated by three global feature flags -
`computeMigrationEnabled`, `computeMigrationFreePercentage`,
`computeMigrationPaidPercentage` - plus a per-org
`computeMigrationEnabled` override that wins in both directions. A
region's compute backing is resolved from a new
`WorkerInstanceGroup.region` column: a container group and its MicroVM
group share one geo `region`, so the migration swaps the resolved worker
queue to the backing group's queue. Orgs are bucketed deterministically
by id, so ramping a percentage down keeps a strict subset rather than
reshuffling, and a region with no compute backing is never touched.
Everything is off by default - behaviour is unchanged unless the flags
are set.
The flags and the worker-region groups are read on the trigger hot path
from in-memory snapshots rather than the database: a small
`createReloadingRegistry` helper loads each at startup and refreshes
them on an interval, so no per-trigger query is added and a percentage
or kill-switch change propagates within the reload interval. A cold
replica whose snapshot hasn't loaded yet reads as not-migrated (the
container path) and self-corrects on the next load - the same cold-start
contract as the datastore / LLM-pricing registries, with a
`reloading_registry_loaded` metric so a never-loaded registry is
alertable.
The same migration decision is consulted at deploy-time template
creation so a migrated org gets a compute template built ahead of its
first run. This runs in shadow mode (best-effort, never fails the
deploy) by default, or - when the `computeMigrationRequireTemplate` flag
is on - in required mode, built synchronously at deploy so the first run
never builds on-demand and template errors surface at deploy time.
So operators keep "which runs ran where" while customers only see
geography: the run's actual worker queue is stored raw, and the geo
region is stamped separately on `TaskRun.region` (and a new ClickHouse
`region` column) at trigger time. Read surfaces - the dashboard, the
API, and the Query/Logs page - show the geo region, falling back to the
worker queue for runs written before the column existed.
Minor follow-ups left out of scope: the percentage flags render as text
inputs on the admin flags page (the catalog UI has no numeric control
type yet), and `createReloadingRegistry` could later gain pub/sub for
sub-second cross-replica propagation if the reload interval proves too
slow.
## Summary
Three improvements to the SDK-bundled agent skills (follow-up to the
skills installer):
- **`trigger-` namespace.** The installed skills (`authoring-tasks`,
`getting-started`, …) had generic names that collide with unrelated
skills in a shared agent skills directory. They're now prefixed —
`trigger-authoring-tasks`, `trigger-getting-started`, etc. — matching
the convention the public skills repo already uses.
- **New `trigger-cost-savings` skill.** An MCP-driven cost audit:
right-sizes machines, flags missing `maxDuration`, spots sequential
triggers that could batch, and reviews schedule frequency, using
`list_runs` / `get_run_details` for live analysis.
- **Bundle the full docs.** `@trigger.dev/sdk` now bundles the entire
"Documentation" section of the docs (157 pages) instead of a curated
55-page subset, so an agent has the complete, version-pinned reference
in `node_modules`.
## How the bundling works
`scripts/bundleSdkDocs.ts` now reads `docs/docs.json`, walks the
"Documentation" dropdown, and copies every page under it into the SDK.
The set tracks the docs navigation automatically — add a page to the nav
and it ships, no skill edits needed. The API reference and Guides &
examples dropdowns are intentionally excluded. A skill's `sources:`
frontmatter is now informational only.
The dropped idea of a dedicated `trigger-config` skill is replaced by
references to the bundled build-extension docs (`config/extensions/*`)
from the `trigger-authoring-tasks` config section and the chat-agent
skills.
## Summary
Typing in the search bar on the task page could clear or reset the input
mid-keystroke. This fixes the re-render race so the field stays stable
while you type.
## Root cause
Two things compounded:
- `SearchInput`'s sync effect depended on `text`, so it re-ran on every
keystroke and could overwrite the input with the URL/controlled value
while focused.
- Each task row unmounted and remounted its activity chart during the
side-panel open/close animation (25 charts at once), forcing heavy
re-renders that the search effect raced against.
## Fix
- `SearchInput` now tracks the last synced value in a ref instead of
comparing against `text`, keeping the effect off the keystroke path. It
only writes to state when the incoming URL/controlled value actually
changes, and never while the input is focused.
- Activity charts are now hidden (`hidden` attribute) instead of
unmounted during the panel animation, so the rows don't churn the tree
and the resize stays smooth.
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
## Summary
The agent skills' deep guidance now ships inside `@trigger.dev/sdk` and
is read from `node_modules`, so it tracks the `@trigger.dev/sdk` version
installed in your project automatically. This updates the Skills page,
the Building with AI step, and the rules-redirect page to drop the old
"pinned to the CLI version, re-run to refresh" framing and describe the
version-pinned reference instead.
Pairs with the SDK/CLI change in #3937. Keep this draft until that
ships, since it describes behavior that is not released yet.
## Summary
The Models page is now split into two tabs. **Your models** shows the
models your project has actually used in the selected time range, with
usage charts (cost over time, tokens over time, calls by model), a
per-model table of calls / cost / avg TTFC / avg tokens-per-sec, and
calls/tokens trend sparklines. **Model library** is the full catalog,
reordered from alphabetical to a relevance-based provider order
(Anthropic, OpenAI, Google, then the rest), newest models first within
each provider, with a "New" badge on models released in the last 7 days.
One time-range selector drives the whole Your models tab, so the charts,
the table, and the sparklines all share the same window. Opening a model
shows its own metrics with an independent range picker and a "View in AI
metrics" link that opens the AI metrics dashboard filtered to that
model. The active tab is kept in the URL so it survives a refresh and is
shareable.
## Prompt caching & cost accuracy
Both the Your models tab and the AI metrics dashboard now surface
prompt-cache usage: a cache-savings column plus per-model cached-tokens
and cache-hit-rate views, and a caching section on the dashboard (hit
rate, cached tokens, estimated savings, and hit rate by model).
Building this surfaced a cost bug. `input_tokens` is the total prompt
count and already includes cache-read and cache-creation tokens, but the
cost pipeline charged the full input at the input price and then added a
separate cache line, so cached tokens were billed twice (and on
Anthropic, cache reads were never discounted because their price is
keyed differently). The input price now applies only to the non-cached
remainder, with cache prices resolved across the provider-specific keys,
so LLM cost and the cache hit-rate metric are accurate. Hit rate is
computed as cached reads over total input.
## Notes
Also fixes React "invalid DOM property" console warnings from the
provider icons (the Llama and DeepSeek SVGs used raw `fill-rule` /
`clip-rule` / `clip-path` attributes), which this page surfaces by
rendering more provider icons.
## Screenshots
**Your models tab:** usage charts and a per-model table with
calls/tokens trend sparklines.
<img width="2560" height="1267" alt="1-your-models-tab"
src="https://github.com/user-attachments/assets/859bd24f-9047-4828-8bbb-83e5882846d6"
/>
**Model library:** provider-relevance ordering with a "New" badge on
models released in the last 7 days.
<img width="2560" height="1267" alt="2-model-library-tab"
src="https://github.com/user-attachments/assets/46dd54b9-80f9-4922-ade9-5935b08dfebc"
/>
**Model detail, Metrics tab:** per-model range picker and a "View in AI
metrics" link.
<img width="2560" height="1267" alt="3-model-detail-metrics"
src="https://github.com/user-attachments/assets/0f65d9d0-6142-4918-93f0-110bb277101a"
/>
**View in AI metrics:** the dashboard deep-linked and filtered to the
selected model.
<img width="2560" height="1267" alt="4-ai-metrics-filtered"
src="https://github.com/user-attachments/assets/821f256c-e305-493c-98c7-eafaf2f57f83"
/>
## Summary
The "What extractNewToolResults returns" reference in the
tool-result-auditing guide did not match the SDK. It listed an `input`
field that `chat.history.extractNewToolResults()` never returns, and
marked `output` as optional when it is always present.
This corrects the block to the real `ChatNewToolResult` shape
(`toolCallId`, `toolName`, `output`, optional `errorText`). Every usage
example in the same guide already reads only those fields, so the
reference now matches both the examples and the code.
## Summary
New `/ai-chat/prompt-caching` guide covering how to cache a chat agent's
prompt prefix with Anthropic prompt caching: the system prompt, the
conversation history (a `prepareMessages` breakpoint), and how caching
interacts with compaction. It also shows how to verify cache hits via
usage and the dashboard, the prefix-stability footguns, and an "Other
providers" section (OpenAI and Google cache automatically; Amazon
Bedrock uses `cachePoint` through `systemProviderOptions`).
Registered under Features in the AI Agents nav, next to Compaction.
---------
Co-authored-by: claude[bot] <41898282+claude[bot]@users.noreply.github.com>
Co-authored-by: Eric Allam <ericallam@users.noreply.github.com>
## Summary
`chat.headStart` now works with the `chat.customAgent` and
`chat.createSession` backends (not just `chat.agent`), and takes a
`triggerConfig` option. These docs cover both.
The Fast starts guide gets a "Handover with custom agents" section
showing how each backend consumes the handover (`consumeHandover`
returning `{ isFinal, skipped }` for custom agents, `turn.handover` for
createSession), including threading `originalMessages` so a resumed tool
round merges into the handed-over assistant. The `chat.headStart` API
section documents `triggerConfig` (tags, queue, machine, and the rest)
on the auto-triggered run.
The reference picks up `ChatTurn.handover`, `turn.complete()` with no
source, `chat.waitForHandover`, and a new `HeadStartHandlerOptions`
table.
Docs for the SDK changes in
[#3963](https://github.com/triggerdotdev/trigger.dev/pull/3963).
## Summary
Docs deploy from the `docs-live` branch via Mintlify, so merging to
`main` no longer publishes docs on its own. To publish, push a
`docs-release-*` tag at the commit you want live. The workflow runs the
Mintlify broken-links check against that commit, then fast-forwards
`docs-live` to it, which is what Mintlify deploys from.
## Design
The ref move uses the GitHub API with `force=false`, making it
fast-forward only: a tag that is not ahead of `docs-live` fails the job
rather than rewinding production. Mintlify's GitHub app reacts to the
resulting push and deploys, so no extra deploy credentials are needed.
Usage:
```bash
git tag docs-release-2026.06.16 # tag the main commit you want live
git push origin docs-release-2026.06.16
```
## Summary
Reworks the scheduled task page right-hand sidebar.
- Adds **Overview** / **Schedules** tabs. The Schedules tab is a
paginated table of all schedules attached to the task, declarative
first.
- Surfaces schedule fields (ID, CRON + human-readable description,
next/last run, status) directly in the Overview property table.
- Sidebar can be dragged much wider (up to 80% of the viewport).
- "No schedules attached" panel explains declarative vs imperative and
links to docs.
- Schedule **create / edit / enable / disable / delete** all happen
inside the existing Sheet — no more navigating to the standalone
schedule page. Toasts confirm each action.
## Test plan
- Open a scheduled task page and verify the new tabs
- Create, edit, enable/disable, and delete a schedule — confirm you stay
on the page and see a toast each time
- Visit a task with no schedules attached and confirm the info panel
renders
- Drag the sidebar wider; confirm pagination shows when there are >25
schedules
## Summary
`chat.headStart` (the warm step-1 fast path) previously handed its
response over only to `chat.agent`. This extends handover to the other
two backends: `chat.customAgent` consumes it with
`conversation.consumeHandover({ payload })` on turn 0, and
`chat.createSession` surfaces it as `turn.handover` (call
`turn.complete()` with no source to finalize a pure-text handover). The
low-level `chat.waitForHandover()` and `accumulator.applyHandover()` are
exported for hand-rolled loops.
It also adds `triggerConfig` to `chat.headStart()` and
`chat.openSession()`, so the auto-triggered handover-prepare run
inherits tags, queue, machine, and the other session run options the
same way `chat.createStartSessionAction()` does. The `chat:{chatId}` tag
is prepended automatically. Because the session is created once on the
first head-start turn (idempotent on the chat id), this is the only
place those options can be set for a head-start chat's lifetime.
## Fix: tool-call resume
When the warm step-1 hands over a pending tool call (rather than pure
text), the agent loop resumes that tool round. For it to merge cleanly
the pipe threads the spliced partial as `originalMessages`, so the
resumed tool-output chunk attaches to the handed-over tool-call instead
of throwing `No tool invocation found`. `MessageAccumulator.addResponse`
now also dedups by id (replace-in-place), so the persisted history
doesn't carry a duplicate assistant message when the resumed response
reuses the partial's id.
Incorporates the `triggerConfig` work from
[#3933](https://github.com/triggerdotdev/trigger.dev/pull/3933) by
@saasjesus, with `createStartSessionAction` extended to also forward
`maxDuration`, `region`, and `lockToVersion` so the two session entry
points stay consistent.
Verified end-to-end against a local environment: handover (pure-text and
tool-call) on both new backends, a `chat.agent` regression pass, and
`triggerConfig` tags and queue landing on the run.
---------
Co-authored-by: saasjesus <armin@chatarmin.com>
### Problem
Firestarter's `didWarmStart: true` means the response was written to a
long-poll socket — not that the runner received it. A silently dead
poller (no FIN, e.g. a VM torn down mid-poll) leaves the dispatched run
stuck in `PENDING_EXECUTING` until the run engine's heartbeat redrive,
and each redrive burns a queue redelivery toward
`TASK_RUN_DEQUEUED_MAX_RETRIES`.
### Change
After a warm-start hit, the supervisor retains the `DequeuedMessage`
(TimerWheel, default 10s), then probes the existing `getLatestSnapshot`
API. If the run is still on the exact dequeued snapshot, no runner ever
acted — it falls through to the regular cold-create path. Recovery: ~10s
+ cold start, no new APIs, no CLI changes.
- **Double-start safe**: `startRunAttempt` runs under a per-run lock and
409s stale snapshot ids, so a reviving runner and the fallback workload
can't both execute; the loser exits before running anything.
- **Probe errors → do nothing**: healthy runners legitimately act late
during platform brownouts (nested attempt-start retries), so falling
back on uncertainty would stampede duplicates. The heartbeat redrive
stays as the backstop (also covers supervisor restarts dropping timers).
- **Off by default**: `TRIGGER_WARM_START_VERIFY_ENABLED` (+
`TRIGGER_WARM_START_VERIFY_DELAY_MS`, 1–60s, default 10s). Disabled =
complete no-op. Works for all workload managers (compute/k8s/docker)
since it hooks the shared dequeue path.
- Emits `warmstart.verify` wide events (`outcome: delivered | fallback |
probe_error`), making the silent-loss rate directly measurable.
Currently the `db:seed` script just hangs on success.
This PR adds `process.exit(0)` to the finally block after db disconnect
so the script exits properly.
---------
Co-authored-by: Chris Arderne <chris@trigger.dev>
Adds `pnpm.overrides` pinning a few transitive deps to their current
releases:
- `js-cookie` → 3.0.7
- `tmp` → 0.2.7
- `brace-expansion` → 1.1.13 / 2.0.3 / 5.0.6 (one entry per major)
Each override is scoped to the affected major range so unaffected majors
aren't dragged forward. Also drops the `fast-xml-builder` override,
which no longer resolves to anything in the tree.
Lockfile-only - no published package's dependencies change.
`js-cookie`/`tmp` parents pin ranges that can't reach the new versions
on their own, so overrides (not a plain lockfile refresh) are needed to
hold them.
## Summary
The script that generates the changeset release PR description was
silently dropping some changelog entries and stripping code examples. In
[#3932](https://github.com/triggerdotdev/trigger.dev/pull/3932), entry
[#3937](https://github.com/triggerdotdev/trigger.dev/pull/3937) was
missing entirely from the Improvements list and
[#3952](https://github.com/triggerdotdev/trigger.dev/pull/3952)'s code
block was gone, even though both were present in the raw changeset
output.
## Root cause
`parsePrBody` parsed the raw changeset body line by line:
- The dependency-bump filter matched any entry whose text *began* with a
backticked package name, so a real changelog entry like ``
`@trigger.dev/sdk` now bundles... `` got thrown out along with the
genuine version-bump lines.
- Only the first line of each bullet was kept, so fenced code blocks,
sub-bullets, and continuation paragraphs were discarded.
## Fix
Group each top-level bullet with its indented continuation (code blocks,
sub-bullets, paragraphs), dedent it, and re-emit it intact. The
dependency filter is now anchored so it only matches lines that are
*entirely* a package bump, leaving real entries that merely start with a
package name.
Verified by replaying #3932's raw body through the script: #3937 returns
to the list, #3952's code block is preserved, and #3936's sub-bullets
nest correctly under their parent.
## Summary
`chat.agent`'s system prompt (the `chat.prompt` text plus any skills
preamble) could not carry a provider cache breakpoint, so the largest
and most stable part of the prompt re-paid full input price on every
turn. `chat.toStreamTextOptions()` now emits the system prompt as a
structured message carrying `providerOptions` when you opt in, so a
provider can cache the system block. Without an option, `system` stays a
plain string, so existing behavior is unchanged.
## API
Three ways to opt in (most specific wins, no deep merge):
```ts
// Anthropic sugar
chat.toStreamTextOptions({ cacheControl: { type: "ephemeral" } });
// provider-agnostic (also covers Amazon Bedrock's cachePoint)
chat.toStreamTextOptions({ systemProviderOptions: { anthropic: { cacheControl: { type: "ephemeral" } } } });
// at the definition site
chat.prompt.set(SYSTEM_PROMPT, { providerOptions: { anthropic: { cacheControl: { type: "ephemeral" } } } });
```
The `cacheControl` shorthand is Anthropic-only; `systemProviderOptions`
is the general form. Pairs with a `prepareMessages` cache breakpoint to
cache the conversation prefix too.
Docs guide: https://github.com/triggerdotdev/trigger.dev/pull/3951
### Summary
Self-serve billing UI is now hidden for managed-billing organizations.
Plan pickers, upgrade actions, billing alerts, and related upgrade
prompts are replaced with a "Contact us" option where appropriate.
Uses the new showSelfServe subscription flag, defaulting to true for
existing self-serve organizations.
### Testing
- [x] billing pages render correctly for self-serve organizations.
- [x] managed-billing organizations no longer see self-serve upgrade
flows.
- [x] "Contact us" actions are shown instead of upgrade actions where
applicable.
### Changelog
Hide self-serve billing flows for managed-billing organizations behind
the new showSelfServe subscription flag.
## Summary
Several optional workflow jobs fail on forks and private mirrors that
lack org-specific secrets or registry permissions. This adds per-job
repository-variable gates so those deployments can switch them off
without editing workflows — matching the pattern from #3901
(`ENABLE_CLAUDE_CODE` / `ENABLE_WORKFLOW_SECURITY_SCAN`).
Two variables, both **default-enabled** (a job runs unless its variable
is explicitly `'false'`), so canonical-repo behaviour is unchanged where
the variables are unset:
**`ENABLE_HELM_PRERELEASE`** — gates the chart-publish jobs that push to
`oci://ghcr.io/<owner>/charts` (needs `write_package` on the owner's
charts namespace):
- `helm-prerelease.yml` → `prerelease` job
- `release-helm.yml` → `release` job
Without the permission these fail with `403: denied: permission_denied:
write_package` on every PR / `helm-v*` tag. The `lint-and-test` jobs
(lint + template + kubeconform, no push) always run, so chart validity
is still enforced everywhere.
**`ENABLE_DEPENDABOT_ALERTS`** — gates the Dependabot notifier crons
that need `DEPENDABOT_ALERTS_TOKEN` / `SLACK_BOT_TOKEN` and post to a
specific Slack:
- `dependabot-critical-alerts.yml` → `alert` job (daily cron)
- `dependabot-weekly-summary.yml` → `summary` job (weekly cron)
On a fork/mirror these otherwise fire on schedule and fail (or post
nowhere) indefinitely.
## Test plan
- Variables unset (default): all jobs run as today.
- `ENABLE_HELM_PRERELEASE=false`: helm `lint-and-test` runs, publish
jobs skip — no 403 on repos lacking `write_package`.
- `ENABLE_DEPENDABOT_ALERTS=false`: the two cron jobs skip cleanly
(neutral, not failed).
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Major dashboard restructure plus the new task landing pages and
self-serve schedules add-on integration.
## Side menu
- Full restructure: standalone Tasks / Runs / Sessions block at the top;
new collapsible sections for AI, Observability, Deployments, Manage
- Persisted collapse state per section in `dashboardPreferences`
- New / updated icons across the menu
- Dashboards section: built-in Run metrics + AI metrics + custom
dashboards, with drag-to-reorder via ReactGridLayout
(`DashboardList.tsx`)
- DevPresence connection indicator in the env selector (DEV + V2)
## Tasks (`_index` — unified Tasks page)
- Replaces the separated Agents / Standard / Schedules listing pages
with one table
- New `UnifiedTaskListPresenter` composes `TaskListPresenter` +
`AgentListPresenter` (shared `currentWorker` lookup)
- Columns: Type (with kind badge), ID, File, Running (numeric for tasks;
running + suspended pills for agents), Activity (24h stacked-by-status),
sticky menu
- Search + "Task type" multi-select filter (URL-synced)
- Client-side pagination at 25/page
- Right-hand "useful links" panel (cookie-persisted state)
- Live-reload SSE: page revalidates on `WORKER_CREATED` so onboarding
`trigger dev` flips the blank state automatically
## Agent landing page (`/agents/$agentParam`)
- New per-agent detail page
- Top tabs (Sessions / Runs) toggle both the chart panel and the table
- Three dashboard-style chart cards: Sessions/Runs activity, LLM spend,
Tokens
- `AgentDetailPresenter` queries ClickHouse for run activity, session
activity (with FINAL on `sessions_v1`), and LLM cost/token activity from
`llm_metrics_v1`
- TimeFilter at the top drives all three charts
- Sticky table header, resizable horizontal handle, sidebar with Test
agent button + properties
- Docs link → `ai-chat/overview`
## Standard Task landing page (`/tasks/standard/$taskParam`)
- New per-task detail page mirroring the Agent layout
- `TaskDetailPresenter` for activity + properties
- Chart panel wrapped in a Card with "Runs by status" header
- Top bar with title, TimeFilter, pagination
- Right sidebar: Test task + identifier, queue, machine, retry, TTL,
payload schema, etc.
## Scheduled Task landing page (`/tasks/scheduled/$taskParam`)
- New per-task detail page mirroring the Agent / Standard layout
- Top-bar actions (right → left): pagination, Bulk replay…, View all
runs, TimeFilter, Create schedule
- Connected schedules mini-table in the sidebar
- **Self-serve schedules add-on integration** (reincarnated from the
now-removed `/schedules` listing page during the `origin/main` merge):
- Bottom usage bar pinned via `grid-rows-[auto_1fr_auto]` — progress
ring + "X/Y of your schedules" + Purchase / Upgrade / Request CTA
- At-limit "Create schedule" intercept dialog
- `PurchaseSchedulesModal` extracted as a shared component
(`apps/webapp/app/components/schedules/PurchaseSchedulesModal.tsx`)
handling increase / decrease / above-quota / need-to-delete states
- New resource action route at
`/resources/orgs/$organizationSlug/schedules-addon`
## Sessions
- Index page: list, filters, blank state, help tooltip rework
- Detail page: combined input/output chronological view (replaces split
tabs)
- Improved raw-message view layout (full-height)
- AI payload UI: `data-*` parts grouped under "AI SDK data parts:" label
- `toSafeUrl` helper guards rendered URLs from streamed content
- Fix: duplicate assistant content on inspector tab switch
## Playground (Test agent)
- Restructured top menu; back button + agent-selector popover
- Improved blank state
- Recent agent chat history moved into the tabbed menu
- Better message-scroll container (full height)
## Dashboards
- New Dashboards landing page (`/dashboards`) — Run metrics, AI metrics,
Create your own CTAs
- `BuiltInDashboards` updated; new `TasksDashboardPresenter` for the
tasks overview
- Custom dashboards section gains drag-to-reorder; cosmetic fix for
active-row drag-handle blending
## PageHeader / shared primitives
- `PageTitle` gains an `accessory` prop supporting string (auto-wrapped
in tooltip) and ReactNode
- Help tooltips on Tasks, Runs, Sessions PageTitles explaining the
concept and sub-categories
- `Card` primitive used for dashboard-style chart panels throughout
## Code review fixes (last batch on this branch)
- ClickHouse activity queries hardened: `FINAL` + `_is_deleted = 0` on
`task_runs_v2` (ReplacingMergeTree); `organization_id` + `project_id`
filters for sort-key prefix; `inserted_at` partition filter on
`llm_metrics_v1`
- `UnifiedTaskListPresenter`: shared `currentWorker` lookup;
slug-collision guard in `mergeRunningStates`; off-by-one fixed in 24h
bucket alignment
- `ScheduleListPresenter`: halved platform RPCs by deriving limit from
`currentPlan` instead of calling `getLimit`
- Sessions detail: stopped IntersectionObserver / scroll listener
re-attach on every chunk; `requestAnimationFrame` deferral on
auto-scroll to avoid virtualizer race
- URL hardening: `?types=` validated against known kinds; new
`parseFiniteInt` helper applied to `from`/`to`/`page` params
- AgentView: HITL resolution buffer now cleared once parts reach a
terminal state (was an unbounded Map on long sessions); subscription
effect deps documented with eslint suppression
- `PurchaseSchedulesModal`: bundle state resets on each open instead of
persisting stale drafts
## Manual testing
Manual smoke-test plan is tracked under
[TRI-10883](https://linear.app/triggerdotdev/issue/TRI-10883), broken
into 20 sub-issues covering onboarding, self-serve schedules, side menu,
the four landing pages, sessions, runs, dashboards, regressions and
performance.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
## Summary
The webapp Docker image build runs `pnpm run build --filter=webapp...`,
which builds `@trigger.dev/sdk` as a dependency. The SDK's `build`
script recently gained a `bundle-docs` step (`tsx
../../scripts/bundleSdkDocs.ts`), but the build couldn't run it in the
pruned image, breaking the image build.
Two things were missing:
- `docker/Dockerfile` copied `scripts/updateVersion.ts` into the builder
stage but not `scripts/bundleSdkDocs.ts`, so the step failed with
`ERR_MODULE_NOT_FOUND`.
- Even with the script present, the repo-level `docs/` tree it reads is
a separate workspace package that isn't in webapp's dependency graph, so
`turbo prune --scope=webapp` excludes it — the script's missing-docs
guard would then fail the build.
## Design
The Dockerfile now copies `bundleSdkDocs.ts` alongside
`updateVersion.ts`. `bundleSdkDocs.ts` skips gracefully when the repo
`docs/` tree is absent, which is exactly the pruned-dependency-build
case (the SDK is compiled there but never published). Publishing always
runs from the full monorepo where `docs/` exists, so the missing-docs
guard still protects releases — it only fires when `docs/` is present
but a cited doc is genuinely missing, rather than when the whole tree
was pruned away. This avoids dragging 27M of docs into a throwaway
builder stage.
## Test plan
- [x] `bundle-docs` from the full monorepo still bundles all cited docs
(exit 0)
- [x] Simulated pruned tree without `docs/` skips cleanly instead of
failing
- [ ] Webapp Docker image build succeeds in CI
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Two defensive fixes to the native realtime backend's run-change
publishing (behind a feature flag, off by default), so turning it on can
never destabilize the run lifecycle.
**Never throws at the caller.** Publish sites run synchronously on the
run-engine event bus and the metadata flush loop. The internal publish
was already wrapped in try/catch, but lazy construction (singleton +
metrics) and record encoding ran before that guard, so a throw could
propagate into a run lifecycle operation. The public
`publishChangeRecord` / `publishManyChangeRecords` helpers now wrap the
whole call and log-and-drop on failure.
**Bounds outage buffering.** The publisher connection caps
`maxRetriesPerRequest` at 1 (vs ioredis's default of 20), so during a
pub/sub Redis outage a publish rejects after ~1 reconnect cycle instead
of holding commands in memory for ~20s. A dropped publish is
latency-only, since the consumer has a periodic backstop full-resolve.
The offline queue stays on, so the first publish after a process boots
still flushes once the connection is ready.
## Summary
Add per-job `if:` gates so deployments that don't want — or can't run —
these jobs can switch them off via repository variables, without editing
workflows.
- `ENABLE_CLAUDE_CODE` gates the Claude jobs: interactive `@claude`, the
CLAUDE.md audit, and the REVIEW.md drift audit.
- `ENABLE_WORKFLOW_SECURITY_SCAN` gates the Zizmor job, which uploads
SARIF and so needs GitHub code scanning enabled.
Both default to **enabled**: a job runs unless its variable is
explicitly set to `'false'`, so behaviour is unchanged anywhere the
variables are unset. The sibling `actionlint` job and the report-only
Trivy scan are untouched.
## Test plan
- [x] `actionlint` clean on the four edited workflows
- [x] YAML parses for all four files
Two expected, non-failure conditions were being logged at `error` level,
which surfaces them as exceptions in error tracking and adds noise
without signal. This downgrades both to `warn`. The first is the
checkpoint-restore path when a `RESTORE` event already exists for a
checkpoint — a benign idempotency skip on a duplicate or retried event.
The second is the `/api/v1/token` endpoint when the authorization code
is invalid or expired, which is the expected steady state while the CLI
polls the endpoint during login; genuinely unexpected failures there
still log at `error`. No behavior or response changes — the token
endpoint still returns 400 in the same cases.
## Summary
Documents the Sessions HTTP API for non-SDK and server-to-server
callers, which until now appeared only in the conceptual
[ai-chat/sessions](https://trigger.dev/docs/ai-chat/sessions) page.
## What's covered
- **Sessions API reference** —
`create`/`list`/`retrieve`/`update`/`close` added to the OpenAPI spec
and a new "Sessions API" group (`management/sessions/*`), mirroring the
Runs API.
- **Channel endpoints** — a reference page for the `.in`/`.out` realtime
HTTP endpoints (append, SSE read, records drain), the wire protocol,
`Last-Event-ID` resume, and the per-direction auth boundary (`.out`
append is secret-key only).
- **Session scopes** — `read:sessions:{id}` / `write:sessions:{id}` in
the authentication docs, with the capability boundary and the 1h token
TTL.
Cross-linked with the SDK-side `ai-chat/sessions` page. Verified by
rendering each page on the Mintlify dev server.
## Summary
`@trigger.dev/sdk` now ships the Trigger.dev agent skills and a curated
snapshot of the docs those skills cite. The skills that `trigger skills`
installs into your coding agent are thin pointers that read this bundled
content from `node_modules`, so the guidance always matches the SDK
version installed in your project. Previously the full skill text was
copied into your repo at install time and went stale until you
reinstalled after an upgrade.
## How it works
The SDK's `files[]` now includes `skills/` (the full skill text) and
`docs/` (a curated snapshot generated at build time). The docs manifest
is derived from each skill's own `sources:` frontmatter, so a skill only
ships the docs it references, and a skill that cites a missing doc fails
the build.
The CLI installs thin skills whose body points the agent at
`node_modules/@trigger.dev/sdk/skills/<name>/SKILL.md` and
`node_modules/@trigger.dev/sdk/docs/`. They keep the high-value "Common
mistakes" anti-patterns inline so the trigger and the guardrails survive
even if the agent does not follow the pointer. `getting-started` stays
self-contained in the CLI because it runs before the SDK is installed.
## Summary
Three fixes that bring custom agent loops (`chat.customAgent`
hand-rolled loops and `chat.createSession`) up to the behavior
`chat.agent` users already get, and that the docs already promise:
- **Continuation runs no longer replay already-answered messages.** A
chat continuing after a cancel, crash, or upgrade re-delivered every
prior user message into the loop's first wait, so the model re-answered
an old message while the real new one had to arrive via steering. The
`.in` resume cursor is now seeded before any listener attaches, using
the same boot logic as `chat.agent`.
- **Mid-stream steering no longer wipes the in-flight response.**
`chat.pipeAndCapture` (also backing `turn.complete()`) streamed without
a server-generated message id, so a `prepareStep` injection regenerated
the assistant id mid-stream and the frontend replaced the partial
message, discarding everything streamed before the injection.
- **Task-backed tools now work from custom agent loops.** A child task
triggered via `ai.toolExecute` failed with "chat.agent session handle is
not initialized" because the parent's chatId only threaded from the
per-turn context that hand-rolled loops never set. It now falls back to
the session handle the `chat.customAgent` wrapper binds at run boot, so
children can stream progress into the chat with `chat.stream.writer({
target: "root" })` (the documented sub-agent pattern).
## Root cause on the replay fix
Attaching any `.in` listener (`chat.createStopSignal`,
`chat.messages.on`, the first wait) opens the SSE tail with
`Last-Event-ID` taken from the seq cursor at attach time. Custom loops
attached before any cursor existed, so S2 replayed from seq 0. The fix
resolves the cursor from the latest turn-complete header and seeds both
manager cursors (`setLastSeqNum` drives the SSE resume point,
`setLastDispatchedSeqNum` gates waiter dispatch) before attach;
`chat.createSession` now creates its stop signal lazily on the first
iteration, after the seed. Seeding only the first cursor after attach
does not work, which is why the earlier attempt at this was reverted.
All three were reproduced red-green against the references ai-chat
project: the replay repro showed the continuation wait consuming a stale
message in 403ms with the real message arriving via steering injection;
post-fix the wait consumes the real message directly with no injection.
Steering now preserves the full in-flight response, and the deepResearch
sub-agent streams its progress parts into a raw-loop parent. Existing
behavior verified unchanged: full SDK unit suite, `chat.agent` steering,
and stop-then-continue on `chat.createSession`.
### Summary
Queue searches previously executed both a count query and a page query
with identical filters. This PR switches filtered searches to `hasMore`
pagination, removing the extra count query while preserving existing
search behavior.
### Testing
cd apps/webapp && pnpm run test ./test/queueListSearch.test.ts --run
passes
### Changelog
Improve filtered queue search performance.
## Summary
Adds observability to the task metadata cache that backs the trigger hot
path. Follow-up to #3930, which made locked-version triggers fall back
to the primary when the read replica returns no row; this makes the
cache's effectiveness (and that fallback) measurable instead of
inferred.
## What it emits
A single bounded counter `task_meta_cache.resolve`, labeled by lookup
path (`locked` / `current`) and the source that satisfied it (`cache` /
`replica` / `writer` / `miss`):
- `cache / total` is the cache hit rate (its inverse is how cold the
cache runs).
- `writer / total` is how often the read replica returned empty for a
row the primary had (the condition #3930 recovers from).
Labels are bounded, with no per-env / worker / slug cardinality.
TRI-10873
Adds a `workload_create_duration_seconds` Prometheus histogram to the
supervisor, observed around the workload manager `create()` call:
- `backend` label: `kubernetes` | `compute` | `docker` — set once from
the configured workload manager
- `outcome` label: `success` | `error` — the per-outcome counts double
as a create error rate
Registered on the supervisor's existing metrics registry, so it's
exposed on the existing `/metrics` endpoint with no config changes.
Notes:
- Covers cold creates only; warm starts and restores return before
reaching `create()`.
- A create may include backend-internal retries, so one observation can
span multiple attempts.
- Fixed low cardinality: 2 active label sets per deployment × 10
buckets.
## Summary
`triggerAndWait` (and other locked-version triggers) could
intermittently fail with `Task '<id>' not found on locked version
'<version>'` for a task that was registered on that version. The
failures came in bursts and recovered on their own, so a retry minutes
later would succeed.
## Root cause
For a locked-version trigger, the queue resolver looks up the task's
`BackgroundWorkerTask` metadata from the read replica (behind a Redis
cache). On a cache miss it queried the replica, and a `null` result was
treated as "task not registered" and turned into a non-retryable 422. A
read replica can return an empty result for a row that already exists on
the primary, so a momentarily-behind replica produced a false negative
even though the locked worker (resolved on the primary in the same
request) clearly had the task.
## Fix
On a cache miss, when the replica returns no row the resolver now
re-checks the primary before concluding the task is missing. If the
primary has the row it is used (and the cache is back-filled); the error
fires only when the primary genuinely lacks it, which is the only case
where the 422 is correct. The extra read happens on the
cache-miss-and-replica-empty path only, so the hot path is unchanged.
Verified with a unit test (replica stub vs. real primary) and end-to-end
against a local streaming replica with replication paused to reproduce
the stale read.
TRI-10868
## Summary
Running `trigger.dev dev` before setting up a project crashed with a raw
`Cannot find matching package.json` stack trace from a transitive
dependency, instead of telling the user what to do next. It happens
whenever `dev` (or `update`) runs in a directory with no `package.json`
in it or any parent directory, for example right after creating an empty
project folder, or when `init` was exited before it scaffolded anything.
The CLI now detects the missing project and prints actionable guidance
pointing at `init`.
## Fix
`dev` runs an embedded package-version check before it loads any project
config. That check resolved `package.json` through a helper that throws
when nothing is found up the tree, and nothing caught it. It is now
wrapped, so a missing `package.json` produces a clear "run init" message
and a clean exit.
The config loader had the same latent crash on the `--skip-update-check`
path. Its resolvers for `package.json`, the lockfile, and the workspace
root all ran before the friendly "couldn't find your trigger.config.ts"
check, so any of them throwing masked it. That check now runs first and
short-circuits before the resolvers touch the filesystem.
Verified live: in an empty directory, `dev`, `dev --skip-update-check`,
and `update` all print a "run init" message and exit cleanly; in a
configured project, `dev` still resolves config and boots normally.
## Summary
Adds the 4.5.0-rc.6 entry to the AI chat changelog, covering the
chat-facing items shipping in
[#3870](https://github.com/triggerdotdev/trigger.dev/pull/3870): the
chat.agent reliability batch, the continuation boot latency fix, the
chat.headStart hydration and reasoning fixes, the chat.createSession
stop and continuation fixes, and the new trigger skills installer.
Should merge alongside the release so the changelog matches the
published version.
## Summary
Documents the two lower-level chat backend APIs and restructures the
Building agents section so it has a sane reading order.
**Custom agents page.** `chat.customAgent()` was effectively
undocumented (one passing mention) and `chat.createSession()` was buried
at the bottom of the Backend page, prompted by a customer asking whether
dropping down a level was supported at all. Both now live on one
dedicated page framed as a composition: register with `customAgent`,
then drive turns with the managed `createSession` iterator or a
hand-rolled primitives loop. The page covers the patterns the managed
lifecycle otherwise handles for you, each verified against a running
agent: seeding history on continuation runs (and why the seed must go
through the turn-0 `addIncoming`, which replaces the accumulator),
persisting the user message before streaming so a mid-stream reload
keeps it, racing `totalUsage` after a stop so the loop cannot wedge, and
the single-message wire shape.
**Backend page.** Now leads with a decision table across the three
abstraction levels and focuses on `chat.agent()`, routing to the new
page. Stale examples that read a plural `messages` field off the wire
payload are fixed (copy-pasting them broke turn accumulation), and the
ChatSessionOptions / ChatTurn reference tables gain their missing rows
(`compaction`, `pendingMessages`, usage fields, `setMessages`,
`prepareStep`).
**Anatomy page + reorder.** The Building agents group opened with the
long How it works mechanics page, a wall right after the Quick Start. A
short Anatomy page now leads the group: the three moving parts, one
annotated example where each region names the page that covers it, and a
routing table. How it works moves to the end of the group as the depth
payoff, matching where peer docs put their internals pages.
All pages visually verified against a local Mintlify build; cross-links
and anchors updated across the section.
## Summary
Two documentation improvements for the AI chat docs.
**Head-start persistence contract.** The fast starts page now documents
what your hooks can rely on across a head-start handover: one stable
assistant `messageId` for the whole turn, `onTurnComplete` as the
canonical persistence point, reasoning parts flowing into durable
history, and how Head Start composes with `hydrateMessages` (the
first-turn history arrives as `incomingMessages`, and the runtime
splices the warm partial onto the hydrated chain, deduplicated by id).
The hydrate examples on the lifecycle hooks and database persistence
pages now upsert their conversation row, since head-start first turns
run without a preload to create it.
**Sessions page.** The page opened with "a durable, task-bound,
bi-directional I/O channel pair", which reads as jargon and omitted run
orchestration entirely. It now leads with the plain mental model (a pair
of durable streams: input carries user messages, output carries
everything the agent produces) plus the Session's role orchestrating
runs, a diagram, a minimal runnable example, and a section on the
one-session-many-runs lifecycle.
Documents behavior shipping in
[#3907](https://github.com/triggerdotdev/trigger.dev/pull/3907).
## Summary
Accuracy fixes across the AI chat docs: drop the non-existent per-call
option from `transport.preload`, clarify that `onValidateMessages` only
fires on turns carrying incoming messages, soften the turn-complete
token-refresh wording (the header is optional), document the new
`onTurnComplete` `error` field and `finishReason`, and correct the
idle-timeout default to 30 seconds.
## Summary
Updates the AI-tooling docs for the new `trigger skills` installer that
shipped in #3868. The Skills page now documents `trigger skills` (skills
bundled with the CLI, version-matched to your SDK) and the four bundled
skills: `authoring-tasks`, `realtime-and-frontend`,
`authoring-chat-agent`, `chat-agent-advanced`. The old Agent Rules page
becomes a short "rules are now skills" redirect (kept because existing
redirects and the CLI link point at it), and the Building with AI
overview collapses the three-way Skills/Rules/MCP comparison into Skills
vs MCP.
Hold until the v4.5 CLI release ships, since `trigger skills` is not on
npm until then.
## Summary
Docs follow-up for #3683 (`TriggerClient` for per-instance SDK
configuration). Adds a dedicated reference page and threads the new
pattern through the existing management + preview-branches docs.
## What's in
**New page** `docs/management/multiple-clients.mdx` — when to use
`TriggerClient` vs `configure()` vs `auth.withAuth`, env-var fallback
rules, isolation contract, namespace surface, `inheritContext` opt-in,
and a when-to-use-what table.
**Updated pages**
- `docs/management/authentication.mdx` — rewrote the `auth.withAuth`
section to reflect the now-ALS-backed semantics (the prior version
warned about concurrency races and pointed at issue #3298 as a tracked
fix; that fix landed in #3683). Added `tr_preview_*` to the key prefix
list. Reframed the multi-target use case to lead with `TriggerClient`,
with `auth.withAuth` as the temporary-override helper.
- `docs/management/overview.mdx` — added a `Multiple clients in one
process` subsection.
- `docs/deployment/preview-branches.mdx` — added a `Triggering across
multiple branches from one process` example.
- `docs/triggering.mdx` — one-liner pointing at the new page for
cross-project triggering.
- `docs/docs.json` — slotted `management/multiple-clients` into the
Management API nav, right after authentication.
Paired with #3683.
## Test plan
- [ ] Mintlify preview renders cleanly
- [ ] Code samples in each updated page run as documented
- [ ] Cross-page links resolve (`/management/multiple-clients`,
`/management/authentication`)
## Summary
7 improvements, 1 bug fix.
## Improvements
- `trigger init` now sets up your AI coding assistant as part of project
setup: pick the MCP server, the agent skills, or both, then scaffold
with the CLI or hand off to your assistant. Adds a new `getting-started`
agent skill that teaches assistants how to bootstrap Trigger.dev
(install the SDK, write `trigger.config.ts`, create a first task, run
`trigger dev`), so the AI-driven setup path works end to end. It ships
in the CLI alongside the existing skills, version-matched to your SDK.
([#3872](https://github.com/triggerdotdev/trigger.dev/pull/3872))
- `dev` and `deploy` now fail with a clear error when two tasks are
defined with the same id, including across different task types (e.g. a
scheduled task and a regular task sharing an id). Previously the second
definition silently overwrote the first, so one of the tasks would
vanish with no warning. Task ids are detected as duplicates during
indexing (naming each offending id and the files it was found in), and
the same rule is enforced server-side when the background worker is
registered.
([#3865](https://github.com/triggerdotdev/trigger.dev/pull/3865))
- `trigger skills` installs Trigger.dev agent skills into your coding
agent so it knows how to write tasks, schedules, realtime, and
chat.agent code. The skills ship with the CLI and are copied into each
tool's native skills directory (Claude Code, Cursor, GitHub Copilot, and
Codex / AGENTS.md), and `trigger dev` offers to install them on first
run. ([#3868](https://github.com/triggerdotdev/trigger.dev/pull/3868))
- Reliability fixes for `chat.agent`. A user message sent while the
agent is streaming is no longer delivered twice (which could run a
duplicate turn), input appends now carry an idempotency key so a retried
send can't duplicate a message, stopping a generation clears the
streaming state so a page reload doesn't replay the stopped turn, and
runs can now carry the full set of dashboard tags instead of being
silently truncated. `onTurnComplete` now fires on errored turns (with
the thrown error attached) and the failed turn's user message is
persisted so it isn't lost on the next run. Custom agents and manual
`chat.writeTurnComplete` callers now trim the output stream, sending a
custom action no longer leaves a second stream reader running, and a
long-lived `watch` subscription no longer grows its dedupe set without
bound. ([#3891](https://github.com/triggerdotdev/trigger.dev/pull/3891))
- Continuation chat boots no longer stall for around 10 seconds before
the first turn. The `session.in` resume cursor is now found with a
non-blocking records read instead of draining an SSE long-poll (which
always waited out its full 5 second inactivity window, twice per boot),
the boot reads run concurrently, and chat snapshots carry the cursor so
subsequent boots skip the scan entirely.
([#3907](https://github.com/triggerdotdev/trigger.dev/pull/3907))
- Record client-side dequeue API latency in the supervisor consumer pool
as a Prometheus histogram
(`queue_consumer_pool_dequeue_duration_seconds`, labelled by `outcome`:
success/empty/error).
([#3887](https://github.com/triggerdotdev/trigger.dev/pull/3887))
- Add `GetProjectEnvironmentsResponseBody` and `ProjectEnvironment`
schemas for the new `GET /api/v1/projects/{projectRef}/environments`
endpoint, which lists the parent environments (dev, staging, preview,
prod) a personal access token can access for a project. Dev is scoped to
the token owner and branch (preview child) environments are excluded.
([#3880](https://github.com/triggerdotdev/trigger.dev/pull/3880))
## Bug fixes
- Fix two `chat.createSession()` bugs: stopping a generation no longer
wedges the run (the turn loop raced a `totalUsage` promise that never
settles after a stop-abort), and continuation runs now wait for the next
message instead of invoking the model with an empty prompt.
([#3920](https://github.com/triggerdotdev/trigger.dev/pull/3920))
<details>
<summary>Raw changeset output</summary>
⚠️⚠️⚠️⚠️⚠️⚠️
`main` is currently in **pre mode** so this branch has prereleases
rather than normal releases. If you want to exit prereleases, run
`changeset pre exit` on `main`.
⚠️⚠️⚠️⚠️⚠️⚠️
# Releases
## @trigger.dev/build@4.5.0-rc.6
### Patch Changes
- Updated dependencies:
- `@trigger.dev/core@4.5.0-rc.6`
## trigger.dev@4.5.0-rc.6
### Patch Changes
- `trigger init` now sets up your AI coding assistant as part of project
setup: pick the MCP server, the agent skills, or both, then scaffold
with the CLI or hand off to your assistant. Adds a new `getting-started`
agent skill that teaches assistants how to bootstrap Trigger.dev
(install the SDK, write `trigger.config.ts`, create a first task, run
`trigger dev`), so the AI-driven setup path works end to end. It ships
in the CLI alongside the existing skills, version-matched to your SDK.
([#3872](https://github.com/triggerdotdev/trigger.dev/pull/3872))
- `dev` and `deploy` now fail with a clear error when two tasks are
defined with the same id, including across different task types (e.g. a
scheduled task and a regular task sharing an id). Previously the second
definition silently overwrote the first, so one of the tasks would
vanish with no warning. Task ids are detected as duplicates during
indexing (naming each offending id and the files it was found in), and
the same rule is enforced server-side when the background worker is
registered.
([#3865](https://github.com/triggerdotdev/trigger.dev/pull/3865))
- `trigger skills` installs Trigger.dev agent skills into your coding
agent so it knows how to write tasks, schedules, realtime, and
chat.agent code. The skills ship with the CLI and are copied into each
tool's native skills directory (Claude Code, Cursor, GitHub Copilot, and
Codex / AGENTS.md), and `trigger dev` offers to install them on first
run. ([#3868](https://github.com/triggerdotdev/trigger.dev/pull/3868))
```bash
trigger skills --target claude-code
```
Replaces the previous `install-rules` command, which stays as an alias.
- Updated dependencies:
- `@trigger.dev/core@4.5.0-rc.6`
- `@trigger.dev/build@4.5.0-rc.6`
- `@trigger.dev/schema-to-json@4.5.0-rc.6`
## @trigger.dev/core@4.5.0-rc.6
### Patch Changes
- Reliability fixes for `chat.agent`. A user message sent while the
agent is streaming is no longer delivered twice (which could run a
duplicate turn), input appends now carry an idempotency key so a retried
send can't duplicate a message, stopping a generation clears the
streaming state so a page reload doesn't replay the stopped turn, and
runs can now carry the full set of dashboard tags instead of being
silently truncated. `onTurnComplete` now fires on errored turns (with
the thrown error attached) and the failed turn's user message is
persisted so it isn't lost on the next run. Custom agents and manual
`chat.writeTurnComplete` callers now trim the output stream, sending a
custom action no longer leaves a second stream reader running, and a
long-lived `watch` subscription no longer grows its dedupe set without
bound. ([#3891](https://github.com/triggerdotdev/trigger.dev/pull/3891))
- Continuation chat boots no longer stall for around 10 seconds before
the first turn. The `session.in` resume cursor is now found with a
non-blocking records read instead of draining an SSE long-poll (which
always waited out its full 5 second inactivity window, twice per boot),
the boot reads run concurrently, and chat snapshots carry the cursor so
subsequent boots skip the scan entirely.
([#3907](https://github.com/triggerdotdev/trigger.dev/pull/3907))
- Record client-side dequeue API latency in the supervisor consumer pool
as a Prometheus histogram
(`queue_consumer_pool_dequeue_duration_seconds`, labelled by `outcome`:
success/empty/error).
([#3887](https://github.com/triggerdotdev/trigger.dev/pull/3887))
- `dev` and `deploy` now fail with a clear error when two tasks are
defined with the same id, including across different task types (e.g. a
scheduled task and a regular task sharing an id). Previously the second
definition silently overwrote the first, so one of the tasks would
vanish with no warning. Task ids are detected as duplicates during
indexing (naming each offending id and the files it was found in), and
the same rule is enforced server-side when the background worker is
registered.
([#3865](https://github.com/triggerdotdev/trigger.dev/pull/3865))
- Add `GetProjectEnvironmentsResponseBody` and `ProjectEnvironment`
schemas for the new `GET /api/v1/projects/{projectRef}/environments`
endpoint, which lists the parent environments (dev, staging, preview,
prod) a personal access token can access for a project. Dev is scoped to
the token owner and branch (preview child) environments are excluded.
([#3880](https://github.com/triggerdotdev/trigger.dev/pull/3880))
## @trigger.dev/python@4.5.0-rc.6
### Patch Changes
- Updated dependencies:
- `@trigger.dev/sdk@4.5.0-rc.6`
- `@trigger.dev/core@4.5.0-rc.6`
- `@trigger.dev/build@4.5.0-rc.6`
## @trigger.dev/react-hooks@4.5.0-rc.6
### Patch Changes
- Updated dependencies:
- `@trigger.dev/core@4.5.0-rc.6`
## @trigger.dev/redis-worker@4.5.0-rc.6
### Patch Changes
- Updated dependencies:
- `@trigger.dev/core@4.5.0-rc.6`
## @trigger.dev/rsc@4.5.0-rc.6
### Patch Changes
- Updated dependencies:
- `@trigger.dev/core@4.5.0-rc.6`
## @trigger.dev/schema-to-json@4.5.0-rc.6
### Patch Changes
- Updated dependencies:
- `@trigger.dev/core@4.5.0-rc.6`
## @trigger.dev/sdk@4.5.0-rc.6
### Patch Changes
- Reliability fixes for `chat.agent`. A user message sent while the
agent is streaming is no longer delivered twice (which could run a
duplicate turn), input appends now carry an idempotency key so a retried
send can't duplicate a message, stopping a generation clears the
streaming state so a page reload doesn't replay the stopped turn, and
runs can now carry the full set of dashboard tags instead of being
silently truncated. `onTurnComplete` now fires on errored turns (with
the thrown error attached) and the failed turn's user message is
persisted so it isn't lost on the next run. Custom agents and manual
`chat.writeTurnComplete` callers now trim the output stream, sending a
custom action no longer leaves a second stream reader running, and a
long-lived `watch` subscription no longer grows its dedupe set without
bound. ([#3891](https://github.com/triggerdotdev/trigger.dev/pull/3891))
- Continuation chat boots no longer stall for around 10 seconds before
the first turn. The `session.in` resume cursor is now found with a
non-blocking records read instead of draining an SSE long-poll (which
always waited out its full 5 second inactivity window, twice per boot),
the boot reads run concurrently, and chat snapshots carry the cursor so
subsequent boots skip the scan entirely.
([#3907](https://github.com/triggerdotdev/trigger.dev/pull/3907))
- Fix `chat.headStart` when `hydrateMessages` is registered. The warm
route's step-1 partial now reaches the agent's accumulator on the
hydrate path, so `onTurnComplete` carries the full first turn (the
head-start user message included), tool-call handovers resume from step
2 instead of re-running step 1, and the assistant `messageId` stays
stable across the handover.
([#3907](https://github.com/triggerdotdev/trigger.dev/pull/3907))
- Preserve reasoning parts across the `chat.headStart` handover.
Extended-thinking models' step-1 reasoning now lands in the durable
session history (and `onTurnComplete`) under the same assistant
`messageId`, with provider metadata intact so Anthropic thinking
signatures survive replays.
([#3907](https://github.com/triggerdotdev/trigger.dev/pull/3907))
- Fix two `chat.createSession()` bugs: stopping a generation no longer
wedges the run (the turn loop raced a `totalUsage` promise that never
settles after a stop-abort), and continuation runs now wait for the next
message instead of invoking the model with an empty prompt.
([#3920](https://github.com/triggerdotdev/trigger.dev/pull/3920))
- Updated dependencies:
- `@trigger.dev/core@4.5.0-rc.6`
## @trigger.dev/plugins@4.5.0-rc.6
### Patch Changes
- Updated dependencies:
- `@trigger.dev/core@4.5.0-rc.6`
</details>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
## Summary
Two `chat.createSession()` bugs that break chats at its abstraction
level:
1. **Stopping a generation wedged the run forever.** `turn.complete()`
bare-awaited the AI SDK's `totalUsage` promise, which never settles
after a stop-abort. The run stayed stuck inside the stopped turn (trace
shows a permanently partial `ai.streamText` span and no further `waiting
for next message`), so the chat could never take another message. Fixed
with the same 2s `Promise.race` guard `chat.agent`'s turn loop already
uses.
2. **Continuation runs invoked the model with an empty prompt.** The
first turn only waited for a message on `preload` boots. A continuation
run (spawned after a cancel, crash, or version upgrade) arrives with the
boot payload stripped, so the loop ran a turn with zero messages and
errored with `AI_InvalidPromptError: messages must not be empty`.
Message-less continuation boots now wait for the next session input
("waiting for first message (continuation)"), and `turn.continuation` is
preserved across the wait so user code can seed stored history off it.
Both reproduced and verified end-to-end against a live environment (stop
followed by a next turn; cancel followed by a continuation turn with
seeded history), plus the existing unit suite.
## Summary
Two read-replica races on the session APIs could break chats whose first
activity lands inside the replication window (or any time the replica
lags):
1. A session's first `.in` append or `.out` subscribe could fail with a
404 for a session that exists on the writer, because the route resolved
the Session row on the replica only.
2. `ensureRunForSession` probed run liveness on the replica, so a probe
miss on a run triggered moments earlier was judged "run is dead" and a
second live run was spawned for the same session. Both runs then
consumed the same input stream, producing duplicated turns and doubled
responses (and doubled LLM cost).
## Fix
Liveness now re-probes the writer before declaring the current run dead
(the old code already fell back to the writer, but only to recover the
friendlyId, after the wrong verdict was made). Session resolution on the
append and subscribe/init routes goes through a new
`resolveSessionWithWriterFallback`, which stays replica-first on the hot
path and only touches the writer on a miss.
Reproduced and verified against a local streaming replica with an
artificial apply delay: pre-fix, a send immediately after session
creation reliably produced either the 404 or two executing runs with a
doubled response; post-fix, the same flow produces exactly one run and
one response.
Also rides along: the local docker replica's default apply delay drops
from 150ms to a realistic 20ms (override via `REPLICA_APPLY_DELAY` when
you want to deliberately widen the race window).
`@trigger.dev/plugins` re-exported
`sanitizeBranchName`/`isValidGitBranchName` from `@trigger.dev/core` as
a convenience forwarder. Nothing actually imports them through this
package — every consumer (webapp, `@trigger.dev/rbac`, …) imports them
directly from `@trigger.dev/core/v3/utils/gitBranch`.
Removing the forwarder keeps the package entry free of **runtime** core
imports (only type re-exports + `buildJwtAbility` remain), so consumers
that bundle `@trigger.dev/plugins` from source don't pull an unrelated
core subpath into their build.
No behavior change; the helpers remain available from
`@trigger.dev/core` where they're defined.
## Problem
The item-streaming endpoint of the two-phase batch API (`POST
/api/v3/batches/:batchId/items`) processed streamed items strictly
sequentially. For a batch of many large payloads, each offloaded to
object storage inline, this serialized N object-store round-trips inside
a single request and could exceed Node's default `server.requestTimeout`
(300s). The webapp then returned `408`, which the SDK reads as `408
terminated` and retries up to 5 times, turning a slow ingest into a
failure that takes tens of minutes to surface.
## Fix
Ingest now runs through `p-map` over the NDJSON async iterable with
bounded concurrency (`STREAMING_BATCH_INGEST_CONCURRENCY`, default 10):
- `p-map` pulls lazily from the stream, so at most `concurrency` items
are read and in-flight at once. Peak memory stays bounded to roughly
`concurrency × STREAMING_BATCH_ITEM_MAXIMUM_SIZE` and request-body
backpressure is preserved.
- Set the env to `1` for fully sequential ingestion (escape hatch).
## Why this is safe (ordering and idempotency unchanged)
- Ordering derives from each item's index (enqueue `timestamp =
batch.createdAt + index`), not enqueue order.
- Dedup is atomic per index in `enqueueBatchItem`.
- The NDJSON parser now stamps oversized-item markers with their emit
position, removing the consumer's sequential `lastIndex` assumption (the
only order-dependent bit).
- The count-check and conditional-seal path is untouched.
## Scope
This speeds up every batch ingested through the streaming endpoint, not
just large-payload batches. Each item does a per-item Redis enqueue
regardless of size, and those now overlap. Large payloads benefit most
because they add an object-store offload round-trip on top of the
enqueue.
## Verification
Added an integration test (`streamBatchItems.test.ts`) that drives the
real service against Postgres + Redis + RunEngine and times a 150-item
batch at increasing concurrency. Object-store offload is modelled as a
fixed per-item latency (local round-trips are too small to compare
meaningfully):
```
runCount=150
large payloads (10ms/item offload):
concurrency=1 1739ms
concurrency=10 192ms (9.1x faster)
concurrency=50 57ms (30.7x faster)
small payloads (Redis enqueue only, no offload):
concurrency=1 90ms
concurrency=10 24ms (3.7x faster)
```
The test asserts correctness at every concurrency (all items accepted,
sealed, enqueued exactly once), that parallel ingest beats the
sequential floor, and that the small-payload case is strictly faster
than sequential, so the win is not specific to large payloads.
Also exercised end-to-end over real HTTP against a local server: a
20-item batch (12MB body) ingests and seals, a re-stream of the sealed
batch returns `sealed: true` with zero re-accepted items (idempotent
retry), and an oversized item still seals at its correct index.
Existing coverage stays green: concurrent ingest of a 100-item batch,
in-flight processing never exceeding the configured concurrency,
concurrent dedup on streaming retry, and emit-position marker indexing.
## Follow-ups (not in this PR)
- SDK pre-offload of large item payloads (send `application/store` refs
instead of raw blobs) to remove object-store work from the request hot
path and shrink the request body.
- Optional `server.requestTimeout` bump as a safety net.
## CI fix
Added `.github/workflows/codeql.yml` to replace GitHub's automatic
("dynamic") CodeQL scanning. The dynamic setup was failing to upload
SARIF results because the auto-generated `GITHUB_TOKEN` lacked the
`security-events: write` permission. The explicit workflow grants that
permission at the job level and pins all actions to commit SHAs,
consistent with the repo's security conventions.
## ✅ Checklist
- [ ] I have followed every step in the [contributing
guide](https://github.com/triggerdotdev/trigger.dev/blob/main/CONTRIBUTING.md)
- [ ] The PR title follows the convention.
- [ ] I ran and tested the code works
---
## Testing
- Integration test (`streamBatchItems.test.ts`) validates correctness
and performance at concurrency 1, 10, and 50 for both large and small
payloads.
- End-to-end verified over real HTTP: 20-item/12MB batch ingests and
seals, idempotent retry returns `sealed: true`, oversized item seals at
correct index.
---
## Changelog
Streaming batch ingest now processes items with bounded concurrency
instead of one at a time, so batches of many large payloads ingest far
faster and no longer time out. Concurrency is configurable via
`STREAMING_BATCH_INGEST_CONCURRENCY` (default 10); set it to 1 for fully
sequential ingestion.
---
## Screenshots
_[Screenshots]_
💯
---------
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
## What
`buildJwtAbility` — the decoder for public-token scope strings
(`read:tags:…`, `read:runs:run_abc`, `admin`, …) — now lives in
`@trigger.dev/plugins` as the single source of truth.
`@trigger.dev/rbac` re-exports it, so the built-in fallback and any auth
plugin interpret a token identically.
Scope strings are split on only the first **two** colons
(`action:type:id`), so a resource id that itself contains colons — e.g.
a tag like `user:123` — is matched in full rather than truncated to its
first segment. (The fallback already did this; this makes it the one
shared implementation.)
`@trigger.dev/plugins` is now **private (unpublished)** and gains a
`@triggerdotdev/source` export condition, so consumers bundle it from
source per-commit like `@trigger.dev/core` instead of resolving a
published version — no cross-version coordination.
## Why
Two hand-maintained copies of the scope grammar drift, and the
difference silently changes what a token grants. One shared decoder
removes that class of bug.
## Notes
- No changeset: `@trigger.dev/plugins` is now private and
`@trigger.dev/rbac` is internal — neither is published.
- Unit coverage for the colon-id path lives in
`internal-packages/rbac/src/ability.test.ts` (now exercising the shared
function).
## Summary
When the realtime runs feed (the backend behind the `realtimeBackend`
feature flag) hydrates a change from a Postgres read replica, the read
can race the replica's apply of the very write that triggered it. The
delivered row then carries the previous change's content, and an
isolated final change (for example a last `metadata.set` before a run
goes quiet) is not corrected until the roughly 20 second backstop poll.
Measured against a replica with deliberate apply delay, every delivery
trailed exactly one change behind and a final change stranded for the
full backstop interval.
## Fix
Publishers stamp each change record with the committed row's
`updatedAt`, taken from writes they already perform, so the stamp costs
no extra queries. The router delays its wake hydrate until the replica's
measured lag has passed, anchored to that timestamp: a record that has
already spent longer than the lag in transit is hydrated immediately, so
only the racing leading edge ever waits. After hydrating, a tripwire
compares each row against its record's watermark. Still-stale rows are
withheld and retried briefly, and each detection feeds the lag estimate.
If retries run out, the rows are delivered anyway (liveness over
freshness) and follow-up re-hydrates emit the fresh version through the
normal working-set diff once the replica catches up, with the backstop
as the terminal net.
Replica lag is sampled reader-side only, and only while feeds are
active. Aurora reports live lag via `aurora_replica_status()`; vanilla
Postgres can only report "caught up or not" (mid-apply lag is not
honestly measurable from a replica), so tripwire observations floor the
estimate there. Deployments without a replica resolve to zero lag and
skip the gate entirely. Tunables live under
`REALTIME_BACKEND_NATIVE_REPLICA_LAG_*`, and
`realtime_native.stale_hydrates` plus
`realtime_native.replica_lag_estimate_ms` make replica health
observable.
Two adjacent fixes: a metadata update that writes nothing no longer
publishes a change record, and buffered parent and root metadata
operations now publish when the flusher writes them, so those changes
wake live feeds instead of waiting for the backstop.
For local testing, `docker-compose` gains an opt-in `database-replica`
service (compose profile `replica`) with a configurable
`recovery_min_apply_delay`, which reproduces replica-lag behavior
deterministically. With the gate disabled this rig reproduces the
one-change-behind delivery exactly; with it enabled, deliveries arrive
with current content at roughly the true replica lag, across write rates
faster and slower than the lag itself.
## Summary
The environment variables page loaded every variable value in the
project, unfiltered by environment. Archiving a preview branch does not
delete its environment variable value rows, so projects that churn
preview branches accumulate values forever, and every page view loaded
all of them. On large projects this made the page loader take many
seconds and stalled the server while deserializing the oversized result.
## Fix
The presenter now loads the displayed environments first and filters the
`values` relation to those environment IDs. That matches the display
semantics exactly (per-user dev environments and active branch
environments included), and the lookup is covered by the existing unique
index on `(variableId, environmentId)`. Values in archived branch
environments are no longer fetched at all.
Covered by a new testcontainers test asserting that values from active
environments (including branch environments) are returned while archived
branch environments are excluded.
## Summary
`test/runsRepositoryCursor.test.ts` pinned its fixture runs to
`createdAt = 2026-06-04T16:55:07Z`. `listRuns` applies the default 7 day
window when no time filter is given, so the fixtures aged out of the
window at 16:55 UTC on 2026-06-11 and all five tests started failing for
every branch, regardless of what the branch changed. The tests were
green on their own CI two days earlier because the fixtures were only
five days old at the time.
This switches the fixture base to a relative timestamp (one hour ago),
so the fixtures stay inside the default window permanently. Verified the
suite goes 5/5 green with this change on the same environment where the
pinned dates fail 5/5.
## Summary
The run trace page loader serialized every span's raw OTel events (with
full properties) into the response, even though the tree UI only renders
the derived `timelineEvents` and the span detail panel refetches what it
needs. On event-heavy traces that inflated both the loader payload and
the server-side heap copies built per request. This PR keeps raw span
events server-side and pairs that with a few related trace-view
improvements:
- A new optional `TRACE_VIEW_EMERGENCY_SPAN_CAP` env var (unset by
default) clamps the trace summary and detailed trace summary span limits
on both event store paths, including the public run trace endpoint, so
operators can bound trace query sizes in one place without retuning the
per-store limits.
- The TreeView virtualizer resolved every rendered row with a linear
scan over the whole tree (and `getNodeProps` did the same via
`findIndex`); rows now resolve through memoized id lookup maps, which
matters once traces reach tens of thousands of spans.
- The run stream SSE lookup now applies the same organization membership
scoping as the rest of the run page presenters, for consistency.
Behavior is unchanged by default: the trace tree renders from the same
`timelineEvents` it always has, and the new cap only takes effect when
set.
## Summary
Three related fixes for `chat.headStart` and continuation boots, found
while investigating customer reports.
**1. `chat.headStart` now works with `hydrateMessages`.** The turn-0
handover splice only ran on the default accumulation path, so agents
registering `hydrateMessages` silently lost the warm route's step-1
response: pure-text turns fired `onTurnComplete` with no assistant
message (and an empty durable write), tool-call turns re-ran step 1 from
scratch under a fresh `messageId`, and the head-start user message never
reached the hydrate hook at all. The first-turn history now reaches
`hydrateMessages` as `incomingMessages`, and the splice runs after both
accumulation branches, deduplicated by the handover `messageId`.
**2. Reasoning parts survive the handover.** The synthesized partial
only mapped text and tool-call parts, so an extended-thinking model's
step-1 reasoning streamed to the browser but never reached durable
history. Reasoning parts now map through with provider metadata, so
Anthropic thinking signatures survive a UIMessage round trip on hydrate
replays.
**3. Continuation boots no longer stall for ~10 seconds.** The `.in`
resume cursor was found by draining an SSE subscription that only closes
after its 5 second inactivity window, and the scan ran twice per boot.
It is now a non-blocking records read of the latest turn-complete
header, runs at most once per boot, the boot reads run concurrently, and
chat snapshots carry the cursor so subsequent boots skip the scan
entirely. Measured locally on a cancel-then-continue repro: pre-turn
continuation latency dropped from ~11s to ~0.5s.
Every fix was verified red-green: new unit tests reproduced each failure
before the fix, and end-to-end smoke tests against a live local stack
covered both handover legs, reasoning persistence with extended thinking
(including a follow-up turn that round-trips the persisted signed
reasoning back to the provider), and the boot timing comparison.
## Rollout
SDK-only; no server change required. A new SDK against a server that
does not serialize record headers degrades to the existing no-cursor
fallback. Old SDKs ignore the new snapshot field, and new SDKs fall back
to the records scan on snapshots written before it existed.
## Summary
HIPAA BAA is offered as a paid add-on on every paid plan. Each paid tier
on the in-app pricing card now has a "HIPAA BAA add-on" row with a
"Request a BAA" link that opens the existing contact dialog pre-filled
with a new `hipaa` inquiry type, prompting the user for their company
name and a brief description of the PHI workload.
The contact form's `feedbackTypes` are restructured to match the
marketing /contact form: every inquiry type carries a Plain label ID and
a "Contact form: ..." thread title, so threads land in Plain identically
whether they come from the dashboard or the marketing site. The
included-compute line on each tier also picks up the credits wording
from the marketing pricing page, and the Enterprise tier lifts its title
above the features row.
The compute suspend flow delays snapshots by `snapshotDelayMs` (~30s) so
short-lived waitpoints skip the snapshot entirely, with the intent that
a run continuing before the delay expires cancels the pending snapshot.
But the only `cancel()` call site was the `/continue` action, which
runners only invoke when restoring from an already-taken snapshot — so
pending snapshots were never cancelled (zero `snapshot.canceled` events
ever emitted in prod). When a run resumed and completed inside the
window, the stale snapshot fired ~30s later anyway, pausing the VM 6–13s
mid warm-start long-poll; the frozen guest couldn't fire its abort timer
or send a FIN, causing stalls and run-engine driven retries.
### Change
- Cancel the pending snapshot on `attempt.complete` — after the platform
accepts the completion, before the HTTP reply (so it can't reorder with
the runner's next `/suspend`).
- Cancel on `runDisconnected` (crash, exit, or run replaced on the
socket).
- Both cancels are guarded by a runnerId match (new
`TimerWheel.peek()`): a stale duplicate runner for a reassigned run must
not cancel the fresh runner's pending snapshot. A missing runnerId falls
through to an unconditional cancel (the pre-existing `/continue`
behavior is unchanged).
Waitpoint suspensions keep the runner socket connected and the attempt
incomplete, so neither hook touches a snapshot that is still wanted.
Known limitation (fail-safe direction): `socket.data.runnerId` is frozen
at the websocket handshake, so after a same-supervisor restore the
disconnect-path guard refuses the cancel. The `attempt.complete` path
uses the runner's current header id and is unaffected.
`ComputeWorkloadManager.create` swallows gateway errors currently, so a
cold start that fails placement (e.g. a netns slot with a busy tap, a
full node disk) silently abandons the dequeued run until the run
engine's `PENDING_EXECUTING` heartbeat timeout redrives it via stall
detection.
### Changes
- Retry `instances.create` with short backoff (default 3 attempts, 250ms
backoff), recording `createAttempts` in the wide event.
- **Only statuses where the create definitely did not commit are
retried**: 500 (agent/fcrun create failed) and 503 (no placement).
502/504 are excluded — the gateway emits those when it fails to reach
the node or read its response, which can happen *after* the agent
committed the create; the gateway only records the instance name on a
clean 201, so a same-name retry would miss the collision check and could
double-create the VM on another node. Network-level fetch failures are
retried (if the gateway processed the create, its name index is
populated and the retry 409s harmlessly). Timeouts are not retried.
- **Retry attempts after a 5xx use a deterministic `-rN` name suffix**:
a failed create can leave its name registered until async cleanup runs.
Attempt 1 keeps the unsuffixed name.
Fixes an intermittent `Test timed out in 30000ms` in the
`streamBatchItems` suite. Not a logic hang — the 30s budget covers
container setup, and each case boots its own per-test Redis container +
a full `RunEngine`, so under CI Docker contention a cold boot can cross
30s (which is why the failure moved between tests).
- New `containerTestWithIsolatedRedisNoClickhouse` fixture (Postgres
clone + per-test Redis, no ClickHouse) — this suite never uses
ClickHouse, but the old fixture's auto `resetClickhouse` forced a
ClickHouse boot + migration onto the cold-start test.
- Raised `testTimeout` 30s → 120s, matching the run-engine convention
for this footprint.
## Summary
A batch of reliability fixes for `chat.agent`:
- A user message sent while the agent is streaming is no longer
delivered twice (which could run a duplicate turn).
- Input appends carry an idempotency key (`X-Part-Id`) so a retried send
can't duplicate a message.
- `onTurnComplete` now fires on errored turns with the thrown error
attached, and the failed turn's user message is persisted so it isn't
lost on the next run.
- Stopping a generation clears the streaming state, so a page reload
doesn't replay the stopped turn.
- Custom agents and manual `chat.writeTurnComplete` callers trim the
output stream, sending a custom action no longer leaves a second stream
reader running, a long-lived `watch` subscription no longer grows its
dedupe set without bound, promoting a queued message to steering no
longer risks a double-send, and runs keep the full set of dashboard
tags.
The `X-Part-Id` header is accepted by current servers (they just don't
dedupe on it yet), so this is safe to ship ahead of the matching server
change.
## Summary
Reliability and authorization fixes for realtime chat sessions:
- Session-stream waitpoint delivery is scoped to the environment, so two
environments using the same session `externalId` can no longer complete
each other's waitpoints.
- The session snapshot-url routes now enforce per-session authorization,
and appending to a session's `out` channel requires secret-key auth, so
a session-scoped token can't read another session's snapshot or forge
assistant output.
- Appends that carry an `X-Part-Id` header are deduplicated on retry, so
a retried send can't duplicate a message.
- Session creation rejects expired sessions (instead of triggering a run
that can never receive input), `externalId` is immutable after creation,
and the sessions list endpoint returns friendly `run_*` ids to match the
single-session routes.
## Rollout
The waitpoint cache key gains an environment prefix. To keep waitpoints
registered by the previous deploy working across the boundary, the drain
reads both the new and the previous key for this release; the legacy
read can be removed a release later once no pre-deploy waitpoints
remain.
## Summary
Adds a second backend for the realtime runs feed (`useRealtimeRun`,
`subscribeToRunsWithTag`, `subscribeToBatch`), built to stay healthy
when a single busy environment has many subscribers watching many runs
at once. It is gated behind a feature flag with the existing backend as
the default, so nothing changes for users until it is enabled per
environment.
## Design
A run change is published once, as a small self-describing record, to a
single per-environment channel. Every feed is then a predicate over that
one stream rather than owning a channel:
- A per-instance router indexes the currently-held feeds by run, tag,
and batch. When a run changes it hydrates the affected rows once and
serializes them once, then fans the result to every matching feed. One
hot shared tag watched by many subscribers costs a single database query
and serialize, not one per subscriber.
- Feeds that don't match a change are never woken, wake delivery per
environment is coalesced on a leading edge (250ms default) so a burst of
changes costs one wake, and cold reads coalesce onto a single
short-TTL-cached resolve.
- An admission gate bounds how many cold ClickHouse resolves run
concurrently, so a mass reconnect across many distinct filters queues
instead of stampeding the database.
- Changes that land while a client is between long-polls are delivered
on its next poll instead of waiting for the periodic backstop: each
environment buffers its recent change records, subscriptions linger
briefly after the last feed closes, and a newly-armed poll replays
exactly the connection's gap.
- The per-connection replay cursors behind that are shared across
instances via Redis (a single timestamp each), so a poll landing on a
different instance behind the load balancer still reads the connection's
true gap instead of falling back to a cold resolve. Cursor reads have a
bounded deadline and degrade to the cold-read path on any Redis trouble.
- Tag subscriptions with multiple tags match runs carrying all of the
tags, mirroring the existing backend's filter semantics, and live
long-polls hold for about 20 seconds to match its cadence.
- The per-environment channel supports Redis Cluster sharded pub/sub, so
the wake path scales horizontally across shards by environment.
- The backend reports its health through OpenTelemetry metrics (delivery
lag, poll resolution paths, backstop outcomes, replay and cursor-store
activity), with a provisioned Grafana dashboard for local development.
Everything is behind the feature flag and tunable via env vars; the
existing backend remains the default.
## Summary
When `RUN_ENGINE_READ_REPLICA_SNAPSHOTS_SINCE_ENABLED` is on,
`RunEngine.getSnapshotsSince` reads from the read replica. During write
spikes the replica can briefly lag, so the snapshot id a runner just
learned from the writer isn't visible there yet: the lookup threw, the
worker route returned a 500, and the runner waited for its next poll —
turning sub-second snapshot notifications into poll-interval latency
exactly when things are busiest. This PR makes the flag safe to enable:
a replica miss of the since snapshot gets one jittered retry on the
replica (most lag windows are shorter than the ~50–200ms wait, so the
writer is never touched), then falls back to the primary, observed via a
new `run_engine.snapshots_since.replica_miss` counter with an `outcome`
attribute (`replica_retry` vs `primary`). Only genuine misses — absent
on the primary too — remain errors.
## Design
- `getExecutionSnapshotsSince` now throws a typed
`ExecutionSnapshotNotFoundError` so the engine can distinguish the
expected lag miss from real failures. The message string is unchanged
and the error never leaves the engine.
- The recovery path only engages when the flag is on, a distinct replica
client is configured, and no transaction client was passed. With the
flag off, the path is behaviorally identical to before.
- Retry delay bounds are configurable
(`RUN_ENGINE_SNAPSHOTS_SINCE_REPLICA_RETRY_MIN_MS`/`MAX_MS`, default
50/200; `MAX_MS=0` skips the replica retry and goes straight to the
primary).
- The warn log fires only when the primary serves the read (the writer
spill is the operationally interesting event); replica-retry recoveries
are counted but quiet. A permanently-missing snapshot id stays an
error-level failure with a `failedDuring` field, so lag metrics aren't
polluted by bogus ids.
- Stale-tail lag (replica has the since snapshot but not newer rows)
deliberately still returns the replica's view; the next poll catches up.
- The since-snapshot anchor lookup is now scoped to the polled run
(`where: { id, runId }`), so a snapshot id from a different run raises
not-found instead of silently anchoring a too-wide window of the run's
snapshots.
## Test plan
All vitest + testcontainers, no mocks. A new `schemaOnlyPrisma` fixture
(migrated-but-empty clone database) simulates a replica that hasn't
caught up, and a real in-memory OTel meter pins the counter semantics
per outcome.
- [x] Replica catches up during the jittered retry window → served by
the replica, `outcome=replica_retry` = 1, primary never consulted
- [x] Replica permanently missing the since snapshot → served by the
primary, `outcome=primary` = 1
- [x] Snapshot missing on both replica and primary → null, counter = 0
- [x] Replica has the since snapshot but lags by one → the replica's
view is served, no fallback (verified discriminating power: the test
fails if reads secretly hit the primary)
- [x] Flag off with a replica configured → primary serves the read
- [x] Transaction client provided → bypasses the replica entirely
- [x] Since snapshot belonging to a different run → null
- [x] Existing getSnapshotsSince + waitpoints suites green; run-engine,
testcontainers, and webapp typechecks pass
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
The supervisor's dequeue round-trip time (`POST
/engine/v1/worker-actions/dequeue`) was measured but only flowed into
wide events and OTel span attributes — there was no Prometheus series,
so latency percentiles and error rates weren't queryable. This adds
`queue_consumer_pool_dequeue_duration_seconds` (histogram, label
`outcome=success|empty|error`) to the existing consumer-pool metrics,
scraped automatically by the existing ServiceMonitors on
queue-raider/schedule-raider/supervisor.
- Records every dequeue call, including failed ones, which previously
emitted no timing at all
- The pool's shared `ConsumerPoolMetrics` instance is injected into each
consumer (mirrors the `BackpressureMetrics` → `BackpressureMonitor`
wiring)
- Buckets extend to 30s because `wrapZodFetch` retries internally (5
attempts, ≥7.5s backoff before a retryable error surfaces)
- Existing `dequeueResponseMs` wide-event/span behavior unchanged
The `dispatch-main-image` job was hard-gated to
`triggerdotdev/trigger.dev` on the `main` ref. This makes it
configurable via repository variables, all defaulting to the current
values:
- `MAIN_IMAGE_DISPATCH_REPO` — the repo allowed to dispatch (default
`triggerdotdev/trigger.dev`).
- `MAIN_IMAGE_DISPATCH_REF_PREFIX` — the ref-name prefix that
dispatches, matched with `startsWith(github.ref_name, …)` (default
`main`).
- `MAIN_IMAGE_DISPATCH_TARGET` — the `repository_dispatch` target
(default `triggerdotdev/cloud`).
The job is additionally gated on `github.event_name == 'push'`. This is
necessary, not just defensive: the gate now keys off `github.ref_name`
rather than the computed image tag, and `ref_name` is still `main` when
`release.yml` invokes this workflow via `workflow_call` during a release
— so without the event guard the job would fire during every release and
fail on the absent `CROSS_REPO_PAT`. A version-equality check can't
replace it because `build-*` tags strip the prefix to the version
output.
Behaviour note: the intended dispatch paths — push to `main`, and push
of a `<prefix>*` tag in a downstream repo — are `push` events and are
unchanged. The one case that no longer dispatches is a manual
`workflow_dispatch` run of `publish.yml` on `main` (it previously did,
via the old `version == 'main'` check). That path is indistinguishable
from a manual release by event name, so `push`-only is the clean
discriminator.
Dispatching still requires `CROSS_REPO_PAT`, so setting the variables
alone doesn't enable anything.
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
## Summary
Adds `GET /api/v1/projects/{projectRef}/environments` (personal access
token auth), which lists the base environments a user can access for a
project — their own dev environment plus the project's staging, preview,
and production environments.
## Details
- Built on the PAT route builder, so it inherits org-membership auth and
the per-resource ability check.
- `dev` is scoped to the token owner; archived environments are
excluded.
- Returns the branchable **parent** preview environment — preview branch
children are not included. A consumer targets the parent; branch-level
overrides are handled separately.
- Sorted to match the dashboard's environment switcher (dev → staging →
preview → prod), and never returns API keys.
Example response:
```json
[
{ "id": "...", "slug": "dev", "type": "DEVELOPMENT", "isBranchableEnvironment": false, "branchName": null, "paused": false },
{ "id": "...", "slug": "stg", "type": "STAGING", "isBranchableEnvironment": false, "branchName": null, "paused": false },
{ "id": "...", "slug": "preview", "type": "PREVIEW", "isBranchableEnvironment": true, "branchName": null, "paused": false },
{ "id": "...", "slug": "prod", "type": "PRODUCTION", "isBranchableEnvironment": false, "branchName": null, "paused": false }
]
```
## Summary
The dashboard's Agent view rendered `source-url` and `file` message
parts by putting their `url` straight into an `href`/`src`. Those URLs
come from streamed agent and tool data, so a tool that emitted something
like `javascript:alert(1)` produced a clickable XSS payload in the
dashboard.
## Fix
A `toSafeUrl` helper now gates every URL before it reaches an
`href`/`src`: it allows only `http:`/`https:`/`blob:` (and
`data:image/...` for inline images) and returns `null` for anything
else. Unsafe values render as plain text instead of a link or image, so
a hostile or malformed URL degrades gracefully rather than becoming
clickable. Safe URLs render exactly as before. Covered by a unit test
over the allow/deny list.
## Summary
During `trigger()` worker-queue resolution, `getWorkerQueue` wrapped any
error from `getDefaultWorkerGroupForProject` into a client-facing
`ServiceValidationError` (HTTP 422) carrying `error.message`. That
method runs `project.findFirst` on the **writer**; when the writer is
unreachable Prisma throws a connection error (P1001) whose message
includes the database host, and that raw message was returned to the API
client and surfaced in the run view via the SDK's `TriggerApiError`.
It also mis-classifies a transient outage: a 422 is not retried by the
SDK, so triggers failed permanently instead of riding out a brief writer
blip.
## Design
This is the only place on the trigger path that folds a *caught* error's
message into a client-facing error — every other DB failure on the path
propagates to the route's generic 500 handler (scrubbed, and retried by
the SDK). So the fix is local:
- Add `isInfrastructureError()` — true for Prisma connection-level
failures (the DB-unreachable family: P1001/P1002/P1008/P1017, plus the
init/panic/unknown client error classes), false for query/validation
errors (e.g. P2002).
- At the wrap site, rethrow infrastructure errors so they reach the
generic 500 handler (no raw message, and retryable). Genuine domain
failures (e.g. "Project not found.") still become a 422.
Only P1001 ("can't reach database server") has been observed in
practice; the rest of the connection family is included as same-class
forward-proofing.
## Test plan
- [x] Unit: `isInfrastructureError` classifies a P1001 (incl. the Prisma
6.x `PrismaClientKnownRequestError` shape) and init errors as
infrastructure; P2002 and a plain `Error` as not
- [x] `getWorkerQueue` rethrows a P1001 unchanged instead of wrapping it
in a `ServiceValidationError`; still wraps a domain failure as a
`ServiceValidationError` — RED on current code, GREEN after
- [ ] (optional) toxiproxy e2e: trigger with the writer cut → HTTP 500
generic body, no DB host in the response
---------
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
## Summary
Under high request load the webapp spends most of its CPU inside
react-router's `matchRoutes`, not in application code.
`@remix-run/router@1.23.2` (the React Router v6 / Remix 2 core)
re-flattens, re-ranks, and recompiles the entire route table on every
request, and with the webapp's ~436 routes that cost dominates once
request rates climb. There is no `NODE_ENV` gate, so production pays it
too.
This adds a pnpm patch that memoizes the parts that depend only on the
static route manifest: it caches the flattened/ranked branches per route
tree, hoists the loop-invariant `decodePath` out of the match loop, and
caches compiled path regexes.
## Benchmark
CPU profile over the same load (100 concurrent tag feeds, ~425 req/s),
`NODE_ENV=production`, before vs after the patch:
| Metric | Before | After |
| --- | --- | --- |
| Active CPU (self-time over the window) | 28.3s | 18.5s (-34%) |
| Route-matching self-time | 19.2s | 7.5s (-61%) |
| Event-loop lag p99 | 322ms | 113ms (-65%) |
| Idle headroom | 26% | 52% |
Application/realtime code was ~0% of CPU in both profiles; the
bottleneck was entirely generic per-request route matching.
## Why a patch instead of an upgrade
The inefficiency is acknowledged upstream
([remix-run/react-router#8653](https://github.com/remix-run/react-router/issues/8653)).
A contributor PR doing exactly this
([remix-run/react-router#14866](https://github.com/remix-run/react-router/pull/14866))
was closed in favor of a narrower fix
([remix-run/react-router#14967](https://github.com/remix-run/react-router/pull/14967),
branch caching only, shipped in React Router v7), with the maintainer
suggesting patch-package as the interim until the Remix 3 route-pattern
rewrite (see
[remix-run/remix#4786](https://github.com/remix-run/remix/discussions/4786)).
We are on the v6-era core and cannot pick up even the partial fix
without a framework migration, so this patch is the sanctioned stopgap,
and it also includes the compiled-regex cache the merged PR left out.
[`patches/README.md`](https://github.com/triggerdotdev/trigger.dev/blob/perf/react-router-route-matching/patches/README.md)
documents the full rationale, the safety argument (deterministic,
internal-only, bounded caches), and when to remove the patch.
## Summary
A new guide for connecting a database to your tasks: where to create the
client, how to size the connection pool against your provider's limit,
when to reach for a pooler, and how to release connections at waits so
you don't hit "too many connections" or crash on resume.
It covers node-postgres, Prisma, Drizzle, and MongoDB, with researched
direct and pooled connection limits for the common Postgres providers
(Supabase, Neon, RDS, PlanetScale) and MongoDB Atlas. The page lives
under Documentation, Troubleshooting, and is linked from the chat agent
docs (overview, lifecycle hooks, chat.local, and the database
persistence pattern).
## Summary
On every `main` build, once the webapp image is pushed to the registry,
the publish workflow emits a cross-repo `repository_dispatch` event
(`main-image-published`) carrying a digest-pinned image ref. Other
repositories in the org can subscribe to that event and build or deploy
from the exact artifact, instead of chasing the moving `main` tag.
## Design
`publish-webapp.yml` now exposes the pushed multi-arch index digest as a
workflow output. `publish.yml` adds a `dispatch-main-image` job (after
`publish-webapp`) that builds `<image_repo>@<digest>` and sends the
dispatch via the same pinned `peter-evans/repository-dispatch` action
already used elsewhere in this repo, authed with `CROSS_REPO_PAT`.
It fires only when the published tag is `main`, so semver releases and
other tag builds are excluded, and only from the canonical repo so forks
never dispatch. The payload is JSON-escaped with `jq`.
## Problem
Two backward-pagination bugs in `ClickHouseRunsRepository.listRunIds`,
both pre-existing (they predate the composite-cursor work in #3852 and
were spotted during/after it):
**1. Wrong slice (straddled pages).** `listRunRows` fetches `page.size +
1` rows to detect `hasMore`. That extra row is the one *farthest from
the cursor* in both directions (forward orders DESC; backward orders
ASC), so it's always the *trailing* element. Forward correctly used
`rows.slice(0, size)`, but backward+`hasMore` used `rows.slice(1, size +
1)` — dropping the row *closest* to the cursor and keeping the has-more
sentinel. The page straddled two logical pages (one run from the correct
previous page + one from the page before it), so paging "newer" across a
boundary **repeated and skipped** runs.
**2. Stranded forward cursor on a partial backward page.** In the
backward `!hasMore` branch, `nextCursor` was `reversedRows.at(page.size
- 1)`. On a partial page (fewer than `page.size` rows — reachable via
`runs.list` by passing a forward page's cursor as `page[before]`), that
index overshoots → `undefined` → `nextCursor` becomes `null`, leaving no
way to page forward again.
## Fix
- **Slice:** both directions now slice `rows.slice(0, size)` (the
sentinel is the trailing element either way).
- **Partial-page cursor:** the backward `!hasMore` branch takes the
oldest row on the page, `rows.at(0)`, for `nextCursor` — equivalent to
the old expression for full pages, correct for partial ones.
Forward pagination, the cursor *values* for full pages, and the `hasMore
=== true` paths were already correct and are unchanged.
## Tests
`runsRepositoryCursor.test.ts` gains two cases (both fail on `main`,
pass here):
- **multi-page backward walk:** forward across all pages, then backward
from the last page — each backward page must *exactly* reproduce the
corresponding forward page (no straddling: `main` returns `{b,c}`
instead of `{c,d}`), and the full traversal covers every run once.
- **partial backward page:** backward onto a partial first page must
still expose a working forward cursor (and paging forward from it
reaches the rest) — `main` returns a `null` nextCursor.
The three existing cursor tests (forward completeness, backward
round-trip, legacy cursor) still pass.
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
## What
`dev` and `deploy` now fail with a clear error when two tasks are
defined with the same id — including across task types (e.g. a scheduled
task and a regular task sharing an id).
## Why
Tasks are registered into the resource catalog keyed by id, so a second
definition with the same id silently overwrote the first. One of the
tasks would just vanish from the worker with no warning — easy to miss,
hard to debug. (Any earlier duplicate-id check ran against the
post-registration task list, which is already de-duplicated, so it never
actually fired.)
## How
- **Detect at registration** (`@trigger.dev/core`):
`StandardResourceCatalog` records a collision when a task id is
registered more than once, capturing the files involved — the only point
where both definitions are visible before the id-keyed map collapses
them. Exposed via `listTaskIdCollisions()`.
- **Fail indexing** (`trigger.dev` CLI): both index workers report
collisions via a new `TASKS_FAILED_TO_INDEX` message;
`indexWorkerManifest` rejects with a new `DuplicateTaskIdsError`. `dev`
renders a dedicated error (offending ids + files + docs link); `deploy`
fails with the same message. Runtime worker boot is unaffected — it
never reads the collisions.
- **Server-side backstop** (webapp): background-worker registration also
rejects duplicate ids with a clear `ServiceValidationError`, so
duplicates are caught even from an older CLI.
## Testing
- Unit tests for collision collection in the catalog and for the
error-message formatting (standard, same-file, and 3+-definition cases).
- Verified end to end against a local webapp: a project with a regular
task and a scheduled task sharing an id now fails `dev` with the
dedicated error; a project with distinct ids still starts normally.
## Changeset
Patch for `@trigger.dev/core` and `trigger.dev`.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
## Summary
- `internal-packages/rbac/src/index.ts` — in `LazyController.load()`'s
catch block, throw an Error when `process.env.REQUIRE_PLUGINS === "1"`
instead of silently falling back. The throw is captured into the lazy
controller's init promise, so it surfaces on the first method call.
- `apps/webapp/app/routes/healthcheck.tsx` — `await
rbac.isUsingPlugin()` after the DB ping. With `REQUIRE_PLUGINS=1` and a
failed plugin load, the throw surfaces here and the healthcheck returns
500 → readiness probe fails → rollout is rolled back. Noop for
self-hosters.
- `.server-changes/require-plugins-fail-fast.md` — server-changes entry.
- `internal-packages/rbac/src/require-plugins.test.ts` — 4 unit tests
covering loader branching: unset → fallback, `=1` → throw,
`forceFallback: true` wins, only exactly `"1"` enforces.
- `internal-packages/testcontainers/src/webapp.ts` — adds
`requirePlugins?: boolean` to `StartWebappOptions`. Implies
`forceRbacFallback: false`.
- `apps/webapp/test/healthcheck-require-plugins.e2e.test.ts` — e2e
closes the loop: spawns a real webapp, hits `/healthcheck` via HTTP,
asserts 500 with `REQUIRE_PLUGINS=1` and 200 without.
## Motivation
Today the RBAC plugin loader catches any plugin-load failure (missing
module, broken transitive dep, init throw) and silently returns the
default fallback implementation. This is the correct behaviour for
self-hosters who don't ship the plugin — but it's dangerous in
deployments where the plugin is expected to load: an
accidentally-missing or broken plugin would silently disable
enforcement.
`REQUIRE_PLUGINS=1` makes the loader fail loudly in those deployments.
The variable name is intentionally plural and generic — future plugin
contracts (audit logs, SSO) can read the same flag without renaming.
Closes
[TRI-9852](https://linear.app/triggerdotdev/issue/TRI-9852/require-plugins1-fail-fast-for-required-plugin-loads).
## Test plan
- [x] `pnpm run test --filter @trigger.dev/rbac` — 38/38 tests pass,
including the 4 new loader tests
- [x] `pnpm run typecheck --filter webapp` — passes
- [x] `pnpm run typecheck --filter @trigger.dev/rbac --filter
@internal/testcontainers` — passes
- [x] e2e test added (`healthcheck-require-plugins.e2e.test.ts`) — CI
runs it via `e2e-webapp.yml`. Couldn't run locally (no Docker daemon
up); CI has Docker provisioned.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
## Summary
Runs routed onto a dedicated scheduled worker queue were showing under a
phantom region in the dashboard, run details, and the API, and slipped
through region filters. They now resolve to their real region
everywhere.
## Fix
A worker queue can carry a `:scheduled` suffix that routes
scheduled-lineage runs onto their own list. That suffix is an internal
routing detail, but it was leaking anywhere the worker queue is read as
a region. A `baseWorkerQueue` helper strips any `:<class>` suffix back
to the base region (region names never contain a colon, so it's
everything before the first colon) and is applied at every region read
site: the runs list, run detail, the public API, and replay's region
override. The runs-replication writer also stores the base region in
ClickHouse so the region filter matches.
## Summary
`trigger init` now sets up your AI coding assistant as part of project
setup. Instead of the old either/or "MCP or CLI" prompt, it offers the
MCP server and agent skills together, then asks whether to scaffold with
the CLI or let your assistant do it.
A new `getting-started` agent skill backs that hand-off: it teaches the
assistant the bootstrap recipe (install the SDK, write
`trigger.config.ts`, scaffold a first task, wire tsconfig/gitignore, run
`trigger dev`) and is explicit about the two steps that genuinely need a
human (`trigger login` and copying the DEV secret key from the
dashboard). It ships in the CLI alongside the existing skills,
version-matched to your SDK.
Prompt-once gating is shared, so opting in or out during `init` means
`trigger dev` won't ask about skills again.
## Summary
The container publish workflows hardcoded `ghcr.io/triggerdotdev/...` as
the image destination. As a result, a fork that builds on push-to-`main`
(or on the worker publish tags) would attempt to push to — and attest —
the upstream packages rather than its own, which fails on permissions
and is surprising besides.
This makes the image destination configurable via a single
`IMAGE_REGISTRY` repository variable, while leaving the upstream
defaults byte-identical:
- **Single source of truth** (`publish.yml`): a `resolve-registry` job
resolves the target registry namespace once — `IMAGE_REGISTRY`
repository variable, defaulting to `ghcr.io/${{ github.repository_owner
}}` — and passes it down to every publish job as an `image_registry`
input. So a fork publishes to its own namespace automatically with no
configuration.
- **Webapp** (`publish-webapp.yml`): the image now lives at
`<registry>/<repo-name>` (e.g. `ghcr.io/<owner>/trigger.dev`). The
provenance attestation and the downstream Trivy scan follow the same
computed repo via the `image_repo` workflow output.
- **Workers** (`publish-worker.yml`, `publish-worker-v4.yml`): build
under `<registry>/<worker-name>`. They keep a `vars.IMAGE_REGISTRY ||
ghcr.io/<owner>` fallback so they still resolve correctly on their
direct `infra-*` / `re2-*` push triggers (which bypass the parent
workflow).
A single `IMAGE_REGISTRY` namespace variable now governs both webapp and
workers (the earlier `WEBAPP_IMAGE_REPO` full-path override is dropped,
removing the full-path/namespace asymmetry). When `IMAGE_REGISTRY` is
unset, every resolved image name is exactly what it is today, so there
is no change for this repo.
## Test plan
- [x] `actionlint` passes on all four workflows
- [ ] On merge, confirm the webapp publish still pushes
`ghcr.io/triggerdotdev/trigger.dev:main` + the commit-SHA tag (defaults
unchanged)
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
## Summary
The `mollifier.decisions` metric only carried an `outcome` label, so for
an org that has the mollifier enabled there was no way to see how often
its triggers pass through the gate instead of being diverted — making it
hard to tell why the trip isn't firing for an opted-in org.
This adds two bounded labels: `enrolled` (`"true"`/`"false"`, the
per-org flag) and `org` (the org id, attached **only** when `enrolled`
is true). For an enrolled org you can now compare directly:
`mollifier.decisions{outcome="pass_through", enrolled="true",
org="<id>"}` vs `{outcome="mollify", ...}`.
## Design
`recordDecision` now takes an options object (`{ reason?, enrolled,
orgId? }`). The `org` label is restricted to the enrolled cohort to keep
cardinality bounded — the guard lives in a pure `decisionLabels` helper,
so a non-enrolled org id can never be attached even if one is passed.
The enrolled set is small and capped operationally.
The per-org flag is resolved once at the top of `evaluateGate`
(in-memory, no DB round-trip on the trigger hot path) so every decision
— including the debounce / one-time-use-token / triggerAndWait bypasses
— is labelled consistently.
## Test plan
- [x] `mollifierGate.test.ts` cascade asserts `enrolled`/`org` on every
gate branch
- [x] `mollifierDecisionLabels.test.ts` (new) proves `org` is dropped
for non-enrolled even when an id is passed (cardinality guard)
- [x] `vitest run mollifierGate mollifierDecisionLabels` — 34/34 pass
- [x] `pnpm run typecheck --filter webapp` clean
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Add an optional network_labels field to the internal compute client's
create and restore request schemas and forward per-VM endpoint labels on
both paths, so a restored VM keeps the same labels as a freshly-booted
one. Mirrors the label the Kubernetes workload manager already sets on
the run pod.
---------
Co-authored-by: nicktrn <55853254+nicktrn@users.noreply.github.com>
## Summary
`trigger skills` installs Trigger.dev agent skills into your coding
agent so it knows how to write Trigger.dev code: tasks, schedules,
realtime, and `chat.agent` AI agents. The skills are `SKILL.md` files
(the open Agent Skills format) bundled with the CLI and copied into each
tool's native skills directory (Claude Code, Cursor, GitHub Copilot, and
Codex / `AGENTS.md`), version-matched to the CLI you run. `trigger dev`
offers to install them on first run, and a one-line always-on pointer is
written into your `CLAUDE.md` / Cursor rules / etc. so the agent always
knows which skills are available and loads the right one on demand.
This replaces the old `install-rules` command, which stays as an alias.
Four skills ship to start: `authoring-tasks`, `realtime-and-frontend`,
`authoring-chat-agent`, and `chat-agent-advanced`.
## Problem
`ClickHouseRunsRepository.listRunIds` / `listRuns` order results by the
composite key `(created_at, run_id)`, but the cursor predicate cut on
`run_id` **alone**:
```ts
.where("run_id < {runId: String}", { runId: cursor })
.orderBy("created_at DESC, run_id DESC")
```
This is only sound when `run_id` lexicographic order matches
`created_at` order. `run_id`s are cuids — only coarsely time-sortable —
so when a burst of runs is created within a sub-second window, the two
orders can diverge. When they do, the next-page predicate (`run_id <
cursor`, where `cursor` is the *last* page element = the smallest
`created_at`, not necessarily the smallest `run_id`):
- **re-includes** rows already returned on a previous page (duplicates),
and
- **skips** rows it should have returned (silent data loss).
For bulk **replay** this caused runs to be replayed more than once
(replay has no idempotency guard). For the dashboard and the `runs.list`
API it could silently repeat or skip runs at page boundaries.
## Fix
Make the cursor predicate match the composite ordering:
- Cursors now encode the full `(created_at, run_id)` key as an **opaque
URL-safe base64 token**
(`base64url({"c":<createdAtMs>,"r":"<runId>"})`), and the query cuts on
the matching tuple — `(created_at, run_id) < (…)` forward / `> (…)`
backward.
- The `ORDER BY` is unchanged, so the query stays aligned with the
table's primary key — no performance regression (the tuple range
predicate is actually more index-friendly than `run_id <` alone).
- Cursors are **server-issued opaque tokens** (the SDK only echoes
`pagination.next` / `pagination.previous` back), so this needs **no
client/SDK update**. Legacy cursors were the bare internal `run_id`;
they're detected by decode failure (a cuid isn't a valid base64-wrapped
JSON payload) and fall back to the old `run_id`-only predicate, so
in-flight cursors keep working and drain naturally. New cursors also no
longer expose a bare internal run id.
- `listRunIds` is now the single cursor-aware list primitive: it returns
`{ runIds, pagination: { nextCursor, previousCursor } }`, and `listRuns`
builds on it (one place constructs cursors). Bulk actions consume the
same method and advance by `pagination.nextCursor`, finishing when it's
`null`.
- `getTaskRunsQueryBuilder` now also selects
`toUnixTimestamp64Milli(created_at) AS created_at_ms`, using a dedicated
`TaskRunListQueryResult` schema. The shared `TaskRunV2QueryResult` stays
`run_id`-only so the run-engine pending-version lookup
(`getPendingVersionIdsQueryBuilder`, which selects only `run_id`)
doesn't fail validation on a column it doesn't query.
## Tests
New `runsRepositoryCursor.test.ts` (testcontainer-backed, real
Postgres→ClickHouse replication):
- **forward** pagination returns every run exactly once when `run_id`
order is the reverse of `created_at` order (reproduces the
duplicate/skip bug — fails on `main`; this
walk-until-`nextCursor`-null-and-assert-complete is exactly the bulk
action's iteration),
- **backward** pagination round-trips to the previous page across a
boundary,
- **legacy** bare-`run_id` cursor still uses the old predicate
(backwards compatibility).
The existing `runsRepository` suites (part1–4) still pass; `part4`'s
`count new runs with listRunIds` test was updated for the new `{ runIds,
pagination }` return shape, and the `clickhouse` `taskRuns`
query-builder snapshots were regenerated for the added `created_at_ms`
column.
## Notes
- Separate, pre-existing issue (out of scope, not introduced here):
`listRuns`' backward display-slicing (`rows.slice(1, size+1)` when
`hasMore`) has an off-by-one that can return a straddled page. Tracked
separately.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Two small hygiene tweaks to **dev-only** images:
- `docker/Dockerfile.postgres`: add `--no-install-recommends` to the
partman install (leaner image, skips unneeded recommended packages).
- `internal-packages/clickhouse/Dockerfile`: run the migration helper as
a non-root user.
Both are local-dev images (the `pnpm run docker` stack) - no impact on
the published webapp image, prod, or self-hosting.
Speeds up and de-flakes the unit-test suite: testcontainers booted once
per vitest worker (per-test isolation kept only where a test runs
background redis work that outlives it), a duration-weighted shard
sequencer so each shard does roughly equal work, the slowest suites
split, two genuine flakes fixed (`streamBatchItems` shared-redis leak;
run-engine waits that relied on fixed sleeps), and transient DockerHub
pulls retried.
**Timings (CI, per-shard wall):** worst unit-test shard ~771s → ~294s;
packages/webapp shards ~250-270s, most internal ~190-240s. All 25 shards
green.
A shard breaks down as ~70s fixed setup (install / image-pull /
generate) + ~70s cold `^build` + the actual container tests. So the
remaining cost is mostly the tests themselves plus that fixed setup.
**Next (separate, timings):**
- **typecheck (~6m24s)** — the slowest check overall; bound by
full-graph `tsc`, not the TS version (a TS6 branch is still ~6m17s). The
real lever is **tsgo** (the Go compiler).
- Possible later: turbo CI caching could trim the ~70s cold build on
*warm* runs, but it's conditional (cold runs rebuild anyway) and doesn't
touch setup or test time — secondary.
`cli-v3` e2e and `sdk-compat` are path-gated (don't run on test-infra
changes) and already comfortably fast.
## Summary
Adds a trace export to the run page. From the new **Export trace** menu
you can
copy a run's full trace to the clipboard as Markdown (for pasting into
an AI
assistant) or download it as a flat Log, a Markdown table, or JSON
Lines.
Internal engine-debug events are filtered out by default, and errors are
surfaced inline with their message.
## Design
The export streams events from the store to the gzipped response one at
a time
and never materialises the span tree, so a trace of any size exports
with
bounded memory and without stalling the server. Output is flat and
chronological: each line carries its own `spanId ← parentSpanId`, so the
hierarchy is reconstructable without nesting. Formats share a single
streaming
pipeline and are pluggable via `?format=log|jsonl|markdown`, so adding a
format
is an isolated change.
## Screenshots
**Export menu**
<img width="370" height="252" alt="trace-export-menu"
src="https://github.com/user-attachments/assets/3d10304a-8c49-4606-b15d-2859b137419f"
/>
**In context**
<img width="2400" height="1802" alt="trace-export-run-page"
src="https://github.com/user-attachments/assets/46c80b30-303b-47c6-9ace-a2fb06f6cb61"
/>
A unit-test shard recently failed on a timing race rather than a real
regression - a run-engine waitpoint test sleeps 1250ms waiting on a
1000ms timeout that's processed by a ~1000ms worker poll, so on a
CPU-starved shard the margin evaporates and the whole matrix goes red.
Because `fail-fast` defaults on, that one flake cancels the sibling
shards, and the only recovery is re-running the entire matrix "just to
be sure" - which is itself slow.
This is the low-risk first pass at that pain:
- `fail-fast: false` on the webapp and internal shard matrices, so one
flaky shard no longer cancels its siblings. "Re-run failed jobs" now
re-runs just the failed shard instead of the whole matrix.
- CI-scoped `retry: process.env.CI ? 2 : 0` on the timing-sensitive
packages (`run-engine`, `redis-worker`, `schedule-engine`). Flakes
self-heal in CI; local runs stay at `retry: 0` so they still surface in
dev. A stopgap until the timing tests are made deterministic.
- `fetch-depth: 1` on the unit-test checkouts - they don't use git
history, so the full clone was wasted setup time across ~20 jobs.
- Reconcile the pre-pull image tags with what testcontainers actually
pulls (`redis:7-alpine` -> `redis:7.2`, `ryuk:0.11.0` -> `ryuk:0.14.0`)
and add `minio/minio:latest` to the webapp pre-pull. Otherwise those
images pull unauthenticated at test time and risk Docker Hub rate-limit
flakes (worst on fork PRs, where the authenticated pre-pull is skipped
entirely).
Deeper follow-ups - bigger runners, turbo remote cache, runtime-weighted
sharding, and the real root-cause fix (container reuse / template-DB
isolation + deterministic timing tests) - are tracked under TRI-10484.
Hardens the webapp Docker image and adds a CVE scan of each published
image.
- Base image `bullseye-slim` → `bookworm-slim` (Debian 12), pinned by
digest. Adds `apt-get upgrade` + `--no-install-recommends` + apt-cache
cleanup across the build stages so OS packages are patched at build
time.
- Moves the `react-email` CLI to `devDependencies` in
`internal-packages/emails` — only the `email dev` preview script uses
it; the runtime render path is `@react-email/render` +
`@react-email/components`. This also drops the bundled `esbuild` binary
from the production image.
- Bumps `goose` v3.26.0 → v3.27.1 and its Go builder image 1.23 → 1.26.
- Adds a reusable Trivy image-scan workflow wired into `publish.yml`, so
every published image (main builds and releases) is scanned for
OS-package CVEs right after it's pushed to GHCR. Report-only (writes to
the run summary), runs alongside the worker publishes so it never blocks
a deploy.
Verified locally: the image builds clean on the new base, and
`@react-email/render` carries no `esbuild` dependency so email rendering
is unaffected.
## Summary
The SDK and core packages run a second, forward-compat typecheck pass
(`tsc --noEmit -p tsconfig.ai-v7.json`) that remaps the `"ai"` import to
the ESM-only AI SDK 7 canary, so we catch source that only compiles
against one major. That pass inherited `composite: true` from the base
tsconfig, which makes `tsc` write a `.tsbuildinfo` even under
`--noEmit`.
Incremental buildinfo caches each file's resolved module format (CJS vs
ESM) and module resolution. When that state goes stale or is replayed,
the v7 pass can report spurious `TS1479` ("CommonJS module ... cannot
`require` an ECMAScript module") errors on the `"ai"` import even though
the source is fine in a clean checkout. Because this pass shares the
typecheck job that gates the Docker image publish, a spurious failure
there blocks publishing.
## Fix
Set `composite: false` and `incremental: false` on both
`tsconfig.ai-v7.json` files. The pass is `--noEmit` only, so it never
needed incremental state. Now each run is a clean, full check that
writes no buildinfo and can't replay stale resolution.
Verified: both `@trigger.dev/sdk` and `@trigger.dev/core` typecheck
green, and neither writes an ai-v7 `.tsbuildinfo` anymore.
## Summary
Opening or closing the Bulk action inspector should not affect the Runs
list, but it was still triggering refresh-related UI behavior. This PR
fixes that and smooths out a few related inspector interactions.
## Changelog
Stop reloading the runs list (and flashing its loading state) when
opening or closing the
Bulk action inspector. Filtering, pagination, and explicit refresh are
unaffected.
## Summary
Documents AI SDK 7 support in the AI Chat docs. Pairs with the SDK
change in #3833.
- The reference compatibility matrix now lists the v7 peer range and
adds an `@ai-sdk/otel` row.
- A new "AI SDK 7 telemetry" section covers the `@ai-sdk/otel` install,
the automatic registration, and the `TRIGGER_AI_SDK_OTEL_AUTOREGISTER`
opt-out.
- The quick start surfaces the supported `ai` versions (v5/v6/v7) up
front, near where you install.
## Summary
1 new feature, 8 improvements, 1 bug fix.
## Highlights
- Add optional `shouldPauseScaling` to the supervisor consumer pool
scaling options to freeze scale-up while it returns true (scale-down
stays allowed).
([#3836](https://github.com/triggerdotdev/trigger.dev/pull/3836))
## Improvements
- The MCP server no longer tells the AI agent to wait for a run to
complete after every `trigger_task` call. Waiting is now opt-in: the
agent only waits when you ask it to (for example "trigger and then wait
for it to finish"). This avoids burning tokens polling runs you didn't
need to block on and keeps responses clearer.
([#3838](https://github.com/triggerdotdev/trigger.dev/pull/3838))
- Update the bundled OpenTelemetry packages to their latest releases
(`@opentelemetry/sdk-node` 0.218.0, `@opentelemetry/core` 2.7.1,
`@opentelemetry/host-metrics` 0.38.3).
([#3810](https://github.com/triggerdotdev/trigger.dev/pull/3810))
- `envvars.upload` now accepts an optional `isSecret` flag, letting you
create the imported variables as secret (redacted) environment
variables. When omitted, variables default to non-secret.
([#3809](https://github.com/triggerdotdev/trigger.dev/pull/3809))
- Offload large trigger payloads to object storage before sending the
trigger API request. The SDK uploads packets at or above the existing
128KB limit and sends an `application/store` pointer instead of
embedding large JSON in the request body. `TriggerTaskRequestBody` now
validates that `application/store` payloads are non-empty storage paths.
([#3785](https://github.com/triggerdotdev/trigger.dev/pull/3785))
- Make mollifier buffer and drainer internals configurable.
`MollifierBuffer` now accepts `ackGraceTtlSeconds`,
`maxRetriesPerRequest`, `reconnectStepMs`, and `reconnectMaxMs` options,
and `MollifierDrainer` accepts `maxBackoffMs` and `backoffFloorMs`. All
default to their previous hardcoded values, so existing behaviour is
unchanged.
([#3822](https://github.com/triggerdotdev/trigger.dev/pull/3822))
- `MollifierDrainer` accepts a `drainBatchSize` option (default 1) that
controls how many entries are popped per env per tick — in-flight
handlers remain capped by the global `concurrency`. `MollifierBuffer`
also gains `getDrainingCount()` / `listStaleDraining()`, backed by a new
`mollifier:draining` ZSET maintained atomically with
pop/ack/fail/requeue (observability-only).
([#3797](https://github.com/triggerdotdev/trigger.dev/pull/3797))
- Adds AI SDK 7 support. The `ai` peer range now includes v7, and the
`chat.agent` / chat surfaces work against v7's ESM-only build. On v7,
install `@ai-sdk/otel` alongside `ai` and the SDK registers it for you
so `experimental_telemetry` spans keep flowing into your run traces (v7
stopped emitting them from `ai` core). v5 and v6 keep working unchanged.
([#3833](https://github.com/triggerdotdev/trigger.dev/pull/3833))
- `useTriggerChatTransport` now recovers when restored session state
points at a session that no longer exists in the current environment
([#3816](https://github.com/triggerdotdev/trigger.dev/pull/3816))
## Bug fixes
- Fix `@trigger.dev/core` build: cast the underlying log record exporter
when calling `forceFlush` so it typechecks against the updated
OpenTelemetry `LogRecordExporter` type (which no longer declares
`forceFlush`).
([#3829](https://github.com/triggerdotdev/trigger.dev/pull/3829))
<details>
<summary>Raw changeset output</summary>
⚠️⚠️⚠️⚠️⚠️⚠️
`main` is currently in **pre mode** so this branch has prereleases
rather than normal releases. If you want to exit prereleases, run
`changeset pre exit` on `main`.
⚠️⚠️⚠️⚠️⚠️⚠️
# Releases
## @trigger.dev/build@4.5.0-rc.5
### Patch Changes
- Updated dependencies:
- `@trigger.dev/core@4.5.0-rc.5`
## trigger.dev@4.5.0-rc.5
### Patch Changes
- The MCP server no longer tells the AI agent to wait for a run to
complete after every `trigger_task` call. Waiting is now opt-in: the
agent only waits when you ask it to (for example "trigger and then wait
for it to finish"). This avoids burning tokens polling runs you didn't
need to block on and keeps responses clearer.
([#3838](https://github.com/triggerdotdev/trigger.dev/pull/3838))
- Update the bundled OpenTelemetry packages to their latest releases
(`@opentelemetry/sdk-node` 0.218.0, `@opentelemetry/core` 2.7.1,
`@opentelemetry/host-metrics` 0.38.3).
([#3810](https://github.com/triggerdotdev/trigger.dev/pull/3810))
- Updated dependencies:
- `@trigger.dev/core@4.5.0-rc.5`
- `@trigger.dev/build@4.5.0-rc.5`
- `@trigger.dev/schema-to-json@4.5.0-rc.5`
## @trigger.dev/core@4.5.0-rc.5
### Patch Changes
- Add optional `shouldPauseScaling` to the supervisor consumer pool
scaling options to freeze scale-up while it returns true (scale-down
stays allowed).
([#3836](https://github.com/triggerdotdev/trigger.dev/pull/3836))
- Fix `@trigger.dev/core` build: cast the underlying log record exporter
when calling `forceFlush` so it typechecks against the updated
OpenTelemetry `LogRecordExporter` type (which no longer declares
`forceFlush`).
([#3829](https://github.com/triggerdotdev/trigger.dev/pull/3829))
- `envvars.upload` now accepts an optional `isSecret` flag, letting you
create the imported variables as secret (redacted) environment
variables. When omitted, variables default to non-secret.
([#3809](https://github.com/triggerdotdev/trigger.dev/pull/3809))
```ts
await envvars.upload("proj_1234", "prod", {
variables: { STRIPE_SECRET_KEY: "sk_live_..." },
isSecret: true,
});
```
- Offload large trigger payloads to object storage before sending the
trigger API request. The SDK uploads packets at or above the existing
128KB limit and sends an `application/store` pointer instead of
embedding large JSON in the request body. `TriggerTaskRequestBody` now
validates that `application/store` payloads are non-empty storage paths.
([#3785](https://github.com/triggerdotdev/trigger.dev/pull/3785))
Payload uploads use the same resolved `ApiClient` as the trigger call
(including `requestOptions.clientConfig`), not only the global
`apiClientManager.client` — so custom `baseURL`, access token, and
preview branch apply to both presign and trigger.
- Update the bundled OpenTelemetry packages to their latest releases
(`@opentelemetry/sdk-node` 0.218.0, `@opentelemetry/core` 2.7.1,
`@opentelemetry/host-metrics` 0.38.3).
([#3810](https://github.com/triggerdotdev/trigger.dev/pull/3810))
## @trigger.dev/plugins@4.5.0-rc.5
### Patch Changes
- Updated dependencies:
- `@trigger.dev/core@4.5.0-rc.5`
## @trigger.dev/python@4.5.0-rc.5
### Patch Changes
- Updated dependencies:
- `@trigger.dev/sdk@4.5.0-rc.5`
- `@trigger.dev/core@4.5.0-rc.5`
- `@trigger.dev/build@4.5.0-rc.5`
## @trigger.dev/react-hooks@4.5.0-rc.5
### Patch Changes
- Updated dependencies:
- `@trigger.dev/core@4.5.0-rc.5`
## @trigger.dev/redis-worker@4.5.0-rc.5
### Patch Changes
- Make mollifier buffer and drainer internals configurable.
`MollifierBuffer` now accepts `ackGraceTtlSeconds`,
`maxRetriesPerRequest`, `reconnectStepMs`, and `reconnectMaxMs` options,
and `MollifierDrainer` accepts `maxBackoffMs` and `backoffFloorMs`. All
default to their previous hardcoded values, so existing behaviour is
unchanged.
([#3822](https://github.com/triggerdotdev/trigger.dev/pull/3822))
- `MollifierDrainer` accepts a `drainBatchSize` option (default 1) that
controls how many entries are popped per env per tick — in-flight
handlers remain capped by the global `concurrency`. `MollifierBuffer`
also gains `getDrainingCount()` / `listStaleDraining()`, backed by a new
`mollifier:draining` ZSET maintained atomically with
pop/ack/fail/requeue (observability-only).
([#3797](https://github.com/triggerdotdev/trigger.dev/pull/3797))
- Updated dependencies:
- `@trigger.dev/core@4.5.0-rc.5`
## @trigger.dev/rsc@4.5.0-rc.5
### Patch Changes
- Updated dependencies:
- `@trigger.dev/core@4.5.0-rc.5`
## @trigger.dev/schema-to-json@4.5.0-rc.5
### Patch Changes
- Updated dependencies:
- `@trigger.dev/core@4.5.0-rc.5`
## @trigger.dev/sdk@4.5.0-rc.5
### Patch Changes
- Adds AI SDK 7 support. The `ai` peer range now includes v7, and the
`chat.agent` / chat surfaces work against v7's ESM-only build. On v7,
install `@ai-sdk/otel` alongside `ai` and the SDK registers it for you
so `experimental_telemetry` spans keep flowing into your run traces (v7
stopped emitting them from `ai` core). v5 and v6 keep working unchanged.
([#3833](https://github.com/triggerdotdev/trigger.dev/pull/3833))
- `useTriggerChatTransport` now recovers when restored session state
points at a session that no longer exists in the current environment
([#3816](https://github.com/triggerdotdev/trigger.dev/pull/3816))
- Offload large trigger payloads to object storage before sending the
trigger API request. The SDK uploads packets at or above the existing
128KB limit and sends an `application/store` pointer instead of
embedding large JSON in the request body. `TriggerTaskRequestBody` now
validates that `application/store` payloads are non-empty storage paths.
([#3785](https://github.com/triggerdotdev/trigger.dev/pull/3785))
Payload uploads use the same resolved `ApiClient` as the trigger call
(including `requestOptions.clientConfig`), not only the global
`apiClientManager.client` — so custom `baseURL`, access token, and
preview branch apply to both presign and trigger.
- Update the bundled OpenTelemetry packages to their latest releases
(`@opentelemetry/sdk-node` 0.218.0, `@opentelemetry/core` 2.7.1,
`@opentelemetry/host-metrics` 0.38.3).
([#3810](https://github.com/triggerdotdev/trigger.dev/pull/3810))
- Updated dependencies:
- `@trigger.dev/core@4.5.0-rc.5`
</details>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
## Summary
The Prisma CLI was missing from production builds of the webapp image,
so anything that shells out to `prisma` at startup failed. The container
entrypoint and the standalone migration step both run `prisma migrate
deploy` / `prisma migrate status`, and those broke with `Command
"prisma" not found`.
## Fix
`prisma` was a `devDependency` of `@trigger.dev/database`. It had only
been landing in the pruned `--prod` install as a side effect of pnpm
auto-installing it as a peer of `@prisma/client`. A recent dependency
change shifted peer resolution so prisma stopped being materialized into
the production tree, and the CLI disappeared from the image.
Moving `prisma` into `dependencies` of `@trigger.dev/database` makes the
CLI an explicit part of production installs. It lands in the webapp
image only: the separately deployed supervisor, coordinator, and
provider images don't reach the database package in their production
trees (`core` only `devDepends` on it, so it isn't transitive), so
they're unaffected.
Verified against a locally built production image: `pnpm --filter
@trigger.dev/database exec prisma --version` now resolves the CLI and
the schema engine instead of failing.
The supervisor can now pause dequeuing - and freeze consumer-pool
scale-up - when a backpressure signal says the cluster can't place more
work, then ramp dequeuing back up gradually once it clears. The signal
is a verdict published to a Redis key by a cluster-side component; the
supervisor reads it on a short refresh and gates `preDequeue` on it.
Off by default (`TRIGGER_DEQUEUE_BACKPRESSURE_ENABLED`). Everything
fails open: a missing, stale, or unreadable verdict never pins the
brake, and the hot-path read is a synchronous cached lookup with no I/O.
The scale-up freeze leaves scale-down untouched, and on release the
resume is ramped so a deep queue isn't hammered all at once.
Dry-run is on by default (`TRIGGER_DEQUEUE_BACKPRESSURE_DRY_RUN`): even
once enabled it only logs what it would have done, and surfaces the
computed state through metrics, until explicitly set to act. Prometheus:
`supervisor_backpressure_engaged`, `_dry_run`,
`_skipped_dequeues_total`.
Refs TRI-5354
## Summary
Scheduled runs and their descendants can now be routed to a dedicated
per-region worker queue, processed by a separate worker fleet, so a
burst of scheduled crons no longer competes with standard and agent runs
for the same queue and inflates their startup latency. It is off by
default and enabled per organization via a feature flag (with a global
default), so nothing changes until it is turned on.
## Design
At trigger time, any run whose lineage originates from a schedule
(`rootTriggerSource === "schedule"`, which already propagates from a
scheduled run down to all of its children) gets its worker queue
suffixed with `:scheduled`. The worker queue name is an opaque string
persisted on the run and used verbatim by enqueue and dequeue, so this
needs no Lua, message-envelope, or concurrency changes. Concurrency
stays keyed by environment and queue, not by worker queue.
On the consumer side, the dequeue endpoint gains an optional
`queueClass` selector. A supervisor sends `queueClass: "scheduled"` and
the server derives the actual queue from the worker's own group, so a
token can only ever reach its own region's queues. A fleet picks its
class with the `TRIGGER_WORKER_QUEUE_CLASS` env var (`default` or
`scheduled`), so a dedicated scheduled fleet can run alongside the
standard one.
Verified end to end against a local managed-worker setup: scheduled runs
route to the dedicated queue, are drained only by the scheduled fleet,
and standard runs are left untouched.
## Summary
The Trigger.dev MCP server told the AI agent to wait for the run to
complete after every `trigger_task` call. The agent followed that
instruction even when the user only wanted to fire-and-forget, which
burned tokens polling runs nobody needed to block on and made responses
less clear.
Waiting is now opt-in. After triggering, the response tells the agent
the run is executing in the background and to only wait if the user
asked it to (for example "trigger and then wait for it to finish"). The
`trigger_task` tool description is updated to match. The
`wait_for_run_to_complete` tool itself is unchanged, so explicit waits
still work.
## Summary
This PR adds packet path validation before key construction and
presigning. Invalid paths are rejected before reaching either
object-store client implementation, ensuring consistent behavior
regardless of the underlying storage configuration.
## Summary
Adds support for Vercel AI SDK 7. The `ai` peer range now includes v7,
and the `chat.agent` / chat surfaces work against v7's ESM-only build.
v5 and v6 keep working unchanged, so this is additive.
## Telemetry on v7
On v7, model-call spans moved out of `ai` core into the separate
`@ai-sdk/otel` adapter, so `experimental_telemetry` alone produces
nothing until an integration is registered. Install `@ai-sdk/otel`
alongside `ai@7` and the SDK registers it once per worker at chat agent
boot, so spans keep flowing into run traces with no extra setup.
If you (or a library you import) already register `@ai-sdk/otel`, the
SDK detects the existing integration and skips its own registration, so
you won't get duplicate spans. Set `TRIGGER_AI_SDK_OTEL_AUTOREGISTER=0`
to disable auto-registration entirely.
## Notes
`ai@7` is ESM-only, which tripped TS1479 in the SDK's CommonJS build.
Runtime value imports from `ai` are isolated behind a paired ESM/CJS
shim so both module formats resolve the right form; type-only imports
stay as direct `import type` at their use sites.
## Summary
Three fixes to the AI agent surface in the dashboard, all surfaced while
verifying AI SDK 7 support (the SDK side is #3833):
- **AI SDK 7 telemetry rendering.** The generation-span inspector and
run metrics now read both the v6 (`ai.*`) and v7 (`gen_ai.*`) telemetry
attribute shapes. On v7 the Messages, Provider, and Tools views showed
empty/unknown before; now they render correctly.
- **HITL approvals in the conversation view.** The agent conversation
view renders human-in-the-loop tool approvals and denials (awaiting
approval, approved, denied with reason) instead of leaving the tool part
blank. This gap predated v7.
- **Chat snapshot store mismatch.** Chat session snapshots now resolve
through a single storage-key helper shared by the SDK write and the
dashboard read. Previously the write applied the default object-store
protocol to the key while the read fell back to a different store, so
the dashboard 404'd on the snapshot and showed only a partial
conversation.
## Summary
On a busy webapp the trace/log/metric ingestion path emits several
`info` logs per insert batch, which makes up the bulk of the service's
log output. This moves that per-batch chatter to `debug` and adds an
opt-in to drop successful HTTP access logs, cutting log volume with no
loss of error signal.
## Details
The per-batch ClickHouse insert logs, the flush scheduler's concurrency
adjustments, and the event-loop utilization sample (already exported as
a metric, so the log line was redundant) now log at `debug`. Error and
warning logs are untouched.
New `HTTP_ACCESS_LOG_DISABLED=1` env var: when set, the HTTP access
logger skips successful (2xx) requests while still logging non-2xx
responses. Defaults off, so existing deployments are unchanged.
## Summary
This PR improves performance across the Environment Variables page.
## Changes
### Targeted value loading
- load only the non-secret (environmentId, key) pairs required by the
page. Secret values continue to be redacted in the UI.
### SSR windowing + virtualization
- SSR-render only the first 50 rows
- hydrate those rows
- virtualize the remaining dataset client-side
- search is now URL-driven during SSR, ensuring deep links such as
`?search=DATABASE_URL`
### Lightweight 'Create' flow
- 'Create' page no longer loads the full Environment Variables dataset.
## Results
Large projects no longer render thousands of rows during SSR.
Example (~11k rendered rows):
Metric | Before | After
-- | -- | --
Document size | ~150 MB | ~5 MB
SSR rows | ~11k | 50
Browser DOM rows | Thousands | ~26–38
## Summary
The mollifier had ~21 behavioural constants baked in as hardcoded values
— the buffer's ack-grace TTL and Redis retry/reconnect tuning, the
drainer's poll interval and backoff envelope, the pre-gate idempotency
claim TTL/wait/poll, the buffered-run mutate-with-fallback wait loop,
the metadata CAS retry budget and backoff, the stale-sweep scan bounds,
and the draining-gauge interval. None could be adjusted without a code
change, which makes tuning the system under production load impossible.
This exposes all of them as `TRIGGER_MOLLIFIER_*` environment variables,
each defaulting to its previous hardcoded value. Behaviour is identical
unless an operator sets a var, so it's a safe no-op deploy.
## Design
The package-level classes (`MollifierBuffer`, `MollifierDrainer` in
`@trigger.dev/redis-worker`) gain optional constructor options
defaulting to the old constants — backward compatible, hence a patch
changeset. The webapp factories and worker bootstraps read the env and
pass them through. The route- and concern-level pure helpers
(mutate-with-fallback, metadata mutation, idempotency claim, stale-sweep
state) keep their existing `?? DEFAULT` option fallbacks and are fed env
values at their call sites, so they stay unit-testable without importing
`env.server`.
## Test plan
- [x] `@trigger.dev/redis-worker` builds
- [x] webapp typecheck passes
- [x] mollifier buffer + drainer testcontainer suites pass (modulo a
couple of pre-existing flaky timing tests)
- [x] Reviewer: confirm the `TRIGGER_MOLLIFIER_*` env var names match
ops conventions
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>
Adds backward-compatible support for large trigger payloads by reusing
the existing object-storage packet flow.
Large payloads are uploaded to object storage before the trigger request
is sent. The trigger API receives a small application/store pointer
payload instead of embedding large JSON bodies in the request.
Small payload behavior is unchanged.
## Summary
Adds a nullable `pricing_unit` column to the LLM model registry's
`llm_models` table, recording how each model is billed ("tokens",
"characters", "images", "minutes", "requests", "free", "not_findable").
It lets pricing-coverage reporting exclude models that aren't priced
per-token (image/video/audio models currently drag the "% priced" number
down even though they can never carry a per-token price), and lays the
groundwork for non-token pricing.
The default model catalog is entirely per-token, so `seed` and
`syncLlmCatalog` set `pricing_unit="tokens"` on those rows. The admin
LLM model form (create + edit) and the admin API get a pricing-unit
selector so admin-curated models can set it; existing rows can stay
unset.
Auto-discovered models get their unit from the model-registry pipeline,
which lands separately.
---------
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
## Summary
When a chat's restored session state points at a session that no longer
exists in the current environment — for example a `sessions` entry that
was persisted against a different trigger environment —
`useTriggerChatTransport` assumed the session was live and never created
a real one. The next message then failed with a 404 and the chat
couldn't send.
## Fix
`callWithAuthRetry` now treats a 404 from a session-PAT-authed call as
"this session doesn't exist here". After the existing 401/403 token
refresh, a 404 recreates the session via `startSession`, drops the stale
`lastEventId` resume cursor (it pointed at another environment's
stream), and retries the send once. When `startSession` isn't configured
the transport throws a clear message instead of a bare 404.
Routine maintenance pass on a few transitive `pnpm.overrides`.
- `fast-uri` / `fast-xml-builder`: add overrides pinning to current
releases (`3.1.2` / `1.1.7`).
- `protobufjs` / `qs`: bump existing override pins that had fallen a
patch behind (`7.5.6` / `6.15.2`).
Overrides-only - no first-party code changes; lockfile regenerated to
match. Verified the affected transitives resolve to the pinned releases
via `pnpm why -r`.
## ✅ Checklist
- [x] I have followed every step in the [contributing
guide](https://github.com/triggerdotdev/trigger.dev/blob/main/CONTRIBUTING.md)
- [x] The PR title follows the convention.
- [x] I ran and tested the code works
---
## Testing
Upgraded the email packages in `internal-packages/emails`:
| Package | Before | After |
| --- | --- | --- |
| `@react-email/components` | `0.0.16` | `1.0.12` |
| `@react-email/render` | `^0.0.12` | `^2.0.8` |
| `react-email` (CLI) | `^2.1.1` | `^6.5.0` |
| `react-dom` | _(missing)_ | `^18.2.0` (now a required peer of render)
|
**Breaking change handled:** `render()` is now async (`Promise<string>`)
in `@react-email/render` v1+. Added `await` in the `aws-ses`, `smtp` and
`null` transports. `EmailClient` and the webapp callers were already
async and needed no changes.
Verification:
- `pnpm run typecheck --filter emails` ✅
- `pnpm run typecheck --filter webapp` (consumer of the `emails`
package) ✅
- **Before/after render comparison**: rendered all 11 templates
(magic-link, invite, welcome, alert-attempt/run/error-group,
deployment-failure/success, mfa-enabled/disabled, bulk-action-complete)
to HTML with both the old and new packages and compared them visually +
via HTML diff. Output is visually identical. The only HTML changes come
from upstream improvements: `<Body>` now wraps content in a
`<table>`/`<td>` for better email-client compatibility, an
`x-apple-disable-message-reformatting` meta tag was added, and CSS
shorthand (e.g. `margin`) is now also emitted as longhand. No visual
regressions; the `CodeBlock`/dracula theme, buttons, and row/column
layouts all render correctly.
No changeset or `.server-changes/` entry is added: `emails` is a private
internal package (not under `packages/`), and there is no user-facing
behavior change.
---
## Changelog
Upgrade `react-email` and `@react-email/{components,render}` in
`internal-packages/emails` to their latest versions and adapt the mail
transports to the now-async `render()` API.
---
## Screenshots
Rendered email templates before vs after the upgrade (visually
identical):
**Before** (`@react-email/components@0.0.16`, `render@0.0.12`)
## Summary
A fresh clone running `trigger dev` against the local webapp shows the
run in the dashboard but an empty trace view.
`EVENT_REPOSITORY_DEFAULT_STORE` was unset in `.env.example`, so span
ingestion fell back to the `postgres` store while the rest of the local
stack (run replication, `CLICKHOUSE_URL`) is ClickHouse-backed.
Defaulting it to `clickhouse_v2` makes dev run traces show up out of the
box.
The reference/example projects (`references/`) now live in their own
repo, https://github.com/triggerdotdev/references, so their heavy,
frequently-changing dependencies are no longer part of this repo's
lockfile and tooling. This removes them here and repoints everything
that referenced them.
- Deletes `references/`; updates the pnpm workspace + lockfile.
- Clears the references-only CI rules and `.vscode` configs.
- Repoints the docs (contributor/agent + one public page) to the new
repo.
- `seed.mts` keeps the local-dev projects (hello-world, d3-chat,
realtime-streams).
Brings the OpenTelemetry packages up to the latest coherent release
across the webapp and the published packages (`@trigger.dev/core`, the
CLI, `@trigger.dev/sdk`) plus
`internal-packages/{tracing,testcontainers}`:
- `@opentelemetry/sdk-node` 0.218.0
- `@opentelemetry/core` 2.7.1
- `@opentelemetry/host-metrics` 0.38.3
We were already on the otel 2.x line, so this is a same-major minor move
- the versions are pinned to `@opentelemetry/sdk-node@0.218.0`'s own
declared dependency set so the experimental (0.2xx) and stable (2.x)
packages stay coherent (mixing them is the usual otel breakage).
**One code change:** otel 0.215 made `forceFlush()` a required method on
`LogRecordExporter`, so `ExternalLogRecordExporterWrapper` (core's
tracing SDK) gains a `forceFlush()` that delegates to the underlying
exporter.
**Notable upgrades along the way:** OTLP exporters can take a custom
HTTP agent (connection pooling/keepAlive on the export path), HTTP
request headers are captured at span creation, and core hot-path perf
improvements in 2.6.1/2.7. `host-metrics` 0.37→0.38 is a clean upgrade.
Patch changeset added for the three published packages. References
projects are intentionally untouched.
Verified: `@trigger.dev/core` / CLI / `@trigger.dev/sdk` build, webapp +
`@internal/tracing` typecheck - all green.
Adds wide-event observability for the supervisor: one flat-keyed JSON
line per dequeue iteration, workload-server route, and run socket
lifecycle event. Events carry `trace_id` sourced from the inbound W3C
traceparent plus `meta.run_id` and related identifiers, so they join
across services by run.
The outbound warm-start POST also forwards the inbound traceparent so
the upstream receiver continues the same trace instead of minting a new
one.
Off by default behind `TRIGGER_WIDE_EVENTS_ENABLED`. With the flag off,
no events are emitted, no ALS state is allocated, and the outbound
warm-start request is unchanged — every call site was audited to confirm
the off path is byte-identical to current behavior.
Dequeue-path phase timings recorded under `phase.<name>.duration_ms`:
`restore`, `warm_start`, `workload_create`. A `path_taken` extra
distinguishes `restore` / `warm_start` / `cold_create` /
`skipped_no_image`.
Refs TRI-9480.
## Summary
The Runs list now updates live without requiring a page refresh. Status
changes and other run fields are updated in place while runs are
executing.
When new runs matching the current filters are created, a "New runs
created" refresh button appears above the list.
Root runs now show a live child-run status breakdown directly in the
status tooltip.
### List live update
- Visible runs update in place while they are still running.
- A "New runs created" refresh button appears when new matching runs are
detected.
- Polling stops when all visible runs have finished and a refresh button
is already shown.
- Polling pauses when the browser tab is not visible.
- Runs list status updates and new-run detection share a single
runs/live polling path.
### Child-status tooltip
- Root run tooltips now display a breakdown of child run statuses.
- Child statuses are loaded when the tooltip opens (after a 400ms hover
delay).
- The tooltip stays up to date while child runs are still changing
state.
- Handles cases where child runs continue running after their parent run
has completed, or have not yet been created.
### Supporting changes
- Added hidden-tab awareness to polling.
- Added safeguards around polling inputs (`runIds` deduping and limits).
## Test plan
- [x] pnpm run typecheck --filter webapp passes
- [x] cd apps/webapp && pnpm run test
./test/presenters/mapRunToLiveFields.test.ts --run passes
- [x] cd apps/webapp && pnpm run test
./test/runsRepository.part2.test.ts --run -t "hasNewRuns" passes
### Manual smoke:
- [x] Active runs update without a page refresh.
- [x] A new matching run shows the refresh banner and the banner actions
work as expected.
- [x] Root run tooltips show live child-status updates and stop polling
once child runs settle.
---------
Co-authored-by: Ekaterina Bulatova <kathiekiwi@Ekaterinas-MacBook-Pro.local>
## Problem
On environments where runs carry a Postgres-backed `taskEventStore`
value (`taskEvent` or `taskEventPartitioned`), OTLP ingest endpoints
(`POST /otel/v1/traces` and `/otel/v1/logs`) were returning HTTP 500.
**Root cause:** The org-scoped ClickHouse factory introduced in a recent
PR routes all OTLP spans through `getEventRepositoryForOrganizationSync`
→ `buildEventRepository`. That function only handles `"clickhouse"` and
`"clickhouse_v2"` store values and throws `Unknown ClickHouse event
repository store: <value>` for anything else. The throw occurred inside
the grouping loop of `#exportEvents`, unwinding the entire method and
returning 500 for the whole batch.
The OpenTelemetry collector's `otlphttp` exporter treats HTTP 500 as
non-retryable and drops the batch — causing real span loss.
**Fix:** Guard the `getEventRepositoryForOrganizationSync` call in
`#exportEvents` so it is only invoked for `clickhouse` / `clickhouse_v2`
store values. All other values are routed directly to the Postgres
`eventRepository`, matching the guard pattern already present in
`resolveEventRepositoryForStore` and `getEventRepositoryForStore` in
`eventRepository/index.server.ts`.
The ClickHouse factory call is also wrapped in a try/catch that falls
back to Postgres so any unexpected store value in a future OTLP batch
degrades gracefully instead of failing the whole request.
## Changes
- `apps/webapp/app/v3/otlpExporter.server.ts` — add Postgres routing
guard and try/catch fallback in `#exportEvents`
## Testing
The `eventRepository/index.server.ts` module already has the same guard
pattern thoroughly covered. The fix brings `#exportEvents` into
alignment with that existing, tested pattern. Manual verification:
confirm OTLP batches containing Postgres-store spans return 200 and
route to the correct repository.
## Summary
The environment variables import API now accepts an optional `isSecret`
flag, so imported variables can be created as secret (redacted)
environment variables instead of plaintext. When the flag is omitted,
variables default to non-secret, preserving existing behavior for CLI
deploys and dashboard imports.
This is useful for tools that push secrets into Trigger.dev (for
example, syncing from a secrets manager) and want them stored as secrets
rather than plain environment variables.
It's available through `envvars.import` in the SDK and the `POST
/api/v1/projects/{projectRef}/envvars/{slug}/import` endpoint, and is
honored for both regular and preview-branch environments.
```ts
await envvars.import("proj_1234", "prod", {
variables: { STRIPE_SECRET_KEY: "sk_live_..." },
isSecret: true,
});
```
## What
Adds [pkg.pr.new](https://pkg.pr.new) continuous preview releases. Every
push to a branch builds the public `@trigger.dev/*` packages and
publishes installable preview builds keyed by commit SHA — **without
touching the npm registry**. pkg.pr.new drops install instructions on
the associated PR:
```
npm i https://pkg.pr.new/@trigger.dev/sdk@<sha>
```
This lets reviewers and users try a branch (SDK, CLI, core, etc.) before
anything is released, separate from the changesets release, the manual
`--snapshot` prerelease, and the chat-prerelease flow.
## How
`.github/workflows/preview-packages.yml` (push trigger) → install →
generate Prisma → **stamp preview version** → build → `pkg-pr-new
publish`.
### The version stamp (the important part)
pkg.pr.new serves previews by SHA but does **not** rewrite the
package.json `version` field. If a preview shipped as `4.5.0-rc.4`, a
consumer who installed it would pin `4.5.0-rc.4` to the preview tarball
in their lockfile/cache — and a later `npm i
@trigger.dev/sdk@4.5.0-rc.4` from npm could resolve to the stale
preview. This is a known, by-design gap in the tool
(stackblitz-labs/pkg.pr.new#250, #390).
`scripts/stamp-preview-version.mjs` runs **before the build** and
rewrites every public package to a unique `0.0.0-preview-<sha>`. The
`0.0.0-` prefix can never satisfy a real semver range, so the collision
is structurally impossible (same convention React/Next canaries use).
Running before the build also means `scripts/updateVersion.ts` bakes the
preview version into the runtime `VERSION` constant, so previews are
self-identifying (`trigger --version`, the `x-trigger-cli-version`
header, the MCP server version) instead of all reporting the RC version.
Sibling `workspace:` specifiers are relaxed to `workspace:*` so `pnpm
pack` resolves them against the rewritten versions — `packages/python`
pins peerDependencies as `workspace:^4.5.0-rc.4`, which would otherwise
be unsatisfiable once the version changes. Non-public deps
(`@trigger.dev/database`, `@internal/*`) are left untouched. All
mutations happen on the ephemeral CI checkout; nothing is committed.
## GitHub App
The pkg.pr.new GitHub App is **already installed** on
`triggerdotdev/trigger.dev` (has been for a while), so no setup is
needed. Confirmed live — this branch's pushes published all 10 public
packages, e.g.
```
pnpm add https://pkg.pr.new/@trigger.dev/sdk@e4dfc59
```
## Fork limitation
pkg.pr.new authenticates with a GitHub Actions OIDC token, which GitHub
does not issue to `pull_request` workflows from forks. The `push`
trigger therefore covers branches pushed to this repo (core team), not
external fork PRs. Fork coverage would need a `workflow_run` two-stage
setup; left out for now.
## Notes
- Pinned `pkg-pr-new@0.0.75` (no Node engine constraint; Node 20 CI is
fine).
- pkg.pr.new
[#525](https://github.com/stackblitz-labs/pkg.pr.new/pull/525) adds a
built-in `--previewVersion` flag (still open). If it lands we can drop
the version-rewrite half of the script, but we'd keep a pre-build stamp
anyway so `updateVersion.ts` picks up the preview version (the flag
rewrites at pack time, too late for the baked `VERSION`).
## Summary
Adds `drainBatchSize` to `MollifierDrainer` (default `1` — preserves
existing behaviour) and wires `TRIGGER_MOLLIFIER_DRAIN_BATCH_SIZE`
through the webapp (default `50`). Each tick the drainer now pops up to
`drainBatchSize` from each chosen env, then dispatches every popped
entry through the shared `concurrency`-bounded `pLimit`. Per-org/per-env
fairness is unchanged — only the in-env pop count grows.
Pre-existing behaviour was one pop per env per tick. For a single-env
burst that single-flighted the drain at the per-tick floor of `pop +
engine.trigger ≈ 50–60 ms`. With buffer entries piling up under a
real-world tenant burst that's tens of minutes of tail latency to fully
materialise — even though PG itself could comfortably sustain the
writes.
## Why this matters — heavy-tail illustration
Scenario: 100 customers in one window — 94 fire 20 triggers each, 5 fire
100, 1 fires 1000. Gate at `THRESHOLD=10/s`, `HOLD_MS=500`. First 10 of
each burst hit PG directly; the rest buffer.
| Customers | Triggers each | PG direct | Buffered each | Total buffered
|
|---|---|---|---|---|
| 94 small | 20 | 10 | 10 | 940 |
| 5 medium | 100 | 10 | 90 | 450 |
| 1 heavy | 1000 | 10 | 990 | 990 |
**With `DRAIN_BATCH_SIZE=50`, `DRAIN_CONCURRENCY=50`, ~50 ms
`engine.trigger`:**
| Tick | Pops | Dispatch waves | Wall-clock |
|---|---|---|---|
| 1 | 94×10 + 5×50 + 1×50 = 1 240 | 25 × 50 ms | ~1 300 ms (94 smalls
done) |
| 2 | 5×40 + 1×50 = 250 | 5 × 50 ms | ~300 ms (5 mediums done) |
| 3–20 | heavy alone, 50/tick | 1 × 50 ms | ~100 ms each |
| Customer class | Buffered fully drained |
|---|---|
| 94 small | **~1.3 s** |
| 5 medium | **~1.6 s** |
| 1 heavy | **~3.4 s** |
**Without batching (one pop per env per tick — current behaviour):**
| Customer class | Buffered fully drained |
|---|---|
| 94 small | ~500 ms |
| 5 medium | ~4.5 s |
| 1 heavy | **~49 s** |
So the heavy single-tenant tail drops from ~49 s to ~3.4 s (~14× faster)
without changing PG load characteristics. Smalls go up slightly in this
scenario (500 ms → 1.3 s) because all 100 envs share one tick's dispatch
queue — that's the trade we accept for the heavy tail; the worst-case
small wait is still inside one tick. PG load is identical either way (50
concurrent inserts at a time, capped by `DRAIN_CONCURRENCY`).
## What changed
**`packages/redis-worker`**
- New `drainBatchSize` option (default 1 — full backward compat).
- `runOnce()` refactored to pop per-env batches in parallel, then
dispatch all popped entries through the existing global `pLimit`.
Mid-batch pop failure aborts only that env's batch and counts as one
failure (same semantic as the old per-env path).
- Removed the now-unused `processOneFromEnv` helper.
**`apps/webapp`**
- `TRIGGER_MOLLIFIER_DRAIN_BATCH_SIZE` env var (default 50, matching
`DRAIN_CONCURRENCY`).
- Wired into `mollifierDrainer.server.ts`.
**Test cloud config** (separate cloud PR):
`TRIGGER_MOLLIFIER_DRAIN_BATCH_SIZE="50"` on the worker service.
Production rollout deferred until we've watched it on test cloud.
## Test plan
- [x] All 25 stub-based drainer tests pass (18 pre-existing + 7 new). 7
new tests under `MollifierDrainer.drainBatchSize`:
- pops up to `drainBatchSize` across ticks
- global `concurrency` cap still holds when batch > concurrency
- mid-batch pop failure isolation
- multi-env batch fan-out in one tick
- **hierarchical org fairness preserved at `drainBatchSize > 1`**
(load-bearing — guards against future regressions to
per-env-instead-of-per-org rotation)
- mixed success/failure accounting in a batched tick
- bounded pops on empty queue (no Lua spam past `drainBatchSize`)
- [x] All pre-existing tests still pass unchanged at default
`drainBatchSize=1` → backward-compat locked.
- [x] `pnpm run build --filter @trigger.dev/redis-worker` clean.
- [x] `pnpm run typecheck --filter webapp` clean.
- [x] `redisTest` block (real Redis via testcontainers) — couldn't run
locally on this branch due to testcontainers runtime discovery; will
validate in CI.
- [ ] Test-cloud smoke after cloud PR lands: fire `burst 50` against a
flagged env and confirm the 50th entry's drain time drops from ~2.5 s to
<200 ms.
## Notes
- Per-tick memory bound: `maxOrgsPerTick × drainBatchSize` entries can
sit in the JS pLimit queue between pop and dispatch. At defaults that's
`500 × 50 = 25 000` × ~5 KB snapshot ≈ ~125 MB worst case per worker —
well within headroom.
- The pre-batch model's strict per-env throughput cap of `1/tick` is
documented as the fairness baseline elsewhere. Org-level fairness is
what callers actually rely on; this change does not weaken that.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Vouches GitHub user
[`ConProgramming`](https://github.com/ConProgramming) (Conner) as an
outside contributor by adding them to `.github/VOUCHED.td`, so their PRs
aren't auto-closed by the vouch check.
Done as a direct edit rather than via the issue flow because they have
no open Vouch Request issue to comment `vouch` on.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
## Summary
Documents the new `tools` option on `chat.agent` (companion to #3790).
Adds a dedicated [Tools](/ai-chat/tools) guide: the three places tools
show up (config, `toStreamTextOptions`, `streamText`), why declaring
them on the config matters for `toModelOutput` across turns, static vs
per-turn tools, the typed `run()` payload,
`InferChatUIMessageFromTools`, the relationship to skills, and the
manual `convertToModelMessages` path for `customAgent` loops.
Threads the option through the rest of the guide: the reference tables,
a happy-path section on the backend page, the types page, and the HITL /
skills / tool-result-auditing patterns. Corrects the sub-agents guide,
where the `toModelOutput` compression was implied to work across turns
but silently degraded from turn 2 without config tools.
Also unstacks the three callouts that were piled under the
`chat.agent()` header on the backend page, and adds a changelog entry.
## Summary
1 new feature, 5 improvements.
## Highlights
- Mollifier `mutateSnapshot` now enforces a tag cap: an `append_tags`
patch carrying `maxTags` returns `"limit_exceeded"` (writing nothing)
when the deduped tag count would exceed the limit, so a buffered run
can't accumulate more tags via the tags API than the trigger validator
allows at creation.
([#3756](https://github.com/triggerdotdev/trigger.dev/pull/3756))
## Improvements
- Mollifier buffer extensions: idempotency dedup, an atomic
`mutateSnapshot` API, metadata CAS, claim primitives, and a
`MollifierSnapshot` type. The buffer's Redis client now reconnects with
jittered backoff so a fleet of clients doesn't stampede Redis in
lockstep after a blip.
([#3752](https://github.com/triggerdotdev/trigger.dev/pull/3752))
- Add `onTerminalFailure` callback to `MollifierDrainerOptions` so the
customer's run lands a SYSTEM_FAILURE PG row even when the drainer
exhausts `maxAttempts` on a retryable PG error. Previously,
retryable-error exhaustion called `buffer.fail()` directly, which
atomically marks FAILED + DELs the entry hash with no PG write — silent
data loss when PG was unreachable across the full retry budget. The
callback fires before `buffer.fail()` on any terminal path (`cause:
"non-retryable"` or `"max-attempts-exhausted"`); throwing a retryable
error from the callback causes the drainer to requeue rather than fail.
([#3754](https://github.com/triggerdotdev/trigger.dev/pull/3754))
- Bump `@s2-dev/streamstore` to `0.22.10` to fix a
`TASK_RUN_UNCAUGHT_EXCEPTION` ("Invalid state: Unable to enqueue") when
a `chat.agent` turn is aborted mid-stream.
([#3792](https://github.com/triggerdotdev/trigger.dev/pull/3792))
- Coerce numeric `concurrencyKey` values to string at the API boundary
across `tasks.trigger`, `tasks.batchTrigger`, and the Phase-2 streaming
batch endpoint.
([#3789](https://github.com/triggerdotdev/trigger.dev/pull/3789))
- Add a `tools` option to `chat.agent`. Declaring your tools here
threads them into the SDK's internal `convertToModelMessages`, so each
tool's `toModelOutput` is re-applied when prior-turn history is
re-converted.
([#3790](https://github.com/triggerdotdev/trigger.dev/pull/3790))
<details>
<summary>Raw changeset output</summary>
⚠️⚠️⚠️⚠️⚠️⚠️
`main` is currently in **pre mode** so this branch has prereleases
rather than normal releases. If you want to exit prereleases, run
`changeset pre exit` on `main`.
⚠️⚠️⚠️⚠️⚠️⚠️
# Releases
## @trigger.dev/redis-worker@4.5.0-rc.4
### Minor Changes
- Mollifier buffer extensions: idempotency dedup, an atomic
`mutateSnapshot` API, metadata CAS, claim primitives, and a
`MollifierSnapshot` type. The buffer's Redis client now reconnects with
jittered backoff so a fleet of clients doesn't stampede Redis in
lockstep after a blip.
([#3752](https://github.com/triggerdotdev/trigger.dev/pull/3752))
- Add `onTerminalFailure` callback to `MollifierDrainerOptions` so the
customer's run lands a SYSTEM_FAILURE PG row even when the drainer
exhausts `maxAttempts` on a retryable PG error. Previously,
retryable-error exhaustion called `buffer.fail()` directly, which
atomically marks FAILED + DELs the entry hash with no PG write — silent
data loss when PG was unreachable across the full retry budget. The
callback fires before `buffer.fail()` on any terminal path (`cause:
"non-retryable"` or `"max-attempts-exhausted"`); throwing a retryable
error from the callback causes the drainer to requeue rather than fail.
([#3754](https://github.com/triggerdotdev/trigger.dev/pull/3754))
### Patch Changes
- Pipeline the per-entry `HGETALL` fetches in
`MollifierBuffer.listEntriesForEnv`. The previous serial implementation
issued one Redis round-trip per runId returned by `LRANGE`, which
dominated stale-sweep wall-time at any meaningful backlog (at the
sweep's default maxCount=1000, this is ~1000 RTTs per env per pass).
Behaviour is unchanged — entries are still skipped when the entry hash
has been torn down by a concurrent drainer ack/fail between the LRANGE
and the HGETALL.
([#3752](https://github.com/triggerdotdev/trigger.dev/pull/3752))
- Mollifier `mutateSnapshot` now enforces a tag cap: an `append_tags`
patch carrying `maxTags` returns `"limit_exceeded"` (writing nothing)
when the deduped tag count would exceed the limit, so a buffered run
can't accumulate more tags via the tags API than the trigger validator
allows at creation.
([#3756](https://github.com/triggerdotdev/trigger.dev/pull/3756))
- Updated dependencies:
- `@trigger.dev/core@4.5.0-rc.4`
## @trigger.dev/build@4.5.0-rc.4
### Patch Changes
- Updated dependencies:
- `@trigger.dev/core@4.5.0-rc.4`
## trigger.dev@4.5.0-rc.4
### Patch Changes
- Bump `@s2-dev/streamstore` to `0.22.10` to fix a
`TASK_RUN_UNCAUGHT_EXCEPTION` ("Invalid state: Unable to enqueue") when
a `chat.agent` turn is aborted mid-stream.
([#3792](https://github.com/triggerdotdev/trigger.dev/pull/3792))
- Updated dependencies:
- `@trigger.dev/core@4.5.0-rc.4`
- `@trigger.dev/build@4.5.0-rc.4`
- `@trigger.dev/schema-to-json@4.5.0-rc.4`
## @trigger.dev/core@4.5.0-rc.4
### Patch Changes
- Coerce numeric `concurrencyKey` values to string at the API boundary
across `tasks.trigger`, `tasks.batchTrigger`, and the Phase-2 streaming
batch endpoint.
([#3789](https://github.com/triggerdotdev/trigger.dev/pull/3789))
- Bump `@s2-dev/streamstore` to `0.22.10` to fix a
`TASK_RUN_UNCAUGHT_EXCEPTION` ("Invalid state: Unable to enqueue") when
a `chat.agent` turn is aborted mid-stream.
([#3792](https://github.com/triggerdotdev/trigger.dev/pull/3792))
## @trigger.dev/plugins@4.5.0-rc.4
### Patch Changes
- Updated dependencies:
- `@trigger.dev/core@4.5.0-rc.4`
## @trigger.dev/python@4.5.0-rc.4
### Patch Changes
- Updated dependencies:
- `@trigger.dev/sdk@4.5.0-rc.4`
- `@trigger.dev/core@4.5.0-rc.4`
- `@trigger.dev/build@4.5.0-rc.4`
## @trigger.dev/react-hooks@4.5.0-rc.4
### Patch Changes
- Updated dependencies:
- `@trigger.dev/core@4.5.0-rc.4`
## @trigger.dev/rsc@4.5.0-rc.4
### Patch Changes
- Updated dependencies:
- `@trigger.dev/core@4.5.0-rc.4`
## @trigger.dev/schema-to-json@4.5.0-rc.4
### Patch Changes
- Updated dependencies:
- `@trigger.dev/core@4.5.0-rc.4`
## @trigger.dev/sdk@4.5.0-rc.4
### Patch Changes
- Add a `tools` option to `chat.agent`. Declaring your tools here
threads them into the SDK's internal `convertToModelMessages`, so each
tool's `toModelOutput` is re-applied when prior-turn history is
re-converted.
([#3790](https://github.com/triggerdotdev/trigger.dev/pull/3790))
```ts
chat.agent({
tools: { readFile, search },
run: async ({ messages, tools, signal }) =>
streamText({ model, messages, tools, abortSignal: signal }),
});
```
Also exports `InferChatUIMessageFromTools<typeof tools>` to derive the
chat `UIMessage` type (typed tool parts) directly from a tool set.
- Updated dependencies:
- `@trigger.dev/core@4.5.0-rc.4`
</details>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Follow-up to #3796, which bumped the slack-client axios paths but left
posthog-node's transitive `axios@1.15.1` in place.
`posthog-node` 4.17.1 → 5.35.6. v5 drops the axios dependency entirely
(it's now fetch-based via `@posthog/core`), so posthog's old axios path
disappears. With #3796 already on main (webapp + d3 references on
`@slack/web-api@7.16.0`), nothing else pins the old line, so the
now-dead `axios@>=1.0.0 <1.15.0` override is removed and axios resolves
to a single patched `1.16.1` repo-wide. This closes the remaining axios
advisories.
Compat: the webapp's usage in `telemetry.server.ts` (`new PostHog(key, {
host })`, `.identify`, `.groupIdentify`, `.capture`) is all object-form
API that v5 preserves; `pnpm run typecheck --filter webapp` passes.
Node: posthog-node v5 requires Node `^20.20.0 || >=22.22.0`. We run
20.20.0 in dev (`.nvmrc`), CI, and the published Docker image
(`node:20.20-bullseye-slim`), so we're compliant.
Bumps `@slack/web-api` 7.9.1 → 7.16.0 in the webapp and the two
`references` examples (d3-chat, d3-openai-agents). 7.16.0 depends on
`axios@^1.16.0`, so every slack-client axios path resolves to 1.16.1
instead of 1.15.1.
This clears the slack and references axios paths. `posthog-node`'s
transitive axios still resolves the older line - that's handled in a
follow-up that upgrades posthog-node to v5 (which drops the axios
dependency entirely and lets us retire the now-stale axios override).
The dependabot axios advisories fully close once both land.
## Summary
A `chat.agent` turn that gets aborted mid-stream (stop generation, idle
suspend, cancellation) could surface a `TASK_RUN_UNCAUGHT_EXCEPTION` —
`TypeError: Invalid state: Unable to enqueue` — in the dashboard. The
run kept working, but the exceptions were loud and confusing.
## Root cause
The realtime stream writer batches chunks through
`@s2-dev/streamstore`'s `BatchTransform`, which holds them for a short
linger window before flushing. When the turn's abort signal fires while
a record is still buffered, the stream's writable is aborted and the
transform's readable controller is closed — but the pending linger
`setTimeout` still fires and calls `controller.enqueue()` on the dead
controller, throwing from a timer callback where nothing can catch it.
Fixed upstream in `@s2-dev/streamstore@0.22.10`, which wraps the linger
flush in a try/catch that discards the closed-controller error. This
bumps the dependency across core, the CLI, and the webapp, and adds a
regression test against the real `BatchTransform`.
Verified end-to-end: a mid-stream stop that previously failed the run
with `TASK_RUN_UNCAUGHT_EXCEPTION` now leaves the run healthy.
## Summary
`chat.agent` now takes a `tools` option. Until now tools only went to
`streamText` inside `run()`, so the SDK had no tools when it
re-converted the persisted `UIMessage` history at the start of each
turn. Any tool with a `toModelOutput` (raw image bytes into an image
content part, or a sub-agent transcript compressed to a summary) had its
transform applied on turn 1 and skipped from turn 2 onward, so the raw
output got JSON-stringified back into the prompt and the model lost the
transformed view.
Declaring `tools` on the config threads them into that conversion, so
`toModelOutput` runs on every turn. The resolved set is handed back,
typed, on the `run()` payload as `tools`:
```ts
const tools = { searchDocs, renderChart };
export const myChat = chat.agent({
tools,
run: async ({ messages, tools, signal }) =>
streamText({ ...chat.toStreamTextOptions({ tools }), messages, abortSignal: signal }),
});
```
`tools` also accepts a per-turn function for tools that depend on the
user or a feature flag. Only `inputSchema` and `toModelOutput` are read
during conversion, never `execute`. Also exports
`InferChatUIMessageFromTools<typeof tools>` to derive the chat
`UIMessage` type from a tool set. No behavior change for agents that
don't declare `tools`.
## Summary
Dashboard surfaces handle buffered runs by falling back to the mollifier
snapshot:
- Run detail, span detail, streams view (`_app.../runs.\$runParam`,
`resources.../spans.\$spanParam`, `resources.../streams.\$streamKey`).
- Redirect routes (`@.runs.\$runParam`, `runs.\$runParam`,
`projects.v3.\$projectRef.runs.\$runParam`).
- Action routes — cancel / replay / idempotency-reset / debug — under
`resources.taskruns/...` and `resources.../idempotencyKey.reset`.
- Logs download.
- Realtime subscription route + per-run resource
(`realtime.v1.runs.\$runId`, `resources.../realtime.v1.*`).
- `CancelRunDialog` gains an `onCancelSubmitted` callback so submit
isn't raced by the Radix `DialogClose` wrapper.
Stacked on the mutations PR.
## Test plan
- [x] \`pnpm run typecheck --filter webapp\` passes
- [x] \`pnpm run test --filter webapp
test/mollifierRealtimeRunResource.test.ts\` passes
- [x] \`pnpm run test --filter webapp
test/mollifierRealtimeRunResourceBuffer.test.ts\` passes
- [x] \`pnpm run test --filter webapp
test/mollifierRealtimeSubscription.test.ts\` passes
- [x] Manual smoke: trigger a buffered run, open it in the dashboard,
replay/cancel from the UI
---
## Ship-gate follow-up fixes
- **Auto-redirect to root span on direct nav** — loader sets `?span=`
from root span (PG) or buffered snapshot spanId before 302'ing, so
bookmark/share-link/direct-nav doesn't leave the panel collapsed.
- **RunPresenter switches from `findFirstOrThrow` to `findFirst` + typed
`RunNotInPgError`** — kills the per-poll `PrismaClient error` log spam
for buffered runs without changing the route-loader's fallback flow.
- **Span detail panel renders for buffered runs** — `SpanPresenter.call`
now falls back to `findRunByIdWithMollifierFallback` +
`buildSyntheticSpanRun` instead of returning undefined and triggering
the "Event not found" toast loop.
- **Logs download for buffered runs returns a gzipped placeholder line**
— replaces the 404 with a content-encoded line explaining the run is
queued. Same org-membership gate as the PG path.
- **Admin Debug-Run button hidden for buffered runs + SpanRun circular
type alias broken** (squashed) — buttons gate on a new `isBuffered` flag
on the synthetic SpanRun. Required grounding SpanRun in
`SpanPresenter.getRun` to break a circular type alias TS no longer
tolerates once `isBuffered` is a literal field on the shape.
- **Replay action requires user auth + org-membership** (🚩 Devin
finding) — `action` was unauthenticated and the PG `findFirst` had no
org filter, so any caller with a valid `runParam` could replay any run.
Buffered fallback inherited the same gap. Fixed to mirror the cancel
route.
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
## Summary
Cancel, replay, reschedule, metadata, tags, and idempotency-key-reset
now succeed against a run that's still in the mollifier buffer.
Mutations are applied to the buffered snapshot via Lua CAS; the drainer
carries the mutation forward when it replays.
Primitives added:
- `mutateWithFallback` — PG-first / buffer-fallback resolver with
bounded-wait safety net for entries that transition mid-mutation.
- `applyMetadataMutation` — buffered metadata PUT mirroring the PG-side
retry loop with CAS atomicity.
- `resolveRunForMutation` — discriminated-union resolver used by route
`findResource` so the route builder's pre-action 404 check sees buffered
runs.
Routes wired (whole files, no GET/POST splits):
- `api.v2.runs.\$runParam.cancel.ts`
- `api.v1.runs.\$runParam.replay.ts`
- `api.v1.runs.\$runParam.reschedule.ts`
- `api.v1.runs.\$runId.metadata.ts`
- `api.v1.runs.\$runId.tags.ts`
- `resetIdempotencyKey.server.ts`
Stacked on the reads PR.
## Test plan
- [x] \`pnpm run typecheck --filter webapp\` passes
- [x] \`pnpm run test --filter webapp
test/mollifierMutateWithFallback.test.ts\` passes
- [x] \`pnpm run test --filter webapp
test/mollifierApplyMetadataMutation.test.ts\` passes
- [x] \`pnpm run test --filter webapp
test/mollifierResolveRunForMutation.test.ts\` passes
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
## Summary
Synthesise QUEUED/FAILED responses from the mollifier buffer when a
TaskRun row hasn't landed in Postgres yet. Wires the synthesis into:
- `ApiRetrieveRunPresenter`
- v1 trace GET route
- v1 spans GET route
- attempts route gains a GET loader (fixes pre-existing Remix "no
loader" 400)
The `readFallback` infra itself lives on the trigger PR (consumed by
`IdempotencyKeyConcern`); this PR adds the route-level
synthetic-rendering primitives.
Stacked on the replay PR.
## Test plan
- [x] \`pnpm run typecheck --filter webapp\` passes
- [x] \`pnpm run test --filter webapp
test/mollifierSyntheticRedirectInfo.test.ts\` passes
- [x] \`pnpm run test --filter webapp
test/mollifierSyntheticSpanRun.test.ts\` passes
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
## Summary
The replay side of the mollifier:
- `DrainerHandler`: reads buffered snapshots and replays them through
`engine.trigger` to materialise PG rows.
- `RunEngine.createCancelledRun`: new public method the handler uses to
write CANCELED rows directly from snapshots (bypass queue + waitpoint,
emit `runCancelled`). Tolerates the cjson empty-table tags edge case
found during validation.
- Drainer fairness: org → env rotation so a heavy env doesn't starve
light ones in the same org.
- Stale-entry sweep + telemetry + alertable gauge so a stuck/offline
drainer surfaces in alerts.
Both the drainer and sweep default-off; nothing fires unless flagged on
(`TRIGGER_MOLLIFIER_DRAINER_ENABLED`,
`TRIGGER_MOLLIFIER_STALE_SWEEP_ENABLED`).
Stacked on the trigger-time decisions PR.
## Test plan
- [x] \`pnpm run typecheck --filter webapp\` passes
- [x] \`pnpm run test --filter webapp
test/mollifierDrainerHandler.test.ts\` passes
- [x] \`pnpm run test --filter webapp test/mollifierStaleSweep.test.ts\`
passes
- [x] \`pnpm run test --filter @internal/run-engine
src/engine/tests/createCancelledRun.test.ts\` passes
- [x] \`pnpm run test --filter @trigger.dev/redis-worker
packages/redis-worker/src/mollifier/drainer.test.ts\` passes
---
## Ship-gate follow-up fix
**Drainer writes SYSTEM_FAILURE on max-attempts exhaustion.** Adds an
`onTerminalFailure` callback on `MollifierDrainerOptions` so the
customer's run lands a SYSTEM_FAILURE PG row even when the drainer
exhausts `MAX_ATTEMPTS` on a retryable PG error (previously
`buffer.fail()` was called with no row written → silent data loss). The
callback runs before `buffer.fail()` on every terminal path
(non-retryable AND max-attempts-exhausted), and re-throwing a retryable
error from the callback causes the drainer to requeue rather than fail.
Bumps `@trigger.dev/redis-worker` to a **minor** changeset (additive
option + new exported types). Includes 5 unit tests covering both
terminal causes plus the requeue-on-retryable-callback-failure path and
no-callback back-compat.
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
## Summary
`concurrencyKey` validation accepted only `z.string().optional()` on the
single-trigger and V2/V3 batch endpoints, and the Phase-2 streaming
NDJSON endpoint accepted `z.record(z.unknown()).optional()` for the
entire `options` field. Callers passing `concurrencyKey: someNumericId`
(e.g. `payload.userId`) either failed schema validation on the first two
paths or sailed through on Phase-2 and then failed downstream at
`prisma.taskRun.create` with `Argument concurrencyKey: Expected String
or Null, provided Int`.
The schema now accepts `string | number` for `concurrencyKey` and
stringifies on the way in, across all three paths. The Phase-2 NDJSON
`options` is tightened to reuse the strict
`BatchTriggerTaskItem.options` shape so it validates identically to the
V2/V3 batch endpoints.
A defensive `typeof === "number"` coercion at the `engine.trigger` call
site in `RunEngineTriggerTaskService` covers in-flight Redis-stored
batch items enqueued before the schema fix — those items are rebuilt
from a `Record<string, unknown>` shape that bypasses the new schema and
would otherwise continue failing for up to their TTL.
## Test plan
- [x] `packages/core/src/v3/schemas/batchItemNDJSON.test.ts` — unit
tests covering numeric→string coercion, string passthrough, no-options,
and rejection of non-string/non-number shapes across
`TriggerTaskRequestBody`, `BatchTriggerTaskItem`, and `BatchItemNDJSON`.
- [x] `apps/webapp/test/engine/triggerTask.test.ts` — `containerTest`
simulating the in-flight Redis batch-item shape (numeric
`concurrencyKey` via `Record<string, unknown>`), verifies the run is
created with `concurrencyKey: "51262"`. Without the worker coercion, the
test reproduces the production stack at `prisma.taskRun.create`.
- [x] `pnpm run typecheck --filter webapp` clean.
- [x] `pnpm run build --filter @trigger.dev/core --filter
@trigger.dev/sdk --filter trigger.dev` clean.
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
## Summary
The trigger hot path's mollifier integration:
- `mollifyTrigger`: when the gate trips, write the engine.trigger
snapshot to the buffer and return a synthesised QUEUED response.
Postgres write is deferred to drainer-replay (next PR in the stack).
- Pre-gate idempotency-key claim: same-key triggers serialise through
Redis so a burst lands in PG / buffer exactly once.
- Read-fallback extensions: `findRunByIdWithMollifierFallback` for the
trigger-time idempotency lookup that must see buffered runs.
- Gate bypasses: `debounce`, `oneTimeUseToken`,
`parentTaskRunId`/`triggerAndWait` skip the mollify path entirely.
- `triggerTask` + `IdempotencyKeyConcern` wired to the above.
All behaviour gated by the master `TRIGGER_MOLLIFIER_ENABLED` switch;
off-state hot path is unchanged (the gate is not even consulted).
Stacked on the buffer extensions PR.
## Test plan
- [x] \`pnpm run typecheck --filter webapp\` passes
- [x] \`pnpm run test --filter webapp test/mollifierMollify.test.ts\`
passes
- [x] \`pnpm run test --filter webapp
test/mollifierIdempotencyClaim.test.ts\` passes
- [x] \`pnpm run test --filter webapp
test/mollifierReadFallback.test.ts\` passes
- [x] \`pnpm run test --filter webapp test/mollifierGate.test.ts\`
passes
- [x] \`pnpm run test --filter webapp test/engine/triggerTask.test.ts\`
passes
---
## Ship-gate follow-up fixes
- **Batch items bypass the mollifier gate** — fixes
`BatchTaskRunItem_taskRunId_fkey` FK violation on batch triggers when
the gate trips. End-state is a drainer-side `BatchTaskRunItem`
create-on-materialise; batch traffic passes through the gate until that
lands.
- **IdempotencyKeyConcern honours buffered-run TTL on expiry** —
buffered path now clears expired idempotency claims (read-side) and
resets the buffer's `mollifier:idempotency:*` SETNX binding (write-side)
so a re-trigger past the customer's TTL lands as a fresh run instead of
echoing the stale buffered runId.
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
## Summary
Buffer-side data layer used by the rest of the mollifier phase-3 stack.
- `buffer.ts` gains entry inspection (`getEntry`), idempotency lookup
(`lookupIdempotency`), in-place snapshot mutation (`mutateSnapshot`),
and dwell tracking. All atomic via Lua.
- `mollifierSnapshot.server.ts`: shared `MollifierSnapshot` type plus
(de)serialise helpers.
- Drops the entry-TTL config and its env var. The drainer is the
recovery mechanism; an entry that survives the drainer should surface as
a stale-sweep alert, not silently TTL away.
Adds methods to the buffer interface; nothing consumes them yet.
Subsequent PRs in the stack wire trigger-time mollify, read-fallback,
and mutation paths against this surface.
## Test plan
- [x] \`pnpm run typecheck --filter webapp\` passes
- [x] \`pnpm run test --filter @trigger.dev/redis-worker
packages/redis-worker/src/mollifier/buffer.test.ts\` passes
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
## Summary
1 improvement, 2 bug fixes.
## Improvements
- Retry `TASK_MIDDLEWARE_ERROR` under the task's retry policy instead of
failing the run on the first attempt. The error was already classified
as retryable by `shouldRetryError`, but `shouldLookupRetrySettings` did
not include it, so the retry flow fell through to `fail_run`. Fixes
#3231. ([#3676](https://github.com/triggerdotdev/trigger.dev/pull/3676))
## Bug fixes
- Fix `TypeError` in `unflattenAttributes` when the input attribute map
contains conflicting dotted key paths (e.g. both `a.b` set to a scalar
and `a.b.c` set to a value). The path-walk loop now applies
last-write-wins when a prior key wrote a primitive, null, or array at an
intermediate slot, matching the existing precedent in
`AttributeFlattener.addAttribute`. Callers no longer crash when handed
malformed external attribute inputs.
([#3762](https://github.com/triggerdotdev/trigger.dev/pull/3762))
- Fix external trace context leaking across runs on warm-started workers
with `processKeepAlive` enabled. Every subsequent run's attempt span was
being exported with the first run's `traceId` and `parentSpanId`,
breaking causal-chain navigation in external APM tools. Runs without an
external trace context are unaffected.
([#3768](https://github.com/triggerdotdev/trigger.dev/pull/3768))
<details>
<summary>Raw changeset output</summary>
⚠️⚠️⚠️⚠️⚠️⚠️
`main` is currently in **pre mode** so this branch has prereleases
rather than normal releases. If you want to exit prereleases, run
`changeset pre exit` on `main`.
⚠️⚠️⚠️⚠️⚠️⚠️
# Releases
## @trigger.dev/build@4.5.0-rc.3
### Patch Changes
- Updated dependencies:
- `@trigger.dev/core@4.5.0-rc.3`
## trigger.dev@4.5.0-rc.3
### Patch Changes
- Updated dependencies:
- `@trigger.dev/core@4.5.0-rc.3`
- `@trigger.dev/build@4.5.0-rc.3`
- `@trigger.dev/schema-to-json@4.5.0-rc.3`
## @trigger.dev/core@4.5.0-rc.3
### Patch Changes
- Retry `TASK_MIDDLEWARE_ERROR` under the task's retry policy instead of
failing the run on the first attempt. The error was already classified
as retryable by `shouldRetryError`, but `shouldLookupRetrySettings` did
not include it, so the retry flow fell through to `fail_run`. Fixes
#3231. ([#3676](https://github.com/triggerdotdev/trigger.dev/pull/3676))
- Fix `TypeError` in `unflattenAttributes` when the input attribute map
contains conflicting dotted key paths (e.g. both `a.b` set to a scalar
and `a.b.c` set to a value). The path-walk loop now applies
last-write-wins when a prior key wrote a primitive, null, or array at an
intermediate slot, matching the existing precedent in
`AttributeFlattener.addAttribute`. Callers no longer crash when handed
malformed external attribute inputs.
([#3762](https://github.com/triggerdotdev/trigger.dev/pull/3762))
- Fix external trace context leaking across runs on warm-started workers
with `processKeepAlive` enabled. Every subsequent run's attempt span was
being exported with the first run's `traceId` and `parentSpanId`,
breaking causal-chain navigation in external APM tools. Runs without an
external trace context are unaffected.
([#3768](https://github.com/triggerdotdev/trigger.dev/pull/3768))
## @trigger.dev/plugins@4.5.0-rc.3
### Patch Changes
- Updated dependencies:
- `@trigger.dev/core@4.5.0-rc.3`
## @trigger.dev/python@4.5.0-rc.3
### Patch Changes
- Updated dependencies:
- `@trigger.dev/core@4.5.0-rc.3`
- `@trigger.dev/build@4.5.0-rc.3`
- `@trigger.dev/sdk@4.5.0-rc.3`
## @trigger.dev/react-hooks@4.5.0-rc.3
### Patch Changes
- Updated dependencies:
- `@trigger.dev/core@4.5.0-rc.3`
## @trigger.dev/redis-worker@4.5.0-rc.3
### Patch Changes
- Updated dependencies:
- `@trigger.dev/core@4.5.0-rc.3`
## @trigger.dev/rsc@4.5.0-rc.3
### Patch Changes
- Updated dependencies:
- `@trigger.dev/core@4.5.0-rc.3`
## @trigger.dev/schema-to-json@4.5.0-rc.3
### Patch Changes
- Updated dependencies:
- `@trigger.dev/core@4.5.0-rc.3`
## @trigger.dev/sdk@4.5.0-rc.3
### Patch Changes
- Updated dependencies:
- `@trigger.dev/core@4.5.0-rc.3`
</details>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
## Summary
Drops the `TaskRun_status_runtimeEnvironmentId_createdAt_id_idx` index
from the `TaskRun` table. After #3742 gated the legacy
`WAITING_FOR_DEPLOY` drain to V1-engine workers only, this index sees
zero scans on both writer and reader replicas. Removing it cuts index
maintenance on every `TaskRun` INSERT/UPDATE.
## Why
The index existed to support `WHERE status = X AND runtimeEnvironmentId
= Y` queries from `ExecuteTasksWaitingForDeployService`, which is
V1-only and no longer triggered on V2 deployments. A code grep across
`apps/webapp` and `internal-packages/run-engine` confirmed no V2
production query uses this access pattern — every other `status:` filter
on `TaskRun` is paired with `id`/`friendlyId`/`parentSpanId` and uses a
different index.
Dropping it also unlocks HOT updates on the dequeue path. The dequeue
`UPDATE` modifies `status` (`QUEUED` -> `DEQUEUED`), and `status` is the
leading column of this index — its presence blocked HOT eligibility for
every `TaskRun` UPDATE. With the index gone, dequeue UPDATEs can become
HOT, reducing WAL bytes and removing the B-tree page contention on this
index's right-edge leaves.
Uses `DROP INDEX CONCURRENTLY` to avoid blocking writes during the drop.
## Sequencing
Should only ship once #3742 has soaked long enough to confirm the index
is genuinely cold (24h+ of zero scans on `pg_stat_user_indexes`).
Make `POST /api/v1/deployments/:deploymentId/background-workers`
idempotent so client-side retries no longer collide on the
`BackgroundWorker` `(project, env, version)` unique index. Helps make
deployments more resilient against the class of indexing failures that
surfaces in the dashboard as "Indexing timed out", e.g. during transient
database issues.
On warm-started workers with `processKeepAlive` enabled, every run's
attempt span was exported with the first run's `traceId` and
`parentSpanId`. `ExternalSpanExporterWrapper` and
`ExternalLogRecordExporterWrapper` captured `externalTraceContext` at
`TracingSDK` construction, and the SDK is memoized for the worker's
lifetime - so the per-run reset of
`StandardTraceContextManager.traceContext` never reached the wrappers.
Reported by a customer running v4.4.x: 33 distinct runs on the same
host/pid showed up in their APM as siblings of one parent span.
Fix: the wrappers now read external context live from the trace context
manager per export. Runs without an external trace context fall through
to the unchanged `externalTraceId` fallback - no behaviour change for
them.
Regression test in
`packages/core/test/externalSpanExporterWrapper.test.ts` asserts that
reassigning the manager between exports produces correctly-parented
spans.
Retries `TASK_MIDDLEWARE_ERROR` under the task's retry policy.
`shouldRetryError` already classed it as retryable, but
`shouldLookupRetrySettings` did not, so the run fell through to
`fail_run` on attempt 1 instead of using the task's `retry` config.
Fixes#3231.
## Summary
Second class of poisoned-row failure in the runs replication path. PR
#3708 plugged lone UTF-16 surrogates; this one handles bare JSON integer
literals outside ClickHouse's `Int64`..`UInt64` range. Recovery stays
purely reactive — the existing `sanitizeRows` walker just gains an extra
branch, so the hot replication path pays nothing on healthy rows.
Fixes the still-firing customer-facing symptom from
[TRI-9755](https://linear.app/triggerdotdev/issue/TRI-9755):
`scan-social-profiles` runs continued to be stranded in `EXECUTING` on
the Tasks page after #3708 deployed. CloudWatch showed `Dropped batch —
ClickHouse JSON parse error but sanitizer found nothing to fix` firing
**8/8 times** since the previous deploy (zero successful sanitizations).
Root cause: upstream JS Number precision loss on a 21-digit Google Plus
ID (`117039831458782873093` → `117039831458782870000`) — the
precision-lossy value still serialises as a bare integer that exceeds
`UInt64.MAX`, which ClickHouse rejects with `INCORRECT_DATA`.
## How the bug ships
The customer task emits an output containing a Poshmark profile's
`spec_format`:
```json
{"key":"gp_id","proper_key":"Gp Id","value":117039831458782870000,"type":"int"}
```
That value is `1.17e20` — comfortably above `UInt64.MAX` (`1.84e19`) but
comfortably below `1e21`. `Number.prototype.toString` only switches to
exponential form at `|value| >= 1e21`, so `JSON.stringify` emits the
bare token `117039831458782870000` and the ClickHouse
`JSON(max_dynamic_paths)` column fails with:
```
Code: 117. DB::Exception: Cannot parse JSON object here: {…}: (while reading the value of key output): (at row 1)
: While executing ParallelParsingBlockInputFormat. (INCORRECT_DATA) (version 25.12.x)
```
Same error verbatim as prod. The same number quoted
(`"117039831458782870000"`) inserts fine — ClickHouse's dynamic JSON
column accepts a `String` subtype on the same path.
## What changed
`apps/webapp/app/v3/eventRepository/sanitizeRowsOnParseError.server.ts`:
- New private `isUnsafeJsonInteger(value)` helper — true iff `value` is
a finite integer-valued JS Number where `|value| < 1e21` (so
`JSON.stringify` emits integer form, not exponent) **and** `value` falls
outside `[Int64.MIN, UInt64.MAX]`.
- `sanitizeUnknownInPlace` gains a number-branch: when the predicate
holds, replace the Number with `String(value)`. The downstream JSON
column dynamic-types the path as String for that row — fine, since the
value was already precision-lossy upstream (no JS Number above 2^53 is
numerically meaningful anyway).
- Float-valued numbers, large floats (>= 1e21), NaN and Infinity are
left alone — `JSON.stringify` emits them with exponents or as `null`,
both of which ClickHouse accepts.
`apps/webapp/test/sanitizeRowsOnParseError.test.ts`: four new unit tests
+ an extension to `sanitizeRows` covering surrogate + integer fixes
counted together across rows. The unit suite now covers:
- Positive value above `UInt64.MAX` (`117039831458782870000` — the
actual prod value)
- Negative value below `Int64.MIN`
- Boundary values pass through (`42`, `Number.MAX_SAFE_INTEGER`, `2^63`)
- Non-integer numbers untouched (floats, `1e25`, NaN, Infinity)
- The actual `scan-social-profiles` nested shape — finds the offending
`gp_id` deep inside
`output.data.profiles[].spec_format[].platform_variables[].value`
`.server-changes/runs-replication-bigint-recovery.md` — release notes
entry.
## Why reactive, not pre-flight
`#prepareJson` runs millions of times per day on the replication hot
path. Walking every JSON tree to look for oversized integers would add
bounded-but-real CPU on every healthy row. `sanitizeRows` only fires
after a ClickHouse parse-error rejection, which is a few times a day
platform-wide. Extending it costs effectively zero on healthy traffic
and gains us recovery on the rare poisoned row.
## Verification
- Reproduced 1:1 in a throwaway Docker
`clickhouse/clickhouse-server:25.12.11.4` (closest available to the prod
`25.12.1.1579` build). Pre-sanitize JSON fails with the exact prod
error; post-sanitize JSON inserts cleanly and the row is readable with
`gp_id` stored as a String subtype.
- `pnpm --filter webapp exec vitest run
test/sanitizeRowsOnParseError.test.ts` — 22/22 passing (18 existing + 4
new).
- `pnpm run typecheck --filter webapp` — clean.
## Test plan
- [x] `pnpm run typecheck --filter webapp`
- [x] Unit tests pass against new + existing cases
- [x] End-to-end Docker ClickHouse repro confirms recovery
- [ ] Post-deploy: confirm `Sanitizing batch after ClickHouse JSON parse
error` warns fire instead of `Dropped batch …` errors when
`scan-social-profiles` outputs trip CH again
- [ ] Post-deploy: confirm `permanentlyDroppedBatches` counter stops
climbing in
`/stp/trigger-app-prod/ecs/replication/service-container/process-logs`
## What this does NOT do
- Doesn't backfill the ~120k+ existing stranded `EXECUTING` rows in
production. Same as #3708 — that needs a reconciliation/backfill sweep
(separate ticket — TRI-9755 fix#3).
- Doesn't address the upstream root cause (the customer task emitting a
JS-Number-precision-lossy big int). That's a customer-task concern; our
replication path needs to be robust to whatever shape arrives.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
## Summary
Stops the legacy V1 `WAITING_FOR_DEPLOY` drain from running on every V2
deployment promotion. The drain queries `TaskRun` by
`status='WAITING_FOR_DEPLOY'`, which only V1-engine runs ever have — V2
runs use `PENDING_VERSION` and are handled out of band. Calling the
drain on V2 promotes produced empty queries against the status index and
unnecessary reader-DB load.
## Fix
Two layers:
1. Gate the enqueue at the call site in `ChangeCurrentDeploymentService`
so it only fires when the deployment's worker is on engine V1.
2. Add a `LEGACY_RUN_ENGINE_WAITING_FOR_DEPLOY_DISABLED` env var
(default `0`). When set to `1`, the service returns immediately from
`call()` — neuters any jobs already sitting in the worker queue from
before the deploy lands.
V1 customers see no change; V2 promotes no longer trigger the drain.
Linear: [TRI-9864](https://linear.app/triggerdotdev/issue/TRI-9864)
(Urgent)
Production incident:
[TRI-9863](https://linear.app/triggerdotdev/issue/TRI-9863) (mitigated
by image revert in cloud#910)
## Bug
`apps/webapp/package.json` declares `"sideEffects": false`. PR #3333
(`71d98b4e`) replaced the previous real method-call retention idiom at
the two `sessionsReplicationInstance` import sites with:
```ts
import { sessionsReplicationInstance } from "...";
void sessionsReplicationInstance;
```
esbuild treats `void <identifier>;` as a pure expression statement under
`sideEffects: false` and **tree-shakes the entire import** — including
the `singleton(...)` call inside `sessionsReplicationInstance.server.ts`
which is the only thing that fires
`initializeSessionsReplicationInstance()`. The sessions→ClickHouse
logical replication worker never starts, the slot is unconsumed, lag
grows.
### How it manifested in production
cloud#907's image bump rolled the `SessionReplicationService` ECS task
on prod at 14:32 UTC. The new container's startup log emitted `🗃️ Runs
replication service enabled` but **not** `🗃️ Sessions replication
service enabled` or `🗃️ Sessions replication service started`.
CloudWatch `OldestReplicationSlotLag` grew at ~220 MB/min and the `High
replication lag` alarm fired at 14:37 UTC. Prod was reverted to the
previous image (cloud#910) to stop the bleed.
### Verification
`grep` of the built bundle `apps/webapp/build/index.js` (built from
`c0365d36`):
- **3** occurrences of `Runs replication` / `runsReplicationInstance`
strings ✅
- **0** occurrences of `Sessions replication` /
`sessionsReplicationInstance` / `SessionsReplicationService` ❌
The runs path survives tree-shaking because `adminWorker.server.ts` and
`admin.api.v1.runs-replication.*` routes have real method calls
(`.start()`, `.teardown()`, `.backfill()`) — observable uses the
tree-shaker must preserve. The sessions singleton has no real callers,
only the `void` no-ops, hence its complete elimination from the bundle.
## Fix
Replace `void sessionsReplicationInstance;` with an assignment to
`globalThis`, an unambiguous observable side effect the bundler cannot
eliminate:
```ts
(globalThis as Record<string, unknown>).__sessionsReplicationInstance =
sessionsReplicationInstance;
```
Applied at both call sites: `apps/webapp/app/entry.server.tsx` and
`apps/webapp/app/v3/services/adminWorker.server.ts`.
Surrounding comments updated to document the bundler interaction so the
next maintainer doesn't reintroduce `void`.
## Out of scope (follow-ups)
- **Robustness improvement**: change `apps/webapp/package.json` from
`"sideEffects": false` to an allowlist that includes
`*Instance.server.ts` files. Prevents the same regression shape via any
future `*Instance` singleton.
- **Build-time check**: add a `grep` post-build step in `publish.yml`
requiring `"Sessions replication"` to appear in
`apps/webapp/build/index.js`. Catches this exact regression at CI time.
## Test plan
- [x] `pnpm run typecheck --filter webapp` clean
- [ ] After merge + publish: confirm new image's
`SessionReplicationService` container logs `🗃️ Sessions replication
service enabled` and `🗃️ Sessions replication service started` at
startup
- [ ] After re-deploying to prod: confirm `OldestReplicationSlotLag`
stops growing and drains
## Summary
On a ClickHouse `Cannot parse JSON object` rejection,
`RunsReplicationService` now sanitizes lone UTF-16 surrogates across the
failing batch via the existing `sanitizeRows` helper and retries once.
If the sanitizer found nothing or the retry also fails, the batch is
dropped loudly with a counter increment, so the surrounding
`#insertWithRetry` layer doesn't spin three more times on a
deterministic failure. Non-parse errors propagate unchanged.
Mirrors the pattern from #3659 (for `ClickhouseEventRepository`) — same
root cause (lone UTF-16 surrogates in user-provided JSON), same recovery
shape, **reusing the same shared helpers** (`sanitizeRows`,
`isClickHouseJsonParseError`, `parseRowNumberFromError`).
Fixes the customer-facing symptom from
[TRI-9755](https://linear.app/triggerdotdev/issue/TRI-9755): a single
row's poisoned `output` JSON used to take down the
`COMPLETED_SUCCESSFULLY` UPDATE events for its 50+ batch-mates,
stranding them in `EXECUTING` in ClickHouse forever and inflating
"Running" counts on the Tasks page. Confirmed in production this is
ongoing — ~120k stale rows accumulated in a single 5-hour burst on
2026-05-18; smaller continuous leak before and after.
## What changed
`apps/webapp/app/services/runsReplicationService.server.ts`:
- Imports the three helpers from
`~/v3/eventRepository/sanitizeRowsOnParseError.server` (no duplication;
no move).
- New private `#insertWithJsonParseRecovery<T>(rows, doInsert,
contextLabel, attempt)` — generic over `TaskRunInsertArray[]` and
`PayloadInsertArray[]`, structurally identical to
`ClickhouseEventRepository.#insertWithJsonParseRecovery`. Try → on parse
error sanitize the whole batch (the `at row N` hint is logged but not
used to slice — semantics under `input_format_parallel_parsing` aren't
stable) → retry once → drop with loud log if sanitizer found nothing OR
retry still fails.
- `#insertTaskRunInserts` and `#insertPayloadInserts` extract a
`doInsert` closure and hand it to the wrapper. Existing error logging,
span recording, and `recordSpanError` are preserved inside the closure.
- New `private _permanentlyDroppedBatches = 0` counter with a public
getter, for ops dashboards and tests (matches the events-repo
convention). One shared counter for both insert sites — granularity
comes from the `contextLabel` (`task_runs_v2` /
`raw_task_runs_payload_v1`) on every log line.
`.server-changes/runs-replication-utf16-recovery.md` — release notes
entry.
## Why no new tests
The shared helpers already have full unit + real-ClickHouse contract
coverage from #3659
(`apps/webapp/test/sanitizeRowsOnParseError.test.ts`,
`apps/webapp/test/otlpUtf16Sanitization.integration.test.ts`). The new
wrapper is a line-for-line structural port. Adding a parallel
integration test would require synthesizing bad data that *escapes* the
preemptive `detectBadJsonStrings` check in `#prepareJson` but still
trips ClickHouse — non-trivial without hand-crafted fixtures and
wouldn't cover any new logic.
## What this does NOT do
- Doesn't touch the ~120k existing stale `EXECUTING` rows in production.
That needs a reconciliation/backfill sweep (separate ticket — TRI-9755
fix#3).
- Doesn't sanitize the `error` column path
(`runsReplicationService.server.ts:932 const errorData = { data:
run.error };`). Reactive recovery will catch it if it ever poisons a
batch, but feeding it through `#prepareJson` like `output` is a cheap
follow-up.
## Test plan
- [x] `pnpm run typecheck --filter webapp` — clean
- [ ] Post-deploy: confirm `permanentlyDroppedBatches` counter stays at
zero (or near-zero) in
`/stp/trigger-app-prod/ecs/replication/service-container/process-logs`,
and watch for `Sanitizing batch after ClickHouse JSON parse error` warns
to confirm recovery is firing on real traffic
- [ ] Post-deploy: confirm the rate of new
"EXECUTING-but-actually-COMPLETED" zombies in ClickHouse flattens
(current rate ≈ tens-to-hundreds per hour platform-wide)
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
## Summary
Updates the AI chat docs to match the slim-wire + field-level merge
behavior shipped in #3719 and the precise `.in/append` cap +
CORS-readable 413 shipped in #3720. No behavior changes here — code is
correct in `main`; the docs were lagging on three patterns customers
copy out of the page.
## What changed
- **`hydrateMessages` examples upsert by id** (in `lifecycle-hooks.mdx`,
`patterns/database-persistence.mdx`, and
`patterns/persistence-and-replay.mdx`). The previous
`stored.push(newMsg)` pattern duplicated the assistant id on HITL
continuations and caused the LLM to receive a tool call with no
`arguments`. The new examples include the rationale inline.
- **`onValidateMessages` example filters to user messages**
(`lifecycle-hooks.mdx`). The previous example called
`validateUIMessages({ messages, tools })` directly, which now throws on
HITL slim wires (the AI SDK schema requires `input` on resolved tool
parts). New example shows the filter pattern, with a Warning callout
explaining why.
- **Merge contract description updated** (`lifecycle-hooks.mdx`). The
old wording said incoming messages are "auto-merged" / "replaced"; the
new description explains the actual field-level overlay (state advances
only).
- **Approval-responded wire example slimmed** (`client-protocol.mdx`).
Shows the minimum shape the agent reads — `state` + `approval` (or
`output` / `errorText` for HITL). Notes that the built-in transports
ship this slim shape by default and that fuller shapes are still
accepted.
- **`/in/append` 413 row and FAQ updated** (`client-protocol.mdx`,
`patterns/trusted-edge-signals.mdx`). Reflects the new precise S2 cap
and the CORS-readable 413.
- **New changelog entry** at the top of `changelog.mdx` covering all of
the above.
The historical `## 512 KiB ceiling removed` entry further down the
changelog is left as-is (it's a snapshot of the prior transition), and
the v4.5 upgrade-guide section is skipped — the merge contract is
backwards compatible.
## Test plan
- Mintlify dev preview renders cleanly with no broken anchors
- Linked references resolve (`/ai-chat/lifecycle-hooks#hydratemessages`,
`/ai-chat/lifecycle-hooks#onvalidatemessages`,
`/ai-chat/patterns/database-persistence#alternative-hydratemessages`,
`/ai-chat/client-protocol#step-3-send-messages-stops-and-actions`,
`/ai-chat/patterns/large-payloads`)
## Summary
3 improvements, 1 bug fix.
## Improvements
- The per-turn merge now overlays the wire copy's tool-part state
advancement onto the agent's existing chain — `state` + the matching
resolution field (`output` / `errorText` / `approval`) come from the
wire, everything else (text, reasoning, tool `input`, provider metadata)
stays whatever the snapshot or `hydrateMessages` returned. Previously a
full-message replace overwrote those fields with whatever the client
shipped, so a slimmed wire copy landed a tool call with no `arguments`
on the next LLM call. Covers `output-available` / `output-error` (HITL
`addToolOutput`) and `approval-responded` / `output-denied` (approval
flow).
- `TriggerChatTransport.sendMessages` and `AgentChat.sendRaw` now slim
assistant messages that carry advanced tool parts. The wire payload is
just `{ id, role, parts: [<state + resolution field>] }` for
`submit-message` continuations; everything else passes through.
Reasoning blobs and full tool inputs no longer ride the wire on every
`addToolOutput` / `addToolApproveResponse`, so continuation payloads
stay well under the `.in/append` cap on long agent loops.
- Add `TriggerClient` for running multiple SDK clients side-by-side,
each with its own auth, preview branch, and baseURL. Useful when a
single process needs to trigger tasks or read runs across multiple
projects, environments, or preview branches without mutating shared
global state.
([#3683](https://github.com/triggerdotdev/trigger.dev/pull/3683))
## Bug fixes
- Fix `chat.agent` HITL continuations on reasoning-heavy turns. Two
changes that work together:
([#3719](https://github.com/triggerdotdev/trigger.dev/pull/3719))
<details>
<summary>Raw changeset output</summary>
⚠️⚠️⚠️⚠️⚠️⚠️
`main` is currently in **pre mode** so this branch has prereleases
rather than normal releases. If you want to exit prereleases, run
`changeset pre exit` on `main`.
⚠️⚠️⚠️⚠️⚠️⚠️
# Releases
## @trigger.dev/build@4.5.0-rc.2
### Patch Changes
- Updated dependencies:
- `@trigger.dev/core@4.5.0-rc.2`
## trigger.dev@4.5.0-rc.2
### Patch Changes
- Updated dependencies:
- `@trigger.dev/build@4.5.0-rc.2`
- `@trigger.dev/core@4.5.0-rc.2`
- `@trigger.dev/schema-to-json@4.5.0-rc.2`
## @trigger.dev/plugins@4.5.0-rc.2
### Patch Changes
- Updated dependencies:
- `@trigger.dev/core@4.5.0-rc.2`
## @trigger.dev/python@4.5.0-rc.2
### Patch Changes
- Updated dependencies:
- `@trigger.dev/sdk@4.5.0-rc.2`
- `@trigger.dev/build@4.5.0-rc.2`
- `@trigger.dev/core@4.5.0-rc.2`
## @trigger.dev/react-hooks@4.5.0-rc.2
### Patch Changes
- Updated dependencies:
- `@trigger.dev/core@4.5.0-rc.2`
## @trigger.dev/redis-worker@4.5.0-rc.2
### Patch Changes
- Updated dependencies:
- `@trigger.dev/core@4.5.0-rc.2`
## @trigger.dev/rsc@4.5.0-rc.2
### Patch Changes
- Updated dependencies:
- `@trigger.dev/core@4.5.0-rc.2`
## @trigger.dev/schema-to-json@4.5.0-rc.2
### Patch Changes
- Updated dependencies:
- `@trigger.dev/core@4.5.0-rc.2`
## @trigger.dev/sdk@4.5.0-rc.2
### Patch Changes
- Fix `chat.agent` HITL continuations on reasoning-heavy turns. Two
changes that work together:
([#3719](https://github.com/triggerdotdev/trigger.dev/pull/3719))
- The per-turn merge now overlays the wire copy's tool-part state
advancement onto the agent's existing chain — `state` + the matching
resolution field (`output` / `errorText` / `approval`) come from the
wire, everything else (text, reasoning, tool `input`, provider metadata)
stays whatever the snapshot or `hydrateMessages` returned. Previously a
full-message replace overwrote those fields with whatever the client
shipped, so a slimmed wire copy landed a tool call with no `arguments`
on the next LLM call. Covers `output-available` / `output-error` (HITL
`addToolOutput`) and `approval-responded` / `output-denied` (approval
flow).
- `TriggerChatTransport.sendMessages` and `AgentChat.sendRaw` now slim
assistant messages that carry advanced tool parts. The wire payload is
just `{ id, role, parts: [<state + resolution field>] }` for
`submit-message` continuations; everything else passes through.
Reasoning blobs and full tool inputs no longer ride the wire on every
`addToolOutput` / `addToolApproveResponse`, so continuation payloads
stay well under the `.in/append` cap on long agent loops.
Note: `onValidateMessages` receives the slim wire on HITL turns. If you
call `validateUIMessages` from `ai` against the full `messages` array it
will reject the slim assistant; filter to user messages (or skip on HITL
turns) — see the updated docstring on `onValidateMessages` for the
recommended pattern.
For `hydrateMessages` hooks that persist the chain, this release also
adds a small helper to the `@trigger.dev/sdk/ai` surface:
```ts
import { chat, upsertIncomingMessage } from "@trigger.dev/sdk/ai";
chat.agent({
hydrateMessages: async ({ chatId, trigger, incomingMessages }) => {
const record = await db.chat.findUnique({ where: { id: chatId } });
const stored = record?.messages ?? [];
if (upsertIncomingMessage(stored, { trigger, incomingMessages })) {
await db.chat.update({ where: { id: chatId }, data: { messages: stored }
});
}
return stored;
},
});
```
It pushes fresh user messages by id, no-ops on HITL continuations (the
incoming shares an id with the existing assistant — the runtime overlays
the new tool-state advance), and skips on non-`submit-message` triggers.
Returns `true` if it mutated `stored` so the caller knows whether to
persist.
Net effect: `chat.addToolOutput(...)` /
`chat.addToolApproveResponse(...)` on multi-step reasoning agents
(OpenAI Responses with `store: false`, Anthropic extended thinking,
etc.) no longer blows the cap and no longer corrupts the LLM input.
- Add `TriggerClient` for running multiple SDK clients side-by-side,
each with its own auth, preview branch, and baseURL. Useful when a
single process needs to trigger tasks or read runs across multiple
projects, environments, or preview branches without mutating shared
global state.
([#3683](https://github.com/triggerdotdev/trigger.dev/pull/3683))
```ts
import { TriggerClient } from "@trigger.dev/sdk";
const prod = new TriggerClient({ accessToken:
process.env.TRIGGER_PROD_KEY });
const preview = new TriggerClient({
accessToken: process.env.TRIGGER_PREVIEW_KEY,
previewBranch: "signup-flow",
});
await prod.tasks.trigger("send-email", payload);
await preview.runs.list({ status: ["COMPLETED"] });
```
- Updated dependencies:
- `@trigger.dev/core@4.5.0-rc.2`
## @trigger.dev/core@4.5.0-rc.2
</details>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
## Summary
Multi-step reasoning agents with HITL tools (OpenAI Responses with
`store: false`, Anthropic extended thinking, etc.) failed on
`chat.addToolOutput(...)` continuations — either the wire payload blew
the `.in/append` cap (reasoning blobs + tool inputs routinely > 512
KiB), or app-side slimming workarounds got overwritten server-side and
the next LLM call landed a tool call with no `arguments`. Both modes are
fixed.
## Design
The per-turn merge in `chat.agent` now overlays only the tool-part state
advances (`output-available` / `output-error` / `approval-responded` /
`output-denied`) from the wire copy onto the hydrated/snapshot chain.
Previously it replaced the entire message, which dropped `input`,
reasoning, and text from the LLM's view whenever the wire was slim.
In parallel, `TriggerChatTransport.sendMessages` and `AgentChat.sendRaw`
now slim the assistant message themselves on `submit-message`
continuations: ship `{ id, role, parts: [<resolved tool part only>] }`,
everything else reconstructed server-side from `hydrateMessages` or the
durable snapshot. Continuation payloads drop from 600 KiB – 1 MiB to ~1
KiB.
`references/ai-chat` `aiChatHydrated.hydrateMessages` now upserts by id
instead of pushing. With slim continuations, a blind push duplicates the
assistant id in the returned chain — the merge updates the first match,
the slim duplicate goes straight to `toModelMessages` with no `input`,
and the LLM 4xx's. This is the canonical pattern customers should mirror
in their own hydrate implementations.
## Test plan
- 11 new tests (slim helper unit + slim+merge integration for HITL,
approval, default no-hydrate branch)
- Full SDK suite: 239 tests pass across 19 files
- End-to-end sweep against `references/ai-chat`: 19 customer-side smoke
tests green; HITL wire bodies confirmed at ~1 KiB (was 600 KiB+); no
provider 4xx errors across OpenAI Responses or Anthropic
## Summary
Two improvements to session `.in/append`:
- Oversize-body 413 responses now carry CORS headers, so browser fetches
see a readable status instead of an opaque `TypeError: Failed to fetch`.
App-side retry-on-disconnect loops no longer spin forever on a
permanently-rejected payload.
- The per-record cap is now computed precisely against S2's actual
ceiling instead of the conservative 512 KiB floor. Legitimate ~600-900
KiB tool outputs (search results, file content) now succeed;
pathological all-quote content that would double under JSON escape still
rejects cleanly.
## Design
S2 enforces a per-record metered size of `8 + 2*H + Σ(header name +
value) + body ≤ 1048576` bytes. With no record headers (our case), the
budget reduces to `body ≤ 1048568`. Verified empirically against cloud
S2 — append succeeds at metered=1048576 and 422s at 1048577 with `record
must have metered size less than 1 MiB`.
The old `MAX_APPEND_BODY_BYTES = 512 KiB` was derived by assuming
worst-case JSON escape doubling (every byte becomes `\"` or `\\`),
giving `(1 MiB - overhead) / 2`. Safe, but rejects ~half the legitimate
input space.
The new flow:
1. Pre-cap the HTTP body at 1 MiB (DoS guard against reading arbitrary
garbage before we can compute the wrap).
2. After reading, `S2RealtimeStreams.#appendPartByName` computes
`Buffer.byteLength(JSON.stringify({data: part, id: partId}), "utf8") +
8` and throws `S2RecordTooLargeError` (a `ServiceValidationError` with
status 413) if it would exceed S2's ceiling. The route's existing error
branch maps the throw to a 413 with a descriptive message.
The 413 CORS fix is a single-line change in `apiBuilder.server.ts` —
`wrapResponse` was being skipped on the body-too-large branch; every
other error branch wraps; the 413 was the exception.
## Test plan
- Empirically verified against cloud S2 with a boundary scan across
`[1048568, 1048569, ..., 1048576]` and across H ∈ {0, 1×5 hdr bytes,
1×14 hdr bytes} — the formula matches exactly
- Browser-side fetch on a 700 KiB POST now resolves with a readable
`status: 413` (no `TypeError: Failed to fetch`)
- A 900 KiB ASCII tool output now passes (would have 413'd at 512 KiB
pre-fix)
## Bug
The `OrganizationDataStoresRegistry` singleton in
`apps/webapp/app/services/dataStores/organizationDataStoresRegistryInstance.server.ts`
was constructed with `$replica`. That client was then used by both the
polling read path *and* by `addDataStore` / `updateDataStore` /
`deleteDataStore` (and their backing `SecretStore.setSecret` upserts).
The write methods route through the read replica, which Postgres rejects
with **error code 25006**:
```
Invalid prisma.secretStore.upsert() invocation:
ConnectorError(ConnectorError { user_facing_error: None, kind: QueryError(PostgresError {
code: "25006",
message: "cannot execute INSERT in a read-only transaction",
...
}), transient: false })
```
User-visible symptom: the admin `/admin/data-stores` "Add data store"
form returns a 400 with this error wrapped, so no
`OrganizationDataStore` row can ever be created via the UI.
The read path (`loadFromDatabase` polling + `SecretStore.getSecret`) is
unaffected because `findMany` + secret read are read-only.
## Fix
Change the registry constructor to take both a writer and a replica:
```ts
constructor(writer: PrismaClient, replica: PrismaClient | PrismaReplicaClient)
```
- `loadFromDatabase()` keeps using `_replica` (and its
`SecretStore.getSecret` calls) — these are background cache-fillers, not
on user-latency-sensitive paths.
- `addDataStore` / `updateDataStore` / `deleteDataStore` (and their
`SecretStore.setSecret` / `deleteSecret` calls) now use `_writer`.
`organizationDataStoresRegistryInstance.server.ts` passes `(prisma,
$replica)` from `~/db.server`. Test sites that constructed with
`(prisma)` now pass `(prisma, prisma)` — the testcontainer exposes a
single client, so the writer/replica split collapses to one connection.
## Files
-
`apps/webapp/app/services/dataStores/organizationDataStoresRegistry.server.ts`
— constructor + read/write split
-
`apps/webapp/app/services/dataStores/organizationDataStoresRegistryInstance.server.ts`
— pass `prisma` alongside `$replica`
- `apps/webapp/test/organizationDataStoresRegistry.test.ts` — 14 call
sites bumped
- `apps/webapp/test/clickhouseFactory.test.ts` — 5 call sites bumped
## Test plan
- [x] Existing `organizationDataStoresRegistry.test.ts` +
`clickhouseFactory.test.ts` still pass (constructor sites updated;
behavior unchanged for tests).
- [ ] After deploy to test cloud, retry `/admin/data-stores` "Add data
store" form for the HIPAA org — should now succeed and the row should
appear.
- [ ] Verify the registry's polling reload picks up the new row within
`ORGANIZATION_DATA_STORES_RELOAD_INTERVAL_MS` (60s default) and the
factory starts routing to the org-scoped instance.
## Summary
When a background worker registers, the engine resolves runs that were
queued before the worker was ready (status `PENDING_VERSION`). That
lookup used to scan a Postgres status index on `TaskRun`. Move it to
ClickHouse: query candidate run ids from `task_runs_v2`, then refetch
the actual rows from Postgres by primary key with a `status =
'PENDING_VERSION'` guard for idempotency.
## Design
The lookup is a pluggable interface on the run engine
(`PendingVersionRunIdLookup`). The webapp wires a ClickHouse-backed
implementation through the org-scoped `clickhouseFactory` using a new
`"engine"` client type, configured by `RUN_ENGINE_CLICKHOUSE_*` env
vars. The URL falls back to `CLICKHOUSE_URL` when unset, so self-hosted
deployments don't need new config to keep working.
When the lookup returns no candidates, one bounded retry is scheduled
~5s later to cover ClickHouse replication lag against `task_runs_v2`.
The Postgres status guard on both the candidate refetch and the inner
`updateMany` prevents double-promotion when a retry races with a
concurrent deploy.
Tests cover three existing PENDING_VERSION cases via a small
Postgres-backed test adapter; new ClickHouse-backed integration tests
will follow.
## Summary
Drops the unused composite Postgres index
`TaskRun_scheduleId_createdAt_idx`. The schedule list view reads from
ClickHouse, so this index served no Prisma query while still being
maintained on every `TaskRun` INSERT/UPDATE. Removing it reduces write
amplification on the primary database.
Sibling to the prior drop of `TaskRun_scheduleId_idx` and the earlier
removal of the `TaskRun.scheduleId` foreign key — all stemming from
migrating schedule-aware reads to ClickHouse.
## Verification
- Sampled `pg_stat_user_indexes` for `TaskRun` over multiple hours —
zero scans against this index.
- Grepped the codebase for any Prisma query filtering
`TaskRun.scheduleId` — none found. All schedule-aware listing routes
through `clickhouseRunsRepository`.
Make the Express server's `keepAliveTimeout` configurable via
`HTTP_KEEPALIVE_TIMEOUT_MS`. Default preserved at 65000 ms — no behavior
change if unset.
Workloads bundled with CLI versions before v4.4.4 use a strict zod enum
for `checkpoint.type` that only allows DOCKER and KUBERNETES. When a
customer's runs are routed via the compute path, those old runners
receive `type: "COMPUTE"` on `/snapshots/since/...` and `/dequeue`
responses and fail validation - blocking silent migration of existing
deployments.
The workload never reads the field - only validates the shape. Rewriting
COMPUTE -> KUBERNETES on the way out lets older runners keep parsing
while the database and internal services keep the real value. Limited to
the two workload-facing endpoints whose response includes a checkpoint;
`/continue`, `/attempts/start`, `/attempts/complete` all return shapes
without one.
Followup to #3114.
## Summary
Stamp every Sentry event with the signed-in user and the tenant (org /
project / env) the request belongs to, so "Users Impacted" counts
distinct humans and events become filterable per tenant.
**Design after review (current):**
- `user.id = real user cuid` (from `requireUser`). "Users Impacted"
counts humans, not tenants.
- Tenant context (org / project / env slugs, IDs, env type) moves
entirely onto tags: `org_slug`, `project_slug`, `env_slug`, `org_id`,
`project_id`, `project_ref`, `environment_id`, `env_type`, plus
`impersonating` when set.
- Backed by an `AsyncLocalStorage` scope established at the HTTP entry.
Each entry point fills what it knows; loaders enrich the same scope with
what they already have.
**Zero new database queries.** The middleware does a regex match only.
Dashboard loaders that already query Prisma gain a couple of extra
selected columns; nothing new round-trips.
## How it's wired
- **Express middleware (`tenantContextResolver.server.ts`)** — parses
the URL with a regex and always opens an ALS scope. Populates whatever
subset of slugs is present: `/orgs/:o` → just `orgSlug`;
`/orgs/:o/projects/:p` adds `projectSlug`; the full triple adds
`envSlug`. Non-tenant paths get an empty scope so loaders can still
enrich.
- **`_app/route.tsx`** — already calls `requireUser`. Adds
`tenantContext.enrich({ userId: user.id })` for every authenticated
dashboard request. No new query.
- **Env layout loader (`_app.orgs.$o.projects.$p.env.$e/route.tsx`)** —
its existing `prisma.project.findFirst` gains two columns in `select`
(`externalRef`, `organization.id`). After it picks an env, calls
`tenantContext.enrich({ orgId, projectId, projectRef, envId, envType
})`. Same query, +2 columns.
- **API path (`apiBuilder.server.ts`)** — wraps every handler in
`tenantContext.run(tenantContextFromAuthEnvironment(authenticationResult.environment),
…)`. The mapper pulls `userId` from `env.orgMember?.userId` (already
selected by `authIncludeBase` — no schema change). Covers
`createLoaderApiRoute`, `createActionApiRoute`, and
`createMultiMethodApiRoute`.
- **Event processor (`sentryTenantContext.server.ts`)** — registered in
`entry.server.tsx` so it lives in the Remix bundle and shares the same
`tenantContext` ALS instance as the middleware and loaders. Stamps
whatever's present; nothing forced.
## Example events from local verification
| URL | `user.id` | Tags |
|-----|-----------|------|
| `/orgs/:o/projects/:p/env/:e/...` | real user cuid | `org_slug`,
`project_slug`, `env_slug`, `org_id`, `project_id`, `project_ref`,
`environment_id`, `env_type` |
| `/orgs/:o/settings` (non-env-scoped) | real user cuid | `org_slug`
only |
| API request with `orgMember` | `orgMember.userId` | full tenant set |
| API request without `orgMember` | (unset) | full tenant set |
## Trade-offs
1. On env-scoped pages, errors that fire before the env layout loader's
enrich callback runs get slugs + `user.id` but not the tenant IDs /
`env_type`. Realistic errors deep in async work get the full set. (Same
race as before, narrower window now that slugs/`user.id` are populated
up-front by the middleware and `_app` enrich.)
2. API requests where the environment has no `orgMember` get tenant tags
but no `user.id`. Those events still show in the issue but don't
contribute to "Users Impacted".
## Out of scope (deferred)
Background workers (`redis-worker`, `schedule-engine`) and socket
handlers. Those entry points don't set `tenantContext.run` yet — their
events ship without tenant attribution until each is wired in a
follow-up.
## Tests
31 unit tests across 4 files. New tests notably cover:
- `parseTenantPath`: org-only, org+project, and full-triple URL
variants.
- `tenantContext.enrich`: in-place patch, no-op outside `run()`,
concurrent-scope isolation, empty-scope + enrich pattern (for non-tenant
pages).
- `tenantContextFromAuthEnvironment`: with and without `orgMember` —
verifies the API path's `user.id` mapping.
- `addTenantContextToEvent`: empty scope, userId-only, slugs-only, full
enrichment, conditional tag emission, preservation of prior `event.user`
fields.
## Test plan
- [ ] `pnpm run typecheck --filter webapp`
- [ ] `pnpm run test --filter webapp -- test/tenantContext.test.ts
test/sentryTenantContext.test.ts test/tenantContextResolver.test.ts
test/tenantContextFromAuthEnvironment.test.ts`
- [ ] Local manual: with `SENTRY_DSN` set, hit a dashboard URL and an
API route, confirm the captured events carry `user.id` + the expected
tag set in Sentry.
- [ ] After ship: confirm "Users Impacted" on a real Sentry issue
reflects distinct users (not tenants).
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
Added `OrganizationDataStore` which allows orgs to have data stored in
specific separate services.
For now this is just used for ClickHouse. When using ClickHouse we get a
client for the factory and pass in the org id.
Particular care has to be made with two hot-insert paths:
1. RunReplicationService
2. OTLPExporter
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: Claude <noreply@anthropic.com>
## Summary
`new TriggerClient({...})` exposes the management API (tasks, runs,
schedules, envvars, batch, queues, deployments, prompts, auth) as an
explicit instance with its own auth, preview branch, and baseURL.
Multiple clients can coexist in one process without mutating shared
global state — useful when a single service triggers across multiple
projects, environments, or preview branches.
```ts
import { TriggerClient } from "@trigger.dev/sdk";
const prod = new TriggerClient({ accessToken: process.env.TRIGGER_PROD_KEY });
const preview = new TriggerClient({
accessToken: process.env.TRIGGER_PREVIEW_KEY,
previewBranch: "signup-flow",
});
await prod.tasks.trigger("send-email", payload);
await preview.runs.list({ status: ["COMPLETED"] });
```
The existing global `configure()` API keeps working unchanged.
## Design
Instance methods enter an `AsyncLocalStorage`-backed scope (`sdkScope`)
before delegating to the existing module-level functions. The four
"pollution" points that previously read globals now consult the scope
first:
- `apiClientManager.{baseURL, accessToken, branchName}` and
`clientOrThrow` — identity fields are scope-only when scoped; `baseURL`
still falls back to `TRIGGER_API_URL` because plumbing (where the API
lives) is not identity.
- `taskContext.{ctx, worker, isWarmStart, isInsideTask}` — masked inside
an isolated scope so a `client.tasks.trigger(...)` from inside a task
doesn't leak the parent's `parentRunId` / `lockToVersion` / `isTest`
into a trigger that hits a different project.
- Inline `getEnvVar("TRIGGER_VERSION")` reads in `shared.ts` go through
a `scopedEnvVar` helper that returns `undefined` inside an isolated
scope.
The `TriggerClient` class itself is a thin wrapper that captures the
scope in its constructor and proxies each namespace method to enter that
scope before calling the existing impl. Generic inference (e.g.
`client.tasks.trigger<typeof t>(...)`) is preserved via `Pick<typeof ns,
keyof curatedSubset>` typings.
Two correctness fixes uncovered along the way are folded in:
- `apiClientManager.setGlobalAPIClientConfiguration` no longer silently
no-ops on the second call. `configure()` now actually overrides as users
expect (this is the root cause behind some "I changed the config but
nothing happened" reports).
- `apiClientManager.runWithConfig` (and therefore `auth.withAuth`) is
now backed by `sdkScope.withScope` instead of "mutate the global and
restore in finally". Two parallel `withAuth` calls with different
configs no longer stomp each other.
Surface curation: instance namespaces drop methods that don't make sense
per-instance — `batch.*AndWait` (runtime-dependent), `schedules.task` /
`schedules.timezones` (definition-time / stateless), `prompts.define`
(definition-time), `auth.configure` / `auth.withAuth` (global-only).
## Test plan
- [x] 9 runtime unit tests in `triggerClient.test.ts` cover: required
accessToken, instance auth + branch headers, no env fallback for
identity fields, no leakage between global and instance, four parallel
calls across two clients stay isolated, taskContext masking +
`inheritContext: true` override, `configure()` second-call override,
parallel `auth.withAuth` isolation.
- [x] 10 type-level assertions in `triggerClient.types.test.ts` using
`expectTypeOf` + `@ts-expect-error` lock in generic inference, return
type passthrough, overload preservation, and curated-surface drift.
- [x] Full SDK suite (219 tests) and core suite (530 tests) pass.
- [x] Webapp typecheck clean.
- [x] End-to-end smoke test against local webapp and a
freshly-provisioned cloud project — six concurrent multi-client triggers
all returned 200 with run IDs, headers per-client as expected.
- [ ] Reviewer: run `references/multi-client` per its `README.md` to
reproduce the smoke test locally.
## Try it
`references/multi-client` is a new reference workspace that exercises
this end-to-end:
- `src/trigger/echo.ts` — trivial target task
- `src/trigger/fanOut.ts` — opens two `TriggerClient`s from inside a
task, fires `echo` through each in parallel
- `src/external/main.ts` — external Node script with two clients
triggering `echo` sequentially and concurrently; logs every outgoing
request's `authorization` + `x-trigger-branch`
- `src/external/isolation.ts` — interleaves global `configure()` and an
instance call, asserts the captured fetch sequence shows no leakage
either way
Sibling to the weekly summary, focused on critical alerts only. Pings
Slack daily while any critical alerts are open; skips the post entirely
when zero, so no daily "all clear" noise.
- Daily 08:00 UTC cron + `workflow_dispatch` with `severity` input
(default `critical`, override to `high`/`medium`/`low` for manual
checks)
- Reuses the existing `dependabot-summary` environment (token, channel,
bot)
- Alerts link at the end is severity-filtered
## Summary
Two docs edits that close a footgun customers persisting transport state
can hit. Clearing `lastEventId` on `chat.endRun()` looks intuitive — the
Run ended, the cursor must be stale — but the cursor is sessionId-keyed,
not runId-keyed. Clearing it forces the next `sendMessages` to subscribe
from `seq_num=0`, which may hit the prior turn's still-durable
`turn-complete` record and close the SSE empty before the new Run's
chunks arrive.
Spells out the invariant in the frontend transport persistence table and
adds a Warning in the `chat.endRun()` reference.
## Test plan
- [x] Mintlify preview renders
- [x] No callout stacking
## Summary
Three post-merge fixes for the AI Agents docs (#3226), all caught by
review after merge.
## Fixes
- **`onTurnComplete` examples now use `db.$transaction`** — both the
Database persistence "Complete example" and the Lifecycle hooks
reference example were doing two separate `await` calls
(`db.chat.update` then `db.chatSession.upsert`). That's the exact
non-atomic pattern the warning earlier on the persistence page calls out
as ❌: a refresh between the two writes reads a stale `lastEventId` and
duplicates the assistant message on resume. Both examples now use the
recommended atomic form.
- **Background injection self-review prose aligned with the code** — the
prose said "gpt-4o-mini" but the example above it had been swapped to
`claude-haiku-4-5`. The Anthropic-sweep script only touched code blocks;
this prose line wasn't picked up.
## Test plan
- [x] Both updated examples use `db.$transaction([...])`
- [x] Prose matches the model used in the code block
- [ ] Mintlify deployment passes
2026-05-21 17:04:13 +01:00
3325 changed files with 262135 additions and 128179 deletions
Add Agent Skills for `chat.agent`. Drop a folder with a `SKILL.md` and any helper scripts/references next to your task code, register it with `skills.define({ id, path })`, and the CLI bundles it into the deploy image automatically — no `trigger.config.ts` changes. The agent gets a one-line summary in its system prompt and discovers full instructions on demand via `loadSkill`, with `bash` and `readFile` tools scoped per-skill (path-traversal guards, output caps, abort-signal propagation).
Built on the [AI SDK cookbook pattern](https://ai-sdk.dev/cookbook/guides/agent-skills) — portable across providers. SDK + CLI only for now; dashboard-editable `SKILL.md` text is on the roadmap.
**AI Prompts** — define prompt templates as code alongside your tasks, version them on deploy, and override the text or model from the dashboard without redeploying. Prompts integrate with the Vercel AI SDK via `toAISDKTelemetry()` (links every generation span back to the prompt) and with `chat.agent` via `chat.prompt.set()` + `chat.toStreamTextOptions()`.
```ts
import{prompts}from"@trigger.dev/sdk";
import{generateText}from"ai";
import{openai}from"@ai-sdk/openai";
import{z}from"zod";
exportconstsupportPrompt=prompts.define({
id:"customer-support",
model:"gpt-4o",
config:{temperature: 0.7},
variables: z.object({
customerName: z.string(),
plan: z.string(),
issue: z.string(),
}),
content:`You are a support agent for Acme.
Customer: {{customerName}} ({{plan}} plan)
Issue: {{issue}}`,
});
constresolved=awaitsupportPrompt.resolve({
customerName:"Alice",
plan:"Pro",
issue:"Can't access billing",
});
constresult=awaitgenerateText({
model: openai(resolved.model??"gpt-4o"),
system: resolved.text,
prompt:"Can't access billing",
...resolved.toAISDKTelemetry(),
});
```
**What you get:**
- **Code-defined, deploy-versioned templates** — define with `prompts.define({ id, model, config, variables, content })`. Every deploy creates a new version visible in the dashboard. Mustache-style placeholders (`{{var}}`, `{{#cond}}...{{/cond}}`) with Zod / ArkType / Valibot-typed variables.
- **Dashboard overrides** — change a prompt's text or model from the dashboard without redeploying. Overrides take priority over the deployed "current" version and are environment-scoped (dev / staging / production independent).
- **Resolve API** — `prompt.resolve(vars, { version?, label? })` returns the compiled `text`, resolved `model`, `version`, and labels. Standalone `prompts.resolve<typeof handle>(slug, vars)` for cross-file resolution with full type inference on slug and variable shape.
- **AI SDK integration** — spread `resolved.toAISDKTelemetry({ ...extra })` into any `generateText` / `streamText` call and every generation span links to the prompt in the dashboard alongside its input variables, model, tokens, and cost.
- **`chat.agent` integration** — `chat.prompt.set(resolved)` stores the resolved prompt run-scoped; `chat.toStreamTextOptions({ registry })` pulls `system`, `model` (resolved via the AI SDK provider registry), `temperature` / `maxTokens` / etc., and telemetry into a single spread for `streamText`.
- **Dashboard** — prompts list with per-prompt usage sparklines; per-prompt detail with Template / Details / Versions / Generations / Metrics tabs. AI generation spans get a custom inspector showing the linked prompt's metadata, input variables, and template content alongside model, tokens, cost, and the message thread.
See [/docs/ai/prompts](https://trigger.dev/docs/ai/prompts) for the full reference — template syntax, version resolution order, override workflow, and type utilities (`PromptHandle`, `PromptIdentifier`, `PromptVariables`).
Add `ai.toolExecute(task)` so you can wire a Trigger subtask in as the `execute` handler of an AI SDK `tool()` while defining `description` and `inputSchema` yourself — useful when you want full control over the tool surface and just need Trigger's subtask machinery for the body.
```ts
constmyTool=tool({
description:"...",
inputSchema: z.object({...}),
execute: ai.toolExecute(mySubtask),
});
```
`ai.tool(task)` (`toolFromTask`) keeps doing the all-in-one wrap and now aligns its return type with AI SDK's `ToolSet`. Minimum `ai` peer raised to `^6.0.116` to avoid cross-version `ToolSet` mismatches in monorepos.
Fix `chat.agent` skills silently missing in `trigger dev` for projects whose task files read `process.env` at module top level (e.g. a third-party SDK client initialized at import). Skill folders now bundle into `.trigger/skills/` reliably regardless of which env vars are set when the CLI launches.
Reject overlong `idempotencyKey` values at the API boundary so they no longer trip an internal size limit on the underlying unique index and surface as a generic 500. Inputs are capped at 2048 characters — well above what `idempotencyKeys.create()` produces (a 64-character hash) and above any realistic raw key. Applies to `tasks.trigger`, `tasks.batchTrigger`, `batch.create` (Phase 1 streaming batches), `wait.createToken`, `wait.forDuration`, and the input/session stream waitpoint endpoints. Over-limit requests now return a structured 400 instead.
Adds `onBoot` to `chat.agent` — a lifecycle hook that fires once per worker process picking up the chat. Runs for the initial run, preloaded runs, AND reactive continuation runs (post-cancel, crash, `endRun`, `requestUpgrade`, OOM retry), before any other hook. Use it to initialize `chat.local`, open per-process resources, or re-hydrate state from your DB on continuation — anywhere the SAME run picking up after suspend/resume isn't enough.
Use `onBoot` (not `onChatStart`) for state setup that must run every time a worker picks up the chat — `onChatStart` fires once per chat and won't run on continuation, leaving `chat.local` uninitialized when `run()` tries to use it.
Preserve the partial assistant message when a chat turn's model stream fails mid-response. `chat.agent` now passes the recovered partial to `onTurnComplete`, and `chat.createSession`'s `turn.complete()` keeps it before rethrowing, instead of dropping the streamed-so-far output.
**AI Agents** — run AI SDK chat completions as durable Trigger.dev agents instead of fragile API routes. Define an agent in one function, point `useChat` at it from React, and the conversation survives page refreshes, network blips, and process restarts.
- **AI SDK `useChat` integration** — a custom [`ChatTransport`](https://sdk.vercel.ai/docs/ai-sdk-ui/transport) (`useTriggerChatTransport`) plugs straight into Vercel AI SDK's `useChat` hook. Text streaming, tool calls, reasoning, and `data-*` parts all work natively over Trigger.dev's realtime streams. No custom API routes needed.
- **First-turn fast path (`chat.headStart`)** — opt-in handler that runs the first turn's `streamText` step in your warm server process while the agent run boots in parallel, cutting cold-start TTFC by roughly half (measured 2801ms → 1218ms on `claude-sonnet-4-6`). The agent owns step 2+ (tool execution, persistence, hooks) so heavy deps stay where they belong. Web Fetch handler works natively in Next.js, Hono, SvelteKit, Remix, Workers, etc.; bridge to Express/Fastify/Koa via `chat.toNodeListener`. New `@trigger.dev/sdk/chat-server` subpath.
- **Multi-turn durability via Sessions** — every chat is backed by a durable Session that outlives any individual run. Conversations resume across page refreshes, idle timeout, crashes, and deploys; `resume: true` reconnects via `lastEventId` so clients only see new chunks. `sessions.list` enumerates chats for inbox-style UIs.
- **Auto-accumulated history, delta-only wire** — the backend accumulates the full conversation across turns; clients only ship the new message each turn. Long chats never hit the 512 KiB body cap. Register `hydrateMessages` to be the source of truth yourself.
- **Lifecycle hooks** — `onPreload`, `onChatStart`, `onValidateMessages`, `hydrateMessages`, `onTurnStart`, `onBeforeTurnComplete`, `onTurnComplete`, `onChatSuspend`, `onChatResume` — for persistence, validation, and post-turn work.
- **Stop generation** — client-driven `transport.stopGeneration(chatId)` aborts mid-stream; the run stays alive for the next message, partial response is captured, and aborted parts (stuck `partial-call` tools, in-progress reasoning) are auto-cleaned.
- **Tool approvals (HITL)** — tools with `needsApproval: true` pause until the user approves or denies via `addToolApprovalResponse`. The runtime reconciles the updated assistant message by ID and continues `streamText`.
- **Steering and background injection** — `pendingMessages` injects user messages between tool-call steps so users can steer the agent mid-execution; `chat.inject()` + `chat.defer()` adds context from background work (self-review, RAG, safety checks) between turns.
- **Actions** — non-turn frontend commands (undo, rollback, regenerate, edit) sent via `transport.sendAction`. Fire `hydrateMessages` + `onAction` only — no turn hooks, no `run()`. `onAction` can return a `StreamTextResult` for a model response, or `void` for side-effect-only.
- **Typed state primitives** — `chat.local<T>` for per-run state accessible from hooks, `run()`, tools, and subtasks (auto-serialized through `ai.toolExecute`); `chat.store` for typed shared data between agent and client; `chat.history` for reading and mutating the message chain; `clientDataSchema` for typed `clientData` in every hook.
- **`chat.toStreamTextOptions()`** — one spread into `streamText` wires up versioned system [Prompts](https://trigger.dev/docs/ai/prompts), model resolution, telemetry metadata, compaction, steering, and background injection.
- **Multi-tab coordination** — `multiTab: true` + `useMultiTabChat` prevents duplicate sends and syncs state across browser tabs via `BroadcastChannel`. Non-active tabs go read-only with live updates.
- **Network resilience** — built-in indefinite retry with bounded backoff, reconnect on `online` / tab refocus / bfcache restore, `Last-Event-ID` mid-stream resume. No app code needed.
See [/docs/ai-chat](https://trigger.dev/docs/ai-chat/overview) for the full surface — quick start, three backend approaches (`chat.agent`, `chat.createSession`, raw task), persistence and code-sandbox patterns, type-level guides, and API reference.
Add read primitives to `chat.history` for HITL flows: `getPendingToolCalls()`, `getResolvedToolCalls()`, `extractNewToolResults(message)`, `getChain()`, and `findMessage(messageId)`. These lift the accumulator-walking logic that customers building human-in-the-loop tools were re-implementing into the SDK.
Use `getPendingToolCalls()` to gate fresh user turns while a tool call is awaiting an answer. Use `extractNewToolResults(message)` to dedup tool results when persisting to your own store — the helper returns only the parts whose `toolCallId` is not already resolved on the chain.
```ts
constpending=chat.history.getPendingToolCalls();
if(pending.length>0){
// an addToolOutput is expected before a new user message
Stamp `gen_ai.conversation.id` (the chat id) on every span and metric emitted from inside a `chat.task` or `chat.agent` run. Lets you filter dashboard spans, runs, and metrics by the chat conversation that produced them — independent of the run boundary, so multi-run chats correlate cleanly. No code changes required on the user side.
`ChatStartSessionParams` gains a typed `clientData` field — folded into the first run's `payload.metadata` so `onPreload` / `onChatStart` see the same shape per-turn `metadata` carries via the transport. The opaque session-level `metadata` field is unchanged.
Allow additional environment API keys to create scoped public access tokens through the Trigger.dev API. Use server-issued public access tokens for batch operations so environment-scoped API keys can read batch results.
Fix `LocalsKey<T>` type incompatibility across dual-package builds. The phantom value-type brand no longer uses a module-level `unique symbol`, so a single TypeScript compilation that resolves the type from both the ESM and CJS outputs (which can happen under certain pnpm hoisting layouts) no longer sees two structurally-incompatible variants of the same type.
The CLI MCP server's agent-chat tools (`start_agent_chat`, `send_agent_message`, `close_agent_chat`) now run on the new Sessions primitive, so AI assistants driving a `chat.agent` get the same idempotent-by-`chatId`, durable-across-runs behavior the browser transport gets. Required PAT scopes go from `write:inputStreams` to `read:sessions` + `write:sessions`.
Unit-test `chat.agent` definitions offline with `mockChatAgent` from `@trigger.dev/sdk/ai/test`. Drives a real agent's turn loop in-process — no network, no task runtime — so you can send messages, actions, and stop signals via driver methods, inspect captured output chunks, and verify hooks fire. Pairs with `MockLanguageModelV3` from `ai/test` for model mocking. `setupLocals` lets you pre-seed `locals` (DB clients, service stubs) before `run()` starts.
The broader `runInMockTaskContext` harness it's built on lives at `@trigger.dev/core/v3/test` — useful for unit-testing any task code, not just chat.
Add MollifierBuffer and MollifierDrainer primitives for trigger burst smoothing.
MollifierBuffer (`accept`, `pop`, `ack`, `requeue`, `fail`, `evaluateTrip`) is a per-env FIFO over Redis with atomic Lua transitions for status tracking. `evaluateTrip` is a sliding-window trip evaluator the webapp gate uses to detect per-env trigger bursts.
MollifierDrainer pops entries through a polling loop with a user-supplied handler. The loop survives transient Redis errors via capped exponential backoff (up to 5s), and per-env pop failures don't poison the rest of the batch — one env's blip is logged and counted as failed for that tick. Rotation is two-level: orgs at the top, envs within each org. The buffer maintains `mollifier:orgs` and `mollifier:org-envs:${orgId}` atomically with per-env queues, so the drainer walks orgs → envs directly without an in-memory cache. The `maxOrgsPerTick` option (default 500) caps how many orgs are scheduled per tick; for each picked org, one env is popped (rotating round-robin within the org). An org with N envs gets the same per-tick scheduling slot as an org with 1 env, so tenant-level drainage throughput is determined by org count rather than env count.
Correct the `expirationTime` docs on `auth.createPublicToken` and the trigger-token helpers: a number is a Unix timestamp in seconds, not milliseconds.
Fix `COULD_NOT_FIND_EXECUTOR` when a task's definition is loaded via `await import(...)` from inside another task's `run()`. The runtime workers now register such tasks with a sentinel file context, and the catalog logs a one-time warning per task id.
Retry `TASK_PROCESS_SIGSEGV` task crashes under the user's retry policy instead of failing the run on the first segfault. SIGSEGV in Node tasks is frequently non-deterministic (native addon races, JIT/GC interaction, near-OOM in native code, host issues), so retrying on a fresh process often succeeds. The retry is gated by the task's existing `retry` config + `maxAttempts` — same path `TASK_PROCESS_SIGTERM` and uncaught exceptions already use — so tasks without a retry policy still fail fast.
Add `region` to the runs list / retrieve API: filter runs by region (`runs.list({ region: "..." })` / `filter[region]=<masterQueue>`) and read each run's executing region from the new `region` field on the response.
**Sessions** — a durable, run-aware stream channel keyed on a stable `externalId`. A Session is the unit of state that owns a multi-run conversation: messages flow through `.in`, responses through `.out`, both survive run boundaries. Sessions back the new `chat.agent` runtime, and you can build on them directly for any pattern that needs durable bi-directional streaming across runs.
```ts
import{sessions,tasks}from"@trigger.dev/sdk";
// Trigger a task and subscribe to its session output in one call
See [/docs/ai-chat/overview](https://trigger.dev/docs/ai-chat/overview) for the full surface — Sessions powers the durable, resumable chat runtime described there.
@@ -15,12 +15,44 @@ Reserve 🔴 for things that would page someone or block a rollback. In this cod
- **Queue / concurrency correctness.** RunQueue, MarQS (V1, legacy), redis-worker — any change to enqueue / dequeue / locking semantics. Re-derive the invariant on paper before flagging or accepting.
- **Missing index on a hot table.** New Prisma queries against `TaskRun`, `TaskRunExecutionSnapshot`, `JobRun`, `Project`, etc. must use an existing index. Check `internal-packages/database/prisma/schema.prisma` for the relevant `@@index` lines — don't guess and don't propose `EXPLAIN`.
- **Recovery-path queries.** Any `TaskRun.findFirst` / `findMany` added to a schedule, run-recovery, or restart loop. Recovery fan-outs (Redis crash, restart storms) turn "rare indexed query" into a DB incident. 🔴 even if indexed.
- **Aggregations on hot tables.** No `COUNT` / `GROUP BY` on `TaskRun` or other multi-million-row tables. Use Redis or ClickHouse for counts.
- **Aggregations on hot tables.** No `COUNT` / `GROUP BY` on `TaskRun` or other tables that can reach billions of rows. Use Redis or ClickHouse for counts.
- **Prod Redis blast-radius.** New code paths that `SCAN` with broad patterns (`*foo*`) on prod-shaped Redis, or `EVAL` Lua with `SCAN` loops inside. Both are 🔴.
- **`@trigger.dev/core` direct import** from anywhere outside the SDK package. Always import from `@trigger.dev/sdk`. Core direct imports are 🔴 — they break the public API contract.
- **Heavy execute-deps imported into request-handler bundles.** Specifically `chat.handover` and similar split-bundle entry points must not transitively import the agent task's execute path. Watch for new imports added at module top-level of route files.
- **V1 engine code modified in a "V2 only" PR.** The `apps/webapp/app/v3/` directory contains both. If the PR description says V2-only but it touches `triggerTaskV1`, `cancelTaskRunV1`, `MarQS`, etc. — 🔴.
## Performance (always review)
Every PR gets a performance pass — not just the ones that look perf-sensitive. For each new query or unit of work, weigh three things: (a) the size of the table it hits, (b) whether it sits on a hot path, (c) whether the data it walks can be deep or wide (run trees, batches). The 🔴 bullets above on indexes, recovery-path queries, aggregations, and Redis `SCAN` are part of this pass — the rest below extends it.
**Treat these tables as large — no scans, no `COUNT` / `GROUP BY`, no unbounded fetch:**
- **Postgres — the `TaskRun` family:** `TaskRun`, `TaskRunExecutionSnapshot`, `Waitpoint`, `BatchTaskRun` and their join tables. Assume billions of rows.
- **ClickHouse — `task_events_v1` / `task_events_v2`.** Partitioned by `toDate(inserted_at)`; `ORDER BY (environment_id, toUnixTimestamp(start_time), trace_id)`. Note `span_id` / `parent_span_id` are NOT in the sort key — span-id lookups can't skip granules, only `environment_id` + a `start_time` window can.
**Hot paths — extra scrutiny on any added query or work:**
- **Trigger + batch trigger** (`triggerTask.server.ts`, `batchTriggerV3.server.ts`) — see `apps/webapp/CLAUDE.md`; do not add DB queries to these.
- **Dequeue / RunQueue** (`dequeueSystem.ts`, run-queue read/lock paths) — runs on every execution.
- **Execution-snapshot creation in the run engine** — any engine function that writes a `TaskRunExecutionSnapshot` runs per state transition; a new query there multiplies by run volume.
- Batch + parent/child fan-out (one run triggers thousands of children).
- Waitpoint / run-dependency chains.
- Tag / attribute many-to-many joins against the run/event tables.
**Anti-patterns (severity):**
- **Per-level fan-out that re-scans a large table once per tree depth** → 🔴. A BFS issuing one query per level (e.g. `parent_span_id IN {thisLevel}`) re-reads the same granules D times for a depth-D tree. Prefer one windowed query + an in-memory tree build.
- **Dropping the partition-pruning predicate** — `inserted_at` for ClickHouse, the `createdAt` window for partitioned Postgres — to "widen" a lookup → 🔴. Without it the query scans every partition. Keep a bounded window even for ancestor / backfill lookups.
- **Unbounded `IN (...)` built from a result set** (a BFS frontier, a batch's child ids) → 🟡. It can reach the row cap (`MAXIMUM_TRACE_SUMMARY_VIEW_COUNT` defaults to 25k). Cap or chunk to ≤1–2k ids per query.
- **Sequential per-level round-trips** where one recursive or windowed query would do → 🟡. N levels = N round-trip latencies stacked.
- **Replacing a single bounded query with a multi-query walk for _every_ call** (not just a rare fallback) → 🔴 on a hot read path, 🟡 elsewhere. Keep the cheap single-query path; branch into the expensive walk only when the cheap one comes up short.
## Always check
- **Tests use testcontainers, not mocks.** Vitest with `redisTest` / `postgresTest` / `containerTest` from `@internal/testcontainers`. Any new `vi.mock(...)` on Redis, Postgres, BullMQ, or other infra is wrong here — 🔴 if added in production-path tests, 🟡 if isolated unit test.
@@ -33,7 +65,7 @@ Reserve 🔴 for things that would page someone or block a rollback. In this cod
## Skip (do NOT flag)
- Anything Prettier / ESLint catches. CI runs both.
- Anything oxfmt / oxlint catches. CI enforces both via the `code-quality` check.
- TypeScript style preferences (`type` vs `interface`) — already covered by repo standards.
- Test coverage exhortations as a generic suggestion. Only flag missing tests when a specific code path is genuinely untested and the path has prior incidents.
-`agentcrumbs` markers (`// @crumbs`, `// #region @crumbs`) and `agentcrumbs` imports — these are temporary debug instrumentation stripped before merge.
The `v3/` directory name is misleading - most code here is actively used by the current V2 engine. Only the specific files below are legacy V1-only code.
The v3 engine (RunEngineVersion `V1`: MarQS queue + Graphile worker) is end-of-life and its execution code has been removed from the webapp. The `app/v3/` directory name is historical: everything under it now serves the current V2 engine (`@internal/run-engine` + `@trigger.dev/redis-worker`).
## V1-Only Files - Never Modify
There is no `V1` execution path anymore. If you find a `RunEngineVersion` branch, the `V1` arm should only reject or finalize gracefully (for example, mark a historical run cancelled in the DB), never run V1 work. Do not reintroduce MarQS, the graphile worker, or the v3 socket.io namespaces.
-`authenticatedSocketConnection.server.ts` (V1 dev WebSocket using DevQueueConsumer)
-`sharedSocketConnection.ts` (V1 shared queue socket using SharedQueueConsumer)
## The deprecation boundary (keep this)
## V1/V2 Branching Pattern
Requests from clients still on v3 (old SDK/CLI) or historical V1 runs must return a clean 4xx, never a 5xx. The boundary lives in:
Some services act as routers that branch on `RunEngineVersion`:
-`services/cancelTaskRun.server.ts` - calls V1 service or `engine.cancelRun()` for V2
-`services/batchTriggerV3.server.ts` - uses marqs for V1 path, run-engine for V2
-`engineDeprecation.server.ts` - the `V3_TRIGGER_DEPRECATION_MESSAGE` / `V3_DEV_DEPRECATION_MESSAGE` / `V3_MIGRATION_URL` upgrade messages.
-`engineVersion.server.ts` - `determineEngineVersion()` still detects a V1 project/run so callers can reject it.
-`services/triggerTask.server.ts`, `services/cancelTaskRun.server.ts`, `services/rescheduleTaskRun.server.ts` - the `V1` arm rejects or finalizes gracefully instead of executing.
-`services/initializeDeployment.server.ts` - the `DEPRECATE_V3_CLI_DEPLOYS_ENABLED`-gated v3 CLI deploy rejection.
-`handleWebsockets.server.ts` - the legacy `trigger dev` websocket closes with the upgrade message.
When editing these shared services, only modify V2 code paths.
- **Background jobs**: `@trigger.dev/redis-worker` (`commonWorker.server.ts`, `alertsWorker.server.ts`, `batchTriggerWorker.server.ts`; `legacyRunEngineWorker.server.ts` still hosts the live batch-completion jobs)
- **Queue operations**: RunQueue inside run-engine (`runQueue.server.ts`), not MarQS
- If the PR also touches `packages/`, just the changeset is sufficient (no `.server-changes/` needed).
The body ships **verbatim in user-facing release notes**. Keep it to 1–2 short sentences, non-technical, written for a dashboard user: describe what changed for them, never the implementation (no header names, endpoints, middleware, storage mechanisms, internal tools). See `.server-changes/README.md` for full guidance.
description: Use this skill when writing or modifying Drizzle ORM schemas, queries, or migrations in this repo — specifically the `@internal/dashboard-agent-db` package (the dashboard agent's conversation datastore). Covers pg-core schema definition, the postgres-js driver, drizzle-kit migrations, and this repo's conventions: a dedicated Postgres schema, foreign-key-free cross-database design, pooler-safe connections, and the access-pattern query layer. Drizzle is NOT the main database — that's Prisma.
Drizzle is used in exactly one place: **`internal-packages/dashboard-agent-db`** (`@internal/dashboard-agent-db`), the in-dashboard agent's conversation store. Everything else in the monorepo is **Prisma** (`@trigger.dev/database`). Keep them separate.
Pinned versions: **`drizzle-orm` ^0.45**, **`drizzle-kit` ^0.31** (dev), **`postgres` ^3.4** (postgres.js driver). drizzle-orm and drizzle-kit are intentionally on different version lines — 0.31.x is the correct companion for 0.45.x, there is no peer dependency between them.
## Critical rules
1.**Drizzle is only the agent's own datastore.** The agent (and its task bundle) must have **no access to the main Prisma database or ClickHouse**. Never import the Prisma client into the agent task or into `@internal/dashboard-agent-db`. Main data is reached via the API, not Drizzle.
2.**Foreign-key-free.** In cloud this DB is a *separate* PlanetScale database, so it can't FK into the main DB. Reference main entities (`organizationId`, `userId`, …) **by id only — never `.references()`**. Joins happen in app code; tenant scoping is enforced in the query layer.
3.**One dedicated Postgres schema.** All tables live under `pgSchema("trigger_dashboard_agent")` so they're schema-qualified and isolated from Prisma's `public` schema (this is what makes the OSS single-database fallback safe).
4.**Pooler-safe connections.** Connections go through a transaction-mode pooler (PlanetScale / PgBouncer-style), so postgres.js must run with **`prepare: false`** — prepared statements don't survive a connection being handed to another client between checkouts.
5.**Node16 module resolution.** Relative imports need explicit **`.js`** extensions (`import { chats } from "./schema.js"`), even though the source is `.ts`.
6.**Scope every user query.** All queries that touch user data go through `src/queries.ts` and are scoped by `organizationId` / `userId`, so callers can't forget the `where`. Don't write ad-hoc cross-tenant queries elsewhere.
## Package layout
```text
internal-packages/dashboard-agent-db/
drizzle.config.ts # drizzle-kit config (schema path, out dir, schemaFilter)
`package.json` points `main`/`types` at `./src/index.ts` (consumed as source, no build step) — same as other simple internal packages.
## Schema (pg-core)
Use `pgSchema(...).table(...)`, not the bare `pgTable`, so tables land in the dedicated schema. ([schemas](https://orm.drizzle.team/docs/schemas), [pg column types](https://orm.drizzle.team/docs/column-types/pg), [indexes](https://orm.drizzle.team/docs/indexes-constraints))
// Inferred row types for the query layer + consumers.
exporttypeChat=typeofchats.$inferSelect;
exporttypeNewChat=typeofchats.$inferInsert;
```
Notes:
-`timestamp(..., { withTimezone: true })` → `timestamp with time zone`. Use `.defaultNow()` for `DEFAULT now()`.
- For a "newest first, nulls last" sort the partial index uses `.desc()`; the *query* uses raw `sql` for `NULLS LAST` (see below).
- Don't add `.references()` — see critical rule 2.
## Client (postgres.js + drizzle)
([connect overview](https://orm.drizzle.team/docs/connect-overview)) One small pool, `prepare: false`. In the agent task create it once in `onBoot` (per-process); in the webapp wrap it in the `singleton(...)` helper.
`drizzle.config.ts` must set **`schemaFilter`** so drizzle-kit only ever manages our schema — never Prisma's `public` (critical in the OSS single-DB fallback):
pnpm run db:generate # diff schema.ts → emit SQL into drizzle/. OFFLINE (no DB needed).
# review the generated drizzle/000N_*.sql before committing
pnpm run db:migrate # apply pending migrations. Needs a real DATABASE URL.
```
- `db:generate` is **offline** — it only reads `schema.ts`, so you can verify a schema change compiles to valid DDL with no database. Use it as a fast check.
- drizzle-kit names migration files with a **random suffix** (`0000_magenta_lilandra.sql`). Don't regenerate a committed migration just to "refresh" it — that churns the filename. After the first migration is committed, schema changes produce a **new** `000N_*.sql`; commit that.
- Generated DDL for a new schema is one `CREATE SCHEMA` + schema-qualified `CREATE TABLE`s + indexes, **no foreign keys** (by design here).
## Common gotchas
- **`prepare: false`** is not optional with a pooler — without it you'll get prepared-statement errors under load.
- **Missing `.js` extension** on a relative import → TS2835 under Node16 resolution.
- **Extra-config callback returns an array** `(t) => [ ... ]` in drizzle-orm 0.36+. The old object form `(t) => ({ ... })` is deprecated.
- **`NULLS LAST` / `NULLS FIRST`** aren't on the `desc()` helper — use raw `sql\`col desc nulls last\`` in `orderBy`.
- **Don't `SELECT *` into list views** — explicitly pick columns so you never ship a megabyte `messages` blob or a session token to a list query.
- **Adding a dependency**: edit `package.json`, then `pnpm i` from the repo root (never `pnpm add`). Mind the repo's `minimumReleaseAge` (3 days) — pin with a caret range and let pnpm resolve an old-enough version.
description: End-to-end smoke test for the public Errors HTTP API (error groups). Seeds failed runs into ClickHouse so the error materialized views populate, then drives the real endpoints against the running webapp — list (with filters + pagination), retrieve, resolve/ignore/unresolve, the `filter[error]` runs filter, user attribution via the `trigger.dev mint-token` -> JWT exchange, and the 401/403/404 negatives. Use for "smoke test the errors API", "test the errors API e2e", "prove the errors endpoints work", or to re-verify after changes.
allowed-tools: Read, Bash
---
# Errors API — end-to-end smoke test
Proves the public Errors API against the **running** webapp with real HTTP. No
mocks. The error data plane is ClickHouse (`errors_v1` + `error_occurrences_v1`,
both materialized-view-fed from `task_runs_v2`) plus Postgres `ErrorGroupState`
for lifecycle status; this skill seeds straight into `task_runs_v2` and lets the
- Attribution: `api.v1.projects.$projectRef.$env.jwt.ts` stamps `act:{sub}` for PAT **and** UAT exchanges; `@trigger.dev/rbac` surfaces `act.sub` through bearer auth; the action handlers read `authentication.actor?.sub`.
`errorId` is `error_<fingerprint>` (round-trips via `ErrorId` in `@trigger.dev/core/v3/isomorphic`).
## Prerequisites
- Webapp running on http://localhost:3030 (`pnpm run dev --filter webapp`). Confirm `curl -s http://localhost:3030/healthcheck`.
- DB seeded (`pnpm run db:seed`), and a local ClickHouse reachable at `CLICKHOUSE_URL` (the `pnpm run docker` stack).
- The CLI built + logged in to localhost:3030 (`pnpm run build --filter trigger.dev`; profile `default` points at localhost:3030). Needed only for the attribution leg.
> Important wiring facts the seed relies on (verified):
> - The MVs read the error type/message from `error.data.*`, so the seeded
> - The MVs only fire for failed statuses: `SYSTEM_FAILURE | CRASHED | INTERRUPTED | COMPLETED_WITH_ERRORS | TIMED_OUT`, and require a non-empty `error_fingerprint`.
> - `GET /api/v1/runs` lists run **ids** from ClickHouse but **hydrates from Postgres** `TaskRun`. So the error-list/detail/action legs work from a ClickHouse-only seed, but the `filter[error]` leg needs a **paired** Postgres `TaskRun` row whose `id` equals the ClickHouse `run_id`.
Run everything from the repo root in one shell. Invoke the built CLI via a
function (a `CLI="node …"` variable won't word-split under zsh):
Q=$(python3 -c "import urllib.parse;print(urllib.parse.quote('INSERT INTO trigger_dev.task_runs_v2 FORMAT JSONEachRow'))")
mkrow(){# status fingerprint errorType message runId
echo"{\"environment_id\":\"$ENV\",\"organization_id\":\"$ORG\",\"project_id\":\"$PROJ\",\"run_id\":\"$5\",\"friendly_id\":\"run_$5\",\"status\":\"$1\",\"environment_type\":\"DEVELOPMENT\",\"engine\":\"V2\",\"task_identifier\":\"$TASK\",\"created_at\":\"$NOW_CH\",\"updated_at\":\"$NOW_CH\",\"error\":{\"data\":{\"type\":\"$3\",\"message\":\"$4\",\"stack\":\"at x (a.ts:1:1)\"}},\"error_fingerprint\":\"$2\",\"task_version\":\"20240101.1\",\"_version\":\"$NOW_MS\",\"_is_deleted\":0}"
# Poll until both fingerprints appear in errors_v1 (the MV is near-instant locally).
for i in $(seq 1 10);do
N=$(curl -s "$CHURL" --data-binary "SELECT count() FROM (SELECT 1 FROM trigger_dev.errors_v1 WHERE environment_id='$ENV' AND error_fingerprint IN ('$FP_A','$FP_B') GROUP BY error_fingerprint)")
["$N"="2"]&& break; sleep 1
done
echo"seeded fingerprints in errors_v1: $N (want 2)"
curl -s "$B/api/v1/errors?filter%5BtaskIdentifier%5D=$TASK&filter%5Bsearch%5D=AlphaBoom&filter%5Bperiod%5D=1d" -H "$H"| python3 -c "import sys,json;print('search:',[e['errorType'] for e in json.load(sys.stdin)['data']])"# ['AlphaBoom']
curl -s "$B/api/v1/errors/$ERRID_B" -H "$H"| st # status ignored | ignoredUntil True | reason known flake
curl -s -X POST "$B/api/v1/errors/$ERRID_A/unresolve" -H "$H" >/dev/null
curl -s "$B/api/v1/errors/$ERRID_A" -H "$H"| st # status unresolved
```
PASS: each transition reflected; `filter[status]=ignored` returns only beta:
```bash
curl -s "$B/api/v1/errors?filter%5BtaskIdentifier%5D=$TASK&filter%5Bstatus%5D=ignored&filter%5Bperiod%5D=1d" -H "$H"| python3 -c "import sys,json;print([e['id'] for e in json.load(sys.stdin)['data']])"# [error_<fpB>]
```
### 5. `filter[error]` on the runs list (paired PG + CH seed)
The runs list hydrates from Postgres, so seed a matching `TaskRun` row + a CH row
curl -s "$B/api/v1/runs?filter%5Berror%5D=error_$FP_R" -H "$H"| python3 -c "import sys,json;d=json.load(sys.stdin);print('runs:',[r['id'] for r in d['data']])"
```
PASS: one run, `run_<RID>` (status maps to `FAILED`). Proves `filter[error]` -> fingerprint -> CH -> PG hydration.
### 6. Attribution — `mint-token` -> JWT exchange records the acting user
PASS: `act.sub` is the user id (matches `cli whoami`), and `detail.resolvedBy` equals that user id (not null). A plain env key leaves it null (step 4). A **PAT** exchanged the same way also stamps `act` — repeat with the stored PAT to confirm `ignoredByUserId` attribution.
unresolve flip status (and `filter[status]` follows); step 5's `filter[error]`
returns the paired run; step 6 records `resolvedBy` = the acting user via the
JWT exchange (null with a plain env key); and step 7 returns 404/401/401/403/200.
A red leg is a bug or a missing prereq — report the exact status + body and file
a Linear issue, don't tune around it.
## Notes / gotchas
- Run files use a unique `$RUN` suffix per invocation, so reruns don't collide and seeded rows stay isolated by their unique task identifier. They are local-dev test rows (90-day ClickHouse TTL); no cleanup required.
- After **adding** the route files, the classic Remix dev compiler may not register them until a dev-server restart (a stale manifest returns Remix's HTML 404 on the new paths). If `POST …/resolve` returns a 404 HTML page rather than 401/200, restart `pnpm run dev --filter webapp`.
- The rbac `act` extraction lives in `@trigger.dev/rbac` (a built dep). After editing it, `pnpm run build --filter @trigger.dev/rbac` and restart the webapp so the attribution leg (step 6) reflects the change.
ENCRYPTION_KEY=ae13021afef0819c3a307ad487071c06 # Must be a random 16 byte hex string. You can generate an encryption key by running `openssl rand -hex 16` in your terminal
MANAGED_WORKER_SECRET=abcdef1234 # Must match the supervisor's MANAGED_WORKER_SECRET
uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2 # zizmor:ignore[artipacked] changesets/action pushes the release branch; no artifact upload here so no leak path
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0 # zizmor:ignore[artipacked] changesets/action pushes the release branch; no artifact upload here so no leak path
You are reviewing a PR to check whether any CLAUDE.md files or .claude/rules/ files need updating.
You are reviewing a PR to check whether any agent instruction files need updating.
In this repo:
- Root shared agent guidance lives in `AGENTS.md`.
- Root `CLAUDE.md` is only a Claude Code adapter that imports `AGENTS.md`.
- Subdirectories may still have scoped `CLAUDE.md` files.
- `.claude/rules/` contains additional Claude Code guidance.
## Your task
1. Run `git diff origin/main...HEAD --name-only` to see which files changed in this PR.
2. For each changed directory, check if there's a CLAUDE.md in that directory or a parent directory.
3. Determine if any CLAUDE.md or .claude/rules/ file should be updated based on the changes. Consider:
2. For each changed directory, check the applicable instruction files: root `AGENTS.md`, any `CLAUDE.md` in that directory or a parent directory, and relevant `.claude/rules/` files.
3. Determine if any instruction file should be updated based on the changes. Consider:
- New files/directories that aren't covered by existing documentation
- Changed architecture or patterns that contradict current CLAUDE.md guidance
- New dependencies, services, or infrastructure that Claude should know about
- Renamed or moved files that are referenced in CLAUDE.md
- Changed architecture or patterns that contradict current agent guidance
- New dependencies, services, or infrastructure that agents should know about
- Renamed or moved files that are referenced in an instruction file
- Changes to build commands, test patterns, or development workflows
## Response format
If NO updates are needed, respond with exactly:
✅ CLAUDE.md files look current for this PR.
✅ Agent instruction files look current for this PR.
If updates ARE needed, respond with a short list:
📝 **CLAUDE.md updates suggested:**
📝 **Agent instruction updates suggested:**
- `AGENTS.md`: [what should be added/changed]
- `path/to/CLAUDE.md`: [what should be added/changed]
- `.claude/rules/file.md`: [what should be added/changed]
Keep suggestions specific and brief. Only flag things that would actually mislead Claude in future sessions.
Keep suggestions specific and brief. Only flag things that would actually mislead agents in future sessions.
Do NOT suggest updates for trivial changes (bug fixes, small refactors within existing patterns).
Do NOT suggest creating new CLAUDE.md files - only updates to existing ones.
Do NOT suggest creating new CLAUDE.md files - only updates to existing instruction files.
uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2 # zizmor:ignore[artipacked] needs persisted git creds for tag push; no artifact upload here so no leak path
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0 # zizmor:ignore[artipacked] needs persisted git creds for tag push; no artifact upload here so no leak path
This directory tracks changes to server-only components (webapp, supervisor, coordinator, etc.) that are not captured by changesets. Changesets only track published npm packages — server changes would otherwise go undocumented.
This directory tracks changes to server-only components (webapp, supervisor, etc.) that are not captured by changesets. Changesets only track published npm packages — server changes would otherwise go undocumented.
## When to add a file
**Server-only PRs**: If your PR only changes `apps/webapp/`, `apps/supervisor/`,`apps/coordinator/`, or other server components (and does NOT change anything in `packages/`), add a `.server-changes/` file.
**Server-only PRs**: If your PR only changes `apps/webapp/`, `apps/supervisor/`, or other server components (and does NOT change anything in `packages/`), add a `.server-changes/` file.
**Mixed PRs** (both packages and server): Just add a changeset as usual. No `.server-changes/` file needed — the changeset covers it.
@@ -31,13 +31,27 @@ Speed up batch queue processing by removing stalls and fixing retry race
The body text (below the frontmatter) is a one-line description of the change. Keep it concise — it will appear in release notes.
### Writing guidance
These entries are public-facing - they ship verbatim in user-visible release notes. A few rules to keep them clean:
- **Write for the user, not the reviewer.** Lead with what the user notices or has to do. If a reader who doesn't know the codebase can't tell what changed for them, rewrite it.
- **One sentence is usually enough.** The body is the bullet in the changelog. If you need a paragraph, you're probably describing the implementation rather than the change.
- **Describe behavior, not implementation.** Skip internal scopes, middleware names, library specifics, framework internals. Users care about what's different for them, not how it's wired.
- **Never name internal tools or infra.** Observability stacks, internal services, infra components, monitoring backends, CI surfaces, AWS specifics - none of these belong in user-facing notes.
Before / after:
- ❌ _"The image verification step now parses the manifest's layer media types and returns a new result the finalizer rejects."_ (describes the wiring; a user can't act on it)
- ✅ _"Deploying with an outdated CLI could produce an image that fails to start on every run. These deploys are now stopped before going live, with a message asking you to upgrade the CLI and re-deploy."_ (what the user sees and does)
## Lifecycle
1. Engineer adds a `.server-changes/` file in their PR
List API endpoints now clamp the page size to a maximum of 100. Requests asking for a larger page size return up to 100 items and keep paginating, rather than pulling an unbounded page.
Organizations without billing alerts now get default spend alert thresholds, so you're notified before usage grows unexpectedly. The billing limit page no longer pre-selects an option before you've set a limit and prompts you to configure one. Alert previews now update immediately after you change your billing limit.
Added `/engine/v1/dev/disconnect` endpoint to auto-cancel runs when the CLI disconnects. Maximum of 500 runs can be cancelled. Uses the bulk action system when there are more than 25 runs to cancel.
When you create a Personal Access Token, the generated token now shows its first and last few characters instead of being fully hidden, so you can confirm you copied the right value.
Realtime run subscriptions can now be configured to read run data straight from the primary database, so a run's latest state is never served from a lagging replica. Off by default; replica reads are unchanged unless you turn it on.
SSO and Directory Sync are no longer restricted to Enterprise plans — get in touch and we can turn them on for your organization whatever plan you're on.
The tasks page no longer runs two queries whose results were never displayed, cutting wasted work on every page load and removing a source of hidden server errors
The runs list on a task's page now updates live — run statuses change and newly triggered runs appear without a manual refresh, matching the main Runs page.
This repository is a pnpm monorepo managed with Turbo. It contains multiple apps and packages that make up the Trigger.dev platform and SDK.
This file provides guidance to Claude Code when working with this repository. Subdirectory CLAUDE.md files provide deeper context when you navigate into specific areas.
## Repository layout
-`apps/webapp`– Remix application that serves as the main API and dashboard.
-`apps/supervisor`– Node application for executing built tasks.
-`packages/*`– Published packages such as `@trigger.dev/sdk`, the CLI (`trigger.dev`), and shared libraries.
-`internal-packages/*`– Internal-only packages used by the webapp and other apps.
-`references/*`– Example projects for manual testing and development of new features.
-`ai/references`– Contains additional documentation including an overview (`repo.md`) and testing guidelines (`tests.md`).
## Build and Development Commands
See `ai/references/repo.md` for a more complete explanation of the workspaces.
This is a pnpm 10.33.2 monorepo using Turborepo. Run commands from root with `pnpm run`.
## Development setup
1. Install dependencies with `pnpm i` (pnpm `10.33.2` and Node.js `20.20.0` are required).
2. Copy `.env.example` to `.env` and generate a random 16 byte hex string for `ENCRYPTION_KEY` (`openssl rand -hex 16`). Update other secrets if needed.
3. Start the local services with Docker:
```bash
pnpm run docker
```
Add `:full` (`pnpm run docker:full`) for the optional observability + chaos tooling. See `docker/docker-compose.extras.yml`.
4. Run database migrations:
```bash
pnpm run db:migrate
```
5. Build the webapp, CLI and SDK packages:
```bash
pnpm run build --filter webapp && pnpm run build --filter trigger.dev && pnpm run build --filter @trigger.dev/sdk
```
6. Launch the development server:
```bash
pnpm run dev --filter webapp
```
The webapp runs on <http://localhost:3030>.
**Adding dependencies:** Edit `package.json` directly instead of using `pnpm add`, then run `pnpm i` from the repo root. See `.claude/rules/package-installation.md` for the full process.
For full setup instructions see `CONTRIBUTING.md`.
```bash
pnpm run docker # Core dev services (Postgres, Redis, Electric, MinIO, ClickHouse, s2-lite)
# pnpm run docker:full # Same + observability stack (Prometheus, Grafana, OTEL) and chaos tooling
pnpm run db:migrate # Run database migrations
pnpm run db:seed # Seed the database (required for reference projects)
## Running tests
- Unit tests use **vitest**. Run all tests:
```bash
pnpm run test
```
- Run tests for a specific workspace (example for `webapp`):
```bash
pnpm run test --filter webapp
```
- Prefer running a single test file from within its directory:
```bash
cd apps/webapp
pnpm run test ./src/components/Button.test.ts
```
If packages in that workspace need to be built first, run `pnpm run build --filter webapp`.
# Build packages (required before running)
pnpm run build --filter webapp && pnpm run build --filter trigger.dev && pnpm run build --filter @trigger.dev/sdk
Refer to `ai/references/tests.md` for details on writing tests. Tests should avoid mocks or stubs and use the helpers from `@internal/testcontainers` when Redis or Postgres are needed.
pnpm run dev --filter webapp # Run webapp (http://localhost:3030)
pnpm run dev --filter trigger.dev --filter "@trigger.dev/*"# Watch CLI and packages
```
## Coding style
- Formatting is enforced using Prettier. Run `pnpm run format` before committing.
- Follow the existing project conventions. Test files live beside the files under test and use descriptive `describe` and `it` blocks.
- Do not commit directly to the `main` branch. All changes should be made in a separate branch and go through a pull request.
### Verifying Changes
## Additional docs
- The root `README.md` describes Trigger.dev and links to documentation.
- The `docs` workspace contains our documentation site, which can be run locally with:
```bash
pnpm run dev --filter docs
```
- `references/README.md` explains how to create new reference projects for manual testing.
The verification command depends on where the change lives:
- **Apps and internal packages** (`apps/*`, `internal-packages/*`): Use `typecheck`. **Never use `build`** for these — building proves almost nothing about correctness.
- **Public packages** (`packages/*`): Use `build`.
```bash
# Apps and internal packages — use typecheck
pnpm run typecheck --filter webapp # ~1-2 minutes
pnpm run typecheck --filter @internal/run-engine
# Public packages — use build
pnpm run build --filter @trigger.dev/sdk
pnpm run build --filter @trigger.dev/core
```
Only run typecheck/build after major changes (new files, significant refactors, schema changes). For small edits, trust the types and let CI catch issues.
## Testing
We use vitest exclusively. **Never mock anything** - use testcontainers instead.
```bash
pnpm run test --filter webapp # All tests for a package
cd internal-packages/run-engine
pnpm run test ./src/engine/tests/ttl.test.ts --run # Single test file
pnpm run build --filter @internal/run-engine # May need to build deps first
```
Test files go next to source files (e.g., `MyService.ts` -> `MyService.test.ts`).
redisTest("should use redis",async({redisOptions})=>{
/* ... */
});
postgresTest("should use postgres",async({prisma})=>{
/* ... */
});
containerTest("should use both",async({prisma,redisOptions})=>{
/* ... */
});
```
## Code Style
### Formatting and linting
Format and lint are enforced by CI (`code-quality` check). Run before committing:
```bash
pnpm run format # oxfmt — auto-fixes formatting
pnpm run lint:fix # oxlint — auto-fixes lint violations
pnpm run lint # oxlint — check only (no fixes)
```
### Imports
**Prefer static imports over dynamic imports.** Only use dynamic `import()` when:
- Circular dependencies cannot be resolved otherwise
- Code splitting is genuinely needed for performance
- The module must be loaded conditionally at runtime
Dynamic imports add unnecessary overhead in hot paths and make code harder to analyze. If you find yourself using `await import()`, ask if a regular `import` statement would work instead.
## Changesets and Server Changes
When modifying any public package (`packages/*` or `integrations/*`), add a changeset:
```bash
pnpm run changeset:add
```
- Default to **patch** for bug fixes and minor changes
- Confirm with maintainers before selecting **minor** (new features)
- **Never** select major without explicit approval
When modifying only server components (`apps/webapp/`, `apps/supervisor/`, etc.) with no package changes, add a `.server-changes/` file instead. See `.server-changes/README.md` for format and documentation.
**Write the description for users, not maintainers.** Both changesets and `.server-changes/` notes ship verbatim in user-visible release notes. Lead with what changed *for the user* - one plain sentence describing behavior, not implementation, and never naming internal tools or infra. The full writing guidance in `.server-changes/README.md` applies to changesets too.
## Dependency Pinning
Zod is pinned to a single version across the entire monorepo (currently `3.25.76`). When adding zod to a new or existing package, use the **exact same version** as the rest of the repo - never a different version or a range. Mismatched zod versions cause runtime type incompatibilities (e.g., schemas from one package can't be used as body validators in another).
## Architecture Overview
### Request Flow
User API call -> Webapp routes -> Services -> RunEngine -> Redis Queue -> Supervisor -> Container execution -> Results back through RunEngine -> ClickHouse (analytics) + PostgreSQL (state)
v3 (engine V1: MarQS + Graphile worker) is end-of-life and its execution code has been removed. The `apps/webapp/app/v3/` directory name is historical - everything there now serves V2 (Run Engine 2.0, `@internal/run-engine` + redis-worker). There is no V1 execution path: a `RunEngineVersion``V1` branch only rejects or finalizes gracefully so v3 clients get a clean 4xx, never a 5xx. Do not reintroduce V1. See `apps/webapp/CLAUDE.md` and `.claude/rules/legacy-v3-code.md`.
### Documentation
Docs live in `docs/` as a Mintlify site (MDX format). See `docs/CLAUDE.md` for conventions.
### Reference Projects
Reference/example projects for testing SDK and platform features live in a separate repo: [`triggerdotdev/references`](https://github.com/triggerdotdev/references). Clone it alongside this repo and use its `projects/hello-world` to manually test changes before submitting PRs. See that repo's README for setup and linking to a local monorepo build.
## Docker Image Guidelines
When updating Docker image references:
- **Always use multiplatform/index digests**, not architecture-specific digests
- Architecture-specific digests cause CI failures on different build environments
- Use the digest from the main Docker Hub page, not from a specific OS/ARCH variant
## Writing Trigger.dev Tasks
Always import from `@trigger.dev/sdk`. Never use `@trigger.dev/sdk/v3` or deprecated `client.defineJob`.
```typescript
import{task}from"@trigger.dev/sdk";
exportconstmyTask=task({
id:"my-task",
run: async(payload:{message: string})=>{
// Task logic
},
});
```
### SDK Documentation Rules
The `rules/` directory contains versioned SDK documentation distributed via the SDK installer. Current version: `rules/manifest.json`. Do NOT update `rules/` or `.claude/skills/trigger-dev-tasks/` unless explicitly asked - these are maintained in separate dedicated passes.
## Testing with the hello-world Reference Project
The reference projects live in the separate [`triggerdotdev/references`](https://github.com/triggerdotdev/references) repo - clone it alongside this repo.
First-time setup:
1.`pnpm run db:seed` to seed the database (creates the References org + hello-world project)
2. Build the CLI/packages you want to test: `pnpm run build --filter trigger.dev`
3. In your `references` clone, follow its README to link to your local monorepo build, then authorize: `cd projects/hello-world && pnpm exec trigger login -a http://localhost:3030`
The preferred way to query for crumbs is to use `pnpm exec agentcrumbs query --app trigger` with the `--limit` option and cursor pagination, and clear existing crumbs before reproducing a bug via `pnpm exec agentcrumbs clear --app trigger`.
This file provides guidance to Claude Code when working with this repository. Subdirectory CLAUDE.md files provide deeper context when you navigate into specific areas.
## Build and Development Commands
This is a pnpm 10.33.2 monorepo using Turborepo. Run commands from root with `pnpm run`.
**Adding dependencies:** Edit `package.json` directly instead of using `pnpm add`, then run `pnpm i` from the repo root. See `.claude/rules/package-installation.md` for the full process.
```bash
pnpm run docker # Core dev services (Postgres, Redis, Electric, MinIO, ClickHouse, s2-lite)
# pnpm run docker:full # Same + observability stack (Prometheus, Grafana, OTEL) and chaos tooling
pnpm run db:migrate # Run database migrations
pnpm run db:seed # Seed the database (required for reference projects)
# Build packages (required before running)
pnpm run build --filter webapp && pnpm run build --filter trigger.dev && pnpm run build --filter @trigger.dev/sdk
pnpm run dev --filter webapp # Run webapp (http://localhost:3030)
pnpm run dev --filter trigger.dev --filter "@trigger.dev/*"# Watch CLI and packages
```
### Verifying Changes
The verification command depends on where the change lives:
- **Apps and internal packages** (`apps/*`, `internal-packages/*`): Use `typecheck`. **Never use `build`** for these — building proves almost nothing about correctness.
- **Public packages** (`packages/*`): Use `build`.
```bash
# Apps and internal packages — use typecheck
pnpm run typecheck --filter webapp # ~1-2 minutes
pnpm run typecheck --filter @internal/run-engine
# Public packages — use build
pnpm run build --filter @trigger.dev/sdk
pnpm run build --filter @trigger.dev/core
```
Only run typecheck/build after major changes (new files, significant refactors, schema changes). For small edits, trust the types and let CI catch issues.
## Testing
We use vitest exclusively. **Never mock anything** - use testcontainers instead.
```bash
pnpm run test --filter webapp # All tests for a package
cd internal-packages/run-engine
pnpm run test ./src/engine/tests/ttl.test.ts --run # Single test file
pnpm run build --filter @internal/run-engine # May need to build deps first
```
Test files go next to source files (e.g., `MyService.ts` -> `MyService.test.ts`).
redisTest("should use redis",async({redisOptions})=>{
/* ... */
});
postgresTest("should use postgres",async({prisma})=>{
/* ... */
});
containerTest("should use both",async({prisma,redisOptions})=>{
/* ... */
});
```
## Changesets and Server Changes
When modifying any public package (`packages/*` or `integrations/*`), add a changeset:
```bash
pnpm run changeset:add
```
- Default to **patch** for bug fixes and minor changes
- Confirm with maintainers before selecting **minor** (new features)
- **Never** select major without explicit approval
When modifying only server components (`apps/webapp/`, `apps/supervisor/`, etc.) with no package changes, add a `.server-changes/` file instead. See `.server-changes/README.md` for format and documentation.
## Dependency Pinning
Zod is pinned to a single version across the entire monorepo (currently `3.25.76`). When adding zod to a new or existing package, use the **exact same version** as the rest of the repo - never a different version or a range. Mismatched zod versions cause runtime type incompatibilities (e.g., schemas from one package can't be used as body validators in another).
## Architecture Overview
### Request Flow
User API call -> Webapp routes -> Services -> RunEngine -> Redis Queue -> Supervisor -> Container execution -> Results back through RunEngine -> ClickHouse (analytics) + PostgreSQL (state)
- **internal-packages/zodworker**: Graphile-worker wrapper (DEPRECATED - use redis-worker)
### Legacy V1 Engine Code
The `apps/webapp/app/v3/` directory name is misleading - most code there is actively used by V2. Only specific files are V1-only legacy (MarQS queue, triggerTaskV1, cancelTaskRunV1, etc.). See `apps/webapp/CLAUDE.md` for the exact list. When you encounter V1/V2 branching in services, only modify V2 code paths. All new work uses Run Engine 2.0 (`@internal/run-engine`) and redis-worker.
### Documentation
Docs live in `docs/` as a Mintlify site (MDX format). See `docs/CLAUDE.md` for conventions.
### Reference Projects
The `references/` directory contains test workspaces for testing SDK and platform features. Use `references/hello-world` to manually test changes before submitting PRs.
## Docker Image Guidelines
When updating Docker image references:
- **Always use multiplatform/index digests**, not architecture-specific digests
- Architecture-specific digests cause CI failures on different build environments
- Use the digest from the main Docker Hub page, not from a specific OS/ARCH variant
## Writing Trigger.dev Tasks
Always import from `@trigger.dev/sdk`. Never use `@trigger.dev/sdk/v3` or deprecated `client.defineJob`.
```typescript
import{task}from"@trigger.dev/sdk";
exportconstmyTask=task({
id:"my-task",
run: async(payload:{message: string})=>{
// Task logic
},
});
```
### SDK Documentation Rules
The `rules/` directory contains versioned SDK documentation distributed via the SDK installer. Current version: `rules/manifest.json`. Do NOT update `rules/` or `.claude/skills/trigger-dev-tasks/` unless explicitly asked - these are maintained in separate dedicated passes.
The preferred way to query for crumbs is to use `pnpm exec agentcrumbs query --app trigger` with the `--limit` option and cursor pagination, and clear existing crumbs before reproducing a bug via `pnpm exec agentcrumbs clear --app trigger`.
@@ -49,7 +49,7 @@ branch are tagged into a release periodically.
```
cd trigger.dev
```
3. Ensure you are on the correct version of Node.js (20.20.0). If you are using `nvm`, there is an `.nvmrc` file that will automatically select the correct version of Node.js when you navigate to the repository.
3. Ensure you are on the correct version of Node.js (24.18.0). If you are using `nvm`, there is an `.nvmrc` file that will automatically select the correct version of Node.js when you navigate to the repository.
4. Run `corepack enable` to use the correct version of pnpm (`10.33.2`) as specified in the root `package.json` file.
@@ -107,24 +107,32 @@ branch are tagged into a release periodically.
## Manual testing using hello-world
We use the `<root>/references/hello-world` subdirectory as a staging ground for testing changes to the SDK (`@trigger.dev/sdk` at `<root>/packages/trigger-sdk`), the Core package (`@trigger.dev/core` at `<root>packages/core`), the CLI (`trigger.dev` at `<root>/packages/cli-v3`) and the platform (The remix app at `<root>/apps/webapp`). The instructions below will get you started on using the `hello-world` for local development of Trigger.dev.
The `hello-world` reference project (and the others) live in a separate repo:
[`triggerdotdev/references`](https://github.com/triggerdotdev/references). Clone it
alongside this repo. It's the staging ground for testing changes to the SDK
(`@trigger.dev/sdk` at `<root>/packages/trigger-sdk`), the Core package
(`@trigger.dev/core` at `<root>/packages/core`), the CLI (`trigger.dev` at
`<root>/packages/cli-v3`) and the platform (the Remix app at `<root>/apps/webapp`).
To exercise your local monorepo changes, the reference project links to your local
build — see the references repo's README for the `pnpm run link` flow.
> Paths below such as `projects/hello-world` are relative to your `references`
> clone, not this repo.
### First-time setup
First, make sure you are running the webapp according to the instructions above. The seed step from setup already created a `hello-world` project under the `References` org with the stable ref `proj_rrkpdguyagvsoktglnod` — log in at http://localhost:3030 with any email to access it. Then:
1. Build the CLI (skip if you already ran the build step in setup)
1. Build the CLI and packages (skip if you already ran the build step in setup)
```sh
pnpm run build --filter trigger.dev
# Make it accessible to `pnpm exec`
pnpm i
pnpm run build --filter trigger.dev --filter "@trigger.dev/*"
```
2. Change into the `<root>/references/hello-world` directory and authorize the CLI to the local server:
2. In your `references` clone, link to your local monorepo build (see its README), then change into `projects/hello-world` and authorize the CLI to the local server:
```sh
cd references/hello-world
cd projects/hello-world
cp .env.example .env
pnpm exec trigger login -a http://localhost:3030
```
@@ -134,7 +142,7 @@ This will open a new browser window and authorize the CLI against your local use
You can optionally pass a `--profile` flag to the `login` command, which will allow you to use the CLI with separate accounts/servers. We suggest using a profile called `local` for your local development:
```sh
cd references/hello-world
cd projects/hello-world
pnpm exec trigger login -a http://localhost:3030 --profile local
# later when you run the dev or deploy command:
pnpm exec trigger dev --profile local
@@ -147,35 +155,35 @@ The following steps should be followed any time you start working on a new featu
1. Make sure the webapp is running on localhost:3030
2. Open a terminal window and build the CLI and packages and watch for changes
2. In this repo, open a terminal window and build the CLI and packages and watch for changes (the reference project links against this build)
```sh
pnpm run dev --filter trigger.dev --filter "@trigger.dev/*"
```
3. Open another terminal window, and change into the `<root>/references/hello-world` directory.
3. Open another terminal window, and change into `projects/hello-world` in your `references` clone.
4. Run the `dev` command, which will register all the local tasks with the platform and allow you to start testing task execution:
```sh
# in <root>/references/hello-world
# in <references-clone>/projects/hello-world
pnpm exec trigger dev
```
If you want additional debug logging, you can use the `--log-level debug` flag:
```sh
# in <root>/references/hello-world
# in <references-clone>/projects/hello-world
pnpm exec trigger dev --log-level debug
```
5. If you make any changes in the CLI/Core/SDK, you'll need to `CTRL+C` to exit the `dev` command and restart it to pickup changes. Any changes to the files inside of the `hello-world/src/trigger` dir will automatically be rebuilt by the `dev` command.
5. If you make any changes in the CLI/Core/SDK, you'll need to `CTRL+C` to exit the `dev` command and restart it to pickup changes. Any changes to the files inside the reference project's `src/trigger` dir will automatically be rebuilt by the `dev` command.
6. Navigate to the `hello-world` project in your local dashboard at localhost:3030 and you should see the list of tasks.
7. Go to the "Test" page in the sidebar and select a task. Then enter a payload and click "Run test". You can tell what the payloads should be by looking at the relevant task file inside the `/references/hello-world/src/trigger` folder. Many of them accept an empty payload.
7. On the Tasks page, open a task and press the "Test" button to open its test page. Then enter a payload and click "Run test". You can tell what the payloads should be by looking at the relevant task file inside the reference project's `src/trigger` folder. Many of them accept an empty payload.
8. Feel free to add additional files in `hello-world/src/trigger` to test out specific aspects of the system, or add in edge cases.
8. Feel free to add additional files in the reference project's `src/trigger` dir to test out specific aspects of the system, or add in edge cases.
## Adding and running migrations
@@ -215,9 +223,15 @@ pnpm exec trigger dev --log-level debug
### PR workflow
1. **Always open your PR in draft status first.** Do not mark it as "Ready for Review" until the steps below are complete.
2. **Address all CodeRabbit code review comments.** Our CI runs an automated code review via CodeRabbit. Go through each comment and either fix the issue or resolve it with a comment explaining why no change is needed.
3. **Wait for all CI checks to pass.** Do not mark the PR as "Ready for Review" until every check is green.
4. **Then mark the PR as "Ready for Review"** so a maintainer can take a look.
2. **Run format and lint locally before pushing:**
```bash
pnpm run format # auto-fixes formatting (oxfmt)
pnpm run lint:fix # auto-fixes lint violations (oxlint)
```
Both are enforced by CI — the `code-quality` check will fail if either produces a diff or errors.
3. **Address all CodeRabbit code review comments.** Our CI runs an automated code review via CodeRabbit. Go through each comment and either fix the issue or resolve it with a comment explaining why no change is needed.
4. **Wait for all CI checks to pass.** Do not mark the PR as "Ready for Review" until every check is green.
5. **Then mark the PR as "Ready for Review"** so a maintainer can take a look.
### Cost/benefit analysis for risky changes
@@ -251,7 +265,7 @@ Most of the time the changes you'll make are likely to be categorized as patch r
## Adding server changes
Changesets only track published npm packages. If your PR only changes server components (`apps/webapp/`, `apps/supervisor/`, `apps/coordinator/`, etc.) with no package changes, add a `.server-changes/` file so the change appears in release notes.
Changesets only track published npm packages. If your PR only changes server components (`apps/webapp/`, `apps/supervisor/`, etc.) with no package changes, add a `.server-changes/` file so the change appears in release notes.
The body text (below the frontmatter) is a one-line description of the change. Keep it concise — it will appear in release notes.
Some files were not shown because too many files have changed in this diff
Show More
Reference in New Issue
Block a user
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
Chris Arderne
