## Summary
Redriving a run out of the dead letter queue published to the hardcoded
Redis channel `rq:redrive`, while the subscriber listens on
`${options.name}:redrive`. For any `RunQueue` not named `rq` the publish
reached zero subscribers, Redis reported success, and the run stayed in
the dead letter queue with no log, error, or metric.
Both sides now derive the channel from the same expression, and
`redriveMessage` logs an error when a redrive publish reaches zero
subscribers instead of failing silently.
The existing "Dead Letter Queue" test now constructs its queue as
`rq-redrive`, so it fails against the old code (verified red before the
fix, green after) and stops the channel names from silently re-locking
to a single magic value.
Fixes#4854
## Summary
A package loaded with `createRequire(import.meta.url)("pkg")` is
invisible to esbuild: the call is never resolved, so the package is
neither bundled nor collected as an external to install in the deployed
image. The deploy succeeds with zero diagnostics and the task fails at
runtime with a module-not-found error, which can surface as something
far more confusing when a library maps errors coarsely (a database
driver loaded this way can look exactly like a connection failure). It
also works fine in `trigger dev` because the local `node_modules`
exists, making the production-only failure extra misleading.
Both `deploy` and `dev` builds now warn about this, pointing at the
exact file and line, with a note showing the exact config that fixes it:
```
▲ [WARNING] "mssql" is loaded with createRequire() but won't be available in the deployed image, so loading it will fail at runtime. The bundler can't follow createRequire() calls, so "mssql" is neither bundled into your code nor installed in the image. [plugin create-require-collector]
src/db.ts:12:14:
12 │ const mssql = createRequire(import.meta.url)("mssql");
╵ ^
To fix this, install "mssql" into the image by adding the additionalPackages build extension to your trigger.config.ts:
import { additionalPackages } from "@trigger.dev/build/extensions/core";
export default defineConfig({
// ...
build: {
extensions: [additionalPackages({ packages: ["mssql"] })],
},
});
Alternatively, replace the createRequire() call with a static import so the package is bundled. Docs: https://trigger.dev/docs/config/extensions/additionalPackages
```
In `dev` the message instead explains that the code works locally but
deploys of it will fail, so the problem is caught while the code is
being written rather than after a deploy.
## How it works
An esbuild plugin scans the bundle's input files outside `node_modules`
for string-literal specifiers passed to `createRequire`-created require
functions: `createRequire(...)("pkg")`, `const req = createRequire(...);
req("pkg")`, `req.resolve("pkg")`, aliased imports, namespace access,
CJS destructuring, and dynamic `import("node:module")` bindings. Sources
are parsed with `@babel/parser` (already in the dependency tree), so
comments, strings, templates, regex literals and JSX can't confuse the
scan; a file that fails to parse is skipped. Relative paths and node
builtins never warn.
A usage only warns when the package will actually be missing from the
image. On deploys the resolved manifest externals are the source of
truth (extension-installed layers are already merged in when the warning
runs); `build.external` alone deliberately does not suppress, because
marking a package external installs nothing when nothing statically
imports it. In dev, which predicts a future deploy, suppression
additionally trusts what extensions declare they install, and stays
silent entirely when that can't be determined (an extension hook throws,
or an older `@trigger.dev/build`'s additionalPackages predates the
declaration hook), so dev never makes a false "deploys will fail" claim.
`additionalPackages` declares its packages via a new diagnostics-only
`BuildExtension` field, `installedPackagesForTarget`, which the bundler
ignores: bundling output is unchanged for existing projects.
Detection is name-based, module-level, and deliberately per-file:
computed specifiers, shadowed names, and require helpers imported from
other files are not followed (those degrade to today's behavior, an
unwarned runtime failure), and scanning is scoped to user code because
bundled libraries legitimately use optional-require patterns that would
drown real findings in noise. Packages named in build-layer install
commands (`RUN npm install ...`) are suppressed individually.
Deploys also now surface esbuild's own bundle warnings for user files
(for example `require()` with a non-literal argument), which were
previously discarded on the deploy path; `trigger dev` already showed
them.
## Summary
A run's finish commit and its follow-up side effects (completing the
associated waitpoint, waking blocked parents, releasing the queue slot,
nudging batch completion) are separate writes across Postgres and Redis.
If a database error landed between them, the child run was already
finished, so the runner's retries hit the "Run is already finished"
guard and the completion signal was lost for good. A parent blocked on
`triggerAndWait` or `batchTriggerAndWait` then stayed waiting forever.
TTL expiry had the same shape: its worker retry returned early on a
non-pending run, and the batch expiry path swallowed a failed
waitpoint-job enqueue.
## Fix
Every finalizing path (attempt success, permanent failure, cancellation,
TTL expiry) now enqueues a durable `ensureRunFinalized` job before the
finish commit, and acks it once the inline side effects all succeed. In
steady state the guard never executes; the cost is one Redis enqueue and
ack per completion.
When the inline path dies in between, the guard fires after a short
delay and re-derives everything from current state: it releases the
run's queue message and concurrency slot, completes a still-pending
associated waitpoint from the run row's output or error, re-runs the
blocked-run fan-out (covering a lost unblock enqueue even after the
waitpoint committed), and re-schedules the batch completion check. Every
leg is idempotent, so racing the inline path is a no-op. The job retries
with a capped backoff for roughly five weeks before dead-lettering, so
it outlives any database outage while a genuinely poisoned item still
becomes visible.
Cancellation gets special handling: CANCELED is the only terminal run
status where execution can still be in flight, so the guard only
re-delivers for a canceled run once its execution snapshot is FINISHED,
re-arming itself until then rather than resuming the parent while the
child is still winding down.
A `finalization_rederivations` counter increments whenever the guard
actually re-delivers a lost signal; it should stay at zero in a healthy
system.
Tests cover six shapes: waitpoint completion lost after the finish
commit, unblock fan-out lost after the waitpoint completed, a failed
guard enqueue failing the completion request with nothing committed, a
stale guard held back during an in-flight cancellation, waitpoint
completion lost during TTL expiry, and the happy path where the guard is
acked and never runs.
Known accepted edge: a guard re-run after a partial inline completion
can re-emit a cached-run completion event for the same span; this only
happens during failure recovery and is bounded to duplicate trace
events.
## Summary
3 improvements.
## Improvements
- Stop shipping compiled test files in the published packages. The
`*.test.ts` sources were being emitted into `dist`, adding dead weight
to every install and leaving modules that `require("vitest")` (not a
dependency) inside the tarball, which tripped tooling that walks every
file in a package.
([#4833](https://github.com/triggerdotdev/trigger.dev/pull/4833))
- Named side channels on a Session: durable, two-way realtime streams
that outlive a single run and are shared across runs. Open a channel
with `sessions.open(id).channel(name)` (or `chat.channel(name)` inside a
`chat.agent`) to get an `.in`/`.out` pair addressed by name rather than
the reserved default pair. Writing a side channel's `.in` does not wake
or trigger a run, so a channel can carry out-of-band data (a stream of
frames, a control signal) that many clients read while the agent
produces it.
([#4815](https://github.com/triggerdotdev/trigger.dev/pull/4815))
```ts
// Inside a chat.agent: stream frames on a named channel, wakes nothing
const frames = chat.channel("screenshots");
await frames.out.append(frame);
frames.in.on((control) => {
/* client control, no suspend */
});
```
Declare channel record types once with `sessions.defineChannel(...)` and
infer them on both the producer and the consumer, including
`useSessionStreamChannel` in React. Channels get a default retention
that keeps them bounded, overridable per channel.
- Session `triggerConfig.tags` now accepts up to 10 tags, matching the
run tag limit. Previously it was capped at 5, which for `chat.agent`
left room for only 4 of your own tags after the automatic
`chat:{chatId}` tag.
([#4832](https://github.com/triggerdotdev/trigger.dev/pull/4832))
<details>
<summary>Raw changeset output</summary>
# Releases
## @trigger.dev/build@4.5.15
### Patch Changes
- Stop shipping compiled test files in the published packages. The
`*.test.ts` sources were being emitted into `dist`, adding dead weight
to every install and leaving modules that `require("vitest")` (not a
dependency) inside the tarball, which tripped tooling that walks every
file in a package.
([#4833](https://github.com/triggerdotdev/trigger.dev/pull/4833))
- Updated dependencies:
- `@trigger.dev/core@4.5.15`
## trigger.dev@4.5.15
### Patch Changes
- Updated dependencies:
- `@trigger.dev/core@4.5.15`
- `@trigger.dev/build@4.5.15`
- `@trigger.dev/schema-to-json@4.5.15`
## @trigger.dev/core@4.5.15
### Patch Changes
- Stop shipping compiled test files in the published packages. The
`*.test.ts` sources were being emitted into `dist`, adding dead weight
to every install and leaving modules that `require("vitest")` (not a
dependency) inside the tarball, which tripped tooling that walks every
file in a package.
([#4833](https://github.com/triggerdotdev/trigger.dev/pull/4833))
- Named side channels on a Session: durable, two-way realtime streams
that outlive a single run and are shared across runs. Open a channel
with `sessions.open(id).channel(name)` (or `chat.channel(name)` inside a
`chat.agent`) to get an `.in`/`.out` pair addressed by name rather than
the reserved default pair. Writing a side channel's `.in` does not wake
or trigger a run, so a channel can carry out-of-band data (a stream of
frames, a control signal) that many clients read while the agent
produces it.
([#4815](https://github.com/triggerdotdev/trigger.dev/pull/4815))
```ts
// Inside a chat.agent: stream frames on a named channel, wakes nothing
const frames = chat.channel("screenshots");
await frames.out.append(frame);
frames.in.on((control) => {
/* client control, no suspend */
});
```
Declare channel record types once with `sessions.defineChannel(...)` and
infer them on both the producer and the consumer, including
`useSessionStreamChannel` in React. Channels get a default retention
that keeps them bounded, overridable per channel.
- Session `triggerConfig.tags` now accepts up to 10 tags, matching the
run tag limit. Previously it was capped at 5, which for `chat.agent`
left room for only 4 of your own tags after the automatic
`chat:{chatId}` tag.
([#4832](https://github.com/triggerdotdev/trigger.dev/pull/4832))
## @trigger.dev/python@4.5.15
### Patch Changes
- Updated dependencies:
- `@trigger.dev/sdk@4.5.15`
- `@trigger.dev/core@4.5.15`
- `@trigger.dev/build@4.5.15`
## @trigger.dev/react-hooks@4.5.15
### Patch Changes
- Named side channels on a Session: durable, two-way realtime streams
that outlive a single run and are shared across runs. Open a channel
with `sessions.open(id).channel(name)` (or `chat.channel(name)` inside a
`chat.agent`) to get an `.in`/`.out` pair addressed by name rather than
the reserved default pair. Writing a side channel's `.in` does not wake
or trigger a run, so a channel can carry out-of-band data (a stream of
frames, a control signal) that many clients read while the agent
produces it.
([#4815](https://github.com/triggerdotdev/trigger.dev/pull/4815))
```ts
// Inside a chat.agent: stream frames on a named channel, wakes nothing
const frames = chat.channel("screenshots");
await frames.out.append(frame);
frames.in.on((control) => {
/* client control, no suspend */
});
```
Declare channel record types once with `sessions.defineChannel(...)` and
infer them on both the producer and the consumer, including
`useSessionStreamChannel` in React. Channels get a default retention
that keeps them bounded, overridable per channel.
- Updated dependencies:
- `@trigger.dev/core@4.5.15`
## @trigger.dev/redis-worker@4.5.15
### Patch Changes
- Updated dependencies:
- `@trigger.dev/core@4.5.15`
## @trigger.dev/rsc@4.5.15
### Patch Changes
- Updated dependencies:
- `@trigger.dev/core@4.5.15`
## @trigger.dev/schema-to-json@4.5.15
### Patch Changes
- Updated dependencies:
- `@trigger.dev/core@4.5.15`
## @trigger.dev/sdk@4.5.15
### Patch Changes
- Stop shipping compiled test files in the published packages. The
`*.test.ts` sources were being emitted into `dist`, adding dead weight
to every install and leaving modules that `require("vitest")` (not a
dependency) inside the tarball, which tripped tooling that walks every
file in a package.
([#4833](https://github.com/triggerdotdev/trigger.dev/pull/4833))
- Named side channels on a Session: durable, two-way realtime streams
that outlive a single run and are shared across runs. Open a channel
with `sessions.open(id).channel(name)` (or `chat.channel(name)` inside a
`chat.agent`) to get an `.in`/`.out` pair addressed by name rather than
the reserved default pair. Writing a side channel's `.in` does not wake
or trigger a run, so a channel can carry out-of-band data (a stream of
frames, a control signal) that many clients read while the agent
produces it.
([#4815](https://github.com/triggerdotdev/trigger.dev/pull/4815))
```ts
// Inside a chat.agent: stream frames on a named channel, wakes nothing
const frames = chat.channel("screenshots");
await frames.out.append(frame);
frames.in.on((control) => {
/* client control, no suspend */
});
```
Declare channel record types once with `sessions.defineChannel(...)` and
infer them on both the producer and the consumer, including
`useSessionStreamChannel` in React. Channels get a default retention
that keeps them bounded, overridable per channel.
- Session `triggerConfig.tags` now accepts up to 10 tags, matching the
run tag limit. Previously it was capped at 5, which for `chat.agent`
left room for only 4 of your own tags after the automatic
`chat:{chatId}` tag.
([#4832](https://github.com/triggerdotdev/trigger.dev/pull/4832))
- Updated dependencies:
- `@trigger.dev/core@4.5.15`
</details>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Project → Settings → Integrations, build settings for GitHub deploys:
- **Trigger config file**: description now says it is auto-detected by
default and a path only overrides that.
- **Use native build server**: rendered only for admins — GitHub
deployments always use the native build server unless an admin opts a
project out. The option to disable native builds in GitHub deployments
will be removed entirely.
## Summary
`SessionTriggerConfig.tags` was capped at 5, while runs (and the [tags
docs](https://trigger.dev/docs/tags)) allow 10. Session trigger tags are
forwarded verbatim as the run tags on every run a session schedules, so
the lower cap was an inconsistency rather than a separate limit. For
`chat.agent` it was worse in practice: the SDK prepends `chat:{chatId}`
automatically and truncates, so users could only get 4 of their own tags
through.
The schema, the SDK truncation points, and the dashboard playground now
all use 10. `chat.agent` users get 9 of their own tags plus the
automatic `chat:{chatId}` tag. Docs updated to say so.
## Summary
Fixes [#4825](https://github.com/triggerdotdev/trigger.dev/issues/4825).
The published `@trigger.dev/sdk`, `@trigger.dev/core` and
`@trigger.dev/build` tarballs included every `*.test.ts` file compiled
into `dist`, plus their `.d.ts` and source maps. Those modules
`require("vitest")`, which is not a dependency of any of the packages,
so the tarballs contained modules that cannot resolve. That is dead
weight on every install, and it trips tooling that walks or bundles
every file in a package.
## Fix
tshy supports an `exclude` list in its `package.json` config that
applies to every dialect build, so each affected package now sets:
```json
"tshy": {
"exclude": ["src/**/*.test.ts"]
}
```
Only `*.test.ts` files are excluded. The public test-helper entry points
(`@trigger.dev/sdk/ai/test`, `@trigger.dev/core/v3/test`) live in
`src/v3/test/` and are still built and exported. The CLI package already
had an equivalent exclude. Type checking and vitest are unaffected
because they run off the package `tsconfig.json`, not the tshy build
config.
Verified with clean builds of all three packages: zero `*.test.*`
artifacts in `dist`, public entry points still present.
## Summary
Adds **named side channels** to a Session: durable, two-way realtime
streams that outlive a single run and are shared across every run of the
session. Today a Session has exactly one reserved `.in`/`.out` pair (the
chat transcript). This lets a session hold any number of *named*
channels alongside it, each its own `.in`/`.out` pair, so an agent can
stream out-of-band data (a feed of frames, telemetry, a control channel)
on a stream separate from the transcript while many clients read it
live.
The two properties a named channel adds over the reserved pair:
1. It is addressed by a name that outlives a run and is shared across
runs, not welded to the chat turn loop.
2. Writing its `.in` does **not** wake or trigger a run. A run observes
it by subscribing; an external client writes it without spawning
anything.
This is the generalization half of the Momentic ask (stream browser
screenshots from a `chat.agent` to the frontend on a channel separate
from the chat). It builds directly on the start-from-latest /
`useSessionStream` subscribe seam from #4811.
## Usage
Declare the channel's record types once and infer them on both sides:
```ts
// channels.ts (shared, client imports it type-only)
import { sessions } from "@trigger.dev/sdk";
export const screenshots = sessions.defineChannel<{ out: ScreenshotFrame; in: ViewportControl }>(
"screenshots"
);
```
Open a channel from a session handle (`sessions.open(id)` returns one
for a known session id). Writing its `.out` is durable, cross-run, and
wakes nothing; a run observes its `.in` by tailing, without suspending:
```ts
import { sessions } from "@trigger.dev/sdk";
import { screenshots } from "./channels";
const channel = sessions.open(sessionId).channel(screenshots);
await channel.out.append(frame); // frame: ScreenshotFrame (typed from the definition)
channel.in.on((control) => { /* ... */ }); // control: ViewportControl, tail, no suspend
```
Passing the definition types `.out.append` / `.in.on` on the producer
side; a bare name string also works, with records typed `unknown`.
An external client writes the `.in` without waking a run, and reads the
`.out` from React:
```ts
sessions.open(sessionId).channel("screenshots").in.send({ paused: true });
const { records } = useSessionStreamChannel<typeof screenshots>("screenshots", {
sessionId,
accessToken,
io: "out",
from: "latest",
maxRecords: 1,
});
```
`session.channel(name)` returns the same `{ in, out }` handle shape as
the reserved pair, so `append` / `pipe` / `writer` / `read` /
`writeControl` / `trimTo` on `.out` and `send` / `on` / `once` / `peek`
on `.in` all carry over. Passing a name other than the declared one is a
type error; a bare-string call without the generic stays valid with
`records` typed `unknown`.
### With `chat.agent`
This is the motivating case: a `chat.agent` answers on the reserved
transcript as usual, and streams screenshot frames on a side channel in
parallel. `chat.channel(name)` opens a channel on the current run's own
Session, so there's no id to thread:
```ts
import { chat } from "@trigger.dev/sdk/ai";
import { streamText } from "ai";
import { screenshots } from "./channels";
export const browserAgent = chat.agent({
id: "browser-agent",
run: async ({ messages, signal }) => {
const frames = chat.channel(screenshots);
// client pause/resume arrives here without waking a turn
frames.in.on((control: ViewportControl) => applyViewport(control));
// frames stream on their own channel, not the chat transcript
driveBrowser({ signal, onFrame: (frame) => frames.out.append(frame) });
// the assistant reply still goes to the reserved transcript
return streamText({ model: openai("gpt-4o"), messages, abortSignal: signal });
},
});
```
`chat.channel(name)` is a shortcut for `chat.session().channel(name)`;
`chat.session()` returns the current run's full `SessionHandle` if you
need it.
The frontend renders the transcript with `useChat` as before, and the
screenshots with `useSessionStreamChannel<typeof
screenshots>("screenshots", { sessionId: chatId, io: "out", from:
"latest", maxRecords: 1 })`: a live view of the newest frame that
survives across turns (each turn is a new run), because the channel is
keyed on the session, not the run.
### From MCP
An MCP client can observe and write a session's channels with two tools,
built on the same apiClient surface as the hook and the dashboard
viewer:
- `read_session_channel` reads records from a channel (or the reserved
pair). It is a point-in-time drain with cursor pagination
(`afterEventId` / `nextCursor`, `maxRecords`); pass `timeoutInSeconds`
to wait for the next record when none exist yet.
- `write_session_channel` appends one record to a channel's `.in` (an
object or a raw string), so an agent can send control input without
waking a run. `.out` is producer-only, so it is not writable here.
### On the session page
The session detail page lists a session's channels (via an S2 prefix
list in the loader) and shows each as a tab beside `Rendered` and `Raw`.
Selecting a channel renders its records in the same table as the Raw
transcript view, sourced from that channel's `out` and `in` streams.
## How it works
**Addressing.** A channel is a stream name segment:
`sessions/{id}/channels/{name}/{io}`. The reserved pair keeps its
two-part `sessions/{id}/{io}` name for back-compat, and the `channels/`
segment means a user channel named `in`/`out` can never collide with it.
The channel dimension is threaded through the session stream manager
(keyed on `(session, channel, io)`, reserved = absent),
`subscribeToSessionStream`, the session apiClient methods, and the
`realtime.v1.sessions.$session.channels.$channel.$io.{ts,append,records}`
routes. The reserved-pair routes are untouched. The start-from-latest
tail path from #4811 is channel-agnostic, so `from: "latest"` and
`maxRecords` compose unchanged.
**No-wake.** The reserved `.in` append route ensures a run and drains
waitpoints so a chat turn advances. The channel `.in` append route
deliberately does neither: the record lands durably and a run picks it
up when it next subscribes, so writing a side channel can't spawn or
resume a run. A named channel's `.in` is therefore subscribe-only from
the run side (`.on` / `.once` / `.peek`); `.wait()` /
`waitWithIdleTimeout()` throw with a message pointing at the observe
methods.
**Auth.** Channel scope folds into the existing resource id
(`sessions:<key>:channels:<channel>`), so no RBAC grammar change. A
channel route authorizes both the channel-folded id and the bare session
id, which means a session-wide token grants every channel while a
channel-scoped token grants only its own. The per-io rule is preserved
per channel: writing `.out` requires secret-key auth so a browser can't
forge frames; `.in` is writable with the session token.
**Retention.** Channel streams are created on demand on first write and
inherit the org's stream retention (bounded age plus delete-on-empty
from the store's default config), the same as the reserved chat streams.
There is no per-channel control-plane call on the write path. Custom
per-channel retention is deferred until the stream store can set config
inline on the on-demand create, which avoids a control-plane round trip.
**Spans.** Channel writes carry `channel` and `io` attributes, an
accessory chip, and the session icon. Clicking a channel span in the
run's span inspector renders the channel's actual records with the same
viewer the run realtime streams use, rather than the raw properties
JSON.
## Verification
- **Unit (core):** the stream manager isolates channels: two channels on
the same `(session, io)` never cross buffers, and a named channel is
isolated from the reserved pair.
- **Full-stack e2e** against a real stack (webapp, stream store,
Postgres, real runs):
- a named `.out` record is readable back **after the triggering run has
gone terminal** (durable, cross-run);
- a channel `.in` append creates **no** run, while a reserved `.in`
append **does** wake one (the differential is the red/green);
- `from: "latest"` on a named channel delivers the live record and does
**not** replay the backlog from the start;
- the span inspector renders a channel span's records, and the MCP
read/write tools round-trip records on a real session;
- an invalid channel name is rejected.
## Notes
- **Channel listing works on the self-hosted store too.** The stream
store's list operation is available on s2-lite, so the session page's
channel list is an OSS feature. It is a control-plane call made once per
session-page load (best-effort; a failure just hides the tabs), not on
the write path.
- **The ~1 MiB per-record cap is unchanged.** Large payloads (e.g. raw
screenshots) still need object-store pointers on the channel rather than
inline bytes; that's independent of this change.
- Docs ride this branch: the side channels guide, the
`useSessionStreamChannel` reference, and the MCP tools list are all
updated here.
## Screenshots
<img width="3444" height="1870" alt="CleanShot 2026-08-28 at 21 46
27@2x"
src="https://github.com/user-attachments/assets/c192aaee-b946-4824-87b7-ca057514d25e"
/>
## Summary
4 improvements, 1 bug fix.
## Improvements
- Native build server deploys now show a single updating build log line
by default; pass `--build-logs full` to stream every line (always used
in CI and when output is not a terminal).
([#4817](https://github.com/triggerdotdev/trigger.dev/pull/4817))
- Realtime stream subscriptions can now refresh an expired access token
and reconnect, via a new optional `refreshAccessToken` option on the
client configuration and the React hooks.
([#4811](https://github.com/triggerdotdev/trigger.dev/pull/4811))
- Subscribe to a realtime stream from its latest record instead of
replaying the whole history. Pass `from: "latest"` to
`useRealtimeStream`, `streams.read()`, or `fetchStream` to start at the
current tail (the latest record, then live updates) instead of replaying
(a live "last value" view), and `maxParts` to keep the accumulated
`parts` array bounded. A reconnect or remount resumes from the last
record it saw, so no records are missed and none are replayed. `from:
"latest"` needs a server that supports it; older servers safely fall
back to a full replay.
([#4811](https://github.com/triggerdotdev/trigger.dev/pull/4811))
`useRealtimeStream` also gains a `lastEventId` option and returns the
`lastEventId` of the last part seen, so you can persist the cursor (for
example across a page reload) and resume exactly where you left off. An
`onParts` callback delivers each throttled batch of parts with their
event ids.
```tsx
const { parts, lastEventId } = useRealtimeStream<Frame>(runId, "frames",
{
from: "latest", // skip history, start at the current tail
maxParts: 1, // keep only the most recent frame
lastEventId: savedCursor, // resume from a persisted cursor
onParts: (batch) => save(batch.at(-1)?.id), // track the cursor
accessToken,
});
```
- Added a `useSessionStream` React hook for reading a session's output
or input channel in realtime. It accumulates records with automatic
resume from the last record you received, and supports `from: "latest"`
(start at the current tail, only new records after you connect),
`maxRecords` (keep a bounded number of records in memory), a
`lastEventId` resume cursor, and an `onRecords` callback that delivers
each throttled batch of records with their event ids.
([#4811](https://github.com/triggerdotdev/trigger.dev/pull/4811))
## Server changes
These changes affect the self-hosted Docker image and Trigger.dev Cloud:
- Task retries that wait in the queue no longer count against the
queue's internal redelivery limit, so runs with many long-delay retries
are not wrongly failed with TASK_RUN_DEQUEUED_MAX_RETRIES.
([#4810](https://github.com/triggerdotdev/trigger.dev/pull/4810))
<details>
<summary>Raw changeset output</summary>
# Releases
## @trigger.dev/build@4.5.14
### Patch Changes
- Updated dependencies:
- `@trigger.dev/core@4.5.14`
## trigger.dev@4.5.14
### Patch Changes
- Native build server deploys now show a single updating build log line
by default; pass `--build-logs full` to stream every line (always used
in CI and when output is not a terminal).
([#4817](https://github.com/triggerdotdev/trigger.dev/pull/4817))
- Updated dependencies:
- `@trigger.dev/core@4.5.14`
- `@trigger.dev/build@4.5.14`
- `@trigger.dev/schema-to-json@4.5.14`
## @trigger.dev/core@4.5.14
### Patch Changes
- Realtime stream subscriptions can now refresh an expired access token
and reconnect, via a new optional `refreshAccessToken` option on the
client configuration and the React hooks.
([#4811](https://github.com/triggerdotdev/trigger.dev/pull/4811))
- Subscribe to a realtime stream from its latest record instead of
replaying the whole history. Pass `from: "latest"` to
`useRealtimeStream`, `streams.read()`, or `fetchStream` to start at the
current tail (the latest record, then live updates) instead of replaying
(a live "last value" view), and `maxParts` to keep the accumulated
`parts` array bounded. A reconnect or remount resumes from the last
record it saw, so no records are missed and none are replayed. `from:
"latest"` needs a server that supports it; older servers safely fall
back to a full replay.
([#4811](https://github.com/triggerdotdev/trigger.dev/pull/4811))
`useRealtimeStream` also gains a `lastEventId` option and returns the
`lastEventId` of the last part seen, so you can persist the cursor (for
example across a page reload) and resume exactly where you left off. An
`onParts` callback delivers each throttled batch of parts with their
event ids.
```tsx
const { parts, lastEventId } = useRealtimeStream<Frame>(runId, "frames",
{
from: "latest", // skip history, start at the current tail
maxParts: 1, // keep only the most recent frame
lastEventId: savedCursor, // resume from a persisted cursor
onParts: (batch) => save(batch.at(-1)?.id), // track the cursor
accessToken,
});
```
## @trigger.dev/python@4.5.14
### Patch Changes
- Updated dependencies:
- `@trigger.dev/core@4.5.14`
- `@trigger.dev/sdk@4.5.14`
- `@trigger.dev/build@4.5.14`
## @trigger.dev/react-hooks@4.5.14
### Patch Changes
- Realtime stream subscriptions can now refresh an expired access token
and reconnect, via a new optional `refreshAccessToken` option on the
client configuration and the React hooks.
([#4811](https://github.com/triggerdotdev/trigger.dev/pull/4811))
- Subscribe to a realtime stream from its latest record instead of
replaying the whole history. Pass `from: "latest"` to
`useRealtimeStream`, `streams.read()`, or `fetchStream` to start at the
current tail (the latest record, then live updates) instead of replaying
(a live "last value" view), and `maxParts` to keep the accumulated
`parts` array bounded. A reconnect or remount resumes from the last
record it saw, so no records are missed and none are replayed. `from:
"latest"` needs a server that supports it; older servers safely fall
back to a full replay.
([#4811](https://github.com/triggerdotdev/trigger.dev/pull/4811))
`useRealtimeStream` also gains a `lastEventId` option and returns the
`lastEventId` of the last part seen, so you can persist the cursor (for
example across a page reload) and resume exactly where you left off. An
`onParts` callback delivers each throttled batch of parts with their
event ids.
```tsx
const { parts, lastEventId } = useRealtimeStream<Frame>(runId, "frames",
{
from: "latest", // skip history, start at the current tail
maxParts: 1, // keep only the most recent frame
lastEventId: savedCursor, // resume from a persisted cursor
onParts: (batch) => save(batch.at(-1)?.id), // track the cursor
accessToken,
});
```
- Added a `useSessionStream` React hook for reading a session's output
or input channel in realtime. It accumulates records with automatic
resume from the last record you received, and supports `from: "latest"`
(start at the current tail, only new records after you connect),
`maxRecords` (keep a bounded number of records in memory), a
`lastEventId` resume cursor, and an `onRecords` callback that delivers
each throttled batch of records with their event ids.
([#4811](https://github.com/triggerdotdev/trigger.dev/pull/4811))
- Updated dependencies:
- `@trigger.dev/core@4.5.14`
## @trigger.dev/redis-worker@4.5.14
### Patch Changes
- Updated dependencies:
- `@trigger.dev/core@4.5.14`
## @trigger.dev/rsc@4.5.14
### Patch Changes
- Updated dependencies:
- `@trigger.dev/core@4.5.14`
## @trigger.dev/schema-to-json@4.5.14
### Patch Changes
- Updated dependencies:
- `@trigger.dev/core@4.5.14`
## @trigger.dev/sdk@4.5.14
### Patch Changes
- Subscribe to a realtime stream from its latest record instead of
replaying the whole history. Pass `from: "latest"` to
`useRealtimeStream`, `streams.read()`, or `fetchStream` to start at the
current tail (the latest record, then live updates) instead of replaying
(a live "last value" view), and `maxParts` to keep the accumulated
`parts` array bounded. A reconnect or remount resumes from the last
record it saw, so no records are missed and none are replayed. `from:
"latest"` needs a server that supports it; older servers safely fall
back to a full replay.
([#4811](https://github.com/triggerdotdev/trigger.dev/pull/4811))
`useRealtimeStream` also gains a `lastEventId` option and returns the
`lastEventId` of the last part seen, so you can persist the cursor (for
example across a page reload) and resume exactly where you left off. An
`onParts` callback delivers each throttled batch of parts with their
event ids.
```tsx
const { parts, lastEventId } = useRealtimeStream<Frame>(runId, "frames",
{
from: "latest", // skip history, start at the current tail
maxParts: 1, // keep only the most recent frame
lastEventId: savedCursor, // resume from a persisted cursor
onParts: (batch) => save(batch.at(-1)?.id), // track the cursor
accessToken,
});
```
- Updated dependencies:
- `@trigger.dev/core@4.5.14`
</details>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
## Summary
A run whose task retries were delayed long enough to go back through the
queue could end up failed with `TASK_RUN_DEQUEUED_MAX_RETRIES` and
status `SYSTEM_FAILURE` even though every attempt had actually executed.
The real failure from the final attempt was replaced by that placeholder
error, and tasks configured for more retries than the queue redelivery
limit never got them.
## Root cause
Retries with a delay at or above the warm-start threshold are requeued
via `tryNackAndRequeue`, which nacks the queue message. `nackMessage`
increments the message attempt counter by default and dead-letters the
message once it reaches the queue retry limit. That counter is meant to
bound dequeues that never reach execution; a task retry after a
completed attempt was being charged against it anyway, so a long-backoff
retry schedule exhausted it.
## Fix
`nackMessage` gains a `resetAttemptCount` option that zeroes the counter
instead of incrementing it. `tryNackAndRequeue` exposes it as
`resetQueueAttempts`, and the attempt-retry path passes it, since a
completed attempt proves the run can start. The dequeue-failure and
stalled `PENDING_EXECUTING` paths keep incrementing, as those are the
genuine "could not start" cases the budget exists for.
Tests cover the queue-level reset (no dead-letter at the limit) and an
engine-level run that retries past the queue limit and finishes with its
own error rather than a system failure.
## Summary
`workload_auth_gate_total` records how each worker action authorizes:
scoped by a
verified environment header, grandfathered by the created-at gate, or
suppressed
by it. It was registered on the Prometheus registry served at
`/metrics`, which is
per-process. With `ENABLE_CLUSTER=1` every Node worker keeps its own
registry, so a
scrape returns whichever process happened to answer and the counter
reads as a
fraction of real traffic.
This moves the counter onto the OpenTelemetry meter the webapp already
uses for its
other engine metrics. Each process exports under its own
`service.instance.id`, so
summing across them gives the true total no matter how many workers a
deployment
runs.
## Attributes
The counter now carries `env_type` and `run_age_bucket` alongside
`outcome` and
`action`.
`run_age_bucket` is the coarse age of the run behind an untokened worker
action
(`lt_1h`, `1h_1d`, `1d_7d`, `7d_30d`, `gt_30d`). It exists so an
operator can size
`WORKLOAD_TOKEN_CUTOFF` before committing to it: set the cutoff far in
the future
and every run is grandfathered, so the age distribution of untokened
traffic is
visible without anything being rejected. Both attributes come off the
run row the
gate already reads, so there is no extra query.
`publish.yml` builds the webapp and worker/supervisor images (and
dispatches the enterprise-image build off the webapp). It currently
triggers on all of `packages/**` and `internal-packages/**`, so a change
confined to a package that never lands in those images still publishes
new images.
This adds `paths` negations for packages that are not built into either
image:
- **packages** (npm-only libraries / CLI): `cli-v3`, `build`, `python`,
`react-hooks`, `rsc`, `schema-to-json`
- **internal-packages** (test/tooling only): `testcontainers`,
`sdk-compat-tests`, `observability-map`
UI-only update of the **Usage** page (`/orgs/…/settings/usage`). **No
logic or data changes**: the loader is byte-identical to `main` and the
usage-bar calculations are unchanged.
### What changed
- **Credits** and **Month-to-date** panels now sit in matching cards,
with the big `$value` and title on one baseline-aligned row and the
progress bar full-width beneath.
- Added a **Set / Update billing limit** link (to the existing Billing
limits page) on the Month-to-date panel.
- The two progress bars share the same height/corners; the Month-to-date
panel shrinks when there's no billing limit to show.
- Removed the progress-bar load animation.
- **Tasks**: moved the "dev environment runs are excluded…" note beside
the title and switched the empty state to the standard `TableBlankRow`.
<img width="3456" height="1364" alt="CleanShot 2026-08-28 at 15 37
52@2x"
src="https://github.com/user-attachments/assets/cb69ec33-1521-47d1-ba0a-51b8afd7eb00"
/>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Native build server deploys (`--native-build`) now show compact build
logs by default: one spinner line updated with the latest message, and
the last 20 lines printed when the build fails. The previous timestamped
line-by-line output is behind `--build-logs full`, and is used
automatically in CI, with `--plain`, or when stdout is not a TTY.
## Summary
Batches created on a run-ops store other than the two the list reads
were missing from the Batches page. No error, nothing logged: the page
just showed fewer batches than exist. This is only reachable once
additional run-ops stores are configured, so nothing changes for anyone
today.
## Fix
The list scanned exactly two databases and merged them by keyset. It now
covers one leg per configured store, in ascending precedence order, all
issued together.
The existing keyset merge generalises without change. Every leg runs the
same query, with the same cursor predicate, ordering and over-fetch, so
a row's rank within its own leg is never worse than its global rank, and
the merged first page is still the true first page. That argument holds
for any number of legs, not just two.
The empty-state check keeps its existing sequential pair, since a
project with no batches is the common case for that path, then issues
the remaining checks in a single round trip.
A store that declares itself an alias of another shares its client by
reference, so it contributes no leg. Scanning it would query the same
database twice for rows the other leg already returned. This matches how
the routing store and the boot checks treat an alias.
The fan-out deliberately fails the page if any store is unreachable,
rather than returning a short page. A tolerant merge would recreate the
same silent absence this change removes, with a wider blast radius.
## Verification
Covered by container tests against real databases: gen-1, legacy and
additional stores merged into one ordered page, paging forward and back
across a boundary that spans stores, and the empty-state check.
Also verified end to end against a live environment with a real corpus:
the missing rows reproduce with the new leg removed and appear correctly
with it present, ordering interleaves across stores as expected, paging
across a store boundary loses and repeats nothing, and the page is
byte-identical to before when no extra store is configured.
Merge precedence is pinned by its own test: one id seeded on two stores,
asserting the higher-authority copy is the one shown. Verified by
mutation, since a union-only test passes regardless of leg order.
## Boot interlocks
Two related boot checks changed alongside the read path, since
configuring an extra store is what makes them reachable.
A store configured while split reads are disabled is dropped in silence:
no client is built, no leg is added, and rows already resident there
disappear from every list with no error. The other two ways the split
ends up disabled already refuse to start; this closes the one that did
not, and names the stores it is refusing. A store that declares itself
an alias of another owns no database, so it is exempt.
The distinct-database probe fails closed, which meant one store being
briefly unreachable collapsed the deployment to single-DB and then
refused the boot entirely. Each target now gets a bounded number of
attempts with a short backoff before the probe gives up. Failing closed
is unchanged once that budget is exhausted, and a genuine duplicate is
still a final answer that is never retried.
Visual-only changes to the Tasks-page onboarding blank state (brand-new
project, dev environment). Formatting, lint, and knip pass via the
pre-push hooks; open the Tasks page for a new project to confirm the
panel, copy button, and step 2 render as intended.
---
## Changelog
Polished the "Set it up with your AI agent" onboarding panel:
top-aligned the badge and switched it to the custom Ask AI sparkle icon,
stopped the copy-prompt button from resizing when it swaps to "Copied
prompt" (the bright check icon now sits beside the label), removed the
sparkle from the button's idle state, removed the spinner next to "Start
the dev server", and widened the gap between the panel text and the copy
button.
---
## Screenshots
<img width="800" height="643" alt="CleanShot 2026-08-27 at 19 04 43"
src="https://github.com/user-attachments/assets/aede4ca1-30d5-4240-aa18-e1a20161973d"
/>
🤖 Generated with [Claude Code](https://claude.com/claude-code)
<!-- conductor-workspace-link -->
---
[Open workspace in
Conductor](https://app.conductor.build/workspace/d4a21ab5-d6fa-4de2-b5f0-4f34abf0b8b9)
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
## Summary
Realtime streams get a live "last value" mode: subscribe from the latest
record instead of replaying the whole history, keep memory bounded, and
resume across reloads. Plus a new `useSessionStream` hook for reading a
Session's channels from React.
## `useRealtimeStream`: start-from-latest, bounded, resumable
```tsx
const { parts, lastEventId } = useRealtimeStream<Frame>(runId, "frames", {
from: "latest", // skip history, only new records after connect
maxParts: 1, // keep just the most recent (bounded memory)
lastEventId: saved, // resume from a persisted cursor (survives reload)
onParts: (batch) => save(batch.at(-1)?.id), // per-batch event ids
accessToken,
});
```
`from`, `lastEventId` (option and return), and the batching also apply
to `streams.read()` and `fetchStream()`.
## `useSessionStream`: read a Session channel from React (new)
A read-only hook for a Session's `out` (default) or `in` channel, with
the same start / bound / resume options. `useSession` is reserved for
two-way (read and write).
```tsx
const { records, lastEventId } = useSessionStream<Frame>(sessionId, {
io: "out",
from: "latest",
maxRecords: 5,
onRecords: (batch) => {/* each throttled batch, with event ids */},
accessToken,
});
```
## Access-token refresh
Long-lived subscriptions can survive token expiry: pass
`refreshAccessToken` and a 401/403 triggers one re-mint and reconnect.
With no refresher, auth errors stay terminal exactly as before.
```tsx
const { parts } = useRealtimeStream<Frame>(runId, "frames", {
accessToken,
// called on a 401/403 to mint a fresh public token from your backend
refreshAccessToken: async () => {
const res = await fetch("/api/realtime-token");
return (await res.json()).token;
},
});
```
It is also available on `useApiClient` / `TriggerAuthContext`, so every
hook under a provider shares one refresher.
## Notes
Server support (S2 `tail_offset` / Redis `$`, and the start-position
header on the run and session SSE routes) ships here; a client passing
`from: "latest"` against an older server degrades safely to a full
replay. Resume, bounded memory, batched callbacks, and token refresh are
client-only.
Supersedes #4808 and #4809, folded in here. Verified end to end on an
isolated stack: `from: "latest"` on the run and session paths against
real S2, `lastEventId` resume across a reload, bounded memory, batched
callbacks, and a real 401 to token-refresh to reconnect.
## Summary
Several of the run-ops router's rules only apply above two stores, and
the fake-slot suites only ever built two, so those rules were untestable
by construction. `clearIdempotencyKey` is the sole caller of the "every
other shard" helper, and with two stores that helper returns a single
entry, which hides a take-the-first bug. The absent-id partition has the
same blind spot: a gen-2 id and a cuid select the same store when only
one other store exists.
Three suites now run at two shards and at three, with the expected value
indexed by topology wherever the rule genuinely changes. The fourth
stays at two and says why in the file, because its N-shard behaviour is
already pinned in `runOpsStore.shardMap.test.ts`.
Two webapp tests defined their own local `RoutingRunStore`. They
compiled against a two-store model whatever the real class did, and one
described a routing rule the code never implemented. Both now build the
real router over the two Postgres stores they already create.
## Validating a test-only change
Every new assertion passed the first time it ran, which proves nothing.
Each was checked by breaking the router in the way the test claims to
guard, then confirming the failure lands in the three-shard arm while
the two-shard arm still passes:
- take-the-first fan-out in the "every other shard" helper
- gen-2 keys moved to the front of the merge precedence order
- the absent-id partition sending every id to the gen-1 pair, which
fails as `expected +0 to be 1`, the shape a silently under-counted
waitpoint takes
- residency routing disabled entirely, caught by 3 of the 5 webapp tests
Each mutation was reverted. No production code changes.
One note for anyone extending these: the webapp resolves
`@internal/run-store` to `dist/`, not to source, so a source edit
without a rebuild makes those two tests assert against the previous
router and pass.
## Summary
4 new features, 12 improvements, 5 bug fixes.
## Improvements
- `trigger.dev deploy` now asks the server whether to build with Depot
or the native build server unless `--native-build`, `--depot-build`, or
`--local-build` is passed, so the native build server can be rolled out
per organization without a CLI change. `--local-bundle` and `--detach`
now require `--native-build`.
([#4803](https://github.com/triggerdotdev/trigger.dev/pull/4803))
- Add an experimental `--local-bundle` deploy flag that runs the install
and bundling steps on your machine and uploads only the build output;
the image is still built remotely. Useful when your project's install
step needs tooling or credentials that only exist locally.
([#4331](https://github.com/triggerdotdev/trigger.dev/pull/4331))
- Send the CLI version header on all API requests so deployments are
attributable to a CLI version
([#4778](https://github.com/triggerdotdev/trigger.dev/pull/4778))
- A message that arrives mid-turn and is not injected into that turn is
now answered as the next turn, instead of being dropped. This is what
the `pendingMessages` docs have always described, and it applies to the
default too: configuring `pendingMessages` without a `shouldInject`
declines every batch, which previously meant every mid-turn message was
lost with no error at either end.
([#4795](https://github.com/triggerdotdev/trigger.dev/pull/4795))
```ts
chat.agent({
id: "my-chat",
pendingMessages: {
onReceived: ({ message }) =>
logger.info("arrived mid-turn", { id: message.id }),
// Only interrupt once the agent has started calling tools.
shouldInject: ({ steps }) => steps.length > 0,
},
run: async ({ messages, signal }) =>
streamText({
model,
messages,
abortSignal: signal,
// Required for injection. Without it nothing injects, and every
// mid-turn message is answered as the next turn instead.
...chat.toStreamTextOptions(),
}),
});
```
A declined message keeps its place in the queue, so it survives a crash
and is answered by whichever run picks the conversation up. An injected
one is consumed at the moment it is injected, so it is never also
answered as a later turn.
- Browser chats now keep the active turn open across page reloads when
older completion records are replayed.
([#4643](https://github.com/triggerdotdev/trigger.dev/pull/4643))
- Add `chat.endAndContinue()` so fully hand-rolled custom chat agents
can hand a conversation off to a fresh run on the latest deployed task
version while preserving unconsumed Session input.
([#4647](https://github.com/triggerdotdev/trigger.dev/pull/4647))
- Custom chat agents now validate and parse client data declared with
`chat.withClientData({ schema })` before passing it to agent code.
([#4646](https://github.com/triggerdotdev/trigger.dev/pull/4646))
## Bug fixes
- Fixes a case where a chat could silently lose a message. If a message
arrived while the agent was between turns and a stop arrived after it,
the cursor the next boot resumed from could point past that message, so
it was never answered and no error was raised. This affected
`chat.agent`, not just custom agents.
([#4644](https://github.com/triggerdotdev/trigger.dev/pull/4644))
Fixes a recovered answer being cut off. After a crash the agent replays
the message it had not answered yet, but it was replaying the stop that
arrived after that message too, so the turn answering it was aborted the
moment it began. A stop is now only applied to the turn that was live
when it arrived. That holds however the stop got there: sent after the
last completed turn, or sent to a chat whose most recent turn was
completed by an older version of the SDK.
One limitation to know about: the recovered answer is persisted
correctly, but a chat page that stayed open across the crash keeps
showing the partial answer it had already received. Reload the page to
see the full recovered answer.
Also fixes a retried send being answered twice. When a send was retried
and its idempotency claim was lost, the agent could consume the same
message a second time.
Custom agent loops can now inspect pending chat input without consuming
it, and consume one record at a time, with `chat.messages.hasPending()`
and `chat.messages.next()`. Records carry stable identifiers so a
redelivery is recognisable.
```ts
if (await chat.messages.hasPending()) {
const record = await chat.messages.next({ timeoutInSeconds: 0 });
if (record) handle(record.payload);
}
```
`hasPending()` answers for messages alone, so a message sitting behind a
stop, or behind a record this version of the SDK does not recognise,
still reports as pending and is still delivered. Anything the agent has
no consumer for is discarded rather than left where it would make every
message queued behind it undeliverable. `chat.messages.next()` returning
`undefined` means no message became consumable before the timeout.
`chat.writeTurnComplete()`'s `sessionInEventId` is the cursor that is
safe to resume from, not the sequence of the record the turn answered.
It is held back behind any message still waiting to be handled, so a
value below the record you just handled is expected.
- Fixed a chat agent hanging after an interrupted turn: when a run was
killed mid-answer (out of memory, crash, or eviction) and only the one
message it was answering was still outstanding, the new run never
replied to it. That message is now re-answered on the new run.
([#4768](https://github.com/triggerdotdev/trigger.dev/pull/4768))
- Fix chat transport discarding the next turn after stopping generation.
`skipToTurnComplete` is now reset when a new message or action is sent,
so a message sent after `stopGeneration` streams normally instead of
leaving the chat stuck in a streaming state.
([#4744](https://github.com/triggerdotdev/trigger.dev/pull/4744))
- Fixes a message sent while the agent was mid-answer being lost if the
run then crashed. The cursor written at the end of each turn could point
past a message that had arrived during that turn but had not been
answered yet, so the next boot skipped it and no error was raised
anywhere. Such a message is now held until a turn actually takes it.
([#4795](https://github.com/triggerdotdev/trigger.dev/pull/4795))
This also removes the in-memory buffer those messages used to sit in, on
both `chat.agent` and `chat.createSession()`, so a message waiting for
its turn is durable rather than only present in the worker that received
it.
## Server changes
These changes affect the self-hosted Docker image and Trigger.dev Cloud:
- Self-hosted instances can now disable the admin dashboard and user
impersonation entirely. See the self-hosting docs for the new setting.
([#4774](https://github.com/triggerdotdev/trigger.dev/pull/4774))
- The dashboard has two new themes, Black and White, plus appearance
options for stronger colors and underlined links.
([#4547](https://github.com/triggerdotdev/trigger.dev/pull/4547))
- Deployment logs no longer jump to the bottom while you are reading
earlier output. Scroll up to pause auto-scroll, and scroll back down or
use the new scroll-to-bottom button in the log header to resume
following.
([#4776](https://github.com/triggerdotdev/trigger.dev/pull/4776))
- Customize the runs list: show, hide, and reorder columns, and add
smart columns that pull a value straight out of a run's payload,
metadata, or output. Your column choices are saved in the page URL, so
you can share a view, bookmark it, or save it straight to your
favorites.
([#4652](https://github.com/triggerdotdev/trigger.dev/pull/4652))
- Stop the browser offering to autofill or save environment variable
values as saved credentials.
([#4777](https://github.com/triggerdotdev/trigger.dev/pull/4777))
- Cut webapp CPU usage by about a quarter on the routes that workers
call most, freeing headroom at the same request rate. Detailed
event-loop blocking traces are no longer recorded by default, because
producing them was itself a large part of that cost.
([#4746](https://github.com/triggerdotdev/trigger.dev/pull/4746))
- When a runs list or runs.list API request spans too much data to
complete, it now returns a clear, actionable error asking you to narrow
the time range, instead of failing with a generic error.
([#4773](https://github.com/triggerdotdev/trigger.dev/pull/4773))
- Improved the performance and reliability of the runs list and the
runs.list API, especially for large projects and filtered views.
([#4763](https://github.com/triggerdotdev/trigger.dev/pull/4763))
- New Vercel connections now get version skew protection turned on
automatically, so each run uses the task version its deployment shipped
with. Automatic atomic deployments are deprecated and no longer offered
when you connect a project, but stay available in your Vercel
integration settings.
([#4741](https://github.com/triggerdotdev/trigger.dev/pull/4741))
- The Staging branch setting now shows an upgrade prompt on plans that
don't include a Staging environment, instead of looking editable and
then silently doing nothing when saved.
([#4784](https://github.com/triggerdotdev/trigger.dev/pull/4784))
<details>
<summary>Raw changeset output</summary>
# Releases
## @trigger.dev/build@4.5.13
### Patch Changes
- Updated dependencies:
- `@trigger.dev/core@4.5.13`
## trigger.dev@4.5.13
### Patch Changes
- `trigger.dev deploy` now asks the server whether to build with Depot
or the native build server unless `--native-build`, `--depot-build`, or
`--local-build` is passed, so the native build server can be rolled out
per organization without a CLI change. `--local-bundle` and `--detach`
now require `--native-build`.
([#4803](https://github.com/triggerdotdev/trigger.dev/pull/4803))
- Add an experimental `--local-bundle` deploy flag that runs the install
and bundling steps on your machine and uploads only the build output;
the image is still built remotely. Useful when your project's install
step needs tooling or credentials that only exist locally.
([#4331](https://github.com/triggerdotdev/trigger.dev/pull/4331))
- Send the CLI version header on all API requests so deployments are
attributable to a CLI version
([#4778](https://github.com/triggerdotdev/trigger.dev/pull/4778))
- Updated dependencies:
- `@trigger.dev/core@4.5.13`
- `@trigger.dev/build@4.5.13`
- `@trigger.dev/schema-to-json@4.5.13`
## @trigger.dev/core@4.5.13
### Patch Changes
- `trigger.dev deploy` now asks the server whether to build with Depot
or the native build server unless `--native-build`, `--depot-build`, or
`--local-build` is passed, so the native build server can be rolled out
per organization without a CLI change. `--local-bundle` and `--detach`
now require `--native-build`.
([#4803](https://github.com/triggerdotdev/trigger.dev/pull/4803))
- Add an experimental `--local-bundle` deploy flag that runs the install
and bundling steps on your machine and uploads only the build output;
the image is still built remotely. Useful when your project's install
step needs tooling or credentials that only exist locally.
([#4331](https://github.com/triggerdotdev/trigger.dev/pull/4331))
- A message that arrives mid-turn and is not injected into that turn is
now answered as the next turn, instead of being dropped. This is what
the `pendingMessages` docs have always described, and it applies to the
default too: configuring `pendingMessages` without a `shouldInject`
declines every batch, which previously meant every mid-turn message was
lost with no error at either end.
([#4795](https://github.com/triggerdotdev/trigger.dev/pull/4795))
```ts
chat.agent({
id: "my-chat",
pendingMessages: {
onReceived: ({ message }) =>
logger.info("arrived mid-turn", { id: message.id }),
// Only interrupt once the agent has started calling tools.
shouldInject: ({ steps }) => steps.length > 0,
},
run: async ({ messages, signal }) =>
streamText({
model,
messages,
abortSignal: signal,
// Required for injection. Without it nothing injects, and every
// mid-turn message is answered as the next turn instead.
...chat.toStreamTextOptions(),
}),
});
```
A declined message keeps its place in the queue, so it survives a crash
and is answered by whichever run picks the conversation up. An injected
one is consumed at the moment it is injected, so it is never also
answered as a later turn.
- Fixes a case where a chat could silently lose a message. If a message
arrived while the agent was between turns and a stop arrived after it,
the cursor the next boot resumed from could point past that message, so
it was never answered and no error was raised. This affected
`chat.agent`, not just custom agents.
([#4644](https://github.com/triggerdotdev/trigger.dev/pull/4644))
Fixes a recovered answer being cut off. After a crash the agent replays
the message it had not answered yet, but it was replaying the stop that
arrived after that message too, so the turn answering it was aborted the
moment it began. A stop is now only applied to the turn that was live
when it arrived. That holds however the stop got there: sent after the
last completed turn, or sent to a chat whose most recent turn was
completed by an older version of the SDK.
One limitation to know about: the recovered answer is persisted
correctly, but a chat page that stayed open across the crash keeps
showing the partial answer it had already received. Reload the page to
see the full recovered answer.
Also fixes a retried send being answered twice. When a send was retried
and its idempotency claim was lost, the agent could consume the same
message a second time.
Custom agent loops can now inspect pending chat input without consuming
it, and consume one record at a time, with `chat.messages.hasPending()`
and `chat.messages.next()`. Records carry stable identifiers so a
redelivery is recognisable.
```ts
if (await chat.messages.hasPending()) {
const record = await chat.messages.next({ timeoutInSeconds: 0 });
if (record) handle(record.payload);
}
```
`hasPending()` answers for messages alone, so a message sitting behind a
stop, or behind a record this version of the SDK does not recognise,
still reports as pending and is still delivered. Anything the agent has
no consumer for is discarded rather than left where it would make every
message queued behind it undeliverable. `chat.messages.next()` returning
`undefined` means no message became consumable before the timeout.
`chat.writeTurnComplete()`'s `sessionInEventId` is the cursor that is
safe to resume from, not the sequence of the record the turn answered.
It is held back behind any message still waiting to be handled, so a
value below the record you just handled is expected.
## @trigger.dev/python@4.5.13
### Patch Changes
- Updated dependencies:
- `@trigger.dev/sdk@4.5.13`
- `@trigger.dev/core@4.5.13`
- `@trigger.dev/build@4.5.13`
## @trigger.dev/react-hooks@4.5.13
### Patch Changes
- Updated dependencies:
- `@trigger.dev/core@4.5.13`
## @trigger.dev/redis-worker@4.5.13
### Patch Changes
- Updated dependencies:
- `@trigger.dev/core@4.5.13`
## @trigger.dev/rsc@4.5.13
### Patch Changes
- Updated dependencies:
- `@trigger.dev/core@4.5.13`
## @trigger.dev/schema-to-json@4.5.13
### Patch Changes
- Updated dependencies:
- `@trigger.dev/core@4.5.13`
## @trigger.dev/sdk@4.5.13
### Patch Changes
- Fixed a chat agent hanging after an interrupted turn: when a run was
killed mid-answer (out of memory, crash, or eviction) and only the one
message it was answering was still outstanding, the new run never
replied to it. That message is now re-answered on the new run.
([#4768](https://github.com/triggerdotdev/trigger.dev/pull/4768))
- Browser chats now keep the active turn open across page reloads when
older completion records are replayed.
([#4643](https://github.com/triggerdotdev/trigger.dev/pull/4643))
- Add `chat.endAndContinue()` so fully hand-rolled custom chat agents
can hand a conversation off to a fresh run on the latest deployed task
version while preserving unconsumed Session input.
([#4647](https://github.com/triggerdotdev/trigger.dev/pull/4647))
- Fix chat transport discarding the next turn after stopping generation.
`skipToTurnComplete` is now reset when a new message or action is sent,
so a message sent after `stopGeneration` streams normally instead of
leaving the chat stuck in a streaming state.
([#4744](https://github.com/triggerdotdev/trigger.dev/pull/4744))
- Custom chat agents now validate and parse client data declared with
`chat.withClientData({ schema })` before passing it to agent code.
([#4646](https://github.com/triggerdotdev/trigger.dev/pull/4646))
- Fixes a message sent while the agent was mid-answer being lost if the
run then crashed. The cursor written at the end of each turn could point
past a message that had arrived during that turn but had not been
answered yet, so the next boot skipped it and no error was raised
anywhere. Such a message is now held until a turn actually takes it.
([#4795](https://github.com/triggerdotdev/trigger.dev/pull/4795))
This also removes the in-memory buffer those messages used to sit in, on
both `chat.agent` and `chat.createSession()`, so a message waiting for
its turn is durable rather than only present in the worker that received
it.
- A message that arrives mid-turn and is not injected into that turn is
now answered as the next turn, instead of being dropped. This is what
the `pendingMessages` docs have always described, and it applies to the
default too: configuring `pendingMessages` without a `shouldInject`
declines every batch, which previously meant every mid-turn message was
lost with no error at either end.
([#4795](https://github.com/triggerdotdev/trigger.dev/pull/4795))
```ts
chat.agent({
id: "my-chat",
pendingMessages: {
onReceived: ({ message }) =>
logger.info("arrived mid-turn", { id: message.id }),
// Only interrupt once the agent has started calling tools.
shouldInject: ({ steps }) => steps.length > 0,
},
run: async ({ messages, signal }) =>
streamText({
model,
messages,
abortSignal: signal,
// Required for injection. Without it nothing injects, and every
// mid-turn message is answered as the next turn instead.
...chat.toStreamTextOptions(),
}),
});
```
A declined message keeps its place in the queue, so it survives a crash
and is answered by whichever run picks the conversation up. An injected
one is consumed at the moment it is injected, so it is never also
answered as a later turn.
- Fixes a case where a chat could silently lose a message. If a message
arrived while the agent was between turns and a stop arrived after it,
the cursor the next boot resumed from could point past that message, so
it was never answered and no error was raised. This affected
`chat.agent`, not just custom agents.
([#4644](https://github.com/triggerdotdev/trigger.dev/pull/4644))
Fixes a recovered answer being cut off. After a crash the agent replays
the message it had not answered yet, but it was replaying the stop that
arrived after that message too, so the turn answering it was aborted the
moment it began. A stop is now only applied to the turn that was live
when it arrived. That holds however the stop got there: sent after the
last completed turn, or sent to a chat whose most recent turn was
completed by an older version of the SDK.
One limitation to know about: the recovered answer is persisted
correctly, but a chat page that stayed open across the crash keeps
showing the partial answer it had already received. Reload the page to
see the full recovered answer.
Also fixes a retried send being answered twice. When a send was retried
and its idempotency claim was lost, the agent could consume the same
message a second time.
Custom agent loops can now inspect pending chat input without consuming
it, and consume one record at a time, with `chat.messages.hasPending()`
and `chat.messages.next()`. Records carry stable identifiers so a
redelivery is recognisable.
```ts
if (await chat.messages.hasPending()) {
const record = await chat.messages.next({ timeoutInSeconds: 0 });
if (record) handle(record.payload);
}
```
`hasPending()` answers for messages alone, so a message sitting behind a
stop, or behind a record this version of the SDK does not recognise,
still reports as pending and is still delivered. Anything the agent has
no consumer for is discarded rather than left where it would make every
message queued behind it undeliverable. `chat.messages.next()` returning
`undefined` means no message became consumable before the timeout.
`chat.writeTurnComplete()`'s `sessionInEventId` is the cursor that is
safe to resume from, not the sequence of the record the turn answered.
It is held back behind any message still waiting to be handled, so a
value below the record you just handled is expected.
- Updated dependencies:
- `@trigger.dev/core@4.5.13`
</details>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
The CLI now asks the server which build path to use before it builds or
uploads anything, so native builds can be rolled out per organization
and per environment type without a CLI release.
```
trigger.dev deploy
│
├─ explicit flag? (--native-build / --local-build / --depot-build)
│ └─ yes → use it, never ask the server
│
└─ GET /api/v1/projects/:ref/:env/deploy-settings (env API key, 5s timeout, one attempt)
│
│ server resolves: native unavailable → org[env type] → org → global[env type] → global → depot
│
├─ { "build_path": "native" | "native_local_bundle" } → that path
├─ { "build_path": "depot" } → Depot
└─ error / timeout / 404 → Depot (fail open)
```
The path comes from four enum feature flags, editable in the global and
per-org admin flag UIs: `deployBuildPath` and `deployBuildPathPreview` /
`Staging` / `Production`. Unset everywhere keeps current behaviour
unchanged; CLIs older than this release never call the endpoint and keep
their current behaviour.
## Summary
`chat.withClientData({ schema }).customAgent()` now parses
`payload.metadata` before passing it to `run`, `chat.messages`, or
`chat.createSession`. Schema defaults and transforms are preserved.
Custom agents without a schema keep the existing pass-through behavior.
This does not change `chat.agent()`. Raw custom agents do not expose an
action schema, so `payload.action` remains `unknown`.
## Validation failures
Invalid client data is logged and never passed to user code. The client
receives a fixed `Invalid client data` error; validator details stay in
the task log and `onClientDataValidationError`.
- Submitted turns and async reads write the error followed by
`turn-complete`, then wait for the next valid frame. This settles the
invalid input before the raw read returns. Callers that need to
coordinate validation with their own persistence or settlement should
omit the schema and validate the full frame in their loop.
- Messageless preload and continuation boots call
`onClientDataValidationError` and wait without writing a terminal frame.
- Active `chat.messages.on()` subscriptions skip invalid frames and call
`onClientDataValidationError` without ending the response. `off()` stops
new frames. A valid frame accepted before `off()` finishes validation
and is delivered; an invalid pending frame is logged without invoking
user callbacks.
- `chat.messages.peek()` throws synchronously.
- Invalid head-start handovers fail closed. A skip ends the run. A real
handover writes the validation error after the warm output, writes
`turn-complete`, and ends the run.
Validation is automatic when a schema is declared. We can make it opt-in
or return a typed failure if maintainers prefer that contract.
## Testing
- `pnpm --filter @trigger.dev/sdk run test -- --run`
- `pnpm --filter @trigger.dev/sdk run typecheck`
- `pnpm run build --filter @trigger.dev/sdk`
- `pnpm run lint`
- Formatting checks pass
## ✅ Checklist
- [x] I followed the contributing guide
- [x] The PR title follows the convention
- [x] I ran and tested the change
## Changelog
Custom chat agents now validate and parse client data declared with
`chat.withClientData({ schema })` before passing it to agent code.
## Screenshots
Not applicable.
---------
Co-authored-by: Eric Allam <eallam@icloud.com>
<!-- ccr-slack-attribution -->
_Requested by **Matt Aitken** · [Slack
thread](https://triggerdotdev.slack.com/archives/C061L2MHW93/p1787615162456839?thread_ts=1787615162.456839&cid=C061L2MHW93)_
**Before:** a `chat.agent` run is killed mid-answer (OOM, crash,
eviction) while the message it was answering is the only one still
outstanding. The new run boots, puts that message and the half-written
reply into its context, and then waits for a message that already
arrived. Nobody ever answers the user; the run sits idle until it times
out.
**After:** the new run re-runs that message as a fresh turn and replies
to it. The half-written reply is dropped. When two or more messages are
outstanding, nothing changes — the interrupted one still goes into
context and the newer ones are re-run, exactly as before.
## ✅ Checklist
- [x] I have followed every step in the [contributing
guide](https://github.com/triggerdotdev/trigger.dev/blob/main/CONTRIBUTING.md)
- [x] The PR title follows the convention.
- [x] I ran and tested the code works
---
## Testing
New regression test in `packages/trigger-sdk/test/recovery-boot.test.ts`
— seeds a partial assistant plus exactly one in-flight user, no
`onRecoveryBoot`, and asserts one turn fires for that user with the
orphan partial dropped from the chain. It fails on `main` (`turnCount`
0, no turn at all) and passes with this change.
- `pnpm exec vitest run` in `packages/trigger-sdk` — 373 passed, 1
skipped (31 files passed, 1 skipped)
- `pnpm exec oxfmt --check` on the changed files — clean
- `pnpm exec oxlint packages/trigger-sdk/src packages/trigger-sdk/test`
— clean
- `pnpm run build --filter @trigger.dev/sdk` — clean
**What it does:** with exactly one in-flight user on a recovery boot,
re-dispatch that user as a fresh turn instead of splicing it into the
seed chain, where it was never answered.
**How:** the recovery-boot smart default made one decision in two halves
— the seed chain and the recovered-turn list — both gated on
`partialAssistant !== undefined && inFlightUsers.length > 0`. The splice
consumes `inFlightUsers[0]` into the chain as "the question the partial
was answering" and dispatches the rest. That only works when there *is*
a rest: at n=1 `recoveredTurns` came out empty, the boot-injected queue
stayed empty, the `session.in` cursor was advanced past the message
anyway, and on a `preload` or continuation boot (no `message` on the
wire payload) neither dispatch site fired. Both branches now require
`length > 1`, so n=1 falls through to the documented default — chain =
`settledMessages`, re-dispatch every in-flight user. The submit-message
boot is unaffected: the existing dedup still drops a queued message
identical to the one already on the wire payload.
Also corrected alongside it: the two SDK docstrings and the
`docs/ai-chat/patterns/recovery-boot.mdx` defaults section, which
described the default as "re-dispatch every user" and never mentioned
the splice.
Follow-up (not in this PR): the webapp e2e OOM helper never streams a
token before throwing, so it exercises the no-partial path only and
would not have caught this. Worth a variant that emits a token first.
---
## Changelog
Fixed a chat agent hanging after an interrupted turn: when a run was
killed mid-answer and only the one message it was answering was still
outstanding, the new run never replied to it. That message is now
re-answered on the new run.
Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: Matt Aitken <matt@mattaitken.com>
Co-authored-by: Eric Allam <eric@trigger.dev>
## Summary
Adds the id-minting half of sharding run data across several databases.
Every entity that co-locates with a run now carries the run's shard key
inside its own id, so its row is routable on its own instead of needing
a directory table or a scatter across shards.
Nothing changes for users yet. With no shard descriptors configured,
every mint path produces exactly the ids it produces today, and the
trigger path issues no extra query.
## Design
A run's mint target travels as a single object carrying the kind and,
when sharded, the shard character. The shard and the caller's region
both occupy index 24 of a run-ops id, so passing them together makes it
impossible for a caller to set two competing sources for one slot.
A child run, a batch and a batch item read the shard from their parent's
id rather than resolving a fresh one, so a run tree never splits across
databases. Three services carried that branch separately, and one had
already drifted, so it now lives in one function.
Waitpoints mint through one shared pure function used by both the webapp
and the run engine. They have to agree byte for byte, because the
routing store refuses a waitpoint whose id is not stamped for the shard
it is being written to:
```ts
mintWaitpointIdForShard(key) // standalone token: the environment's shard
mintWaitpointIdFor(anchorId) // co-located: the anchor's shard, or a cuid
```
The core is always freshly minted rather than derived from the anchor,
since a derived body would be byte-identical to the run's own id.
One latent bug fixed on the way: the failed-run path duplicated the mint
branch inline and had drifted, so a child of a sharded parent would have
been written to a different database from its parent.
## Guarding the create sites
The expensive failure here is a waitpoint minted without its anchor's
shard: one of the five create sites writes through a path that has no
stamp check, so a miss there strands a blocked run with nothing logged.
An enumerated census plus a source scan fails when a new create site
appears, when an existing one stops passing its anchor, or when a site
is added to a file the scan does not yet cover.
The census was written before any site was converted, so it went red on
the first commit and green as the last site landed. Both holes an
earlier draft had, a file-granular count and a scan that missed the
directory these mints used to live in, were confirmed closed by
reintroducing them and watching the guard fail.
## Before enabling a shard
Merging this is inert: with the mint list empty the resolver returns
before it reads anything, and
ids are identical to a measured `main` baseline. Verified against a live
shard locally, including
that the resolver issues no query across thirty triggers with no shard
configured.
Enabling is gated on two other pull requests, both open, both by the
same author, each of which owns
the file involved:
- **#4781** adds the gen-2 shard arm to read-through. Without it a gen-2
run cannot wait on a token
at all: the wait route resolves the waitpoint through read-through,
which is shard-blind, so the
wait fails. Do not set the mint list before it merges.
- **#4780** generalises the distinct-database sentinel. Without it a
shard pointed at the same
physical database as the gen-1 store boots without complaint, which
voids the disjointness the
fan-out sums rely on.
Testing also turned up a silent read-path gap that neither pull request
covers: the paths that
hydrate runs from ClickHouse through a fixed pair of Postgres clients
drop gen-2 rows on the floor,
so the runs list would show fewer rows than its own count with nothing
logged. That needs its own
change before a shard carries real traffic, and it is filed as such.
## Notes for reviewers
Four commits in the middle of the stack do not typecheck in isolation: a
signature change and its call-site repairs are separate commits, so
bisecting inside the stack needs care. Commit `845ab06` also understates
itself, since it rewrites the primary trigger path's mint alongside the
failed-run path it names.
No changeset and no server-changes entry: every path is inert while the
feature is off, so there is nothing to tell users yet.
---------
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
## Summary
Raw `chat.customAgent()` loops can now call `chat.endAndContinue()` to
move the Session to a fresh run. The managed loop already used the same
server operation through `chat.requestUpgrade()`, but raw loops could
not call it directly.
Call the method between turns after detaching input listeners from the
old run. Await it and return immediately. Unconsumed `.in` records stay
on the Session for the continuation run.
I put this on the `chat` namespace next to the other raw chat
primitives. Happy to move it if maintainers prefer a different API
placement.
## Testing
- `pnpm exec vitest run` in `packages/trigger-sdk` (374 tests)
- Focused webapp Session E2E tests (3 tests)
- `pnpm run build` in `packages/trigger-sdk`
- Webapp typecheck
- `pnpm run format`
- `pnpm run lint`
## Checklist
- [x] I followed the contributing guide
- [x] The PR title follows the convention
- [x] I tested the change
## Changelog
Allow custom chat agents to rotate to a new task version without
dropping unconsumed Session input.
---------
Co-authored-by: Eric Allam <eallam@icloud.com>
Follow-up to
[#4644](https://github.com/triggerdotdev/trigger.dev/pull/4644), now
rebased onto main so the diff is just these three commits.
## Summary
Two ways a chat could lose a user message, both pre-existing and both
raised while reviewing #4644.
A message arriving while a turn was streaming was handed to that turn's
push handler and parked in an in-memory array. The router counts a
record handed to a handler as terminally decided, so it stopped holding
the resume floor behind it, and the turn boundary published a cursor
past a message that existed only in that process. A crash before the
next turn lost it, silently. Measured: with the message at sequence 1,
the boundary published `session-in-event-id: 1`, so a resume skipped it.
Separately, a message the agent declined to inject was discarded with
the turn. Never injected, never written to the wire buffer, never
answered. That was also the documented default, since a
`pendingMessages` config without `shouldInject` declines every batch.
## Design
Notification and consumption are now separate concerns on the router.
`observe` reports that a record arrived without taking it, so the record
stays queued and keeps holding the floor. It is rejected on an
`at-arrival` route: an observer there would either have to count as a
listener, which would stop an unconsumed stop being discarded and bring
back a wedged mailbox, or watch records it cannot affect. `take` removes
exactly one queued record.
The managed loop and the `chat.createSession()` iterator now only
subscribe when there is a steering config to feed, and injection is the
point of consumption. A declined batch never reaches the take, so its
records stay queued and become later turns. Both in-memory wire buffers
are gone, so a message waiting for its turn is durable rather than
living in whichever worker received it.
The floor doubles as the wake cursor: `awaitWake` registers with it and
the server completes the waitpoint immediately if anything sits after
that sequence. An over-advanced floor was therefore also a missed wake.
It is now recorded on the wait span so a run that never woke can be
diagnosed from its trace.
## Verification
Both fixes have a red and green pair, each checked against the
unmodified source rather than only observed to pass:
- the resume cursor test fails on the parent branch and passes here
- the declined-message test fails without the second commit and passes
with it
Also 8 new router tests for `observe` and `take`. Suites green at 385
for the SDK and 886 for core.
## Not addressed
A `pendingMessages` config with no `chat.toStreamTextOptions()` spread
still swallows messages, because nothing drains the queue at all. Same
shape, different trigger, tracked separately.
## Summary
Reloading a browser chat mid-turn can replay a completion event for an
older input and close the active turn too early.
This persists the last browser-owned input sequence and reuses it on
reconnect, so older completion events are ignored. The sequence is
cleared after the matching boundary, and reconnect avoids the
settled-peek shortcut while that sequence is active.
The persisted field is optional, so sessions without it keep their
existing behavior.
## Testing
- `pnpm --dir packages/trigger-sdk run test ./src/v3/chat.test.ts
./test/chat-turn-correlation.test.ts --run` — 67 passed
- `pnpm --dir packages/trigger-sdk run test --run` — 32 files, 379 tests
passed
- `pnpm run build --filter @trigger.dev/sdk`
- `pnpm run format`
- `pnpm run lint`
## Changelog
Browser chats now keep the active turn open across page reloads when
older completion records are replayed.
## ✅ Checklist
- [x] I have followed every step in the [contributing
guide](https://github.com/triggerdotdev/trigger.dev/blob/main/CONTRIBUTING.md)
- [x] The PR title follows the convention.
- [x] I ran and tested the code works
💯
Co-authored-by: Matt Aitken <matt@mattaitken.com>
## Summary
Adds `chat.messages.hasPending()` and `chat.messages.next()` so a custom
agent
loop can inspect pending chat input without consuming it and take one
record at a
time, and fixes four ways a chat could mishandle input across a restart:
a
message silently lost, a recovered answer cut off by a stop the user had
already
pressed, a retried send answered twice, and a record the agent had no
consumer
for blocking every message queued behind it.
```ts
if (await chat.messages.hasPending()) {
const record = await chat.messages.next({ timeoutInSeconds: 0 });
if (record) handle(record.payload);
}
```
## Why the fixes came together
`session.in` carries records for consumers whose delivery needs differ.
A user
message must be delivered eventually, so it can wait arbitrarily long
for a turn
to take it. A stop only means anything to the turn that is live when it
lands.
Progress along the channel was tracked as one sequence number, and one
number
cannot say "control applied through 7, message 3 still owed" at the same
time.
Each of the bugs above is that mismatch surfacing somewhere different.
So instead of a rule per symptom, records are now classified once and
handed to
one route, and each route declares two things: whether it holds a record
when no
consumer is ready, and whether a record it never handled has to survive
into the
next boot. The resume cursor, the replay window and the
discard-the-unowned
behaviour are then derived from route state rather than maintained
beside it, and
`hasPending()` answers from the message queue instead of the head of a
buffer
shared with every other kind.
The wire is unchanged. Both cursors on the turn boundary keep their
meanings, so
existing chats resume as before and there is no webapp change.
## Behaviour worth calling out
`chat.writeTurnComplete()`'s `sessionInEventId` is the cursor that is
safe to
resume from, not the sequence of the record the turn answered. It is
held back
behind any message still waiting to be handled, so a value below the
record you
just handled is expected rather than a sign of a lost turn.
The stop fix also covers chats whose most recent turn was completed by
an older
SDK, by resolving the replay window from the channel when the boundary
does not
carry one. The trade there is deliberate: a stop that landed in the
moments
before boot and was never applied is dropped along with the replayed
ones,
because a stop the user can press again beats a stale one killing an
answer they
are waiting for.
## Verification
Thirteen reproductions against a local stack, each driving real runs
rather than
mocks, covering the documented `next()`/`hasPending()` loop, suspend and
resume, a
crash between consuming a message and writing turn-complete, a retried
send whose
idempotency claim is lost, and a continuation boot that must not replay
answered
messages. Where applicable each was also run against `main`, so the
fixes are
differences rather than assertions.
Five further legs on a deployed environment, which the earlier revisions
of this
branch did not cover at all: a message appended while the run is
genuinely
checkpointed, a message appended while the run is dead, the
stop-after-crash case
on the real crash path, and both version-skew directions (a newer worker
resuming
an older worker's turn boundary, and an older worker resuming a newer
one's).
Two of those restart fixes also have a browser-driven red and green pair
on a
deployed environment, staged identically on both sides and differing
only in the
SDK. For the lost-message fix, the unanswered message is replayed and
answered in
full here, and is never replayed at all on the released SDK. For the
stop fix,
both sides replay the message and diverge on the stop itself: it is
declined here
and the answer completes, while the released SDK re-applies it and the
recovered
answer dies before it streams.
The routing decision itself is a pure state machine, so it also has a
property
test over every interleaving of the record kinds crossed with each crash
point,
checked by mutation to confirm it fails when the cursor arithmetic or
the replay
window is broken.
## Known and not addressed here
The read of the woken record is unbounded, so a wake with nothing to
read makes
`wait()` outlive its own waitpoint. Tested and not a deadlock, since the
read
defers to the next record, but bounding it is a separate change with its
own
test.
Separately, and not caused by this branch: a run that crashes while a
message is
still queued is not replaced until the next inbound append, so that
message waits
rather than being recovered on its own. Worth its own issue.
Also not caused by this branch, but worth knowing when reading the
release note: a
chat page that stayed open across the crash keeps showing the partial
answer it
already received, so the recovered answer only appears after a reload.
The answer
itself is persisted correctly. The gap is on the client, which does not
apply a
re-delivered turn over a partial it already holds.
---------
Co-authored-by: Eric Allam <eric@trigger.dev>
Co-authored-by: Eric Allam <eallam@icloud.com>
## ✅ Checklist
- [x] I have followed every step in the [contributing
guide](https://github.com/triggerdotdev/trigger.dev/blob/main/CONTRIBUTING.md)
- [x] The PR title follows the convention.
- [x] I ran and tested the code works
---
## Testing
Reproduced with `useTriggerChatTransport` + `useChat` and the stop
pattern from the ai-chat frontend docs:
1. Send a message so a turn is streaming.
2. Call `transport.stopGeneration(chatId)`, then `useChat`'s `stop()`.
3. Send another message.
Before this change the second turn never renders: no parts arrive,
`status` stays `streaming`, and the session stays `isStreaming: true`,
so a stop button stays on screen until the page is reloaded. The run
itself is fine and everything persists, so a reload shows the full
response.
Cause: `stopGeneration` sets `state.skipToTurnComplete = true`, and the
read loop only clears that when it sees a `TURN_COMPLETE` record. The
abort closes the reader before that record arrives, so the flag survives
into the next turn and every record of that turn is skipped, including
its own `TURN_COMPLETE`.
After this change the same sequence streams the second turn normally.
Verified against 4.5.11 and 4.5.12 (both affected) with the equivalent
patch applied to the built SDK.
---
## Changelog
Reset `skipToTurnComplete` when a new chat turn or action is sent, so a
message sent after `stopGeneration` streams normally instead of leaving
the chat stuck in a streaming state.
---------
Co-authored-by: Devin AI <devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: nicktrn <55853254+nicktrn@users.noreply.github.com>
Gives read-through and idempotency their gen-2 shard arms, so an id that
names its own shard is read there and nowhere else.
#4764 has landed, so this now targets `main` directly and no longer
depends on an unmerged branch. It builds on what that PR supplied:
`resolveShard`, `runOpsShardHandles` and the keyed router.
TRI-13431
## What changes
**Read-through routes by `resolveShard`, not by the binary residency
classifier.** A gen-2 id reads its own shard's replica once and probes
no other store. A gen-1 v1 id still reads new only.
**Callers now declare `idKind`.** A cuid gives no way to tell a run id
from a waitpoint id, and the two must route differently:
- a legacy-classified **run** id reads the legacy replica only — there
is no cuid run migration, so the new-store probe cannot find it;
- a cuid **waitpoint** keeps the new-first pair probe, which is
load-bearing because a cuid waitpoint can be co-located with its run on
the new store.
There is no default, because a default would pick one of those arms
silently. The field `runId` is renamed to `id`, since it carried both
kinds already.
**`ReadThroughResult` carries `found`.** `source` is an open-ended union
once shards exist, so a consumer testing found-ness by listing the hit
sources reads a gen-2 hit as a miss. One consumer did exactly that.
Discriminating on `found` makes that class of bug a compile error rather
than something a reviewer has to spot.
**Idempotency resolves its client through one shard-keyed map.** Both
call sites go through `clientForShardKey`, so they cannot disagree about
which store owns an id. An absent key takes an explicit logged branch to
the fallback, not a silent legacy default. The `classify` seam is
retyped to return a `ShardKey`: `Residency` (`"NEW"`) and the reserved
shard keys (`"new"`) differ only by case, and `ShardKey` collapses to
`string`, so the compiler would not have caught feeding one into the
other.
The dead `isMigrated` branch is deleted. Nothing implemented it, and the
one production comment recorded that omitting it was deliberate.
**`PostgresRunStore._residency` widens to `ShardKey`.** Still unused;
the store stays unaware of its siblings.
## Two behaviour fixes found while doing the above
**An unconfigured shard key logs and returns not-found instead of
throwing.** The waitpoint route takes the id from a URL parameter, and
any base32hex core plus `[a-z0-9]` plus `"2"` parses as gen-2. The route
turns a throw into a 500, so throwing here would let any authenticated
client generate 500s and error logs by guessing shard chars, of which
there are 36. An error-logged not-found is neither silent nor a
misroute. Throwing stays correct on the router path, where ids are
minted rather than received.
**The two cross-seam batch hydration sites were gen-2 blind.**
`hydrateRunsAcrossSeam` and `ApiBatchResultsPresenter` classified with
the binary `ownerEngine`, so a gen-2 run id joined the gen-1 `new`
group, missed there, and — classifying dedicated-family — never reached
the legacy probe either. The id was dropped from a bulk-action page and
from batch results with no error. Both now partition ids by shard key
and read each configured shard once.
Also: a gen-2 waitpoint that missed its shard replica fell back to the
gen-1 new writer, a different database, silently disabling
read-your-writes for the freshly minted token that fallback exists to
serve. It now falls back to its own shard's writer.
## Merge safety
Inert while `RUN_OPS_SHARDS` is unset: the shard maps are empty, so
every gen-2 arm is unreachable, and gen-2 minting is not live yet.
The one live change is the gen-1 run arm, and it removes work rather
than adding it. `RoutingRunStore.findRun` never forwards the caller's
client object — it routes by id and reads only the client's presence and
replica brand — so `readRunForEvent`'s "new" closure already resolved a
legacy-classified run id to the legacy store. The arm removes a
duplicated read of the legacy replica. A test pins this, because a
future caller passing a raw client and a run id would lose the
pre-cutover 27-char case, which is new-resident but classifies legacy.
## Testing
14 tests added, testcontainers throughout, no mocks. 22 affected test
files pass; typecheck, lint, format and knip are clean.
Both arms were verified by neutralising them and confirming the new
tests fail. The batch-results test needed rewriting after that check:
the first version passed with the fix neutralised, because it used one
container as both the gen-1 new client and the shard replica, so it was
not testing what it claimed.
Note for review: run testcontainer suites in small batches. Sixteen at
once starves Docker and everything times out at 60 seconds.
The run-ops legacy-guard baseline is refreshed in its own commit. The
baseline is keyed by line number, so partitioning the batch-results read
shifted four pre-existing entries and added one. Baselined violations in
that file go from four to five, all reads; the new one is the shard read
beside two gen-1 reads already there.
No changeset and no `.server-changes` entry: a user notices nothing
while the flag is unset.
Three bugs on the project integrations page, one commit each for the two
reported ones and four for the follow-ups found while fixing them.
## `chore`: remove unreachable code on the integrations page (TRI-12645)
Two notification panels in `VercelSettingsPanel` could never render:
1. The **"Failed to load Vercel settings"** panel was gated on a
`hasError` state whose setter is never called anywhere, so it was
permanently `false`.
2. The **"connection expired"** banner *inside* the `connectedProject`
branch was unreachable: `VercelSettingsPresenter` only populates
`connectedProject` on its success exit, which hardcodes `authInvalid:
false`, while both `authInvalid: true` exits return `connectedProject:
undefined`.
Removing them makes the surrounding `!showAuthInvalid` guards vacuous,
and the `onboardingData?.authInvalid` disjunct redundant — the loader
already folds onboarding auth state into `authInvalid` before it reaches
the component.
**No behaviour change.** An org with a connected project and an expired
token still gets the banner, from the branch below (untouched).
## `fix`: gate Staging settings on plans without a Staging environment
(TRI-12646)
The ticket's premise was inverted, and I've corrected it there. In Git
settings, **Preview** is the row that's correctly gated; **Staging** is
the one with no gate at all:
- Preview swaps its switch for an Upgrade button, and
`projectSettings.server.ts` neutralises a forged
`previewDeploymentsEnabled=on`.
- Staging was a plain always-editable `Input`, and
`validateStagingBranch` only checked the branch existed on GitHub. An
org without a staging environment could type a tracking branch, hit
Save, get a success toast, and have it silently do nothing.
Staging and Preview environments are created together for projects on a
plan that includes them, so gating one and not the other was an
oversight.
The Staging row now mirrors the Preview row. Server-side it ignores the
submitted branch when there's no staging environment, but **preserves
the stored branch rather than clearing it** — deliberately different
from the Preview handling. Forcing a boolean off is harmless; forcing a
*string* off would wipe a tracking branch the org had already configured
the first time they saved after losing the environment.
The Vercel write path had the same gap: `update-config` /
`complete-onboarding` / `update-env-mapping` never re-derived available
env slugs server-side, so `["stg","preview"]` could be persisted for a
project with neither environment, and
`createDefaultVercelIntegrationData` turned preview on unconditionally.
Both now filter against the project's actual environments, via a pure
`restrictConfigToAvailableEnvSlugs` helper that only touches keys
present on the input.
## `fix`: show build settings when the GitHub app is disabled
(TRI-13488)
The page wrapped Git settings, the Vercel section **and** build settings
in one `githubAppEnabled` guard, so with the GitHub app off it rendered
an empty container.
The Vercel section genuinely depends on GitHub — it can't sync
environment variables or link deployments without a connected repo — so
it stays gated. Build settings don't: they also apply to CLI deploys run
with `--native-build-server`, exactly as the section's own description
states. They now render regardless.
## `fix`: stop the Vercel onboarding modal spinning forever (TRI-13488)
`computeInitialState` starts in `loading-projects` whenever the org has
a Vercel integration but no onboarding data yet, and the effect that
escapes it waits for `availableProjects !== undefined`. When
`getOnboardingData` returns `null` — it does that on any thrown error,
and when the org integration row is missing — nothing ever arrives.
The empty-array case self-resolves (`[] !== undefined`), so this is
specifically the null case. The route can tell "still loading" from
"loaded nothing" because its fetcher always requests
`?vercelOnboarding=true`; it now passes that down and the modal explains
the failure with a retry and a link to check the integration's access on
Vercel.
## `fix`: match staging and preview environments consistently
(TRI-13488)
The four places that ask "does this project have a staging / preview
environment?" disagreed. `VercelSettingsPresenter` matched on type with
no parent filter, so any preview *branch* row satisfied it — branches
are `PREVIEW` rows too. `GitHubSettingsPresenter` and
`ProjectSettingsService` matched on slug instead.
Slug is the weaker key: it's derived at creation time and legacy rows
can carry something else, which is why
`memberDevelopmentEnvironmentWhere` deliberately avoids it. All four now
match on `type` plus `parentEnvironmentId: null`, which excludes
branches without depending on the slug being canonical.
## `fix`: explain when no Vercel environment can be mapped to Staging
(TRI-13488)
Reported while reviewing the branch. The Staging build settings show
*"Set a Vercel environment for Staging first."* whenever the project has
a staging environment and no mapping — but the control that sets the
mapping only rendered when the Vercel project had at least one custom
environment:
```
hint: hasStagingEnvironment && !configValues.vercelStagingEnvironment
control: hasStagingEnvironment && customEnvironments.length > 0
```
So a Vercel project with no custom environments, or one whose custom
environments failed to fetch (the presenter swallows that error to
`[]`), got an instruction with nothing to act on. Both conditions
predate this PR.
The mapping row now always renders alongside the hint and explains what
to do when there's nothing to choose from, and the build-settings hint
says the same thing.
## `chore`: remove the remaining dead code (TRI-13488)
- The `"installing"` `OnboardingState` is unproducible — no `setState`
call yields it — so its redirect effect, switch arm, `isLoadingState`
conjunct and the `vercelAppInstallPath` import it was the only user of
are all dead.
- `(state as string) !== "completed"` sits in a branch where TypeScript
has already narrowed `"completed"` out; the cast is what let it compile.
- `hideSectionToggles` was only ever passed alongside
`layout="settings"` but only read inside `layout="card"` blocks, so it
could never take effect. Removed the prop entirely.
- Unused bindings and the helpers only they referenced: `envSlugLabel`,
`_formatSelectedEnvs`, `_CompleteOnboardingForm`,
`_handleFinishOnboarding`, and the rest.
No behaviour change in that commit.
## Not included
The three overlapping modal-open effects in
`settings.integrations/route.tsx` are left alone — they're defensive
against a close-then-reopen race, and untangling them is a behavioural
risk with no user-visible payoff.
## Verification
`pnpm run typecheck --filter webapp`, `pnpm run lint` and `pnpm run
knip` are clean. New `apps/webapp/test/vercelIntegrationConfig.test.ts`
covers the slug restriction and the default-config seeding (both pure
functions); 39 tests pass across it and the three existing
Vercel/project-settings files.
The new `projectId` + `slug` query is served by the existing
`@@unique([projectId, slug, orgMemberId])` prefix — same access pattern
as the preview check it mirrors.
refs TRI-12645, TRI-12646, TRI-13488
## Summary
Adds a `RunStore` decorator that mirrors execution snapshots into Redis
alongside Postgres, plus the orphan-key sweep and the fault-injection
suite that prove the write protocol converges after a crash. Nothing
constructs it, so merging this changes no behaviour: the configuration,
the production wiring and the Redis client all arrive in later work.
The execution-state log is the hottest table in the run graph, and
moving it out of Postgres has to happen without a big-bang cutover. This
is the attachment point for that: a decorator that wraps the existing
storage interface and intercepts only the methods that touch snapshots,
so none of the many callers change.
## Design
Write order is the correctness property, and the two orders differ on
purpose.
A transition writes Postgres first and Redis second. A crash in the gap
leaves a run whose latest snapshot is stale, which is the state the
heartbeat stall watchdog already heals in production today.
A birth writes Redis first and Postgres second. A crash there leaves an
unreachable key for a run that does not exist. Postgres first would
instead leave a run with no snapshot at all, which the engine treats as
a hard error, so the run would be stuck.
Each order is chosen so the state a crash leaves behind is the harmless
one. A lost cross-store write is never recovered by a transaction or an
outbox; recovery is always the existing stall and repair job. A failed
append retries, then hands the run to that job, and never rethrows,
because Postgres has already committed and a throw would turn a healable
gap into a caller-visible error.
Inside a transaction the Redis half is staged and flushed only after the
commit, so a rollback cannot leave Redis holding a transition that never
happened.
Reads are shape matched. Two of the snapshot reads take arbitrary Prisma
arguments, and a key-value store cannot answer an arbitrary query, so
the decorator recognises exactly the shapes the engine sends and
delegates everything else. A miss falls back to Postgres, which is also
how runs created before any cutover keep working.
The sweep reaps under two rules, because neither can see what the other
leaves behind. A finished run whose keyspace never received its
completion expiry gets one applied. A keyspace with no run row at all,
past an age threshold, is deleted; that is a crashed birth, which is
non-terminal so it carries no expiry and has no run row, so the first
rule can never match it.
## Inertness
Three independent reasons this is a no-op if merged alone:
- Nothing constructs the decorator or the Redis store outside tests.
- No configuration reaches it, so the dial stays at its off position,
which is a pass-through that makes no Redis call.
- The existing Postgres store gains an off-by-default flag and two
optional input fields. Both default to today's behaviour, and only the
decorator would ever supply them.
## Notes for review
The snapshot id and the creation instant are both minted by the
decorator and written into both stores, so one snapshot has one identity
and one timestamp wherever it is read. Without that, the two stores
disagree on values that later tooling has to compare, and the cursor for
a snapshot window resolved from one store misfilters the window walked
in the other.
Three defects in this work passed the full existing test suites before
being found by review rather than by a test: the decorator wrote no wait
cycle at all, the snapshot window dropped the ordering used to give each
completed waitpoint its position in a batch, and the two stores stamped
different creation times. The common cause was that no test drove a
snapshot that actually carried waitpoints, and that the parity suite
compared a timestamp against a value it had just read back from the row
it was checking. Both gaps now have tests.
## Summary
The run-ops boot interlocks and the migration entrypoint each assume
exactly two run-ops
databases. This generalizes them to any number, so a deployment that
configures
`RUN_OPS_SHARDS` gets the same safety guarantees it gets today with two
stores: no two stores
may point at one database, every store that owns its own database must
replicate to
ClickHouse, and every store must have its schema migrated.
With `RUN_OPS_SHARDS` unset, nothing changes. The distinctness check
over a two-element set is
the pairwise compare it replaces, replication coverage is the check it
was, and the entrypoint
runs the same two migration invocations.
A shard may declare `aliasOf: "new"`, which shares an existing store's
client by reference. An
aliased shard is not its own database, so it is exempt from the
distinctness check and needs no
replication slot of its own. Every check keys that exemption on the
declared field, never on
client object identity: two client objects can sit over one database,
which identity comparison
cannot see.
## Design
**Distinctness.** `probeDistinctDatabases` compared two URLs. It now
delegates to
`probeDistinctStores`, which reads every fingerprint in parallel and
groups them by system
identifier and database name. Any two stores under one key refuse the
boot. The old pairwise
entry point stays, so its existing container tests are the proof that
set uniqueness over one
pair gives the verdict it gave before. Fail-closed is unchanged: a probe
that cannot answer
returns not-distinct, because "distinct" is a positive claim a failed
probe cannot support.
**Co-residency.** The advisory runs once per store against the control
plane. The legacy
emission keeps its exact call shape and its untagged metric series, so
an existing dashboard
does not change. Each shard emits its own point carrying its shard key.
Every store emits
before any enforcement throw, so one offending store never costs another
store its metric.
**Replication.** `buildReplicationSources` appends one source per shard
that owns its own
database, taking the slot, publication and origin generation its
descriptor declares.
`assertReplicationCoversSplit` then requires a source per such shard.
That check also closes a hole it inherited. The descriptor parser
validates uniqueness among
shards only, so a shard could take the slot name, publication name or
origin generation of the
legacy or the new source. The replication service does validate this,
but it throws from its
constructor, and the caller reaches that constructor only after shutting
the bootstrap instance
down:
```ts
if (sources.length > 1) {
await service.shutdown(); // legacy stream stops here
service = new RunsReplicationService({ ... }); // throws: duplicate slotName
}
```
The throw was not a `SplitReplicationMisconfiguredError`, so the process
stayed up with no
replication at all, legacy included, behind one logged line. That is the
silent ClickHouse
under-count the error exists to prevent. The check now runs at the boot
gate, before anything is
torn down, and raises a subclass the existing exit path already
recognizes. A correct deployment
already satisfies it, because two consumers on one WAL slot is a data
race that cannot work.
**Migrations.** Every shard runs the identical schema, so a new shard is
the existing migrations
against a new DSN. The runner image has no `jq`, so a small node script
prints one DSN per line
and the entrypoint loops over them. The loop is a `for` and not a `while
read` pipeline: a
pipeline subshell swallows a failed migration on any iteration but the
last, which would let a
broken shard boot. Tracing stays off across the capture and the loop,
because `set -x` prints an
assignment and a DSN carries credentials.
Verified end to end against real Postgres containers for the fingerprint
probes, and against the
real shell block with a stubbed migration command: an aliased shard is
skipped, `directUrl` wins
over `url`, a failing shard stops the container on the first failure,
and a malformed descriptor
stops it before it migrates anything.
Stacked on #4764.
---------
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Deployments currently leave little analytical trace. This PR makes every
deployment emit two analytics events to enable useful queries. It also
enables comparing deployments across build paths, CLI versions,
runtimes, and orgs.
### Where the events come from
```
trigger deploy
│
▼
initialize ─────────────────────────────▶ ✨ deployment.initialized
│ createdAt
▼
PENDING waiting for a build slot ┐
│ startedAt │ queue time
▼ ┘
INSTALLING build server installs deps ┐
│ installedAt (native paths only) │ install time
▼ ┘
BUILDING the image is built ┐
│ builtAt │ building time
▼ ┘
DEPLOYING indexing + registry push ┐
│ deployedAt / failedAt / canceledAt │ deploying time
▼ ┘
DEPLOYED · FAILED · TIMED_OUT · CANCELED
│
└───────────────────────────────────▶ ✨ deployment.finished
```
`deployment.finished` fires exactly once, whichever way the deployment
ends, and is backdated to cover the deployment's real lifetime. Not
every path visits every state (Depot deploys skip PENDING/INSTALLING,
for example) — a phase duration is simply omitted when its state was
never entered.
### What each event carries
- **Which path built it**: `depot`, `native`, or `native_local_bundle`
- **How it ended**: status, plus an error class and message when it
failed
- **How long each phase took**: queue, install, building, deploying, and
total — derived from the timestamps above
- **Who and with what**: org, project, environment, runtime, CLI
version, and how the deploy was triggered (CLI, GitHub, Vercel)
With that, one query gives failure rate per build path, duration
percentiles per phase, adoption per CLI version, or a per-org health
table.
### Fixes that ride along
- The old `deployment.outcome` span was silently dropped ~95% of the
time (it was subject to trace sampling). The new events opt out of
sampling explicitly, so every deployment is counted.
- The fail/timeout/finalize transitions were racy: a late timeout could
overwrite a successful deployment. They now use guarded writes, so
exactly one caller wins the terminal transition — and exactly one event
is emitted.
- Canceled deployments previously recorded nothing; they do now.
- The deployment's CLI version is now stored at initialization (new
nullable column), so even deploys that fail early are attributable to a
CLI release.
- Telemetry is flushed on shutdown (the last batch used to be lost on
every webapp deploy), and an optional second exporter can mirror just
these events into a dedicated dataset.
Part of the RunOps N-way sharding work.
This lets the webapp hold N run-ops stores, configured by a single
`RUN_OPS_SHARDS` JSON descriptor, and routes to them through the
existing keyed router. **Inert with `RUN_OPS_SHARDS` unset** — the
topology, the wiring and `ROUTING_ENABLED` are byte-identical to today.
## What's here
- **`RUN_OPS_SHARDS`** — a zod-validated JSON array of shard descriptors
(`key`, `region`, `url`, `replicaUrl`, `directUrl`, `replication`,
`knobs`, `aliasOf`), validated at boot in the `parseMachinePresetCsv`
style. Unset or `[]` → no shards.
- **One run-ops client factory** —
`buildRunOpsWriterClient`/`buildRunOpsReplicaClient` collapse into one
`buildRunOpsClient` parameterized by role and resolved pool knobs. The
control-plane builders (`buildWriterClient`/`buildReplicaClient`) are a
separate path and stay untouched; every resolved value matches the
former builders.
- **Shard loop in `selectRunOpsTopology`** — one client pair per
descriptor; an `aliasOf: "new"` descriptor reuses the new store's
clients by reference and opens no pool.
- **N-way `buildRunStore`** — builds N dedicated stores + the keyed
router via a new `RoutingRunStore.fromShards`, keeping the two-store
compat router when no shards are configured.
- **`UnknownShardKey`** — raised when an id resolves to an unconfigured
key; never falls back to another store. `fromShards` injects
`resolveShard` so a gen-2 id routes to its own shard.
- **Per-shard transaction resilience** — each shard gets its own retry
budget.
- **Mint bound** — `computeMintShard` intersects the active mint list
with the configured descriptor keys, so a key with no descriptor is
never minted into.
- **Boot table** — logs `key`, address fingerprint (host:port/db, no
credentials), and role, only when shards are configured.
## Ordering constraint
Do **not** configure a `RUN_OPS_SHARDS` descriptor in any environment
until the routing-semantics change (TRI-13427) lands — three fan-out
sites still truncate at N>2. Merging this PR alone is safe (inert with
the var unset); configuring a descriptor is what must wait.
## Testing
- Run-store corpus: green with zero test-file diffs (the bit-identical
proof for the compat router).
- `runOpsDbTopology.test.ts` 17/17, `runStore.server.test.ts` 4/4,
`runOpsMigration` family 149/149.
- New unit suites: descriptor validation, pool-knob value tables,
`fromShards` routing + `UnknownShardKey`, boot-table formatter, mint
bound.
- typecheck (webapp + run-store), knip, lint, format: pass.
## Changelog
Internal run-ops sharding infrastructure. No changeset or
`.server-changes`: the change is inert with `RUN_OPS_SHARDS` unset and
has no user-visible behaviour.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
The environment variable key and value inputs did not set an
autocomplete attribute, so browsers could offer to autofill or save
typed values as saved credentials. This sets `autoComplete="off"` on
those inputs in both the create and edit forms, matching the
`autoComplete="off"` convention already used on the other
credential-name inputs.
`autoComplete="off"` is a best-effort hint. Browsers may still ignore it
for password-typed fields, so this is defense-in-depth hardening, not a
hard guarantee that a password manager cannot store the value.
Auto-scroll now only follows while you are at the bottom. Scrolling up
pauses it; scrolling back to the bottom, or clicking the new
scroll-to-bottom button in the log header, resumes it. When you are at
the bottom the same button scrolls to the top. Switching to another
deployment starts at the bottom again.
## What
Makes `RoutingRunStore` correct when the run-ops layer routes across
more than two Postgres stores. Today it routes between a gen-1 `new`
dedicated database and a `legacy` control-plane database; this
generalizes every routing policy to N shards while keeping the two-store
behaviour byte-identical.
The change sets the four routing decisions that were implicit in code
order, and fixes one hazard that failed silently:
- **Id → shard key.** The router resolves a shard key with
`resolveShard` instead of the binary residency classifier, so a gen-2 id
reaches its own shard through the keyed map.
- **Membership vs routing.** `#distinctStores` (one entry per physical
database, aliases excluded by a declared `aliasOf`) drives every sum,
probe, and merge; `#shards` drives routing. An aliased shard can no
longer make a sum count one database twice.
- **Probe order.** A keyless lookup stays a sequential short-circuit at
two stores; above two it fans out in parallel, picks by precedence,
tolerates a single down leg, and keeps the canonical not-found throw on
the legacy leg.
- **Precedence and duplicates.** One merge helper across all four merge
sites. A duplicate id confined to `{new, legacy}` stays silent (the
known drain-mirror case); any other cross-shard duplicate increments
`runops_shard_duplicate_id_total` and logs at error level.
- **Disjoint sum (the silent hazard).** `countPendingWaitpoints` and the
waitpoint collector now partition absent ids by shard and **union by
id** rather than summing counts. A drain-mirrored waitpoint on both
gen-1 stores is counted once, so a blocked run can no longer hang
forever on a double-counted pending waitpoint.
- **Waitpoint completion.** A gen-2 waitpoint completes on its own
shard, overriding the legacy pins; a cuid waitpoint keeps its two-member
gen-1-pair probe unchanged.
- **Fail-loud creates.** A create with no shard key throws instead of
silently defaulting to `new`. An id resolving to an unconfigured shard
throws instead of being dropped.
Two new counters are exported: `runops_shard_duplicate_id_total` and
`runops_waitpoint_probe_fallback_total`.
## Why it is safe to merge
With only `{new, legacy}` configured every generalized rule reduces to
today's behaviour. `resolveShard` returns exactly what the old
classifier returned for every id shape that exists today, and no gen-2
id is minted yet. The only intentional behaviour change is the fail-loud
create throw; an enumeration of production call sites confirmed no
caller trips it.
## Testing
- New container-free algebra suite (50 cases) over probe order,
precedence, the duplicate alarm, the disjoint-sum partition, the
waitpoint probes, and the fail-loud paths.
- New `runOpsStore.nShardMatrix.test.ts` runs a four-store matrix
(legacy + new + two gen-2 shards) against real Postgres containers: the
disjoint-sum union, the alias topology, cross-tree completion,
pagination merges, and mixed-id hydration.
- New `makeNShardRunOpsPostgresTest(k)` fixture in
`@internal/testcontainers`.
- Full run-store corpus green: 71 files, 480 tests. Typecheck, lint,
format, and knip all clean.
## Notes
- Draft: opened for review; not marking ready yet.
- No changeset or `.server-changes` file: internal routing
infrastructure, no user-visible behaviour change.
- TRI-13427.
Switching between deployments in the dashboard re-fetched the whole
build log stream from record zero and re-rendered the list line by line
every time. Logs are now cached per deployment for the lifetime of the
tab: revisiting a deployment shows its logs immediately, and the stream
is resumed from the next unread record rather than restarted. Finished
deployments whose stream has been read through the `finalized` event are
served entirely from the cache.
### Changes
The stream/cache logic moved out of the route into a `useDeploymentLogs`
hook. On each deployment switch it seeds state from the cache, resumes
the S2 read session at `nextSeqNum`, and writes back on cleanup or
natural session end. Completion is derived from the stream's own
`finalized` event (plus a terminal deployment status), not from the
session closing, so a session cut short by token expiry or a proxy
cannot pin a truncated log in the cache.
Memory is bounded by a small LRU (`deploymentLogsCache`): at most 20
deployments and 20,000 log lines in total, least recently viewed evicted
first. The most recently viewed deployment is always kept, so a single
very large log can temporarily exceed the line budget on its own.
Records are batched into one state update per tick instead of one per
line.
## Summary
When a runs list query is too expensive to complete, it now fails with a
clear, actionable error instead of a generic 500.
Previously, a runs list query that exceeded ClickHouse resource limits
threw an opaque error. On the public `runs.list` API that surfaced as a
retryable 500, so a customer task calling it would keep retrying a query
that could never succeed. On the dashboard it rendered as a generic
error page with no hint about what to do.
## Fix
The ClickHouse client now tags resource-limit failures (memory, time,
rows, bytes) with their error type, and the runs repository maps those
to a dedicated `RunsListQueryError` (HTTP 422).
- `runs.list` API returns 422 with a message telling the user to narrow
their `created_at` range, plus an `x-should-retry: false` header so the
SDK does not retry it.
- The dashboard runs list (and the errors, scheduled, standard-task,
agents, and webhooks list views) render a shared error state with the
same guidance, so a too-broad time filter is recoverable by the user.
Adds an `ADMIN_DASHBOARD_ENABLED` env var (default: enabled) that turns
the admin dashboard and user impersonation off for an entire instance.
When disabled:
- every admin dashboard page redirects away, and the admin navigation
isn't rendered
- existing impersonation cookies are ignored, and any lingering session
is actively terminated with an audit record
- every flow that could start an impersonation responds 404, and no
impersonation tokens are minted
Stopping an impersonation always works regardless of the flag, so
nothing gets stuck. Machine-to-machine admin API endpoints are not
affected. The variable is documented for self-hosters; instances that
don't set it are unaffected.
## Summary
Adds the read comparator for the in-progress migration of the run
execution-snapshot log from Postgres to Redis. The comparator samples a
single read against both stores, normalizes the two results to one
shape, and reports any per-field difference with a tagged metric. It
never serves a read itself: the diff layer imports only types, so it
cannot hold a store client, and a test enforces that by failing if any
value import appears.
Also adds a combined Postgres-and-Redis test fixture and two shared test
utilities (a cluster-slot assertion and a generic fault-injection
harness) that the parallel Redis-store work reuses.
Everything here is inert. Nothing constructs the comparator, so merging
changes no runtime behavior. It becomes active only when a later change
turns on compare mode.
## Notes
The divergence classes separate real differences (scalar, ordering,
waitpoint id set, validity, missing on one side) from two expected
classes that must not be driven to zero: a rotated idempotency key, and
a Redis-only surplus at a since-cursor tie. The since comparison is
direction sensitive: a Postgres-only entry at the cursor is always a
lost write, never an expected tie.
Skew protection resolves a run's worker by (environmentId, externalId,
status=DEPLOYED). A miss parks the run and then expires it, so
deployments
predating the feature — which already carry the same value in commitSHA
— need
externalId populated to stay reachable. Vercel instant-rollback is the
sharpest
case, which is why the scope is the current promotion plus a recent
window
rather than current alone.
Follows the existing backfill shape: admin PAT, keyset cursor over
environments,
per-environment action results, pMap, dryRun defaulting to true. Reuses
normalizeExternalDeploymentId so a backfilled id is byte-identical to
what a
build writes, and the update re-checks externalId IS NULL so a deploy
landing
mid-backfill keeps its own id.
Refs TRI-13464.
## Summary
Improves the performance and reliability of the runs list and the
`runs.list` API, especially for large projects and filtered views.
## What changed
- **Filtered runs-list queries use `PREWHERE`.** Immutable and
additive-only filters (tags, task identifier, version, queue, region,
machine, and the rest) are applied in `PREWHERE` on the `task_runs_v2
FINAL` scan, so ClickHouse filters, and uses the tags skip index, before
it reconciles versions and materialises the wide columns. Same results,
far less memory per query. `status` stays in `WHERE`: it changes across
a run's versions, so filtering it before `FINAL` could return stale
rows.
- **The runs-list ClickHouse pool gets per-query guardrails**, all
env-configurable: a `max_execution_time` paired with the client request
timeout, a per-query `max_memory_usage`, a `max_threads` cap, and
`readonly`. Each bounds a single query to itself, so a heavy query can't
affect other queries, and they are safe as pool-level settings only
because this pool is read-only.
- **Billing and bulk count reads move to the read pool**, off the write
pool.
Defaults are conservative for self-hosters; production values are set
via env.
Default setup doesn't run CodeQL on pull requests from forks, so
external contributions are stuck on PR checks that never come. Advanced
setup fixes this.
Languages, categories and `main` coverage match the current default
setup. The bare `pull_request` trigger (no `branches` filter) keeps
stacked PRs scanned, whose base isn't `main`.
Default setup has to be disabled in Settings -> Code security for these
uploads to be accepted. Until it is, the CodeQL check here fails with
`CodeQL analyses from advanced configurations cannot be processed when
the default setup is enabled`.
Adds an experimental `--local-bundle` flag to native build deployments:
the project is installed and bundled on the local machine (exactly like
in the depot path) and only the resulting build context is uploaded. The
remote build then runs just the container image build.
### Design
- The uploaded artifact is the same build context classic deploys
produce: bundled output, a synthesized package.json with the resolved
externals, build.json, and the generated Containerfile. The bundle is
secret-free: build.json is deliberately scrubbed because it is copied
into the image, and build-arg values never enter the bundle at all.
- Build-arg values are sent with the deployment initialization request
instead, stored encrypted (aes-256-gcm) in a new
`WorkerDeployment.buildEnvVars` column, and cleared on every terminal
status transition. They exist at rest only for the active build window,
always encrypted.
- A dedicated `GET /api/v1/deployments/:id/build-env-vars` endpoint
returns the decrypted values to the same principals that can already
read the environment's variables. It answers with an empty record for
deployments without stored values or in a terminal state, keeping secret
access to a single auditable route.
- Size limits are enforced server side and pre-checked client side. If
the server does not acknowledge storing the values, the CLI fails fast
instead of letting the remote build run without them.
- A `--from-bundle <dir>` mode builds a deployment image straight from
such a bundle directory, skipping config loading and bundling entirely.
In attach mode it fetches the stored build-arg values through the new
endpoint.
- Env var syncing (the `syncEnvVars` extension) happens client side,
before the deployment initializes, since the remote side never sees the
unscrubbed manifest.
- Bundle artifacts use a distinct type and storage prefix so the server
can always distinguish them from source uploads.
Builds the Redis-backed half of the waitpoint coordinator, beside the
Postgres coordinator that #4753 extracted. Adds the coordination
protocol as Lua scripts, the run-ops-format waitpoint id scheme, and the
key layout. **No caller wires any of it up.**
Refs TRI-13440.
## Inert by construction
Merging this changes nothing observable. 3180 insertions, **zero
deletions**, nine new or additively-edited files.
- `WaitpointStoreCoordinator` is never constructed outside its own tests
and the benchmark.
- No env var, no config plumbing, no connection. It takes `redisOptions`
as a constructor argument.
- `waitpointSystem.ts` is untouched. Every live waitpoint operation
still runs on Postgres through the coordinator merged in #4753.
- No changeset and no `.server-changes` note — nothing here is
user-facing yet.
Deploying this needs no Redis or MemoryDB instance. That becomes a
prerequisite when a later change routes traffic onto the store behind a
per-organisation flag.
## What's here
**Nine Lua scripts**, each atomic on one hash tag. Seven mutate state —
create-if-absent, register-or-report, complete, idempotency reserve,
absorb, deliver, clear. One reads state (`runReadBlockState`) and is
separate because the pending, delivered and edge sets must be read as
one consistent view. One discards an idempotency loser.
**Two hash tags, deliberately.** `wp:{waitpointId}` holds a waitpoint's
record, status, completion envelope and watcher hash. `wp:run:{runId}:*`
holds one run's pending set, delivered set and edge set. A waitpoint has
N watchers, so it cannot live under any single run's tag.
**Waitpoint ids** reuse the run-ops body layout: a 24-char base32hex
core, a type char (`r`/`b`/`d`/`m`), and version char `w`. RUN and BATCH
ids derive from their anchor's core, so create-if-absent is idempotent
with no lock. `parseWaitpointId` is total and never throws.
**The single-slot guard.** Every script invocation goes through one
private wrapper that asserts all keys share a hash tag. A single-node
test server accepts what a real cluster rejects, so this assertion is
the only enforcement — and it is mutation-tested: removing it fails a
test.
## Measured
Against the same population of real Postgres rows:
| | store | postgres |
|---|---|---|
| pending count (the blocked/unblocked gate) | 0.13 ms p50 | 3.32 ms p50
|
| full-payload read | 1.45 ms p50 | 7.70 ms p50 |
Both are lower bounds: the benchmark charges Postgres a `COUNT(*)`,
while the resume-time read is a join with a partial select plus
filtering in JavaScript.
Store-only paths, no Postgres counterpart: block+complete+deliver 0.88
ms p50; 100-watcher fan-out 13.8 ms; a 1001-edge fan-in 149.8 ms, flat
at 0.15 ms per edge and round-trip bound rather than algorithmic.
The benchmark lives in `*.bench.test.ts` and is excluded from the
default suite.
## Review notes
- **The type surfaces are not reconciled yet, on purpose.** `types.ts`
(from #4753) carries the coordinator interface; `storeCoordinator.ts`
declares its own operation types because this was built in parallel. The
wiring change reconciles them.
- **The read-time resolver is not here.** Another lane froze its
contract while this was in flight, and its frozen types are not yet on
main. Building a second copy would fork a just-frozen contract.
- **Teardown is one-shard while registration is two-shard.** A terminal
clear leaves a run registered as a watcher on the waitpoints it was
blocked on, because the watcher hash is under a different tag and no
script may span slots. Recorded, not fixed here — it needs a retention
decision, and nothing observes it while the code is unwired.
## Verification
79 tests in the coordinator suite, 58 in the id suite. `typecheck` on
run-engine and webapp, `build` on core, `knip`, `oxfmt` and `oxlint` all
clean. The engine corpus passes 82/82.
Every invariant is mutation-tested rather than merely asserted. A
whole-branch review ran 14 mutants and killed 12; the two survivors were
fixed with their own mutation checks.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
## Summary
Adds the shard-selection stage of run-id minting.
`resolveMintShard(env)` returns which run-ops database an environment
mints its new run roots into: the active shard list, then a fleet-wide
override, then a per-environment or per-organization pin, then a
rendezvous hash of the environment id.
That half is inert. Nothing calls `resolveMintShard`, no deployment has
any of the new flags set, and an empty active list returns the current
answer without reading anything.
**The other half is not inert, and it is where review effort belongs.**
To stamp a grace window this needs a read-then-write under a lock, so it
rewrites the global feature-flag write path that `runOpsMintKind`
already depends on in production. See below.
## Placement
Resolution reads the active list from a global flag, applies the grace
window, and then picks:
- a fleet-wide override if one is set, which is how a cutover completes
without visiting each organization. `new` holds the whole fleet on the
current id format.
- otherwise a per-environment or per-organization pin. `new` holds one
organization back while the rest move, which is how a canary works.
- otherwise a rendezvous hash, so adding a shard moves only about
1/(N+1) of environments and removing one moves only its own.
Two hash details are load-bearing. Scores are 64-bit `sha256(envId \0
key)`, because a 32-bit score collides at our environment count and an
undetected tie would resolve by iteration order. The parsed key list is
sorted, because otherwise two deployments listing the same shards in a
different CSV order would place environments differently.
A pin or override naming a shard that has left the active list falls
through to the hash and reports once. Honouring it would leak the drain
the active list exists to perform, and throwing would fail triggers
whenever a pinned shard drains.
## Why the active list is a flag and not an environment variable
A deploy rolls for hours, so two pods hold two different environment
values at the same time. A list held in the environment therefore splits
the fleet for the length of the rollout, with new pods placing an
environment on one shard and old pods on another. A grace window
measured in seconds cannot cover that, and the same knob times the
existing mint-kind flip so it cannot simply be lengthened. An
environment variable also cannot record its own flip time, and an
operator cannot know a rollout's end in advance.
So the list, its grace stamp and the override are global flags, written
server-side against the control-plane clock under an advisory lock. This
branch adds no environment variables.
## The write path, which is live
Stamping generalises to any number of graced flag groups in one
transaction under one lock. That has three consequences a reviewer
should look at directly:
- It closes a real bug. `runOpsMintKind` is an editable control on the
global flags page, and that page previously wrote it with a bare upsert:
no lock, no stamp. An operator flipping mint kind through the UI got an
ungraced flip, so every pod crossed the cutover at a different moment.
Verified against a running instance, before and after.
- A graced group is all-or-nothing. Submitting its primary writes the
group with a fresh stamp; omitting it deletes the primary and its stamp
together, because a stamp left without its primary keeps being served
and would mint into a shard just removed.
- The advisory lock takes the previous id as well as the current one, in
a fixed order, so writers on an older release still serialise during a
rollout. The legacy id can be dropped one release after this ships.
This folds with #4751 rather than replacing it: its `unlockLockedFlags`
rule decides what the sweep may delete, and the graced groups keep their
stamp under the lock. Both sets of tests pass.
## Notes for review
Determinism is a property of the pure core for fixed inputs. The wrapper
supplies the clock, the same split `effectiveMintKind` already uses. A
failed read of the list falls back to the current id format rather than
guessing.
Six flags appear in the admin pages immediately. The two pins are
per-organization, so they render read-only on the global page. The list,
its stamp and the override are deployment-wide, so they render read-only
in the organization dialog.
Nothing bounds the active list against shards that actually exist. That
is safe while nothing mints, but the change that carries a shard key
into an id must land after the shard descriptors bound the list, or
bound it itself.
Builds on
[#4754](https://github.com/triggerdotdev/trigger.dev/pull/4754), which
added the store this contract belongs to.
## Why
Two migrations are moving to Redis in parallel, and execution snapshots
reference completed waitpoints across the boundary between them. If the
record shape is agreed only once both halves are built, the correction
lands mid-rollout: dual-write is live, real keys are in Redis, and
changing the entry format then means two versions of the entry
coexisting plus a migration for whatever was already written. Agreeing
it now, while nothing writes a pointer, makes that same correction a
type edit.
The reserved-and-empty field is the same argument one level down. The
entry format is what dual-write writes, so adding a field to it later
splits the format in two. Reserving it before any write means the format
never changes after writes begin.
## Summary
Adds the type contract for carrying completed waitpoints alongside the
Redis-backed execution-snapshot store: a `{cycleSeq, count}` pointer on
the snapshot entry, the record shape that pointer resolves to, and the
read-time resolver signature. Nothing constructs or reads a pointer yet,
so this is inert on merge.
The record shape has to reproduce
`enhanceExecutionSnapshotWithWaitpoints` field for field, because that
is what the executor consumes. A conformance test runs the real function
against a reference resolver over an exhaustive grid of 6144 input
combinations, derived from every `Waitpoint` column the function reads
rather than hand-picked.
## Design
`completedWaitpoints` is reserved on the entry type and always unset.
`append()` rejects a set value, because the pointer's physical home is
the `<snapshotId>#c` sidecar field rather than the entry JSON. The
append script mints both halves after the client serializes the entry,
and the entry JSON has to stay byte-identical to the Postgres row so the
two can be compared during a dual-write rollout.
Two rules are worth calling out, both found by making the test fail
rather than by reading the code:
* `records` is the authoritative waitpoint set, not `order`. Only batch
waits carry an index, so `order` is empty for a single `triggerAndWait`
while the Postgres join still holds the id. Comparing id sets over
`order` would serve the previous wait cycle's records.
* `deriveFromRun` requires a non-null `completedByTaskRunId`.
`Waitpoint.completedByTaskRun` is `onDelete: SetNull`, so an orphaned
RUN waitpoint keeps its output with no run left to derive from. Those
records carry their output inline instead.
`tsconfig.freeze-test.json` typechecks the conformance test, which the
package build config excludes. Without it, renaming a field in the
frozen type compiles clean and every test stays green, so the literal
assertions in the test would only pin the test's own writer.
## Fixes carried along
Auditing the contract surfaced three defects in the append script, each
with a regression test that fails when the fix is reverted:
* A new wait cycle now clears any `records` left on a reused key. A
`seq` counter lost to eviction can re-mint a `cycleSeq` whose key still
holds another cycle's records, and `order` and `count` are overwritten
together, so the mismatch check could not see the drift.
* A carry-forward now attaches a pointer only if the current keyspace
incarnation actually minted that cycle. The previous key-exists check
adopted a dead incarnation's records under a count that agreed with
them, reporting no mismatch.
* The cycle-key size metric now counts `records`, not only `order`. It
reported 7 bytes for a 20 KB key, so the high-water log could never fire
on the field that grows.
Carries over the self-hosted ClickHouse fix from #4546 by @Leafgard,
whose commits are preserved here, plus follow-up polish. Opened in-repo
because the fork is org-owned, which GitHub's "Allow edits from
maintainers" doesn't cover.
fixes#4343
## What was wrong
Two independent problems in `hosting/docker/clickhouse/`:
1. **The `<profiles>` block never applied.** It sits in `override.xml`,
mounted under `config.d` - but ClickHouse only reads profile settings
from the users config tree. Verified on the pinned image: before this
change `max_block_size` sat at its default `65409` with `changed=0`, so
the advertised low-memory settings had never taken effect at all.
2. **Every ClickHouse system log table was enabled and unbounded.** On a
sub-16GB machine their background merges outgrow the memory cap;
ClickHouse's [low-RAM
guide](https://clickhouse.com/docs/operations/tips) recommends disabling
them. The dev stack already does this - `hosting/docker` never got it.
## What this does
- `clickhouse/override.xml`: disables the high-frequency telemetry
tables, and bounds the ones worth keeping with a config-level `<ttl>` -
`query_log` and `part_log` at 7 days, `error_log` at 30. A config-level
TTL survives log-table recreation, unlike `ALTER ... MODIFY TTL`.
- New `clickhouse/users-override.xml`, mounted at
`users.d/override.xml`: carries the profile settings so they actually
apply, completes the sub-16GB set with `max_threads=1`, and zeroes the
memory/query profilers, whose samples were the main source feeding
`trace_log`.
- `webapp/docker-compose.yml`: adds the `users.d` mount.
## Verification
Ran `clickhouse/clickhouse-server:26.2` with these exact mounts, and
`25.12` to cover the documented 25.8 floor:
- All 9 profile settings report `changed=1`, and a custom
`CLICKHOUSE_USER` inherits them.
- `users.d` merges rather than replaces: the `default` user, its
password, `access_management` and the `readonly` profile all survive, so
the compose healthcheck still passes.
- `remove="1"` is a clean no-op on keys absent from a given version - no
empty section, no accidental table, no startup error - so pinning
`CLICKHOUSE_IMAGE_TAG` to an older supported tag won't crash-loop.
- TTLs land in the real DDL: `TTL event_date + toIntervalDay(7)` /
`(30)`.
- In-place upgrade on a populated volume: clean restart, data preserved,
and ClickHouse lazily renames the pre-existing `query_log`/`error_log`
to `query_log_0`/`error_log_0` as it applies the new retention.
## Notes for review
- **`part_log` is kept (bounded) rather than disabled.** It appears in
neither report behind this change and isn't on ClickHouse's sub-16GB
list, but it's the merge history you'd need to diagnose a recurrence.
Measured at ~0.18 KiB per part event under insert churn - about 10x
cheaper than `text_log` over the same window - so a TTL bounds it rather
than removing it.
- **The profile settings go live for the first time here.** On larger
machines that's a real, intended throughput change: `max_threads=1`,
`max_download_threads=1`, parallel parsing and formatting off.
- **Disabling a log table stops new writes but doesn't delete existing
data.** Reclaiming disk on an existing deployment needs `DROP TABLE
system.<name> SYNC`, including the `*_log_0` leftovers.
## Known gaps, deliberately not in this PR
- The Helm chart carries the same ineffective `<profiles>` block in
`values.yaml` and mounts nothing into `users.d`, so this fix isn't
currently expressible there.
- `background_schedule_pool_log` is enabled by default with no TTL and
is disabled by neither stack.
- The dev stack's disable list has drifted from this one.
- The compose healthcheck still logs a query every 5 seconds.
---------
Co-authored-by: Yann SEGET <yann.seget@actemium.ch>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Extracts every Postgres waitpoint and edge operation out of
`WaitpointSystem` into a `WaitpointCoordinator` seam with one Postgres
implementation, so a different coordination backend can be plugged in
later without any caller changing.
Pure refactor. Zero behaviour change, and zero test-file diffs — the
existing engine corpus is the characterisation test.
## What moved
`WaitpointCoordinator` (`waitpointCoordinator/types.ts`, declared with
`type`) has nine members: `clearRunBlockState`, `readRunBlockState`,
`registerBlocks`, `registerBlocksLockless`, `complete`,
`createDateTimeWaitpoint`, `createManualWaitpoint`,
`mintAssociatedWaitpointData`, `createAssociatedWaitpoint`.
`LegacyPostgresWaitpointCoordinator` implements them against the run-ops
store. Its dependencies are `{ runStore, prisma, logger }` only, so it
structurally cannot reach the run lock, the worker, or the event bus —
orchestration stays in `WaitpointSystem`, which keeps all ten public
signatures, all six `worker.enqueue` sites, the racepoints, the snapshot
transitions, and the event emissions.
Two register methods rather than one with a flag, so "the batch path
issues no extra query" is structural instead of conditional. Both share
one private edge-write helper.
## Six notes for reviewers — please read before "simplifying" any of
these
1. **`nanoid(24)` is called twice with different values on purpose**, in
each create path: once for the upsert `where` key, once for
`create.data`. Hoisting either to a shared constant makes the where-key
match the create-key, turning a guaranteed-miss upsert into a possible
update. In `createManualWaitpoint` both calls plus
`WaitpointId.generate()` stay *inside* the retry loop so each attempt
tries a fresh key.
2. **The two enqueue conditions are deliberately asymmetric.** DATETIME
enqueues `finishWaitpoint` unconditionally after a non-cached create,
with `availableAt: completedAfter`. MANUAL enqueues only when `timeout`
is set. That is existing behaviour, not an oversight. The coordinator
returns a discriminated union on `kind` rather than a boolean so the
enqueue is structurally unreachable on the cached path.
3. **One false clause was deleted from a moved comment.** The old
comment on the full-clear delete claimed the caller's `tx` is not
forwarded. The code does forward it, and `PostgresRunStore` uses `tx ??
this.prisma`, so a single store joins the caller's transaction — only
the routing store strips it. The rest of that comment is unchanged.
4. **The MANUAL timeout enqueue now sits outside the P2002 retry loop.**
Safe because the worker is Redis-backed and cannot raise
`Prisma.PrismaClientKnownRequestError`, so the loop never retried on it.
**If a Postgres-backed enqueue is ever swapped in, that equivalence
breaks silently.**
5. **The coordinator caches `runStore`/`prisma`/`logger` at
construction**, where the old code read `this.$.*` per call. Equivalent
only because nothing reassigns them: one assignment at
`engine/index.ts`, and the `resources` object is a `const` that is never
mutated.
6. **Two comments in other files are now stale and were left alone** —
`engine/index.ts` and `completeWaitpointCrossSeamGuard.test.ts` both
describe routing as the first statement of
`waitpointSystem.completeWaitpoint`. Both tests still pass, because that
guard sits in `index.ts` before the delegation. Left untouched to keep
this diff to three files.
## Preserved verbatim
The `unnest` edge CTE rather than a `Waitpoint` join; the pending count
as a separate statement after the edge write (READ COMMITTED needs its
own snapshot); completion's `findWaitpointOnPrimary` re-read through the
*resolved handle* while the blocked-run fan-out goes back through the
*router*; the residency and colocate hints, with colocation objects
built only in the Postgres arm and the count keeping its `runId`
argument; `ON CONFLICT DO NOTHING` and the `(taskRunId, waitpointId,
batchIndex)` multi-index edge semantics; the unread `batchId` select,
which rides inside two `logger.debug` payloads.
`internal-packages/run-store/` is untouched, so the CTE and the conflict
semantics never moved.
## Verification
| Check | Result |
| --- | --- |
| Engine corpus | 61/61 files, 353 passed, 1 skipped, **0 failed**
(baseline: 352 passed, 1 failed) |
| Test-file diffs | **empty** |
| `run-engine` typecheck | `tsc --noEmit -p tsconfig.build.json` exits 0
|
| `webapp` typecheck | 146 errors on this branch, **146 identical errors
at baseline** — pre-existing, none added |
The webapp typecheck does not pass. The failures are pre-existing
(`PrismaPg` not assignable to `never`; missing `@trigger.dev/rbac`
exports) and the sorted error lists are byte-identical to the merge
base, so this branch adds none — but the criterion is genuinely unmet
and needs a separate fix.
No changeset and no `.server-changes` note: internal refactor with no
user-visible change.
## Follow-ups this surfaced
- The dominant RUN waitpoint is still created outside the seam —
`buildRunAssociatedWaitpoint` now mints through the coordinator, but the
row is inserted nested inside `createRun`/`createFailedRun`. That needs
its own packet before a second backend lands, or the commonest waitpoint
gets split across two of them.
- `clearRunBlockState` overloads opposite outcomes on `undefined` versus
`[]`: `undefined` clears every edge, `[]` clears none. Both callers are
correct today; worth splitting when the file is next touched.
- A stray non-`.sql` entry in `internal-packages/clickhouse/schema/`
breaks every `containerTest` in the repo, because the testcontainers
migration reader `readFile`s every `readdir` entry without filtering
despite a comment claiming it filters. Hit this during setup; unrelated
to this change and left for a separate fix.
## Summary
On a self-hosted instance, saving anything on the global admin feature
flags page also deleted the two read-only flags,
`defaultWorkerInstanceGroupId` and `taskEventRepository`. Losing the
first one leaves deployed runs with no default worker group. Neither
deletion showed up in the confirm dialog, so the flags disappeared
silently.
## Root cause
The page submits only the flags its UI is managing, and strips the
read-only ones from the payload unless "Unlock read-only flags" is
ticked. The action treated every catalog key absent from that payload as
"the admin unset this", and protected the locked keys only when the
instance was managed cloud. Anywhere else, both locked rows fell
straight into the delete sweep.
The protection now keys off what the client says it was editing rather
than off the deployment:
```ts
const canDeleteLocked = params.unlockLockedFlags && !params.isManagedCloud;
...
} else if (canDeleteLocked || !GLOBAL_LOCKED_FLAGS.includes(key)) {
keysToDelete.push(key);
}
```
Exactly one case changes: a locked flag, on a non managed-cloud
instance, with the flags not unlocked, is now kept instead of deleted.
Managed cloud behaviour is bit for bit identical, and ticking the unlock
box still gives a self-hosted instance full control. The write moves
into `replaceGlobalFeatureFlags` so it can be driven directly in tests
against a real Postgres.
Fixes found while reviewing #4547, stacked on that branch so they can be
reviewed on their own and merged into it. One commit per fix.
## Write-path correctness
**Refuse account writes while impersonating.** The five
`dashboardPreferences` writers already no-op for an impersonating admin,
but the three profile writers added next to them did not, and
`requireUserId` returns the impersonated user's id. Both gates now
refuse up front and say so, rather than the preference writers silently
no-opping while the page reports success.
**Preserve unknown keys on a full-blob write.**
`mutateDashboardPreferences` parses the JSON column, hands the result to
a mutator and persists the whole object back. zod strips keys it does
not declare, so a deploy that predates a preference field drops it on
the next write through that path — and
`updateCurrentProjectEnvironmentId` sits on the navigation hot path.
`preserveUnknownKeys` re-attaches them at the write. Note this cannot
help deploys already running, so it makes this the last release able to
strip rather than retroactively protecting the fields added in #4547.
**Scope hidden-sidebar writes to what was shown.** The customize dialog
builds its hidden map from the sections it can see and the write
replaced `hiddenItems` wholesale. The profile page has no org in scope,
so it resolves sections from the most-recently-updated project's org:
confirming there dropped hidden ids belonging to sections that org's
flags exclude. The payload now carries the ids the dialog rendered and
the write only replaces those. Submissions without the list stay
authoritative.
**Consider both addresses when checking email ownership.** The check
only looked at the address the user already had; it now considers the
current and submitted address together, so an org managing either one
governs the change. Validation moved ahead of the check, and
`emailDomainOf` splits on the last `@`.
## Interaction
**Revert unsaved themes, debounce contrast saves.** The theme and
system-theme selects stamp `data-theme` before the write lands. When it
fails, the loader returns the value it always had — so
`useSystemThemeSync`'s effect deps are unchanged and React's vdom diff
sees no change either, and nothing rewrites the attribute. The page kept
rendering a theme that was never stored while the select showed the
stored one. The stored pair is now re-applied explicitly, as the side
menu's switcher already did. The contrast slider is debounced because
Radix commits on every arrow keypress, so a keyboard user crossing the
range fired one write per step.
**Tick More options for themes outside the short list.** The appearance
submenu offers System, Light and Dark; Black and White live on the
profile page. With one of those stored, every row read as unselected.
## Subtraction
**Drop the profile update rate limiter.** It covered one of four paths
that write the same column — `resources.preferences.sidemenu` and
`.favorites` take unlimited authenticated writes and go through the
locked read-modify-write, which is more expensive than the single narrow
`jsonb_set` this capped. It was also what made the contrast slider
unusable by keyboard. If preference writes want limiting, it belongs in
one place covering all of them.
**Resolve email ownership when the dialog opens.** It fans out one SSO
status lookup per organization the user belongs to and ran in the
profile loader on every page view, purely to pick which body the dialog
renders. The action re-derives it before writing either way, so the
check that guards the write now has one call site instead of two.
## Testing
`typecheck --filter webapp` and `lint` clean. New unit tests for
`preserveUnknownKeys`, `mergeHiddenItems` and `emailDomainOf`;
`themePreference`, `mergeHiddenItems` and `ssoManagedIdentity` suites
pass locally (26 tests). The rest of the webapp suite needs
testcontainers and is left to CI.
No changeset or `.server-changes` entry: everything here fixes code on
the parent branch that has not shipped. The one exception worth a
maintainer's call is `mergeHiddenItems`, which also touches the side
menu's own customize path.
## What this does
Rounds out the theme work behind the existing `hasThemeSwitcher` flag.
**Two new themes.** Black and White sit alongside Dark and Light. They
inherit their neighbour's whole token set and only pin their surfaces
flat, so sections are separated by grid lines rather than layered fills.
**`System` is now configurable at both ends.** You choose which theme
the OS light setting lands on (Light or White) and which the dark
setting lands on (Dark or Black).
**Two accessibility toggles.**
- *Stronger colors* — swaps tinted status chips for solid fills, drops
decorative icon accents to monochrome, and darkens chart series that
didn't clear 3:1 on a white plot.
- *Underline links* — underlines body-text links, so an underline always
means the preference is on rather than being a hover style.
**Contrast slider.** Stores a 0–100 position within the active theme's
own range rather than a shared scale, so 35% stays 35% when you switch
themes. Each theme maps it in CSS, which keeps `system` working before
hydration.
**Appearance in the account popover.** A submenu listing the themes with
a check against the current one, plus a link through to the full set on
your profile. Picking one applies immediately rather than waiting for
the write to round-trip.
**Profile page.** Each row now saves on its own — no submit button. Name
and email show their value inline with an edit button; the email row is
read-only when an identity provider owns the address.
**A `/storybook/colors` audit page.** Renders every colour-carrying
pattern in the app once per theme plus once under Stronger colors, and
measures contrast ratios off the live DOM rather than a hard-coded
table, so it can't go stale.
---
## Demo
https://github.com/user-attachments/assets/d56cd4d8-719f-4ec5-a990-e04cdb98def1
---
## Compatibility
The stored preference shape is unchanged (`version: "1"`), and the four
new fields are all optional. The retired `classic` theme falls back to
Dark, whose palette at contrast 0 is what Classic shipped.
One deliberate change worth knowing: the default contrast moves from 50
to 0, so existing users who never touched the slider will see slightly
less contrast than before. That's what makes 0 mean "the base palette".
---
## Testing
Switched between every theme from both the account popover and the
profile page, in the expanded and collapsed rail, checking `data-theme`
follows and survives a reload. Dragged the contrast slider in each theme
and confirmed the percentage label tracks the handle and resnaps if a
save fails. Checked both accessibility toggles across the
`/storybook/colors` page, which is also where the contrast ratios were
read from. Confirmed the Appearance entry stays hidden for a non-admin
while the flag is off.
<!-- conductor-workspace-link -->
---
[Open workspace in
Conductor](https://app.conductor.build/workspace/fee50611-7623-4422-bada-ed1cba317ed1)
---------
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Adds `RedisSnapshotStore` to `@internal/run-store`: a Redis-backed,
append-only store for a run's execution-state log, as an alternative to
keeping that log in Postgres.
Nothing constructs it. No existing code path can reach it, so merging
this changes no behaviour. The store, the wiring that would use it, and
the switch that would enable it are deliberately separate changes.
## Design
Four keys per run, plus one key per wait cycle, all sharing a `{runId}`
hash tag. Every mutation for a run therefore lands in one cluster slot,
and each operation is a single Lua script.
No script mints a key name. Dynamic keys are derived from `KEYS[1]` by
string surgery, because ioredis applies `keyPrefix` only to the KEYS
array: a key built inside Lua would be unprefixed while the client wrote
a prefixed one.
Retention is keyed to run completion. A non-terminal run's keys carry no
expiry at all, since a suspended run can wait indefinitely with nothing
left to refresh a TTL. The terminal transition sets the completion
expiry once, and a write arriving after completion re-applies that same
expiry rather than a live one, so a stale client cannot resurrect a key.
Entry JSON round-trips byte for byte. No script calls `cjson`, and the
values the store assigns itself live in their own hash fields instead of
being patched into the caller's document.
Sizes are observed, never enforced. Entry and cycle-key bytes are
recorded, with a warning above a configurable mark. Nothing rejects,
truncates, or spills.
`append` takes an optional expected-current-snapshot argument. Left out,
it advances the pointer unconditionally, matching the Postgres behaviour
it replaces. Supplied, it advances only on a match and otherwise reports
the conflict without writing.
Covered by 48 tests against a real Redis container, including the
retention transitions, the single-slot guarantee under a key prefix, and
tenant-scoped reads.
Connecting a Vercel project now writes
TRIGGER_AUTOMATIC_SKEW_VERSION_PROTECTION=1
(plain, create-if-absent only - an existing value, including "0", is
never
touched; presence is target-containment aware, branch-scoped records do
not
count, a truncated env listing skips the write). The onboarding wizard
no
longer offers automatic atomic deployments (default off); the settings
row is
labelled Deprecated and enabling it requires confirming a dialog that
points
to task version skew protection and the docs (TRI-13001).
New deployment/version-skew-protection page: the skew problem, the
--external-id primitive and its reuse behaviour, runtime discovery (call
option, configure(), TRIGGER_EXTERNAL_DEPLOYMENT_ID, and the gated
platform/CI/generic commit-SHA variables with the build-time caveat),
the
manual any-platform recipe, waiting/expiry semantics, precedence, and
automatic skew protection on Vercel. Deprecation callouts on the atomic
deployments page and the Vercel integration page; --external-id/--force
added to the CLI deploy reference; redirect from
deployment/vercel-skew-protection so existing webapp links resolve
(TRI-13002).
## Summary
Makes the runs list customizable. A new **Display** control lets you
show, hide, and reorder columns, and add **smart columns** that pull a
single value out of a run's payload, metadata, or output by JSON path
(e.g. `$.failed`, `$.order.total`). Column choices live in the page URL,
so a view can be bookmarked or shared. Applies to the global runs list
and every per-task / scheduled / agent / webhook / error list, which all
share one table.
ID, Task, and Status can be reordered but not hidden. Smart columns are
display-only (no sort or filter, which would defeat the ClickHouse sort
key and cursor).
## How it works
Columns come from a shared registry; the Postgres `select` is derived
from the visible columns, so a run's large payload/output are only
hydrated when a smart column actually references them. All JSON parsing
for smart columns happens client-side, respecting the packet content
type, parsed once per source per row. Offloaded (too-large) values and
paths that aren't present render distinct placeholders rather than
fetching per row. The live poll carries the same sources so smart-column
values update in place.
Scalar columns stay always-selected for now: the shared list presenter
has a fixed output shape consumed by several routes and the live poll,
and narrowing individual scalar fields would add no real query cost
benefit on a single-row read. The select derivation is already
column-driven, so tightening this later is a one-line change.
## Screenshots
<img width="590" height="1028" alt="CleanShot 2026-08-21 at 16 48 17@2x"
src="https://github.com/user-attachments/assets/86b39856-bfcc-47c0-85ed-ee6ccddc3590"
/>
<img width="1924" height="1528" alt="CleanShot 2026-08-21 at 16 48
27@2x"
src="https://github.com/user-attachments/assets/6c766249-6d5b-45be-9330-c6caa75af7f7"
/>
<!-- conductor-workspace-link -->
---
[Open workspace in
Conductor](https://app.conductor.build/workspace/d6911080-2140-4de1-b88a-1b0623593caa)
---------
Co-authored-by: James Ritchie <james@trigger.dev>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
## What
`RoutingRunStore` held two named store fields, `#new` and `#legacy`, and
took its routing policy from the order the statements happened to run
in. It now holds a `Map<ShardKey, RunStore>`, and the three policies
that were implicit are readable data:
- **`#probeOrder`** (`new` → `legacy`) — the sequential probe for a
lookup with no routable id. The first non-null result wins, and the
*last* entry owns the canonical not-found throw.
- **`#precedence`** (`legacy` → `new`) — ascending authority for a
merge, so the highest-authority shard wins a duplicate id.
- **`#idlessRouteShard`** (`new`) and **`#idlessWaitpointShard`**
(`legacy`) — the two id-less defaults, which differ by role and were
previously two unrelated literals in unrelated methods.
The two orders are the **reverse of each other**, which is why they are
separate fields rather than one ordering. Nine sites observe the
result-array order and must iterate `#probeOrder`; five decide a value
by which shard wins a duplicate and must iterate `#precedence`. Five
more sum counts and are order-independent, because addition commutes.
Four helpers absorb the twenty-six hand-written fan-outs —
`#probeFirst`, `#fanOut(order, fn)`, `#fanOutPartitioned`,
`#shardsExcept` — and `#shardKeyOf` replaces the inline
residency-to-store ternaries. `#fanOut` takes its order as an argument
so every call site states which policy it uses.
The constructor keeps its exact options type. No union arm, no `shards`
member: that would loosen the excess-property check and silently retire
the `@ts-expect-error onLegacyRead` lock in the test corpus. N-way
construction is a later change.
## One behaviour change
`findManyTaskRunWaitpoints` merged its edge rows NEW-first into a
last-wins dedupe, so a duplicate edge id resolved to the **legacy** row
— the opposite of the rule the other four merges follow, and the
opposite of what `dedupeEdgesById`'s own comment claimed. No test pinned
it in either direction.
It now resolves NEW-wins, consistent with every sibling merge, and a new
test pins the winner so it cannot drift back silently.
Reaching this case needs one edge id present on both stores at the same
time, with no routable `taskRunId`. That only arises from drain
mirroring. The drain seam is removed (`runOpsStore.test.ts`, "fan-out
spans NEW+LEGACY with no drain seam"), so **no new duplicates can be
created** — but removing the code does not delete rows it previously
wrote, and this class still carries comments treating mirrored rows as a
live data condition. Whether any historical duplicate edge rows persist
is an empirical question about production data, not something this diff
settles.
If such a row is hit, the two copies either agree — in which case the
winner is immaterial — or they have diverged, in which case NEW is the
authoritative copy by the router's own precedence rule. So the corrected
behaviour is at least as correct as the old one in every reachable case.
Everything else is behaviour-preserving.
## How it was verified
- **`internal-packages/run-store`: 69 files, 379 tests pass.** The
corpus is the regression gate for this refactor. 67 of the 68
pre-existing test files are byte-identical; the one that differs
(`runOpsStore.mixedResidency.test.ts`) changes only `//` comments.
- **`internal-packages/run-engine`: 12 files, 69 tests pass** — every
file that constructs the router, exercised at runtime.
- **The `@ts-expect-error onLegacyRead` lock still fires.**
`tsconfig.build.json` excludes `*.test.ts`, so a green typecheck does
not cover it. A scratch probe confirmed `tsc` still reports `TS2353` for
`onLegacyRead` and no error for the three real options.
- **All 48 construction sites outside the package compile unchanged.**
`tsconfig.check.json` also excludes `*.test.ts`, so the 25 webapp test
files were checked with the test exclusion dropped and compared against
the same check on the base commit: 614 errors before, 614 after, zero
present in one and not the other. Those 614 are pre-existing in
never-typechecked test files.
- `typecheck` passes for `run-store`, `run-engine` and `webapp`. `knip`
reports nothing in `run-store`.
## Also
Refreshes the sixteen stale `runOpsStore.ts` line references in
`runOpsStore.mixedResidency.test.ts`, each verified against the symbol
it names.
## Notes for the reviewer
- The riskiest possible mistake in this diff is a fan-out passing the
wrong order — the compiler cannot catch it, because both orders are
`readonly ShardKey[]`. The five `#precedence` sites are `#findRunsOpen`,
`findRunsByIdempotencyKeys`, `#collectManyWaitpoints`,
`findManyTaskRunWaitpoints` and `findManyWaitpointTags`. Those are the
lines worth the closest read.
- Four sites previously derived "the other store" by object identity
(`home === this.#new ? ...`). They now compare keys. The two are
equivalent: in single-database mode both keys map to the same store
object, and when the stores are distinct, identity and key comparison
agree.
- No changeset and no `.server-changes` note: the package is internal
and the one behaviour change is unreachable in production, so a release
note would tell a user nothing.
- Two CI checks fail for reasons that predate this branch and reproduce
on the base commit: `lint` (~16 unknown `react/*` rules make
`.oxlintrc.json` fail to parse, which disables oxlint entirely —
including the two `trigger-runops` fences) and `knip` (`unrun`, an
unused devDependency on the default branch). Both want their own fix.
## Summary
Adds a second generation of run-ops id, plus the resolver that reads a
store key straight out of an id. A gen-2 id keeps the existing
26-character layout, but the character at index 24 becomes a routing
shard key instead of a region code, and the version character at index
25 becomes `"2"`. Nothing mints gen-2 ids yet, so this is inert on
merge.
## Design
The version character is a single character, so the gen-1 and gen-2
shape checks can never both match. That is what makes the two
generations provably disjoint rather than disjoint by convention.
```ts
resolveShard(id) // gen-2 body -> its shard key, [a-z0-9]
// gen-1 v1 body -> "new"
// anything else -> "legacy"
```
`resolveShard` is total: it returns a key for any input string,
including an empty or malformed one, and never throws.
`classifyResidency` keeps its signature and its two values, and now
reports gen-2 ids as part of the dedicated family, so existing consumers
of that boolean are unaffected.
The body stays 26 characters rather than 27 deliberately. The older
27-character format is still in the wild and has to keep resolving to
legacy, and a longer gen-2 shape would need probabilistic disambiguation
against it. A rare misroute is not an acceptable property for a routing
key.
The one behavior change is that a 26-character body ending in `"2"` now
routes by its shard key instead of falling back to legacy. Two test
assertions pinned the old result and are updated here. A repository-wide
search confirms they are the only two of their kind.
Verified against the full run-store corpus (68 files, 370 tests) with no
test-file changes there, plus the run-engine residency and waitpoint
suites. No changeset: the new surface has no caller, so a version bump
would tell a user nothing.
Cuts CPU on the `engine/v1/worker-actions/*` routes a managed supervisor
calls, and adds the benchmark harness the numbers come from.
Measured on a local stack: **on-CPU per completed run 9.07ms → 6.59ms
(−27%)**, busy fraction 45.6% → 33.8%, with every worker-action p50 down
23–27%. Load was 5,000 runs / 24 virtual supervisors / 90s window /
30,120 requests / 0 errors.
Query-count work from the same investigation is deliberately **not**
here — it will follow as a separate PR.
## The three changes
**1. Split the event-loop monitor in two (~14% of on-CPU, plus ~5pp of
GC).**
`eventLoopMonitor.server.ts` installs a global `async_hooks` hook:
`init` writes a `Map` entry for *every* async resource the process
creates, `before` calls `process.hrtime()` and `context.active()` on
every one. Enabling any async hook also puts V8 on the slow path for
promise instrumentation process-wide. `EVENT_LOOP_MONITOR_ENABLED`
defaulted to `"1"`, so this was the shipping configuration.
The blocked-loop detector is now opt-in (`EVENT_LOOP_MONITOR_ENABLED`,
default `0`). The event-loop *utilization* gauge — a single interval
timer with no per-request cost — moves to its own flag
(`EVENT_LOOP_UTILIZATION_MONITOR_ENABLED`, default `1`) and stays on, so
the useful half survives without the expensive half.
A/B under identical load:
| | monitor on | monitor off | change |
|---|---|---|---|
| on-CPU per run | 9.08ms | 7.25ms | −20% |
| GC self time | 9.80% | 5.05% | −4.75pp |
| dequeue p50 | 76.6ms | 62.8ms | −18% |
| attempts/start p50 | 56.3ms | 43.5ms | −23% |
**2. Bucket route matching by first static path segment (10.4% → 3.9% of
on-CPU).**
`patches/@remix-run__router@1.23.3.patch` already memoized flattened
branches and compiled path regexes. What remained was the linear scan:
`matchRouteBranch` walked the ranked branch list calling `matchPath` per
branch across 521 route files, so every worker-action request paid a
scan proportional to the whole route table.
Branches are now indexed by their lowercased leading segment, with one
always-considered list for branches whose leading segment is dynamic,
splat or optional (and for root/pathless paths). A request walks only
its own bucket merged with that list. Route-matching self time dropped
64% (3.6s → 1.3s over a 90s window).
Ordering is preserved exactly: both lists hold indexes into the already
rank-sorted branch array and are walked in ascending-index order, so the
first match found is the same branch the full scan would have found.
Bucketing lowercases on both sides, so case-insensitive matching still
resolves and `caseSensitive: true` routes are still rejected by
`matchPath` itself. A pathname whose own leading segment can't be
bucketed falls back to the full scan.
Verified equivalent to the unpatched matcher over 20,050 pathnames
(literal, dynamic, splat, optional, case variants, basenames,
percent-encoded) with zero mismatches.
`apps/webapp/test/routeMatchingPatch.test.ts` pins the matching
semantics rather than the optimisation, so it still passes without the
patch.
**3. Demote per-heartbeat and per-dequeue `info` logs to `debug`.**
These are the two highest-rate engine calls and each wrote a synchronous
structured log line on every request. Synchronous `console` writes can
block the loop when stdout backs up, which costs more than the ~1.3% CPU
share suggests.
## The harness
Two benchmarks, neither in the default suite (they run for minutes,
attach the V8 profiler, and report numbers rather than assert on them).
See `apps/webapp/test/bench/README.md`.
- `apps/webapp/test/bench/engineHttp.bench.test.ts` — spawns a real
webapp against throwaway Postgres/Redis containers, seeds a production
environment with a promoted managed deployment, and drives a closed-loop
supervisor pool through the full lifecycle. Profiling runs over CDP
rather than `--cpu-prof` so it covers only the measured window instead
of being swamped by boot, and `performance.eventLoopUtilization()` is
sampled *inside* the webapp process.
-
`internal-packages/run-engine/src/engine/bench/runEngineLifecycle.bench.test.ts`
— drives `RunEngine` directly, profiling enqueue and lifecycle
separately so engine cost isn't mixed with request-stack overhead.
- `apps/webapp/test/bench/analyzeProfile.ts` — dependency-free
`.cpuprofile` analyzer that symbolicates through the build's source maps
and ranks CPU by package, self time and total time. Percentages are
shares of on-CPU time (V8's `(idle)`/`(program)` excluded).
`startWebapp` gains `overrideEnv`, applied after the worker-disable
defaults, so the HTTP bench can re-enable the run engine worker that
drains the master queue into the worker queues a supervisor dequeues
from.
The local OTel collector gains a traces pipeline. It only defined a
metrics pipeline, so pointing `INTERNAL_OTEL_TRACE_EXPORTER_URL` at it
locally failed and the webapp silently fell back to the console span
logger.
## Configuration
For operators upgrading:
- `EVENT_LOOP_MONITOR_ENABLED` (now defaults to `0`) — the
per-async-resource blocked-loop detector. Set to `1` to restore the
previous behaviour and keep emitting `event-loop-blocked` spans.
- `EVENT_LOOP_UTILIZATION_MONITOR_ENABLED` (new, defaults to `1`) — the
`nodejs.event_loop.utilization` gauge. Unchanged in behaviour; it just
has its own flag now so it survives turning the detector off.
## Notes for review
- `pnpm-lock.yaml` changes only because the router patch content
changed, which changes its patch hash.
- One thing the profile ruled out: with a real OTLP collector receiving
spans, tracing costs ~1.7% of on-CPU at 100% sampling and ~0.8% at the
production rate. Span shipping is not a hidden cost, so nothing here
touches it.
- Caveats on the numbers: a laptop, not production hardware, so DB and
Redis *latency* are unrepresentative (client-side CPU is what's ranked);
single webapp process; throughput varies ~5% run to run, which is why
the claims rest on on-CPU per run rather than req/s.
## Verification
- 20,050-pathname router equivalence check vs the unpatched matcher,
zero mismatches
- `apps/webapp/test/routeMatchingPatch.test.ts` (12 cases) passes
- webapp e2e smoke suite (68 tests) passes through the patched router
- run-engine suites covering the snapshot/attempt paths pass
- `typecheck`, `format`, `lint`, `knip` clean
Two small tweaks to the `Switch` primitive, so every variant and call
site picks them up:
1. **Track is 2px shorter.** `large` 44 → 42px, `medium` 32 → 30px,
`small` 24 → 22px. The checked thumb travel drops by the same 2px so the
thumb stays flush at both ends.
2. **Holding the switch down stretches the thumb into an oval** pointing
the way it's about to travel — rightwards when off, leftwards when on.
Pure CSS via `group-active:`, no new state or handlers.
The thumb's `transition` shorthand doesn't cover `width`, so it's now
`transition-[translate,width,background-color]` (same 150ms
duration/easing as before). `size-N` on the thumb became `h-N w-N` so
the press rule overrides the same `width` utility.
Verified in headless Chrome across all five variants in both states:
correct widths at rest, thumb flush at both ends, stretch grows the
right direction, and no overflow of the track.
<img width="266" height="108" alt="CleanShot 2026-08-21 at 10 16 14"
src="https://github.com/user-attachments/assets/ee95a399-0a40-48c4-a325-a1166b3bd88a"
/>
🤖 Generated with [Claude Code](https://claude.com/claude-code)
<!-- conductor-workspace-link -->
---
[Open workspace in
Conductor](https://app.conductor.build/workspace/c1ce8d0f-9ed2-4fbc-8084-a3989484cc53)
---------
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
## Summary
The platform notifications admin page can now save a notification as a
draft without committing to a schedule, then publish it later by
entering start and end dates. Drafts stay hidden from the webapp panel,
the CLI, and the "What's new" changelog until they are published.
## Design
A draft is an `isDraft` flag on `PlatformNotification`, not nullable
dates, so the existing index and every read query stay intact. All three
reader queries filter on the flag, so a draft can never surface
regardless of its placeholder dates. Publishing writes the real start
and end dates and clears the flag; the publish dialog validates the
range and shows inline errors. Editing a draft keeps it a draft, with
the schedule fields hidden until publish.
Also folds in a small tweak: the "Send preview to me" test button now
appears when editing a notification, not just when creating one.
## Summary
The Queue Metrics dashboard UI is gated by a per-org feature flag, so
there was no way to look at it for a real org without turning it on for
every member of that org. An admin impersonating into an org now sees
the metrics UI there regardless of the flag, so it can be checked
against real data before anyone else in the org sees it.
Nothing changes for a normal session: a member of an org whose flag is
off still gets the classic Queues page, and the gated sub-routes still
404.
## Design
The gate had no request and only resolved the org flag. It now takes the
request and resolves impersonation itself, rather than each caller
computing a boolean and passing it in, so the rule lives in one place
and a new call site cannot forget it. Seven call sites gate on this,
which is exactly why.
Two things narrow the bypass:
- It keys on **impersonation**, not `user.admin`. Impersonation is
scoped to one org and is deliberate; keying on admin status would
silently hand every admin the preview in their own day-to-day orgs.
- It yields to the **view-as-user** toggle. That toggle exists so an
impersonating admin can see what the member sees, and unreleased UI
leaking through it would make it lie. Suppressing a read-only view there
stays inside the display-only contract in `hasAdminDisplayAccess` (added
in #4421).
The bypass also stays behind the gate's existing org-membership lookup.
Since the acting user id is the impersonation target, that lookup is
what keeps the preview confined to the org actually being impersonated
into.
Verified end-to-end against a running instance across the matrix: member
with the flag off gets the classic view and 404s; the same org under
impersonation gets the metrics view and a 200; flipping view-as-user
returns it to the member's exact experience and back; and the flag-on
path is unchanged. An admin who is merely a member, not impersonating,
still gets the classic view.
One thing worth flagging: a few route comments say that with the flag
off no metrics reads fire. That remains true for every member session
and for the org as a whole, but an admin actively previewing does
exercise that org's real Redis and ClickHouse reads. That is inherent to
previewing, and bounded to one admin session.
## Summary
Follow-up to #4738. Splits the dashboard agent's base URL into two: the
instance that hosts the agent project (used for sessions), and the
instance the agent acts against as the user (used by its read-tools).
#4738 only needed the first, but moved the second along with it, which
breaks the tools when the agent runs on a different instance than the
webapp.
## Root cause
The agent's read-tools call the API as the logged-in user via a
delegated user-actor token. The webapp signs that token with its own
`SESSION_SECRET`, scoped to its own `userId` and `environmentId`, so it
can only be verified by, and only resolves the user's data on, that same
instance. #4738 routed the injected `apiOrigin` those tools use to the
agent's host instance, so the token no longer verifies and the data
isn't there.
## Fix
`dashboardAgentApiOrigin()` stays the agent's host instance (sessions,
task triggers, realtime, the `in` forward). A new
`dashboardAgentUserApiOrigin()` returns the webapp's own origin
(`API_ORIGIN ?? APP_ORIGIN`) and is injected into the run metadata the
tools use. Same-instance deployments resolve both to the same host, so
behavior is unchanged there.
## Summary
Lets the dashboard agent point at a specific Trigger instance instead of
assuming it runs on the same instance as the webapp. Adds an optional
`DASHBOARD_AGENT_BASE_URL`; when unset it falls back to the SDK default.
## Root cause
The agent's session start, token mint, head start, in-proxy and the
client transport all built the agent's base URL from the webapp's own
origin (`API_ORIGIN ?? APP_ORIGIN`). That only holds when the agent
project runs on the same instance as the webapp. When it runs elsewhere,
`DASHBOARD_AGENT_SECRET_KEY` belongs to that other instance, so the
webapp's own API rejects it with an "Invalid API key" and the chat can't
start.
## Fix
`dashboardAgentApiOrigin()` now returns `DASHBOARD_AGENT_BASE_URL` or
the SDK default, never the webapp origin. A concrete default (rather
than an unset value) keeps it independent of `TRIGGER_API_URL`, which a
webapp may point at a different host. Every server call site already
routes through that helper; the client transport reads the value from
the root loader via a new `useDashboardAgentBaseUrl` hook.
<!-- ccr-slack-attribution -->
_Requested by **Matt Aitken** · [Slack
thread](https://triggerdotdev.slack.com/archives/C045W9WM3E1/p1786741966214949?thread_ts=1786741966.214949&cid=C045W9WM3E1)_
`idempotencyKeys.reset()` now honours an explicitly passed `scope` even
when the key material happens to be 64 characters long.
**Before:** `resetIdempotencyKey` treated *any* 64-character string as
an already-computed hash and sent it to the API verbatim. That
short-circuit ran before the scope logic, so if your key material is
itself a 64-character digest (a common pattern when you hash your own
dedup identity) the `scope` you passed was silently discarded and the
un-hashed material went on the wire. The server stores the hash, so the
reset matched no run and returned 404 every single time. Key material of
any other length worked fine, which made this look arbitrary.
**After:** a 64-character key with an explicit `scope` is sent verbatim
first and, only when that attempt comes back a definitive not-found,
retried as the derived scope hash. Every call that worked before behaves
identically, and the previously impossible case now resolves on the
fallback.
## How
A 64-character string is forwarded unchanged, exactly as before, when:
- the idempotency key catalog recognises it (it came from
`idempotencyKeys.create()` in this process), or
- no `scope` was passed, so there is nothing to derive a hash from, or
- the scope hash cannot be derived (e.g. `scope: "run"` outside a task
context with no `parentRunId`).
Otherwise the key is ambiguous: it may be raw material the caller wants
hashed with the scope, or it may already be the stored hash. Reset sends
the verbatim value first because that is what every previous version
sent, so anything that resolved before still resolves with the same
single request, the same target run, and the same errors. The derived
hash is the new behaviour, so it only runs once the verbatim attempt has
failed with a 404, a definitive "no run under this key". Any other error
(a 503, a connection error) leaves the verbatim key's state unknown, and
resetting a different key on unknown state would be an untargeted write
the caller never asked for, so those errors surface unchanged. That has
an honest cost: when the endpoint answers 503 for a miss it cannot
confirm, the caller sees the 503 and retries rather than silently
falling through to the derived key. When both attempts miss, the
verbatim attempt's 404 is surfaced, again matching what previous
versions threw.
A side benefit of this order: a key from `idempotencyKeys.create()`
reset with a `scope` from a cold process resolves in a single request,
because the created key is itself the stored value.
`isIdempotencyKey` is deliberately left alone: it applies the same
length rule on the trigger path, but it is self-consistent there, and
changing it would invalidate already-stored keys.
The `attachedOptions?.key` / `attachedOptions?.scope` fallbacks below
the old guard were unreachable (every catalog entry is a 64-character
digest, so it always hit the short-circuit first) and re-deriving from
them produces the identical hash anyway. They are removed rather than
left as dead code.
---
## ✅ Checklist
- [x] I have followed every step in the [contributing
guide](https://github.com/triggerdotdev/trigger.dev/blob/main/CONTRIBUTING.md)
- [x] The PR title follows the convention.
- [x] I ran and tested the code works
---
## Testing
Tests in `packages/core/src/v3/idempotencyKeys.test.ts` drive the real
`resetIdempotencyKey` against a local HTTP server and assert on the
exact values that reach the wire, in order. Nothing is mocked. They
cover:
- 64-character material + explicit `scope` derives the global- and
run-scoped hash once the verbatim key misses (fails without this change)
- the verbatim key wins when runs exist under both the verbatim value
and the derived hash, so the pre-existing target is preserved
- keys from `idempotencyKeys.create()` are forwarded unchanged: catalog
hit, no scope, and scope with a cold catalog (the last now a single
request)
- a transient failure of the verbatim attempt surfaces its error without
ever touching the derived key
- error surfacing: a double miss reports the key the caller passed, and
a non-404 from the fallback is not swallowed
- ordinary short material is still hashed, and underivable run/attempt
scopes still send a 64-character key verbatim while still throwing for
shorter material
```
pnpm run test ./src/v3/idempotencyKeys.test.ts --run # 18 passed
pnpm run build --filter @trigger.dev/core # clean
pnpm run format && pnpm run lint # clean
```
---
## Changelog
`idempotencyKeys.reset()` now works when your idempotency key is itself
64 characters long. Previously any 64-character key was assumed to be
already hashed, so passing one along with a `scope` silently ignored the
scope and the reset never found a matching run.
---
## Follow-ups (not in this PR)
- `docs/idempotency.mdx` describes the `idempotencyKey` parameter of
`reset()` as "the 64-character hash string" in one place while showing
raw material plus `{ scope: "global" }` a few lines later. Worth
reconciling.
- No surface currently exposes the stored hash that the reset endpoint
matches on: `ctx.run.idempotencyKey`, the run page and the
`idempotency_key` query column all show the user-provided key. That is
what leads people to send a value reset cannot match.
---------
Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: Matt Aitken <matt@mattaitken.com>
## Summary
Enforces `react/react-compiler` as an error for the webapp now that all
reported compiler diagnostics are fixed or narrowly scoped. Removes the
unused lazy-ref helper made obsolete by the ref initialization cleanup.
## Summary
Scopes React Compiler diagnostics to route statements where refs
intentionally coordinate virtualized views, live reload state, transport
lifecycles, and deferred callbacks. Other compiler diagnostics remain
active in those routes.
## Summary
Scopes React Compiler diagnostics to component and hook statements where
refs intentionally coordinate editors, animations, polling, deferred
callbacks, and other imperative integrations. Other compiler diagnostics
remain active in those components.
## Summary
Replaces render-time ref initialization with lazy state for frozen form
defaults, the tooltip's virtual positioning element, and the side menu's
first-paint visuals. Editable alert fields now update immutable state
snapshots.
## Summary
Scopes React Compiler diagnostics for component and hook effects that
intentionally synchronize with navigation, submissions, browser APIs,
streams, timers, or authoritative server values. Each suppression stays
on the reported synchronization call rather than disabling analysis for
the component.
## Summary
Derives controlled tab, tag, and checkbox values directly during render
instead of copying them through effects. Modal drafts now reset from
their open event, and the route-backed alert dialog renders open
immediately without a mount-time state update.
## Summary
Scopes state synchronization that intentionally resets editable drafts
from authoritative server values, deployment state, or programmatic
filter changes. These values cannot be derived during render without
removing user control between resets.
## Summary
Removes manual memoization where derived values are already rebuilt each
render, narrows the dashboard watch callback to a stable chat
identifier, and scopes two intentional memoization patterns that protect
local edits and serialized synchronization.
## Summary
Makes stable dashboard history refs explicit memo inputs and scopes the
remaining compiler diagnostics to callbacks whose local handlers or
lifetime-stable values cannot be represented accurately in dependency
arrays.
## Summary
Captures chat-history age when the menu opens so rerenders cannot change
labels mid-view. The waitpoint deadline form also reuses one intentional
wall-clock snapshot for all calculations in a render.
## Summary
Records when live metric responses arrive and uses that timestamp to
evaluate gauge freshness and waiting duration. Cached or failed
responses remain untrusted until revalidated, while rendered values stay
stable between polling updates.
## Summary
Derives session and API key expiry states from a timestamp captured by
each route loader. Every status on a page now uses one consistent point
in time instead of changing according to when an individual component
rerenders.
## Summary
Uses explicit bucket timestamps when rendering usage charts instead of
anchoring missing timestamps to the current render time. Tooltips now
remain stable across rerenders, and examples use a deterministic
timestamp.
## Summary
Keeps render inputs and shared regular expressions immutable. Grouped
selects now compute each section's shortcut offset directly from
preceding sections, which also makes numeric shortcuts follow the
displayed item order reliably.
## Summary
Calls dashboard hooks directly instead of passing them as ordinary
callback values, and subscribes to optional Ariakit stores through an
unconditional hook. This keeps hook ordering stable while preserving the
existing behavior when a provider is absent.
## Summary
Adds targeted lint suppressions for components built around libraries
that React Compiler intentionally declines to memoize, plus one
unsupported function-reference pattern. Each suppression is scoped to
the affected component so other compiler diagnostics remain actionable.
## Summary
Enables exhaustive React Hook dependency checking and resolves the
existing violations across the dashboard and React hooks package.
Effects and callbacks now track current values without introducing
request, subscription, or render loops.
## Design
Dependencies are included directly when the hook lifecycle should follow
them. Timers, Remix fetchers, and realtime subscriptions use stable
callbacks or latest-value refs where restarting work would change
behavior.
Unnecessary memoization was removed where ordinary derivation is
clearer. Full lint and typechecks for the webapp and React hooks package
pass.
## Summary
`goose up` against `internal-packages/clickhouse/schema` panics on
`main` today, so ClickHouse migrations cannot be applied from a fresh
checkout. Renumbering the external deployment id migration from 040 to
041 clears it.
## Root cause
Two migrations claim version 40.
[#4615](https://github.com/triggerdotdev/trigger.dev/pull/4615) added
`040_create_task_events_search_v2.sql`, and
[#4661](https://github.com/triggerdotdev/trigger.dev/pull/4661) added
`040_add_task_runs_v2_external_deployment_id.sql` a day later. #4661 was
opened before #4615 merged, so 040 was genuinely free at branch time,
and because the two files have different names there is no textual
conflict for git or a rebase to surface. Both merged green, and no
workflow in this repo runs `goose`, so the collision only shows up the
first time someone actually migrates.
goose parses the numeric filename prefix as the version and refuses
duplicates:
```
panic: goose: duplicate version 40 detected:
.../040_create_task_events_search_v2.sql
.../040_add_task_runs_v2_external_deployment_id.sql
```
It aborts while collecting the directory, before executing any SQL, so
nothing was half applied and there is no migration state to repair.
This migration gets renumbered rather than the `task_events_search_v2`
one because goose keys on the version number and not the filename:
version 40 is already recorded wherever 040 has been applied, so
renaming that file would re-run an applied migration.
Verified with a full `goose up` against ClickHouse 26.2.19.43 (the image
pinned in `internal-packages/testcontainers`): migrations apply cleanly
through version 41, and `task_runs_v2.external_deployment_id` lands as
`String DEFAULT ''`.
## Summary
Speeds up webapp test jobs by balancing measured work across runners,
reducing repeated container setup, and ensuring test workers release
shutdown resources promptly. Unit tests run across 24 duration-aware
shards, while E2E tests run across two balanced shards.
## Design
`RunEngine` shutdown now closes processing resources before support
resources, continues cleanup if one close fails, and reuses one shutdown
promise for concurrent callers. Redis workers clear completed shutdown
deadlines so finished tests no longer wait on idle timers.
Container-heavy suites are split only where it improves parallelism, and
repeated replication and engine fixtures are consolidated where one
end-to-end case provides coverage. Timing weights are refreshed for all
affected files.
Dependency installation overlaps container pulls, and both workflows use
WarpBuild's Node setup action.
<!-- ccr-slack-attribution -->
_Requested by **Iss** · [Slack
thread](https://triggerdotdev.slack.com/archives/C045W9WM3E1/p1787161814493949)_
**Before:** archiving a branch dropped the query string on the way back
to the branches list, so the list reset to page 1. Working down a long
list meant re-navigating to the page you were on after every archive.
**After:** you land back on the exact page you archived from, with
`page`, `search` and `showArchived` intact.
The archive action now redirects to the page the request came from
instead of rebuilding a bare branches path.
## How
The archive dialog already submits the page it was opened from as a
hidden `redirectPath` field (`${location.pathname}${location.search}`),
and the failure path already redirected to it — only the success path
ignored it and rebuilt the path with `branchesPath`/`branchesDevPath`,
which have no query string. Both paths now redirect to the submitted
path, run through the existing `sanitizeRedirectPath` helper to keep the
redirect same-origin (the same idiom used by
`resources.batches.$batchId.check-completion`).
## ✅ Checklist
- [x] I have followed every step in the [contributing
guide](https://github.com/triggerdotdev/trigger.dev/blob/main/CONTRIBUTING.md)
- [x] The PR title follows the convention.
- [x] I ran and tested the code works
---
## Testing
Three files change:
- `apps/webapp/app/routes/resources.branches.archive.tsx` — the fix.
- `apps/webapp/test/archiveBranchRedirect.test.ts` — new test that
drives the archive action and asserts the redirect `Location`: the query
string survives on both success and failure, and an off-origin
`redirectPath` falls back to `/`. Reverting the fix makes two of the
three cases fail, so the test covers the regression.
- `.server-changes/archive-branch-keeps-list-page.md` — release-note
entry, since this is a user-facing server-only change.
Also ran `pnpm run typecheck` and `oxlint` for `apps/webapp` — both
clean.
---
## Changelog
Archiving a branch now returns you to the same page of the branches list
instead of resetting it to page 1.
---
## Screenshots
_None — no visual change._
---------
Co-authored-by: Claude <noreply@anthropic.com>
## Problem
Every merge to main touching the agent queued a gated `staging`+`prod`
deploy that sat `pending` on a reviewer approval nobody grants
routinely. Because the gated runs never completed, they never drained
the concurrency queue and cancelled each other, so the Actions tab
filled with never-completing runs and the agent only ever actually
deployed via a manual dispatch + approval.
The reviewer gate bought nothing here: the agent deploys with
`--skip-promotion`, so a deploy lands **dormant** and nothing goes live
until the consuming webapp flips `DASHBOARD_AGENT_VERSION`. Promotion is
already a deliberate act (the env-var flip); gating the dormant deploy
on top of that just created the pile-up.
## Change
- **Remove the reviewer gate** by dropping the required-reviewers rule
on the `dashboard-agent-*` environments (repo-settings change, done).
The `environment:` key **stays** so the per-environment scoped deploy
token still resolves — no secret migration.
- **`workflow_dispatch` `ref` input** — deploy a specific commit SHA,
branch, or tag; defaults to the ref the run launches from. Checkout uses
`github.event.inputs.ref || github.sha`.
- **Require the ref to be an ancestor of `main`.** Constrains which
commit gets deployed to merged code only. A push is always main's tip
(passes trivially); a dispatched unmerged ref is rejected before the
deploy step. Because an explicit `ref:` checkout doesn't create
remote-tracking branches, `origin/main` is fetched explicitly before
`git merge-base --is-ancestor`.
- **`cancel-in-progress: false`** (kept). Cancelling the runner wouldn't
stop the remote build (it finishes server-side), and a superseding
concurrent deploy would race the same project's indexer. With the gate
gone, deploys are short, so a brief queue can't pile up.
- `max-parallel: 1` stays (parallel deploys of the same project race at
the indexer).
## Owner actions (repo settings — not in the diff)
1. **Remove required-reviewers** on `dashboard-agent-staging` and
`dashboard-agent-prod` — done.
2. **Add a deployment branch policy** on both environments restricting
deployments to `main`. This is the authoritative token guard:
`workflow_dispatch` runs the workflow file from the selected ref, so the
in-file ancestor check alone can't protect `TRIGGER_ACCESS_TOKEN` (a
branch could edit the check out). GitHub enforces the branch policy
server-side against `GITHUB_REF` regardless of file contents. With it in
place, the workflow only runs (and the token is only exposed) when
dispatched from `main`, and the in-file check then constrains the
independent `ref` input to merged commits.
## Pile-up root cause
The stacking was caused by the **reviewer gate** (runs waited forever,
so the queue never drained), not by `cancel-in-progress`. Removing the
gate is what fixes it; `cancel-in-progress` stays `false`.
Two defects that surface when a run parked on an external deployment id
gets pushed by a debounce key. Both were reproduced against a local
instance before being fixed.
## 1. The run is expired before it is due
```
now | status | statusReason | delayUntil | expiredAt
13:57:06 | EXPIRED | EXTERNAL_DEPLOYMENT_NOT_FOUND | 14:01:37 | 13:57:02
```
Killed 4m35s before its own scheduled start, blaming a missing
deployment.
**Why.** The park deadline is armed **once**, when the run is first
parked, from `max(now, delayUntil) + deadline`. Debounce pushes
`delayUntil` out afterwards and nothing re-arms it:
- `rescheduleDelayedRun` reschedules `enqueueDelayedRun:<id>`, not
`expireParkedExternalDeploymentRun:<id>`
- the redis-worker reschedule is an update-only `ZADD … XX`, and a
parked run has no `enqueueDelayedRun` job, so that call is a silent
no-op
Repeat triggers on one key walk `delayUntil` away from a deadline that
no longer moves. Once it crosses, the run dies while parked and not yet
due.
**Fix.** The expiry job already loads `delayUntil`, so it re-arms from
the current value and returns instead of expiring a run that is not due.
The guard lives in the expiry job rather than the debounce path
deliberately: it covers **every** caller that moves `delayUntil`, so a
future call site can't reintroduce this by forgetting to re-arm. It
stays bounded by the debounce max-duration contract, so a hot key can't
postpone expiry indefinitely.
## 2. The run reports itself as delayed while it is parked
```
RUN_CREATED | PENDING_VERSION | Run is waiting for a deployment of 'debounce-test-2'
DELAYED | DELAYED | Delayed run was rescheduled to a future date ← after one debounce push
```
The row stays `PENDING_VERSION`; the latest snapshot claims `DELAYED`,
so the run page describes a parked run as delayed. Happens on the
*first* push.
**Fix.** `rescheduleRun` hardcoded `DELAYED`/`DELAYED`. The snapshot
statuses are now supplied by the caller and **default to `DELAYED`**, so
the ordinary delayed path is byte-identical, and `rescheduleDelayedRun`
passes the parked statuses through when the run is parked.
## Reproducing
Repeated triggers on one debounce key against an id that hasn't landed:
```bash
curl … -d '{"options":{"externalDeploymentId":"x","debounce":{"key":"k","delay":"5m"}}}'
```
Three triggers correctly fold into one parked run; the defects show up
on the pushes.
## Testing
Two tests, each verified red before green and failing alone:
- a run whose delay was pushed past the deadline stays `PENDING_VERSION`
instead of expiring
- a debounce push on a parked run leaves a
`RUN_CREATED`/`PENDING_VERSION` snapshot, not `DELAYED`
`56 passed` across parking, pendingVersion, delayedRunSystem and
debounce; `43 passed` in `PostgresRunStore`. Typecheck, lint, format
clean.
## Notes
- Stacks on #4665, so it lands after the whole external-deployment-id
series.
- No changeset: this fixes unreleased behaviour introduced by the stack
below it, so no user has seen it.
- Both found by Devin's review on #4664, and both confirmed end to end
on a local instance before fixing.
Deployments page: an always-visible External ID column after Deployed
by, and an External ID row in the deployment inspector under Worker
type, both showing an en dash when a deploy carried no id. The Vercel
Linked column now renders before Git, still only when a Vercel
integration is connected. Also corrects the blank-row colSpan, which was
already off by one before this column existed.
Run inspector: an External deployment ID row between Version and SDK
version, read from the run annotations, so an operator can see which id
a run was pinned to - including a run that expired before its deployment
ever arrived, where the locked version is empty but the id is the whole
story. Buffered runs read the id from the same annotations rather than
reporting none.
Long ids are head-truncated with the full value behind the copy button:
a commit SHA is meaningful in its prefix, and the inspector panel can be
narrowed to 250px, where an unbroken 40-character SHA would otherwise
scroll the properties list sideways and push the copy button off-panel
(TRI-12923, TRI-13000).
The SDK discovers an external deployment id at runtime (explicit
TRIGGER_EXTERNAL_DEPLOYMENT_ID always; platform commit-SHA variables and
generic fallbacks when TRIGGER_AUTOMATIC_SKEW_VERSION_PROTECTION=1) and
sends it alongside lockToVersion; the server resolves precedence
(version > external id > current). An id held by a deployed deployment
pins the run to that worker; an in-flight or unknown id parks the run in
PENDING_VERSION with the id in TaskRun.annotations, wakes it pinned when
a deployment carrying the id finalizes (ClickHouse candidates, Postgres
authoritative), and expires it after a deadline that re-checks Postgres
before acting. Parking outranks delaying and preserves delayUntil. The
id is projected to ClickHouse task_runs_v2.external_deployment_id during
replication. Redis cache for id-to-worker resolution, guarded
version-aware writes.
Ids are not unique. Several deployments can hold one id - a --force
rebuild is the ordinary way to get there - so resolution always picks
the highest version among the candidates, never the newest by timestamp.
The rule is applied identically on both paths that can bind a run to a
worker: resolveExternalDeployment at trigger time, and
PendingVersionSystem when a landing deployment wakes a parked run.
Version comparison is numeric on the counter half, so 20260807.10
outranks 20260807.9.
A run whose id never lands expires at the deadline with
EXTERNAL_DEPLOYMENT_NOT_FOUND and an error naming the id it waited for,
which is what a failed build or a typo looks like from the caller.
Default deadline is one hour (EXTERNAL_DEPLOYMENT_PARK_DEADLINE_MS).
Debounce registration happens in both the parked and the delayed branch
through one helper, so a debounced run that parks still binds its
debounce key; without it every later trigger for the same key created
another parked run, and all of them executed when the deployment landed.
The two DELAYED-only status checks in DebounceSystem also accept
PENDING_VERSION, without which the lock-contention fallback would
rethrow a 5xx the SDK retries and amplifies, and the fast path would
push every trigger on a parked key through the redlock.
Resolution is skipped in development. A dev environment cannot hold a
WorkerDeployment - trigger dev registers a BackgroundWorker with nothing
behind it, and deploy --env refuses dev - so an external deployment id
there could only ever park, and the parked run then expired against the
dev TTL while a connected dev worker sat idle. The id is still annotated
so the dashboard shows what the app sent (TRI-13000).
A deploy can carry an opaque external id (commit SHA, CI run id, release
tag). Repeating an id that already deployed returns the existing version
as a no-op instead of rebuilding; an id with a build in flight is
rejected with 409 naming that version; a failed id rebuilds freely.
--force is non-destructive to deployments that already succeeded - both
persist and the higher version wins - but cancels a build still in
flight, so one id never has two live builds racing to define it.
Cancelling writes a terminal status and appends a finalized event, which
aborts a build the platform drives; a build it does not drive keeps
running but can never land, and the CLI says so. Ids are deliberately
not unique - reuse is resolved in application code by highest version,
never timestamps. The no-op path mints no build credentials and no event
stream (TRI-12923).
What that means for callers: a --force rebuild leaves two deployments
holding one id, and runs triggered with it go to the higher version once
the rebuild lands, so the takeover needs no separate promotion. Until a
successful build exists for an id, runs triggered with it park and then
expire rather than falling back to current - a failed build is therefore
visible to the caller as expired runs, not as runs on the wrong release.
An external deployment id is an opaque, caller-chosen name for a release
- a commit SHA, a CI run id, a release tag. This adds the shared
contract that both halves of the feature read, and nothing else: no
deploy writes one yet and no trigger sends one.
ExternalDeploymentId is defined once and reused by
InitializeDeploymentRequestBody.externalId and
TriggerTaskRequestBody.options.externalDeploymentId, so a value accepted
by one half can never be rejected by the other. A value that is blank
once trimmed is treated as absent rather than rejected, so an unset CI
variable expanding to an empty string is not a 400. The 128 character
limit fits a SHA-256 commit hash with room for composite ids, and
EXTERNAL_DEPLOYMENT_ID_MAX_LENGTH is the single source of truth that the
request schemas and the CLI both read.
RunAnnotations.externalDeploymentId records the request, not the
outcome: lockedToVersionId and taskVersion are overwritten when a run
locks, whereas this stays true forever, and it can carry the pin for a
run parked before its deployment exists.
Also lands the runtime discovery helpers as pure functions over an
environment reader: the explicit TRIGGER_EXTERNAL_DEPLOYMENT_ID
variable, the platform and CI commit-SHA table, and the
TRIGGER_AUTOMATIC_SKEW_VERSION_PROTECTION gate. Nothing calls them yet.
refs TRI-13000
Migrations only, no code reads them yet. Postgres: nullable non-unique
externalId on WorkerDeployment plus a CONCURRENTLY-built (environmentId,
externalId) index in its own migration file. ClickHouse:
external_deployment_id String DEFAULT '' on task_runs_v2 (plain String,
not LowCardinality - commit SHAs are high-cardinality). Part of task run
version skew protection (TRI-12998).
## Summary
Move tree selection onto semantic tree items and use native expansion
buttons.
Dashboard and story tree rows now share mouse and keyboard selection
through `getNodeProps`. Expand and collapse affordances are named
buttons instead of clickable layout elements.
Base: [#4700](https://github.com/triggerdotdev/trigger.dev/pull/4700)
## Summary
Use native controls for sortable columns and selectable prompt versions.
Table headers keep filter actions separate from sort buttons, prompt
version rows expose pressed state, and a redundant deployment click
interceptor is removed.
Base: [#4699](https://github.com/triggerdotdev/trigger.dev/pull/4699)
## Summary
Make time-filter mode selection keyboard accessible.
Duration and exact-range modes now use native pressed buttons. Nested
date, duration, and quick-select controls no longer depend on click
propagation blockers.
Base: [#4698](https://github.com/triggerdotdev/trigger.dev/pull/4698)
## Summary
Replace mouse-only dashboard actions with native buttons.
Copy, remove, and stop-generation controls now expose keyboard focus and
accessible names. Hover-revealed actions remain mounted so keyboard
users can discover them, and a decorative clipboard icon no longer
captures clicks.
Base: [#4697](https://github.com/triggerdotdev/trigger.dev/pull/4697)
## Summary
Use native label and checkbox behavior for `CheckboxWithLabel` and
enforce `jsx-a11y/no-noninteractive-element-interactions`.
The component no longer simulates checkbox activation with click
handlers on non-interactive wrappers. Native change events now drive the
controlled checked state.
Base: [#4696](https://github.com/triggerdotdev/trigger.dev/pull/4696)
## Summary
Require accessible names for dashboard controls.
Filter menu action items and chart color controls now expose explicit
names. The chart legend action uses a native button, while lint depth
and spacer-cell configuration match the rendered control structure.
Base: [#4695](https://github.com/triggerdotdev/trigger.dev/pull/4695)
## Summary
Finish associating dashboard form labels with their controls and enforce
`jsx-a11y/label-has-associated-control`.
Repeated data store dialogs use unique generated IDs, story controls and
notification filters have explicit associations, and display-only status
text no longer uses label elements.
Base: [#4694](https://github.com/triggerdotdev/trigger.dev/pull/4694)
## Summary
Associate internal model administration labels with their form controls.
The model editor, creator, and tester now use explicit `htmlFor` and
`id` pairs. Section titles that do not label controls now use headings
instead of label elements.
Base: [#4693](https://github.com/triggerdotdev/trigger.dev/pull/4693)
## Summary
Enable foundational JSX accessibility checks for image text alternatives
and valid ARIA roles.
The avatar color picker now has an explicit accessible name and
decorative image alternative. Dashboard chat styling props no longer
reuse the reserved DOM `role` name.
Base: [#4692](https://github.com/triggerdotdev/trigger.dev/pull/4692)
## Summary
Add explicit types to native dashboard buttons and enforce
`react/button-has-type`.
This prevents action buttons from accidentally submitting a surrounding
form. Shared button primitives retain their caller-selected submit and
reset semantics with documented lint exceptions.
Base: [#4691](https://github.com/triggerdotdev/trigger.dev/pull/4691)
## Summary
Remove redundant React fragments from the dashboard and enforce
`react/jsx-no-useless-fragment`.
The cleanup returns existing nodes, arrays, and empty states directly
without adding wrapper elements.
Base: [#4689](https://github.com/triggerdotdev/trigger.dev/pull/4689)
## Summary
Keep component and renderer identities stable across dashboard renders.
Inline icon components, chart renderers, table cells, and select render
callbacks now use module-level implementations. Oxlint enforces the
pattern across the dashboard.
Base: [#4688](https://github.com/triggerdotdev/trigger.dev/pull/4688)
## Summary
Enforce stable React hook ordering in the dashboard and React hooks
package.
Conditional hook calls now keep a consistent order, and overloaded
realtime stream arguments are resolved before entering the shared hook
implementation.
Base: `main`
`@grpc/grpc-js` sat at 1.12.6 in the lockfile. `dockerode` is the only
consumer and already declares `^1.11.1`, so a scoped override is enough:
```json
"@grpc/grpc-js@>=1.12.0 <1.12.7": "1.12.7"
```
Pinned exactly to stay on the 1.12 line; a caret would pull 1.14.x.
## Summary
Listing schedules could block the event loop for seconds. A page of 100
timezone-aware schedules spent over two seconds on cron arithmetic
alone, after the database work was already done, which stalls every
other request on that process. The same page now resolves in tens of
milliseconds.
## Root cause and fix
`cron-parser` walks the calendar unit by unit, and under a named
timezone every step goes through luxon. Parsing an expression is cheap
(single-digit microseconds); *stepping* it is not, ranging from a couple
of hundred microseconds for a common expression to several milliseconds
for a sparse one like `0 0 29 2 *`. The presenter did three independent
walks per row, one backwards for "last run" and two forwards (re-parsing
each time) for the next run and the occurrence after it. At 100 rows
that is 300 calendar walks in one uninterrupted tick.
Run times now resolve for the whole page in one pass, in a new
`resolveScheduleTimings` that takes plain values rather than Prisma rows
so it can be tested and benchmarked on its own.
- **Nominal times are cached per `(cron, timezone)`** against a single
`now` pinned for the batch, so cost scales with the number of distinct
expressions instead of the number of rows. Rows in one response also
stop disagreeing about the current time.
- **The backwards walk is opt-in.** It is the most expensive of the
three and only the dashboard renders the column; the public API never
returned it at all.
- **Windowless schedules take one step instead of two.** The second step
only measures the interval to the following occurrence, and that
interval reaches the result solely through `min(intervalMs,
max(MINIMUM_SCHEDULE_RANGE_MS, windowMs))`. With no window `windowMs` is
0, and `CronPattern` rejects expressions with a seconds field, so
occurrences are always at least `MINIMUM_SCHEDULE_RANGE_MS` apart and
that `min` can never bind. It is also the costlier step, since it walks
a whole period rather than the remainder of the current one.
- **`nextScheduledTimestamps` steps one parsed expression** instead of
re-parsing per step, which also helps the single-schedule callers.
Behaviour is unchanged, error semantics included: a malformed expression
still throws for the next run and still degrades to an undefined last
run.
## Verification
Measured inside a real request against a live environment, 100
schedules: sparse expressions went from 2250-2652 ms to 23-30 ms, and
five distinct timezone expressions from 463-500 ms to 9.7-10.6 ms.
The new suite checks the optimized code against an inline copy of the
previous implementation across eleven cron and timezone combinations
plus five DST transitions, so the rewrite is verified as
behaviour-preserving rather than just faster. Separate tests pin the
invariant the single-step path depends on, so if sub-minute crons are
ever allowed they fail loudly instead of the timings quietly going
wrong.
Worth knowing for later: `cron-parser` v5 is a much faster rewrite on
exactly this workload (`prev()` under a timezone drops from roughly 2700
to 60 microseconds), but it is a breaking API change across several call
sites including the schedule engine, so it belongs on its own. The
differential test added here is the tool to de-risk it.
## Summary
Allow the logs search schema migration to run on ClickHouse versions
that require text index options to be literals.
## Root cause
The text index declared `lowerUTF8(search_text)` as a preprocessor
option. Some ClickHouse versions reject that column expression while
parsing index settings. The projected `search_text` is already
normalized to lowercase before insertion, so removing the redundant
preprocessor preserves search behavior.
Verified with the task events search integration tests.
## What
The one-off worker container boot is billed to whichever test resolves
the fixture first. This moves it into a `beforeAll` with its own
timeout.
## Why
vitest runs the fixture chain *inside* the test timer:
```js
// @vitest/runner 4.1.7
setFn(task, withTimeout(...withFixtures(handler)..., timeout, ...))
```
There is no `fixtureTimeout`. So booting Postgres (plus `CREATE
DATABASE`, schema push, ClickHouse and Redis) lands on the first test
and consumes a budget sized for test work.
That is why losing the image pre-pull on fork PRs was fatal rather than
merely slower: the extra ~10s crossed the 60s cap. Since fork time is
roughly internal + 10s and forks exceed 60s, internal runs were already
clearing that cap by under 10s — a latent flake regardless of forks.
## How
`withWarmup` wraps each fixture family and lazily registers a
`beforeAll` on first touch, with its own generous timeout. Registration
is lazy so only files that actually use a family pay for it —
`@internal/testcontainers` is imported by hundreds of test files, many
of which only need Redis. It registers once per file, since `isolate`
gives each file a fresh module registry.
Eight families are wrapped. `isolatedRedisTest`,
`replicationContainerTest` and `postgresAndRedisTest` are deliberately
untouched: they use per-test containers by design, so there is no
one-off boot to hoist.
No test file or CI changes, and it applies to every package using these
fixtures.
## Verification
Proven by mutation. `src/warmup.test.ts` runs container tests under a
deliberately tight cap:
| | Result |
| --- | --- |
| with the warm-up | passes |
| warm-up neutered | fails, `Test timed out` |
It is kept as a regression test — without it, unwrapping a fixture would
break nothing visibly.
`triggerFailedTask.call.test.ts`, one of the five shard casualties,
passes locally in 20.4s.
## Also here
`@internal/testcontainers` had no `test` script, so `turbo run test
--filter "@internal/*"` skipped the package and its existing
`heteroDedicated.test.ts` never ran in CI. Adding the script (matching
the sibling packages') runs both files; verified green through turbo
exactly as CI invokes it.
## What
Three corrections to the pre-pull lists, each verified against what the
suites actually use.
## Changes
**`ryuk:0.11.0` -> `0.14.0`** in `e2e-webapp.yml` and
`e2e-webapp-auth-full.yml`. The installed testcontainers hardcodes the
image it starts:
```js
// testcontainers@11.14.0 build/reaper/reaper.js
: ImageName.fromString("testcontainers/ryuk:0.14.0").string;
```
So those two lines were pre-pulling an image nothing starts, and the one
actually used was never pre-pulled. The other three workflows already
say 0.14.0.
**`postgres:17` added** to `unit-tests-webapp.yml`. The webapp suite
references `docker.io/postgres:17` across 10 files but only
`postgres:14` was pre-pulled. `unit-tests-internal.yml` already pulls
both.
**Electric pinned to its digest** in `unit-tests-webapp.yml`. The tests
run `electricsql/electric:1.2.4@sha256:20da...` while the pre-pull asked
for the bare tag, so the pre-pull did not necessarily populate the
manifest the tests then request.
## Not changed
The otel collector and s2 images are pulled by other workflows but are
not used by the webapp suite, so they are deliberately not added here.
`postgresAndRedisTest` uses per-test containers by design and needs
nothing pre-pulled.
## What
The `Pre-pull testcontainer images` step is gated on
`env.DOCKERHUB_USERNAME`. Fork PRs receive no repository secrets, so
that variable is empty and the step is skipped along with the DockerHub
login it was grouped with.
## Why
With the pre-pull skipped, testcontainers pulls images lazily — inside
the first test that resolves the fixture, against that test's
`testTimeout`. On PR #4534 that pushed five webapp shards past their 60s
cap across three runs, each failing as `Test timed out in 60000ms` while
42 of 43 files in the shard passed.
Measured cost of the missing pre-pull, comparing the delta from vitest
start to the first container fixture on the same runner class:
| Run | Delta |
| --- | --- |
| internal x2 | +139.9s, +139.4s |
| fork x2 | +149.7s, +149.4s |
A 10.0s penalty, bimodal to within 0.3s.
Note the pulls themselves succeed anonymously — there are no rate-limit
errors in any of the failing logs. Only the login needs credentials, so
the pre-pull can run unconditionally.
## Scope
Removes the `if:` from the pre-pull step in all five workflows that have
one. The DockerHub login stays gated, since it genuinely needs secrets.
## Summary
Memoize shared context values so provider renders do not unnecessarily
rerender every consumer. Oxlint now enforces this pattern for the rest
of the dashboard.
Base: [#4677](https://github.com/triggerdotdev/trigger.dev/pull/4677)
## Summary
Enable lint rules that prefer direct iteration and concise function
callback types.
The existing code now uses direct iteration where no index is needed,
and callback contracts use function types consistently.
Base: [#4675](https://github.com/triggerdotdev/trigger.dev/pull/4675)
## Summary
Enable JSX cleanup rules for shorthand fragments and self-closing
components.
The existing JSX is automatically simplified, and future components will
follow the same concise form.
Base: [#4673](https://github.com/triggerdotdev/trigger.dev/pull/4673)
## Summary
Enable small cleanup rules for redundant boolean expressions, object
ownership checks, assignments, and object construction.
The existing call sites now use the simpler equivalent forms, keeping
future code consistent without changing behavior.
Base: [#4672](https://github.com/triggerdotdev/trigger.dev/pull/4672)
## Summary
Enable additional lint rules that catch unsafe optional-chain
assertions, inherited-property iteration, anonymous symbols, and unsafe
external links.
The existing violations now use explicit values and own-property checks,
so the rules can prevent those patterns from returning.
Adds an optional priority class for run pods.
```
KUBERNETES_RUN_POD_PRIORITY_CLASS_NAME
```
When set, the value is applied as `priorityClassName` on the run pod
spec. When unset, pods are created exactly as before.
Off by default, and inert unless set. It sits beside the existing
`KUBERNETES_SCHEDULER_NAME` option and follows the same conditional
shape:
```ts
...(env.KUBERNETES_RUN_POD_PRIORITY_CLASS_NAME
? { priorityClassName: env.KUBERNETES_RUN_POD_PRIORITY_CLASS_NAME }
: {}),
```
## Verification
`typecheck --filter supervisor`, `format` and `lint` clean. No changeset
or `.server-changes/` note: off by default, no user-visible behaviour
change.
## What
Runs that carry no external trace context (schedules, task-to-task
triggers) fall back to a trace id generated once in the [`TracingSDK`
constructor](https://github.com/triggerdotdev/trigger.dev/blob/main/packages/core/src/v3/otel/tracingSDK.ts#L165).
With `experimental_processKeepAlive` the SDK outlives the run, so every
run on a warm process is exported to the external OTLP endpoint under
that one id.
Across our production traces, 80.3% contained spans from more than one
run, worst case 25. Per-run cost and latency attribution is unusable as
a result. This is the same warm-start hazard c043c4a6a fixed for the
external-context path, which left the fallback captured at construction.
## How
`FallbackExternalTraceIds` hands out one id per internal trace, shared
by the span and log wrappers so a run's spans and logs agree.
The id is keyed off the record's own internal trace id rather than
ambient state at export time, because batch processors drain
asynchronously and a run's records routinely export after the next run
has started. The map is bounded and evicts least-recently-used, so a run
that is still exporting can't lose its id.
Granularity follows the internal trace, so a run and the runs it
triggers stay on one trace.
**Risk:** the wrappers only exist when `exporters` / `logExporters` are
configured, so deployments that don't export externally are untouched.
Nothing outside `tracingSDK.ts` changes.
**Known gap (pre-existing):** sampling and id selection still branch on
ambient `getExternalTraceContext()`, so records draining across a run
boundary in mixed mode are misplaced in both directions. It can't use
the approach here — the external id comes from the run's incoming
`traceparent`, which isn't carried on the record — so closing it means
capturing `internalTraceId -> external context` in a span processor.
Happy to follow up separately.
---
## Testing
`packages/core` suite passes. `pnpm run format` and `pnpm run lint:fix`
produce no diff.
Six cases in `externalSpanExporterWrapper.test.ts`, each
mutation-checked rather than just observed passing: one id per run,
stability within a run, correct id when records drain after the next run
started (spans and logs together), external export stays off when
unconfigured, retention of a run still exporting while the map churns,
and the bound itself.
**CI:** the five failing `webapp` shards are the ones containing
`containerTest` suites. Fork PRs receive no repository secrets, so
`unit-tests-webapp.yml` skips the DockerHub login and the image pre-pull
(both gated on `env.DOCKERHUB_USERNAME`) and the container tests time
out at 60s. Same five shards across five runs, every failure a 60s
timeout, and those shards pass on internal PRs. Happy to be corrected if
you can run them with secrets available.
---
## Changelog
Unrelated runs are no longer merged into a single trace in your external
observability tool when they happen to execute on the same warm worker
process. A run and the runs it triggers still share one trace, so a run
tree stays together.
---
## Screenshots
_n/a_
---
_Supersedes #4526 (auto-closed before I was vouched) and #4533 (opened
ready rather than as a draft). GitHub won't reopen either._
---------
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-authored-by: Iss <74388823+isshaddad@users.noreply.github.com>
Co-authored-by: nicktrn <55853254+nicktrn@users.noreply.github.com>
<!-- ccr-slack-attribution -->
_Requested by **Matt Aitken** · [Slack
thread](https://triggerdotdev.slack.com/archives/C0BKB98B84W/p1787045331358929)_
Copy-only reword of the banner shown to org admins who have not set a
billing limit yet.
**Before** — the banner read "Protect your organization from unexpected
usage spikes." with a button labelled "Configure billing limit".
**After** — it reads "Add a billing limit to your account to prevent
overspending" with a button labelled "Billing limit settings".
The new wording names the action up front and matches the destination it
sends you to, so the banner reads as a settings link rather than a
one-off setup step.
## ✅ Checklist
- [x] I have followed every step in the [contributing
guide](https://github.com/triggerdotdev/trigger.dev/blob/main/CONTRIBUTING.md)
- [x] The PR title follows the convention.
- [x] I ran and tested the code works
---
## Testing
Formatting and linting pass (`oxfmt --check`, `oxlint`). No tests or
snapshots assert this copy. The change is two string literals in one
component, with no behaviour attached.
---
## Changelog
Reworded the billing-limit banner for organizations without a limit
configured, and relabelled its button to "Billing limit settings".
---
## How
Both strings live in `NoLimitConfiguredBanner` in
`apps/webapp/app/components/billing/OrgBanner.tsx`: the heading is the
`canManageBillingLimits` branch of the banner's children, and the label
is the `<span>` inside the `LinkButton`. Only those two literals
changed. The button still points at `v3BillingLimitsPath(organization)`
(`/orgs/{slug}/settings/billing-limits`), so routing, permissions and
the non-admin variant of the message are untouched.
Co-authored-by: Claude <noreply@anthropic.com>
Adds KUBERNETES_RUNNER_SECURITY_CONTEXT (off | baseline | restricted), selecting how constrained the run container is.
baseline drops the capability bounding set and blocks privilege escalation. restricted additionally pins the container to a non-root uid, chosen by runtime so bun images get their own.
Default is off, so this is inert on merge.
## Summary
A run triggered with a `ttl` could get permanently stuck showing as
queued. If the run started executing and was then requeued after a
failure (a stalled heartbeat, a worker dying mid-run) once its TTL had
already elapsed, the next dequeue pass silently dropped it from every
queue structure. The run stayed QUEUED in the database forever, and
nothing (dequeue, the TTL consumer, queue repair) could ever see it
again.
## Root cause
Enqueue registers a TTL entry for the TTL consumer, and the first
dequeue removes it ("the run is executing, not expired"). A nack rewrote
the message preserving the original `ttlExpiresAt` without
re-registering that entry. The next dequeue pass then took the
expired-TTL branch: remove the run from the queue sorted sets and leave
the message for the TTL consumer to finalize. But the consumer's entry
was gone, so nothing ever finalized the run.
The fix has two halves:
- `nackMessage` drops `ttlExpiresAt` from the rewritten message. TTL
only applies to runs that have never been dequeued (the same contract as
`includeTtl` on re-enqueues), so a requeued run stays dequeuable and is
never expired by its original deadline.
- The dequeue expired-TTL branches now (re-)register the TTL entry
instead of assuming it exists, so any message still carrying a lapsed
`ttlExpiresAt` with no TTL entry (including ones written before this
fix) finalizes as EXPIRED instead of orphaning.
## Verification
New engine test suite `ttlNackRequeue.test.ts` (testcontainers, real
Redis and Postgres). All four tests fail before the fix and pass after:
- a heartbeat-stalled EXECUTING run with a lapsed TTL is requeued and
dequeued again instead of orphaned (the full production failure chain)
- requeue-after-failure strips `ttlExpiresAt` so later dequeues do not
treat the run as expired
- a lapsed-TTL message whose TTL entry is missing is re-registered by
dequeue and finalized as EXPIRED, for both plain and concurrency-key
queues
Also ran the existing ttl, heartbeats, dequeuing and attemptFailures
engine suites plus the full run-queue suite (149 tests) against the
change.
Adds two optional env vars that rewrite the registry host of run pod images at pod creation, so a supervisor can pull from a registry in its own region. Off by default and inert unless both are set. Exact host-prefix matching, so look-alike hosts pass through untouched.
Replaces the hardcoded runner seccomp profile path with KUBERNETES_RUNNER_SECCOMP_PROFILE_PATH, and the node-24-only condition with KUBERNETES_RUNNER_SECCOMP_PROFILE_RUNTIMES (none | node-24-plus | all).
Both defaults reproduce current behaviour, so this is inert on merge. Widening the scope or turning attachment off becomes a config change rather than a deploy.
The supervisor now supports routing an organization's runs to specific
nodes. `KUBERNETES_ORG_PLACEMENT_OVERRIDES` takes JSON keyed by the
internal org ID, adding node selector entries and tolerations to that
org's run pods, e.g. to route an org onto a dedicated, tainted node
pool:
```json
{"<orgId>": {"nodeSelector": {"pool": "dedicated"}, "tolerations": "dedicated=runs:NoSchedule"}}
```
The node selector merges over the defaults (the override wins on key
collision, with a warning logged). Tolerations append to the existing
runner and scheduled-run sets. Overrides are validated at startup
similar to `KUBERNETES_RUNNER_TOLERATIONS`.
Exposed in the Helm chart as
`supervisor.config.kubernetes.orgPlacementOverrides`, where tolerations
can also be given as a list.
## What & why
The dashboard agent can now run its model calls through AWS Bedrock
instead of the direct Anthropic API, chosen by a single env switch. It's
**off by default** (`DASHBOARD_AGENT_MODEL_PROVIDER` unset ⇒
`anthropic`), so merging changes nothing at runtime — the Bedrock path
is a dormant branch until an operator sets the switch and AWS config.
The default Anthropic path is byte-for-byte unchanged.
This also carries a related tenant-isolation hardening for the agent's
delegated token (kept together deliberately — both land the agent on
Bedrock for HIPAA readiness). Refs: TRI-13251, TRI-11032.
## What's inside
**Provider seam** —
`internal-packages/dashboard-agent/src/model-provider.ts`: the registry
now holds both `anthropic` and `bedrock`; `resolveDashboardAgentModel()`
maps the canonical `"anthropic:<id>"` strings the managed prompts carry
to the active provider, and the cache-breakpoint helpers emit the active
provider's shape — Anthropic `cacheControl` vs Bedrock `cachePoint`.
Managed prompt strings stay canonical, so stored prompts don't change
meaning. Unmapped model ids throw rather than shipping a guaranteed-404
profile. All agent, watch, compaction and title callsites route through
the resolver; the `dashboardAgentModelKey` locals override (test mock
injection) is preserved.
**Cache telemetry** — `step-cache.ts`: cache token usage is read from
the active provider (Anthropic reports it on provider metadata; Bedrock
reports the write on metadata and the read via standard usage), so
`gen_ai.usage.cache_*` is populated on both. This also fixes a latent
ordering bug where step attributes could null-overwrite the prompt-cache
read count.
**Webapp callsites** — `dashboardAgentHeadStart.server.ts` and the
head-start route resolve the model and the cache breakpoint through the
same seam, so the warm-up prefix and the following turn share one
provider. The head-start firing gate is provider-aware: on Bedrock it
gates on `AWS_REGION` and lets the SDK resolve credentials (IAM role /
static keys / session token / bearer), so a role-based deploy still
warms; on Anthropic it stays `Boolean(ANTHROPIC_API_KEY)`.
`app/env.server.ts` gains the optional AWS vars and validates
`DASHBOARD_AGENT_MODEL_PROVIDER`. `ANTHROPIC_API_KEY` is untouched and
not required on a Bedrock deploy.
**Tenant-isolation hardening** —
`internal-packages/rbac/src/fallback.ts`: for a **scoped** context, the
OSS `authenticateUserActor` now applies the same membership floor as the
session path — a delegated user-actor token whose user is not a member
of the scoped org/project is denied (403). Unscoped tokens keep their
prior behavior (no tenant claim, no lookup). The user lookup falls back
replica→primary so replication lag can't spuriously 401 a just-joined
member. Members and admins are unaffected. Previously this invariant
held only through per-route discipline; this makes it structural.
## Enabling Bedrock (later, ops)
- Set `DASHBOARD_AGENT_MODEL_PROVIDER=bedrock` **identically** in both
the webapp and the agent task container — the webapp warms the cache
prefix and the task reads it, so a split would silently miss the cache.
- Set `AWS_REGION` and provide credentials the Bedrock SDK can resolve
(IAM role preferred). For v1 this runs **without** an Anthropic API key.
Note: with no Anthropic key set, rollback is "turn the agent off", not
"unset the switch" (unsetting falls back to the Anthropic provider,
which then has no key).
- Two things to confirm before rollout: the Sonnet inference-profile id
is validated against the SDK's own model-id union but still warrants a
live smoke test; and Bedrock prompt caching for Sonnet is a 5-minute
window (not Anthropic's 1h), so input-token cost rises when flipped.
## Testing
Unit tests cover both provider paths: the provider switch and
per-provider cache shapes, a structural regex asserting Bedrock ids are
real inference profiles (not an echo of the table), the split-metadata
cache telemetry, and real-Postgres RBAC tests — member allowed, scoped
non-member denied (org-only and project-only), missing user → 401, admin
non-member exempt, unscoped success. `typecheck --filter webapp` and the
dashboard-agent + rbac suites pass.
## Summary
When an AI SDK tool call failed inside a run, the span showed up under
the "Errors only" filter but the span inspector gave no hint of what
went wrong. The exception was recorded on the span all along; the
`ai.toolCall` and `ai.embed` inspector views just never rendered span
events. Failed tool call and embedding spans now show the standard error
block (message plus stack trace) below the Input section.
## Root cause
Generic spans render exception span events via the `SpanEvents`
component, but the AI-specific span entities replace the whole panel
with their own layout and dropped the events entirely. The span's events
are now passed into `AIToolCallSpanDetails` and `AIEmbedSpanDetails` and
rendered with the same `SpanEvents` component the generic view uses.
Errored generation spans (`ai.generateText` and friends) use a tabbed
view and still don't surface errors; that needs its own design pass and
is left for a follow-up.
## The bug
A concurrency key is an unrestricted client string
(`ConcurrencyKeySchema` is `z.union([z.string(),
z.number()]).transform(String)`), and `concurrencyKeySection` does no
escaping, so `*` reaches the queue raw. `queueKey` then renders it as
`...:queue:<q>:ck:*`, which is byte-identical to the wildcard member the
CK scripts keep in the master queue to mean "this base queue has
concurrency-key work".
Every CK script ends with the same pair:
```lua
-- Rebalance master queue with ck:* member
redis.call('ZADD', masterQueueKey, earliestIdx[2], ckWildcardName)
-- Remove old-format entry from master queue (transition cleanup)
redis.call('ZREM', masterQueueKey, queueName)
```
`ckWildcardName` is `toCkWildcard(message.queue)`, and for a `*`-keyed
run that returns the identical string, so the cleanup on the second line
deletes what the rebalance on the first line just wrote.
The master queue then has no entry for that base queue, while `ckIndex`
and the variant queues still hold the work. **Every concurrency key on
the queue stops being dequeued**, not just the `*` one. It is silent,
and it only recovers if some later write happens to re-add the member.
Reproduced before the fix:
```
master queue AFTER normal ck enqueue: ["{org:...}:queue:task/my-task:ck:*"]
master queue AFTER ck='*' enqueue: []
ckIndex members (work still queued): [":ck:user-1", ":ck:*"]
dequeued: []
```
Blast radius is bounded to the environment that triggers it, so it is
self-inflicted rather than cross-tenant, but a single trigger stalls the
queue for everything on it.
## The fix
Guard the cleanup so it never removes the wildcard member:
```lua
if queueName ~= ckWildcardName then
redis.call('ZREM', masterQueueKey, queueName)
end
```
Applied to all 10 CK scripts (4 enqueue, 6 ack/nack/dead-letter). No
key-format change and no migration: a queue already stranded in Redis is
repaired by its next write.
I considered rejecting `*` at the API boundary instead and rejected it.
Existing Redis state and `TaskRun.concurrencyKey` rows already hold raw
`:`-bearing and `*` keys, so changing key construction would orphan
in-flight messages and split concurrency accounting mid-deploy. Boundary
validation would still be reasonable as belt-and-braces later, but the
Lua guard alone fixes it including for state already out there.
## Testing
`ckWildcardKey.test.ts` covers the enqueue, ack and nack paths. All
three pass with the guard and **all three fail without it**, verified by
reverting. Full `src/run-queue/` suite is green (166 tests).
## Note for #4367
The virtual-time branch adds three more CK scripts with the same pattern
(`enqueueMessageCkVtimeTracked`, `enqueueMessageWithTtlCkVtimeTracked`,
`nackMessageCkVtimeTracked`). They do not exist on main so they are not
in this PR; the same guard needs applying there, and I will do that on
that branch.
## Summary
`lefthook.yml` has been in the repo since #4147, but nothing installs
lefthook and nothing runs `lefthook install`, so the pre-push hook it
describes has never fired for anyone. #3977 had removed the `lefthook`
devDependency a week before #4147 landed, and #4147 only added the
config file.
This supplies the missing half:
```diff
+ "prepare": "lefthook install",
+ "lefthook": "^2.1.10",
"onlyBuiltDependencies": [
+ "lefthook",
```
With those in place, `pnpm install` wires the hook up on clone, and the
format and lint checks actually run before a push instead of first
failing in CI.
Also here: the pre-push jobs run in parallel rather than in sequence,
and `CONTRIBUTING.md` documents the hook, including how to skip it and
the fact that GitButler only runs hooks when "Run hooks" is enabled in
its settings.
`lefthook@2.1.10` is the current release.
<!-- ccr-slack-attribution -->
_Requested by **Matt Aitken** · [Slack
thread](https://triggerdotdev.slack.com/archives/C045W9WM3E1/p1786948064703889?thread_ts=1786948064.703889&cid=C045W9WM3E1)_
Adds `gtremper` to `.github/VOUCHED.td`.
Before: PRs from `gtremper` are auto-closed by the vouch check.
After: `gtremper` is vouched, so their PRs stay open and run CI.
Done as a direct file edit rather than the issue-comment flow because
there is no open Vouch Request issue for this user, matching the
precedent in #3804.
Co-authored-by: Claude <noreply@anthropic.com>
## Summary
`packages/core` declared `socket.io`, the server package, but never
imported it. Its only Socket.IO usage is the client:
```
packages/core/src/v3/zodSocket.ts
packages/core/src/v3/runEngineWorker/supervisor/session.ts
import { io } from "socket.io-client";
```
The only occurrence of `socket.io` outside those client imports was the
`package.json` line itself. Since `@trigger.dev/core` is published, that
line meant every consumer installed a server package nothing in the tree
imports.
`socket.io-client` is untouched. `apps/webapp` and `apps/supervisor`
keep their own `socket.io` dependencies, so the server side is
unaffected.
Found with `pnpm run knip:deps`, which the repo already ships.
`pnpm run typecheck` passes across all 57 workspaces, and
`@trigger.dev/core` builds clean.
Stacked on #4639.
## Summary
`find-my-way` was resolving `9.3.0` even though its only parent,
`fastify@5.8.5`, declares `^9.0.0` and so already permitted a newer
release. The lockfile had not re-resolved since. This adds a floor so it
lands on a current 9.x:
```json
"find-my-way@>=9 <9.7.0": "^9.7.0"
```
It resolves to `9.7.0`. Nothing outside the 9.x line is touched, and no
parent is asked to accept anything its declared range did not already
allow.
The whole path is development only: `find-my-way` arrives through
`fastify`, which arrives through `evalite`, a devDependency of
`apps/webapp` used by the `eval:dev` harness.
Stacked on #4638.
## Summary
The existing `ip-address` override is scoped to a single parent,
`@jsonhero/json-infer-types>ip-address`. A second parent reaches
`ip-address` independently: `express-rate-limit@8.6.0`, which is itself
pinned by our `@modelcontextprotocol/sdk@>=1.26.0>express-rate-limit`
override. That path was resolving `10.2.0` while the scoped path
resolved `10.5.0`, so the tree carried two copies.
This adds a matching scoped override for the second parent:
```json
"express-rate-limit>ip-address": "^10.3.1"
```
`express-rate-limit` declares `^10.2.0`, so this asks nothing of it that
its own range did not already allow. The tree now resolves a single
`ip-address@10.5.0`.
The existing `@jsonhero/json-infer-types` override stays: that package
declares `ip-address: ^8.1.0`, so removing it brings an 8.x copy back.
Stacked on #4637.
## Summary
`nanoid` was pinned at exactly `3.3.8` in five manifests. Two of those
five never imported it: in `internal-packages/schedule-engine` and
`internal-packages/webhook-engine` the only occurrence of the string
`nanoid` in the entire package was the `package.json` line itself. Both
are removed rather than bumped.
The three that genuinely use it move to `3.3.18`, a version already
present in the tree via `postcss`, so this pulls in nothing new.
| Package | Uses it | Change |
| --- | --- | --- |
| `internal-packages/schedule-engine` | no | removed |
| `internal-packages/webhook-engine` | no | removed |
| `apps/webapp` | yes | `3.3.8` to `3.3.18` |
| `packages/core` | yes | `3.3.8` to `3.3.18` |
| `internal-packages/run-engine` | yes | `3.3.8` to `3.3.18` |
| `packages/redis-worker` | yes | `^5.0.7` to `^5.1.16` |
`redis-worker` is on the 5.x line and is included because its declared
range already permitted a newer release; the lockfile had simply not
re-resolved, leaving it on `5.1.2`.
The unused declarations were found with `pnpm run knip:deps`, which the
repo already ships.
`pnpm run typecheck` passes across all 57 workspaces.
## Summary
`socket.io` was pinned at exactly `4.7.4` in three manifests
(`apps/webapp`, `apps/supervisor`, `packages/core`). That pin capped
`engine.io` at 6.5.4, because 4.7.4 declares `engine.io: ~6.5.2`.
Moving all three pins to `4.8.3` lifts that cap: 4.8.3 declares
`engine.io: ~6.6.0`. The webapp's direct `engine.io` devDependency moves
from `^6.5.4` to `^6.6.7` to match.
These are direct dependencies, so they are bumped in place rather than
forced with an override.
## Result
The tree previously carried two `engine.io` copies. It now carries one:
```
engine.io@6.6.8
└─┬ socket.io@4.8.3
├── @trigger.dev/core (dependencies)
├─┬ react-email
│ └── emails (devDependencies)
├── supervisor (dependencies)
└── webapp (dependencies)
```
`react-email` was already resolving `socket.io@4.8.3` in this same tree,
so that combination was already running here before this change.
## Servers move, clients do not
This bumps `socket.io` (the server) only. `socket.io-client` stays at
`4.7.5` in `packages/core` and `packages/cli-v3`, deliberately: the fix
is server-side, and clients ship inside user deployments, so leaving
them alone keeps the blast radius small. That means a 4.8.3 server will
be talking to 4.7.5 clients indefinitely, which is worth being explicit
about.
That pairing is safe because neither wire protocol changed. Both
versions report the same protocol numbers:
| | 4.7.4 | 4.8.3 |
| --- | --- | --- |
| Socket.IO protocol (`socket.io-parser`) | 5 | 5 |
| Engine.IO protocol (`engine.io-parser`) | 4 | 4 |
The version bump moves `socket.io-parser` 4.2.6 to 4.2.7 and `engine.io`
6.5.4 to 6.6.8, but the protocol constants each exports are unchanged.
The 4.8.0 changes are additive on the client (custom transport
implementations, a `tryAllTransports` option) and bug fixes on the
server.
Verified rather than assumed, with a cross-version matrix covering both
transports and both directions:
```
PASS server 4.8.3 <- client 4.7.5 websocket / polling
PASS server 4.8.3 <- client 4.8.3 websocket / polling
PASS server 4.7.4 <- client 4.7.5 websocket / polling
PASS server 4.7.4 <- client 4.8.3 websocket / polling
```
Each case exercised connect, a server-initiated emit, `emitWithAck`,
room join, room broadcast, and a binary payload. Compatibility holds in
both directions, so there is no upgrade-ordering requirement between
server and client.
`pnpm run typecheck` passes across all 57 workspaces.
Stacked on #4634.
## Summary
`axios` was resolving to 1.16.1 through `@slack/web-api`, which declares
`^1.16.0`. The lockfile had simply not re-resolved since, so the tree
sat on an older 1.x release than the range allows.
This adds a scoped override so the 1.x line picks up a current release:
```json
"axios@>=1.15.2 <1.18.0": "^1.18.0"
```
It resolves to 1.19.0. No parent bump is needed, since `^1.16.0` already
permits it, and `@slack/web-api` is the only consumer.
Stacked on #4633 so the two lockfile changes do not collide.
## Summary
`npm-run-all` has had no release since 4.1.5 in 2018, and pnpm now
covers the one thing we used it for. The webapp's `build` script was its
only consumer anywhere in the repo, so the dependency goes away
entirely.
`run-s build:**` becomes an explicit chain:
```
pnpm run build:remix && pnpm run build:server && pnpm run build:otlpworker && pnpm run build:sentry && pnpm run upload:sourcemaps
```
## Why this shape
I compared both forms side by side against the real `run-s` before
swapping:
| Behaviour | `run-s build:**` | explicit chain |
| --- | --- | --- |
| Scripts selected | remix, server, otlpworker, sentry | identical |
| Order | declaration order | identical |
| `upload:sourcemaps` matched by the glob | no | no |
| Second script fails | aborts, third never runs | identical |
| Exit code on failure | `1` | `1` |
`pnpm run --sequential "/^build:/"` was the closer-looking option, but
it keeps running scripts after one fails, so it is not a faithful
replacement.
The one thing given up is that `build:**` automatically picked up any
new `build:*` script, where the chain has to be edited. With four
entries that felt like the better trade.
`pnpm run build --filter webapp` passes end to end locally, all five
steps in order.
## Summary
Following a link straight into an organization's settings (for example
the usage limit link in a billing email) and then clicking "Back to app"
took you to `/`, which resolves to whichever organization you last had
selected, not the one whose settings you were looking at. The button now
links to the organization in the URL, so you land back in the org you
came from.
The org index route already redirects to the best project in that org,
so the destination is unchanged apart from being the right org.
Account settings still links to `/`, since that page is not org scoped
and has no org to return to.
## Summary
A number of `pnpm.overrides` entries had drifted behind the releases
they were written against. An override fixes the resolved version
outright, so in every one of these cases the tree was pinned to the
floor value rather than picking up later releases in the same line. This
raises each floor to a current release, and widens the selectors that
were scoped to an exact upper bound so they keep matching.
| Override | Before | After |
| --- | --- | --- |
| `body-parser` (under `express@^4`) | `1.20.3` | `^1.20.6` |
| `tar` | `7.5.19` | `7.5.21` |
| `hono` | `4.12.25` | `4.12.34` |
| `undici` (6.x) | `6.27.0` | `6.28.0` |
| `undici` (7.x) | `7.28.0` | `7.29.0` |
| `js-yaml` (3.x) | `3.14.2` | `3.15.1` |
| `js-yaml` (4.x) | `4.1.1` | `4.3.1` |
| `dompurify` | `^3.4.1` | `^3.4.13` |
| `vite` | `^6.4.2` | `^6.4.3` |
| `protobufjs` | `^7.5.6` | `^7.6.5` |
| `socket.io-parser` | `^4.2.6` | `^4.2.7` |
| `postcss` | `^8.5.10` | `^8.5.23` |
| `fast-uri` | `^3.1.2` | `^3.1.5` |
| `brace-expansion` (1.x) | `1.1.13` | `1.1.18` |
| `brace-expansion` (2.x) | `2.0.3` | `2.1.4` |
| `brace-expansion` (5.x) | `5.0.6` | `5.0.9` |
| `ip-address` (under `@jsonhero/json-infer-types`) | `^10.2.0` |
`^10.3.1` |
Every parent's declared range still accepts the new resolution, so
nothing is forced outside its stated bounds by this change.
Two of these changed a default rather than just moving version.
`js-yaml` 4.2.0 stopped resolving underscore-separated scalars such as
`1_000` as numbers, which is the YAML 1.2 behaviour, and there are none
in any YAML in this repo. `brace-expansion` 2.1.x now caps expansion
size by default, well above anything a real glob produces, and
`minimatch` calls it with no options. Neither is reachable from how we
use them.
`undici@5.29.0` and `vite@4.4.9` are left alone: their parents cap below
the newer lines, so moving either would mean taking the parent across a
major.
Verified with a clean install, and `pnpm run typecheck` passes.
## Summary
The server half of hosted webhooks: the public ingress endpoint,
signature verification, the delivery pipeline (Postgres partitioned
storage + ClickHouse for ordering), the in-app partition manager, the
HTTP API, and the dashboard (Deliveries, Endpoints, and the in-app test
console).
The public SDK and docs half is #4537. That PR carries the user-facing
API (`webhook()`, `chat.event` / `chat.channels`, the
`@trigger.dev/slack` connector) and builds on the shared
`@trigger.dev/core` schemas that ship here.
## Shipping behind a flag
A `WEBHOOK_ENABLED` env var (default off) gates the public ingress route
and the engine worker plus partition cron, so merging and deploying this
changes nothing in production until it is flipped on per environment.
The dashboard is separately gated per org by the `hasWebhooksAccess`
feature flag.
## Note on packages
This PR includes the `@trigger.dev/core` schema additions the server
compiles against, but carries no changeset. Core is not consumed
independently of the SDK, so it is released together with the SDK via
#4537. Keeping its changeset off `main` means no release cut from `main`
publishes it early.
## What
Makes two transaction-resilience behaviors real and env-var
configurable, defaults set to the good values, so we can tune during and
after the Aug 15 database patch window without a redeploy:
- **maxWait 2s → 10s** (TRI-12982): how long Prisma waits to borrow a
connection before it can `BEGIN`. A restart freeze holds the pool full,
and the only thing that errored was transaction starts giving up at 2s.
- **Retry transaction-start P2028-at-acquisition** (TRI-12984): when
Prisma can't borrow a connection within `maxWait` it raises P2028
(`Unable to start a transaction in the given time`) and **no SQL ran**,
so retrying is safe. Scoped narrowly: only that error (never P2024
pool-exhaustion), 2 attempts, jittered backoff, and a token-bucket
budget so a mass freeze can't amplify into a retry storm.
## Env vars (`DATABASE_*` convention)
Generic defaults:
| var | default |
|---|---|
| `DATABASE_TRANSACTION_MAX_WAIT_MS` | `10000` |
| `DATABASE_TRANSACTION_START_RETRY_ENABLED` | `true` (kill switch) |
| `DATABASE_TRANSACTION_START_RETRY_MAX_ATTEMPTS` | `2` |
| `DATABASE_TRANSACTION_START_RETRY_BACKOFF_MIN_MS` | `50` |
| `DATABASE_TRANSACTION_START_RETRY_BACKOFF_MAX_MS` | `250` |
| `DATABASE_TRANSACTION_START_RETRY_BUDGET_PER_SEC` | `50` |
| `DATABASE_TRANSACTION_START_RETRY_BUDGET_BURST` | `100` |
Per-writer-pool overrides, each falling back to the generic when unset
(same pattern as the per-client pool/connect-timeout work):
`RUN_OPS_DATABASE_TRANSACTION_*` and
`RUN_OPS_LEGACY_DATABASE_TRANSACTION_*` (all 7 knobs each). Transactions
only open on writer pools, so those are the only pools with their own
knobs. Each pool gets its **own** token bucket, so a storm on one pool
can't drain another's retry budget.
## Design
- The retry primitives live in `internal-packages/database` and never
read `process.env` (IoC): a P2028-at-acquisition classifier, a
`TokenBucketRetryBudget`, and `withTransactionStartRetry`, folded into
the `$transaction` helper via a new `startRetry` option. Config is
resolved at the app boundary and threaded in.
- The `$transaction` helper is the chokepoint (wraps the whole
transaction), not the per-statement `$allOperations` extension.
- The run engine's writes go through `PostgresRunStore`'s own
`.$transaction(...)`, not the webapp helper, so both the helper and the
two `PostgresRunStore` sites apply maxWait + retry (sharing the per-pool
config). Builds on the `options?: { timeout, maxWait }` seam added in
#4514.
- Webapp `$transaction` call sites get the default `maxWait` + retry
injected at one merge point, so no call site needed editing.
## Evidence
- Unit red/green in `internal-packages/database`: reverting the helper
wiring turned the acquisition-retry test red (`Unable to start a
transaction in the given time`), re-applying it green. Full package
suite 25/25. Covers: classifier (P2028-acq yes, P2024 no, in-tx P2028
no), retry (retry-then-succeed, no-retry P2024, stop at maxAttempts,
disabled, budget-exhausted, jitter bounds), token bucket, and
`$transaction` wiring.
- Typecheck clean: webapp, run-store, run-engine.
- Full-stack run: bounded queue-ay pass (15 projects, real dev runs
through the run-engine `PostgresRunStore` transaction path). 13 pass;
the 2 failures are one documented known-failure and one
stale-worker-state flake that passes 2/2 with this change active on a
fresh app.
- Boots cleanly with per-pool overrides set.
## Configuration & rollout
Ship **inert** first (zero behavior change), then flip to the good
values **live via env** — no redeploy needed for either.
### Inert — behaves exactly as today
```
DATABASE_TRANSACTION_MAX_WAIT_MS=2000 # Prisma's built-in default (change defaults to 10000)
DATABASE_TRANSACTION_START_RETRY_ENABLED=false # disable the new retry entirely
```
`maxWait=2000` is what every path used before (Prisma's default; the
run-store sites and the helper passed no maxWait). `retry=false`
short-circuits `withTransactionStartRetry` to a single run and makes the
serialization-retry exclusion a no-op. Verified on the pooler-freeze
rig: identical fail-fast P2028 at ~2003ms with zero retries —
byte-for-byte current behavior, across all pools.
### Production ("good") — the baked defaults
Rely on defaults (nothing to set) or set explicitly:
```
DATABASE_TRANSACTION_MAX_WAIT_MS=10000
DATABASE_TRANSACTION_START_RETRY_ENABLED=true
DATABASE_TRANSACTION_START_RETRY_MAX_ATTEMPTS=3 # 3 attempts (2 retries); ~30s acquisition tolerance covers a ~20-25s freeze
DATABASE_TRANSACTION_START_RETRY_BACKOFF_MIN_MS=50
DATABASE_TRANSACTION_START_RETRY_BACKOFF_MAX_MS=250
DATABASE_TRANSACTION_START_RETRY_BUDGET_PER_SEC=50
DATABASE_TRANSACTION_START_RETRY_BUDGET_BURST=100
```
Per-pool overrides `RUN_OPS_DATABASE_TRANSACTION_*` and
`RUN_OPS_LEGACY_DATABASE_TRANSACTION_*` (all seven knobs each) are
optional and fall back to the generic set — not needed for v1; the
generic set covers the control-plane, run-ops, and run-ops-legacy writer
pools. Readers open no transactions and take nothing.
**Guardrail:** the retry only engages when a pool's `pool_timeout` >
`maxWait`. Prod is fine (`DATABASE_POOL_TIMEOUT=60` >> 10). Do not set
any writer pool's `pool_timeout` at or under `maxWait`, or saturation
failures flip from retryable P2028 to non-retryable P2024 and the retry
silently stops helping.
### Rollback
Env flip (set inert) or revert. Retry only fires where no SQL ran, and
the per-pool token bucket caps a storm. No migration.
refs TRI-13295, TRI-12982, TRI-12984
<!-- ccr-slack-attribution -->
_Requested by **Matt Aitken** · [Slack
thread](https://triggerdotdev.slack.com/archives/C045W9WM3E1/p1786732623292829?thread_ts=1786732623.292829&cid=C045W9WM3E1)_
**Before:** you pause an environment, then a deploy lands (or a
background worker is created, or an admin changes the
concurrency/burst-factor). The environment starts picking up runs again
even though the dashboard still shows it as paused.
**After:** a paused environment stays paused until it is resumed, no
matter what else pushes its concurrency limit.
Pausing an environment sets `paused` in the database and writes a `0`
env concurrency limit into the run queue — the `0` is the only thing
that actually stops dequeueing. Any caller that pushed the limit without
an explicit value (`finalizeDeployment`, `createBackgroundWorker`, the
two admin environment routes) rewrote the real limit and silently
un-paused the environment.
## ✅ Checklist
- [x] I have followed every step in the [contributing
guide](https://github.com/triggerdotdev/trigger.dev/blob/main/CONTRIBUTING.md)
- [x] The PR title follows the convention.
- [x] I ran and tested the code works
---
## Testing
`apps/webapp/test/pauseEnvironment.server.test.ts` gains two
`containerTest` cases that wire a real `RunEngine` (real Redis) in place
of the stubbed app singleton and assert the actual run-queue env limit:
- pause a PRODUCTION env → limit is `0` → run the real
`FinalizeDeploymentService` → limit is still `0`, plus a control on a
running env in the same test proving that deploy path really does push
the limit (so the `0` can't just mean "nothing happened").
- pause → resume → the real limit is restored, so the clamp can't
regress resuming.
Both cases fail on `main` (`expected 17 to be +0` and `expected +0 to be
17`) and pass with this change. `pnpm run typecheck --filter webapp` is
clean.
---
## Changelog
Fix paused environments starting to run work again after a deploy.
---
## How
The clamp lives in the shared `updateEnvConcurrencyLimits` helper in
`apps/webapp/app/v3/runQueue.server.ts`, so every present and future
caller is covered: when no explicit limit is passed and the environment
is paused, `0` is written instead of the stored maximum. An
explicitly-passed limit still wins, which is what pausing itself relies
on. The resume path now passes the post-update environment state (its
in-memory copy was read before the un-pause and would otherwise be
clamped back to `0`), and the helper no longer mutates the caller's
environment object — that aliasing made a pause followed by a resume on
the same object write `0` twice. The existing `!paused` guards in
`allocateConcurrency` and the queue-level guard in
`createBackgroundWorker` are left in place as defence in depth, and
queue-level `TaskQueue.paused` behaviour is untouched.
---------
Co-authored-by: Claude <noreply@anthropic.com>
## Summary
After a deployment promotion or rollback, newly triggered runs could
keep dispatching onto the previously deployed version for up to 30
seconds. Runs now resolve the current version fresh on every dequeue, so
a promotion or rollback takes effect immediately.
## Fix
The dequeue path resolved the worker version through a 30s in-process
cache that nothing invalidated on promotion, and it loaded the worker's
entire task and queue set only to keep the single row matching the run.
Both go away: the resolve now fetches just the matched task and queue by
unique index and reads them fresh, so there is no cache left to serve a
stale version.
```
- cache.get(env:current) # 30s TTL, never invalidated -> stale
- worker + ALL tasks + ALL queues
+ worker + one task WHERE slug=... + one queue WHERE id/name=... # fresh
```
A kill-switch env var (`RUN_OPS_WORKER_VERSION_FRESH_READ_ENABLED`,
default on) falls back to the old cached path without a code deploy.
Verified end-to-end on an isolated stack: a run triggered after a
mid-stream promotion now dequeues onto the new version, with the
previous stale behavior reproduced first.
## Summary
When resolving the current worker for a development environment,
`findCurrentWorkerFromEnvironment` loaded the entire `BackgroundWorker`
row,
including the large `metadata` JSON, even though it only ever returns a
handful
of small fields. It is a frequently-run query, so the wasted payload
adds up:
every call pulled data it immediately threw away.
## Fix
Add a `select` to the development-environment lookup listing exactly the
fields
the function returns (`id`, `friendlyId`, `version`, `sdkVersion`,
`cliVersion`,
`supportsLazyAttempts`, `engine`). The query plan is unchanged, still a
single-row indexed lookup; only the row width shrinks. No behavior
change: the
dropped columns were never read.
## Summary
The worker-version resolve path fetched every column of every
`BackgroundWorkerTask` for a worker (`include: { tasks: true }`), plus
full `WorkerDeployment` and `TaskQueue` rows, just to match one task at
dequeue. That pulls large JSON columns none of this path reads (task
`payloadSchema`/`config`/`queueConfig`/`description`, deployment
`externalBuildData`/`buildServerMetadata`/`errorData`/`git`, queue
`rateLimit`), so each resolve transfers and deserializes far more than
it uses.
## Fix
Replace the includes with explicit `select`s of only the columns dequeue
reads, in both the passthrough resolver and the app resolver:
- task: `id`, `slug`, `machineConfig`, `retryConfig`,
`maxDurationInSeconds`
- deployment: `id`, `friendlyId`, `imageReference`, `imagePlatform`
- queue: `id`, `name` (the queue matcher keys on both)
The shared `ResolvedWorkerVersion` element types narrow to match
(mirrored in the cache), which also shrinks each cached worker-version
entry.
## Impact
The `tasks` read fetches every task of a worker to match one, so its
cost scales with task count and payload-schema size. For a worker with
~70 registered tasks, dropping the unread columns cuts the per-query
transfer roughly:
| Task shape | Before | After | Reduction |
|---|---|---|---|
| Light (no payload schema, small config) | ~28 KB | ~14 KB | ~54% |
| Typical (mixed schemas / config) | ~62 KB | ~14 KB | ~77% |
| Schema-heavy (large `payloadSchema`) | ~200 KB | ~14 KB | ~93% |
The `after` size is roughly fixed because the kept columns are small;
the win grows with how heavy the dropped JSON is. Narrowing `deployment`
(four JSON columns off a single row) and `queues` saves further on top.
No behavior change: pure read-shape narrowing, no flag and no schema
change, so rollback is a plain revert. Verified with a red/green
run-engine test that asserts the resolved task, deployment, and queue
carry only the used columns, plus the queue feature-matrix runs (batch,
retry-policy, machine-preset, plain trigger) that exercise the kept
columns.
## What
A relation with `onDelete: Cascade | SetNull` whose child FK column has
no index makes every parent delete fire a cascade that sequentially
scans the whole child table. That has shipped three times recently and
had to be fixed after the fact (#4554 `ProjectAlert.channelId`, #4555
`EnvironmentVariableValue.valueReferenceId`, #4588
`PersonalAccessToken.userId`).
This adds a schema-aware CI guard that catches the next one before it
merges.
## How
`apps/webapp/scripts/fkCascadeIndexGuard.ts` parses both Prisma schemas
(`@trigger.dev/database`, `@internal/run-ops-database`) and flags any
`onDelete: Cascade | SetNull` relation whose leading FK scalar is not
the leading column of some index (`@@index` / `@@unique` / `@@id` /
field-level `@id`/`@unique`) on the child model. A leading FK column
lets the cascade's `WHERE fk = $1` use the index instead of a seq scan.
It is modeled on the existing `runOpsLegacyGuard` (same `--check` gate,
same baseline-regenerate pattern), and it is lighter: it only reads
`schema.prisma` as text, so its CI job needs no Prisma client generation
and no raised heap.
## Why a baseline, not a hard rule
Not every unindexed cascade FK is a live bug. When the parent is only
ever soft-deleted, the cascade never fires, so the missing index is
harmless. Hard vs soft delete lives in application code
(`parent.delete()` vs `parent.update({ deletedAt })`), not in the
schema, and a `deletedAt` column proves neither direction. So the guard
makes no such judgment: it flags every unindexed cascade FK uniformly
and carries a baseline of the 72 currently-accepted cases. Only
violations **not** in the baseline fail `--check`.
The value is the forcing function: a newly added cascade FK stops CI and
makes the author answer "is the parent ever hard-deleted?" Add the index
if yes; regenerate the baseline with a reason if no.
## Wiring
- `apps/webapp/package.json`: `guard:fk-cascade-index` script
(regenerate with no args, gate with `-- --check`).
- `.github/workflows/fk-cascade-guard.yml`: the reusable workflow.
- `.github/workflows/pr_checks.yml`: runs on webapp-affecting changes,
aggregated into `all-checks`.
## Verification
- The three already-fixed columns are correctly seen as indexed (absent
from the baseline).
- `--check` passes on the current schemas (72 baselined, 0 new).
- A synthetic new unindexed cascade FK fails with exit 1 and an
actionable message.
- Adding `@@index([fk])`, or a composite leading with the FK, clears it.
No false positives.
- `oxfmt` and `oxlint` clean on the new script.
## Rollback
Pure tooling addition, no runtime code, no schema or data change. Revert
to remove.
## Summary
Clarifies when to add a changeset or a `.server-changes/` file. The
friction that keeps coming up is treating these as "I touched a public
package or a server app, so I owe a note." They are user-facing release
notes that go straight into the changelog customers read, not a catalog
of every change.
The guidance now leads with the real test: would a user or customer care
about this change? Add a note when the change is something they would
notice, act on, or want to hear about. Skip it otherwise, even when a
public package or server app is touched, for example:
- internal-only or admin-only changes, refactors, test-only changes,
chores
- performance or query tuning with no user-visible behavior change
- public packages that are not consumed independently (e.g.
`@trigger.dev/redis-worker`), where a version bump means nothing to a
user
Anyone who wants the exact history reads the commits.
Updates every place that encoded the old "touched a package or app, so
add a note" rule so they agree: `AGENTS.md`,
`.server-changes/README.md`, `CONTRIBUTING.md`, `CHANGESETS.md`,
`.claude/rules/server-apps.md`, and `.claude/REVIEW.md` (the last drives
automated review flagging, so it stops flagging exactly the changes the
new guidance says to skip). Also handles the mixed-PR case where the
package change needs no changeset but the server change is user-facing.
## Summary
The notifications admin list loaded every interaction row for the
notifications on the current page just to show three per-notification
counters (seen, clicked, dismissed), then counted them in memory. On
notifications with many interactions this made the page slow to load and
heavy on memory, even though only 20 notifications are shown.
## Fix
Compute the counters in a single grouped aggregate in the database
instead, returning one row per notification rather than one row per
interaction:
```sql
SELECT "notificationId",
COUNT(*) AS seen,
COUNT(*) FILTER (WHERE "webappClickedAt" IS NOT NULL) AS clicked,
COUNT(*) FILTER (WHERE "webappDismissedAt" IS NOT NULL OR "cliDismissedAt" IS NOT NULL) AS dismissed
FROM "PlatformNotificationInteraction"
WHERE "notificationId" IN (...)
GROUP BY "notificationId"
```
Behavior is unchanged; notifications with no interactions report zero.
The generated deploy Containerfile now starts from the prebuilt base
images published by base-images/ (`triggerdotdev/node` and
`triggerdotdev/bun` on DockerHub, pinned by digest) instead of
installing system packages during every project's build. Uncustomized
projects run no apt at all and their base layers are identical across
every project, so worker nodes cache one copy fleet-wide. The build
stage uses the -build toolchain variant for uncustomized and
package-only projects; projects with image instructions build FROM base
so instructions and their downloads run exactly once.
### Notes
- User packages install in their own sorted RUN with --allow-downgrades
(a pin of a preinstalled package is a downgrade against the prebuilt
base), preceded by a dpkg repair whenever instructions came first, since
apt-get install refuses to run on state a dpkg -i instruction left
broken.
- Deployed runtime images inherit newer package versions than today's
live-archive installs (the published bases upgrade everything to their
snapshot), plus the base images' OCI labels. Runtime env, user, workdir,
and entrypoint are unchanged.
## Summary
Since the move from the Remix compiler to Vite
([#4188](https://github.com/triggerdotdev/trigger.dev/pull/4188)), the
"App version" on the organization settings page shows `v0.0.0` unless
the image was built from a semver release tag (which bakes in
`BUILD_APP_VERSION`). Self-hosted builds and any image built from `main`
are affected. This restores the real version.
## Root cause
The Vite SSR bundle resolves workspace packages to TS source via the
`@triggerdotdev/source` condition, so `@trigger.dev/core`'s `VERSION`
constant is bundled as its raw `"0.0.0"` placeholder.
`scripts/updateVersion.ts` still stamps the real version at build time,
but only into the packages' dist output, which the bundle no longer
reads. The old Remix compiler bundled the stamped dist, which is why
this used to work.
The fix is a small Vite plugin that applies the same substitution to the
source version modules of `@trigger.dev/core` and `@trigger.dev/sdk`
during bundling. Beyond the settings page, this also restores real
values in the `trigger-version` request header and the version
attributes the bundled packages emit.
Verified by building the server bundle and confirming the VERSION
constants carry the package versions, with no `"0.0.0"` occurrences left
in the build output.
## Summary
Adds execution-window product surfaces for both declarative and
imperative schedules.
- Declarative schedules can set `window` through `schedules.task()`,
with support for whole-minute, hour, and percentage values.
- Imperative schedules can create, update, clear, and inspect windows
through the API and dashboard.
- Schedule API responses preserve `nextRun` as the nominal CRON time and
expose `nextRunEffectiveAt` as the stable assigned time.
- The dashboard displays configured windows alongside assigned
upcoming-run times.
- Deploy output summarizes declarative schedules and suggests adding a
wider window when the default 60-second placement range is used.
## Design
Window validation remains authoritative on the server and ensures each
window is compatible with the schedule cadence. Omitting a window uses
the default 60-second range, while explicit zero-duration windows remain
supported.
Deployment summaries are derived from the deployment's stored task
metadata, so they reflect the declarations associated with that
deployment.
## Summary
Projects that configure their own `metricExporters` or `metricReaders`
in `trigger.config.ts` were losing task metrics on nearly every run, and
seeing an unexplained `Failed to flush tracingSDK` alongside
`OTLPExporterError: Bad Request` in their run logs. Spans and logs kept
working, so the runs otherwise looked healthy.
## Root cause and fix
Every configured exporter gets its own `PeriodicExportingMetricReader`,
and `meterProvider.forceFlush()` fans out across all readers with
`Promise.all`, so two collections can land on the same millisecond.
`@opentelemetry/host-metrics` divides by the elapsed interval to compute
`process.cpu.utilization`
([common.ts](https://github.com/open-telemetry/opentelemetry-js-contrib/blob/main/packages/host-metrics/src/stats/common.ts)),
so a zero interval yields `0/0`. `JSON.stringify(NaN)` is `null`, and a
collector rejects `"asDouble": null` with a 400 that drops the
**entire** request, not just the offending point.
`flush()` and `shutdown()` now walk the metric readers one at a time, so
collections can no longer share a timestamp. Each reader is isolated, so
one failing reader cannot skip the readers behind it, and every failure
is logged with the reader that produced it. The first error is still
rethrown, so callers see failures exactly as before.
As a second layer, non-finite data points are dropped just before our
own export, so a metric that divides by zero cannot take the rest of the
batch with it. Exporters and readers supplied through
`trigger.config.ts` are untouched by that filter and still receive raw
data.
The trade-off is that configured exporters now flush after the built-in
one rather than alongside it, so flush latency is the sum rather than
the max.
An internal test package's dependency on core was replaced with a local
helper, because core now needs that package in `devDependencies` and the
two together formed a workspace cycle.
## Verification
Tested against a real collector in a container: a batch containing a
`NaN` reading is rejected with a 400 without the fix and accepted with
it, and a single flush is asserted to collect from one reader at a time.
## Summary
Fair queue consumers could leak the per-tenant concurrency slots that
gate admission. Slots were freed on some paths and skipped on others,
and once enough leaked slots accumulated for a tenant, every queue that
tenant owned stopped being served until someone cleared the set by hand.
This PR frees slots on every path and, more importantly, makes the
remaining failure modes self-healing.
## Design
The fix applies one rule uniformly: releasing a concurrency slot is
best-effort cleanup and must never block the message's primary state
transition. Blocking completion re-delivers the message, which
duplicates customer work; blocking a retry loses the attempt increment,
so the message can circle forever; blocking a reclaim strands the
message in flight. A leaked slot is the better failure in every one of
those trades because it is the only one that is recoverable. A failed
release is therefore logged and the transition proceeds.
Leaked slots then heal through two mechanisms:
- `reserve` re-admits a message that is already a member of its own
concurrency set, since re-admitting it does not increase concurrency. A
message whose earlier release failed can no longer be blocked by its own
leftover slot.
- A reconcile loop periodically removes any set member with no in-flight
record (interval configurable via `reconcileIntervalMs`, default 60s).
The check-and-remove is atomic, and it is sound because a message is
always registered in flight before its slot is reserved, so a member
with no in-flight record can only be a leak. This also covers leaks this
PR cannot prevent directly, such as a release that resolves the wrong
concurrency group from queue metadata.
Ordering hardening from earlier revisions stays: slots are released
before the in-flight record needed to describe them is discarded, the
release Lua scripts write the message back to the queue before removing
it from in-flight (Lua does not roll back on error), and dangling
in-flight entries with no payload are dropped instead of being rescanned
forever.
Every guard test was verified to fail without its specific fix,
including the duplicate-execution case: completing a message while its
slot release fails used to re-deliver and re-execute it.
Every publish now also pushes an immutable per-publish tag alongside the
mutable one, named after the snapshot date and commit (e.g.
`22-bookworm-20260812-45444a7`), so previously published digests stay
tag-referenced after republishes. Shipped CLI releases pin those
digests, so they must remain resolvable indefinitely.
Merging triggers a republish; the fresh tag-protected digests will then
be pinned by #4602 before it merges.
## Summary
Follow-up to #4595. Dashboard pages under an environment loaded every
environment in the project on each page just to resolve the one named in
the URL. On projects with many preview branches that meant reading
hundreds of (mostly archived) rows on every page load.
## Fix
The environment-scoped layout loader now scopes its lookup to the slug
in the URL (`where: { slug: envParam }`), resolving the current
environment through the `projectId, slug` composite index instead of
loading the whole project. Archived branches stay viewable by slug.
`BatchListPresenter` is bounded to the current environment, since every
batch in that list already belongs to it.
Verified on a project seeded with 2,000 archived branch environments:
the layout lookup drops from all environments to one, and both a normal
environment page and an archived branch page render correctly.
## Summary
4 new features, 24 improvements, 10 bug fixes.
## Highlights
- Allow `trigger deploy` to authenticate with an environment API key
from `TRIGGER_ACCESS_TOKEN`.
([#4561](https://github.com/triggerdotdev/trigger.dev/pull/4561))
## Improvements
- Chat in the browser now reconnects when the connection drops mid-turn,
instead of leaving the reply stuck as if it were still generating.
Reports can be fetched as structured data with the `json` format, and
the shortest report period is now one minute (`1m`, `30m`, `1h`, `7d`).
The `mint-token` command's help is clearer too: a token minted without
`--cap` is read-only, and `--ttl` shows the correct maximum lifetime of
7 days.
([#4418](https://github.com/triggerdotdev/trigger.dev/pull/4418))
- The dev environment onboarding now tracks real progress. After you run
`init`, the setup checklist marks your project as initialized, and it
updates live as your dev server connects and your tasks register. The
blank state also adds a "Copy AI agent prompt" button that copies a
ready-to-paste setup prompt (pre-filled with your project reference) for
Claude Code, Cursor, or any coding agent.
([#4563](https://github.com/triggerdotdev/trigger.dev/pull/4563))
The `init` scaffold now imports from `@trigger.dev/sdk` instead of the
deprecated `@trigger.dev/sdk/v3` subpath.
- Deployed images now ship dependencies and bundled task code as
separate layers. Repeat deploys with unchanged dependencies typically
push and pull far less data, making deploys and worker image pulls
faster.
([#4551](https://github.com/triggerdotdev/trigger.dev/pull/4551))
- The current-worker API now reports each task's queue, so you can see
which tasks write to a given queue.
([#4525](https://github.com/triggerdotdev/trigger.dev/pull/4525))
- Watch-mode chat streams now survive quiet windows and page reloads,
and a reply cut off by a lost connection shows an error instead of
appearing finished. Aborting a resumed subscription only closes your
local stream — call `stopGeneration(chatId)` or pass `stopOnAbort: true`
to stop the run. Also fixed a race where quickly restarting a stream
could break stop and reconnect, and stopping a chat now hands it back to
your other tabs instead of leaving them read-only.
([#4516](https://github.com/triggerdotdev/trigger.dev/pull/4516))
## Server changes
These changes affect the self-hosted Docker image and Trigger.dev Cloud:
- The dashboard agent now has a monthly message allowance and plan-based
limits on watches. Queries stay read-only with clearer errors when busy,
and messages with unusual characters no longer fail to send.
([#4516](https://github.com/triggerdotdev/trigger.dev/pull/4516))
- Meet the dashboard agent: a chat in every environment that answers
questions about your runs, queues, errors and health with real data and
links, replacing Ask AI everywhere it used to appear. Investigate a
failed run, an error, a backed-up queue or a run that hasn't started to
get a worked-through answer — what happened, why, and how to fix it,
with every claim linked to the runs, errors and deploys behind it. It
reads your data read-only, works on preview and dev branches with that
branch's own data, and reads the same everywhere — dashboard, terminal,
editor. A very long chat keeps working: the agent summarises the earlier
part and carries on.
**Watch…** on a run, queue, error or the health report tells you when
things change: a run finishes, a queue clears or grows past a number you
pick, an error comes back, an environment recovers. The answer arrives
in the chat and, if you want, by email, Slack or webhook — and the agent
can look into bad news on its own. A watch reaches you on any browser
you sign in from, without opening the chat first.
A sample of conversations is scored automatically so the agent keeps
getting better; only the score and a one-line summary are kept, never
your messages, data or code, and we can switch it off for your
organization on request. Ask the agent instead of the Docs buttons in
page headers — they stay there when the agent isn't available to you.
Separately, a queue's wait times, peak depth, throughput and throttling
can now be read from the API.
([#4418](https://github.com/triggerdotdev/trigger.dev/pull/4418))
- Add backend support for delaying cron schedules within a specified
window with a minimum of 60 seconds.
([#4566](https://github.com/triggerdotdev/trigger.dev/pull/4566))
- Reduced recurring background database load from the billing-limit
recovery check, so paused environments are reconciled with less
overhead.
([#4590](https://github.com/triggerdotdev/trigger.dev/pull/4590))
- Validating a schedule when deploying or updating a schedule now does
less work on projects with many preview branches, so those operations
stay fast as branches accumulate.
([#4598](https://github.com/triggerdotdev/trigger.dev/pull/4598))
- Project pages now load faster for projects with a large number of
preview branches, by no longer loading archived branch environments that
aren't shown.
([#4595](https://github.com/triggerdotdev/trigger.dev/pull/4595))
- Database queries that filter on a list of values now reuse cached
query plans more consistently, instead of forcing the database to
re-plan whenever the list length changes.
([#4480](https://github.com/triggerdotdev/trigger.dev/pull/4480))
- Routine cleanup of old dashboard agent data now runs on its own
schedule.
([#4599](https://github.com/triggerdotdev/trigger.dev/pull/4599))
- Database connection metrics are now reported for every configured
database connection instead of only the primary one, and stay accurate
regardless of connection type.
([#4541](https://github.com/triggerdotdev/trigger.dev/pull/4541))
- Deployment-related API endpoints now draw from their own generous rate
limit budget, configurable via the `DEPLOYMENT_RATE_LIMIT_*` environment
variables, so runtime API traffic no longer competes with deployments
for the same per-environment budget.
([#4565](https://github.com/triggerdotdev/trigger.dev/pull/4565))
- Deleting or editing a secret environment variable is now fast and no
longer slows down as a project accumulates variables.
([#4555](https://github.com/triggerdotdev/trigger.dev/pull/4555))
- Speed up personal access token lookups by indexing them on their owner
([#4588](https://github.com/triggerdotdev/trigger.dev/pull/4588))
- Switching project or organization in the sidebar now keeps you on the
same page instead of sending you back to Tasks. Pages for a specific
run, deploy or other single item open the matching list instead.
([#4585](https://github.com/triggerdotdev/trigger.dev/pull/4585))
- Reduced database load when loading the dashboard by removing an unused
organization member count that was being calculated on every page
navigation.
([#4587](https://github.com/triggerdotdev/trigger.dev/pull/4587))
- The environment variables page now loads a page at a time, keeping it
fast for projects with a large number of variables. Search matches
variable names across every page.
([#4597](https://github.com/triggerdotdev/trigger.dev/pull/4597))
- Groundwork for an alternative database connection driver, gated behind
configuration and disabled by default, so there is no change to default
behavior.
([#4539](https://github.com/triggerdotdev/trigger.dev/pull/4539))
- Deleting an alert channel is now fast and no longer slows down as a
project builds up alert history.
([#4554](https://github.com/triggerdotdev/trigger.dev/pull/4554))
- Reduced internal overhead on the API under high load.
([#4532](https://github.com/triggerdotdev/trigger.dev/pull/4532))
- Out-of-date upgrade prompts no longer appear in the dashboard: the
"V4" badges and the notices saying preview branches and the queues table
need V4 have been removed. The side menu still warns you when a project
is on v3, with updated wording and a link to the v4 upgrade guide.
([#4589](https://github.com/triggerdotdev/trigger.dev/pull/4589))
- Make background worker registration cheaper for projects with many
scheduled tasks by scoping declarative schedule reconciliation to the
current environment and dropping redundant schedule lookups.
([#4577](https://github.com/triggerdotdev/trigger.dev/pull/4577))
- Speed up setting and importing environment variables for projects with
many variables.
([#4579](https://github.com/triggerdotdev/trigger.dev/pull/4579))
- Loading the deployments list is now faster, especially when filtering
by deployment status on projects with many deployments.
([#4591](https://github.com/triggerdotdev/trigger.dev/pull/4591))
- Fixed the billing limits page timing out for organizations with many
preview branches, especially while a spend limit was being enforced. The
page now loads quickly, so you can raise or resolve your limit without
delay. ([#4594](https://github.com/triggerdotdev/trigger.dev/pull/4594))
- Fix the Concurrency page showing the plan's default concurrency for
the dev environment instead of the environment's actual limit.
([#4596](https://github.com/triggerdotdev/trigger.dev/pull/4596))
- Creating an organization sometimes left you back on the creation form
even though the organization had already been created, so clicking
Create again made a duplicate. Creating an organization now completes
and takes you to your new organization.
([#4530](https://github.com/triggerdotdev/trigger.dev/pull/4530))
- Ensure creating a project completes instead of returning to its
creation form after a navigation error.
([#4584](https://github.com/triggerdotdev/trigger.dev/pull/4584))
- Renaming a project now keeps you on the project settings page and
tells you what happened, instead of silently moving you to the tasks
page or clearing the form with no explanation.
([#4601](https://github.com/triggerdotdev/trigger.dev/pull/4601))
- Fixed support threads showing no account details for some customers,
so the team can see your plan, organizations and projects when you get
in touch.
([#4575](https://github.com/triggerdotdev/trigger.dev/pull/4575))
- In the light theme, the Format, Clear and Copy buttons on the query
editor no longer blend into the query text behind them.
([#4592](https://github.com/triggerdotdev/trigger.dev/pull/4592))
- The health report now says start latency is "unknown" when there is no
data for it, instead of showing a healthy-looking 0ms
([#4544](https://github.com/triggerdotdev/trigger.dev/pull/4544))
- Realtime streams written inside a chat session run now use the same
backend as the session itself, and runs are no longer created against a
backend that cannot serve them.
([#4564](https://github.com/triggerdotdev/trigger.dev/pull/4564))
- The grouped "watch updates" notification now shows the total number of
results waiting, instead of only the most recent batch's count.
([#4525](https://github.com/triggerdotdev/trigger.dev/pull/4525))
<details>
<summary>Raw changeset output</summary>
# Releases
## @trigger.dev/build@4.5.11
### Patch Changes
- Updated dependencies:
- `@trigger.dev/core@4.5.11`
## trigger.dev@4.5.11
### Patch Changes
- Chat in the browser now reconnects when the connection drops mid-turn,
instead of leaving the reply stuck as if it were still generating.
Reports can be fetched as structured data with the `json` format, and
the shortest report period is now one minute (`1m`, `30m`, `1h`, `7d`).
The `mint-token` command's help is clearer too: a token minted without
`--cap` is read-only, and `--ttl` shows the correct maximum lifetime of
7 days.
([#4418](https://github.com/triggerdotdev/trigger.dev/pull/4418))
- Allow `trigger deploy` to authenticate with an environment API key
from `TRIGGER_ACCESS_TOKEN`.
([#4561](https://github.com/triggerdotdev/trigger.dev/pull/4561))
- The dev environment onboarding now tracks real progress. After you run
`init`, the setup checklist marks your project as initialized, and it
updates live as your dev server connects and your tasks register. The
blank state also adds a "Copy AI agent prompt" button that copies a
ready-to-paste setup prompt (pre-filled with your project reference) for
Claude Code, Cursor, or any coding agent.
([#4563](https://github.com/triggerdotdev/trigger.dev/pull/4563))
The `init` scaffold now imports from `@trigger.dev/sdk` instead of the
deprecated `@trigger.dev/sdk/v3` subpath.
- Deployed images now ship dependencies and bundled task code as
separate layers. Repeat deploys with unchanged dependencies typically
push and pull far less data, making deploys and worker image pulls
faster.
([#4551](https://github.com/triggerdotdev/trigger.dev/pull/4551))
- Updated dependencies:
- `@trigger.dev/core@4.5.11`
- `@trigger.dev/build@4.5.11`
- `@trigger.dev/schema-to-json@4.5.11`
## @trigger.dev/core@4.5.11
### Patch Changes
- Chat in the browser now reconnects when the connection drops mid-turn,
instead of leaving the reply stuck as if it were still generating.
Reports can be fetched as structured data with the `json` format, and
the shortest report period is now one minute (`1m`, `30m`, `1h`, `7d`).
The `mint-token` command's help is clearer too: a token minted without
`--cap` is read-only, and `--ttl` shows the correct maximum lifetime of
7 days.
([#4418](https://github.com/triggerdotdev/trigger.dev/pull/4418))
- The current-worker API now reports each task's queue, so you can see
which tasks write to a given queue.
([#4525](https://github.com/triggerdotdev/trigger.dev/pull/4525))
## @trigger.dev/python@4.5.11
### Patch Changes
- Updated dependencies:
- `@trigger.dev/core@4.5.11`
- `@trigger.dev/sdk@4.5.11`
- `@trigger.dev/build@4.5.11`
## @trigger.dev/react-hooks@4.5.11
### Patch Changes
- Updated dependencies:
- `@trigger.dev/core@4.5.11`
## @trigger.dev/redis-worker@4.5.11
### Patch Changes
- Updated dependencies:
- `@trigger.dev/core@4.5.11`
## @trigger.dev/rsc@4.5.11
### Patch Changes
- Updated dependencies:
- `@trigger.dev/core@4.5.11`
## @trigger.dev/schema-to-json@4.5.11
### Patch Changes
- Updated dependencies:
- `@trigger.dev/core@4.5.11`
## @trigger.dev/sdk@4.5.11
### Patch Changes
- Chat in the browser now reconnects when the connection drops mid-turn,
instead of leaving the reply stuck as if it were still generating.
Reports can be fetched as structured data with the `json` format, and
the shortest report period is now one minute (`1m`, `30m`, `1h`, `7d`).
The `mint-token` command's help is clearer too: a token minted without
`--cap` is read-only, and `--ttl` shows the correct maximum lifetime of
7 days.
([#4418](https://github.com/triggerdotdev/trigger.dev/pull/4418))
- Watch-mode chat streams now survive quiet windows and page reloads,
and a reply cut off by a lost connection shows an error instead of
appearing finished. Aborting a resumed subscription only closes your
local stream — call `stopGeneration(chatId)` or pass `stopOnAbort: true`
to stop the run. Also fixed a race where quickly restarting a stream
could break stop and reconnect, and stopping a chat now hands it back to
your other tabs instead of leaving them read-only.
([#4516](https://github.com/triggerdotdev/trigger.dev/pull/4516))
- Updated dependencies:
- `@trigger.dev/core@4.5.11`
</details>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
## Summary
On the Concurrency page, the dev environment row's Total always showed
the plan's included dev concurrency, even when the environment's limit
had been raised. The row's own "Extra concurrency" value was already
derived from the real limit, so the two columns could disagree with each
other.
## Root cause
The Total cell renders `planConcurrencyLimit + allocation`, where
`allocation` is the state behind the editable prod/staging inputs. Dev
environments are deliberately excluded from that allocation map (dev
concurrency is not purchasable), so the dev row's allocation always
resolved to 0 and the Total fell back to the plan value. The dev row now
renders the environment's actual `maximumConcurrencyLimit` instead.
## What & why
Org avatars disappeared from the sidebar, replaced by alt text spilling
across it. Two bugs stacked: the document img-src CSP pins the Google
favicon endpoint org avatars are stored as, but Google 302-redirects it
to `tN.gstatic.com` and CSP re-checks the redirect target, so the avatar
is refused. Changelog images served from `trigger.dev` in the agent chat
were also missing from the allowlist. And `Avatar.tsx` had no clipping
and no error fallback, so a refused image degraded into overflowing alt
text.
## What's inside
**CSP allowlist** — `app/utils/cspImageOrigins.ts`: the base sources
gain the four gstatic shards `t0`–`t3.gstatic.com`, path-pinned to
`/faviconV2`, plus `https://trigger.dev/changelog/` as a path prefix. No
wildcards — the no-wildcard beacon policy stands. The shard hosts are
Google-operated with no public write path, so the enumeration is as
narrow as the existing `s2/favicons` entry; if Google ever adds a `t4`,
the failure mode is one broken avatar, not a broken page.
**Avatar fallback** — `app/components/primitives/Avatar.tsx`: the image
box clips, and a failed load falls back to the globe icon. That covers
failures before hydration too — `onError` never replays for a node that
already failed, so a ref checks `complete && naturalWidth === 0` at
attach time. The error state resets when the URL changes
(`key={avatar.url}`).
**Radio card theming** — `app/components/primitives/RadioButton.tsx`: in
the dark themes the checked radio card rendered darker than the
unchecked ones. Unchecked cards now sit on `background-bright`
(near-black in dark, unchanged white in light) and the checked card uses
the `surface-control` tokens, so selection reads black → grey in dark
themes; light theme keeps its current look. The API keys route keeps its
indigo checked-hover via an explicit override.
## Testing
The CSP test helper now implements CSP's real path-matching rule
(trailing slash = prefix, otherwise exact, query ignored) and asserts
the pins hold: the gstatic redirect target passes, `beacon.png` on
gstatic, a `t9` shard, and non-changelog `trigger.dev` paths stay
blocked. 39 tests green plus webapp typecheck. Verified against a
running webapp that the served directive contains the new sources.
## What & why
The dashboard agent's upkeep — retention deletes and the investigation
sweep — ran as cron jobs on the webapp's common worker, even though it
only touches the agent's own datastore. This moves that upkeep into the
agent's Trigger project as scheduled tasks (TRI-13182).
## What's inside
**Retention** — `internal-packages/dashboard-agent/src/maintenance.ts`,
a daily task (03:00 UTC). Deletes turn evals older than 30 days,
hard-deletes chats soft-deleted more than 30 days ago, and purges
terminal watches and submission rows older than 7 days. It used to run
every 5 minutes; nothing needs a hard delete that fast, so it is daily
now, draining in bounded batches and warning if it hits the cap. It
retries (3 attempts) because the next run is a day away. It connects
with `DASHBOARD_AGENT_DATABASE_URL`, falling back to `DATABASE_URL` like
every other task in the package (the deletes are confined to the agent's
own Postgres schema), and skips when neither is set.
**Investigation sweep** — `src/investigation-sweep.ts`, every 5 minutes,
same as before: settles investigation cards stuck `in_progress`
(30-minute window, attempt cap, force-abandon note). It keeps the fast
cadence because it fixes live state the UI is showing.
**What stays in the webapp.** The watch finalize/deliver sweep and batch
rearm: they cover a dead agent-side tick chain — a backstop can't live
inside the thing it backstops — and they need the main database and the
alerts worker. The org-deletion chat purge also stays: deletion must not
depend on the agent project being deployed. The removed cron job keeps a
cron-less tombstone entry so already-queued items drain cleanly; remove
it in a follow-up.
**Test plumbing** — the drizzle migration replayer that webapp tests
hand-rolled is now exported once from
`@internal/dashboard-agent-db/testing`; the moved tests live in the
agent package as `src/*.test.ts` against real Postgres.
## Testing
Agent package: retention passes (backlog drain, batch cap, no-op guard,
chat-delete cascade) and the sweep, on testcontainers Postgres. Webapp:
the watch/chat suites, plus a test that a settlement card stops the
dashboard spinner. Full typecheck on both.
Publishes the base images deployed task containers will build on:
`triggerdotdev/node:{21,22,24,26}-bookworm` and
`triggerdotdev/bun:1.3-node20-bookworm`, each with a `-build` toolchain
variant, multi-arch, built from a pinned Debian snapshot so every
published layer is reproducible from recorded inputs and carries a
GitHub provenance attestation.
Publishing runs on manual dispatch and on merges touching
`base-images/`. Pull requests build without pushing.
## What
`CheckScheduleService.call` loaded **every** environment of a project
(`{ id, type, archivedAt }`, no filter) and then immediately narrowed to
just the requested `environmentIds` via
`resolveProjectScopedEnvironments`. It only ever uses the requested envs
(to reject foreign env ids and reject archived branches). On a
preview-heavy project that meant loading hundreds of archived branch
rows to validate one, on a path called in a per-scheduled-task loop on
the deploy path (`createBackgroundWorker` -> `syncDeclarativeSchedules`)
and from `upsertTaskSchedule`.
The query is index-backed and individually fast (rows_read/returned = 1
per predicate), so this is about result-set width / egress and wasted
work at scale (~580k calls/24h observed via Insights), not a slow plan.
## Change
Bound the `environments` relation load to `boundedIn(environmentIds)`:
```ts
environments: {
where: { id: { in: boundedIn(environmentIds) } },
select: { id: true, type: true, archivedAt: true },
}
```
Returns `<=` the number of requested envs (usually 1) instead of the
whole project. Both existing behaviors are preserved:
- **Foreign-id rejection**: the relation is still scoped to the project,
so a requested id belonging to another project never comes back and
`resolveProjectScopedEnvironments` reports it as `foreign` (a missing
requested id is already treated as foreign).
- **Archived-branch rejection**: a requested id that is an archived
branch still comes back with `archivedAt` set, so the downstream `Can't
add or edit a schedule for an archived branch` check still fires.
`archivedAt` is kept in the select deliberately, so this bounds by id
rather than filtering archived rows out.
## Evidence (isolated stack, seeded 1 prod env + 40 archived branch
envs)
Local `EXPLAIN (ANALYZE)` of the exact environments sub-select:
| | rows returned | buffers |
|---|---|---|
| before (unbounded) | **41** | shared hit=12 |
| after (`id IN (requested)`) | **1** (`Rows Removed by Filter: 40`) |
shared hit=4 |
Same `RuntimeEnvironment_projectId_idx`, no plan change. Rows to the
client drop to `len(environmentIds)`, which is the point.
**Unit (vitest, testcontainers, real Postgres):**
`apps/webapp/test/checkSchedule.test.ts` extended to prove, on real
rows, that the bounded load returns only the requested env (1 of 10),
still reports a foreign id as foreign, and still surfaces an archived
branch when it is the requested one. 5/5 pass.
**Full e2e (both execution modes, real stack):** a purpose-built project
with two declarative `schedules.task`s.
- `trigger dev`: dev worker created, both schedules synced through the
edited `checkSchedule` loop, no errors.
- `trigger deploy` (managed deployment): PRODUCTION worker registered,
both schedules synced against the **prod** environment through the same
loop, prod + dev schedule instances active, no errors.
`typecheck --filter webapp` clean.
## Rollout / rollback
Straight deploy, no flag, no migration. Rollback is revert-only
(read-path narrowing, no data change). Old and in-flight rows read
correctly under both the old and new code.
## Out of scope
The two lower-priority sibling reads in the ticket (the Query/metrics
env id->slug map and the env-var repository fan-out) are left for
follow-ups; they need caching / per-method scoping rather than this
single bound.
## What
The environment variables settings page loaded **every** variable in the
project in one shot, with a nested `values` read plus a `valueReference`
(SecretReference) sub-load that was selected but never read. For a
project with many variables this pulled `variables × environments` value
rows (~18k for large projects) on every page load, plus a matching
~18k-row `SecretReference IN` query.
This paginates the presenter by variable key and removes the dead
include.
- Remove the never-read `valueReference: { select: { key } }` include →
the `SecretReference` query is gone entirely.
- Paginate the parent variable query: `count` + `orderBy key` +
`skip/take`, page size 50 → the value read is bounded to `pageSize ×
environments` per page.
- Scope the count and the page to variables that have a value in a
displayed environment (`values: { some: { environmentId: { in } } }`),
so `totalCount`/`totalPages` and the `skip/take` window match what
actually renders (no phantom empty pages from variables that live only
in archived branches or another member's dev env).
- Display order comes from the DB `orderBy: { key: "asc" }` — the
presenter no longer re-sorts each page with `localeCompare`, which under
pagination could disagree with the DB collation at page boundaries.
- The secret-value lookup (`SecretStore` keys) and the updater lookup
(`user` by id) are now scoped to the current page instead of the whole
project.
- Search moves server-side (variable key, case-insensitive) and drives
both the count and the page; the UI gains standard pagination controls.
## Why
The two correlated ~18k-row control-plane queries flagged in the ticket
come from this settings-page presenter, not from any hot path. Both are
index-covered (`rows_read == rows_returned`); the issue is the sheer
volume fetched in one burst. Bounding it per page removes the burst.
## Evidence
Measured on an isolated stack with a seeded project of 1000 variables ×
3 environments (3000 value rows), using Prisma's emitted-SQL log:
| | SecretReference query | value rows fetched |
| --- | --- | --- |
| before | 1 | 3000 |
| after | **0** | **150** (page 1) + one `count` |
`EXPLAIN` on Prisma's verbatim statements (index confirmed via
`enable_seqscan=off`; the local table is too small for the planner to
choose them by default):
- `count` (`WHERE projectId AND EXISTS(values in displayed envs)`) →
Hash Join: Index Scan on `EnvironmentVariable_pkey` + Bitmap Index Scan
on `EnvironmentVariableValue_environmentId_idx`
- paginated parent (`WHERE projectId AND EXISTS(...) ORDER BY key
LIMIT/OFFSET`) → Nested Loop Semi Join: Index Scan on
`EnvironmentVariable_projectId_key_key` (**no Sort node**) driving an
Index-Only Scan on
`EnvironmentVariableValue_variableId_environmentId_key`
- nested values (`variableId = ANY … AND environmentId = ANY …`) → index
scan on `EnvironmentVariableValue_environmentId_idx`
- `SecretStore` keys (`key = ANY …`) → index scan on
`SecretStore_key_idx`
No new index required. Verified in the browser on the seeded project: 20
pages, page navigation, server-side search (matches across all pages),
last page renders, no app console errors. `typecheck`, `oxlint`, `oxfmt`
all clean.
## Behavior change
The previous client-side search matched variable **name and value** (and
environment type / branch name). Values are encrypted at rest and
resolved separately, so they cannot be searched server-side under
pagination. Search is now **variable-name only**, server-side,
case-insensitive. Projects with fewer than one page of variables see no
pagination bar and no visible change.
## Rollout / rollback
Pure read-path change on a dashboard loader, no schema or data
migration. Rollback is a straight revert.
## Screenshots
<img width="2400" height="1794" alt="01-page1"
src="https://github.com/user-attachments/assets/d4a7effd-d167-4dd6-92f4-6e9174818acd"
/>
<img width="2400" height="1794" alt="02-search-single"
src="https://github.com/user-attachments/assets/cd113ca9-ff87-431f-b2f6-7f7d36f2b32a"
/>
## What
Several project pages loaded **every** `RuntimeEnvironment` row for a
project, including the archived preview-branch environments that are
never shown in the UI. On a project with heavy preview-branch usage that
means thousands of rows per load, producing a large result set and a
rare multi-second tail on the environment lookup (~30s outlier observed
via Insights on `RuntimeEnvironment` projectId lookups, fingerprint
`f2b3ecab…`).
The tail is dominated by the size of the result being
parsed/transferred, not by the query plan (it already used
`RuntimeEnvironment_projectId_idx` with no over-read). So the fix is to
stop returning archived branch environments.
## Diagnosis correction
The ticket framed this as a "large `projectId IN` list" and suggested
bounding the IN list / cursor pagination. It's actually a Prisma
**nested relation load** on a *single-project* `project.findFirst`, so
the `IN (...)` holds one projectId and the trailing `OFFSET $1` is
Prisma's relation-subquery artifact. The 4,644 rows in the observed
execution were **one project with ~4,644 environments** (accumulated
archived branches), not many projects.
## Change
Filter the `environments` relation load to `archivedAt: null` (base envs
never archive, so only archived preview branches are excluded):
- `ProjectPresenter.server.ts`
-
`orgs.$organizationSlug.projects.$projectParam.{concurrency,apikeys,environment-variables,settings}.ts`
(best-env resolvers)
And remove an **unused** `environments` select from
`DeploymentListPresenter.server.ts` (it was selected but never read).
`loadProjectEnvironments` (replay route) already filters `archivedAt:
null` + env type; this change follows that existing precedent.
## Evidence (isolated stack, seeded one project with 2,000 archived
branch envs + 4 active)
`EXPLAIN (ANALYZE)` of the exact presenter sub-select:
| | rows returned | index |
|---|---|---|
| before (unfiltered) | **2004** | `RuntimeEnvironment_projectId_idx` |
| after (`archivedAt IS NULL`) | **4** (`Rows Removed by Filter: 2000`)
| same index, no plan change |
500x fewer rows to the client, which is what removes the parse-on-load
tail. No new index needed. `typecheck --filter webapp` clean. UI
verified: project layout, Deploys page, and the concurrency best-env
redirect all render with the 2,000 archived branches present in the DB
and zero console errors.
## Rollout / rollback
Straight deploy, no migration. Rollback is revert-only (read-path
filter, no data change). Old and in-flight rows read correctly under
both the old and new code.
## Limitation
A project with thousands of *active* branches would still load them all;
in practice active branches are few (branches are archived when their
work merges). Hard-bounding active branches would be a larger change and
is out of scope here.
## Summary
Opening the billing limits page while a spend limit was being enforced
could time out with no response for organizations with many preview
branches. That is exactly the moment the page matters: it is the only
self-serve way to raise or resolve the limit. The page now loads fast
regardless of how many environments the organization has.
## Root cause and fix
The loader's queued-run count ran one ClickHouse count per billable
environment, sequentially, with no timeout, and the environment list
included every archived preview branch ever created. Thousands of
environments times one round trip each held the response open past the
edge timeout.
The count is now a single org-level ClickHouse query filtered on
environment type, capped server-side with max_execution_time. If the
count fails, the loader falls back to 0 (the page hides the count label
at 0) instead of throwing, so the recovery panel stays reachable even
when the count errors. The billing-limit bulk-cancel path also stops
enumerating archived environments.
Restores the page-header docs buttons removed in #4529 / #4418, shown
only when the dashboard agent is unavailable (feature flag off, or pages
outside the environment layout). The buttons are restored verbatim at
their original spots — 22 sites across 21 files — wrapped in a small
`WhenAgentUnavailable` gate that reads the agent context (SSR-safe, no
hydration flicker).
Also: in the light theme, the query editor's Format/Clear/Copy toolbar
gets a translucent white background (`light:bg-white/80`) instead of
transparent, so it no longer blends into the code behind it.
<img width="1215" height="133" alt="Screenshot 2026-08-12 at 17 26 02"
src="https://github.com/user-attachments/assets/4bdba825-9cbd-4df4-b6ca-0ea6691a534b"
/>
<!-- ccr-slack-attribution -->
_Requested by **Eric Allam** · [Slack
thread](https://triggerdotdev.slack.com/archives/C0BEM9Z73TM/p1786528784863449)_
**Before:** you're on the API keys page in project X, you switch to
project Y in the sidebar, and you land on project Y's Tasks page. Same
for switching organization. Every switch threw away the page you were
looking at.
**After:** you land on project Y's API keys page. Switching organization
does the same thing, one project down. Pages that name a single thing —
a run, a batch, a queue, a schedule, a deploy, a session, an error group
— can't exist in another project, so those take you to the matching list
page instead (a run page takes you to Runs).
The environment is still chosen exactly as it is today: nothing tries to
guess it in the browser.
---
## Testing
- New `apps/webapp/app/utils/pageSwitching.test.ts` (35 tests). It reads
the compiled Remix route manifest, so the portable-page list can't
silently drift from the routes:
- every environment page that names no resource survives an environment
switch — the same pages the old slug swap kept
- the two branch lists are the only pages an environment switch keeps
and a project switch drops
- the pages gated per organization — Logs, Query and the queue metrics
dashboard — travel with an environment or project switch but not an
organization switch, and that list is derived from the route sources so
a new gated page cannot be missed
- every portable page points at a route that exists
- every one of the 19 environment routes that takes a resource id
truncates to a list page, with the id gone
- portable pages resolve to themselves, so switching twice lands in the
same place
- every rejection case: leading slash, `//`, absolute URL, `..`,
percent-encoded traversal, `javascript:`, unknown page — each falls back
to Tasks rather than being sanitised into something
- Manual: switch project and organization from API keys, project
settings, a run page, and a queue page.
- `pnpm run typecheck --filter webapp` passes.
- The rest of the webapp suite needs Docker for testcontainers, which
wasn't available here; all colocated pure unit tests under `app/utils/`
pass (15 files, 150 tests).
---
## Changelog
Switching project or organization in the sidebar keeps you on the same
page instead of sending you back to Tasks. Pages for a specific run,
deploy or other single item open the matching list instead.
---
## How
The switcher links already pointed at `/orgs/:org/projects/:project` and
`/orgs/:org`, whose `_index` loaders resolve the best environment (and,
for the organization, the best project) and redirect. So the page
travels as a search param on those links, and each loader appends it to
the path it already builds:
- `app/utils/pageSwitching.ts` — one pure module.
`environmentPortablePage(suffix)` and `projectPortablePage(suffix)` walk
up the suffix until they find an entry in an allowlist of portable
pages, and answer with the environment root if they find none. The
result is therefore always a literal from that closed set, which is what
makes it safe to concatenate into a redirect target; there is no regex
sanitising. The allowlist is built from the landing pages already listed
in `deeplinkPages.ts` plus the handful of nested pages that file doesn't
know about, so this isn't a new URL-shape table.
- `app/hooks/useEnvironmentSwitcher.ts` — `usePageSwitcher()` derives
the current page by slicing the environment layout route match's
pathname off the current pathname, so there's no route table on the
client either. The query string and hash are dropped on a project or
organization switch, since filters encode task slugs and ids scoped to
the project you're leaving.
- Both `_index` loaders re-validate the page through the same function
before using it.
Two things worth a look:
- **The environment switcher's truncation gap is fixed as a side
effect.** It had a hand-written switch covering `runs/:runParam`,
`deployments/:deploymentParam` and `schedules/:scheduleParam`; the other
16 id-bearing routes carried their id straight into the new environment
(e.g. `queues/:queueParam`, `batches/:batchParam`,
`errors/:fingerprint`, `sessions/:sessionParam`). All three switchers
now share one truncation, and the test asserts it covers every such
route in the manifest.
- **Portability turned out to be two properties, not one.** Preview
branches and dev branches render under any environment slug of their
project — both loaders pass a hardcoded environment type and the project
slug and never read `envParam` — so an environment switch keeps them,
exactly as swapping the slug did before. A project or organization
switch still falls back to Tasks, since the project you land in may have
no preview branches. A test locks the environment half: every id-free
page below an environment has to survive an environment switch.
---
## Screenshots
_n/a — no visual change; only where the switcher links point._
💯
---------
Co-authored-by: Claude <noreply@anthropic.com>
Plain sends `customer.externalId` as an explicit `null` rather than
omitting the key. The schema validated it with `z.string().optional()`,
which accepts `undefined` but rejects `null`, so every customer we don't
set an `externalId` for got a 400 instead of a card — while the rest
worked, which made it look intermittent.
`email`, `externalId` and `thread` are now `nullish`. One of
email/externalId is still required, and the route's existing email
fallback resolves these customers.
Three related fixes in the same path:
- The route returned `{ cards: [] }` when no user matched. Plain records
an integration error for any requested key it doesn't get back, so that
surfaced as a broken card rather than a hidden one. Every requested key
is now answered, with `components: null` where there's no data.
- The impersonation link is offered only when the customer matched on
`externalId` — a value we set ourselves. An email match is a weaker
claim, since the address on a Plain customer isn't verified and for
customers created outside our own writes it comes from whoever sent the
message. Email-matched customers get the account rows without a
one-click impersonation link.
- The not-found log recorded raw customer identifiers; it now keeps
presence flags only.
The schema and the response helper moved to
`app/utils/plainCustomerCards.ts` so they can be unit-tested without
pulling in the db and env modules.
## Testing
`app/utils/plainCustomerCards.test.ts` — 11 tests covering the null
shapes, the every-key-answered response, and the missing-vs-zero
distinction. Verified locally.
Split out of #4571, which bundled this with an unrelated impersonation
fix.
## Summary
The two personal-access-token lookups by `userId` (one also filtering
`revokedAt is null`, the other also filtering `name`) had no index on
`userId`, so each did a full sequential scan of the
`PersonalAccessToken` table to return a single row. `userId` is also an
unindexed foreign key.
## Fix
Add a single `@@index([userId])`. A user owns only a handful of PATs, so
once `userId` is indexed each lookup touches a few rows and the residual
`revokedAt` / `name` filter is trivial. Both query shapes lead with
`userId =`, so one index serves both and a composite would only add
write cost. The migration uses `CREATE INDEX CONCURRENTLY IF NOT
EXISTS`, which is online-safe under write load and reversible by
dropping the index.
Verified with a seeded local EXPLAIN: both queries go from a full
sequential scan to an index scan on the new index.
## What
`OrganizationsPresenter.#getOrganizations` selected a Prisma
`_count.members` relation on every org-list load (hit on nearly every
dashboard navigation). Prisma lowers that relation `_count` to a
whole-`OrgMember`-table `GROUP BY organizationId` aggregate joined onto
`Organization`. The computed `membersCount` field is read by **nothing**
in the webapp, so the entire aggregate scan is wasted work.
This removes the `_count` select and the `membersCount` field. The query
keeps only the indexed `EXISTS` membership filter and the org/project
selects.
## Why it's safe
- `membersCount` has zero consumers (whole-webapp grep finds the name
only at the point of assignment). It was added in #1796 (2023) and has
been unused since.
- The member count shown on the org settings/team page comes from a
separate presenter query, not this one. No user-visible change.
## Evidence (generated SQL, before/after, seeded isolated stack)
Before (with `_count.members`):
```sql
SELECT ..., COALESCE(aggr._aggr_count_members, 0)
FROM "Organization"
LEFT JOIN (SELECT "organizationId", COUNT(*) AS _aggr_count_members
FROM "OrgMember" GROUP BY "organizationId") aggr ON ...
WHERE EXISTS (... "userId" = $1 ...) AND "deletedAt" IS NULL
ORDER BY "createdAt" DESC
```
After:
```sql
SELECT id, slug, title, avatar, "featureFlags"
FROM "Organization"
WHERE EXISTS (... "userId" = $1 ...) AND "deletedAt" IS NULL
ORDER BY "createdAt" DESC
```
The whole-table `GROUP BY` aggregate is gone. The only remaining
`OrgMember` access is the `EXISTS` on the caller's own membership
(indexed by `userId`, a handful of rows). This is the single largest
read-amplification query on the control-plane database (~1.39B rows
read/day, ~719s DB CPU/day per Insights); removing it takes that portion
to zero.
Webapp typecheck passes.
## Rollout
Straight deploy, zero blast radius. Rollback is a plain revert, no data
migration.
refs TRI-13170
## What
The `billingLimit.reconcileTick` worker calls
`getOrgIdsWithBillingPauseSource()` on
`BILLING_LIMIT_RECONCILE_INTERVAL_MS` (~every 90s) to find which orgs
currently have billing-limit-paused environments. Two problems:
1. `RuntimeEnvironment.pauseSource` had no index, so `WHERE pauseSource
= 'BILLING_LIMIT'` was a **sequential scan of the whole table** on the
control-plane primary, every tick.
2. Prisma `distinct` dedups **after** fetching, so it read every paused
row (thousands) to produce a handful of distinct org ids.
This PR:
- Adds a **partial index** on `RuntimeEnvironment (pauseSource,
organizationId) WHERE pauseSource IS NOT NULL`. Nearly all rows have
`pauseSource = null`, so the index stays tiny. Second column lets the DB
satisfy the distinct-org lookup from the index. Defined in SQL (Prisma
can't express partial indexes), matching the existing partial-unique
indexes on this model.
- Switches the query from `findMany({ distinct })` to
`groupBy(["organizationId"])`, pushing DISTINCT into the DB so it
returns only the distinct orgs.
## Evidence
**Correctness** — colocated `postgresTest` (testcontainers, no mocks):
multiple `BILLING_LIMIT` envs in one org collapse to one org id,
`pauseSource = null` envs are excluded, each org id returned once. 5/5
tests in `billingLimitReconciliation.test.ts` pass.
**Plan change** — `EXPLAIN ANALYZE` on a synthetic table (200k rows,
5,250 `BILLING_LIMIT` across ~40 orgs, mirroring the test-side numbers
from the investigation):
| | Before (no index) | After (partial index) |
|---|---|---|
| Plan | Seq Scan (194,750 rows removed by filter) | Bitmap Index Scan
on partial index |
| Buffers | 1355 | 51 (index 6 + heap 45) |
| Exec time | 6.06 ms | 0.59 ms |
Index size 56 kB vs table 11 MB. The key win: cost now scales with the
paused-env count, not total table size, which matters most on prod where
the table is far larger.
## Rollout & rollback
- **Index**: `CREATE INDEX CONCURRENTLY IF NOT EXISTS`, in its own
migration file. Pre-apply the index manually on the control-plane
primary before deploying the migration (the migration is a no-op if the
index already exists).
- **Query change** is behavior-equivalent (same distinct org set), so no
flag needed.
- **Rollback**: revert the deploy and drop the index. No data migration
either direction.
## Notes / limitations
- The planner uses a Bitmap Heap Scan, so `organizationId` is still read
from the heap (45 blocks for the matched rows only, not the whole
table). A pure index-only scan isn't chosen for the bitmap path; the
second index column keeps that open for the index-scan path at
negligible cost.
refs TRI-13169
## What
Adds a composite index `@@index([environmentId, status, id])` to
`WorkerDeployment`.
The public deployments list (`GET /api/v1/deployments`) filters by
`status` and paginates by `id` descending. The existing indexes cover
`(environmentId, createdAt)` and the PK, but nothing covers `status`. So
for a status filter Postgres walks back through the environment's
deployments discarding non-matching statuses, reading roughly 350 rows
for every 1 returned (p99 ~1.1s on the busiest environments). The new
index makes the status filter index-satisfied and lets `id` serve both
the cursor range and the `ORDER BY id DESC`, bounding the read to a
single page.
Full composite (not partial) because callers filter by arbitrary status
values with no single dominant one.
## Query
```sql
SELECT ... FROM "WorkerDeployment"
WHERE "environmentId" = $1 AND "status" = $2 [AND "id" < $3]
ORDER BY "id" DESC LIMIT $4;
```
Source: `apps/webapp/app/routes/api.v1.deployments.ts`.
## Evidence
Reproduced on an isolated stack: one environment seeded with 7,000
deployments, the filtered status appearing 1 in 333 rows.
Before (no index):
```
Seq Scan on "WorkerDeployment" (rows=21)
Rows Removed by Filter: 6979
Buffers: shared hit=206
Execution Time: 2.9 ms (+ a sort for id desc)
```
After (with the index):
```
Index Scan Backward using "WorkerDeployment_environmentId_status_id_idx"
Index Cond: (environmentId = $1 AND status = $2)
Buffers: shared hit=23
Execution Time: 0.43 ms
```
Rows-removed-by-filter drops to 0; buffers 206 -> 23. The cursor
(mid-pagination) variant uses the same index with all three predicates
as the index condition. A dense/common status keeps the cheap PK
backward scan (already fine); the index targets exactly the rare-status
paths that were amplified.
End-to-end against the running webapp API: `?status=FAILED` returns the
correct newest-first page and paginates correctly across pages, and the
emitted SQL matches the query above.
## Rollout
- Index only, `CREATE INDEX CONCURRENTLY IF NOT EXISTS` in its own
migration file. Online-safe under write load.
- Pre-apply the index in production before the migration deploys, per
repo convention (the migration is then a no-op).
- Rollback: drop the index. No data migration.
refs TRI-13171
Plan enforcement for the dashboard agent — message quota and watch
limits — plus the component gallery, fixes and test hardening from the
same stack (#4548, #4549, #4550, #4552, #4556 merged here).
## Plan enforcement
([TRI-12863](https://linear.app/triggerdotdev/issue/TRI-12863))
**Agent message quota.** The Free-plan allowance becomes a real
server-side limit with a durable counter. New `agent_message_usage`
table keyed `(organization_id, period)` — deliberately not joined to
chats, so deleting a chat can't free quota within the period. Both send
paths count one user message (wakes never count) and refuse at the cap
with `403 message_quota_reached`, which the client renders as an upgrade
panel, never a silent drop. The refusal code is a single shared constant
on both sides.
**Watch limits.** A watch whose window exceeds the plan's
`agentWatchMaxHours`, or that would push the org past its
`agentWatchers` count, is refused with `watch_limit_reached` (409 on the
API, an upgrade hint on the card). Plan limits only tighten the existing
code ceilings (`min(plan, 24h)`, per-chat cap of 3 still applies). A
plan limit of zero means zero, not unlimited. Questions answerable
instantly are answered before any plan refusal — a one-shot consumes no
slot and never sees an upgrade nag.
**Fails open by design.** Cloud ships the actual per-plan numbers
separately (TRI-12863 P0). Until then absent limits resolve to the
unlimited sentinel and the upgrade UI is gated on billing presence —
self-hosted sees no cap, no upsell, with tests proving the fallback.
Both quotas are nudges, not security boundaries: a failing limit read
never blocks a send.
## Component gallery
An admin-only gallery of every agent card state: five
`storybook.agent-*` pages (chat UI, view blocks, report, investigation,
watch) with their shared shell and manifest, demo fixtures, two
demo-only cards, toast examples, and the screenshot script. No LLM and
no data — every state renders from fixtures under
`dashboard-agent/demo/`, never reachable from a production path.
Designers and reviewers can look at every state, including the report
states, without seeding anything.
## And fixes
**SDK: watch-mode chat subscriptions survive quiet windows** (TRI-13065,
TRI-13070) — watch mode keeps reconnecting across empty long-poll
windows and only stops on abort or a settled session; a passive
subscriber can no longer stop a turn it doesn't own (`stopOnAbort` is
explicit, default off). Review findings fixed alongside: a superseded
stream's async teardown no longer removes the live successor's abort
controller or multi-tab claim, and stopping a generation hands the chat
back to the user's other tabs.
**Query boundary pinned end-to-end**
([TRI-11165](https://linear.app/triggerdotdev/issue/TRI-11165)) — a
route-level test drives `api.v1.query` with a real signed environment
JWT (writes refused before ClickHouse, a read passes); `readonly=1` made
non-overridable; a per-turn cap stops the model burning a turn rewriting
a query it can't fix (deterministic SQL errors only — busy/transport
rejections don't count).
**chat.agent durability regression suite**
([TRI-11166](https://linear.app/triggerdotdev/issue/TRI-11166)) —
testcontainers-backed coverage of the two audit criticals (cross-tenant
isolation, no duplicate mid-stream turn, both control-broken) plus
crash-resume, cursor-based refresh, clean rollback of a mid-write turn
failure (torn by a real constraint violation), and OOM-restart replay.
**Investigation sweep backoff** — stale investigations get an attempt
counter and backoff so a poison row can't pin the sweep queue head
(migration `0005`: `sweep_attempts`, `last_sweep_attempt_at`).
## Screenshots
<img width="1440" height="791" alt="Screenshot 2026-08-06 at 00 36 19"
src="https://github.com/user-attachments/assets/6a68cd42-8580-469d-afe7-e28d1eef18e1"
/>
🤖 Generated with [Claude Code](https://claude.com/claude-code)
- New DB fields on Schedule and ScheduleInstance
- Use `queueTimestamp` for the "effectiveAt" delayed start time,
propagate it to Clickhouse TaskRun table
- Disable fastpath for delayed jobs
- Add schedule timing logic, API endpoints with windows, persistence
- Calculate phase for every schedule, only persist when window is
non-null
- Additional o11y for phased rollout
## Summary
Runs created for a Session were triggered without a realtime streams
version, so they fell through to the `realtimeStreamsVersion` column
default of `v1`. A Session's own `.in` / `.out` channels are always
`v2`, so any run-scoped `streams.append()` or `streams.pipe()` call made
inside a session run wrote to a different backend than the session it
belongs to, and stayed there for the life of the run.
The API trigger routes were never affected. They call
`determineRealtimeStreamsVersion` with the client's
`x-trigger-realtime-streams-version` header and always pass an explicit
value, so a current SDK asking for v2 gets it. Only the internal callers
that build trigger options by hand were leaning on the column default,
which no env var can influence because that path never calls the
resolver at all.
## The version resolver
Fixing the call site exposed a second problem in
`determineRealtimeStreamsVersion`. Its two paths disagreed: an explicit
`v2` was checked against the S2 configuration first, but when the caller
expressed no preference it returned `REALTIME_STREAMS_DEFAULT_VERSION`
verbatim with no check. A deployment that set the default to `v2`
without configuring S2 therefore stamped runs `v2`, nothing failed at
trigger time, and every later read or write against those runs' streams
threw `Realtime streams v2 is required for this run but S2 configuration
is missing` for the life of the run.
Both paths now resolve through one pure function that takes its
configuration rather than reading `env`:
```ts
const requested = streamVersion ?? config.defaultVersion;
if (requested !== "v2") return "v1";
const hasCredentials = Boolean(config.accessToken) || config.skipAccessTokens;
return hasCredentials && Boolean(config.basin) ? "v2" : "v1";
```
## The basin requirement
`resolveStreamBasin` resolves run, session and organization basins ahead
of the global setting, so a deployment that provisions a basin per
organization can serve v2 with no global basin at all. Gating purely on
the global setting would degrade every run there to `v1`.
`determineRealtimeStreamsVersion` therefore takes an optional
organization basin, and every caller that holds one passes it, including
the session path:
```ts
basin: organizationBasinName ?? env.REALTIME_STREAMS_S2_BASIN,
```
This is deliberately the resolved basin and not the
`REALTIME_STREAMS_PER_ORG_BASINS_ENABLED` flag. The flag says the
feature is on, not that a given organization has been provisioned, and
provisioning happens out of band. Keying off the flag would stamp `v2`
on runs for unprovisioned organizations, recreating the failure this
removes.
**This widens behaviour for explicit `v2` requests**, which previously
required the global basin: a provisioned organization on a per-org
deployment now resolves `v2` where it used to get `v1`. That is
intentional, and it makes every path agree.
## Scope
Only newly created runs change. A run already stamped `v1` keeps that
version for its lifetime by design, since readers resolve the backend
from the same column and its existing streams have to stay readable.
Scheduled runs reach the same column default through
`scheduleEngine.server.ts` and are deliberately left alone: that one is
a policy question about `REALTIME_STREAMS_DEFAULT_VERSION` rather than
an inconsistency inside a single feature.
## Verification
A full-stack e2e boots the real webapp plus Postgres, Redis and s2-lite,
creates a Session through the public API so the run comes from the real
trigger path, appends records the way `streams.append()` does, and
asserts three things at once: the version stamped on the run, that the
payload is readable from S2, and that no key exists in Redis. It appends
at a realistic record size so the route's body cap and S2's per-record
cap are both exercised. Reverting the session-path change flips all
three observations, so it fails against the old behaviour rather than
passing vacuously.
Unit tests cover the resolver matrix, including organization-basin-only
and credential-only configurations; two of them fail against the
previous resolver.
Also verified by hand against a local stack: a real `chat.agent` session
run writing 8 records of 250KB through `streams.append()` put 2,049,072
bytes into S2 with no Redis key, while the same agent with the
session-path change removed put 2,102,360 bytes into Redis and nothing
into S2.
fix(webapp): include Tailwind in production image
## Summary
Include `tailwindcss` in the webapp production dependencies so
self-hosted Docker images can render emails that use React Email's
Tailwind component.
Watch is the agent noticing something later: you ask it to tell you when
a condition holds, and it answers when it does — or when it can't any
more.
A watch is a **durable one-shot promise**. The condition is checked on a
schedule by deterministic code (no LLM in the checks), the answer lands
in the chat once, and then the watch is over. Ten kinds: three on a run,
five on a queue, error recurrence, health recovery.
## Stack
Stacked on **#4529** (UI), which is stacked on **#4418** (chat, reports,
investigate). Merge those first. **#4516** (storybook gallery) sits on
top of this branch.
## How to review
[**GUIDEBOOK.md**](https://github.com/triggerdotdev/trigger.dev/blob/feat/dashboard-agent-flows-watch/internal-packages/dashboard-agent/GUIDEBOOK.md)
on this branch is the behaviour reference — it states the conditions
rather than the code, so you can predict what happens without running
anything. "The ten watch kinds, and what makes each fire" and "Creating
a watch" describe exactly this PR, and the tables there are the spec the
code is written against.
## What's inside
- **Ten watch kinds**, one deterministic check each
(`dashboardAgentWatch*Checks.ts`), with the spec union in
`dashboard-agent-contracts/src/watch.ts`.
- **Scheduling** — each watch schedules its own next check; due watches
of one `(environment, cadence)` group can be checked together in one
batch pass, with a sweep as the backstop for expiry, redelivery and
retention.
- **Delivery** — the in-chat wake and card, an optional email alert (new
`DASHBOARD_AGENT_WATCH` alert channel, so it shows on the project's
Alerts page with one-click unsubscribe), and an optional investigation
when the outcome needs attention.
- **Submission ledger** — `watch_submissions`, keyed `(chat_id,
client_request_id)`, so a retried card submission replays the recorded
outcome instead of creating a second watch.
- **Watch token** — a delayed-execution credential accepted only by the
watch endpoints, re-checked against the user's live access on every
tick.
- **Unread work** — the panel polls for wakes that landed while it was
closed, so a chat can go unread and light the launcher dot.
## Key decisions
**A check result is a 4-way, and only two of them are verdicts.**
`satisfied` / `terminal_unsatisfied` are answers; `pending` and
`unavailable` are not. Any exception inside any check is caught in one
place and becomes `unavailable` with an unverified observation — a check
that failed is never evidence.
**A completed window is an answer, and whether it is good or bad news is
declared per kind, never inferred.** There is a table for that in the
guidebook: `run_failed` completing its window is *good* news ("hasn't
failed"), `backlog_drain` completing it is not. One rule overrides the
table: a window that completed on an unverified observation is neutral
and says only that the watch ended without a confirmed answer. **An
unreadable source is never a negative answer** — and, because
investigations only open on `attention`, it never starts one either.
**Identity is `(chat, project, environment)` plus the condition,**
enforced by a partial unique index over active rows
(`watches_chat_active_identity_key`), not by the read-then-insert check.
Cadence, window, note and `ticks` are deliberately not part of it. Two
different chats may watch the same thing — a watch is a promise to a
chat.
**The server resolves the target's name, whatever the model calls it.**
The model can't tell a task queue (`task/<id>`) from a custom queue, so
both spellings are tried and the stored one wins — and the rewrite
happens **before** identity and before the row is written, so the
identity, the checks, the link and the wording all see one spelling.
**Freshness fences.** Depth falls back from the live counter to the
newest 60 s ClickHouse bucket, which only counts as current within 60 s
of now. A non-current reading at or below the *quiet line* is refused as
`unavailable` rather than believed, so a stale empty bucket is never
read as "drained". The stall streak is the one piece of carried state:
it lives in the previous check's facts and *freezes* on an unreadable
reading rather than breaking.
**Chain reliability.** There is no shared cron — each watch (or batch
group) schedules its own next tick, so the failure mode to review is the
chain dying. A failed batch check is caught, the next tick is scheduled
anyway and the run resolves rather than failing, so the chain survives a
check that couldn't run; the sweep re-arms groups and finalizes anything
still active past its deadline, even when delivery isn't configured.
Wake redelivery is id-deduped rather than conditional, because the sweep
can't know whether the user was already told. Access is re-authorized on
**every** check against the primary — replica lag would extend access
the user has already lost.
**Wording lives in one place.** `watch-wording.ts` is read by the card,
banner, toast, email and the agent's own narration, and the numbers come
from the frozen observation rather than a fresh read, so a retry
produces the same sentence. Replay reproduces the **recorded** decision
instead of deciding again — the transcript is append-once, so a second
decision would contradict it forever.
**Cancellation is the ending without an answer** — no resolution, no
wake. One exception, decided during testing: a watch the *user*
cancelled leaves a single neutral transcript line ("Stopped watching
…"), keyed off the watch id so a retry can't repeat it. The other four
reasons stay silent.
**Email is opt-in and only a fired watch emails.** An expiry is narrated
in the chat and nowhere else. Both gates (agent access, a configured
email transport) are checked at subscribe time *and* again at delivery,
and the subscription outcome is frozen on the ledger row so a retry
replays it. Neither gate is a plan check.
**One watch offer per turn.** The prompt and the renderer guard this
independently — if the turn already proposed a watch card, the action
button is dropped, because the card is the better affordance. Two eval
cases pin the prompt side: exactly one offer with the line last and the
button after it, and zero offers when the rendered card already carries
one — deterministic assertions, over a real-model run.
## Testing
Unit tests (vitest, testcontainers, no mocks) under
`apps/webapp/test/dashboardAgentWatch*.test.ts` and
`internal-packages/dashboard-agent/src/watch-*.test.ts` cover the
invariants above: the 4-way check results and the freshness fences,
identity/dedup and the submission ledger, queue-name resolution, the
batch chain surviving a failed check, sweep boundaries and alert-once,
tenancy and the watch token's scope, and the wording snapshot. The
load-bearing ones were verified by control-breaking the guard first and
checking the test goes red.
Live-tested end to end against a local stack, following the guidebook:
all ten watch kinds firing and expiring, cancellation, the email pair (a
fired watch mails, an expired one does not), and watch recovery from a
health report.
## Summary
Background worker registration runs on every deploy and every `trigger
dev` file save. Its declarative-schedule reconcile loaded every
declarative schedule for the whole project across all environments, then
re-fetched the deletion candidates it already had in memory. For
projects with many scheduled tasks or many environments, that meant
reading tens of thousands of rows on each registration. This scopes the
load to the environment being registered, drops the redundant re-fetch,
and selects only the columns the reconcile needs.
It also fixes the schedule-limit count (`getUsedSchedulesCount`), which
joined `TaskSchedule` and `RuntimeEnvironment` without a project
constraint and could scan those tables in full. Pushing `projectId` onto
both joins gives it a project-scoped index path with the same result.
Follow-up to
[#4522](https://github.com/triggerdotdev/trigger.dev/pull/4522), which
batched the delete side of the same reconcile.
## Summary
Setting or importing environment variables ran a conflict pre-check that
loaded every variable in the project and every value across all of its
environments, only to decide whether the submitted keys already had a
value in the target environments. On projects with many variables and
environments that meant reading tens of thousands of rows on each
create/import call.
This scopes the pre-check to the submitted keys and target environments,
so it reads only the rows it actually inspects (submitted keys × target
envs), wrapped in `boundedIn` to keep the prepared-statement cache
stable. Same conflict detection, a handful of rows instead of the whole
project's env-var values.
Stacked on #4418. Merge that first.
The UI slice of the dashboard agent: the side panel, the chat transport
wiring, message and card rendering, suggested prompts, and chat history.
#4418 works without this — the system is simply invisible. The diff is
mostly components, so the notes below cover only the three decisions you
can't read off the markup. Behavior and a hands-on walkthrough live in
GUIDEBOOK.md, which lands with #4525.
## Decisions worth knowing
- **Action rows always render at the end of a turn.** The model's
emission order isn't trusted for layout, so action blocks are split out
of the stream and appended last. Display only — `answered` stays keyed
on the emission index.
- **The last-chat memory is org-true.** It's keyed by the chat's own
organization, and a foreign or deleted chat comes back as a 404 the
client treats as gone, rather than an empty chat it keeps around.
- **A dead stream self-heals from the settled transcript.** Terminal
records are written to the chat row after the client's stream closes, so
the panel re-reads it. The poll gate is any unfinished turn — a dangling
tool part, not just an open investigation.
## Notes
- Gated by `canAccessDashboardAgent`; no behavior change with the flag
off.
- Page marks: `handle.agentPageContext` on 47 routes, ~20 lines each.
- Entry points: Ask Trigger button, ⌘J, Help & Feedback. The old ⌘I and
`?aiHelp=` links keep working.
## Screenshots
<img width="1440" height="788" alt="Screenshot 2026-08-07 at 15 14 29"
src="https://github.com/user-attachments/assets/f4e89e8d-13ed-4be3-a88d-d5cca3ece0fa"
/>
## What & why
This is the system behind the Dashboard Agent — an assistant that
answers questions about a project's runs, errors, queues, deploys and
health, and can investigate failures end to end.
The agent runs as a chat.agent task in its own Trigger project. It has
no access to the main database or ClickHouse; all platform data is read
through the public API using a delegated, read-only user token.
Everything here is behind `canAccessDashboardAgent` and inert with the
flag off. The UI that mounts the panel lands in #4529.
## Stack
`#4418` (this, base) ← `#4529` UI ← `#4525` Watch ← `#4516` storybook
gallery. The scenario/contract reference for the whole stack is
`internal-packages/dashboard-agent/GUIDEBOOK.md` (it lands on the Watch
branch): it states, per feature, what makes each thing happen and where
that is decided.
## What's inside
**Agent runtime and tools** — `internal-packages/dashboard-agent`:
prompt, tool set (API reads, TRQL query, docs, navigation,
evidence/investigations, repo source), conversation compaction, a
prompt-prefix token budget pinned by snapshot test, and sampled
LLM-judged turn evals. The package cannot import webapp server code,
which is what makes the "no DB access" claim structural rather than a
convention.
**Contracts** — `internal-packages/dashboard-agent-contracts`:
`trigger://` URIs, intents, and the block envelope every rendered card
travels in.
**Conversation store** — `internal-packages/dashboard-agent-db`: drizzle
over postgres-js in its own `trigger_dashboard_agent` Postgres schema,
plus one additive migration.
**Auth boundary** — the user-actor token gains an optional environment
claim; one guard (`userActorEnvironment.server.ts`) enforces it so
routes don't each re-derive the rule. Token minting, cap ceiling, and
the RBAC fallback path for self-hosted.
**Transport** — webapp resource routes that mint the token and proxy
each turn, and SDK-side mid-turn reconnect.
**Public API the agent reads through** — orgs, projects, environments,
runs, queue metrics, workers, a run's commit metadata, repo snapshot,
reports, and `POST /api/v1/query`.
**Reports** — the health report's layout is declared once and shared by
the card, the markdown surface and the JSON/MCP surface, so the same
report reads the same in the dashboard, the terminal and an editor.
**Block renderers** — the report and investigation cards the flows above
already emit (`app/components/dashboard-agent/`). The panel that hosts
them, and the rest of the chat UI, is #4529.
**Query safety and CSP** — see below.
## Key decisions
- **The agent is a separate Trigger project, not webapp code.** It reads
platform data over the public API with a delegated user-actor token
whose `cap` ceilings it to read scopes. No Prisma, no ClickHouse, no
webapp imports.
- **The PAT-only auth helper now refuses user-actor tokens.** This is an
intentional behavioral change: its callers consume only a bare userId
and do not enforce delegated-token capabilities. Actor-aware routes
continue through the scoped route builders instead.
- **RBAC fallback builds a delegated token's ability from its own cap**,
never the blanket ability a PAT gets (read-only when the token declares
none). Without this, the agent's read-only cap would buy a write JWT on
self-hosted.
- **Org creation checks RBAC only for user-actor tokens, and only after
the env gate**, so an install with `ORG_CREATION_API_ENABLED` off
returns 404 rather than 403, and an ordinary PAT never consults an
ability the route has no org to scope. Both orderings are pinned by
test.
- **The query path is read-only in depth.** TRQL rejects write
statements at the grammar level (they don't parse, rather than being
filtered), ClickHouse runs with `readonly=1`, and the org/project/env
filters are injected server-side from the credential — the request body
cannot widen scope. An unparseable query denies instead of falling
through to the permissive resource.
- **Document-wide img-src CSP.** Remote images are an
outbound-request/exfiltration surface, so the policy permits only
own-origin/data/blob, the required SSO avatar hosts, and the favicon
endpoint. Operators can add exact origins through CSP_IMG_SRC_ALLOWLIST;
wildcard hosts and bare schemes are intentionally not allowed.
- **The chat transport reconnects on a mid-turn EOF**
(`@trigger.dev/sdk`). A body that ends without a turn-complete is
terminal only when the server says `X-Session-Settled: true`; otherwise
the transport resubscribes from `lastEventId` with bounded backoff, and
any record re-earns the budget. Previously a closed long-poll window or
a proxy restart left the reply stuck as if still generating.
- **Conversations live in their own datastore**, schema-scoped and
foreign-key-free (it references `organizationId`/`userId` by id, because
in cloud it is a different database). It is a display read-model for the
History tab and transport resume; `chat.agent`'s object-store snapshot
remains the model's source of truth.
- **Deterministic first.** Reports and health checks contain no LLM —
they are computed from the same data the dashboard shows, and the model
only narrates and links them. That is what makes a number in an answer
auditable.
## Testing
- 63 new test files, run with `pnpm run test --filter webapp` and
per-package vitest. Heaviest coverage on the auth boundary
(`userActorPatOnlyBoundary`, `userActorTokenClaimsAndScopes`,
`contextlessPatRoutes`, `rbacFallbackBranch`), TRQL read-only, the
report layout, and the SDK reconnect.
- The agent package has a separate eval lane (`pnpm run test:evals`,
`vitest.eval.config.ts`) that hits the real model, so it never runs in
`pnpm test`.
- Live-tested against a local stack scenario by scenario; the GUIDEBOOK
lists the condition each behaviour is expected under, which is what
those runs were checked against.
## Changelog
`.server-changes/dashboard-agent.md`, plus changesets for
`@trigger.dev/core` (report schemas), `@trigger.dev/sdk` (chat
reconnect) and the CLI's `mint-token` help text.
Most deploy-flow API calls shared the general per-environment rate limit
bucket with all of that environment's runtime traffic, so an org with
heavy API usage could intermittently 429 its own deploys; the
`/api/v*/deployments` endpoints themselves were fully exempt from rate
limits as a stopgap
([#2774](https://github.com/triggerdotdev/trigger.dev/pull/2774)), which
promised a dedicated limiter as the follow-up. This is that follow-up:
the whole deploy-flow group now runs on its own budget, separate from
runtime API limits.
### Design
A new `deploymentRateLimiter` covers every endpoint the deploy flow
depends on: the `/api/v*/deployments` group, the env API key exchange
(`/api/v1/projects/:ref/:env`), build-time env var resolution and sync
(`/envvars`, `/envvars/:slug/import`), preview branches,
`/api/v1/remote-build-provider-status` and `/api/v1/artifacts`. The
general API limiter whitelists the same shared path list, so exactly one
limiter applies to each path and the two can't drift apart.
Buckets are keyed per environment for environment API keys and per token
for the PAT-authenticated phase of a CLI deploy (whoami, key exchange,
branches). The deploy budget is controlled via the
`DEPLOYMENT_RATE_LIMIT_*` env vars.
## Summary
Adds a docs page for LLM observability: every opted-in Vercel AI SDK
call inside a task becomes its own span in the run trace, carrying the
model, provider, token counts, cost, and latency. The page covers
turning it on per call with `experimental_telemetry: { isEnabled: true
}`, what each span inspector tab shows (Overview, Messages, Tools, and a
Prompt tab when linked), linking a call to its prompt version with
`toAISDKTelemetry()`, and querying usage across runs with TRQL against
the `llm_metrics` table.
It sits in the AI dropdown under Features, next to
[Prompts](https://trigger.dev/docs/ai/prompts), and cross-links the
[Query](https://trigger.dev/docs/observability/query) page.
It is explicit that capture is opt-in per call (not automatic) and only
covers Vercel AI SDK calls, and notes the `@ai-sdk/otel` requirement on
AI SDK 7. Every API name, span tab, and TRQL column was checked against
the SDK and the live query schema.
## Also in this PR
Corrects one bullet in the [AI Agents
overview](https://trigger.dev/docs/ai-chat/overview): it claimed an
in-progress chat resumes on the new version after a redeploy, which
contradicts the version-upgrades and backend pages. Chat agent runs are
pinned to the version they started on; moving onto new code is an
explicit version upgrade.
## Summary
Adds a "Build a chat agent" guide to the AI agents section, surfaces the
ClickHouse chat agent example in the guides index and the AI agents
overview,
and refreshes the five existing workflow guides so their code is
current.
## Details
The pattern guides (prompt chaining, routing, parallelization,
orchestrator,
evaluator-optimizer) still used retired models and dated APIs. Updated
them to
current Anthropic Claude models (claude-haiku-4-5 for lightweight
classifier
roles, claude-sonnet-4-5 for the main work) and modernized the code:
- route-question uses generateObject for the routing decision instead of
generateText plus manual JSON parsing.
- verify-news-article uses ModelMessage in place of the renamed
CoreMessage.
- Fixed translate-and-refine discarding its recursive refinement result,
so
refined translations never returned to the caller.
- Fixed an invalid JSON test payload in generate-translate-copy.
The pattern concepts are unchanged; only the example code was stale.
## Summary
The dev environment "Get set up" panel used to be a static list of CLI
commands that only disappeared once your tasks registered, so nothing
ever changed after you ran `init` and people assumed it was stuck. It
now tracks real progress: `trigger init` records the project as
initialized, so step 1 checks off, and the panel updates live as the dev
server connects and your tasks register.
It also adds a prominent "Copy AI agent prompt" button, presented as a
clear alternative ("or") to the manual CLI steps, that copies a
ready-to-paste setup prompt pre-filled with your project reference for
Claude Code, Cursor, or any coding agent.
## Notes
- Adds a `Project.initializedAt` column (migration
`20260811065646_add_project_initialized_at`); the CLI `init` command
calls a new project-scoped `POST /api/v1/projects/:ref/init` best-effort
at the end of setup.
- The `init` scaffold now imports from `@trigger.dev/sdk` instead of the
deprecated `/v3` subpath.
## Screenshots
<img width="2400" height="1794" alt="v7-redesigned-card"
src="https://github.com/user-attachments/assets/c2fb4fa1-9484-4700-8bd3-110d66f5a44e"
/>
🚀 Publish Trigger.dev Docker / typecheck (push) Failing after 0s
🚀 Publish Trigger.dev Docker / units (push) Failing after 0s
🚀 Publish Trigger.dev Docker / publish-webapp (push) Has been skipped
🚀 Publish Trigger.dev Docker / publish-worker-v4 (push) Has been skipped
🚀 Publish Trigger.dev Docker / scan-webapp (push) Has been skipped
🚀 Publish Trigger.dev Docker / scan-supervisor (push) Has been skipped
🦋 Changesets PR / Create Release PR (push) Has been cancelled
🚀 Publish Trigger.dev Docker / 📣 Dispatch main image (push) Has been cancelled
## Summary
Adds an opt-in path to serve a run's per-run configuration reads from
the control-plane read replica instead of the primary, reducing primary
database load during task execution. The managed-worker dequeue resolves
each run's environment, organization, and environment variables before
starting the run; those rows are stable for the life of a run, so they
can safely come from the replica.
Gated by `CONTROL_PLANE_DEQUEUE_READS_FROM_REPLICA`, defaulting to `"0"`
(reads from the primary, unchanged from today). Set it to `"1"` to route
the reads to the replica. The env-var read is scoped to the
dequeue/resolution path (`resolveVariablesForEnvironment`); dashboard
env-var reads and writes always stay on the primary. When no read
replica is configured, `$replica` transparently falls back to the
writer, so single-database self-host is unchanged either way.
Verified end-to-end against a real primary/replica split, in both
`trigger dev` and deployed (managed-worker) runs: with the flag on, env
vars inject correctly and a value set immediately before triggering a
deployed run is present on the run.
## Why this change
`EnvironmentVariableValue.valueReference` is an `onDelete: SetNull`
foreign key. Deleting a `SecretReference` (the env var edit/delete path
for secret values) fires the cascade `UPDATE ONLY
"EnvironmentVariableValue" SET "valueReferenceId" = NULL WHERE $1 =
"valueReferenceId"`. That cascade is scan-shaped: with no index on
`valueReferenceId`, it reads the entire table to find the rows
referencing the deleted secret. The parent `SecretReference` delete does
almost no work itself; its latency is dominated by this cascade.
## Diagnosis
`EnvironmentVariableValue` was indexed on `environmentId` and
`(variableId, environmentId)`, but not on `valueReferenceId`. The SET
NULL cascade therefore did a full sequential scan of the whole table.
Two sibling SET NULL cascades on the same delete
(`OrganizationIntegration.tokenReferenceId`,
`User.mfaSecretReferenceId`) are index-backed and stay fast, which
isolates the missing index as the cause.
## Change
Add `@@index([valueReferenceId])` on `EnvironmentVariableValue`, created
with `CREATE INDEX CONCURRENTLY IF NOT EXISTS` so `prisma migrate
deploy` stays safe on a live table.
## Benchmark (local, seeded)
Local Postgres seeded with 1,000,000 `EnvironmentVariableValue` rows,
`EXPLAIN (ANALYZE, BUFFERS)` on the SET NULL cascade with zero matching
rows (the worst case: reads the whole table, affects nothing):
| | before | after |
|---|---|---|
| plan | Seq Scan (1M rows) | Bitmap Index Scan |
| execution | 183 ms | 2.8 ms |
In a variant where the secret matched several thousand rows, the parent
`SecretReference` delete's
`EnvironmentVariableValue_valueReferenceId_fkey` trigger dropped from
216 ms to 88 ms (the residual is the heap work of nulling those rows).
## Expected impact
The cascade drops from a full-table sequential scan to a targeted index
lookup. The win grows with the table, so the benefit is larger than the
seeded numbers above.
## Risks
- One extra btree to maintain on `EnvironmentVariableValue` writes;
small, single-column, and it should be pre-created before the migration
deploys (per the repo index rules).
- No behavior change: same rows nulled, no ordering or result-set
change, read paths untouched.
Companion to the same fix on `ProjectAlert.channelId`.
## Why this change
Deleting a `ProjectAlertChannel` fires the FK cascade `DELETE FROM ONLY
"ProjectAlert" WHERE $1 = "channelId"`. That cascade is scan-shaped:
with no index on `channelId`, it reads the entire `ProjectAlert` table
to find the few child rows belonging to the deleted channel. The parent
`DELETE ProjectAlertChannel` does almost no work itself; its latency is
dominated by this cascade. `ProjectAlert` is append-heavy and grows over
time, so the scan cost only increases.
## Diagnosis
`ProjectAlert` had no index on `channelId` (only `pkey` + a `friendlyId`
unique). The cascade therefore did a full sequential scan of the whole
table. The sibling `ProjectAlertStorage` cascade on the same delete is
index-backed and stays fast, which isolates the missing index as the
cause.
## Change
Add `@@index([channelId])` on `ProjectAlert`, created with `CREATE INDEX
CONCURRENTLY IF NOT EXISTS` so `prisma migrate deploy` stays safe on a
live table.
## Benchmark (local, seeded)
Local Postgres seeded with 1,000,000 `ProjectAlert` rows across 50
channels (~20k rows per channel), `EXPLAIN (ANALYZE, BUFFERS)` on the
cascade delete:
| | before | after |
|---|---|---|
| plan | Seq Scan (1M rows) | Bitmap Index Scan |
| direct child delete | 740 ms | 22 ms |
| parent delete `ProjectAlert_channelId_fkey` trigger | 77.7 ms | 23.8
ms |
## Expected impact
The cascade drops from a full-table sequential scan to a targeted index
lookup. The win grows with the table: the more rows in `ProjectAlert`,
the more a scan costs and the more the index saves, so the benefit is
larger than the seeded numbers above.
## Risks
- One extra btree to maintain on every `ProjectAlert` insert; acceptable
for a single-column index on a high-insert table, and it should be
pre-created before the migration deploys (per the repo index rules).
- No behavior change: no rows orphaned, no ordering or result-set
change, read paths untouched.
## Follow-up
`ProjectAlert`'s other cascade FK columns (`projectId`, `environmentId`,
`workerDeploymentId`) are also unindexed, but their parents are
soft-deleted rather than physically removed, so those cascades do not
currently fire. Lower priority unless a hard-delete path is introduced.
## What
Follow-up to #4539. The driver-adapter work is inert until a client
flips to the pg driver adapter, but the moment one does, our database
observability degrades: the OTel metrics pipeline reads pool stats from
Prisma's `$metrics`, which is owned by the Rust engine's `quaint` pool.
Under the adapter, `pg.Pool` owns the pool, so those gauges read zero.
The pipeline also only ever scraped a single client (the control-plane
writer singleton).
This PR makes database metrics driver-agnostic and per-client:
- Every configured client registers a metrics source: control-plane
writer/replica, run-ops writer/replica, legacy writer/replica.
Previously only the control-plane writer singleton was scraped.
- Each OTel instrument is observed per client with `db_client` and
`db_driver` (`quaint` | `pg-adapter`) attributes. `db_client` uses our
canonical datasource-role labels (`control-plane-writer`,
`control-plane-replica`, `run-ops-writer`, `run-ops-replica`,
`legacy-run-ops-writer`, `legacy-run-ops-replica`) — the same strings
used for the `db.datasource` span attribute, so a metric and a trace
point at the same pool.
- Pool figures come from the authoritative source per driver:
- **pg-adapter**: `pg.Pool` (`totalCount`/`idleCount`/`waitingCount`,
plus cumulative opened/closed from `connect`/`remove` events).
- **quaint**: the Rust engine's `$metrics` pool gauges/counters, exactly
as before.
- Query counters and duration histograms still come from `$metrics` for
both drivers (the Rust engine executes queries in both cases).
- New `db.pool.connections.waiting` gauge (pg.Pool exposes this; quaint
reports 0).
- Stops exporting Prisma metrics from the Prometheus `/metrics` route.
Pool observability now lives entirely in the OTel pipeline, per driver,
per client.
## Why
So we can flip any client (including the control-plane writer, the
primary desync-fix target) to the driver adapter without losing pool
visibility. Existing dashboards keyed on the same metric names keep
working; they gain a per-client dimension.
## Testing
Unit (`apps/webapp/app/utils/databaseMetrics.server.test.ts`): the pure
normalizer — quaint reads pool from `$metrics`; adapter reads pool from
`pg.Pool` and keeps engine query metrics; `busy` never goes negative;
graceful zeroing when `$metrics` is unavailable (adapter still reports
live pool figures).
Live smoke test against a prod-shaped local stack: three
physically-distinct Postgres DBs (control-plane, run-ops, legacy) behind
dual PgBouncers, split mode on, with a mix of adapter and quaint
clients. Reading the actual emitted OTel metrics, every pool shows up as
its own series:
```
db.pool.connections.total{db_client="control-plane-writer", db_driver="pg-adapter"} = 1
db.pool.connections.total{db_client="control-plane-replica", db_driver="quaint"} = 1
db.pool.connections.total{db_client="run-ops-writer", db_driver="pg-adapter"} = 1
db.pool.connections.total{db_client="run-ops-replica", db_driver="quaint"} = 1
db.pool.connections.total{db_client="legacy-run-ops-writer", db_driver="quaint"} = 1
db.pool.connections.total{db_client="legacy-run-ops-replica",db_driver="quaint"} = 1
db.client.queries.total{db_client="control-plane-writer",db_driver="pg-adapter"} = incrementing
db.client.queries.duration.count{db_client="control-plane-writer",db_driver="pg-adapter"} = incrementing
```
Confirms: metrics are attributed per pool with the correct driver;
adapter pools' figures come from `pg.Pool`; and query counters/duration
histograms keep incrementing under the pg adapter. Also verified
`/metrics` (Prometheus) now returns zero `prisma_*` series while still
serving the app's own metrics.
`pnpm run typecheck --filter webapp` passes.
## Notes
- `/metrics` (Prometheus) no longer includes `prisma_*` series. Anything
scraping that endpoint for Prisma metrics should read the equivalent
`db.*` metrics from the OTel exporter instead.
- **PgBouncer + `?schema=` gotcha (separate from this PR, worth flagging
for rollout):** since #4539 parses `?schema=` from the DSN and passes `{
schema }` to the adapter, node-postgres sends `search_path` as a startup
parameter. A transaction-mode PgBouncer rejects that with `FATAL:
unsupported startup parameter: search_path`. Our prod control-plane DSNs
use the default `public` schema with no `?schema=` param, so this is
latent, but any client we flip to the adapter must not carry `?schema=`
in its DSN (or the pooler needs `ignore_startup_parameters =
search_path`).
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Deploy images previously shipped node_modules and the bundled task code
in a single layer, so every deploy re-pushed and re-pulled the full
dependency tree even when nothing in it changed. The generated
Containerfile now copies `/app/node_modules` as its own layer and the
app files separately. With unchanged dependencies the dependency layer
is identical across deploys, so registries and workers already have it
and only the code layer moves.
When the health report had no start-latency measurement for the window,
it printed a confident "p95 0ms" and graded it healthy. It now shows
"unknown" for that metric and skips grading it, so an absent measurement
can't read as a green signal.
A genuinely measured 0ms is still shown as 0ms: the loader keeps "no
measurement" distinct from a measured zero instead of coercing both to
0.
## Summary
Bumps the transitive `mermaid` in the lockfile from `11.14.0` to
`11.16.1`.
`mermaid` has no direct dependents here. It arrives through
`streamdown`,
which declares it as a hard dependency even though diagram rendering is
gated
behind the optional `@streamdown/mermaid` plugin, which we don't
install.
`streamdown@2.5.0` is its latest release, and its declared range
(`^11.12.2`)
already permits `11.16.1`, so this was a stale lockfile pin rather than
a
range conflict.
Done as a scoped override rather than a bare lockfile refresh, so the
floor
survives a lockfile regenerated from an older base:
```json
"mermaid@>=11 <11.16.1": "^11.16.1"
```
Net effect is 96 fewer lockfile lines, contained to mermaid's own
subtree.
`11.16.1` swapped out its parser, so the `langium` / `chevrotain@12` /
`vscode-languageserver-*` chain drops in favour of a single
`@chevrotain/types`, and `lodash-es` and `uuid@11` are no longer pulled
at
all.
The override goes away once `streamdown` makes `mermaid` an optional
peer of
its diagram plugin instead of a hard dependency.
🚀 Publish Trigger.dev Docker / typecheck (push) Failing after 0s
🚀 Publish Trigger.dev Docker / units (push) Failing after 0s
🚀 Publish Trigger.dev Docker / publish-webapp (push) Has been skipped
🚀 Publish Trigger.dev Docker / publish-worker-v4 (push) Has been skipped
🚀 Publish Trigger.dev Docker / scan-webapp (push) Has been skipped
🚀 Publish Trigger.dev Docker / scan-supervisor (push) Has been skipped
🦋 Changesets PR / Create Release PR (push) Has been cancelled
🚀 Publish Trigger.dev Docker / 📣 Dispatch main image (push) Has been cancelled
📚 Docs Checks / check-broken-links (push) Has been cancelled
🧭 Helm Chart Prerelease / lint-and-test (push) Has been cancelled
Workflow Checks / Actionlint (push) Has been cancelled
Workflow Checks / Zizmor (push) Has been cancelled
🧭 Helm Chart Prerelease / prerelease (push) Has been cancelled
## What
Adds an opt-in path to run each Prisma client through
**`@prisma/adapter-pg`** (the node-postgres driver) instead of the
built-in engine driver, controlled by a **per-client env var, all off by
default**:
| env var | client |
|---|---|
| `CONTROL_PLANE_DATABASE_WRITER_DRIVER_ADAPTER` | control-plane writer
|
| `CONTROL_PLANE_DATABASE_REPLICA_DRIVER_ADAPTER` | control-plane
replica |
| `RUN_OPS_DATABASE_WRITER_DRIVER_ADAPTER` | new run-ops writer |
| `RUN_OPS_DATABASE_REPLICA_DRIVER_ADAPTER` | new run-ops replica |
| `RUN_OPS_LEGACY_DATABASE_WRITER_DRIVER_ADAPTER` | legacy run-ops
writer |
| `RUN_OPS_LEGACY_DATABASE_REPLICA_DRIVER_ADAPTER` | legacy run-ops
replica |
With every flag unset the construction path is byte-identical to today
(`datasources` URL + Rust engine), so this is inert until a flag is
turned on. Per-client granularity allows enabling the adapter only where
it's wanted.
## How
- Enables the `driverAdapters` preview feature on both schemas
(`@trigger.dev/database` and `@internal/run-ops-database`). This keeps
the **Rust query engine** — it does NOT add `queryCompiler` — so query
behavior, result types, and engine tracing spans are unchanged.
- A shared `buildDriverAdapterPool` builds each client's `pg.Pool` with
an explicit `max`, a bounded `connectionTimeoutMillis` (the
node-postgres pool otherwise waits unbounded on acquire), and an
`onPoolError` handler (an unhandled idle-connection error would
otherwise crash the process). Threaded through all four client builders
via a `useDriverAdapter` flag.
- Adds `@prisma/adapter-pg` + `@types/pg` to the webapp; `pg` is already
pinned at `8.15.6` (adapter-pg 6.x requires `pg < 8.17`).
## Connect-failure handling (the important correctness/security bit)
Under the adapter an unreachable DB no longer surfaces as
`PrismaClientInitializationError` / `P1001`; it becomes a `P2010`
"Database not reachable: <host>" (or a raw
`ECONNREFUSED`/`ENOTFOUND`-class error). Two handlers are updated so a
client on the adapter behaves like today:
- **`isInfrastructureError`** now recognizes those shapes (P2010 with a
connectivity message, and raw connectivity errno codes). Without this,
the DB **hostname would leak into API-client-facing errors** and the
failure would go unlogged. Security-relevant.
- **`isPrismaRetriableError`** treats the adapter's pool-acquire timeout
("timeout exceeded when trying to connect") as retriable, preserving the
`P2024` retry behavior the adapter otherwise drops.
## Evidence
Validated on an isolated stack that mirrors the production DB topology
(chained PgBouncers in front of writer + reader):
- **Behavioral parity:** raw-query results and Prisma error codes/`meta`
are byte-identical between the engine driver and the adapter across the
queried shapes (unique-constraint `meta.target`, record-not-found,
transaction-timeout, serialization-failure, etc.).
- **Feature matrix:** a full 380-project queue-ay pass shows no
adapter-caused regressions — pass/fail parity between adapter-off and
adapter-on, with the residual failures being pre-existing
known-failures/flakes common to both.
## Rollout / rollback
All flags default off; enable per client via env var, roll back by
unsetting and redeploying (no data migration). Recommended first target
is a single writer; enable one client at a time.
## Follow-ups (not in this PR)
- `$metrics`-based pool observability is removed under the adapter (the
Prometheus route + `db.pool.connections.*` instruments); the metrics
replacement (via `pg.Pool` counters) lands in a separate PR.
- Note for operators: on the adapter path, interactive-transaction
`maxWait` does not bound pool acquisition — `connectionTimeoutMillis`
does.
## Note on connection-string parameters
The adapter pool is built from the base DSN, so Prisma-specific DSN
parameters that node-postgres does not understand are not honored when a
client is on the adapter:
- **Prisma TLS spellings** (`sslaccept`, `sslcert`, etc.) —
node-postgres uses `sslmode`/`ssl` instead. Our production DSNs do not
use these Prisma-specific TLS params, but any deployment whose DSN
relies on them must be checked before enabling a flag.
- `pgbouncer=true` and `statement_cache_size` — effectively moot under
the adapter, which uses no persistent named prepared statements.
`connection_limit`, `pool_timeout`, and `schema` are handled explicitly
(passed as `max`/`connectionTimeoutMillis` and PrismaPg's `{schema}`
option).
refs TRI-13039
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
⚒️ Publish Worker (v4) / build (supervisor) (push) Has been cancelled
## Summary
Prisma expands `in` / `notIn` into one bind parameter per element, so
every distinct list
length is a separate prepared statement. Where the length tracks data
volume (a batch size,
a run-graph fan-out, a prior query's id set) one call site can mint
hundreds of them. Each
is used about once, but inserting it evicts an entry that was being
reused, so the cost
lands on unrelated queries sharing the pooler's statement cache. An
unbounded list also
risks the 65535 bind-parameter ceiling.
`boundedIn()` pads a filter list to the next power of two by repeating
its last element.
`IN` and `NOT IN` ignore duplicates, so results are unchanged, and a
call site drops from
one statement per length to at most `log2(cap)`. Applied to all existing
sites.
## Enforcement
Two oxlint rules require the helper: a list filter must be an inline
array literal or a
`boundedIn()` call.
- The first covers filters reached through `where` / `having` /
`cursor`, and deliberately
never descends into `data`, `create`, `update`, `set` or `equals`. A key
named `in` in
those positions is user data, not a predicate, and rewriting it would
corrupt what gets
stored or compared.
- The second covers bare filter objects passed to where-building
helpers, which the first
cannot see. It found five sites in the run-graph batch loaders that were
otherwise
invisible.
Both rules follow filters through the shapes they are actually written
in: conditional
expressions, logical-and objects, spread-conditional properties,
computed keys, and call
arguments. An array literal only counts as fixed-arity when nothing
spreads into it, since
`[...new Set(ids)]` has a runtime length. Twelve sites were hidden
behind those shapes
until the rules handled them.
Scoped to `in` and `notIn`. The scalar-list filters `hasSome` and
`hasEvery` compile to
`&& $1` and `@> $1`, passing the whole array as a single bind parameter,
so their arity never
reaches the statement text and there is nothing to bound.
Both rules are `error`, so new call sites fail CI. That ratchet has
already caught four
sites added by other PRs while this one was in review.
## Notes
`boundedIn` pads by repeating rather than with null: `x NOT IN (a, b,
NULL)` is never true,
so null-padding a `notIn` filter would silently return no rows. Lists
above 32768 are
returned unchanged so padding can never push a query past the parameter
limit.
Route modules reach the helper through `~/db.server` rather than
importing the database
barrel directly, since a value import of that barrel into a module that
also exports a React
component is only safe while dead-code elimination prunes it.
Measured on a local rig: 300 distinct list lengths produce 300 prepared
statements
unpadded, 10 padded. Verified end-to-end against a local stack with the
full task-suite
sweep, which surfaced no regressions.
## What
The internal tracing `ParentBasedSampler` in `tracer.server.ts` left
`remoteParentSampled` at its default of `AlwaysOn`. Any request arriving
with a `traceparent` whose sampled flag was set got recorded in full,
bypassing `INTERNAL_OTEL_TRACE_SAMPLING_RATE` entirely. Because the SDK
propagates its (always-sampled) trace context on calls back to the
platform from inside running tasks, the large majority of API server
spans inherited a sampled parent and ignored the divisor. The sampling
knob was effectively inert on the busiest service.
This registers a custom propagator
(`NonInheritingTraceContextPropagator`) that stops adopting the inbound
trace as the parent:
- `inject` still delegates to the standard W3C trace + baggage
propagators, so outbound propagation is unchanged.
- `extract` drops the parent span (`trace.deleteSpan`) while preserving
baggage, so every incoming request roots its own trace and the ratio
sampler applies uniformly.
`remoteParentSampled` is also set to the ratio sampler as a
belt-and-suspenders fallback, in case an inbound sampled parent ever
reaches the sampler another way.
Two effects: the divisor becomes effective on the API server, and the
API no longer stitches onto (and inflates) the propagated task-run
traces, which is where the very large, un-thinnable trace chains came
from. Rooting each request removes those chains rather than only
diluting them.
Only the internal APM trace pipeline
(`INTERNAL_OTEL_TRACE_EXPORTER_URL`) is affected. The user-facing
run-trace pipeline (`otel.v1.traces` -> ClickHouse) is a separate path
and is untouched. The only consumer of the global propagator's `extract`
is the OTel HTTP/Express auto-instrumentation, so the blast radius is
inbound-request trace shape.
## Evidence (local full-stack red/green, divisor 10)
A local OTLP/JSON sink counting spans; a driver fires N requests at a
real endpoint, each carrying a distinct sampled `traceparent`, then
counts how many spans/traces carry that run's marker.
| run | code | sent | kept traces | kept fraction |
| --- | --- | --- | --- | --- |
| before | unmodified | 500 | 500 | 1.00 |
| after | this PR | 500 | 67 | 0.134 |
| after | this PR | 2000 | 213 | 0.1065 |
Before: 100% of inherited-sampled requests kept, divisor ignored. After:
~10% kept (the divisor), converging on it at larger N. In every
after-run each kept request is a single self-rooted trace (kept spans ==
kept distinct traces), confirming the inherited chains are gone, not
just thinned. `typecheck` passes.
## Rollout / rollback
No flag. Behavior stays governed by the existing
`INTERNAL_OTEL_TRACE_SAMPLING_RATE`. Rollback is a straight revert with
no data migration.
## Notes
Internal dashboards that count raw span or request volume from this
pipeline will read lower once this ships. That is expected: those counts
were inflated by the bypass, not a real drop in traffic.
Latency/percentile monitors retain plenty of samples at the current
divisor.
refs TRI-13031
## What
A trigger request carrying a Unicode NUL (`U+0000`) in the **idempotency
key** or **debounce key** reached `prisma.taskRun.create()` and failed
the insert, so the caller got an opaque 500 and the run was never
created.
These two keys are stored in `jsonb` columns (`idempotencyKeyOptions`,
`debounce`), and Postgres rejects a NUL inside a `jsonb` value with
`SQLSTATE 22P05` ("unsupported Unicode escape sequence ... cannot be
converted to text"). This fix strips the NUL from both keys at the
single trigger-input chokepoint (`#buildEngineTriggerInput`), which
every trigger path flows through (single, batch item, mollified, and
drainer replay).
Stripping matches the existing precedent for run errors and task events.
It does not change dedup behaviour: the idempotency **dedup identity**
is the hashed key (a clean 64-char digest), computed independently of
the raw key we clean, so dedup keeps working exactly as before. For
debounce the key is used directly, so the cleaned key also becomes the
grouping key, an acceptable change for input that is already malformed.
## Why not payload / metadata / tags
Those are `text` columns fed by `JSON.stringify`, which escapes a NUL to
a safe escape sequence, so they do not hit this failure on the normal
JSON path. (A raw NUL in a `text` column throws a different code,
`22021`, and is not what triggers this issue.) The observed failures are
the `jsonb` `22P05` variant, which is only reachable via the two key
fields.
## Evidence
Red then green (containerTest, real Postgres): with the fix reverted,
triggering through the real service with a NUL in
`idempotencyKeyOptions.key` / `debounce.key` fails with the exact
`22P05` signature; with the fix, the run is created and the stored key
has the NUL removed.
Full-stack e2e (isolated stack, real HTTP): `POST
/api/v1/tasks/:taskId/trigger` with a NUL inside
`idempotencyKeyOptions.key` (`"acme<NUL>inc"`) and, separately,
`debounce.key` (`"grp<NUL>1"`):
- both returned `HTTP 200` with a created run (previously `500`)
- stored `idempotencyKeyOptions` = `{ "key": "acmeinc", "scope": "run"
}` (7 chars, NUL removed)
- stored `debounce.key` = `"grp1"` (4 chars, NUL removed)
- both runs render in the dashboard
Unit tests cover the helper (strip, no-op fast path, object-reference
reuse, null/undefined pass-through).
## Rollout / rollback
Server-only webapp change, no flag. Zero behaviour change for clean
input; only affects inputs that previously 500'd. Rollback is a straight
revert, no data migration.
## Known limitation
A raw NUL in a plain-string idempotency key (not created via
`idempotencyKeys.create()`) lands in a `text` column and throws `22021`
instead. That variant is not addressed here because stripping it would
change the dedup identity, so it warrants a separate decision. Not
observed in practice.
refs TRI-13030
📦 Preview packages (pkg.pr.new) / Build and publish previews (push) Has been cancelled
📚 Publish docs / publish (push) Has been cancelled
Implementing PlanetScale Insights improvement.
## Summary
Validating a schedule (creating or updating one through the API or the
dashboard, and deploying a project that declares schedules) looks up the
newest version of a task by slug. That lookup reads *every* version of
the task and sorts them to return one. A project gains a row per task on
every deploy, so the work grows with the project's age: the oldest
projects pay the most, and dev-mode redeploys make it worse. This was
picked because it was the largest single consumer of database time on
the schedules path, and the fix is a sort key with no index behind it.
## Fix
`BackgroundWorkerTask` is indexed on `(projectId, slug)`, which serves
the equality but not the `ORDER BY createdAt DESC`. Postgres seeks the
index, then bitmap-scans and top-N sorts the whole group to produce a
single row. Adding `createdAt` to the index lets it scan backward and
stop at the first row.
The same call site also selected all 21 columns, including five JSON
blobs, to read one field (`triggerSource`), so it now selects that field
alone.
## Benchmark
Local Postgres 17, 997,000 seeded rows / 748 MB, group sizes chosen to
match the distribution seen in production.
| Group size | Before | After |
| --- | --- | --- |
| 15,000 versions of one task | 11.118 ms, 1,510 buffers, 15,000 rows
scanned | 0.027 ms, 4 buffers, 1 row |
| 2,000 versions of one task | 2.081 ms, 1,455 buffers, 2,000 rows
scanned | 0.022 ms, 4 buffers, 1 row |
```
before: Limit -> Sort (top-N heapsort) -> Bitmap Heap Scan
after: Limit -> Index Scan Backward using BackgroundWorkerTask_projectId_slug_createdAt_idx
```
An ascending index scanned backward is enough here, so no descending
index is needed.
## Impact and risk
Real-world gain lands between the two rows above and scales with how
many deploys a project has accumulated. Projects with few deploys will
see little change, since there is barely anything to sort.
The new index costs noticeably more than the existing two-column one: 43
MB against 7.3 MB on the benchmark rig. Adding `createdAt` makes every
key unique, which defeats btree deduplication, so this is a real disk
and write cost rather than a rounding error. Writes to this table happen
at deploy time, not on the run path, so the write amplification is
acceptable. The existing `(projectId, slug)` index is now a redundant
prefix and could be dropped, but this PR keeps it so index usage can be
observed before removing it.
Behavior is unchanged: same predicate, same ordering, same row returned.
The narrowed select is the only code change, and the field it keeps is
the only one the caller read.
Deploy note: the migration is
`20260806100000_add_background_worker_task_project_id_slug_created_at_index`
and uses `CREATE INDEX CONCURRENTLY IF NOT EXISTS`, so it can be
pre-applied by hand before the deploy.
## Summary
`syncDeclarativeSchedules` runs on every background-worker creation
(every deploy, and every file save during `trigger dev`). It issued one
instance-delete per declarative schedule the current worker no longer
declares, in a loop, and the overwhelming majority of those deletes
matched zero rows. This collapses the loop into at most two set-based
statements and skips the instance delete entirely when the current
environment owns no instance of the schedule.
## Why so many, and mostly no-op
The loop runs once per entry in `missingSchedules`, which starts as
every DECLARATIVE schedule for the whole project across all its
environments (the query filters only by `projectId`). A schedule leaves
that set only when a declared task matches it by `taskIdentifier`
**and** the schedule already has an instance in the current environment.
That last clause is the amplifier. When a task's schedule has no
instance in the current environment, the create branch inserts a
brand-new `TaskSchedule` row with an instance for this environment
rather than adding an instance to the existing row. So the same
scheduled task, once it has run in dev and been deployed to prod, exists
as two separate schedule rows: one carrying a dev instance, one carrying
a prod instance.
On a dev worker sync of that project:
- the dev-instance row matches the declared task and is removed from the
set
- the prod-instance row has the same `taskIdentifier` but no dev
instance, so it stays in the set and gets `deleteMany(taskScheduleId =
prodRow, environmentId = dev)`, which matches zero rows
So every declarative task that has been synced in another environment
contributes one guaranteed no-op delete per sync, and the count scales
with (declarative tasks x environments), plus any leftover rows from
renamed or removed tasks. A project does not need to have dropped a
schedule to generate these; it just needs the same declarative tasks
present in more than one environment, which is the normal
develop-in-dev, deploy-to-prod case.
## Fix
The candidate schedules are already loaded with their instances, so the
branch is decided in memory:
- schedules with no instances (or only current-environment instances)
are removed in a single `taskSchedule.deleteMany`
- schedules that still have another environment's instance have only the
current environment's instance detached, in a single
`taskScheduleInstance.deleteMany`, and only when such an instance
actually exists
Behavior is unchanged (cascade delete still removes the instances of a
deleted schedule); the difference is statement count. A zero-row delete
writes no WAL and creates no dead tuples, so the removed work was pure
query and commit overhead.
Verified with a testcontainer test (red before, green after) counting
the emitted deletes across the no-op, batched-detach, and
schedule-delete cases, and end to end through `trigger dev`: three
declarative schedules created, surviving a re-sync, then two removed in
a single batched delete with the third preserved.
Debouncing with a `delay` longer than an hour did nothing at all.
The engine applied a server-side ceiling on how long a debounced run
could be pushed back, measured from the run's `createdAt` and defaulting
to one hour. A run is only pushed back while its new execution time
stays inside that ceiling, so a `delay` at or above it could never push
anything: the waiting run was released, the trigger started its own run,
and the next trigger repeated it. A `delay: "12h"` produced one run per
trigger, each correctly delayed by 12h, with no error raised and nothing
on the run to show the debounce key had been ignored.
The ceiling is now unset by default. A debounce key with no `maxDelay`
keeps collapsing triggers for as long as they keep arriving, which is
what the docs have always described. Self-hosters who want a bound can
still set `RUN_ENGINE_MAXIMUM_DEBOUNCE_DURATION_MS`.
That has a consequence worth stating plainly, so the docs now carry a
warning for it: with no `maxDelay`, a continuously triggered key never
executes. Set `maxDelay` when the work has to happen eventually.
**Failing fast on an unusable `maxDelay`.** A caller who sets `maxDelay`
no longer than their `delay` hits exactly the dead end described above,
so that pair is now rejected at trigger time instead of silently
behaving as if no debounce were set:
```
debounce.maxDelay (1h) must be longer than debounce.delay (12h). A debounced run is only
pushed back while it stays inside maxDelay, so with these values every trigger would create
its own run.
```
An unparseable `maxDelay` is rejected too, rather than quietly falling
back to no bound at all, and so is a `delay` given as a date rather than
a duration, which could never work because the value is re-applied on
every push.
The same check runs against a configured server ceiling, so a
self-hosted deployment that sets
`RUN_ENGINE_MAXIMUM_DEBOUNCE_DURATION_MS` gets the error rather than the
silent failure this PR is about. With no `maxDelay` and no configured
ceiling, which is the default, there is nothing to conflict with and
nothing is rejected.
The docs, the `TriggerOptions` JSDoc and the engine option all now state
that the room available to push is the gap between `delay` and
`maxDelay`. The run engine suite gains the case that motivated this:
four triggers on one key with a 12h delay now collapse to a single run.
Passing `debounce` in the per-item options of a batch trigger did
nothing when the items were an array. The option was accepted by the
types and by the API, then dropped before the request went out, so every
item created its own run instead of collapsing onto the debounce key.
Four public entry points were affected: `task.batchTrigger`,
`task.batchTriggerAndWait`, `tasks.batchTrigger`, and
`tasks.batchTriggerAndWait`. The streaming (async iterable) forms of the
same calls were already correct, as were `batch.trigger`,
`batch.triggerAndWait`, `batch.triggerByTask`, and
`batch.triggerByTaskAndWait`.
`useTaskTrigger` in `@trigger.dev/react-hooks` had the same silent drop
on the single-trigger path, so that is fixed here too. It also drops
`machine`, `priority`, `region`, `idempotencyKeyTTL`, and
`idempotencyKeyOptions`; those are left alone, since forwarding them is
a behaviour change beyond this bug.
Each batch item builder constructs its options field by field, which is
why one of them could fall behind without anything catching it.
TypeScript did not help: the literal is returned from a `.map` callback
inside `Promise.all`, so excess-property checking never fired against
the `BatchItemNDJSON[]` annotation, and the server's schema silently
strips unknown keys. A misspelled option name therefore reproduced this
bug with no compile error and no server error. Every builder now ends in
`satisfies BatchItemNDJSON`, which does catch it:
```
error TS2561: Object literal may only specify known properties, but 'debounceTYPO'
does not exist in type '{ ... debounce?: {...} | undefined; }'.
Did you mean to write 'debounce'?
```
The new test drives all six public batch surfaces in both array and
async-iterable form and asserts on the NDJSON that actually reaches the
wire. Each item carries a distinct debounce key so the test catches a
wrong item-to-option pairing, not just a wholesale drop.
Fixes#3304
## Summary
Adds a docs page for developers who already have a working Vercel AI SDK
chat app (`useChat` on the client, an `app/api/chat/route.ts` calling
`streamText`) and want to move it to `chat.agent`. There was no page
covering that path. `ai-chat/upgrade-guide` reads like it should be the
one, but it covers moving prerelease `chat.agent` code to the Sessions
release, which is a different reader.
The page is structured around what stays, what goes, and what is new,
because the reassuring part of this migration is how much is untouched:
the `streamText` call, model config, tool definitions, `useChat`, and
all message rendering carry over as-is. What gets deleted is the route
handler, the persistence glue wired into it, and any resumable-stream
setup. What is new is the agent task, two server actions, and
`useTriggerChatTransport`.
Covers moving tools onto the agent config so `toModelOutput` survives
past turn one, where existing database persistence goes
(`hydrateMessages` plus the turn hooks), a short section on what
durability you get once you are across, a note that
Hono/SvelteKit/Express follow the same shape, and a gotchas list built
from the mistakes this specific migration produces.
## Head Start
The one thing this migration makes worse is the opening response of a
new chat. The route handler answered out of a warm process; the agent
run has to be dequeued and booted first. That is the complaint the page
has to answer head on, so Head Start gets a full section rather than a
closing aside, plus a callout up top next to the "what changes" table so
nobody plans the migration without knowing it exists.
The section walks the four steps: splitting tool schemas away from tool
executes (the bundle-isolation constraint the whole feature rests on),
building the handler, mounting it back at `app/api/chat/route.ts` with
the original auth check wrapped around it, and the transport option.
Both server actions stay, because Head Start only owns the first turn.
Three gotchas go with it: a slow first turn without Head Start, Head
Start on but the route bundle still heavy, and the route timing out
because the handler holds the SSE response open for the whole turn
rather than just step 1.
The coding-agent prompt names Head Start as explicitly out of scope, so
an agent handed the migration does not attempt the tool split
unprompted.
Also fixes the `chat.headStart` example on `ai-chat/fast-starts`, which
set `stopWhen: stepCountIs(15)` after the spread.
`toStreamTextOptions()` pins `stopWhen` to `stepCountIs(1)`, so
overriding it makes the warm handler run steps the agent is supposed to
own (and `stepCountIs` was never imported in that snippet either).
## Migration prompt
The page also ships a copy-pasteable prompt for handing the migration to
a coding agent. It tells the agent to run `npx trigger.dev@latest
skills` first, so it picks up guidance version-pinned to the SDK
actually installed in the project, then read `quick-start.md`,
`frontend.md`, and `reference.md` (with `llms.txt` as the index) before
editing anything. The instructions are explicit about preserving the
existing model, prompt, and tool schemas rather than rewriting them.
Registered in `docs.json` under Agents, directly after Quick Start, so
it is picked up by the generated `llms.txt` and the per-page `.md`
variants.
## What
Rate-limit the API by **environment** rather than per API key.
Previously the limiter keyed its bucket on the hash of the full
`Authorization` header — one bucket per key. With additional environment
API keys (`tr_*_sk_*`), an environment can mint many keys and each got
its own full bucket, so more keys = higher effective rate limit. This
collapses all of an environment's keys onto a single shared
per-environment bucket, so the ceiling is exactly the configured limit
regardless of key mix.
## How
- `authorizationRateLimitMiddleware` now lets the override return `{
config?, identifier? }`. `identifier`, when present, is the rate limit
bucket key; otherwise it falls back to the hashed `Authorization` header
(unchanged legacy behavior, still used by `engineRateLimiter` and any
unauthenticated fallthrough).
- `apiRateLimiter`'s override resolves the environment id and uses it as
the identifier:
- **Additional keys** (`isAdditionalApiKey`) resolve via a new
`resolveAdditionalApiKeyRateLimitScope()` — a **scope-agnostic** keyHash
→ (environmentId, org limiter config) lookup. It is deliberately
permissive (restricted keys resolve too) because it's used **only for
bucketing, never as an auth decision** — request auth still goes through
the RBAC bearer controller, which enforces scopes. Revoked/expired keys
are excluded so they can't hold a bucket warm.
- **Root/legacy keys** reuse the environment already resolved by
`authenticateAuthorizationHeader` and key on `environment.id` too.
- The identifier is always the stable environment id, never the secret
key (which can rotate and would split the bucket).
- The whole override result is cached per key by the existing SWR cache,
so **no extra per-request lookup and no separate Redis mapping** is
added.
## Behavior notes
- Root + additional keys of the same environment now share one bucket
(ceiling = configured limit, not a multiple of it). Restricted
additional keys are included — they were the biggest gap, since they
authenticate via the RBAC controller and previously fell back to per-key
buckets.
- **Public JWTs** keep their existing fixed-window, per-token bucketing.
- One-time bucket reset on deploy (bucket keys change); harmless.
## Tests
- New: two tokens resolving to the same identifier share one bucket.
- New: with no identifier, bucketing stays per-key (legacy behavior
preserved).
- Updated existing override tests to the new `{ config }` return shape.
Base: `feat/multi-keys-surface`. Closes TRI-12888.
## Summary
Projects can create, inspect, expire, and revoke multiple API keys for
each environment. Plaintext values are shown only at creation; stored
credentials are hashed and the API keys page displays only an obfuscated
suffix afterward.
Self-hosted installations support full-access additional keys by
default. Authorization extensions can provide additional access presets
and optional task selection. Additional keys can also mint scoped public
access tokens through the Trigger.dev API without receiving the
environment signing key.
## Feature notes
- Only admin+ can create API keys (Developer can make in Development
branch).
- JWT self-signing will be a server call when used with new `_ak_` keys.
- JWTs with long expiry can keep working even with api key deleted (gets
priveleges from api key, signed with root key)
- Unfiltered session listings intentionally preserve the existing broad
task-read behavior. Filtered listings enforce task-level scopes for
every requested task.
- Buffered runs without a task identifier are not safely authorizable,
so cancel/replay requests fail closed rather than resolving an unscoped
run.
- Batch and waitpoint endpoints intentionally return server-minted,
narrowly scoped public tokens to all callers. These tokens have bounded
lifetimes and may remain valid until expiry after API-key revocation.
## Deployment notes
Deploy the management UI and public-token endpoint with new key creation
disabled. Enable creation for selected organizations after the
authentication path and released SDK have been verified, then expand
availability gradually.
Revoking an API key prevents new bearer requests and new token minting.
Public tokens already minted by that key remain valid until their own
expiration because they are signed by the environment signing key.
## TODO
- [x] Add "Created by" to the key table
- [x] Document that streamed batch ingestion is non-atomic and may
partially accept items before a validation or authorization error.
## Follow-ups
- [x] Add an organization-level feature flag for the API key management
UI and creation action.
- [x] Document rollout ordering: enable additional-key lookup before
enabling issuance.
- [x] Add a system-wide gate that can stop new key issuance without
disabling authentication for existing keys.
- [x] Replace the generic SDK compatibility warning with the first
published compatible version. Old SDK will mint an unusable token if
given an `_ak_` key.
- [x] Add public documentation covering creation, storage, expiration,
revocation, SDK compatibility, and public-token lifetime behavior.
- [x] Add observability for key creation, revocation, policy preparation
failures, and public-token mint failures.
- [ ] Exercise create, copy-once display, authenticate, mint, expire,
and revoke flows end to end before broad enablement.
Adds a shared `pageMeta()` helper and 74 route declarations, so a title
reads `run_abc | Runs | Trigger.dev` — the specific thing first, then
the page. Org pages also carry the organization: `Team | Acme |
Trigger.dev`. Inside a project no scope is added, because the dashboard
switches projects in every tab at once.
Page names are unchanged; what's new is that a page says which one it is
at all. Three wording changes on purpose: the queue page now names the
queue, the model page names the model, and entity pages carry their
section.
## Findings addressed
- **Report bot edited the wrong comment.** The comment-lookup step
matched on the marker body text with no author predicate, so it would
silently PATCH a human's comment that happened to quote the marker
(GitHub gates comment editing on write access, not authorship, so it
never 403'd). Now constrained to `.user.login == "github-actions[bot]"`,
the same identity `helm-prerelease.yml` already pins.
- **A required check asserted facts about the whole webapp namespace.**
`webappSymbols.test.ts` asserted that nobody anywhere in `apps/webapp`
(walking locals, params, object keys) declares names like
`createJWT`/`updateEnvVars`, so an unrelated PR naming a local variable
failed a required check with a message pointing at nothing. Those
negative self-tests move onto a package-owned fixture tree; the positive
resolution assertions stay required (their absence rotted the tool
before) but now name the list to edit.
- **The suite ran twice on shared paths.** `obsmap` and `internal` path
filters shared four generic paths (`package.json`, both lockfiles,
`pr_checks.yml`), so any lockfile bump ran the observability-map suite
in both jobs. Dropped from `obsmap` (where `internal` already covers
them). The test that should have caught it only checked the package's
own source path; it now asserts the two filters' path intersection is
empty.
- **PR-comment footer** reworded: it said the report gates nothing,
which is true of the report but misled now that the tool's test suite
does gate webapp PRs. Names both failure directions and where to read
the rules.
- **Nightly corpus** comment corrected (stale entry count; the
failure-notification gap is documented, not silently implied).
## Review
Two adversarial reviewers ran over the diff; both findings were verified
and fixed: a hollow fixture assertion (a shared name satisfied either
walker branch — now one name per declaration form, revert-confirmed) and
a filter-intersection test that could be fooled by apostrophes in
comment prose (now strips comment lines first). Full package suite green
(877 passed), typecheck and format clean.
## Summary
On the run-ops store, creating a run could intermittently fail with a
"Transaction already closed" error, and the run would never be created.
Single-write run creates no longer run inside an interactive
transaction, so a brief database write stall can't blow the transaction
budget and drop the run.
## Fix
The dedicated run-ops `createRun` / `createFailedRun` wrapped a single
nested `taskRun.create` in an interactive `$transaction`. Its default 5s
budget is wall-clock from `BEGIN`, so when a write briefly stalls the
transaction expires before the create completes and throws, even though
the statement itself is fast at the database.
A single-write create does not need an interactive transaction: Prisma's
implicit nested create is already atomic and holds no app-side budget,
so it now runs directly. Only the `triggerAndWait` path (run plus its
associated waitpoint, two writes that must commit together) keeps an
interactive transaction, now with headroom over the default.
Verified with a red/green test against the real split topology
(reproduces the exact expiry on the unchanged code, green after) and an
end-to-end run created and completed through the dedicated store.
## Summary
Follow-on to #4513. The database connect timeout is now honored, but a
single global value has to serve three separate databases at once
(control-plane, legacy run-ops, and run-ops). This adds optional
per-client overrides for the Prisma pool and connect timeouts, one pair
for the writer and one for the read replica of each of the three
databases, each falling back to the shared `DATABASE_POOL_TIMEOUT` /
`DATABASE_CONNECTION_TIMEOUT` when unset.
That lets one database's clients run a fail-fast connect timeout (with a
bounded pool wait) while another keeps more headroom, without a single
knob forcing the same tradeoff everywhere. No behavior change until an
override is set.
It also tags each client's queries with its specific datasource
(`control-plane` / `legacy-run-ops` / `run-ops`, writer or replica) via
the `db.datasource` span attribute, so telemetry can attribute
connection behavior to a specific database instead of just
writer-vs-replica.
## Summary
Allow projects using TypeScript 7 to enable `emitDecoratorMetadata()`
without adding the TypeScript 6 compiler to every Trigger.dev CLI
installation. Addresses #4500.
## Fix
The extension now resolves TypeScript from the project and
feature-detects the legacy compiler API. TypeScript 5 and 6 continue
using the project's compiler, while TypeScript 7 projects can install
Microsoft's optional `@typescript/typescript6` compatibility package
alongside TypeScript 7.
When no compatible compiler API is available, the build reports an
actionable installation error. The extension documentation includes
setup commands for npm, pnpm, and Bun.
Verified with TypeScript 5, TypeScript 6, TypeScript 7 with and without
the compatibility package, emitted decorator metadata, packed ESM and
CommonJS consumers, package export checks, and typechecking.
## Summary
Triggering a run with a very large `priority` could fail run creation
outright with an opaque database error. `priority` is multiplied by 1000
and stored in a 32-bit integer column, with nothing bounding it, so a
big enough value overflowed the column and the create failed. The
trigger now caps the value to the highest supported priority instead of
erroring, so the run is still created.
## Fix
`priorityMs` (the stored `priority * 1000`) now goes through a
`clampPriorityMs` helper before the write. It rounds to a whole number
and clamps into the column range at both ends, so only a valid integer
ever reaches the column and an out-of-range priority caps rather than
failing. Single and batch triggers share the write path, so both are
covered.
## Summary
Every Prisma client built its connection URL with a `connection_timeout`
query param, but the Postgres connector's parameter is
`connect_timeout`. The misspelled param is silently ignored, so all
clients fell back to Prisma's 5s default instead of the configured
timeout. When establishing a new connection briefly took longer than 5s
(for example during connection spikes), it failed with `Can't reach
database server` even though the database was healthy.
## Fix
All four client builders now construct their connection URL through one
shared helper (`buildPrismaConnectionUrl`) that sets `connect_timeout`,
so the configured value actually applies, and the parameter name lives
in exactly one place. Covered by a unit test.
## Summary
Upgrade the monorepo to TypeScript 7.0.2 and update package build
tooling for compatibility with the native compiler.
## Design
Package builds now use `tshy` 4, while the packages still using `tsup`
move to `tsdown`. The few scripts that depend on the legacy TypeScript
compiler API use an explicit TypeScript 6 alias; declaration portability
coverage invokes the TypeScript 7 CLI directly.
Turbo is updated so workspace tasks can read the regenerated pnpm
lockfile.
---------
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
A run controller must call the continue route to resume, so the
supervisor already knows synchronously that any checkpoint still running
for that run is pointless. It only acted on that for the compute path.
The continue route now cancels it for the Kubernetes path too, matching
what completion already does since #4493. Called after the reply so the
runner is never delayed, and skipped when there is no checkpoint client
or when the compute path owns the run. The request is bounded by a 5s
timeout so a hung call cannot leave the handler pending.
`checkpoint_cancel_requests_total{result}` records the outcome, using
the same label names as the delete path where they overlap: `sent`,
`no_client`, `not_applicable`, `http_error`.
No changeset: `CheckpointClient` is a server-only internal API, same as
#4493.
refs TRI-12915
A static observability scorer for the webapp's route entry points,
Lighthouse-style. The idea comes from evlog's `map` command, but that
tool has no Remix adapter and checks for its own logging API, so the
idea is ported rather than the tool.
It scans all 427 loader/action entry points in `apps/webapp/app/routes`
with the TypeScript compiler API and scores each against five checks:
error-classification, auth-boundary, auth-scope, request-context and
audit-trail. Current output on the real tree is **19/100** over 412
measured entry points.
```
cd internal-packages/observability-map
pnpm exec tsx src/cli.ts # terminal report
pnpm exec tsx src/cli.ts --json # machine output
pnpm exec tsx src/cli.ts api/v1/token # one entry, per-check detail
```
The two findings at the top of the fix list are real: `/auth/sso` and
`/api/v1/authorization-code` mint or exchange credentials
unauthenticated, and `/_app/orgs/:organizationSlug/settings/team`
resolves its org from a URL slug and gates each mutating branch on an
RBAC check alone, which per `apps/webapp/CLAUDE.md` is not the tenant
floor on self-hosted.
Decisions worth knowing, all with the reasoning in the README:
- The score started at 83 during development and fell to 19. Every drop
was a perverse incentive being removed, not a regression: routes were
being paid for having no error handling, two checks were reading the
same fact, suppressing a failure raised the score, and a no-op `catch
(e) { throw e }` was worth 50 points a route.
- **A mutation corpus is the tool's main defence.** 44 entries apply
semantics-preserving edits to a copy of the real route tree and assert
the score cannot rise, per route as well as globally, because a mean can
hide one route going up by taking another down. One entry runs as a live
expected failure: `try { String(0); }` with a deciding catch is a known
open hole worth 19 to 44, and it is disclosed rather than quietly
excluded.
- `audit-trail` and `request-context` are reported as headline figures
rather than one finding repeated hundreds of times. Both still count in
full where they should.
- A cohort change moves the number without anything in the codebase
getting better. Widening the sensitive cohort from 26 to 67 took the
global from 15 to 19 with no webapp change at all, so the report prints
per-check applicability and what the global would be without each one.
CI: a report-only job posts a sticky comment when a PR moves the report,
and says nothing when it does not. The package's own tests gate through
`pr_checks.yml`. The diff-scoped merge gate is still deferred until the
report has been used in anger.
524 tests plus the corpus. No runtime or dependency changes to anything
that ships.
<!-- GitButler Footer Boundary Top -->
---
This is **part 1 of 4 in a stack** made with GitButler:
- <kbd> 4 </kbd> #4485
- <kbd> 3 </kbd> #4484
- <kbd> 2 </kbd> #4483
- <kbd> 1 </kbd> #4455👈
<!-- GitButler Footer Boundary Bottom -->
When a run reaches a terminal state, ask the checkpoint service to
reclaim the storage its checkpoints occupied. Storage for finished runs
is not otherwise reclaimed, so nothing frees it today.
**Off by default** behind `DELETE_CHECKPOINTS_ON_COMPLETION`, and the
service-side handler ships separately, so merging this changes no
behaviour.
## Where the tenancy comes from
Addressing a run's checkpoints needs org, project, environment,
deployment version and run id. All five are already in hand at
`attempt.complete`, and three are **signed** by the deployment token:
| Value | Source | Trust |
| -- | -- | -- |
| org | claim `org_id` | signed |
| environment | claim `environment_id` | signed |
| deployment version | claim `deployment_version` | signed |
| project ref | `x-trigger-workload-project-ref` header |
runner-supplied |
| run | route param | runner-supplied |
`authorizeWorkloadRequest` previously returned only `environment_id`,
and only in enforce mode, so it now also returns the verified `claims`.
That difference is deliberate and documented on the method: claims are
used to address a run's **own** resources locally, never to scope the
platform, which is why `environmentId` stays enforce-only.
The two runner-supplied values are safe because the signed ones are
outermost - a runner lying about either can only name something inside
its own org and environment, and a project ref that doesn't pair with
its signed environment matches nothing. The run id is read from
`params.runFriendlyId`, the same value the platform just validated,
rather than from the body or a header. Where both a claim and a header
exist (`deployment_version`), the claim wins.
## Placement
The call sits after `reply.json(...)`, so the runner sees no added
latency - the same shape the suspend route already uses. The service
enqueues and returns 202, so it is one fast local hop.
Terminal means `RUN_FINISHED` **or `RUN_PENDING_CANCEL`** - a run
cancelled mid-execution never restores, and skipping it would leave its
storage behind. Retries are excluded deliberately: reclamation is
per-run, so a retry is covered by the final completion.
Also gated on `!snapshotService`, so it stays inert where checkpoints
aren't the kind this reclaims.
## Observability
`checkpoint_delete_requests_total{result}` counts `sent` **and every
reason we decide not to send**: `disabled`, `not_terminal`, `no_claims`,
`no_project_ref`, `http_error`.
The negative labels are the point - without them, "no requests are
happening" looks identical to the feature being switched off.
`no_claims` is reachable even under enforcement, since enforce only
rejects a *present-but-invalid* token; an absent or legacy id still
passes with no claims attached.
## Notes for review
- **No changeset**: `CheckpointClient` is `core/v3/serverOnly`, an
internal service-to-service API rather than customer-facing surface.
- **No `.server-changes/` note**: there is nothing a dashboard user
would notice here. Happy to add one if you disagree.
- `pnpm run typecheck` can't complete in my checkout -
`@trigger.dev/database` fails to build on a missing `tsc` in the pnpm
store, unrelated to this diff. Verified with `tsc --noEmit` against the
supervisor project instead: **zero errors in `apps/supervisor/src`**.
Worth noting it caught a real bug here - the completion response is
wrapped, so the status is `data.result.attemptStatus`.
refs TRI-12789
Adds System Preferences, Dark and Light themes, gated by the
`hasThemeSwitcher` feature flag (off by default — dark stays the default
theme for everyone).
Old theme is now "Classic"and set as default.
"System preferences" theme has both Light and Dark modes and uses your
laptop settings to use a correct one.
It has less color accents (specifically less colored text), and they are
the same for both modes, only grayscale values change between them. And
Light/Dark themes can be used separately.
New Contrast setting is available for System Preferences, Dark and Light
themes - it changes the contrast for the whole app. All new visual
Settings live in Account.
## Summary
The webapp's server bundle imports `prop-types` directly, but the
package was declared only as a `devDependency`. A production install
therefore leaves it out and the built server fails to boot:
```
Failed to start server: Error [ERR_MODULE_NOT_FOUND]: Cannot find package 'prop-types'
imported from /triggerdotdev/apps/webapp/build/server/assets/server-build-*.js
```
Moving it to `dependencies` is the whole change.
## Why the bundle imports it
Nothing in the webapp's own code uses `prop-types` — there is no
reference to it, or to `PropTypes`, anywhere under `apps/webapp/app`. It
arrives through `recharts`, whose `react-smooth` dependency still
declares `propTypes` on its components.
That was invisible until recently. While `recharts` was resolved at
runtime, its `prop-types` import was satisfied inside `recharts`' own
dependency tree, which is production all the way down. #4486 added
`recharts` and `victory-vendor` to `ssr.noExternal` to fix a hydration
mismatch on every server-rendered chart; that inlines `react-smooth`
into the server bundle, which moves its `prop-types` import into the
webapp's own resolution scope — where the package was not available in
production.
So the bundling change was correct about *which* d3-shape build both
sides resolve, and wrong about what the production runtime would be able
to find.
## Verification
`docker/Dockerfile` builds the runtime dependencies with `pnpm install
--prod` against a `turbo prune --scope=webapp --docker` output, so I
reproduced exactly that: pruned the workspace, installed with `--prod`,
and imported `prop-types` from `apps/webapp`.
| | result |
| -- | -- |
| `main` as it stands (devDependency only) | `FAILS:
ERR_MODULE_NOT_FOUND` |
| with this change | `prop-types resolves OK` |
It resolves both as a CommonJS `require` and as an ESM `import`, which
is the form the bundle uses.
I also checked this is not one symptom of a wider problem: of the 169
bare specifier roots the server bundle imports, `prop-types` is the
**only** one that is a devDependency and not a production dependency.
The rest are node builtins or production dependencies.
The hydration fix from #4486 is unaffected — the rebuilt bundle still
carries the rounding d3-path build.
## Notes
`prop-types` is inert in production (its entry point swaps in
`factoryWithThrowingShims`), so this adds a 124 KB package that does no
work at runtime. It has to be resolvable regardless, because the import
is real.
An alternative would be adding `prop-types` to `ssr.noExternal` so it is
inlined and needs no runtime resolution. That keeps the dependency list
honest about the fact that the webapp itself does not use it, at the
cost of bundling a CommonJS package into the ESM server output. This
route is the smaller, better-understood change.
Worth following up separately: a check that every bare import in the
server bundle resolves from a production install would have caught this
before it landed. Local development installs every devDependency, so the
gap is invisible when the built server is run from a working tree.
The dequeue brake released the moment its signal became unreadable.
`refresh()` caught any error from `source.read()` and set the verdict to
`null`, which `computeEngaged()` treats as not-engaged — so a few failed
reads dropped an engaged brake, silently, with no log and no metric.
That handling was symmetric while the risk is not. A source that has
stopped answering correlates with the pressure the brake exists for, so
releasing on read failure gives up protection at exactly the wrong
moment; holding too long only costs throughput.
Now a failed read keeps the last verdict instead of discarding it. The
verdict then ages normally, so the existing `maxVerdictAgeMs` check
becomes the grace window and still bounds how long a dead source can
hold the brake — a permanently unreachable source releases it rather
than pinning dequeuing forever. Because `computeEngaged()` only consults
staleness for an *engaged* verdict, a released one is unaffected and
stays released.
The default grace moves from 15s to 120s, comparable to how long the
brake normally stays engaged.
One guard worth calling out: holding is only safe when something bounds
it, so when `maxVerdictAgeMs` is unset the previous discard behaviour is
kept. Otherwise an unbounded hold could pin the brake indefinitely.
Read failures were previously invisible — the catch block neither logged
nor counted. Adds a `read_failures_total` counter, plus an error log on
the transition into failure rather than once per tick, since the refresh
loop runs every second.
The post-release ramp needs no change: it anchors off the
engaged-to-released transition, so a grace-window release still ramps
back up instead of snapping to full rate, which is what you want after a
blind period.
Tests cover holding while reads fail, releasing past the max age, and
the existing unbounded-config paths are unchanged.
## Summary
Self-hosted Kubernetes deployments can now add tolerations to run pods,
so runs
can schedule onto tainted nodes. Previously the only way to do this was
to patch
the supervisor.
`KUBERNETES_RUNNER_TOLERATIONS` takes a comma separated list of
`key=value:effect`, or `key:effect` to tolerate any value. It applies to
every
run pod, and for runs from a schedule tree it merges with the existing
`KUBERNETES_SCHEDULED_RUN_TOLERATIONS`. Left unset, nothing changes: no
tolerations are added and the pod spec leaves the field off entirely.
The Helm chart takes it as a list:
```yaml
supervisor:
config:
kubernetes:
runnerTolerations:
- dedicated=runs:NoSchedule
- spot:NoExecute
```
## Naming
The issue proposed `KUBERNETES_WORKER_TOLERATIONS`. This ships as
`KUBERNETES_RUNNER_TOLERATIONS` instead, because `RUNNER_*` is already
the prefix
for run pod settings (`RUNNER_HEARTBEAT_INTERVAL_SECONDS`,
`RUNNER_ADDITIONAL_ENV_VARS`, and `DOCKER_RUNNER_NETWORKS` for the
Docker
equivalent), whereas "worker" refers to the supervisor itself throughout
this app.
## Validation
Keys and values are checked against the Kubernetes naming rules when the
supervisor starts, so `dedicated=prod runs:NoSchedule` fails immediately
with a
message naming the offending entry. Without that check a bad value is
accepted at
startup and then rejected by the API server on every pod create, which
stops all
runs with the cause buried in an API error.
`KUBERNETES_WORKER_NODETYPE_LABEL` is
trimmed and validated for the same reason: surrounding whitespace is not
valid in
a label value, so a padded value fails every pod create today.
## Node selector off switch
`KUBERNETES_WORKER_NODETYPE_LABEL` accepts an empty string to skip the
node
selector entirely, so runs schedule on any node. This already worked and
the Helm
chart has always shipped it empty, but it was not documented. It is now.
The issue also asked for general node affinity configuration. That is
not
included: the node selector off switch plus tolerations covers the
reported
problem, and a free form affinity setting is a much larger config
surface to
commit to.
Fixes#4458
## Summary
The four charts above the queues table aggregated over **at most the 25
queues on the current page**. They reused the loader's already-paginated
queue array as a ClickHouse `queue IN (...)` filter, so paging or
re-sorting changed the values, and a name search matching nothing
blanked the whole chart row. The stat tiles above them were already
environment-wide, so the two rows disagreed.
They now read `env_metrics`, the environment-level rollup that already
exists for exactly this (the built-in Queues dashboard and the health
report read it). That is both correct and queue-count-independent: no
`GROUP BY queue` across an entire environment, and no client-side
summing.
Note this is not only a paging artifact: page 1 under-reported too. On
the seeded environment below, page 1 read 82% saturation against a true
87%, because the environment's running total is not the sum of one page
of per-queue gauges.
Three related fixes ride along.
**Scheduling delay and throttling sawed to zero.** Both are
event-driven, so at the 10-second bucket a short range picks, most
buckets hold no samples at all and were drawn as `0ms`. Measured over a
1-hour window: **232 of 349 buckets had no scheduling-delay samples**. A
bucket where nothing started is not a bucket where nothing waited, so
the line was both ugly and wrong. TRQL grows a `minBucketSeconds` floor,
plumbed through the metric resource route, and the hero tiles set 60s.
Buckets that still have no samples render as a gap instead of a dive to
zero.
**The floor must not feed a width-dependent headline.** Two of the four
headlines are not peaks, so widening the plotted buckets moved them:
- **Throttled** is a share of buckets that saw any throttling, so a
single brief throttle came to mark a whole minute instead of ten
seconds: the same seeded events read 17% at 10s and 85% at 60s.
- **Scheduling delay p95** is a percentile, and merging quantile states
over a wider bucket yields a p95 between the sub-buckets' own. Two 240s
samples among twenty in one 10-second sub-bucket give a worst-of-six p95
of 240,000ms against a merged 60-second p95 of 5,000ms — a 48x
understatement of a headline whose tooltip claims it is the worst in the
window.
Both charts keep the floor, since a readable line was the point of it.
Their headlines now come from a second query at the range's natural
bucket width, via an optional `readout` on the tile, so each means what
its tooltip says regardless of how the plotted buckets are sized.
Saturation and backlog are genuinely width-invariant (a max of maxes is
the same at any width), so they are unchanged and issue no extra query.
Both caught by Devin in review; I had wrongly lumped p95 in with the
peaks.
**Charts reported a hydration mismatch on every render.** Recharts
resolved victory-vendor's CJS entry on the server and its ESM entry in
the browser. Those bundle different d3-shape builds, and the CJS one
predates d3-path's digit rounding, so every server-rendered curve
carried full-precision coordinates while the client rounded to 3
decimals:
```
Server: M0,3C0.9305555555555555,3,1.8611111111111112,3,...
Client: M0,3C0.931,3,1.861,3,...
```
Bundling recharts for SSR makes both sides resolve the same ESM build.
Verified: 45 of 45 server-rendered chart curves now match the client,
and the page loads with an empty console.
## Verification
An isolated stack with 40 seeded queues (20 heavily loaded, 20 idle) and
90 minutes of 10-second buckets written into `queue_metrics_raw_v1`, so
the real materialized views built `queue_metrics_v1`, `env_metrics_v1`
and the 5m rollup. Ground truth for the environment: 260 running against
a limit of 300 (**87% saturation**), 800 queued.
| | before | after |
| -- | -- | -- |
| Saturation, page 1 | 82% peak | **87% peak** |
| Saturation, page 2 | 5% peak | **87% peak** |
| Backlog / delay, page 2 | "No activity" | **800 peak / 59.5s** |
| Name search matching nothing | all four charts blank | charts stay
environment-wide |
| Metric refetches on a page change | 4, each painting a skeleton | **0,
no skeleton** |
| Buckets drawn as 0ms with no samples | 232 of 349 | **0** |
| Throttled readout | 17% | **17%**, unchanged by the wider buckets |
| Worst-p95 readout source | plotted buckets | **natural width**, so a
sub-minute spike is not averaged away |
| Crosshair reach, hovering one detail-page chart | 2 of 4 others | **4
of 4** |
| SSR chart curves mismatching the client | 45 | **0** |
The bucket floor was measured across ranges: it widens 10s to 60s at 30m
and 1h, and is correctly a no-op at 12h (300s) and 7d (3600s). One extra
request per page load, for the throttled readout.
The built-in Queues dashboard, which reads `env_metrics` independently,
agrees at 86.7% and 260 of 300.
`internal-packages/tsql` suite green (612 tests), including 5 new ones
for the floor that fail without it. Webapp typecheck, oxfmt and oxlint
clean. Spot-checked the Run metrics dashboard and the per-queue detail
page for SSR regressions from bundling recharts: both render, console
clean.
The queue detail page carries the same event-driven series, so its
scheduling delay, throttling and per-key mean delay take the same
treatment.
## Screenshots
<img width="2540" height="580" alt="after-page1-charts"
src="https://github.com/user-attachments/assets/6cd23f9c-e7fd-4918-bcfa-b1d3340b16d1"
/>
## Rollout
Already behind the per-organization `queueMetricsUiEnabled` flag, so
only gated orgs see any of it. Blast radius is chart values on one page
plus the SSR bundling of recharts; rollback is a revert with no data
migration.
## Stated limitations
- `wait_ms_count` and the quantile state both only count `wait_ms > 0`,
so "nothing started in this bucket" and "everything started instantly"
are indistinguishable in storage. Both render as a gap. Distinguishing
them needs a schema change, which is not in this PR.
- The queue name search deliberately no longer narrows the charts. It
only did so incidentally and incorrectly before (first 25 matches, and
blanked on zero matches). Search-scoped charts would need the full
unpaginated matching set and a server-side aggregate; worth its own
ticket if we want it.
- Bundling recharts for SSR grows the server bundle slightly. That is
the cost of both sides resolving one d3-shape build.
- The plotted delay line is a smoothed 60-second view, so a sub-minute
spike above the one-minute warning threshold can fail to colour the line
even though the headline reports it and colours itself.
- Every chart inside one synced group shares the floor, because the
hover crosshair is a reference line on a category x-axis and only draws
where the hovered bucket exists in the other chart's own data. That
costs the queue detail page's gauges some resolution (1 minute instead
of 10 seconds) in exchange for the crosshair working across the row.
Separately, while taking the screenshots I found a pre-existing
rendering bug unrelated to this change: a **perfectly flat** saturation
series draws no line at all (the readout still shows the right
percentage), which looks like the threshold gradient's offset
degenerating when the series min equals its max. It reproduces on
`main`, so it is not a regression here and I have left it alone; filed
as its own issue.
Refs TRI-12784
GET /api/v1/reports/health threw `no catalog registered for report
"health"` in production (fine in dev): the catalog registered itself as
a side effect of a bare import, which the SSR build tree-shakes under
`"sideEffects": false`. Verified on the built server bundle — main's is
missing the catalog, this branch's carries it.
Fix: catalogs are values on the report registry entries; the resolver
reads them from there and the mutable register-at-import step is gone.
## Summary
Environment API keys backed by the additional-key table can authenticate
API requests using their stored effective scopes. Revoked and expired
keys are rejected, branch environments retain their existing routing
behavior, and last-used timestamps are updated on a throttled
best-effort basis.
## Design
API route builders receive the resolved ability and reject restricted
keys on routes without an authorization declaration. Existing
deployment, environment variable, queue, run, task, batch, session, and
waitpoint routes declare the resources they access.
Trigger and batch responses return server-signed public access tokens,
so additional keys never need access to the environment signing secret.
Root-key rotation also keeps public tokens valid for the existing grace
window.
## Feature notes
- Root environment keys remain unrestricted for backward compatibility.
Additional keys enforce their persisted scopes and fail closed on routes
without an authorization declaration.
- Machine-key requests never exchange one credential for another.
Additional keys cannot retrieve the root key, and rotated root keys are
not upgraded
during their grace window.
- Public JWT validation remains host-owned, while installed RBAC plugins
continue to supply root-key abilities.
- Unfiltered session and run listings preserve existing broad task-read
behavior. Filtered requests enforce the supplied task identifiers.
- Related-run summaries remain embedded in run retrieval for API
compatibility. Retrieving or mutating a related run independently still
requires
permission for that run.
- Queue management authorizes at collection scope, matching the queue
permissions currently issued.
- Batch responses deliberately include server-signed public access
tokens for all clients. Selected-task credentials continue using their
original
credential for per-item authorization.
- Two-phase batches authorize declared task identifiers before creation
and authorize every streamed item. Streaming paths that cannot declare
the
complete task set remain fail closed.
- Authentication telemetry records successful credential resolution
separately from subsequent resource-authorization failures.
- API keys are high-entropy random tokens. SHA-256 is intentionally used
for deterministic indexed lookup, not password hashing.
## Deployment notes
The schema migration must be present before this code is deployed.
Because bearer resolution runs on every authenticated request, deploy
the resolver with additional-key lookup disabled, verify root-key and
public-token parity, then enable lookup before any additional keys can
be issued.
The multi-task authorization tightening changes the result for narrowly
scoped tokens that request tasks outside their grants. Observe
would-deny results before enforcing that check. Request-idempotency keys
are also newly isolated by environment and task, so a retry crossing the
deployment boundary may execute once more before old cache entries
expire.
## Follow-ups
- [x] Add a system-wide kill switch for additional-key lookup, defaulted
off for the initial deployment.
- [x] Add authentication observability by credential kind, result,
latency, and lookup path without recording credential values.
- [ ] ~Add would-deny observability and an independent enforcement
switch for multi-task authorization.~
- [ ] ~Add an independent switch for server-issued batch tokens while
root-key parity is verified.~
- [ ] Confirm every API route reachable by a restricted key has an
explicit authorization declaration or intentionally fails closed.
- [x] Verify root-key rotation, revoked-key grace, and public-token
validation through each bearer resolver path.
Restructures the Kubernetes self-hosting guide around two explicit paths
- an **evaluation install** (bundled datastores, one command) and a
**production install** (external datastores, your own secrets) - so
every configuration decision belongs to one path or the other instead of
being a flat list of options with caveats.
Also in this pass:
- Adds an architecture overview (component-to-`values.yaml` map) and a
post-install "verify it" step.
- Consolidates the previously scattered upgrade notes into a single
collapsible group, and cuts implementation detail and historical asides
that no longer apply.
- Removes a duplicated object-storage section (two configs in two
styles) and trims the Docker ClickHouse note down to what a self-hoster
needs to act on.
## Summary
The Queues list and queue detail pages opened on a 1 day window, and
went back to it every time you navigated between queues or reloaded.
They now default to the last hour, and the period you pick is remembered
across navigations and refreshes.
## Design
The last period is stored in a `queueMetricsPeriod` cookie, written
client-side whenever a `period` lands in the URL and read by both
loaders. A cookie rather than localStorage because the queues list
renders its per-queue metrics columns server-side: with localStorage the
page would paint the 1 hour default and then re-fetch, and the picker
would flash the wrong window.
Both pages resolve the window once, in one place, and pass it down:
```ts
period: resolveQueueMetricsPeriod({
period: value("period"), // a usable period in the URL wins
from: value("from"), // an absolute range means "no period"
to: value("to"),
defaultPeriod, // otherwise the remembered default from the loader
}),
```
That keeps the picker pill and every chart query on the same value, so
no call site falls back to its own default. Periods the picker could
never produce (a hand-edited `?period=garbage`, or a window past the 30
day retention) fall back to the default, and the picker renders the
resolved window rather than the raw search param so the label can't
disagree with the data. Absolute from/to ranges, including drag-to-zoom,
are not remembered, since they would pin later visits to a window that
has gone stale.
While wiring that up: the two queue-metric queries that go straight to
ClickHouse (the list table and the concurrency-keys endpoint) never
applied the org's `queryPeriodDays` limit, so a hand-typed `?period=`
read further back than the plan allows. Everything behind
`/resources/metric` is already clipped that way by `executeQuery`; both
of these now clip with the same limit, capped at the retention window,
and the plan cap is resolved once per load and handed to the page
instead of each route deriving its own copy from the client-side
subscription.
Verified on both pages: default with no cookie is 1 hr, picking 6 hrs
survives navigating away and back to a param-free URL and a hard reload,
clearing the cookie returns to 1 hr, an oversized period falls back
without being remembered, and an absolute range still renders as a
range.
The Help & Feedback → "Contact us" form in the sidebar intermittently
failed to send. The `<Feedback>` dialog was nested inside the Help
popover, so clicking **Send** closed the popover and unmounted the form
mid-submit — canceling the `POST /resources/feedback` before it went
out. The message was silently lost (the success toast still shows). A
race, so it "worked sometimes"; the standalone "I'm stuck!" path was
unaffected.
**Fix:** host the Feedback dialog *outside* the popover (same pattern as
`AskAIRoot`) and open it from the menu item, so closing the popover no
longer tears down the form. `Feedback` gains an optional controlled
`open`/`setOpen` mode; existing `button`-triggered usages are unchanged.
## Changes
- `Feedback.tsx` — optional controlled `open`/`setOpen`; `button` now
optional.
- `HelpAndFeedbackPopover.tsx` — "Contact us…" opens a `<Feedback>`
hosted outside `PopoverContent`.
- `.server-changes/fix-sidebar-feedback.md` — user-facing note.
## Testing
Webapp typecheck passes. Sidebar "Contact us…" now sends on every
attempt (Network: `POST /resources/feedback` → `204`, never
`(canceled)`); "I'm stuck!" and the `?feedbackPanel=` open path
unchanged.
## Summary
The task, scheduled task and agent pages now name their runs table with
its own title bar, and the controls that page the table sit beside it
rather than in the bar at the top of the page. The top bar keeps just
the date filter.
Two agent page layout bugs are fixed along the way: scrolling a wide
runs table sideways dragged the charts off screen with it, and the
details panel stopped short of the bottom of the window.
## Fix
The charts moved because the runs table had no horizontal scroller of
its own. `stickyHeader` swaps the table's `overflow-x-auto` for
`overflow-visible`, so the overflow escaped up to the page scroll box,
and setting only `overflow-y-auto` on that box leaves the computed
`overflow-x` at `visible`, which CSS then promotes to `auto`. The chart
grid is a sibling inside that box, so it scrolled too. The table now
keeps its own scroller (the same rule the queues list already documents)
and the page box clips x so this cannot recur.
The short panel was a second `PageContainer` wrapping the agent routes.
`PageContainer` is `grid-rows-[auto_1fr]`, so a lone child lands in the
`auto` row and its `h-full` resolves against content height instead of
the viewport.
This also reverts the global tooltip `max-w-[230px]` introduced in
[#4131](https://github.com/triggerdotdev/trigger.dev/pull/4131), so
longer tooltips are no longer squeezed into a narrow column.
### Agent overview page showing table now scrolling
<img width="3452" height="1648" alt="CleanShot 2026-08-01 at 12 04
38@2x"
src="https://github.com/user-attachments/assets/ef1ac55d-8ffb-4278-983b-031ed21c1f55"
/>
## Summary
Self-hosted deployments now run ClickHouse from the official
[`clickhouse/clickhouse-server`](https://hub.docker.com/r/clickhouse/clickhouse-server)
image instead of `bitnamilegacy/clickhouse`. Bitnami's free image
catalog is EOL and the frozen legacy archive tops out at ClickHouse
25.7.5, below the 25.8 minimum the platform requires since v4.5.0, which
broke every ClickHouse insert on chart-bundled deployments. Both stacks
now default to 26.2, the same version the platform is developed and
tested against.
Existing deployments keep their ClickHouse data with no manual
migration.
Fixes#4197.
## Details
**Docker Compose**: the `clickhouse` service uses the official image
with its native env vars, plus the recommended `nofile` ulimits. It
reuses the same named volume as before: a `data-paths.xml` config
override points ClickHouse at the `data/` subdirectory of the volume,
which is exactly the layout the Bitnami image used, so old volumes work
in place (including SQL-created users) and fresh installs get the
identical layout. The service follows the required-secrets model:
`CLICKHOUSE_PASSWORD` must be set, matching the other services.
**Helm chart**: the Bitnami ClickHouse subchart is replaced by a
chart-owned single-node StatefulSet and Service running the official
image (non-root, HTTP `/ping` probes, config overrides mounted into
`config.d`, and the same `data-paths.xml` layout compatibility). On
upgrade, the chart automatically adopts the data PVC left behind by the
old subchart (`data-<release>-clickhouse-shard0-0`) via `lookup`, and
`fsGroup` relabeling handles the uid change on first mount. Both the
ClickHouse server and the webapp read the password from the same
chart-managed datastore secret (auto-generated and retained across
upgrades), so the server credential and the app's connection URL always
match. Existing `clickhouse.*` values keep working: `auth` (including
`existingSecret`/`existingSecretKey`), `persistence` (including
`global.storageClass`), `resources`, `secure`, `external.*`,
`configdFiles`, and now `nodeSelector`/`tolerations`/`affinity`.
Bitnami-only keys (`shards`, `replicaCount`, `keeper`,
`resourcesPreset`) are gone; default `resources` requests/limits match
what the old preset applied. The docs state the 25.8 minimum for
bring-your-own ClickHouse.
## Upgrade caveats
An adversarial review of the upgrade path found a few cohorts that need
awareness (all documented):
- **GitOps tools that render with `helm template`** (no cluster access):
PVC auto-detection can't run, so `clickhouse.persistence.existingClaim`
must be set to the old PVC name or ClickHouse starts on a fresh empty
volume. Documented in the values file and the Kubernetes self-hosting
docs. Tools that run real helm installs (e.g. Flux) adopt automatically.
- **A pinned `CLICKHOUSE_IMAGE_TAG`** pointing at a Bitnami tag must be
updated to an official image tag; documented in the Docker self-hosting
docs.
- **Storage without `fsGroup` support** (NFS, hostPath): set
`clickhouse.volumePermissions.enabled: true` for a one-time
ownership-fixing init container.
- **Rollback is not automatic**: once the official image has run, file
ownership changes and the Bitnami image can no longer read the volume
without a manual chown, and ClickHouse does not support downgrades
across the version gap.
## Verification
- Full upgrade simulation for Compose, twice (before and after rebasing
onto the required-secrets release): booted the ClickHouse service from
the old compose file on `main` (Bitnami), wrote thousands of rows, then
brought the same project up with this branch's compose file. The
official 26.2 server came up healthy on the same volume with all rows
intact, SQL-created users working, and writes succeeding.
- Adoption scenarios tested against real containers: old volume + root
entrypoint (Compose), old volume owned by the Bitnami uid + non-root 101
with fsGroup-style group permissions (Kubernetes), and fresh volumes for
both.
- `helm lint`, `helm template` (default values, `existingClaim` set,
external ClickHouse, volumePermissions/scheduling toggles, and the
production example) and kubeconform all pass, mirroring the release CI
steps. The rendered webapp Deployment and ClickHouse StatefulSet resolve
to the same datastore secret key.
- Inserts using
`input_format_json_infer_array_of_dynamic_from_array_of_different_types`
(the setting that fails on 25.7.5) succeed on the upgraded volume.
## Upgrade preflight and docs
A production upgrade report on this branch surfaced two hazards that
predate this PR — both landed in chart 4.5.6 (#4316) — so they are fixed
here rather than left for the next person to hit.
**`secrets.existingSecret` gained two required keys.** The webapp
started reading `PROVIDER_SECRET` and `COORDINATOR_SECRET`, and when
`existingSecret` is set the chart generates nothing, so a missing key
only surfaced as a `CreateContainerConfigError` partway through the
webapp rollout. The pre-install/pre-upgrade validation now looks the
Secret up and fails with the complete list of missing keys, leaving the
running release untouched. It is skipped under `helm template` and
client-side dry-run, where `lookup` cannot read the cluster.
**Bundled datastore credentials moved into the chart-managed Secret**
(`<release>-clickhouse`/`admin-password` →
`trigger-datastore`/`clickhouse-admin-password`). The chart wires both
ends itself, but consumers outside it — maintenance CronJobs, Grafana
datasources, secret syncs — have to be repointed. A new `## Upgrading`
section in the Kubernetes docs carries the old→new mapping, the two new
keys, and a pointer to the ClickHouse image notes.
The existingSecret key list in the docs also named
`OBJECT_STORE_ACCESS_KEY_ID`/`OBJECT_STORE_SECRET_ACCESS_KEY`, which are
env var names rather than keys the chart reads; corrected to the real
key names and the condition under which they apply.
Verified on a throwaway kind cluster with `--dry-run=server`: a
pre-4.5.6 Secret fails with both key names listed, the documented
`kubectl patch` clears it, and default values, `existingClaim`, external
ClickHouse, volumePermissions/scheduling and the production example all
still render. A real `helm install` followed by an upgrade against an
incomplete Secret aborts with the release still at revision 1 and
`deployed`. `helm lint`, the CI render and kubeconform (59 resources, 0
invalid) pass.
---------
Co-authored-by: nicktrn <55853254+nicktrn@users.noreply.github.com>
## Summary
A single run output, trace span, or payload carrying JSON that
ClickHouse can't ingest (for example nesting past its depth limit) used
to fail the whole insert batch, so unrelated runs and spans silently
disappeared from the runs list, traces, and logs. This keeps the rest of
the batch and handles the offending row instead of dropping everything
around it.
## Fix
Recovery is per-table, matched to what each table needs:
- **Runs** (`task_runs_v2`) keep their status. We follow ClickHouse's
failing-row hint to strip just the un-ingestable JSON column(s) so the
run still lands (its output reads from Postgres on the detail page), up
to a configurable limit (`RUN_REPLICATION_MAX_POISON_STRIPS_PER_BATCH`,
default `1`). Past the limit we stop and land the batch with
`allow_errors` in a single pass, skipping the remainder. Cost stays a
fixed handful of inserts no matter how large or poisoned a flush is.
- **Trace events and payloads** (high volume, append-only) recover with
a single `allow_errors` insert: the good rows land in one pass and only
the un-ingestable rows are skipped.
Before falling back, a lightweight sanitizer still repairs what it can
losslessly (lone UTF-16 surrogates, out-of-range integers) so a
repairable row lands in full.
To read the failing-row hint we patch `@clickhouse/client-common`: its
error parser truncates the server response and discards the `(at row N)`
position, so the patch preserves the full text for the recovery path to
read.
## Summary
Adds an admin-only "AI agent" storybook page exploring an animated
identity for the dashboard agent: a resting dot logo that animates while
the agent is thinking, then settles once it is done.
The lead experiment is a 5x5 dot matrix. Shapes are five-line string
bitmaps, a bright head walks each shape's route on a fixed beat, and it
only hands off between shapes on a dot the two share, so the rhythm
never breaks. It comes with 26 faces, six gradient palettes, and light
and dark treatments. Two earlier prototypes (a crisp logo that scatters
into orbiting dots, and a dotted triangle on tilted 3D orbits) are kept
in their own tabs for comparison. Everything is plain canvas code with
no new dependencies.
Also adds an `ask-ai` Button variant: secondary styling with a soft
trigger-green border and padding tuned around the leading logo. The
variant supplies the agent logo itself, so callers write `<Button
variant="ask-ai/small">Ask AI</Button>`. Passing a `LeadingIcon`
overrides it, which is how the thinking animation gets driven.
No release note: the storybook is admin gated and the button variant is
not used in product UI yet.
## Summary
Opening a filter sub-menu that has its own search field left the cursor
outside it, so you had to click into the field before you could type.
The cursor now lands in the search field every time a sub-menu opens.
`ComboBox` now focuses its input whenever the popover is open and the
field is present, so the cursor lands there both when a menu opens
normally and when a sub-menu mounts its field late. It is a no-op
wherever focus already worked.
Verified in the dashboard against the Tags menu: before, the field
mounted with focus still on the popover container; after, it mounts
focused and accepts typing straight away.
The pod-count backpressure source read
`apiserver_storage_objects{resource="pods"}` from an apiserver
`/metrics` scrape. That gauge is a periodically-refreshed cached count,
and it is served by whichever apiserver replica the scrape lands on —
replicas disagree with each other at the same instant, by enough to
swamp the engage/release hysteresis band. Engage and release timing was
therefore partly a function of scrape routing.
This replaces it with a single `limit=1` list of the workload namespace
and computes `remainingItemCount + items.length`. One pod object
transferred, no informer, no watch cache.
Two request-shape constraints are load-bearing and called out in the
code: passing a label or field selector makes the apiserver omit
`remainingItemCount` entirely, and setting `resourceVersion` serves a
cached count rather than a quorum read. Neither is passed.
`remainingItemCount` is only set when the list is truncated, so
`_continue` is the truncation signal — if it is absent the returned page
is the whole collection and `items.length` is already exact. If the list
*is* truncated and the count is missing or implausible, the fetcher
throws rather than guessing.
Failure semantics are unchanged: a throw lands in the monitor's existing
catch, exactly as the previous parse did. The hysteresis, verdict shape,
and gauge are untouched. RBAC is unchanged — the existing role already
grants `pods: list`.
The `/metrics` non-resource grant in the deployment role becomes unused,
and the scrape-timeout env var is now a slight misnomer. Both left alone
deliberately: the grant may be wanted again for other apiserver signals,
and renaming the var would need a coordinated config change for no
behavioural gain.
Tests cover the not-truncated, truncated, missing-count, negative-count
and timeout paths.
A non-delayed run used to get two execution snapshots the moment it was
triggered: `RUN_CREATED` nested in the run-create transaction,
immediately followed by `QUEUED` from its own `BEGIN`/`INSERT`/`COMMIT`.
It now gets a single `QUEUED` snapshot written inside the create, and
the trigger path only publishes to the queue. One fewer row per run on
`TaskRunExecutionSnapshot`, and one fewer round trip on the trigger hot
path.
`EnqueueSystem` gains a `publishRun` seam that enqueues without writing
a snapshot. Every re-enqueue path (waitpoint resume, checkpoint restore,
delayed enqueue, pending version, retry requeue) still calls
`enqueueRun` and writes its own `QUEUED`, so only the first enqueue
changes. The `QUEUED` snapshot still commits before the queue message,
so a dequeue sees a dequeueable status exactly as before.
Two things for reviewers. Nesting the write skips
`createExecutionSnapshot`, which is what emits
`executionSnapshotCreated` and therefore the run timeline's `[engine]
QUEUED` entry, so the trigger path now emits it directly, the same way
the dequeue and attempt-start paths already do for their nested creates.
And `RUN_CREATED` is still written when a dequeued run has no background
worker yet, so the status and both `statuses.ts` helpers stay live and
existing rows keep reading correctly.
Delayed runs are untouched: `DELAYED` then `QUEUED` are two genuinely
different moments and stay two snapshots.
Rollback is a revert. Create-and-enqueue happen in one request in one
process, so no in-flight run needs both code paths to agree during a
rollout.
One note for whoever debugs this path later. The `QUEUED` snapshot now
commits before the queue publish, so a failed publish leaves the run
recorded as `QUEUED` with no queue message. That state was already
reachable, since the publish was never part of the snapshot transaction,
but it used to be recorded as `RUN_CREATED`, which was distinctive
because it never otherwise persisted. `QUEUED` with no message is
indistinguishable from a run waiting on a concurrency slot, so
trigger-time publish failure is now one more cause of an apparently
stuck queued run.
## What
`startRunAttempt` — the run controller's first call when a run starts —
had no retry on transient connection errors. A brief connection blip on
that call would abandon the start and send the run back through the
queue, delaying its first attempt.
This adds a jittered backoff retry, matching the existing
`continueRunExecution` path with a shorter budget, so a transient blip
is ridden out in place instead of bouncing the run.
## Why a shorter budget
The continue path retries generously. Start-attempt keeps a tighter
budget (6 attempts, ~25-40s jittered) so it rides out a transient blip
but never keeps retrying past the point the run would already have been
requeued.
## Safety
Retrying is safe: start-attempt is guarded server-side by the snapshot
id — a retry after a start has already committed is rejected, so it can
never double-start an attempt. A pure connection error (the common case)
never reached the server.
## Scope
One retry-options object on `startRunAttempt`; no other behavior change.
Warm starts share this path and get the same resilience.
## Summary
`batchTriggerAndWait()` could leave a parent run waiting forever. The
2-phase batch API blocks the parent on the batch's waitpoint as soon as
the batch is created, but the batch is only sealed at the end of item
streaming. If streaming never completed, nothing sealed the batch,
nothing completed the waitpoint, and the parent stayed suspended with no
timeout and no way to recover.
Supersedes #4016, which added the reaper alone.
## Fix
Admission for item streaming was being decided twice. Batch creation
passes its own rate limiter, which fixes `expectedCount` and blocks the
parent, and then the item stream had to pass the general API limiter as
well, competing with unrelated traffic. A second limiter could therefore
veto work the first had already committed the parent to. Creation now
mints a bounded grant that the item stream spends, so an admitted batch
can finish streaming. The grant is capped per batch rather than
exempting the path, and every failure mode (no grant, spent grant,
unreachable store) falls back to the normal limiter.
That makes stranding much rarer but not impossible, since a request
timeout or a crash can still end streaming for good. So a seal-timeout
reaper aborts any batch still unsealed after `BATCH_SEAL_TIMEOUT_MS` and
completes the parent's waitpoint with an error, letting
`batchTriggerAndWait()` reject instead of hang. It is race-safe against
a late seal, and it is only scheduled for batches that actually block a
parent, so fire-and-forget batches cost nothing.
Finally, the batches page used to report "Batch completion checked." for
these batches while doing nothing, because the completion path returns
early on an unsealed batch. It now says the batch cannot be resumed.
Rate limiting is no longer the reason a batch strands, so the reaper's
default stays at 30 minutes, comfortably above the SDK's worst-case
stream-retry budget.
## Verification
Unit and container tests cover the grant cap, the bypass ordering (it
runs after the authorization check, so it can never skip
authentication), and the reaper's abort, seal race, idempotency, and
no-waitpoint cases.
Also verified end-to-end against a running stack. With the general limit
exhausted, batch creation and other API calls returned 429 while a
granted batch still streamed and sealed; an ungranted batch id was rate
limited rather than bypassed; and the grant cut off exactly at its
configured attempt count. Reproducing the stranded state on a real
parent run, the batch was aborted at the timeout, the waitpoint
completed with an error, and the parent resumed and finished instead of
hanging. A parentless batch left unsealed was untouched well past the
reaper window.
## Verified against deployed runs
The reaper was proven end to end with a real deployed run (locally-run
supervisor, containerised
run) and a real network fault, rather than a simulated one: toxiproxy
severs the phase 2 item
stream mid-flight so every SDK stream retry genuinely fails, while phase
1 still succeeds. Only
the batch calls traverse the fault, so control-plane traffic is
untouched.
The reproduction is the shape that actually strands a parent: the task
catches the
`BatchTriggerError` the SDK throws and carries on, so the phase 1 block
outlives the thrown error
and the parent hangs at its next suspension point.
With the reaper disabled, the parent sat in `EXECUTING_WITH_WAITPOINTS`
for over 24 minutes holding
two blockers, and stayed stuck across a full infrastructure restart:
```
type | status | has_timeout
BATCH | PENDING | f <- orphan, completedAfter NULL
DATETIME | COMPLETED | t <- the wait already elapsed
```
With the reaper enabled the same task under the same fault completed in
about 75 seconds with zero
blockers left, the batch `ABORTED`, and its waitpoint completed carrying
the error.
Two conditions are required to observe this at all, which is worth
knowing for any future test:
the run must be deployed rather than `trigger dev` (dev runs execute in
process and finish while
still holding blocker rows), and the wait after the caught error must
exceed the checkpoint
threshold, or it is served in process and never suspends.
### Why completing the batch waitpoint is sufficient
`batchTriggerAndWait` runs create, then stream, then wait. A phase 2
failure throws before the wait
is ever reached, and the reaper only fires on an unsealed batch, so the
parent is never suspended
awaiting the batch when it runs. The parent therefore does not need a
synthetic result, only to stop
being blocked. Note this reasoning depends on that ordering: if the wait
were ever reached with an
unsealed batch, completing the batch waitpoint alone would not settle
the caller.
## Follow-ups
- Batches stranded before this ships still need a one-off recovery; the
reaper only schedules at creation time.
- That same property leaves a gap if the process dies between creating
the batch and scheduling the job. A periodic sweep would close it, but
wants a supporting index.
- When a partially streamed batch aborts, children already enqueued keep
running while the parent fails. Left as-is deliberately, since
cancelling triggered work is a bigger semantic call.
<!-- ccr-slack-attribution -->
_Requested by **Eric Allam** · [Slack
thread](https://triggerdotdev.slack.com/archives/C0BEM9Z73TM/p1785491472104199)_
## Checklist
- [ ] I have followed every step in the [contributing
guide](https://github.com/triggerdotdev/trigger.dev/blob/main/CONTRIBUTING.md)
- [x] The PR title follows the convention.
- [ ] I ran and tested the code works
---
## Testing
Static checks only, all clean:
- `pnpm run typecheck --filter trigger.dev`
- `pnpm run format`
- `pnpm run lint`
No live task was triggered against a running project, so the "ran and
tested" box above is left unchecked.
---
## Changelog
**Before:** triggering a task through the MCP server always ran it in
the project's default region. There was no way to pick one.
**After:** the `trigger_task` tool accepts an optional `region` option,
so you can choose the region a run executes in.
**How:** `region: z.string().optional()` was added to
`TriggerTaskInput.options` in `packages/cli-v3/src/mcp/schemas.ts`. No
call-site change was needed — `tools/tasks.ts` passes `options` through
verbatim, and `TriggerTaskRequestBody.options.region` already existed.
The tool description in `docs/mcp-tools.mdx` gained a matching line, and
a patch changeset is included.
There is no batch-trigger MCP tool, so there is no sibling tool to
mirror this change on.
---
## Screenshots
N/A — no UI changes.
Co-authored-by: Claude <noreply@anthropic.com>
`test/envParamRoute.ownership.test.ts` fails on main: 3 of its 4 tests
throw
```
Error: [vitest] No "hasAdminDisplayAccess" export is defined on the
"~/services/session.server" mock. Did you forget to return it from "vi.mock"?
```
#4421 added a `hasAdminDisplayAccess(user)` call to the `env.$envParam`
loader, and the test's `vi.mock` of `session.server` only returns
`requireUser`, so the call blows up. Both changes were green in their
own PR and only conflict once merged together, which is why nobody
caught it.
The mock now mirrors the real implementation rather than returning a
constant, so it stays correct if the test's user fixture is ever varied.
No assertions were changed: the tests were right, the mock was stale.
Worth flagging separately: no workflow runs on push to main, so this has
been red since #4421 landed without showing up anywhere. Every PR opened
since has inherited the failure.
## Summary
Keeps the queue metrics ClickHouse tests stable as wall-clock time
advances.
## Root cause
The fixtures used fixed timestamps. Once those timestamps crossed the
tables' 30-day retention boundary, ClickHouse immediately expired the
inserted aggregate rows and all six tests read empty results.
The fixtures now derive a recent minute-aligned timestamp once per test
file. The second 10-second bucket and ranking window are derived from
the same anchor, preserving deterministic bucket relationships while
keeping rows inside both the raw and aggregate table TTLs.
Verified with `pnpm --filter @internal/clickhouse exec vitest run
src/queueMetrics.test.ts`.
Members added by SSO just-in-time provisioning or Directory Sync never
got
their per-member DEVELOPMENT environments - only invite acceptance and
project creation created them. `trigger dev` returned "Environment not
found" for those members and the dashboard had no dev view.
ensureOrgMember now queues provisioning for every membership it settles,
so
both paths are covered and members missing environments are repaired on
their next sync. Provisioning runs as a common-worker job to keep
sign-in
and directory webhooks off the per-project write loop. A failed enqueue
surfaces for Directory Sync, whose worker retries the idempotent effect,
and is swallowed for sign-in, where the next login enqueues again.
Environment creation now tolerates a concurrent creator so the
project-creation loop and the job cannot collide on the unique index.
Also fixes environment resolution ignoring dev-environment ownership: a
member without their own dev environment could be handed a colleague's
and
have it persisted as their dashboard preference.
## What
Two changes, shipped together:
1. **SDK migration (TRI-12460).** `@team-plain/typescript-sdk` is
deprecated. Move the webapp to its successors — `@team-plain/graphql`
(client) and `@team-plain/ui-components` (`uiComponent` builder).
Behaviour-preserving: the `PlainClient` customer upsert + thread
creation move to the new `client.mutation.*({ input })` shape; the
client now throws on failure, so `sendToPlain` wraps its calls and logs,
staying best-effort.
2. **Org tenant attribution (TRI-12461).** When org context is
available, `sendToPlain` now upserts a Plain tenant keyed by `externalId
= org_id`, links the customer to it, and stamps the created thread with
that tenant — so support threads become attributable to a Trigger.dev
org. Wired into the four add-on quota requests and the plan-cancellation
feedback (which already have org context). The tenant steps are isolated
in their own try/catch and the thread's `tenantIdentifier` is gated on
their success, so a tenant failure never blocks thread creation.
## Not affected
- `customer.externalId` stays `User.id` — the customer cards +
impersonation link are unchanged.
- No ticket content leaves Plain.
- Callers without a single org (e.g. the feedback widget) are unchanged
— the org params are optional.
## Deploy prerequisite
The webapp's Plain API key needs three **new** scopes for attribution to
work (it already has `customer:create`, `customer:edit`,
`thread:create`):
- [x] `tenant:create`
- [x] `tenant:edit`
- [x] `customerTenantMembership:create`
Until granted, nothing breaks — `sendToPlain` logs the forbidden error
and creates the thread without attribution.
## Testing
- `pnpm typecheck --filter webapp` passes; oxfmt + oxlint clean.
- Ran the real `sendToPlain` end-to-end via a throwaway vitest harness
against live Plain — confirmed the code path executes; the live write is
gated only by the key scopes above.
## Summary
A query sent to the query API with a typo in it, like a column name that
does not exist, was being reported as a server error. That put customer
SQL mistakes into our error alerting, where they made up almost all of
the volume on one of our noisiest alerts, and it drowned out the
failures that are actually ours to fix. This makes the level match who
is at fault, and fixes two related problems found alongside it.
## Invalid queries are the caller's, not ours
The query API route already got this right. It checks for `QueryError`,
logs at warn, and returns a 400, with a comment saying the system
handles it gracefully and no alert is needed.
The layer underneath ignored that. `executeTSQL` logged every exception
out of its catch block at error, including the compile failures the
route was about to turn into a 400, and error-level logs are forwarded
to error reporting.
The TSQL package already draws the line we need:
```ts
export class ExposedTSQLError extends BaseTSQLError {
/** An exception that can be exposed to the user. */
}
export class InternalTSQLError extends BaseTSQLError {
/** An internal exception in the TSQL engine. */
}
```
`SyntaxError` and `QueryError` extend the first. So the catch block now
branches on `ExposedTSQLError` and logs those at warn, keeping error for
`InternalTSQLError` and anything unanticipated, which is a genuine
compiler bug.
## SQL the caller wrote is their mistake, not ours
The same asymmetry showed up one level down. A query that compiles fine
can still be rejected by ClickHouse at execution, and most of those
rejections mean the caller's SQL is wrong rather than that we generated
something bad.
This is where the volume actually is. Checking production, one error
group alone, a missing `GROUP BY` on the public query API
(`NOT_AN_AGGREGATE`), accounts for over a million events across hundreds
of users. It is by far the largest error group in the project, and
classifying only by resource limit would have left every one of those at
error level.
So rejections are split three ways in `ClickhouseClient`, which is the
only place holding the parsed `ClickHouseError` and its symbolic type.
By the time the error reaches `executeTSQL` it has been wrapped and the
type is gone, and the type never appears in the message text, so it
cannot be recovered by string matching.
- **Resource limits** (memory ceiling, timeout, row/byte caps) log at
warn. The query is valid, it just asked for more than it is allowed to
spend.
- **Invalid SQL** (`NOT_AN_AGGREGATE`, `UNKNOWN_IDENTIFIER`,
`SYNTAX_ERROR`, the type and parse families) logs at warn **only when
the caller wrote the SQL**.
- **Everything else** keeps alerting.
That gate matters. The client is shared, so the identical rejection on
TRQL *we* generated is our bug and has to stay at error. Callers opt in
with `userAuthoredQuery`:
| caller | who wrote the SQL | opts in |
| --- | --- | --- |
| public query API | the customer | yes |
| query editor | the customer | yes |
| agent charts | the agent's model | yes |
| built-in dashboard tiles | us, in code | no |
| queue metric cards | us, in code | no |
| health report | us, in code | no |
The agent is the one judgement call. Its TRQL is not typed by a person,
but it is also not something a code fix makes correct, so a query it
gets wrong is not worth waking anyone for. The same endpoint serves
built-in tiles whose TRQL we do write, so the opt-in lives with the
caller rather than the route.
Separately, when one of these queries did fail, the log recorded the
generated ClickHouse SQL but not the query the caller actually wrote,
which made the reports hard to act on. `queryWithStats` takes an
optional `logFields` that `executeTSQL` uses to attach the original
TSQL.
## Events were attributed to the wrong request
Chasing the above turned up something broader: only a tenth of the
events on that alert pointed at the query API. The rest were pinned to
unrelated requests that happened to be in flight at the same time, so
the alert looked like the trigger endpoint was failing.
`Sentry.init` runs with `skipOpenTelemetrySetup: true`, because we
register our own OTel pipeline. That skips `initOpenTelemetry`, and one
of the things it does is:
```js
api.context.setGlobalContextManager(new SentryContextManager());
```
The async-context strategy is still installed, but `withIsolationScope`
only marks the OTel context and delegates the actual fork to that
context manager:
```js
// "We depend on the otelContextManager to handle the context/hub"
return api.context.with(ctx.setValue(SENTRY_FORK_ISOLATION_SCOPE_CONTEXT_KEY, true), ...)
```
`provider.register()` installed a plain
`AsyncLocalStorageContextManager`, which does not know that key. The
lookup found no scopes on the context and fell back to the
process-global default isolation scope, so every request wrote its
request data into the same object and the last writer won.
The tracer now registers `SentryContextManager`, which subclasses
`AsyncLocalStorageContextManager`, so OTel behaviour is unchanged. It is
also registered on the path where tracing is disabled, which previously
never called `register()` at all and so had no context manager of its
own.
Tenant tags were always correct, because those come from our own async
local storage rather than the isolation scope. That is why the
attribution being wrong was not obvious.
This affects every error report the webapp sends, not just the query
API.
## Verification
`internal-packages/clickhouse`: 76 tests pass, including eight covering
each level decision against a real ClickHouse container. Three pairs pin
the gate open and shut at both layers: an invalid query, a compile
failure, and a real limit breach driven with `max_rows_to_read` each log
at warn with `userAuthoredQuery` and at error without it.
The isolation fix has a test that reproduces the leak before asserting
the fix. Two overlapping requests each tag their own isolation scope;
with the plain context manager the slower one reads back the other's
tag, and with `SentryContextManager` each reads back its own.
Measured separately against a faithful reproduction of the server's
wiring (own OTel pipeline, CommonJS entry) at 200 concurrent requests:
per-request attribution goes from 0.5% to 100%, while span nesting,
context propagation across awaits, and distinct trace IDs are identical
before and after.
Adds the storage model and authorization contracts needed for multiple
environment API keys. Credentials are represented by hashed values,
revocation and expiration state, and persisted effective scopes.
The built-in authorization fallback exposes full-access policy
preparation, while optional authorization extensions can supply
additional presets and task-aware scope generation. This change does not
create, display, or authenticate additional keys.
## Summary
Three related changes, each independently gated:
**Queue metrics and health.** Per-queue depth, throughput (enqueued,
started, completed), concurrency, whether a queue is throttled, and
scheduling delay (how long a run waits between becoming eligible and
actually starting), plus a per concurrency-key breakdown for keyed
queues. Collected from inside the run queue itself, stored in
ClickHouse, and surfaced on the Queues list, a new per-queue detail
page, the task pages, and the run inspector. The question it answers is
"does this queue have enough concurrency to keep up, and if not, which
key or which limit is the constraint".
**Percent-based queue concurrency limits.** A queue's concurrency
override can now be expressed as a percentage of the environment limit,
stored as the source of truth and re-materialized whenever the
environment limit changes. Absolute overrides above the environment
limit are now **rejected with a 400** instead of being silently capped,
which is a behavior change on `POST
/api/v1/queues/:queue/concurrency/override`.
**The `health` report.** A server-computed verdict on whether work is
flowing, whether the runs that do start are healthy, and whether
telemetry is fresh, rendered as text with sparklines. Available as `GET
/api/v1/reports/:key`, `trigger report`, and the `get_report` MCP tool
(plus a `report` MCP prompt, which shows up as a slash command in hosts
that support prompts).
With the flags off, the Queues page renders the pre-metrics component
verbatim, nothing is emitted, and nothing is written to ClickHouse.
## Configuration
Two independent gates, on purpose. Emission is global so data accrues
for everyone before anyone can look at it; the view is per organization
so it can be turned on for one org at a time without a deploy.
**Runtime flags (no restart)**
| Flag | Store | Gates |
| --- | --- | --- |
| `queue_metrics:enabled` | run-queue Redis key (`"1"`/`"0"`, off by
default) | All emission, gauges and counters. Cached in-process for 10s
with stale-while-revalidate, warmed eagerly at boot so the first op
after a deploy is not dropped. |
| `queue_metrics:gauge_sample_rate` | run-queue Redis key, `0..1` |
Fraction of queue ops that emit a gauge. Counters are never sampled, so
throughput stays exact at any rate. |
| `queueMetricsUiEnabled` | feature-flag catalog: global `FeatureFlag`
row, per-org `Organization.featureFlags` override wins | Whether an org
sees the metrics view at all: the Queues list variant, the queue detail
route, the built-in Queues dashboard, the concurrency-keys endpoint, and
the metrics blocks on task pages and the run inspector. Off by default;
a gated org gets a 404 on the detail route rather than an empty page. |
Both Redis keys are readable and writable from `/admin/queue-metrics`
(super-admin UI, with a live per-shard stream-health table) and
`GET`/`POST /admin/api/v1/queue-metrics` (admin PAT). The admin surface
uses its own Redis client, so it works on any instance regardless of
whether that instance runs the emitter or the consumer.
**Environment variables (boot time)**
| Variable | Default | Notes |
| --- | --- | --- |
| `QUEUE_METRICS_EMIT_ENABLED` | `0` | Constructs the emitter and
injects it into the run engine. Without it the run queue has no emitter
at all. |
| `QUEUE_METRICS_CONSUMER_ENABLED` | `0` | Boots the stream consumer on
this instance. Independent of emission, so consumers can be sized
separately from the API. |
| `QUEUE_METRICS_STREAM_SHARD_COUNT` | `4` | Stream shards, hashed per
queue. |
| `QUEUE_METRICS_CONSUMER_BATCH_SIZE` | `1000` | Poll batch equals
insert batch, so an ack can never outrun a write. |
| `QUEUE_METRICS_REDIS_{HOST,PORT,USERNAME,PASSWORD,TLS_DISABLED}` |
falls back to the run-queue Redis | Set `HOST` to move the metrics
stream onto a dedicated instance so a metrics backlog cannot compete
with the run queue for memory. Self-hosters can leave it unset and get a
single-Redis deployment. |
| `QUEUE_METRICS_COUNTER_STREAM_MAXLEN` | `2000000` shared, `8000000`
dedicated | Bound on how much a stalled consumer can hold. The default
is deliberately lower when the stream shares the queue-critical Redis. |
| `QUEUE_METRICS_COUNTER_ODOMETER_TTL_SECONDS` | `604800` | TTL on the
per-queue cumulative counter key, refreshed on every write, so only
queues idle for the whole window are purged. |
| `QUEUE_METRICS_MAX_QUEUE_NAMES_PER_ENV` | `1000` | Distinct queue
names tracked per environment; overflow collapses into `__overflow__`. |
| `QUEUE_METRICS_MAX_CONCURRENCY_KEYS_PER_QUEUE` | `10000` | Same idea
one level down, per queue. |
| `QUEUE_METRICS_GAUGE_SAMPLE_RATE` | `1` | Default for the live
sample-rate key above. |
| `QUEUE_METRICS_QUERY_TABLES_VISIBLE` | `0` | Lists the queue-metrics
tables in the Query page, its schema docs, the schema API and the AI
query context. Off keeps them unlisted while the feature is dark; a
query naming them still runs either way. |
| `QUEUE_METRICS_CLICKHOUSE_URL` | falls back to the shared wiring |
Runs queue metrics on their own ClickHouse service: the consumer's
inserts and every queue-metrics read go through it, so a metrics-heavy
chart refresh never competes with runs-list or trace reads. Unset
reproduces the previous split exactly (inserts on `CLICKHOUSE_URL`,
reads on the query pool). |
| `QUEUE_METRICS_CLICKHOUSE_READER_URL` | the write URL | Reader split,
so the consumer's inserts can never land on a read endpoint. |
|
`QUEUE_METRICS_CLICKHOUSE_{KEEP_ALIVE_ENABLED,KEEP_ALIVE_IDLE_SOCKET_TTL_MS,MAX_OPEN_CONNECTIONS,LOG_LEVEL,COMPRESSION_REQUEST}`
| `1`, unset, `10`, `info`, `1` | Pool tuning, matching the other
per-workload ClickHouse clients. |
Migrations to apply: ClickHouse `036_create_queue_metrics_v1.sql`, and a
Postgres migration adding the nullable
`TaskQueue.concurrencyLimitOverridePercent`. Both are additive.
## How collection works
Queue operations produce two kinds of signal, and they have opposite
failure modes, so they are handled differently.
**Gauges** (queued, running, queue limit, env queued, env running, env
limit, throttled, plus keys-with-backlog and worst-key wait on keyed
queues) are read *inside* the same Redis script that performs the
enqueue or dequeue, so the reading is atomic with the operation it
describes rather than a racy follow-up read. The script returns them on
its reply and the app forwards them to the stream. Gauges are sampled
and drop-tolerant: they are aggregated with `max`, so a lost reading
costs resolution, never correctness.
**Counters** (enqueued, started, completed, plus nack and dead-lettered)
are cumulative odometers. Each event increments a per-queue key on the
metrics Redis and emits the absolute total, and ClickHouse takes the
difference across buckets at read time. This is the important property
of the design: a summed-delta counter undercounts permanently on any
lost event, while a cumulative one self-heals, because the next
surviving reading restates the whole total. Only bucket granularity can
be lost, never the total. A queue returning after its odometer TTL
expired restarts at 1 and reset detection handles it, which is safe
precisely because expiry only spans a window with no activity.
Both land on one sharded Redis stream. A consumer reads it with a
consumer group, reclaims stale pending entries on a 15s interval rather
than on every poll, maps one entry to one or two ClickHouse rows
(whole-queue and, for keyed queues, per-key), and acks only after the
insert lands. Each batch carries a dedup token derived from its
stream-entry ids, and the target tables set
`non_replicated_deduplication_window`, so a retried batch cannot
double-count either the raw rows or the aggregates that hang off them.
Consumer and emitter both emit OTel metrics
(`queue_metrics.emitter.emitted`,
`queue_metrics.consumer.{entries,rows_inserted,insert_errors,insert_duration,stream_depth,group_lag,pending,lag_unknown}`);
stream depth and group lag are the two worth alerting on, and
`lag_unknown` exists because Redis can report a null lag after a trim,
which must not be read as zero.
## Storage and read path
`queue_metrics_raw_v1` is a short landing table with a 6 hour TTL. Four
aggregate tiers are materialized straight from raw, never cascaded off
each other, each with a 30 day TTL:
- `queue_metrics_v1`, 10 second buckets per queue, the default read path
- `queue_metrics_5m_v1`, 5 minute buckets per queue, for wide ranges and
cross-queue ranking
- `env_metrics_v1`, 10 second buckets per environment, queue-independent
so it stays cheap at any range
- `queue_metrics_ck_v1`, 10 second buckets per concurrency key
Every tier is an MV from raw because the counter states do not survive a
cascade: their merge is order sensitive, so a `-MergeState` chain off
the 10s table inflates the result, and the same property means an
aggregate state may only be merged inside one queue. That constraint is
now enforced by the query engine rather than by reviewer discipline: a
column can declare a `mergeGroupKey`, and any query that references it
without grouping by, or pinning to a single value of, every named key
fails to compile with an actionable message.
On the read side, TRQL gains three tables (`queue_metrics`,
`env_metrics`, and a `queue_metrics_by_key` that is hidden from the
editor, schema docs and schema API but still queryable, so per-key rows
can never silently merge into a plain per-queue query), plus
`deltaSumTimestampMerge` and `quantilesTDigestMerge`. Two schema-level
optimizations ride along: a table can declare coarser rollups, so a
query whose bucket interval is 5 minutes or wider is routed to the 5m
table with no change to the query itself, and it can opt into the
ClickHouse query cache with time bounds floored to a fixed grid, so the
auto-refreshing dashboards actually share cache entries instead of
missing on every tick. Both are caller-side substitutions, so the
printer stays unaware of physical layout.
All of this can also live on its own ClickHouse service. A table
declares the pool its reads run on, the three queue-metrics tables name
the dedicated one, and the ingestion consumer writes through the same
client, so both directions move together with one env var and nothing
else routes differently.
The other engine change is opt-in gap filling: charts can request rows
for empty buckets, where counters zero-fill and gauges carry forward.
Grouped gauge series are densified per group and carried inside a
partition, so a quiet queue's line holds its last value without bleeding
another queue's value into it.
## Queue concurrency limits
`concurrencyLimitOverridePercent` on `TaskQueue` is the source of truth
when an override is set as a percentage; the absolute `concurrencyLimit`
is materialized from it (floored, clamped to at least 1 so a percentage
can never act as a pause, and never above the environment limit). Every
path that changes an environment limit now recalculates the
environment's percent-based overrides afterwards, outside the
transaction, and pushes changed limits to the engine. The push is
attempted even when the stored value did not change, so a previously
failed sync self-heals rather than leaving the database and the engine
diverged; paused queues are skipped so a recalculation cannot
effectively unpause one.
The API accepts exactly one of `concurrencyLimit` or `percent`, and the
reject-instead-of-clamp change above means a request asking for more
than the environment allows now fails loudly. The percent bound (greater
than 0, at most 100) is defined once and shared by the zod schema, the
dashboard mutation handler and the service, so the three cannot drift.
The concurrency-keys table on a queue is now paginated against the
ClickHouse per-key tier, ranked by peak backlog with the total on every
row from a single scan, and only the keys on the current page are
enriched with live counts from Redis. That replaces a hard top-50 cap
with something whose cost is a function of page size rather than key
cardinality.
## The health report
`GET /api/v1/reports/:key?period=&format=markdown|ansi|json`. The
verdict is computed on the server and is deterministic, not
model-generated. Three independent analyzers run over one input
snapshot: flow (is work moving, and if not, is the cause a limit,
throttling, one bad queue, or dead-lettering), execution (are the runs
that start succeeding, and at what latency), and liveness (how fresh is
the telemetry). When telemetry is genuinely stale, the first two are
forced to unknown and every actionable field is stripped, so no surface
ever advises action off stale data.
Authorization is per query table rather than a blanket query grant: a
JWT must be scoped to every table the report reads (`runs`,
`env_metrics`, `queue_metrics`), so a narrowly scoped token cannot pull
a report that reads more than it was granted. `period` is validated as a
shorthand with a 90 day ceiling at the edge. The report catalog is a
registry of `{ load, interpret }` entries, so the next report is a new
entry and no change to the route, the view model, the renderers, the CLI
or the MCP tool.
`trigger mcp` no longer launches the install wizard when stdout is a
TTY, which fixed a real failure: hosts spawn the server over a PTY, so
the wizard would open and the client would time out waiting for a server
that never started. The wizard now needs `trigger mcp --install`.
## The part that is live regardless of every flag
The enqueue and dequeue scripts now return a 2-tuple so a gauge reading
can ride back on the reply. Every return site in the eight affected
scripts is wrapped, and a `nil` original is converted to `false` on the
way out, because a raw `nil` in the first slot would make Lua truncate
the multi-bulk reply and silently drop the gauge on the throttled and
empty-queue paths. The reply shape and the destructuring on the app side
are exercised on every queue operation whether or not metrics are
enabled, so that is the part of `run-engine` worth the closest review.
One behavior fix in the same area: the scheduling-delay anchor is set
only on a run's first entry into the queue. Anchoring it to trigger time
on re-enqueues made waitpoint and checkpoint resumes report the entire
wait as scheduling delay. Queue ordering is untouched, so a re-enqueued
run keeps its position, and nacks deliberately keep the original anchor
because a rolled-back dequeue is the same continuous wait.
A pending-version promotion still anchors to trigger time, on purpose:
that promotion is the run's first real entry into the queue, since the
trigger deliberately held it back waiting for a worker version, and the
TTL is armed at the same point for the same reason. The consequence is
worth naming, because it is a judgement call: a run that waits on a
deployment reports that wait as scheduling delay on its queue, which is
time unrelated to queue capacity.
## Verification
Unit and integration suites across the new package, the run queue, the
mapping layer, the query engine and ClickHouse (including a test that
applies migration 036 through the same splitter CI uses, and a
regression test that inserts the same batch three times to prove the
aggregates do not inflate). Beyond that, the whole path was driven end
to end against a live stack with real runs: emitter to Redis stream to
consumer to ClickHouse to the dashboards, for both the local dev path
and the deployed path where a supervisor drives the dequeue, with
assertions on exact counter reconstruction per queue and per concurrency
key, throttling, environment saturation, scheduling delay, and a
deliberate mid-stream reading drop to confirm the cumulative counters
still reconstruct the correct total. The gated-off state was checked on
every touched surface.
The dedicated ClickHouse service was verified against a second,
separately-schema'd instance: with it configured, the driven counters
reconstruct exactly on the dedicated instance, the shared instance gains
no rows for that window, a read through the query API returns the value
that exists only on the dedicated instance, and a `runs` query still
succeeds (it would fail outright if it were mis-routed to a service
without that table). With the variable unset, the full suite passes
unchanged.
---------
Co-authored-by: Katia Bulatova <katia@trigger.dev>
Co-authored-by: Katia Bulatova <katherine.bulatova@gmail.com>
Co-authored-by: James Ritchie <james@trigger.dev>
<!-- ccr-slack-attribution -->
_Requested via [Slack
thread](https://triggerdotdev.slack.com/archives/C097ZHVKZFA/p1785249693523749)_
## Summary
Accepting an old invitation could change the role of someone who was
already in the organization. A long-pending invite can carry a lower
role than the member has since been promoted to, so accepting it was a
silent demotion. When the accepting user was the organization's only
Owner, the role layer refused that demotion, and the refusal (an
expected, protective outcome) was logged as an error.
An invitation now only sets a role on a membership the accept actually
created, and people who are already in an organization are skipped when
invitations are sent.
## How
`acceptInvite` already skipped the `OrgMember` create when it found an
existing membership, but the `rbac.setUserRole` call below it was gated
only on `invite.rbacRoleId`. It now also tracks whether this accept
created the membership. A create that loses the unique-constraint race
counts as pre-existing, since whichever flow won it owns that
membership's role.
Skipping existing members outright would regress one case: a member with
no RBAC role at all would never receive the invitation's role.
`ensureOrgMember` handles that with `healMissingRoleAssignment`, which
fills in a null role but never overwrites a real one, so
`assignInviteRbacRole` takes the same gate. An established role is never
touched; an absent one is filled in.
`assignInviteRbacRole` branches on the result's machine-readable `code`
instead of logging every refusal at `error`. `last_owner` goes to
`logger.info`, matching the two directory-sync role paths; everything
else, including a refusal that carries no code, goes to `logger.warn`.
The helper is best-effort and never throws, so no outcome it produces
warrants `error`. No string matching on the error text is involved.
`inviteMembers` resolves the organization's members by email and skips
those addresses before creating invites. The invite table's
`@@unique([organizationId, email])` only dedupes *pending invites*, so
it could never catch this.
## Invite surfaces
Skipping addresses means a batch can now come back empty, and neither
caller handled that:
- The dashboard action built its redirect from
`invites[0].organization`, so a batch where every address was skipped
threw a `TypeError` that reached the admin as a raw error string. It
also reported the submitted count rather than the created one. It now
names what it skipped ("No invitations sent: 1 already a member of this
organization") and counts what it actually created.
- The invites API derived `alreadyInvited` as "everything not created",
so an existing member was reported as though they had already been
invited. `inviteMembers` now returns the two groups separately and the
endpoint reports `alreadyMembers` alongside `alreadyInvited`.
## Testing
`apps/webapp/test/member.server.test.ts` passes 16/16 locally, up from
12.
Getting there needed a harness fix. The `~/db.server` mock did not
export `Prisma`, so any code reaching
`PrismaNamespace.PrismaClientKnownRequestError` threw before it could
branch, leaving every duplicate-key path in `member.server.ts`
unreachable from tests. The mock now re-exports the real `Prisma`, and
there is a case covering the pending-invite skip.
New cases: the invite role is applied when the accept creates the
membership; it is not applied when the member already has a role; it is
applied when an existing member has no role assigned; the organization
is still joined when the assignment is refused with `last_owner`; and
`inviteMembers` reports members separately from pending invites. Forcing
the gate off fails exactly the "already has a role" case, so the
coverage is load-bearing.
`pnpm run typecheck --filter webapp` and `oxfmt --check` both pass.
## Changelog
Accepting an old invitation could change the role of someone who was
already in the organization. An invitation now leaves an existing
member's role untouched, people who are already in an organization are
no longer sent invitations to it, and the invite form says which
addresses it skipped instead of failing with an unhelpful error.
---
## ✅ Checklist
- [x] I have followed every step in the [contributing
guide](https://github.com/triggerdotdev/trigger.dev/blob/main/CONTRIBUTING.md)
- [x] The PR title follows the convention.
- [x] I ran and tested the code works.
## Screenshots
No visual changes. The invite form's toast copy changes, as described
above.
---------
Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: Matt Aitken <matt@mattaitken.com>
## Summary
Several docs pages rendered identical `<title>` tags, which weakens
search indexing and makes results ambiguous. Each affected page now has
a unique, descriptive title while keeping its existing sidebar label
unchanged.
Alongside the retitles:
- Removed two stale build-system upgrade pages that were no longer in
the navigation, with redirects to the current package upgrade guide.
- Redirected the build-extensions group index to its overview page so
the two URLs stop sharing a title.
- Dropped a leftover orphaned API reference page (its old URL already
redirects to the management overview).
No links break: nothing in the docs points at the removed pages, and
every redirect target exists.
Long organization, project, and environment names in the side menu were
cut off mid-character. They now fade out at the right edge like the rest
of the side menu items already did.
### Example of faded long names:
<img width="246" height="200" alt="CleanShot 2026-07-28 at 22 59 24"
src="https://github.com/user-attachments/assets/efa60b87-286f-4ab0-9d4e-490ef2de53e5"
/>
## Summary
The confirmation dialog for leaving a team or removing a teammate was
still built on the old `Alert` primitive: the entire question sat in the
title, there was no header divider or `Esc` affordance, and the footer
used small buttons pinned to the right.
It now uses the standard `Dialog` layout the rest of the dashboard uses.
The title is static ("Remove team member" / "Leave team"), the question
moves into the body with the person's name and the organization
highlighted, and the footer is a bordered row with medium Cancel and
confirm buttons. A member who has not set a name is now identified by
their email instead of "them".
Verified against a local dashboard on both dialogs. Confirming a removal
posts the member id, deletes the membership and shows the success toast.
Cancel, `Esc`, and Enter while Cancel is focused all close the dialog
without issuing a request, leaving the member in place.
No release note needed: this is a visual restyle of an existing dialog
with no behaviour change.
The wait docs describe the 5 second compute-billing threshold as if it
were also the suspension threshold. It isn't, and the gap is confusing
when you're sizing a poll interval:
- **Compute** stops being charged for any wait longer than 5 seconds.
- **Concurrency** is only released once the machine has been snapshotted
and shut down. For `wait.for` and `wait.until` that happens 60 seconds
into the wait — a shorter wait stays `EXECUTING` and holds its
concurrency slot for the whole wait, even though the compute is free.
So `await wait.for({ seconds: 30 })` in a polling loop never releases
its slot, which looks like a bug if the docs told you waits over 5
seconds checkpoint.
## Changes
**`docs/snippets/paused-execution-free.mdx`** — rendered on `/wait`,
`/wait-for` and `/wait-until`. Drops "we checkpoint and" from the
billing sentence so it's purely about compute, then adds one paragraph
for the concurrency half.
**`docs/queue-concurrency.mdx`** — the "Waits and concurrency" section
states flatly that waiting runs don't consume slots. Adds a short
subsection for the time-based exception.
**`docs/how-to-reduce-your-spend.mdx`** — "Waits longer than 5 seconds
automatically checkpoint your task, meaning you don't pay for compute" →
the compute claim only. Code comments follow, plus a pointer that
waiting doesn't always free concurrency.
**`docs/how-it-works.mdx`** — the Checkpoint-Resume walkthrough used
`wait.for({ seconds: 30 })` as *the* example of a wait that suspends.
Bumped to 5 minutes and noted the sub-60s exception.
No behaviour change — docs only.
Adds an opt-in way for operators to route deployed runs' API traffic
through a different origin than the public one, per organization. Set
`INTERNAL_API_ORIGIN` on the webapp and enable the
`internalApiOriginEnabled` feature flag (globally or per org, with the
org override winning in both directions): deployed runs for enabled orgs
then get `TRIGGER_API_URL` set to the internal origin instead of
`API_ORIGIN`. Useful for gradually moving run traffic onto a private
network path.
## Design
The origin is resolved when an attempt starts, so flag changes take
effect on the next attempt and roll back the same way, with no task
redeploys. The org override is read fresh per attempt; the global
default comes from the cached flags registry (a cold read fails safe to
the public origin). When `INTERNAL_API_ORIGIN` is unset the flag is a
no-op and no extra queries run, so existing deployments are unaffected.
Dev runs always use the public origin, and `TRIGGER_STREAM_URL` remains
unchanged.
<!-- ccr-slack-attribution -->
_Requested by **Eric Allam** · [Slack
thread](https://triggerdotdev.slack.com/archives/C0AU83M3136/p1785222101937829?thread_ts=1785207509.304669&cid=C0AU83M3136)_
Removes two dead Remix routes and the helpers only they used.
`app/routes/sync.traces.runs.$traceId.ts` (`/sync/traces/runs/:traceId`)
and `app/routes/sync.traces.$traceId.ts` (`/sync/traces/:traceId`) were
added with the original ElectricSQL run page and lost their only
consumers when the dashboard hooks that called them were deleted.
Nothing in the repo references either route today.
Also removed, because the deleted routes were their only callers:
- `OtelTraceIdSchema`, `RESERVED_ELECTRIC_SHAPE_PARAMS`, `TraceScope`,
`buildElectricTraceWhereClause` from `app/v3/electricShape.server.ts`
(the file stays — `UNSAFE_REALTIME_TAG_CHARS` /
`sanitizeRealtimeTagForSql` / `sanitizeRealtimeTagsForSql` are still
used by `realtime.v1.runs.ts` and `realtimeClient.server.ts`)
- the loader-specific cases in
`apps/webapp/test/spanTraceRoutes.replicaLag.test.ts` and
`internal-packages/run-store/src/runOpsStore.routesSpanTraceReadView.replicaLag.test.ts`
`app/utils/longPollingFetch.ts` is untouched —
`realtimeClient.server.ts` still uses it. `runOpsStore.ts` /
`PostgresRunStore.ts` are untouched too; the unrouted-lookup mechanism
there is generic and stays.
As a plain code fact: the run lookup these loaders performed keyed on
`TaskRun.traceId` alone, which is not an index-backed query shape. That
is noted only as context for why the code is not worth keeping around
unused.
### Judgement call worth a maintainer's opinion
The request was specifically about `/sync/traces/runs/:traceId`, the
route that looks up a run by `traceId`. This PR **also** deletes its
sibling `/sync/traces/:traceId`. The reasoning:
- both routes came in with the same ElectricSQL run-page work
- both lost their only consumers in the same later commit
- neither has any caller anywhere in the repo
- they share the same helper module, so keeping one means keeping the
helpers half-used
If you would rather keep the sibling, reverting just that one file
deletion is easy and does not affect the rest of this PR — say the word
and I will restore it along with the helpers it needs.
## ✅ Checklist
- [x] I have followed every step in the [contributing
guide](https://github.com/triggerdotdev/trigger.dev/blob/main/CONTRIBUTING.md)
- [x] The PR title follows the convention.
- [x] I ran and tested the code works
---
## Testing
Verification run locally from the repo root:
| Command | Result |
| --- | --- |
| `pnpm run format` | clean, no changes produced |
| `pnpm run lint:fix` | clean |
| `pnpm run lint` | pass (exit 0, no findings) |
| `pnpm run typecheck --filter webapp` | pass |
| `pnpm run typecheck --filter @internal/run-store` | pass |
A ripgrep sweep for `sync.traces`, `sync/traces`, `syncTraceRunsLoader`,
`buildElectricTraceWhereClause`, `OtelTraceIdSchema` and
`RESERVED_ELECTRIC_SHAPE_PARAMS` (excluding `node_modules`) returns zero
hits.
**Not fully verified:** both edited test files are testcontainers suites
and need a Docker runtime, which was not available in my environment. I
confirmed each file *collects* correctly with exactly the three intended
remaining tests and no import errors — notably, dropping the
`session.server` / `controlPlaneResolver.server` / `longPollingFetch` /
`env.server` mocks does not break module loading for the surviving
loaders. The assertions themselves then failed only on `Could not find a
working container runtime strategy`. CI should be the real signal here.
Per `apps/webapp/CLAUDE.md`, `pnpm run build --filter webapp` was
deliberately not run.
---
## Changelog
Removed two unused sync routes left over from the original ElectricSQL
run page, along with the helpers and tests that existed only to serve
them. No behaviour change — neither route had any caller.
---
## Screenshots
_n/a — no user-visible surface changes._
💯
---------
Co-authored-by: Claude <noreply@anthropic.com>
🚀 Publish Trigger.dev Docker / units (push) Failing after 1s
🚀 Publish Trigger.dev Docker / typecheck (push) Failing after 3s
🚀 Publish Trigger.dev Docker / publish-webapp (push) Has been skipped
🚀 Publish Trigger.dev Docker / publish-worker-v4 (push) Has been skipped
🚀 Publish Trigger.dev Docker / scan-webapp (push) Has been skipped
🚀 Publish Trigger.dev Docker / scan-supervisor (push) Has been skipped
🦋 Changesets PR / Create Release PR (push) Has been cancelled
🧭 Helm Chart Prerelease / lint-and-test (push) Has been cancelled
Workflow Checks / Actionlint (push) Has been cancelled
Workflow Checks / Zizmor (push) Has been cancelled
🚀 Publish Trigger.dev Docker / 📣 Dispatch main image (push) Has been cancelled
🧭 Helm Chart Prerelease / prerelease (push) Has been cancelled
## Summary
PRs opened by the claude GitHub app fail both the agent instructions
audit and the REVIEW.md drift audit before Claude gets a chance to run.
`claude-code-action` refuses any actor whose account type is not `User`
unless the actor is listed in `allowed_bots`:
```
Workflow initiated by non-human actor: claude (type: Bot).
Add bot to allowed_bots list or use '*' to allow all bots.
```
So those PRs land with two permanently red checks and no audit coverage
at all. Both workflows already allowlist Devin; this adds the claude app
alongside it.
## Why this does not open the workflows up to outside contributors
`allowed_bots` is only consulted for non-`User` actors. Humans,
contributor or maintainer, take the separate write-permission path and
are unaffected by what is in the list.
Beyond that, both jobs are guarded by
`github.event.pull_request.head.repo.full_name == github.repository`, so
a fork PR skips the job entirely, and they trigger on `pull_request`
rather than `pull_request_target`, so a fork-triggered run would get no
API key and a read-only token anyway.
The bot is named explicitly instead of using `"*"`, which would let
every bot trigger these audits, dependabot's PR stream included.
## Summary
The batches list page orders by `(createdAt DESC, id DESC)`, which is
why [#4361](https://github.com/triggerdotdev/trigger.dev/pull/4361)
added a matching index on `BatchTaskRun`. That index only landed in
`@trigger.dev/database`.
The dedicated run-ops database has its own migration history, so it
never received the index. `BatchListPresenter` reads both databases and
merges, so for environments whose batches live in the dedicated database
the page kept falling back to a scan and in-memory sort, which is the
exact behaviour #4361 set out to fix.
## Fix
Adds the index to the run-ops schema with its own migration. `CREATE
INDEX CONCURRENTLY IF NOT EXISTS`, so it is a no-op where the index
already exists and still records its ledger row.
The second half is the interesting part. Because the two packages own
separate migration histories, a run-graph schema change has to be
authored twice, and nothing made the miss visible: the run-ops status
check truthfully reports "up to date" against its own history, so the
apply step just skips.
`schemaParity.test.ts` compares the physical shape of every model the
run-ops schema declares against its counterpart in
`@trigger.dev/database`: scalar fields with their attributes, plus
`@@index`, `@@unique`, `@@id` and `@@map`. Relation navigation fields
are excluded, since the run-ops schema deliberately drops relations that
would cross a database boundary while keeping the scalar FK column. A
field counts as a relation when its type resolves to a model name, which
keeps enum-typed columns in scope.
Two models are listed as run-ops-only: `CompletedWaitpoint` and
`WaitpointRunConnection`, both explicit FK-free replacements for a
control-plane implicit many-to-many, since an implicit m2m carries a
foreign key that cannot resolve across databases. The test also asserts
that exception list is exhaustive, so a new unpaired model fails rather
than being silently skipped.
Confirmed the guard actually fails: reverting the index turns
`BatchTaskRun` red with the missing `@@index` named in the diff.
🚀 Publish Trigger.dev Docker / typecheck (push) Failing after 1s
🚀 Publish Trigger.dev Docker / units (push) Failing after 0s
🚀 Publish Trigger.dev Docker / publish-webapp (push) Has been skipped
🚀 Publish Trigger.dev Docker / publish-worker-v4 (push) Has been skipped
🚀 Publish Trigger.dev Docker / scan-webapp (push) Has been skipped
🚀 Publish Trigger.dev Docker / scan-supervisor (push) Has been skipped
🧭 Helm Chart Release / lint-and-test (push) Has been cancelled
🚀 Publish Trigger.dev Docker / 📣 Dispatch main image (push) Has been cancelled
🧭 Helm Chart Release / release (push) Has been cancelled
## Summary
2 new features, 9 improvements, 3 bug fixes.
## Highlights
- Allow additional environment API keys to create scoped public access
tokens through the Trigger.dev API. Use server-issued public access
tokens for batch operations so environment-scoped API keys can read
batch results.
([#4387](https://github.com/triggerdotdev/trigger.dev/pull/4387))
## Improvements
- Preserve the partial assistant message when a chat turn's model stream
fails mid-response. `chat.agent` now passes the recovered partial to
`onTurnComplete`, and `chat.createSession`'s `turn.complete()` keeps it
before rethrowing, instead of dropping the streamed-so-far output.
([#4348](https://github.com/triggerdotdev/trigger.dev/pull/4348))
## Server changes
These changes affect the self-hosted Docker image and Trigger.dev Cloud:
- Favorite any dashboard page to a new Favorites section in the side
menu, and customize the sidebar by renaming favorites, hiding items, and
reordering items and sections.
([#4375](https://github.com/triggerdotdev/trigger.dev/pull/4375))
- List API endpoints now clamp the page size to a maximum of 100.
Requests asking for a larger page size return up to 100 items and keep
paginating, rather than pulling an unbounded page.
([#4360](https://github.com/triggerdotdev/trigger.dev/pull/4360))
- Organizations without billing alerts now get default spend alert
thresholds, so you're notified before usage grows unexpectedly. The
billing limit page no longer pre-selects an option before you've set a
limit and prompts you to configure one. Alert previews now update
immediately after you change your billing limit.
([#4328](https://github.com/triggerdotdev/trigger.dev/pull/4328))
- When you create a Personal Access Token, the generated token now shows
its first and last few characters instead of being fully hidden, so you
can confirm you copied the right value.
([#4363](https://github.com/triggerdotdev/trigger.dev/pull/4363))
- Add metrics to the realtime backend that measure how often a single
changed run is served to multiple subscriptions in one batch.
([#4341](https://github.com/triggerdotdev/trigger.dev/pull/4341))
- Realtime run subscriptions can now be configured to read run data
straight from the primary database, so a run's latest state is never
served from a lagging replica. Off by default; replica reads are
unchanged unless you turn it on.
([#4378](https://github.com/triggerdotdev/trigger.dev/pull/4378))
- SSO and Directory Sync are no longer restricted to Enterprise plans —
get in touch and we can turn them on for your organization whatever plan
you're on.
([#4393](https://github.com/triggerdotdev/trigger.dev/pull/4393))
- Improved supervisor observability: it now reports metrics for its
outbound requests, making failed calls to upstream services easier to
monitor.
([#4350](https://github.com/triggerdotdev/trigger.dev/pull/4350))
- The runs list on a task's page now updates live — run statuses change
and newly triggered runs appear without a manual refresh, matching the
main Runs page.
([#4377](https://github.com/triggerdotdev/trigger.dev/pull/4377))
- Speed up the Batches list page for environments with a large number of
batches, which could previously time out while loading.
([#4361](https://github.com/triggerdotdev/trigger.dev/pull/4361))
- Container startup no longer prints database and ClickHouse connection
strings (with credentials) to the logs.
([#4346](https://github.com/triggerdotdev/trigger.dev/pull/4346))
- The tasks page no longer runs two queries whose results were never
displayed, cutting wasted work on every page load and removing a source
of hidden server errors
([#4380](https://github.com/triggerdotdev/trigger.dev/pull/4380))
<details>
<summary>Raw changeset output</summary>
# Releases
## @trigger.dev/build@4.5.8
### Patch Changes
- Updated dependencies:
- `@trigger.dev/core@4.5.8`
## trigger.dev@4.5.8
### Patch Changes
- Updated dependencies:
- `@trigger.dev/core@4.5.8`
- `@trigger.dev/build@4.5.8`
- `@trigger.dev/schema-to-json@4.5.8`
## @trigger.dev/core@4.5.8
### Patch Changes
- Allow additional environment API keys to create scoped public access
tokens through the Trigger.dev API. Use server-issued public access
tokens for batch operations so environment-scoped API keys can read
batch results.
([#4387](https://github.com/triggerdotdev/trigger.dev/pull/4387))
## @trigger.dev/python@4.5.8
### Patch Changes
- Updated dependencies:
- `@trigger.dev/sdk@4.5.8`
- `@trigger.dev/core@4.5.8`
- `@trigger.dev/build@4.5.8`
## @trigger.dev/react-hooks@4.5.8
### Patch Changes
- Updated dependencies:
- `@trigger.dev/core@4.5.8`
## @trigger.dev/redis-worker@4.5.8
### Patch Changes
- Updated dependencies:
- `@trigger.dev/core@4.5.8`
## @trigger.dev/rsc@4.5.8
### Patch Changes
- Updated dependencies:
- `@trigger.dev/core@4.5.8`
## @trigger.dev/schema-to-json@4.5.8
### Patch Changes
- Updated dependencies:
- `@trigger.dev/core@4.5.8`
## @trigger.dev/sdk@4.5.8
### Patch Changes
- Preserve the partial assistant message when a chat turn's model stream
fails mid-response. `chat.agent` now passes the recovered partial to
`onTurnComplete`, and `chat.createSession`'s `turn.complete()` keeps it
before rethrowing, instead of dropping the streamed-so-far output.
([#4348](https://github.com/triggerdotdev/trigger.dev/pull/4348))
- Allow additional environment API keys to create scoped public access
tokens through the Trigger.dev API. Use server-issued public access
tokens for batch operations so environment-scoped API keys can read
batch results.
([#4387](https://github.com/triggerdotdev/trigger.dev/pull/4387))
- Updated dependencies:
- `@trigger.dev/core@4.5.8`
</details>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
## Summary
The project Integrations page now uses the same settings layout as the
org SSO page: a centered column of titled rows with dividers, instead of
headings over bordered boxes. GitHub, Vercel and build settings read as
one consistent list, and the page titles itself "Integrations".
Confirmations persist rather than vanishing once you move past them
(`GitHub app: Installed`, `Vercel project: Connected`), plan-gated rows
offer an Upgrade button instead of a dead toggle, a disabled toggle
explains why in place and highlights the control that unlocks it, and
warnings are rows with a hazard icon and their recovery action on the
right. Copy throughout leads with the outcome instead of restating the
field label.
Two fixes along the way: a nested `<form>` in the Vercel panel that
failed hydration and silently truncated the page, and every settings row
carrying a few pixels more space above its title than below its
description.
### Before
<img width="1160" height="1972" alt="CleanShot 2026-07-26 at 21 56
42@2x"
src="https://github.com/user-attachments/assets/ed0fd676-36d8-4eb7-a16e-827a24f007d9"
/>
### After
<img width="1358" height="4455" alt="CleanShot 2026-07-26 at 19 14
28@2x"
src="https://github.com/user-attachments/assets/6a635e6a-c0eb-4a4c-a68f-fcde4d25e8a6"
/>
## Summary
Favorite any dashboard page and it appears in a new "Favorites" section
at the top of the side menu. The star next to the page title (or
Option+F) saves the exact view, filters and tabs included, with a name
derived from the URL ("Runs: Completed successfully, last 7d", "Run:
05hrqq9n") that you can rename inline from each item's hover menu.
The sidebar is customizable too: "Customize sidebar" (on section header
menus and in each "More" menu) opens a modal where you can reorder
sections, drag items into a new order, hide items behind a per-section
"More" popover, and rename or remove favorites. Changes apply on
Confirm, Reset restores the default layout without touching favorites,
and everything is stored per user in dashboard preferences.
## Screenshots
| Favorites in the side menu | Customize sidebar modal |
| --- | --- |
| 
| 
|
## Design notes
- Favorite links carry a small marker search param so the favorite, not
its identical main menu item, highlights as active. Markers from shared
or stale links are cleaned on load, and changing any filter hands the
highlight back to the regular menu item.
- Preference writes are serialized with a row lock: several writers
(debounced collapse and width saves, favorite toggles, the customize
modal) can land concurrently and would otherwise clobber each other's
read-modify-write of the JSON column.
- Option+F is matched on `event.code` with a raw listener because macOS
reports Option-modified letters as symbols, which the `event.key` based
shortcut hook can't capture.
Verified end-to-end in the browser: star toggle and shortcut, instant
section appearance, inline rename and staged modal removal, filter-aware
labels and unique active states, shared-link normalization, drag
reordering, and persistence across reloads.
## Summary
Additional environment API keys can use SDK APIs that require public
access tokens. The SDK detects the additional-key format and asks the
Trigger.dev server to mint scoped tokens instead of attempting to sign
them locally.
Root environment keys retain their existing local-signing behavior.
Trigger and batch clients also prefer server-issued tokens returned in
response headers while preserving compatibility with older servers.
## Deployment notes
This package update is safe to publish before servers expose additional
key creation. Existing root keys continue to use the current path, while
an additional key used with an older server fails with an actionable
upgrade error.
## Summary
The runs list on a task's page now updates live, matching the main Runs
page. Run rows update their status, duration, and cost in place as runs
progress, and a "N new runs" button appears in the header when newer
runs come in so you can pull them into the list without a manual
refresh. This applies to both standard and scheduled task pages.
## Design
It reuses the Runs page's polling hook. A task page scopes its runs by
the task in the URL path rather than a `tasks` query filter, so the hook
now takes an optional task slug and scopes new-run detection to it. The
"new runs" button sits in the header, outside the deferred runs table,
so the count is lifted to the page and the click action is passed
through a ref. That keeps the table streaming on first load instead of
blocking the header on the runs query.
When newer runs come in, a `1 new run` button appears in the task page
header, to the left of the time filter. Clicking it pulls the new runs
into the list.
Adds a gitignore rule for `**/docs/superpowers/` so locally-generated
planning and design scratch docs under that path aren't committed;
preventive only, no-op for existing tree.
The SSO & Directory Sync settings page decided access by comparing the
organization's plan code against the literal string `"enterprise"`. The
webapp now reads a `hasSso` entitlement from plan limits.
## Changes
- **`settings.sso` route** — `planAllowsSso` reads `limits.hasSso`
rather than the plan code; the loader and the action gate on a shared
`getSsoEntitlement` helper.
- **`platform.v3.server`** — new `getSsoEntitlement(orgId)` returning
`entitled | not_entitled | unknown`, behind a new SWR cache namespace
(60s fresh / 120s stale, memory + Redis). This replaces an uncached
billing round-trip that previously ran on every settings load, so the
page gets cheaper than it was.
- **`directorySyncEffects`** — the entitlement is now checked before
applying membership effects, per organization and memoised across a
batch.
- **`@trigger.dev/platform` 1.2.0 → 1.3.0** — required, see below.
## Behaviour worth reviewing
**Revocation now stops SCIM.** Previously the plan check existed only on
the settings page, so an org that lost access kept receiving
directory-sync pushes indefinitely; only the config UI froze. Provision
*and* deprovision are gated, so a revoked entitlement can't remove
members either.
**An unreadable entitlement throws instead of skipping.** Effects are
idempotent and the worker retries, so retrying is lossless where
dropping would silently lose a directory change. It's raised at `warn`
level so a transient billing blip doesn't page anyone.
**The login path is deliberately untouched.** A hard entitlement check
there turns a billing outage into a login outage. Consequence: an org
that loses the entitlement keeps its existing SSO logins working until
the connection is removed. Gating sign-in is a separate decision.
**Self-hosted is unaffected.** With no billing service configured the
helper returns `entitled`, leaving plugin presence and the kill switch
as the only gates — a self-hoster who installed the plugin isn't locked
out of it.
## The dependency bump is load-bearing
The `Limits` schema is a plain `z.object`, so it *strips* unknown keys.
On 1.2.0 the `hasSso` field was silently discarded during parsing and
read as `undefined` no matter what billing sent — a structural accessor
would not have helped. Verified against both builds:
```
1.2.0 → parsed: true | hasSso survives: false
1.3.0 → parsed: true | hasSso survives: true
```
This PR therefore cannot merge before 1.3.0 is published, which it now
is.
## Testing
`apps/webapp/test/directorySyncEffects.server.test.ts` — 7 tests over
the gate: applies when entitled, skips provision and deprovision when
not, throws a warn-level retryable error when unreadable, resolves once
per org across a batch, and gates per org so one unentitled org doesn't
block another.
`pnpm run typecheck --filter webapp` passes (18/18), oxfmt and oxlint
clean.
**Before:** `ip-address` resolved twice in `pnpm-lock.yaml` — `8.1.0`
under `@jsonhero/json-infer-types`, and `10.0.1` under
`express-rate-limit`.
**After:** a single `ip-address@10.2.0` entry, shared by both chains.
**How:** `express-rate-limit@8.2.1` pinned `ip-address` to an exact
version, so the parent itself had to move — `8.5.1` onwards declares a
range instead, and `@modelcontextprotocol/sdk` already allows `^8.2.1`,
so scoping that parent to `^8.6.0` lets `ip-address` resolve on its own.
`@jsonhero/json-infer-types` caps `ip-address` at `^8.1.0` and is
already at its latest published release, so that chain gets a scoped
override instead of a parent bump. `jsbn` and `sprintf-js` drop out of
the tree as a side effect.
Both overrides are parent-scoped, so the `cli-v3` chain is deliberately
untouched: it resolves `@modelcontextprotocol/sdk` 1.25.2, which
declares `express-rate-limit ^7.5.0` and pulls in no `ip-address` at
all.
`pnpm-lock.yaml` regenerated. `package.json` and `pnpm-lock.yaml` are
the only two files changed.
Nothing in the repo imports `ip-address` or `express-rate-limit`
directly. Both chains are transitive under `apps/webapp` —
`@jsonhero/schema-infer` (used by `TestTaskPresenter.server.ts`) and
`@vercel/sdk` — so no published `@trigger.dev/*` package is affected.
---
## Testing
- `pnpm install --lockfile-only` regenerates cleanly, and `pnpm install
--frozen-lockfile --lockfile-only` passes, so the lockfile matches the
manifests.
- Package churn is limited to the intended set: `express-rate-limit`
8.2.1 to 8.6.0, `ip-address` 8.1.0 and 10.0.1 collapsing to 10.2.0, and
`jsbn` / `sprintf-js` removed. No other resolution moved.
- `@jsonhero/json-infer-types` only calls `new Address4()` / `new
Address6()` inside a try/catch to classify strings. Ran that exact logic
against both `8.1.0` and `10.2.0` over 27 inputs (v4, v6, zone IDs,
CIDR, IPv4-mapped, malformed, empty, non-strings): identical results in
all 27. Both are still CJS named exports in `10.2.0`, with the same
`engines` floor.
- Drove the real `inferSchema()` path from `@jsonhero/schema-infer` with
`ip-address` forced to `10.2.0`; it still detects `ipv4` and `ipv6`
formats correctly.
- `express-rate-limit` 8.6.0 keeps the same `express` peer range (`>=
4.11`) and the same node floor as 8.2.1. Its new `debug` dependency
resolves to a version already present in the tree.
- `oxfmt --check` passes on the modified `package.json`.
- Both bumped versions clear the repo's `minimumReleaseAge` window; the
newest `express-rate-limit` (8.6.1) and `ip-address` (10.2.1+) releases
do not yet, which is why this lands on 8.6.0 and 10.2.0.
- Not run here: a full monorepo install, typecheck and test suite. No
TypeScript changed, and neither package leaks types into ours —
`ip-address` is not referenced in `json-infer-types`' or
`schema-infer`'s declaration files — so CI should be the judge of the
wider suite.
---
## Changelog
Routine dependency maintenance, no behaviour change. No changeset or
`.server-changes/` entry: the diff touches only the root `package.json`
and `pnpm-lock.yaml`, not `packages/*`, `integrations/*`, `apps/webapp/`
or `apps/supervisor/`.
Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: nicktrn <55853254+nicktrn@users.noreply.github.com>
<!-- ccr-slack-attribution -->
_Requested via [Slack
thread](https://triggerdotdev.slack.com/archives/C097ZHVKZFA/p1785082528841609)_
`TaskListPresenter` created promises that nothing ever consumed. Two of
the three deferred metrics promises it returned had no reader, no
`await` and no `.catch()`, so when the query behind one of them failed
the rejection had nowhere to go.
## Before / After
**Before**
- `TaskListPresenter.call()` returned four things: `tasks`, `activity`,
`runningStats` and `durations`. Its only caller reads `tasks` and
`runningStats`.
- Every load of the tasks page therefore fired two ClickHouse queries
whose results were thrown away.
- If either of those two queries failed, the resulting promise rejection
was unhandled — nothing was awaiting it and nothing had attached an
error handler, so it surfaced as an unhandled rejection at the process
level rather than as an error anyone could attribute to a request.
**After**
- `TaskListPresenter.call()` returns `tasks` and `runningStats` only.
- Two fewer queries run per tasks-page load.
- There is no longer an unconsumed promise that can reject without a
handler. `runningStats` is awaited by its caller, so its failures
continue to be handled the way they always were.
Nothing changes on screen: the tasks page renders `hourlyActivity` and
`runningStates`, and neither of the removed values fed either of those.
## How
The removed values were verified unreferenced before deleting anything:
- `TaskListPresenter` has exactly one caller,
`UnifiedTaskListPresenter`, which reads `taskResult.tasks` and
`taskResult.runningStats` and nothing else.
- No file anywhere in the repo — app code, tests, or type re-exports —
reads an `activity` or `durations` field off the presenter's result.
- `UnifiedTaskListPresenter` builds its own
`unifiedTaskListHourlyActivity` query for the 24h chart the page
actually renders, which is what made the presenter's separate 7-day
daily activity data redundant.
- `getDailyTaskActivity` and `getAverageDurations` on
`ClickHouseEnvironmentMetricsRepository` had no callers other than the
two lines being deleted, so they and their now-orphaned helpers and
types were removed too.
Changes:
- `apps/webapp/app/presenters/v3/TaskListPresenter.server.ts` — drop the
`activity` and `durations` fields (both from the main return and from
the no-current-worker early return) and the two repository calls behind
them. Drop the unreferenced `TaskActivity` type alias. The "don't await
this" comment on the remaining `runningStats` promise now spells out
that the caller has to consume it.
- `apps/webapp/app/services/environmentMetricsRepository.server.ts` —
remove `getDailyTaskActivity` and `getAverageDurations` from the
`EnvironmentMetricsRepository` interface and its ClickHouse
implementation, along with `fillInDailyTaskActivity` and the
`DailyTaskActivity` / `AverageDurations` types.
`getCurrentRunningStats` is the control that shows the diagnosis is
right. It throws on query failure in exactly the same way as the two
removed methods — `if (queryError) throw queryError` — but it never
produced an unhandled rejection, because `UnifiedTaskListPresenter`
passes its promise into a `Promise.all(...).then(...)` chain that the
route then awaits. Same failure mode, opposite outcome, and the only
difference is whether anything consumes the promise.
Follow-ups, not in this PR:
- `AgentListPresenter` returns three sparkline promises in the same
shape and they look similarly unconsumed. Left alone here to keep this
change reviewable.
- With these two callers gone, the `getTaskActivity` and
`getAverageDurations` query builders in `@internal/clickhouse` have no
remaining callers in this repo. Whether to remove them is a separate
call for someone who owns that package.
## ✅ Checklist
- [x] I have followed every step in the [contributing
guide](https://github.com/triggerdotdev/trigger.dev/blob/main/CONTRIBUTING.md)
- [x] The PR title follows the convention.
- [x] I ran and tested the code works
---
## Testing
- `pnpm run typecheck --filter webapp` — passes. This is the meaningful
check here: it proves nothing still references the removed fields,
methods or types.
- `pnpm run format` and `pnpm run lint:fix` — clean, no changes
produced.
- No test file referenced the removed symbols, so no test needed
updating.
---
## Changelog
Server-only change, so this carries a `.server-changes/` note rather
than a changeset:
`.server-changes/task-list-remove-unused-metrics-queries.md`.
> The tasks page no longer runs two queries whose results were never
displayed, cutting wasted work on every page load and removing a source
of hidden server errors
---
## Screenshots
_No visual change — the removed data was never rendered._
Co-authored-by: Claude <noreply@anthropic.com>
## ✅ Checklist
- [x] I have followed every step in the [contributing
guide](https://github.com/triggerdotdev/trigger.dev/blob/main/CONTRIBUTING.md)
- [x] The PR title follows the convention.
- [ ] I ran and tested the code works — n/a, this PR only deletes two
workflow files
---
## Summary
**Before:** two scheduled workflows in this repo posted Dependabot
digests to Slack — a critical-alert check every morning at 08:00 UTC,
and a summary of all open alerts on Mondays at 08:00 UTC.
**After:** neither runs. This reporting is handled centrally now, so the
two in-repo workflows were duplicating it.
**How:** deletes `.github/workflows/dependabot-critical-alerts.yml` and
`.github/workflows/dependabot-weekly-summary.yml`. Both were
self-contained — inline shell, no shared scripts or composite actions —
so nothing else in `.github/` referenced them.
Dependabot itself is unchanged: `.github/dependabot.yml`, alerts, and
version updates all keep working. This removes only the two Slack
notifiers.
The `ENABLE_DEPENDABOT_ALERTS` repository variable existed only to
switch these two workflows off. Nothing else reads it, so it can be
removed from the repository settings if it's set.
---
## Testing
No runtime code changes — this PR only removes two scheduled workflow
files. Verified that nothing else in the repo references either
filename, either workflow name, or the `ENABLE_DEPENDABOT_ALERTS`
variable.
---
## Changelog
Removed the two in-repo scheduled workflows that posted Dependabot
digests to Slack.
Co-authored-by: Claude <noreply@anthropic.com>
🦋 Changesets PR / Create Release PR (push) Has been cancelled
🚀 Publish Trigger.dev Docker / 📣 Dispatch main image (push) Has been cancelled
🚀 Publish Trigger.dev Docker / typecheck (push) Failing after 2s
🚀 Publish Trigger.dev Docker / units (push) Failing after 2s
🚀 Publish Trigger.dev Docker / publish-webapp (push) Has been skipped
🚀 Publish Trigger.dev Docker / publish-worker-v4 (push) Has been skipped
🚀 Publish Trigger.dev Docker / scan-webapp (push) Has been skipped
🚀 Publish Trigger.dev Docker / scan-supervisor (push) Has been skipped
## Summary
The realtime runs feed hydrates run rows from read replicas, which means
it needs a replica-lag gate to avoid serving a run's previous state
right after a write. Setting
`REALTIME_BACKEND_NATIVE_RUN_READS_FROM_PRIMARY=1` reads those rows from
each run store's primary instead, so there is no lag to gate against: no
probe, no wake delay, no stale-read retries. Off by default, so nothing
changes unless you set it.
## Design
The run stores already decide replica-vs-primary from the *brand* on the
read client they are handed: a branded replica keeps the read on the
owning store's replica, an unbranded writer escalates it to that store's
own primary. So this is a one-line choice at the hydrator, and it stays
correct across topologies. With the run-ops split on, each leg lands on
its own writer and the caller's client is never forwarded across
databases; with the split off, it is the single database's primary.
```ts
const runReader = new RunHydrator({
readClient: runReadsFromPrimary ? prisma : $replica,
runStore,
});
```
The same flag skips constructing the lag estimator, since probing a
replica the feed no longer reads would be measuring the wrong thing.
Independently, `AuroraReplicaLagSource` detected Aurora by letting
`aurora_replica_status()` fail, on the assumption that the app-level
catch made that free. It isn't: an unresolvable function is a query
error the driver reports to the error log on every sample, so a
non-Aurora replica produced a continuous stream of error events while
the estimator quietly fell through to its next candidate. It now
resolves the function with `to_regproc` and memoizes the answer, so the
unparseable call never reaches the wire.
## Summary
When a `chat.agent` (or `chat.createSession`) turn's model stream fails
mid-response (e.g. a transport timeout like `UND_ERR_BODY_TIMEOUT`), the
assistant output that already streamed was dropped: `onTurnComplete`
fired with `responseMessage: undefined`, and the manual loop's
`turn.complete()` rethrew without keeping the partial. Apps that
register `hydrateMessages` are hit hardest, since boot-time tail-replay
recovery is off by design.
This preserves the streamed-so-far assistant output while still
reporting the turn as errored, so persistence and recovery keep it.
## Scope of behavior change
Only the **error path** changes. Successful turns are unaffected: the
same chunks stream to the client in the same order, and
backpressure/cancel behave as before. Everything here is a correctness
improvement on a turn that hit a source-stream failure.
## What it does
Follow-up to #4304 (`chat.pipeAndCapture`), extending the same
partial-recovery to the two loops that lacked it:
- **`chat.agent`**: taps the response stream (via a `TransformStream`,
so pass-through backpressure and cancel are preserved) to buffer chunks,
and on a source-stream failure reconstructs the partial (preferring the
`onFinish` message). It's surfaced on the error-path `onTurnComplete`
(`responseMessage`, `rawResponseMessage`, `uiMessages`, `newUIMessages`,
`newMessages`) and committed to the accumulator so the next turn and the
reboot snapshot keep it.
- **`chat.createSession` / `turn.complete()`**: the reconstructed
partial is accumulated (so `turn.uiMessages` reflects it and the caller
can persist after catching) before `turn.complete()` rethrows.
`onBeforeTurnComplete` stays skipped on the error path (it hands out a
writer for a stream that has already broken).
## Correctness properties (each covered by a regression test)
Each test below was confirmed to fail without its fix:
- The recovered partial reaches `onTurnComplete` and the next turn's
accumulated messages.
- An already-committed (possibly enriched) response is not overwritten
if a post-response hook then throws.
- Incomplete tool parts are cleaned from the recovered partial (text
kept), so the UI and model views agree and the next turn isn't poisoned.
- A prior turn's model-only compaction survives an errored turn (append
only the new tail, don't reconvert the full history).
- A reconstructed fragment that reuses an existing message id does not
clobber the complete message.
- Queued `chat.response` data parts are folded into the recovered
partial, matching the success path.
- `newMessages` (model delta) stays symmetric with `newUIMessages`.
## Tests
New `chat-agent-source-stream-error.test.ts` covers the cases above. The
full `@trigger.dev/sdk` unit suite passes and the package build is green
across all supported runtimes (Node 20 to 26, Bun, Deno, Cloudflare
Workers).
Adds an optional `TRIGGER_WARM_START_DISPATCH_URL`. The warm-start
dispatch request uses it when set, otherwise falls back to
`TRIGGER_WARM_START_URL`, so the dispatch target can differ from the
default warm-start URL. No behavior change when unset.
## Summary
Several list endpoints accepted an unbounded page size (`perPage` /
`per_page` / `pageSize`). An unbounded page lets one request pull an
arbitrarily large result set and do a proportional amount of work, which
is a poor default for a shared API.
This clamps the page size to 100 on every list endpoint that was
uncapped, matching the existing cap on `api.v1.runs` and
`api.v1.sessions`. Clamping rather than rejecting keeps existing clients
working: a request for a larger page returns up to 100 items and offset
pagination continues from there.
## Endpoints capped
- `api.v1.schedules` (`perPage`)
- `api.v1.queues` (`perPage`)
- `resources.…versions` (`per_page`)
- `resources.…queues` (`per_page`)
- `admin.api.v1.…engine.report` (`per_page`)
- `admin.api.v1.llm-models` (`pageSize`)
Already capped, left as-is: `api.v1.runs`, `api.v1.sessions`,
`api.v1.deployments`.
## Summary
The batches list page orders by `createdAt DESC, id DESC` filtered by
environment and a created-at window, but the only supporting index on
`BatchTaskRun` was `(runtimeEnvironmentId, id)`. That index can't
satisfy the `createdAt` ordering, so on environments with a large number
of batches the query fell back to a full table scan and in-memory sort,
which could run long enough to hit the statement timeout.
## Fix
Adds `(runtimeEnvironmentId, createdAt DESC, id DESC)` on
`BatchTaskRun`. The query now reads straight from the index in order
with no sort step, returning a page with only a handful of heap fetches
instead of scanning the whole environment slice.
The migration uses `CREATE INDEX CONCURRENTLY IF NOT EXISTS`, so it
takes no table lock and is a no-op if the index already exists.
Adds Prometheus metrics so the supervisor's outbound HTTP calls are
observable - including client-side failures that previously only
surfaced as a log line.
- `supervisor_outbound_request_total{name, method, status, outcome}` -
counts every outbound request. `outcome` separates a transport failure
(`network_error`), an HTTP error response (`http_error`), a response
that failed schema validation (`invalid_response`), and success (`ok`).
- `supervisor_outbound_request_duration_seconds{name, outcome}` -
latency histogram. Leaner labels than the counter (no `status`) to avoid
bucket×label cardinality; buckets match the existing dequeue-latency
histogram since these calls share the same retrying HTTP client and
long-poll envelope.
Coverage:
- The warm-start request (a one-off `fetch`) - instrumented inline; the
response status code is now also included in the failure log (it was
previously dropped).
- All worker API client calls (`SupervisorHttpClient`: dequeue, run
attempt start/complete, heartbeats, snapshots, continue, suspend,
debug-log, connect) - routed through a single instrumented `request()`
helper that reports via an optional `onHttpRequestComplete` callback on
the client, which the supervisor wires into the counter + histogram.
Low cardinality by design: `name` is a **static per-endpoint label**
(e.g. `dequeue`, `start_run_attempt`), never the interpolated URL - so
no run/snapshot IDs land in labels, mirroring the templated `route`
labels on the inbound HTTP server.
Registered on the existing metrics registry, exposed on `/metrics` with
no new wiring. Internal-only change (no package release needed), so the
changelog note is a single `.server-changes` entry.
## Summary
The container entrypoint runs under `set -x`, which echoes every command
to the logs with its variables expanded. Several startup guards
reference full database connection strings, so the DSN (including the
password) was printed to the container logs on every boot. This turns
tracing off around those lines so connection strings are never traced,
while leaving migration behavior and ordinary startup logging unchanged.
## Fix
The leaking lines are the `[ -n "$RUN_OPS_DATABASE_URL" ]` and `[ -n
"$RUN_OPS_LEGACY_DIRECT_URL" ]` guards, and the ClickHouse block (its `[
-n "$CLICKHOUSE_URL" ]` guard plus the lines that build `GOOSE_DBSTRING`
from `CLICKHOUSE_URL`). `set -x` prints each of these with the
credential expanded. Tracing is now disabled around each region and
restored afterward, so non-secret tracing is preserved everywhere else.
The existing legacy-migration subshell already protected its own command
body; this adds the missing protection for the guards and the ClickHouse
block.
```sh
{ set +x; } 2>/dev/null
if [ -n "$RUN_OPS_DATABASE_URL" ]; then
set -x
...
```
## Verification
Built the webapp image and ran it with dummy sentinel connection strings
whose password token is `S3NTINEL_PW_DoNotLog`, then grepped the boot
logs.
Before (unmodified), the token appears in the traced guards:
```
+ [ -n postgresql://user:S3NTINEL_PW_DoNotLog@fake-host:6432/run-ops ]
+ [ -n postgresql://user:S3NTINEL_PW_DoNotLog@fake-host:5432/legacy ]
+ [ -n https://default:S3NTINEL_PW_DoNotLog@fake-host:8443 ]
```
After, `grep S3NTINEL_PW_DoNotLog` on the same run returns nothing, and
the normal "skipping ... migrations" lines still log.
Pins `tar` to `7.5.19` via a root `pnpm.overrides` entry, replacing a
stale range override (`tar@>=7 <7.5.11`) that no longer matched any
installed copy.
The single override collapses all resolved `tar` copies onto one
version:
- `packages/cli-v3` — direct dependency (was 7.5.13)
- `@kubernetes/client-node` (apps/supervisor) — transitive (was 7.5.13)
- `cacache` — transitive (was 6.2.1)
- `giget` — transitive (was 6.2.1)
No source changes; cli-v3's published `^7.5.13` spec already permits
`7.5.19`, so no changeset is needed.
## Summary
The split run-store's id-set read path (`#findRunsByIdSet`, used by the
runs-list hydrate, the realtime hydrator, and engine sweeps) queried the
new store for the entire id set and then probed the legacy store for the
misses. A run's residency is a total function of its id (run-ops ids
live in the new store, every other id in legacy), so each id belongs to
exactly one store. Route each id to its owner and query each store only
for its own ids, in parallel. Same result set, and while a split is
active with most runs still on legacy it removes a wasted new-store
query from every id-set read.
## Change
`#findRunsByIdSet` now partitions the ids by `classifyResidency` and
runs one bounded query per store (skipping an empty side), in parallel,
mirroring `expireRunsBatch` and the single-run `#route`. `finalizeRows`
still applies orderBy/take/skip globally over the merged set.
This drops the id-set path's cross-store fallback, which existed to
prefer the new-store copy when the same id was present in both stores.
That collision cannot arise when each id maps to exactly one store
(nothing writes a legacy-shaped id into the new store), so the fallback
is dead code. The two id-set tests that asserted "new copy wins on
collision" now assert the routing invariant: a legacy-shaped id resolves
to the legacy store and the path never consults the new store.
The open-predicate path (`#findRunsOpen`) is unchanged: an open `where`
has no id to route on, so it still unions both stores and dedupes.
## Default billing alerts + billing limit page UX
- New orgs get default billing alerts: $5, $100, $500, $1000, $2500.
Existing orgs are backfilled by a billing-side data migration (companion
[PR](https://github.com/triggerdotdev/cloud/pull/1657)).
- The billing limit form starts with nothing selected for orgs that
never set a limit — the save button appears once an option is picked.
- The yellow banner now also shows on the billing limits page itself,
asking to configure a limit. Hidden everywhere for members who can't
manage billing.
- Also fixes billing limit alert preview.
Tests
- `apps/webapp/test/billingLimitsRoute.test.ts` — dirty logic for
empty/selected mode
- `apps/webapp/test/billingAlertsDefaults.test.ts` — default values
- `apps/webapp/test/billingAlertsFormat.test.ts` — preview after a limit
change
## Summary
Adds two counters to the native realtime backend so we can see how much
duplicate row serialization the change router does per batch. When a run
changes it can match several held feeds at once (a run subscription plus
one or more tag/list feeds), and today each matching feed serializes
that run's wire value independently. These counters quantify that
fan-out so we can decide whether a shared serialization step is worth
it.
## What they measure
- `realtime_native.emission_run_serializations`: total wire-value
serializations performed across feeds per batch (what the current path
does).
- `realtime_native.emission_distinct_serializations`: distinct (columns,
run) rows those serializations cover (what a serialize-once-per-batch
step would do).
Average feeds-per-run is `run_serializations / distinct_serializations`,
and `1 - distinct / run_serializations` is the serialization work a
shared step could save. Wired through a new optional `onEmissionFanout`
callback on the router. No behavior change.
## Summary
5 improvements, 5 bug fixes.
## Improvements
- Add `node-24` and `node-26` as supported `runtime` options in
`trigger.config.ts`. The `experimental-node-24` and
`experimental-node-26` names are now deprecated aliases and emit a
deprecation warning; switch to `node-24` / `node-26` instead.
([#4337](https://github.com/triggerdotdev/trigger.dev/pull/4337))
```ts
import { defineConfig } from "@trigger.dev/sdk";
export default defineConfig({
runtime: "node-24",
project: "<your-project-ref>",
});
```
- Avoid logging task run environment variable values at debug level
([#4336](https://github.com/triggerdotdev/trigger.dev/pull/4336))
- Custom chat agent loops get two ergonomic wins for owning the turn
loop. ([#4304](https://github.com/triggerdotdev/trigger.dev/pull/4304))
`chat.writeTurnComplete()` now returns the turn boundary's resume
cursors (`lastEventId` for the output stream and `sessionInEventId` for
the input stream), so you can persist them straight from the task
instead of round-tripping them back from the client.
```ts
const { lastEventId, sessionInEventId } = await
chat.writeTurnComplete();
await db.chats.update(chatId, { lastEventId, sessionInEventId });
```
`chat.pipeAndCapture()` no longer throws when a stream is stopped or
fails. It now returns a `PipeAndCaptureResult` whose `message` holds any
partial output captured before the stop or failure, alongside a typed
`status` (`"complete" | "aborted" | "error"`) and, on failure, the
`error`. Read the message off the result:
```ts
const { message, status, error } = await chat.pipeAndCapture(result, {
signal,
});
if (message) conversation.addResponse(message);
if (status === "error") logger.error("turn failed", { error });
```
Note: `pipeAndCapture` previously resolved to `UIMessage | undefined`.
Update call sites to read `.message` from the returned result.
- Suppress a build-time warning that could appear in Vite-based projects
when the optional `@ai-sdk/otel` package is not installed.
([#4188](https://github.com/triggerdotdev/trigger.dev/pull/4188))
## Bug fixes
- Fixes intermittent `trigger dev` run crashes where a run could fail at
boot with a cryptic `Cannot find module .../dev-run-worker.mjs` after a
rebuild had cleaned up the build directory the run was launched against.
Dev runs now retry cleanly instead of hard-crashing when their build
directory is missing, the dev watchdog no longer removes the build tree
of a still-running session, and a run assigned to a worker version that
was superseded by a rebuild now fails fast with a clear message instead
of silently hanging until it times out.
([#4276](https://github.com/triggerdotdev/trigger.dev/pull/4276))
## Server changes
These changes affect the self-hosted Docker image and Trigger.dev Cloud:
- Refreshed the side menu: separate organization and account menus, a
new project switcher, and the menu is now resizable by dragging its
edge. The account Profile page has also been redesigned.
([#4066](https://github.com/triggerdotdev/trigger.dev/pull/4066))
- Allow different organization members to use the same development
branch name without sharing or colliding with each other's branch
environments.
([#4323](https://github.com/triggerdotdev/trigger.dev/pull/4323))
- Limit account settings email input to 254 characters.
([#4330](https://github.com/triggerdotdev/trigger.dev/pull/4330))
- Prevent duplicate Staging and Preview environments when account setup
requests overlap
([#4261](https://github.com/triggerdotdev/trigger.dev/pull/4261))
- Fix the docs link on the empty Prompts page, which pointed to a page
that no longer exists.
([#4247](https://github.com/triggerdotdev/trigger.dev/pull/4247))
<details>
<summary>Raw changeset output</summary>
# Releases
## @trigger.dev/build@4.5.7
### Patch Changes
- Updated dependencies:
- `@trigger.dev/core@4.5.7`
## trigger.dev@4.5.7
### Patch Changes
- Fixes intermittent `trigger dev` run crashes where a run could fail at
boot with a cryptic `Cannot find module .../dev-run-worker.mjs` after a
rebuild had cleaned up the build directory the run was launched against.
Dev runs now retry cleanly instead of hard-crashing when their build
directory is missing, the dev watchdog no longer removes the build tree
of a still-running session, and a run assigned to a worker version that
was superseded by a rebuild now fails fast with a clear message instead
of silently hanging until it times out.
([#4276](https://github.com/triggerdotdev/trigger.dev/pull/4276))
- Add `node-24` and `node-26` as supported `runtime` options in
`trigger.config.ts`. The `experimental-node-24` and
`experimental-node-26` names are now deprecated aliases and emit a
deprecation warning; switch to `node-24` / `node-26` instead.
([#4337](https://github.com/triggerdotdev/trigger.dev/pull/4337))
```ts
import { defineConfig } from "@trigger.dev/sdk";
export default defineConfig({
runtime: "node-24",
project: "<your-project-ref>",
});
```
- Avoid logging task run environment variable values at debug level
([#4336](https://github.com/triggerdotdev/trigger.dev/pull/4336))
- Updated dependencies:
- `@trigger.dev/core@4.5.7`
- `@trigger.dev/build@4.5.7`
- `@trigger.dev/schema-to-json@4.5.7`
## @trigger.dev/core@4.5.7
### Patch Changes
- Add `node-24` and `node-26` as supported `runtime` options in
`trigger.config.ts`. The `experimental-node-24` and
`experimental-node-26` names are now deprecated aliases and emit a
deprecation warning; switch to `node-24` / `node-26` instead.
([#4337](https://github.com/triggerdotdev/trigger.dev/pull/4337))
```ts
import { defineConfig } from "@trigger.dev/sdk";
export default defineConfig({
runtime: "node-24",
project: "<your-project-ref>",
});
```
## @trigger.dev/python@4.5.7
### Patch Changes
- Updated dependencies:
- `@trigger.dev/sdk@4.5.7`
- `@trigger.dev/core@4.5.7`
- `@trigger.dev/build@4.5.7`
## @trigger.dev/react-hooks@4.5.7
### Patch Changes
- Updated dependencies:
- `@trigger.dev/core@4.5.7`
## @trigger.dev/redis-worker@4.5.7
### Patch Changes
- Updated dependencies:
- `@trigger.dev/core@4.5.7`
## @trigger.dev/rsc@4.5.7
### Patch Changes
- Updated dependencies:
- `@trigger.dev/core@4.5.7`
## @trigger.dev/schema-to-json@4.5.7
### Patch Changes
- Updated dependencies:
- `@trigger.dev/core@4.5.7`
## @trigger.dev/sdk@4.5.7
### Patch Changes
- Custom chat agent loops get two ergonomic wins for owning the turn
loop. ([#4304](https://github.com/triggerdotdev/trigger.dev/pull/4304))
`chat.writeTurnComplete()` now returns the turn boundary's resume
cursors (`lastEventId` for the output stream and `sessionInEventId` for
the input stream), so you can persist them straight from the task
instead of round-tripping them back from the client.
```ts
const { lastEventId, sessionInEventId } = await
chat.writeTurnComplete();
await db.chats.update(chatId, { lastEventId, sessionInEventId });
```
`chat.pipeAndCapture()` no longer throws when a stream is stopped or
fails. It now returns a `PipeAndCaptureResult` whose `message` holds any
partial output captured before the stop or failure, alongside a typed
`status` (`"complete" | "aborted" | "error"`) and, on failure, the
`error`. Read the message off the result:
```ts
const { message, status, error } = await chat.pipeAndCapture(result, {
signal,
});
if (message) conversation.addResponse(message);
if (status === "error") logger.error("turn failed", { error });
```
Note: `pipeAndCapture` previously resolved to `UIMessage | undefined`.
Update call sites to read `.message` from the returned result.
- Suppress a build-time warning that could appear in Vite-based projects
when the optional `@ai-sdk/otel` package is not installed.
([#4188](https://github.com/triggerdotdev/trigger.dev/pull/4188))
- Updated dependencies:
- `@trigger.dev/core@4.5.7`
</details>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Wraps the workload_auth_gate_total Counter in the singleton helper (same
pattern as reloadingRegistry.server.ts) so a dev hot reload doesn't
crash with "A metric with the name workload_auth_gate_total has already
been registered". No production behavior change.
## Summary
Adds `node-24` and `node-26` as first-class `runtime` options in
`trigger.config.ts`. Previously these Node versions were only reachable
via the `experimental-node-24` / `experimental-node-26` names.
Those experimental names are now **deprecated aliases**: they still
resolve to `node-24` / `node-26` for backwards compatibility, but
loading a config that uses them prints a deprecation warning pointing at
the new name.
```ts
export default defineConfig({
runtime: "node-24",
project: "<your-project-ref>",
});
```
## Details
- `ConfigRuntime` (the public config schema) now accepts `node-24` and
`node-26` directly; the internal `BuildRuntime` already supported them,
so base images and the deploy path are unchanged.
- `resolveBuildRuntime` passes the new names straight through and keeps
mapping the experimental aliases to their replacements.
- Renamed the runtime helper from `isExperimentalConfigRuntime` to
`isDeprecatedConfigRuntime` and added `deprecatedRuntimeReplacement` so
the CLI can name the replacement in its warning.
- Docs snippet updated to list the new versions and flag the deprecated
names.
Add `.worktrees/` to the repo `.gitignore`.
The pre-push hook runs `oxfmt --check .` and `oxlint .` over the whole
tree, and those tools only read the in-repo ignore files (not a user's
global gitignore). Local git-worktree checkouts placed under
`.worktrees/` therefore got linted/formatted, failing the hook on
unrelated code. Ignoring the directory keeps both tools out of worktree
checkouts. No source changes.
Add a Prometheus gauge `workload_token_enforcement_mode` set to 1 for
the active `WORKLOAD_TOKEN_ENFORCEMENT` value
(`disabled`/`log`/`enforce`), emitted at startup on the shared registry.
The existing mint/verify counters don't distinguish `log` from `enforce`
(the verify outcome is recorded before the reject decision), so
dashboards can't tell which mode a cluster is running. This gauge makes
the active mode queryable at a glance. Supervisor typecheck passes.
Restores the debug panel on the **Tasks** and **Runs** pages, and makes
the data it shows copyable.
Admin/impersonation only — no change for regular users, so there's no
`.server-changes`
<img width="909" height="1420" alt="CleanShot 2026-07-22 at 12 05 27@2x"
src="https://github.com/user-attachments/assets/ce2da167-dc23-422f-83f3-4f4aee9ed32c"
/>
## Summary
The empty-state panel on the Prompts page linked to a docs path that no
longer exists, so the "Prompts docs" button returned a 404. It now
points to the current prompts documentation at /docs/ai/prompts,
matching the link already used in the page header.
## What
The organization side menu previously showed an "Enterprise" badge next
to the SSO & Directory Sync item for any org not on the enterprise plan.
That badge is now removed so the item renders without it.
## Screenshot (before)
<img width="1428" height="649" alt="CleanShot 2026-07-10 at 08 24 41"
src="https://github.com/user-attachments/assets/b9787363-972f-4dd5-bf61-486680f49f4c"
/>
## Summary
Limits user account email addresses to 254 characters in profile
settings and onboarding. Oversized values are rejected before the
uniqueness lookup, and the form fields enforce the same limit in the
browser.
## Fix
Both email update flows use a shared bounded email schema. Basic
validation completes before the uniqueness lookup runs.
## Summary
Allow each organization member to use the same development branch name
without colliding with another member's environment. Fixes#4320.
## Fix
Development branches now use the existing member-scoped project, slug,
and organization-member key for upserts. Preview branches retain their
project-wide shortcode behavior.
New development branches receive distinct shortcodes while keeping their
readable, member-scoped slugs. Existing branches continue to resolve
through the member-scoped key, so this requires no migration or
backfill.
Two small corrections to the organization **Usage** page credits
display.
### 1. Label the credits panel "Credits" (was "Promo credits")
The panel surfaces any credit balance, not only promo-code redemptions,
so "Promo credits" is misleading when the credits come from another
source. Renamed the heading to "Credits".
### 2. Don't show "Included usage" for Enterprise orgs
Enterprise inherits the Pro plan's `includedUsage` value, so the Usage
bar rendered an "Included usage: $50" tier marker for Enterprise
organizations. Enterprise bills against prepaid credits rather than a
per-month included-usage tier, so the marker was misleading. The
`tierLimit` marker is now suppressed for Enterprise (`plan.type ===
"enterprise"`).
Verified with `pnpm run typecheck --filter webapp`.
Redesign of the main side menu: separates Projects and Accounts from the
Organization menu and makes the menu resizable.
**Main changes**
- **Organization & Account menus**: the top-left is now a dedicated
organization menu (Settings, Usage, Billing, Team, SSO, integrations),
with a separate account menu beside it (Profile, PATs, Security,
Logout).
- **Project switcher**: a new Project section above the Environment
selector.
- **Resizable side menu**: drag the right edge to set a custom width
(saved per user), or click the edge to collapse/expand.
- **Environment selector**: reworked to match the Project menu,
including dev-branch handling.
- **Account Profile page**: redesigned into the Security page's
row-and-divider layout.
Preview URL: https://samejr-org-menu-update.triggerlabs.dev/https://github.com/user-attachments/assets/9b199576-6037-4ea6-9bdb-3ee15265b8c2
## Summary
Two ergonomic additions for custom chat-agent loops that own the turn
loop (`chat.customAgent`, `chat.createSession`, and the hand-rolled
primitives).
`chat.writeTurnComplete()` now resolves to `{ lastEventId }`, the resume
cursor for the start of the next turn. A custom loop can persist it
straight from the task instead of round-tripping it back from the client
after the turn ends. The value was already produced internally by the
turn-complete write; the public wrapper simply discarded it.
`chat.pipeAndCapture()` no longer throws when a stream is stopped or
fails. It now resolves to a `PipeAndCaptureResult` carrying any partial
`message` captured before the stop or failure, a typed `status`
(`"complete" | "aborted" | "error"`), and the `error` on failure.
Previously a failed stream threw and the partial was lost, and an abort
was captured only when the AI SDK happened to fire `onFinish` in time.
```ts
const { message, status, error } = await chat.pipeAndCapture(result, { signal });
if (message) conversation.addResponse(message);
if (status === "error") logger.error("turn failed", { error });
const { lastEventId } = await chat.writeTurnComplete();
await db.chats.update(chatId, { lastEventId });
```
## Design
`pipeAndCapture` wraps the pipe in a `try/catch` and classifies the
outcome from the abort signal (a stop drains the source stream cleanly
rather than throwing) versus a thrown error. It also races the
`onFinish` capture against a timeout so a hard stop that prevents
`onFinish` from firing can't hang the caller. This mirrors the capture
path `chat.agent` already uses internally.
The `finishReason` from `onFinish` is surfaced too, since it was already
captured on the built-in path.
The internal `turn.complete()` helper keeps its existing contract: it
still returns `UIMessage | undefined`, still throws on a genuine stream
failure, and still discards output on a full run cancel.
## Breaking change
`chat.pipeAndCapture` previously resolved to `UIMessage | undefined`.
Call sites now read `.message` off the result. This is a young,
low-level API; the docs examples are updated in this PR.
Fixes TRI-12078
## Summary
Prevents concurrent environment setup requests from creating duplicate
Staging and Preview environments.
## Fix
Adds database-enforced uniqueness for root Staging and Preview
environments.
If two requests race, the losing request loads the environment created
by the winner and continues successfully instead of creating a duplicate
or returning an error.
## Summary
Upgrades the workspace to TypeScript 6.0.3 and applies the compiler,
type, and build configuration changes required to preserve package
layouts and existing runtime behavior, apart from correcting the HTTP
status field used for deployment connection errors.
## Compatibility
- Centralizes TypeScript 6.0.3 through the pnpm workspace catalog.
- Replaces compiler options and module resolution modes that TypeScript
6 no longer accepts.
- Restores explicit Node types where TypeScript 6 no longer includes
them transitively.
- Adds explicit declaration build roots that preserve each package's
existing output layout.
- Patches tsup to stop injecting the removed `baseUrl` option during
declaration builds.
- Uses type-only assertions for stricter typed-array and stream
definitions without changing runtime behavior.
- Reads the EventSource v3 HTTP status from `code`, so deployment
connection errors include it correctly.
- Keeps standalone CLI compatibility fixtures pinned to their existing
TypeScript version and lockfiles.
`turbo run typecheck` and the complete PR test suite are green.
## What
The `/promo` page heading rendered with overlapping lines — the two
lines of "Promo codes are for new accounts" collided.
## Why
The page used `Header2` stretched to display sizes (`sm:text-2xl
md:text-3xl lg:text-4xl`), but `Header2` bakes in a fixed `leading-6`
(24px). A 36px font in a 24px line box makes wrapped lines overlap. It
only showed at `sm`+ widths and only on headings that wrap to 2+ lines,
which is why it slipped through — the short single-line headings on the
same page looked fine.
## Fix
Switch both headings to `Header1` — the page-title primitive the sibling
login pages (`login._index`, `login.magic`) already use for exactly this
size. Add `leading-tight` (relative line-height, scales with font size,
and this heading uniquely wraps to two lines) and `pb-4` to match the
login pages' spacing convention.
## Testing
Manually verified the signed-in view (`/promo` while logged in) renders
as two clean, non-overlapping lines across breakpoints. Pure CSS/layout
change — no automated test.
## Summary
5 improvements, 9 bug fixes.
## Breaking changes
- Self-hosted deployments no longer ship shared default credentials;
fresh installs generate their own. If yours still uses a previously
published default, set a unique value before upgrading, or set
`ALLOW_INSECURE_DEFAULT_SECRETS=true` to keep booting while you migrate.
([#4316](https://github.com/triggerdotdev/trigger.dev/pull/4316))
## Improvements
- Require explicit browser approval for CLI and MCP login, with
resilient polling while approval is pending.
([#4316](https://github.com/triggerdotdev/trigger.dev/pull/4316))
- Deployed task telemetry now reports the deployment identifier (e.g.
`deployment_abc123`) in the `worker.id` attribute, instead of an opaque
internal value. Upgrade to get the readable identifier in your own
OpenTelemetry exporters.
([#4316](https://github.com/triggerdotdev/trigger.dev/pull/4316))
- Prevent prototype pollution when applying run metadata operations or
reconstructing nested telemetry attributes, while preserving legitimate
`constructor` and `prototype` fields.
([#4316](https://github.com/triggerdotdev/trigger.dev/pull/4316))
- Add helpers to mint and verify the deployment-scoped token used to
authenticate run controllers to the platform.
([#4316](https://github.com/triggerdotdev/trigger.dev/pull/4316))
## Server changes
These changes affect the self-hosted Docker image and Trigger.dev Cloud:
- Added optional request rate limiting for telemetry ingestion
endpoints.
([#4316](https://github.com/triggerdotdev/trigger.dev/pull/4316))
- Background-worker deployment lookups are now scoped to the
authenticated environment.
([#4316](https://github.com/triggerdotdev/trigger.dev/pull/4316))
- Updating a GitHub App installation from the callback flow is now
scoped to your own organization, so an installation ID belonging to
another organization can no longer be used to refresh that
organization's installation record. The GitHub App installation session
is also now single-use, so completing an installation callback
invalidates its state and it can no longer be replayed.
([#4316](https://github.com/triggerdotdev/trigger.dev/pull/4316))
- Scope schedule and environment-variable writes to the caller's project
and environment
([#4316](https://github.com/triggerdotdev/trigger.dev/pull/4316))
- Reject compute snapshot callbacks that do not match the snapshot
request that created them.
([#4316](https://github.com/triggerdotdev/trigger.dev/pull/4316))
- Require secret-key authentication to initialize the session out
(agent→client) stream, matching the append route.
([#4316](https://github.com/triggerdotdev/trigger.dev/pull/4316))
- Live run and trace subscriptions now validate their identifiers more
strictly and only return data from your own organization.
([#4316](https://github.com/triggerdotdev/trigger.dev/pull/4316))
- Window-function names in the query compiler are now validated against
the allowlist, matching how other function calls are handled.
([#4316](https://github.com/triggerdotdev/trigger.dev/pull/4316))
- Authenticate run controllers to the platform with a signed,
deployment-scoped token.
([#4316](https://github.com/triggerdotdev/trigger.dev/pull/4316))
- Verify that worker actions (starting, completing, and continuing a
run, and reading its snapshots) target a run belonging to the caller's
environment.
([#4316](https://github.com/triggerdotdev/trigger.dev/pull/4316))
<details>
<summary>Raw changeset output</summary>
# Releases
## @trigger.dev/build@4.5.6
### Patch Changes
- Updated dependencies:
- `@trigger.dev/core@4.5.6`
## trigger.dev@4.5.6
### Patch Changes
- Require explicit browser approval for CLI and MCP login, with
resilient polling while approval is pending.
([#4316](https://github.com/triggerdotdev/trigger.dev/pull/4316))
- Deployed task telemetry now reports the deployment identifier (e.g.
`deployment_abc123`) in the `worker.id` attribute, instead of an opaque
internal value. Upgrade to get the readable identifier in your own
OpenTelemetry exporters.
([#4316](https://github.com/triggerdotdev/trigger.dev/pull/4316))
- Updated dependencies:
- `@trigger.dev/core@4.5.6`
- `@trigger.dev/build@4.5.6`
- `@trigger.dev/schema-to-json@4.5.6`
## @trigger.dev/core@4.5.6
### Patch Changes
- Prevent prototype pollution when applying run metadata operations or
reconstructing nested telemetry attributes, while preserving legitimate
`constructor` and `prototype` fields.
([#4316](https://github.com/triggerdotdev/trigger.dev/pull/4316))
- Require explicit browser approval for CLI and MCP login, with
resilient polling while approval is pending.
([#4316](https://github.com/triggerdotdev/trigger.dev/pull/4316))
- Add helpers to mint and verify the deployment-scoped token used to
authenticate run controllers to the platform.
([#4316](https://github.com/triggerdotdev/trigger.dev/pull/4316))
## @trigger.dev/python@4.5.6
### Patch Changes
- Updated dependencies:
- `@trigger.dev/core@4.5.6`
- `@trigger.dev/build@4.5.6`
- `@trigger.dev/sdk@4.5.6`
## @trigger.dev/react-hooks@4.5.6
### Patch Changes
- Updated dependencies:
- `@trigger.dev/core@4.5.6`
## @trigger.dev/redis-worker@4.5.6
### Patch Changes
- Updated dependencies:
- `@trigger.dev/core@4.5.6`
## @trigger.dev/rsc@4.5.6
### Patch Changes
- Updated dependencies:
- `@trigger.dev/core@4.5.6`
## @trigger.dev/schema-to-json@4.5.6
### Patch Changes
- Updated dependencies:
- `@trigger.dev/core@4.5.6`
## @trigger.dev/sdk@4.5.6
### Patch Changes
- Updated dependencies:
- `@trigger.dev/core@4.5.6`
</details>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
## Summary
5 improvements, 5 bug fixes.
## Improvements
- Add experimental Node.js 24 and 26 task runtimes. Set `runtime` to
`experimental-node-24` or `experimental-node-26` in `trigger.config.ts`.
([#4085](https://github.com/triggerdotdev/trigger.dev/pull/4085))
- Add `defaultRegion` to the project GET and list API responses; null
when unset.
([#4146](https://github.com/triggerdotdev/trigger.dev/pull/4146))
## Server changes
These changes affect the self-hosted Docker image and Trigger.dev Cloud:
- Transient internal sync failures are now retried quietly instead of
surfacing as errors.
([#4270](https://github.com/triggerdotdev/trigger.dev/pull/4270))
- Optionally route ClickHouse read traffic to a read replica while
writes stay on the primary. Set `CLICKHOUSE_READER_URL` to move all
reads, or target the busiest paths with `RUNS_LIST_CLICKHOUSE_URL` (runs
list) and `EVENTS_READER_CLICKHOUSE_URL` (traces, spans, logs). All
optional; unset keeps current behavior.
([#4081](https://github.com/triggerdotdev/trigger.dev/pull/4081))
- Remove the deprecated realtime stream write endpoint used by retired
v3 task clients.
([#4250](https://github.com/triggerdotdev/trigger.dev/pull/4250))
- Fix batchTrigger requests that set a per-item idempotency key failing
with an error instead of creating and deduplicating the runs
([#4271](https://github.com/triggerdotdev/trigger.dev/pull/4271))
- Speed up idempotency checks on `batchTrigger` calls that use
idempotency keys. Large batches against a task with a big run history no
longer degrade to multi-second lookups.
([#4255](https://github.com/triggerdotdev/trigger.dev/pull/4255))
- The "Preview branches" usage on the Limits page now counts only
preview branches.
([#4283](https://github.com/triggerdotdev/trigger.dev/pull/4283))
- Avoid opening a redundant database connection pool when the legacy and
primary databases are the same server, preventing connection usage from
doubling.
([#4253](https://github.com/triggerdotdev/trigger.dev/pull/4253))
- Fix pages occasionally loading unstyled or failing to load during a
deploy. The dashboard now reloads automatically to recover.
([#4282](https://github.com/triggerdotdev/trigger.dev/pull/4282))
<details>
<summary>Raw changeset output</summary>
# Releases
## @trigger.dev/build@4.5.5
### Patch Changes
- Updated dependencies:
- `@trigger.dev/core@4.5.5`
## trigger.dev@4.5.5
### Patch Changes
- Add experimental Node.js 24 and 26 task runtimes. Set `runtime` to
`experimental-node-24` or `experimental-node-26` in `trigger.config.ts`.
([#4085](https://github.com/triggerdotdev/trigger.dev/pull/4085))
- Updated dependencies:
- `@trigger.dev/core@4.5.5`
- `@trigger.dev/build@4.5.5`
- `@trigger.dev/schema-to-json@4.5.5`
## @trigger.dev/core@4.5.5
### Patch Changes
- Add experimental Node.js 24 and 26 task runtimes. Set `runtime` to
`experimental-node-24` or `experimental-node-26` in `trigger.config.ts`.
([#4085](https://github.com/triggerdotdev/trigger.dev/pull/4085))
- Add `defaultRegion` to the project GET and list API responses; null
when unset.
([#4146](https://github.com/triggerdotdev/trigger.dev/pull/4146))
## @trigger.dev/python@4.5.5
### Patch Changes
- Updated dependencies:
- `@trigger.dev/core@4.5.5`
- `@trigger.dev/build@4.5.5`
- `@trigger.dev/sdk@4.5.5`
## @trigger.dev/react-hooks@4.5.5
### Patch Changes
- Updated dependencies:
- `@trigger.dev/core@4.5.5`
## @trigger.dev/redis-worker@4.5.5
### Patch Changes
- Updated dependencies:
- `@trigger.dev/core@4.5.5`
## @trigger.dev/rsc@4.5.5
### Patch Changes
- Updated dependencies:
- `@trigger.dev/core@4.5.5`
## @trigger.dev/schema-to-json@4.5.5
### Patch Changes
- Updated dependencies:
- `@trigger.dev/core@4.5.5`
## @trigger.dev/sdk@4.5.5
### Patch Changes
- Updated dependencies:
- `@trigger.dev/core@4.5.5`
</details>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
## What
Run the shared Trivy image scan on every published image through a
single reusable workflow.
## Changes
- Generalise the image-scan workflow (`trivy-image-webapp.yml` ->
`trivy-image.yml`) - it was already parameterised by `image-ref`; only
the run-summary label was image-specific.
- `publish-worker-v4.yml`: expose `version` + `image_repo` as workflow
outputs (single-entry matrix, so unambiguous).
- `publish.yml`: run the shared scan from each publish job
(`scan-webapp`, `scan-supervisor`).
Report-only (writes a table to the run summary), OS packages only
(`vuln-type: os` - library deps stay with Dependabot), never blocks the
publish.
## Stacked on #4284 — tests only
This PR contains **only the tests** that guard the production fixes in
#4284 (its base). Review #4284 first; this branch adds no production
code.
## What
Caller-driven replica-lag and idempotency guards for every fixed site:
- Each guard **drives the real exported caller** (route loader/action,
presenter `.call()`, service, or engine method) against a **real
Postgres** with the owning replica frozen via the shared
`laggingReplica` testcontainer primitive — never a store-seam
reimplementation.
- For a **fixed** site the guard goes **RED when the production change
is reverted**; for a **tolerated read-view** site it's a caller-driven
**GREEN** proof the miss self-heals (returns null/empty, no mutation,
row live on primary).
- The **global-scope idempotency** guard drives the real dedup + claim
path through a **real `MollifierBuffer` over a Redis testcontainer**
(real SETNX/poll/publish), and covers the cross-DB **andWait** waitpoint
wiring and the **expired/failed clear-and-recreate** reacquire cases.
Run with `vitest --no-file-parallelism` (testcontainers). Verified
GREEN, and revert→RED verified per fixed site.
## What & why
Two related correctness fixes for the run-ops DB split. Under the split,
run-store reads can route to a **lagging read replica**; a just-written
run/waitpoint/batch can then be missed, causing a wrong decision.
**1. Read-your-writes → owning primary.** Surfaced first as an
intermittent `wait.until({ idempotencyKey })` re-wait on retry. Auditing
the run-store read surface found the same class at sibling sites (some
gating mutations or returning spurious 404s, others
tolerable/self-healing). Reads that must observe their own writes now
route to the owning **primary**
(`findRun`/`findWaitpoint`/`findBatchTaskRunByFriendlyId` →
`*OnPrimary`, a primary re-read on a miss, or a retryable 404 where the
SDK polls). Read-view reads stay on the replica. All additive — the
happy path is unchanged.
**2. Global-scope idempotency across the split.** A `global`-scope key
carries no per-run salt, so the same `(env, task, key)` triggered
concurrently from parents resident on **different** run-ops DBs could
dedup-miss on each DB and create a duplicate (the per-DB unique index
can't enforce cross-DB uniqueness). Such triggers (global scope, or
scope-absent, while split is active) are serialized through the existing
Redis idempotency claim, the loser resolves the winner by id across both
DBs, and the claim is reacquired on the expired/failed
clear-and-recreate path. `run`/`attempt` scope embed the run id and
never contend.
## Stacked for review
This is the **base** of a 2-PR stack, split so review is easier:
- **This PR** — production code only (34 files).
- **Stacked tests PR →
https://github.com/triggerdotdev/trigger.dev/pull/4285** — the
caller-driven guards (55 test files) on top of this branch.
## Validation
Local run-ops split, **both 2-DB and 3-DB**, fresh boot on this branch:
SDK canary 64/71 (only the known concurrency/input-streams/s3 failures),
quarantine sweep **0 unexpected** (340 pass / 16 known / 4 local) in
each topology, dashboard e2e 0 failed. No product regressions.
## What
- Runs `apps/webapp/scripts/runOpsLegacyGuard.ts --check` as its own PR
job (`runops-guard`), so code that reaches a run-graph table through the
control-plane Prisma client instead of the RunStore fails the build.
- Adds a `trigger-runops` oxlint plugin with two fast, in-editor rules
scoped to `apps/webapp/app`: one for direct `prisma.taskRun`-style
access, one for a control-plane client wired into a read-through slot.
These are the cheap fence; the guard is the type-aware gate.
- Fixes `CancelTaskRunService.callV1`: historical V1 runs are
legacy-resident, so its two finalize writes now go through
`runOpsLegacyPrisma` instead of the control-plane client (they'd miss
the row once legacy is a separate database).
- Regenerates the guard baseline, which had drifted stale (it referenced
files deleted in an earlier PR).
## Why
The guard existed but ran nowhere, so its baseline rotted and a real
residency gap (the V1 cancel writes) sat undetected. Wiring it into CI
turns it into a ratchet against new control-plane run-graph access.
## Verification
Local, against a clean regen: `oxfmt --check`, `oxlint .`, `guard
--check`, and `typecheck --filter webapp` all pass. Remaining baseline
entries are 4 batch-results router reads through type-opaque `as
PrismaReplicaClient` casts (correct at runtime, accepted) + 2 sanctioned
legacy annotations.
## Summary
Correctness and performance fixes for deployments that split run data
across more than one database. Single-database / self-hosted deployments
are unaffected (they collapse to a single read/write path).
- **Batches list (dashboard):** for some organizations the Batches list
could hide older batches or show them out of order. It now orders and
paginates by creation time (with the id as a stable tiebreak), so every
batch appears exactly once, newest first. The pagination cursor format
changes; older in-flight cursors simply restart from the first page.
- **Reads:** waitpoint and snapshot lookups that are keyed by a single
run now read only the database that holds that run instead of querying
both, removing redundant queries on hot paths (unblock, snapshot reads).
- **Writes:** environment-scoped writes with no owning run (standalone
wait tokens, waitpoint tags, idempotency-key resets) now land in the
same database as that environment's runs, rather than defaulting to the
other one. An idempotency-key reset also falls back to the other
database when it matches nothing, so a reset still clears the key
wherever the run actually lives.
## Notes
Verified end-to-end against multi-database setups: run-keyed reads and
env-scoped writes land on the correct database with no cross-database
writes, and the batches list surfaces every batch in creation order. New
tests cover the batches ordering/reachability and the write-residency
routing.
## Problem
The webapp's HTML references content-hashed `/build` assets, and each
running
instance contains exactly one build and returns 404 for asset hashes it
doesn't
have. During a rolling deploy a client can hold HTML from one build
while a
request for one of its assets is served by an instance on a different
build →
missing styles or a failed chunk load.
## What this does
On a `/build` stylesheet/script/chunk load failure, the client does a
**bounded
full-document reload** (at most 2 per 5 minutes, tracked in
`sessionStorage`) so
the page reloads onto a single consistent build. That's the whole
mechanism — no
polling, no `fetch` interception, no blocking overlay, no form
snapshotting.
- `apps/webapp/app/components/StaleAssetRecovery.tsx` — authored as a
typed,
lint-checked function and serialized to an inline script via
`.toString()` (so
the logic is real, reviewable code, not an opaque string), injected
before
`<Links />`, production only.
- Detection: capture-phase `error` listener for
`<link>`/`<script>`/modulepreload
failures under `/build/`, plus an `unhandledrejection` guard for
dynamic-import
failures.
- Guards: once-per-page re-entrancy guard, the bounded reload budget,
and a
`navigator.onLine` check so it never reloads into an offline error page.
- Unit tests in `StaleAssetRecovery.test.ts`.
## Relationship to #4260
Replaces the recovery introduced in #4260 (reverted in #4280) with a
much
smaller, reload-only approach — the previous version intercepted `fetch`
and
could turn a data request into a navigation, and showed a full-screen
overlay on
any asset error; this drops both.
## `/build-version` compatibility shim
`apps/webapp/server.ts` adds a tiny `GET /build-version` endpoint (build
id only,
`no-store`). A previously-deployed client build polls it after an asset
failure
and reloads once it sees a newer build, so those older tabs recover in
one reload
instead of getting stuck. Temporary — safe to remove once older clients
have
cycled out. It deliberately does **not** re-add an `X-Build-Id` response
header.
## Also
Restores the `.server-changes` writing guidance in
`.claude/rules/server-apps.md`
(reverted alongside #4260).
## Self-hosting note
Recovery is most reliable when your load balancer keeps a client on one
instance
for the duration of a deploy (short session stickiness) — the reload
then lands
on a consistent build in one hop.
Removes the deprecated realtime stream write action kept for retired v3
task clients. Supported clients use the targeted stream write routes,
while the existing stream read loader remains unchanged.
Standard `git revert` of #4260.
Its client-side stale-asset recovery is net-negative during normal
deploys:
- The `fetch` interception treats any `?_data=` request (Remix loader
**and** action traffic) as a navigation and, on a build-id mismatch,
`location.assign`es the tab to the fetched URL — an open dashboard tab
can be hard-navigated to a raw data URL during a rolling deploy, losing
unsaved input.
- Any transient `/build` asset error (a network blip, an extension, an
unrelated failed dynamic import) blanks the page behind a full-screen
overlay for ~60s before offering a manual reload.
- It serialized form field values to `sessionStorage` to restore them
across the reload.
This returns the webapp to the pre-#4260 baseline as a fast, low-risk
step.
Follow-ups (separate PRs):
- a minimal reload-only recovery to replace this,
- restore the unrelated `.claude/rules/server-apps.md` docs tidy-up from
#4260 (via cherry-pick),
- a load-balancer stickiness change addressing the root cause.
## Summary
Adds experimental Node.js 24 and 26 task runtimes through the
`experimental-node-24` and `experimental-node-26` config values.
Existing runtime defaults and the `node`, `node-22`, and `bun` behavior
remain unchanged. The unprefixed `node-24` and `node-26` config values
remain unavailable until the runtimes are ready for general use.
## Design
Experimental config values normalize to canonical runtime identifiers
before build manifests are created, keeping deployment metadata and
execution behavior consistent. Kubernetes task pods also use the
runtime-default seccomp profile so modern Node.js versions fall back
from io_uring to checkpoint-compatible system calls.
## Summary
`batchTrigger` requests that set a per-item `idempotencyKey` failed with
a 500 when the run-store is split across databases: the per-item
idempotency lookup errored before any run was created. Batches without
per-item keys, single `trigger` idempotency, and batch-level
(`idempotency-key` header) idempotency were unaffected.
## Root cause
`findRunsByIdempotencyKeys` built its `UNION ALL` of per-key
point-lookups with `@trigger.dev/database`'s `Prisma.sql` /
`Prisma.join`, then executed it on whichever store client it was handed.
On the dedicated run-ops store that client is a *separate* generated
Prisma client, and a `Sql` object from a different generated client is
not recognized: the bare `$queryRaw(Prisma.join(...))` form dropped the
query text entirely (`Argument \`query\` is missing`). The
tagged-template form is no better here: joining nested `Prisma.sql`
fragments across the two clients mis-numbers the bound parameters
(`syntax error at or near "$1"`).
## Fix
Build the lookup as a plain parameterized string and run it via
`$queryRawUnsafe` with positional placeholders and bound values, so it
no longer depends on which generated client executes it. The query text
contains only static SQL and integer placeholders; every value
(`runtimeEnvironmentId`, `taskIdentifier`, each key) is bound, so it is
not a raw-interpolation site. Same per-key point-lookup shape as before,
no change on the single-client path.
Verified end-to-end against a bundled build with the run-store split
enabled: before the fix, `batchTrigger` with a per-item key 500s; after,
it returns the runs and dedups correctly across fresh, repeat, and mixed
batches.
The signup → Attio sync (`attio.server.ts` `#assert`) logged every
non-2xx response at `error` level and threw the same way regardless of
status. Transient upstream failures (5xx/429) are retried by the common
worker and self-heal, so treating them as errors created false alerts
for something that isn't actually a bug.
Now `#assert` splits the two cases:
- **5xx / 429** — Logged at `warn` and thrown with `logLevel: "warn"`,
so they continue to be retried but don't raise error-level alerts. This
reuses the same pattern the worker already honors
(`directorySyncEffects`).
- **4xx** — Unchanged: logged at `error` and thrown, so genuine
integration bugs (schema, permissions, auth, etc.) remain visible.
There is no behavior change to retries or the signup flow. This is a
server-only change.
---------
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
## Summary
Adds optional configuration to send ClickHouse read traffic to a
separate instance (for example a read replica) while writes stay on the
primary `CLICKHOUSE_URL`. This lets operators offload read load (runs
list, traces, logs, queries) from the cluster that handles inserts.
Fully backwards compatible: with nothing new set, every client resolves
to `CLICKHOUSE_URL` exactly as before.
## What it adds
- `CLICKHOUSE_READER_URL` (optional): a single reader endpoint that the
read-only clients fall back to. Read clients resolve `<own URL> ??
CLICKHOUSE_READER_URL ?? CLICKHOUSE_URL`. The task-events client (which
both inserts events and reads traces, spans, and logs) is built as a
reader/writer pair so queries use the reader while inserts stay on
`CLICKHOUSE_URL`.
- `RUNS_LIST_CLICKHOUSE_URL` (optional): a dedicated client for the runs
list (dashboard list, runs list API, live reload, child-status counts),
so the highest-traffic read path can target its own instance.
## Safety
Only read-only clients fall back to the reader: logs, query, admin, runs
list, the pending-version lookup, and the realtime run-id resolver. The
query page is constrained to read-only (the TSQL parser rejects anything
that is not a `SELECT`, and a `readonly` setting is applied). The
task-events client routes inserts to the writer and queries to the
reader per method, so a write can never reach the reader. Pure-write
clients (event inserts, replication) always use `CLICKHOUSE_URL`.
Note: this PR targets a baseline branch rather than `main` so the diff
stays scoped to the read-replica changes. It will be retargeted to
`main` before merge.
---------
Co-authored-by: Eric Allam <eallam@icloud.com>
## Summary
Updates the internal development, CI, and runtime-image Node version to
24.18.0. SDK compatibility coverage continues to include Node 20, 22,
24, and 26.
The Node type definitions and the package-manager lockfiles now resolve
against Node 24 types.
## Summary
Adds a set of PAT-authenticated management API endpoints so orgs,
projects, members/invites, environment variables, and a few
project/environment settings can be managed programmatically (scripting,
automation) rather than only through the dashboard. Each route is a thin
wrapper over the **existing** service the dashboard already uses, with
the same authorization applied at the route layer - no new business
logic.
## Endpoints
**Organizations**
- `POST /api/v1/orgs` - create an org (`createOrganization`)
- `PATCH /api/v1/orgs/:orgParam` - rename (title)
- `DELETE /api/v1/orgs/:orgParam` - soft-delete
(`DeleteOrganizationService`; keeps the active-subscription guard)
**Members & invites**
- `GET /api/v1/orgs/:orgParam/members` - list members + pending invites
- `DELETE /api/v1/orgs/:orgParam/members/:memberId` - remove a member
(last-member guarded)
- `POST /api/v1/orgs/:orgParam/invites` - invite by email
(`inviteMembers`, sends the invite email)
- `DELETE /api/v1/orgs/:orgParam/invites/:inviteId` - revoke an invite
**Projects**
- `PATCH /api/v1/projects/:projectRef` - rename
(`ProjectSettingsService`)
- `DELETE /api/v1/projects/:projectRef` - soft-delete
(`DeleteProjectService`)
- `PUT /api/v1/projects/:projectRef/default-region` - set the default
region by worker-group name (`SetDefaultRegionService`)
- project GET/list now return `defaultRegion` (worker-group name, or
null when unset)
**Environments**
- `POST /api/v1/projects/:projectRef/:env/pause` and `/resume`
(`PauseEnvironmentService`)
- `POST /api/v1/projects/:projectRef/:env/regenerate-api-key` - rotate
the env secret key (`regenerateApiKey`, RBAC `write:apiKeys`)
- env var create now accepts an optional `isSecret` flag
## Auth & authorization
- All routes authenticate with a **Personal Access Token**
(`Authorization: Bearer tr_pat_...`).
- Org/project routes are built on the PAT route builders in
`apiBuilder.server.ts`: `createLoaderPATApiRoute` (already existed) and
**`createActionPATApiRoute`** (added here - the loader builder had no
mutation counterpart). The builder runs auth, resolves the org/project
role-floor via `context`, and enforces a declarative `authorization`
block using the same RBAC actions the dashboard applies
(`manage:organization` / `read:members` / `manage:members` /
`manage:project`). Handlers keep a membership-scoped query as the floor,
so a non-member gets a 404. This also gives these routes `tenantContext`
user attribution (Sentry) and `ServiceValidationError`-to-status mapping
for free.
- **Membership floor (important).** The OSS RBAC fallback grants a
permissive ability, so `ability.can(...)` can't reject a non-member on
self-hosted. Every handler therefore resolves the target scoped to the
caller's membership (`members: { some: { userId } }`) → 404 for
non-members. `authorization` is the *role* gate; this is the *tenant*
gate. `resolveOrganizationForApiUser`
(`organizationApiAccess.server.ts`) is the org-tier version of the
existing `findProjectByRef` - org-addressed PAT routes are new, so no
such helper existed before.
- Env-tier routes reuse the existing `authorizePatEnvironmentAccess`
(`write:apiKeys`).
### What `createActionPATApiRoute` gives you
A route is pure declaration - the builder handles auth, RBAC,
validation, tracing, and error mapping:
```ts
export const action = createActionPATApiRoute(
{
method: "PUT", // one verb, or ["PATCH", "DELETE"] for multi-verb routes
params: ParamsSchema,
body: SetDefaultRegionRequestBody, // zod-validated
context: async ({ projectRef }) => { // resolve the org for the RBAC role-floor
const project = await prisma.project.findFirst({
where: { externalRef: projectRef, deletedAt: null },
select: { organizationId: true },
});
return project ? { organizationId: project.organizationId } : {};
},
authorization: { action: "manage", resource: () => ({ type: "project" }) },
},
async ({ params, body, authentication, ability }) => {
// auth + authz already enforced. Just do the work.
// `throw new ServiceValidationError("Region not found", 400)` → mapped to that status.
return json({ ok: true });
}
);
```
Handled for you, so handlers stay thin:
- **Method allowlist** - `method` accepts a verb or an array; any other
verb → `405` with an `Allow` header, *before* auth runs:
```ts
const allowedMethods = method ? (Array.isArray(method) ? method :
[method]) : undefined;
if (allowedMethods && !(allowedMethods as
string[]).includes(request.method.toUpperCase())) {
return json({ error: "Method not allowed" }, { status: 405, headers: {
Allow: allowedMethods.join(", ") } });
}
```
- **PAT / user-actor auth** in a single roundtrip → `401` on
missing/invalid/revoked token.
- **RBAC** - `context` computes the caller's role-floor for the target
org/project; `authorization` gates it → `403` with a structured error
body.
- **Sentry attribution** - `tenantContext.enrich({ userId })` so events
from the handler carry the acting user.
- **Typed errors** - a thrown `ServiceValidationError` is mapped to its
`.status` (default 400); anything else → `500`, and expected boundary
errors are logged as `warn` (kept out of Sentry).
- **Validation** - params / query / headers / body are all zod-checked →
`400` with details.
## Notes for reviewers
- Everything wraps an existing service; the intent is API parity for
things that are currently dashboard-only, not new behaviour.
- `createActionPATApiRoute` is new shared infra (the PAT + RBAC mutation
builder that didn't exist). It's self-contained - the loader builder and
existing routes are untouched.
- `@trigger.dev/core` gets one additive field (`defaultRegion` on the
project response, optional/nullable for client-server version skew) -
changeset included, patch.
- `removeTeamMember`'s last-member guard is now atomic (Serializable
transaction via the `$transaction` helper, with retry), so the dashboard
and API both get it server-side. Added a `## Transactions` rule to
`apps/webapp/CLAUDE.md` (always use the `$transaction` helper);
migrating the remaining direct usages is tracked in TRI-11698.
## Open questions
- ~~Is PAT the right auth (vs OAT for automation)?~~ **Resolved: PAT.**
Organization Access Tokens are currently internal-only (used by the
image builder) and not user-accessible, so they can't back this yet.
- Should any of these be gated behind a flag or scope?
- Naming/shape of the routes.
## Summary
Batch triggers that use per-item idempotency keys could take seconds
instead of milliseconds when the target task had a large run history.
This keeps the idempotency lookup fast regardless of how many runs a
task has accumulated.
## Root cause
The batch path checks which items already have runs by looking up their
idempotency keys with a single `WHERE runtimeEnvironmentId = ? AND
taskIdentifier = ? AND idempotencyKey IN (...)` query. On a very large
`TaskRun` table Postgres underestimates the row count of a specific
`(environment, task)` pair, so once the `IN` list grows past a handful
of keys it stops doing per-key index probes and instead scans every run
for that `(environment, task)` and filters the keys in memory. The cost
is then flat and large regardless of how many keys are being checked,
and a routine `ANALYZE` does not correct the estimate at that table
size.
## Fix
Look each idempotency key up on its own, batched into a `UNION ALL` of
point lookups (chunked, run with bounded concurrency). Each branch is an
equality on all three columns of the unique index, so the planner can
only do a per-key index probe and can never fall back to the range scan.
Same results, same columns, confined to the batch trigger path.
### Problem
Webapp HTML references content-hashed /build assets, and each Docker
image contains exactly one build with a hard 404 for unknown hashes.
During a rolling deploy, a client holding HTML from the old build may
request old asset hashes from a replica running the new image, causing
missing styles or failed chunk loads.
The page should recover automatically once a compatible build becomes
available, without reload loops or unnecessary interruptions during
normal deployments.
### What changed
- Build changes alone do nothing — no polling, no automatic reloads.
- If a CSS or JavaScript asset fails to load, a recovery overlay is
shown immediately.
- While the server still reports the same build, the client polls for a
newer build using exponential backoff (up to ~60s). As soon as a newer
build is detected, the page reloads automatically.
- If no newer build appears within the timeout, recovery falls back to a
manual Reload action.
- If recovery still fails after the automatic reload, the client stops
retrying and displays a final recovery screen instead of entering a
reload loop.
- Recovery preserves form values and scroll position across the
automatic reload.
Updates the ClickHouse chat agent example page to match the upgraded
example (triggerdotdev/examples#124), which is now a fullstack
generative-UI chat app rather than an agent-only project.
## What changed
- **Overview / tech stack / features** rewritten: Next.js chat app
(`useChat` + `useTriggerChatTransport`, no API route), a
`renderVisualization` tool taking json-render specs rendered with
`@json-render/shadcn` + shadcn charts (Recharts) + mapcn point maps, and
a shared catalog that generates both the system-prompt component
reference and tool-call validation.
- **The agent section** now shows the versioned [AI
Prompt](https://trigger.dev/docs/ai/prompts) pattern (`prompts.define()`
+ `chat.prompt.set()` + `chat.toStreamTextOptions({ registry })`), with
a warning that `experimental_telemetry` comes from the stored prompt —
the docs previously showed a static `system:` string, which silently
ships no LLM observability.
- **New sections** for the shared catalog, the `renderVisualization`
tool, the Next.js chat UI and registry.
- **Relevant code links** updated to the new `src/` layout.
- **Learn more** cards now include Frontend and AI Prompts.
Note: merge after triggerdotdev/examples#124 lands, so the GitHub file
links resolve.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
## Summary
When the run-ops split is enabled, the legacy run-ops database client
was always constructed as its own connection pool, even when it points
at the same database as the primary (control-plane) client. On setups
where those two DSNs resolve to the same physical database, this opened
a second, redundant pool and doubled the number of connections used
against that database. This change makes the legacy client reuse the
primary client's pool whenever their DSNs point at the same database,
and only open a separate pool when they genuinely differ.
## Fix
A small `sameDatabaseTarget` comparison (host, port, database name,
user) decides whether the legacy DSN points at the same database as the
primary. When it does, the legacy handle reuses the primary client by
reference, so no second pool is opened. When the DSNs diverge, the
legacy client is built independently as before, so the split still works
once the databases are actually separate.
Two smaller changes ride along:
- An optional per-pool limit for the run-ops read replica, which
connects unpooled and so draws raw backend connections; unset, it falls
back to the existing default and behaviour is unchanged.
- A startup warning about a missing legacy replica URL is now suppressed
when the legacy client shares the primary pool, where it would be
misleading.
## Verification
Booted the webapp end-to-end in three modes and confirmed the pools
opened as expected via the client's own startup logs and live backend
connection counts: split off (single pool), split on with a shared
database (legacy reuses the primary pool, no doubling), and split on
with separate databases (legacy opens its own pool).
## Summary
2 new features, 11 improvements, 5 bug fixes.
## Breaking changes
- Trigger.dev v3 is no longer supported. For self-hosted deployments,
4.5.0 is the last version we officially support for running v3; stay on
4.5.0 or upgrade to v4. v3 triggers, batch triggers, reschedules, and
deploys now return a clear upgrade message instead of running.
([#4236](https://github.com/triggerdotdev/trigger.dev/pull/4236))
## Improvements
- You can now mark environment variables synced via the `syncEnvVars`
build extension as secrets. Return `{ name, value, isSecret: true }`
from your callback and those variables are stored redacted in the
dashboard, just like manually created secret env vars.
([#4203](https://github.com/triggerdotdev/trigger.dev/pull/4203))
- Remove the legacy `--mcp` and `--mcp-port` options from the `dev`
command. Run the dedicated `trigger mcp` command to start the
Trigger.dev MCP server.
([#4246](https://github.com/triggerdotdev/trigger.dev/pull/4246))
- Removed the unused `ResourceMonitor` export from
`@trigger.dev/core/v3/serverOnly`. It was a server-side logging helper
with no remaining consumers.
([#4244](https://github.com/triggerdotdev/trigger.dev/pull/4244))
- Removed the unused `@trigger.dev/core/v3/zodNamespace` export and the
legacy v3 socket message schemas. These were only used by the
now-retired v3 engine and have no v4 consumers.
([#4236](https://github.com/triggerdotdev/trigger.dev/pull/4236))
## Bug fixes
- Fix a `chat.agent` message-loss race where sending a message right
after an action (such as an undo) could drop the follow-up's response
from the UI until a refresh.
([#4234](https://github.com/triggerdotdev/trigger.dev/pull/4234))
## Server changes
These changes affect the self-hosted Docker image and Trigger.dev Cloud:
- Added `EVENT_REPOSITORY_POSTGRES_WRITES_DISABLED` to skip all
PostgreSQL task-event writes for deployments that store task events in
ClickHouse. Leave it off unless `EVENT_REPOSITORY_DEFAULT_STORE` is
`clickhouse_v2`, otherwise task events are lost.
([#4242](https://github.com/triggerdotdev/trigger.dev/pull/4242))
- Promo credits: a /promo signup landing page, redeeming a promo code
when a new org selects a plan, and showing remaining credits on the
usage page.
([#4138](https://github.com/triggerdotdev/trigger.dev/pull/4138))
- Speed up retrieving a background worker by version. The endpoint no
longer runs a slow lookup that scanned the full task table for large
deployments; it now reuses data it already loads, so the response is the
same but returns much faster.
([#4245](https://github.com/triggerdotdev/trigger.dev/pull/4245))
- Clearer login error when an email address is blocked by the
WHITELISTED_EMAILS setting: the message now explains the address isn't
allowed on this instance instead of the ambiguous "This email is
unauthorized".
([#4220](https://github.com/triggerdotdev/trigger.dev/pull/4220))
- Make the native build server the default in project build settings.
It's now opt-out, stored as a new `disableNativeBuildServer` key. Also
clarifies in the UI that build settings apply to GitHub-triggered and
native build server deployments.
([#3980](https://github.com/triggerdotdev/trigger.dev/pull/3980))
- Optionally process high-volume telemetry ingestion in parallel for
higher throughput under heavy load by setting
`OTEL_TRANSFORM_WORKER_POOL_ENABLED=1`. Off by default.
([#4232](https://github.com/triggerdotdev/trigger.dev/pull/4232))
- Add a `REALTIME_BACKEND_DEFAULT` env var to choose the default
realtime backend (`electric`, `native`, or `shadow`) for environments
whose org has no per-org override. Defaults to `electric`, so existing
behavior is unchanged.
([#4231](https://github.com/triggerdotdev/trigger.dev/pull/4231))
- Clarified on the Regions page that a region only affects where your
runs execute, not where your data is stored. This shows as a tooltip on
the Location column and in the confirmation dialog when you change your
default region.
([#4226](https://github.com/triggerdotdev/trigger.dev/pull/4226))
- Improved the reliability of how run data is read and written.
([#4237](https://github.com/triggerdotdev/trigger.dev/pull/4237))
- Fixed stale login errors: an error from a previous login attempt (for
example a rejected email address) no longer keeps reappearing on the
login page and no longer makes later, successful attempts look like they
failed.
([#4220](https://github.com/triggerdotdev/trigger.dev/pull/4220))
- The Errors page now shows better details for each error. Errors that
don't carry a message — such as errors thrown without a message, or
values thrown that aren't `Error` objects — get a meaningful title
instead of all reading "Unknown error", and are grouped by their name
(or value) rather than collapsed into a single group. The error type now
shows the actual error name, and stack traces now appear where
previously they were missing.
([#4225](https://github.com/triggerdotdev/trigger.dev/pull/4225))
- Return a clear client error when SSO form submissions use an
unsupported content type
([#4238](https://github.com/triggerdotdev/trigger.dev/pull/4238))
- Query page: extracting fields from a run's output with JSON functions
(such as JSONExtractString or JSONExtractInt) no longer fails with an
"illegal type: JSON" error.
([#4221](https://github.com/triggerdotdev/trigger.dev/pull/4221))
<details>
<summary>Raw changeset output</summary>
# Releases
## @trigger.dev/build@4.5.4
### Patch Changes
- You can now mark environment variables synced via the `syncEnvVars`
build extension as secrets. Return `{ name, value, isSecret: true }`
from your callback and those variables are stored redacted in the
dashboard, just like manually created secret env vars.
([#4203](https://github.com/triggerdotdev/trigger.dev/pull/4203))
- Updated dependencies:
- `@trigger.dev/core@4.5.4`
## trigger.dev@4.5.4
### Patch Changes
- Remove the legacy `--mcp` and `--mcp-port` options from the `dev`
command. Run the dedicated `trigger mcp` command to start the
Trigger.dev MCP server.
([#4246](https://github.com/triggerdotdev/trigger.dev/pull/4246))
- Updated dependencies:
- `@trigger.dev/core@4.5.4`
- `@trigger.dev/build@4.5.4`
- `@trigger.dev/schema-to-json@4.5.4`
## @trigger.dev/core@4.5.4
### Patch Changes
- Removed the unused `ResourceMonitor` export from
`@trigger.dev/core/v3/serverOnly`. It was a server-side logging helper
with no remaining consumers.
([#4244](https://github.com/triggerdotdev/trigger.dev/pull/4244))
- Removed the unused `@trigger.dev/core/v3/zodNamespace` export and the
legacy v3 socket message schemas. These were only used by the
now-retired v3 engine and have no v4 consumers.
([#4236](https://github.com/triggerdotdev/trigger.dev/pull/4236))
## @trigger.dev/python@4.5.4
### Patch Changes
- Updated dependencies:
- `@trigger.dev/sdk@4.5.4`
- `@trigger.dev/core@4.5.4`
- `@trigger.dev/build@4.5.4`
## @trigger.dev/react-hooks@4.5.4
### Patch Changes
- Updated dependencies:
- `@trigger.dev/core@4.5.4`
## @trigger.dev/redis-worker@4.5.4
### Patch Changes
- Updated dependencies:
- `@trigger.dev/core@4.5.4`
## @trigger.dev/rsc@4.5.4
### Patch Changes
- Updated dependencies:
- `@trigger.dev/core@4.5.4`
## @trigger.dev/schema-to-json@4.5.4
### Patch Changes
- Updated dependencies:
- `@trigger.dev/core@4.5.4`
## @trigger.dev/sdk@4.5.4
### Patch Changes
- Fix a `chat.agent` message-loss race where sending a message right
after an action (such as an undo) could drop the follow-up's response
from the UI until a refresh.
([#4234](https://github.com/triggerdotdev/trigger.dev/pull/4234))
- Updated dependencies:
- `@trigger.dev/core@4.5.4`
</details>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
The dashboard was redesigned and two pages moved, but the docs still
described the old sidebar:
- **Schedules** no longer has its own sidebar page — schedules are
managed from the **Tasks** page (open a scheduled task to create / view
/ edit / enable-disable / delete them).
- The standalone list-based **Test** page is deprecated — you test a
task from its own **Test** button now.
## Changes
- `tasks/scheduled.mdx`: rewrote the "attaching schedules" and "testing
schedules" sections for the Tasks-based flow, added a "managing
schedules in the dashboard" section, and added explicit callouts noting
both pages moved (so readers — and search — aren't pointed at a page
that no longer exists). Re-shot the four schedule screenshots and fixed
a mislabeled alt text.
- `run-tests.mdx`, `snippets/step-run-test.mdx`,
`guides/examples/sentry-error-tracking.mdx`: replaced "select the Test
page in the sidebar" with the task-first flow plus a callout, and
refreshed `test-dashboard.png`.
TRI-11939
The `ResourceMonitor` server-side logging helper is no longer used. It
periodically logged the webapp process own memory, disk, and CPU usage
behind the `RESOURCE_MONITOR_ENABLED` flag (off by default), and was
also exported from `@trigger.dev/core/v3/serverOnly` with no other
consumers.
This removes the helper, its `@trigger.dev/core` export, the webapp
wiring, and the `RESOURCE_MONITOR_ENABLED` env var. The supervisor has
its own unrelated `ResourceMonitor` class, which is left untouched.
## Summary
Adds `EVENT_REPOSITORY_POSTGRES_WRITES_DISABLED` (default off), which
makes the task-event store skip all PostgreSQL `TaskEvent` writes. It's
for deployments that store task events in ClickHouse
(`EVENT_REPOSITORY_DEFAULT_STORE=clickhouse_v2`) and no longer want the
PostgreSQL copy.
## How it works
The guard sits at the single postgres write boundary,
`TaskEventStore.create` / `createMany`, so it covers every write path
(OTLP ingestion and run-lifecycle events) with one check. Reads are
untouched (`findMany` / trace queries / streaming), so existing
PostgreSQL events remain readable.
Leave it off unless the default store is `clickhouse_v2`, otherwise task
events for any run still routed to PostgreSQL would be dropped.
## What
The `GET
/api/v1/projects/:projectRef/background-workers/:envSlug/:version`
endpoint loaded each file's tasks through the nested `files.tasks`
relation. Prisma resolves that as a separate query:
```sql
SELECT id, slug, "fileId" FROM "BackgroundWorkerTask" WHERE "fileId" IN (...)
```
`BackgroundWorkerTask.fileId` is not indexed — the FK constraint exists,
but Postgres does not auto-create an index for foreign keys — so on a
large table this can only run as a sequential scan, which gets
progressively slower as the table grows and was observed taking minutes
per call in production.
The loader already loads every task for the worker via `tasks: true`,
which uses the indexed `workerId` relation, and those rows already
include `fileId`. This PR groups task slugs by `fileId` in memory from
that already-loaded data and drops the `files.tasks` include entirely.
## Behavior change (latent bug fix)
The response shape is unchanged, but there is a semantic correction for
**source files reused across worker versions** (files are de-duplicated
by `@@unique([projectId, contentHash])`, so one file row can be linked
to many workers).
- **Before:** `file.tasks` came from the `BackgroundWorkerFile.tasks`
relation, i.e. *every* `BackgroundWorkerTask` with that `fileId` —
across all workers sharing the file. So a worker's manifest could list
tasks it doesn't actually have.
- **After:** `file.tasks` is grouped from the queried worker's own
tasks, so it reflects only that worker version's tasks.
Verified on a local DB: 460 files are referenced by tasks from more than
one worker; of 6819 (worker, file) pairs, 6 differ — all one file where
the old union leaked a task slug (`cancellation-test`) into worker
versions that never had it. The new per-worker behavior is the correct
one for a worker-version manifest. (Thanks to the automated review for
flagging this.)
## Analysis
Captured the exact SQL before/after by instrumenting Prisma against real
data (a worker with 62 files):
- **Before:** 5 statements, including the `WHERE "fileId" IN (...)`
scan.
- **After:** 4 statements; the `fileId` query is gone and the other four
are identical.
EXPLAIN of the two access paths:
```
Before WHERE "fileId" IN (...)
Seq Scan on "BackgroundWorkerTask"
Filter: ("fileId" = ANY (...)) -- reads the whole table, scales with table size
After WHERE "workerId" IN (...)
Index Scan using "BackgroundWorkerTask_workerId_slug_key"
Index Cond: ("workerId" = ...) -- bounded by matching rows, scale-independent
```
No new index is required: the `workerId` access path is already covered
by the existing `BackgroundWorkerTask_workerId_slug_key` unique index.
## Testing
- `pnpm run typecheck --filter webapp` passes.
- Query capture + EXPLAIN performed against a local database seeded with
real worker/file/task data.
Routine dependency maintenance.
- Pin a few high-fanout transitive deps to current patched versions via
`pnpm.overrides`: `form-data`, `ws`, `undici`, `hono`. Lockfile-only (no
published-package dependency changes); net shrinks via dedup.
- Upgrade `nodemailer` 8 → 9 in `internal-packages/emails` (private
package). The SES transport already uses SESv2 and the
`createTransport`/`sendMail` API is unchanged, so no code changes were
needed. `@types/nodemailer` stays at 8 (no 9.x published yet; types are
compatible).
Verified locally: `pnpm i` clean; `pnpm run typecheck --filter emails`
and `--filter webapp` both pass.
## Summary
Run-graph data (runs, batches, waitpoints, and their related tables) can
now live in a database separate from the control plane, with every read
and write routed to the correct database by each run's residency. This
makes reading and writing run data more reliable once the two are split,
and is a no-op for single-database installs.
## Design
- Run-graph table access goes through the run-store router, which
selects the legacy or the new run-ops store per run instead of assuming
one shared client.
- The legacy run-ops client is now independently pointable, so legacy
run data can be served from its own database (and replica) rather than
the control-plane connection.
- Run-graph writes go straight to the run-graph database instead of
being forwarded through the control plane, and replication targets are
split so runs in the new database still replicate to analytics without
under-counting.
- Read-through slots refuse the control-plane client, so a missing
residency fails loudly instead of silently reading the wrong database.
- Migration `20260710120000_drop_remaining_run_graph_seam_foreign_keys`
drops the foreign keys that still crossed the run-graph / control-plane
seam, which is what lets the two live in separate databases.
The split stays off unless explicitly enabled and the two databases are
confirmed physically distinct; startup fails closed otherwise.
Verified by running the full dashboard end-to-end suite against both a
single-database configuration and a three-database configuration
(control plane, the new database, and a physically separate legacy
database), with runs on both residencies. No misrouted reads in either
configuration.
## Summary
Refreshes the docs for the v3 sunset: v3 (SDK v3) is end of life, and
4.5.0 is the last version we officially support for running v3.
- The self-hosting overview, plus the docker and kubernetes
version-locking sections, now tell self-hosters on v3 to stay on 4.5.0
or migrate to v4. 4.5.1 and later reject v3 triggers and deploys with an
upgrade message.
- The migration guide's deprecation notice was still written in the
future tense (with dates that have since passed); it now describes v3 as
retired and adds the self-hosted 4.5.0 cutoff. This is the page the
server's upgrade message links to.
- Fixes a stale "v3 project" reference in the CLI overview.
The Mintlify preview will render the callouts for a visual check.
## Summary
v3 (the engine that ran the SDK v3 era, internally
`RunEngineVersion.V1`) is end-of-life. Following the removal of the v3
execution apps
([#4194](https://github.com/triggerdotdev/trigger.dev/pull/4194)) and
the legacy dev websocket
([#4198](https://github.com/triggerdotdev/trigger.dev/pull/4198)), this
removes the remaining v3 execution stack from the server.
Clients still on v3 (an old SDK or CLI that has not upgraded) keep
getting a clear "upgrade to v4" response. Triggers, batch triggers,
reschedules, and deploys that resolve to v3 are rejected with a graceful
4xx pointing at the migration guide, never a 5xx, so a stale client
cannot affect server health. Self-hosted instances still running v3
should stay on the 4.5.x release line until they migrate.
## What is removed
- The MarQS queue and its shared/dev queue consumers.
- The v3 socket.io namespaces (coordinator, provider, shared-queue) and
the v3 run lifecycle services (attempt, checkpoint, and batch-resume).
- The graphile-worker background job system; all live jobs already run
on `@trigger.dev/redis-worker`.
- The `DEPRECATE_V3_ENABLED` flag: v3 is now rejected unconditionally,
so the flag is gone.
- Unused v3 exports from `@trigger.dev/core` (the `v3/zodNamespace`
subpath and the legacy socket message catalogs) and the now-dead MarQS
environment variables.
## What stays
The v4 engine is untouched. The graceful v3 rejection boundary stays,
`determineEngineVersion` still detects a v3 project so it can reject it,
and the batch service plus batch-completion worker stay for current
clients. Live queue concurrency limits and metrics now read from the v4
run engine instead of MarQS, and a brand-new dev environment now
defaults to v4.
## Dependency cleanup
Removes webapp dependencies left unused by this change: `seedrandom` and
`semver` (only the removed v3 code used them) plus a set that was
already dead, their orphaned `@types` packages, and two dead files. Adds
a `knip:deps` script and a `knip.json` config so unused dependencies can
be found the same way going forward.
## Summary
Two threads on the run-ops split path.
Read path: per-item run reads are batched into grouped queries, a
waitpoint's connected-run reads are bounded, and the dedicated-schema
relation hydrators fetch only the requested columns instead of whole
rows. Retrieve also falls back to the other database when a routed read
misses, so a run whose physical residency diverges from its id shape is
still found rather than returning a spurious not-found. Fewer and
lighter queries on the run read path, with no change to results.
Mint-kind flip safety: flipping which database new runs mint to is now a
deterministic wall-clock cutover, for both per-org and global flips. For
a grace window every process resolves the same database, so a flip
cannot route two concurrent triggers that share an idempotency key to
different databases (which would bypass the per-database unique
constraint and create a duplicate run).
Supersedes the earlier #4205 and #4208.
Draft: validation in progress.
Switches the native build server from opt-in to opt-out in project build
settings.
- It's now enabled by default, stored as a new
\`disableNativeBuildServer\` opt-out key so previously-saved
\`useNativeBuildServer: false\` values aren't treated as deliberate
opt-outs.
- The "Use native build server" checkbox is checked by default;
unchecking it persists the opt-out.
- Brief wording: clarifies build settings apply to GitHub-triggered and
native build server deployments, and the native build server hint no
longer says "in the future".
## Summary
Under high OTLP ingest volume, the whole decode, transform, and enrich
pipeline runs on the request event loop, so a single CPU core becomes
the ceiling while the rest sit idle. This adds an opt-in worker pool
that moves decode, transform, and LLM-cost enrichment onto worker
threads, keeping the main thread free for I/O. It is off by default
(`OTEL_TRANSFORM_WORKER_POOL_ENABLED`), so behavior is unchanged unless
enabled.
## Design
Workers do decode, filter, convert, and enrich (including LLM pricing
match). The main thread stays the single database reader: it loads the
pricing registry and broadcasts the compiled model rows to the workers
(re-broadcasting on every reload), so workers never touch the database.
The pure transform is extracted into a dependency-light module (no
Prisma/Redis/ClickHouse imports) so it can run inside a worker.
Importantly, the main thread keeps the existing single consolidated
insert path, so ClickHouse insert batching and part count are unchanged.
The parallelism buys CPU headroom, not more insert streams (which would
add merge pressure).
The worker is bundled as a standalone file at build time and ships in
the existing image with no Dockerfile change. In local load testing the
pool sustained roughly 2.6x the throughput of the single-thread path and
kept the main thread responsive under load.
## Summary
Sending a chat message immediately after an action (for example an undo)
could make the message's response vanish from the UI. The transport
opened a response stream that closed on the *earlier* turn's completion
instead of waiting for the send's own turn. The agent still produced and
persisted the answer, so it reappeared on refresh. Same "disappearing
message" class as
[#4176](https://github.com/triggerdotdev/trigger.dev/pull/4176),
different cause.
## Fix
A send's response stream had no way to tell whether a `turn-complete`
belonged to its turn. `POST /realtime/v1/sessions/:id/in/append` now
returns the appended record's sequence number, and the transport skips
any turn-complete whose `session-in-event-id` (the agent's committed
`.in` cursor) is below that seq, closing only on its own turn. Older
webapps omit the seq, in which case the transport falls back to the
previous behavior, so the SDK and server can ship independently.
Because the fix spans the SDK and the server, both a webapp deploy and
an SDK release are needed for the full effect.
Verified end to end with the ai-chat reference app:
undo-then-immediate-send loses the follow-up's answer before the fix and
streams it inline after, with a revert-the-guard run reproducing the
loss on the same script. Unit tests cover the skip and the no-seq
fallback.
## Summary
The default realtime backend was hardcoded to Electric. This adds a
`REALTIME_BACKEND_DEFAULT` env var (`electric` | `native` | `shadow`,
default `electric`) that chooses the backend for any environment whose
org has no `realtimeBackend` override. Behavior is unchanged unless you
set it; per-org overrides still win.
The default is applied at every point where the per-org flag falls
through: the initial value, the flag lookup default, and the error
fallback.
## Problem
Several display/grouping issues in the **Errors** feature, all rooted in
how the ClickHouse error materialized views (`errors_mv_v1`,
`error_occurrences_mv_v1`) read the stored error JSON produced by
`parseError`:
1. **Messageless errors show "Unknown error".** An empty message falls
straight through `coalesce(nullIf(message,''), 'Unknown error')` to the
literal, even though the error's class `name` is available (e.g. an
Effect tagged error `ListMessagesError` with no message).
2. **Unrelated errors collapse into one group.**
`calculateErrorFingerprint` keys on `type : message : stack`, where
`type` is always the union tag (`BUILT_IN_ERROR`, …), `message` is
empty, and the stack isn't read — so every messageless built-in error
(and every string/custom error) hashes to the same constant input → one
fingerprint.
3. **error_type shows the internal tag.** `coalesce(type, name, …)`
always resolves to `type` (always present), so the column shows
`BUILT_IN_ERROR` instead of the real class name.
4. **Stack traces never populate.** The MVs read `error.data.stack`, but
the serializer stores the trace under `stackTrace` — so the column is
always empty.
## Fix
All display changes are `ALTER TABLE … MODIFY QUERY` on the two views
(migration `035`); the fingerprint change is in the webapp.
- **Fingerprint** (`errorFingerprinting.ts`): fall back **message → name
→ raw**. Messageless errors now group by class name (or raw value for
non-Error throws); message-bearing errors are **unchanged**
(short-circuits at `message`), so existing groups don't split — only
currently-messageless errors get their own group going forward.
- **error_message**: same `message → name → raw` fallback before
`'Unknown error'`.
- **error_type**: coalesce `name → code → 'Error'` (drops the reliance
on the union tag). Built-in → class name, internal → `code`,
string/custom → `Error`.
- **stack trace**: read `error.data.stackTrace`. Bounded as before
(serializer caps 50 frames / 1024 chars per line; MV clips to 2000
chars).
## Migration notes
- `MODIFY QUERY` swaps the view query in place (no drop/recreate gap);
Down restores the previous query.
- **Existing rows are left unchanged** — changes apply only to rows
inserted after the migration. No backfill.
## Tests
`errorFingerprinting.test.ts` — 57 pass, incl. new cases for messageless
class names, string/custom raw values, and stability of message-bearing
fingerprints.
Fixes the display-derivation half of TRI-11938 (error_type + stack
trace); relates to TRI-9254 and TRI-9250.
Applying a directory-sync effect that would demote the org's last Owner
(a
group remap, or a provision) previously threw and 500'd the settings
save. Now
rbac.setUserRole reports code:"last_owner" and applyEffect skips just
that
member (they keep Owner) while the rest of the batch applies.
Adds the machine-readable RoleAssignmentResult.code to the plugin
contract so
callers can tell the last-owner guard apart from a real failure.
## Summary
The RBAC and SSO auth plugins can own their own database client, but
they could only read `DATABASE_URL`, so every connection they opened
landed on the primary. The host webapp now resolves writer and
read-replica URLs from its env (the same fallback chain its own Prisma
clients use: control-plane URL first, then the default) and passes them
to the plugins at create time via a shared `PluginDatabaseConfig`, along
with separate connection limits for writes (default 2) and reads
(default 5, tunable via `RBAC_DATABASE_*_CONNECTION_LIMIT` and
`SSO_DATABASE_*_CONNECTION_LIMIT`).
A plugin can then route hot-path reads (per-request auth checks, login
routing) to the read replica and keep only rare mutations on the
primary. With no replica configured, or no plugin installed, nothing
changes: the OSS fallback ignores the new option and keeps reading
through the Prisma clients it is already given.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
## Summary
This adds an always-visible info tooltip on the Location column and a
note in the "set default region" confirmation dialog making it explicit.
It also removes the obsolete "V4" badge from the Regions page title.
## What & why
Signup promo credits. A new logged-out `/promo?code=<code>` landing page
validates the code and carries it through signup via a cookie. When the
new organization is activated by selecting a plan, the code is redeemed
and its credits are applied; the usage page then shows the remaining
promo credits and their expiry.
## Notes
- The code is redeemed at **plan selection**, not org creation: the
credit grant targets the org's usage allowance, which only exists once a
plan is selected — applying at creation would have nothing to grant
onto. Redemption is best-effort and never blocks plan selection.
- Pairs with the corresponding billing-service change (promo code
validate/apply/credits + grant issuance); the two are released together.
## Testing
Verified locally end to end: `/promo` shows the offer, a new account
carries the code through signup, selecting the Free plan redeems it, and
the usage page shows the remaining credits.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
## What
Adds per-variable secret support to the `syncEnvVars` build extension.
Return `{ name, value, isSecret: true }` and the variable is stored as a
secret (redacted in the dashboard, value non-revealable), just like a
manually created secret env var. Secret and non-secret variables can be
mixed in one callback.
```ts
syncEnvVars(async () => [
{ name: "PUBLIC_API_URL", value: "https://api.example.com" },
{ name: "DATABASE_URL", value: "postgres://...", isSecret: true },
]);
```
## How
Env vars flow through the build pipeline as a flat name→value map, and
the import API's `isSecret` is per-call. So secret vars are carried
through the layer + manifest in parallel `secretEnv` / `secretParentEnv`
maps, and at deploy time they go up in a second `importEnvVars` call
with `isSecret: true` (the plain vars in the first call). The record
form (`{ KEY: "value" }`) is unchanged and stays non-secret.
## Commits
- `feat(core)`: carry secret env vars through the build layer + manifest
schema
- `feat(build)`: partition `isSecret` vars in `syncEnvVars`
- `feat(cli)`: merge secret layers and import them with `isSecret: true`
at deploy
- `test(build)`: cover the partitioning + document `isSecret`
## Testing
- vitest covers the partitioning (secret/non-secret × child/parent) and
that the record form stays non-secret.
- Verified against a local webapp that the deploy's import contract
stores the secret var redacted (`isSecret: true`) and the plain var
visible.
Closes TRI-11099
Creating a Prisma migration now formats its schema first, keeping
migration-related schema edits consistently formatted without adding
work to the repository-wide format command. Run `pnpm run format:prisma`
to format either schema on demand.
## Summary
A Query page (TRQL) query that pulls fields out of a run's `output` with
JSON functions (`JSONExtractString`, `JSONExtractInt`, `JSONHas`, and
the rest of the family) failed with "The first argument of function ...
should be a string containing JSON, illegal type: JSON". Those queries
now work.
## Root cause and fix
`output` is a native ClickHouse `JSON` column, but `JSONExtract*`,
`JSONHas`, `JSONLength`, and `JSONType` all expect a String containing
JSON text. The compiler already swaps in the column's String companion
(`output_text`) when a JSON column is selected or compared, but not
inside function-call arguments, so it emitted `JSONExtractInt(output,
'x')` against the native column.
The fix prints the companion column for the first argument of these
functions when it resolves to a bare JSON field, keeping the table alias
when qualified (so it works in JOINs):
JSONExtractInt(output, 'x') -> JSONExtractInt(output_text, 'x')
JSONExtractArrayRaw(assumeNotNull(output), 'y') ->
JSONExtractArrayRaw(assumeNotNull(output_text), 'y')
It also reaches through value-preserving passthrough wrappers like
`assumeNotNull(...)`, while leaving value-changing wrappers like
`toJSONString(output)` on the native column (that argument is already a
String). The swap is also semantically correct, not just a type fix:
`output_text` is the unwrapped data JSON that the TRQL `output` model
already represents, so field paths line up.
Covered by printer unit tests and a ClickHouse integration test that
runs the whole family (plus the wrapped and `toJSONString` cases)
against a real native-JSON column. Both new cases fail with the exact
"illegal type: JSON" error without the fix.
## ✅ Checklist
- [x] I have followed every step in the [contributing
guide](https://github.com/triggerdotdev/trigger.dev/blob/main/CONTRIBUTING.md)
- [x] The PR title follows the convention.
- [x] I ran and tested the code works
---
## Testing
- Verified the rule emits exactly five errors for un-awaited throws of
the known
async redirect helpers while ignoring awaited throws, returned promises,
and
synchronous `redirect(...)`.
- Verified `--fix` inserts `await` in async functions and produces a
clean
second lint run.
- Verified synchronous functions remain diagnostic-only so autofix
cannot
introduce invalid syntax.
- Ran `pnpm run format`, `pnpm run lint`,
`pnpm run typecheck --filter webapp`, and `git diff --check`.
---
## Changelog
Adds an Oxlint rule that prevents async redirect helpers from being
thrown
without awaiting their `Response`. Existing violations are fixed, the
autofix
is limited to async functions, and the plugin uses an explicit ESM
extension.
---
## Screenshots
See the test-results comment for CLI evidence.
💯
Link to Devin session:
https://app.devin.ai/sessions/e60ad7610773401da3d3040cf1252337
Requested by: @ericallam
---------
Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: Eric Allam <eric@trigger.dev>
- Combine SDK and dashboard bulk actions docs
- Fix API reference pages for bulk actions
- Fix weird rendering on bulk actions page
## Todo
- [ ] not sure about having the SDK+dashboard combined and under "Using
the dashboard"... need to find the right place
## Summary
The limits page didn't spell out that large payloads offload to object
storage automatically, and its single "512KB" note conflated two
different thresholds. This clarifies the behaviour.
On the way in, the SDK uploads any trigger or batch-item payload over
128KB to object storage before sending, so large triggers and batches
don't hit the request body limit (`trigger` / `triggerAndWait` since
4.5.0, `batchTrigger` / `batchTriggerAndWait` since 4.5.2). On
retrieval, payloads and outputs over 512KB are stored in object storage
and returned as a presigned URL from `runs.retrieve`.
## Summary
Magic link login could appear completely broken: submitting your email
on the login page showed a stale "This email is unauthorized" error
instead of the "we've sent you a magic link" confirmation, even when the
address was fine.
This PR reverts
[#4215](https://github.com/triggerdotdev/trigger.dev/pull/4215) (whose
diagnosis and fix turned out to be wrong) and fixes the actual bug,
which was in how login errors are stored and consumed.
## Root cause
Two session bugs compounded on the login page:
- The `/login` loader read the flashed `auth:error` without committing
the session. A Remix flash is only consumed when the session is
committed after the read, so once any attempt flashed an error (for
example an address rejected on an instance with `WHITELISTED_EMAILS`
set), it stayed in the session cookie and reappeared on every later
`/login` visit, making successful attempts look like failures.
- The `/login/magic` action stored its validation and rate limit errors
with `session.set`, which survives every later read and commit, so those
errors stuck permanently.
[#4215](https://github.com/triggerdotdev/trigger.dev/pull/4215) had
instead diagnosed a server-only module leaking into the client bundle
and crashing navigation. Checking the shipped images' client bundles via
their sourcemaps shows `.server` modules were always stubbed out, so
that change fixed nothing and is reverted here.
## Fix
- `/login` reads the flashed error and commits the session when one was
present, so an error renders once and clears. The `redirectTo` branch
now surfaces the error too instead of leaving it in the cookie.
- `/login/magic` flashes its errors instead of `set`ting them.
Verified end-to-end on a live preview environment: a rejected address
shows the error once and a reload clears it; a valid address lands on
the confirmation screen with the address named; GitHub, Google, and SSO
login paths are untouched by this diff.
## Summary
1 improvement, 2 bug fixes.
## Breaking changes
- Removed support for the end-of-life v3 `trigger dev` CLI. Starting a
dev session with an old v3 CLI now returns an upgrade message instead of
connecting - upgrade to the v4 CLI to continue using `trigger dev`.
([#4198](https://github.com/triggerdotdev/trigger.dev/pull/4198))
## Bug fixes
- Fix TS2742 ("inferred type cannot be named") when exporting a
`chat.agent` from a project with declaration emit: `ChatTaskWirePayload`
and `ChatInputChunk` are now declared in the public
`@trigger.dev/sdk/chat` subpath, so inferred agent types emit portable
declarations and the wire types are directly importable.
([#4218](https://github.com/triggerdotdev/trigger.dev/pull/4218))
## Server changes
These changes affect the self-hosted Docker image and Trigger.dev Cloud:
- Reduce primary database load on the runs page by serving its
empty-state check from ClickHouse instead of Postgres.
([#4202](https://github.com/triggerdotdev/trigger.dev/pull/4202))
- Fixed submitting your email on the login page reloading back to an
empty form instead of showing the magic link confirmation screen.
([#4215](https://github.com/triggerdotdev/trigger.dev/pull/4215))
<details>
<summary>Raw changeset output</summary>
# Releases
## @trigger.dev/build@4.5.3
### Patch Changes
- Updated dependencies:
- `@trigger.dev/core@4.5.3`
## trigger.dev@4.5.3
### Patch Changes
- Updated dependencies:
- `@trigger.dev/build@4.5.3`
- `@trigger.dev/core@4.5.3`
- `@trigger.dev/schema-to-json@4.5.3`
## @trigger.dev/python@4.5.3
### Patch Changes
- Updated dependencies:
- `@trigger.dev/sdk@4.5.3`
- `@trigger.dev/build@4.5.3`
- `@trigger.dev/core@4.5.3`
## @trigger.dev/react-hooks@4.5.3
### Patch Changes
- Updated dependencies:
- `@trigger.dev/core@4.5.3`
## @trigger.dev/redis-worker@4.5.3
### Patch Changes
- Updated dependencies:
- `@trigger.dev/core@4.5.3`
## @trigger.dev/rsc@4.5.3
### Patch Changes
- Updated dependencies:
- `@trigger.dev/core@4.5.3`
## @trigger.dev/schema-to-json@4.5.3
### Patch Changes
- Updated dependencies:
- `@trigger.dev/core@4.5.3`
## @trigger.dev/sdk@4.5.3
### Patch Changes
- Fix TS2742 ("inferred type cannot be named") when exporting a
`chat.agent` from a project with declaration emit: `ChatTaskWirePayload`
and `ChatInputChunk` are now declared in the public
`@trigger.dev/sdk/chat` subpath, so inferred agent types emit portable
declarations and the wire types are directly importable.
([#4218](https://github.com/triggerdotdev/trigger.dev/pull/4218))
- Updated dependencies:
- `@trigger.dev/core@4.5.3`
## @trigger.dev/core@4.5.3
</details>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
## Summary
Exporting a `chat.agent` from a project with `declaration: true` failed
with TS2742: the inferred type of the agent references
`ChatTaskWirePayload`, which was declared in an internal module not
reachable through the package exports map, so tsc could only name it via
a file path into `node_modules` and refused to emit. Consumers had to
hand-mirror the wire type and annotate their export.
## Fix
`ChatTaskWirePayload` and `ChatInputChunk` are now declared in
`@trigger.dev/sdk/chat` (a public subpath) and re-exported type-only
from the internal shared module, so every internal import is unchanged
and the browser/server module split is untouched. Declaration emit for
an inferred agent type now produces a portable specifier:
```ts
export declare const chatAgent: Task<"chat-agent", import("@trigger.dev/sdk/chat").ChatTaskWirePayload<MyUIMessage, MyClientData>, unknown>;
```
As a side effect the wire types are now directly importable, which is
what affected users were reconstructing by hand.
## Verification
Reproduced against the built 4.5.2-equivalent package: a consumer
fixture with declaration emit produced `import("<file
path>/ai-shared.js")` in its declaration (the TS2742 trigger); after the
fix the same fixture emits the public specifier with zero diagnostics. A
regression test now builds that consumer simulation in a temp directory
on every test run: it copies the built package into a fake node_modules
(copied, not symlinked, because tsc only applies exports-map naming to
real node_modules paths), compiles the fixture with the TypeScript API,
and asserts no errors, no relative-path imports, and no internal module
references in the emit.
## Summary
Submitting your email on the login page could reload back to an empty
login form instead of showing the "we've sent you a magic link"
confirmation. The magic link email was still sent, so it looked like
nothing happened.
## Root cause
The `/login/magic` route imported a server-only cookie module
(`magicLinkEmailCookie.server.ts`) whose top-level `env.NODE_ENV` read
got bundled into the route's client JS. On the client `env` is
undefined, so the module threw a `TypeError` at module eval, which
aborted Remix's client-side navigation to the confirmation and
hard-reloaded back to `/login`. It only surfaced in production builds
(local dev auto-logs-in, and local prod builds happen to tree-shake the
module out), which is why it slipped through.
## Fix
The email-link strategy already stores the submitted address in the
session (`auth:email`), so the separate cookie was redundant. Deleted
the cookie module and read the address from the session in the loader.
With the module gone, nothing server-only can leak into the client
bundle regardless of tree-shaking.
Verified the confirmation renders with the email address, the SSO
domain-policy redirect (with the email prefilled) still works, and a
production build no longer bundles the module.
Condense the kept rationale comments (logLevel/warn, last-Owner dedup,
role
overwrite) and drop the obvious function-header comments that just
restated
the code. No behavior change.
## Summary
Excludes the non-published plugins workspace from Changesets release
planning so it cannot drive public package version bumps.
## Verification
Ran `pnpm run changeset:version` with temporary changesets for
`@trigger.dev/plugins` and `@trigger.dev/core`; the ignored workspace
produced no release-driver updates, and the public package changeset
versioned normally.
Directory-sync effects are idempotent and the accounts-webhook worker
retries
the whole event, so a single failed attempt (typically a role assignment
losing a serializable race during a backfill burst) is self-healing
rather
than alert-worthy. Tag those thrown errors with logLevel "warn" so the
worker
logs at warn instead of error, keeping them visible for triage without
paging.
## Summary
The runs page's empty-state check (whether an environment has ever had a
run, which decides between the "getting started" and "no runs match your
filters" states) ran a `findFirst` against the Postgres `TaskRun` table.
This moves it to ClickHouse, the same store the runs list itself reads
from, so the check no longer queries `TaskRun`.
## Design
Only the runs list triggers the check now (via an `includeHasAnyRuns`
flag); the other presenters that reuse `NextRunListPresenter` (API,
schedule detail, waitpoint detail, error group) no longer issue it. When
the list is empty it runs `SELECT 1 FROM task_runs_v2 ... LIMIT 1`
filtered on the full `(organization_id, project_id, environment_id)`
sort-key prefix with a configurable `created_at` lower bound
(`RUN_LIST_HAS_RUNS_LOOKBACK_DAYS`, default 30), so it hits the primary
index and reads minimal granules.
Results are cached in a tiered memory + Redis SWR cache. Only positive
("has runs") results are cached, so an environment with no runs is
always re-checked and its first run shows up immediately.
Follow-up to #4194 (v3 execution app + core-helper removal). The v3
(engine V1) is end-of-lifed and enforced off in prod, so this removes a
self-contained slice of the remaining dead v3 code while **keeping every
user-facing deprecation message** - a user still on v3 must still be
told to upgrade.
## Legacy dev websocket
`app/v3/handleWebsockets.server.ts` backs the `/ws` transport used
**only** by the legacy v3 `trigger dev` CLI (v4 dev uses a different
transport). It's now authenticate-then-close with
`V3_DEV_DEPRECATION_MESSAGE`, so an old CLI is still told what to do -
only the legacy `AuthenticatedSocketConnection` / `DevQueueConsumer`
execution behind it (which can no longer run) is removed.
- Deleted `app/v3/authenticatedSocketConnection.server.ts` (its only
consumer).
- `engineDeprecation.server.ts` and the deprecation message constants
are untouched.
## Docs
Deleted the intentionally-legacy "Docker (legacy)" self-hosting page
(`open-source-self-hosting.mdx`) and redirected
`/open-source-self-hosting` (+ the existing
`/v3/open-source-self-hosting` alias) to `/self-hosting/overview`;
repointed the two inbound links. The current `self-hosting/*` docs
already describe the v4 (single supervisor) setup.
## Deliberately out of scope
Despite the branch name, this PR does **not** touch MarQS or the
socket.io coordinator/provider namespaces. Investigation found MarQS is
entangled with **live v2** queue/metrics/concurrency/project-cleanup
code (`runQueue`, `queueSizeLimits`, `taskRunConcurrencyTracker`,
`EnvironmentQueuePresenter`, `registerProjectMetrics`, `deleteProject`),
so it needs a per-file reviewed pass, not a bulk delete. That remainder
stays on TRI-11883.
refs TRI-11883
## Summary
4 improvements, 5 bug fixes.
## Improvements
- Add SDK and API client helpers for run bulk actions.
([#4105](https://github.com/triggerdotdev/trigger.dev/pull/4105))
- Large batch payloads now offload to object storage instead of riding
inline in the trigger request. `batchTrigger` and `batchTriggerAndWait`
(and the by-id and by-task variants) offload any per-item payload over
128KB before sending, the same way single `trigger` and `triggerAndWait`
already do, so a big batch no longer blows past the API body limit.
([#4165](https://github.com/triggerdotdev/trigger.dev/pull/4165))
- Removed internal helpers that were only used by the end-of-life v3
self-hosted compute providers.
([#4194](https://github.com/triggerdotdev/trigger.dev/pull/4194))
- Add an `onEvent` callback to `TriggerChatTransport` /
`useTriggerChatTransport` that emits typed lifecycle events for sends,
stream connects, first chunk, and turn completion. Send-success metrics,
time-to-first-token, and "sent but never answered" watchdogs become a
few lines of client code.
([#4187](https://github.com/triggerdotdev/trigger.dev/pull/4187))
```ts
onEvent: (event) => {
if (event.type === "message-sent") metrics.timing("chat.send_ms",
event.durationMs);
if (event.type === "first-chunk") metrics.timing("chat.ttft_ms",
event.sinceSendMs ?? 0);
},
```
## Bug fixes
- fix(cli): honor the MCP server's `--dev-only` flag
([#4199](https://github.com/triggerdotdev/trigger.dev/pull/4199))
- Fix chat turns that throw (for example from an `onTurnStart` hook)
leaking their message listener, which lost or duplicated messages sent
during later turns.
([#4176](https://github.com/triggerdotdev/trigger.dev/pull/4176))
- Fix `chat.agent` and `chat.createSession` permanently dropping user
messages when several arrived during a single turn: every buffered
message is now dispatched as its own turn instead of only the first.
([#4176](https://github.com/triggerdotdev/trigger.dev/pull/4176))
- Fix chat continuation runs replaying already-answered messages: turns
delivered while the run was suspended now advance the session.in resume
cursor, so a new run picks up exactly where the previous one left off.
([#4176](https://github.com/triggerdotdev/trigger.dev/pull/4176))
- Fix `chat.createSession` swallowing a message sent shortly after
stopping a turn: the turn's message listener now detaches when the
stream settles, so those messages run as the next turn.
([#4176](https://github.com/triggerdotdev/trigger.dev/pull/4176))
<details>
<summary>Raw changeset output</summary>
# Releases
## @trigger.dev/build@4.5.2
### Patch Changes
- Updated dependencies:
- `@trigger.dev/core@4.5.2`
## trigger.dev@4.5.2
### Patch Changes
- fix(cli): honor the MCP server's `--dev-only` flag
([#4199](https://github.com/triggerdotdev/trigger.dev/pull/4199))
- Updated dependencies:
- `@trigger.dev/core@4.5.2`
- `@trigger.dev/build@4.5.2`
- `@trigger.dev/schema-to-json@4.5.2`
## @trigger.dev/core@4.5.2
### Patch Changes
- Add SDK and API client helpers for run bulk actions.
([#4105](https://github.com/triggerdotdev/trigger.dev/pull/4105))
- Large batch payloads now offload to object storage instead of riding
inline in the trigger request. `batchTrigger` and `batchTriggerAndWait`
(and the by-id and by-task variants) offload any per-item payload over
128KB before sending, the same way single `trigger` and `triggerAndWait`
already do, so a big batch no longer blows past the API body limit.
([#4165](https://github.com/triggerdotdev/trigger.dev/pull/4165))
- Removed internal helpers that were only used by the end-of-life v3
self-hosted compute providers.
([#4194](https://github.com/triggerdotdev/trigger.dev/pull/4194))
## @trigger.dev/python@4.5.2
### Patch Changes
- Updated dependencies:
- `@trigger.dev/core@4.5.2`
- `@trigger.dev/sdk@4.5.2`
- `@trigger.dev/build@4.5.2`
## @trigger.dev/react-hooks@4.5.2
### Patch Changes
- Updated dependencies:
- `@trigger.dev/core@4.5.2`
## @trigger.dev/redis-worker@4.5.2
### Patch Changes
- Updated dependencies:
- `@trigger.dev/core@4.5.2`
## @trigger.dev/rsc@4.5.2
### Patch Changes
- Updated dependencies:
- `@trigger.dev/core@4.5.2`
## @trigger.dev/schema-to-json@4.5.2
### Patch Changes
- Updated dependencies:
- `@trigger.dev/core@4.5.2`
## @trigger.dev/sdk@4.5.2
### Patch Changes
- Add SDK and API client helpers for run bulk actions.
([#4105](https://github.com/triggerdotdev/trigger.dev/pull/4105))
- Fix chat turns that throw (for example from an `onTurnStart` hook)
leaking their message listener, which lost or duplicated messages sent
during later turns.
([#4176](https://github.com/triggerdotdev/trigger.dev/pull/4176))
- Fix `chat.agent` and `chat.createSession` permanently dropping user
messages when several arrived during a single turn: every buffered
message is now dispatched as its own turn instead of only the first.
([#4176](https://github.com/triggerdotdev/trigger.dev/pull/4176))
- Fix chat continuation runs replaying already-answered messages: turns
delivered while the run was suspended now advance the session.in resume
cursor, so a new run picks up exactly where the previous one left off.
([#4176](https://github.com/triggerdotdev/trigger.dev/pull/4176))
- Fix `chat.createSession` swallowing a message sent shortly after
stopping a turn: the turn's message listener now detaches when the
stream settles, so those messages run as the next turn.
([#4176](https://github.com/triggerdotdev/trigger.dev/pull/4176))
- Add an `onEvent` callback to `TriggerChatTransport` /
`useTriggerChatTransport` that emits typed lifecycle events for sends,
stream connects, first chunk, and turn completion. Send-success metrics,
time-to-first-token, and "sent but never answered" watchdogs become a
few lines of client code.
([#4187](https://github.com/triggerdotdev/trigger.dev/pull/4187))
```ts
onEvent: (event) => {
if (event.type === "message-sent") metrics.timing("chat.send_ms",
event.durationMs);
if (event.type === "first-chunk") metrics.timing("chat.ttft_ms",
event.sinceSendMs ?? 0);
},
```
- Large batch payloads now offload to object storage instead of riding
inline in the trigger request. `batchTrigger` and `batchTriggerAndWait`
(and the by-id and by-task variants) offload any per-item payload over
128KB before sending, the same way single `trigger` and `triggerAndWait`
already do, so a big batch no longer blows past the API body limit.
([#4165](https://github.com/triggerdotdev/trigger.dev/pull/4165))
- Updated dependencies:
- `@trigger.dev/core@4.5.2`
## @trigger.dev/plugins@4.5.2
### Patch Changes
- Updated dependencies:
- `@trigger.dev/core@4.5.2`
</details>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
## Summary
UI/layout/copy pass over the org **SSO & Directory Sync** settings page
(formerly "Identity & Access"). No logic, gates, flags, or data flow
changed — server-side auth (`manage:sso`), Enterprise entitlement,
action validation, and data loading are all untouched.
- Renamed the nav item, page title, and meta from "Identity & Access" to
"SSO & Directory Sync".
- Added a reusable `SettingsLayout` component system (container,
section, header, row, block, actions) modeled on `/account/security`,
and refactored the SSO page onto it (section titles, dividers, left
title/subtitle + right action rows).
- Tightened all UI copy: concise, active voice, consistent labels, no
em-dashes.
- `Select` primitive: additive `wrap`, `popoverClassName`, and
`placement` props (all default to prior behavior) so role options show a
bright title with a wrapping description, right-aligned popover, and no
horizontal overflow.
- Removed the external-link arrow icon from buttons that open a modal;
kept it only on genuinely external actions (Contact us, Open in new
tab).
- Polished the admin portal link dialog: smaller description, tighter
spacing, `ClipboardField` with a permanent copy button, removed the
redundant Copy link button, and a provider-aware Open label (e.g. "Open
in WorkOS") derived from the link host with a safe fallback.
### SSO page UI
<img width="3568" height="2550" alt="CleanShot 2026-07-08 at 18 52
11@2x"
src="https://github.com/user-attachments/assets/009d2437-7552-4ff0-a457-64744a9fcd88"
/>
### Login with SSO and normal email test (local)
https://github.com/user-attachments/assets/b33a4ce9-c1fa-45c9-bd3c-077cb6fc9473
## Test plan
- [ ] Non-Enterprise org: SSO page shows the upsell state
- [ ] Enterprise org, non-Owner without `manage:sso`: 403
- [ ] Enterprise Owner: verify domains, configure SSO, connect
directory, JIT/default/group role selects, and enforcement toggle all
work
- [ ] Role select popovers: bright title + wrapping description,
right-aligned, no horizontal scroll
- [ ] Admin portal dialog: copy button works, "Open in WorkOS" opens the
portal in a new tab
---------
Co-authored-by: Cursor <cursoragent@cursor.com>
## What
Adds a new Example projects page: **ClickHouse chat agent** — a
`chat.agent()` that answers questions about your data by writing and
running SQL against ClickHouse Cloud via the official Node.js ClickHouse
client.
The page follows the existing example-project format (overview,
features, GitHub repo card, how-it-works with code excerpts, relevant
code links) and is registered in `docs.json` in alphabetical order.
## Note
The GitHub repo card links to
`triggerdotdev/examples/tree/main/clickhouse-chat-agent`, which lands in
a companion examples PR — merge that one first.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
v3 (engine V1) is end-of-lifed and the v3 clusters are gone, so this
removes the dead v3 execution code from the monorepo. It's the first
pass of TRI-11824 - the webapp v3 code paths are deliberately left
untouched and gated for a follow-up.
## Apps
Deletes the three v3-only execution apps and their build wiring:
- `apps/coordinator`, `apps/kubernetes-provider`, `apps/docker-provider`
- `.github/workflows/publish-worker.yml` - it built only those three;
the v4 worker publish is a separate workflow
- Their references in `.changeset/config.json`, `.cursorignore`,
`CHANGESETS.md`, `CONTRIBUTING.md`, `.server-changes/README.md`
- `pnpm-lock.yaml` regenerated to prune the apps and their app-only
dependencies (`socket.io`, `@kubernetes/client-node`, `p-queue`,
`execa`, `prom-client`, `tinyexec`)
## Core
Removes the helpers in `@trigger.dev/core` that only those apps used -
`ProviderShell`, `SimpleLogger`, the `Exec`/process helpers,
`isExecaChildProcess`, `getTextBody`, and `testDockerCheckpoint`. Each
was verified to have no remaining consumers anywhere in the repo.
Kept the helpers still used elsewhere: `ExponentialBackoff` (warm-start
client), `HttpReply`/`getJsonBody` (serverOnly http server),
`SimpleStructuredLogger` (widely used), and
`ZodNamespace`/`ZodSocketConnection` (still referenced by legacy v3
webapp code, hence the follow-up pass).
The `./v3/apps` and `./v3/serverOnly` export subpaths remain - only dead
members were trimmed from their barrels, so no `package.json` exports
changed.
## Verification
`@trigger.dev/core` builds, and `typecheck` passes for core, supervisor,
cli-v3, run-engine, redis-worker, and webapp.
refs TRI-11824
### Problem
The agent playground chat builds its realtime transport baseURL from
apiOrigin, but points it at a same-origin /resources/... dashboard
route. When API_ORIGIN differs from APP_ORIGIN, the in/append POST goes
cross-origin, fails the CORS preflight, and messages never reach the
agent ("Failed to fetch").
It only reproduces where the two origins differ — not locally, where
both default to localhost:3030.
Fixes#4149.
### Fix
Build the base URL from window.location.origin (falling back to
apiOrigin on SSR), so realtime traffic stays same-origin — the same
approach AgentView.tsx already uses.
### Testing
Typecheck passes. The CORS path only manifests when API_ORIGIN !=
APP_ORIGIN, so verify on test-cloud (can't reproduce locally).
## Summary
Deleting a project triggered an unbounded database cleanup that scanned
the project's entire run history, so deleting a project with many runs
could be very slow. Project deletion is a soft delete again: run data is
retained and the deletion completes quickly.
## Fix
Project deletion ran a cascade hard-delete whose `BulkActionItem` step
filtered through a relation to `TaskRun` scoped by `projectId`. Prisma
compiles that to an `EXISTS`-join over the project's entire `TaskRun`
set (a large, hot table with no `projectId` index), and it ran on every
project deletion unconditionally.
Removing the cascade-cleanup call restores the prior soft-delete
behaviour: queues are removed, the project is marked deleted, and run
data is retained. The cascade-cleanup service (added in
[#4117](https://github.com/triggerdotdev/trigger.dev/pull/4117)) had no
other callers, so it and its test are deleted.
## Summary
Realtime streams (AI-agent token streaming and run streams) now default
to v2 for self-hosters, backed by a bundled [s2-lite](https://s2.dev)
service. Self-hosting previously shipped no S2 configuration, so streams
ran on the Redis-backed v1 path and there were no docs for wiring up v2.
Both the Docker Compose stack and the Helm chart now provision s2-lite
with persistent storage and set the stream env vars out of the box.
## What's included
- **Docker Compose**: a persistent `s2` service (s2-lite), a basin init
spec, and the `REALTIME_STREAMS_S2_*` plus
`REALTIME_STREAMS_DEFAULT_VERSION=v2` env on the webapp. `.env.example`
documents the v1 fallback and hosted-S2 options.
- **Helm**: an `s2` StatefulSet, PVC, Service and ConfigMap (runs as the
non-root image user via `fsGroup`), an `s2` values block, and webapp env
wiring with an existing-secret path for hosted S2.
- **Docs**: the `REALTIME_STREAMS_S2_*` and
`REALTIME_STREAMS_DEFAULT_VERSION` vars in the webapp env reference,
plus a "Realtime streams" section in the Docker and Kubernetes
self-hosting guides.
## Notes
- The OSS code default stays `v1`; v2 becomes the default purely through
the self-hosting artifacts, so non-self-host deployments are unaffected.
Disabling s2, or setting the version back to `v1`, cleanly reverts to
Redis-backed v1.
- With v2 enabled, the bundled s2 service is a required dependency for
streaming: if it is down, streams error while the task itself still
runs. That is the intended trade for the better v2 path.
- You can point at a hosted S2 at s2.dev instead of the bundled server.
New `/billing-limits` page covering the full [billing limits
feature](https://trigger.dev/changelog/billing-limits) : the three limit
options (plan / custom / no limit), billing alerts (% of limit or dollar
thresholds), what happens when the limit is reached, the recovery flow,
the soft-limits caveat, and the billing limit marker on the Usage page.
## Summary
posthog-js sent product analytics to PostHog Cloud directly from the
browser. This points `api_host` at a same-origin `/ph` path that
forwards to PostHog Cloud EU server-side, following PostHog's standard
first-party reverse-proxy setup.
## How it works
A resource route forwards each request server-side, splitting by path:
`/ph/static/*` and `/ph/array/*` go to the asset host, everything else
(analytics events, feature flags) goes to the ingest host. It rewrites
the `Host` header, strips the `/ph` prefix, and streams the response
back. Only PostHog's own cookies are forwarded, so the app session
cookie stays first-party. Upstream hosts default to PostHog Cloud EU,
overridable via `POSTHOG_INGEST_HOST` / `POSTHOG_ASSETS_HOST`.
It also sets `cross_subdomain_cookie` so a single PostHog session is
shared across the marketing site and app.
Verified locally: static assets return 200 from the EU asset host, and
analytics events return 200 through the ingest host.
## Summary
`sendMessage` from `useChat` gives no feedback about whether a message
actually reached the backend, and the `fetch` override is wire-level: it
requires knowing endpoint semantics, cannot attribute requests to
messages, and misses the headStart first-turn POST entirely. This adds a
typed `onEvent` observability callback to `TriggerChatTransport` /
`useTriggerChatTransport` so send-success metrics, time-to-first-token,
and "sent but never answered" watchdogs become a few lines of client
code.
## Example
```ts
const transport = useTriggerChatTransport({
task: "my-chat",
accessToken: ({ chatId }) => mintChatAccessToken(chatId),
onEvent: (event) => {
switch (event.type) {
case "message-sent":
// Durably acknowledged by the session's input stream, not just "request accepted".
metrics.increment("chat.message_sent", { source: event.source });
metrics.timing("chat.send_duration_ms", event.durationMs);
break;
case "message-send-failed":
metrics.increment("chat.message_send_failed", { status: event.status });
break;
case "first-chunk":
metrics.timing("chat.ttft_ms", event.sinceSendMs ?? 0);
break;
case "turn-completed":
metrics.timing("chat.turn_duration_ms", event.sinceSendMs ?? 0);
break;
}
},
});
```
## Design
One callback, one discriminated union (`ChatTransportEvent`):
- `message-sent` / `message-send-failed`: terminal send outcomes with
`messageId`, a `source` discriminator (submit, regenerate, steer,
action, stop, head-start), `durationMs`, `bodyBytes`, the append's
idempotency key (`partId`, also stored on the server-side record), and
error + HTTP status on failure. `message-sent` means the append was
durably acknowledged, after any internal token-refresh retries.
- `stream-connected` (with a `resumed` flag and the cursor it connected
from), `first-chunk` (chunk type plus `sinceSendMs` for
time-to-first-token), `turn-completed` (`sinceSendMs` full-turn latency
and the agent's committed input cursor), and `stream-error` follow the
response side, so a send can be paired with the answer that should
follow it. `messageId` on response events is client-side attribution
from the last turn-producing send on that chat.
Emissions sit at the transport's existing choke points, covering every
send path uniformly (including steering and headStart, which the fetch
override cannot observe). Exceptions thrown by the callback are
swallowed: observability can never break the chat. The React hook keeps
the callback live across renders instead of freezing the first-render
closure.
## Verification
Unit tests drive the transport directly with the `fetch` override as the
network stub (send success/failure per source, stream lifecycle, resumed
flag, field enrichment, callback exceptions swallowed). Verified
end-to-end against a realistic metrics setup in the ai-chat reference
app (counters, send-duration and TTFT histograms, and both watchdogs
built purely on these events): a healthy two-turn chat produces exactly
the expected event sequence and TTFT values; an oversized append records
`message_send_failed` with status 413; and killing the worker after a
durable send fires both `sent_but_no_stream` and `sent_but_unanswered`,
reproducing and detecting the "message disappeared" failure mode that
motivated this feature.
## Summary
The run page could show an AI generation cost well above what the
provider actually charged, most visibly for OpenRouter and Vercel AI
Gateway requests where a heavily cache-read prompt was priced at the
full input rate. When the provider reports an exact per-request cost, we
now use that instead of catalog pricing.
## Fix
Gateway and OpenRouter include the exact per-request cost in
`ai.response.providerMetadata` (`openrouter.usage.cost` /
`gateway.cost`). That figure already reflects the cache-read discount
and the real per-provider rate, which the catalog cannot reconstruct:
cache-read counts do not arrive in `gen_ai.usage.*`, and per-model
catalog prices drift from what the provider billed, in either direction.
So provider-reported cost is now preferred, and the catalog is used only
when no provider cost is present.
Fallback routing is covered by the same change: when OpenRouter routes
to a different model, `gen_ai.response.model` already carries the served
model, so the cost follows the served model and the provider's own
figure makes it exact.
`extractProviderCost` now runs on every AI span, so it gets a cheap
`"cost"` substring guard to skip the JSON parse on reasoning-model spans
whose provider metadata carries large reasoning text and no cost field.
Regression tests cover the cache-discount overcharge, fallback
served-model pricing, gateway cost, and the catalog fallback path.
## Summary
`batchTrigger` and `batchTriggerAndWait` (and the by-id and by-task
variants) now offload any per-item payload over 128KB to object storage
before sending, the same way single `trigger`/`triggerAndWait` already
do since
[#3785](https://github.com/triggerdotdev/trigger.dev/pull/3785). A batch
of large items no longer inflates the request body past the API limit.
## Demo
A live local run: `batchTriggerAndWait` of 5 items × 300KB (1.5MB
total). Each item offloads to object storage, so the receiver run rows
hold a 65-byte `application/store` pointer instead of the 300KB body,
and every item round-trips (received == sent).
<img width="1000" height="494" alt="batch large-payload offload demo"
src="https://github.com/user-attachments/assets/77ae3958-97d6-4b5c-ab25-39b217caefbc"
/>
## Design
Both the array and streaming batch paths funnel through
`executeBatchTwoPhase`, so offloading happens once there: each item is
measured, then offloaded through the existing
`conditionallyExportPacket` when it crosses 128KB, with bounded
concurrency so a big batch doesn't fire an unbounded number of presigned
PUTs.
Because items are offloaded before the request, SDK batches arrive as
small `application/store` references, so the server-side inline offload
during item ingest (parallelised in
[#3777](https://github.com/triggerdotdev/trigger.dev/pull/3777)) mostly
no longer fires for them.
Every trigger and item also carries its pre-offload serialised size as
`options.payloadSize`. The trigger span records that value, so an
offloaded payload shows its real size instead of the size of the small
object-store reference (previously the span measured the reference).
Container runtimes (cri-o / containerd / podman) can't pull
zstd-compressed layers carried in a Docker v2s2 manifest
(`application/vnd.docker.image.rootfs.diff.tar.zstd`). A deploy built
with an outdated CLI can produce exactly that combination - and today
it's promoted to current and then fails every run at image-pull time.
This extends the pre-promotion image check (#4049) to also inspect the
manifest's layer media types. If any layer uses the unpullable zstd/v2s2
media type, the deploy is rejected at finalize with a clear message to
upgrade the CLI and re-deploy, instead of silently shipping a version
that can't start.
The manifest is already returned by the existing ECR `BatchGetImage`
call, so there's no extra registry request for single-arch images.
Parsing is a lenient Zod schema and **fails open** - a manifest we can't
read never blocks a deploy. Manifest lists / OCI indexes (no top-level
`layers[]`) and OCI zstd (`...tar+zstd`, which runtimes support) pass
unaffected.
Also clarifies in the contributor docs that changesets and
`.server-changes/` notes are user-facing and should be written for
users, not maintainers.
refs TRI-11702
<!-- ccr-slack-attribution -->
_Requested by **Matt Aitken** · [Slack
thread](https://triggerdotdev.slack.com/archives/C032WA2S43F/p1783430373189849?thread_ts=1783430373.189849&cid=C032WA2S43F)_
## ✅ Checklist
- [x] The PR title follows the convention.
- [x] I ran and tested the code works (typecheck of the edited files is
clean; see Testing)
---
## Testing
**Before:** the webapp run-replication test shard failed on nearly every
PR because assertions waited a fixed 1s for rows to replicate from
Postgres → ClickHouse and intermittently checked before the row arrived
under CI load.
**After:** those assertions poll (up to 30s, 250ms interval) until the
rows land, so they pass as soon as replication completes and stop
flaking, without slowing the happy path.
These tests are testcontainers-backed (need Docker + Postgres +
ClickHouse), so the full suite is exercised in CI. Locally I confirmed
the edited `runsReplicationService.part1..part8.test.ts` files
type-check with no new errors.
---
## Changelog
**How:** wrapped the ~21 present-row assertions across
`runsReplicationService.part1..part8.test.ts` in `vi.waitFor`, matching
the existing poll pattern in `part9.test.ts`. Left absence assertions
(expecting 0 rows / no spans) on a fixed settle delay since there is
nothing to poll for. Tests only — no production code changed.
Note: this does NOT touch the `subscribe()` startup race in
`internal-packages/replication/src/client.ts` (a riskier, separate
follow-up).
💯
---
_Generated by [Claude
Code](https://claude.ai/code/session_01KtUdSLKrK17eFVuRYXT6uj)_
---------
Co-authored-by: Claude <noreply@anthropic.com>
## Summary
Adds SDK and API support for run bulk actions. You can now create bulk
cancel or replay actions from `@trigger.dev/sdk` using run IDs or the
same filters as `runs.list()`, then retrieve, list, poll, or abort the
action by its `bulk_` handle.
Tests, docs, changesets added.
## Design
The dashboard bulk action service now accepts structured filters instead
of reading directly from a dashboard request, so the dashboard and API
share the same creation path. Replay actions created through the API are
attributed with the existing `api` trigger source, while
dashboard-created actions keep `dashboard`.
The SDK exposes the new surface under `runs.bulk.*`, including
`targetRegion` for replay region overrides and cursor pagination for
listing bulk actions.
## Filters and runIds
Nuance on filters. If `filter` is provided, it MUST have at least one
key. This is to remove the footgun of passing no filter and selecting
all runs.
```typescript
{ action: "cancel", runIds: ["run_1"] } // valid
{ action: "cancel", runIds: [] } // invalid, min(1)
{ action: "cancel", filter: { status: "FAILED" } } // valid
{ action: "cancel", filter: {} } // invalid
{ action: "cancel", filter: {}, runIds: ["run_1"] } // invalid
```
## Summary
Three fixes to the run-ops database split (the Cloud-only mode where
run-lifecycle rows live on a dedicated Postgres). All are inert in the
default single-database deployment.
The main fix: on the batch trigger paths, a parentless batch's item runs
chose their physical store from a fresh per-org mint-flag read at
processing time, so flipping an org's flag mid-batch could land an item
in a different store than its batch, breaking the `TaskRun.batchId`
foreign key (or silently orphaning the item). The other two harden the
split's safety nets: the schema-parity test now actually compares
columns, and the read fan-out gate now signals when it has been silently
disabled.
## Batch item residency
`RunEngineBatchTriggerService` (api.v2) and the BatchQueue item callback
(api.v3) now anchor each item's id mint on the batch's own friendlyId,
mirroring the already-safe `BatchTriggerV3Service`. Residency is a pure
id-shape check, so an item can no longer diverge from its batch across a
mid-batch flag flip. The pre-failed-run fallback is anchored the same
way (it also sets `batchId`), and the shared mint branch is consolidated
into one helper so every mint path stays in lockstep. No new database
queries; single-database mode is unchanged (a cuid-shaped batch
friendlyId yields a cuid item).
## Schema parity test
The parity test previously read only the dedicated schema and matched
model headers with regexes, so it never compared columns and could not
catch a run-subgraph column that diverged between the two physical
schemas. It now parses both schemas and asserts bidirectional
scalar-column parity (type, nullability, array-ness, default) across the
run-subgraph models, and fails on any field line it can't parse. Scoped
to the run-subgraph models so unrelated control-plane edits don't break
it.
## Read fan-out signal
The split read fan-out gate is decided by the object identity of the NEW
vs control-plane clients. It now warns when both run-ops URLs are set
but the NEW client isn't a distinct instance (fan-out silently off), and
a new test exercises the real topology-into-gate wiring so a future
refactor that aliases the clients can't disable fan-out unnoticed.
## Verification
New unit and glue tests cover all three changes; the DB-backed
residency, store-routing, and topology suites pass against real
Postgres; `typecheck` is clean for both packages.
## Summary
Sending a message to a chat whose run had ended could make the message
vanish: the continuation run replayed already-answered messages, never
processed the new one, and a page refresh lost it entirely. Chasing that
report surfaced four composing message-loss bugs in the chat session
runtime; this PR fixes all of them, each with a regression test.
## The fixes
1. **Stale resume cursor.** Records delivered while a run was suspended
(the waitpoint path) advanced the SSE resume counter but not the
committed-consume cursor, so the `session-in-event-id` header stamped on
turn-completes went stale by one record per suspended turn. Continuation
boots seed from that header, which is what made them replay
already-processed messages. `session.in.wait()` now advances both
cursors.
2. **Only the first buffered message dispatched.** Messages arriving
during a turn are consumed into a buffer whose end-of-turn pickup
dispatched only the first entry; the buffer was recreated each turn, so
the rest were discarded, and since consuming a record commits the cursor
the loss was permanent. A continuation boot's replay delivers several
records back-to-back, which put the user's new message at index 1 or
later. The buffer now outlives the turn and drains one message per turn
in both `chat.agent` and `chat.createSession` (whose equivalent buffer
was never read at all).
3. **Post-stop window in `chat.createSession`.** The turn's message
listener stayed attached through the stopped turn's post-stream work, so
a message sent shortly after stopping a turn was consumed into the dead
steering queue and lost. The listener now detaches when the stream
settles, matching the `chat.agent` loop.
4. **Handler leak on errored turns.** A turn that threw outside the
streaming section (for example from an `onTurnStart` hook) leaked its
message listener. Previously that silently lost mid-turn messages; with
the loop-level buffer it would have duplicated them instead. The
subscription handle is now detached by the turn's catch/finally, and
`chat.createSession` defensively detaches its prior turn's listener when
user code exits a turn without `complete()`/`done()`.
## Verification
Reproduced end-to-end with the ai-chat reference project before the fix
(message consumed but never answered, two replayed turns, gone on
refresh) and verified after (single clean turn, survives refresh,
turn-complete cursors strictly advancing). Regression tests in
`packages/trigger-sdk/test/pending-message-drain.test.ts` cover all
four, each verified red against the unfixed behavior. A smoke sweep of
the standard chat scenarios (basic send, multi-turn, suspend/resume,
mid-stream refresh, stop, steering, cancel + continue, and the
`createSession` variant) passes on the final branch state.
## Summary
When the platform database is briefly unreachable while a run is
resuming from a wait, the run no longer fails with
`TASK_EXECUTION_ABORTED`. The worker now retries the resume through the
outage instead of aborting on the first blip.
## Root cause
Resuming a run calls the engine's `continue` worker-action endpoint.
That route caught every error and returned a `422`, which the worker's
HTTP client treats as non-retryable. So a transient Prisma
infrastructure error (for example `P1001` "Can't reach database server")
was flattened into a permanent failure: the worker gave up, force-killed
the run process, and completed it with `TASK_EXECUTION_ABORTED`.
## Fix
- The `continue` route now lets infrastructure errors propagate to the
generic 500 handler (message scrubbed, and retryable by the worker's
HTTP client), the same treatment the trigger path already gives them via
`isInfrastructureError`. Genuine validation errors (snapshot mismatch,
invalid state) still return `422`, so a stale retry stays non-retryable.
Resuming is idempotent server-side (guarded by the snapshot id), so
retrying is safe.
- The worker's `continueRunExecution` calls (both the
runner-to-supervisor and supervisor-to-engine hops) retry with a longer,
jittered backoff so they can ride out an outage lasting tens of seconds,
and the jitter keeps a fleet of resuming runs from stampeding the
database the moment it recovers.
Builds on #3960, which scrubbed the leaked message on these routes but
left the status non-retryable.
No changeset: this is a server-side behaviour fix recorded via
`.server-changes`. The `@trigger.dev/core` edits are internal run-engine
worker plumbing, not a public API change.
## Summary
Running `pnpm run db:migrate` locally left `defaultPrices.ts` and
`modelCatalog.ts` in `llm-model-catalog` showing as modified every time,
a ~10k-line diff that only ever changed formatting. This stops the
churn.
## Root cause
The root `db:migrate` script ends in `&& turbo run generate`, which runs
the `generate` script in every package that has one, including this one.
The generator writes its output with `JSON.stringify` (quoted keys, no
trailing commas), but the checked-in copies had been reformatted by
oxfmt (unquoted keys, trailing commas). So the generator output never
matched what was committed, even though the parsed data was identical.
## Fix
Add the two generated files to `.oxfmtrc.json`'s ignore list and commit
the raw generator output, matching how other codegen files in the repo
are already handled (e.g. the tsql grammar). Generation is
deterministic, so `generate` and `format` are both no-ops on a clean
tree now.
No changeset: internal package, dev tooling only, no runtime or public
API change.
## Summary
Depends on #4136.
Docker Compose self-hosting now uses the maintained `latest` image tag
by default instead of the frozen prerelease tag. The version-locking
docs keep pointing production users at explicit versioned tags when they
want pinned upgrades.
The engine `triggerTask` suite was a single 2447-line file with 23
`containerTest` cases, each spinning its own Postgres + Redis. vitest
shards by whole file, so all 23 container setups landed on one shard and
dominated its wall-clock. The recorded entry in `test-timings.json`
badly under-counts the real cost (it does not capture the
per-`containerTest` container startup that dominates on CI), so the
duration-sharding sequencer treated the file as light and stacked it,
producing one ~21 minute shard.
Splitting does not reduce the number of container setups; it lets those
23 cases distribute across shards instead of stacking on one. The webapp
unit-test stage is gated by its slowest shard, so this cuts the stage's
wall-clock roughly in half.
## CI timing (before vs after)
Real CI wall-clock of the `Unit Tests: Webapp` shards (`--shard=i/10`).
"Before" is sampled from recent runs on other branches (unsplit file,
from `main`); "after" is this PR.
| Shard | Before (s) | After (s) |
|------:|-----------:|----------:|
| 1 | 250 | 359 |
| 2 | 444 | 411 |
| 3 | 497 | 659 |
| 4 | **1257** | 284 |
| 5 | 545 | 641 |
| 6 | 284 | 644 |
| 7 | 244 | 214 |
| 8 | 340 | 445 |
| 9 | 188 | 395 |
| 10 | 234 | 567 |
| **Slowest shard (gates the stage)** | **~1247s (≈21m)** | **659s
(≈11m)** |
| Sum of all shards | 4283 | 4619 |
Before: shard 4 is the long pole at 1237s / 1247s / 1257s across three
sampled runs (the `triggerTask` file plus whatever else the packer put
with it). After: the six pieces spread across shards, the slowest drops
to 659s. The small rise in summed time is the extra per-file container
startup, paid in parallel across shards, so the gating number still
falls by about 10 minutes.
## Change
Split into six per-concern files that share a `triggerTaskTestHelpers`
module (the `vi.mock` calls stay per-file, since vitest hoists them):
- `triggerTask.test.ts` (3): trigger + concurrencyKey coercion
- `triggerTask.idempotency.test.ts` (4): idempotency + queue resolution
- `triggerTask.debounce.test.ts` (4): retries + debounce validation
- `triggerTask.mollifier.test.ts` (4): mollifier call-site behaviour
- `triggerTask.metadataCache.test.ts` (4): DefaultQueueManager task
metadata cache
- `triggerTask.residency.test.ts` (4): child run residency inheritance
All 23 cases are preserved. The file's `test-timings.json` entry is
split across the new files so bin-packing stays balanced.
While rewriting these files, cleanup was moved to `onTestFinished(() =>
engine.quit())` so an `engine`/`Redis` leaked on a failing assertion no
longer persists on the worker-scoped Redis and cascades into later cases
(`hookTimeout` raised to 60s so the after-cleanup gets the full budget).
Prisma lookups switched from `findUnique` to `findFirst` to match the
repo convention.
Verified: all six files run green locally (23/23), oxlint and oxfmt
clean.
## Summary
On the run-ops database split, a run that waits (`triggerAndWait`,
`batchTriggerAndWait`, `wait.forToken`) could hang forever after its
wait had already completed. The runner reads a resume from
`/snapshots/since` exactly once: if that read returned the resume
snapshot without its completed-waitpoints, the runner logged "executing
without completed waitpoints", advanced its cursor, and never re-read
it, so the awaiting run never continued.
## Root cause
The resume snapshot and its completed-waitpoint rows were written as two
separate commits. This regressed when the split replaced Prisma's atomic
nested `connect` with an FK-free insert (in
[#4163](https://github.com/triggerdotdev/trigger.dev/pull/4163)), and
`/snapshots/since` is served from a read replica. A fetch landing in the
sub-millisecond gap between the two commits, or a multi-reader replica
serving the snapshot from a different point in time than its join rows,
delivered an empty resume. Because the runner consumes each snapshot
once and treats an empty resume as terminal, a single stale read was
fatal and produced a permanent, nondeterministic hang.
## Fixes
- Commit a snapshot and its completed-waitpoint links in one
transaction, restoring the atomicity the split removed.
- Repair the completed-waitpoints from the owning primary when a
multi-reader replica serves the snapshot without its join rows. This
covers single-waitpoint resumes, which carry no
`completedWaitpointOrder` and so were missed by the count-based repair.
- Read the primary in the checkpoint `WAIT_FOR_BATCH` pre-check, so a
batch that already resumed is not re-suspended into a stall.
- Fall back to the primary when a waitpoint token misses both read
replicas, so a token completed immediately after it was minted no longer
returns a spurious 404.
- Route batch-item creation by `batchTaskRunId`, consistent with the
batch-completion count and the row's foreign key.
- Reject control-plane-only relation selects on the dedicated schema
with a clear error instead of an opaque Prisma failure, and stop
`createDateTimeWaitpoint` bypassing residency routing through a caller
transaction.
Verified against the deployed split topology: a resume snapshot and its
completed-waitpoints are now always delivered together, so the runner
can no longer drop a resume.
Extend the SSO plugin contract for directory sync and apply membership
effects
from the accounts webhook worker: provision users in mapped groups (role
from
group mapping, else the org default role), deprovision on removal, and
keep a
sticky-removal tombstone so JIT never silently re-adds a removed user.
JIT and
Directory Sync coexist; roles default to Developer (the JIT default-role
picker
has no 'None'). Changing a group's role in the dashboard re-applies it
to that
group's current members immediately. The Directory Sync settings section
(group→role mapping, external-domain + manual-membership policy,
deferred Save)
appears once a domain is verified — independent of SSO — gated by the
hasSso
flag. The settings page polls the whole page while entitled with
override-aware
drafts so in-progress edits are never clobbered.
## Summary
On the run-ops split, NEW-residency runs could hang. Time-based waits
(`wait.for`, `wait.until`, `delay`, waitpoint tokens),
`batchTriggerAndWait`, and attempt starts stalled and never resumed.
Each was a run-ops read or update that hit the wrong database: either
the owning store's read replica when it needed read-your-writes, or the
wrong store entirely because it routed by an id that does not encode
residency.
## Fixes
**Waitpoint resume (the main hang).** The managed resume path reads a
run's completed waitpoints by snapshot id
(`findSnapshotCompletedWaitpointIds`). Snapshot ids are cuids, which
always classify to the legacy store, so a NEW run's join rows (which
live on the new store) were never found. The resumed run saw zero
completed waitpoints and hung. It now fans out across both stores and
merges, like its sibling readers.
**Batch completion.** Batch item completion
(`updateManyBatchTaskRunItems`) routed by the item id, which is also a
cuid, so a NEW batch's items were updated on the wrong store, matched
zero rows, and the batch was treated as already complete (its parent's
`batchTriggerAndWait` then hung). It now routes by the batch id, which
does encode residency, matching the sibling `countBatchTaskRunItems`.
**Read-your-writes on the resume path.** The block-time
pending-waitpoint check (`countPendingWaitpoints`) and the attempt-start
lock check (`findRun` in `startRunAttempt`) both read the owning store's
replica with no read-your-writes guarantee, so a just-committed
waitpoint completion or dequeue lock could be missed under replica lag
and strand the run. Both now read the owning primary.
Each fix ships with a two-database store or engine test that reproduces
the hang and passes with the fix.
## Summary
The run-ops runs-replication source now takes its connection URL from
`RUN_REPLICATION_RUN_OPS_DATABASE_URL`, required whenever the run-ops
split is enabled.
The runs replicator speaks the Postgres streaming replication protocol,
which cannot run through a transaction pooler, so it needs its own
direct endpoint separate from the app's `RUN_OPS_DATABASE_URL` (which
may point at a pooler). When the split is on and this is unset, boot
fails via `SplitReplicationMisconfiguredError` rather than silently
falling back to a wrong endpoint.
## What
Adds the missing `COPY scripts/retry-prisma-generate.mjs` to the
supervisor `Containerfile` builder stage, before `RUN pnpm run
generate`.
## Why
The `generate` scripts in `internal-packages/database` and
`internal-packages/run-ops-database` shell out to
`scripts/retry-prisma-generate.mjs`. The supervisor build never copied
that file into the image, so `pnpm run generate` failed:
```
@internal/run-ops-database:generate: Error: Cannot find module '/app/scripts/retry-prisma-generate.mjs'
```
This is the same failure class as #4156 (webapp Dockerfile). The
supervisor `Containerfile` is the **only other** build file that runs
`pnpm run generate` — the coordinator / docker-provider /
kubernetes-provider Containerfiles don't, so this completes the fix.
## Verification
Local `docker build` of the supervisor `Containerfile` builder target —
result appended below once the build completes.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
## Build-blocker hotfix
`publish.yml` on `main` is failing to build the image after #4154
merged:
```
@trigger.dev/database:generate: Error: Cannot find module '/triggerdotdev/scripts/retry-prisma-generate.mjs'
… ERROR: process "/bin/sh -c pnpm run generate" did not complete successfully: exit code: 1
```
## Cause
#4154 (Windows-CI hardening) changed the `generate` scripts of
`@trigger.dev/database` and `@internal/run-ops-database` to call `node
../../scripts/retry-prisma-generate.mjs`. But `docker/Dockerfile`'s
`builder` stage does `COPY docker/scripts ./scripts` (replacing the
scripts dir) and then copies back only the specific root scripts it
needs (`updateVersion.ts`, `bundleSdkDocs.ts`) before `RUN pnpm run
generate` — the new `retry-prisma-generate.mjs` wasn't copied, so `pnpm
run generate` can't find it and the image build fails.
`publish.yml` only runs on push to `main` (not on PRs), so #4154's PR CI
never built the image and this slipped through.
## Fix
One line — copy the retry script alongside the other root scripts before
the generate step:
```dockerfile
COPY --chown=node:node scripts/retry-prisma-generate.mjs scripts/retry-prisma-generate.mjs
```
## Verification
Built locally with `docker build --target builder` (the stage that runs
`pnpm run generate`) to confirm the generate step now passes — result
appended once it finishes.
No changeset / `.server-changes` — Dockerfile/build-only change, no
package or server-runtime change.
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
## Problem
The run-ops split mints NEW-store run ids as **27-char base62 KSUIDs**.
The supervisor writes the run id into the Kubernetes pod name
(`runner-<id>`), and pod names must be DNS-1123 labels (lowercase
`[a-z0-9-]`) — so uppercase base62 ids make k8s reject the pod (422) and
**those runs never launch** (they loop in `PENDING_EXECUTING` until the
heartbeat-stall handler nacks them, forever). `.toLowerCase()` can't fix
it: base62 has both `A`(10) and `a`(36) as distinct symbols, so folding
collides distinct ids and destroys sort order.
## Fix: change the encoding, not the structure
Mint a **26-char lowercase base32hex** run id:
```
run_<24-char base32hex core><region char><version char>
[ 6-byte ms timestamp ][ 9 CSPRNG bytes ]
```
- **base32hex** (RFC 4648 §7, alphabet `0-9a-v`): lowercase,
order-preserving, DNS-safe; 15 bytes → exactly 24 chars, no padding.
Hand-rolled encode/decode (no new dependency).
- **48-bit ms timestamp** in the leading bytes → plain string sort ==
creation order at millisecond resolution.
- **72 bits CSPRNG** entropy; PK unique constraint is the backstop (no
retry loop).
- **region / version** are raw positional chars (read via one `charAt`
before decoding/routing), version = `"1"`.
DNS-safe from birth and hyphen-free, so **firekeeper is unchanged** —
`runner-<id>-attempt-N` → strip `runner-`, cut at first hyphen still
recovers the exact id incl. region+version.
## Residency discriminator: length → version char
`classifyKind`/`classifyResidency` (`runOpsResidency.ts`) previously
distinguished NEW vs LEGACY by **id length**. That gets ambiguous with a
third format. It now discriminates on the **version char at a fixed
position** (`isRunOpsIdBody`: 26 chars, `[25] === "1"`, base32hex
alphabet) → NEW; everything else → LEGACY. Total, never throws. The
`Residency` (NEW/LEGACY) contract the routing store consumes is
unchanged; the `"ksuid"` `ResidencyKind` label is retained only because
it's the persisted `runOpsMintKsuid` feature-flag value.
## Scope / verification
- Generator + discriminator in `@trigger.dev/core` isomorphic; mint path
+ all id-shape call sites swept (~40 webapp files); changeset added
(`@trigger.dev/core` patch).
- Core unit tests (encode/decode round-trip + property, generator shape,
ms sort-order incl. intra-second, parse partitioned-vs-legacy,
firekeeper round-trip): **24 pass**. `@trigger.dev/core` builds; webapp
typechecks; format/lint clean.
## Open decisions (flagged, not silently chosen)
1. **Backward-compat**: existing 27-char base62 KSUID runs now classify
LEGACY. On test cloud these are the broken/looping runs that never
completed, so this is acceptable — but worth a conscious call before
prod. No transitional length-recognition added (keeps the discriminator
clean).
2. **Storage collation**: the sort guarantee is byte-order — if the
run-ops id column is `TEXT` with default locale collation it's silently
not honored. Confirm whether `COLLATE "C"` / `BYTEA` is needed on the
run-ops schema.
3. **Region sourcing** wiring — see `regionCharForRegion` /
`REGION_CODES`.
---
## ⚠️ Required migration — deploy in lockstep
This PR renames a persisted feature-flag key/value and an env var. These
are **not** changed by the code alone and must be migrated when this
deploys, or affected orgs silently fall back to `cuid` minting (no crash
— `defaultValue: "cuid"`):
1. **Env var** (terraform): `RUN_OPS_MINT_KSUID_ENABLED` →
`RUN_OPS_MINT_ENABLED` (carry the value over).
2. **DB** `organization.featureFlags`: migrate both the key and value
together:
- key `runOpsMintKsuid` → `runOpsMintKind`
- value `"ksuid"` → `"runOpsId"`
Until an org's flag row is migrated, its `runOpsMintKind` lookup misses
and it mints `cuid` (legacy) — so no NEW-store ids for that org until
the data lands.
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
## Summary
The **Run ID** and **Batch ID** filters on the runs list, batches list,
and logs view rejected valid IDs. The input showed an error and the
**Apply** button stayed disabled, so filtering by an affected run or
batch ID from the dashboard was impossible.
The filter validators hard-coded exact friendly-id character lengths.
Friendly IDs come in three generations that all still exist in the data
(`<prefix>_` plus a 21-char nanoid, a 25-char cuid, or a 27-char ksuid),
and the hard-coded lengths never covered all three at once.
## Fix
All the ID filter validators (run, batch, waitpoint, schedule) now share
one helper, `makeFriendlyIdValidator`
(`apps/webapp/app/utils/friendlyId.ts`), which validates by prefix plus
a base62 body of any known generator length (21 / 25 / 27). The cuid and
ksuid lengths are sourced from core so the helper tracks any future
change to those formats. Unit tests assert it accepts the output of the
real id generators and rejects malformed input.
Downstream was already unaffected: run/batch route params and
URL-applied filters use unconstrained validation, so only the manual
filter inputs needed the fix.
## What
Adds the ability to **automatically migrate the dedicated run-ops
database** (the NEW DB in the run-ops split), matching how every other
database in the system is migrated. Follow-up to the run-ops split
activation.
## Changes
- **Migrate runner** — new
`internal-packages/run-ops-database/scripts/migrate.mjs`, exposed as
`db:migrate:deploy` / `db:migrate:status`. Connects via
`RUN_OPS_DATABASE_URL` (the same var the app uses) and expands `${VAR}`
refs like Prisma's dotenv.
- **Self-host** — `docker/scripts/entrypoint.sh` runs the run-ops
migration on boot when the DB is configured, gated by
`SKIP_RUN_OPS_MIGRATIONS`. Single-DB installs never set the URL, so it's
a clean no-op.
- **Single env-var family** — the run-ops DB is now addressed by one
canonical `RUN_OPS_*` family, connect path and migrations resolving the
identical URL:
- `RUN_OPS_DATABASE_URL` (writer) — replaces `TASK_RUN_DATABASE_URL`
- `RUN_OPS_LEGACY_DATABASE_URL` — replaces
`TASK_RUN_LEGACY_DATABASE_URL`
- `RUN_OPS_DATABASE_READ_REPLICA_URL` — replaces
`TASK_RUN_DATABASE_READ_REPLICA_URL`
- the old `TASK_RUN_*` aliases, the `??` coalesce, the
`runOpsNewDatabaseUrl` indirection, and the migrate-only `directUrl` are
all removed (consumers read `env.RUN_OPS_DATABASE_URL` directly).
`directUrl` was dropped because it was only ever used by `prisma
migrate` (never the app runtime) to bypass a pooler for advisory locks —
premature here since the run-ops connection isn't wired to the app yet.
If a pooler is later introduced for the app, a direct URL can be
reintroduced then.
## Safety
- **Pure rename** — nothing deployed sets any `TASK_RUN_*` var (the
split isn't activated anywhere yet; `.env.example`, docker-compose, and
cloud already use `RUN_OPS_*`), so there is no config migration.
- **Single-DB / self-host** — no new required env var; entrypoint and
migrate are no-ops when `RUN_OPS_DATABASE_URL` is unset.
- **Cloud** — runs migrations as pre-deploy ECS tasks (companion cloud
PR), calling these same `db:migrate:deploy` / `db:migrate:status`
commands.
## Verification
- Live migration against a fresh scratch DB with only
`RUN_OPS_DATABASE_URL` set: both migrations applied, no `P1012`/`P1013`;
`${VAR}` expansion, idempotent re-run, `status`, and no-op skip all
pass.
- Schema parity 4/4; `typecheck --filter webapp` 18/18; affected
split/replication tests 34/34.
## Scope
This delivers automatic migrations only. Enabling the app to *use* the
new DB (setting `RUN_OPS_DATABASE_URL` + `RUN_OPS_SPLIT_ENABLED` on the
service) is a separate activation step.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
## Problem
`LogicalReplicationClient` uses a Redlock leader lock to guarantee a
single active consumer per Postgres logical replication slot. The lock
resource was keyed on the client `name`:
```
logical-replication-client:${this.options.name}
```
A slot permits exactly one consumer, so the lock's job is to serialize
consumers **of a given slot**. Keying it on `name` breaks that whenever
two clients target the same slot with different names — most notably
across a rolling deploy where the client `name` changes but `slotName`
does not. Both acquire *distinct* locks, both consider themselves
leader, and the second to reach `START_REPLICATION` hits `replication
slot "<slot>" is active for PID <n>`. Because that query was
fire-and-forget and its failure was only logged (no retry), the consumer
stopped and replication stalled until the process was restarted.
## Fix
**1. Key the leader lock on `slotName`** — the actual single-consumer
resource:
```
logical-replication-client:${this.options.slotName}
```
Consumers of the same slot now contend on the same lock and hand off
cleanly across restarts/deploys; different slots stay independent.
`name` is kept for logging and the pg `application_name`.
**2. Self-healing resubscribe** (`resubscribeOnFailure`, opt-in) —
instead of logging-and-dying, a client re-subscribes with exponential
backoff after a lost election or a failed `START_REPLICATION`, so a
rolling deploy self-heals: the incoming pod retries until the draining
pod releases the slot, then takes over. Safety:
- `#cleanupAttempt()` unconditionally ends the pg client (freeing the
walsender) and releases the leader lock before rescheduling — retries
never leak connections/locks.
- `shutdown()` sets an intentional-stop latch re-checked after every
`await` in `subscribe()` (and aborts the lock-acquire spin), so a
resubscribe can never race or outlive an intentional shutdown.
- Backoff resets only on genuine stream start, so a permanently stuck
slot backs off to the ceiling and logs loudly rather than tight-looping;
an epoch guard neutralises stale `START_REPLICATION` catches.
Runs- and sessions-replication opt in and use `shutdown()` for all
intentional stops.
**3. Observability** — the admin runs-replication status route probed
the old name-keyed Redis key (would report `leader:false` for every
source after fix#1); now probes the slot-keyed key.
## Tests
`internal-packages/replication/src/client.test.ts` (real Postgres +
Redis containers):
- same-slot/different-name → second client must not double-lead or race
into "slot is active" (the regression)
- a failing `START_REPLICATION` retry loop must not leak connections or
locks
- `shutdown()` during an in-flight `subscribe()` must not leave a zombie
leader
- `subscribe()` after `shutdown()` re-arms `resubscribeOnFailure`
- self-heals once the leader releases the slot
Plus the multi-source wiring test updated to the slot-keyed lock keys.
## Rollout
With the self-healing resubscribe, this ships as a **plain rolling
deploy** — the incoming pods retry across the one-time lock-key
transition and take over once the old pods drain (a brief replication
stall that the durable slot replays on reconnect — no data loss). No
stop-before-start required.
## What
Extends the ClickHouse runs-replication service to fan in from multiple
Postgres sources (the control-plane DB and the run-ops DB) instead of a
single source, plus the admin operations to run and observe it.
- **Multi-source fan-in** (`services/runsReplicationService.server.ts`,
new `runsReplicationInstance.server.ts`,
`runsReplicationGlobal.server.ts`): factors the replication service into
per-source instances and a coordinator so a single ClickHouse target is
fed from more than one Postgres source.
- **Admin ops** (`routes/admin.api.v1.runs-replication.status.ts`,
`admin.api.v1.runs-replication.backfill.ts`,
`v3/services/adminWorker.server.ts`): adds a status endpoint reporting
per-source replication state and updates the backfill entrypoint for the
multi-source shape.
## Why
PR7 of the run-ops split stack, and the final piece: once run state can
live in a separate run-ops DB (earlier PRs), the analytics replication
into ClickHouse has to consume both sources so runs remain queryable
regardless of residency. Behavior-changing for the replication service
internals; the ClickHouse-facing output is unchanged (still one runs
stream), and single-source operation is preserved when the split is not
enabled.
## Tests
New vitest coverage: `runsReplicationInstance.test.ts` (per-source
instance behavior) and `runsReplicationService.part8`/`part9` suites
exercising the multi-source coordinator. Testcontainers-backed
(ClickHouse + Postgres); no mocks.
## Notes
Draft, **stacked on #4118** (`runops/pr06-write-path`). Review that
first; this diff is against it.
Server-change / changeset note to be added at stack-assembly time.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
## What
Routes the webapp write path through the run-ops split seam:
trigger/batch minting, idempotency-key resolution, and the run-lifecycle
services now determine residency and dispatch writes to the correct
store.
- **Trigger & batch** (`runEngine/services/triggerTask.server.ts`,
`batchTrigger.server.ts`, `createBatch.server.ts`,
`streamBatchItems.server.ts`, `v3/services/batchTriggerV3.server.ts`):
mint ids with the run-ops-aware minting and route creation/streaming
through the store; batch children inherit the parent's residency.
- **Idempotency** (`runEngine/concerns/idempotencyKeys.server.ts` + new
`idempotencyResidency.server.ts`): idempotency-key lookup/dedup is
residency-aware so a keyed retrigger resolves against the store that
owns the original run.
- **Run lifecycle services** (`createCheckpoint`,
`createTaskRunAttempt`, `enqueueDelayedRun`, `expireEnqueuedRun`,
`finalizeTaskRun`, `resumeBatchRun`, `cancelDevSessionRuns`,
`executeTasksWaitingForDeploy`, `triggerFailedTask`): resolve their
target run through the store rather than a fixed client.
- **Reads that fan out from writes** (`runsRepository` +
`clickhouseRunsRepository`, `BulkActionV2` + batch read-through,
realtime `sessions`/`runReader`, alerts
`deliverAlert`/`performTaskRunAlerts`): route through the read-through
resolver.
- `9535ae63d` — resolves the parent run through an injectable run store
in `TriggerFailedTaskService`.
- `bf8f7c881` — drops the "known-migrated" concept from write-path and
read repos; residency is id-shape only.
- `515b897ea` — self-defaults `resolveWaitpointThroughReadThrough` to
the safe run-ops clients.
## Why
PR6 of the run-ops split stack. This is the write-path counterpart to
the read foundation in the previous PRs: with it in place, both reads
and writes route through the seam. Additive when the split is disabled
(id-shape resolution collapses to the control-plane client);
behavior-changing on the minting, idempotency, and lifecycle paths when
enabled.
## Tests
Large new/expanded vitest suite under `apps/webapp/test/` and colocated
service tests: trigger-task and batch-trigger store routing, residency
inheritance, idempotency dedup residency + legacy-authority, bulk-action
read routing, cancel-dev-session routing, alerts store routing,
runs-repository read-through, realtime session/run-reader read-through
and stream-registration routing, and the waitpoint read-through default.
Testcontainers-backed; no mocks.
## Notes
Draft, **stacked on #4117** (`runops/pr05-webapp-foundation`). Review
that first; this diff is against it.
Server-change / changeset note to be added at stack-assembly time.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
## What
Wires the run-ops split into the webapp: database topology, environment
flags, split-mode gating, and the control-plane resolver/cache layer
that the run-store and run-engine seams from the previous PR plug into.
- **DB topology & env** (`apps/webapp/app/db.server.ts`,
`env.server.ts`, `entry.server.tsx`): adds the run-ops database
clients/topology and the environment variables that configure and gate
the split.
- **runOpsMigration module** (new
`apps/webapp/app/v3/runOpsMigration/`): the webapp-side machinery —
`splitMode.server.ts`, `controlPlaneResolver.server.ts` +
`controlPlaneCache.server.ts`, `readThrough.server.ts`,
`crossSeamGuard.server.ts`, `distinctDbSentinel.server.ts`, id-minting
helpers (`mintBatchFriendlyId`, `runOpsMintKind`,
`resolveInheritedMintKind`), `runOpsCascadeCleanup.server.ts`, the split
read gate, and route/unblock catalogs.
- **Store/engine wiring** (`app/v3/runStore.server.ts`,
`runEngine.server.ts`, `runEngineHandlers.server.ts` + new
`runEngineHandlersShared.server.ts`): points the webapp's store/engine
construction at the resolver, and factors shared handler logic out so
both seams use one path.
- **Read-path touch-ups**: `runtimeEnvironment.server.ts`,
`eventRepository/index.server.ts`, `taskRunHeartbeatFailed.server.ts`,
`engineVersion.server.ts` route their run/environment lookups
read-through the resolver.
- `413a94511` — interlocks split mode against the native realtime
backend so the two aren't enabled in an incompatible combination (see
`.server-changes/run-ops-split-realtime-interlock.md`).
- `dc74c57fd` — drops the earlier "known-migrated" read layer; residency
is determined by id-shape only.
## Why
PR5 of the run-ops split stack. This is the webapp foundation layer: it
stands up the DB topology, flags, and resolver/cache the rest of the
stack depends on, and repoints webapp read paths through the resolver.
Additive when the split is not enabled (existing single-DB behavior
preserved behind flags); behavior-changing on the read-through paths and
the realtime interlock.
## Tests
New vitest coverage across `apps/webapp/test/` and colocated
`*.server.test.ts` files: db topology, split mode, split read gate,
cross-seam guard, mint cutover / flip latency, control-plane cache,
control-plane resolver, distinct-db sentinel, read-through loaders
(route loaders, run-detail loaders, `findEnvironmentFromRun`), and the
run-engine handlers. Testcontainers-backed; no mocks. `pnpm-lock.yaml`
synced for the two new webapp deps.
## Notes
Draft, **stacked on #4116** (`runops/pr04-store-engine`). Review that
first; this diff is against it.
Server-change / changeset note to be added at stack-assembly time.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
## What
Introduces the run-store routing seam and the run-engine read seams that
let run lifecycle operations be dispatched to either the control-plane
database or a separately-generated run-ops database, depending on where
a run/batch resides.
- **run-store** (`internal-packages/run-store`): adds `runOpsStore.ts`
and substantially expands `PostgresRunStore.ts` so the store can resolve
residency and route reads/writes to the correct backing client.
`types.ts` grows the routing/residency types; `NoopRunStore.ts` is
removed.
- **run-engine** (`internal-packages/run-engine`): adds
`engine/controlPlaneResolver.ts` and routes the per-system read paths
(dequeue, enqueue, waitpoint, checkpoint, run-attempt, ttl, delayed-run,
execution-snapshot, pending-version, debounce, batch) through the
resolver/store instead of talking to a single Prisma client directly.
`engine/errors.ts`, `engine/types.ts`, and `engine/index.ts` are
extended to support injecting the store/resolver.
Three fixes are included on top of the seam work:
- `c6cadd85f` — routes read-your-writes to the owning store's
**writer**, not its lagging replica, so an operation immediately reading
back what it just wrote sees a consistent result.
- `05c912e05` — normalizes run-ops-generation Prisma errors to the
control-plane error class at the store **write boundary**, so
`instanceof` checks and the `P2002` → 422 handling continue to work
across the separately-generated run-ops Prisma client.
- `88d12907f` — resolves NEW-resident batches in
`ApiBatchResultsPresenter` by routing the batch read through the store,
so a dedicated-DB batch resolves instead of returning 404.
The change is heavily test-first: the bulk of the diff is new
unit/integration coverage for the store routing, residency, and each
run-engine system's control-plane resolver path.
## Why
PR4 of the run-ops split stack (PR1–PR3 land the ClickHouse
test-container and earlier plumbing). This PR is the read-path
foundation: it adds the seam and read-routing but leaves the write path
to route through the same seam in a later PR. Behavior-changing where
the three fixes above touch existing read-your-writes /
error-normalization / batch-resolution paths; otherwise additive (new
store module, new resolver, injectable dependencies with existing
single-client behavior preserved when no dedicated store is configured).
## Tests
Extensive new vitest coverage under `run-store/src/*.test.ts` (routing,
residency, dual-schema select, cross-generation error normalization,
read-after-write, idempotency dedup, mixed residency, waitpoint
co-location) and `run-engine/src/engine/**/*.test.ts` (per-system
`controlPlaneResolver` tests, injectability, block-edge residency,
waitpoint read residency, trigger-create routing, lifecycle router).
Testcontainers-backed; no mocks.
## Notes
Draft, **stacked on #4114** (`runops/pr03-clickhouse-tc`). Review that
first; this diff is against it.
Server-change / changeset note to be added at stack-assembly time.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Adds a lefthook `pre-push` job that runs the same checks as CI
code-quality (`oxfmt --check .` + `oxlint .`), so formatting/lint
failures are caught locally before they reach a PR. Uses the existing
lefthook setup - no new tooling.
Caveat noted in the config: GitButler uses its own git implementation
and only runs hooks when "Run hooks" is enabled in its per-project
settings; with that off, this protects plain `git push`. Enable that
setting to have it fire on `but push` too.
`db:migrate` ran `prisma migrate deploy` for
`@internal/run-ops-database`, which requires the dedicated run-ops DB
(:5434) that isn't up in a default local — breaking local `db:migrate`
for everyone; excluding it with `--filter=!@internal/run-ops-database`
restores it.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
## Summary
Getting the CLI talking to a local instance meant the browser magic-link
login, which is no good when you're driving things headlessly (an agent,
a container, or just no browser to hand). The seed already prints dev
secret keys for the batch-limit orgs, so it now also mints a personal
access token for the seeded `local@trigger.dev` user and prints a
ready-to-run `export TRIGGER_ACCESS_TOKEN=...` next to them.
Re-seeding stays idempotent: it decrypts and reprints the existing
`local-dev-cli` token rather than piling up a new one on every run.
<!-- GitButler Footer Boundary Top -->
---
This is **part 2 of 2 in a stack** made with GitButler:
- <kbd> 2 </kbd> #4135👈
- <kbd> 1 </kbd> #4137
<!-- GitButler Footer Boundary Bottom -->
---------
Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>
<!-- ccr-slack-attribution -->
_Requested by **Eric Allam** · [Slack
thread](https://triggerdotdev.slack.com/archives/C061L2MHW93/p1783083021892389?thread_ts=1783083021.892389&cid=C061L2MHW93)_
## ✅ Checklist
- [ ] I have followed every step in the [contributing
guide](https://github.com/triggerdotdev/trigger.dev/blob/main/CONTRIBUTING.md)
- [x] The PR title follows the convention.
- [ ] I ran and tested the code works
---
## Testing
N/A — this change only removes two changeset markdown files; there are
no code changes to test.
---
## Changelog
### What changed
Removes two `@trigger.dev/core` changesets that were added for changes
that are not user-facing package changes:
- `.changeset/runops-core-residency.md` — internal run-ops residency
classifier + ksuid mint/decode primitives
- `.changeset/telnet-dev-logs.md` — dev-only
`@trigger.dev/core/v3/telnetLogServer` module
### Why
Per the convention that changesets should only be added when there are
actual user-facing package changes, these two do not qualify. Removing
them keeps the release (currently the automated v4.5.1 PR #4126) from
bumping `@trigger.dev/core` for internal/dev-only additions.
---
_Generated by [Claude
Code](https://claude.ai/code/session_019xCdLwoozZm4Gr5ZHLYiws)_
Co-authored-by: Claude <noreply@anthropic.com>
## Summary
Stable v4 Docker image builds now also publish `v4` and `latest` tags,
giving Docker-based self-hosters a maintained floating tag to use after
a stable release. The Kubernetes guide now uses the current Helm chart
line and current pinned examples, so new installs and upgrades resolve
to the 4.5 chart line instead of the 4.0 line.
## Design
The publish workflows add the floating tags only for stable `v4.x.x`
image tags. Prerelease and `main` builds keep their existing tags.
## What
- Adds `composeTaskRunVersion` to `@internal/clickhouse` (exported from
the package index). It packs a small `originGeneration` epoch into the
top 8 bits of the ReplacingMergeTree version and keeps the producer's
own LSN in the low 56 bits, so `task_runs_v2` rows replicated from more
than one Postgres producer become globally comparable while preserving
in-producer ordering. Single-producer setups never call it and keep
using the raw LSN version.
- Adds unit coverage for the helper in `taskRuns.test.ts` (bit layout,
ordering, epoch precedence, range validation).
- Adds two run-engine tests that exercise the cross-Postgres-version
testcontainer fixture:
- `heteroPostgresFixture.test.ts` — a smoke test asserting byte-identity
and identical `ORDER BY` (under a pinned ICU collation) across two
different Postgres major versions.
- `crossVersionCompat.test.ts` — mirrors the run-engine's real raw-SQL
surfaces and asserts byte-identical, ordering-identical results across
the two versions. Ships with an env-gated block (skipped by default)
that can be pointed at a real dedicated database in CI.
## Why
Third PR in the run-ops split stack. It is purely additive: it
introduces one new exported helper plus tests and changes no runtime
call sites, so on its own it has no runtime behavior change. It lays
down the version-composition primitive and the cross-version
compatibility proof that later PRs in the stack rely on.
## Tests
- New unit tests for `composeTaskRunVersion` in
`internal-packages/clickhouse/src/taskRuns.test.ts` (run against a real
ClickHouse testcontainer).
- New cross-version tests in
`internal-packages/run-engine/src/engine/tests/` running against real
Postgres containers of two different major versions (no mocks). The
env-gated dedicated-database block is skipped unless its URL is set.
## Notes
Draft, **stacked on #4113** (`runops/pr02-db-foundation`). Review that
first; this diff is against it.
Server-change / changeset note to be added at stack-assembly time.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
## Summary
`pnpm run db:seed` failed with `SyntaxError: The requested module
'./app/models/organization.server' does not provide an export named
'createOrganization'`, even though that export exists. Renaming the seed
entry
point from `seed.mts` to `seed.ts` runs it as CommonJS and fixes the
failure.
No seed logic changes.
## Root cause
`seed.mts` is an ES module, but the server modules it imports (`.ts`
files, and
no package declares `"type": "module"`) resolve as CommonJS. tsx
compiles those
to CommonJS using esbuild's getter-based export shape
(`Object.defineProperty(exports, name, { get })`), which Node's
`cjs-module-lexer` does not detect when it links the ESM importer. The
named
exports look absent, so linking throws before any code runs.
The seed only ever needed the `.mts` extension: it has no top-level
`await` and
no `import.meta`. Running it as `.ts` keeps the whole import chain
CommonJS to
CommonJS and never crosses the lexer boundary. Verified the seed runs to
completion after the change.
Possibly caused by Node 22 upgrade?
## What
The **dedicated run-ops database** foundation for the split: a
standalone Prisma package plus the infra to run and migrate it.
- **`internal-packages/run-ops-database`** — a new Prisma package
(`@internal/run-ops-database`) whose schema mirrors the run-execution
tables that will live on the dedicated DB, with its own generated
client, migrations, and migration runner.
- **`prisma/schema.parity.test.ts`** — a parity test that guards the
run-ops schema against drift from the control-plane schema for the
mirrored tables.
- **Docker** — a Postgres 17 service (`docker/Dockerfile.postgres17`,
`docker/docker-compose.yml`) so the dedicated DB is available locally
under the run-ops compose profile.
- **Testcontainers** — hetero fixtures (PG14 legacy + PG17 dedicated) so
later PRs can exercise cross-database behaviour with real containers
rather than mocks.
## Why
This is the **second PR in the run-ops split stack**, stacked on the
core primitives. It stands up the dedicated database and its tooling.
There is **no runtime wiring** into the webapp here — the app does not
read or write this DB yet; that arrives in later PRs. On its own this PR
only adds a package, a docker service, and test fixtures.
## Tests
Schema-parity test for the run-ops schema; hetero testcontainer fixture
smoke test.
## Notes
- Draft, **stacked on #4112** (`runops/pr01-core-residency`). Review
that one first; this diff is against it.
- Server-change / changeset note to be added at stack-assembly time.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
## What
Foundation for the run-ops database split: an isomorphic **id-shape
residency classifier** and the **ksuid mint primitives**, added to
`@trigger.dev/core` under `v3/isomorphic`.
- **`runOpsResidency.ts`** — classifies a run id by its shape: 25-char
cuid → `LEGACY`, 27-char ksuid → `NEW`. Pure and environment-free (safe
on both client and server).
- **`friendlyId.ts`** — ksuid mint primitives and id helpers.
- Both exported via `v3/isomorphic/index.ts`.
## Why
This is the **base of a stacked series** implementing the run-ops DB
split (routing run-execution data to a dedicated database by id-shape).
Later PRs in the series consume this classifier and these primitives to
route reads and writes across the two databases.
On its own this PR is **purely additive** — new isomorphic helpers with
unit tests, no runtime wiring, and no behaviour change to existing code
paths.
## Tests
Unit tests for the classifier (`runOpsResidency.test.ts`) and the id /
mint primitives (`friendlyId.test.ts`).
## Notes
- Draft, stacked on `main`; subsequent PRs in the series build on top of
this one.
- A changeset for `@trigger.dev/core` will be added before this is
marked ready for review.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The dashboard agent deploy runs a matrix leg per environment (staging,
prod). Running them in parallel deployed the same Trigger.dev project
twice at once, and one leg failed to create the background worker while
the other succeeded. `max-parallel: 1` runs the legs sequentially so a
deploy never races itself.
Also aligns the workflow's Node version to the repo standard (`22.23.1`,
per `.nvmrc`).
Verified: re-running the staging leg on its own (no parallel prod)
deployed cleanly.
## Summary
The dashboard agent runs as a `chat.agent` in its own Trigger.dev
project, deployed separately from the app that starts its sessions. This
adds independent, non-disruptive deploys for it: a new workflow deploys
the agent with `--skip-promotion` (so a deploy never becomes the current
version on its own), and the app pins its sessions to a chosen version.
## How it works
`DASHBOARD_AGENT_VERSION` (unset by default) pins agent sessions to a
specific deployed version; when unset, sessions run on the project
environment's current version. The pin is passed on session start and
head start and is forwarded to every continuation run, so a pinned
session stays on its version for its whole life. Cutting over to a new
build becomes a config change (set the version) rather than a redeploy,
and rollback is flipping it back.
The workflow (`dashboard-agent-deploy.yml`) runs a leg per environment
(staging and prod), each gated by its own environment, and triggers on
pushes to `main` that touch the agent or its store (also available via
manual dispatch). Deploy versions are per-environment, so each
environment pins to its own leg's version.
Follow-up to #4120, adding regression coverage for the pause/resume path
in `PauseEnvironmentService`.
Three tests against the real service with testcontainers Postgres (no
mocks), seeded org/project/environment rows, and the same
`AuthenticatedEnvironment` coercion production uses:
1. **resumes a manually paused env** - the actual regression: pause
leaves `pauseSource` null, resume must succeed. Verified this fails with
the exact pre-#4120 symptom (false billing-limit error) when the fix is
reverted locally.
2. **rejects resume of a billing-limit paused env** - the guard still
blocks manual resume while `pauseSource = BILLING_LIMIT`, and the env
stays paused.
3. **manual pause while billing-limit paused is a no-op** - returns
success without overwriting `pauseSource`, so billing-limit converge can
still find and unpause the environment.
Tests-only change, no runtime behavior touched.
Stream dev logs over a local telnet/TCP socket. `trigger dev` mirrors
its terminal output on port 6767 by default (override with
--telnet-logs-port or TRIGGER_DEV_TELNET_LOGS_PORT, 0 disables). webapp,
supervisor, and coordinator each expose an opt-in stream gated on a
per-service *_TELNET_LOGS_PORT env var. New
@trigger.dev/core/v3/telnetLogServer module (localhost-only,
backpressure-safe, plain-text) plus optional static Logger.onLog /
SimpleStructuredLogger.onLog sinks.
Then you (or your agent) can use `nc` to connect and filter out the
stream.
<img width="1103" height="239" alt="image"
src="https://github.com/user-attachments/assets/b4d47efc-8a57-4185-a159-10f2806627ae"
/>
Makes the app error page always render full screen — it previously
inherited the width/offset of whatever container the error boundary was
mounted in (e.g. the centered `max-w-xs` column in the root boundary),
so `min-h-screen` alone couldn't fill the viewport. The root container
now uses `fixed inset-0 z-50` to break out and cover the full screen
regardless of nesting.
Also changes the "Go to homepage" shortcut from `Cmd/Ctrl+G` (which
collides with the browser's native "Find Again") to `Enter`.
## Bug
Manually pausing an environment works, but resuming it always fails
with:
> This environment is paused because your organization reached its
billing limit. Resolve the limit on the billing limits settings page to
resume.
even when no billing limit is in effect. Once paused by a user, an
environment cannot be resumed at all.
## Root cause
A manual pause leaves `RuntimeEnvironment.pauseSource` as `NULL` (only
billing-limit enforcement sets `BILLING_LIMIT`). The resume path in
`PauseEnvironmentService` guards its `updateMany` with:
```ts
NOT: { pauseSource: EnvironmentPauseSource.BILLING_LIMIT }
```
Prisma's `NOT` on a nullable field translates to SQL `!=`, which
excludes `NULL` rows. So the update matches zero rows for every
user-paused environment, and the zero-count branch (meant to catch a
race with billing-limit pausing) returns the misleading billing-limit
error.
Introduced in #3996 (the guard is correct for `BILLING_LIMIT` rows; it
just also swallows `NULL`).
## Fix
Explicitly include `pauseSource: null` rows:
```ts
OR: [
{ pauseSource: null },
{ NOT: { pauseSource: EnvironmentPauseSource.BILLING_LIMIT } },
]
```
Billing-limit-paused environments are still blocked from manual resume,
both by the `getManualPauseEnvironmentResult` guard and by this clause.
## Verification
Reproduced locally: paused an environment via `PauseEnvironmentService`
(DB shows `paused = true`, `pauseSource = NULL`), resume returned the
billing-limit error with `updateMany` matching 0 rows. With the fix,
resume succeeds and the environment unpauses. Billing-paused rows remain
excluded by the same clause.
## Summary
Long task names in the task landing page side menu pushed the **Test**
button off the edge of the panel instead of truncating. The heading now
truncates with an ellipsis so the Test button always stays in view, on
the standard and agent task pages.
## Root cause
The side menu lives in a fixed-width resizable panel with `overflow:
hidden`. Its header row is a grid item, and a grid item's default
`min-width: auto` lets it grow to its content's width. The title
`<span>` uses `truncate` (`white-space: nowrap`), whose min-content is
the full, untruncated name, so the header row expanded past the panel
and the Test button was clipped off the edge.
Adding `min-w-0` to the header container lets it shrink back to the
panel width so the title truncates. The scheduled task page already had
this class; the standard and agent pages did not.
Bumps the internal/toolchain Node version to the latest 22.x LTS
(`22.23.1`) and standardises it across the repo. Scope is the **platform
toolchain + the repo's own runtime images** (all `20 → 22` *upgrades*,
off the now-EOL node 20).
### Main changes
- Node `20.20.2 → 22.23.1` across all CI workflows, `.nvmrc`,
`CONTRIBUTING.md`, and the OSS `docker/Dockerfile` (digest-pinned).
- `@types/node → 22.20.0` (root dep + pnpm `overrides`, so the whole
workspace resolves to it); lockfile regenerated.
- `sdk-compat` matrix: adds Node 24 + 26 (keeps 20, still in `engines`).
- **App runtime images → node 22** (were on EOL node 20):
`apps/coordinator` → `node:22.23.1-bookworm-slim`;
`apps/docker-provider` + `apps/kubernetes-provider` → `node:22-alpine`
(reusing the exact digest `apps/supervisor` already runs, so all four
worker images are now identical). Stage aliases renamed off `node-20`.
### Possible issues / test notes
- `@types/node` 22.x can surface new TS errors — typecheck (now on 22)
is the gate.
- **Smoke-test the v3 worker path** — `coordinator` (`crictl`/CRI calls)
and the docker/kubernetes providers (talking to their daemons) now run
on node 22 (alpine/musl for the providers). Upgrade off EOL so low-risk,
but it's deployed runtime code with its own `publish-worker.yml`
pipeline.
## Summary
Two related timezone bugs in the dashboard.
1. The date/time tooltip could show a UTC offset label that contradicted
the time it displayed. A viewer whose machine clock differs from their
saved timezone (or when a date falls in the other DST phase) would see
something like `Local (UTC +0)` next to a value that isn't at +0.
2. A user's timezone preference silently failed to save whenever their
browser reported a zone like `UTC`, `Etc/UTC`, or `Asia/Kolkata`,
leaving their timestamps stuck in a previously-saved timezone.
## Offset label
The "Local" row formatted its time using the viewer's configured
timezone but computed the `(UTC +n)` label from `new
Date().getTimezoneOffset()`, the browser's offset at the current moment.
Those are two independent sources, so they disagreed when the configured
timezone differed from the machine, and also when the displayed date was
in the opposite DST phase. The label is now derived from the same date
and timezone used to render the row (via `Intl.DateTimeFormat` with
`timeZoneName: "longOffset"`), so it always matches the displayed time.
## Preference persistence
`/resources/timezone` validated the incoming zone against
`Intl.supportedValuesOf("timeZone")`, which lists only canonical zone
ids. Browsers report zones that aren't in that list via
`resolvedOptions().timeZone`, notably `UTC` (and `Etc/UTC`,
`Asia/Kolkata`, `GMT`), so those requests returned 400 and the
preference was never stored. Validation now checks whether the runtime
can resolve the zone at all, which accepts every real zone and still
rejects invalid input.
Added unit tests for both.
## What & why
Two related fixes to how new cloud organizations get onboarded onto the
Free plan.
### 1. Route new cloud orgs through plan selection
New cloud organizations were created already activated, so they skipped
the plan-selection step and went straight to creating projects — which
meant their plan and usage limits were never set up. They're now created
deactivated and routed through plan selection, which activates them once
a plan is chosen. Self-hosted installs have no plan-selection step, so
they're activated immediately on creation and are unaffected.
The `Organization.v3Enabled` field is renamed to `isActivated` to better
describe what it now gates. It's mapped to the existing `v3Enabled`
column, so there's no data migration — only a schema/code rename.
### 2. Allow selecting the Free plan without GitHub verification
Choosing the Free plan no longer requires connecting and verifying a
GitHub account. The plan is applied immediately when selected. This
removes:
- the "Connect to GitHub" dialog and the GitHub-verified badge from the
plan picker
- the account-rejected state
- the now-unreachable GitHub-connect return routes
## Notes
- These changes pair with the corresponding change in the billing
service that applies the Free plan directly; they should be released
together.
## Testing
Verified locally end to end: a new cloud org is routed to plan
selection, the Free plan applies in one click with no GitHub step, the
org is activated, its usage allowance is provisioned, and it lands on
the new-project page.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Once this is merged, oxlint is at a pretty sensible baseline.
**Enable `no-unused-vars`, `typescript/consistent-type-imports`, and
`import/no-duplicates` lint rules**
Turns on three previously-disabled oxlint rules across the monorepo and
fixes all violations:
- **`no-unused-vars`** – enabled as an error with standard ignore
patterns: unused function arguments are ignored by default (`args:
"none"`), variables/caught errors/destructured array elements prefixed
with `_` are allowed, and rest siblings are permitted.
- **`typescript/consistent-type-imports`** – enforced as an error; all
type-only imports now use the `import type` syntax.
- **`import/no-duplicates`** – enforced as an error; duplicate import
statements from the same module have been merged.
The remaining commits clean up the violations found across the codebase:
removing unused variables/imports/type aliases, adding `_` prefixes to
intentionally unused bindings, fixing duplicate imports, and converting
value imports to `import type` where appropriate.
# Trigger.dev v4.5.0
4.5.0 is the GA of the AI Agents platform. Everything built during the
prerelease line (durable agents, Sessions, AI Prompts) is now stable on
the `latest` tag, alongside a set of SDK and runtime improvements.
## AI Agents (`chat.agent`)
Run Vercel AI SDK chat completions as durable Trigger.dev tasks instead
of fragile API routes. A conversation runs as one long-lived task keyed
on `chatId`, so it survives page refreshes, network blips, redeploys,
and crashes, and every turn is a span in the dashboard.
```ts
import { chat } from "@trigger.dev/sdk/ai";
import { streamText, stepCountIs } from "ai";
import { anthropic } from "@ai-sdk/anthropic";
export const myChat = chat.agent({
id: "my-chat",
run: async ({ messages, signal }) => {
return streamText({
...chat.toStreamTextOptions(), // system prompt, compaction, steering, telemetry
model: anthropic("claude-sonnet-4-5"),
messages,
abortSignal: signal,
stopWhen: stepCountIs(15),
});
},
});
```
## Sessions
The durable primitive underneath `chat.agent`, usable on its own: a
run-aware, bidirectional stream channel keyed on a stable `externalId`
whose `.in` / `.out` streams survive run boundaries (suspend, crash,
idle-timeout, redeploy). One Session spans many runs, which makes it a
good fit for agent inboxes and approval flows.
```ts
import { sessions } from "@trigger.dev/sdk";
// Create the session and trigger its first run (idempotent on externalId)
await sessions.start({
type: "inbox",
externalId: userId,
taskIdentifier: "inbox-agent",
});
const session = sessions.open(userId);
await session.in.send({ text: "hello" });
const stream = await session.out.read({ signal: AbortSignal.timeout(30_000) });
for await (const chunk of stream) console.log(chunk); // durable across run swaps
```
## AI Prompts
Define prompt templates as code, versioned on every deploy, and override
the text or model from the dashboard without redeploying
(environment-scoped). Each generation links back to its prompt version
for usage, cost, and latency.
```ts
import { prompts } from "@trigger.dev/sdk";
import { z } from "zod";
export const supportPrompt = prompts.define({
id: "customer-support",
model: "gpt-4o",
variables: z.object({ customerName: z.string(), issue: z.string() }),
content: `You are a support agent for Acme.
Customer: {{customerName}}
Issue: {{issue}}`,
});
// Honors any active dashboard override, else the current deployed version
const resolved = await supportPrompt.resolve({ customerName: "Alice", issue: "Can't log in" });
// resolved.text, resolved.model, resolved.version
```
## `useChat` integration
`useTriggerChatTransport` is a Vercel AI SDK `ChatTransport` that runs
`useChat` over Trigger.dev realtime with no API routes. Text, tool
calls, reasoning, and `data-*` parts stream natively, and it works with
AI SDK v5, v6, and now v7.
## First-turn fast path (`chat.headStart`)
Runs the first turn in your warm server process while the agent boots in
parallel, cutting cold-start time-to-first-chunk roughly in half
(measured ~2.8s to ~1.2s). Available via the new
`@trigger.dev/sdk/chat-server` subpath.
## Human-in-the-loop, stop, and steering
The agent control surface: tool approvals (`needsApproval` +
`addToolApprovalResponse`), client-driven stop-generation, mid-execution
steering (`pendingMessages`), and between-turn context injection
(`chat.inject` / `chat.defer`), all durable across the conversation.
## Agent Skills
`skills.define({ id, path })` bundles a `SKILL.md` folder into your
deploy image. The agent gets a one-line summary up front and loads the
full instructions plus scoped `bash` / `readFile` tools on demand
(progressive disclosure), so a capability is something the model reaches
for rather than a pre-declared typed tool.
## `trigger skills` for coding assistants
`trigger skills` installs version-pinned Trigger.dev skills plus a
bundled docs snapshot into Claude Code, Cursor, GitHub Copilot, and
Codex, so your assistant's Trigger.dev knowledge stays current with your
installed SDK version. `trigger init` now offers to set up the MCP
server and skills too.
## Model library
A new Models page in the dashboard: a catalog of models grouped by
provider with context window, capabilities, and input / output pricing
per 1M tokens, plus a "Your models" tab showing per-model usage, cost,
and cache-hit sparklines from your actual traffic.
## Dev branches
Run multiple local `trigger dev` sessions in parallel (separate git
worktrees or coding agents) without runs colliding, each isolated with
its own dashboard, via `trigger dev --branch <name>`.
## `TriggerClient`
An instantiable client so one process can trigger and read across
projects, environments, and preview branches, each with its own auth and
baseURL, with no shared global state.
```ts
import { TriggerClient } from "@trigger.dev/sdk";
const prod = new TriggerClient({ accessToken: process.env.TRIGGER_PROD_KEY });
const preview = new TriggerClient({
accessToken: process.env.TRIGGER_PREVIEW_KEY,
previewBranch: "signup-flow",
});
await prod.tasks.trigger("send-email", { to: "user@example.com" });
await preview.runs.list({ status: ["COMPLETED"] });
```
## SDK and runtime
- AI SDK 7 support (v5 and v6 still supported), with OpenTelemetry
telemetry auto-wired
- Large trigger-payload offload: trigger payloads at or above 128KB
upload to object storage automatically, using the same auth and baseURL
as the trigger call
- Region support on the runs API: filter runs by region and read each
run's executing region (also on MCP `list_runs`)
- Duplicate task-id detection: `dev` and `deploy` fail with a clear
error instead of silently overwriting
- `envvars.upload` gains an `isSecret` flag to import redacted secret
variables
- Retry hardening: `TASK_MIDDLEWARE_ERROR` now retries under the task's
retry policy
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Removes the release-candidate framing from the docs for the 4.5 GA: the
`@rc` install caveats on the `skills` CLI command, and the AI Agents and
Prompts release-candidate banner (a shared snippet used across the
ai-chat and prompts pages) plus the `>=4.5.0-rc.0` compatibility pin in
the AI reference.
## Summary
Adds the `4.5.0` GA entry to the AI chat agents changelog covering the
`chat.agent` changes in that release: a new `apiClient` option on
`chat.headStart` and `chat.createStartSessionAction` for pointing chat
sessions at a different project or environment, the fix for messaging
chat agents deployed to a preview branch, and the fix for Head Start
handovers when the agent also defines a `prepareMessages` hook.
All four changes are from
[#4018](https://github.com/triggerdotdev/trigger.dev/pull/4018).
Bumps the `@remix-run/*` family in the webapp from 2.17.4 to 2.17.5 to
keep dependencies current.
2.17.5 pulls `@remix-run/router` 1.23.2 → 1.23.3, so the local
route-matching perf patch was rebased onto 1.23.3 (regenerated via `pnpm
patch`). It is functionally identical to the previous one - the only
difference is that 1.23.3 already hoists `decodePath` out of the match
loop upstream, so that hunk is dropped; the per-route-tree branch cache
and the compiled-path cache are unchanged. Also updated the
`@remix-run/dev>tar-fs` override key to track the new dev version.
Verified locally against latest main: `typecheck --filter webapp`
passes, `--frozen-lockfile` is consistent, and the dev server boots and
server-renders pages cleanly (route matching exercised via the patched
router).
## ✅ Checklist
- [ ] I have followed every step in the [contributing
guide](https://github.com/triggerdotdev/trigger.dev/blob/main/CONTRIBUTING.md)
- [x] The PR title follows the convention.
- [ ] I ran and tested the code works
---
## Testing
`pnpm run typecheck --filter @internal/clickhouse` passes. This only
adds a ClickHouse input-format setting to existing insert calls; the
setting affects type inference for newly-inserted/merged data and is
non-destructive to existing rows.
---
## Changelog
Sets
`input_format_json_infer_array_of_dynamic_from_array_of_different_types
= 1` on every native-JSON insert path:
- `task_runs_v2` (`output`, `error`) — `insertTaskRuns`,
`insertTaskRunsCompactArrays`, and the async-insert variants
- `task_events_v1` / `task_events_v2` (`attributes`)
- `metrics_v1`
- `sessions_v1`
### Why
Our JSON columns contain arrays with mixed element types (e.g.
`[{"key":"value"}, "string", "string"]`). With this setting off — which
is the effective default under `24.12` compatibility — ClickHouse infers
those as deeply nested unnamed `Tuple(JSON, Nullable(String), …)` types.
ClickHouse 26.2 introduced `input_format_binary_max_type_complexity`
(default 1000), and those tuple type trees exceeded the limit, causing
background merges to fail with **Code 117**.
With the setting on (the default since 25.8), mixed-type arrays are
inferred as a single `Array(Dynamic)` — a simpler, flatter type
representation that never approaches the complexity limit, even once the
upstream default limit is restored.
Setting this explicitly at insert time keeps behavior deterministic and
version-controlled, so it does not depend on the server profile or a
future compatibility bump. This is a forward-only change: it only
affects newly inserted/merged data and does not rewrite existing parts.
Our read path re-serializes these columns to strings (`toJSONString` via
the materialized `*_text` columns), so the internal Tuple →
Array(Dynamic) representation change is transparent to the application.
### Companion server-side setting
To also apply this on the ClickHouse side (covers merges and any writes
not going through these code paths), set it on the default user:
```sql
ALTER USER default SETTINGS input_format_json_infer_array_of_dynamic_from_array_of_different_types = 1;
```
---
## Screenshots
_N/A_
💯🤖 Generated with [Claude Code](https://claude.com/claude-code)
https://claude.ai/code/session_01AaChyhestFMBYBWh6bgcCF
---
_Generated by [Claude
Code](https://claude.ai/code/session_01AaChyhestFMBYBWh6bgcCF)_
---------
Co-authored-by: Claude <noreply@anthropic.com>
## Summary
Fixes flaky CLI v3 E2E tests by removing fixture-level parallelism.
The suite was using `describe.concurrent`, but each fixture test mutates
its fixture workspace during setup: removing `node_modules`,
renaming/restoring lockfiles, and running package installs. On
Windows/npm this can race or hit file-lock/cache contention, causing
intermittent failures.
## Fix
Run the CLI v3 E2E fixtures serially instead of concurrently.
## Expected impact
This should make the Windows/npm E2E job more stable. The E2E step may
slow down from ~30s to roughly ~60–90s in typical runs, with a
conservative upper bound around ~2 minutes.
Fixes#4046
## Problem
When a single run creates more than 1000 idempotency keys (e.g. a large
batch trigger where each item calls `idempotencyKeys.create()`), the
original key and scope metadata is silently dropped for all but the most
recent 1000 keys.
`idempotencyKeys.create()` returns a plain 64-char hash and stores the
`{ key, scope }` mapping in an in-process catalog keyed by that hash.
That catalog was a fixed-size **LRU capped at 1000 entries**. Once a run
creates more than 1000 keys, the earliest mappings are evicted, so when
the SDK later looks them up to attach `idempotencyKeyOptions` to the
trigger call, it finds nothing and sends `undefined`. The affected runs
then:
- report `ctx.run.idempotencyKey` as the raw hash instead of the
user-provided key
- have no `idempotencyKeyScope`
- show empty `idempotency_key` / `idempotency_key_scope` in the
dashboard and analytics
Deduplication still works (the hash is intact); only the human-readable
metadata is lost, which makes the failure silent and hard to notice.
## Fix
- Replace the LRU catalog with an unbounded in-memory catalog, so every
key created within a run keeps its metadata regardless of how many are
created.
- Clear the catalog at each run boundary via
`resetExecutionEnvironment()` (both dev and managed workers), matching
how every other per-run manager is reset. Deployed workers reuse one
process across many runs (warm starts), so this bounds memory to a
single run's keys instead of accumulating across runs — which is the
reason the size cap existed in the first place.
## Tests
- New public-API test creates 3000 keys and asserts all of them
(including the first) retain their key/scope — this fails on `main` and
passes with the fix.
- New test for the in-memory catalog covers store/retrieve/overwrite,
large-N retention (no eviction), and `clear()`.
- New test asserts the catalog is emptied after a run-boundary reset.
- Replaces the previous LRU catalog + its eviction tests.
Verified: `@trigger.dev/core` and `trigger.dev` both build; all
idempotency tests pass. Changeset added (patch).
The `Exec` helper in `@trigger.dev/core` logs command args at debug
level (and in its output/error metadata). For commands that take a
credential directly on the command line - `--password`, `--token`,
`--secret`, etc. - that value is logged verbatim, so turning on debug
logging can surface secrets in log sinks.
This masks the value of known credential-bearing flags (both `--flag
value` and `--flag=value` forms) before the args are logged. The
executed command is untouched - only the logged copy is redacted. Added
a small unit test for the redaction helper.
Adds a `redis_worker.queue.oldest_message_age` observable gauge (labeled
`worker_name`) and `SimpleQueue.oldestMessageAge()`, reporting the age
of the
oldest overdue message in each queue. Generic queue-stall signal: 0
while a
queue drains healthily, rising only when due work sits undrained
(blocked
dequeue, dead consumer, backpressure) — even when no items are being
processed.
## Summary
Upgrades the dashboard form layer from `@conform-to` 0.9 to 1.x. No
behaviour change is intended; this is the conform API migration only.
conform 1.x peer-depends on `zod` `^3.21 || ^4`, so it runs on the
current zod 3 and is a prerequisite for upgrading the repo to zod 4:
conform 0.9 imports `ZodNativeEnum`, `ZodEffects`, and `ZodPipeline`,
all removed in zod 4, so the webapp cannot build against zod 4 until
conform is on 1.x. Landing this first (on zod 3) keeps the zod 4 PR
focused on zod alone.
## Testing
Did a bunch of local smoke tests and E2E playwright tests, several
rounds of different reviewers, all clean.
Invite acceptance could fail for cloud organizations with many projects
because the whole flow ran inside a single transaction and did too much
work before it completed. In larger orgs, that pushed the transaction
past its timeout and blocked the invite from being accepted.
This PR moves the expensive parts of invite acceptance out of the
transaction, excludes deleted projects from environment setup, fixes
error handling on /invites, and adds regression coverage for the failure
cases.
Fixed trace rendering for child and nested runs in large traces.
Dashboard and trace API responses now load the requested run's trace
subtree instead of depending on the run span appearing in the initial
trace slice.
## Summary
Two robustness fixes in the dashboard's error handling, found while
testing the billing-limit pause/resume flow.
## Toast cookie overflow
Toast messages are flashed into the `__message` session cookie, which
the session store rejects once the serialized cookie passes the
browser's ~4KB limit. Any call site that flashes a raw caught error (a
verbose database or validation message, for example) could turn a toast
into a failed request. `setErrorMessage` / `setSuccessMessage` now clamp
the message length, so a toast can never overflow the cookie. This
protects every toast helper at once.
## Pause/resume reporting
Resuming an environment that is paused by a billing limit is an
expected, user-actionable state, but `PauseEnvironmentService` threw it,
and the service's catch reports every throw at error level. It now
returns that case as a failure result, so callers still surface the
message to the user while genuine errors keep reporting.
## Summary
When a scheduled task fires for an organization that is out of
entitlements, the trigger can't proceed. That's an expected outcome, but
it was being logged at error level and surfaced as a failure.
## Fix
The trigger callback now classifies an out-of-entitlements result as its
own error type (`OUT_OF_ENTITLEMENTS`), and the schedule engine logs
both that and the existing queue-limit result as warnings rather than
errors. The run still doesn't fire and the `schedule_execution_failure`
metric still records the outcome (now tagged `out_of_entitlements`), so
nothing about observability or behavior changes beyond the log level.
## Summary
The billing limits page and the usage-limit banners now require a
dedicated `manage:billing-limits` permission instead of the broader
billing permission. This lets a role be granted control over billing
limits independently of subscription and payment management.
## Details
Both the loader and action of the billing limits settings page check
`manage:billing-limits`. The "Configure billing limit" and "Resolve"
actions in the limit banners (the no-limit-configured, grace, and
rejected states) gate on the same permission. The subscription and
billing pages keep using the existing billing permission.
## Summary
When replaying runs in bulk from a deployed environment, you can now
choose which region the replayed runs run in. The bulk action inspector
shows an "Override region" dropdown that defaults to "Don't override",
which keeps each run in its original region, so replaying a selection
that spans multiple regions doesn't silently re-route anything. Pick a
region and every matched run is replayed there instead.
The dropdown only appears for the replay action in a deployed
environment with more than one region available; cancel actions and
development environments don't show it.
## Design
The selected region is carried through the bulk action as a dedicated
`replayRegion` param, kept separate from the run-list selection filters
so it can't be confused with a region selection filter. When the action
runs, each replay passes it through to the existing region override on
the replay service, which already falls back to each run's original
region when no override is set. "Don't override" is a sentinel value
that the action normalizes away so the service only ever sees a real
region or nothing.
---------
Co-authored-by: Eric Allam <eric@trigger.dev>
## Summary
Improves and unifies the run-activity charts by extracting a shared set
of chart primitives and adopting them on the three task landing pages
(agent, standard, scheduled), with the density and label fixes also
carried over to the dashboard and custom query charts.
Main changes a reviewer should know about:
- **Shared primitives (DRY).** New `ChartCard` (title +
maximize/fullscreen), `ChartSyncContext` (cross-chart hover + zoom
state), `useXAxisTicks` (width-aware tick selection),
`activityTimeAxis`, and `statusColors`, plus a server-side
`activitySeries.server.ts` holding `chooseBucketSeconds`, status
grouping, and the zero-fill helpers. The three task routes and both
presenters were refactored onto these, removing roughly 3x duplicated
tick logic, status-color tables, and bucket-ladder code.
- **Denser bars on short ranges.** Server-side bucketing now uses
`chooseBucketSeconds` (nice-interval ladder, ~72 target, capped at 120
buckets) instead of the hardcoded 1h/6h/1d ladder, so a 5m or 1h range
no longer collapses into a single bar.
- **Width-aware x-axis labels.** Labels are selected to fit the measured
plot width (always first + last, evenly spaced, de-duplicated by
rendered text), stay horizontal, and reflow on panel/window resize.
Y-axis values default to compact form (8K, 1.2M) in
`ChartBar`/`ChartLine`.
- **Synced hover line.** Hovering one of the agent page's three charts
draws a dashed vertical line at the same bucket on the *other* two, and
suppresses it on the hovered chart. It is opt-in via
`ChartSyncProvider`, so single-chart pages are unaffected.
- **Maximize button.** Each chart gets a fullscreen dialog toggle
(reuses the existing dashboard-widget pattern, `v` shortcut while
hovered).
- **Drag-to-zoom on task pages.** Dragging across a task chart sets the
Time/Date filter (`from`/`to` URL params, clearing `period`/`cursor`),
with a From/To tooltip shown during the drag.
- **Custom query charts.** Long categorical x labels (run IDs, task
names) middle-truncate and auto-rotate only when needed, and label
thinning is now width-aware for both bar and line variants. Dashboard
line-chart label density is also width-aware, tuned by a
`TIME_AXIS_LABEL_SPACING_PX` constant.
- **Tests.** 46 new unit tests for the pure logic (bucket selection,
tick spacing, time-axis formatting, zoom range, truncation).
## Intentionally unchanged
- **No click/drag zoom on the dashboard or custom query charts.**
Drag-to-zoom is wired up on the task landing pages only; zooming the
dashboard and custom charts is deliberately deferred to a separate
follow-up PR. A plain click (without a drag) on a task chart is a no-op.
- **The 25 mini activity charts on the Task list (`_index`) page are
untouched.** They are hand-rolled raw-Recharts sparklines kept
deliberately lightweight and do not use these primitives.
- **Other raw-Recharts sparklines are untouched** (the usage sparkline,
errors and prompts pages).
- **No ClickHouse query semantics changed** beyond the bucket-interval
parameter (same filters, same FINAL / `_is_deleted` handling).
- **Webapp-only.** No public package (`packages/*`) changes, so there is
no changeset; the `.server-changes/` entries cover it.
---
## Testing
Added 46 unit tests covering server-side bucket selection, width-aware
tick spacing, time-axis formatting, zoom-range math, and categorical
label truncation (`pnpm --filter webapp run test`), and `pnpm run
typecheck --filter webapp` passes. Manually exercised each task landing
page (agent, standard, scheduled) plus the dashboard and custom query
charts, stepping the Date/Time filter through 5m, 1h, 24h, 7d, and 30d
to confirm dense bars on short ranges, non-overlapping labels that
reflow on resize, the synced hover line across the agent charts, the
maximize button, and drag-to-zoom updating the filter.
---
## Changelog
The activity charts on the task landing pages and the dashboard and
custom query charts now share one set of reusable primitives. X-axis
labels are width-aware so they never overlap and reflow when a panel
resizes, y-axis values are abbreviated (8K, 1.2M), and short time ranges
render dense bars instead of collapsing into a single bar. Hovering any
agent chart mirrors a vertical line on the others, every chart gains a
maximize button, and dragging across a task chart zooms the Time/Date
filter. Long categorical labels such as run IDs and task names
middle-truncate and auto-rotate only when needed.
---
https://github.com/user-attachments/assets/6be09e38-3a0e-4947-b6e3-4839daa2fbe0
## Summary
Adds a single env flag, `DEPRECATE_V3_ENABLED` (default off), that
gracefully winds down the v3 engine (`RunEngineVersion.V1`). While it's
off nothing changes, so self-hosted instances still on v3 keep working.
When it's on:
- Triggers that resolve to v3 are rejected with a clear, actionable
error pointing at the [v4 migration
guide](https://trigger.dev/docs/migrating-from-v3), instead of silently
creating runs that never execute. This covers single triggers, batches,
scheduled fires, replays, and `triggerAndWait`, which all funnel through
one place.
- The legacy `trigger dev` websocket used by v3 CLIs is closed with an
upgrade message (v4 CLIs use a different dev transport).
- The v3 shared-queue consumer refuses to start, so no deployed v3 runs
are dequeued.
- The v3 run-lifecycle background jobs (heartbeat timeout, TTL expiry,
retry, resume batch/dependency, delayed-run enqueue, and scheduled
fires) become no-ops, so abandoned v3 runs stop generating database
load.
This builds on the existing deploy deprecation flag, which already
rejects v3 CLI deploys.
## Design
Enforcement is read through one helper, `isV3Disabled()`. Every gate
combines it with a per-run or per-project engine check (`isV3Disabled()
&& engine === "V1"`), so a v4 run that happens to reach a shared service
behaves exactly as before. v4 (V2) is never affected.
The flag is a hard switch, not a drain: when it's on, in-flight v3 runs
are abandoned in place rather than failed or expired, which is the
intended behaviour for the final shutdown.
## Summary
The `skills` installer command (`npx trigger.dev@latest skills`) ships
in the release candidate but is not yet on the stable release, so
running it with `@latest` fails today. Adds a warning on each docs page
that shows the command, telling users to run it with the `@rc` tag until
it lands in `@latest`.
The commands themselves stay on `@latest`, so nothing needs reverting
once the next stable release ships, just the warnings.
Pages updated:
- `/skills`
- `/mcp-agent-rules`
- `/building-with-ai`
## Summary
Adds Billing Limits to the webapp.
Customers can set a monthly spend cap. When usage crosses the limit,
billable environments enter a grace period. If the limit is not resolved
before grace expires, new triggers are rejected until the organization
increases or removes the limit.
The Tasks page logged a burst of React hydration errors (#421, "this
Suspense boundary received an update before it finished hydrating") on
every load, one per task row for the Running and Activity cells. The
page still worked, but it spammed the console.
The Running and Activity (24h) cells stream in via Remix `defer()` +
`<Suspense>`/`<Await>`, two boundaries per row. A streamed boundary
stays in React's "hydrating" state until its data arrives; if the
backing queries are slow enough that the data is still in flight after
the page loads, a normal background re-render (a server-sent-events
update, a panel layout effect, a revalidation) hits the boundary and
React bails it to client rendering and throws #421. With N rows that is
2N errors. It never reproduced locally because those queries return
instantly there.
Fix: wrap the two cells in `ClientOnly` so they mount after hydration.
The stats still load asynchronously (the task list renders immediately),
but there is no longer an SSR boundary to bail. In the slow-query case
those cells already client-rendered (that was the bail); this just makes
it explicit and silent.
Verified by simulating slow stat queries against a local build: the
errors go from 2-per-row to zero, and the cells render correctly once
the data resolves.
The supervisor image build has been failing since `@trigger.dev/core`
gained
an `ai` peer dependency. `turbo prune` (2.5.4) generates a pruned
lockfile
that references the `ai@6.0.116(zod@3.25.76)` snapshot without including
the
entry itself, which causes `pnpm fetch --frozen-lockfile` to abort.
Bumping to 2.10.0 fixes the pnpm v9 peer dep snapshot pruning. Updated
both
Containerfiles for consistency.
Example failure here:
https://github.com/triggerdotdev/trigger.dev/actions/runs/28225353375/job/83618124564
Broken since:
c06005b3
## Summary
Adds a dedicated, always-on Performance section to the repo PR review
guide (`.claude/REVIEW.md`), so every review weighs new work against
table size, hot paths, and how deep or wide the data it walks can get.
It names the tables to treat as huge (the `TaskRun` family in Postgres,
`task_events_v1`/`v2` in ClickHouse), the hot paths that warrant extra
scrutiny (trigger and batch trigger, dequeue, execution-snapshot writes,
OTEL ingestion, trace and run-list reads), the deep and wide shapes that
turn one run into a large tree or batch, and five named anti-patterns
with severities: per-level re-scans, dropping the partition-pruning
predicate, unbounded `IN` lists, sequential per-level round-trips, and
losing the single-query fast path.
Co-authored-by: Matt Aitken <matt@trigger.dev>
## Summary
The in-dashboard agent's datastore now runs migrations over a direct
(non-pooler) connection. A transaction-mode pooler can't run the
migrator (no advisory locks, no multi-statement DDL), so when the
agent's database sits behind a pooler the migration step needs a
separate direct connection.
The application keeps connecting over the pooled
`DASHBOARD_AGENT_DATABASE_URL`. Only the migration entry points changed
(`drizzle.config.ts`, `migrate.mjs`, `migrate-status.mjs`); the runtime
client is untouched.
## Connection resolution (migrations)
```
DASHBOARD_AGENT_DIRECT_URL direct agent connection (used for migrations)
DASHBOARD_AGENT_DATABASE_URL pooled agent connection (preserves current behavior)
DIRECT_URL main direct connection (single-database fallback)
DATABASE_URL last resort
```
Mirrors the existing `DATABASE_URL` / `DIRECT_URL` split. Fully
backward-compatible: with nothing new set, resolution is identical to
before. The agent-specific vars take precedence over the main
`DIRECT_URL`, so a separate agent database is never migrated against the
wrong one. When the agent falls back to the main single database,
migrations now prefer its direct connection.
<img width="2400" height="1794" alt="chat-ui-closed"
src="https://github.com/user-attachments/assets/35016a72-c6d2-4b6a-8760-b1da5b4a166c"
/>
<img width="2400" height="1794" alt="chat-ui-open"
src="https://github.com/user-attachments/assets/25b3df60-9aa2-40fd-a1ba-1447eeee4c52"
/>
## Summary
The in-dashboard agent was launched from a button pinned to the
bottom-right of every page, which floated over page controls (for
example the run inspector's action bar). It now opens from a compact
"Chat" button on the far right of the page header, and the same button
toggles to "Collapse" while the panel is open. The panel and launcher
are labelled "Chat" in the UI.
The launcher only renders on env-scoped pages where the agent is enabled
(same feature flag gating), so it stays hidden for everyone who doesn't
have it. The existing "Ask AI" support button is untouched; it stays in
place until the agent is turned on by default.
## How it works
`DashboardAgent` (env layout) shares the open/close state through a
small context, and `NavBar` renders a launcher that self-hides whenever
that context is absent. No floating overlay, and the launcher can't
appear on pages where the agent can't open.
## Summary
The in-dashboard agent button was rendered for all admins and
impersonators regardless of the `hasDashboardAgentAccess` flag, so it
appeared even where the agent is disabled (for example, floating over
the run inspector controls). It is now gated by the flag for everyone,
so it stays hidden until the flag is turned on.
## Rollout
Both levers default off, so nothing changes for users until deliberately
enabled:
- **Per-org:** set `hasDashboardAgentAccess` on an org's feature flags
to enable the agent for just that org.
- **All admins:** set `DASHBOARD_AGENT_ADMIN_PREVIEW=1` to give admins
and impersonators an everywhere-preview, independent of the per-org
flag.
Previously admins bypassed the flag unconditionally, which is why the
button showed up before the agent was ready to ship.
Closes this feature request:
[https://triggerdev.featurebase.app/p/isolated-dev-sessions-for-multiple-local-trigger-dev-instances](https://triggerdev.featurebase.app/p/isolated-dev-sessions-for-multiple-local-trigger-dev-instances)
### Feature notes:
- CLI `trigger dev` works as before
- `trigger dev --branch my-branch` to create a new branch and run
against it.
- `trigger dev archive --branch my-branch` to archive (or in webapp).
- New webapp page to manage and archive dev branches, currently feature
flagged.
### Implementation details:
- No changes to data model, no backfill. `isBranchableEnvironment`
column is ignored for dev branches, we use `parentEnvironmentId IS NULL`
instead.
- `x-trigger-branch` overloaded for preview and dev branches
- New `TRIGGER_DEV_BRANCH` env var available locally.
`TRIGGER_PREVIEW_BRANCH` overloaded for child runs.
- Lots of new glue code to sanitise the branch checks.
### Rollout
- Deploy webapp/API changes (all backwards compatible)
- Manual tests on some orgs
- Deploy docs, release CLI, flip feature flag for webapp feature
### NB
- `api.v1.projects.$projectRef.environments.ts` will return
`isBranchableEnvironment: true` for all dev environments.
### Prerequisites
- [x] Typecheck will not pass until we make a new release of
`@trigger.dev/platform` and bump it here
A deployment could be marked deployed and promoted to current without
its image ever landing in the registry. Finalize trusted the CLI: the v1
path never pushed or checked, and the v2/v3 path skips its own push when
the CLI sends `skipPushToRegistry` - which the local-build path always
does. In the happy path the CLI pushes the image itself, so this stayed
latent. But any deviation - `--no-push`/`--load`, a push that lands in a
different registry, or an old CLI - promoted a version whose image can't
be pulled, so every run failed at pull time while the deploy itself
reported success.
This adds a registry existence check after push and before finalize. If
the image isn't there, the deploy fails loudly instead of promoting a
version that can't start. The check is ECR-only (a no-op for other
registries, so self-hosted setups are unaffected) and uses
`BatchGetImage`, which the deploy role already allows. It fails open on
an ambiguous registry error so the check can't itself turn into a deploy
outage. The image reference is the platform-generated value and the
lookup is bound to the configured registry host; the CLI-supplied digest
is validated before use.
Can be turned off with `DEPLOY_IMAGE_VERIFICATION_ENABLED=0` for setups
that push images out of band (e.g. an air-gapped registry the platform
can't reach).
refs TRI-11243
SSO settings page: resolve plan before the role check. A non-Enterprise
org now renders the upsell state for every role instead of showing a
"permission denied" panel to non-Owners for a feature their org can't
use yet. manage:sso is only enforced once the org is actually entitled.
Extracts EMPTY_SSO_STATUS and uses throwPermissionDenied().
Also removes the client-side SSO session fetch guard. It monkeypatched
global window.fetch, which made it the initiator of every request and
obfuscated the real call site on any 4xx/5xx. Session revocation is
still enforced server-side on every authenticated request and surfaces
as a logout redirect on the next navigation/refresh, so the client guard
was UX-only and not worth the cross-cutting cost.
The wait.for() idempotency example passed the idempotency options as a
second argument, but wait.for() takes a single options object — so the
second object was silently ignored and the key never applied. Merged the
fields into the single options object, matching the wait.until()
example.
Supersedes #4021.
Prefixes the dashboard feedback form thread titles with `Web app:` so
support inbox threads coming from the in-app contact form are easy to
tell apart from those submitted on the marketing site, which previously
shared an identical `Contact form:` prefix.
## Summary
The dashboard agent's database migrations could be silently skipped when
its database is shared with another Drizzle application, leaving the
`trigger_dashboard_agent` schema uncreated and a later migration failing
with `schema "trigger_dashboard_agent" does not exist`.
## Root cause
Drizzle's migrator decides what to run by reading the most recent row
from its journal table by `created_at`, and skipping any migration dated
at or before it. The dashboard-agent runner used Drizzle's default
journal table, `drizzle.__drizzle_migrations`, which every Drizzle app
shares by default. When the database is shared, another app's journal
row dated between two of our migrations makes the migrator treat the
earlier one (the `CREATE SCHEMA`) as already applied and run a later one
against a schema that was never created.
## Fix
- Track the dashboard-agent migrations in a dedicated journal table
(`drizzle.__dashboard_agent_migrations`), in both the deploy runner
(`migrate.mjs`) and the drizzle-kit config, so its history is
independent of any other Drizzle app sharing the database. The table
stays in the `drizzle` schema so the first migration's `CREATE SCHEMA
"trigger_dashboard_agent"` does not collide with it.
- Make the first two migrations idempotent (`CREATE SCHEMA/TABLE/INDEX
IF NOT EXISTS`) so databases that already tracked them under the old
journal table re-run cleanly after the rename instead of erroring on the
bare `CREATE SCHEMA`.
Verified against a Postgres seeded to reproduce the skip: the old
default-table path fails as above, the dedicated-table path creates the
schema and all tables, and re-running on an already-migrated database is
a clean no-op.
## Summary
The auto-generated changeset release PR (`changeset-release/main`)
builds its `## Improvements` / `## Bug fixes` summary with
`scripts/enhance-release-pr.mjs`. The script deduplicated summary
entries by PR number, so when a single PR shipped more than one
changeset, only the first entry survived and the rest were silently
dropped from the summary. The dropped entries still appeared in the raw
`<details>` block, which is how the mismatch surfaced (for example in
[#3998](https://github.com/triggerdotdev/trigger.dev/pull/3998), where
one PR's four changesets showed up as a single summary line).
## Fix
Deduplicate on the full entry text rather than the bare PR number. The
entry text embeds the PR link, so:
- the same changeset rendered once per package section still collapses
to one,
- distinct changesets from the same PR are each kept,
- identical descriptions from different PRs stay separate.
Verified against the raw changeset output from
[#3998](https://github.com/triggerdotdev/trigger.dev/pull/3998): that
PR's changesets went from 1 to all 4 in the generated summary.
## Summary
Adds a `RUN_ENGINE_DEQUEUE_DISABLED_WORKER_QUEUES` setting that refuses
worker dequeue requests for the listed worker queues (or base regions),
so their runs stay queued instead of being handed to workers that can't
run them. Blocked dequeues are counted via a
`run_engine.dequeue.blocked` OTel counter (labeled by `worker_queue` and
`region`).
## Summary
The `runqueue.workerQueue.length` gauge only reported a worker queue's
depth while runs were being dequeued from it. When dequeues stop, the
metric goes stale or missing, so a queue that has backed up because
nothing is draining it can't be alerted on. This adds a small observer
that refreshes the observed set of worker queues from the
`WorkerInstanceGroup` records on an interval, so every active worker
queue (and its scheduled split variant) keeps reporting its length
regardless of dequeue activity.
The observer is off by default and enabled per service via
`RUN_ENGINE_WORKER_QUEUE_OBSERVER_ENABLED`, reads from the read replica,
and skips a configurable set of cloud providers
(`RUN_ENGINE_WORKER_QUEUE_OBSERVER_EXCLUDED_CLOUD_PROVIDERS`, default
`digitalocean`). When enabled it is the source of truth for the observed
set, so the per-dequeue registration is skipped on that instance, and it
groups by worker queue so the per-instance duplicates collapse to the
true depth.
Also removes the unused `GET`/`POST /api/v1/workers` endpoints. Their
only consumer was a CLI command group that is no longer registered.
## Verification
Verified end to end against a local stack: the gauge reports each worker
queue's length with no dequeues happening, excludes the configured
providers, includes hidden groups, and the removed endpoints return as
if they never existed. Added a run-engine test
(`workerQueueObservation.test.ts`).
## Summary
Adds an in-dashboard AI agent: a chat panel, reachable from any
environment
page, that answers questions about your runs, errors, tasks, and
analytics,
diagnoses why a run failed, charts your data, reads your connected
repo's
source, and answers product and how-to questions. It is gated behind the
`hasDashboardAgentAccess` feature flag (global or per-org, default off),
so
this PR ships disabled: the launcher is hidden unless the flag is
enabled.
## Design
The agent runs as a standalone `chat.agent` Trigger task in its own
internal
package, with no access to the webapp database, Prisma, or ClickHouse.
It reads
the user's data over the public API, acting as the user via a
short-lived
delegated user-actor token minted server-side each turn (never in the
browser),
building on
[#3997](https://github.com/triggerdotdev/trigger.dev/pull/3997). The
error and analytics tools use
[#4005](https://github.com/triggerdotdev/trigger.dev/pull/4005)
and the TRQL query API.
The first turn of a new chat streams from a warm webapp route (Head
Start) while
the durable agent boots in parallel. Structured answers (a run-failure
diagnosis
card, a live chart) render through a small typed view catalog rather
than
arbitrary markup. A knowledge lane forwards product and how-to questions
to the
support assistant.
Conversation history lives in a separate Drizzle-backed store on its own
Postgres schema, kept as a display read-model so it can never corrupt
the
agent's model context.
The SDK changes add an `apiClient` option to
`chat.createStartSessionAction` and
`chat.headStart`, and keep the Head Start tool-approval tail intact
across a
custom `prepareMessages` hook so prompt caching and Head Start compose.
Adds an in-process backpressure signal that pauses dequeuing when the
Kubernetes cluster is saturated, so work overflows cheaply in the queue
instead of piling up as unschedulable pods. Saturation is read by
scraping the apiserver's total pod-object count
(`apiserver_storage_objects{resource="pods"}`) and applying an
engage/release threshold with hysteresis - a single lightweight
aggregate scrape, not a pod listing.
Backpressure sources are now evaluated independently and OR'd: each
source has its own enable and dry-run flag, and the supervisor engages
if any enabled source trips. This adds the pod-count source alongside
the existing one without changing it, and is extensible to more sources
later. Off by default.
The scrape uses the in-cluster kubeconfig over `https` so TLS verifies
against the cluster CA (the fetch-options helper attaches the CA as an
`https.Agent`, which the global `fetch` ignores - that path silently
dropped the CA). Enabling the pod-count source requires the supervisor's
service account to be granted `get` on the `/metrics` non-resource URL;
that RBAC and the per-deployment env wiring are operator-side and live
elsewhere.
New config (pod-count source):
`TRIGGER_DEQUEUE_BACKPRESSURE_POD_COUNT_ENABLED` (default false),
`_POD_COUNT_DRY_RUN` (default true), `_POD_COUNT_ENGAGE` /
`_POD_COUNT_RELEASE` (hysteresis thresholds), `_POD_COUNT_REFRESH_MS`
(scrape interval, default 5s). The existing source's flags are
unchanged.
Observability: a `supervisor_cluster_pod_count` gauge, and the pod-count
monitor's metrics are namespaced (`supervisor_backpressure_pod_count_*`)
so the existing backpressure metrics keep their names.
## Summary
Fixes the fair-queue weighted environment shuffle, which biased
environment ordering whenever fair-queue biases are enabled (the default
configuration).
## Root cause
`#weightedShuffle` in `fairQueueSelectionStrategy.ts` computed the total
weight once and drew its random pivot against that full-set total on
every iteration, but never decremented the total as items were removed
from the working set. After the first pick, the pivot frequently
overshot the sum of the remaining items, so the inner selection loop ran
off the end and clamped to the last remaining element. The result
systematically over-selected whichever environment sat at the tail of
the set.
The first slot stayed fair (the full total is correct on the first
draw), but later positions were ordered by environment iteration order
rather than by the intended concurrency-limit and available-capacity
weighting. For four equal-weight environments, the final position landed
on one env ~9% of the time and another ~42%, instead of ~25% each.
The two sibling selection paths (`#weightedRandomQueueOrder` and
`#selectTopEnvs`) already decrement the total before splicing; this
brings the env shuffle in line with them.
## Fix
```ts
result.push(items[index].envId);
totalWeight -= items[index].weight;
items.splice(index, 1);
```
Adds a regression test that runs the weighted shuffle over equal-weight
envs with biases enabled and asserts each env lands in every position
roughly uniformly. It fails on the old code (tail position ~37%) and
passes with the fix.
Reported in #4001.
Deploy success/failure wasn't easily observable: compute template
creation only logged, and the terminal deployment statuses (deployed /
failed / timed-out) weren't traced — so deploy health couldn't be seen
without querying the database.
This adds two spans:
- `compute.template.create` around template creation at finalize,
tagging the resolved mode and per-preset outcome. `resolveMode` now
returns its decision (mode + reason) so the span can record why a mode
was chosen.
- `deployment.outcome` via a small shared helper
(`recordDeploymentOutcome`) emitted at every terminal-status write —
finalize (deployed), fail / index-failed / background-worker (failed),
and timeout (timed out) — so deploy success/failure is queryable by
status and reason.
The helper is best-effort (org/project/env enrichment where cheaply
available) and never throws, so telemetry can't break a deploy.
## Summary
Refreshes the SVG artwork for the main task icon and the cached task
variant shown on the run trace span view.
The cached icon (previously a hardcoded blue "T" in a dashed border) now
lives alongside `TaskIcon` in `TaskIcon.tsx` and is drawn with
`currentColor`, so it inherits the `text-tasks` theme color like the
other span icons instead of ignoring it. The standalone
`TaskCachedIcon.tsx` file is removed and its two import sites updated.
## Summary
Sessions started from the agent Test playground were tagged with a
`"playground"` tag that rendered in the Sessions table's Tags column.
They are now flagged with a real `Session.isTest` boolean (mirroring
`TaskRun.isTest`) and surfaced as a dedicated **Test** column with a
check icon, to the left of Tags, on both the Sessions page and the Agent
landing page, plus a matching **Test** property on the session detail
page. This mirrors how Standard and Scheduled task runs already indicate
test runs.
## Design
`isTest` is a new `Session` column (Postgres) replicated into ClickHouse
`sessions_v1` alongside the existing fields. The Sessions list reads
`isTest` from Postgres for display (ClickHouse only supplies the ordered
session IDs), so the column renders correctly without a ClickHouse
backfill.
The playground action now sets `isTest: true` on session create instead
of writing the `"playground"` tag. The triggered run still carries
`playground:true` in its own tags (unchanged). A migration backfills
existing sessions, setting `isTest = true` and stripping the
now-redundant `"playground"` tag where it is present, so the list and
detail views render consistently without read-time tag filtering.
## Summary
Updates the task icons used across the dashboard. `TaskIcon` and its
small variant now use a new burst glyph, and `TasksIcon` adopts the
previous task glyph (the rounded square). Both still render with
`currentColor`, so they inherit text color exactly as before.
Export names are unchanged, so every existing usage (side menu, task and
queue views, run filters) picks up the new artwork with no other code
changes.
## Summary
The logs search page (behind a feature flag) ran ClickHouse out of
memory when browsing back over long time ranges. This keeps it within
bounded memory and fixes a pagination bug that could skip or duplicate
rows at a page boundary.
## Fix
Memory: the list query reads in sort-key order, which opens one read
stream per part in the window, and on object storage those per-part read
buffers dominate peak memory, so it scaled with the number of parts
scanned. Two changes bound it:
- The logs ClickHouse client caps the per-part read buffers via new
env-tunable settings. The object-storage-only setting is opt-in, so it
is never sent to a ClickHouse version that lacks it.
- Recent-first window narrowing: rows come back newest first, so the
presenter probes the most recent window and only widens toward the full
requested range when a page is short. A busy environment fills a page
from a few recent parts instead of scanning the whole range; a quiet one
still returns every row in a couple of cheap reads.
Correctness: the keyset cursor ordered on (triggered_timestamp,
trace_id), which is not unique because the spans of a trace share both,
so rows at a tie could be skipped or duplicated across pages. The cursor
and ORDER BY now include span_id, and the cursor is versioned so stale
cursors reset to the first page.
Guards: the effective page size is capped, and the existing per-query
memory limit lets a pathological wide browse fail with an error instead
of taking the node down.
## ClickHouse 26.2
The memory fix relies on lazy materialization deferring the wide
attributes column to the output rows, which only holds on 26.x. Cloud
already runs 26.2, so this moves the dev stack, testcontainers, and CI
to match. The ClickHouse test suite passes on 26.2.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
## Summary
Adds read methods to `RunStore` (`findRun`, `findRunOrThrow`,
`findRuns`) and routes every Postgres read of `TaskRun` through them,
mirroring how writes already go through the store. Behavior-preserving:
each relocated read keeps its exact query, field selection, and database
client (writer, replica, or transaction). This lets `TaskRun` reads be
retargeted to a different backing store later without touching call
sites.
Stacked on #3981 (the write adapter); that PR is the base of this one.
## Scope
In scope: the run engine, webapp services, presenters, and route
loaders. Three reads that pulled `TaskRun` in through a parent model's
relation `include` (alert delivery, batch results, attempt-dependency
cancellation) are decomposed to fetch the run(s) through the store and
stitch them back, since a relation include would not follow `TaskRun` to
a new table.
Left reading the existing table (out of scope): the legacy MarQS paths,
the legacy trigger idempotency read, and one raw-SQL recovery script
(commented for revisiting at cutover).
## Notes
Reads default to the read replica; callers pass the writer or a
transaction client wherever the original read did, so writer-vs-replica
behavior is unchanged.
Follow-up to #3992, which gated the send runner-side - but only for new
runner images. Existing runners still POST a debug log per line.
When `SEND_RUN_DEBUG_LOGS` is off (default), the route now drops the
request immediately: `skipBodyParsing` skips the body read/parse, a bare
handler returns 204, no wide event. The route stays registered so it
avoids the `No route match` error log; the only per-request log left is
the framework's `logger.debug` trace, suppressed at the default `info`
level. Still counted by request metrics, and 204 is non-retryable so no
retry storm.
Adds a `skipBodyParsing` flag to the internal HTTP server.
Runners were POSTing a debug log to the supervisor for every log line -
one request per line, unbatched and unconditional. The supervisor
already has a `SEND_RUN_DEBUG_LOGS` toggle (off by default) that
discards them on receipt, but the runner fired the request regardless,
so the traffic hit the supervisor either way.
This gates the send at the source. The runner now reads
`TRIGGER_SEND_RUN_DEBUG_LOGS` (off by default, injected by the
supervisor from its existing `SEND_RUN_DEBUG_LOGS` setting) and skips
the POST entirely when disabled. Local log output is unchanged. Dev runs
use a separate path and are unaffected.
## Summary
Adds an environment-scoped HTTP API over the Errors feature, mirroring
the runs API. Task-run failures are grouped by a fingerprint into "error
groups," and this exposes everything you can do with them in the
dashboard:
- `GET /api/v1/errors` lists error groups, with
`filter[taskIdentifier]`, `filter[version]`, `filter[status]`
(`unresolved`/`resolved`/`ignored`), `filter[search]`, a time range, and
cursor pagination.
- `GET /api/v1/errors/{errorId}` retrieves a single group (summary,
lifecycle state, affected versions).
- `POST /api/v1/errors/{errorId}/{resolve,ignore,unresolve}` changes its
state.
- `GET /api/v1/runs?filter[error]={errorId}` lists the runs behind a
group.
Request and response schemas are exported from `@trigger.dev/core/v3` so
the SDK can reuse them, and all endpoints are documented in the API
reference (OpenAPI). `errorId` is the `error_<fingerprint>` friendly id.
## Attribution
State changes record who made them. A plain environment API key has no
user, so `resolvedBy`/`ignoredByUserId` stay null. When the caller uses
an environment JWT obtained by exchanging a personal access token or a
delegated user token at `POST /api/v1/projects/:ref/:env/jwt`, that
exchange now stamps an `act` delegation claim, and the write endpoints
read `act.sub` to attribute the change to the acting user. This is the
first endpoint to consume the `act` claim, so two small pieces of
plumbing ride along: the exchange stamps `act` for personal-access-token
subjects too (it was delegated-token-only), and the public-JWT
bearer-auth path surfaces `act.sub` to the handler.
Built on the delegated-token work in #3997.
## Summary
The CLAUDE.md audit job (`.github/workflows/claude-md-audit.yml`)
frequently hits its 15-turn cap before it finishes reviewing a PR, so
the job fails without posting a verdict. For example, the audit job
failed on [this
run](https://github.com/triggerdotdev/trigger.dev/actions/runs/27837408945/job/82390460772?pr=3990).
This raises `--max-turns` from 15 to 25 to give the review room to
complete, and pins `--model claude-opus-4-8` (the job previously
inherited the action default model).
## Summary
Adds a short-lived, delegated token (`tr_uat_...`) that authenticates
against the API as a user without handing out a long-lived personal
access token. You mint one from a PAT, optionally narrow it to a set of
scopes, and give it a lifetime; the API then treats requests as that
user, subject to their role.
`trigger.dev mint-token` is the entry point (it uses your stored PAT):
```bash
UAT=$(trigger.dev mint-token --ttl 3600 --cap read:runs)
```
The token works anywhere a PAT does for user-level endpoints, and can be
exchanged for an environment JWT at `POST
/api/v1/projects/:ref/:env/jwt` to reach environment-scoped data (the
same exchange a PAT supports).
## How it works
A user-actor token is a short-lived JWT verified by a new first-class
`authenticateUserActor` method on the RBAC plugin. Self-hosters get a
built-in fallback; role-aware enforcement comes from the plugin.
Effective permissions are the intersection of the user's role and the
token's optional scope cap, so a token is only ever narrower than the
user, never broader.
Minting is restricted to personal access tokens (a token can't mint
another one, and an environment key can't mint one). Tokens default to a
1 hour lifetime (max 365 days). When exchanged for an environment JWT,
the user is stamped on it for attribution and the scope cap is carried
through.
## Summary
Replaces the multi-select popover task type filter on the Tasks page
with a single-select segmented control: **All** plus icon-only
**Agent**, **Standard**, and **Scheduled** segments. Each segment has a
tooltip showing its label and a number-key shortcut (0-3), and the
search field no longer autofocuses so the shortcuts work on page load.
## ✅ Checklist
- [x] I have followed every step in the [contributing
guide](https://github.com/triggerdotdev/trigger.dev/blob/main/CONTRIBUTING.md)
- [x] The PR title follows the convention.
- [x] I ran and tested the code works
## Summary
The Personal Access Tokens page now shows each token's maximum role in a
new column, so you can see at a glance what a token is capped to. The
column only appears when an RBAC plugin is installed, and shows "-" for
tokens with no cap. Its header tooltip reuses the same explanation shown
in the create-token panel.
## Summary
Adds the `4.5.0-rc.7` entry to the AI chat changelog, covering the
agent-facing changes in
[v4.5.0-rc.7](https://github.com/triggerdotdev/trigger.dev/releases/tag/v4.5.0-rc.7):
- `chat.headStart` now works with the `chat.customAgent` and
`chat.createSession` backends, not just `chat.agent`
- Opt-in Anthropic system-prompt caching via
`chat.toStreamTextOptions()`
- Three custom-agent-loop fixes: continuation replay, mid-stream
steering, and task-backed tools
- `trigger skills` follow-ups: `trigger-` namespacing, SDK-bundled docs,
and a new cost-savings skill
Generic, non-agent rc.7 items (the CLI uninitialized-project error
message, run-span cost fields) are intentionally left out to keep this
changelog scoped to AI chat agents.
Pushes new organizations and users into the Attio CRM at signup time,
for Customer Success (TRI-10431).
- Orgs → Attio `workspaces`, users → Attio `users`, keyed on Attio's
built-in unique `workspace_id` / `user_id` so writes are idempotent
upserts.
- Runs on the common Redis worker (not inline), so a slow or unavailable
Attio never blocks the signup path; failures retry (3 attempts).
- Hooks: user-created (alongside the existing Loops call) and
org-created (`createOrganization`).
- Gated behind `ATTIO_API_KEY`, no key means the sync is skipped
entirely, so OSS / self-hosted installs are unaffected.
Only creation is covered here (the record "shell"); spend, runs, plan
changes, churn, and role/relationship linking are populated by the
scheduled full sync, tracked separately.
**Deploy note:** requires an Attio API key set as `ATTIO_API_KEY` in the
webapp env, with scopes **Records (read-write)** + **Object
Configuration (read)**, the assert/upsert endpoint reads object config
to resolve the matching attribute. Without the key the sync no-ops.
---------
Co-authored-by: Matt Aitken <matt@mattaitken.com>
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
## ✅ Checklist
- [x] I have followed every step in the [contributing
guide](https://github.com/triggerdotdev/trigger.dev/blob/main/CONTRIBUTING.md)
- [x] The PR title follows the convention.
- [x] I ran and tested the code works
---
## Testing
Ran the webapp locally with the change applied; it compiles and serves.
The edit only swaps the chart card title string from "LLM spend" to "LLM
spend ($)" on the agent landing page.
---
## Changelog
The agent dashboard "LLM spend" chart label now includes the currency
unit, reading "LLM spend ($)".
---
## Screenshots
_[Screenshots]_
💯
A batch of technical-SEO fixes across the docs, all reader-facing
(titles, links, redirects):
- Canonicalize the duplicate CLI command pages: the bare `/cli-dev` and
`/cli-deploy` paths now permanently redirect to their `-commands`
equivalents, and a duplicate navigation entry is removed.
- Give the three pages that all rendered as "Overview" distinct titles
(Building with AI, self-hosting overview, Management API overview), with
sidebar labels unchanged.
- Replace the generic "Learn more" links in the introduction's
build-extension list with descriptive anchor text.
- Switch two http links to https in the Supabase guides, point a
troubleshooting page's help link to Discord, and add missing meta
descriptions to three help and troubleshooting pages.
## Summary
Several dashboard routes performed actions a restricted role should not
be able to do (cancel or replay runs, manage prompt versions, invite and
manage members, manage billing) without any permission check. This adds
role-based permission enforcement to those routes, and disables the
matching UI controls (with a tooltip) when the current role lacks
permission.
Covered actions:
- Runs: cancel and replay (single, bulk create, bulk abort)
- Prompts: create or edit override versions, and promote a version to
current
- Members: invite, resend invite, revoke invite
- Billing: change plan, billing alerts, and the customer portal
## How
Each affected route now goes through the `dashboardLoader` /
`dashboardAction` route builders with an `authorization` block declaring
the required permission (or a per-intent check where one route handles
several intents). Existing tenancy and data-scoping queries are
untouched; this only layers permission checks on top. The UI follows
disable-don't-hide: controls stay visible but disabled with a "You don't
have permission to ..." tooltip.
Two reusable pieces support this: `checkPermissions(ability, checks)`
turns a set of checks into a boolean map a loader returns to the client,
and `PermissionButton` / `PermissionLink` disable the underlying control
and show a tooltip when a permission flag is false.
## Behaviour
No change in the default configuration: permissions are permissive, so
every control stays enabled and every route behaves as before. The
checks only take effect when an RBAC plugin is installed. This also
makes role assignment on invite-accept non-fatal, so a failure there
cannot block joining an org.
Verified with `pnpm run typecheck --filter webapp`; `checkPermissions`
has unit tests.
## Summary
Adds a "Stopping generation" section to the Custom agents page. It
documents how stop works when you drop down from `chat.agent` to
`chat.createSession`: pass `turn.signal` (a combined stop-and-cancel
`AbortSignal`) to `streamText`, and `turn.complete()` cleans up the
aborted partial, accumulates it as its own assistant message, and keeps
the run alive for the next turn. `turn.stopped` distinguishes a user
stop from a full run cancel.
Until now the createSession stop story only existed as scattered fields
in the reference table; the client side (`transport.stopGeneration`) and
the `chat.agent` run-callback signals were documented, but not the
custom-agent turn loop. Steering for these backends is already covered
on the pending messages page, which this page links to.
## Summary
Adds a "Duration and cost while paused" section to the human-in-the-loop
page. It explains that a HITL pause (a no-execute tool waiting on
`addToolOutput`) suspends the run and frees compute, so the human's
thinking time does not count against `maxDuration` (which measures
active CPU time and excludes suspended waitpoint time, the same as
`wait.for`). Customers don't need to raise `maxDuration` or end the run
to support long human waits.
This was a recurring point of confusion: readers assumed the pause holds
the run open and burns the budget. Also updates the how-it-works
pseudocode ("Agent suspends (compute freed)") and links `wait.for` and
`maxDuration` on first mention.
The global feature flags admin page had a few rough edges.
The percentage flags are numeric (`z.coerce.number()`) but rendered as
free-text inputs, so you could type non-numeric values that only failed
validation after submitting - and the error surfaced behind the confirm
dialog. The control-type detection now recognises numbers and renders a
proper number input, with the min/max range as the placeholder so the
type is clear even when the field is unset. The save error also shows
inside the confirm dialog now, not just behind it.
The action buttons were unreachable without zooming out. The admin
layout wrapped each page in a plain block, so `h-full` page content
overran the viewport by the height of the tab bar and got clipped by the
`overflow-hidden` body. Making the layout a flex column bounds each page
to the space below the tabs, so the existing per-page scroll works and
the feature flags page scrolls like the Users/Orgs tabs. Also capped the
confirm dialog's diff list so its footer stays on screen when there are
many changes.
## Summary
Prisma infrastructure failures (P1xxx-class: database unreachable, timed
out, connection dropped, engine init/panic) carry the database hostname
in their `.message`. This captures them centrally for observability and
ensures they never reach API clients verbatim.
## Design
A `$allOperations` client extension on the writer and replica clients
logs infrastructure errors with the originating model and operation,
then rethrows the **original** error unchanged — call sites that branch
on `error.code` (unique-violation idempotency, not-found handling) and
transaction retries keep working. Only infrastructure errors are logged;
routine query/validation errors (P2xxx) are left alone.
`$allOperations` can't see the transaction boundary (`$transaction` is a
client method, not an operation), so infrastructure errors surfacing
from `$transaction()` without a Prisma code — e.g.
`PrismaClientInitializationError` — are logged separately at the
transaction wrapper, where the existing coded-error path would otherwise
miss them.
`clientSafeErrorMessage()` swaps an infrastructure error's message for
`"Internal Server Error"` at the API routes that previously returned
`error.message` raw. Status codes, headers, and every non-infrastructure
message are unchanged.
## Test plan
- [x] P2002 / P2025 rethrow with code intact and are not logged
- [x] Statement errors inside `$transaction` keep their code (retry
logic intact)
- [x] Raw queries wrapped without crashing on the undefined model
- [x] A genuine connectivity failure is logged with model/operation/code
- [x] `clientSafeErrorMessage` obfuscates infra messages, preserves all
others
- [x] `pnpm run typecheck --filter webapp` (12/12)
## Note
Overlaps with #3391 (Prisma 7 migration) on
`apps/webapp/app/db.server.ts` — coordinate rebasing.
Replicates `TaskRun.planType` into the `task_runs_v2` ClickHouse table
so run analytics can group by plan type.
Adds a `plan_type` column (goose migration `033`,
`LowCardinality(String)`), the replication insert mapping, and the
matching schema/column/type entries - same shape as the recent `region`
addition. Write-once at trigger, so it just rides along on existing
replicated rows. Internal analytics only; not exposed in the Query API.
## Summary
7 improvements.
## Improvements
- `@trigger.dev/sdk` now bundles the Trigger.dev agent skills and a
curated snapshot of the docs those skills reference. The skills that
`trigger skills` installs into your coding agent read this content from
node_modules, so the guidance your AI assistant follows is pinned to the
SDK version installed in your project and stays current across upgrades
instead of going stale until the next reinstall.
([#3937](https://github.com/triggerdotdev/trigger.dev/pull/3937))
- Running a CLI command like `dev`, `deploy`, `preview`, or `update`
before initializing a project no longer crashes with a raw `Cannot find
matching package.json` stack trace. The CLI now detects the missing
project and points you to `npx trigger.dev@latest init` instead.
([#3929](https://github.com/triggerdotdev/trigger.dev/pull/3929))
- The agent skills installed by `trigger skills` are now namespaced with
a `trigger-` prefix (e.g. `trigger-authoring-tasks`,
`trigger-getting-started`) so they don't collide with unrelated skills
in your coding agent's skills directory. Adds a `trigger-cost-savings`
skill for auditing and reducing compute spend (right-sizing machines,
`maxDuration`, batching, debounce), and `@trigger.dev/sdk` now bundles
the full Trigger.dev documentation so your agent can read the complete,
version-pinned reference directly from node_modules.
([#3970](https://github.com/triggerdotdev/trigger.dev/pull/3970))
- The run span API response now includes `cachedCost` and
`cacheCreationCost` on the `ai` object, alongside the existing
`inputCost` / `outputCost` / `totalCost`. `inputCost` reflects only the
non-cached input, so these fields let you reconstruct the full cost
breakdown for prompt-cached calls.
([#3958](https://github.com/triggerdotdev/trigger.dev/pull/3958))
- `chat.headStart` now works with the `chat.customAgent` and
`chat.createSession` backends, not only `chat.agent`. The warm step-1
response hands over to your loop the same way it does for a managed
agent. ([#3963](https://github.com/triggerdotdev/trigger.dev/pull/3963))
In a `chat.customAgent` loop, consume the handover on turn 0:
```ts
const conversation = new chat.MessageAccumulator();
const { isFinal, skipped } = await conversation.consumeHandover({
payload });
if (skipped) return; // warm handler aborted, so exit without a turn
if (isFinal) {
await chat.writeTurnComplete(); // step 1 is the response, no streamText
} else {
const result = streamText({ model, messages: conversation.modelMessages,
tools });
// Pass originalMessages so the handed-over tool round merges into the
// step-1 assistant instead of starting a new message.
const response = await chat.pipeAndCapture(result, {
originalMessages: conversation.uiMessages,
});
if (response) await conversation.addResponse(response);
}
```
With `chat.createSession`, the iterator surfaces it as `turn.handover`;
call `turn.complete()` with no argument on a final handover. The
lower-level `chat.waitForHandover()` and `accumulator.applyHandover()`
are also exported for hand-rolled loops.
- Cache your chat agent's system prompt with Anthropic prompt caching.
`chat.toStreamTextOptions()` now emits the system prompt as a cacheable
message when you opt in, so a large, stable system block is billed at
cache-read rates on every turn instead of full price.
([#3952](https://github.com/triggerdotdev/trigger.dev/pull/3952))
```ts
// at the streamText call site (Anthropic sugar)
streamText({
...chat.toStreamTextOptions({ cacheControl: { type: "ephemeral" } }),
messages,
});
// provider-agnostic equivalent
chat.toStreamTextOptions({
systemProviderOptions: { anthropic: { cacheControl: { type: "ephemeral"
} } },
});
// or where the prompt is defined
chat.prompt.set(SYSTEM_PROMPT, {
providerOptions: { anthropic: { cacheControl: { type: "ephemeral" } } },
});
```
Without an option, `system` stays a plain string. Pairs with a
`prepareMessages` cache breakpoint to cache the conversation prefix
across turns too.
- Three fixes for custom agent loops (`chat.customAgent`,
`chat.createSession`, and hand-rolled `MessageAccumulator` loops):
([#3936](https://github.com/triggerdotdev/trigger.dev/pull/3936))
- Continuation runs no longer replay already-answered user messages into
the first turn. The `.in` resume cursor is now seeded before any
listener attaches (the same boot logic `chat.agent` uses), so a chat
that continues after a cancel, crash, or upgrade only sees genuinely new
messages.
- Steering a hand-rolled loop mid-stream no longer wipes the in-flight
assistant response. `chat.pipeAndCapture` now stamps a server-generated
message id on the stream, so a `prepareStep` injection keeps the partial
text instead of replacing the message.
- Task-backed tools (`ai.toolExecute`) now work from custom agent loops:
the parent's session is threaded to the child run, so child tasks can
stream progress into the chat with `chat.stream.writer({ target: "root"
})` instead of failing with "session handle is not initialized".
<details>
<summary>Raw changeset output</summary>
⚠️⚠️⚠️⚠️⚠️⚠️
`main` is currently in **pre mode** so this branch has prereleases
rather than normal releases. If you want to exit prereleases, run
`changeset pre exit` on `main`.
⚠️⚠️⚠️⚠️⚠️⚠️
# Releases
## @trigger.dev/build@4.5.0-rc.7
### Patch Changes
- Updated dependencies:
- `@trigger.dev/core@4.5.0-rc.7`
## trigger.dev@4.5.0-rc.7
### Patch Changes
- `@trigger.dev/sdk` now bundles the Trigger.dev agent skills and a
curated snapshot of the docs those skills reference. The skills that
`trigger skills` installs into your coding agent read this content from
node_modules, so the guidance your AI assistant follows is pinned to the
SDK version installed in your project and stays current across upgrades
instead of going stale until the next reinstall.
([#3937](https://github.com/triggerdotdev/trigger.dev/pull/3937))
- Running a CLI command like `dev`, `deploy`, `preview`, or `update`
before initializing a project no longer crashes with a raw `Cannot find
matching package.json` stack trace. The CLI now detects the missing
project and points you to `npx trigger.dev@latest init` instead.
([#3929](https://github.com/triggerdotdev/trigger.dev/pull/3929))
- The agent skills installed by `trigger skills` are now namespaced with
a `trigger-` prefix (e.g. `trigger-authoring-tasks`,
`trigger-getting-started`) so they don't collide with unrelated skills
in your coding agent's skills directory. Adds a `trigger-cost-savings`
skill for auditing and reducing compute spend (right-sizing machines,
`maxDuration`, batching, debounce), and `@trigger.dev/sdk` now bundles
the full Trigger.dev documentation so your agent can read the complete,
version-pinned reference directly from node_modules.
([#3970](https://github.com/triggerdotdev/trigger.dev/pull/3970))
- Updated dependencies:
- `@trigger.dev/core@4.5.0-rc.7`
- `@trigger.dev/build@4.5.0-rc.7`
- `@trigger.dev/schema-to-json@4.5.0-rc.7`
## @trigger.dev/core@4.5.0-rc.7
### Patch Changes
- The run span API response now includes `cachedCost` and
`cacheCreationCost` on the `ai` object, alongside the existing
`inputCost` / `outputCost` / `totalCost`. `inputCost` reflects only the
non-cached input, so these fields let you reconstruct the full cost
breakdown for prompt-cached calls.
([#3958](https://github.com/triggerdotdev/trigger.dev/pull/3958))
## @trigger.dev/python@4.5.0-rc.7
### Patch Changes
- Updated dependencies:
- `@trigger.dev/sdk@4.5.0-rc.7`
- `@trigger.dev/core@4.5.0-rc.7`
- `@trigger.dev/build@4.5.0-rc.7`
## @trigger.dev/react-hooks@4.5.0-rc.7
### Patch Changes
- Updated dependencies:
- `@trigger.dev/core@4.5.0-rc.7`
## @trigger.dev/redis-worker@4.5.0-rc.7
### Patch Changes
- Updated dependencies:
- `@trigger.dev/core@4.5.0-rc.7`
## @trigger.dev/rsc@4.5.0-rc.7
### Patch Changes
- Updated dependencies:
- `@trigger.dev/core@4.5.0-rc.7`
## @trigger.dev/schema-to-json@4.5.0-rc.7
### Patch Changes
- Updated dependencies:
- `@trigger.dev/core@4.5.0-rc.7`
## @trigger.dev/sdk@4.5.0-rc.7
### Patch Changes
- `@trigger.dev/sdk` now bundles the Trigger.dev agent skills and a
curated snapshot of the docs those skills reference. The skills that
`trigger skills` installs into your coding agent read this content from
node_modules, so the guidance your AI assistant follows is pinned to the
SDK version installed in your project and stays current across upgrades
instead of going stale until the next reinstall.
([#3937](https://github.com/triggerdotdev/trigger.dev/pull/3937))
- `chat.headStart` now works with the `chat.customAgent` and
`chat.createSession` backends, not only `chat.agent`. The warm step-1
response hands over to your loop the same way it does for a managed
agent. ([#3963](https://github.com/triggerdotdev/trigger.dev/pull/3963))
In a `chat.customAgent` loop, consume the handover on turn 0:
```ts
const conversation = new chat.MessageAccumulator();
const { isFinal, skipped } = await conversation.consumeHandover({
payload });
if (skipped) return; // warm handler aborted, so exit without a turn
if (isFinal) {
await chat.writeTurnComplete(); // step 1 is the response, no streamText
} else {
const result = streamText({ model, messages: conversation.modelMessages,
tools });
// Pass originalMessages so the handed-over tool round merges into the
// step-1 assistant instead of starting a new message.
const response = await chat.pipeAndCapture(result, {
originalMessages: conversation.uiMessages,
});
if (response) await conversation.addResponse(response);
}
```
With `chat.createSession`, the iterator surfaces it as `turn.handover`;
call `turn.complete()` with no argument on a final handover. The
lower-level `chat.waitForHandover()` and `accumulator.applyHandover()`
are also exported for hand-rolled loops.
- Add `triggerConfig` support to `chat.headStart()` and
`chat.openSession()`, so the auto-triggered handover-prepare run
inherits tags, queue, machine, and other session trigger options the
same way `chat.createStartSessionAction()` does. The `chat:{chatId}` tag
is prepended automatically.
([#3963](https://github.com/triggerdotdev/trigger.dev/pull/3963))
```ts
export const POST = chat.headStart({
agentId: "my-agent",
triggerConfig: { tags: ["org:acme"], queue: "chat" },
run: async ({ chat }) => streamText({ ...chat.toStreamTextOptions(),
model }),
});
```
Because the session is created once on the first head-start turn and is
idempotent on the chat id, this is the only place to set those options
for a head-start chat's lifetime. `chat.createStartSessionAction()` now
also forwards `maxDuration`, `region`, and `lockToVersion` so both
session entry points stay consistent.
- Cache your chat agent's system prompt with Anthropic prompt caching.
`chat.toStreamTextOptions()` now emits the system prompt as a cacheable
message when you opt in, so a large, stable system block is billed at
cache-read rates on every turn instead of full price.
([#3952](https://github.com/triggerdotdev/trigger.dev/pull/3952))
```ts
// at the streamText call site (Anthropic sugar)
streamText({
...chat.toStreamTextOptions({ cacheControl: { type: "ephemeral" } }),
messages,
});
// provider-agnostic equivalent
chat.toStreamTextOptions({
systemProviderOptions: { anthropic: { cacheControl: { type: "ephemeral"
} } },
});
// or where the prompt is defined
chat.prompt.set(SYSTEM_PROMPT, {
providerOptions: { anthropic: { cacheControl: { type: "ephemeral" } } },
});
```
Without an option, `system` stays a plain string. Pairs with a
`prepareMessages` cache breakpoint to cache the conversation prefix
across turns too.
- Three fixes for custom agent loops (`chat.customAgent`,
`chat.createSession`, and hand-rolled `MessageAccumulator` loops):
([#3936](https://github.com/triggerdotdev/trigger.dev/pull/3936))
- Continuation runs no longer replay already-answered user messages into
the first turn. The `.in` resume cursor is now seeded before any
listener attaches (the same boot logic `chat.agent` uses), so a chat
that continues after a cancel, crash, or upgrade only sees genuinely new
messages.
- Steering a hand-rolled loop mid-stream no longer wipes the in-flight
assistant response. `chat.pipeAndCapture` now stamps a server-generated
message id on the stream, so a `prepareStep` injection keeps the partial
text instead of replacing the message.
- Task-backed tools (`ai.toolExecute`) now work from custom agent loops:
the parent's session is threaded to the child run, so child tasks can
stream progress into the chat with `chat.stream.writer({ target: "root"
})` instead of failing with "session handle is not initialized".
- The agent skills installed by `trigger skills` are now namespaced with
a `trigger-` prefix (e.g. `trigger-authoring-tasks`,
`trigger-getting-started`) so they don't collide with unrelated skills
in your coding agent's skills directory. Adds a `trigger-cost-savings`
skill for auditing and reducing compute spend (right-sizing machines,
`maxDuration`, batching, debounce), and `@trigger.dev/sdk` now bundles
the full Trigger.dev documentation so your agent can read the complete,
version-pinned reference directly from node_modules.
([#3970](https://github.com/triggerdotdev/trigger.dev/pull/3970))
- Updated dependencies:
- `@trigger.dev/core@4.5.0-rc.7`
## @trigger.dev/plugins@4.5.0-rc.7
### Patch Changes
- Updated dependencies:
- `@trigger.dev/core@4.5.0-rc.7`
</details>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Adds an opt-in mechanism to route a configurable percentage of
organizations onto the compute (MicroVM) backing of their region at
trigger time, without changing their stored region settings.
Routing is gated by three global feature flags -
`computeMigrationEnabled`, `computeMigrationFreePercentage`,
`computeMigrationPaidPercentage` - plus a per-org
`computeMigrationEnabled` override that wins in both directions. A
region's compute backing is resolved from a new
`WorkerInstanceGroup.region` column: a container group and its MicroVM
group share one geo `region`, so the migration swaps the resolved worker
queue to the backing group's queue. Orgs are bucketed deterministically
by id, so ramping a percentage down keeps a strict subset rather than
reshuffling, and a region with no compute backing is never touched.
Everything is off by default - behaviour is unchanged unless the flags
are set.
The flags and the worker-region groups are read on the trigger hot path
from in-memory snapshots rather than the database: a small
`createReloadingRegistry` helper loads each at startup and refreshes
them on an interval, so no per-trigger query is added and a percentage
or kill-switch change propagates within the reload interval. A cold
replica whose snapshot hasn't loaded yet reads as not-migrated (the
container path) and self-corrects on the next load - the same cold-start
contract as the datastore / LLM-pricing registries, with a
`reloading_registry_loaded` metric so a never-loaded registry is
alertable.
The same migration decision is consulted at deploy-time template
creation so a migrated org gets a compute template built ahead of its
first run. This runs in shadow mode (best-effort, never fails the
deploy) by default, or - when the `computeMigrationRequireTemplate` flag
is on - in required mode, built synchronously at deploy so the first run
never builds on-demand and template errors surface at deploy time.
So operators keep "which runs ran where" while customers only see
geography: the run's actual worker queue is stored raw, and the geo
region is stamped separately on `TaskRun.region` (and a new ClickHouse
`region` column) at trigger time. Read surfaces - the dashboard, the
API, and the Query/Logs page - show the geo region, falling back to the
worker queue for runs written before the column existed.
Minor follow-ups left out of scope: the percentage flags render as text
inputs on the admin flags page (the catalog UI has no numeric control
type yet), and `createReloadingRegistry` could later gain pub/sub for
sub-second cross-replica propagation if the reload interval proves too
slow.
## Summary
Three improvements to the SDK-bundled agent skills (follow-up to the
skills installer):
- **`trigger-` namespace.** The installed skills (`authoring-tasks`,
`getting-started`, …) had generic names that collide with unrelated
skills in a shared agent skills directory. They're now prefixed —
`trigger-authoring-tasks`, `trigger-getting-started`, etc. — matching
the convention the public skills repo already uses.
- **New `trigger-cost-savings` skill.** An MCP-driven cost audit:
right-sizes machines, flags missing `maxDuration`, spots sequential
triggers that could batch, and reviews schedule frequency, using
`list_runs` / `get_run_details` for live analysis.
- **Bundle the full docs.** `@trigger.dev/sdk` now bundles the entire
"Documentation" section of the docs (157 pages) instead of a curated
55-page subset, so an agent has the complete, version-pinned reference
in `node_modules`.
## How the bundling works
`scripts/bundleSdkDocs.ts` now reads `docs/docs.json`, walks the
"Documentation" dropdown, and copies every page under it into the SDK.
The set tracks the docs navigation automatically — add a page to the nav
and it ships, no skill edits needed. The API reference and Guides &
examples dropdowns are intentionally excluded. A skill's `sources:`
frontmatter is now informational only.
The dropped idea of a dedicated `trigger-config` skill is replaced by
references to the bundled build-extension docs (`config/extensions/*`)
from the `trigger-authoring-tasks` config section and the chat-agent
skills.
## Summary
Typing in the search bar on the task page could clear or reset the input
mid-keystroke. This fixes the re-render race so the field stays stable
while you type.
## Root cause
Two things compounded:
- `SearchInput`'s sync effect depended on `text`, so it re-ran on every
keystroke and could overwrite the input with the URL/controlled value
while focused.
- Each task row unmounted and remounted its activity chart during the
side-panel open/close animation (25 charts at once), forcing heavy
re-renders that the search effect raced against.
## Fix
- `SearchInput` now tracks the last synced value in a ref instead of
comparing against `text`, keeping the effect off the keystroke path. It
only writes to state when the incoming URL/controlled value actually
changes, and never while the input is focused.
- Activity charts are now hidden (`hidden` attribute) instead of
unmounted during the panel animation, so the rows don't churn the tree
and the resize stays smooth.
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
## Summary
The agent skills' deep guidance now ships inside `@trigger.dev/sdk` and
is read from `node_modules`, so it tracks the `@trigger.dev/sdk` version
installed in your project automatically. This updates the Skills page,
the Building with AI step, and the rules-redirect page to drop the old
"pinned to the CLI version, re-run to refresh" framing and describe the
version-pinned reference instead.
Pairs with the SDK/CLI change in #3937. Keep this draft until that
ships, since it describes behavior that is not released yet.
## Summary
The Models page is now split into two tabs. **Your models** shows the
models your project has actually used in the selected time range, with
usage charts (cost over time, tokens over time, calls by model), a
per-model table of calls / cost / avg TTFC / avg tokens-per-sec, and
calls/tokens trend sparklines. **Model library** is the full catalog,
reordered from alphabetical to a relevance-based provider order
(Anthropic, OpenAI, Google, then the rest), newest models first within
each provider, with a "New" badge on models released in the last 7 days.
One time-range selector drives the whole Your models tab, so the charts,
the table, and the sparklines all share the same window. Opening a model
shows its own metrics with an independent range picker and a "View in AI
metrics" link that opens the AI metrics dashboard filtered to that
model. The active tab is kept in the URL so it survives a refresh and is
shareable.
## Prompt caching & cost accuracy
Both the Your models tab and the AI metrics dashboard now surface
prompt-cache usage: a cache-savings column plus per-model cached-tokens
and cache-hit-rate views, and a caching section on the dashboard (hit
rate, cached tokens, estimated savings, and hit rate by model).
Building this surfaced a cost bug. `input_tokens` is the total prompt
count and already includes cache-read and cache-creation tokens, but the
cost pipeline charged the full input at the input price and then added a
separate cache line, so cached tokens were billed twice (and on
Anthropic, cache reads were never discounted because their price is
keyed differently). The input price now applies only to the non-cached
remainder, with cache prices resolved across the provider-specific keys,
so LLM cost and the cache hit-rate metric are accurate. Hit rate is
computed as cached reads over total input.
## Notes
Also fixes React "invalid DOM property" console warnings from the
provider icons (the Llama and DeepSeek SVGs used raw `fill-rule` /
`clip-rule` / `clip-path` attributes), which this page surfaces by
rendering more provider icons.
## Screenshots
**Your models tab:** usage charts and a per-model table with
calls/tokens trend sparklines.
<img width="2560" height="1267" alt="1-your-models-tab"
src="https://github.com/user-attachments/assets/859bd24f-9047-4828-8bbb-83e5882846d6"
/>
**Model library:** provider-relevance ordering with a "New" badge on
models released in the last 7 days.
<img width="2560" height="1267" alt="2-model-library-tab"
src="https://github.com/user-attachments/assets/46dd54b9-80f9-4922-ade9-5935b08dfebc"
/>
**Model detail, Metrics tab:** per-model range picker and a "View in AI
metrics" link.
<img width="2560" height="1267" alt="3-model-detail-metrics"
src="https://github.com/user-attachments/assets/0f65d9d0-6142-4918-93f0-110bb277101a"
/>
**View in AI metrics:** the dashboard deep-linked and filtered to the
selected model.
<img width="2560" height="1267" alt="4-ai-metrics-filtered"
src="https://github.com/user-attachments/assets/821f256c-e305-493c-98c7-eafaf2f57f83"
/>
## Summary
The "What extractNewToolResults returns" reference in the
tool-result-auditing guide did not match the SDK. It listed an `input`
field that `chat.history.extractNewToolResults()` never returns, and
marked `output` as optional when it is always present.
This corrects the block to the real `ChatNewToolResult` shape
(`toolCallId`, `toolName`, `output`, optional `errorText`). Every usage
example in the same guide already reads only those fields, so the
reference now matches both the examples and the code.
## Summary
New `/ai-chat/prompt-caching` guide covering how to cache a chat agent's
prompt prefix with Anthropic prompt caching: the system prompt, the
conversation history (a `prepareMessages` breakpoint), and how caching
interacts with compaction. It also shows how to verify cache hits via
usage and the dashboard, the prefix-stability footguns, and an "Other
providers" section (OpenAI and Google cache automatically; Amazon
Bedrock uses `cachePoint` through `systemProviderOptions`).
Registered under Features in the AI Agents nav, next to Compaction.
---------
Co-authored-by: claude[bot] <41898282+claude[bot]@users.noreply.github.com>
Co-authored-by: Eric Allam <ericallam@users.noreply.github.com>
## Summary
`chat.headStart` now works with the `chat.customAgent` and
`chat.createSession` backends (not just `chat.agent`), and takes a
`triggerConfig` option. These docs cover both.
The Fast starts guide gets a "Handover with custom agents" section
showing how each backend consumes the handover (`consumeHandover`
returning `{ isFinal, skipped }` for custom agents, `turn.handover` for
createSession), including threading `originalMessages` so a resumed tool
round merges into the handed-over assistant. The `chat.headStart` API
section documents `triggerConfig` (tags, queue, machine, and the rest)
on the auto-triggered run.
The reference picks up `ChatTurn.handover`, `turn.complete()` with no
source, `chat.waitForHandover`, and a new `HeadStartHandlerOptions`
table.
Docs for the SDK changes in
[#3963](https://github.com/triggerdotdev/trigger.dev/pull/3963).
## Summary
Docs deploy from the `docs-live` branch via Mintlify, so merging to
`main` no longer publishes docs on its own. To publish, push a
`docs-release-*` tag at the commit you want live. The workflow runs the
Mintlify broken-links check against that commit, then fast-forwards
`docs-live` to it, which is what Mintlify deploys from.
## Design
The ref move uses the GitHub API with `force=false`, making it
fast-forward only: a tag that is not ahead of `docs-live` fails the job
rather than rewinding production. Mintlify's GitHub app reacts to the
resulting push and deploys, so no extra deploy credentials are needed.
Usage:
```bash
git tag docs-release-2026.06.16 # tag the main commit you want live
git push origin docs-release-2026.06.16
```
## Summary
Reworks the scheduled task page right-hand sidebar.
- Adds **Overview** / **Schedules** tabs. The Schedules tab is a
paginated table of all schedules attached to the task, declarative
first.
- Surfaces schedule fields (ID, CRON + human-readable description,
next/last run, status) directly in the Overview property table.
- Sidebar can be dragged much wider (up to 80% of the viewport).
- "No schedules attached" panel explains declarative vs imperative and
links to docs.
- Schedule **create / edit / enable / disable / delete** all happen
inside the existing Sheet — no more navigating to the standalone
schedule page. Toasts confirm each action.
## Test plan
- Open a scheduled task page and verify the new tabs
- Create, edit, enable/disable, and delete a schedule — confirm you stay
on the page and see a toast each time
- Visit a task with no schedules attached and confirm the info panel
renders
- Drag the sidebar wider; confirm pagination shows when there are >25
schedules
## Summary
`chat.headStart` (the warm step-1 fast path) previously handed its
response over only to `chat.agent`. This extends handover to the other
two backends: `chat.customAgent` consumes it with
`conversation.consumeHandover({ payload })` on turn 0, and
`chat.createSession` surfaces it as `turn.handover` (call
`turn.complete()` with no source to finalize a pure-text handover). The
low-level `chat.waitForHandover()` and `accumulator.applyHandover()` are
exported for hand-rolled loops.
It also adds `triggerConfig` to `chat.headStart()` and
`chat.openSession()`, so the auto-triggered handover-prepare run
inherits tags, queue, machine, and the other session run options the
same way `chat.createStartSessionAction()` does. The `chat:{chatId}` tag
is prepended automatically. Because the session is created once on the
first head-start turn (idempotent on the chat id), this is the only
place those options can be set for a head-start chat's lifetime.
## Fix: tool-call resume
When the warm step-1 hands over a pending tool call (rather than pure
text), the agent loop resumes that tool round. For it to merge cleanly
the pipe threads the spliced partial as `originalMessages`, so the
resumed tool-output chunk attaches to the handed-over tool-call instead
of throwing `No tool invocation found`. `MessageAccumulator.addResponse`
now also dedups by id (replace-in-place), so the persisted history
doesn't carry a duplicate assistant message when the resumed response
reuses the partial's id.
Incorporates the `triggerConfig` work from
[#3933](https://github.com/triggerdotdev/trigger.dev/pull/3933) by
@saasjesus, with `createStartSessionAction` extended to also forward
`maxDuration`, `region`, and `lockToVersion` so the two session entry
points stay consistent.
Verified end-to-end against a local environment: handover (pure-text and
tool-call) on both new backends, a `chat.agent` regression pass, and
`triggerConfig` tags and queue landing on the run.
---------
Co-authored-by: saasjesus <armin@chatarmin.com>
### Problem
Firestarter's `didWarmStart: true` means the response was written to a
long-poll socket — not that the runner received it. A silently dead
poller (no FIN, e.g. a VM torn down mid-poll) leaves the dispatched run
stuck in `PENDING_EXECUTING` until the run engine's heartbeat redrive,
and each redrive burns a queue redelivery toward
`TASK_RUN_DEQUEUED_MAX_RETRIES`.
### Change
After a warm-start hit, the supervisor retains the `DequeuedMessage`
(TimerWheel, default 10s), then probes the existing `getLatestSnapshot`
API. If the run is still on the exact dequeued snapshot, no runner ever
acted — it falls through to the regular cold-create path. Recovery: ~10s
+ cold start, no new APIs, no CLI changes.
- **Double-start safe**: `startRunAttempt` runs under a per-run lock and
409s stale snapshot ids, so a reviving runner and the fallback workload
can't both execute; the loser exits before running anything.
- **Probe errors → do nothing**: healthy runners legitimately act late
during platform brownouts (nested attempt-start retries), so falling
back on uncertainty would stampede duplicates. The heartbeat redrive
stays as the backstop (also covers supervisor restarts dropping timers).
- **Off by default**: `TRIGGER_WARM_START_VERIFY_ENABLED` (+
`TRIGGER_WARM_START_VERIFY_DELAY_MS`, 1–60s, default 10s). Disabled =
complete no-op. Works for all workload managers (compute/k8s/docker)
since it hooks the shared dequeue path.
- Emits `warmstart.verify` wide events (`outcome: delivered | fallback |
probe_error`), making the silent-loss rate directly measurable.
Currently the `db:seed` script just hangs on success.
This PR adds `process.exit(0)` to the finally block after db disconnect
so the script exits properly.
---------
Co-authored-by: Chris Arderne <chris@trigger.dev>
Adds `pnpm.overrides` pinning a few transitive deps to their current
releases:
- `js-cookie` → 3.0.7
- `tmp` → 0.2.7
- `brace-expansion` → 1.1.13 / 2.0.3 / 5.0.6 (one entry per major)
Each override is scoped to the affected major range so unaffected majors
aren't dragged forward. Also drops the `fast-xml-builder` override,
which no longer resolves to anything in the tree.
Lockfile-only - no published package's dependencies change.
`js-cookie`/`tmp` parents pin ranges that can't reach the new versions
on their own, so overrides (not a plain lockfile refresh) are needed to
hold them.
## Summary
The script that generates the changeset release PR description was
silently dropping some changelog entries and stripping code examples. In
[#3932](https://github.com/triggerdotdev/trigger.dev/pull/3932), entry
[#3937](https://github.com/triggerdotdev/trigger.dev/pull/3937) was
missing entirely from the Improvements list and
[#3952](https://github.com/triggerdotdev/trigger.dev/pull/3952)'s code
block was gone, even though both were present in the raw changeset
output.
## Root cause
`parsePrBody` parsed the raw changeset body line by line:
- The dependency-bump filter matched any entry whose text *began* with a
backticked package name, so a real changelog entry like ``
`@trigger.dev/sdk` now bundles... `` got thrown out along with the
genuine version-bump lines.
- Only the first line of each bullet was kept, so fenced code blocks,
sub-bullets, and continuation paragraphs were discarded.
## Fix
Group each top-level bullet with its indented continuation (code blocks,
sub-bullets, paragraphs), dedent it, and re-emit it intact. The
dependency filter is now anchored so it only matches lines that are
*entirely* a package bump, leaving real entries that merely start with a
package name.
Verified by replaying #3932's raw body through the script: #3937 returns
to the list, #3952's code block is preserved, and #3936's sub-bullets
nest correctly under their parent.
## Summary
`chat.agent`'s system prompt (the `chat.prompt` text plus any skills
preamble) could not carry a provider cache breakpoint, so the largest
and most stable part of the prompt re-paid full input price on every
turn. `chat.toStreamTextOptions()` now emits the system prompt as a
structured message carrying `providerOptions` when you opt in, so a
provider can cache the system block. Without an option, `system` stays a
plain string, so existing behavior is unchanged.
## API
Three ways to opt in (most specific wins, no deep merge):
```ts
// Anthropic sugar
chat.toStreamTextOptions({ cacheControl: { type: "ephemeral" } });
// provider-agnostic (also covers Amazon Bedrock's cachePoint)
chat.toStreamTextOptions({ systemProviderOptions: { anthropic: { cacheControl: { type: "ephemeral" } } } });
// at the definition site
chat.prompt.set(SYSTEM_PROMPT, { providerOptions: { anthropic: { cacheControl: { type: "ephemeral" } } } });
```
The `cacheControl` shorthand is Anthropic-only; `systemProviderOptions`
is the general form. Pairs with a `prepareMessages` cache breakpoint to
cache the conversation prefix too.
Docs guide: https://github.com/triggerdotdev/trigger.dev/pull/3951
### Summary
Self-serve billing UI is now hidden for managed-billing organizations.
Plan pickers, upgrade actions, billing alerts, and related upgrade
prompts are replaced with a "Contact us" option where appropriate.
Uses the new showSelfServe subscription flag, defaulting to true for
existing self-serve organizations.
### Testing
- [x] billing pages render correctly for self-serve organizations.
- [x] managed-billing organizations no longer see self-serve upgrade
flows.
- [x] "Contact us" actions are shown instead of upgrade actions where
applicable.
### Changelog
Hide self-serve billing flows for managed-billing organizations behind
the new showSelfServe subscription flag.
## Summary
Several optional workflow jobs fail on forks and private mirrors that
lack org-specific secrets or registry permissions. This adds per-job
repository-variable gates so those deployments can switch them off
without editing workflows — matching the pattern from #3901
(`ENABLE_CLAUDE_CODE` / `ENABLE_WORKFLOW_SECURITY_SCAN`).
Two variables, both **default-enabled** (a job runs unless its variable
is explicitly `'false'`), so canonical-repo behaviour is unchanged where
the variables are unset:
**`ENABLE_HELM_PRERELEASE`** — gates the chart-publish jobs that push to
`oci://ghcr.io/<owner>/charts` (needs `write_package` on the owner's
charts namespace):
- `helm-prerelease.yml` → `prerelease` job
- `release-helm.yml` → `release` job
Without the permission these fail with `403: denied: permission_denied:
write_package` on every PR / `helm-v*` tag. The `lint-and-test` jobs
(lint + template + kubeconform, no push) always run, so chart validity
is still enforced everywhere.
**`ENABLE_DEPENDABOT_ALERTS`** — gates the Dependabot notifier crons
that need `DEPENDABOT_ALERTS_TOKEN` / `SLACK_BOT_TOKEN` and post to a
specific Slack:
- `dependabot-critical-alerts.yml` → `alert` job (daily cron)
- `dependabot-weekly-summary.yml` → `summary` job (weekly cron)
On a fork/mirror these otherwise fire on schedule and fail (or post
nowhere) indefinitely.
## Test plan
- Variables unset (default): all jobs run as today.
- `ENABLE_HELM_PRERELEASE=false`: helm `lint-and-test` runs, publish
jobs skip — no 403 on repos lacking `write_package`.
- `ENABLE_DEPENDABOT_ALERTS=false`: the two cron jobs skip cleanly
(neutral, not failed).
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Major dashboard restructure plus the new task landing pages and
self-serve schedules add-on integration.
## Side menu
- Full restructure: standalone Tasks / Runs / Sessions block at the top;
new collapsible sections for AI, Observability, Deployments, Manage
- Persisted collapse state per section in `dashboardPreferences`
- New / updated icons across the menu
- Dashboards section: built-in Run metrics + AI metrics + custom
dashboards, with drag-to-reorder via ReactGridLayout
(`DashboardList.tsx`)
- DevPresence connection indicator in the env selector (DEV + V2)
## Tasks (`_index` — unified Tasks page)
- Replaces the separated Agents / Standard / Schedules listing pages
with one table
- New `UnifiedTaskListPresenter` composes `TaskListPresenter` +
`AgentListPresenter` (shared `currentWorker` lookup)
- Columns: Type (with kind badge), ID, File, Running (numeric for tasks;
running + suspended pills for agents), Activity (24h stacked-by-status),
sticky menu
- Search + "Task type" multi-select filter (URL-synced)
- Client-side pagination at 25/page
- Right-hand "useful links" panel (cookie-persisted state)
- Live-reload SSE: page revalidates on `WORKER_CREATED` so onboarding
`trigger dev` flips the blank state automatically
## Agent landing page (`/agents/$agentParam`)
- New per-agent detail page
- Top tabs (Sessions / Runs) toggle both the chart panel and the table
- Three dashboard-style chart cards: Sessions/Runs activity, LLM spend,
Tokens
- `AgentDetailPresenter` queries ClickHouse for run activity, session
activity (with FINAL on `sessions_v1`), and LLM cost/token activity from
`llm_metrics_v1`
- TimeFilter at the top drives all three charts
- Sticky table header, resizable horizontal handle, sidebar with Test
agent button + properties
- Docs link → `ai-chat/overview`
## Standard Task landing page (`/tasks/standard/$taskParam`)
- New per-task detail page mirroring the Agent layout
- `TaskDetailPresenter` for activity + properties
- Chart panel wrapped in a Card with "Runs by status" header
- Top bar with title, TimeFilter, pagination
- Right sidebar: Test task + identifier, queue, machine, retry, TTL,
payload schema, etc.
## Scheduled Task landing page (`/tasks/scheduled/$taskParam`)
- New per-task detail page mirroring the Agent / Standard layout
- Top-bar actions (right → left): pagination, Bulk replay…, View all
runs, TimeFilter, Create schedule
- Connected schedules mini-table in the sidebar
- **Self-serve schedules add-on integration** (reincarnated from the
now-removed `/schedules` listing page during the `origin/main` merge):
- Bottom usage bar pinned via `grid-rows-[auto_1fr_auto]` — progress
ring + "X/Y of your schedules" + Purchase / Upgrade / Request CTA
- At-limit "Create schedule" intercept dialog
- `PurchaseSchedulesModal` extracted as a shared component
(`apps/webapp/app/components/schedules/PurchaseSchedulesModal.tsx`)
handling increase / decrease / above-quota / need-to-delete states
- New resource action route at
`/resources/orgs/$organizationSlug/schedules-addon`
## Sessions
- Index page: list, filters, blank state, help tooltip rework
- Detail page: combined input/output chronological view (replaces split
tabs)
- Improved raw-message view layout (full-height)
- AI payload UI: `data-*` parts grouped under "AI SDK data parts:" label
- `toSafeUrl` helper guards rendered URLs from streamed content
- Fix: duplicate assistant content on inspector tab switch
## Playground (Test agent)
- Restructured top menu; back button + agent-selector popover
- Improved blank state
- Recent agent chat history moved into the tabbed menu
- Better message-scroll container (full height)
## Dashboards
- New Dashboards landing page (`/dashboards`) — Run metrics, AI metrics,
Create your own CTAs
- `BuiltInDashboards` updated; new `TasksDashboardPresenter` for the
tasks overview
- Custom dashboards section gains drag-to-reorder; cosmetic fix for
active-row drag-handle blending
## PageHeader / shared primitives
- `PageTitle` gains an `accessory` prop supporting string (auto-wrapped
in tooltip) and ReactNode
- Help tooltips on Tasks, Runs, Sessions PageTitles explaining the
concept and sub-categories
- `Card` primitive used for dashboard-style chart panels throughout
## Code review fixes (last batch on this branch)
- ClickHouse activity queries hardened: `FINAL` + `_is_deleted = 0` on
`task_runs_v2` (ReplacingMergeTree); `organization_id` + `project_id`
filters for sort-key prefix; `inserted_at` partition filter on
`llm_metrics_v1`
- `UnifiedTaskListPresenter`: shared `currentWorker` lookup;
slug-collision guard in `mergeRunningStates`; off-by-one fixed in 24h
bucket alignment
- `ScheduleListPresenter`: halved platform RPCs by deriving limit from
`currentPlan` instead of calling `getLimit`
- Sessions detail: stopped IntersectionObserver / scroll listener
re-attach on every chunk; `requestAnimationFrame` deferral on
auto-scroll to avoid virtualizer race
- URL hardening: `?types=` validated against known kinds; new
`parseFiniteInt` helper applied to `from`/`to`/`page` params
- AgentView: HITL resolution buffer now cleared once parts reach a
terminal state (was an unbounded Map on long sessions); subscription
effect deps documented with eslint suppression
- `PurchaseSchedulesModal`: bundle state resets on each open instead of
persisting stale drafts
## Manual testing
Manual smoke-test plan is tracked under
[TRI-10883](https://linear.app/triggerdotdev/issue/TRI-10883), broken
into 20 sub-issues covering onboarding, self-serve schedules, side menu,
the four landing pages, sessions, runs, dashboards, regressions and
performance.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
## Summary
The webapp Docker image build runs `pnpm run build --filter=webapp...`,
which builds `@trigger.dev/sdk` as a dependency. The SDK's `build`
script recently gained a `bundle-docs` step (`tsx
../../scripts/bundleSdkDocs.ts`), but the build couldn't run it in the
pruned image, breaking the image build.
Two things were missing:
- `docker/Dockerfile` copied `scripts/updateVersion.ts` into the builder
stage but not `scripts/bundleSdkDocs.ts`, so the step failed with
`ERR_MODULE_NOT_FOUND`.
- Even with the script present, the repo-level `docs/` tree it reads is
a separate workspace package that isn't in webapp's dependency graph, so
`turbo prune --scope=webapp` excludes it — the script's missing-docs
guard would then fail the build.
## Design
The Dockerfile now copies `bundleSdkDocs.ts` alongside
`updateVersion.ts`. `bundleSdkDocs.ts` skips gracefully when the repo
`docs/` tree is absent, which is exactly the pruned-dependency-build
case (the SDK is compiled there but never published). Publishing always
runs from the full monorepo where `docs/` exists, so the missing-docs
guard still protects releases — it only fires when `docs/` is present
but a cited doc is genuinely missing, rather than when the whole tree
was pruned away. This avoids dragging 27M of docs into a throwaway
builder stage.
## Test plan
- [x] `bundle-docs` from the full monorepo still bundles all cited docs
(exit 0)
- [x] Simulated pruned tree without `docs/` skips cleanly instead of
failing
- [ ] Webapp Docker image build succeeds in CI
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Two defensive fixes to the native realtime backend's run-change
publishing (behind a feature flag, off by default), so turning it on can
never destabilize the run lifecycle.
**Never throws at the caller.** Publish sites run synchronously on the
run-engine event bus and the metadata flush loop. The internal publish
was already wrapped in try/catch, but lazy construction (singleton +
metrics) and record encoding ran before that guard, so a throw could
propagate into a run lifecycle operation. The public
`publishChangeRecord` / `publishManyChangeRecords` helpers now wrap the
whole call and log-and-drop on failure.
**Bounds outage buffering.** The publisher connection caps
`maxRetriesPerRequest` at 1 (vs ioredis's default of 20), so during a
pub/sub Redis outage a publish rejects after ~1 reconnect cycle instead
of holding commands in memory for ~20s. A dropped publish is
latency-only, since the consumer has a periodic backstop full-resolve.
The offline queue stays on, so the first publish after a process boots
still flushes once the connection is ready.
## Summary
Add per-job `if:` gates so deployments that don't want — or can't run —
these jobs can switch them off via repository variables, without editing
workflows.
- `ENABLE_CLAUDE_CODE` gates the Claude jobs: interactive `@claude`, the
CLAUDE.md audit, and the REVIEW.md drift audit.
- `ENABLE_WORKFLOW_SECURITY_SCAN` gates the Zizmor job, which uploads
SARIF and so needs GitHub code scanning enabled.
Both default to **enabled**: a job runs unless its variable is
explicitly set to `'false'`, so behaviour is unchanged anywhere the
variables are unset. The sibling `actionlint` job and the report-only
Trivy scan are untouched.
## Test plan
- [x] `actionlint` clean on the four edited workflows
- [x] YAML parses for all four files
Two expected, non-failure conditions were being logged at `error` level,
which surfaces them as exceptions in error tracking and adds noise
without signal. This downgrades both to `warn`. The first is the
checkpoint-restore path when a `RESTORE` event already exists for a
checkpoint — a benign idempotency skip on a duplicate or retried event.
The second is the `/api/v1/token` endpoint when the authorization code
is invalid or expired, which is the expected steady state while the CLI
polls the endpoint during login; genuinely unexpected failures there
still log at `error`. No behavior or response changes — the token
endpoint still returns 400 in the same cases.
## Summary
Documents the Sessions HTTP API for non-SDK and server-to-server
callers, which until now appeared only in the conceptual
[ai-chat/sessions](https://trigger.dev/docs/ai-chat/sessions) page.
## What's covered
- **Sessions API reference** —
`create`/`list`/`retrieve`/`update`/`close` added to the OpenAPI spec
and a new "Sessions API" group (`management/sessions/*`), mirroring the
Runs API.
- **Channel endpoints** — a reference page for the `.in`/`.out` realtime
HTTP endpoints (append, SSE read, records drain), the wire protocol,
`Last-Event-ID` resume, and the per-direction auth boundary (`.out`
append is secret-key only).
- **Session scopes** — `read:sessions:{id}` / `write:sessions:{id}` in
the authentication docs, with the capability boundary and the 1h token
TTL.
Cross-linked with the SDK-side `ai-chat/sessions` page. Verified by
rendering each page on the Mintlify dev server.
## Summary
`@trigger.dev/sdk` now ships the Trigger.dev agent skills and a curated
snapshot of the docs those skills cite. The skills that `trigger skills`
installs into your coding agent are thin pointers that read this bundled
content from `node_modules`, so the guidance always matches the SDK
version installed in your project. Previously the full skill text was
copied into your repo at install time and went stale until you
reinstalled after an upgrade.
## How it works
The SDK's `files[]` now includes `skills/` (the full skill text) and
`docs/` (a curated snapshot generated at build time). The docs manifest
is derived from each skill's own `sources:` frontmatter, so a skill only
ships the docs it references, and a skill that cites a missing doc fails
the build.
The CLI installs thin skills whose body points the agent at
`node_modules/@trigger.dev/sdk/skills/<name>/SKILL.md` and
`node_modules/@trigger.dev/sdk/docs/`. They keep the high-value "Common
mistakes" anti-patterns inline so the trigger and the guardrails survive
even if the agent does not follow the pointer. `getting-started` stays
self-contained in the CLI because it runs before the SDK is installed.
## Summary
Three fixes that bring custom agent loops (`chat.customAgent`
hand-rolled loops and `chat.createSession`) up to the behavior
`chat.agent` users already get, and that the docs already promise:
- **Continuation runs no longer replay already-answered messages.** A
chat continuing after a cancel, crash, or upgrade re-delivered every
prior user message into the loop's first wait, so the model re-answered
an old message while the real new one had to arrive via steering. The
`.in` resume cursor is now seeded before any listener attaches, using
the same boot logic as `chat.agent`.
- **Mid-stream steering no longer wipes the in-flight response.**
`chat.pipeAndCapture` (also backing `turn.complete()`) streamed without
a server-generated message id, so a `prepareStep` injection regenerated
the assistant id mid-stream and the frontend replaced the partial
message, discarding everything streamed before the injection.
- **Task-backed tools now work from custom agent loops.** A child task
triggered via `ai.toolExecute` failed with "chat.agent session handle is
not initialized" because the parent's chatId only threaded from the
per-turn context that hand-rolled loops never set. It now falls back to
the session handle the `chat.customAgent` wrapper binds at run boot, so
children can stream progress into the chat with `chat.stream.writer({
target: "root" })` (the documented sub-agent pattern).
## Root cause on the replay fix
Attaching any `.in` listener (`chat.createStopSignal`,
`chat.messages.on`, the first wait) opens the SSE tail with
`Last-Event-ID` taken from the seq cursor at attach time. Custom loops
attached before any cursor existed, so S2 replayed from seq 0. The fix
resolves the cursor from the latest turn-complete header and seeds both
manager cursors (`setLastSeqNum` drives the SSE resume point,
`setLastDispatchedSeqNum` gates waiter dispatch) before attach;
`chat.createSession` now creates its stop signal lazily on the first
iteration, after the seed. Seeding only the first cursor after attach
does not work, which is why the earlier attempt at this was reverted.
All three were reproduced red-green against the references ai-chat
project: the replay repro showed the continuation wait consuming a stale
message in 403ms with the real message arriving via steering injection;
post-fix the wait consumes the real message directly with no injection.
Steering now preserves the full in-flight response, and the deepResearch
sub-agent streams its progress parts into a raw-loop parent. Existing
behavior verified unchanged: full SDK unit suite, `chat.agent` steering,
and stop-then-continue on `chat.createSession`.
### Summary
Queue searches previously executed both a count query and a page query
with identical filters. This PR switches filtered searches to `hasMore`
pagination, removing the extra count query while preserving existing
search behavior.
### Testing
cd apps/webapp && pnpm run test ./test/queueListSearch.test.ts --run
passes
### Changelog
Improve filtered queue search performance.
## Summary
Adds observability to the task metadata cache that backs the trigger hot
path. Follow-up to #3930, which made locked-version triggers fall back
to the primary when the read replica returns no row; this makes the
cache's effectiveness (and that fallback) measurable instead of
inferred.
## What it emits
A single bounded counter `task_meta_cache.resolve`, labeled by lookup
path (`locked` / `current`) and the source that satisfied it (`cache` /
`replica` / `writer` / `miss`):
- `cache / total` is the cache hit rate (its inverse is how cold the
cache runs).
- `writer / total` is how often the read replica returned empty for a
row the primary had (the condition #3930 recovers from).
Labels are bounded, with no per-env / worker / slug cardinality.
TRI-10873
Adds a `workload_create_duration_seconds` Prometheus histogram to the
supervisor, observed around the workload manager `create()` call:
- `backend` label: `kubernetes` | `compute` | `docker` — set once from
the configured workload manager
- `outcome` label: `success` | `error` — the per-outcome counts double
as a create error rate
Registered on the supervisor's existing metrics registry, so it's
exposed on the existing `/metrics` endpoint with no config changes.
Notes:
- Covers cold creates only; warm starts and restores return before
reaching `create()`.
- A create may include backend-internal retries, so one observation can
span multiple attempts.
- Fixed low cardinality: 2 active label sets per deployment × 10
buckets.
## Summary
`triggerAndWait` (and other locked-version triggers) could
intermittently fail with `Task '<id>' not found on locked version
'<version>'` for a task that was registered on that version. The
failures came in bursts and recovered on their own, so a retry minutes
later would succeed.
## Root cause
For a locked-version trigger, the queue resolver looks up the task's
`BackgroundWorkerTask` metadata from the read replica (behind a Redis
cache). On a cache miss it queried the replica, and a `null` result was
treated as "task not registered" and turned into a non-retryable 422. A
read replica can return an empty result for a row that already exists on
the primary, so a momentarily-behind replica produced a false negative
even though the locked worker (resolved on the primary in the same
request) clearly had the task.
## Fix
On a cache miss, when the replica returns no row the resolver now
re-checks the primary before concluding the task is missing. If the
primary has the row it is used (and the cache is back-filled); the error
fires only when the primary genuinely lacks it, which is the only case
where the 422 is correct. The extra read happens on the
cache-miss-and-replica-empty path only, so the hot path is unchanged.
Verified with a unit test (replica stub vs. real primary) and end-to-end
against a local streaming replica with replication paused to reproduce
the stale read.
TRI-10868
## Summary
Running `trigger.dev dev` before setting up a project crashed with a raw
`Cannot find matching package.json` stack trace from a transitive
dependency, instead of telling the user what to do next. It happens
whenever `dev` (or `update`) runs in a directory with no `package.json`
in it or any parent directory, for example right after creating an empty
project folder, or when `init` was exited before it scaffolded anything.
The CLI now detects the missing project and prints actionable guidance
pointing at `init`.
## Fix
`dev` runs an embedded package-version check before it loads any project
config. That check resolved `package.json` through a helper that throws
when nothing is found up the tree, and nothing caught it. It is now
wrapped, so a missing `package.json` produces a clear "run init" message
and a clean exit.
The config loader had the same latent crash on the `--skip-update-check`
path. Its resolvers for `package.json`, the lockfile, and the workspace
root all ran before the friendly "couldn't find your trigger.config.ts"
check, so any of them throwing masked it. That check now runs first and
short-circuits before the resolvers touch the filesystem.
Verified live: in an empty directory, `dev`, `dev --skip-update-check`,
and `update` all print a "run init" message and exit cleanly; in a
configured project, `dev` still resolves config and boots normally.
## Summary
Adds the 4.5.0-rc.6 entry to the AI chat changelog, covering the
chat-facing items shipping in
[#3870](https://github.com/triggerdotdev/trigger.dev/pull/3870): the
chat.agent reliability batch, the continuation boot latency fix, the
chat.headStart hydration and reasoning fixes, the chat.createSession
stop and continuation fixes, and the new trigger skills installer.
Should merge alongside the release so the changelog matches the
published version.
## Summary
Documents the two lower-level chat backend APIs and restructures the
Building agents section so it has a sane reading order.
**Custom agents page.** `chat.customAgent()` was effectively
undocumented (one passing mention) and `chat.createSession()` was buried
at the bottom of the Backend page, prompted by a customer asking whether
dropping down a level was supported at all. Both now live on one
dedicated page framed as a composition: register with `customAgent`,
then drive turns with the managed `createSession` iterator or a
hand-rolled primitives loop. The page covers the patterns the managed
lifecycle otherwise handles for you, each verified against a running
agent: seeding history on continuation runs (and why the seed must go
through the turn-0 `addIncoming`, which replaces the accumulator),
persisting the user message before streaming so a mid-stream reload
keeps it, racing `totalUsage` after a stop so the loop cannot wedge, and
the single-message wire shape.
**Backend page.** Now leads with a decision table across the three
abstraction levels and focuses on `chat.agent()`, routing to the new
page. Stale examples that read a plural `messages` field off the wire
payload are fixed (copy-pasting them broke turn accumulation), and the
ChatSessionOptions / ChatTurn reference tables gain their missing rows
(`compaction`, `pendingMessages`, usage fields, `setMessages`,
`prepareStep`).
**Anatomy page + reorder.** The Building agents group opened with the
long How it works mechanics page, a wall right after the Quick Start. A
short Anatomy page now leads the group: the three moving parts, one
annotated example where each region names the page that covers it, and a
routing table. How it works moves to the end of the group as the depth
payoff, matching where peer docs put their internals pages.
All pages visually verified against a local Mintlify build; cross-links
and anchors updated across the section.
## Summary
Two documentation improvements for the AI chat docs.
**Head-start persistence contract.** The fast starts page now documents
what your hooks can rely on across a head-start handover: one stable
assistant `messageId` for the whole turn, `onTurnComplete` as the
canonical persistence point, reasoning parts flowing into durable
history, and how Head Start composes with `hydrateMessages` (the
first-turn history arrives as `incomingMessages`, and the runtime
splices the warm partial onto the hydrated chain, deduplicated by id).
The hydrate examples on the lifecycle hooks and database persistence
pages now upsert their conversation row, since head-start first turns
run without a preload to create it.
**Sessions page.** The page opened with "a durable, task-bound,
bi-directional I/O channel pair", which reads as jargon and omitted run
orchestration entirely. It now leads with the plain mental model (a pair
of durable streams: input carries user messages, output carries
everything the agent produces) plus the Session's role orchestrating
runs, a diagram, a minimal runnable example, and a section on the
one-session-many-runs lifecycle.
Documents behavior shipping in
[#3907](https://github.com/triggerdotdev/trigger.dev/pull/3907).
## Summary
Accuracy fixes across the AI chat docs: drop the non-existent per-call
option from `transport.preload`, clarify that `onValidateMessages` only
fires on turns carrying incoming messages, soften the turn-complete
token-refresh wording (the header is optional), document the new
`onTurnComplete` `error` field and `finishReason`, and correct the
idle-timeout default to 30 seconds.
## Summary
Updates the AI-tooling docs for the new `trigger skills` installer that
shipped in #3868. The Skills page now documents `trigger skills` (skills
bundled with the CLI, version-matched to your SDK) and the four bundled
skills: `authoring-tasks`, `realtime-and-frontend`,
`authoring-chat-agent`, `chat-agent-advanced`. The old Agent Rules page
becomes a short "rules are now skills" redirect (kept because existing
redirects and the CLI link point at it), and the Building with AI
overview collapses the three-way Skills/Rules/MCP comparison into Skills
vs MCP.
Hold until the v4.5 CLI release ships, since `trigger skills` is not on
npm until then.
## Summary
Docs follow-up for #3683 (`TriggerClient` for per-instance SDK
configuration). Adds a dedicated reference page and threads the new
pattern through the existing management + preview-branches docs.
## What's in
**New page** `docs/management/multiple-clients.mdx` — when to use
`TriggerClient` vs `configure()` vs `auth.withAuth`, env-var fallback
rules, isolation contract, namespace surface, `inheritContext` opt-in,
and a when-to-use-what table.
**Updated pages**
- `docs/management/authentication.mdx` — rewrote the `auth.withAuth`
section to reflect the now-ALS-backed semantics (the prior version
warned about concurrency races and pointed at issue #3298 as a tracked
fix; that fix landed in #3683). Added `tr_preview_*` to the key prefix
list. Reframed the multi-target use case to lead with `TriggerClient`,
with `auth.withAuth` as the temporary-override helper.
- `docs/management/overview.mdx` — added a `Multiple clients in one
process` subsection.
- `docs/deployment/preview-branches.mdx` — added a `Triggering across
multiple branches from one process` example.
- `docs/triggering.mdx` — one-liner pointing at the new page for
cross-project triggering.
- `docs/docs.json` — slotted `management/multiple-clients` into the
Management API nav, right after authentication.
Paired with #3683.
## Test plan
- [ ] Mintlify preview renders cleanly
- [ ] Code samples in each updated page run as documented
- [ ] Cross-page links resolve (`/management/multiple-clients`,
`/management/authentication`)
## Summary
7 improvements, 1 bug fix.
## Improvements
- `trigger init` now sets up your AI coding assistant as part of project
setup: pick the MCP server, the agent skills, or both, then scaffold
with the CLI or hand off to your assistant. Adds a new `getting-started`
agent skill that teaches assistants how to bootstrap Trigger.dev
(install the SDK, write `trigger.config.ts`, create a first task, run
`trigger dev`), so the AI-driven setup path works end to end. It ships
in the CLI alongside the existing skills, version-matched to your SDK.
([#3872](https://github.com/triggerdotdev/trigger.dev/pull/3872))
- `dev` and `deploy` now fail with a clear error when two tasks are
defined with the same id, including across different task types (e.g. a
scheduled task and a regular task sharing an id). Previously the second
definition silently overwrote the first, so one of the tasks would
vanish with no warning. Task ids are detected as duplicates during
indexing (naming each offending id and the files it was found in), and
the same rule is enforced server-side when the background worker is
registered.
([#3865](https://github.com/triggerdotdev/trigger.dev/pull/3865))
- `trigger skills` installs Trigger.dev agent skills into your coding
agent so it knows how to write tasks, schedules, realtime, and
chat.agent code. The skills ship with the CLI and are copied into each
tool's native skills directory (Claude Code, Cursor, GitHub Copilot, and
Codex / AGENTS.md), and `trigger dev` offers to install them on first
run. ([#3868](https://github.com/triggerdotdev/trigger.dev/pull/3868))
- Reliability fixes for `chat.agent`. A user message sent while the
agent is streaming is no longer delivered twice (which could run a
duplicate turn), input appends now carry an idempotency key so a retried
send can't duplicate a message, stopping a generation clears the
streaming state so a page reload doesn't replay the stopped turn, and
runs can now carry the full set of dashboard tags instead of being
silently truncated. `onTurnComplete` now fires on errored turns (with
the thrown error attached) and the failed turn's user message is
persisted so it isn't lost on the next run. Custom agents and manual
`chat.writeTurnComplete` callers now trim the output stream, sending a
custom action no longer leaves a second stream reader running, and a
long-lived `watch` subscription no longer grows its dedupe set without
bound. ([#3891](https://github.com/triggerdotdev/trigger.dev/pull/3891))
- Continuation chat boots no longer stall for around 10 seconds before
the first turn. The `session.in` resume cursor is now found with a
non-blocking records read instead of draining an SSE long-poll (which
always waited out its full 5 second inactivity window, twice per boot),
the boot reads run concurrently, and chat snapshots carry the cursor so
subsequent boots skip the scan entirely.
([#3907](https://github.com/triggerdotdev/trigger.dev/pull/3907))
- Record client-side dequeue API latency in the supervisor consumer pool
as a Prometheus histogram
(`queue_consumer_pool_dequeue_duration_seconds`, labelled by `outcome`:
success/empty/error).
([#3887](https://github.com/triggerdotdev/trigger.dev/pull/3887))
- Add `GetProjectEnvironmentsResponseBody` and `ProjectEnvironment`
schemas for the new `GET /api/v1/projects/{projectRef}/environments`
endpoint, which lists the parent environments (dev, staging, preview,
prod) a personal access token can access for a project. Dev is scoped to
the token owner and branch (preview child) environments are excluded.
([#3880](https://github.com/triggerdotdev/trigger.dev/pull/3880))
## Bug fixes
- Fix two `chat.createSession()` bugs: stopping a generation no longer
wedges the run (the turn loop raced a `totalUsage` promise that never
settles after a stop-abort), and continuation runs now wait for the next
message instead of invoking the model with an empty prompt.
([#3920](https://github.com/triggerdotdev/trigger.dev/pull/3920))
<details>
<summary>Raw changeset output</summary>
⚠️⚠️⚠️⚠️⚠️⚠️
`main` is currently in **pre mode** so this branch has prereleases
rather than normal releases. If you want to exit prereleases, run
`changeset pre exit` on `main`.
⚠️⚠️⚠️⚠️⚠️⚠️
# Releases
## @trigger.dev/build@4.5.0-rc.6
### Patch Changes
- Updated dependencies:
- `@trigger.dev/core@4.5.0-rc.6`
## trigger.dev@4.5.0-rc.6
### Patch Changes
- `trigger init` now sets up your AI coding assistant as part of project
setup: pick the MCP server, the agent skills, or both, then scaffold
with the CLI or hand off to your assistant. Adds a new `getting-started`
agent skill that teaches assistants how to bootstrap Trigger.dev
(install the SDK, write `trigger.config.ts`, create a first task, run
`trigger dev`), so the AI-driven setup path works end to end. It ships
in the CLI alongside the existing skills, version-matched to your SDK.
([#3872](https://github.com/triggerdotdev/trigger.dev/pull/3872))
- `dev` and `deploy` now fail with a clear error when two tasks are
defined with the same id, including across different task types (e.g. a
scheduled task and a regular task sharing an id). Previously the second
definition silently overwrote the first, so one of the tasks would
vanish with no warning. Task ids are detected as duplicates during
indexing (naming each offending id and the files it was found in), and
the same rule is enforced server-side when the background worker is
registered.
([#3865](https://github.com/triggerdotdev/trigger.dev/pull/3865))
- `trigger skills` installs Trigger.dev agent skills into your coding
agent so it knows how to write tasks, schedules, realtime, and
chat.agent code. The skills ship with the CLI and are copied into each
tool's native skills directory (Claude Code, Cursor, GitHub Copilot, and
Codex / AGENTS.md), and `trigger dev` offers to install them on first
run. ([#3868](https://github.com/triggerdotdev/trigger.dev/pull/3868))
```bash
trigger skills --target claude-code
```
Replaces the previous `install-rules` command, which stays as an alias.
- Updated dependencies:
- `@trigger.dev/core@4.5.0-rc.6`
- `@trigger.dev/build@4.5.0-rc.6`
- `@trigger.dev/schema-to-json@4.5.0-rc.6`
## @trigger.dev/core@4.5.0-rc.6
### Patch Changes
- Reliability fixes for `chat.agent`. A user message sent while the
agent is streaming is no longer delivered twice (which could run a
duplicate turn), input appends now carry an idempotency key so a retried
send can't duplicate a message, stopping a generation clears the
streaming state so a page reload doesn't replay the stopped turn, and
runs can now carry the full set of dashboard tags instead of being
silently truncated. `onTurnComplete` now fires on errored turns (with
the thrown error attached) and the failed turn's user message is
persisted so it isn't lost on the next run. Custom agents and manual
`chat.writeTurnComplete` callers now trim the output stream, sending a
custom action no longer leaves a second stream reader running, and a
long-lived `watch` subscription no longer grows its dedupe set without
bound. ([#3891](https://github.com/triggerdotdev/trigger.dev/pull/3891))
- Continuation chat boots no longer stall for around 10 seconds before
the first turn. The `session.in` resume cursor is now found with a
non-blocking records read instead of draining an SSE long-poll (which
always waited out its full 5 second inactivity window, twice per boot),
the boot reads run concurrently, and chat snapshots carry the cursor so
subsequent boots skip the scan entirely.
([#3907](https://github.com/triggerdotdev/trigger.dev/pull/3907))
- Record client-side dequeue API latency in the supervisor consumer pool
as a Prometheus histogram
(`queue_consumer_pool_dequeue_duration_seconds`, labelled by `outcome`:
success/empty/error).
([#3887](https://github.com/triggerdotdev/trigger.dev/pull/3887))
- `dev` and `deploy` now fail with a clear error when two tasks are
defined with the same id, including across different task types (e.g. a
scheduled task and a regular task sharing an id). Previously the second
definition silently overwrote the first, so one of the tasks would
vanish with no warning. Task ids are detected as duplicates during
indexing (naming each offending id and the files it was found in), and
the same rule is enforced server-side when the background worker is
registered.
([#3865](https://github.com/triggerdotdev/trigger.dev/pull/3865))
- Add `GetProjectEnvironmentsResponseBody` and `ProjectEnvironment`
schemas for the new `GET /api/v1/projects/{projectRef}/environments`
endpoint, which lists the parent environments (dev, staging, preview,
prod) a personal access token can access for a project. Dev is scoped to
the token owner and branch (preview child) environments are excluded.
([#3880](https://github.com/triggerdotdev/trigger.dev/pull/3880))
## @trigger.dev/python@4.5.0-rc.6
### Patch Changes
- Updated dependencies:
- `@trigger.dev/sdk@4.5.0-rc.6`
- `@trigger.dev/core@4.5.0-rc.6`
- `@trigger.dev/build@4.5.0-rc.6`
## @trigger.dev/react-hooks@4.5.0-rc.6
### Patch Changes
- Updated dependencies:
- `@trigger.dev/core@4.5.0-rc.6`
## @trigger.dev/redis-worker@4.5.0-rc.6
### Patch Changes
- Updated dependencies:
- `@trigger.dev/core@4.5.0-rc.6`
## @trigger.dev/rsc@4.5.0-rc.6
### Patch Changes
- Updated dependencies:
- `@trigger.dev/core@4.5.0-rc.6`
## @trigger.dev/schema-to-json@4.5.0-rc.6
### Patch Changes
- Updated dependencies:
- `@trigger.dev/core@4.5.0-rc.6`
## @trigger.dev/sdk@4.5.0-rc.6
### Patch Changes
- Reliability fixes for `chat.agent`. A user message sent while the
agent is streaming is no longer delivered twice (which could run a
duplicate turn), input appends now carry an idempotency key so a retried
send can't duplicate a message, stopping a generation clears the
streaming state so a page reload doesn't replay the stopped turn, and
runs can now carry the full set of dashboard tags instead of being
silently truncated. `onTurnComplete` now fires on errored turns (with
the thrown error attached) and the failed turn's user message is
persisted so it isn't lost on the next run. Custom agents and manual
`chat.writeTurnComplete` callers now trim the output stream, sending a
custom action no longer leaves a second stream reader running, and a
long-lived `watch` subscription no longer grows its dedupe set without
bound. ([#3891](https://github.com/triggerdotdev/trigger.dev/pull/3891))
- Continuation chat boots no longer stall for around 10 seconds before
the first turn. The `session.in` resume cursor is now found with a
non-blocking records read instead of draining an SSE long-poll (which
always waited out its full 5 second inactivity window, twice per boot),
the boot reads run concurrently, and chat snapshots carry the cursor so
subsequent boots skip the scan entirely.
([#3907](https://github.com/triggerdotdev/trigger.dev/pull/3907))
- Fix `chat.headStart` when `hydrateMessages` is registered. The warm
route's step-1 partial now reaches the agent's accumulator on the
hydrate path, so `onTurnComplete` carries the full first turn (the
head-start user message included), tool-call handovers resume from step
2 instead of re-running step 1, and the assistant `messageId` stays
stable across the handover.
([#3907](https://github.com/triggerdotdev/trigger.dev/pull/3907))
- Preserve reasoning parts across the `chat.headStart` handover.
Extended-thinking models' step-1 reasoning now lands in the durable
session history (and `onTurnComplete`) under the same assistant
`messageId`, with provider metadata intact so Anthropic thinking
signatures survive replays.
([#3907](https://github.com/triggerdotdev/trigger.dev/pull/3907))
- Fix two `chat.createSession()` bugs: stopping a generation no longer
wedges the run (the turn loop raced a `totalUsage` promise that never
settles after a stop-abort), and continuation runs now wait for the next
message instead of invoking the model with an empty prompt.
([#3920](https://github.com/triggerdotdev/trigger.dev/pull/3920))
- Updated dependencies:
- `@trigger.dev/core@4.5.0-rc.6`
## @trigger.dev/plugins@4.5.0-rc.6
### Patch Changes
- Updated dependencies:
- `@trigger.dev/core@4.5.0-rc.6`
</details>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
## Summary
Two `chat.createSession()` bugs that break chats at its abstraction
level:
1. **Stopping a generation wedged the run forever.** `turn.complete()`
bare-awaited the AI SDK's `totalUsage` promise, which never settles
after a stop-abort. The run stayed stuck inside the stopped turn (trace
shows a permanently partial `ai.streamText` span and no further `waiting
for next message`), so the chat could never take another message. Fixed
with the same 2s `Promise.race` guard `chat.agent`'s turn loop already
uses.
2. **Continuation runs invoked the model with an empty prompt.** The
first turn only waited for a message on `preload` boots. A continuation
run (spawned after a cancel, crash, or version upgrade) arrives with the
boot payload stripped, so the loop ran a turn with zero messages and
errored with `AI_InvalidPromptError: messages must not be empty`.
Message-less continuation boots now wait for the next session input
("waiting for first message (continuation)"), and `turn.continuation` is
preserved across the wait so user code can seed stored history off it.
Both reproduced and verified end-to-end against a live environment (stop
followed by a next turn; cancel followed by a continuation turn with
seeded history), plus the existing unit suite.
## Summary
Two read-replica races on the session APIs could break chats whose first
activity lands inside the replication window (or any time the replica
lags):
1. A session's first `.in` append or `.out` subscribe could fail with a
404 for a session that exists on the writer, because the route resolved
the Session row on the replica only.
2. `ensureRunForSession` probed run liveness on the replica, so a probe
miss on a run triggered moments earlier was judged "run is dead" and a
second live run was spawned for the same session. Both runs then
consumed the same input stream, producing duplicated turns and doubled
responses (and doubled LLM cost).
## Fix
Liveness now re-probes the writer before declaring the current run dead
(the old code already fell back to the writer, but only to recover the
friendlyId, after the wrong verdict was made). Session resolution on the
append and subscribe/init routes goes through a new
`resolveSessionWithWriterFallback`, which stays replica-first on the hot
path and only touches the writer on a miss.
Reproduced and verified against a local streaming replica with an
artificial apply delay: pre-fix, a send immediately after session
creation reliably produced either the 404 or two executing runs with a
doubled response; post-fix, the same flow produces exactly one run and
one response.
Also rides along: the local docker replica's default apply delay drops
from 150ms to a realistic 20ms (override via `REPLICA_APPLY_DELAY` when
you want to deliberately widen the race window).
`@trigger.dev/plugins` re-exported
`sanitizeBranchName`/`isValidGitBranchName` from `@trigger.dev/core` as
a convenience forwarder. Nothing actually imports them through this
package — every consumer (webapp, `@trigger.dev/rbac`, …) imports them
directly from `@trigger.dev/core/v3/utils/gitBranch`.
Removing the forwarder keeps the package entry free of **runtime** core
imports (only type re-exports + `buildJwtAbility` remain), so consumers
that bundle `@trigger.dev/plugins` from source don't pull an unrelated
core subpath into their build.
No behavior change; the helpers remain available from
`@trigger.dev/core` where they're defined.
## Problem
The item-streaming endpoint of the two-phase batch API (`POST
/api/v3/batches/:batchId/items`) processed streamed items strictly
sequentially. For a batch of many large payloads, each offloaded to
object storage inline, this serialized N object-store round-trips inside
a single request and could exceed Node's default `server.requestTimeout`
(300s). The webapp then returned `408`, which the SDK reads as `408
terminated` and retries up to 5 times, turning a slow ingest into a
failure that takes tens of minutes to surface.
## Fix
Ingest now runs through `p-map` over the NDJSON async iterable with
bounded concurrency (`STREAMING_BATCH_INGEST_CONCURRENCY`, default 10):
- `p-map` pulls lazily from the stream, so at most `concurrency` items
are read and in-flight at once. Peak memory stays bounded to roughly
`concurrency × STREAMING_BATCH_ITEM_MAXIMUM_SIZE` and request-body
backpressure is preserved.
- Set the env to `1` for fully sequential ingestion (escape hatch).
## Why this is safe (ordering and idempotency unchanged)
- Ordering derives from each item's index (enqueue `timestamp =
batch.createdAt + index`), not enqueue order.
- Dedup is atomic per index in `enqueueBatchItem`.
- The NDJSON parser now stamps oversized-item markers with their emit
position, removing the consumer's sequential `lastIndex` assumption (the
only order-dependent bit).
- The count-check and conditional-seal path is untouched.
## Scope
This speeds up every batch ingested through the streaming endpoint, not
just large-payload batches. Each item does a per-item Redis enqueue
regardless of size, and those now overlap. Large payloads benefit most
because they add an object-store offload round-trip on top of the
enqueue.
## Verification
Added an integration test (`streamBatchItems.test.ts`) that drives the
real service against Postgres + Redis + RunEngine and times a 150-item
batch at increasing concurrency. Object-store offload is modelled as a
fixed per-item latency (local round-trips are too small to compare
meaningfully):
```
runCount=150
large payloads (10ms/item offload):
concurrency=1 1739ms
concurrency=10 192ms (9.1x faster)
concurrency=50 57ms (30.7x faster)
small payloads (Redis enqueue only, no offload):
concurrency=1 90ms
concurrency=10 24ms (3.7x faster)
```
The test asserts correctness at every concurrency (all items accepted,
sealed, enqueued exactly once), that parallel ingest beats the
sequential floor, and that the small-payload case is strictly faster
than sequential, so the win is not specific to large payloads.
Also exercised end-to-end over real HTTP against a local server: a
20-item batch (12MB body) ingests and seals, a re-stream of the sealed
batch returns `sealed: true` with zero re-accepted items (idempotent
retry), and an oversized item still seals at its correct index.
Existing coverage stays green: concurrent ingest of a 100-item batch,
in-flight processing never exceeding the configured concurrency,
concurrent dedup on streaming retry, and emit-position marker indexing.
## Follow-ups (not in this PR)
- SDK pre-offload of large item payloads (send `application/store` refs
instead of raw blobs) to remove object-store work from the request hot
path and shrink the request body.
- Optional `server.requestTimeout` bump as a safety net.
## CI fix
Added `.github/workflows/codeql.yml` to replace GitHub's automatic
("dynamic") CodeQL scanning. The dynamic setup was failing to upload
SARIF results because the auto-generated `GITHUB_TOKEN` lacked the
`security-events: write` permission. The explicit workflow grants that
permission at the job level and pins all actions to commit SHAs,
consistent with the repo's security conventions.
## ✅ Checklist
- [ ] I have followed every step in the [contributing
guide](https://github.com/triggerdotdev/trigger.dev/blob/main/CONTRIBUTING.md)
- [ ] The PR title follows the convention.
- [ ] I ran and tested the code works
---
## Testing
- Integration test (`streamBatchItems.test.ts`) validates correctness
and performance at concurrency 1, 10, and 50 for both large and small
payloads.
- End-to-end verified over real HTTP: 20-item/12MB batch ingests and
seals, idempotent retry returns `sealed: true`, oversized item seals at
correct index.
---
## Changelog
Streaming batch ingest now processes items with bounded concurrency
instead of one at a time, so batches of many large payloads ingest far
faster and no longer time out. Concurrency is configurable via
`STREAMING_BATCH_INGEST_CONCURRENCY` (default 10); set it to 1 for fully
sequential ingestion.
---
## Screenshots
_[Screenshots]_
💯
---------
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
## What
`buildJwtAbility` — the decoder for public-token scope strings
(`read:tags:…`, `read:runs:run_abc`, `admin`, …) — now lives in
`@trigger.dev/plugins` as the single source of truth.
`@trigger.dev/rbac` re-exports it, so the built-in fallback and any auth
plugin interpret a token identically.
Scope strings are split on only the first **two** colons
(`action:type:id`), so a resource id that itself contains colons — e.g.
a tag like `user:123` — is matched in full rather than truncated to its
first segment. (The fallback already did this; this makes it the one
shared implementation.)
`@trigger.dev/plugins` is now **private (unpublished)** and gains a
`@triggerdotdev/source` export condition, so consumers bundle it from
source per-commit like `@trigger.dev/core` instead of resolving a
published version — no cross-version coordination.
## Why
Two hand-maintained copies of the scope grammar drift, and the
difference silently changes what a token grants. One shared decoder
removes that class of bug.
## Notes
- No changeset: `@trigger.dev/plugins` is now private and
`@trigger.dev/rbac` is internal — neither is published.
- Unit coverage for the colon-id path lives in
`internal-packages/rbac/src/ability.test.ts` (now exercising the shared
function).
## Summary
When the realtime runs feed (the backend behind the `realtimeBackend`
feature flag) hydrates a change from a Postgres read replica, the read
can race the replica's apply of the very write that triggered it. The
delivered row then carries the previous change's content, and an
isolated final change (for example a last `metadata.set` before a run
goes quiet) is not corrected until the roughly 20 second backstop poll.
Measured against a replica with deliberate apply delay, every delivery
trailed exactly one change behind and a final change stranded for the
full backstop interval.
## Fix
Publishers stamp each change record with the committed row's
`updatedAt`, taken from writes they already perform, so the stamp costs
no extra queries. The router delays its wake hydrate until the replica's
measured lag has passed, anchored to that timestamp: a record that has
already spent longer than the lag in transit is hydrated immediately, so
only the racing leading edge ever waits. After hydrating, a tripwire
compares each row against its record's watermark. Still-stale rows are
withheld and retried briefly, and each detection feeds the lag estimate.
If retries run out, the rows are delivered anyway (liveness over
freshness) and follow-up re-hydrates emit the fresh version through the
normal working-set diff once the replica catches up, with the backstop
as the terminal net.
Replica lag is sampled reader-side only, and only while feeds are
active. Aurora reports live lag via `aurora_replica_status()`; vanilla
Postgres can only report "caught up or not" (mid-apply lag is not
honestly measurable from a replica), so tripwire observations floor the
estimate there. Deployments without a replica resolve to zero lag and
skip the gate entirely. Tunables live under
`REALTIME_BACKEND_NATIVE_REPLICA_LAG_*`, and
`realtime_native.stale_hydrates` plus
`realtime_native.replica_lag_estimate_ms` make replica health
observable.
Two adjacent fixes: a metadata update that writes nothing no longer
publishes a change record, and buffered parent and root metadata
operations now publish when the flusher writes them, so those changes
wake live feeds instead of waiting for the backstop.
For local testing, `docker-compose` gains an opt-in `database-replica`
service (compose profile `replica`) with a configurable
`recovery_min_apply_delay`, which reproduces replica-lag behavior
deterministically. With the gate disabled this rig reproduces the
one-change-behind delivery exactly; with it enabled, deliveries arrive
with current content at roughly the true replica lag, across write rates
faster and slower than the lag itself.
## Summary
The environment variables page loaded every variable value in the
project, unfiltered by environment. Archiving a preview branch does not
delete its environment variable value rows, so projects that churn
preview branches accumulate values forever, and every page view loaded
all of them. On large projects this made the page loader take many
seconds and stalled the server while deserializing the oversized result.
## Fix
The presenter now loads the displayed environments first and filters the
`values` relation to those environment IDs. That matches the display
semantics exactly (per-user dev environments and active branch
environments included), and the lookup is covered by the existing unique
index on `(variableId, environmentId)`. Values in archived branch
environments are no longer fetched at all.
Covered by a new testcontainers test asserting that values from active
environments (including branch environments) are returned while archived
branch environments are excluded.
## Summary
`test/runsRepositoryCursor.test.ts` pinned its fixture runs to
`createdAt = 2026-06-04T16:55:07Z`. `listRuns` applies the default 7 day
window when no time filter is given, so the fixtures aged out of the
window at 16:55 UTC on 2026-06-11 and all five tests started failing for
every branch, regardless of what the branch changed. The tests were
green on their own CI two days earlier because the fixtures were only
five days old at the time.
This switches the fixture base to a relative timestamp (one hour ago),
so the fixtures stay inside the default window permanently. Verified the
suite goes 5/5 green with this change on the same environment where the
pinned dates fail 5/5.
## Summary
The run trace page loader serialized every span's raw OTel events (with
full properties) into the response, even though the tree UI only renders
the derived `timelineEvents` and the span detail panel refetches what it
needs. On event-heavy traces that inflated both the loader payload and
the server-side heap copies built per request. This PR keeps raw span
events server-side and pairs that with a few related trace-view
improvements:
- A new optional `TRACE_VIEW_EMERGENCY_SPAN_CAP` env var (unset by
default) clamps the trace summary and detailed trace summary span limits
on both event store paths, including the public run trace endpoint, so
operators can bound trace query sizes in one place without retuning the
per-store limits.
- The TreeView virtualizer resolved every rendered row with a linear
scan over the whole tree (and `getNodeProps` did the same via
`findIndex`); rows now resolve through memoized id lookup maps, which
matters once traces reach tens of thousands of spans.
- The run stream SSE lookup now applies the same organization membership
scoping as the rest of the run page presenters, for consistency.
Behavior is unchanged by default: the trace tree renders from the same
`timelineEvents` it always has, and the new cap only takes effect when
set.
## Summary
Three related fixes for `chat.headStart` and continuation boots, found
while investigating customer reports.
**1. `chat.headStart` now works with `hydrateMessages`.** The turn-0
handover splice only ran on the default accumulation path, so agents
registering `hydrateMessages` silently lost the warm route's step-1
response: pure-text turns fired `onTurnComplete` with no assistant
message (and an empty durable write), tool-call turns re-ran step 1 from
scratch under a fresh `messageId`, and the head-start user message never
reached the hydrate hook at all. The first-turn history now reaches
`hydrateMessages` as `incomingMessages`, and the splice runs after both
accumulation branches, deduplicated by the handover `messageId`.
**2. Reasoning parts survive the handover.** The synthesized partial
only mapped text and tool-call parts, so an extended-thinking model's
step-1 reasoning streamed to the browser but never reached durable
history. Reasoning parts now map through with provider metadata, so
Anthropic thinking signatures survive a UIMessage round trip on hydrate
replays.
**3. Continuation boots no longer stall for ~10 seconds.** The `.in`
resume cursor was found by draining an SSE subscription that only closes
after its 5 second inactivity window, and the scan ran twice per boot.
It is now a non-blocking records read of the latest turn-complete
header, runs at most once per boot, the boot reads run concurrently, and
chat snapshots carry the cursor so subsequent boots skip the scan
entirely. Measured locally on a cancel-then-continue repro: pre-turn
continuation latency dropped from ~11s to ~0.5s.
Every fix was verified red-green: new unit tests reproduced each failure
before the fix, and end-to-end smoke tests against a live local stack
covered both handover legs, reasoning persistence with extended thinking
(including a follow-up turn that round-trips the persisted signed
reasoning back to the provider), and the boot timing comparison.
## Rollout
SDK-only; no server change required. A new SDK against a server that
does not serialize record headers degrades to the existing no-cursor
fallback. Old SDKs ignore the new snapshot field, and new SDKs fall back
to the records scan on snapshots written before it existed.
## Summary
HIPAA BAA is offered as a paid add-on on every paid plan. Each paid tier
on the in-app pricing card now has a "HIPAA BAA add-on" row with a
"Request a BAA" link that opens the existing contact dialog pre-filled
with a new `hipaa` inquiry type, prompting the user for their company
name and a brief description of the PHI workload.
The contact form's `feedbackTypes` are restructured to match the
marketing /contact form: every inquiry type carries a Plain label ID and
a "Contact form: ..." thread title, so threads land in Plain identically
whether they come from the dashboard or the marketing site. The
included-compute line on each tier also picks up the credits wording
from the marketing pricing page, and the Enterprise tier lifts its title
above the features row.
The compute suspend flow delays snapshots by `snapshotDelayMs` (~30s) so
short-lived waitpoints skip the snapshot entirely, with the intent that
a run continuing before the delay expires cancels the pending snapshot.
But the only `cancel()` call site was the `/continue` action, which
runners only invoke when restoring from an already-taken snapshot — so
pending snapshots were never cancelled (zero `snapshot.canceled` events
ever emitted in prod). When a run resumed and completed inside the
window, the stale snapshot fired ~30s later anyway, pausing the VM 6–13s
mid warm-start long-poll; the frozen guest couldn't fire its abort timer
or send a FIN, causing stalls and run-engine driven retries.
### Change
- Cancel the pending snapshot on `attempt.complete` — after the platform
accepts the completion, before the HTTP reply (so it can't reorder with
the runner's next `/suspend`).
- Cancel on `runDisconnected` (crash, exit, or run replaced on the
socket).
- Both cancels are guarded by a runnerId match (new
`TimerWheel.peek()`): a stale duplicate runner for a reassigned run must
not cancel the fresh runner's pending snapshot. A missing runnerId falls
through to an unconditional cancel (the pre-existing `/continue`
behavior is unchanged).
Waitpoint suspensions keep the runner socket connected and the attempt
incomplete, so neither hook touches a snapshot that is still wanted.
Known limitation (fail-safe direction): `socket.data.runnerId` is frozen
at the websocket handshake, so after a same-supervisor restore the
disconnect-path guard refuses the cancel. The `attempt.complete` path
uses the runner's current header id and is unaffected.
`ComputeWorkloadManager.create` swallows gateway errors currently, so a
cold start that fails placement (e.g. a netns slot with a busy tap, a
full node disk) silently abandons the dequeued run until the run
engine's `PENDING_EXECUTING` heartbeat timeout redrives it via stall
detection.
### Changes
- Retry `instances.create` with short backoff (default 3 attempts, 250ms
backoff), recording `createAttempts` in the wide event.
- **Only statuses where the create definitely did not commit are
retried**: 500 (agent/fcrun create failed) and 503 (no placement).
502/504 are excluded — the gateway emits those when it fails to reach
the node or read its response, which can happen *after* the agent
committed the create; the gateway only records the instance name on a
clean 201, so a same-name retry would miss the collision check and could
double-create the VM on another node. Network-level fetch failures are
retried (if the gateway processed the create, its name index is
populated and the retry 409s harmlessly). Timeouts are not retried.
- **Retry attempts after a 5xx use a deterministic `-rN` name suffix**:
a failed create can leave its name registered until async cleanup runs.
Attempt 1 keeps the unsuffixed name.
Fixes an intermittent `Test timed out in 30000ms` in the
`streamBatchItems` suite. Not a logic hang — the 30s budget covers
container setup, and each case boots its own per-test Redis container +
a full `RunEngine`, so under CI Docker contention a cold boot can cross
30s (which is why the failure moved between tests).
- New `containerTestWithIsolatedRedisNoClickhouse` fixture (Postgres
clone + per-test Redis, no ClickHouse) — this suite never uses
ClickHouse, but the old fixture's auto `resetClickhouse` forced a
ClickHouse boot + migration onto the cold-start test.
- Raised `testTimeout` 30s → 120s, matching the run-engine convention
for this footprint.
## Summary
A batch of reliability fixes for `chat.agent`:
- A user message sent while the agent is streaming is no longer
delivered twice (which could run a duplicate turn).
- Input appends carry an idempotency key (`X-Part-Id`) so a retried send
can't duplicate a message.
- `onTurnComplete` now fires on errored turns with the thrown error
attached, and the failed turn's user message is persisted so it isn't
lost on the next run.
- Stopping a generation clears the streaming state, so a page reload
doesn't replay the stopped turn.
- Custom agents and manual `chat.writeTurnComplete` callers trim the
output stream, sending a custom action no longer leaves a second stream
reader running, a long-lived `watch` subscription no longer grows its
dedupe set without bound, promoting a queued message to steering no
longer risks a double-send, and runs keep the full set of dashboard
tags.
The `X-Part-Id` header is accepted by current servers (they just don't
dedupe on it yet), so this is safe to ship ahead of the matching server
change.
## Summary
Reliability and authorization fixes for realtime chat sessions:
- Session-stream waitpoint delivery is scoped to the environment, so two
environments using the same session `externalId` can no longer complete
each other's waitpoints.
- The session snapshot-url routes now enforce per-session authorization,
and appending to a session's `out` channel requires secret-key auth, so
a session-scoped token can't read another session's snapshot or forge
assistant output.
- Appends that carry an `X-Part-Id` header are deduplicated on retry, so
a retried send can't duplicate a message.
- Session creation rejects expired sessions (instead of triggering a run
that can never receive input), `externalId` is immutable after creation,
and the sessions list endpoint returns friendly `run_*` ids to match the
single-session routes.
## Rollout
The waitpoint cache key gains an environment prefix. To keep waitpoints
registered by the previous deploy working across the boundary, the drain
reads both the new and the previous key for this release; the legacy
read can be removed a release later once no pre-deploy waitpoints
remain.
## Summary
Adds a second backend for the realtime runs feed (`useRealtimeRun`,
`subscribeToRunsWithTag`, `subscribeToBatch`), built to stay healthy
when a single busy environment has many subscribers watching many runs
at once. It is gated behind a feature flag with the existing backend as
the default, so nothing changes for users until it is enabled per
environment.
## Design
A run change is published once, as a small self-describing record, to a
single per-environment channel. Every feed is then a predicate over that
one stream rather than owning a channel:
- A per-instance router indexes the currently-held feeds by run, tag,
and batch. When a run changes it hydrates the affected rows once and
serializes them once, then fans the result to every matching feed. One
hot shared tag watched by many subscribers costs a single database query
and serialize, not one per subscriber.
- Feeds that don't match a change are never woken, wake delivery per
environment is coalesced on a leading edge (250ms default) so a burst of
changes costs one wake, and cold reads coalesce onto a single
short-TTL-cached resolve.
- An admission gate bounds how many cold ClickHouse resolves run
concurrently, so a mass reconnect across many distinct filters queues
instead of stampeding the database.
- Changes that land while a client is between long-polls are delivered
on its next poll instead of waiting for the periodic backstop: each
environment buffers its recent change records, subscriptions linger
briefly after the last feed closes, and a newly-armed poll replays
exactly the connection's gap.
- The per-connection replay cursors behind that are shared across
instances via Redis (a single timestamp each), so a poll landing on a
different instance behind the load balancer still reads the connection's
true gap instead of falling back to a cold resolve. Cursor reads have a
bounded deadline and degrade to the cold-read path on any Redis trouble.
- Tag subscriptions with multiple tags match runs carrying all of the
tags, mirroring the existing backend's filter semantics, and live
long-polls hold for about 20 seconds to match its cadence.
- The per-environment channel supports Redis Cluster sharded pub/sub, so
the wake path scales horizontally across shards by environment.
- The backend reports its health through OpenTelemetry metrics (delivery
lag, poll resolution paths, backstop outcomes, replay and cursor-store
activity), with a provisioned Grafana dashboard for local development.
Everything is behind the feature flag and tunable via env vars; the
existing backend remains the default.
## Summary
When `RUN_ENGINE_READ_REPLICA_SNAPSHOTS_SINCE_ENABLED` is on,
`RunEngine.getSnapshotsSince` reads from the read replica. During write
spikes the replica can briefly lag, so the snapshot id a runner just
learned from the writer isn't visible there yet: the lookup threw, the
worker route returned a 500, and the runner waited for its next poll —
turning sub-second snapshot notifications into poll-interval latency
exactly when things are busiest. This PR makes the flag safe to enable:
a replica miss of the since snapshot gets one jittered retry on the
replica (most lag windows are shorter than the ~50–200ms wait, so the
writer is never touched), then falls back to the primary, observed via a
new `run_engine.snapshots_since.replica_miss` counter with an `outcome`
attribute (`replica_retry` vs `primary`). Only genuine misses — absent
on the primary too — remain errors.
## Design
- `getExecutionSnapshotsSince` now throws a typed
`ExecutionSnapshotNotFoundError` so the engine can distinguish the
expected lag miss from real failures. The message string is unchanged
and the error never leaves the engine.
- The recovery path only engages when the flag is on, a distinct replica
client is configured, and no transaction client was passed. With the
flag off, the path is behaviorally identical to before.
- Retry delay bounds are configurable
(`RUN_ENGINE_SNAPSHOTS_SINCE_REPLICA_RETRY_MIN_MS`/`MAX_MS`, default
50/200; `MAX_MS=0` skips the replica retry and goes straight to the
primary).
- The warn log fires only when the primary serves the read (the writer
spill is the operationally interesting event); replica-retry recoveries
are counted but quiet. A permanently-missing snapshot id stays an
error-level failure with a `failedDuring` field, so lag metrics aren't
polluted by bogus ids.
- Stale-tail lag (replica has the since snapshot but not newer rows)
deliberately still returns the replica's view; the next poll catches up.
- The since-snapshot anchor lookup is now scoped to the polled run
(`where: { id, runId }`), so a snapshot id from a different run raises
not-found instead of silently anchoring a too-wide window of the run's
snapshots.
## Test plan
All vitest + testcontainers, no mocks. A new `schemaOnlyPrisma` fixture
(migrated-but-empty clone database) simulates a replica that hasn't
caught up, and a real in-memory OTel meter pins the counter semantics
per outcome.
- [x] Replica catches up during the jittered retry window → served by
the replica, `outcome=replica_retry` = 1, primary never consulted
- [x] Replica permanently missing the since snapshot → served by the
primary, `outcome=primary` = 1
- [x] Snapshot missing on both replica and primary → null, counter = 0
- [x] Replica has the since snapshot but lags by one → the replica's
view is served, no fallback (verified discriminating power: the test
fails if reads secretly hit the primary)
- [x] Flag off with a replica configured → primary serves the read
- [x] Transaction client provided → bypasses the replica entirely
- [x] Since snapshot belonging to a different run → null
- [x] Existing getSnapshotsSince + waitpoints suites green; run-engine,
testcontainers, and webapp typechecks pass
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
The supervisor's dequeue round-trip time (`POST
/engine/v1/worker-actions/dequeue`) was measured but only flowed into
wide events and OTel span attributes — there was no Prometheus series,
so latency percentiles and error rates weren't queryable. This adds
`queue_consumer_pool_dequeue_duration_seconds` (histogram, label
`outcome=success|empty|error`) to the existing consumer-pool metrics,
scraped automatically by the existing ServiceMonitors on
queue-raider/schedule-raider/supervisor.
- Records every dequeue call, including failed ones, which previously
emitted no timing at all
- The pool's shared `ConsumerPoolMetrics` instance is injected into each
consumer (mirrors the `BackpressureMetrics` → `BackpressureMonitor`
wiring)
- Buckets extend to 30s because `wrapZodFetch` retries internally (5
attempts, ≥7.5s backoff before a retryable error surfaces)
- Existing `dequeueResponseMs` wide-event/span behavior unchanged
The `dispatch-main-image` job was hard-gated to
`triggerdotdev/trigger.dev` on the `main` ref. This makes it
configurable via repository variables, all defaulting to the current
values:
- `MAIN_IMAGE_DISPATCH_REPO` — the repo allowed to dispatch (default
`triggerdotdev/trigger.dev`).
- `MAIN_IMAGE_DISPATCH_REF_PREFIX` — the ref-name prefix that
dispatches, matched with `startsWith(github.ref_name, …)` (default
`main`).
- `MAIN_IMAGE_DISPATCH_TARGET` — the `repository_dispatch` target
(default `triggerdotdev/cloud`).
The job is additionally gated on `github.event_name == 'push'`. This is
necessary, not just defensive: the gate now keys off `github.ref_name`
rather than the computed image tag, and `ref_name` is still `main` when
`release.yml` invokes this workflow via `workflow_call` during a release
— so without the event guard the job would fire during every release and
fail on the absent `CROSS_REPO_PAT`. A version-equality check can't
replace it because `build-*` tags strip the prefix to the version
output.
Behaviour note: the intended dispatch paths — push to `main`, and push
of a `<prefix>*` tag in a downstream repo — are `push` events and are
unchanged. The one case that no longer dispatches is a manual
`workflow_dispatch` run of `publish.yml` on `main` (it previously did,
via the old `version == 'main'` check). That path is indistinguishable
from a manual release by event name, so `push`-only is the clean
discriminator.
Dispatching still requires `CROSS_REPO_PAT`, so setting the variables
alone doesn't enable anything.
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
## Summary
Adds `GET /api/v1/projects/{projectRef}/environments` (personal access
token auth), which lists the base environments a user can access for a
project — their own dev environment plus the project's staging, preview,
and production environments.
## Details
- Built on the PAT route builder, so it inherits org-membership auth and
the per-resource ability check.
- `dev` is scoped to the token owner; archived environments are
excluded.
- Returns the branchable **parent** preview environment — preview branch
children are not included. A consumer targets the parent; branch-level
overrides are handled separately.
- Sorted to match the dashboard's environment switcher (dev → staging →
preview → prod), and never returns API keys.
Example response:
```json
[
{ "id": "...", "slug": "dev", "type": "DEVELOPMENT", "isBranchableEnvironment": false, "branchName": null, "paused": false },
{ "id": "...", "slug": "stg", "type": "STAGING", "isBranchableEnvironment": false, "branchName": null, "paused": false },
{ "id": "...", "slug": "preview", "type": "PREVIEW", "isBranchableEnvironment": true, "branchName": null, "paused": false },
{ "id": "...", "slug": "prod", "type": "PRODUCTION", "isBranchableEnvironment": false, "branchName": null, "paused": false }
]
```
## Summary
The dashboard's Agent view rendered `source-url` and `file` message
parts by putting their `url` straight into an `href`/`src`. Those URLs
come from streamed agent and tool data, so a tool that emitted something
like `javascript:alert(1)` produced a clickable XSS payload in the
dashboard.
## Fix
A `toSafeUrl` helper now gates every URL before it reaches an
`href`/`src`: it allows only `http:`/`https:`/`blob:` (and
`data:image/...` for inline images) and returns `null` for anything
else. Unsafe values render as plain text instead of a link or image, so
a hostile or malformed URL degrades gracefully rather than becoming
clickable. Safe URLs render exactly as before. Covered by a unit test
over the allow/deny list.
## Summary
During `trigger()` worker-queue resolution, `getWorkerQueue` wrapped any
error from `getDefaultWorkerGroupForProject` into a client-facing
`ServiceValidationError` (HTTP 422) carrying `error.message`. That
method runs `project.findFirst` on the **writer**; when the writer is
unreachable Prisma throws a connection error (P1001) whose message
includes the database host, and that raw message was returned to the API
client and surfaced in the run view via the SDK's `TriggerApiError`.
It also mis-classifies a transient outage: a 422 is not retried by the
SDK, so triggers failed permanently instead of riding out a brief writer
blip.
## Design
This is the only place on the trigger path that folds a *caught* error's
message into a client-facing error — every other DB failure on the path
propagates to the route's generic 500 handler (scrubbed, and retried by
the SDK). So the fix is local:
- Add `isInfrastructureError()` — true for Prisma connection-level
failures (the DB-unreachable family: P1001/P1002/P1008/P1017, plus the
init/panic/unknown client error classes), false for query/validation
errors (e.g. P2002).
- At the wrap site, rethrow infrastructure errors so they reach the
generic 500 handler (no raw message, and retryable). Genuine domain
failures (e.g. "Project not found.") still become a 422.
Only P1001 ("can't reach database server") has been observed in
practice; the rest of the connection family is included as same-class
forward-proofing.
## Test plan
- [x] Unit: `isInfrastructureError` classifies a P1001 (incl. the Prisma
6.x `PrismaClientKnownRequestError` shape) and init errors as
infrastructure; P2002 and a plain `Error` as not
- [x] `getWorkerQueue` rethrows a P1001 unchanged instead of wrapping it
in a `ServiceValidationError`; still wraps a domain failure as a
`ServiceValidationError` — RED on current code, GREEN after
- [ ] (optional) toxiproxy e2e: trigger with the writer cut → HTTP 500
generic body, no DB host in the response
---------
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
## Summary
Under high request load the webapp spends most of its CPU inside
react-router's `matchRoutes`, not in application code.
`@remix-run/router@1.23.2` (the React Router v6 / Remix 2 core)
re-flattens, re-ranks, and recompiles the entire route table on every
request, and with the webapp's ~436 routes that cost dominates once
request rates climb. There is no `NODE_ENV` gate, so production pays it
too.
This adds a pnpm patch that memoizes the parts that depend only on the
static route manifest: it caches the flattened/ranked branches per route
tree, hoists the loop-invariant `decodePath` out of the match loop, and
caches compiled path regexes.
## Benchmark
CPU profile over the same load (100 concurrent tag feeds, ~425 req/s),
`NODE_ENV=production`, before vs after the patch:
| Metric | Before | After |
| --- | --- | --- |
| Active CPU (self-time over the window) | 28.3s | 18.5s (-34%) |
| Route-matching self-time | 19.2s | 7.5s (-61%) |
| Event-loop lag p99 | 322ms | 113ms (-65%) |
| Idle headroom | 26% | 52% |
Application/realtime code was ~0% of CPU in both profiles; the
bottleneck was entirely generic per-request route matching.
## Why a patch instead of an upgrade
The inefficiency is acknowledged upstream
([remix-run/react-router#8653](https://github.com/remix-run/react-router/issues/8653)).
A contributor PR doing exactly this
([remix-run/react-router#14866](https://github.com/remix-run/react-router/pull/14866))
was closed in favor of a narrower fix
([remix-run/react-router#14967](https://github.com/remix-run/react-router/pull/14967),
branch caching only, shipped in React Router v7), with the maintainer
suggesting patch-package as the interim until the Remix 3 route-pattern
rewrite (see
[remix-run/remix#4786](https://github.com/remix-run/remix/discussions/4786)).
We are on the v6-era core and cannot pick up even the partial fix
without a framework migration, so this patch is the sanctioned stopgap,
and it also includes the compiled-regex cache the merged PR left out.
[`patches/README.md`](https://github.com/triggerdotdev/trigger.dev/blob/perf/react-router-route-matching/patches/README.md)
documents the full rationale, the safety argument (deterministic,
internal-only, bounded caches), and when to remove the patch.
## Summary
A new guide for connecting a database to your tasks: where to create the
client, how to size the connection pool against your provider's limit,
when to reach for a pooler, and how to release connections at waits so
you don't hit "too many connections" or crash on resume.
It covers node-postgres, Prisma, Drizzle, and MongoDB, with researched
direct and pooled connection limits for the common Postgres providers
(Supabase, Neon, RDS, PlanetScale) and MongoDB Atlas. The page lives
under Documentation, Troubleshooting, and is linked from the chat agent
docs (overview, lifecycle hooks, chat.local, and the database
persistence pattern).
## Summary
On every `main` build, once the webapp image is pushed to the registry,
the publish workflow emits a cross-repo `repository_dispatch` event
(`main-image-published`) carrying a digest-pinned image ref. Other
repositories in the org can subscribe to that event and build or deploy
from the exact artifact, instead of chasing the moving `main` tag.
## Design
`publish-webapp.yml` now exposes the pushed multi-arch index digest as a
workflow output. `publish.yml` adds a `dispatch-main-image` job (after
`publish-webapp`) that builds `<image_repo>@<digest>` and sends the
dispatch via the same pinned `peter-evans/repository-dispatch` action
already used elsewhere in this repo, authed with `CROSS_REPO_PAT`.
It fires only when the published tag is `main`, so semver releases and
other tag builds are excluded, and only from the canonical repo so forks
never dispatch. The payload is JSON-escaped with `jq`.
## Problem
Two backward-pagination bugs in `ClickHouseRunsRepository.listRunIds`,
both pre-existing (they predate the composite-cursor work in #3852 and
were spotted during/after it):
**1. Wrong slice (straddled pages).** `listRunRows` fetches `page.size +
1` rows to detect `hasMore`. That extra row is the one *farthest from
the cursor* in both directions (forward orders DESC; backward orders
ASC), so it's always the *trailing* element. Forward correctly used
`rows.slice(0, size)`, but backward+`hasMore` used `rows.slice(1, size +
1)` — dropping the row *closest* to the cursor and keeping the has-more
sentinel. The page straddled two logical pages (one run from the correct
previous page + one from the page before it), so paging "newer" across a
boundary **repeated and skipped** runs.
**2. Stranded forward cursor on a partial backward page.** In the
backward `!hasMore` branch, `nextCursor` was `reversedRows.at(page.size
- 1)`. On a partial page (fewer than `page.size` rows — reachable via
`runs.list` by passing a forward page's cursor as `page[before]`), that
index overshoots → `undefined` → `nextCursor` becomes `null`, leaving no
way to page forward again.
## Fix
- **Slice:** both directions now slice `rows.slice(0, size)` (the
sentinel is the trailing element either way).
- **Partial-page cursor:** the backward `!hasMore` branch takes the
oldest row on the page, `rows.at(0)`, for `nextCursor` — equivalent to
the old expression for full pages, correct for partial ones.
Forward pagination, the cursor *values* for full pages, and the `hasMore
=== true` paths were already correct and are unchanged.
## Tests
`runsRepositoryCursor.test.ts` gains two cases (both fail on `main`,
pass here):
- **multi-page backward walk:** forward across all pages, then backward
from the last page — each backward page must *exactly* reproduce the
corresponding forward page (no straddling: `main` returns `{b,c}`
instead of `{c,d}`), and the full traversal covers every run once.
- **partial backward page:** backward onto a partial first page must
still expose a working forward cursor (and paging forward from it
reaches the rest) — `main` returns a `null` nextCursor.
The three existing cursor tests (forward completeness, backward
round-trip, legacy cursor) still pass.
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
## What
`dev` and `deploy` now fail with a clear error when two tasks are
defined with the same id — including across task types (e.g. a scheduled
task and a regular task sharing an id).
## Why
Tasks are registered into the resource catalog keyed by id, so a second
definition with the same id silently overwrote the first. One of the
tasks would just vanish from the worker with no warning — easy to miss,
hard to debug. (Any earlier duplicate-id check ran against the
post-registration task list, which is already de-duplicated, so it never
actually fired.)
## How
- **Detect at registration** (`@trigger.dev/core`):
`StandardResourceCatalog` records a collision when a task id is
registered more than once, capturing the files involved — the only point
where both definitions are visible before the id-keyed map collapses
them. Exposed via `listTaskIdCollisions()`.
- **Fail indexing** (`trigger.dev` CLI): both index workers report
collisions via a new `TASKS_FAILED_TO_INDEX` message;
`indexWorkerManifest` rejects with a new `DuplicateTaskIdsError`. `dev`
renders a dedicated error (offending ids + files + docs link); `deploy`
fails with the same message. Runtime worker boot is unaffected — it
never reads the collisions.
- **Server-side backstop** (webapp): background-worker registration also
rejects duplicate ids with a clear `ServiceValidationError`, so
duplicates are caught even from an older CLI.
## Testing
- Unit tests for collision collection in the catalog and for the
error-message formatting (standard, same-file, and 3+-definition cases).
- Verified end to end against a local webapp: a project with a regular
task and a scheduled task sharing an id now fails `dev` with the
dedicated error; a project with distinct ids still starts normally.
## Changeset
Patch for `@trigger.dev/core` and `trigger.dev`.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
## Summary
- `internal-packages/rbac/src/index.ts` — in `LazyController.load()`'s
catch block, throw an Error when `process.env.REQUIRE_PLUGINS === "1"`
instead of silently falling back. The throw is captured into the lazy
controller's init promise, so it surfaces on the first method call.
- `apps/webapp/app/routes/healthcheck.tsx` — `await
rbac.isUsingPlugin()` after the DB ping. With `REQUIRE_PLUGINS=1` and a
failed plugin load, the throw surfaces here and the healthcheck returns
500 → readiness probe fails → rollout is rolled back. Noop for
self-hosters.
- `.server-changes/require-plugins-fail-fast.md` — server-changes entry.
- `internal-packages/rbac/src/require-plugins.test.ts` — 4 unit tests
covering loader branching: unset → fallback, `=1` → throw,
`forceFallback: true` wins, only exactly `"1"` enforces.
- `internal-packages/testcontainers/src/webapp.ts` — adds
`requirePlugins?: boolean` to `StartWebappOptions`. Implies
`forceRbacFallback: false`.
- `apps/webapp/test/healthcheck-require-plugins.e2e.test.ts` — e2e
closes the loop: spawns a real webapp, hits `/healthcheck` via HTTP,
asserts 500 with `REQUIRE_PLUGINS=1` and 200 without.
## Motivation
Today the RBAC plugin loader catches any plugin-load failure (missing
module, broken transitive dep, init throw) and silently returns the
default fallback implementation. This is the correct behaviour for
self-hosters who don't ship the plugin — but it's dangerous in
deployments where the plugin is expected to load: an
accidentally-missing or broken plugin would silently disable
enforcement.
`REQUIRE_PLUGINS=1` makes the loader fail loudly in those deployments.
The variable name is intentionally plural and generic — future plugin
contracts (audit logs, SSO) can read the same flag without renaming.
Closes
[TRI-9852](https://linear.app/triggerdotdev/issue/TRI-9852/require-plugins1-fail-fast-for-required-plugin-loads).
## Test plan
- [x] `pnpm run test --filter @trigger.dev/rbac` — 38/38 tests pass,
including the 4 new loader tests
- [x] `pnpm run typecheck --filter webapp` — passes
- [x] `pnpm run typecheck --filter @trigger.dev/rbac --filter
@internal/testcontainers` — passes
- [x] e2e test added (`healthcheck-require-plugins.e2e.test.ts`) — CI
runs it via `e2e-webapp.yml`. Couldn't run locally (no Docker daemon
up); CI has Docker provisioned.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
## Summary
Runs routed onto a dedicated scheduled worker queue were showing under a
phantom region in the dashboard, run details, and the API, and slipped
through region filters. They now resolve to their real region
everywhere.
## Fix
A worker queue can carry a `:scheduled` suffix that routes
scheduled-lineage runs onto their own list. That suffix is an internal
routing detail, but it was leaking anywhere the worker queue is read as
a region. A `baseWorkerQueue` helper strips any `:<class>` suffix back
to the base region (region names never contain a colon, so it's
everything before the first colon) and is applied at every region read
site: the runs list, run detail, the public API, and replay's region
override. The runs-replication writer also stores the base region in
ClickHouse so the region filter matches.
## Summary
`trigger init` now sets up your AI coding assistant as part of project
setup. Instead of the old either/or "MCP or CLI" prompt, it offers the
MCP server and agent skills together, then asks whether to scaffold with
the CLI or let your assistant do it.
A new `getting-started` agent skill backs that hand-off: it teaches the
assistant the bootstrap recipe (install the SDK, write
`trigger.config.ts`, scaffold a first task, wire tsconfig/gitignore, run
`trigger dev`) and is explicit about the two steps that genuinely need a
human (`trigger login` and copying the DEV secret key from the
dashboard). It ships in the CLI alongside the existing skills,
version-matched to your SDK.
Prompt-once gating is shared, so opting in or out during `init` means
`trigger dev` won't ask about skills again.
## Summary
The container publish workflows hardcoded `ghcr.io/triggerdotdev/...` as
the image destination. As a result, a fork that builds on push-to-`main`
(or on the worker publish tags) would attempt to push to — and attest —
the upstream packages rather than its own, which fails on permissions
and is surprising besides.
This makes the image destination configurable via a single
`IMAGE_REGISTRY` repository variable, while leaving the upstream
defaults byte-identical:
- **Single source of truth** (`publish.yml`): a `resolve-registry` job
resolves the target registry namespace once — `IMAGE_REGISTRY`
repository variable, defaulting to `ghcr.io/${{ github.repository_owner
}}` — and passes it down to every publish job as an `image_registry`
input. So a fork publishes to its own namespace automatically with no
configuration.
- **Webapp** (`publish-webapp.yml`): the image now lives at
`<registry>/<repo-name>` (e.g. `ghcr.io/<owner>/trigger.dev`). The
provenance attestation and the downstream Trivy scan follow the same
computed repo via the `image_repo` workflow output.
- **Workers** (`publish-worker.yml`, `publish-worker-v4.yml`): build
under `<registry>/<worker-name>`. They keep a `vars.IMAGE_REGISTRY ||
ghcr.io/<owner>` fallback so they still resolve correctly on their
direct `infra-*` / `re2-*` push triggers (which bypass the parent
workflow).
A single `IMAGE_REGISTRY` namespace variable now governs both webapp and
workers (the earlier `WEBAPP_IMAGE_REPO` full-path override is dropped,
removing the full-path/namespace asymmetry). When `IMAGE_REGISTRY` is
unset, every resolved image name is exactly what it is today, so there
is no change for this repo.
## Test plan
- [x] `actionlint` passes on all four workflows
- [ ] On merge, confirm the webapp publish still pushes
`ghcr.io/triggerdotdev/trigger.dev:main` + the commit-SHA tag (defaults
unchanged)
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
## Summary
The `mollifier.decisions` metric only carried an `outcome` label, so for
an org that has the mollifier enabled there was no way to see how often
its triggers pass through the gate instead of being diverted — making it
hard to tell why the trip isn't firing for an opted-in org.
This adds two bounded labels: `enrolled` (`"true"`/`"false"`, the
per-org flag) and `org` (the org id, attached **only** when `enrolled`
is true). For an enrolled org you can now compare directly:
`mollifier.decisions{outcome="pass_through", enrolled="true",
org="<id>"}` vs `{outcome="mollify", ...}`.
## Design
`recordDecision` now takes an options object (`{ reason?, enrolled,
orgId? }`). The `org` label is restricted to the enrolled cohort to keep
cardinality bounded — the guard lives in a pure `decisionLabels` helper,
so a non-enrolled org id can never be attached even if one is passed.
The enrolled set is small and capped operationally.
The per-org flag is resolved once at the top of `evaluateGate`
(in-memory, no DB round-trip on the trigger hot path) so every decision
— including the debounce / one-time-use-token / triggerAndWait bypasses
— is labelled consistently.
## Test plan
- [x] `mollifierGate.test.ts` cascade asserts `enrolled`/`org` on every
gate branch
- [x] `mollifierDecisionLabels.test.ts` (new) proves `org` is dropped
for non-enrolled even when an id is passed (cardinality guard)
- [x] `vitest run mollifierGate mollifierDecisionLabels` — 34/34 pass
- [x] `pnpm run typecheck --filter webapp` clean
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Add an optional network_labels field to the internal compute client's
create and restore request schemas and forward per-VM endpoint labels on
both paths, so a restored VM keeps the same labels as a freshly-booted
one. Mirrors the label the Kubernetes workload manager already sets on
the run pod.
---------
Co-authored-by: nicktrn <55853254+nicktrn@users.noreply.github.com>
## Summary
`trigger skills` installs Trigger.dev agent skills into your coding
agent so it knows how to write Trigger.dev code: tasks, schedules,
realtime, and `chat.agent` AI agents. The skills are `SKILL.md` files
(the open Agent Skills format) bundled with the CLI and copied into each
tool's native skills directory (Claude Code, Cursor, GitHub Copilot, and
Codex / `AGENTS.md`), version-matched to the CLI you run. `trigger dev`
offers to install them on first run, and a one-line always-on pointer is
written into your `CLAUDE.md` / Cursor rules / etc. so the agent always
knows which skills are available and loads the right one on demand.
This replaces the old `install-rules` command, which stays as an alias.
Four skills ship to start: `authoring-tasks`, `realtime-and-frontend`,
`authoring-chat-agent`, and `chat-agent-advanced`.
## Problem
`ClickHouseRunsRepository.listRunIds` / `listRuns` order results by the
composite key `(created_at, run_id)`, but the cursor predicate cut on
`run_id` **alone**:
```ts
.where("run_id < {runId: String}", { runId: cursor })
.orderBy("created_at DESC, run_id DESC")
```
This is only sound when `run_id` lexicographic order matches
`created_at` order. `run_id`s are cuids — only coarsely time-sortable —
so when a burst of runs is created within a sub-second window, the two
orders can diverge. When they do, the next-page predicate (`run_id <
cursor`, where `cursor` is the *last* page element = the smallest
`created_at`, not necessarily the smallest `run_id`):
- **re-includes** rows already returned on a previous page (duplicates),
and
- **skips** rows it should have returned (silent data loss).
For bulk **replay** this caused runs to be replayed more than once
(replay has no idempotency guard). For the dashboard and the `runs.list`
API it could silently repeat or skip runs at page boundaries.
## Fix
Make the cursor predicate match the composite ordering:
- Cursors now encode the full `(created_at, run_id)` key as an **opaque
URL-safe base64 token**
(`base64url({"c":<createdAtMs>,"r":"<runId>"})`), and the query cuts on
the matching tuple — `(created_at, run_id) < (…)` forward / `> (…)`
backward.
- The `ORDER BY` is unchanged, so the query stays aligned with the
table's primary key — no performance regression (the tuple range
predicate is actually more index-friendly than `run_id <` alone).
- Cursors are **server-issued opaque tokens** (the SDK only echoes
`pagination.next` / `pagination.previous` back), so this needs **no
client/SDK update**. Legacy cursors were the bare internal `run_id`;
they're detected by decode failure (a cuid isn't a valid base64-wrapped
JSON payload) and fall back to the old `run_id`-only predicate, so
in-flight cursors keep working and drain naturally. New cursors also no
longer expose a bare internal run id.
- `listRunIds` is now the single cursor-aware list primitive: it returns
`{ runIds, pagination: { nextCursor, previousCursor } }`, and `listRuns`
builds on it (one place constructs cursors). Bulk actions consume the
same method and advance by `pagination.nextCursor`, finishing when it's
`null`.
- `getTaskRunsQueryBuilder` now also selects
`toUnixTimestamp64Milli(created_at) AS created_at_ms`, using a dedicated
`TaskRunListQueryResult` schema. The shared `TaskRunV2QueryResult` stays
`run_id`-only so the run-engine pending-version lookup
(`getPendingVersionIdsQueryBuilder`, which selects only `run_id`)
doesn't fail validation on a column it doesn't query.
## Tests
New `runsRepositoryCursor.test.ts` (testcontainer-backed, real
Postgres→ClickHouse replication):
- **forward** pagination returns every run exactly once when `run_id`
order is the reverse of `created_at` order (reproduces the
duplicate/skip bug — fails on `main`; this
walk-until-`nextCursor`-null-and-assert-complete is exactly the bulk
action's iteration),
- **backward** pagination round-trips to the previous page across a
boundary,
- **legacy** bare-`run_id` cursor still uses the old predicate
(backwards compatibility).
The existing `runsRepository` suites (part1–4) still pass; `part4`'s
`count new runs with listRunIds` test was updated for the new `{ runIds,
pagination }` return shape, and the `clickhouse` `taskRuns`
query-builder snapshots were regenerated for the added `created_at_ms`
column.
## Notes
- Separate, pre-existing issue (out of scope, not introduced here):
`listRuns`' backward display-slicing (`rows.slice(1, size+1)` when
`hasMore`) has an off-by-one that can return a straddled page. Tracked
separately.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Two small hygiene tweaks to **dev-only** images:
- `docker/Dockerfile.postgres`: add `--no-install-recommends` to the
partman install (leaner image, skips unneeded recommended packages).
- `internal-packages/clickhouse/Dockerfile`: run the migration helper as
a non-root user.
Both are local-dev images (the `pnpm run docker` stack) - no impact on
the published webapp image, prod, or self-hosting.
Speeds up and de-flakes the unit-test suite: testcontainers booted once
per vitest worker (per-test isolation kept only where a test runs
background redis work that outlives it), a duration-weighted shard
sequencer so each shard does roughly equal work, the slowest suites
split, two genuine flakes fixed (`streamBatchItems` shared-redis leak;
run-engine waits that relied on fixed sleeps), and transient DockerHub
pulls retried.
**Timings (CI, per-shard wall):** worst unit-test shard ~771s → ~294s;
packages/webapp shards ~250-270s, most internal ~190-240s. All 25 shards
green.
A shard breaks down as ~70s fixed setup (install / image-pull /
generate) + ~70s cold `^build` + the actual container tests. So the
remaining cost is mostly the tests themselves plus that fixed setup.
**Next (separate, timings):**
- **typecheck (~6m24s)** — the slowest check overall; bound by
full-graph `tsc`, not the TS version (a TS6 branch is still ~6m17s). The
real lever is **tsgo** (the Go compiler).
- Possible later: turbo CI caching could trim the ~70s cold build on
*warm* runs, but it's conditional (cold runs rebuild anyway) and doesn't
touch setup or test time — secondary.
`cli-v3` e2e and `sdk-compat` are path-gated (don't run on test-infra
changes) and already comfortably fast.
## Summary
Adds a trace export to the run page. From the new **Export trace** menu
you can
copy a run's full trace to the clipboard as Markdown (for pasting into
an AI
assistant) or download it as a flat Log, a Markdown table, or JSON
Lines.
Internal engine-debug events are filtered out by default, and errors are
surfaced inline with their message.
## Design
The export streams events from the store to the gzipped response one at
a time
and never materialises the span tree, so a trace of any size exports
with
bounded memory and without stalling the server. Output is flat and
chronological: each line carries its own `spanId ← parentSpanId`, so the
hierarchy is reconstructable without nesting. Formats share a single
streaming
pipeline and are pluggable via `?format=log|jsonl|markdown`, so adding a
format
is an isolated change.
## Screenshots
**Export menu**
<img width="370" height="252" alt="trace-export-menu"
src="https://github.com/user-attachments/assets/3d10304a-8c49-4606-b15d-2859b137419f"
/>
**In context**
<img width="2400" height="1802" alt="trace-export-run-page"
src="https://github.com/user-attachments/assets/46c80b30-303b-47c6-9ace-a2fb06f6cb61"
/>
A unit-test shard recently failed on a timing race rather than a real
regression - a run-engine waitpoint test sleeps 1250ms waiting on a
1000ms timeout that's processed by a ~1000ms worker poll, so on a
CPU-starved shard the margin evaporates and the whole matrix goes red.
Because `fail-fast` defaults on, that one flake cancels the sibling
shards, and the only recovery is re-running the entire matrix "just to
be sure" - which is itself slow.
This is the low-risk first pass at that pain:
- `fail-fast: false` on the webapp and internal shard matrices, so one
flaky shard no longer cancels its siblings. "Re-run failed jobs" now
re-runs just the failed shard instead of the whole matrix.
- CI-scoped `retry: process.env.CI ? 2 : 0` on the timing-sensitive
packages (`run-engine`, `redis-worker`, `schedule-engine`). Flakes
self-heal in CI; local runs stay at `retry: 0` so they still surface in
dev. A stopgap until the timing tests are made deterministic.
- `fetch-depth: 1` on the unit-test checkouts - they don't use git
history, so the full clone was wasted setup time across ~20 jobs.
- Reconcile the pre-pull image tags with what testcontainers actually
pulls (`redis:7-alpine` -> `redis:7.2`, `ryuk:0.11.0` -> `ryuk:0.14.0`)
and add `minio/minio:latest` to the webapp pre-pull. Otherwise those
images pull unauthenticated at test time and risk Docker Hub rate-limit
flakes (worst on fork PRs, where the authenticated pre-pull is skipped
entirely).
Deeper follow-ups - bigger runners, turbo remote cache, runtime-weighted
sharding, and the real root-cause fix (container reuse / template-DB
isolation + deterministic timing tests) - are tracked under TRI-10484.
Hardens the webapp Docker image and adds a CVE scan of each published
image.
- Base image `bullseye-slim` → `bookworm-slim` (Debian 12), pinned by
digest. Adds `apt-get upgrade` + `--no-install-recommends` + apt-cache
cleanup across the build stages so OS packages are patched at build
time.
- Moves the `react-email` CLI to `devDependencies` in
`internal-packages/emails` — only the `email dev` preview script uses
it; the runtime render path is `@react-email/render` +
`@react-email/components`. This also drops the bundled `esbuild` binary
from the production image.
- Bumps `goose` v3.26.0 → v3.27.1 and its Go builder image 1.23 → 1.26.
- Adds a reusable Trivy image-scan workflow wired into `publish.yml`, so
every published image (main builds and releases) is scanned for
OS-package CVEs right after it's pushed to GHCR. Report-only (writes to
the run summary), runs alongside the worker publishes so it never blocks
a deploy.
Verified locally: the image builds clean on the new base, and
`@react-email/render` carries no `esbuild` dependency so email rendering
is unaffected.
## Summary
The SDK and core packages run a second, forward-compat typecheck pass
(`tsc --noEmit -p tsconfig.ai-v7.json`) that remaps the `"ai"` import to
the ESM-only AI SDK 7 canary, so we catch source that only compiles
against one major. That pass inherited `composite: true` from the base
tsconfig, which makes `tsc` write a `.tsbuildinfo` even under
`--noEmit`.
Incremental buildinfo caches each file's resolved module format (CJS vs
ESM) and module resolution. When that state goes stale or is replayed,
the v7 pass can report spurious `TS1479` ("CommonJS module ... cannot
`require` an ECMAScript module") errors on the `"ai"` import even though
the source is fine in a clean checkout. Because this pass shares the
typecheck job that gates the Docker image publish, a spurious failure
there blocks publishing.
## Fix
Set `composite: false` and `incremental: false` on both
`tsconfig.ai-v7.json` files. The pass is `--noEmit` only, so it never
needed incremental state. Now each run is a clean, full check that
writes no buildinfo and can't replay stale resolution.
Verified: both `@trigger.dev/sdk` and `@trigger.dev/core` typecheck
green, and neither writes an ai-v7 `.tsbuildinfo` anymore.
## Summary
Opening or closing the Bulk action inspector should not affect the Runs
list, but it was still triggering refresh-related UI behavior. This PR
fixes that and smooths out a few related inspector interactions.
## Changelog
Stop reloading the runs list (and flashing its loading state) when
opening or closing the
Bulk action inspector. Filtering, pagination, and explicit refresh are
unaffected.
## Summary
Documents AI SDK 7 support in the AI Chat docs. Pairs with the SDK
change in #3833.
- The reference compatibility matrix now lists the v7 peer range and
adds an `@ai-sdk/otel` row.
- A new "AI SDK 7 telemetry" section covers the `@ai-sdk/otel` install,
the automatic registration, and the `TRIGGER_AI_SDK_OTEL_AUTOREGISTER`
opt-out.
- The quick start surfaces the supported `ai` versions (v5/v6/v7) up
front, near where you install.
## Summary
1 new feature, 8 improvements, 1 bug fix.
## Highlights
- Add optional `shouldPauseScaling` to the supervisor consumer pool
scaling options to freeze scale-up while it returns true (scale-down
stays allowed).
([#3836](https://github.com/triggerdotdev/trigger.dev/pull/3836))
## Improvements
- The MCP server no longer tells the AI agent to wait for a run to
complete after every `trigger_task` call. Waiting is now opt-in: the
agent only waits when you ask it to (for example "trigger and then wait
for it to finish"). This avoids burning tokens polling runs you didn't
need to block on and keeps responses clearer.
([#3838](https://github.com/triggerdotdev/trigger.dev/pull/3838))
- Update the bundled OpenTelemetry packages to their latest releases
(`@opentelemetry/sdk-node` 0.218.0, `@opentelemetry/core` 2.7.1,
`@opentelemetry/host-metrics` 0.38.3).
([#3810](https://github.com/triggerdotdev/trigger.dev/pull/3810))
- `envvars.upload` now accepts an optional `isSecret` flag, letting you
create the imported variables as secret (redacted) environment
variables. When omitted, variables default to non-secret.
([#3809](https://github.com/triggerdotdev/trigger.dev/pull/3809))
- Offload large trigger payloads to object storage before sending the
trigger API request. The SDK uploads packets at or above the existing
128KB limit and sends an `application/store` pointer instead of
embedding large JSON in the request body. `TriggerTaskRequestBody` now
validates that `application/store` payloads are non-empty storage paths.
([#3785](https://github.com/triggerdotdev/trigger.dev/pull/3785))
- Make mollifier buffer and drainer internals configurable.
`MollifierBuffer` now accepts `ackGraceTtlSeconds`,
`maxRetriesPerRequest`, `reconnectStepMs`, and `reconnectMaxMs` options,
and `MollifierDrainer` accepts `maxBackoffMs` and `backoffFloorMs`. All
default to their previous hardcoded values, so existing behaviour is
unchanged.
([#3822](https://github.com/triggerdotdev/trigger.dev/pull/3822))
- `MollifierDrainer` accepts a `drainBatchSize` option (default 1) that
controls how many entries are popped per env per tick — in-flight
handlers remain capped by the global `concurrency`. `MollifierBuffer`
also gains `getDrainingCount()` / `listStaleDraining()`, backed by a new
`mollifier:draining` ZSET maintained atomically with
pop/ack/fail/requeue (observability-only).
([#3797](https://github.com/triggerdotdev/trigger.dev/pull/3797))
- Adds AI SDK 7 support. The `ai` peer range now includes v7, and the
`chat.agent` / chat surfaces work against v7's ESM-only build. On v7,
install `@ai-sdk/otel` alongside `ai` and the SDK registers it for you
so `experimental_telemetry` spans keep flowing into your run traces (v7
stopped emitting them from `ai` core). v5 and v6 keep working unchanged.
([#3833](https://github.com/triggerdotdev/trigger.dev/pull/3833))
- `useTriggerChatTransport` now recovers when restored session state
points at a session that no longer exists in the current environment
([#3816](https://github.com/triggerdotdev/trigger.dev/pull/3816))
## Bug fixes
- Fix `@trigger.dev/core` build: cast the underlying log record exporter
when calling `forceFlush` so it typechecks against the updated
OpenTelemetry `LogRecordExporter` type (which no longer declares
`forceFlush`).
([#3829](https://github.com/triggerdotdev/trigger.dev/pull/3829))
<details>
<summary>Raw changeset output</summary>
⚠️⚠️⚠️⚠️⚠️⚠️
`main` is currently in **pre mode** so this branch has prereleases
rather than normal releases. If you want to exit prereleases, run
`changeset pre exit` on `main`.
⚠️⚠️⚠️⚠️⚠️⚠️
# Releases
## @trigger.dev/build@4.5.0-rc.5
### Patch Changes
- Updated dependencies:
- `@trigger.dev/core@4.5.0-rc.5`
## trigger.dev@4.5.0-rc.5
### Patch Changes
- The MCP server no longer tells the AI agent to wait for a run to
complete after every `trigger_task` call. Waiting is now opt-in: the
agent only waits when you ask it to (for example "trigger and then wait
for it to finish"). This avoids burning tokens polling runs you didn't
need to block on and keeps responses clearer.
([#3838](https://github.com/triggerdotdev/trigger.dev/pull/3838))
- Update the bundled OpenTelemetry packages to their latest releases
(`@opentelemetry/sdk-node` 0.218.0, `@opentelemetry/core` 2.7.1,
`@opentelemetry/host-metrics` 0.38.3).
([#3810](https://github.com/triggerdotdev/trigger.dev/pull/3810))
- Updated dependencies:
- `@trigger.dev/core@4.5.0-rc.5`
- `@trigger.dev/build@4.5.0-rc.5`
- `@trigger.dev/schema-to-json@4.5.0-rc.5`
## @trigger.dev/core@4.5.0-rc.5
### Patch Changes
- Add optional `shouldPauseScaling` to the supervisor consumer pool
scaling options to freeze scale-up while it returns true (scale-down
stays allowed).
([#3836](https://github.com/triggerdotdev/trigger.dev/pull/3836))
- Fix `@trigger.dev/core` build: cast the underlying log record exporter
when calling `forceFlush` so it typechecks against the updated
OpenTelemetry `LogRecordExporter` type (which no longer declares
`forceFlush`).
([#3829](https://github.com/triggerdotdev/trigger.dev/pull/3829))
- `envvars.upload` now accepts an optional `isSecret` flag, letting you
create the imported variables as secret (redacted) environment
variables. When omitted, variables default to non-secret.
([#3809](https://github.com/triggerdotdev/trigger.dev/pull/3809))
```ts
await envvars.upload("proj_1234", "prod", {
variables: { STRIPE_SECRET_KEY: "sk_live_..." },
isSecret: true,
});
```
- Offload large trigger payloads to object storage before sending the
trigger API request. The SDK uploads packets at or above the existing
128KB limit and sends an `application/store` pointer instead of
embedding large JSON in the request body. `TriggerTaskRequestBody` now
validates that `application/store` payloads are non-empty storage paths.
([#3785](https://github.com/triggerdotdev/trigger.dev/pull/3785))
Payload uploads use the same resolved `ApiClient` as the trigger call
(including `requestOptions.clientConfig`), not only the global
`apiClientManager.client` — so custom `baseURL`, access token, and
preview branch apply to both presign and trigger.
- Update the bundled OpenTelemetry packages to their latest releases
(`@opentelemetry/sdk-node` 0.218.0, `@opentelemetry/core` 2.7.1,
`@opentelemetry/host-metrics` 0.38.3).
([#3810](https://github.com/triggerdotdev/trigger.dev/pull/3810))
## @trigger.dev/plugins@4.5.0-rc.5
### Patch Changes
- Updated dependencies:
- `@trigger.dev/core@4.5.0-rc.5`
## @trigger.dev/python@4.5.0-rc.5
### Patch Changes
- Updated dependencies:
- `@trigger.dev/sdk@4.5.0-rc.5`
- `@trigger.dev/core@4.5.0-rc.5`
- `@trigger.dev/build@4.5.0-rc.5`
## @trigger.dev/react-hooks@4.5.0-rc.5
### Patch Changes
- Updated dependencies:
- `@trigger.dev/core@4.5.0-rc.5`
## @trigger.dev/redis-worker@4.5.0-rc.5
### Patch Changes
- Make mollifier buffer and drainer internals configurable.
`MollifierBuffer` now accepts `ackGraceTtlSeconds`,
`maxRetriesPerRequest`, `reconnectStepMs`, and `reconnectMaxMs` options,
and `MollifierDrainer` accepts `maxBackoffMs` and `backoffFloorMs`. All
default to their previous hardcoded values, so existing behaviour is
unchanged.
([#3822](https://github.com/triggerdotdev/trigger.dev/pull/3822))
- `MollifierDrainer` accepts a `drainBatchSize` option (default 1) that
controls how many entries are popped per env per tick — in-flight
handlers remain capped by the global `concurrency`. `MollifierBuffer`
also gains `getDrainingCount()` / `listStaleDraining()`, backed by a new
`mollifier:draining` ZSET maintained atomically with
pop/ack/fail/requeue (observability-only).
([#3797](https://github.com/triggerdotdev/trigger.dev/pull/3797))
- Updated dependencies:
- `@trigger.dev/core@4.5.0-rc.5`
## @trigger.dev/rsc@4.5.0-rc.5
### Patch Changes
- Updated dependencies:
- `@trigger.dev/core@4.5.0-rc.5`
## @trigger.dev/schema-to-json@4.5.0-rc.5
### Patch Changes
- Updated dependencies:
- `@trigger.dev/core@4.5.0-rc.5`
## @trigger.dev/sdk@4.5.0-rc.5
### Patch Changes
- Adds AI SDK 7 support. The `ai` peer range now includes v7, and the
`chat.agent` / chat surfaces work against v7's ESM-only build. On v7,
install `@ai-sdk/otel` alongside `ai` and the SDK registers it for you
so `experimental_telemetry` spans keep flowing into your run traces (v7
stopped emitting them from `ai` core). v5 and v6 keep working unchanged.
([#3833](https://github.com/triggerdotdev/trigger.dev/pull/3833))
- `useTriggerChatTransport` now recovers when restored session state
points at a session that no longer exists in the current environment
([#3816](https://github.com/triggerdotdev/trigger.dev/pull/3816))
- Offload large trigger payloads to object storage before sending the
trigger API request. The SDK uploads packets at or above the existing
128KB limit and sends an `application/store` pointer instead of
embedding large JSON in the request body. `TriggerTaskRequestBody` now
validates that `application/store` payloads are non-empty storage paths.
([#3785](https://github.com/triggerdotdev/trigger.dev/pull/3785))
Payload uploads use the same resolved `ApiClient` as the trigger call
(including `requestOptions.clientConfig`), not only the global
`apiClientManager.client` — so custom `baseURL`, access token, and
preview branch apply to both presign and trigger.
- Update the bundled OpenTelemetry packages to their latest releases
(`@opentelemetry/sdk-node` 0.218.0, `@opentelemetry/core` 2.7.1,
`@opentelemetry/host-metrics` 0.38.3).
([#3810](https://github.com/triggerdotdev/trigger.dev/pull/3810))
- Updated dependencies:
- `@trigger.dev/core@4.5.0-rc.5`
</details>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
## Summary
The Prisma CLI was missing from production builds of the webapp image,
so anything that shells out to `prisma` at startup failed. The container
entrypoint and the standalone migration step both run `prisma migrate
deploy` / `prisma migrate status`, and those broke with `Command
"prisma" not found`.
## Fix
`prisma` was a `devDependency` of `@trigger.dev/database`. It had only
been landing in the pruned `--prod` install as a side effect of pnpm
auto-installing it as a peer of `@prisma/client`. A recent dependency
change shifted peer resolution so prisma stopped being materialized into
the production tree, and the CLI disappeared from the image.
Moving `prisma` into `dependencies` of `@trigger.dev/database` makes the
CLI an explicit part of production installs. It lands in the webapp
image only: the separately deployed supervisor, coordinator, and
provider images don't reach the database package in their production
trees (`core` only `devDepends` on it, so it isn't transitive), so
they're unaffected.
Verified against a locally built production image: `pnpm --filter
@trigger.dev/database exec prisma --version` now resolves the CLI and
the schema engine instead of failing.
The supervisor can now pause dequeuing - and freeze consumer-pool
scale-up - when a backpressure signal says the cluster can't place more
work, then ramp dequeuing back up gradually once it clears. The signal
is a verdict published to a Redis key by a cluster-side component; the
supervisor reads it on a short refresh and gates `preDequeue` on it.
Off by default (`TRIGGER_DEQUEUE_BACKPRESSURE_ENABLED`). Everything
fails open: a missing, stale, or unreadable verdict never pins the
brake, and the hot-path read is a synchronous cached lookup with no I/O.
The scale-up freeze leaves scale-down untouched, and on release the
resume is ramped so a deep queue isn't hammered all at once.
Dry-run is on by default (`TRIGGER_DEQUEUE_BACKPRESSURE_DRY_RUN`): even
once enabled it only logs what it would have done, and surfaces the
computed state through metrics, until explicitly set to act. Prometheus:
`supervisor_backpressure_engaged`, `_dry_run`,
`_skipped_dequeues_total`.
Refs TRI-5354
## Summary
Scheduled runs and their descendants can now be routed to a dedicated
per-region worker queue, processed by a separate worker fleet, so a
burst of scheduled crons no longer competes with standard and agent runs
for the same queue and inflates their startup latency. It is off by
default and enabled per organization via a feature flag (with a global
default), so nothing changes until it is turned on.
## Design
At trigger time, any run whose lineage originates from a schedule
(`rootTriggerSource === "schedule"`, which already propagates from a
scheduled run down to all of its children) gets its worker queue
suffixed with `:scheduled`. The worker queue name is an opaque string
persisted on the run and used verbatim by enqueue and dequeue, so this
needs no Lua, message-envelope, or concurrency changes. Concurrency
stays keyed by environment and queue, not by worker queue.
On the consumer side, the dequeue endpoint gains an optional
`queueClass` selector. A supervisor sends `queueClass: "scheduled"` and
the server derives the actual queue from the worker's own group, so a
token can only ever reach its own region's queues. A fleet picks its
class with the `TRIGGER_WORKER_QUEUE_CLASS` env var (`default` or
`scheduled`), so a dedicated scheduled fleet can run alongside the
standard one.
Verified end to end against a local managed-worker setup: scheduled runs
route to the dedicated queue, are drained only by the scheduled fleet,
and standard runs are left untouched.
## Summary
The Trigger.dev MCP server told the AI agent to wait for the run to
complete after every `trigger_task` call. The agent followed that
instruction even when the user only wanted to fire-and-forget, which
burned tokens polling runs nobody needed to block on and made responses
less clear.
Waiting is now opt-in. After triggering, the response tells the agent
the run is executing in the background and to only wait if the user
asked it to (for example "trigger and then wait for it to finish"). The
`trigger_task` tool description is updated to match. The
`wait_for_run_to_complete` tool itself is unchanged, so explicit waits
still work.
## Summary
This PR adds packet path validation before key construction and
presigning. Invalid paths are rejected before reaching either
object-store client implementation, ensuring consistent behavior
regardless of the underlying storage configuration.
## Summary
Adds support for Vercel AI SDK 7. The `ai` peer range now includes v7,
and the `chat.agent` / chat surfaces work against v7's ESM-only build.
v5 and v6 keep working unchanged, so this is additive.
## Telemetry on v7
On v7, model-call spans moved out of `ai` core into the separate
`@ai-sdk/otel` adapter, so `experimental_telemetry` alone produces
nothing until an integration is registered. Install `@ai-sdk/otel`
alongside `ai@7` and the SDK registers it once per worker at chat agent
boot, so spans keep flowing into run traces with no extra setup.
If you (or a library you import) already register `@ai-sdk/otel`, the
SDK detects the existing integration and skips its own registration, so
you won't get duplicate spans. Set `TRIGGER_AI_SDK_OTEL_AUTOREGISTER=0`
to disable auto-registration entirely.
## Notes
`ai@7` is ESM-only, which tripped TS1479 in the SDK's CommonJS build.
Runtime value imports from `ai` are isolated behind a paired ESM/CJS
shim so both module formats resolve the right form; type-only imports
stay as direct `import type` at their use sites.
## Summary
Three fixes to the AI agent surface in the dashboard, all surfaced while
verifying AI SDK 7 support (the SDK side is #3833):
- **AI SDK 7 telemetry rendering.** The generation-span inspector and
run metrics now read both the v6 (`ai.*`) and v7 (`gen_ai.*`) telemetry
attribute shapes. On v7 the Messages, Provider, and Tools views showed
empty/unknown before; now they render correctly.
- **HITL approvals in the conversation view.** The agent conversation
view renders human-in-the-loop tool approvals and denials (awaiting
approval, approved, denied with reason) instead of leaving the tool part
blank. This gap predated v7.
- **Chat snapshot store mismatch.** Chat session snapshots now resolve
through a single storage-key helper shared by the SDK write and the
dashboard read. Previously the write applied the default object-store
protocol to the key while the read fell back to a different store, so
the dashboard 404'd on the snapshot and showed only a partial
conversation.
## Summary
On a busy webapp the trace/log/metric ingestion path emits several
`info` logs per insert batch, which makes up the bulk of the service's
log output. This moves that per-batch chatter to `debug` and adds an
opt-in to drop successful HTTP access logs, cutting log volume with no
loss of error signal.
## Details
The per-batch ClickHouse insert logs, the flush scheduler's concurrency
adjustments, and the event-loop utilization sample (already exported as
a metric, so the log line was redundant) now log at `debug`. Error and
warning logs are untouched.
New `HTTP_ACCESS_LOG_DISABLED=1` env var: when set, the HTTP access
logger skips successful (2xx) requests while still logging non-2xx
responses. Defaults off, so existing deployments are unchanged.
## Summary
This PR improves performance across the Environment Variables page.
## Changes
### Targeted value loading
- load only the non-secret (environmentId, key) pairs required by the
page. Secret values continue to be redacted in the UI.
### SSR windowing + virtualization
- SSR-render only the first 50 rows
- hydrate those rows
- virtualize the remaining dataset client-side
- search is now URL-driven during SSR, ensuring deep links such as
`?search=DATABASE_URL`
### Lightweight 'Create' flow
- 'Create' page no longer loads the full Environment Variables dataset.
## Results
Large projects no longer render thousands of rows during SSR.
Example (~11k rendered rows):
Metric | Before | After
-- | -- | --
Document size | ~150 MB | ~5 MB
SSR rows | ~11k | 50
Browser DOM rows | Thousands | ~26–38
## Summary
The mollifier had ~21 behavioural constants baked in as hardcoded values
— the buffer's ack-grace TTL and Redis retry/reconnect tuning, the
drainer's poll interval and backoff envelope, the pre-gate idempotency
claim TTL/wait/poll, the buffered-run mutate-with-fallback wait loop,
the metadata CAS retry budget and backoff, the stale-sweep scan bounds,
and the draining-gauge interval. None could be adjusted without a code
change, which makes tuning the system under production load impossible.
This exposes all of them as `TRIGGER_MOLLIFIER_*` environment variables,
each defaulting to its previous hardcoded value. Behaviour is identical
unless an operator sets a var, so it's a safe no-op deploy.
## Design
The package-level classes (`MollifierBuffer`, `MollifierDrainer` in
`@trigger.dev/redis-worker`) gain optional constructor options
defaulting to the old constants — backward compatible, hence a patch
changeset. The webapp factories and worker bootstraps read the env and
pass them through. The route- and concern-level pure helpers
(mutate-with-fallback, metadata mutation, idempotency claim, stale-sweep
state) keep their existing `?? DEFAULT` option fallbacks and are fed env
values at their call sites, so they stay unit-testable without importing
`env.server`.
## Test plan
- [x] `@trigger.dev/redis-worker` builds
- [x] webapp typecheck passes
- [x] mollifier buffer + drainer testcontainer suites pass (modulo a
couple of pre-existing flaky timing tests)
- [x] Reviewer: confirm the `TRIGGER_MOLLIFIER_*` env var names match
ops conventions
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>
Adds backward-compatible support for large trigger payloads by reusing
the existing object-storage packet flow.
Large payloads are uploaded to object storage before the trigger request
is sent. The trigger API receives a small application/store pointer
payload instead of embedding large JSON bodies in the request.
Small payload behavior is unchanged.
## Summary
Adds a nullable `pricing_unit` column to the LLM model registry's
`llm_models` table, recording how each model is billed ("tokens",
"characters", "images", "minutes", "requests", "free", "not_findable").
It lets pricing-coverage reporting exclude models that aren't priced
per-token (image/video/audio models currently drag the "% priced" number
down even though they can never carry a per-token price), and lays the
groundwork for non-token pricing.
The default model catalog is entirely per-token, so `seed` and
`syncLlmCatalog` set `pricing_unit="tokens"` on those rows. The admin
LLM model form (create + edit) and the admin API get a pricing-unit
selector so admin-curated models can set it; existing rows can stay
unset.
Auto-discovered models get their unit from the model-registry pipeline,
which lands separately.
---------
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
## Summary
When a chat's restored session state points at a session that no longer
exists in the current environment — for example a `sessions` entry that
was persisted against a different trigger environment —
`useTriggerChatTransport` assumed the session was live and never created
a real one. The next message then failed with a 404 and the chat
couldn't send.
## Fix
`callWithAuthRetry` now treats a 404 from a session-PAT-authed call as
"this session doesn't exist here". After the existing 401/403 token
refresh, a 404 recreates the session via `startSession`, drops the stale
`lastEventId` resume cursor (it pointed at another environment's
stream), and retries the send once. When `startSession` isn't configured
the transport throws a clear message instead of a bare 404.
Routine maintenance pass on a few transitive `pnpm.overrides`.
- `fast-uri` / `fast-xml-builder`: add overrides pinning to current
releases (`3.1.2` / `1.1.7`).
- `protobufjs` / `qs`: bump existing override pins that had fallen a
patch behind (`7.5.6` / `6.15.2`).
Overrides-only - no first-party code changes; lockfile regenerated to
match. Verified the affected transitives resolve to the pinned releases
via `pnpm why -r`.
## ✅ Checklist
- [x] I have followed every step in the [contributing
guide](https://github.com/triggerdotdev/trigger.dev/blob/main/CONTRIBUTING.md)
- [x] The PR title follows the convention.
- [x] I ran and tested the code works
---
## Testing
Upgraded the email packages in `internal-packages/emails`:
| Package | Before | After |
| --- | --- | --- |
| `@react-email/components` | `0.0.16` | `1.0.12` |
| `@react-email/render` | `^0.0.12` | `^2.0.8` |
| `react-email` (CLI) | `^2.1.1` | `^6.5.0` |
| `react-dom` | _(missing)_ | `^18.2.0` (now a required peer of render)
|
**Breaking change handled:** `render()` is now async (`Promise<string>`)
in `@react-email/render` v1+. Added `await` in the `aws-ses`, `smtp` and
`null` transports. `EmailClient` and the webapp callers were already
async and needed no changes.
Verification:
- `pnpm run typecheck --filter emails` ✅
- `pnpm run typecheck --filter webapp` (consumer of the `emails`
package) ✅
- **Before/after render comparison**: rendered all 11 templates
(magic-link, invite, welcome, alert-attempt/run/error-group,
deployment-failure/success, mfa-enabled/disabled, bulk-action-complete)
to HTML with both the old and new packages and compared them visually +
via HTML diff. Output is visually identical. The only HTML changes come
from upstream improvements: `<Body>` now wraps content in a
`<table>`/`<td>` for better email-client compatibility, an
`x-apple-disable-message-reformatting` meta tag was added, and CSS
shorthand (e.g. `margin`) is now also emitted as longhand. No visual
regressions; the `CodeBlock`/dracula theme, buttons, and row/column
layouts all render correctly.
No changeset or `.server-changes/` entry is added: `emails` is a private
internal package (not under `packages/`), and there is no user-facing
behavior change.
---
## Changelog
Upgrade `react-email` and `@react-email/{components,render}` in
`internal-packages/emails` to their latest versions and adapt the mail
transports to the now-async `render()` API.
---
## Screenshots
Rendered email templates before vs after the upgrade (visually
identical):
**Before** (`@react-email/components@0.0.16`, `render@0.0.12`)
## Summary
A fresh clone running `trigger dev` against the local webapp shows the
run in the dashboard but an empty trace view.
`EVENT_REPOSITORY_DEFAULT_STORE` was unset in `.env.example`, so span
ingestion fell back to the `postgres` store while the rest of the local
stack (run replication, `CLICKHOUSE_URL`) is ClickHouse-backed.
Defaulting it to `clickhouse_v2` makes dev run traces show up out of the
box.
The reference/example projects (`references/`) now live in their own
repo, https://github.com/triggerdotdev/references, so their heavy,
frequently-changing dependencies are no longer part of this repo's
lockfile and tooling. This removes them here and repoints everything
that referenced them.
- Deletes `references/`; updates the pnpm workspace + lockfile.
- Clears the references-only CI rules and `.vscode` configs.
- Repoints the docs (contributor/agent + one public page) to the new
repo.
- `seed.mts` keeps the local-dev projects (hello-world, d3-chat,
realtime-streams).
Brings the OpenTelemetry packages up to the latest coherent release
across the webapp and the published packages (`@trigger.dev/core`, the
CLI, `@trigger.dev/sdk`) plus
`internal-packages/{tracing,testcontainers}`:
- `@opentelemetry/sdk-node` 0.218.0
- `@opentelemetry/core` 2.7.1
- `@opentelemetry/host-metrics` 0.38.3
We were already on the otel 2.x line, so this is a same-major minor move
- the versions are pinned to `@opentelemetry/sdk-node@0.218.0`'s own
declared dependency set so the experimental (0.2xx) and stable (2.x)
packages stay coherent (mixing them is the usual otel breakage).
**One code change:** otel 0.215 made `forceFlush()` a required method on
`LogRecordExporter`, so `ExternalLogRecordExporterWrapper` (core's
tracing SDK) gains a `forceFlush()` that delegates to the underlying
exporter.
**Notable upgrades along the way:** OTLP exporters can take a custom
HTTP agent (connection pooling/keepAlive on the export path), HTTP
request headers are captured at span creation, and core hot-path perf
improvements in 2.6.1/2.7. `host-metrics` 0.37→0.38 is a clean upgrade.
Patch changeset added for the three published packages. References
projects are intentionally untouched.
Verified: `@trigger.dev/core` / CLI / `@trigger.dev/sdk` build, webapp +
`@internal/tracing` typecheck - all green.
Adds wide-event observability for the supervisor: one flat-keyed JSON
line per dequeue iteration, workload-server route, and run socket
lifecycle event. Events carry `trace_id` sourced from the inbound W3C
traceparent plus `meta.run_id` and related identifiers, so they join
across services by run.
The outbound warm-start POST also forwards the inbound traceparent so
the upstream receiver continues the same trace instead of minting a new
one.
Off by default behind `TRIGGER_WIDE_EVENTS_ENABLED`. With the flag off,
no events are emitted, no ALS state is allocated, and the outbound
warm-start request is unchanged — every call site was audited to confirm
the off path is byte-identical to current behavior.
Dequeue-path phase timings recorded under `phase.<name>.duration_ms`:
`restore`, `warm_start`, `workload_create`. A `path_taken` extra
distinguishes `restore` / `warm_start` / `cold_create` /
`skipped_no_image`.
Refs TRI-9480.
## Summary
The Runs list now updates live without requiring a page refresh. Status
changes and other run fields are updated in place while runs are
executing.
When new runs matching the current filters are created, a "New runs
created" refresh button appears above the list.
Root runs now show a live child-run status breakdown directly in the
status tooltip.
### List live update
- Visible runs update in place while they are still running.
- A "New runs created" refresh button appears when new matching runs are
detected.
- Polling stops when all visible runs have finished and a refresh button
is already shown.
- Polling pauses when the browser tab is not visible.
- Runs list status updates and new-run detection share a single
runs/live polling path.
### Child-status tooltip
- Root run tooltips now display a breakdown of child run statuses.
- Child statuses are loaded when the tooltip opens (after a 400ms hover
delay).
- The tooltip stays up to date while child runs are still changing
state.
- Handles cases where child runs continue running after their parent run
has completed, or have not yet been created.
### Supporting changes
- Added hidden-tab awareness to polling.
- Added safeguards around polling inputs (`runIds` deduping and limits).
## Test plan
- [x] pnpm run typecheck --filter webapp passes
- [x] cd apps/webapp && pnpm run test
./test/presenters/mapRunToLiveFields.test.ts --run passes
- [x] cd apps/webapp && pnpm run test
./test/runsRepository.part2.test.ts --run -t "hasNewRuns" passes
### Manual smoke:
- [x] Active runs update without a page refresh.
- [x] A new matching run shows the refresh banner and the banner actions
work as expected.
- [x] Root run tooltips show live child-status updates and stop polling
once child runs settle.
---------
Co-authored-by: Ekaterina Bulatova <kathiekiwi@Ekaterinas-MacBook-Pro.local>
## Problem
On environments where runs carry a Postgres-backed `taskEventStore`
value (`taskEvent` or `taskEventPartitioned`), OTLP ingest endpoints
(`POST /otel/v1/traces` and `/otel/v1/logs`) were returning HTTP 500.
**Root cause:** The org-scoped ClickHouse factory introduced in a recent
PR routes all OTLP spans through `getEventRepositoryForOrganizationSync`
→ `buildEventRepository`. That function only handles `"clickhouse"` and
`"clickhouse_v2"` store values and throws `Unknown ClickHouse event
repository store: <value>` for anything else. The throw occurred inside
the grouping loop of `#exportEvents`, unwinding the entire method and
returning 500 for the whole batch.
The OpenTelemetry collector's `otlphttp` exporter treats HTTP 500 as
non-retryable and drops the batch — causing real span loss.
**Fix:** Guard the `getEventRepositoryForOrganizationSync` call in
`#exportEvents` so it is only invoked for `clickhouse` / `clickhouse_v2`
store values. All other values are routed directly to the Postgres
`eventRepository`, matching the guard pattern already present in
`resolveEventRepositoryForStore` and `getEventRepositoryForStore` in
`eventRepository/index.server.ts`.
The ClickHouse factory call is also wrapped in a try/catch that falls
back to Postgres so any unexpected store value in a future OTLP batch
degrades gracefully instead of failing the whole request.
## Changes
- `apps/webapp/app/v3/otlpExporter.server.ts` — add Postgres routing
guard and try/catch fallback in `#exportEvents`
## Testing
The `eventRepository/index.server.ts` module already has the same guard
pattern thoroughly covered. The fix brings `#exportEvents` into
alignment with that existing, tested pattern. Manual verification:
confirm OTLP batches containing Postgres-store spans return 200 and
route to the correct repository.
## Summary
The environment variables import API now accepts an optional `isSecret`
flag, so imported variables can be created as secret (redacted)
environment variables instead of plaintext. When the flag is omitted,
variables default to non-secret, preserving existing behavior for CLI
deploys and dashboard imports.
This is useful for tools that push secrets into Trigger.dev (for
example, syncing from a secrets manager) and want them stored as secrets
rather than plain environment variables.
It's available through `envvars.import` in the SDK and the `POST
/api/v1/projects/{projectRef}/envvars/{slug}/import` endpoint, and is
honored for both regular and preview-branch environments.
```ts
await envvars.import("proj_1234", "prod", {
variables: { STRIPE_SECRET_KEY: "sk_live_..." },
isSecret: true,
});
```
## What
Adds [pkg.pr.new](https://pkg.pr.new) continuous preview releases. Every
push to a branch builds the public `@trigger.dev/*` packages and
publishes installable preview builds keyed by commit SHA — **without
touching the npm registry**. pkg.pr.new drops install instructions on
the associated PR:
```
npm i https://pkg.pr.new/@trigger.dev/sdk@<sha>
```
This lets reviewers and users try a branch (SDK, CLI, core, etc.) before
anything is released, separate from the changesets release, the manual
`--snapshot` prerelease, and the chat-prerelease flow.
## How
`.github/workflows/preview-packages.yml` (push trigger) → install →
generate Prisma → **stamp preview version** → build → `pkg-pr-new
publish`.
### The version stamp (the important part)
pkg.pr.new serves previews by SHA but does **not** rewrite the
package.json `version` field. If a preview shipped as `4.5.0-rc.4`, a
consumer who installed it would pin `4.5.0-rc.4` to the preview tarball
in their lockfile/cache — and a later `npm i
@trigger.dev/sdk@4.5.0-rc.4` from npm could resolve to the stale
preview. This is a known, by-design gap in the tool
(stackblitz-labs/pkg.pr.new#250, #390).
`scripts/stamp-preview-version.mjs` runs **before the build** and
rewrites every public package to a unique `0.0.0-preview-<sha>`. The
`0.0.0-` prefix can never satisfy a real semver range, so the collision
is structurally impossible (same convention React/Next canaries use).
Running before the build also means `scripts/updateVersion.ts` bakes the
preview version into the runtime `VERSION` constant, so previews are
self-identifying (`trigger --version`, the `x-trigger-cli-version`
header, the MCP server version) instead of all reporting the RC version.
Sibling `workspace:` specifiers are relaxed to `workspace:*` so `pnpm
pack` resolves them against the rewritten versions — `packages/python`
pins peerDependencies as `workspace:^4.5.0-rc.4`, which would otherwise
be unsatisfiable once the version changes. Non-public deps
(`@trigger.dev/database`, `@internal/*`) are left untouched. All
mutations happen on the ephemeral CI checkout; nothing is committed.
## GitHub App
The pkg.pr.new GitHub App is **already installed** on
`triggerdotdev/trigger.dev` (has been for a while), so no setup is
needed. Confirmed live — this branch's pushes published all 10 public
packages, e.g.
```
pnpm add https://pkg.pr.new/@trigger.dev/sdk@e4dfc59
```
## Fork limitation
pkg.pr.new authenticates with a GitHub Actions OIDC token, which GitHub
does not issue to `pull_request` workflows from forks. The `push`
trigger therefore covers branches pushed to this repo (core team), not
external fork PRs. Fork coverage would need a `workflow_run` two-stage
setup; left out for now.
## Notes
- Pinned `pkg-pr-new@0.0.75` (no Node engine constraint; Node 20 CI is
fine).
- pkg.pr.new
[#525](https://github.com/stackblitz-labs/pkg.pr.new/pull/525) adds a
built-in `--previewVersion` flag (still open). If it lands we can drop
the version-rewrite half of the script, but we'd keep a pre-build stamp
anyway so `updateVersion.ts` picks up the preview version (the flag
rewrites at pack time, too late for the baked `VERSION`).
## Summary
Adds `drainBatchSize` to `MollifierDrainer` (default `1` — preserves
existing behaviour) and wires `TRIGGER_MOLLIFIER_DRAIN_BATCH_SIZE`
through the webapp (default `50`). Each tick the drainer now pops up to
`drainBatchSize` from each chosen env, then dispatches every popped
entry through the shared `concurrency`-bounded `pLimit`. Per-org/per-env
fairness is unchanged — only the in-env pop count grows.
Pre-existing behaviour was one pop per env per tick. For a single-env
burst that single-flighted the drain at the per-tick floor of `pop +
engine.trigger ≈ 50–60 ms`. With buffer entries piling up under a
real-world tenant burst that's tens of minutes of tail latency to fully
materialise — even though PG itself could comfortably sustain the
writes.
## Why this matters — heavy-tail illustration
Scenario: 100 customers in one window — 94 fire 20 triggers each, 5 fire
100, 1 fires 1000. Gate at `THRESHOLD=10/s`, `HOLD_MS=500`. First 10 of
each burst hit PG directly; the rest buffer.
| Customers | Triggers each | PG direct | Buffered each | Total buffered
|
|---|---|---|---|---|
| 94 small | 20 | 10 | 10 | 940 |
| 5 medium | 100 | 10 | 90 | 450 |
| 1 heavy | 1000 | 10 | 990 | 990 |
**With `DRAIN_BATCH_SIZE=50`, `DRAIN_CONCURRENCY=50`, ~50 ms
`engine.trigger`:**
| Tick | Pops | Dispatch waves | Wall-clock |
|---|---|---|---|
| 1 | 94×10 + 5×50 + 1×50 = 1 240 | 25 × 50 ms | ~1 300 ms (94 smalls
done) |
| 2 | 5×40 + 1×50 = 250 | 5 × 50 ms | ~300 ms (5 mediums done) |
| 3–20 | heavy alone, 50/tick | 1 × 50 ms | ~100 ms each |
| Customer class | Buffered fully drained |
|---|---|
| 94 small | **~1.3 s** |
| 5 medium | **~1.6 s** |
| 1 heavy | **~3.4 s** |
**Without batching (one pop per env per tick — current behaviour):**
| Customer class | Buffered fully drained |
|---|---|
| 94 small | ~500 ms |
| 5 medium | ~4.5 s |
| 1 heavy | **~49 s** |
So the heavy single-tenant tail drops from ~49 s to ~3.4 s (~14× faster)
without changing PG load characteristics. Smalls go up slightly in this
scenario (500 ms → 1.3 s) because all 100 envs share one tick's dispatch
queue — that's the trade we accept for the heavy tail; the worst-case
small wait is still inside one tick. PG load is identical either way (50
concurrent inserts at a time, capped by `DRAIN_CONCURRENCY`).
## What changed
**`packages/redis-worker`**
- New `drainBatchSize` option (default 1 — full backward compat).
- `runOnce()` refactored to pop per-env batches in parallel, then
dispatch all popped entries through the existing global `pLimit`.
Mid-batch pop failure aborts only that env's batch and counts as one
failure (same semantic as the old per-env path).
- Removed the now-unused `processOneFromEnv` helper.
**`apps/webapp`**
- `TRIGGER_MOLLIFIER_DRAIN_BATCH_SIZE` env var (default 50, matching
`DRAIN_CONCURRENCY`).
- Wired into `mollifierDrainer.server.ts`.
**Test cloud config** (separate cloud PR):
`TRIGGER_MOLLIFIER_DRAIN_BATCH_SIZE="50"` on the worker service.
Production rollout deferred until we've watched it on test cloud.
## Test plan
- [x] All 25 stub-based drainer tests pass (18 pre-existing + 7 new). 7
new tests under `MollifierDrainer.drainBatchSize`:
- pops up to `drainBatchSize` across ticks
- global `concurrency` cap still holds when batch > concurrency
- mid-batch pop failure isolation
- multi-env batch fan-out in one tick
- **hierarchical org fairness preserved at `drainBatchSize > 1`**
(load-bearing — guards against future regressions to
per-env-instead-of-per-org rotation)
- mixed success/failure accounting in a batched tick
- bounded pops on empty queue (no Lua spam past `drainBatchSize`)
- [x] All pre-existing tests still pass unchanged at default
`drainBatchSize=1` → backward-compat locked.
- [x] `pnpm run build --filter @trigger.dev/redis-worker` clean.
- [x] `pnpm run typecheck --filter webapp` clean.
- [x] `redisTest` block (real Redis via testcontainers) — couldn't run
locally on this branch due to testcontainers runtime discovery; will
validate in CI.
- [ ] Test-cloud smoke after cloud PR lands: fire `burst 50` against a
flagged env and confirm the 50th entry's drain time drops from ~2.5 s to
<200 ms.
## Notes
- Per-tick memory bound: `maxOrgsPerTick × drainBatchSize` entries can
sit in the JS pLimit queue between pop and dispatch. At defaults that's
`500 × 50 = 25 000` × ~5 KB snapshot ≈ ~125 MB worst case per worker —
well within headroom.
- The pre-batch model's strict per-env throughput cap of `1/tick` is
documented as the fairness baseline elsewhere. Org-level fairness is
what callers actually rely on; this change does not weaken that.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Vouches GitHub user
[`ConProgramming`](https://github.com/ConProgramming) (Conner) as an
outside contributor by adding them to `.github/VOUCHED.td`, so their PRs
aren't auto-closed by the vouch check.
Done as a direct edit rather than via the issue flow because they have
no open Vouch Request issue to comment `vouch` on.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
## Summary
Documents the new `tools` option on `chat.agent` (companion to #3790).
Adds a dedicated [Tools](/ai-chat/tools) guide: the three places tools
show up (config, `toStreamTextOptions`, `streamText`), why declaring
them on the config matters for `toModelOutput` across turns, static vs
per-turn tools, the typed `run()` payload,
`InferChatUIMessageFromTools`, the relationship to skills, and the
manual `convertToModelMessages` path for `customAgent` loops.
Threads the option through the rest of the guide: the reference tables,
a happy-path section on the backend page, the types page, and the HITL /
skills / tool-result-auditing patterns. Corrects the sub-agents guide,
where the `toModelOutput` compression was implied to work across turns
but silently degraded from turn 2 without config tools.
Also unstacks the three callouts that were piled under the
`chat.agent()` header on the backend page, and adds a changelog entry.
## Summary
1 new feature, 5 improvements.
## Highlights
- Mollifier `mutateSnapshot` now enforces a tag cap: an `append_tags`
patch carrying `maxTags` returns `"limit_exceeded"` (writing nothing)
when the deduped tag count would exceed the limit, so a buffered run
can't accumulate more tags via the tags API than the trigger validator
allows at creation.
([#3756](https://github.com/triggerdotdev/trigger.dev/pull/3756))
## Improvements
- Mollifier buffer extensions: idempotency dedup, an atomic
`mutateSnapshot` API, metadata CAS, claim primitives, and a
`MollifierSnapshot` type. The buffer's Redis client now reconnects with
jittered backoff so a fleet of clients doesn't stampede Redis in
lockstep after a blip.
([#3752](https://github.com/triggerdotdev/trigger.dev/pull/3752))
- Add `onTerminalFailure` callback to `MollifierDrainerOptions` so the
customer's run lands a SYSTEM_FAILURE PG row even when the drainer
exhausts `maxAttempts` on a retryable PG error. Previously,
retryable-error exhaustion called `buffer.fail()` directly, which
atomically marks FAILED + DELs the entry hash with no PG write — silent
data loss when PG was unreachable across the full retry budget. The
callback fires before `buffer.fail()` on any terminal path (`cause:
"non-retryable"` or `"max-attempts-exhausted"`); throwing a retryable
error from the callback causes the drainer to requeue rather than fail.
([#3754](https://github.com/triggerdotdev/trigger.dev/pull/3754))
- Bump `@s2-dev/streamstore` to `0.22.10` to fix a
`TASK_RUN_UNCAUGHT_EXCEPTION` ("Invalid state: Unable to enqueue") when
a `chat.agent` turn is aborted mid-stream.
([#3792](https://github.com/triggerdotdev/trigger.dev/pull/3792))
- Coerce numeric `concurrencyKey` values to string at the API boundary
across `tasks.trigger`, `tasks.batchTrigger`, and the Phase-2 streaming
batch endpoint.
([#3789](https://github.com/triggerdotdev/trigger.dev/pull/3789))
- Add a `tools` option to `chat.agent`. Declaring your tools here
threads them into the SDK's internal `convertToModelMessages`, so each
tool's `toModelOutput` is re-applied when prior-turn history is
re-converted.
([#3790](https://github.com/triggerdotdev/trigger.dev/pull/3790))
<details>
<summary>Raw changeset output</summary>
⚠️⚠️⚠️⚠️⚠️⚠️
`main` is currently in **pre mode** so this branch has prereleases
rather than normal releases. If you want to exit prereleases, run
`changeset pre exit` on `main`.
⚠️⚠️⚠️⚠️⚠️⚠️
# Releases
## @trigger.dev/redis-worker@4.5.0-rc.4
### Minor Changes
- Mollifier buffer extensions: idempotency dedup, an atomic
`mutateSnapshot` API, metadata CAS, claim primitives, and a
`MollifierSnapshot` type. The buffer's Redis client now reconnects with
jittered backoff so a fleet of clients doesn't stampede Redis in
lockstep after a blip.
([#3752](https://github.com/triggerdotdev/trigger.dev/pull/3752))
- Add `onTerminalFailure` callback to `MollifierDrainerOptions` so the
customer's run lands a SYSTEM_FAILURE PG row even when the drainer
exhausts `maxAttempts` on a retryable PG error. Previously,
retryable-error exhaustion called `buffer.fail()` directly, which
atomically marks FAILED + DELs the entry hash with no PG write — silent
data loss when PG was unreachable across the full retry budget. The
callback fires before `buffer.fail()` on any terminal path (`cause:
"non-retryable"` or `"max-attempts-exhausted"`); throwing a retryable
error from the callback causes the drainer to requeue rather than fail.
([#3754](https://github.com/triggerdotdev/trigger.dev/pull/3754))
### Patch Changes
- Pipeline the per-entry `HGETALL` fetches in
`MollifierBuffer.listEntriesForEnv`. The previous serial implementation
issued one Redis round-trip per runId returned by `LRANGE`, which
dominated stale-sweep wall-time at any meaningful backlog (at the
sweep's default maxCount=1000, this is ~1000 RTTs per env per pass).
Behaviour is unchanged — entries are still skipped when the entry hash
has been torn down by a concurrent drainer ack/fail between the LRANGE
and the HGETALL.
([#3752](https://github.com/triggerdotdev/trigger.dev/pull/3752))
- Mollifier `mutateSnapshot` now enforces a tag cap: an `append_tags`
patch carrying `maxTags` returns `"limit_exceeded"` (writing nothing)
when the deduped tag count would exceed the limit, so a buffered run
can't accumulate more tags via the tags API than the trigger validator
allows at creation.
([#3756](https://github.com/triggerdotdev/trigger.dev/pull/3756))
- Updated dependencies:
- `@trigger.dev/core@4.5.0-rc.4`
## @trigger.dev/build@4.5.0-rc.4
### Patch Changes
- Updated dependencies:
- `@trigger.dev/core@4.5.0-rc.4`
## trigger.dev@4.5.0-rc.4
### Patch Changes
- Bump `@s2-dev/streamstore` to `0.22.10` to fix a
`TASK_RUN_UNCAUGHT_EXCEPTION` ("Invalid state: Unable to enqueue") when
a `chat.agent` turn is aborted mid-stream.
([#3792](https://github.com/triggerdotdev/trigger.dev/pull/3792))
- Updated dependencies:
- `@trigger.dev/core@4.5.0-rc.4`
- `@trigger.dev/build@4.5.0-rc.4`
- `@trigger.dev/schema-to-json@4.5.0-rc.4`
## @trigger.dev/core@4.5.0-rc.4
### Patch Changes
- Coerce numeric `concurrencyKey` values to string at the API boundary
across `tasks.trigger`, `tasks.batchTrigger`, and the Phase-2 streaming
batch endpoint.
([#3789](https://github.com/triggerdotdev/trigger.dev/pull/3789))
- Bump `@s2-dev/streamstore` to `0.22.10` to fix a
`TASK_RUN_UNCAUGHT_EXCEPTION` ("Invalid state: Unable to enqueue") when
a `chat.agent` turn is aborted mid-stream.
([#3792](https://github.com/triggerdotdev/trigger.dev/pull/3792))
## @trigger.dev/plugins@4.5.0-rc.4
### Patch Changes
- Updated dependencies:
- `@trigger.dev/core@4.5.0-rc.4`
## @trigger.dev/python@4.5.0-rc.4
### Patch Changes
- Updated dependencies:
- `@trigger.dev/sdk@4.5.0-rc.4`
- `@trigger.dev/core@4.5.0-rc.4`
- `@trigger.dev/build@4.5.0-rc.4`
## @trigger.dev/react-hooks@4.5.0-rc.4
### Patch Changes
- Updated dependencies:
- `@trigger.dev/core@4.5.0-rc.4`
## @trigger.dev/rsc@4.5.0-rc.4
### Patch Changes
- Updated dependencies:
- `@trigger.dev/core@4.5.0-rc.4`
## @trigger.dev/schema-to-json@4.5.0-rc.4
### Patch Changes
- Updated dependencies:
- `@trigger.dev/core@4.5.0-rc.4`
## @trigger.dev/sdk@4.5.0-rc.4
### Patch Changes
- Add a `tools` option to `chat.agent`. Declaring your tools here
threads them into the SDK's internal `convertToModelMessages`, so each
tool's `toModelOutput` is re-applied when prior-turn history is
re-converted.
([#3790](https://github.com/triggerdotdev/trigger.dev/pull/3790))
```ts
chat.agent({
tools: { readFile, search },
run: async ({ messages, tools, signal }) =>
streamText({ model, messages, tools, abortSignal: signal }),
});
```
Also exports `InferChatUIMessageFromTools<typeof tools>` to derive the
chat `UIMessage` type (typed tool parts) directly from a tool set.
- Updated dependencies:
- `@trigger.dev/core@4.5.0-rc.4`
</details>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Follow-up to #3796, which bumped the slack-client axios paths but left
posthog-node's transitive `axios@1.15.1` in place.
`posthog-node` 4.17.1 → 5.35.6. v5 drops the axios dependency entirely
(it's now fetch-based via `@posthog/core`), so posthog's old axios path
disappears. With #3796 already on main (webapp + d3 references on
`@slack/web-api@7.16.0`), nothing else pins the old line, so the
now-dead `axios@>=1.0.0 <1.15.0` override is removed and axios resolves
to a single patched `1.16.1` repo-wide. This closes the remaining axios
advisories.
Compat: the webapp's usage in `telemetry.server.ts` (`new PostHog(key, {
host })`, `.identify`, `.groupIdentify`, `.capture`) is all object-form
API that v5 preserves; `pnpm run typecheck --filter webapp` passes.
Node: posthog-node v5 requires Node `^20.20.0 || >=22.22.0`. We run
20.20.0 in dev (`.nvmrc`), CI, and the published Docker image
(`node:20.20-bullseye-slim`), so we're compliant.
Bumps `@slack/web-api` 7.9.1 → 7.16.0 in the webapp and the two
`references` examples (d3-chat, d3-openai-agents). 7.16.0 depends on
`axios@^1.16.0`, so every slack-client axios path resolves to 1.16.1
instead of 1.15.1.
This clears the slack and references axios paths. `posthog-node`'s
transitive axios still resolves the older line - that's handled in a
follow-up that upgrades posthog-node to v5 (which drops the axios
dependency entirely and lets us retire the now-stale axios override).
The dependabot axios advisories fully close once both land.
## Summary
A `chat.agent` turn that gets aborted mid-stream (stop generation, idle
suspend, cancellation) could surface a `TASK_RUN_UNCAUGHT_EXCEPTION` —
`TypeError: Invalid state: Unable to enqueue` — in the dashboard. The
run kept working, but the exceptions were loud and confusing.
## Root cause
The realtime stream writer batches chunks through
`@s2-dev/streamstore`'s `BatchTransform`, which holds them for a short
linger window before flushing. When the turn's abort signal fires while
a record is still buffered, the stream's writable is aborted and the
transform's readable controller is closed — but the pending linger
`setTimeout` still fires and calls `controller.enqueue()` on the dead
controller, throwing from a timer callback where nothing can catch it.
Fixed upstream in `@s2-dev/streamstore@0.22.10`, which wraps the linger
flush in a try/catch that discards the closed-controller error. This
bumps the dependency across core, the CLI, and the webapp, and adds a
regression test against the real `BatchTransform`.
Verified end-to-end: a mid-stream stop that previously failed the run
with `TASK_RUN_UNCAUGHT_EXCEPTION` now leaves the run healthy.
## Summary
`chat.agent` now takes a `tools` option. Until now tools only went to
`streamText` inside `run()`, so the SDK had no tools when it
re-converted the persisted `UIMessage` history at the start of each
turn. Any tool with a `toModelOutput` (raw image bytes into an image
content part, or a sub-agent transcript compressed to a summary) had its
transform applied on turn 1 and skipped from turn 2 onward, so the raw
output got JSON-stringified back into the prompt and the model lost the
transformed view.
Declaring `tools` on the config threads them into that conversion, so
`toModelOutput` runs on every turn. The resolved set is handed back,
typed, on the `run()` payload as `tools`:
```ts
const tools = { searchDocs, renderChart };
export const myChat = chat.agent({
tools,
run: async ({ messages, tools, signal }) =>
streamText({ ...chat.toStreamTextOptions({ tools }), messages, abortSignal: signal }),
});
```
`tools` also accepts a per-turn function for tools that depend on the
user or a feature flag. Only `inputSchema` and `toModelOutput` are read
during conversion, never `execute`. Also exports
`InferChatUIMessageFromTools<typeof tools>` to derive the chat
`UIMessage` type from a tool set. No behavior change for agents that
don't declare `tools`.
## Summary
Dashboard surfaces handle buffered runs by falling back to the mollifier
snapshot:
- Run detail, span detail, streams view (`_app.../runs.\$runParam`,
`resources.../spans.\$spanParam`, `resources.../streams.\$streamKey`).
- Redirect routes (`@.runs.\$runParam`, `runs.\$runParam`,
`projects.v3.\$projectRef.runs.\$runParam`).
- Action routes — cancel / replay / idempotency-reset / debug — under
`resources.taskruns/...` and `resources.../idempotencyKey.reset`.
- Logs download.
- Realtime subscription route + per-run resource
(`realtime.v1.runs.\$runId`, `resources.../realtime.v1.*`).
- `CancelRunDialog` gains an `onCancelSubmitted` callback so submit
isn't raced by the Radix `DialogClose` wrapper.
Stacked on the mutations PR.
## Test plan
- [x] \`pnpm run typecheck --filter webapp\` passes
- [x] \`pnpm run test --filter webapp
test/mollifierRealtimeRunResource.test.ts\` passes
- [x] \`pnpm run test --filter webapp
test/mollifierRealtimeRunResourceBuffer.test.ts\` passes
- [x] \`pnpm run test --filter webapp
test/mollifierRealtimeSubscription.test.ts\` passes
- [x] Manual smoke: trigger a buffered run, open it in the dashboard,
replay/cancel from the UI
---
## Ship-gate follow-up fixes
- **Auto-redirect to root span on direct nav** — loader sets `?span=`
from root span (PG) or buffered snapshot spanId before 302'ing, so
bookmark/share-link/direct-nav doesn't leave the panel collapsed.
- **RunPresenter switches from `findFirstOrThrow` to `findFirst` + typed
`RunNotInPgError`** — kills the per-poll `PrismaClient error` log spam
for buffered runs without changing the route-loader's fallback flow.
- **Span detail panel renders for buffered runs** — `SpanPresenter.call`
now falls back to `findRunByIdWithMollifierFallback` +
`buildSyntheticSpanRun` instead of returning undefined and triggering
the "Event not found" toast loop.
- **Logs download for buffered runs returns a gzipped placeholder line**
— replaces the 404 with a content-encoded line explaining the run is
queued. Same org-membership gate as the PG path.
- **Admin Debug-Run button hidden for buffered runs + SpanRun circular
type alias broken** (squashed) — buttons gate on a new `isBuffered` flag
on the synthetic SpanRun. Required grounding SpanRun in
`SpanPresenter.getRun` to break a circular type alias TS no longer
tolerates once `isBuffered` is a literal field on the shape.
- **Replay action requires user auth + org-membership** (🚩 Devin
finding) — `action` was unauthenticated and the PG `findFirst` had no
org filter, so any caller with a valid `runParam` could replay any run.
Buffered fallback inherited the same gap. Fixed to mirror the cancel
route.
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
## Summary
Cancel, replay, reschedule, metadata, tags, and idempotency-key-reset
now succeed against a run that's still in the mollifier buffer.
Mutations are applied to the buffered snapshot via Lua CAS; the drainer
carries the mutation forward when it replays.
Primitives added:
- `mutateWithFallback` — PG-first / buffer-fallback resolver with
bounded-wait safety net for entries that transition mid-mutation.
- `applyMetadataMutation` — buffered metadata PUT mirroring the PG-side
retry loop with CAS atomicity.
- `resolveRunForMutation` — discriminated-union resolver used by route
`findResource` so the route builder's pre-action 404 check sees buffered
runs.
Routes wired (whole files, no GET/POST splits):
- `api.v2.runs.\$runParam.cancel.ts`
- `api.v1.runs.\$runParam.replay.ts`
- `api.v1.runs.\$runParam.reschedule.ts`
- `api.v1.runs.\$runId.metadata.ts`
- `api.v1.runs.\$runId.tags.ts`
- `resetIdempotencyKey.server.ts`
Stacked on the reads PR.
## Test plan
- [x] \`pnpm run typecheck --filter webapp\` passes
- [x] \`pnpm run test --filter webapp
test/mollifierMutateWithFallback.test.ts\` passes
- [x] \`pnpm run test --filter webapp
test/mollifierApplyMetadataMutation.test.ts\` passes
- [x] \`pnpm run test --filter webapp
test/mollifierResolveRunForMutation.test.ts\` passes
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
## Summary
Synthesise QUEUED/FAILED responses from the mollifier buffer when a
TaskRun row hasn't landed in Postgres yet. Wires the synthesis into:
- `ApiRetrieveRunPresenter`
- v1 trace GET route
- v1 spans GET route
- attempts route gains a GET loader (fixes pre-existing Remix "no
loader" 400)
The `readFallback` infra itself lives on the trigger PR (consumed by
`IdempotencyKeyConcern`); this PR adds the route-level
synthetic-rendering primitives.
Stacked on the replay PR.
## Test plan
- [x] \`pnpm run typecheck --filter webapp\` passes
- [x] \`pnpm run test --filter webapp
test/mollifierSyntheticRedirectInfo.test.ts\` passes
- [x] \`pnpm run test --filter webapp
test/mollifierSyntheticSpanRun.test.ts\` passes
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
## Summary
The replay side of the mollifier:
- `DrainerHandler`: reads buffered snapshots and replays them through
`engine.trigger` to materialise PG rows.
- `RunEngine.createCancelledRun`: new public method the handler uses to
write CANCELED rows directly from snapshots (bypass queue + waitpoint,
emit `runCancelled`). Tolerates the cjson empty-table tags edge case
found during validation.
- Drainer fairness: org → env rotation so a heavy env doesn't starve
light ones in the same org.
- Stale-entry sweep + telemetry + alertable gauge so a stuck/offline
drainer surfaces in alerts.
Both the drainer and sweep default-off; nothing fires unless flagged on
(`TRIGGER_MOLLIFIER_DRAINER_ENABLED`,
`TRIGGER_MOLLIFIER_STALE_SWEEP_ENABLED`).
Stacked on the trigger-time decisions PR.
## Test plan
- [x] \`pnpm run typecheck --filter webapp\` passes
- [x] \`pnpm run test --filter webapp
test/mollifierDrainerHandler.test.ts\` passes
- [x] \`pnpm run test --filter webapp test/mollifierStaleSweep.test.ts\`
passes
- [x] \`pnpm run test --filter @internal/run-engine
src/engine/tests/createCancelledRun.test.ts\` passes
- [x] \`pnpm run test --filter @trigger.dev/redis-worker
packages/redis-worker/src/mollifier/drainer.test.ts\` passes
---
## Ship-gate follow-up fix
**Drainer writes SYSTEM_FAILURE on max-attempts exhaustion.** Adds an
`onTerminalFailure` callback on `MollifierDrainerOptions` so the
customer's run lands a SYSTEM_FAILURE PG row even when the drainer
exhausts `MAX_ATTEMPTS` on a retryable PG error (previously
`buffer.fail()` was called with no row written → silent data loss). The
callback runs before `buffer.fail()` on every terminal path
(non-retryable AND max-attempts-exhausted), and re-throwing a retryable
error from the callback causes the drainer to requeue rather than fail.
Bumps `@trigger.dev/redis-worker` to a **minor** changeset (additive
option + new exported types). Includes 5 unit tests covering both
terminal causes plus the requeue-on-retryable-callback-failure path and
no-callback back-compat.
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
## Summary
`concurrencyKey` validation accepted only `z.string().optional()` on the
single-trigger and V2/V3 batch endpoints, and the Phase-2 streaming
NDJSON endpoint accepted `z.record(z.unknown()).optional()` for the
entire `options` field. Callers passing `concurrencyKey: someNumericId`
(e.g. `payload.userId`) either failed schema validation on the first two
paths or sailed through on Phase-2 and then failed downstream at
`prisma.taskRun.create` with `Argument concurrencyKey: Expected String
or Null, provided Int`.
The schema now accepts `string | number` for `concurrencyKey` and
stringifies on the way in, across all three paths. The Phase-2 NDJSON
`options` is tightened to reuse the strict
`BatchTriggerTaskItem.options` shape so it validates identically to the
V2/V3 batch endpoints.
A defensive `typeof === "number"` coercion at the `engine.trigger` call
site in `RunEngineTriggerTaskService` covers in-flight Redis-stored
batch items enqueued before the schema fix — those items are rebuilt
from a `Record<string, unknown>` shape that bypasses the new schema and
would otherwise continue failing for up to their TTL.
## Test plan
- [x] `packages/core/src/v3/schemas/batchItemNDJSON.test.ts` — unit
tests covering numeric→string coercion, string passthrough, no-options,
and rejection of non-string/non-number shapes across
`TriggerTaskRequestBody`, `BatchTriggerTaskItem`, and `BatchItemNDJSON`.
- [x] `apps/webapp/test/engine/triggerTask.test.ts` — `containerTest`
simulating the in-flight Redis batch-item shape (numeric
`concurrencyKey` via `Record<string, unknown>`), verifies the run is
created with `concurrencyKey: "51262"`. Without the worker coercion, the
test reproduces the production stack at `prisma.taskRun.create`.
- [x] `pnpm run typecheck --filter webapp` clean.
- [x] `pnpm run build --filter @trigger.dev/core --filter
@trigger.dev/sdk --filter trigger.dev` clean.
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
## Summary
The trigger hot path's mollifier integration:
- `mollifyTrigger`: when the gate trips, write the engine.trigger
snapshot to the buffer and return a synthesised QUEUED response.
Postgres write is deferred to drainer-replay (next PR in the stack).
- Pre-gate idempotency-key claim: same-key triggers serialise through
Redis so a burst lands in PG / buffer exactly once.
- Read-fallback extensions: `findRunByIdWithMollifierFallback` for the
trigger-time idempotency lookup that must see buffered runs.
- Gate bypasses: `debounce`, `oneTimeUseToken`,
`parentTaskRunId`/`triggerAndWait` skip the mollify path entirely.
- `triggerTask` + `IdempotencyKeyConcern` wired to the above.
All behaviour gated by the master `TRIGGER_MOLLIFIER_ENABLED` switch;
off-state hot path is unchanged (the gate is not even consulted).
Stacked on the buffer extensions PR.
## Test plan
- [x] \`pnpm run typecheck --filter webapp\` passes
- [x] \`pnpm run test --filter webapp test/mollifierMollify.test.ts\`
passes
- [x] \`pnpm run test --filter webapp
test/mollifierIdempotencyClaim.test.ts\` passes
- [x] \`pnpm run test --filter webapp
test/mollifierReadFallback.test.ts\` passes
- [x] \`pnpm run test --filter webapp test/mollifierGate.test.ts\`
passes
- [x] \`pnpm run test --filter webapp test/engine/triggerTask.test.ts\`
passes
---
## Ship-gate follow-up fixes
- **Batch items bypass the mollifier gate** — fixes
`BatchTaskRunItem_taskRunId_fkey` FK violation on batch triggers when
the gate trips. End-state is a drainer-side `BatchTaskRunItem`
create-on-materialise; batch traffic passes through the gate until that
lands.
- **IdempotencyKeyConcern honours buffered-run TTL on expiry** —
buffered path now clears expired idempotency claims (read-side) and
resets the buffer's `mollifier:idempotency:*` SETNX binding (write-side)
so a re-trigger past the customer's TTL lands as a fresh run instead of
echoing the stale buffered runId.
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
## Summary
Buffer-side data layer used by the rest of the mollifier phase-3 stack.
- `buffer.ts` gains entry inspection (`getEntry`), idempotency lookup
(`lookupIdempotency`), in-place snapshot mutation (`mutateSnapshot`),
and dwell tracking. All atomic via Lua.
- `mollifierSnapshot.server.ts`: shared `MollifierSnapshot` type plus
(de)serialise helpers.
- Drops the entry-TTL config and its env var. The drainer is the
recovery mechanism; an entry that survives the drainer should surface as
a stale-sweep alert, not silently TTL away.
Adds methods to the buffer interface; nothing consumes them yet.
Subsequent PRs in the stack wire trigger-time mollify, read-fallback,
and mutation paths against this surface.
## Test plan
- [x] \`pnpm run typecheck --filter webapp\` passes
- [x] \`pnpm run test --filter @trigger.dev/redis-worker
packages/redis-worker/src/mollifier/buffer.test.ts\` passes
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
## Summary
1 improvement, 2 bug fixes.
## Improvements
- Retry `TASK_MIDDLEWARE_ERROR` under the task's retry policy instead of
failing the run on the first attempt. The error was already classified
as retryable by `shouldRetryError`, but `shouldLookupRetrySettings` did
not include it, so the retry flow fell through to `fail_run`. Fixes
#3231. ([#3676](https://github.com/triggerdotdev/trigger.dev/pull/3676))
## Bug fixes
- Fix `TypeError` in `unflattenAttributes` when the input attribute map
contains conflicting dotted key paths (e.g. both `a.b` set to a scalar
and `a.b.c` set to a value). The path-walk loop now applies
last-write-wins when a prior key wrote a primitive, null, or array at an
intermediate slot, matching the existing precedent in
`AttributeFlattener.addAttribute`. Callers no longer crash when handed
malformed external attribute inputs.
([#3762](https://github.com/triggerdotdev/trigger.dev/pull/3762))
- Fix external trace context leaking across runs on warm-started workers
with `processKeepAlive` enabled. Every subsequent run's attempt span was
being exported with the first run's `traceId` and `parentSpanId`,
breaking causal-chain navigation in external APM tools. Runs without an
external trace context are unaffected.
([#3768](https://github.com/triggerdotdev/trigger.dev/pull/3768))
<details>
<summary>Raw changeset output</summary>
⚠️⚠️⚠️⚠️⚠️⚠️
`main` is currently in **pre mode** so this branch has prereleases
rather than normal releases. If you want to exit prereleases, run
`changeset pre exit` on `main`.
⚠️⚠️⚠️⚠️⚠️⚠️
# Releases
## @trigger.dev/build@4.5.0-rc.3
### Patch Changes
- Updated dependencies:
- `@trigger.dev/core@4.5.0-rc.3`
## trigger.dev@4.5.0-rc.3
### Patch Changes
- Updated dependencies:
- `@trigger.dev/core@4.5.0-rc.3`
- `@trigger.dev/build@4.5.0-rc.3`
- `@trigger.dev/schema-to-json@4.5.0-rc.3`
## @trigger.dev/core@4.5.0-rc.3
### Patch Changes
- Retry `TASK_MIDDLEWARE_ERROR` under the task's retry policy instead of
failing the run on the first attempt. The error was already classified
as retryable by `shouldRetryError`, but `shouldLookupRetrySettings` did
not include it, so the retry flow fell through to `fail_run`. Fixes
#3231. ([#3676](https://github.com/triggerdotdev/trigger.dev/pull/3676))
- Fix `TypeError` in `unflattenAttributes` when the input attribute map
contains conflicting dotted key paths (e.g. both `a.b` set to a scalar
and `a.b.c` set to a value). The path-walk loop now applies
last-write-wins when a prior key wrote a primitive, null, or array at an
intermediate slot, matching the existing precedent in
`AttributeFlattener.addAttribute`. Callers no longer crash when handed
malformed external attribute inputs.
([#3762](https://github.com/triggerdotdev/trigger.dev/pull/3762))
- Fix external trace context leaking across runs on warm-started workers
with `processKeepAlive` enabled. Every subsequent run's attempt span was
being exported with the first run's `traceId` and `parentSpanId`,
breaking causal-chain navigation in external APM tools. Runs without an
external trace context are unaffected.
([#3768](https://github.com/triggerdotdev/trigger.dev/pull/3768))
## @trigger.dev/plugins@4.5.0-rc.3
### Patch Changes
- Updated dependencies:
- `@trigger.dev/core@4.5.0-rc.3`
## @trigger.dev/python@4.5.0-rc.3
### Patch Changes
- Updated dependencies:
- `@trigger.dev/core@4.5.0-rc.3`
- `@trigger.dev/build@4.5.0-rc.3`
- `@trigger.dev/sdk@4.5.0-rc.3`
## @trigger.dev/react-hooks@4.5.0-rc.3
### Patch Changes
- Updated dependencies:
- `@trigger.dev/core@4.5.0-rc.3`
## @trigger.dev/redis-worker@4.5.0-rc.3
### Patch Changes
- Updated dependencies:
- `@trigger.dev/core@4.5.0-rc.3`
## @trigger.dev/rsc@4.5.0-rc.3
### Patch Changes
- Updated dependencies:
- `@trigger.dev/core@4.5.0-rc.3`
## @trigger.dev/schema-to-json@4.5.0-rc.3
### Patch Changes
- Updated dependencies:
- `@trigger.dev/core@4.5.0-rc.3`
## @trigger.dev/sdk@4.5.0-rc.3
### Patch Changes
- Updated dependencies:
- `@trigger.dev/core@4.5.0-rc.3`
</details>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
## Summary
Drops the `TaskRun_status_runtimeEnvironmentId_createdAt_id_idx` index
from the `TaskRun` table. After #3742 gated the legacy
`WAITING_FOR_DEPLOY` drain to V1-engine workers only, this index sees
zero scans on both writer and reader replicas. Removing it cuts index
maintenance on every `TaskRun` INSERT/UPDATE.
## Why
The index existed to support `WHERE status = X AND runtimeEnvironmentId
= Y` queries from `ExecuteTasksWaitingForDeployService`, which is
V1-only and no longer triggered on V2 deployments. A code grep across
`apps/webapp` and `internal-packages/run-engine` confirmed no V2
production query uses this access pattern — every other `status:` filter
on `TaskRun` is paired with `id`/`friendlyId`/`parentSpanId` and uses a
different index.
Dropping it also unlocks HOT updates on the dequeue path. The dequeue
`UPDATE` modifies `status` (`QUEUED` -> `DEQUEUED`), and `status` is the
leading column of this index — its presence blocked HOT eligibility for
every `TaskRun` UPDATE. With the index gone, dequeue UPDATEs can become
HOT, reducing WAL bytes and removing the B-tree page contention on this
index's right-edge leaves.
Uses `DROP INDEX CONCURRENTLY` to avoid blocking writes during the drop.
## Sequencing
Should only ship once #3742 has soaked long enough to confirm the index
is genuinely cold (24h+ of zero scans on `pg_stat_user_indexes`).
Make `POST /api/v1/deployments/:deploymentId/background-workers`
idempotent so client-side retries no longer collide on the
`BackgroundWorker` `(project, env, version)` unique index. Helps make
deployments more resilient against the class of indexing failures that
surfaces in the dashboard as "Indexing timed out", e.g. during transient
database issues.
On warm-started workers with `processKeepAlive` enabled, every run's
attempt span was exported with the first run's `traceId` and
`parentSpanId`. `ExternalSpanExporterWrapper` and
`ExternalLogRecordExporterWrapper` captured `externalTraceContext` at
`TracingSDK` construction, and the SDK is memoized for the worker's
lifetime - so the per-run reset of
`StandardTraceContextManager.traceContext` never reached the wrappers.
Reported by a customer running v4.4.x: 33 distinct runs on the same
host/pid showed up in their APM as siblings of one parent span.
Fix: the wrappers now read external context live from the trace context
manager per export. Runs without an external trace context fall through
to the unchanged `externalTraceId` fallback - no behaviour change for
them.
Regression test in
`packages/core/test/externalSpanExporterWrapper.test.ts` asserts that
reassigning the manager between exports produces correctly-parented
spans.
Retries `TASK_MIDDLEWARE_ERROR` under the task's retry policy.
`shouldRetryError` already classed it as retryable, but
`shouldLookupRetrySettings` did not, so the run fell through to
`fail_run` on attempt 1 instead of using the task's `retry` config.
Fixes#3231.
## Summary
Second class of poisoned-row failure in the runs replication path. PR
#3708 plugged lone UTF-16 surrogates; this one handles bare JSON integer
literals outside ClickHouse's `Int64`..`UInt64` range. Recovery stays
purely reactive — the existing `sanitizeRows` walker just gains an extra
branch, so the hot replication path pays nothing on healthy rows.
Fixes the still-firing customer-facing symptom from
[TRI-9755](https://linear.app/triggerdotdev/issue/TRI-9755):
`scan-social-profiles` runs continued to be stranded in `EXECUTING` on
the Tasks page after #3708 deployed. CloudWatch showed `Dropped batch —
ClickHouse JSON parse error but sanitizer found nothing to fix` firing
**8/8 times** since the previous deploy (zero successful sanitizations).
Root cause: upstream JS Number precision loss on a 21-digit Google Plus
ID (`117039831458782873093` → `117039831458782870000`) — the
precision-lossy value still serialises as a bare integer that exceeds
`UInt64.MAX`, which ClickHouse rejects with `INCORRECT_DATA`.
## How the bug ships
The customer task emits an output containing a Poshmark profile's
`spec_format`:
```json
{"key":"gp_id","proper_key":"Gp Id","value":117039831458782870000,"type":"int"}
```
That value is `1.17e20` — comfortably above `UInt64.MAX` (`1.84e19`) but
comfortably below `1e21`. `Number.prototype.toString` only switches to
exponential form at `|value| >= 1e21`, so `JSON.stringify` emits the
bare token `117039831458782870000` and the ClickHouse
`JSON(max_dynamic_paths)` column fails with:
```
Code: 117. DB::Exception: Cannot parse JSON object here: {…}: (while reading the value of key output): (at row 1)
: While executing ParallelParsingBlockInputFormat. (INCORRECT_DATA) (version 25.12.x)
```
Same error verbatim as prod. The same number quoted
(`"117039831458782870000"`) inserts fine — ClickHouse's dynamic JSON
column accepts a `String` subtype on the same path.
## What changed
`apps/webapp/app/v3/eventRepository/sanitizeRowsOnParseError.server.ts`:
- New private `isUnsafeJsonInteger(value)` helper — true iff `value` is
a finite integer-valued JS Number where `|value| < 1e21` (so
`JSON.stringify` emits integer form, not exponent) **and** `value` falls
outside `[Int64.MIN, UInt64.MAX]`.
- `sanitizeUnknownInPlace` gains a number-branch: when the predicate
holds, replace the Number with `String(value)`. The downstream JSON
column dynamic-types the path as String for that row — fine, since the
value was already precision-lossy upstream (no JS Number above 2^53 is
numerically meaningful anyway).
- Float-valued numbers, large floats (>= 1e21), NaN and Infinity are
left alone — `JSON.stringify` emits them with exponents or as `null`,
both of which ClickHouse accepts.
`apps/webapp/test/sanitizeRowsOnParseError.test.ts`: four new unit tests
+ an extension to `sanitizeRows` covering surrogate + integer fixes
counted together across rows. The unit suite now covers:
- Positive value above `UInt64.MAX` (`117039831458782870000` — the
actual prod value)
- Negative value below `Int64.MIN`
- Boundary values pass through (`42`, `Number.MAX_SAFE_INTEGER`, `2^63`)
- Non-integer numbers untouched (floats, `1e25`, NaN, Infinity)
- The actual `scan-social-profiles` nested shape — finds the offending
`gp_id` deep inside
`output.data.profiles[].spec_format[].platform_variables[].value`
`.server-changes/runs-replication-bigint-recovery.md` — release notes
entry.
## Why reactive, not pre-flight
`#prepareJson` runs millions of times per day on the replication hot
path. Walking every JSON tree to look for oversized integers would add
bounded-but-real CPU on every healthy row. `sanitizeRows` only fires
after a ClickHouse parse-error rejection, which is a few times a day
platform-wide. Extending it costs effectively zero on healthy traffic
and gains us recovery on the rare poisoned row.
## Verification
- Reproduced 1:1 in a throwaway Docker
`clickhouse/clickhouse-server:25.12.11.4` (closest available to the prod
`25.12.1.1579` build). Pre-sanitize JSON fails with the exact prod
error; post-sanitize JSON inserts cleanly and the row is readable with
`gp_id` stored as a String subtype.
- `pnpm --filter webapp exec vitest run
test/sanitizeRowsOnParseError.test.ts` — 22/22 passing (18 existing + 4
new).
- `pnpm run typecheck --filter webapp` — clean.
## Test plan
- [x] `pnpm run typecheck --filter webapp`
- [x] Unit tests pass against new + existing cases
- [x] End-to-end Docker ClickHouse repro confirms recovery
- [ ] Post-deploy: confirm `Sanitizing batch after ClickHouse JSON parse
error` warns fire instead of `Dropped batch …` errors when
`scan-social-profiles` outputs trip CH again
- [ ] Post-deploy: confirm `permanentlyDroppedBatches` counter stops
climbing in
`/stp/trigger-app-prod/ecs/replication/service-container/process-logs`
## What this does NOT do
- Doesn't backfill the ~120k+ existing stranded `EXECUTING` rows in
production. Same as #3708 — that needs a reconciliation/backfill sweep
(separate ticket — TRI-9755 fix#3).
- Doesn't address the upstream root cause (the customer task emitting a
JS-Number-precision-lossy big int). That's a customer-task concern; our
replication path needs to be robust to whatever shape arrives.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
## Summary
Stops the legacy V1 `WAITING_FOR_DEPLOY` drain from running on every V2
deployment promotion. The drain queries `TaskRun` by
`status='WAITING_FOR_DEPLOY'`, which only V1-engine runs ever have — V2
runs use `PENDING_VERSION` and are handled out of band. Calling the
drain on V2 promotes produced empty queries against the status index and
unnecessary reader-DB load.
## Fix
Two layers:
1. Gate the enqueue at the call site in `ChangeCurrentDeploymentService`
so it only fires when the deployment's worker is on engine V1.
2. Add a `LEGACY_RUN_ENGINE_WAITING_FOR_DEPLOY_DISABLED` env var
(default `0`). When set to `1`, the service returns immediately from
`call()` — neuters any jobs already sitting in the worker queue from
before the deploy lands.
V1 customers see no change; V2 promotes no longer trigger the drain.
Linear: [TRI-9864](https://linear.app/triggerdotdev/issue/TRI-9864)
(Urgent)
Production incident:
[TRI-9863](https://linear.app/triggerdotdev/issue/TRI-9863) (mitigated
by image revert in cloud#910)
## Bug
`apps/webapp/package.json` declares `"sideEffects": false`. PR #3333
(`71d98b4e`) replaced the previous real method-call retention idiom at
the two `sessionsReplicationInstance` import sites with:
```ts
import { sessionsReplicationInstance } from "...";
void sessionsReplicationInstance;
```
esbuild treats `void <identifier>;` as a pure expression statement under
`sideEffects: false` and **tree-shakes the entire import** — including
the `singleton(...)` call inside `sessionsReplicationInstance.server.ts`
which is the only thing that fires
`initializeSessionsReplicationInstance()`. The sessions→ClickHouse
logical replication worker never starts, the slot is unconsumed, lag
grows.
### How it manifested in production
cloud#907's image bump rolled the `SessionReplicationService` ECS task
on prod at 14:32 UTC. The new container's startup log emitted `🗃️ Runs
replication service enabled` but **not** `🗃️ Sessions replication
service enabled` or `🗃️ Sessions replication service started`.
CloudWatch `OldestReplicationSlotLag` grew at ~220 MB/min and the `High
replication lag` alarm fired at 14:37 UTC. Prod was reverted to the
previous image (cloud#910) to stop the bleed.
### Verification
`grep` of the built bundle `apps/webapp/build/index.js` (built from
`c0365d36`):
- **3** occurrences of `Runs replication` / `runsReplicationInstance`
strings ✅
- **0** occurrences of `Sessions replication` /
`sessionsReplicationInstance` / `SessionsReplicationService` ❌
The runs path survives tree-shaking because `adminWorker.server.ts` and
`admin.api.v1.runs-replication.*` routes have real method calls
(`.start()`, `.teardown()`, `.backfill()`) — observable uses the
tree-shaker must preserve. The sessions singleton has no real callers,
only the `void` no-ops, hence its complete elimination from the bundle.
## Fix
Replace `void sessionsReplicationInstance;` with an assignment to
`globalThis`, an unambiguous observable side effect the bundler cannot
eliminate:
```ts
(globalThis as Record<string, unknown>).__sessionsReplicationInstance =
sessionsReplicationInstance;
```
Applied at both call sites: `apps/webapp/app/entry.server.tsx` and
`apps/webapp/app/v3/services/adminWorker.server.ts`.
Surrounding comments updated to document the bundler interaction so the
next maintainer doesn't reintroduce `void`.
## Out of scope (follow-ups)
- **Robustness improvement**: change `apps/webapp/package.json` from
`"sideEffects": false` to an allowlist that includes
`*Instance.server.ts` files. Prevents the same regression shape via any
future `*Instance` singleton.
- **Build-time check**: add a `grep` post-build step in `publish.yml`
requiring `"Sessions replication"` to appear in
`apps/webapp/build/index.js`. Catches this exact regression at CI time.
## Test plan
- [x] `pnpm run typecheck --filter webapp` clean
- [ ] After merge + publish: confirm new image's
`SessionReplicationService` container logs `🗃️ Sessions replication
service enabled` and `🗃️ Sessions replication service started` at
startup
- [ ] After re-deploying to prod: confirm `OldestReplicationSlotLag`
stops growing and drains
## Summary
On a ClickHouse `Cannot parse JSON object` rejection,
`RunsReplicationService` now sanitizes lone UTF-16 surrogates across the
failing batch via the existing `sanitizeRows` helper and retries once.
If the sanitizer found nothing or the retry also fails, the batch is
dropped loudly with a counter increment, so the surrounding
`#insertWithRetry` layer doesn't spin three more times on a
deterministic failure. Non-parse errors propagate unchanged.
Mirrors the pattern from #3659 (for `ClickhouseEventRepository`) — same
root cause (lone UTF-16 surrogates in user-provided JSON), same recovery
shape, **reusing the same shared helpers** (`sanitizeRows`,
`isClickHouseJsonParseError`, `parseRowNumberFromError`).
Fixes the customer-facing symptom from
[TRI-9755](https://linear.app/triggerdotdev/issue/TRI-9755): a single
row's poisoned `output` JSON used to take down the
`COMPLETED_SUCCESSFULLY` UPDATE events for its 50+ batch-mates,
stranding them in `EXECUTING` in ClickHouse forever and inflating
"Running" counts on the Tasks page. Confirmed in production this is
ongoing — ~120k stale rows accumulated in a single 5-hour burst on
2026-05-18; smaller continuous leak before and after.
## What changed
`apps/webapp/app/services/runsReplicationService.server.ts`:
- Imports the three helpers from
`~/v3/eventRepository/sanitizeRowsOnParseError.server` (no duplication;
no move).
- New private `#insertWithJsonParseRecovery<T>(rows, doInsert,
contextLabel, attempt)` — generic over `TaskRunInsertArray[]` and
`PayloadInsertArray[]`, structurally identical to
`ClickhouseEventRepository.#insertWithJsonParseRecovery`. Try → on parse
error sanitize the whole batch (the `at row N` hint is logged but not
used to slice — semantics under `input_format_parallel_parsing` aren't
stable) → retry once → drop with loud log if sanitizer found nothing OR
retry still fails.
- `#insertTaskRunInserts` and `#insertPayloadInserts` extract a
`doInsert` closure and hand it to the wrapper. Existing error logging,
span recording, and `recordSpanError` are preserved inside the closure.
- New `private _permanentlyDroppedBatches = 0` counter with a public
getter, for ops dashboards and tests (matches the events-repo
convention). One shared counter for both insert sites — granularity
comes from the `contextLabel` (`task_runs_v2` /
`raw_task_runs_payload_v1`) on every log line.
`.server-changes/runs-replication-utf16-recovery.md` — release notes
entry.
## Why no new tests
The shared helpers already have full unit + real-ClickHouse contract
coverage from #3659
(`apps/webapp/test/sanitizeRowsOnParseError.test.ts`,
`apps/webapp/test/otlpUtf16Sanitization.integration.test.ts`). The new
wrapper is a line-for-line structural port. Adding a parallel
integration test would require synthesizing bad data that *escapes* the
preemptive `detectBadJsonStrings` check in `#prepareJson` but still
trips ClickHouse — non-trivial without hand-crafted fixtures and
wouldn't cover any new logic.
## What this does NOT do
- Doesn't touch the ~120k existing stale `EXECUTING` rows in production.
That needs a reconciliation/backfill sweep (separate ticket — TRI-9755
fix#3).
- Doesn't sanitize the `error` column path
(`runsReplicationService.server.ts:932 const errorData = { data:
run.error };`). Reactive recovery will catch it if it ever poisons a
batch, but feeding it through `#prepareJson` like `output` is a cheap
follow-up.
## Test plan
- [x] `pnpm run typecheck --filter webapp` — clean
- [ ] Post-deploy: confirm `permanentlyDroppedBatches` counter stays at
zero (or near-zero) in
`/stp/trigger-app-prod/ecs/replication/service-container/process-logs`,
and watch for `Sanitizing batch after ClickHouse JSON parse error` warns
to confirm recovery is firing on real traffic
- [ ] Post-deploy: confirm the rate of new
"EXECUTING-but-actually-COMPLETED" zombies in ClickHouse flattens
(current rate ≈ tens-to-hundreds per hour platform-wide)
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
## Summary
Updates the AI chat docs to match the slim-wire + field-level merge
behavior shipped in #3719 and the precise `.in/append` cap +
CORS-readable 413 shipped in #3720. No behavior changes here — code is
correct in `main`; the docs were lagging on three patterns customers
copy out of the page.
## What changed
- **`hydrateMessages` examples upsert by id** (in `lifecycle-hooks.mdx`,
`patterns/database-persistence.mdx`, and
`patterns/persistence-and-replay.mdx`). The previous
`stored.push(newMsg)` pattern duplicated the assistant id on HITL
continuations and caused the LLM to receive a tool call with no
`arguments`. The new examples include the rationale inline.
- **`onValidateMessages` example filters to user messages**
(`lifecycle-hooks.mdx`). The previous example called
`validateUIMessages({ messages, tools })` directly, which now throws on
HITL slim wires (the AI SDK schema requires `input` on resolved tool
parts). New example shows the filter pattern, with a Warning callout
explaining why.
- **Merge contract description updated** (`lifecycle-hooks.mdx`). The
old wording said incoming messages are "auto-merged" / "replaced"; the
new description explains the actual field-level overlay (state advances
only).
- **Approval-responded wire example slimmed** (`client-protocol.mdx`).
Shows the minimum shape the agent reads — `state` + `approval` (or
`output` / `errorText` for HITL). Notes that the built-in transports
ship this slim shape by default and that fuller shapes are still
accepted.
- **`/in/append` 413 row and FAQ updated** (`client-protocol.mdx`,
`patterns/trusted-edge-signals.mdx`). Reflects the new precise S2 cap
and the CORS-readable 413.
- **New changelog entry** at the top of `changelog.mdx` covering all of
the above.
The historical `## 512 KiB ceiling removed` entry further down the
changelog is left as-is (it's a snapshot of the prior transition), and
the v4.5 upgrade-guide section is skipped — the merge contract is
backwards compatible.
## Test plan
- Mintlify dev preview renders cleanly with no broken anchors
- Linked references resolve (`/ai-chat/lifecycle-hooks#hydratemessages`,
`/ai-chat/lifecycle-hooks#onvalidatemessages`,
`/ai-chat/patterns/database-persistence#alternative-hydratemessages`,
`/ai-chat/client-protocol#step-3-send-messages-stops-and-actions`,
`/ai-chat/patterns/large-payloads`)
## Summary
3 improvements, 1 bug fix.
## Improvements
- The per-turn merge now overlays the wire copy's tool-part state
advancement onto the agent's existing chain — `state` + the matching
resolution field (`output` / `errorText` / `approval`) come from the
wire, everything else (text, reasoning, tool `input`, provider metadata)
stays whatever the snapshot or `hydrateMessages` returned. Previously a
full-message replace overwrote those fields with whatever the client
shipped, so a slimmed wire copy landed a tool call with no `arguments`
on the next LLM call. Covers `output-available` / `output-error` (HITL
`addToolOutput`) and `approval-responded` / `output-denied` (approval
flow).
- `TriggerChatTransport.sendMessages` and `AgentChat.sendRaw` now slim
assistant messages that carry advanced tool parts. The wire payload is
just `{ id, role, parts: [<state + resolution field>] }` for
`submit-message` continuations; everything else passes through.
Reasoning blobs and full tool inputs no longer ride the wire on every
`addToolOutput` / `addToolApproveResponse`, so continuation payloads
stay well under the `.in/append` cap on long agent loops.
- Add `TriggerClient` for running multiple SDK clients side-by-side,
each with its own auth, preview branch, and baseURL. Useful when a
single process needs to trigger tasks or read runs across multiple
projects, environments, or preview branches without mutating shared
global state.
([#3683](https://github.com/triggerdotdev/trigger.dev/pull/3683))
## Bug fixes
- Fix `chat.agent` HITL continuations on reasoning-heavy turns. Two
changes that work together:
([#3719](https://github.com/triggerdotdev/trigger.dev/pull/3719))
<details>
<summary>Raw changeset output</summary>
⚠️⚠️⚠️⚠️⚠️⚠️
`main` is currently in **pre mode** so this branch has prereleases
rather than normal releases. If you want to exit prereleases, run
`changeset pre exit` on `main`.
⚠️⚠️⚠️⚠️⚠️⚠️
# Releases
## @trigger.dev/build@4.5.0-rc.2
### Patch Changes
- Updated dependencies:
- `@trigger.dev/core@4.5.0-rc.2`
## trigger.dev@4.5.0-rc.2
### Patch Changes
- Updated dependencies:
- `@trigger.dev/build@4.5.0-rc.2`
- `@trigger.dev/core@4.5.0-rc.2`
- `@trigger.dev/schema-to-json@4.5.0-rc.2`
## @trigger.dev/plugins@4.5.0-rc.2
### Patch Changes
- Updated dependencies:
- `@trigger.dev/core@4.5.0-rc.2`
## @trigger.dev/python@4.5.0-rc.2
### Patch Changes
- Updated dependencies:
- `@trigger.dev/sdk@4.5.0-rc.2`
- `@trigger.dev/build@4.5.0-rc.2`
- `@trigger.dev/core@4.5.0-rc.2`
## @trigger.dev/react-hooks@4.5.0-rc.2
### Patch Changes
- Updated dependencies:
- `@trigger.dev/core@4.5.0-rc.2`
## @trigger.dev/redis-worker@4.5.0-rc.2
### Patch Changes
- Updated dependencies:
- `@trigger.dev/core@4.5.0-rc.2`
## @trigger.dev/rsc@4.5.0-rc.2
### Patch Changes
- Updated dependencies:
- `@trigger.dev/core@4.5.0-rc.2`
## @trigger.dev/schema-to-json@4.5.0-rc.2
### Patch Changes
- Updated dependencies:
- `@trigger.dev/core@4.5.0-rc.2`
## @trigger.dev/sdk@4.5.0-rc.2
### Patch Changes
- Fix `chat.agent` HITL continuations on reasoning-heavy turns. Two
changes that work together:
([#3719](https://github.com/triggerdotdev/trigger.dev/pull/3719))
- The per-turn merge now overlays the wire copy's tool-part state
advancement onto the agent's existing chain — `state` + the matching
resolution field (`output` / `errorText` / `approval`) come from the
wire, everything else (text, reasoning, tool `input`, provider metadata)
stays whatever the snapshot or `hydrateMessages` returned. Previously a
full-message replace overwrote those fields with whatever the client
shipped, so a slimmed wire copy landed a tool call with no `arguments`
on the next LLM call. Covers `output-available` / `output-error` (HITL
`addToolOutput`) and `approval-responded` / `output-denied` (approval
flow).
- `TriggerChatTransport.sendMessages` and `AgentChat.sendRaw` now slim
assistant messages that carry advanced tool parts. The wire payload is
just `{ id, role, parts: [<state + resolution field>] }` for
`submit-message` continuations; everything else passes through.
Reasoning blobs and full tool inputs no longer ride the wire on every
`addToolOutput` / `addToolApproveResponse`, so continuation payloads
stay well under the `.in/append` cap on long agent loops.
Note: `onValidateMessages` receives the slim wire on HITL turns. If you
call `validateUIMessages` from `ai` against the full `messages` array it
will reject the slim assistant; filter to user messages (or skip on HITL
turns) — see the updated docstring on `onValidateMessages` for the
recommended pattern.
For `hydrateMessages` hooks that persist the chain, this release also
adds a small helper to the `@trigger.dev/sdk/ai` surface:
```ts
import { chat, upsertIncomingMessage } from "@trigger.dev/sdk/ai";
chat.agent({
hydrateMessages: async ({ chatId, trigger, incomingMessages }) => {
const record = await db.chat.findUnique({ where: { id: chatId } });
const stored = record?.messages ?? [];
if (upsertIncomingMessage(stored, { trigger, incomingMessages })) {
await db.chat.update({ where: { id: chatId }, data: { messages: stored }
});
}
return stored;
},
});
```
It pushes fresh user messages by id, no-ops on HITL continuations (the
incoming shares an id with the existing assistant — the runtime overlays
the new tool-state advance), and skips on non-`submit-message` triggers.
Returns `true` if it mutated `stored` so the caller knows whether to
persist.
Net effect: `chat.addToolOutput(...)` /
`chat.addToolApproveResponse(...)` on multi-step reasoning agents
(OpenAI Responses with `store: false`, Anthropic extended thinking,
etc.) no longer blows the cap and no longer corrupts the LLM input.
- Add `TriggerClient` for running multiple SDK clients side-by-side,
each with its own auth, preview branch, and baseURL. Useful when a
single process needs to trigger tasks or read runs across multiple
projects, environments, or preview branches without mutating shared
global state.
([#3683](https://github.com/triggerdotdev/trigger.dev/pull/3683))
```ts
import { TriggerClient } from "@trigger.dev/sdk";
const prod = new TriggerClient({ accessToken:
process.env.TRIGGER_PROD_KEY });
const preview = new TriggerClient({
accessToken: process.env.TRIGGER_PREVIEW_KEY,
previewBranch: "signup-flow",
});
await prod.tasks.trigger("send-email", payload);
await preview.runs.list({ status: ["COMPLETED"] });
```
- Updated dependencies:
- `@trigger.dev/core@4.5.0-rc.2`
## @trigger.dev/core@4.5.0-rc.2
</details>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
## Summary
Multi-step reasoning agents with HITL tools (OpenAI Responses with
`store: false`, Anthropic extended thinking, etc.) failed on
`chat.addToolOutput(...)` continuations — either the wire payload blew
the `.in/append` cap (reasoning blobs + tool inputs routinely > 512
KiB), or app-side slimming workarounds got overwritten server-side and
the next LLM call landed a tool call with no `arguments`. Both modes are
fixed.
## Design
The per-turn merge in `chat.agent` now overlays only the tool-part state
advances (`output-available` / `output-error` / `approval-responded` /
`output-denied`) from the wire copy onto the hydrated/snapshot chain.
Previously it replaced the entire message, which dropped `input`,
reasoning, and text from the LLM's view whenever the wire was slim.
In parallel, `TriggerChatTransport.sendMessages` and `AgentChat.sendRaw`
now slim the assistant message themselves on `submit-message`
continuations: ship `{ id, role, parts: [<resolved tool part only>] }`,
everything else reconstructed server-side from `hydrateMessages` or the
durable snapshot. Continuation payloads drop from 600 KiB – 1 MiB to ~1
KiB.
`references/ai-chat` `aiChatHydrated.hydrateMessages` now upserts by id
instead of pushing. With slim continuations, a blind push duplicates the
assistant id in the returned chain — the merge updates the first match,
the slim duplicate goes straight to `toModelMessages` with no `input`,
and the LLM 4xx's. This is the canonical pattern customers should mirror
in their own hydrate implementations.
## Test plan
- 11 new tests (slim helper unit + slim+merge integration for HITL,
approval, default no-hydrate branch)
- Full SDK suite: 239 tests pass across 19 files
- End-to-end sweep against `references/ai-chat`: 19 customer-side smoke
tests green; HITL wire bodies confirmed at ~1 KiB (was 600 KiB+); no
provider 4xx errors across OpenAI Responses or Anthropic
## Summary
Two improvements to session `.in/append`:
- Oversize-body 413 responses now carry CORS headers, so browser fetches
see a readable status instead of an opaque `TypeError: Failed to fetch`.
App-side retry-on-disconnect loops no longer spin forever on a
permanently-rejected payload.
- The per-record cap is now computed precisely against S2's actual
ceiling instead of the conservative 512 KiB floor. Legitimate ~600-900
KiB tool outputs (search results, file content) now succeed;
pathological all-quote content that would double under JSON escape still
rejects cleanly.
## Design
S2 enforces a per-record metered size of `8 + 2*H + Σ(header name +
value) + body ≤ 1048576` bytes. With no record headers (our case), the
budget reduces to `body ≤ 1048568`. Verified empirically against cloud
S2 — append succeeds at metered=1048576 and 422s at 1048577 with `record
must have metered size less than 1 MiB`.
The old `MAX_APPEND_BODY_BYTES = 512 KiB` was derived by assuming
worst-case JSON escape doubling (every byte becomes `\"` or `\\`),
giving `(1 MiB - overhead) / 2`. Safe, but rejects ~half the legitimate
input space.
The new flow:
1. Pre-cap the HTTP body at 1 MiB (DoS guard against reading arbitrary
garbage before we can compute the wrap).
2. After reading, `S2RealtimeStreams.#appendPartByName` computes
`Buffer.byteLength(JSON.stringify({data: part, id: partId}), "utf8") +
8` and throws `S2RecordTooLargeError` (a `ServiceValidationError` with
status 413) if it would exceed S2's ceiling. The route's existing error
branch maps the throw to a 413 with a descriptive message.
The 413 CORS fix is a single-line change in `apiBuilder.server.ts` —
`wrapResponse` was being skipped on the body-too-large branch; every
other error branch wraps; the 413 was the exception.
## Test plan
- Empirically verified against cloud S2 with a boundary scan across
`[1048568, 1048569, ..., 1048576]` and across H ∈ {0, 1×5 hdr bytes,
1×14 hdr bytes} — the formula matches exactly
- Browser-side fetch on a 700 KiB POST now resolves with a readable
`status: 413` (no `TypeError: Failed to fetch`)
- A 900 KiB ASCII tool output now passes (would have 413'd at 512 KiB
pre-fix)
## Bug
The `OrganizationDataStoresRegistry` singleton in
`apps/webapp/app/services/dataStores/organizationDataStoresRegistryInstance.server.ts`
was constructed with `$replica`. That client was then used by both the
polling read path *and* by `addDataStore` / `updateDataStore` /
`deleteDataStore` (and their backing `SecretStore.setSecret` upserts).
The write methods route through the read replica, which Postgres rejects
with **error code 25006**:
```
Invalid prisma.secretStore.upsert() invocation:
ConnectorError(ConnectorError { user_facing_error: None, kind: QueryError(PostgresError {
code: "25006",
message: "cannot execute INSERT in a read-only transaction",
...
}), transient: false })
```
User-visible symptom: the admin `/admin/data-stores` "Add data store"
form returns a 400 with this error wrapped, so no
`OrganizationDataStore` row can ever be created via the UI.
The read path (`loadFromDatabase` polling + `SecretStore.getSecret`) is
unaffected because `findMany` + secret read are read-only.
## Fix
Change the registry constructor to take both a writer and a replica:
```ts
constructor(writer: PrismaClient, replica: PrismaClient | PrismaReplicaClient)
```
- `loadFromDatabase()` keeps using `_replica` (and its
`SecretStore.getSecret` calls) — these are background cache-fillers, not
on user-latency-sensitive paths.
- `addDataStore` / `updateDataStore` / `deleteDataStore` (and their
`SecretStore.setSecret` / `deleteSecret` calls) now use `_writer`.
`organizationDataStoresRegistryInstance.server.ts` passes `(prisma,
$replica)` from `~/db.server`. Test sites that constructed with
`(prisma)` now pass `(prisma, prisma)` — the testcontainer exposes a
single client, so the writer/replica split collapses to one connection.
## Files
-
`apps/webapp/app/services/dataStores/organizationDataStoresRegistry.server.ts`
— constructor + read/write split
-
`apps/webapp/app/services/dataStores/organizationDataStoresRegistryInstance.server.ts`
— pass `prisma` alongside `$replica`
- `apps/webapp/test/organizationDataStoresRegistry.test.ts` — 14 call
sites bumped
- `apps/webapp/test/clickhouseFactory.test.ts` — 5 call sites bumped
## Test plan
- [x] Existing `organizationDataStoresRegistry.test.ts` +
`clickhouseFactory.test.ts` still pass (constructor sites updated;
behavior unchanged for tests).
- [ ] After deploy to test cloud, retry `/admin/data-stores` "Add data
store" form for the HIPAA org — should now succeed and the row should
appear.
- [ ] Verify the registry's polling reload picks up the new row within
`ORGANIZATION_DATA_STORES_RELOAD_INTERVAL_MS` (60s default) and the
factory starts routing to the org-scoped instance.
## Summary
When a background worker registers, the engine resolves runs that were
queued before the worker was ready (status `PENDING_VERSION`). That
lookup used to scan a Postgres status index on `TaskRun`. Move it to
ClickHouse: query candidate run ids from `task_runs_v2`, then refetch
the actual rows from Postgres by primary key with a `status =
'PENDING_VERSION'` guard for idempotency.
## Design
The lookup is a pluggable interface on the run engine
(`PendingVersionRunIdLookup`). The webapp wires a ClickHouse-backed
implementation through the org-scoped `clickhouseFactory` using a new
`"engine"` client type, configured by `RUN_ENGINE_CLICKHOUSE_*` env
vars. The URL falls back to `CLICKHOUSE_URL` when unset, so self-hosted
deployments don't need new config to keep working.
When the lookup returns no candidates, one bounded retry is scheduled
~5s later to cover ClickHouse replication lag against `task_runs_v2`.
The Postgres status guard on both the candidate refetch and the inner
`updateMany` prevents double-promotion when a retry races with a
concurrent deploy.
Tests cover three existing PENDING_VERSION cases via a small
Postgres-backed test adapter; new ClickHouse-backed integration tests
will follow.
## Summary
Drops the unused composite Postgres index
`TaskRun_scheduleId_createdAt_idx`. The schedule list view reads from
ClickHouse, so this index served no Prisma query while still being
maintained on every `TaskRun` INSERT/UPDATE. Removing it reduces write
amplification on the primary database.
Sibling to the prior drop of `TaskRun_scheduleId_idx` and the earlier
removal of the `TaskRun.scheduleId` foreign key — all stemming from
migrating schedule-aware reads to ClickHouse.
## Verification
- Sampled `pg_stat_user_indexes` for `TaskRun` over multiple hours —
zero scans against this index.
- Grepped the codebase for any Prisma query filtering
`TaskRun.scheduleId` — none found. All schedule-aware listing routes
through `clickhouseRunsRepository`.
Make the Express server's `keepAliveTimeout` configurable via
`HTTP_KEEPALIVE_TIMEOUT_MS`. Default preserved at 65000 ms — no behavior
change if unset.
Workloads bundled with CLI versions before v4.4.4 use a strict zod enum
for `checkpoint.type` that only allows DOCKER and KUBERNETES. When a
customer's runs are routed via the compute path, those old runners
receive `type: "COMPUTE"` on `/snapshots/since/...` and `/dequeue`
responses and fail validation - blocking silent migration of existing
deployments.
The workload never reads the field - only validates the shape. Rewriting
COMPUTE -> KUBERNETES on the way out lets older runners keep parsing
while the database and internal services keep the real value. Limited to
the two workload-facing endpoints whose response includes a checkpoint;
`/continue`, `/attempts/start`, `/attempts/complete` all return shapes
without one.
Followup to #3114.
## Summary
Stamp every Sentry event with the signed-in user and the tenant (org /
project / env) the request belongs to, so "Users Impacted" counts
distinct humans and events become filterable per tenant.
**Design after review (current):**
- `user.id = real user cuid` (from `requireUser`). "Users Impacted"
counts humans, not tenants.
- Tenant context (org / project / env slugs, IDs, env type) moves
entirely onto tags: `org_slug`, `project_slug`, `env_slug`, `org_id`,
`project_id`, `project_ref`, `environment_id`, `env_type`, plus
`impersonating` when set.
- Backed by an `AsyncLocalStorage` scope established at the HTTP entry.
Each entry point fills what it knows; loaders enrich the same scope with
what they already have.
**Zero new database queries.** The middleware does a regex match only.
Dashboard loaders that already query Prisma gain a couple of extra
selected columns; nothing new round-trips.
## How it's wired
- **Express middleware (`tenantContextResolver.server.ts`)** — parses
the URL with a regex and always opens an ALS scope. Populates whatever
subset of slugs is present: `/orgs/:o` → just `orgSlug`;
`/orgs/:o/projects/:p` adds `projectSlug`; the full triple adds
`envSlug`. Non-tenant paths get an empty scope so loaders can still
enrich.
- **`_app/route.tsx`** — already calls `requireUser`. Adds
`tenantContext.enrich({ userId: user.id })` for every authenticated
dashboard request. No new query.
- **Env layout loader (`_app.orgs.$o.projects.$p.env.$e/route.tsx`)** —
its existing `prisma.project.findFirst` gains two columns in `select`
(`externalRef`, `organization.id`). After it picks an env, calls
`tenantContext.enrich({ orgId, projectId, projectRef, envId, envType
})`. Same query, +2 columns.
- **API path (`apiBuilder.server.ts`)** — wraps every handler in
`tenantContext.run(tenantContextFromAuthEnvironment(authenticationResult.environment),
…)`. The mapper pulls `userId` from `env.orgMember?.userId` (already
selected by `authIncludeBase` — no schema change). Covers
`createLoaderApiRoute`, `createActionApiRoute`, and
`createMultiMethodApiRoute`.
- **Event processor (`sentryTenantContext.server.ts`)** — registered in
`entry.server.tsx` so it lives in the Remix bundle and shares the same
`tenantContext` ALS instance as the middleware and loaders. Stamps
whatever's present; nothing forced.
## Example events from local verification
| URL | `user.id` | Tags |
|-----|-----------|------|
| `/orgs/:o/projects/:p/env/:e/...` | real user cuid | `org_slug`,
`project_slug`, `env_slug`, `org_id`, `project_id`, `project_ref`,
`environment_id`, `env_type` |
| `/orgs/:o/settings` (non-env-scoped) | real user cuid | `org_slug`
only |
| API request with `orgMember` | `orgMember.userId` | full tenant set |
| API request without `orgMember` | (unset) | full tenant set |
## Trade-offs
1. On env-scoped pages, errors that fire before the env layout loader's
enrich callback runs get slugs + `user.id` but not the tenant IDs /
`env_type`. Realistic errors deep in async work get the full set. (Same
race as before, narrower window now that slugs/`user.id` are populated
up-front by the middleware and `_app` enrich.)
2. API requests where the environment has no `orgMember` get tenant tags
but no `user.id`. Those events still show in the issue but don't
contribute to "Users Impacted".
## Out of scope (deferred)
Background workers (`redis-worker`, `schedule-engine`) and socket
handlers. Those entry points don't set `tenantContext.run` yet — their
events ship without tenant attribution until each is wired in a
follow-up.
## Tests
31 unit tests across 4 files. New tests notably cover:
- `parseTenantPath`: org-only, org+project, and full-triple URL
variants.
- `tenantContext.enrich`: in-place patch, no-op outside `run()`,
concurrent-scope isolation, empty-scope + enrich pattern (for non-tenant
pages).
- `tenantContextFromAuthEnvironment`: with and without `orgMember` —
verifies the API path's `user.id` mapping.
- `addTenantContextToEvent`: empty scope, userId-only, slugs-only, full
enrichment, conditional tag emission, preservation of prior `event.user`
fields.
## Test plan
- [ ] `pnpm run typecheck --filter webapp`
- [ ] `pnpm run test --filter webapp -- test/tenantContext.test.ts
test/sentryTenantContext.test.ts test/tenantContextResolver.test.ts
test/tenantContextFromAuthEnvironment.test.ts`
- [ ] Local manual: with `SENTRY_DSN` set, hit a dashboard URL and an
API route, confirm the captured events carry `user.id` + the expected
tag set in Sentry.
- [ ] After ship: confirm "Users Impacted" on a real Sentry issue
reflects distinct users (not tenants).
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
Added `OrganizationDataStore` which allows orgs to have data stored in
specific separate services.
For now this is just used for ClickHouse. When using ClickHouse we get a
client for the factory and pass in the org id.
Particular care has to be made with two hot-insert paths:
1. RunReplicationService
2. OTLPExporter
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: Claude <noreply@anthropic.com>
## Summary
`new TriggerClient({...})` exposes the management API (tasks, runs,
schedules, envvars, batch, queues, deployments, prompts, auth) as an
explicit instance with its own auth, preview branch, and baseURL.
Multiple clients can coexist in one process without mutating shared
global state — useful when a single service triggers across multiple
projects, environments, or preview branches.
```ts
import { TriggerClient } from "@trigger.dev/sdk";
const prod = new TriggerClient({ accessToken: process.env.TRIGGER_PROD_KEY });
const preview = new TriggerClient({
accessToken: process.env.TRIGGER_PREVIEW_KEY,
previewBranch: "signup-flow",
});
await prod.tasks.trigger("send-email", payload);
await preview.runs.list({ status: ["COMPLETED"] });
```
The existing global `configure()` API keeps working unchanged.
## Design
Instance methods enter an `AsyncLocalStorage`-backed scope (`sdkScope`)
before delegating to the existing module-level functions. The four
"pollution" points that previously read globals now consult the scope
first:
- `apiClientManager.{baseURL, accessToken, branchName}` and
`clientOrThrow` — identity fields are scope-only when scoped; `baseURL`
still falls back to `TRIGGER_API_URL` because plumbing (where the API
lives) is not identity.
- `taskContext.{ctx, worker, isWarmStart, isInsideTask}` — masked inside
an isolated scope so a `client.tasks.trigger(...)` from inside a task
doesn't leak the parent's `parentRunId` / `lockToVersion` / `isTest`
into a trigger that hits a different project.
- Inline `getEnvVar("TRIGGER_VERSION")` reads in `shared.ts` go through
a `scopedEnvVar` helper that returns `undefined` inside an isolated
scope.
The `TriggerClient` class itself is a thin wrapper that captures the
scope in its constructor and proxies each namespace method to enter that
scope before calling the existing impl. Generic inference (e.g.
`client.tasks.trigger<typeof t>(...)`) is preserved via `Pick<typeof ns,
keyof curatedSubset>` typings.
Two correctness fixes uncovered along the way are folded in:
- `apiClientManager.setGlobalAPIClientConfiguration` no longer silently
no-ops on the second call. `configure()` now actually overrides as users
expect (this is the root cause behind some "I changed the config but
nothing happened" reports).
- `apiClientManager.runWithConfig` (and therefore `auth.withAuth`) is
now backed by `sdkScope.withScope` instead of "mutate the global and
restore in finally". Two parallel `withAuth` calls with different
configs no longer stomp each other.
Surface curation: instance namespaces drop methods that don't make sense
per-instance — `batch.*AndWait` (runtime-dependent), `schedules.task` /
`schedules.timezones` (definition-time / stateless), `prompts.define`
(definition-time), `auth.configure` / `auth.withAuth` (global-only).
## Test plan
- [x] 9 runtime unit tests in `triggerClient.test.ts` cover: required
accessToken, instance auth + branch headers, no env fallback for
identity fields, no leakage between global and instance, four parallel
calls across two clients stay isolated, taskContext masking +
`inheritContext: true` override, `configure()` second-call override,
parallel `auth.withAuth` isolation.
- [x] 10 type-level assertions in `triggerClient.types.test.ts` using
`expectTypeOf` + `@ts-expect-error` lock in generic inference, return
type passthrough, overload preservation, and curated-surface drift.
- [x] Full SDK suite (219 tests) and core suite (530 tests) pass.
- [x] Webapp typecheck clean.
- [x] End-to-end smoke test against local webapp and a
freshly-provisioned cloud project — six concurrent multi-client triggers
all returned 200 with run IDs, headers per-client as expected.
- [ ] Reviewer: run `references/multi-client` per its `README.md` to
reproduce the smoke test locally.
## Try it
`references/multi-client` is a new reference workspace that exercises
this end-to-end:
- `src/trigger/echo.ts` — trivial target task
- `src/trigger/fanOut.ts` — opens two `TriggerClient`s from inside a
task, fires `echo` through each in parallel
- `src/external/main.ts` — external Node script with two clients
triggering `echo` sequentially and concurrently; logs every outgoing
request's `authorization` + `x-trigger-branch`
- `src/external/isolation.ts` — interleaves global `configure()` and an
instance call, asserts the captured fetch sequence shows no leakage
either way
Sibling to the weekly summary, focused on critical alerts only. Pings
Slack daily while any critical alerts are open; skips the post entirely
when zero, so no daily "all clear" noise.
- Daily 08:00 UTC cron + `workflow_dispatch` with `severity` input
(default `critical`, override to `high`/`medium`/`low` for manual
checks)
- Reuses the existing `dependabot-summary` environment (token, channel,
bot)
- Alerts link at the end is severity-filtered
## Summary
Two docs edits that close a footgun customers persisting transport state
can hit. Clearing `lastEventId` on `chat.endRun()` looks intuitive — the
Run ended, the cursor must be stale — but the cursor is sessionId-keyed,
not runId-keyed. Clearing it forces the next `sendMessages` to subscribe
from `seq_num=0`, which may hit the prior turn's still-durable
`turn-complete` record and close the SSE empty before the new Run's
chunks arrive.
Spells out the invariant in the frontend transport persistence table and
adds a Warning in the `chat.endRun()` reference.
## Test plan
- [x] Mintlify preview renders
- [x] No callout stacking
## Summary
Three post-merge fixes for the AI Agents docs (#3226), all caught by
review after merge.
## Fixes
- **`onTurnComplete` examples now use `db.$transaction`** — both the
Database persistence "Complete example" and the Lifecycle hooks
reference example were doing two separate `await` calls
(`db.chat.update` then `db.chatSession.upsert`). That's the exact
non-atomic pattern the warning earlier on the persistence page calls out
as ❌: a refresh between the two writes reads a stale `lastEventId` and
duplicates the assistant message on resume. Both examples now use the
recommended atomic form.
- **Background injection self-review prose aligned with the code** — the
prose said "gpt-4o-mini" but the example above it had been swapped to
`claude-haiku-4-5`. The Anthropic-sweep script only touched code blocks;
this prose line wasn't picked up.
## Test plan
- [x] Both updated examples use `db.$transaction([...])`
- [x] Prose matches the model used in the code block
- [ ] Mintlify deployment passes
## Summary
Lands the full AI Agents documentation surface alongside the v4.5
release candidate of `@trigger.dev/sdk`. Covers `chat.agent` end to end
— defining agents, lifecycle hooks, the frontend transport, sub-agents,
recovery from cancel/crash/OOM, AI Prompts integration — and the
Sessions primitive that backs it.
## Coverage
- **Conceptual**: Overview, Quick Start, How it works.
- **Building agents**: Backend (`chat.agent` / `chat.createSession` /
raw primitives), Lifecycle hooks, Frontend transport, Server-side
`AgentChat`, Sessions reference, `chat.local` state primitive,
TypeScript types.
- **Features**: AI Prompts integration, Fast starts (Preload + Head
Start), Compaction, Pending Messages (steering), Background Injection
(`chat.inject` + `chat.defer`), Actions (undo / regenerate / edit),
Error handling.
- **Patterns (13)**: Sub-agents, Branching conversations, Code sandbox,
Database persistence, Persistence and replay, HITL, Tool result
auditing, Large payloads, Agent skills, OOM resilience, Recovery boot,
Trusted edge signals, Version upgrades.
- **Reference**: API Reference, Client Protocol (wire format), Testing
harness (`mockChatAgent`), MCP server tools, Upgrade guide, Changelog.
## Structure changes
- Top-level nav: AI → **Agents**, with sub-groups for *Building agents /
Features / Patterns / Reference*.
- New RC banner snippet on every page links to the supported AI SDK
versions table on the API Reference.
- All examples use Anthropic with `stopWhen: stepCountIs(15)`.
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
## Summary
2 bug fixes.
## Bug fixes
- Fix `chat.agent` skills silently missing in `trigger dev` for projects
whose task files read `process.env` at module top level (e.g. a
third-party SDK client initialized at import). Skill folders now bundle
into `.trigger/skills/` reliably regardless of which env vars are set
when the CLI launches.
([#3690](https://github.com/triggerdotdev/trigger.dev/pull/3690))
- Fix `COULD_NOT_FIND_EXECUTOR` when a task's definition is loaded via
`await import(...)` from inside another task's `run()`. The runtime
workers now register such tasks with a sentinel file context, and the
catalog logs a one-time warning per task id.
([#3688](https://github.com/triggerdotdev/trigger.dev/pull/3688))
<details>
<summary>Raw changeset output</summary>
⚠️⚠️⚠️⚠️⚠️⚠️
`main` is currently in **pre mode** so this branch has prereleases
rather than normal releases. If you want to exit prereleases, run
`changeset pre exit` on `main`.
⚠️⚠️⚠️⚠️⚠️⚠️
# Releases
## @trigger.dev/build@4.5.0-rc.1
### Patch Changes
- Updated dependencies:
- `@trigger.dev/core@4.5.0-rc.1`
## trigger.dev@4.5.0-rc.1
### Patch Changes
- Fix `chat.agent` skills silently missing in `trigger dev` for projects
whose task files read `process.env` at module top level (e.g. a
third-party SDK client initialized at import). Skill folders now bundle
into `.trigger/skills/` reliably regardless of which env vars are set
when the CLI launches.
([#3690](https://github.com/triggerdotdev/trigger.dev/pull/3690))
- Fix `COULD_NOT_FIND_EXECUTOR` when a task's definition is loaded via
`await import(...)` from inside another task's `run()`. The runtime
workers now register such tasks with a sentinel file context, and the
catalog logs a one-time warning per task id.
([#3688](https://github.com/triggerdotdev/trigger.dev/pull/3688))
- Updated dependencies:
- `@trigger.dev/core@4.5.0-rc.1`
- `@trigger.dev/build@4.5.0-rc.1`
- `@trigger.dev/schema-to-json@4.5.0-rc.1`
## @trigger.dev/core@4.5.0-rc.1
### Patch Changes
- Fix `COULD_NOT_FIND_EXECUTOR` when a task's definition is loaded via
`await import(...)` from inside another task's `run()`. The runtime
workers now register such tasks with a sentinel file context, and the
catalog logs a one-time warning per task id.
([#3688](https://github.com/triggerdotdev/trigger.dev/pull/3688))
## @trigger.dev/plugins@4.5.0-rc.1
### Patch Changes
- Updated dependencies:
- `@trigger.dev/core@4.5.0-rc.1`
## @trigger.dev/python@4.5.0-rc.1
### Patch Changes
- Updated dependencies:
- `@trigger.dev/core@4.5.0-rc.1`
- `@trigger.dev/build@4.5.0-rc.1`
- `@trigger.dev/sdk@4.5.0-rc.1`
## @trigger.dev/react-hooks@4.5.0-rc.1
### Patch Changes
- Updated dependencies:
- `@trigger.dev/core@4.5.0-rc.1`
## @trigger.dev/redis-worker@4.5.0-rc.1
### Patch Changes
- Updated dependencies:
- `@trigger.dev/core@4.5.0-rc.1`
## @trigger.dev/rsc@4.5.0-rc.1
### Patch Changes
- Updated dependencies:
- `@trigger.dev/core@4.5.0-rc.1`
## @trigger.dev/schema-to-json@4.5.0-rc.1
### Patch Changes
- Updated dependencies:
- `@trigger.dev/core@4.5.0-rc.1`
## @trigger.dev/sdk@4.5.0-rc.1
### Patch Changes
- Updated dependencies:
- `@trigger.dev/core@4.5.0-rc.1`
</details>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
## Summary
`trigger.dev dev` was silently dropping registered `chat.agent` skills
for any project whose task files read `process.env` at module top level
— e.g. a third-party SDK client initialized at import. The agent would
boot fine, but `skill.local()` failed at runtime with `ENOENT` because
the skill folder was never copied into `.trigger/skills/`.
## Design
The CLI ran two indexer passes in dev: the worker's own indexer (with
the full env it eventually executes tasks in), and a separate
skill-discovery indexer with only the CLI process's env. Top-level reads
of vars like `TRIGGER_API_URL` imported cleanly in the worker pass and
threw in the skill pass — the latter caught the error, warned, and
skipped skill copying. Failure was silent enough that `skill.local()`
only surfaced it at task runtime.
The skill registry is already part of the worker manifest. This PR drops
the duplicate pass and copies skill folders from that manifest after the
worker initializes. One indexer instead of two; a bad `SKILL.md` now
surfaces as a startup error instead of silently disappearing skills.
Deploy is unaffected — its skill discovery uses the project's
environment variables (fetched via the API, which fills in
`TRIGGER_API_URL` etc.), so the dev failure mode doesn't reach there.
## Test plan
- [x] New `references/agent-skills` reference project with
`skills.define` + a task that calls `skill.local()` and runs a bundled
script
- [x] On `main`, adding a top-level
`process.env.TRIGGER_API_URL!.includes(...)` read in any task file
reproduces the symptom: warning at dev startup, no `.trigger/skills/`
folder, `skill.local()` fails with ENOENT
- [x] On this branch, same project boots clean and `skill.local()` works
end-to-end
- [x] Deploy still works end-to-end with the new reference project
On a warm worker process, a task whose `task()` definition is loaded via
`await import(...)` from inside another task's `run()` could end up
permanently missing from the catalog: the `task()` call fired with no
`_currentFileContext` set, `registerTaskMetadata` silently returned, and
Node's ESM module cache then blocked the worker's setContext + re-import
recovery from ever firing the call again. Subsequent runs of that task
on the same warm process failed with `COULD_NOT_FIND_EXECUTOR` until the
process hit `maxExecutionsPerProcess` and exited.
All five of these had to coincide on the same worker for the bug to
surface:
1. `processKeepAlive` enabled (so catalog state survives across runs).
2. A `run()` function (or lifecycle hook) does `await import(...)`.
3. The import's transitive static graph reaches a `task()` /
`schemaTask()` call.
4. The task containing the dynamic import is the **first** task to run
on a given warm worker process — so the dropped `task()` calls fire on
this process for the first time, are silently dropped, and Node's module
cache locks the wrong outcome in.
5. A subsequent run for one of the dropped task ids lands on the same
warm worker before it recycles.
The runtime workers now set a sentinel file context (`<no-context>`)
around the `executor.execute(...)` call, so `task()` invocations firing
during a run register normally. The catalog detects the sentinel and
emits a one-time `console.warn` per task id so the pattern stays visible
without spamming. The indexer never sets this context, so deploy-time
behavior is unchanged.
Repro is `references/hello-world/src/trigger/dynamicImportRepro*.ts`.
Verified end-to-end against a deployed image with firestarter
warm-starts on: pre-fix saw `COULD_NOT_FIND_EXECUTOR` on children that
landed on the parent-poisoned worker; post-fix all 23/23 runs succeeded
and the warning surfaces in the parent's run trace.
## Summary
44 improvements, 1 bug fix.
## Improvements
- **AI Prompts** — define prompt templates as code alongside your tasks,
version them on deploy, and override the text or model from the
dashboard without redeploying. Prompts integrate with the Vercel AI SDK
via `toAISDKTelemetry()` (links every generation span back to the
prompt) and with `chat.agent` via `chat.prompt.set()` +
`chat.toStreamTextOptions()`.
([#3629](https://github.com/triggerdotdev/trigger.dev/pull/3629))
- **Code-defined, deploy-versioned templates** — define with
`prompts.define({ id, model, config, variables, content })`. Every
deploy creates a new version visible in the dashboard. Mustache-style
placeholders (`{{var}}`, `{{#cond}}...{{/cond}}`) with Zod / ArkType /
Valibot-typed variables.
- **Dashboard overrides** — change a prompt's text or model from the
dashboard without redeploying. Overrides take priority over the deployed
"current" version and are environment-scoped (dev / staging / production
independent).
- **Resolve API** — `prompt.resolve(vars, { version?, label? })` returns
the compiled `text`, resolved `model`, `version`, and labels. Standalone
`prompts.resolve<typeof handle>(slug, vars)` for cross-file resolution
with full type inference on slug and variable shape.
- **AI SDK integration** — spread `resolved.toAISDKTelemetry({ ...extra
})` into any `generateText` / `streamText` call and every generation
span links to the prompt in the dashboard alongside its input variables,
model, tokens, and cost.
- **`chat.agent` integration** — `chat.prompt.set(resolved)` stores the
resolved prompt run-scoped; `chat.toStreamTextOptions({ registry })`
pulls `system`, `model` (resolved via the AI SDK provider registry),
`temperature` / `maxTokens` / etc., and telemetry into a single spread
for `streamText`.
- **Management SDK** — `prompts.list()`, `prompts.versions(slug)`,
`prompts.promote(slug, version)`, `prompts.createOverride(slug, body)`,
`prompts.updateOverride(slug, body)`, `prompts.removeOverride(slug)`,
`prompts.reactivateOverride(slug, version)`.
- **Dashboard** — prompts list with per-prompt usage sparklines;
per-prompt detail with Template / Details / Versions / Generations /
Metrics tabs. AI generation spans get a custom inspector showing the
linked prompt's metadata, input variables, and template content
alongside model, tokens, cost, and the message thread.
- Adds `onBoot` to `chat.agent` — a lifecycle hook that fires once per
worker process picking up the chat. Runs for the initial run, preloaded
runs, AND reactive continuation runs (post-cancel, crash, `endRun`,
`requestUpgrade`, OOM retry), before any other hook. Use it to
initialize `chat.local`, open per-process resources, or re-hydrate state
from your DB on continuation — anywhere the SAME run picking up after
suspend/resume isn't enough.
([#3543](https://github.com/triggerdotdev/trigger.dev/pull/3543))
- **AI SDK `useChat` integration** — a custom
[`ChatTransport`](https://sdk.vercel.ai/docs/ai-sdk-ui/transport)
(`useTriggerChatTransport`) plugs straight into Vercel AI SDK's
`useChat` hook. Text streaming, tool calls, reasoning, and `data-*`
parts all work natively over Trigger.dev's realtime streams. No custom
API routes needed.
- **First-turn fast path (`chat.headStart`)** — opt-in handler that runs
the first turn's `streamText` step in your warm server process while the
agent run boots in parallel, cutting cold-start TTFC by roughly half
(measured 2801ms → 1218ms on `claude-sonnet-4-6`). The agent owns step
2+ (tool execution, persistence, hooks) so heavy deps stay where they
belong. Web Fetch handler works natively in Next.js, Hono, SvelteKit,
Remix, Workers, etc.; bridge to Express/Fastify/Koa via
`chat.toNodeListener`. New `@trigger.dev/sdk/chat-server` subpath.
- **Multi-turn durability via Sessions** — every chat is backed by a
durable Session that outlives any individual run. Conversations resume
across page refreshes, idle timeout, crashes, and deploys; `resume:
true` reconnects via `lastEventId` so clients only see new chunks.
`sessions.list` enumerates chats for inbox-style UIs.
- **Auto-accumulated history, delta-only wire** — the backend
accumulates the full conversation across turns; clients only ship the
new message each turn. Long chats never hit the 512 KiB body cap.
Register `hydrateMessages` to be the source of truth yourself.
- **Lifecycle hooks** — `onPreload`, `onChatStart`,
`onValidateMessages`, `hydrateMessages`, `onTurnStart`,
`onBeforeTurnComplete`, `onTurnComplete`, `onChatSuspend`,
`onChatResume` — for persistence, validation, and post-turn work.
- **Stop generation** — client-driven `transport.stopGeneration(chatId)`
aborts mid-stream; the run stays alive for the next message, partial
response is captured, and aborted parts (stuck `partial-call` tools,
in-progress reasoning) are auto-cleaned.
- **Tool approvals (HITL)** — tools with `needsApproval: true` pause
until the user approves or denies via `addToolApprovalResponse`. The
runtime reconciles the updated assistant message by ID and continues
`streamText`.
- **Steering and background injection** — `pendingMessages` injects user
messages between tool-call steps so users can steer the agent
mid-execution; `chat.inject()` + `chat.defer()` adds context from
background work (self-review, RAG, safety checks) between turns.
- **Actions** — non-turn frontend commands (undo, rollback, regenerate,
edit) sent via `transport.sendAction`. Fire `hydrateMessages` +
`onAction` only — no turn hooks, no `run()`. `onAction` can return a
`StreamTextResult` for a model response, or `void` for side-effect-only.
- **Typed state primitives** — `chat.local<T>` for per-run state
accessible from hooks, `run()`, tools, and subtasks (auto-serialized
through `ai.toolExecute`); `chat.store` for typed shared data between
agent and client; `chat.history` for reading and mutating the message
chain; `clientDataSchema` for typed `clientData` in every hook.
- **`chat.toStreamTextOptions()`** — one spread into `streamText` wires
up versioned system [Prompts](https://trigger.dev/docs/ai/prompts),
model resolution, telemetry metadata, compaction, steering, and
background injection.
- **Multi-tab coordination** — `multiTab: true` + `useMultiTabChat`
prevents duplicate sends and syncs state across browser tabs via
`BroadcastChannel`. Non-active tabs go read-only with live updates.
- **Network resilience** — built-in indefinite retry with bounded
backoff, reconnect on `online` / tab refocus / bfcache restore,
`Last-Event-ID` mid-stream resume. No app code needed.
- **Sessions** — a durable, run-aware stream channel keyed on a stable
`externalId`. A Session is the unit of state that owns a multi-run
conversation: messages flow through `.in`, responses through `.out`,
both survive run boundaries. Sessions back the new `chat.agent` runtime,
and you can build on them directly for any pattern that needs durable
bi-directional streaming across runs.
([#3542](https://github.com/triggerdotdev/trigger.dev/pull/3542))
- Add `ai.toolExecute(task)` so you can wire a Trigger subtask in as the
`execute` handler of an AI SDK `tool()` while defining `description` and
`inputSchema` yourself — useful when you want full control over the tool
surface and just need Trigger's subtask machinery for the body.
([#3546](https://github.com/triggerdotdev/trigger.dev/pull/3546))
- Type `chat.createStartSessionAction` against your chat agent so
`clientData` is typed end-to-end on the first turn:
([#3684](https://github.com/triggerdotdev/trigger.dev/pull/3684))
- Add `region` to the runs list / retrieve API: filter runs by region
(`runs.list({ region: "..." })` / `filter[region]=<masterQueue>`) and
read each run's executing region from the new `region` field on the
response.
([#3612](https://github.com/triggerdotdev/trigger.dev/pull/3612))
- Add `TRIGGER_BUILD_SKIP_REWRITE_TIMESTAMP=1` escape hatch for local
self-hosted builds whose buildx driver doesn't support
`rewrite-timestamp` alongside push (e.g. orbstack's default `docker`
driver).
([#3618](https://github.com/triggerdotdev/trigger.dev/pull/3618))
- Reject overlong `idempotencyKey` values at the API boundary so they no
longer trip an internal size limit on the underlying unique index and
surface as a generic 500. Inputs are capped at 2048 characters — well
above what `idempotencyKeys.create()` produces (a 64-character hash) and
above any realistic raw key. Applies to `tasks.trigger`,
`tasks.batchTrigger`, `batch.create` (Phase 1 streaming batches),
`wait.createToken`, `wait.forDuration`, and the input/session stream
waitpoint endpoints. Over-limit requests now return a structured 400
instead.
([#3560](https://github.com/triggerdotdev/trigger.dev/pull/3560))
- **AI SDK `useChat` integration** — a custom
[`ChatTransport`](https://sdk.vercel.ai/docs/ai-sdk-ui/transport)
(`useTriggerChatTransport`) plugs straight into Vercel AI SDK's
`useChat` hook. Text streaming, tool calls, reasoning, and `data-*`
parts all work natively over Trigger.dev's realtime streams. No custom
API routes needed.
- **First-turn fast path (`chat.headStart`)** — opt-in handler that runs
the first turn's `streamText` step in your warm server process while the
agent run boots in parallel, cutting cold-start TTFC by roughly half
(measured 2801ms → 1218ms on `claude-sonnet-4-6`). The agent owns step
2+ (tool execution, persistence, hooks) so heavy deps stay where they
belong. Web Fetch handler works natively in Next.js, Hono, SvelteKit,
Remix, Workers, etc.; bridge to Express/Fastify/Koa via
`chat.toNodeListener`. New `@trigger.dev/sdk/chat-server` subpath.
- **Multi-turn durability via Sessions** — every chat is backed by a
durable Session that outlives any individual run. Conversations resume
across page refreshes, idle timeout, crashes, and deploys; `resume:
true` reconnects via `lastEventId` so clients only see new chunks.
`sessions.list` enumerates chats for inbox-style UIs.
- **Auto-accumulated history, delta-only wire** — the backend
accumulates the full conversation across turns; clients only ship the
new message each turn. Long chats never hit the 512 KiB body cap.
Register `hydrateMessages` to be the source of truth yourself.
- **Lifecycle hooks** — `onPreload`, `onChatStart`,
`onValidateMessages`, `hydrateMessages`, `onTurnStart`,
`onBeforeTurnComplete`, `onTurnComplete`, `onChatSuspend`,
`onChatResume` — for persistence, validation, and post-turn work.
- **Stop generation** — client-driven `transport.stopGeneration(chatId)`
aborts mid-stream; the run stays alive for the next message, partial
response is captured, and aborted parts (stuck `partial-call` tools,
in-progress reasoning) are auto-cleaned.
- **Tool approvals (HITL)** — tools with `needsApproval: true` pause
until the user approves or denies via `addToolApprovalResponse`. The
runtime reconciles the updated assistant message by ID and continues
`streamText`.
- **Steering and background injection** — `pendingMessages` injects user
messages between tool-call steps so users can steer the agent
mid-execution; `chat.inject()` + `chat.defer()` adds context from
background work (self-review, RAG, safety checks) between turns.
- **Actions** — non-turn frontend commands (undo, rollback, regenerate,
edit) sent via `transport.sendAction`. Fire `hydrateMessages` +
`onAction` only — no turn hooks, no `run()`. `onAction` can return a
`StreamTextResult` for a model response, or `void` for side-effect-only.
- **Typed state primitives** — `chat.local<T>` for per-run state
accessible from hooks, `run()`, tools, and subtasks (auto-serialized
through `ai.toolExecute`); `chat.store` for typed shared data between
agent and client; `chat.history` for reading and mutating the message
chain; `clientDataSchema` for typed `clientData` in every hook.
- **`chat.toStreamTextOptions()`** — one spread into `streamText` wires
up versioned system [Prompts](https://trigger.dev/docs/ai/prompts),
model resolution, telemetry metadata, compaction, steering, and
background injection.
- **Multi-tab coordination** — `multiTab: true` + `useMultiTabChat`
prevents duplicate sends and syncs state across browser tabs via
`BroadcastChannel`. Non-active tabs go read-only with live updates.
- **Network resilience** — built-in indefinite retry with bounded
backoff, reconnect on `online` / tab refocus / bfcache restore,
`Last-Event-ID` mid-stream resume. No app code needed.
- Retry `TASK_PROCESS_SIGSEGV` task crashes under the user's retry
policy instead of failing the run on the first segfault. SIGSEGV in Node
tasks is frequently non-deterministic (native addon races, JIT/GC
interaction, near-OOM in native code, host issues), so retrying on a
fresh process often succeeds. The retry is gated by the task's existing
`retry` config + `maxAttempts` — same path `TASK_PROCESS_SIGTERM` and
uncaught exceptions already use — so tasks without a retry policy still
fail fast.
([#3552](https://github.com/triggerdotdev/trigger.dev/pull/3552))
- The public interfaces for a plugin system. Initially consolidated
authentication and authorization interfaces.
([#3499](https://github.com/triggerdotdev/trigger.dev/pull/3499))
- Add MollifierBuffer and MollifierDrainer primitives for trigger burst
smoothing.
([#3614](https://github.com/triggerdotdev/trigger.dev/pull/3614))
## Bug fixes
- Fix `LocalsKey<T>` type incompatibility across dual-package builds.
The phantom value-type brand no longer uses a module-level `unique
symbol`, so a single TypeScript compilation that resolves the type from
both the ESM and CJS outputs (which can happen under certain pnpm
hoisting layouts) no longer sees two structurally-incompatible variants
of the same type.
([#3626](https://github.com/triggerdotdev/trigger.dev/pull/3626))
<details>
<summary>Raw changeset output</summary>
⚠️⚠️⚠️⚠️⚠️⚠️
`main` is currently in **pre mode** so this branch has prereleases
rather than normal releases. If you want to exit prereleases, run
`changeset pre exit` on `main`.
⚠️⚠️⚠️⚠️⚠️⚠️
# Releases
## @trigger.dev/sdk@4.5.0-rc.0
### Minor Changes
- **AI Prompts** — define prompt templates as code alongside your tasks,
version them on deploy, and override the text or model from the
dashboard without redeploying. Prompts integrate with the Vercel AI SDK
via `toAISDKTelemetry()` (links every generation span back to the
prompt) and with `chat.agent` via `chat.prompt.set()` +
`chat.toStreamTextOptions()`.
([#3629](https://github.com/triggerdotdev/trigger.dev/pull/3629))
```ts
import { prompts } from "@trigger.dev/sdk";
import { generateText } from "ai";
import { openai } from "@ai-sdk/openai";
import { z } from "zod";
export const supportPrompt = prompts.define({
id: "customer-support",
model: "gpt-4o",
config: { temperature: 0.7 },
variables: z.object({
customerName: z.string(),
plan: z.string(),
issue: z.string(),
}),
content: `You are a support agent for Acme.
Customer: {{customerName}} ({{plan}} plan)
Issue: {{issue}}`,
});
const resolved = await supportPrompt.resolve({
customerName: "Alice",
plan: "Pro",
issue: "Can't access billing",
});
const result = await generateText({
model: openai(resolved.model ?? "gpt-4o"),
system: resolved.text,
prompt: "Can't access billing",
...resolved.toAISDKTelemetry(),
});
```
**What you get:**
- **Code-defined, deploy-versioned templates** — define with
`prompts.define({ id, model, config, variables, content })`. Every
deploy creates a new version visible in the dashboard. Mustache-style
placeholders (`{{var}}`, `{{#cond}}...{{/cond}}`) with Zod / ArkType /
Valibot-typed variables.
- **Dashboard overrides** — change a prompt's text or model from the
dashboard without redeploying. Overrides take priority over the deployed
"current" version and are environment-scoped (dev / staging / production
independent).
- **Resolve API** — `prompt.resolve(vars, { version?, label? })` returns
the compiled `text`, resolved `model`, `version`, and labels. Standalone
`prompts.resolve<typeof handle>(slug, vars)` for cross-file resolution
with full type inference on slug and variable shape.
- **AI SDK integration** — spread `resolved.toAISDKTelemetry({ ...extra
})` into any `generateText` / `streamText` call and every generation
span links to the prompt in the dashboard alongside its input variables,
model, tokens, and cost.
- **`chat.agent` integration** — `chat.prompt.set(resolved)` stores the
resolved prompt run-scoped; `chat.toStreamTextOptions({ registry })`
pulls `system`, `model` (resolved via the AI SDK provider registry),
`temperature` / `maxTokens` / etc., and telemetry into a single spread
for `streamText`.
- **Management SDK** — `prompts.list()`, `prompts.versions(slug)`,
`prompts.promote(slug, version)`, `prompts.createOverride(slug, body)`,
`prompts.updateOverride(slug, body)`, `prompts.removeOverride(slug)`,
`prompts.reactivateOverride(slug, version)`.
- **Dashboard** — prompts list with per-prompt usage sparklines;
per-prompt detail with Template / Details / Versions / Generations /
Metrics tabs. AI generation spans get a custom inspector showing the
linked prompt's metadata, input variables, and template content
alongside model, tokens, cost, and the message thread.
See [/docs/ai/prompts](https://trigger.dev/docs/ai/prompts) for the full
reference — template syntax, version resolution order, override
workflow, and type utilities (`PromptHandle`, `PromptIdentifier`,
`PromptVariables`).
- Adds `onBoot` to `chat.agent` — a lifecycle hook that fires once per
worker process picking up the chat. Runs for the initial run, preloaded
runs, AND reactive continuation runs (post-cancel, crash, `endRun`,
`requestUpgrade`, OOM retry), before any other hook. Use it to
initialize `chat.local`, open per-process resources, or re-hydrate state
from your DB on continuation — anywhere the SAME run picking up after
suspend/resume isn't enough.
([#3543](https://github.com/triggerdotdev/trigger.dev/pull/3543))
```ts
const userContext = chat.local<{ name: string; plan: string }>({ id:
"userContext" });
export const myChat = chat.agent({
id: "my-chat",
onBoot: async ({ clientData, continuation }) => {
const user = await db.user.findUnique({ where: { id: clientData.userId }
});
userContext.init({ name: user.name, plan: user.plan });
},
run: async ({ messages, signal }) =>
streamText({ model: openai("gpt-4o"), messages, abortSignal: signal }),
});
```
Use `onBoot` (not `onChatStart`) for state setup that must run every
time a worker picks up the chat — `onChatStart` fires once per chat and
won't run on continuation, leaving `chat.local` uninitialized when
`run()` tries to use it.
- **AI Agents** — run AI SDK chat completions as durable Trigger.dev
agents instead of fragile API routes. Define an agent in one function,
point `useChat` at it from React, and the conversation survives page
refreshes, network blips, and process restarts.
([#3543](https://github.com/triggerdotdev/trigger.dev/pull/3543))
```ts
import { chat } from "@trigger.dev/sdk/ai";
import { streamText } from "ai";
import { openai } from "@ai-sdk/openai";
export const myChat = chat.agent({
id: "my-chat",
run: async ({ messages, signal }) =>
streamText({ model: openai("gpt-4o"), messages, abortSignal: signal }),
});
```
```tsx
import { useChat } from "@ai-sdk/react";
import { useTriggerChatTransport } from "@trigger.dev/sdk/chat/react";
const transport = useTriggerChatTransport({ task: "my-chat",
accessToken, startSession });
const { messages, sendMessage } = useChat({ transport });
```
**What you get:**
- **AI SDK `useChat` integration** — a custom
[`ChatTransport`](https://sdk.vercel.ai/docs/ai-sdk-ui/transport)
(`useTriggerChatTransport`) plugs straight into Vercel AI SDK's
`useChat` hook. Text streaming, tool calls, reasoning, and `data-*`
parts all work natively over Trigger.dev's realtime streams. No custom
API routes needed.
- **First-turn fast path (`chat.headStart`)** — opt-in handler that runs
the first turn's `streamText` step in your warm server process while the
agent run boots in parallel, cutting cold-start TTFC by roughly half
(measured 2801ms → 1218ms on `claude-sonnet-4-6`). The agent owns step
2+ (tool execution, persistence, hooks) so heavy deps stay where they
belong. Web Fetch handler works natively in Next.js, Hono, SvelteKit,
Remix, Workers, etc.; bridge to Express/Fastify/Koa via
`chat.toNodeListener`. New `@trigger.dev/sdk/chat-server` subpath.
- **Multi-turn durability via Sessions** — every chat is backed by a
durable Session that outlives any individual run. Conversations resume
across page refreshes, idle timeout, crashes, and deploys; `resume:
true` reconnects via `lastEventId` so clients only see new chunks.
`sessions.list` enumerates chats for inbox-style UIs.
- **Auto-accumulated history, delta-only wire** — the backend
accumulates the full conversation across turns; clients only ship the
new message each turn. Long chats never hit the 512 KiB body cap.
Register `hydrateMessages` to be the source of truth yourself.
- **Lifecycle hooks** — `onPreload`, `onChatStart`,
`onValidateMessages`, `hydrateMessages`, `onTurnStart`,
`onBeforeTurnComplete`, `onTurnComplete`, `onChatSuspend`,
`onChatResume` — for persistence, validation, and post-turn work.
- **Stop generation** — client-driven `transport.stopGeneration(chatId)`
aborts mid-stream; the run stays alive for the next message, partial
response is captured, and aborted parts (stuck `partial-call` tools,
in-progress reasoning) are auto-cleaned.
- **Tool approvals (HITL)** — tools with `needsApproval: true` pause
until the user approves or denies via `addToolApprovalResponse`. The
runtime reconciles the updated assistant message by ID and continues
`streamText`.
- **Steering and background injection** — `pendingMessages` injects user
messages between tool-call steps so users can steer the agent
mid-execution; `chat.inject()` + `chat.defer()` adds context from
background work (self-review, RAG, safety checks) between turns.
- **Actions** — non-turn frontend commands (undo, rollback, regenerate,
edit) sent via `transport.sendAction`. Fire `hydrateMessages` +
`onAction` only — no turn hooks, no `run()`. `onAction` can return a
`StreamTextResult` for a model response, or `void` for side-effect-only.
- **Typed state primitives** — `chat.local<T>` for per-run state
accessible from hooks, `run()`, tools, and subtasks (auto-serialized
through `ai.toolExecute`); `chat.store` for typed shared data between
agent and client; `chat.history` for reading and mutating the message
chain; `clientDataSchema` for typed `clientData` in every hook.
- **`chat.toStreamTextOptions()`** — one spread into `streamText` wires
up versioned system [Prompts](https://trigger.dev/docs/ai/prompts),
model resolution, telemetry metadata, compaction, steering, and
background injection.
- **Multi-tab coordination** — `multiTab: true` + `useMultiTabChat`
prevents duplicate sends and syncs state across browser tabs via
`BroadcastChannel`. Non-active tabs go read-only with live updates.
- **Network resilience** — built-in indefinite retry with bounded
backoff, reconnect on `online` / tab refocus / bfcache restore,
`Last-Event-ID` mid-stream resume. No app code needed.
See [/docs/ai-chat](https://trigger.dev/docs/ai-chat/overview) for the
full surface — quick start, three backend approaches (`chat.agent`,
`chat.createSession`, raw task), persistence and code-sandbox patterns,
type-level guides, and API reference.
- Add read primitives to `chat.history` for HITL flows:
`getPendingToolCalls()`, `getResolvedToolCalls()`,
`extractNewToolResults(message)`, `getChain()`, and
`findMessage(messageId)`. These lift the accumulator-walking logic that
customers building human-in-the-loop tools were re-implementing into the
SDK. ([#3543](https://github.com/triggerdotdev/trigger.dev/pull/3543))
Use `getPendingToolCalls()` to gate fresh user turns while a tool call
is awaiting an answer. Use `extractNewToolResults(message)` to dedup
tool results when persisting to your own store — the helper returns only
the parts whose `toolCallId` is not already resolved on the chain.
```ts
const pending = chat.history.getPendingToolCalls();
if (pending.length > 0) {
// an addToolOutput is expected before a new user message
}
onTurnComplete: async ({ responseMessage }) => {
const newResults = chat.history.extractNewToolResults(responseMessage);
for (const r of newResults) {
await db.toolResults.upsert({ id: r.toolCallId, output: r.output,
errorText: r.errorText });
}
};
```
- **Sessions** — a durable, run-aware stream channel keyed on a stable
`externalId`. A Session is the unit of state that owns a multi-run
conversation: messages flow through `.in`, responses through `.out`,
both survive run boundaries. Sessions back the new `chat.agent` runtime,
and you can build on them directly for any pattern that needs durable
bi-directional streaming across runs.
([#3542](https://github.com/triggerdotdev/trigger.dev/pull/3542))
```ts
import { sessions, tasks } from "@trigger.dev/sdk";
// Trigger a task and subscribe to its session output in one call
const { runId, stream } = await tasks.triggerAndSubscribe("my-task",
payload, {
externalId: "user-456",
});
for await (const chunk of stream) {
// ...
}
// Enumerate existing sessions (powers inbox-style UIs without a
separate index)
for await (const s of sessions.list({ type: "chat.agent", tag:
"user:user-456" })) {
console.log(s.id, s.externalId, s.createdAt, s.closedAt);
}
```
See [/docs/ai-chat/overview](https://trigger.dev/docs/ai-chat/overview)
for the full surface — Sessions powers the durable, resumable chat
runtime described there.
### Patch Changes
- Add Agent Skills for `chat.agent`. Drop a folder with a `SKILL.md` and
any helper scripts/references next to your task code, register it with
`skills.define({ id, path })`, and the CLI bundles it into the deploy
image automatically — no `trigger.config.ts` changes. The agent gets a
one-line summary in its system prompt and discovers full instructions on
demand via `loadSkill`, with `bash` and `readFile` tools scoped
per-skill (path-traversal guards, output caps, abort-signal
propagation).
([#3543](https://github.com/triggerdotdev/trigger.dev/pull/3543))
```ts
const pdfSkill = skills.define({ id: "pdf-extract", path:
"./skills/pdf-extract" });
chat.skills.set([await pdfSkill.local()]);
```
Built on the [AI SDK cookbook
pattern](https://ai-sdk.dev/cookbook/guides/agent-skills) — portable
across providers. SDK + CLI only for now; dashboard-editable `SKILL.md`
text is on the roadmap.
- Add `ai.toolExecute(task)` so you can wire a Trigger subtask in as the
`execute` handler of an AI SDK `tool()` while defining `description` and
`inputSchema` yourself — useful when you want full control over the tool
surface and just need Trigger's subtask machinery for the body.
([#3546](https://github.com/triggerdotdev/trigger.dev/pull/3546))
```ts
const myTool = tool({
description: "...",
inputSchema: z.object({ ... }),
execute: ai.toolExecute(mySubtask),
});
```
`ai.tool(task)` (`toolFromTask`) keeps doing the all-in-one wrap and now
aligns its return type with AI SDK's `ToolSet`. Minimum `ai` peer raised
to `^6.0.116` to avoid cross-version `ToolSet` mismatches in monorepos.
- Stamp `gen_ai.conversation.id` (the chat id) on every span and metric
emitted from inside a `chat.task` or `chat.agent` run. Lets you filter
dashboard spans, runs, and metrics by the chat conversation that
produced them — independent of the run boundary, so multi-run chats
correlate cleanly. No code changes required on the user side.
([#3543](https://github.com/triggerdotdev/trigger.dev/pull/3543))
- Type `chat.createStartSessionAction` against your chat agent so
`clientData` is typed end-to-end on the first turn:
([#3684](https://github.com/triggerdotdev/trigger.dev/pull/3684))
```ts
import { chat } from "@trigger.dev/sdk/ai";
import type { myChat } from "@/trigger/chat";
export const startChatSession = chat.createStartSessionAction<typeof
myChat>("my-chat");
// In the browser, threaded from the transport's typed startSession
callback:
const transport = useTriggerChatTransport<typeof myChat>({
task: "my-chat",
startSession: ({ chatId, clientData }) => startChatSession({ chatId,
clientData }),
// ...
});
```
`ChatStartSessionParams` gains a typed `clientData` field — folded into
the first run's `payload.metadata` so `onPreload` / `onChatStart` see
the same shape per-turn `metadata` carries via the transport. The opaque
session-level `metadata` field is unchanged.
- Unit-test `chat.agent` definitions offline with `mockChatAgent` from
`@trigger.dev/sdk/ai/test`. Drives a real agent's turn loop in-process —
no network, no task runtime — so you can send messages, actions, and
stop signals via driver methods, inspect captured output chunks, and
verify hooks fire. Pairs with `MockLanguageModelV3` from `ai/test` for
model mocking. `setupLocals` lets you pre-seed `locals` (DB clients,
service stubs) before `run()` starts.
([#3543](https://github.com/triggerdotdev/trigger.dev/pull/3543))
The broader `runInMockTaskContext` harness it's built on lives at
`@trigger.dev/core/v3/test` — useful for unit-testing any task code, not
just chat.
- Add `region` to the runs list / retrieve API: filter runs by region
(`runs.list({ region: "..." })` / `filter[region]=<masterQueue>`) and
read each run's executing region from the new `region` field on the
response.
([#3612](https://github.com/triggerdotdev/trigger.dev/pull/3612))
- Updated dependencies:
- `@trigger.dev/core@4.5.0-rc.0`
## @trigger.dev/build@4.5.0-rc.0
### Patch Changes
- Add Agent Skills for `chat.agent`. Drop a folder with a `SKILL.md` and
any helper scripts/references next to your task code, register it with
`skills.define({ id, path })`, and the CLI bundles it into the deploy
image automatically — no `trigger.config.ts` changes. The agent gets a
one-line summary in its system prompt and discovers full instructions on
demand via `loadSkill`, with `bash` and `readFile` tools scoped
per-skill (path-traversal guards, output caps, abort-signal
propagation).
([#3543](https://github.com/triggerdotdev/trigger.dev/pull/3543))
```ts
const pdfSkill = skills.define({ id: "pdf-extract", path:
"./skills/pdf-extract" });
chat.skills.set([await pdfSkill.local()]);
```
Built on the [AI SDK cookbook
pattern](https://ai-sdk.dev/cookbook/guides/agent-skills) — portable
across providers. SDK + CLI only for now; dashboard-editable `SKILL.md`
text is on the roadmap.
- Updated dependencies:
- `@trigger.dev/core@4.5.0-rc.0`
## trigger.dev@4.5.0-rc.0
### Patch Changes
- Add Agent Skills for `chat.agent`. Drop a folder with a `SKILL.md` and
any helper scripts/references next to your task code, register it with
`skills.define({ id, path })`, and the CLI bundles it into the deploy
image automatically — no `trigger.config.ts` changes. The agent gets a
one-line summary in its system prompt and discovers full instructions on
demand via `loadSkill`, with `bash` and `readFile` tools scoped
per-skill (path-traversal guards, output caps, abort-signal
propagation).
([#3543](https://github.com/triggerdotdev/trigger.dev/pull/3543))
```ts
const pdfSkill = skills.define({ id: "pdf-extract", path:
"./skills/pdf-extract" });
chat.skills.set([await pdfSkill.local()]);
```
Built on the [AI SDK cookbook
pattern](https://ai-sdk.dev/cookbook/guides/agent-skills) — portable
across providers. SDK + CLI only for now; dashboard-editable `SKILL.md`
text is on the roadmap.
- Add `TRIGGER_BUILD_SKIP_REWRITE_TIMESTAMP=1` escape hatch for local
self-hosted builds whose buildx driver doesn't support
`rewrite-timestamp` alongside push (e.g. orbstack's default `docker`
driver).
([#3618](https://github.com/triggerdotdev/trigger.dev/pull/3618))
- The CLI MCP server's agent-chat tools (`start_agent_chat`,
`send_agent_message`, `close_agent_chat`) now run on the new Sessions
primitive, so AI assistants driving a `chat.agent` get the same
idempotent-by-`chatId`, durable-across-runs behavior the browser
transport gets. Required PAT scopes go from `write:inputStreams` to
`read:sessions` + `write:sessions`.
([#3546](https://github.com/triggerdotdev/trigger.dev/pull/3546))
- MCP `list_runs` tool: add a `region` filter input and surface each
run's executing region in the formatted summary.
([#3612](https://github.com/triggerdotdev/trigger.dev/pull/3612))
- Updated dependencies:
- `@trigger.dev/core@4.5.0-rc.0`
- `@trigger.dev/build@4.5.0-rc.0`
- `@trigger.dev/schema-to-json@4.5.0-rc.0`
## @trigger.dev/core@4.5.0-rc.0
### Patch Changes
- Add Agent Skills for `chat.agent`. Drop a folder with a `SKILL.md` and
any helper scripts/references next to your task code, register it with
`skills.define({ id, path })`, and the CLI bundles it into the deploy
image automatically — no `trigger.config.ts` changes. The agent gets a
one-line summary in its system prompt and discovers full instructions on
demand via `loadSkill`, with `bash` and `readFile` tools scoped
per-skill (path-traversal guards, output caps, abort-signal
propagation).
([#3543](https://github.com/triggerdotdev/trigger.dev/pull/3543))
```ts
const pdfSkill = skills.define({ id: "pdf-extract", path:
"./skills/pdf-extract" });
chat.skills.set([await pdfSkill.local()]);
```
Built on the [AI SDK cookbook
pattern](https://ai-sdk.dev/cookbook/guides/agent-skills) — portable
across providers. SDK + CLI only for now; dashboard-editable `SKILL.md`
text is on the roadmap.
- Reject overlong `idempotencyKey` values at the API boundary so they no
longer trip an internal size limit on the underlying unique index and
surface as a generic 500. Inputs are capped at 2048 characters — well
above what `idempotencyKeys.create()` produces (a 64-character hash) and
above any realistic raw key. Applies to `tasks.trigger`,
`tasks.batchTrigger`, `batch.create` (Phase 1 streaming batches),
`wait.createToken`, `wait.forDuration`, and the input/session stream
waitpoint endpoints. Over-limit requests now return a structured 400
instead.
([#3560](https://github.com/triggerdotdev/trigger.dev/pull/3560))
- **AI Agents** — run AI SDK chat completions as durable Trigger.dev
agents instead of fragile API routes. Define an agent in one function,
point `useChat` at it from React, and the conversation survives page
refreshes, network blips, and process restarts.
([#3543](https://github.com/triggerdotdev/trigger.dev/pull/3543))
```ts
import { chat } from "@trigger.dev/sdk/ai";
import { streamText } from "ai";
import { openai } from "@ai-sdk/openai";
export const myChat = chat.agent({
id: "my-chat",
run: async ({ messages, signal }) =>
streamText({ model: openai("gpt-4o"), messages, abortSignal: signal }),
});
```
```tsx
import { useChat } from "@ai-sdk/react";
import { useTriggerChatTransport } from "@trigger.dev/sdk/chat/react";
const transport = useTriggerChatTransport({ task: "my-chat",
accessToken, startSession });
const { messages, sendMessage } = useChat({ transport });
```
**What you get:**
- **AI SDK `useChat` integration** — a custom
[`ChatTransport`](https://sdk.vercel.ai/docs/ai-sdk-ui/transport)
(`useTriggerChatTransport`) plugs straight into Vercel AI SDK's
`useChat` hook. Text streaming, tool calls, reasoning, and `data-*`
parts all work natively over Trigger.dev's realtime streams. No custom
API routes needed.
- **First-turn fast path (`chat.headStart`)** — opt-in handler that runs
the first turn's `streamText` step in your warm server process while the
agent run boots in parallel, cutting cold-start TTFC by roughly half
(measured 2801ms → 1218ms on `claude-sonnet-4-6`). The agent owns step
2+ (tool execution, persistence, hooks) so heavy deps stay where they
belong. Web Fetch handler works natively in Next.js, Hono, SvelteKit,
Remix, Workers, etc.; bridge to Express/Fastify/Koa via
`chat.toNodeListener`. New `@trigger.dev/sdk/chat-server` subpath.
- **Multi-turn durability via Sessions** — every chat is backed by a
durable Session that outlives any individual run. Conversations resume
across page refreshes, idle timeout, crashes, and deploys; `resume:
true` reconnects via `lastEventId` so clients only see new chunks.
`sessions.list` enumerates chats for inbox-style UIs.
- **Auto-accumulated history, delta-only wire** — the backend
accumulates the full conversation across turns; clients only ship the
new message each turn. Long chats never hit the 512 KiB body cap.
Register `hydrateMessages` to be the source of truth yourself.
- **Lifecycle hooks** — `onPreload`, `onChatStart`,
`onValidateMessages`, `hydrateMessages`, `onTurnStart`,
`onBeforeTurnComplete`, `onTurnComplete`, `onChatSuspend`,
`onChatResume` — for persistence, validation, and post-turn work.
- **Stop generation** — client-driven `transport.stopGeneration(chatId)`
aborts mid-stream; the run stays alive for the next message, partial
response is captured, and aborted parts (stuck `partial-call` tools,
in-progress reasoning) are auto-cleaned.
- **Tool approvals (HITL)** — tools with `needsApproval: true` pause
until the user approves or denies via `addToolApprovalResponse`. The
runtime reconciles the updated assistant message by ID and continues
`streamText`.
- **Steering and background injection** — `pendingMessages` injects user
messages between tool-call steps so users can steer the agent
mid-execution; `chat.inject()` + `chat.defer()` adds context from
background work (self-review, RAG, safety checks) between turns.
- **Actions** — non-turn frontend commands (undo, rollback, regenerate,
edit) sent via `transport.sendAction`. Fire `hydrateMessages` +
`onAction` only — no turn hooks, no `run()`. `onAction` can return a
`StreamTextResult` for a model response, or `void` for side-effect-only.
- **Typed state primitives** — `chat.local<T>` for per-run state
accessible from hooks, `run()`, tools, and subtasks (auto-serialized
through `ai.toolExecute`); `chat.store` for typed shared data between
agent and client; `chat.history` for reading and mutating the message
chain; `clientDataSchema` for typed `clientData` in every hook.
- **`chat.toStreamTextOptions()`** — one spread into `streamText` wires
up versioned system [Prompts](https://trigger.dev/docs/ai/prompts),
model resolution, telemetry metadata, compaction, steering, and
background injection.
- **Multi-tab coordination** — `multiTab: true` + `useMultiTabChat`
prevents duplicate sends and syncs state across browser tabs via
`BroadcastChannel`. Non-active tabs go read-only with live updates.
- **Network resilience** — built-in indefinite retry with bounded
backoff, reconnect on `online` / tab refocus / bfcache restore,
`Last-Event-ID` mid-stream resume. No app code needed.
See [/docs/ai-chat](https://trigger.dev/docs/ai-chat/overview) for the
full surface — quick start, three backend approaches (`chat.agent`,
`chat.createSession`, raw task), persistence and code-sandbox patterns,
type-level guides, and API reference.
- Stamp `gen_ai.conversation.id` (the chat id) on every span and metric
emitted from inside a `chat.task` or `chat.agent` run. Lets you filter
dashboard spans, runs, and metrics by the chat conversation that
produced them — independent of the run boundary, so multi-run chats
correlate cleanly. No code changes required on the user side.
([#3543](https://github.com/triggerdotdev/trigger.dev/pull/3543))
- Fix `LocalsKey<T>` type incompatibility across dual-package builds.
The phantom value-type brand no longer uses a module-level `unique
symbol`, so a single TypeScript compilation that resolves the type from
both the ESM and CJS outputs (which can happen under certain pnpm
hoisting layouts) no longer sees two structurally-incompatible variants
of the same type.
([#3626](https://github.com/triggerdotdev/trigger.dev/pull/3626))
- Unit-test `chat.agent` definitions offline with `mockChatAgent` from
`@trigger.dev/sdk/ai/test`. Drives a real agent's turn loop in-process —
no network, no task runtime — so you can send messages, actions, and
stop signals via driver methods, inspect captured output chunks, and
verify hooks fire. Pairs with `MockLanguageModelV3` from `ai/test` for
model mocking. `setupLocals` lets you pre-seed `locals` (DB clients,
service stubs) before `run()` starts.
([#3543](https://github.com/triggerdotdev/trigger.dev/pull/3543))
The broader `runInMockTaskContext` harness it's built on lives at
`@trigger.dev/core/v3/test` — useful for unit-testing any task code, not
just chat.
- Retry `TASK_PROCESS_SIGSEGV` task crashes under the user's retry
policy instead of failing the run on the first segfault. SIGSEGV in Node
tasks is frequently non-deterministic (native addon races, JIT/GC
interaction, near-OOM in native code, host issues), so retrying on a
fresh process often succeeds. The retry is gated by the task's existing
`retry` config + `maxAttempts` — same path `TASK_PROCESS_SIGTERM` and
uncaught exceptions already use — so tasks without a retry policy still
fail fast.
([#3552](https://github.com/triggerdotdev/trigger.dev/pull/3552))
- Add `region` to the runs list / retrieve API: filter runs by region
(`runs.list({ region: "..." })` / `filter[region]=<masterQueue>`) and
read each run's executing region from the new `region` field on the
response.
([#3612](https://github.com/triggerdotdev/trigger.dev/pull/3612))
- **Sessions** — a durable, run-aware stream channel keyed on a stable
`externalId`. A Session is the unit of state that owns a multi-run
conversation: messages flow through `.in`, responses through `.out`,
both survive run boundaries. Sessions back the new `chat.agent` runtime,
and you can build on them directly for any pattern that needs durable
bi-directional streaming across runs.
([#3542](https://github.com/triggerdotdev/trigger.dev/pull/3542))
```ts
import { sessions, tasks } from "@trigger.dev/sdk";
// Trigger a task and subscribe to its session output in one call
const { runId, stream } = await tasks.triggerAndSubscribe("my-task",
payload, {
externalId: "user-456",
});
for await (const chunk of stream) {
// ...
}
// Enumerate existing sessions (powers inbox-style UIs without a
separate index)
for await (const s of sessions.list({ type: "chat.agent", tag:
"user:user-456" })) {
console.log(s.id, s.externalId, s.createdAt, s.closedAt);
}
```
See [/docs/ai-chat/overview](https://trigger.dev/docs/ai-chat/overview)
for the full surface — Sessions powers the durable, resumable chat
runtime described there.
## @trigger.dev/plugins@4.5.0-rc.0
### Patch Changes
- The public interfaces for a plugin system. Initially consolidated
authentication and authorization interfaces.
([#3499](https://github.com/triggerdotdev/trigger.dev/pull/3499))
- Updated dependencies:
- `@trigger.dev/core@4.5.0-rc.0`
## @trigger.dev/python@4.5.0-rc.0
### Patch Changes
- Updated dependencies:
- `@trigger.dev/sdk@4.5.0-rc.0`
- `@trigger.dev/core@4.5.0-rc.0`
- `@trigger.dev/build@4.5.0-rc.0`
## @trigger.dev/react-hooks@4.5.0-rc.0
### Patch Changes
- Updated dependencies:
- `@trigger.dev/core@4.5.0-rc.0`
## @trigger.dev/redis-worker@4.5.0-rc.0
### Patch Changes
- Add MollifierBuffer and MollifierDrainer primitives for trigger burst
smoothing.
([#3614](https://github.com/triggerdotdev/trigger.dev/pull/3614))
MollifierBuffer (`accept`, `pop`, `ack`, `requeue`, `fail`,
`evaluateTrip`) is a per-env FIFO over Redis with atomic Lua transitions
for status tracking. `evaluateTrip` is a sliding-window trip evaluator
the webapp gate uses to detect per-env trigger bursts.
MollifierDrainer pops entries through a polling loop with a
user-supplied handler. The loop survives transient Redis errors via
capped exponential backoff (up to 5s), and per-env pop failures don't
poison the rest of the batch — one env's blip is logged and counted as
failed for that tick. Rotation is two-level: orgs at the top, envs
within each org. The buffer maintains `mollifier:orgs` and
`mollifier:org-envs:${orgId}` atomically with per-env queues, so the
drainer walks orgs → envs directly without an in-memory cache. The
`maxOrgsPerTick` option (default 500) caps how many orgs are scheduled
per tick; for each picked org, one env is popped (rotating round-robin
within the org). An org with N envs gets the same per-tick scheduling
slot as an org with 1 env, so tenant-level drainage throughput is
determined by org count rather than env count.
- Updated dependencies:
- `@trigger.dev/core@4.5.0-rc.0`
## @trigger.dev/rsc@4.5.0-rc.0
### Patch Changes
- Updated dependencies:
- `@trigger.dev/core@4.5.0-rc.0`
## @trigger.dev/schema-to-json@4.5.0-rc.0
### Patch Changes
- Updated dependencies:
- `@trigger.dev/core@4.5.0-rc.0`
</details>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
## Summary
Two CI workflows were blocking the v4.5.0-rc.0 release PR (#3563) and
would block every future changeset release PR.
### 1. `changesets-pr.yml` — self-report `All PR Checks`
The changesets bot pushes commits authored by `GITHUB_TOKEN`. By GitHub
design, `GITHUB_TOKEN`-authored pushes can't trigger downstream
workflows (loop-prevention). That means `pr_checks.yml` never fires on
release-PR commits, leaving the required `All PR Checks` status
permanently `Expected — Waiting for status to be reported`. The PR can't
merge.
The fix: after `changesets/action` creates the PR, post a `success`
check with the exact `All PR Checks` context onto the PR's head SHA.
GitHub's required-check evaluation is satisfied by any check with the
right context name — the source doesn't have to be `pr_checks.yml`.
**Why this is safe:** the release PR only mechanically bumps
`package.json`, `pnpm-lock.yaml`, and `CHANGELOG.md` from changesets
that were already on `main` (and already ran full CI when they merged).
If a human ever pushes a commit to `changeset-release/main`,
`pr_checks.yml` fires on that push (real user, not `GITHUB_TOKEN`) and
posts its own `All PR Checks` status — last write wins for the same
context on the same SHA, so the human-push result overrides the
auto-success.
### 2. `vouch-check-pr.yml` — exempt `github-actions[bot]`
The `require-draft` job auto-closes any non-draft PR whose author is not
a `MEMBER`/`OWNER`/`COLLABORATOR`, with an explicit allowlist for
`devin-ai-integration[bot]` and `dependabot[bot]`. The changesets bot
publishes as `github-actions[bot]` with `author_association:
CONTRIBUTOR`, so every release PR was getting auto-closed on open with a
"please re-open as draft" comment. Add `github-actions[bot]` to the
exemption list.
## Test plan
- [ ] After merge, the next changeset bot push to
`changeset-release/main` should post `All PR Checks: success` on the
release PR's head SHA, and the PR should not get auto-closed by `Vouch -
Check PR`.
- [ ] Confirm `pr_checks.yml` still fires + gates normal
(human-authored) PRs to `main`.
## Summary
Pre-existing typecheck errors in `references/ai-chat` against the
current SDK shape. Unblocks `pnpm exec tsc --noEmit` in the reference
project.
## What changed
Three categories of fixes inside `references/ai-chat`. No SDK changes.
### 1. `payload.messages` → `payload.message`
The wire payload is now delta-only — one new message per trigger,
optional. Old code in two raw-task files reads `payload.messages`
(plural array) which no longer exists.
```ts
// before
const messages = await conversation.addIncoming(currentPayload.messages, ...);
// after
const messages = await conversation.addIncoming(
currentPayload.message ? [currentPayload.message] : [],
...
);
```
Same fix to the `chat.messages.on` handler, reading `msg.message`
(singular) instead of `msg.messages[length - 1]`.
### 2. `clientData` non-null assertion in `cf-trust-test`
`ChatTurnContext.clientData` is typed as `?: TClientData` on
`onTurnStart` / `run` event objects even when the agent declares a
`clientDataSchema`. The runtime validates against the schema before the
hook fires, so it's structurally non-null — but TypeScript can't know
that. Non-null assert for now.
Follow-up worth filing: narrow `ChatTurnContext.clientData` to
non-optional when the agent has a `clientDataSchema`. Same friction the
docs friction-test subagent flagged.
### 3. `stress-emit.parseConfig` retyped against `ModelMessage[]`
The `run` callback hands `messages: ModelMessage[]`, not `UIMessage[]`.
Update `parseConfig` to accept `ModelMessage[]` and pull text from
`content` (string or array-of-parts).
## Test plan
- [x] `pnpm exec tsc --noEmit` in `references/ai-chat` passes (was 8
errors, now 0)
## Summary
Type `chat.createStartSessionAction` against the chat agent so
`clientData` is typed end-to-end on the first turn. Closes the gap where
`useTriggerChatTransport`'s `startSession` callback already hands you a
typed `clientData` (via the transport generic) but the server-side
action couldn't accept it without untyped routing through the `metadata`
field.
## Design
`ChatStartSessionParams` gains a typed `clientData` field via the new
generic:
```ts
export type ChatStartSessionParams<TChat extends AnyTask = AnyTask> = {
chatId: string;
clientData?: InferChatClientData<TChat>;
triggerConfig?: Partial<SessionTriggerConfig>;
metadata?: Record<string, unknown>;
};
function createChatStartSessionAction<TChat extends AnyTask = AnyTask>(
taskId: string,
options?: CreateChatStartSessionActionOptions
): (params: ChatStartSessionParams<TChat>) => Promise<ChatStartSessionResult>
```
When provided, `clientData` is folded into the first run's
`triggerConfig.basePayload.metadata`, so `onPreload` / `onChatStart` see
the same shape per-turn `metadata` carries via the transport. The opaque
session-level `metadata` field stays exactly as before — it lands on the
Session row, not the run payload.
## Usage
```ts
// actions.ts
import { chat } from "@trigger.dev/sdk/ai";
import type { myChat } from "@/trigger/chat";
export const startChatSession = chat.createStartSessionAction<typeof myChat>("my-chat");
```
```tsx
// Chat.tsx
const transport = useTriggerChatTransport<typeof myChat>({
task: "my-chat",
accessToken: ({ chatId }) => mintChatAccessToken(chatId),
startSession: ({ chatId, clientData }) =>
startChatSession({ chatId, clientData }),
});
```
## Test plan
- [x] `pnpm run build --filter @trigger.dev/sdk` passes
- [ ] Verify a `chat.agent` with `clientDataSchema` reads the typed
clientData from `onPreload` payload metadata on the first turn
(`OBJECT_STORE_BASE_URL`) and a named protocol provider
(`OBJECT_STORE_DEFAULT_PROTOCOL=s3`), chat.agent session snapshot writes
landed in the named provider but reads fell through to the default — so
the recovery boot couldn't find the snapshot it had just written.
After a mid-stream cancel, the missing snapshot triggered a fallback
replay path that dropped the user's follow-up message, leaving the chat
stuck in `submitted` indefinitely.
Fix:
- New `/api/v1/sessions/:id/snapshot-url` route handles PUT + GET
symmetrically — both prefix unprefixed keys with
`OBJECT_STORE_DEFAULT_PROTOCOL` so they always round-trip through the
same store.
- `Session.chatSnapshotStoragePath` persists the resolved URI on first
write so future protocol changes don't strand existing snapshots.
Reads prefer the stored URI and fall back to the computed default for
pre-column sessions.
- SDK calls `createChatSnapshotUploadUrl` / `getChatSnapshotUrl`; the
generic v1/v2 packets endpoints are unchanged.
## Test plan
- [x] Configure local with two providers (R2 default + MinIO `s3` named)
and `OBJECT_STORE_DEFAULT_PROTOCOL=s3`.
- [x] Reproduce hang: send a message, cancel mid-stream, send another —
without the fix it hangs in `submitted`; with the fix it streams.
- [x] Snapshot lands in the `s3`-protocol bucket and
`Session.chatSnapshotStoragePath` is set after first write.
- [x] SDK unit tests pass; webapp typecheck passes.
Env-var lookups via `GET
/api/v1/projects/:projectRef/envvars/:slug/:name` run a Prisma
`findMany` on `EnvironmentVariableValue` filtered by `environmentId` +
`isSecret`. The only existing indexes are the primary key and a unique
on `(variableId, environmentId)`, so `environmentId` is never the
leading column — the planner falls back to a Parallel Seq Scan over the
whole table to find what is, in practice, a handful of rows per
environment.
Two changes:
- Add a btree index on `EnvironmentVariableValue(environmentId)` so the
planner switches to an index scan. The composite `(variableId,
environmentId)` unique stays in place; the new index is purely additive.
- Route the `findMany` inside `getEnvironmentWithRedactedSecrets`
through the read replica via a new `replicaClient` constructor param on
the repository (defaulting to `$replica`, mirroring how `prismaClient`
defaults to `prisma`). Writes and read-after-write methods stay on the
primary.
## Test plan
- [ ] `pnpm run typecheck --filter webapp`
- [ ] Confirm `EXPLAIN` plan flips from Parallel Seq Scan to an index
scan
- [ ] Existing env-var route tests still pass
## Summary
- Dashboard loaders for runs / sessions / batches / schedule-detail
threw bare `Error("X not found")` when a slug didn't resolve. Remix
surfaces this as a 500 and Sentry captures it via auto-instrumentation,
producing ongoing noise from real users following stale preview-branch
or deleted-resource links (the URLs in those Sentry events all carry
`?_data=routes/...`, i.e. client-side revalidation, not full-page
navigation).
- Added a `throwNotFound(statusText)` helper in
`app/utils/httpErrors.ts` that throws a Response with status 404,
matching the established pattern in sibling routes (agents, alerts,
bulk-actions, etc.).
- Migrated 5 loader sites to `throwNotFound` (4× "Environment not
found", 1× "Schedule not found").
- Migrated 1 loader site (`runs._index` project branch) to
`redirectWithErrorMessage("/", request, "Project not found")` to match
the pre-existing convention used by every other dashboard route's
project-not-found branch.
- Intentionally **not** touched: bare `throw new Error("X not found")`
inside `resources.*` action routes (sit inside try/catch blocks that
already redirect with a flash message), the invariant assertion in
`vercel.connect.tsx`, and the admin config check in
`admin.api.v1.runs-replication.backfill.ts`.
## Where the fix is visible
Normal browser navigation to these URLs doesn't reach the buggy loaders
— the parent env-layout
(`_app.orgs.$organizationSlug.projects.$projectParam.env.$envParam/route.tsx`)
already filters missing envs/projects and redirects/404s before the
child loader runs. The bug fires exclusively when Remix calls a single
child loader via `?_data=routes/...`, which happens during client-side
navigation or `useRevalidator`. That matches every Sentry event URL.
## Test plan
- [x] Unit test for the new helper —
`apps/webapp/test/httpErrors.test.ts`
- [x] `pnpm run typecheck --filter webapp` clean
- [x] Manual verification via Playwright on `main` vs this branch (6
cases): main returns 500 for each defective `_data` URL; branch returns
404 or 204 + `X-Remix-Redirect` as designed
- [x] Verified user-visible 404 catch boundary on `schedules/<missing>`
(the one case reachable via normal nav)
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
## Summary
Five hardening fixes across `@trigger.dev/sdk`, `@trigger.dev/core`, and
`@trigger.dev/build`.
- `tasks.triggerAndSubscribe` now forwards caller `requestOptions`
(custom API keys, per-request overrides) to the underlying
`apiClient.triggerTask` call instead of silently dropping them.
- `SSEStreamSubscription` no longer retries permanent client errors
forever. The default `nonRetryableStatuses` widens from `[404, 410]` to
`[400, 404, 409, 410, 422]`, so a malformed session-stream request fails
fast instead of busy-looping under bounded backoff.
- Session writer falls back to manually wiring the caller's
`AbortSignal` on Node 18, where `AbortSignal.any` is unavailable.
Caller-driven cancellation now propagates on every supported runtime.
- `TriggerChatTransport` throws immediately when a `chat.handover`
response is missing `X-Trigger-Chat-Access-Token`, instead of silently
downgrading every subsequent turn back to the handover path. `dispose()`
aborts every active `session.out` subscription before tearing the
coordinator down, so unmount/navigation no longer leaves SSE readers in
flight.
- Removed the experimental `@trigger.dev/build/extensions/secureExec`
build extension. It will return alongside the sandbox feature it was
built to support.
## Test plan
- [ ] `pnpm run build --filter @trigger.dev/sdk --filter
@trigger.dev/core --filter @trigger.dev/build`
- [ ] `pnpm --filter @trigger.dev/sdk test --run` (183 tests, including
chat / chat-server / sessions / handover)
- [ ] `pnpm --filter @trigger.dev/core test --run`
- [ ] Manually trigger a `chat.handover` whose response strips
`X-Trigger-Chat-Access-Token`, and confirm the transport throws
synchronously rather than degrading.
- [ ] Unmount a chat UI mid-stream and confirm the active `session.out`
SSE connection closes immediately.
## Summary
The S2 access-token cache key was `${basin}:${streamPrefix}` — purely
server-derived but blind to the **scope/ops list** hardcoded one method
away. When the ops list changes in code (e.g. #3644 added `trim` so
`chat.agent`'s per-turn trim chain can issue `AppendRecord.trim()`),
pre-deploy tokens still in cache get returned to SDK callers for up to
the token's TTL (24h default), surfacing as `Operation not permitted`
403s on any op outside the old scope.
## Fix
Lift the ops list to a module constant and fold its sorted-join
fingerprint into the cache key:
```ts
const S2_TOKEN_OPS = ["append", "create-stream", "trim"] as const;
const S2_TOKEN_OPS_FINGERPRINT = [...S2_TOKEN_OPS].sort().join(",");
// in getS2AccessToken
const cacheKey = `${this.basin}:${this.streamPrefix}:${S2_TOKEN_OPS_FINGERPRINT}`;
// in s2IssueAccessToken
scope: { /* ... */ ops: [...S2_TOKEN_OPS], /* ... */ }
```
The fingerprint is derived from the single source of truth, so any
future scope change auto-invalidates without anyone remembering to bump
a literal version. The Unkey L1 (in-memory LRU) and L2 (Redis) layers
share the same key derivation, so both reset together on the next deploy
with no manual cache busting.
## Test plan
- [ ] `pnpm run typecheck --filter webapp`
- [ ] Run a multi-turn `chat.agent` chat via `references/ai-chat` and
confirm no `chat.agent: trim failed; will retry next turn` warn span
fires across turn-completes.
## Summary
The prerelease (snapshot) path of the release workflow fails immediately
whenever `main` carries an active `.changeset/pre.json` (i.e. during an
in-progress RC cycle, like the current v4 RC):
```
🦋 error Snapshot release is not allowed in pre mode
🦋 To resolve this exit the pre mode by running `changeset pre exit`
```
This blocks `chat-prerelease` snapshots from main even though the
snapshots are unrelated to the RC cycle.
Adds a conditional `changeset pre exit` step right before `Snapshot
version` in the prerelease job. The job runs on a checkout with
`persist-credentials: false`, so the `pre.json` deletion stays on the
runner's working tree — main's persisted pre-mode state is untouched,
and v4 RC publishes keep working normally.
## Test plan
- [ ] Re-run the `🦋 Changesets Release` workflow with `type=prerelease`,
`ref=main`, `prerelease_tag=chat-prerelease` and confirm it gets past
the snapshot step and publishes.
- [ ] Confirm `.changeset/pre.json` on `main` is unchanged after the
run.
## Summary
Companion to #3536, which patched routes that already had a leaking
`catch (e) { return json({error: e.message}, 500) }`. That pattern can't
reach routes which have no catch in the first place — when those throw,
Remix's default error path serializes `error.message` into the response
body, and the SDK then wraps the leaked string as `TriggerApiError`.
Across 28 raw api.v1 loaders/actions plus one dashboard polling
endpoint, each handler now:
- Wraps its body in `try { ... } catch (error) { ... }`.
- Re-throws `Response` instances so auth helpers' `throw json(...)` /
`throw redirect(...)` pass through unchanged.
- Logs non-Response errors via `logger.error` so server-side visibility
is preserved.
- Returns a generic body — `{"error": "Internal Server Error"}` 500 for
raw API routes, or `{ changelogs: [] }` 200 for the polling widget
(degrade silently across transient blips; the consumer hook already
coped with empty payloads).
For six routes where #3536 left an inner try/catch covering only a
service call (`alertChannels`, `batches.results`,
`deployments.finalize`, `deployments.background-workers`,
`deployments.promote`, `projects.background-workers`): an outer
try/catch is added so auth/parsing failures are also sanitized. Inner
typed-error handling (`ServiceValidationError` → 422 with message, etc.)
is preserved exactly.
For two routes whose existing catch returned 400 + `error.message`
(`api.v1.authorization-code`, `api.v1.orgs.\$orgParam.projects` action):
the body is sanitized to a generic per-route string. **Status code stays
400** — clients that key on the 4xx/5xx distinction (and the SDK's
no-retry-on-4xx behavior) are unaffected.
## Test plan
- [x] \`pnpm run typecheck --filter webapp\`
- [x] Per-route synthetic-throw probe: inject \`throw new
Error("SYNTHETIC ...")\` at the top of each catch'd try, curl the route
with a dummy bearer, confirm the response body is the generic shape and
that the synthetic message lands server-side via \`logger.error\`. 29
routes verified.
- [x] Real-P1001 probe on the envvars loader: \`docker stop database\`
mid-flight, confirm response is generic 500 (not the leaked Prisma
message).
- [x] Sampled legitimate 4xx/2xx paths across each pattern variant
(naked-wrap, partial-expanded, 400-preserved) to confirm the wraps don't
interfere with normal control flow.
## Summary
`TriggerChatTransport`, `AgentChat`, and `chat.createStartSessionAction`
now accept a string-or-function `baseURL` so callers can route per
endpoint — e.g. `.in/append` through a trusted edge proxy while keeping
`.out` SSE direct. The same surfaces add a `fetch` override for header
injection, custom retries, or proxy rewrites that go beyond URL routing.
SSE GETs are covered too via a new `fetchClient` option on
`SSEStreamSubscription`.
```ts
// TriggerChatTransport / AgentChat — endpoints: "in" | "out"
baseURL: ({ endpoint }) =>
endpoint === "out" ? DIRECT : PROXY,
fetch: (url, init, ctx) => {
init.headers = new Headers(init.headers);
init.headers.set("traceparent", currentTraceparent());
return globalThis.fetch(url, init);
},
// chat.createStartSessionAction — endpoints: "sessions" | "auth"
chat.createStartSessionAction("my-agent", {
baseURL: ({ endpoint }) => (endpoint === "sessions" ? PROXY : DIRECT),
});
```
`streamBaseURL` on `TriggerChatTransport` is kept as a backwards-compat
alias and continues to win for the `"out"` endpoint when set.
Plain-string `baseURL` still applies to every endpoint, matching prior
behavior.
## Summary
Hot-loop writers — `streams.writer` / `streams.pipe` on the run-scoped
side, `chat.response.write` / `chat.stream.*` on the session side — were
issuing a fresh `PUT` to mint S2 credentials for every chunk. On run
streams, each PUT also pushed the streamId onto
`TaskRun.realtimeStreams`,
so a chat-agent turn writing N chunks produced N PUTs and N duplicate
array pushes against the same row.
The SDK now caches the initialize response per cache slot: `(runId,
key)`
for run streams, the session id for session streams. First call PUTs as
before; subsequent calls reuse the cached promise. Hot-loop writers do
one PUT per slot for the lifetime of the cache.
S2 access tokens have a 1-day TTL. If a writer's `wait()` rejects (auth
error, expired token, network blip), the cache evicts the matching slot
so the next call re-PUTs and mints fresh credentials, identity-checked
so a concurrent caller's fresh promise isn't accidentally cleared.
## chat.agent guardrail
`streams.pipe / writer / append / read` called inside a `chat.agent` run
now logs a one-time warning pointing at `chat.response.write` /
`chat.stream.*` — `streams.*` is run-scoped and isn't visible on the
chat session. The ai-chat docs are updated to drop the old guidance
toward run-scoped streams.
## Summary
- Introduce the Mollifier: a Redis-backed buffer for `trigger()` API
calls during traffic spikes, with a per-env trip evaluator and a drainer
ack-loop.
- Phase 1 is dual-write monitoring — every mollified trigger is buffered
to Redis AND continues to `engine.trigger`. No customer-facing behaviour
change.
- Telemetry events: `mollifier.would_mollify`, `mollifier.buffered`,
`mollifier.drained`, plus the `mollifier.decisions` counter.
- Gated behind a feature flag (default off).
## Test plan
- [x] `pnpm run test --filter @trigger.dev/redis-worker`
- [x] `pnpm run test --filter webapp -- mollifier`
- [x] Manual: with flag off, no behaviour change vs main
- [x] Manual: with flag on + threshold lowered, observe
`mollifier.buffered` + `mollifier.drained` log pairs with matching
`runId`
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
## Summary
Two papercuts new contributors hit running this repo locally:
1. Fresh clones default to v1 (Redis-only) realtime streams, so Sessions
and `chat.agent` error with `"S2 configuration is missing"`, even though
the `s2` service is already in `docker/docker-compose.yml` and pre-seeds
a `trigger-local` basin. Wire `REALTIME_STREAMS_S2_*` to it in
`.env.example` so the new-contributor flow just works. (Also drop the s2
healthcheck: the image is distroless, so the `wget` check always reports
unhealthy.)
2. Two clones can't both run `pnpm run docker` because ports, project
name, and container names are all hardcoded. Parameterize every host
port as `${VAR:-default}`, drive the project name via
`COMPOSE_PROJECT_NAME` (with a top-level `name:` field as the default),
prefix container names with `${CONTAINER_PREFIX:-}`, and pass
`--env-file .env` so compose reads the same root `.env` the webapp does.
The "Running multiple instances side by side" block in `.env.example`
lists every overridable knob.
Also split the optional services (`electric-shard-1`, `ch-ui`,
`toxiproxy`, `nginx-h2`, `otel-collector`, `prometheus`, `grafana`) into
`docker-compose.extras.yml` behind a new `pnpm run docker:full` script.
The core stack keeps everything the webapp actually needs to boot:
postgres, redis, electric, minio, clickhouse + migrator, s2-lite.
Defaults match every previous hardcoded value, so existing setups keep
working without touching `.env`.
## Test plan
- [x] `pnpm run docker` on a clean clone brings up the core services on
the standard ports under the `triggerdotdev-docker` project name.
- [x] Setting `COMPOSE_PROJECT_NAME=triggerdotdev-docker-alt` + the
`*_HOST_PORT` overrides in `.env` brings up a second stack alongside the
default one with no port or container-name clashes.
- [x] Webapp boots cleanly against the default `.env.example` values;
`/healthcheck` returns 200, no S2 errors.
- [x] s2-lite basin `trigger-local` accepts an append + read via the
same REST endpoints the webapp uses.
- [x] `pnpm run docker:full` brings up the optional services alongside
the core ones in the same project.
## Summary
Long-running chat agents were filling `session.out` forever — every
`chat.agent` turn appended to the same S2 stream with no trim, and the
Sessions dashboard re-streamed the entire history from `seq_num=0` on
every page load. After this change the agent appends an S2 `trim`
command record after each `trigger:turn-complete`, pointing back at the
previous turn-complete's seq_num. `session.out` stays roughly one turn
long at steady state, regardless of session age.
`trigger:turn-complete` and `trigger:upgrade-required` move from
`chunk.type`-shaped data records into header-form control records under
a uniform `trigger-control` namespace. Built-in transports
(`TriggerChatTransport`, `AgentChat`, the dashboard's `AgentView`)
handle the new shape transparently. Custom transports need a one-line
filter on the `trigger-control` header — see the rewritten "Records on
session.out" section in the client-protocol docs.
The Sessions detail page in the dashboard fetches the agent's per-turn
S3 snapshot via a presigned URL and seeds the transcript view, then
SSE-tails from the snapshot's `lastOutEventId`. Bandwidth and
time-to-first-render scale with unread turns instead of session
lifetime.
Resume contract is now explicit: single-turn-boundary resume always
works (the prior turn-complete is still on the stream), the S2 trim is
eventually consistent over 10-60s, and multi-turn-away resume falls back
to a snapshot reload.
## Summary
The PUT handler at `/realtime/v1/streams/:runId/:target/:streamId` ran
`taskRun.update({ realtimeStreams: { push: streamId } })` on every call,
even when the `streamId` was already present. SDK call patterns that
re-initialize the same stream key on every chunk produce a per-write row
UPDATE, duplicate entries pile up in the array, and the row-lock + TOAST
rewrite cost grows unbounded on long-running stream sessions.
## Fix
Mirror the sibling append handler: read the array first and only push
when the `streamId` isn't already present. Identical behavior for
first-time stream creation; repeat creates short-circuit to a single
indexed read. The dashboard's per-run stream listing keeps working
because the first create still records the entry.
## Test plan
- [ ] A fresh PUT for a new `(run, streamId)` adds the entry to the
array
- [ ] A repeat PUT for the same pair leaves the array unchanged
- [ ] 404 is returned when the run doesn't exist; 400 when the run is
completed
## Summary
- Adds a `beforeSend` rule in `apps/webapp/sentry.server.ts` that
collapses Prisma `P1001` ("Can't reach database server") errors into a
single Sentry issue regardless of which call site threw, by setting
`event.fingerprint = ["prisma-p1001-db-unreachable"]` and tagging
`db_unreachable:true`.
- Matches both `err.code === "P1001"` (Prisma's `KnownRequestError` when
a connection drops mid-query) and `err.errorCode === "P1001"`
(`InitializationError` when the client fails to connect at startup).
- Implemented as a small extensible `FINGERPRINT_RULES` table so further
fan-out errors can be added with one entry.
## Verification
End-to-end verified locally with `debug: true` on the SDK:
- Real Prisma `P1001` thrown from a loader (DB stopped mid-request) is
captured by Sentry's Remix auto-instrumentation
- `beforeSend` fires with `originalException.code === "P1001"`, rule
matches
- `event.fingerprint = ["prisma-p1001-db-unreachable"]` and
`tags.db_unreachable = "true"` applied
- Event lands in Sentry under the new fingerprint
## Test plan
- [ ] Deploy to staging; confirm P1001 events appear under a single
`prisma-p1001-db-unreachable` issue rather than fanning out
- [ ] Confirm `db_unreachable:true` tag is filterable in Sentry
- [ ] Verify non-P1001 errors are unaffected (event passes through
`beforeSend` untouched)
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
## Summary
Local self-hosted deploys (`trigger deploy --local-build --push
--builder orbstack` or any other buildx setup using the **docker**
driver) fail at the push step with:
```
ERROR: failed to build: failed to solve:
exporter option "rewrite-timestamp" conflicts with "unpack"
```
The docker driver auto-enables `unpack=true` when pushing, and that's
incompatible with `rewrite-timestamp` (which the CLI sets for
reproducible-build hashing).
Adds a simple env-var opt-out so contributors can keep using their
default builder. The flag is only read by the local-build code path;
remote/cloud builds are unaffected.
```bash
TRIGGER_BUILD_SKIP_REWRITE_TIMESTAMP=1 \
pnpm exec trigger deploy --profile default --local-build --push --builder orbstack
```
The trade-off: skipping `rewrite-timestamp` means layer timestamps
reflect actual build time, so two identical builds produce different
layer hashes. Fine for a local-dev registry; the only real consumer of
timestamp-stability is registry-layer cache hit rates.
## Test plan
- [x] Manual: ran `trigger deploy --profile default --local-build --push
--builder orbstack` against the localhost webapp + a local Docker
registry on port 5001 — first failed with the rewrite-timestamp/unpack
error, then succeeded after setting
`TRIGGER_BUILD_SKIP_REWRITE_TIMESTAMP=1`.
- [x] Full chat.agent smoke sweep (15 tests, including suspend/resume,
deepResearch subtask, AgentChat orchestrator) against the deployed image
— all pass.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
`TriggerChatTransport` had a single `baseURL` option covering both the
`.in/append` POSTs and the long-lived `.out` SSE subscription. Customers
wanting to route the SSE through a proxy (e.g. a Cloudflare worker
capturing JA4 fingerprints for bot detection) had to send every append
through the proxy too, adding a hop to every user message.
New optional `streamBaseURL` overrides the SSE base URL only; appends
keep using `baseURL`. Falls back to `baseURL` when unset, so existing
transports are unchanged.
```ts
const transport = new TriggerChatTransport({
task: "ai-chat",
baseURL: "https://api.trigger.dev",
streamBaseURL: "https://chat-proxy.example.com",
accessToken,
startSession,
});
```
Verified with a new test in `chat.test.ts` that asserts `.in/append`
routes through `baseURL` and `.out` SSE routes through `streamBaseURL`.
All existing tests still pass.
`dorny/paths-filter` defaults to OR semantics across the pattern array,
so the leading `**` matched every file and the `!...` excludes were
no-ops. The `code` filter has been returning `true` for every PR since
#3615.
Split into two filter steps: `code` moves into its own step with
`predicate-quantifier: every` so excludes actually subtract. The two
re-include workflow files become a separate `typecheck_self` filter that
the `typecheck` job ORs into its `if:`.
Side effect: workflow-file-only PRs that don't touch `pr_checks.yml` or
`typecheck.yml` no longer trigger typecheck. Previously they did because
the filter was broken-true.
## Summary
Codify two rules for ClickHouse migration authors that came out of the
029/030 ordering incident on the TRI-9367 test cloud deploy:
1. **Number files to `max(existing) + 1`, never slot in below the
latest.** Goose runs in strict mode in the cloud deploy pipeline and
refuses to apply a missing version below the current version — slotting
a file in below an already-applied number blocks the next deploy.
2. **DDL must be idempotent** (`ADD COLUMN IF NOT EXISTS`, `DROP COLUMN
IF EXISTS`, `CREATE TABLE IF NOT EXISTS`, etc.) so a retry or
out-of-order apply (`goose up --allow-missing` for local recovery,
manual fixups) is a no-op rather than an error.
## Where the rules live
- `internal-packages/clickhouse/CLAUDE.md` — full rules + example for
migration authors (and AI agents writing migrations).
- `.claude/REVIEW.md` — added a 🔴 finding under "What makes a 🔴
Important finding" so PR reviewers flag either fault as blocking.
The existing migration files are left untouched; the idempotency
requirement applies going forward.
## Test plan
- [ ] Next ClickHouse migration PR uses `IF NOT EXISTS` / `IF EXISTS`
forms
- [ ] No new migration files numbered below an already-applied version
on test/prod
## Summary
Renumber `029_add_task_kind_to_task_runs_v2.sql` →
`031_add_task_kind_to_task_runs_v2.sql` to fix a deploy-blocking
out-of-order migration, and make the DDL idempotent with `ADD COLUMN IF
NOT EXISTS` / `DROP COLUMN IF EXISTS`.
## Root cause
- Migration `030_create_sessions_v1.sql` landed on main on 2026-04-28
(PR #3417) and was applied to test cloud ClickHouse on a subsequent
deploy. Current goose version on test ClickHouse: **30**.
- Migration `029_add_task_kind_to_task_runs_v2.sql` was authored later
on 2026-05-10 as part of the Sessions primitive PR series (`be1a6cf8`).
- The next test cloud deploy failed because goose strict-mode refused to
apply a missing version *before* the current version:
```
goose run: error: found 1 missing migrations before current version 30:
version 29: 029_add_task_kind_to_task_runs_v2.sql
```
## Fix
1. **Rename to `031_*`** (next available number after 030). Goose now
treats it as a new migration after 030 and applies it cleanly on
test/prod where the column does not yet exist.
2. **Make the DDL idempotent** (`ADD COLUMN IF NOT EXISTS`). The
original 029 may have been applied in environments that ran goose with
`--allow-missing` (e.g. some local dev databases) — those would have the
column already, and the rename causes goose to see 031 as new and
re-attempt the ADD. Idempotent DDL keeps that path safe. The `Down`
mirrors with `DROP COLUMN IF EXISTS`.
## Test plan
- [ ] Test cloud deploy (after this lands) successfully runs the
ClickHouse migration step
- [ ] `task_kind` column shows up on `trigger_dev.task_runs_v2`
post-migration
- [ ] Local environments that had previously applied 029 do not error on
the next `goose up`
## Summary
Adds `.changeset/pre.json` to put the repo into changesets pre mode with
dist-tag `rc`. After this merges, the changesets bot regenerates the
existing release PR as `chore: release v4.5.0-rc.0`. Merging that PR
publishes the first release candidate of 4.5.0 to npm under `@rc`.
The pre-mode plumbing landed in #3628. The release content (chat.agent +
sessions + ai prompts + dashboard server-changes) landed in #3629.
## What ships when the bot PR merges
Under dist-tag `rc`:
-
`@trigger.dev/{sdk,core,build,react-hooks,redis-worker,plugins,python,rsc,schema-to-json}@4.5.0-rc.0`
- `trigger.dev@4.5.0-rc.0`
Plus:
- Docker image `ghcr.io/triggerdotdev/trigger.dev:v4.5.0-rc.0`
(immutable tag only — `:v4-beta` is not touched)
- Helm chart `oci://ghcr.io/triggerdotdev/charts/trigger.dev:4.5.0-rc.0`
- GitHub release `v4.5.0-rc.0` marked as Pre-release (no Latest badge)
What does NOT happen:
- npm `latest` stays at 4.4.6
- No marketing-site changelog PR (gated on `is_prerelease != 'true'`)
- Docker `:latest` not touched (we never push it anyway in this repo)
## Iteration
For subsequent rc.N: add a regular changeset to main, bot regenerates
the release PR as `v4.5.0-rc.N`. Merge to ship.
## Exiting pre mode
When ready to ship stable: `pnpm exec changeset pre exit`, push, merge
regenerated PR. That publishes `4.5.0` under `latest` and fires the
marketing-site dispatch.
## Summary
Refocuses the v4.5.0 changeset and server-changes content on the
public-facing AI features story, replacing the pre-release-internal diff
framing that had accumulated in `.changeset/` and `.server-changes/`.
Pairs with the RC support PR — the next bot regeneration will pick up
this content.
## What's in here
### Changeset rewrites
- **`chat-agent.md` rewritten as the headline AI Agents entry** —
written from the `docs/ai-chat/` surface (not from internal pre-release
diffs). Covers useChat integration, multi-turn durability via Sessions,
lifecycle hooks, stop generation, tool approvals (HITL), pending
messages + background injection, actions, typed state primitives,
`chat.toStreamTextOptions()`, multi-tab coordination, network
resilience, and the first-turn fast path (`chat.headStart`).
- **New `ai-prompts.md`** — announces the Prompts feature publicly for
the first time. Code-defined templates, deploy-versioning, dashboard
overrides, AI SDK telemetry integration, `chat.agent` integration via
`chat.prompt.set()` + `chat.toStreamTextOptions()`, full management SDK.
- **`sessions-primitive.md` expanded** — calls out
`tasks.triggerAndSubscribe()` and `sessions.list` as standalone
primitives (not just chat.agent infrastructure).
- **`chat-agent-on-boot-hook.md` trimmed** — drops "if you previously…"
pre-release migration framing.
- **Deletes 4 changesets** that described pre-release-internal
migrations or were circular ("groundwork for the upcoming chat.agent" —
chat.agent ships in the same release).
### Server-changes rewrites (`.server-changes/`)
Five new entries for the dashboard surface of the AI feature set:
- Agents list page
- Agent Playground
- Sessions dashboard
- Prompts dashboard (list with usage sparklines + detail with template /
Generations / Metrics / Versions tabs + override UI)
- Models registry (provider-grouped catalog with cross-tenant usage
metrics)
- AI generation span inspector on run traces
- Runs list Task source filter (Standard / Scheduled / Agent)
- Run-detail Agent view (segmented control)
Each entry is 1–2 sentences, no bullets, no implementation file paths —
fits as a single bullet in a future changelog.
Three older `.server-changes/` files were merged or split into the
cleaner taxonomy above and deleted.
## Out of scope
Non-AI-feature server-changes (admin-tabs, queue-length-cap fix,
worker-deployment race, streamdown upgrade, etc.) and changesets
(idempotency-key cap, sigsegv retry, locals-key fix, plugin auth, region
filters, etc.) are untouched.
## Summary
Enables shipping `X.Y.Z-rc.N` prereleases of `@trigger.dev/*` via
changesets pre mode. RCs publish under the `rc` npm dist-tag, never
claim `latest`, and don't trigger marketing-site changelog PRs. The
plumbing is hyphen-in-version detection in `release.yml` — no separate
workflow, no opt-in flag at publish time.
Validated end-to-end against a sandbox repo (real npm publishes, Docker
builds, Helm chart pushes, GitHub releases) before porting back. Full RC
lifecycle tested: pre enter → rc.0 → iterate to rc.1 → pre exit →
stable. Plus interaction with the existing release-branch hotfix flow.
## What changes
### `release.yml`
- New `is_prerelease` output (hyphen-in-version)
- GitHub release adds `--prerelease` flag for RC publishes (Pre-release
badge, not Latest)
- `dispatch-changelog` job gated on `is_prerelease != 'true'` — no
marketing-site PR per RC
### Docker workflows
- Removes the `:v4-beta` floating tag entirely from `publish-webapp.yml`
and `publish-worker-v4.yml`. v4 is GA; the tag is a misnomer and is
already inconsistent with the npm side (npm `v4-beta` dist-tag was
frozen at 4.0.4 months ago while Docker `:v4-beta` kept bumping).
Self-hosters should pin to a versioned tag going forward — the last
value of `:v4-beta` stays frozen wherever it currently points.
### CLI version-check fix
(`packages/cli-v3/src/utilities/initialBanner.ts`)
Switches the "new version available" comparison from JavaScript
`localeCompare` to `semver.lt`. The old comparison handled `X.Y.Z-rc.N`
vs `X.Y.Z` incorrectly — a user on `4.5.0-rc.0` would never be prompted
to upgrade once `4.5.0` stable shipped (lex order put the prerelease
ahead of the bare version). Real semver gets this right.
Stable users were never affected: the check queries the `@latest`
dist-tag, which by convention never points at a prerelease.
## How an RC actually publishes after this
1. `pnpm exec changeset pre enter rc` on main, push the `pre.json`
2. Bot regenerates the release PR as `chore: release v<X.Y.Z>-rc.0`
3. Merge → `release.yml` runs `changeset publish` which reads
`pre.json.tag` and publishes under `--tag rc`. GitHub release marked
Pre-release. No marketing-site dispatch.
4. Iterate by adding changesets normally; bot bumps to `rc.1`, `rc.2`, …
5. When ready: `pnpm exec changeset pre exit`, push, merge regenerated
PR → stable ships under `latest` and the marketing-site dispatch fires.
## Summary
Adds a Region column and Region filter (under More filters) to the runs
list dashboard, the same filter on the public runs list API
(`filter[region]`), and a matching `region` input on the MCP `list_runs`
tool. Each run's executing region is also surfaced as a new optional
`region` field on the runs list and run retrieve responses, populated
from the worker instance group's `masterQueue` identifier.
Useful when you run tasks across multiple regions and want to slice the
runs list — or your existing run-querying scripts — by where the run
actually executed.
## Design
The filter value in the URL / API is the `masterQueue` identifier (the
same string already persisted on `TaskRun` and replicated to ClickHouse
as `worker_queue`), so the query just becomes `worker_queue IN (...)`
with no server-side translation. The Region dropdown options come from a
new resource loader backed by `RegionsPresenter`, which now also exposes
`masterQueue` alongside the existing region metadata.
```ts
// public API
const runs = await runs.list({ region: ["us-east-1", "eu-west-1"] });
// each item: { id, status, ..., region?: "us-east-1" }
```
```ts
// MCP
list_runs({ environment: "prod", region: "us-east-1" })
```
## Summary
The trigger-task hotpath used to early-return without a DB query when a
caller passed both a queue override and a per-trigger TTL — the hottest
configuration on the trigger API. Adding `triggerSource` to the resolver
so the runs-list "Source" filter could distinguish STANDARD / SCHEDULED
/
AGENT runs removed those early-returns, costing +2 DB queries per
trigger
on non-locked calls and +1 on locked calls.
This change caches `BackgroundWorkerTask` metadata (`ttl`,
`triggerSource`,
`queueId`, `queueName`) in Redis so the resolver can satisfy every
caller
configuration with a single `HGET` on the warm path. PG fallback on miss
back-fills the cache.
Follow-up to #3542.
## Design
Two key spaces:
- `task-meta:env:{envId}` — the "current worker" view, refreshed at
every
deploy promotion. 24h safety TTL.
- `task-meta:by-worker:{workerId}` — used for `lockToVersion` triggers.
Immutable post-create. 30d sliding TTL so historical workers age out.
Cache writes use Lua scripts via `defineCommand` so `DEL` + `HSET` +
`EXPIRE` land atomically — concurrent readers never see the empty
intermediate state of a naive pipeline. Read-path back-fill uses
single-field upserts so concurrent back-fills don't wipe each other's
siblings.
The cache lives behind its own `TASK_META_CACHE_REDIS_*` env-var prefix
that falls back to the default `REDIS_*` set, so operators can route the
cache to a dedicated Redis instance if they want.
The service/instance file split (`taskMetadataCache.server.ts` for the
pure class, `taskMetadataCacheInstance.server.ts` for the env-wired
singleton) mirrors the existing `runsReplicationService` /
`runsReplicationInstance` pattern.
## Test plan
- [ ] `pnpm run typecheck --filter webapp`
- [ ] `pnpm run test ./test/engine/triggerTask.test.ts --run` — 8
existing tests untouched + 5 new tests covering warm cache, cold
miss with back-fill, queue + ttl path, by-worker vs env keyspace,
and the promotion cache write
- [ ] End-to-end against a dev worker: registering writes both keyspaces
with the expected TTLs, and `redis-cli HGETALL
"tr:task-meta:env:<envId>"`
returns the cached entries
## Benchmark
Measured `DefaultQueueManager.resolveQueueProperties` against a real
Postgres + Redis (vitest `containerTest`, single-host docker). 500
sequential calls and 2,000 parallel calls (concurrency=50) per scenario,
request shaped as `{ taskId, queue: "bench-queue", ttl: "5m" }` — the
hot path this PR restores.
```
sequential (one in flight at a time):
[noop cache (baseline)] n=500 mean=1.423ms p50=1.394ms p95=1.735ms p99=2.629ms max=11.100ms
[redis cache, cold ] n=500 mean=1.346ms p50=1.283ms p95=1.688ms p99=2.463ms max=5.058ms
[redis cache, warm ] n=500 mean=0.084ms p50=0.078ms p95=0.105ms p99=0.156ms max=1.129ms
speedup (warm vs baseline, sequential): 16.95x
parallel (concurrency=50):
[noop cache (baseline)] n=2000 mean=10.069ms p50=8.850ms p95=14.718ms p99=31.887ms total=405ms ops/s=4,940
[redis cache, warm ] n=2000 mean=0.614ms p50=0.568ms p95=1.189ms p99=1.432ms total=25ms ops/s=80,389
throughput speedup (warm vs baseline, parallel): 16.27x
```
Read:
- **Warm cache cuts resolver latency 17×** at p50 — from ~1.4 ms to ~78
µs per call.
- **Cold cache is on par with baseline** — the extra `HGET` miss adds
<50 µs against the two Postgres queries that follow, so the worst case
is not worse than today.
- **Under burst load (50 concurrent triggers)**, the baseline's p99
jumps to ~32 ms as Postgres connections queue up; warm stays at ~1.4 ms.
The cache moves the saturation point from ~5k ops/s (PG pool) to ~80k
ops/s (single-client Redis pipelining).
Caveats: single-host docker, local Postgres + Redis, resolver-only
measurement (excludes the rest of the trigger transaction). Prod adds
region-local Redis RTT (~0.3–0.8 ms) which shifts warm absolute numbers
up but keeps the ratio intact.
## Summary
A "Google auth conflict" Sentry alert fires whenever a user signs in via
Google whose Google account is linked to one user row but whose
Google-provided email is now on a *different* user row. The handler in
`apps/webapp/app/models/user.server.ts:236` already does the right thing
— it returns the existing auth-linked user and skips the update path so
neither row gets mutated — but it logs the situation with
`logger.error`, which routes to Sentry as an exception and pages the
on-call channel.
There's no exception to chase here: the branch is the intended outcome
for a known data shape (user changed their email on one account after
originally signing up via Google on another). Downgrading the call to
`logger.warn` keeps the diagnostic record in our logs (with all the same
context fields — email, both user IDs, authIdentifier) but stops it
firing the production error alert.
## Change
- `logger.error` → `logger.warn` for the conflict branch in
`findOrCreateGoogleUser`. Context payload is unchanged.
## Test plan
- [x] Typecheck only — there's no behavioural change to test, the log
level is the entire diff.
## Summary
`LocalsKey<T>` (the type returned by `locals.create()`) was branded with
a
module-level `declare const __local: unique symbol`. Each such
declaration
is its own nominal type, and `tshy` emits separate `.d.ts` files for the
ESM and CJS outputs — each gets its own `__local` symbol. Under certain
pnpm hoisting layouts a single TypeScript compilation can resolve
`LocalsKey` from both the ESM source path and the CJS dist path within
the same call site, producing two structurally-incompatible variants of
the same type. TS surfaces this as the misleading error:
```
Argument of type 'LocalsKey<X>' is not assignable to parameter of type
'LocalsKey<X>'. Property '[__local]' is missing in type 'LocalsKey<X>'
but required in type 'BrandLocal<X>'.
```
The error has been hitting CI on PRs opened since the chat.agent stack
landed (e.g. #3625 typecheck job), but doesn't reproduce on developer
machines where the pnpm node_modules layout was built up incrementally.
## Fix
Replace the `unique symbol` brand with an optional phantom field that
carries `T` at the type level:
```ts
// before
declare const __local: unique symbol;
type BrandLocal<T> = { [__local]: T };
export type LocalsKey<T> = BrandLocal<T> & {
readonly id: string;
readonly __type: unique symbol;
};
// after
export type LocalsKey<T> = {
readonly id: string;
readonly __type: symbol;
/** Phantom carrier for the value type — never read at runtime. */
readonly __valueType?: T;
};
```
The ESM and CJS `.d.ts` outputs now produce structurally identical
types,
so cross-output resolution no longer produces a mismatch. `T` is still
carried at the type level via the optional phantom field. The runtime
shape is unchanged — `manager.ts` was already casting via `as unknown`,
which is no longer needed.
## Test plan
- [ ] `pnpm run typecheck --filter @trigger.dev/core --filter
@trigger.dev/sdk`
- [ ] `pnpm run build --filter @trigger.dev/core --filter
@trigger.dev/sdk`
(clean rebuild) — confirms the ESM and CJS dist `.d.ts` outputs
no longer carry distinct `unique symbol` declarations
- [ ] `pnpm --filter @trigger.dev/core test test/mockTaskContext.test.ts
--run`
- [ ] `pnpm --filter @trigger.dev/sdk test test/mockChatAgent.test.ts
--run`
Follow-up to #3615. The `code` filter currently fires typecheck for any
change outside `docs/`, `.changeset/`, `hosting/`, or `.github/` - so a
docs-only PR like #3623 (touching `references/ai-chat/.env.example` +
`README.md`) triggered the typecheck job. None of the `references/*`
packages declare a `typecheck` script either, so even when a real code
change lands there, `turbo run typecheck` skips them. Running the job is
pure cost.
Tightens the filter to also exclude:
- `references/**` - playground projects, none of them contribute to
`turbo run typecheck` today
- `**/*.md` - markdown anywhere
- `**/.env.example` - example env files anywhere
Two known gaps left open:
- references/ have no real CI typecheck coverage. Separate question -
either add `typecheck` scripts to each (or top-level `tsc -p`), or
accept playground status.
- `changes` job still runs (it's a path-filter step) but the dependent
jobs all skip on irrelevant PRs.
## Summary
A complete Next.js reference project that exercises `chat.agent`
end-to-end, plus the CLI MCP tools that let Claude Code, Cursor, and
similar IDE agents drive a deployed `chat.agent` task from the editor.
Builds on #3545.
## Design
`references/ai-chat` is a full Next.js app: prisma-backed persistence,
multi-chat sidebar, per-chat model picker, debug panel, tool examples
(`getCurrentTime`, `searchHackerNews`, `createGithubIssue`, PR review
helpers, code sandbox), and smoke tests. It's intended both as a
copy-paste starting point and as a place to regression-test SDK changes.
The CLI gains MCP tools (`start_agent_chat`, `send_agent_message`,
`close_agent_chat`, `list_agents`) so an IDE agent can converse with a
deployed `chat.agent` task. The dev runtime adds one-shot OOM kill on
the run controller and skills bundling in the build pipeline.
Top of the chat.agent stack: a full Next.js reference project that
exercises chat.agent end-to-end, plus the CLI MCP tools that drive
agent runs from Claude Code / Cursor / etc.
references/ai-chat:
- Full Next.js app with prisma persistence, multi-chat sidebar,
per-chat model picker, debug panel, tool examples, smoke tests
- Reference tools: getCurrentTime, searchHackerNews, createGithubIssue,
PR review helpers, code sandbox
- chat-client-test orchestrator for concurrent-send stress
- references/hello-world chatAgent + triggerAndSubscribe examples
CLI MCP tooling for chat.agent:
- mcp/tools/agentChat.ts (start_agent_chat, send_agent_message,
close_agent_chat)
- mcp/tools/agents.ts + tasks.ts (list agents, agent run details)
- dev-run-controller OOM kill + taskRunProcessPool tweaks
- dev/managed entry-point hooks for skills bundling
- buildWorker + bundleSkills (agent skills support)
Includes ai-tool-helpers + mcp-agent-chat-sessions changesets, plus
the streamdown@2 patch and pnpm-lock reconciliation.
(Will be renamed to feature/ai-chat-reference-and-cli before push.)
fix(cli): preserve lastEventId after sendMessage fallback to avoid stale turn-complete replay
## Summary
A chat-aware run inspector and a `/playground` UI for testing
`chat.agent` tasks interactively. Builds on #3543's runtime.
## Design
The run inspector grows a new tab that renders the conversation chain
for any `chat.agent`-kind run. It subscribes to the run's session
streams, threads chat parts through a per-message renderer, and uses a
shared markdown + Shiki component for code highlighting (also used by
the test-payload panel).
The playground is a standalone `/playground` route that lets you drive a
deployed chat agent from the dashboard — pick a task, send messages,
watch tool calls render, and see span detail on every turn. The matching
`/agents` list view shows all deployed agents in the project.
Concurrent `POST /api/v1/deployments` requests for the same environment
race on the `WorkerDeployment(environmentId, version)` unique
constraint. Both requests read the same latest deployment via
`findFirst`, compute the same next version via
`calculateNextBuildVersion`, and both attempt
`prisma.workerDeployment.create()` — one wins, the other crashes with
Prisma `P2002`. The bug is a classic TOCTOU between the version read and
the version write; it's been latent since the version-assignment logic
was first added but only fires when two deploys land within milliseconds
of each other (CI matrices, retried CLI calls, webhook-triggered
redeploys).
## Approach
Extracts the version assignment + create into a small helper
`createDeploymentWithNextVersion`
(`apps/webapp/app/v3/services/initializeDeployment/createDeploymentWithNextVersion.server.ts`).
The helper retries on `P2002 (environmentId, version)` up to 5 times
with randomised 5–50ms jitter so N concurrent racers don't loop in
lockstep. Each attempt re-reads the latest version, recomputes via
`calculateNextBuildVersion`, and re-runs the caller's `buildData`
callback so version-dependent fields (image ref tag, friendlyId) are
always consistent with the version actually persisted. A `logger.warn`
fires per collision so the retry rate is observable in production logs.
When retries are exhausted, the helper throws a dedicated
`DeploymentVersionCollisionError` carrying `environmentId`, `attempts`,
and `lastAttemptedVersion`, with the original
`PrismaClientKnownRequestError` attached as `cause`. Sentry walks the
`cause` chain natively, so contention exhaustion shows up as a
distinguishable wrapper exception linked to the underlying `P2002`
rather than a generic unique-constraint violation that looks identical
to every other duplicate-key bug.
The behavioural change is limited to "catch P2002 and retry instead of
crashing." The image ref computation stays inside the builder callback
(same call site as before the refactor), so ECR / non-ECR behaviour, S2
stream creation order, and all downstream side effects are unchanged.
## Non-goals
- No new database migrations, no schema changes, no isolation-level /
locking changes. A serialisable transaction or advisory lock would also
fix this; retry-on-conflict is the smaller change that keeps the
existing version-allocation logic intact.
- Does not touch the analogous `calculateNextBuildVersion` call in
`createBackgroundWorker.server.ts`, which likely has the same race shape
against `BackgroundWorker`'s unique constraint — flagged as a follow-up.
## Test plan
- [x] `pnpm run typecheck --filter webapp` passes (no new errors in the
modified files).
- [x] Three real-Postgres tests in
`apps/webapp/test/createDeploymentWithNextVersion.test.ts` via
`containerTest`:
- 5 concurrent calls all produce distinct, persistable versions
(`Set(versions).size === concurrency`). The naive read-then-create
version of the helper fails this test with the exact same `P2002` seen
in production; the retry version passes.
- Non-`P2002` errors raised from the `buildData` callback propagate
immediately without retry, builder invoked exactly once.
- With `maxRetries: 0`, concurrent racers surface the wrapped
`DeploymentVersionCollisionError` (not a raw `P2002`); `environmentId`,
`attempts`, `lastAttemptedVersion` are populated and `error.cause.code
=== "P2002"`.
- [x] Existing `apps/webapp/test/getDeploymentImageRef.test.ts` still
green (the file was untouched in the final diff).
## Follow-ups (not in this PR)
- `createBackgroundWorker.server.ts` likely has the same TOCTOU shape
against its background-worker version unique constraint — should use the
same helper.
- Sentry visibility check: confirm `error.cause` chain renders as a
linked exception in the Sentry UI when the wrapped error fires (requires
a sandboxed triggering of the exhaustion path).
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
## Summary
Adds `chat.agent({...})`, a durable conversational task runtime, plus
the browser-side `TriggerChatTransport` + `AgentChat` that drive it from
a React or Next.js app. Conversations survive page refreshes, network
blips, idle suspend, and process restarts, with built-in tools, HITL
approvals, multi-turn state, and stop-mid-stream cancellation. Builds on
#3542.
## Design
Each `/in/append` request carries at most one new message. The agent
reconstructs prior history at run boot from an object-store snapshot
plus a `session.out` replay tail, so conversation context lives
server-side instead of bloating the wire. Awaited snapshot writes after
every `onTurnComplete` keep the chain durable across idle suspend.
Registering `hydrateMessages` short-circuits both paths for customers
who own their own conversation store.
Lifecycle hooks — `onChatStart`, `onTurnStart`, `onTurnComplete`,
`onAction`, `onValidateMessages`, `hydrateMessages` — cover validation,
persistence, and post-turn work. `chat.history` exposes read primitives
(`getPendingToolCalls`, `getResolvedToolCalls`, `extractNewToolResults`,
`findMessage`, `all`) for HITL flows. `chat.local` gives per-run typed
state with Proxy access and dirty tracking. `chat.headStart` bridges
first-turn TTFC via a customer HTTP handler. `oomMachine` opts a chat
into one-shot OOM-retry on a larger machine.
`TriggerChatTransport` is a `Transport` implementation for Vercel's
ai-sdk `useChat`: delta-only wire sends, SSE reconnection with
`lastEventId` resume, stop/abort cleanup, dynamic `accessToken` refresh,
`X-Peek-Settled` fast-close. `AgentChat` is the direct programmatic
equivalent. A cross-tab coordinator does leader election so multiple
open tabs share a single SSE.
```ts
import { chat } from "@trigger.dev/sdk/ai";
import { streamText } from "ai";
export const myChat = chat.agent({
id: "my-chat",
run: async ({ messages, signal }) =>
streamText({ model: openai("gpt-4o"), messages, abortSignal: signal }),
});
```
Adds the chat.agent({...}) task definition (server runtime) and the
browser-side TriggerChatTransport + AgentChat that drives it from a
React or Next.js app. The runtime sits on top of the Sessions primitive
and handles the durable conversational task lifecycle.
Server runtime:
- chat.agent({...}) — session-aware task definition
- Lifecycle hooks: onChatStart, onTurnStart, onTurnComplete, onAction,
onValidateMessages, hydrateMessages
- chat.history read primitives for HITL flows
- chat.local, chat.headStart, chat.handover, oomMachine
- Delta-only wire + S3 snapshot reconstruction at run boot
- Actions are no longer turns
Browser transport:
- TriggerChatTransport (ai-sdk Transport): delta-only wire sends,
SSE reconnection with lastEventId resume, stop/abort cleanup,
dynamic accessToken refresh
- AgentChat: direct programmatic API
- useTriggerChatTransport (React hook)
- chat-tab-coordinator: cross-tab leader election
Includes the chat-agent, chat-agent-delta-wire-snapshots,
chat-history-read-primitives, chat-head-start, chat-actions-no-turn,
chat-session-attributes, agent-skills, and mock-chat-agent-test-harness
changesets.
Adds a Mon 08:00 UTC workflow that posts a summary of open Dependabot
alerts and PRs to Slack. Uses env-scoped secrets so the alerts PAT and
Slack token are only available to this workflow.
The `code` paths filter currently matches `**` minus a tiny exclusion
list, so a PR that only touches `.github/workflows/*.yml` still flips
`code == true` and runs typecheck (~2 min on the runner).
Exclude `.github/**` from `code`, then re-include just `pr_checks.yml`
and `typecheck.yml` so a change to either of those still triggers the
full code check matrix.
Effect:
- workflow-only PRs (this one, future dependabot/codeql/etc.) skip
typecheck; `all-checks` treats the skipped job as non-failure so the
required status passes.
- modifying `pr_checks.yml` or `typecheck.yml` themselves still triggers
typecheck.
- the existing per-suite filters (`webapp`, `packages`, `internal`,
`cli`, `sdk`) already re-include the specific workflows that gate them,
so they're unaffected.
## Summary
A `/sessions` dashboard for inspecting durable Sessions, an `AGENT` /
`SCHEDULED` task-kind filter for the runs list, and the server-side
hardening (rate-limit exemption for packets, retry-with-backoff on
stream appends, typed too-large-chunk error) that the `chat.agent`
runtime in #3543 needs. Builds on the Sessions primitive shipped in
#3417.
## Design
The Sessions list + detail routes mirror the run inspector pattern.
`TaskTriggerSource` gains `AGENT` and `SCHEDULED` values, persisted on
`BackgroundWorker.taskKind` and `TaskRun.taskKind` (plus a matching
Clickhouse column), so the runs list can filter by kind.
New `@trigger.dev/core` modules — `sessionStreams`, `inputStreams`, a
`sessionStreamInstance` for realtime streams, and the
`realtime-streams-api` / `session-streams-api` surfaces — expose the
typed shapes that chat.agent will use to drive `session.out`.
`ChatChunkTooLargeError` lets the runtime drop oversized chunks with a
typed surface instead of failing the run. `s2Append` retries transient
failures with exponential backoff. `/api/v[12]/packets/*` is exempt from
customer rate limits so chat snapshot reads and writes don't get
throttled under load.
## Stack
Part of a 4-PR stack. Merge bottom-up.
1. **This PR** (#3542) → `main`
2. #3543 → #3542 — `chat.agent` runtime + browser transport
3. #3545 → #3543 — agent-view dashboard
4. #3546 → #3545 — ai-chat reference + MCP tooling
Replaces #3173 (closed).
<!-- GitButler Footer Boundary Top -->
---
This is **part 5 of 5 in a stack** made with GitButler:
- <kbd> 5 </kbd> #3612
- <kbd> 4 </kbd> #3546
- <kbd> 3 </kbd> #3545
- <kbd> 2 </kbd> #3543
- <kbd> 1 </kbd> #3542👈
<!-- GitButler Footer Boundary Bottom -->
`tasks.trigger`, `tasks.batchTrigger`, `batch.create`,
`wait.createToken`, `wait.forDuration`, and the input/session stream
waitpoint endpoints all accept a caller-supplied `idempotencyKey` and
store it verbatim against a composite-unique index on `TaskRun`,
`BatchTaskRun`, or `Waitpoint`. The schemas had no length cap, so a
sufficiently long high-entropy key produced an index row larger than the
underlying storage layer can hold. The insert failed at the database,
and the caller saw a generic 500 from
`RunEngineTriggerTaskService.call()` / `CreateBatchService` / waitpoint
creation, depending on the endpoint.
Keys produced by `idempotencyKeys.create()` are 64-character SHA-256
hashes and never trip this — it only manifests for direct REST callers
(or SDK callers passing a raw string they generated themselves).
Low-entropy keys also sail through, because the storage layer compresses
repeated bytes before they reach the index, which is why the failure
mode is intermittent and tied to caller-side key shape.
## Fix
Add `.max(2048, "<field> must be 2048 characters or less")` to the seven
schemas that feed an indexed `idempotencyKey` column:
- `TriggerTaskRequestBody.options.idempotencyKey`
- `BatchTriggerTaskItem.options.idempotencyKey`
- `CreateBatchRequestBody.idempotencyKey`
- `CreateWaitpointTokenRequestBody.idempotencyKey`
- `CreateInputStreamWaitpointRequestBody.idempotencyKey`
- `CreateSessionStreamWaitpointRequestBody.idempotencyKey`
- `WaitForDurationRequestBody.idempotencyKey`
Plus the `idempotency-key` HTTP header on the trigger route (and the
three batch routes that re-export `HeadersSchema`). The header schema is
lifted out of `api.v1.tasks.$taskId.trigger.ts` into
`apps/webapp/app/v3/triggerHeaders.server.ts` so it can be exercised in
tests without dragging the route's import-time side effects.
The 2048 character ceiling is chosen to sit safely under the per-row
index limit while staying generous against existing callers — keys that
fit before still fit. Oversized keys now return a structured Zod 400
instead of a generic 500.
Limit is documented under `Idempotency key` in `docs/limits.mdx` and as
a `<Note>` on `docs/idempotency.mdx`.
## Test plan
- [x] 15 schema unit tests added
(`packages/core/src/v3/schemas/idempotencyKey.test.ts`,
`apps/webapp/test/routes/triggerHeaders.test.ts`) —
rejection-with-message + boundary acceptance for each capped schema. The
webapp test exercises the extracted `TriggerHeadersSchema` directly with
no mocks.
- [x] `pnpm run build --filter @trigger.dev/core`
- [x] `pnpm run typecheck --filter webapp`
- [x] End-to-end verified locally: baseline (small key) → 200; 3000-char
high-entropy header → 400 with the expected Zod error; same key at the
2048 boundary → 200; same key with the cap reverted → the database
rejected the insert and the route returned 500 to the caller. Cap
restored.
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds Sessions, a durable, run-aware stream primitive that scopes
session.in / session.out records to a session (not a single run).
Records survive run boundaries; reconnect-from-last-event-id is built in.
Server foundation:
- New /realtime/v1/sessions/:session/:io/append + /records routes
- sessionRunManager + sessionsRepository + clickhouseSessionsRepository
- mintRunToken for short-lived per-session tokens
- s2Append retry-with-backoff + undici cause diagnostics
- /api/v[12]/packets/* exempt from customer rate limits
- BackgroundWorker schema gains taskKind enum (TASK, AGENT, SCHEDULED)
- TaskRun.taskKind column + clickhouse 029_add_task_kind_to_task_runs_v2
Core types:
- new sessionStreams, inputStreams, realtimeStreams packages in @trigger.dev/core
- session-streams-api / realtime-streams-api surface
Sessions dashboard UI (the primitive's own viewer):
- /sessions index + detail routes
- SessionsTable, SessionFilters, SessionStatus, CloseSessionDialog
- AGENT/SCHEDULED filter in RunFilters + TaskTriggerSource
Includes the sessions-primitive changeset.
Switching between the Users and Organizations tabs in the admin
dashboard now keeps the current `?search=` value, so you can flip
between the two without re-typing your filter. Other admin tabs don't
take `search` and so don't carry it.
`pr_checks` runs the full matrix on every PR. #3609 touched only
`apps/webapp/app/routes/admin.tsx` and still ran the 4-job CLI e2e
matrix and 5-job sdk-compat suite.
Adds a `changes` job using `dorny/paths-filter` and gates each tier:
- webapp + e2e-webapp: `apps/webapp/**`, `packages/**`,
`internal-packages/**`
- packages: `packages/**`
- internal: `internal-packages/**` + `packages/**` (cross-deps)
- e2e (cli-v3): `packages/{cli-v3,build,core,schema-to-json}/**`
- sdk-compat: `packages/{trigger-sdk,core}/**`
`.configs/**`, `package.json`, `pnpm-lock.yaml`, `pnpm-workspace.yaml`,
`turbo.json` are also included in every filter since they affect the
whole workspace.
Inlines the `units` reusable-workflow children so each can be gated
independently (status check names also flatten from `units / webapp /
...` to `webapp / ...`). `unit-tests.yml` is unaffected - still used by
`publish.yml`.
Adds an `all-checks` gate that always runs and short-circuits to success
when every dependent is success-or-skipped. With this in place a single
required status check (`All PR Checks`) is enough; before this,
`paths-ignore` would have left required checks Pending on docs/changeset
PRs ([gh
docs](https://docs.github.com/en/actions/managing-workflow-runs/skipping-workflow-runs)).
## Summary
- Recommend deploying NodeLocal DNS and lowering `ndots` to `1` in the
Kubernetes self-hosting guide.
- Recommend storing task events in ClickHouse
(`EVENT_REPOSITORY_DEFAULT_STORE=clickhouse_v2`) in both the Docker and
Kubernetes guides, plus a new row in the webapp env var reference.
## Summary
- Users on production are hitting `QuotaExceededError: Failed to execute
'setItem' on 'Storage'` when navigating runs, because their localStorage
is full of orphaned `panel-group-react-aria<n>-:<rid>:` entries.
- Each entry is a session-unique key written by the resizable panel
library; they accumulated to thousands per user over the last two months
and now block legitimate `setItem` calls (the run-view inspector can no
longer persist its layout, and the page crashes mid-render).
- This PR evicts the legacy entries once on client boot. The leak itself
is already plugged by the v1.1.3 upgrade in #XXXX — this is the cleanup
that recovers the wasted quota on existing users' machines.
## Root cause (already fixed, for context)
In v0.4.1 of the underlying library, `PanelGroupImpl` defaulted
`autosaveStrategy` to `"localStorage"` unconditionally — so *every*
`PanelGroup` wrote to localStorage on every autosave trigger, including
the four in `QueryEditor`, the one in `ReplayRunDialog`, the storybook
routes, etc. Without an `autosaveId`, the key fell back to
`panel-group-${useId()}`, and React Aria's `useId()` produces a new
session-unique prefix each visit. Result: entries accumulated without
bound across sessions.
The condition was introduced when
[#3282](https://github.com/triggerdotdev/trigger.dev/pull/3282) removed
the wrapper's explicit `autosaveStrategy="cookie"` override (to fix HTTP
431 cookie-size errors). That worked, but the library default that took
over silently caused this leak.
The v1.1.3 upgrade in the resizable-panel PR changed the default to
`autosaveStrategy = autosaveId ? "localStorage" : undefined`, so no new
entries are being written. Existing residue still needs to be removed
from users' browsers.
## Changes
- New file
[`apps/webapp/app/clientBeforeFirstRender.ts`](apps/webapp/app/clientBeforeFirstRender.ts)
— exports a `clientBeforeFirstRender()` function that runs
synchronously, before React hydrates. Encapsulates a small cleanup
helper that scans `localStorage` and removes:
- Every key starting with `panel-group-react-aria` (the legacy
auto-generated keys).
- The orphan `panel-run-parent-v2` key from before the autosaveId v2→v3
bump.
- [`apps/webapp/app/entry.client.tsx`](apps/webapp/app/entry.client.tsx)
— imports and invokes `clientBeforeFirstRender()` once, before
`hydrateRoot()`. This guarantees the cleanup completes before any
`ResizablePanelGroup` mounts and tries to write.
The cleanup is wrapped in `try/catch` so private-browsing /
disabled-storage scenarios fail silently. Idempotent: subsequent loads
find no matching keys and exit immediately.
## Test plan
- [x] Locally seed ~50 fake `panel-group-react-aria…` entries plus a
`panel-run-parent-v2` entry via DevTools console, hard reload → legacy
entries gone, real entries (`panel-run-parent-v3`, `panel-run-tree`)
preserved.
- [x] Idempotency: reload a second time, no errors, no state changes.
- [x] Add a control entry (`panel-run-parent-v3-but-different-suffix`) —
confirmed not over-matched.
- [x] Simulate broken `Storage.setItem` throwing — page still renders,
cleanup swallows the error.
- [x] Typecheck clean.
## Notes
- Customer report: `QuotaExceededError: Failed to execute 'setItem' on
'Storage': Setting the value of 'panel-run-parent-v3' exceeded the
quota.`
- The cleanup runs once per page load. Once a user has loaded the app
after this deploys, their localStorage is clean and the function becomes
a no-op forever.
## Summary
Follow-up to #3561. The drift-audit workflow timed out on PR #3542 (92
files, +5962 lines) by hitting `--max-turns 15` before reaching a
verdict, leaving a red ❌ on that PR with no sticky comment.
## Changes
- `--max-turns` bumped from 15 to 30.
- Prompt now opens with an explicit "Strategy" section: read REVIEW.md
once, scan the file-list only, open at most 5 files (3-5 on PRs >50
files), and bias toward finishing over exploring.
- Final rule: *"when in doubt between one more file read and finish now
— finish now."*
The audit is allowed to miss things. It is not allowed to time out and
leave a red X.
## Test plan
- [ ] Verify this PR's audit posts `✅ REVIEW.md looks current for this
PR.` (small diff)
- [ ] After merge, retry the audit on #3542 or a similarly large PR and
confirm it completes
## Summary
Local ClickHouse was burning ~325% CPU endlessly merging its own
telemetry tables (`metric_log`, `asynchronous_metric_log`, `part_log`,
`trace_log`) after the container had been running long enough to
accumulate hundreds of GB of system-log data. OrbStack Helper reflected
this on the host (~400% CPU).
These tables are not used by anything in the dev stack. They only exist
for ClickHouse to log itself, so disabling them eliminates the merge
churn entirely.
## Changes
- Adds `docker/config/clickhouse-disable-system-logs.xml`, mounted into
`/etc/clickhouse-server/config.d/`, that removes the noisy system log
tables via `<table remove="1"/>`.
- Mounts the override file in `docker/docker-compose.yml`.
After applying, idle CPU dropped from 325% to ~12% on my machine.
## Test plan
- [ ] `pnpm run docker` brings up the stack cleanly
- [ ] `docker stats clickhouse` shows low idle CPU
- [ ] App functionality unaffected (system log tables are not queried by
the webapp)
## Summary
Queues that use concurrency keys can no longer bypass the per-queue
length cap, and the "Queued | Running" columns in the dashboard now show
the true total across all CK variants instead of 0.
The cap and the dashboard both relied on `ZCARD` of the base queue key,
but CK-keyed runs live under `<base>:ck:<variant>` keys. Any queue that
used concurrency keys read 0 — letting a single CK variant grow
unbounded past the user's configured cap.
## Fix
Two per-base-queue counters are maintained inside the CK Lua scripts:
`<base>:lengthCounter` and `<base>:runningCounter`. Non-CK
enqueue/dequeue paths are untouched.
Counters are lazy-initialized the first time a CK enqueue (or nack)
lands on a queue: the Lua script sums `ZCARD` across the variants
tracked by `ckIndex`, sets the counter, then `INCR`s. Pre-existing CK
backlog on already-populated queues is captured automatically — no batch
migration required.
`INCR`/`DECR` is gated on `ZADD`/`SADD` returning 1 (a new entry vs an
idempotent no-op), so duplicate enqueues or re-dequeues don't inflate
the counter.
The counter is `SET` with a 24-hour TTL on init. `INCR`/`DECR` do not
extend the TTL, so the counter expires daily and the next CK operation
re-seeds it from `ckIndex`. This bounds any drift that accumulates
during the rolling-deploy overlap window — where old (un-Tracked) and
new (Tracked) webapp instances briefly coexist — to ≤24 hours, with no
admin sweep or background reconciler needed.
Read paths pipeline `ZCARD`/`SCARD` on the base key + `GET` on the
counter and sum. A missing counter is treated as 0, so pure non-CK
queues see the same answer as before.
The counter-aware scripts ship alongside the originals with a `Tracked`
suffix for rolling-deploy safety; a follow-up PR will drop the originals
once this has rolled out.
## Test plan
- [ ] `pnpm run test --filter @internal/run-engine` — 116 tests pass,
including a new `ckCounters.test.ts` covering lazy init from
pre-existing backlog, churn, floor-at-zero, the non-CK regression case,
mixed CK + non-CK on the same base queue, idempotent re-enqueue
(ZADD-already-exists), 24h TTL on the counter, and nack re-seeding after
counter expiry.
- [ ] Verified end-to-end against a live local environment:
- Triggered 24 CK enqueues across 4 variants → `lengthCounter=16`,
`runningCounter=8`, dashboard showed Queued=16 / Running=8 for the CK
queue.
- Set the env queue cap to 16, triggered 12 more enqueues → 8 succeeded,
4 rejected with `QueueSizeLimitExceededError`.
- Deleted the counter on a queue with 31 messages already sitting in CK
variants, triggered one more enqueue → counter materialized to 31 from
the `ckIndex` sum, then INCR'd.
## Summary
Consolidates the webapp's authentication and authorization into a small
set of route helpers, replacing the ad-hoc `requireUser` /
`requireUserId` / `authenticatedEnvironmentForAuthentication` calls
scattered across routes. Same security model, but the per-request flow
(authenticate → authorize → load) now lives in one place per route
family.
Introduces a plugin seam (`@trigger.dev/plugins`) that lets the cloud
build install a richer RBAC implementation without touching webapp code.
The OSS fallback keeps the pre-RBAC permissive behaviour intact, so
self-hosted deployments work unchanged.
Adds a comprehensive end-to-end auth test suite that didn't exist before
— 193 `it()` blocks (vitest reports ~199 after `it.each` expansion)
covering API key, PAT and JWT auth across the public API surface, plus
dashboard session auth for admin pages.
## Changes
### Plugin contract — `@trigger.dev/plugins`
`RoleBaseAccessController` interface authoritative for both OSS
(fallback) and cloud (enterprise plugin):
- `authenticateBearer(request, { allowJWT? })` — API-key / public-JWT
auth, returns env + ability
- `authenticateSession(request, { userId, organizationId?, projectId?
})` — dashboard auth, caller resolves `userId` from the session cookie
and passes it in (no `helpers.getSessionUserId` callback — decouples the
plugin host from session-cookie code)
- `authenticatePat(request, { organizationId?, projectId? })` — PAT
auth, returns identity + `lastAccessedAt` so the host can throttle the
per-request update
- `authenticateAuthorize*` variants for the auth-and-check-in-one-call
cases
- `isUsingPlugin(): Promise<boolean>` — capability flag for UI /
branching where plugin-present-ness matters; replaces the
sentinel-string coupling that had `personalAccessToken.server` matching
`"RBAC plugin not installed"` literally
### Dashboard auth (started, partial rollout)
Admin and settings pages migrated to a unified `dashboardLoader` /
`dashboardAction` helper that authenticates the session, runs an
authorization check, and exposes the result to the route. Other
dashboard routes still on the old pattern; remaining migration tracked
in TRI-8730.
Migrated routes:
- `admin.*` (14 admin / back-office / feature-flags / LLM-models /
notifications / orgs / concurrency pages)
- `_app.orgs.$organizationSlug.settings.team`
- `_app.orgs.$organizationSlug.settings.roles`
### API / realtime / engine auth (complete for the migrated families)
71 routes migrated to a unified `apiBuilder` that centralizes Bearer /
PAT / Public-JWT authentication and applies the per-route authorization
check before the handler runs. Includes:
- `api.v1.*` and `api.v2.*` and `api.v3.*` — tasks, runs, batches,
queues, prompts, deployments, query, sessions, waitpoints, packets,
workers, idempotency keys
- `realtime.v1.*` — runs, batches, sessions, streams
- `engine.v1.*` — dev / worker-action protocols
29 routes still on the legacy `authenticateApiRequest*` helpers —
tracked as a post-deploy follow-up in TRI-9228.
Multi-resource auth direction is now explicit at the call site via
`anyResource(...)` (OR) and `everyResource(...)` (AND). Bare arrays no
longer typecheck — fixes a class of bug where a JWT scoped to one
resource could implicitly access others under OR semantics.
PAT auth path consolidated: was three DB queries per request (legacy
`authenticateApiRequestWithPersonalAccessToken` findFirst +
`rbac.authenticatePat` join + `lastAccessedAt` update). Now one query in
the steady state — plugin returns `lastAccessedAt`, host smart-skips the
update via JS-side throttle when fresh.
Side effect: action aliases preserved historic JWT scope semantics where
the new model is stricter (e.g. a `write:tasks` JWT now also satisfies
`trigger` / `batchTrigger` / `update` actions on the same resource —
matched at the auth boundary, not in the route handler).
### Backwards-compat fixes
The strict-match model regressed several real-world JWT shapes. Each
preserved via explicit `anyResource(...)` entries in the route's authz
block:
- **Batch retrieve routes** (`api.v1.batches.$batchId`, `api.v2.*`,
`realtime.v1.batches.*`) accept `read:runs` JWTs again (pre-RBAC
literal-match superScope behaviour)
- **Runs list routes** (`api.v1.runs`, `realtime.v1.runs`) accept
type-level `read:tasks` / `read:tags` on unfiltered queries (matched the
legacy `Object.keys` iteration semantic)
- **PAT/OAT auth shape** normalized through `toAuthenticated` so all
auth methods return the same slim `AuthenticatedEnvironment` (was:
API-key returned the slim shape but PAT/OAT returned raw Prisma
`Decimal` / no `orgMember`)
- **Scope `:` preservation** in resource ids — `read:tags:env:staging`
now correctly identifies the tag id as `env:staging`, not `env`
### Slim `AuthenticatedEnvironment`
Extracted to `@trigger.dev/core/v3/auth/environment` — a structural
shape independent of `@trigger.dev/database`. The plugin contract
returns this; webapp consumers import from there; the cloud plugin
(Drizzle) returns the same shape without Prisma's `Decimal` class
leaking into the public surface. Lets internal-packages (run-engine,
etc.) refer to `AuthenticatedEnvironment` without pulling Prisma in.
### Auth test suite (new — `*.e2e.full.test.ts`)
193 e2e tests run against a real spawned webapp + Postgres (no mocks).
Coverage matrix:
- **API key auth** — read / write / trigger / batchTrigger / deploy
actions across runs, batches, deployments, prompts, queues, query,
sessions, input-streams, waitpoints, tasks, idempotency keys; multi-key
resources (a run carries batch / tag / task identifiers — auth must
accept any matching scope)
- **Personal Access Token auth** — comprehensive matrix: scope match,
scope mismatch, missing scope, expired token, malformed token
- **Public JWT auth** — sub-vs-URL environment resolution, expired JWTs,
signature verification, scope checking, otu (one-time-use) token
semantics, branch-environment signing-key fallback
- **Dashboard session auth** — admin-only pages reject non-admins;
per-action gating
- **Cross-cutting edge cases** — revoked API key grace window, JWT
cross-environment isolation, MissingResource branch behaviour
### Hygiene cleanups
- Deleted dead `app/services/authorization.server.ts` (legacy
`checkAuthorization` + types — no live consumers post-migration) and its
orphaned test
- Dropped the never-populated `scopes` field from
`ApiAuthenticationResultSuccess`
- `scheduleEmail` moved out of `email.server.ts` into its own module —
breaks a `commonWorker → marqs/V1` import chain that was poisoning the
auth test graph
- OSS Roles page shows a deployment-aware empty state ("Roles aren't
available in this self-hosted deployment" vs the plan-upsell copy) via
`rbac.isUsingPlugin()`
- Team action handler: explicit per-intent ability gates
(`manage:billing` for purchase-seats, `manage:members` for set-role +
remove-member with self-leave carve-out)
### Cross-repo coordination
All public-package contract changes paired in `triggerdotdev/cloud#763`
(rbac-packages branch) — the enterprise plugin implements the same
`RoleBaseAccessController` interface against Drizzle.
## Test plan
- [x] `pnpm run typecheck --filter webapp` clean
- [x] `pnpm --filter webapp exec vitest run --config
vitest.e2e.full.config.ts` — 193/193 pass (requires Docker for
testcontainers)
- [x] Spot-check an authed API endpoint with a valid + invalid API key
against a local stack
- [x] Spot-check the migrated admin pages render and gate non-admins
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
## Summary
Adds `.claude/REVIEW.md` — a repo-specific source of truth for what AI /
agent code reviewers should treat as critical in this codebase
(rolling-deploy safety, hot-table indexes, recovery-path queries,
testcontainers usage, etc.). Pairs with a Claude-based PR audit that
flags drift between REVIEW.md and the code as it evolves.
## How the audit works
Mirrors the existing `.github/workflows/claude-md-audit.yml` pattern. On
non-draft, non-fork PRs that touch code, `anthropics/claude-code-action`
reads REVIEW.md, samples the PR diff, and posts a sticky comment with up
to 3 of:
- `[stale]` — rule cites a path / function / table that's been removed
or renamed
- `[contradiction]` — code in the PR violates a current rule
- `[missing]` — PR introduces a new pattern future reviewers should know
about
- `[obsolete]` — rule asserts a constraint the repo has moved past
If nothing's off, posts `✅ REVIEW.md looks current for this PR.`
## Test plan
- [ ] Convert this PR to ready-for-review, confirm the audit runs and
posts a sticky comment
- [ ] Verify the audit doesn't run on fork PRs (gated by
`head.repo.full_name == github.repository`)
- [ ] Verify suggestions are actionable on at least one follow-up PR
Closes
[TRI-9234](https://linear.app/triggerdotdev/issue/TRI-9234/retry-task-process-sigsegv-errors-respecting-user-retry-config)
## What this changes
SIGSEGV crashes (`TASK_PROCESS_SIGSEGV`) will now be **retried when an
attempt fails**, in line with the task's configured retry settings
(`retry.maxAttempts` etc.) — the same path SIGTERM and uncaught
exceptions already use. Previously SIGSEGV was hard-classified as
non-retriable and failed the run on the first segfault, ignoring the
user's retry policy.
Tasks without a retry policy still fail fast on the first SIGSEGV.
Behaviour is unchanged for OOM kills (separate machine-bump retry path)
and SIGKILL_TIMEOUT.
## Deploy
**Only the webapp needs to ship.** The retry decision lives entirely in
the webapp:
- V2 path: `internal-packages/run-engine` (bundled into the webapp)
- V1 path: `apps/webapp/app/v3/services/completeAttempt.server.ts`
No supervisor, CLI, SDK, or customer-task-image changes required.
Customers do not need to redeploy. The `@trigger.dev/core` changeset is
just keeping the public package in sync — the published npm version
isn't what makes the fix work.
## Why retry
SIGSEGV in Node tasks is frequently non-deterministic across processes:
- **Native addon races** (`sharp`, `canvas`, `better-sqlite3`,
`node-rdkafka`, `bcrypt`, …) — libuv thread-pool work stepping on V8
handles. Different heap layout / thread schedule on a fresh process →
retry often succeeds.
- **JIT / GC interaction** — V8 turbofan deopt or GC during a native
callback. Timing-dependent.
- **Near-OOM in native code** — when RSS approaches the cgroup limit,
native allocations fail and poorly-written addons dereference NULL →
SIGSEGV instead of clean OOM-kill.
- **Host / hardware issues** — bit flips, kernel quirks. Retry lands on
a different host.
The genuinely deterministic case (a user-code bug always tripping the
same addon) is real, but a subset — and `maxAttempts` bounds the damage.
## Pre-existing inconsistency this resolves
- `shouldRetryError` returned `false` for `TASK_PROCESS_SIGSEGV` →
`fail_run`.
- `shouldLookupRetrySettings` already listed `TASK_PROCESS_SIGSEGV` as
retry-config-aware — but that branch was unreachable because
`shouldRetryError` short-circuited first in `retrying.ts:86-90`.
- We already retry `TASK_RUN_UNCAUGHT_EXCEPTION` (clearly a user-code
bug) under the user's retry policy; refusing to retry SIGSEGV was the
odd one out.
## Test plan
- [x] `pnpm exec vitest run test/errors.test.ts` in `packages/core` —
26/26 pass (4 new)
- [x] `pnpm run build --filter @trigger.dev/core`
- [ ] CI green on PR
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
## Summary
1 improvement, 1 bug fix.
## Improvements
- Fail attempts on uncaught exceptions instead of hanging to
`MAX_DURATION_EXCEEDED`. A Node `EventEmitter` (e.g. `node-redis`)
emitting `"error"` with no `.on("error", ...)` listener escalates to
`uncaughtException`, which the worker previously reported but did not
act on — runs drifted to maxDuration with empty attempts. They now fail
fast with the original error and status `FAILED`, and respect the task's
normal retry policy. You should still attach `.on("error", ...)`
listeners to long-lived clients to handle errors gracefully.
([#3529](https://github.com/triggerdotdev/trigger.dev/pull/3529))
## Bug fixes
- Fix dev workers spinning at 100% CPU after the parent CLI disconnects.
Orphaned `trigger-dev-run-worker` (and indexer) processes were caught in
an `uncaughtException` feedback loop: a periodic IPC send via
`process.send` would throw `ERR_IPC_CHANNEL_CLOSED` once the parent
closed the channel, which re-entered the same handler that itself called
`process.send`, scheduled via `setImmediate` and amplified by
source-map-support's `prepareStackTrace`. Fixed by (1) silently dropping
packets in `ZodIpcConnection` when the channel is disconnected, (2)
adding a `process.on("disconnect", ...)` handler in dev workers so they
exit cleanly when the CLI closes the IPC channel, and (3) wrapping all
`uncaughtException`-path `process.send` calls in a `safeSend` guard that
checks `process.connected` and swallows synchronous throws.
([#3491](https://github.com/triggerdotdev/trigger.dev/pull/3491))
<details>
<summary>Raw changeset output</summary>
# Releases
## @trigger.dev/build@4.4.6
### Patch Changes
- Updated dependencies:
- `@trigger.dev/core@4.4.6`
## trigger.dev@4.4.6
### Patch Changes
- Fix dev workers spinning at 100% CPU after the parent CLI disconnects.
Orphaned `trigger-dev-run-worker` (and indexer) processes were caught in
an `uncaughtException` feedback loop: a periodic IPC send via
`process.send` would throw `ERR_IPC_CHANNEL_CLOSED` once the parent
closed the channel, which re-entered the same handler that itself called
`process.send`, scheduled via `setImmediate` and amplified by
source-map-support's `prepareStackTrace`. Fixed by (1) silently dropping
packets in `ZodIpcConnection` when the channel is disconnected, (2)
adding a `process.on("disconnect", ...)` handler in dev workers so they
exit cleanly when the CLI closes the IPC channel, and (3) wrapping all
`uncaughtException`-path `process.send` calls in a `safeSend` guard that
checks `process.connected` and swallows synchronous throws.
([#3491](https://github.com/triggerdotdev/trigger.dev/pull/3491))
- Fail attempts on uncaught exceptions instead of hanging to
`MAX_DURATION_EXCEEDED`. A Node `EventEmitter` (e.g. `node-redis`)
emitting `"error"` with no `.on("error", ...)` listener escalates to
`uncaughtException`, which the worker previously reported but did not
act on — runs drifted to maxDuration with empty attempts. They now fail
fast with the original error and status `FAILED`, and respect the task's
normal retry policy. You should still attach `.on("error", ...)`
listeners to long-lived clients to handle errors gracefully.
([#3529](https://github.com/triggerdotdev/trigger.dev/pull/3529))
- Updated dependencies:
- `@trigger.dev/core@4.4.6`
- `@trigger.dev/build@4.4.6`
- `@trigger.dev/schema-to-json@4.4.6`
## @trigger.dev/core@4.4.6
### Patch Changes
- Fix dev workers spinning at 100% CPU after the parent CLI disconnects.
Orphaned `trigger-dev-run-worker` (and indexer) processes were caught in
an `uncaughtException` feedback loop: a periodic IPC send via
`process.send` would throw `ERR_IPC_CHANNEL_CLOSED` once the parent
closed the channel, which re-entered the same handler that itself called
`process.send`, scheduled via `setImmediate` and amplified by
source-map-support's `prepareStackTrace`. Fixed by (1) silently dropping
packets in `ZodIpcConnection` when the channel is disconnected, (2)
adding a `process.on("disconnect", ...)` handler in dev workers so they
exit cleanly when the CLI closes the IPC channel, and (3) wrapping all
`uncaughtException`-path `process.send` calls in a `safeSend` guard that
checks `process.connected` and swallows synchronous throws.
([#3491](https://github.com/triggerdotdev/trigger.dev/pull/3491))
- Fail attempts on uncaught exceptions instead of hanging to
`MAX_DURATION_EXCEEDED`. A Node `EventEmitter` (e.g. `node-redis`)
emitting `"error"` with no `.on("error", ...)` listener escalates to
`uncaughtException`, which the worker previously reported but did not
act on — runs drifted to maxDuration with empty attempts. They now fail
fast with the original error and status `FAILED`, and respect the task's
normal retry policy. You should still attach `.on("error", ...)`
listeners to long-lived clients to handle errors gracefully.
([#3529](https://github.com/triggerdotdev/trigger.dev/pull/3529))
## @trigger.dev/python@4.4.6
### Patch Changes
- Updated dependencies:
- `@trigger.dev/core@4.4.6`
- `@trigger.dev/build@4.4.6`
- `@trigger.dev/sdk@4.4.6`
## @trigger.dev/react-hooks@4.4.6
### Patch Changes
- Updated dependencies:
- `@trigger.dev/core@4.4.6`
## @trigger.dev/redis-worker@4.4.6
### Patch Changes
- Updated dependencies:
- `@trigger.dev/core@4.4.6`
## @trigger.dev/rsc@4.4.6
### Patch Changes
- Updated dependencies:
- `@trigger.dev/core@4.4.6`
## @trigger.dev/schema-to-json@4.4.6
### Patch Changes
- Updated dependencies:
- `@trigger.dev/core@4.4.6`
## @trigger.dev/sdk@4.4.6
### Patch Changes
- Updated dependencies:
- `@trigger.dev/core@4.4.6`
</details>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
### Style updates to the notifications
- Tightened up the typography
- Brighter background to make it stand out a bit more
- A bit more padding to make it more readable
- Show the close button on hover instead
- Turned the notification into a separate component as it's shared on
the admin page modal
- Minor tweaks to the behavior of toggling the notification beween
open/closed side menu states
### Before
<img width="224" height="313" alt="before"
src="https://github.com/user-attachments/assets/c9a9377c-4a3b-4477-921a-3c86385d3f0b"
/>
### After (with image)
<img width="239" height="284" alt="CleanShot 2026-05-11 at 17 22 01"
src="https://github.com/user-attachments/assets/311b4dbc-4853-4e6c-9f83-8173b38bd466"
/>
### After (no image)
<img width="239" height="189" alt="after"
src="https://github.com/user-attachments/assets/884e062b-3608-4cb3-a462-d50597257753"
/>
---------
Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com>
## Summary
Make `taskIdentifier` optional on the run-queue message schema. No
behavior change in this PR; readers continue to accept payloads that
include the field. A separate change will stop writing it on the wire to
shrink the per-run payload that lives in Redis while runs wait to be
dequeued.
## Design
The field is written into every payload at enqueue time but no consumer
reads it back on the dequeue path. Both the run-engine and supervisor
derive `taskIdentifier` from the loaded `TaskRun` row instead. Relaxing
the schema first means readers tolerate payloads that omit it, so the
writer-side change can ship without producing schema-parse errors during
a rolling deploy.
`projectId` is left required: `WorkerQueueResolver.#getOverride` reads
it for project-scoped runtime worker-queue overrides.
## Test plan
- [x] `pnpm run typecheck --filter @internal/run-engine`
- [x] `pnpm run typecheck --filter webapp`
- [x] `pnpm run test ./src/run-queue/tests/enqueueMessage.test.ts
./src/run-queue/tests/workerQueueResolver.test.ts --run` (28/28 passing)
## Summary
TTL expiration on queued runs was being scheduled twice: once via a
per-run `expireRun` worker job (the original implementation) and once
via the batch TTL system (added more recently). Both paths attempt to
flip the same run to `EXPIRED`. The per-run job almost always won the
race, leaving the batch consumer to observe runs already expired by the
older path.
This collapses TTL expiration onto the batch path so every queued TTLed
run goes through a single Redis-backed sorted set + batch consumer
instead of also getting its own scheduled redis-worker job.
## Design
`engine.trigger` and `delayedRunSystem.enqueueDelayedRun` no longer call
`ttlSystem.scheduleExpireRun`. The remaining `enqueueSystem.enqueueRun({
includeTtl: true })` already adds the run to the TTL sorted set;
`TtlSystem.expireRunsBatch` flips it to `EXPIRED` when the TTL fires.
Delayed runs get the same coverage by passing `includeTtl: true` on
their post-delay enqueue, so the TTL is armed from the moment the run
enters the queue (matching how the old job behaved —
`parseNaturalLanguageDuration` is evaluated at enqueue time).
The new path explicitly does not re-expire runs once they have been
allocated a concurrency slot. That is intentional: TTL is for runs that
are queued and have never started. Once a run has a slot it is on its
way to executing.
## Test plan
- [x] `pnpm run test --filter @internal/run-engine
./src/engine/tests/ttl.test.ts` — 15 tests, including a new "Re-enqueued
runs are not expired by TTL once they have started" that locks in the
queued-and-never-started contract.
- [x] `pnpm run test --filter @internal/run-engine
./src/engine/tests/delays.test.ts` — 5 tests, including "Delayed run
with a ttl" which now also asserts the TTL is armed from queue-enter
time, not `createdAt`.
- [x] `pnpm run test --filter @internal/run-engine
./src/engine/tests/lazyWaitpoint.test.ts` — 12 tests.
- [x] `pnpm run typecheck --filter @internal/run-engine`.
## Summary
- Adds admin-only editors on the back-office org page for
`Organization.maximumProjectCount` and
`Organization.batchRateLimitConfig`, alongside the existing API rate
limit editor.
- Splits the back-office org page into per-section components
(`ApiRateLimitSection`, `BatchRateLimitSection`, `MaxProjectsSection`)
so each tool is self-contained — adding new sections later doesn't bloat
the route.
- Generalizes the rate-limit form into a reusable `RateLimitSection`
component + `RateLimitDomain` server config so API and batch share the
same UI, validation, and action handler. Each domain only owns its env
defaults, DB column, and logger key.
- "Saved." banner and validation errors are scoped to the section that
submitted, not the page.
Heads-up: the API rate-limit log key was renamed
`admin.backOffice.rateLimit` → `admin.backOffice.apiRateLimit` for
symmetry with the new `admin.backOffice.batchRateLimit`.
## Test plan
- [ ] As an admin, visit `/admin/back-office/orgs/:orgId` and confirm
all three sections render with the org's current values (or system
defaults).
- [ ] Edit and save each section; confirm only that section shows the
"Saved." banner.
- [ ] Submit invalid input (e.g. `0` tokens, malformed interval);
confirm errors render in the offending form only and the other sections
stay closed.
- [ ] Confirm a non-admin user is redirected away from the route.
- [ ] After saving a rate-limit override, hit the org with traffic and
confirm the new limit is enforced (API rate limit + batch rate limit
code paths read the column at request time).
## Summary
When ElastiCache demotes a primary to replica — during a Multi-AZ
failover or a vertical node-type change — the demoting primary issues an
`UNBLOCKED` reply to any in-flight blocking commands (`BLPOP`, `BRPOP`,
`BLMOVE`, `XREADGROUP ... BLOCK`, etc.) to clear them before the role
flips. ioredis surfaces these as `ReplyError` to caller code.
The shared `defaultReconnectOnError` added in #3548 only matches
`READONLY` and `LOADING`. This extends it to `UNBLOCKED` so the
disconnect-reconnect-retry cycle handles BLPOP-shaped errors the same
way the existing two cases handle non-blocking-command errors.
## Fix
```ts
export function defaultReconnectOnError(err: Error): boolean | 1 | 2 {
const msg = err.message ?? "";
if (
msg.startsWith("READONLY") ||
msg.startsWith("LOADING") ||
msg.startsWith("UNBLOCKED")
) {
return 2;
}
return false;
}
```
Returning `2` tells ioredis to disconnect, reconnect, and re-issue the
command. For a BLPOP that means a fresh BLPOP against the new primary
instead of the `UNBLOCKED` error escaping to the caller.
## Test plan
- [ ] CI green
- [ ] Trigger a Multi-AZ failover or a vertical scale event on an
ElastiCache replication group whose clients are running blocking
commands and confirm no `UNBLOCKED` errors surface to caller code during
the cutover.
## Summary
During an ElastiCache role swap (failover) or node-type change (vertical
scale), the ioredis TCP/TLS connection stays open but the server starts
answering with `READONLY` (the client is talking to a node that became a
replica) or `LOADING` (node still loading data from disk). Without an
explicit hook, those errors surface to caller code as `ReplyError`
instances — every write op on the affected connection fails until the
cluster fully cuts over.
This PR adds `reconnectOnError` to every prod ioredis client so the
disconnect + reconnect + retry cycle absorbs these errors and caller
code never sees them.
## Fix
```ts
export function defaultReconnectOnError(err: Error): boolean | 1 | 2 {
const msg = err.message ?? "";
if (msg.startsWith("READONLY") || msg.startsWith("LOADING")) return 2;
return false;
}
```
Returning `2` tells ioredis to disconnect, reconnect, and re-issue the
failed command. After reconnect, DNS / SG state routes the new socket to
a writable node.
The helper lives in `@internal/redis` and is wired into both the shared
`createRedisClient` (which covers RunQueue, schedule-engine,
redis-worker, and every other internal-package consumer) and the direct
`new Redis(...)` call sites in the webapp.
V1-only marqs files are intentionally not migrated.
## Test plan
- [x] `pnpm run typecheck --filter webapp`
- [x] `pnpm run typecheck --filter @internal/run-engine`
- [x] Verified end-to-end against a live ElastiCache vertical-scale
event — caller-surfaced errors went from tens of thousands during the
cutover window down to a handful per ioredis client
- [ ] Confirm steady-state behavior unchanged after deploy
## Summary
- Run-view inspector panel was glitching out on Firefox: visual flicker
on close, locking up at min size, and intermittent `panelHasSpace`
invariant errors. Root cause is the underlying `react-window-splitter`
library's collapse animation, which uses `@react-spring/rafz` and
interacts poorly with Firefox.
- Disabled the library's collapse animation on Firefox only, app-wide
(every consumer of `RESIZABLE_PANEL_ANIMATION`). Chromium and Safari
behaviour is unchanged.
## Changes
- **Firefox animation skip** in `RESIZABLE_PANEL_ANIMATION` —
UA-detected at module load, resolves to `undefined` for Firefox so the
library's animation actor completes in one frame instead of running its
rAF loop.
- **Inspector min raised 50px → 250px** so dragging can't shrink the
panel into a near-useless width.
- **`autosaveId` bumped `v2` → `v3`** to invalidate stale persisted
snapshots (the library has a `// TODO` branch that ignores prop changes
for already-registered panels, so existing users would otherwise still
see the old 50px min).
- **`react-window-splitter` pinned** to exact `0.4.1` to protect the
patch from drifting if line offsets change in a patch release.
- **Two hunks added to the existing `@window-splitter/state` patch:**
- Removed the library's auto-collapse-on-drag block entirely. Every
collapsible panel in the app is parent-controlled, and that block was
triggering state-machine deadlocks when handlers were no-ops.
Drag-to-collapse is now disabled across the app; collapse is only
triggered explicitly (close button, ESC, URL change, etc.).
- In `getDeltaForEvent`, fall back to the panel's `default` before its
`min` when expanding — so the first ever click on a span opens the
inspector at 500px, not 250px.
## Local testing confirmed
- [x] Firefox: open a run, click various spans → panel opens instantly
at 500px, drags freely between 250px and max, closes instantly to 0. No
console errors.
- [x] Chrome/Chromium: same flow, but with smooth open/close animation
as before.
- [x] Safari: same as Chrome.
- [x] Reload mid-session → panel restores cleanly to the dragged size.
- [x] Other resizable panels in the app (logs, deployments, schedules,
batches, bulk-actions, runs index) still animate on Chromium/Safari.
## Notes
- Linear: TRI-8584
- Branch contains intermediate commits exploring an unsuccessful
snapshot-validator approach; they're reverted by the final commit.
Cumulative diff is 6 files. Squash on merge if you'd prefer a clean
history.
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
## Summary
Adds a Redis pub/sub reload path to the webapp's in-memory LLM pricing
registry. When enabled on a process, the registry reloads from the
database whenever a publish lands on the configured channel — instead of
waiting for the existing 5-minute interval. Lets pricing/model changes
propagate to cost enrichment within seconds.
Subscription is **off by default** and opt-in per process. Only
OTel-ingesting services need real-time freshness; dashboard and worker
services run fine on the periodic interval and shouldn't pile onto each
publish with a full-table reload.
## Design
When `LLM_PRICING_RELOAD_PUBSUB_ENABLED=true`, subscribes via
`createRedisClient` against `COMMON_WORKER_REDIS_*` and listens on
`LLM_PRICING_RELOAD_CHANNEL` (default `llm-registry:reload`). The
5-minute periodic reload stays as a backstop, and a SIGTERM/SIGINT
handler closes the subscription cleanly.
The publisher side lives outside this PR — any process running in the
same Redis namespace can trigger a reload by `PUBLISH
llm-registry:reload <anything>`. Includes a `.server-changes/` note for
the changelog.
### Debounced reload
Bursts of publishes are coalesced. The first publish schedules a reload
at T+`LLM_PRICING_RELOAD_DEBOUNCE_MS` (default 1s); subsequent publishes
during that window are no-ops because the trailing reload picks up
everything when it queries the DB. Bounds reload rate to at most 1 per
debounce window regardless of publisher chattiness, so a runaway
upstream publisher can't fan out into a flood of full-table-scan
reloads.
## Test plan
- [ ] With `LLM_PRICING_RELOAD_PUBSUB_ENABLED=false` (default):
`redis-cli PUBSUB NUMSUB llm-registry:reload` returns `0` while the
webapp is up
- [ ] With it set to `true`: returns `>= 1`
- [ ] `redis-cli PUBLISH llm-registry:reload test` returns `1` (one
subscriber received) on a subscribed process
- [ ] Mutate an `LlmModel` row externally, publish on the channel,
observe the registry's match() picks up the change without waiting for
the 5-min tick
- [ ] Publish 100x in rapid succession; confirm only one reload fires
within the debounce window
When a webapp API route's catch-all 500 branch handles a non-typed
exception, it returns the raw `error.message` to the caller. If the
exception originates from an internal subsystem (the ORM client, an
infra dependency, etc.) the server-side error string is surfaced
verbatim in the response body — exposing implementation details the API
surface shouldn't carry.
The leak shows up in three shapes across the routes:
- `return json({ error: error.message }, { status: 500 })`
- `return json({ error: error instanceof Error ? error.message :
"Internal Server Error" }, { status: 500 })`
- ``return json({ error: `Internal server error: ${error.message}` }, {
status: 500 })``
(plus a couple of analogous neverthrow-Result variants on admin routes.)
## Fix
Across 19 webapp routes, replace each leaking branch with a generic body
(`"Something went wrong"` / `"Internal Server Error"` to match the
file's existing fallback) and add `logger.error(...)` so full visibility
is preserved server-side. Catch blocks that branch on typed user-input
errors (`ServiceValidationError`, `EngineServiceValidationError`,
`OutOfEntitlementError`, `PrismaClientKnownRequestError`) are left
intact — those messages are constructed deliberately and intended to be
customer-facing.
## Test plan
- [x] `pnpm run typecheck --filter webapp`
- [x] Per-route manual probe: inject a synthetic `Error` at the top of
the catch'd `try` block (or fake the wrapped call's rejection / Result
error), curl the route with the dev API key, confirm the response body
changed from the synthetic message verbatim → generic body. 21/21 leak
sites verified end-to-end.
- [x] 4xx-typed-error paths spot-checked: throwing
`ServiceValidationError` from inside the catch'd try still surfaces its
message at 422 as intended.
## Summary
Stamps the active OpenTelemetry `trace_id` and `span_id` onto every
Sentry event captured from the webapp, so engineers can copy a
`trace_id` from a Sentry issue and search for the corresponding trace in
any OTel-aware backend. Also adds an `otel_sampled` tag to indicate
whether the trace was head-sampled — a cheap signal for whether the link
will resolve to span data or hit a missing trace.
## Why
Sentry and OTel were OTel-disconnected: `apps/webapp/sentry.server.ts`
initialised Sentry with `skipOpenTelemetrySetup: true`, and no
error-capture site (`logger.server.ts`, the Remix-wrapped `handleError`,
the root `ErrorBoundary`) attached OTel context to the event. With many
spans/sec across services, getting from a Sentry issue to its trace was
guesswork.
## Approach
Single global Sentry event processor, registered immediately after
`Sentry.init`. On each event it reads
`trace.getActiveSpan()?.spanContext()` via `@opentelemetry/api`, then
writes:
- `event.contexts.trace.trace_id` and `event.contexts.trace.span_id`
(Sentry's native trace context fields)
- `event.tags.otel_sampled` = `"true"` | `"false"` (derived from
`traceFlags`)
If no active span (module-load errors, scheduled timers without a
context, primary cluster process), the processor returns the event
unmodified — Sentry's default propagation context fills in.
Implementation is co-located in `apps/webapp/sentry.server.ts` (no
separate helper module — `sentry.server.ts` is built standalone by
esbuild and a separate import would have required a new bundling step).
Helper functions are exported so the unit tests can reach them without
re-running `Sentry.init`.
## Non-goals (deliberate)
- No sample rate change. ~95% of Sentry events will carry a `trace_id`
that returns no spans in the tracing backend (head-sampled out). The
`otel_sampled` tag makes that obvious at a glance. Raising find-rate is
a separate conversation with cost trade-offs.
- No user/org tags or `Sentry.setUser` (would need auth-helper +
per-request scope wiring across multiple worker entrypoints — separate
ticket).
- Webapp image only. No changes to supervisor or CLI workers.
## Test plan
- [x] Unit tests in `apps/webapp/test/sentryTraceContext.server.test.ts`
— 9 tests covering: helper returns \`undefined\` with no active span;
returns \`traceId\`/\`spanId\`/\`sampled=true\` for a recording span;
returns \`sampled=false\` for a non-recording span; processor leaves the
event unchanged with no active span; processor stamps
\`trace_id\`/\`span_id\` onto \`contexts.trace\`; preserves existing
\`contexts.trace\` fields; tags \`otel_sampled\` correctly for both
sampled and non-sampled cases; never throws if \`@opentelemetry/api\`
access throws.
- [x] \`pnpm run typecheck --filter webapp\` passes.
- [x] Manually verified end-to-end against a sandboxed Sentry project:
confirmed both sampled and non-sampled traces correctly populate
\`contexts.trace.trace_id\` matching the OTel ids logged from the
loader, and the \`otel_sampled\` tag appears with the expected value.
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
## Summary
Both Claude Code workflows (`claude.yml` and `claude-md-audit.yml`)
authenticated via `CLAUDE_CODE_OAUTH_TOKEN`, which broke when the org
disabled Claude subscription access for Claude Code:
> Your organization has disabled Claude subscription access for Claude
Code · Use an Anthropic API key instead, or ask your admin to enable
access
This switches both workflows to `anthropic_api_key: ${{
secrets.ANTHROPIC_API_KEY }}` (secret already added to the repo).
## Test plan
- [ ] Confirm `📝 CLAUDE.md Audit` runs to completion on this PR
- [ ] Confirm `@claude` mention in a PR comment still triggers the
`Claude Code` workflow successfully
When a Node EventEmitter (e.g. node-redis) emits an "error" event with
no
listener attached, Node escalates it to process.on("uncaughtException")
in
the task worker. The worker reported the error via the
UNCAUGHT_EXCEPTION
IPC event but did not exit, and the supervisor-side handler in
taskRunProcess only logged the message at debug level — leaving the
run()
promise orphaned until maxDuration fired and producing empty attempts
(durationMs=0, costInCents=0).
The supervisor now rejects the in-flight attempt with an
UncaughtExceptionError and gracefully terminates the worker (preserving
the OTEL flush window) on UNCAUGHT_EXCEPTION. The attempt fails fast
with
TASK_EXECUTION_FAILED, surfacing the original error name, message, and
stack trace, and falls under the normal retry policy. This mirrors the
existing indexing-side behavior in indexWorkerManifest. Apply the same
handling to unhandled promise rejections, which Node already routes
through uncaughtException by default.
- Tags webapp images by full commit SHA on `main` pushes
(`ghcr.io/triggerdotdev/trigger.dev:<sha>`) so any commit can be
resolved to a digest easily.
- Adds OCI labels (`source`, `revision`, `version`, `created`) so
`docker inspect`, vulnerability scanners, and
registry browsers see source/commit/version directly.
- Signs each pushed digest with SLSA build provenance via
`actions/attest-build-provenance@v4.1.0` (pinned by SHA), enabling `gh
attestation verify oci://...` against the source commit and workflow.
Large deploys (projects with many tasks or source files) blocked the
webapp event loop for several seconds inside Prisma's client-side
serializer on `BackgroundWorker.create`, tail-latencying every other
in-flight request on the same Node process. The `metadata` JSON column
was being written with the full deploy manifest — every task's config,
every queue and prompt, and the full source of every file — all of which
already live on dedicated columns or in dedicated tables.
Fix: project the manifest to `{ packageVersion, contentHash, tasks: [{
id, filePath, schedule }] }` on insert. The only post-write read site is
`changeCurrentDeployment`, which feeds `tasks[].schedule` into
`syncDeclarativeSchedules` at deploy promotion. The retained top-level
keys and per-task `filePath` are kept solely so
`BackgroundWorkerMetadata.safeParse` still succeeds on read.
## Test plan
- [ ] Deploy a project with declarative schedules; verify schedules are
created on first deploy
- [ ] Modify / remove schedules across subsequent deploys; verify sync
- [ ] Roll back to a previous deploy; verify `changeCurrentDeployment`
re-syncs schedules
- [ ] Inspect `BackgroundWorker.metadata` on a fresh deploy — should be
a small object, not the full manifest
`dac9c83bd` added `ignoreErrors: /^ServiceValidationError(?::|$)/` in
`apps/webapp/sentry.server.ts` to drop SVEs before they reach Sentry.
The
filter only matches when the captured event's *type* is
`ServiceValidationError`, but nine call sites in the webapp catch SVE
(and
analogous user-input error types — `OutOfEntitlementError`,
`CreateDeclarativeScheduleError`, `QueryError`) and call
`logger.error("wrapper message", { error: e })` *before* the type check.
The captured event is then titled with the wrapper message, with the
inner
error buried in `extra.error` — invisible to the SDK filter. Result: a
steady stream of expected user-input failures escalating as
`error`-level
events when they should be `warn`.
Each catch block now type-discriminates first, logs expected types at
`warn`,
and keeps unknown-error fall-throughs at `error`. For service sites that
wrap into SVE (`createBackgroundWorker`,
`createDeploymentBackgroundWorkerV4`),
the inner error is logged at `error` before wrapping — mirrors the
`waitpointCompletionPacket.server.ts` pattern from `dac9c83bd`.
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Fixes#3520. The bundled bitnami clickhouse subchart was pinned at
`9.3.7` (clickhouse `25.6.1-debian-12-r0`), which hits a memory-tracker
accounting bug under sustained ingest - the global counter overflows to
~7 EiB and every query gets rejected by OvercommitTracker until the pod
is restarted. Self-hosters running 4.0.5 through 4.4.5 are exposed
regardless of chart version since the subchart pin hadn't moved.
Bumping to `9.4.4` (clickhouse `25.7.5-debian-12-r0`) pulls in the
25.7.x memory-tracker fixes. This is also the latest publicly packaged
release at `oci://registry-1.docker.io/bitnamicharts` - that registry
has been frozen since 2025-08-28 (Bitnami catalog changes), but the
chart source remains under Apache 2 on `bitnami/charts`. The image
continues to resolve via `bitnamilegacy/clickhouse` per the existing
`values.yaml` override, since `bitnami/clickhouse` itself moved to
paid-only.
Verified locally: `helm dependency update` + `helm lint` + `helm
template` + kubeconform across all 57 rendered manifests. Rendered
statefulset image is
`docker.io/bitnamilegacy/clickhouse:25.7.5-debian-12-r0`.
## Summary
Move from a single shared S2 basin to **per-org basins** with retention
tied to the org's billing plan. Stops S2 from deleting streams out from
under live chat sessions when basin retention fires before the chat
ends, and unlocks per-org cost attribution.
OSS / s2-lite installs are unaffected: provisioning is gated by
`REALTIME_STREAMS_PER_ORG_BASINS_ENABLED` (default `false`), and the
read precedence falls back to the global basin env var when an entity
has no stamped basin.
```
basin = run.streamBasinName ?? session.streamBasinName ?? env.REALTIME_STREAMS_S2_BASIN
```
## Design
Three nullable `streamBasinName` columns (`Organization`, `TaskRun`,
`Session`) plus a provisioner that idempotently creates the basin and
reconfigures retention on plan changes. The trigger and session-create
paths stamp the org's basin onto new rows; the realtime read path picks
the basin from the entity context.
Admin routes back-fill existing orgs and force-reconfigure a single org.
## Test plan
- [x] `pnpm run typecheck --filter webapp --filter @internal/run-engine`
- [x] Backfill admin route end-to-end (provision + DB stamp + S2 basin
config).
- [x] Reconfigure on plan change (all retention tiers).
- [x] chat.agent multi-turn drives streams into the per-org basin.
- [x] Legacy fallback when entity has no stamped basin.
- [x] Provisioner is a no-op when the flag is off.
Adds zizmor alongside the actionlint job from #3503. Both now run as
parallel jobs in a single `.github/workflows/workflow-checks.yml`,
triggered on `.github/workflows/**` and `.github/actions/**` changes.
Zizmor is configured with `unpinned-uses: hash-pin` policy via
`.github/zizmor.yml`, so any future unpinned action will fail CI.
Findings upload SARIF to the Security tab alongside CodeQL.
Bulk of the diff is cleanup of the findings zizmor surfaced on first
run. `zizmor --fix=all` handled most of them mechanically; the rest were
judgment calls.
<img width="2284" height="2028" alt="CleanShot 2026-05-01 at 18 53
50@2x"
src="https://github.com/user-attachments/assets/4f58cbb1-0168-40fb-a523-017f2ba625a1"
/>
## Performance
- **Per-request DB hit**: `getUserId` runs `getEffectiveSessionDuration`
(User lookup + Org `aggregate`) on *every* authenticated request,
including each fetcher poll. Consider caching the effective duration in
the session cookie with a short TTL (e.g. 60s) and revalidating in the
background.
- **Double session commit in `root.tsx`**: `getUser` already runs the
expiry check; then `commitAuthenticatedSessionLazy` commits the cookie
again. Fine, but doubles `Set-Cookie` headers on every page load — worth
a quick perf check.
## Correctness / Edge cases
- **Lazy backfill assumes a root.tsx hit first**: users whose first
post-deploy request is a fetcher/API route (`/resources/*`) skip the
backfill until they navigate to a page. Not a security hole, but
`getUserId` could backfill itself for completeness.
- **No upper bound on `Organization.maxSessionDuration`**: admin API
accepts `1` second, which would instant-logout every member on next
request. Add a `min(60)` (or `min(300)` to match the lowest user option)
to the Zod schema.
- **No clock-skew tolerance**: `isSessionExpired` is exact-millisecond.
Multi-instance deploys with skewed clocks could log users out a few
seconds early/late. Probably fine for the 5-min minimum, but worth
noting.
## Security
- **Auto-logout audit log lacks IP/orgId**: HIPAA forensics typically
wants source IP and which org context. Currently logs only `userId` +
path. IP isn't PII for audit purposes; orgIds help correlate. Add both.
- **Cookie `Max-Age` is 1 year regardless of user's setting**:
intentional (server-side `issuedAt` is the source of truth), but
reviewers will ask. Add a one-line comment on the cookie config
explaining why.
## API surface
- **`maxSessionDuration` is admin-PAT only**: no in-app UI for org
owners to set/change their own cap. If this is "Trigger staff sets it
during HIPAA onboarding", say so in the PR description; otherwise add an
org-settings UI.
- **Auto-submit dropdown has no confirmation**: misclicking "5 minutes"
immediately shortens the user's session window with no undo. Consider a
save button or 3-sec undo toast.
## Schema / migration
- **`User.sessionDuration NOT NULL DEFAULT 31556952`**: instant on PG
11+ (metadata-only), but call out in the PR description so reviewers
don't worry about a table rewrite on the User table.
- **No DB-level constraint matching `SESSION_DURATION_OPTIONS`**: if the
option list changes, existing users keep orphaned values. The dropdown's
tag-along behaviour hides this — fine for now, but if you ever drop an
option you'll need a backfill.
## UX
- **Session expiry only fires on next request**: an idle authenticated
tab keeps showing UI past the cap (until SSE/polling catches it, ~60s).
Add a client-side timer based on the user's effective duration that
triggers a fetcher to `/account` or `/logout` at expiry.
- **No "you were signed out" message on logout**: users hitting their
cap are bounced to `/` with no explanation. Was intentionally reverted
in this PR — call that out so reviewers don't request it.
## Tests
- Unit coverage on `sessionDuration.server.ts` is solid (215 lines).
Missing: integration test for `getUserId` → expired session → redirect
to `/logout`, and one for the loader's clamping fix (the most recent
bug). Add at least the second one to lock in the regression.
---------
Co-authored-by: Matt Aitken <matt@mattaitken.com>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Reported by external contributor. The supervisor template hardcoded a
short DNS name for `OTEL_EXPORTER_OTLP_ENDPOINT`, which the supervisor
then propagates verbatim into runner pods
(`apps/supervisor/src/workloadManager/kubernetes.ts:196`). When runners
are spawned in a different namespace via
`supervisor.config.kubernetes.namespace`, the short name doesn't resolve
and span/log export silently fails - runs complete fine but the
dashboard shows nothing.
Same FQDN pattern the chart already uses for
`TRIGGER_WORKLOAD_API_DOMAIN` (line 203). Verified with `helm template
trigger . --namespace my-ns` - renders
`http://trigger-webapp.my-ns.svc.cluster.local:3030/otel`.
Cheers Niels
## Summary
Delete 34 `.server-changes/*.md` files that should have been cleaned up
automatically when v4.4.5 (#3406) was merged but were stranded by a
workflow race.
## Why these are stale
The `update-lockfile` job in `.github/workflows/changesets-pr.yml` is
what cleans up consumed `.server-changes/*.md` files on the release
branch. When v4.4.5 was merged on 2026-05-01, the post-merge workflow
run on `main` failed at `pnpm install --frozen-lockfile` (stale lockfile
in the merge commit), and `cancel-in-progress: true` cancelled the
in-flight run from the previous push — so `update-lockfile` never
reached the cleanup step.
Result: the 34 files described changes that v4.4.5 already shipped, and
they were re-appearing in the v4.4.6 release PR (#3501) under "Server
changes" plus showing up as deletions in its diff.
## What this PR keeps
- `fix-rollback-schedule-sync.md` — genuinely new for v4.4.6 (#3468),
the only server change introduced after v4.4.5
- `README.md`, `.gitkeep` — directory infrastructure
- `dev-cli-disconnect-md` — leaving alone (typo'd filename from March,
no `.md` extension, not picked up by the cleanup glob anyway)
## After merge
The next run of `changesets-pr.yml` will refresh #3501 with a "Server
changes" section that only lists the v4.4.6 entry, and the only
`.server-changes/` deletion in its diff will be
`fix-rollback-schedule-sync.md`.
## Related
- #3505 is the proper underlying fix — collapses the three-job graph
into a single atomic commit by `changesets/action` so this race can't
strand the cleanup again. This PR is just the one-time catch-up for the
files that already got stranded.
Follow-up to the v4.4.5 release incident where the release PR (#3406)
was merged with a stale lockfile and stale Chart.yaml, breaking npm +
helm releases. The two automation jobs (`update-lockfile`,
`bump-chart-version`) got cancelled mid-flight by `cancel-in-progress`
when the merge fired the workflow again on `main`.
This restructures `changeset:version` so all the post-version-bump
fixups happen in the same script and end up in a single atomic commit on
`changeset-release/main`, via `changesets/action`'s normal commit step.
Pattern borrowed from Cloudflare workers-sdk, Astro, shadcn/ui.
## Before
```
push: main
└── release-pr (changeset version → bumps package.jsons, opens PR)
└── update-lockfile (separate job, separate commit)
└── bump-chart-version (separate job, separate commit)
```
Three jobs, three commits to the release branch.
## After
```
push: main
└── release-pr
└── changesets/action runs:
changeset version
pnpm install --lockfile-only
node scripts/bump-helm-chart.mjs
node scripts/cleanup-server-changes.mjs
...all staged and committed as ONE commit by the action
```
One job, one commit.
Adds an `actionlint` job that runs on changes to `.github/workflows/**`
and `.github/actions/**`. Catches workflow bugs at PR time — expression
typos, deprecated runner labels, broken matrices, and shellcheck issues
in `run:` blocks.
Run from the official `docker://rhysd/actionlint` image, digest-pinned
alongside everything else.
Existing workflows had 6 shellcheck findings, all fixed.
Updates the compute private beta page with the May 1 release entry, plus
a deploy-time warning when `us-east-1-next` is the project default.
The new What's new entry, verbatim:
### May 1, 2026
- **Cold starts are faster across all machine sizes.** Every preset
starts faster, including `micro` and `small-1x` - there's no longer a
cold-start penalty for picking a smaller machine.
- **First runs after a deploy are faster on every preset.** Boot
snapshot creation is significantly quicker across the board, so the cold
path is consistently snappier.
- **`large-1x` and `large-2x` no longer hard-fail.** They're still not
recommended - cold-start performance trails the smaller presets and
we're ironing out reliability issues.
Follow-up to #3472 and #3479.
Orphaned `trigger-dev-run-worker` processes were pinning CPU at 100%
after the dev CLI exited — stuck in an uncaughtException feedback loop
where a closed IPC channel kept throwing `ERR_IPC_CHANNEL_CLOSED` back
into a handler that itself called `process.send`.
Fix:
- `ZodIpcConnection` no-ops sends when the channel is disconnected.
- Dev workers exit on `process.disconnect` instead of being re-parented
to init.
- All worker `uncaughtException` handlers route through a `safeSend`
guard so the handler can never re-enter itself.
Verified end-to-end: `kill -9` of the dev CLI now cleans up all child
workers within ~2s.
Last action still firing the Node 20 deprecation warning after #3494.
`buildjet/setup-node@v4.0.4` (the latest tag) declares `runs: using:
'node20'` and the repo hasn't shipped a node24 update.
Workflows here run on `ubuntu-latest` (not buildjet runners), so the
buildjet fork wasn't giving us anything we don't get from
`actions/setup-node` directly. Swapping to `actions/setup-node@v6.4.0`
(node24 runtime) silences the warning.
## Summary
Each successful PAT (`PersonalAccessToken`) or OAT
(`OrganizationAccessToken`) authentication issues a `prisma.X.update({
lastAccessedAt: new Date() })` to bump the timestamp. For tokens used at
high frequency (CLI clients, integrations) this generates a per-request
DB write that is mostly redundant — the `lastAccessedAt` field is only
surfaced on the settings page so users can decide which tokens to
revoke, and "within the last 5 minutes" is plenty of granularity for
that.
## Design
Replace each unconditional `update` with a conditional `updateMany`
whose `WHERE` requires the existing `lastAccessedAt` to be `NULL` or
strictly older than 5 minutes:
```ts
await prisma.personalAccessToken.updateMany({
where: {
id: personalAccessToken.id,
OR: [
{ lastAccessedAt: null },
{ lastAccessedAt: { lt: new Date(Date.now() - PAT_LAST_ACCESSED_THROTTLE_MS) } },
],
},
data: { lastAccessedAt: new Date() },
});
```
The conditional runs inside the SQL `UPDATE`, so concurrent auths can't
race into a double-write.
No schema change. No migration. No new infrastructure. Throttle is a
hardcoded constant (`5 * 60 * 1000`) — easy to revisit.
## Test plan
- [x] `pnpm run typecheck --filter webapp`
- [x] `pnpm vitest run ./test/services/personalAccessToken.test.ts
./test/services/organizationAccessToken.test.ts` — 6/6 pass, verifying
the throttle `WHERE` clause is constructed correctly and the `update` is
skipped on token-not-found / wrong-prefix paths
Follow-up to v4.4.5 release. The `bump-chart-version` job on the release
PR was cancelled before it could run, so Chart.yaml was merged still
pointing at 4.4.4. The helm release job ([failed
run](https://github.com/triggerdotdev/trigger.dev/actions/runs/25218553990/job/73947054128))
caught it via its version-match guard.
Once this merges I'll re-run the helm release workflow manually.
## ✅ Checklist
- [x] I have followed every step in the [contributing
guide](https://github.com/triggerdotdev/trigger.dev/blob/main/CONTRIBUTING.md)
- [x] The PR title follows the convention.
- [x] I ran and tested the code works
---
## Testing
- Reviewed the code flow for deployment rollback
(`ChangeCurrentDeploymentService`) and confirmed it was missing schedule
sync
- Verified all 4 callers of `ChangeCurrentDeploymentService` (UI
rollback, UI promote, API promote, finalize deployment) are now covered
- Ran `pnpm run typecheck --filter webapp` — passes cleanly
---
## Changelog
When rolling back (or manually promoting) a deployment, declarative
schedules were not being synced to match the target deployment's worker
metadata. Schedules remained as configured by the most recent deployment
rather than reflecting the target version's schedule configuration.
This fix adds a call to `syncDeclarativeSchedules` in
`ChangeCurrentDeploymentService` after the deployment promotion is
updated. It parses the target deployment's stored
`BackgroundWorkerMetadata` to restore the correct schedule state. This
covers both rollback and promote paths (UI and API). Errors are handled
gracefully so they don't block the deployment change itself.
---
## Screenshots
N/A — backend-only change.
💯
Link to Devin session:
https://app.devin.ai/sessions/0debf012b58c4132be778f8ea88cd2b6
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: nick <55853254+nicktrn@users.noreply.github.com>
## Summary
The v4.4.5 release PR (#3406) was merged before the automated
lockfile-update job in
[\`changesets-pr.yml\`](.github/workflows/changesets-pr.yml) could push
its commit. As a result main now has \`package.json\` bumped to
\`4.4.5\` but \`pnpm-lock.yaml\` still pinned to \`4.4.4\`.
This blocks every subsequent \`pnpm install --frozen-lockfile\` run,
including:
- \`release.yml\` for v4.4.5 publish ([run
#25217579660](https://github.com/triggerdotdev/trigger.dev/actions/runs/25217579660))
— never published packages to npm
- \`changesets-pr.yml\` on the next push to main ([run
#25217579645](https://github.com/triggerdotdev/trigger.dev/actions/runs/25217579645))
## Root cause (from CI logs)
\`\`\`
ERR_PNPM_OUTDATED_LOCKFILE Cannot install with "frozen-lockfile" because
pnpm-lock.yaml is not up to date with <ROOT>/packages/build/package.json
- @trigger.dev/core (lockfile: workspace:4.4.4, manifest:
workspace:4.4.5)
\`\`\`
Regenerated via \`pnpm install --lockfile-only\` against current main.
The diff is exactly what the canceled \`update-lockfile\` job would have
produced:
- 12 \`workspace:4.4.4\` → \`workspace:4.4.5\` specifier bumps
- pnpm metadata refresh (deprecation annotations on transitive deps, one
optional \`bufferutil\` peer resolution on \`react-email\`)
No new direct dependencies, no version drops.
## Follow-ups (separate PRs)
1. **Re-run release.yml** via \`workflow_dispatch\` (\`type: release\`,
\`ref\` = merge commit on main once this lands) to actually publish
4.4.5 to npm.
2. **Workflow fix** to prevent recurrence: fold the lockfile update into
\`changeset:version\` so the \`release-pr\` job creates a single commit
with version bumps + lockfile in sync. Removes the race window where the
release PR is mergeable before \`update-lockfile\` runs.
Dependabot's first auto-bump PR (#3495) was auto-closed because
`dependabot[bot]` isn't in the vouch list and isn't exempt from the
require-draft check.
Two changes:
- Add `dependabot[bot]` to `.github/VOUCHED.td` so the vouch check
passes.
- Add `dependabot[bot]` to the require-draft exception in
`vouch-check-pr.yml` (alongside `devin-ai-integration[bot]`) so its PRs
aren't closed for being non-draft.
Without both, dependabot bumps will keep getting closed and we lose the
weekly action update flow that #3494 set up.
## Summary
8 new features, 18 improvements, 11 bug fixes.
## Breaking changes
- Add server-side deprecation gate for deploys from v3 CLI versions
(gated by `DEPRECATE_V3_CLI_DEPLOYS_ENABLED`). v4 CLI deploys are
unaffected.
([#3415](https://github.com/triggerdotdev/trigger.dev/pull/3415))
## Improvements
- Add `--no-browser` flag to `init` and `login` to skip auto-opening the
browser during authentication. Also error loudly when `init` is run
without `--yes` under non-TTY stdin (previously default-and-exited
silently, leaving the project half-initialized). Both commands now show
an `Examples` section in `--help`.
([#3483](https://github.com/triggerdotdev/trigger.dev/pull/3483))
- Add `isReplay` boolean to the run context (`ctx.run.isReplay`),
derived from the existing `replayedFromTaskRunFriendlyId` database
field. Defaults to `false` for backwards compatibility.
([#3454](https://github.com/triggerdotdev/trigger.dev/pull/3454))
- Redact the `resolveWaitpoint` runtime log so it only emits `id` and
`type` instead of the full completed waitpoint. Previously the log
printed the entire waitpoint (including `output`) to stdout in
production runs, which could leak sensitive payloads. The value returned
by `wait.forToken()` is unchanged.
([#3490](https://github.com/triggerdotdev/trigger.dev/pull/3490))
- Add `SessionId` friendly ID generator and schemas for the new durable
Session primitive. Exported from `@trigger.dev/core/v3/isomorphic`
alongside `RunId`, `BatchId`, etc. Ships the
`CreateSessionStreamWaitpoint` request/response schemas alongside the
main Session CRUD.
([#3417](https://github.com/triggerdotdev/trigger.dev/pull/3417))
- Truncate large error stacks and messages to prevent OOM crashes. Stack
traces are capped at 50 frames (keeping top 5 + bottom 45 with an
omission notice), individual stack lines at 1024 chars, and error
messages at 1000 chars. Applied in parseError, sanitizeError, and OTel
span recording.
([#3405](https://github.com/triggerdotdev/trigger.dev/pull/3405))
## Server changes
These changes affect the self-hosted Docker image and Trigger.dev Cloud:
- Add a "Back office" tab to `/admin` and a per-organization detail page
at `/admin/back-office/orgs/:orgId`. The first action available on that
page is editing the org's API rate limit: admins can save a
`tokenBucket` override (refill rate, interval, max tokens) and see a
plain-English preview of the resulting sustained rate and burst
allowance. Writes are audit-logged via the server logger.
([#3434](https://github.com/triggerdotdev/trigger.dev/pull/3434))
- Optional `DEPLOY_REGISTRY_ECR_DEFAULT_REPOSITORY_POLICY` env var to
apply a default repository policy when the webapp creates new ECR repos
([#3467](https://github.com/triggerdotdev/trigger.dev/pull/3467))
- Ship the Errors page to all users, with a polish + bug-fix pass:
pinned "No channel" item in the Slack alert channel picker,
viewer-timezone alert timestamps via Slack's `<!date^>` token, Activity
sparkline peak tooltip, centered loading spinner and bug-icon empty
state on the error detail page, ellipsis on the Configure alerts
trigger.
([#3477](https://github.com/triggerdotdev/trigger.dev/pull/3477))
- Configure the set of machine presets to build boot snapshots for at
deploy time via `COMPUTE_TEMPLATE_MACHINE_PRESETS` (CSV of preset names,
default `small-1x`). Use `COMPUTE_TEMPLATE_MACHINE_PRESETS_REQUIRED`
(CSV, default = full PRESETS list) to scope which preset failures fail a
required-mode deploy. Optional preset failures are logged and don't
block the deploy.
([#3492](https://github.com/triggerdotdev/trigger.dev/pull/3492))
- Regenerating a RuntimeEnvironment API key no longer invalidates the
previous key immediately. The old key is recorded in a new
`RevokedApiKey` table with a 24 hour grace window, and
`findEnvironmentByApiKey` falls back to it when the submitted key
doesn't match any live environment. The grace window can be ended early
(or extended) by updating `expiresAt` on the row.
([#3420](https://github.com/triggerdotdev/trigger.dev/pull/3420))
- Add the `Session` primitive — a durable, task-bound, bidirectional I/O
channel that outlives a single run and acts as the run manager for
`chat.agent`. Ships the Postgres `Session` + `SessionRun` tables,
ClickHouse `sessions_v1` + replication service, the `sessions` JWT
scope, and the public CRUD + realtime routes (`/api/v1/sessions`,
`/realtime/v1/sessions/:session/:io`) including `end-and-continue` for
server-orchestrated run handoffs and session-stream waitpoints.
([#3417](https://github.com/triggerdotdev/trigger.dev/pull/3417))
- Add `KUBERNETES_POD_DNS_NDOTS_OVERRIDE_ENABLED` flag (off by default)
that overrides the cluster default and sets `dnsConfig.options.ndots` on
runner pods (defaulting to 2, configurable via
`KUBERNETES_POD_DNS_NDOTS`). Kubernetes defaults pods to `ndots: 5`, so
any name with fewer than 5 dots — including typical external domains
like `api.example.com` — is first walked through every entry in the
cluster search list (`<ns>.svc.cluster.local`, `svc.cluster.local`,
`cluster.local`) before being tried as-is, turning one resolution into
4+ CoreDNS queries (×2 with A+AAAA). Using a lower `ndots` value reduces
DNS query amplification in the `cluster.local` zone.
Note: before enabling, make sure no code path relies on search-list
expansion for names with dots ≥ the configured value — those names will
hit their as-is form first and could resolve externally before falling
back to the cluster search path.
([#3441](https://github.com/triggerdotdev/trigger.dev/pull/3441))
- Vercel integration option to disable auto promotions
([#3376](https://github.com/triggerdotdev/trigger.dev/pull/3376))
- Make it clear in the admin that feature flags are global and should
rarely be changed.
([#3408](https://github.com/triggerdotdev/trigger.dev/pull/3408))
- Admin worker groups API: add GET loader and expose more fields on
POST. ([#3390](https://github.com/triggerdotdev/trigger.dev/pull/3390))
- Add 60s fresh / 60s stale SWR cache to `getEntitlement` in
`platform.v3.server.ts`. Eliminates a synchronous billing-service HTTP
round trip on every trigger. Reuses the existing `platformCache` (LRU
memory + Redis) pattern already used for `limits` and `usage`. Cache key
is `${orgId}`. Errors return a permissive `{ hasAccess: true }` fallback
(existing behavior) and are also cached to prevent thundering-herd on
billing outages.
([#3388](https://github.com/triggerdotdev/trigger.dev/pull/3388))
- Show a `MicroVM` badge next to the region name on the regions page.
([#3407](https://github.com/triggerdotdev/trigger.dev/pull/3407))
- Increase default maximum project count per organization from 10 to 25
([#3409](https://github.com/triggerdotdev/trigger.dev/pull/3409))
- Merge execution snapshot creation into the dequeue taskRun.update
transaction, reducing 2 DB commits to 1 per dequeue operation
([#3395](https://github.com/triggerdotdev/trigger.dev/pull/3395))
- Add per-worker Node.js heap metrics to the OTel meter —
`nodejs.memory.heap.used`, `nodejs.memory.heap.total`,
`nodejs.memory.heap.limit`, `nodejs.memory.external`,
`nodejs.memory.array_buffers`, `nodejs.memory.rss`. Host-metrics only
publishes RSS, which overstates V8 heap by the external + native
footprint; these give direct heap visibility per cluster worker so
`NODE_MAX_OLD_SPACE_SIZE` can be sized against observed heap peaks
rather than RSS.
([#3437](https://github.com/triggerdotdev/trigger.dev/pull/3437))
- Tag Prisma spans with `db.datasource: "writer" | "replica"` so
monitors and trace queries can distinguish the writer pool from the
replica pool. Applies to all `prisma:engine:*` spans (including
`prisma:engine:connection` used by the connection-pool monitors) and the
outer `prisma:client:operation` span.
([#3422](https://github.com/triggerdotdev/trigger.dev/pull/3422))
- Clarify the cross-region intent in the Terraform and AI-prompt helpers
on the Add Private Connection page. Both already default
`supported_regions` to `["us-east-1", "eu-central-1"]`; added an inline
comment / parenthetical so the user understands why both regions are
listed (Trigger.dev runs in both, so the service must be consumable from
either).
([#3465](https://github.com/triggerdotdev/trigger.dev/pull/3465))
- Add `RUN_ENGINE_READ_REPLICA_SNAPSHOTS_SINCE_ENABLED` flag (default
off) to route the Prisma reads inside `RunEngine.getSnapshotsSince`
through the read-only replica client. Offloads the snapshot polling
queries (fired by every running task runner) from the primary. When
disabled, behavior is unchanged.
([#3423](https://github.com/triggerdotdev/trigger.dev/pull/3423))
- Stop creating TaskRunTag records and _TaskRunToTaskRunTag join table
entries during task triggering. The denormalized runTags string array on
TaskRun already stores tag names, making the M2M relation redundant
write overhead.
([#3369](https://github.com/triggerdotdev/trigger.dev/pull/3369))
- Stop writing per-tick state (`lastScheduledTimestamp`,
`nextScheduledTimestamp`, `lastRunTriggeredAt`) on `TaskSchedule` and
`TaskScheduleInstance`. The schedule engine now carries the previous
fire time forward via the worker queue payload, eliminating ~270K
dead-tuple-driven autovacuums per year on these hot tables and the
associated `IO:XactSync` mini-spikes on the writer. Customer-facing
`payload.lastTimestamp` semantics are unchanged.
([#3476](https://github.com/triggerdotdev/trigger.dev/pull/3476))
- Replace the expensive DISTINCT query for task filter dropdowns with a
dedicated TaskIdentifier registry table backed by Redis. Environments
migrate automatically on their next deploy, with a transparent fallback
to the legacy query for unmigrated environments. Also fixes duplicate
dropdown entries when a task changes trigger source, and adds
active/archived grouping for removed tasks. Moves BackgroundWorkerTask
reads in the trigger hot path to the read replica.
([#3368](https://github.com/triggerdotdev/trigger.dev/pull/3368))
- Public Access Tokens (PATs) minted before an API key rotation now keep
working during the 24h grace window. `validatePublicJwtKey` falls back
to any non-expired `RevokedApiKey` rows for the signing environment when
the primary signature check against the env's current `apiKey` fails.
The fallback query only runs on the failure path, so the hot success
path is unchanged.
([#3464](https://github.com/triggerdotdev/trigger.dev/pull/3464))
- Batch items that hit the environment queue size limit now fast-fail
without
retries and without creating pre-failed TaskRuns.
([#3352](https://github.com/triggerdotdev/trigger.dev/pull/3352))
- Show the cancel button in the runs list for runs in `DEQUEUED` status.
`DEQUEUED` was missing from `NON_FINAL_RUN_STATUSES` so the list hid the
button even though the single run page allowed it.
([#3421](https://github.com/triggerdotdev/trigger.dev/pull/3421))
- Reduce 5xx feedback loops on hot debounce keys by quantizing
`delayUntil`,
adding an unlocked fast-path skip, and gracefully handling redlock
contention in `handleDebounce` so the SDK no longer retries into a herd.
([#3453](https://github.com/triggerdotdev/trigger.dev/pull/3453))
- Fix RSS memory leak in the realtime proxy routes. `/realtime/v1/runs`,
`/realtime/v1/runs/:id`, and `/realtime/v1/batches/:id` called `fetch()`
into Electric with no abort signal, so when a client disconnected mid
long-poll, undici kept the upstream socket open and buffered response
chunks that would never be consumed — retained only in RSS, invisible to
V8 heap tooling. Thread `getRequestAbortSignal()` through
`RealtimeClient.streamRun/streamRuns/streamBatch` to `longPollingFetch`
and cancel the upstream body in the error path. Isolated reproducer
showed ~44 KB retained per leaked request; signal propagation releases
it cleanly.
([#3442](https://github.com/triggerdotdev/trigger.dev/pull/3442))
- Fix memory leak where every aborted SSE connection pinned the full
request/response graph on Node 20, caused by `AbortSignal.any()` in
`sse.ts` retaining its source signals indefinitely (see
nodejs/node#54614, nodejs/node#55351). Also clear the
`setTimeout(abort)` timer in `entry.server.tsx` so successful HTML
renders don't pin the React tree for 30s per request.
([#3430](https://github.com/triggerdotdev/trigger.dev/pull/3430))
- Preserve filters on the queues page when submitting modal actions.
([#3471](https://github.com/triggerdotdev/trigger.dev/pull/3471))
- Fix Redis connection leak in realtime streams and broken abort signal
propagation.
**Redis connections**: Non-blocking methods (ingestData, appendPart,
getLastChunkIndex) now share a single Redis connection instead of
creating one per request. streamResponse still uses dedicated
connections (required for XREAD BLOCK) but now tears them down
immediately via disconnect() instead of graceful quit(), with a 15s
inactivity fallback.
**Abort signal**: request.signal is broken in Remix/Express due to a
Node.js undici GC bug (nodejs/node#55428) that severs the signal chain
when Remix clones the Request internally. Added getRequestAbortSignal()
wired to Express res.on("close") via httpAsyncStorage, which fires
reliably on client disconnect. All SSE/streaming routes updated to use
it. ([#3399](https://github.com/triggerdotdev/trigger.dev/pull/3399))
- Prevent dashboard crash (React error #31) when span accessory item
text is not a string. Filters out malformed accessory items in
SpanCodePathAccessory instead of passing objects to React as children.
([#3400](https://github.com/triggerdotdev/trigger.dev/pull/3400))
- Upgrade Remix packages from 2.1.0 to 2.17.4 to address security
vulnerabilities in React Router
([#3372](https://github.com/triggerdotdev/trigger.dev/pull/3372))
- Fix Vercel integration settings page (remove redundant section
toggles) and improve the Vercel onboarding flow so the modal closes
after connecting a GitHub repo and the marketplace `next` URL is
preserved across the GitHub app install redirect.
([#3424](https://github.com/triggerdotdev/trigger.dev/pull/3424))
<details>
<summary>Raw changeset output</summary>
# Releases
## @trigger.dev/build@4.4.5
### Patch Changes
- Updated dependencies:
- `@trigger.dev/core@4.4.5`
## trigger.dev@4.4.5
### Patch Changes
- Add `--no-browser` flag to `init` and `login` to skip auto-opening the
browser during authentication. Also error loudly when `init` is run
without `--yes` under non-TTY stdin (previously default-and-exited
silently, leaving the project half-initialized). Both commands now show
an `Examples` section in `--help`.
([#3483](https://github.com/triggerdotdev/trigger.dev/pull/3483))
- Updated dependencies:
- `@trigger.dev/core@4.4.5`
- `@trigger.dev/build@4.4.5`
- `@trigger.dev/schema-to-json@4.4.5`
## @trigger.dev/core@4.4.5
### Patch Changes
- Add `isReplay` boolean to the run context (`ctx.run.isReplay`),
derived from the existing `replayedFromTaskRunFriendlyId` database
field. Defaults to `false` for backwards compatibility.
([#3454](https://github.com/triggerdotdev/trigger.dev/pull/3454))
- Redact the `resolveWaitpoint` runtime log so it only emits `id` and
`type` instead of the full completed waitpoint. Previously the log
printed the entire waitpoint (including `output`) to stdout in
production runs, which could leak sensitive payloads. The value returned
by `wait.forToken()` is unchanged.
([#3490](https://github.com/triggerdotdev/trigger.dev/pull/3490))
- Add `SessionId` friendly ID generator and schemas for the new durable
Session primitive. Exported from `@trigger.dev/core/v3/isomorphic`
alongside `RunId`, `BatchId`, etc. Ships the
`CreateSessionStreamWaitpoint` request/response schemas alongside the
main Session CRUD.
([#3417](https://github.com/triggerdotdev/trigger.dev/pull/3417))
- Truncate large error stacks and messages to prevent OOM crashes. Stack
traces are capped at 50 frames (keeping top 5 + bottom 45 with an
omission notice), individual stack lines at 1024 chars, and error
messages at 1000 chars. Applied in parseError, sanitizeError, and OTel
span recording.
([#3405](https://github.com/triggerdotdev/trigger.dev/pull/3405))
## @trigger.dev/python@4.4.5
### Patch Changes
- Updated dependencies:
- `@trigger.dev/core@4.4.5`
- `@trigger.dev/build@4.4.5`
- `@trigger.dev/sdk@4.4.5`
## @trigger.dev/react-hooks@4.4.5
### Patch Changes
- Updated dependencies:
- `@trigger.dev/core@4.4.5`
## @trigger.dev/redis-worker@4.4.5
### Patch Changes
- Updated dependencies:
- `@trigger.dev/core@4.4.5`
## @trigger.dev/rsc@4.4.5
### Patch Changes
- Updated dependencies:
- `@trigger.dev/core@4.4.5`
## @trigger.dev/schema-to-json@4.4.5
### Patch Changes
- Updated dependencies:
- `@trigger.dev/core@4.4.5`
## @trigger.dev/sdk@4.4.5
### Patch Changes
- Updated dependencies:
- `@trigger.dev/core@4.4.5`
</details>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
The webapp's compute template creation hardcoded a single machine preset
(`small-1x`) at deploy time, regardless of which presets a project
actually uses. Tasks running on any other preset paid full cold-snapshot
creation cost on first run.
Two new env vars:
- `COMPUTE_TEMPLATE_MACHINE_PRESETS` - CSV of preset names to build boot
snapshots for during deploy. Defaults to `small-1x` so existing deploys
don't change behavior.
- `COMPUTE_TEMPLATE_MACHINE_PRESETS_REQUIRED` - CSV of presets whose
failure fails a required-mode deploy. Defaults to the full `PRESETS`
list. Optional preset failures are logged but don't block the deploy.
The compute client now sends the multi-config request shape; the service
evaluates per-preset outcomes against the required set and surfaces a
combined failure message when a required preset fails.
Both env vars are validated at boot via the env schema - unknown preset
names or `_REQUIRED` entries that aren't a subset of `_PRESETS` fail
loudly at startup rather than silently per-deploy.
Most actions in this repo were several major versions behind, which is
why every CI run has been emitting Node 20 deprecation warnings.
Pinning every action to a commit SHA (with the version as a trailing
comment) means each CI run uses the exact code that was reviewed when
the bump landed, instead of whatever a maintainer last pointed the major
tag at. Dependabot is configured to group all action bumps into one
weekly PR with a 7-day cooldown.
Worth flagging:
- The Claude Code action ships ~daily but the model is set separately
via `--model` in `claude_args`, so SHA-pinning the action gives
reproducibility without locking the model.
- The kubeconform container is digest-pinned
(`docker://image:tag@sha256:...`). Dependabot's github-actions ecosystem
doesn't track `docker://` references ([explicit TODO in
dependabot-core](https://github.com/dependabot/dependabot-core/blob/main/github_actions/lib/dependabot/github_actions/file_parser.rb)),
so it needs manual bumps either way - but the digest pin protects
against tag repointing for free.
Redact the `resolveWaitpoint` runtime log so it only emits `id` and
`type` instead of the full completed waitpoint. Previously the log
printed the entire waitpoint (including `output`) to stdout in
production runs, which could leak sensitive payloads. The value returned
by `wait.forToken()` is unchanged.
The Activity peak count tooltip in the errors list rendered a `<button>`
(from `SimpleTooltip`'s default `TooltipTrigger`) inside the row's `<a>`
link (`TableCell to={errorPath}`). Interactive content nested inside
other interactive content is invalid HTML and triggers accessibility
warnings. Adding `asChild` to `SimpleTooltip` makes the existing
`<span>` the trigger directly, removing the nested `<button>`.
Closes the most common friction point hit while setting up a fresh
project from an agent harness: the CLI auto-opens the user's default
browser during auth and there is no supported way to skip it (the
existing `isLinuxServer()` path only triggers when `xdg-open` is missing
entirely).
`--no-browser` on `login` and `init` prints the URL and waits to be
visited from any browser. The flag threads through the embedded
`login()` call inside `init`.
While here:
- `init` now errors loudly when stdin is non-TTY without `--yes` instead
of default-and-exiting silently at the first prompt (which left the
project half-initialized: deps installed, no config or example file).
- Both commands gain an `Examples` block in `--help` rendered between
the description and the arguments/options list, so `--help | head`
surfaces the common invocations.
Other commands also call `login()` embedded and would benefit from
`--no-browser` too, but kept this PR scoped to the cases the friction
log called out.
## Summary
Each scheduled-task tick previously issued **3 Prisma `UPDATE`s**
against
`TaskSchedule.lastRunTriggeredAt`,
`TaskScheduleInstance.lastScheduledTimestamp`,
and `TaskScheduleInstance.nextScheduledTimestamp`. All three were pure
denormalization — every value can be derived without persisting.
After this PR `TaskSchedule` and `TaskScheduleInstance` become **near
read-only**:
writes happen only on schedule create / update / delete (rare admin
actions),
so the per-tick autovacuum churn on these hot tables disappears.
## Design
The previous fire time travels forward through the **schedule worker
payload**,
not through the database. Concretely:
- The `schedule.triggerScheduledTask` worker payload gains an optional
`lastScheduleTime: z.coerce.date().optional()` field.
- When the engine fires a schedule, it re-enqueues the next tick with
`lastScheduleTime = scheduleTimestamp` (the just-fired time).
- When the next tick dequeues, `payload.lastTimestamp` is sourced from
`params.lastScheduleTime` directly. No DB round-trip, no cron-derivation
drift across DST boundaries, no caveats around recently-edited cron
expressions.
`payload.lastTimestamp` keeps its `Date | undefined` SDK shape.
First-ever
fires still report `undefined`, so customer `if
(!payload.lastTimestamp)`
first-run patterns keep working.
For Redis jobs that were enqueued **before** this change (which lack
`lastScheduleTime` in their payload), the engine falls back to
`instance.lastScheduledTimestamp` once. Once those drain, the column is
never read again. Revert is code-only; the columns stay in place and can
be dropped in a follow-up once the rollout is stable.
## Files
- `internal-packages/schedule-engine/*` — engine refactor,
`workerCatalog`
schema field, `TriggerScheduleParams` extension, tests updated to assert
on the worker-payload flow rather than DB readbacks.
- `internal-packages/database/prisma/schema.prisma` — `/// @deprecated`
triple-slash docstrings on the three columns. No migration.
- `apps/webapp/app/presenters/v3/ScheduleListPresenter.server.ts` —
drops
the `lastRunTriggeredAt` Prisma select; "Last run" cell is approximated
from the cron expression's previous slot, gated on `schedule.createdAt`
so brand-new schedules show "–". UI is best-effort; the runs page is the
source of truth.
- `apps/webapp/app/v3/utils/calculateNextSchedule.server.ts` — adds a
`previousScheduledTimestamp` helper for the UI cell above. Public API
responses (`api.v1.schedules.*`) already compute `nextRun` from cron and
don't expose `lastTimestamp` — no public API change.
- `references/scheduled-tasks/` — new reference project with declarative
schedules at multiple cadences and three throw-on-fail validators
(`first-fire-detector`, `interval-validator`, `upcoming-validator`) for
E2E-verifying the worker-payload flow.
Refs TRI-8891
## Test plan
- [x] `pnpm run typecheck --filter @internal/schedule-engine --filter
webapp`
- [x] `pnpm run build --filter @trigger.dev/core`
- [x] `pnpm run test --filter @internal/schedule-engine` — integration
test
asserts first-fire `lastTimestamp === undefined`, second fire carries
the previous fire's timestamp exactly.
- [x] E2E against local webapp via `references/scheduled-tasks`:
- Fresh schedules attached → all three deprecated columns stay `NULL`
after
multiple fires.
- Redis payload at second fire contains
`"lastScheduleTime":"<previous fire timestamp>"`.
- `TaskRun.payload` and the every-minute task's returned output both
confirm
`lastTimestamp = null` on first fire and `lastTimestamp = <prev fire>`
on
second fire, exactly 60s apart.
- All three throw-on-FAIL validators completed successfully on every
non-first fire.
- [x] Schedules REST API end-to-end (`POST` / `GET` / `PUT` / `activate`
/
`deactivate` / `DELETE`) — `nextRun` recomputed live from cron + tz on
every response, no reads of deprecated columns.
## What this does
Polish + bug-fix pass on the Errors page so it can ship to everyone.
Touches the Slack alert config UX, errors list, error detail page, and
unhides the SideMenu entry for non-admins.
## Decisions
**"No channel" item over standalone Remove button**
Chose pinning a `<XMarkIcon /> No channel` `SelectItem` above the
channel list. Rejected the standalone "Remove channel" link in a
`<Hint>` — color/hover behaviour clashed with the sibling `<TextLink>`,
and "channel selection" is the right context for clearing. Server action
already deletes the channel when `slackChannel=""` is submitted.
**Slack `<!date^>` token over per-user TZ field for alerts**
Chose Slack's native `<!date^TS^…>` token so each viewer sees timestamps
in their own timezone (UTC fallback). Rejected per-user/per-org TZ
schema work — works for multi-region channels for free. Email/dashboard
TZ source-of-truth filed as TRI-8885 / TRI-8886.
**Make errors GA**
## Summary
`POST /api/v1/sessions` was secret-key-only because the customer browser
flow runs through `chat.createStartSessionAction` (server-side, holds
the secret key). But the `cli-v3` MCP `start_agent_chat` tool is itself
a server-side surface — developer's CLI/IDE acting as their own server —
and only holds a JWT minted from the user's PAT. Without JWT support on
this route the entire MCP agent toolkit (`start_agent_chat`,
`send_agent_message`, `close_agent_chat`) is blocked at session
creation.
Add `allowJWT: true` plus an `authorization` block requiring the
`write:sessions` (or `admin`) super-scope.
## Why a wildcard `sessions` resource
Resource scoping by `taskIdentifier` isn't possible at auth-resolve time
— action routes don't pass `body` to the `resource` callback, and the
task name only lives in the body. So the resource is `sessions: "*"` and
the super-scope does the actual gating. The JWT-issuer (cli-v3 MCP,
customer servers wrapping their own auth helpers, etc.) decides which
scopes to mint, which is where per-task narrowing lives.
## Test plan
- [x] Verified end-to-end against local:
`mcp__trigger__start_agent_chat` → `send_agent_message("pong")` →
`send_agent_message("echo")` → `close_agent_chat` all succeed. Two
assistant turns reuse the same runId (continuation in the idle window).
- [ ] Browser-mediated `chat.createStartSessionAction` flow continues to
work unchanged (still uses secret-key path under the hood).
- [ ] Loader (GET) and other session routes — unchanged, no scope drift.
## Notes
This unblocks T17 in the [ai-chat e2e smoke
catalog](https://github.com/triggerdotdev/trigger.dev/blob/feature/tri-7532-ai-sdk-chat-transport-and-chat-task-system/.claude/skills/ai-chat-e2e/SMOKE-TESTS.md)
(which lives in the feature branch's skill catalog, not this repo).
Pairs with the cli-v3 MCP fix on the feature branch (`feat: AI SDK
custom useChat transport & chat.task harness`, PR #3173) — that PR's
`agentChat.ts` change makes the call shape correct (`taskIdentifier` +
`triggerConfig`); this PR opens the door for the JWT to actually pass.
## Summary
Self-hosters that operate the webapp's ECR account separately from the
account running the EKS workers (e.g., a shared platform account that
hosts the registry plus per-team accounts that host clusters) currently
hit a 403 Forbidden the first time **any** project is deployed:
```
Failed to pull image "<acct-A>.dkr.ecr.<region>.amazonaws.com/<namespace>/proj_…:…":
unexpected status from HEAD request to .../v2/.../manifests/sha256:…: 403 Forbidden
```
`ensureEcrRepositoryExists` in
`apps/webapp/app/v3/getDeploymentImageRef.server.ts` calls
`CreateRepository` and `PutLifecyclePolicy`, but never
`SetRepositoryPolicy` — so the new repo inherits the AWS default (only
the registry-owner account can read/pull). Workers in the cluster
account get 403 every single deploy. The only workarounds today are
running a one-off post-create script or pre-creating every repo by hand.
## Proposed change
Add an optional env var:
```
DEPLOY_REGISTRY_ECR_DEFAULT_REPOSITORY_POLICY (V4 mirror: V4_DEPLOY_REGISTRY_ECR_DEFAULT_REPOSITORY_POLICY)
```
Raw IAM policy JSON. When set, the webapp calls `SetRepositoryPolicy`
immediately after `CreateRepository` so every new repo carries that
policy from creation. Operators control the principal/actions; we don't
bake in any opinions about cross-account boundaries.
Example value (for the typical self-host case — grant pull to the
cluster account):
```json
{
"Version": "2012-10-17",
"Statement": [{
"Sid": "AllowClusterAccountPull",
"Effect": "Allow",
"Principal": {"AWS": "arn:aws:iam::<cluster-account-id>:root"},
"Action": [
"ecr:GetDownloadUrlForLayer",
"ecr:BatchGetImage",
"ecr:BatchCheckLayerAvailability"
]
}]
}
```
## Why env var (not a chart-level field)
- Mirrors the shape of the sibling vars (`DEPLOY_REGISTRY_ECR_TAGS`,
`DEPLOY_REGISTRY_ECR_ASSUME_ROLE_ARN`, etc.) which are already
operator-supplied via `webapp.extraEnvVars` in self-host setups.
- Cloud is unaffected — the env var is optional, unset by default;
existing behavior unchanged.
- Existing repos are unaffected — only newly-created repos get the
policy.
- `RepositoryCreationTemplate` from the AWS provider isn't an
alternative here: it only applies to repos created via
pull-through-cache or replication, not to `ecr:CreateRepository` API
calls.
## Implementation
- `apps/webapp/app/env.server.ts` — declare
`DEPLOY_REGISTRY_ECR_DEFAULT_REPOSITORY_POLICY` and the V4 fallback.
- `apps/webapp/app/v3/registryConfig.server.ts` — propagate
`ecrDefaultRepositoryPolicy` to `RegistryConfig`.
- `apps/webapp/app/v3/getDeploymentImageRef.server.ts` —
`createEcrRepository` accepts the policy; if set, calls
`SetRepositoryPolicy` after `PutLifecyclePolicy`.
- `docs/self-hosting/env/webapp.mdx` — documentation row added under
**Deploy & Registry**.
## Verification
Verified end-to-end against a self-hosted Trigger.dev on EKS where the
ECR account is separate from the cluster account:
- **Without the env var** (current `main`): the new project's first run
pod stays in `ImagePullBackOff` with `403 Forbidden`.
- **With the env var set** to a JSON granting
`ecr:BatchGetImage`/`GetDownloadUrlForLayer`/`BatchCheckLayerAvailability`
to the cluster account: a fresh `trigger.dev deploy --env prod` followed
by a `hello-world` run completes in ~5s end-to-end on the first try.
Manually also confirmed that existing repos are untouched (the call only
fires inside `createEcrRepository`, which only runs when
`DescribeRepositories` returned `RepositoryNotFoundException`).
## Out of scope
- Chart values surface for this — operators already pass the existing
ECR vars via `webapp.extraEnvVars`, so this follows the same pattern.
Happy to add a first-class chart field in a follow-up if that's the
preferred direction.
- IAM-policy validation in the webapp — we forward the JSON verbatim to
AWS and surface AWS's error messages on misuse, matching how
`DEPLOY_REGISTRY_ECR_TAGS` is handled today.
This is a draft pending CI / CodeRabbit pass — happy to iterate on
direction (e.g., split into per-action env vars, or extend the chart
values schema) if any of the above choices feels off.
---------
Co-authored-by: nicktrn <55853254+nicktrn@users.noreply.github.com>
Queues page action handler was rebuilding the redirect URL with only
`?page=`, so any pause/resume/override modal confirmation wiped the
user's search query. With hundreds of queues filtered down to a handful,
every confirmation dropped you back to the unfiltered list - and
pagination still pointed at the previous numeric page, so you'd land on
a different slice than you came from.
Swap the manual rebuild for `url.search` so the full querystring
(including any future filter params) flows through. Drops the now-unused
`SearchParamsSchema.parse` call inside `action`; the loader still
validates on the way back.
## Summary
Follow-up to #3420. PATs (public access tokens) minted before an API key
rotation 401'd immediately on the realtime stream endpoints, even though
the rotation flow advertises a 24h overlap. This fixes the gap.
## Root cause
PATs are JWTs signed with the env's `apiKey` at mint time. When that
secret is rotated, `validatePublicJwtKey`
(`apps/webapp/app/services/realtime/jwtAuth.server.ts`) only verifies
the signature against `environment.parentEnvironment?.apiKey ??
environment.apiKey` — i.e. the env's *current* canonical key. Any PAT in
the wild signed with the previous key fails signature verification →
401, even within the grace window.
#3420 wired up the grace-window fallback in two places —
`findEnvironmentByApiKey` (raw secret-key auth) and `api.v1.auth.jwt.ts`
(signs new JWTs with the canonical key when minting from an old one) —
but the *verify* path for already-issued PATs was never updated.
In a typical app, `POST /api/v1/tasks/.../trigger` (Bearer secret) keeps
working through rotation because that path has the fallback, but `GET
/realtime/v1/streams/run_*/...` and `POST
/realtime/v1/streams/run_*/input/...` 401 for runs that were already in
flight when the rotation happened.
## Fix
After the primary `validateJWT` against the env's current `apiKey`, fall
back to non-expired `RevokedApiKey` rows for the signing env (parent env
when the request is against a child) — but **only on the failure path**,
so the hot success path is unchanged. Uses `$replica` to match the rest
of the auth path.
Symmetrical to the `findEnvironmentByApiKey` two-step from #3420.
## Changes
- `apps/webapp/app/services/realtime/jwtAuth.server.ts` —
`validateAgainstRevokedApiKeys` helper invoked only on `!result.ok`
- `apps/webapp/app/models/runtimeEnvironment.server.ts` —
`findEnvironmentById` also selects `parentEnvironment.id` so we can
scope the revoked-keys lookup to the correct env
## Test plan
E2E verified locally via curl against `GET /realtime/v1/runs/{runId}`
(PAT-authenticated):
- [x] Pre-rotation, PAT signed with K1 → **200** with run body
- [x] Simulate rotation (insert `RevokedApiKey` row + flip env `apiKey`
to K2 in a single transaction, mirroring `regenerateApiKey`)
- [x] Same PAT (K1) within grace window → **200** with run body —
fallback hits
- [x] Fresh PAT signed with K2 → **200** — current key still works
- [x] Set `RevokedApiKey.expiresAt` to past → **401** — fallback finds
no live row
- [x] Bogus signature (no rotation) → **401**
- [x] Cleanup verified: env `apiKey` restored, `RevokedApiKey` row
deleted
- [x] `pnpm run typecheck --filter webapp` passes
## Summary
Several boundary catches and customer-input validation paths were
logging at `error` level for failures the system already handles
gracefully — disconnect on auth failure, return undefined, skip retries,
etc. This batch routes them to `warn` (which stays in stdout) or counts
them as OTel metrics, so visibility is preserved without surfacing them
as alerts.
## Changes
**New helper / pattern:**
- `apiBuilder.server.ts` — `logBoundaryError(message, error, url)`
inspects the inner error type at loader/action boundary catches;
downgrades to `warn` for `AbortError`, `ServiceValidationError`, and
`EngineServiceValidationError`.
- `platform.v3.server.ts` — `platform_client.failures_total` OTel
counter with `{function, kind}` labels; helper
`recordPlatformFailure(fn, kind)` replaces the previous error-level
logging across all `BillingClient` wrappers.
**Log-level downgrades:**
- `handleSocketIo.server.ts` — `Worker authentication failed` → warn
(system disconnects on failure; refs TRI-8863)
- `waitpointSystem.ts` — when `runStatus === "CANCELED"` in the
suspended-without-checkpoint branch, skip the throw and warn instead
(benign cancel-vs-resume race, nothing to resume)
- `runAttemptSystem.ts` — `flushedMetadata` parse/validate failures →
warn (customer-side data shape, system returns gracefully)
- `batch-queue/index.ts` — final-attempt failures with
`result.skipRetries` → warn (callbacks already opted out of retry, e.g.
queue size limit hit)
- `queryPerformanceMonitor.server.ts` — slow queries → warn
(observability signal, not an application error)
- `timeoutDeployment.server.ts` — deployment-state mismatch in the
timeout job → warn (timeout-vs-completion race)
**Inner error preservation:**
- `waitpointCompletionPacket.server.ts` — `logger.error(uploadError)`
before throwing the `ServiceValidationError` wrapper, so the underlying
upload error stays visible.
## Why
The pattern across all of these is the same: a boundary log treated any
thrown/returned error as `error` regardless of cause, even when the
cause was an expected, system-handled condition (client disconnect,
customer quota, race condition, schema validation of customer data).
That made the logs noisy and made it harder to spot real bugs.
Where the underlying signal is still useful operationally (slow queries,
billing call failures), we route it to OTel metrics with low-cardinality
labels so dashboards and alerts can be tuned independently of error
logs.
## Test plan
- [ ] `pnpm run typecheck --filter webapp`
- [ ] `pnpm run build --filter @internal/run-engine`
- [ ] Trigger a run on hello-world and verify task lifecycle is
unaffected
- [ ] Cancel a suspended run and verify the cancel-while-suspended
branch in `waitpointSystem.ts` returns `{status: "skipped"}` instead of
throwing
- [ ] Confirm `platform_client.failures_total` counter shows up in
metrics with `{function, kind}` labels when the billing client errors
Today the helm prerelease workflow only fires on PRs that touch
`hosting/k8s/helm/**`. Two consequences we ran into:
1. The `changeset-release/main` PR's prerelease comment goes stale once
the release branch gets force-pushed without a helm-touching commit (the
bot's `Chart.yaml` bump alone doesn't seem to refire the trigger
reliably).
2. The release PR's chart references an `appVersion` (e.g. `v4.4.5`)
whose Docker images don't exist until *after* merge + tag. So that
prerelease chart can't actually be installed end-to-end.
Renames the workflow to `helm-prerelease.yml` and adds two new triggers:
- **`push: main`** with `paths: hosting/k8s/helm/**` -> rolling
prereleases versioned `<base>-main.<sha>`. `appVersion` stays at
whatever `Chart.yaml` has (i.e. last released), so installs pull real
images. Tests that chart structure is deployable, even if the app code
is one release behind.
- **`workflow_dispatch`** with optional `app_version` input -> manually
trigger a prerelease and optionally override `appVersion` (e.g. pin to
`main` or a specific tag). Useful for testing chart + app-version
combinations on demand.
PR behavior unchanged: same `<base>-pr<N>.<sha>` versioning, same
posted/updated comment.
Why not also bypass paths for `changeset-release/main`? The release PR's
chart references not-yet-built `v4.4.5` images, so those prereleases
aren't actually installable. The rolling main prerelease covers the
testable case better.
Why not SHA-pin `appVersion` to a built image like `main-<sha>`? Bigger
change - the docker publish workflows currently only push `:main` (no
SHA-suffixed tag). Worth doing later if we want first-class "install one
chart, get exactly that commit's app code" testing, but out of scope
here.
Diff is mostly a rename. Substantive changes:
- new `push` and `workflow_dispatch` triggers
- `prerelease` job `if:` extended for the new event types
- version logic branches per event
- new "Override appVersion" step (workflow_dispatch only)
- new "Write run summary" step so non-PR runs surface the install
instructions
- PR comment steps gated on `github.event_name == 'pull_request'`
- concurrency group falls back to `github.ref` for non-PR runs
> ⚠️ **Not released yet.** This PR is the server-side foundation only.
The SDK changes that customers will actually use (`chat.agent`
migration, `chat.createStartSessionAction`, `useTriggerChatTransport`
updates) live on a separate branch and ship together in an upcoming
`@trigger.dev/sdk` prerelease. Until that prerelease is published, this
surface is reachable only via direct HTTP.
## What this gives Trigger.dev users
A new first-class primitive, **Session**, for durable, task-bound,
bidirectional I/O that outlives any single run. Sessions are the run
manager for `chat.agent` going forward, and they unblock anything else
that needs "one identifier, many runs over time" with a stable channel
pair the client can write to and subscribe to.
### Use cases unblocked
- **Chat agents that persist across many runs.** One session per chat
(keyed on your own `chatId` via `externalId`), turns 1..N attach to the
same Session, the UI subscribes once and keeps receiving output as new
runs take over.
- **Approval loops and long-running tasks with user feedback.** The task
waits on `.in`, the client writes to `.in`, the server enforces
no-writes-after-close.
- **Workflow progress streams that live past the run.** Subscribe to
`.out` after the task finishes to replay history.
- **Resume-next-day flows.** A session is a durable row, not a transient
stream. Send a message a day later and the server triggers a fresh run
on the same session.
### How it works (Session-as-run-manager)
A Session row is task-bound (`taskIdentifier` + `triggerConfig` are
required) and owns its current run via `currentRunId` +
`currentRunVersion` for optimistic claim. Three trigger paths:
1. **Session create** — `POST /api/v1/sessions` creates the row and
triggers the first run synchronously.
2. **Append-time probe** — `POST
/realtime/v1/sessions/:session/in/append` checks if the current run is
alive; if it has terminated (idle exit, crash, etc.), the server
triggers a new run before processing the append.
3. **End-and-continue handoff** — `POST
/api/v1/sessions/:session/end-and-continue`, called by the running
agent, triggers a fresh run and atomically swaps `currentRunId`. Used by
`chat.requestUpgrade()` for version handoffs.
Every triggered run is recorded in the `SessionRun` audit table with a
reason (`initial`, `continuation`, `upgrade`, `manual`).
## Public API surface
### Control plane
- `POST /api/v1/sessions` — create. Idempotent on `(env, externalId)`.
Triggers the first run, returns the session and a session-scoped public
access token. Returns 409 if the upserted row is already closed.
- `GET /api/v1/sessions/:session` — retrieve by friendlyId
(`session_abc...`) or by your own externalId (server disambiguates by
prefix).
- `GET /api/v1/sessions` — list with filters (`type`, `tag`,
`taskIdentifier`, `externalId`, derived `status` ACTIVE/CLOSED/EXPIRED,
created-at range) and cursor pagination. Backed by ClickHouse.
- `PATCH /api/v1/sessions/:session` — update tags / metadata /
externalId.
- `POST /api/v1/sessions/:session/close` — terminate. Idempotent,
hard-blocks new server-brokered writes.
- `POST /api/v1/sessions/:session/end-and-continue` — agent-only handoff
to a fresh run.
### Realtime
- `PUT /realtime/v1/sessions/:session/:io` — initialize a channel.
Returns S2 credentials in headers so high-throughput clients can write
direct to S2.
- `GET /realtime/v1/sessions/:session/:io` — SSE subscribe. Supports
Last-Event-ID resume and an opt-in `X-Peek-Settled: 1` header that
fast-closes the stream when the upstream is already settled
(`trigger:turn-complete`), eliminating long-poll wait on
reconnect-on-reload paths.
- `POST /realtime/v1/sessions/:session/:io/append` — server-side
appends.
- `POST /api/v1/runs/:runFriendlyId/session-streams/wait` — runs wait on
a session stream as a waitpoint, with a race-check to avoid suspending
if data already landed.
### Auth scopes
`sessions` is a new resource type. `read:sessions:{id}`,
`write:sessions:{id}`, `admin:sessions:{id}` flow through the existing
JWT validator. Session-scoped public access tokens minted by the server
replace browser-held trigger-task tokens for chat-style flows — the
browser never sees a run identifier or a run-scoped token in steady
state.
## What's coming after this PR
- **SDK + chat.agent migration**: separate branch, separate PR, ships in
the next `@trigger.dev/sdk` prerelease alongside this server deploy.
Customers using the prerelease `chat.agent` will follow the [upgrade
guide](https://github.com/triggerdotdev/trigger.dev/blob/docs/tri-7532-ai-sdk-chat-transport-and-chat-task-system/docs/ai-chat/upgrade-guide.mdx).
- **Dashboard surfaces**: dedicated agent list, agent playground, agent
view on the run dashboard. Tracking separately.
## Implementation notes
- **Postgres `Session` table**: scalar scoping columns (`projectId`,
`runtimeEnvironmentId`, `environmentType`, `organizationId`) without
FKs, matching the January TaskRun FK-removal decision. Point-lookup
indexes only — list queries go to ClickHouse. Terminal markers
(`closedAt`, `expiresAt`) are write-once.
- **ClickHouse `sessions_v1`**: ReplacingMergeTree, partitioned by
month, ordered by `(org_id, project_id, environment_id, created_at,
session_id)`. Tags indexed via `tokenbf_v1` skip index.
- **`SessionsReplicationService`**: mirrors `RunsReplicationService`
exactly — leader-locked logical replication consumer,
`ConcurrentFlushScheduler`, retry with exponential backoff + jitter,
identical metric shape. Dedicated slot + publication so the two consume
independently.
- **S2 keys**: `sessions/{addressingKey}/{out|in}`. The existing
`runs/{runId}/{streamId}` key format for run-scoped streams is
untouched.
- **Optimistic claim**: `ensureRunForSession` triggers a run upfront
(cheap to cancel if it loses the race), then attempts an `updateMany`
keyed on `currentRunVersion`. Loser cancels its triggered run and reuses
the winner's. No DB lock held across the trigger.
### What did NOT change
Run-scoped `streams.pipe` / `streams.input` and the existing
`/realtime/v1/streams/{runId}/...` routes are unchanged. Sessions are
net-new — not a reshaping of the current streams API.
## Deploy notes
- Set `SESSION_REPLICATION_CLICKHOUSE_URL` and
`SESSION_REPLICATION_ENABLED=1` to enable the replication consumer.
- The `Session` table needs `REPLICA IDENTITY FULL` set on the prod
source DB before the publication is created (same one-time DDL we did
for `TaskRun`). Required for delete events to carry full column values.
- Cross-form authorization on the `GET /api/v1/sessions/:session` loader
(a JWT minted for either form authorizes both URL forms). Action routes
are URL-form-specific, matching how the SDK mints PATs.
## Verification
- Webapp typecheck clean (10/10).
- `apps/webapp/test/sessionsReplicationService.test.ts` — round-trip
tests for insert/update/delete through Postgres logical replication into
ClickHouse via testcontainers.
- Live end-to-end against local dev: create + retrieve (both forms) +
update + close, `.out.initialize` + `.out.append` x2 + `.in.send` +
`.out.subscribe` over SSE, list with all filter combinations +
pagination, `end-and-continue` swap, `X-Peek-Settled` fast-close
(verified in browser via reconnect-on-reload and via curl). Replicated
row lands in ClickHouse within ~1s.
- Multi-round Devin + CodeRabbit review feedback addressed
(read-after-write paths use `prisma` writer, info-leak on auth-routes
masked as 403, peek-settled discriminator parsing fix, etc.).
## Test plan
- [ ] `pnpm run typecheck --filter webapp`
- [ ] `pnpm run test --filter webapp
./test/sessionsReplicationService.test.ts --run`
- [ ] Start the webapp with `SESSION_REPLICATION_CLICKHOUSE_URL` and
`SESSION_REPLICATION_ENABLED=1`. Confirm the slot and publication
auto-create on boot.
- [ ] `POST /api/v1/sessions` and verify the row replicates to
`trigger_dev.sessions_v1` within a couple of seconds.
- [ ] `POST /api/v1/sessions/:id/close`, then confirm `POST
/realtime/v1/sessions/:id/out/append` returns 400.
- [ ] Reuse a closed session's `externalId` on `POST /api/v1/sessions`
and confirm 409.
- [ ] `GET /realtime/v1/sessions/:id/out` with `X-Peek-Settled: 1` after
a turn completes and confirm `X-Session-Settled: true` response header +
immediate close.
## Changes
Three changes in
`internal-packages/run-engine/src/engine/systems/debounceSystem.ts`, in
order of impact:
1. **Fast-path skip before the lock.** In `handleExistingRun`, do an
unlocked read of `delayUntil` (and `createdAt` for the max-duration
check) from the run row before entering `runLock.lock("handleDebounce",
...)`. If `newDelayUntil <= currentDelayUntil` and the run is still
within its max-duration window, return the existing run immediately
without taking the lock. Safe because debounce is monotonic-forward only
— a stale read either matches reality or undershoots, both of which
decay correctly (re-checked properly inside the lock by whichever caller
is actually pushing forward). Trailing-mode triggers carrying
`updateData` still take the lock so the data update is applied.
2. **Quantize `newDelayUntil`.** Round the computed `newDelayUntil` to
1-second buckets (configurable via `quantizeNewDelayUntilMs`, set to 0
to disable). Without quantization, every call has a slightly larger
`newDelayUntil` than the last and they all pass the fast-path check.
With it, concurrent callers on the same key share a target time and ~95%
short-circuit. User-visible effect: a debounced run might fire up to 1s
earlier than the strict spec — non-issue for typical debounce use cases
(chat summarization, batched notifications, etc.).
3. **Graceful lock-contention fallback.** Wrap the `runLock.lock(...)`
call so `LockAcquisitionTimeoutError` and Redlock `ExecutionError` /
`ResourceLockedError` return the existing run id with success instead of
propagating a 5xx. Debounce is best-effort: if we can't take the lock,
the herd is already updating it for us; fall in line. This kills the 5xx
→ SDK-retry feedback loop. With (1)+(2) this rarely fires; without them
it's the difference between 5xx and 200.
Defaults preserve current behaviour aside from quantization (1s) and
fast-path (on). Both are configurable via `RunEngineOptions.debounce`.
## ✅ Checklist
- [x] I have followed every step in the [contributing
guide](https://github.com/triggerdotdev/trigger.dev/blob/main/CONTRIBUTING.md)
- [x] The PR title follows the convention.
- [x] I ran and tested the code works
---
## Changelog
Reduce 5xx feedback loops on hot debounce keys by quantizing
`delayUntil`, adding an unlocked fast-path skip before the redlock, and
gracefully handling redlock contention in `handleDebounce` so the SDK no
longer retries into a herd.
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
## Summary
Adds `isReplay` boolean to the run context (`ctx.run.isReplay`),
following the same pattern as the existing `isTest`. The value is
derived from the existing `replayedFromTaskRunFriendlyId` database
field, so no schema migration is needed.
## ✅ Checklist
- [x] I have followed every step in the [contributing
guide](https://github.com/triggerdotdev/trigger.dev/blob/main/CONTRIBUTING.md)
- [x] The PR title follows the convention.
- [x] I ran and tested the code works
---
## Testing
- Verified `@trigger.dev/core` builds successfully
- Verified `webapp` typechecks successfully
- All new fields use `default(false)` for backwards compatibility
---
## Changelog
- Added `isReplay` to `TaskRun` and `V3TaskRun` schemas in `common.ts`
- Added `RUN_IS_REPLAY` semantic attribute and wired it in `taskContext`
- Propagated `isReplay` through the dequeue system, run attempt system,
and all execution context construction paths (V1 + V2)
- Added `isReplay` to `DequeuedMessage` and
`TaskRunExecutionLazyAttemptPayload` schemas
- Added patch changeset for `@trigger.dev/core`
- Updated docs: added `isReplay` to context reference, added "Detecting
replays" section to replaying page
---
💯
Link to Devin session:
https://app.devin.ai/sessions/1d6f1b3cc39a4623b72d05bf00f2d70c
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: nick <55853254+nicktrn@users.noreply.github.com>
Fork PRs can't access org secrets or push to GHCR, so these two
`pull_request` jobs hard-fail with no path to passing:
- `claude-md-audit` - needs `CLAUDE_CODE_OAUTH_TOKEN`
- `helm-pr-prerelease` `prerelease` job - needs `packages: write` to
push the chart
Hit this on #3449. Approving the run didn't help; the jobs ran and
failed at the privileged step. The chart-validation `lint-and-test` job
is fork-safe and stays untouched - that remains the merge gate for Helm
changes.
Gate both jobs on same-repo head:
```yaml
if: github.event.pull_request.head.repo.full_name == github.repository
```
Other PR workflows already handle forks fine: `pr_checks`
(typecheck/units/e2e/sdk-compat) falls back to anonymous DockerHub pulls
when secrets are missing.
## Summary
When the official Helm chart is deployed with an external ClickHouse and
`clickhouse.external.existingSecret` set — the documented path for not
committing secrets to `values.yaml` — the webapp pod crash-loops on
startup:
```
goose run: parse "http://default:${CLICKHOUSE_PASSWORD}@<host>:8123?secure=false": net/url: invalid userinfo
```
Context in vouch request #3443. Re-opening in draft status per bot
policy (previous attempt was #3445, closed by automation because it
wasn't draft; no changes to the patch).
## Root cause
Two pieces interact:
1. `hosting/k8s/helm/templates/_helpers.tpl` renders `CLICKHOUSE_URL`
(and `RUN_REPLICATION_CLICKHOUSE_URL`) with a shell-style literal
`${CLICKHOUSE_PASSWORD}` expecting bash expansion at container start.
2. `docker/scripts/entrypoint.sh` does `export
GOOSE_DBSTRING="$CLICKHOUSE_URL"` — single-pass POSIX sh substitution,
so the inner `${...}` survives as literal text and goose rejects it.
Reproduces against the latest published chart
(`oci://ghcr.io/triggerdotdev/charts/trigger:4.0.5`) and `main`.
## Fix
Switch the two helpers (external + `existingSecret` branch) from
shell-style `${CLICKHOUSE_PASSWORD}` to Kubernetes'
`$(CLICKHOUSE_PASSWORD)`. Kubelet substitutes `$(VAR)` at pod-creation
time from earlier env entries, and the chart already declares
`CLICKHOUSE_PASSWORD` from the Secret immediately before
`CLICKHOUSE_URL`, so the URL reaches the entrypoint with the real
password already inlined. No entrypoint change, no image change. The
plain-password branch (no `existingSecret`) is unchanged.
Operator caveat added as template comments: `CLICKHOUSE_PASSWORD` must
be URL-userinfo-safe since kubelet substitutes verbatim without
percent-encoding. Hex-encoded passwords (e.g. `openssl rand -hex 32`)
are safe by construction.
## Verification
- `helm template` against `external.existingSecret` now renders `value:
"http://default:$(CLICKHOUSE_PASSWORD)@<host>:8123?secure=false"` (was
`${CLICKHOUSE_PASSWORD}`).
- `helm template` against the plain-password branch is byte-identical to
before.
- Deployed end-to-end on a staging EKS cluster (Meistrari platform):
webapp container reaches `goose: successfully migrated database to
version: 6`, Node.js ClickHouse client connects at runtime.
## Alternatives considered
- **Change `entrypoint.sh`** to `eval` / `envsubst` the URL — larger
surface, touches every deployment mode (Docker Compose + k8s) and every
container image.
- **Mirror the Postgres pattern** (chart reads the full URL via
`valueFrom.secretKeyRef`, as in `trigger-v4.postgres.useSecretUrl`) —
cleaner long-term but requires a new `values.yaml` field and a migration
path for existing users. Happy to follow up with that as a separate PR
if the minimal fix here isn't the preferred direction.
## Changeset
None added — the Helm chart isn't versioned through `@changesets/cli`
(docs/chart-only PRs historically merge without a changeset, e.g.
#2671). Happy to add one if the policy changed.
Closes#3443.
## Summary
Two fixes that together get a fresh-machine setup working from
`CONTRIBUTING.md` end-to-end with no manual workarounds:
### `CONTRIBUTING.md`
- Fix wrong path in the migration walkthrough: `cd packages/database` →
`cd internal-packages/database`. The current path doesn't exist; this
breaks step 2 for every contributor adding a migration.
- Renumber duplicate `4.` steps in **Adding migrations** and the skipped
`5.` in the hello-world **Running** section.
- Combine three sequential `pnpm run build --filter ...` calls into one
(Turbo parallelizes filters): `pnpm run build --filter webapp --filter
trigger.dev --filter @trigger.dev/sdk`.
- Add a `pnpm run db:seed` step after migrate. The seed creates the
local user, `References` org, and reference projects (including
`hello-world` with the stable `proj_rrkpdguyagvsoktglnod`). Removes the
manual instruction to edit the `externalRef` column in Postgres.
- Mention ClickHouse and the ClickHouse migrator alongside
Postgres/Redis in the Docker step (they're already part of `pnpm run
docker`, just invisible in the docs).
- Remove the V1-era **Add sample jobs** section.
`references/job-catalog` no longer exists; the hello-world flow above
replaces it.
### `turbo.json`
Scope `db:seed` to `webapp#db:seed → webapp#build`. The previous
root-level entry queued `build` for every workspace package — including
`references-*`, `docs`, `kubernetes-provider`, `coordinator`, etc. Only
`webapp` actually has a `db:seed` script, so the rest of those builds
were dead weight. Worse: a single broken reference (today,
`references-realtime-hooks-test` failing under Turbopack with
`node:fs/promises`) kills the whole seed pipeline.
After the change, `turbo run db:seed --dry-run` plan drops from 27 tasks
to 20 — only `webapp` and its real transitive workspace deps. Reference
projects no longer block seeding.
## Test plan
- [x] Fresh-machine setup followed end-to-end on a wiped Postgres +
ClickHouse: migrate → seed → build → webapp → CLI login → `trigger dev`
→ triggered `hello-world`, run completed with `{"message":"Hello,
world!"}`.
- [x] `turbo run db:seed --dry-run=json` confirms 20 tasks, all webapp
deps, no reference packages.
- [ ] CI green on the renamed turbo task name.
## Summary
Fixes an RSS-only memory leak in the three realtime proxy routes
(`/realtime/v1/runs`, `/realtime/v1/runs/:id`,
`/realtime/v1/batches/:id`). Client disconnects during an in-flight
long-poll would leave the upstream fetch to Electric running with no way
to abort it, so undici kept the socket open and buffered response chunks
that would never be consumed.
## Root cause
All three routes flow through
`RealtimeClient.streamRun/streamRuns/streamBatch` → `#streamRunsWhere` →
`#performElectricRequest` → `longPollingFetch(url, { signal })`. The
chain was already signal-aware, but `#streamRunsWhere` hardcoded
`signal=undefined` when calling `#performElectricRequest`, so no signal
ever reached `longPollingFetch`.
When a downstream client aborts a long-poll mid-flight:
1. Express tears down the downstream response socket.
2. The `longPollingFetch` promise has already resolved (it returns as
soon as upstream headers arrive) and handed back `new
Response(upstream.body, {...})`.
3. `undici` keeps the upstream socket open and continues buffering
chunks into the `ReadableStream` that nothing will ever read from.
4. The upstream connection is eventually closed by Electric's own poll
timeout (~20s). During that window the per-request buffers stay in
native memory.
These buffers live below V8's accounting — no `heapUsed` or `external`
growth, no sign in heap snapshots, only RSS. An isolated standalone
reproducer (`fetch` against a slow-streaming upstream, discard the
`Response` before consuming its body) measures **~44 KB retained per
leaked request** after GC. That's consistent with the undici socket +
receive buffer + HTTP parser state for a long-lived chunked response.
The pattern is the shape documented in
[nodejs/undici#1108](https://github.com/nodejs/undici/issues/1108) and
[#2143](https://github.com/nodejs/undici/issues/2143).
## What changed
- **`realtimeClient.server.ts`** — add optional `signal` parameter to
`streamRun`, `streamRuns`, `streamBatch`, and the shared
`#streamRunsWhere`; thread it through to `#performElectricRequest`
instead of hardcoding `undefined`.
- **`realtime.v1.runs.$runId.ts`, `realtime.v1.runs.ts`,
`realtime.v1.batches.$batchId.ts`** — pass `getRequestAbortSignal()`
(from `httpAsyncStorage.server.ts`) at the call site. This is the signal
wired to `res.on('close')` and fires reliably on downstream disconnect.
- **`longPollingFetch.ts`** — belt-and-suspenders: cancel the upstream
body explicitly in the error path, and treat `AbortError` as a clean
`499` instead of a `500`. This both releases undici's buffers
deterministically on error and avoids spurious 500s in request logs when
a client legitimately walks away.
## Verification
Standalone reproducer: slow upstream server streams 32 KB chunks every
100 ms for 5 seconds per request. The proxy does `fetch(url)` with
varying signal/cancel strategies, creates `new Response(upstream.body,
...)`, and discards it without consuming the body (simulating the leak
path).
Results from 1 000 parallel fetches per variant, measured post-GC:
| variant | Δ heap | Δ external | Δ RSS |
| --- | --- | --- | --- |
| A. no signal, body never consumed (the bug) | +0.3 MB | 0 MB | **+59.4
MB** |
| B. signal propagated, aborted after headers (this fix) | −0.1 MB | 0
MB | +15.4 MB |
| C. no signal, explicit `res.body.cancel()` | 0 MB | 0 MB | −25.4 MB |
10-round sustained test of variant B to distinguish accumulating
retention from one-time allocator overhead:
```
round 1/10 Δ=+3.2 MB round 6/10 Δ=-12.5 MB
round 2/10 Δ=-7.6 MB round 7/10 Δ=-11.9 MB
round 3/10 Δ=-11.7 MB round 8/10 Δ=-2.6 MB
round 4/10 Δ=+3.2 MB round 9/10 Δ=-8.0 MB
round 5/10 Δ=-1.2 MB round 10/10 Δ=-12.6 MB
```
RSS oscillates in a 49-65 MB band with no upward trend — signal
propagation fully releases the buffers.
## Risk
- Behavior change only on aborted long-polls: the upstream fetch now
cancels promptly instead of running to its natural timeout. This saves
both memory and outbound traffic to Electric.
- `AbortError` now surfaces as `499` rather than `500`. Any dashboard or
alert that counts 500s in request logs will see slightly fewer of them;
this is the intended behavior.
- Signal-aware parameter is optional on
`RealtimeClient.streamRun/streamRuns/streamBatch`, so callers that don't
opt in get the previous behavior.
## Test plan
- [ ] Existing realtime integration tests pass
- [ ] Dashboard realtime views (runs list, batch details) continue
working normally across tab open/close cycles
- [ ] Under a burst of aborted long-polls, server RSS returns to
baseline rather than climbing
Adds @GautamBytes to the vouch list so they can contribute to the
repository.
Closes#3307
Co-authored-by: claude[bot] <41898282+claude[bot]@users.noreply.github.com>
Co-authored-by: Eric Allam <ericallam@users.noreply.github.com>
Adds a minimal end-to-end test harness that spawns the compiled webapp
as a child
process against a throwaway Postgres container, plus a baseline of 8
auth-behaviour
tests. These tests will be used as a regression check before and after
the upcoming
apiBuilder RBAC migration to confirm auth behaviour is unchanged.
## What's included
**`internal-packages/testcontainers/src/webapp.ts`** (new)
Spawns `build/server.js` with a dynamically allocated port, polls
`/healthcheck`,
and exposes `WebappInstance` and `startTestServer()` (postgres container
+ webapp +
PrismaClient in one call). Key details:
- Uses `process.execPath` so the correct Node binary is found in forked
test processes
- Sets `NODE_PATH` to `node_modules/.pnpm/node_modules` so pnpm-hoisted
transitive
deps (e.g. `eventsource-parser`) resolve correctly inside the subprocess
- Overrides both `PORT` and `REMIX_APP_PORT` so Vite's automatic `.env`
loading
doesn't override the dynamically allocated port
**`internal-packages/testcontainers/package.json`**
Adds `./webapp` sub-path export so tests can `import from
"@internal/testcontainers/webapp"`.
**`internal-packages/testcontainers/src/index.ts`**
Exports `createPostgresContainer` (used internally by `webapp.ts`).
**`apps/webapp/test/helpers/seedTestEnvironment.ts`** (new)
Creates a minimal org → project → environment row set with random
suffixes.
**`apps/webapp/test/api-auth.e2e.test.ts`** (new)
8 tests across two suites:
- API-key bearer: valid key (auth passes, 404), missing header (401),
invalid key (401), error body shape
- JWT bearer: valid JWT on JWT-enabled route (passes), valid JWT on
non-JWT route (401), empty-scope JWT (403), wrong signing key (401)
## How to run
```bash
# Build required first (one-time)
pnpm run build --filter webapp
cd apps/webapp && pnpm exec vitest run test/api-auth.e2e.test.ts
```
## Test plan
- [x] All 8 tests pass against the current webapp build
- [x] Webapp healthcheck returns 200 on startup
- [ ] CI passes
---------
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Adds `KUBERNETES_POD_DNS_NDOTS_OVERRIDE_ENABLED` flag (off by default)
that overrides the cluster default and sets `dnsConfig.options.ndots` on
runner pods (defaulting to 2, configurable via
`KUBERNETES_POD_DNS_NDOTS`).
Kubernetes defaults pods to `ndots: 5`, so any name with fewer than 5
dots, including typical external domains like `api.example.com`, is
first walked through every entry in the cluster search list
(`<ns>.svc.cluster.local`, `svc.cluster.local`, `cluster.local`) before
being tried as-is, turning one resolution into 4+ CoreDNS queries (×2
with A+AAAA).
Using a lower `ndots` value reduces DNS query amplification in the
`cluster.local` zone.
## Summary
- The `processMasterQueueForEnvironment` call in the priority test was
racing against background `processQueueForWorkerQueue` jobs scheduled
50ms after each trigger
- With a 50ms debounce (`processWorkerQueueDebounceMs: 50`) and runs
triggered sequentially, the RunQueue Worker could process those jobs
mid-sequence, pushing partial batches to the worker queue in the wrong
overall priority order
- `masterQueueConsumersDisabled: true` only blocks the shard-level
polling loops — it does not prevent the RunQueue's own Worker from
processing these debounced jobs
- Fix: add `worker.disabled: true` to the test 1 engine config, which
propagates to `workerOptions.disabled` in the RunQueue constructor and
prevents the Worker from starting
## Test plan
- [x] Both priority tests pass: `pnpm run test
./src/engine/tests/priority.test.ts --run`
- [x] Test 1 log confirms no `✅ Starting run engine worker` or worker
loop messages — workers fully disabled
- [x] Test 2 unaffected (uses master queue consumers for automatic
promotion, no `disabled` flag added)
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Wires up automatic Helm chart releases to ride along with the existing
changeset-driven package release flow.
Today `Chart.yaml` is bumped by hand and `release-helm.yml` fires only
when a human pushes a `helm-v*` tag. With this, the changeset release PR
also carries a `Chart.yaml` bump so main always matches the published
version, and `release.yml` invokes `release-helm.yml` via
`workflow_call` after Docker images are published.
`helm-v${VERSION}` tag is pushed as a marker (same GITHUB_TOKEN trick as
`v.docker.*`). Manual `helm-v*` tag flow still works. Chart.yaml
consistency check in `release-helm.yml` is the safety net if the bump
job ever drifts.
First rollout: the open `changeset-release/main` PR has stale
Chart.yaml. Bump it manually on that branch before merging, otherwise
the first automated helm release fails at the consistency check.
## Summary
Adds direct V8 heap and process-memory gauges to the webapp's
OpenTelemetry meter. The webapp already exports per-cluster-worker
Node.js runtime metrics (event-loop lag / utilization, active handles,
active requests, libuv threadpool size) via a custom meter under the
`trigger.dev` scope. Heap and memory were missing; this PR adds them
alongside, in the same observable-batch pattern.
## New gauges
| Metric | Source | Unit |
| --- | --- | --- |
| `nodejs.memory.heap.used` | `process.memoryUsage().heapUsed` | bytes |
| `nodejs.memory.heap.total` | `process.memoryUsage().heapTotal` | bytes
|
| `nodejs.memory.heap.limit` | `v8.getHeapStatistics().heap_size_limit`
| bytes |
| `nodejs.memory.external` | `process.memoryUsage().external` | bytes |
| `nodejs.memory.array_buffers` | `process.memoryUsage().arrayBuffers` |
bytes |
| `nodejs.memory.rss` | `process.memoryUsage().rss` | bytes |
Gated by the existing `INTERNAL_OTEL_NODEJS_METRICS_ENABLED` flag, same
as the adjacent event-loop / handle gauges. Zero overhead when disabled.
## Why
`@opentelemetry/host-metrics` publishes `process.memory.usage`, which is
RSS only. RSS is the sum of V8 heap, external memory (Buffers, etc.),
native code, and thread stacks. Without a direct heap metric it is not
possible to size the V8 heap cap (`--max-old-space-size`) from metrics
alone, because RSS overstates heap by the external + native footprint. A
worker can have a 4 GB RSS with a 2.5 GB heap and 1.5 GB of buffers; the
former constrains `--max-old-space-size`, the latter does not.
`nodejs.memory.heap.limit` also surfaces the configured
`--max-old-space-size` (read from
`v8.getHeapStatistics().heap_size_limit`), so operators can see the
current limit in the same dashboard as actual usage rather than
cross-referencing container environment variables.
## Risk
Minimal. Observable gauges are sampled at the configured metric-export
interval. `v8.getHeapStatistics()` and `process.memoryUsage()` are each
microsecond-level calls, and six gauges are added to the same batch
callback that already reads ~20 other Node.js runtime values per sample.
Same registration pattern as the existing event-loop metrics in the
file.
## Test plan
- [ ] Deploy and confirm the six new gauges appear at the configured
exporter
- [ ] In cluster mode, confirm per-worker granularity (one series per
cluster worker, tagged by `process.executable.name` /
`service.instance.id`)
- [ ] Confirm `nodejs.memory.heap.limit` reports the configured
`--max-old-space-size` value in bytes
## Summary
- New **Back office** tab at `/admin`, per-org detail page at
`/admin/back-office/orgs/:orgId` designed to host future per-org admin
actions (project count, delete account, YC deals).
- First action: edit an organization's API rate limit — tokenBucket
override (refill rate, interval, max tokens), with a live plain-English
preview (e.g. *"1,500 requests per minute · 750 request burst
allowance"*). Writes are audit-logged via the server logger.
- Cleanup: removed unused `v2?` / `v3?` columns from the admin orgs list
(display only — Prisma select untouched).
## Test plan
- [ ] Back office tab visible in admin nav and highlighted when on a
sub-route
- [ ] `/admin/orgs` shows a Back office "Open" link per row; no v2/v3
columns
- [ ] Empty state at `/admin/back-office` links back to `/admin/orgs`
- [ ] Detail page renders the effective rate limit in view mode; Edit
reveals the form
- [ ] Save writes `Organization.apiRateLimiterConfig`, returns to view
mode, shows "Rate limit saved." banner
- [ ] Invalid values surface inline field errors and keep edit mode
- [ ] Non-admins hitting any new route are redirected to `/`
- [ ] Server logs show `admin.backOffice.rateLimit` info line per
mutation
## Summary
Fixes a server-side memory leak in the webapp's SSE helper. Every
aborted SSE connection (client tab close, navigation, timeout) was
pinning its full request/response graph indefinitely on Node 20, so any
long-running webapp process accumulated retained memory proportional to
streaming-request churn.
## Root cause
`apps/webapp/app/utils/sse.ts` combined four abort signals via
`AbortSignal.any([requestAbortSignal, timeoutSignal,
internalController.signal])`. The composite signal tracks its source
signals in an internal `Set<WeakRef>` registered against a
`FinalizationRegistry`; under sustained traffic those entries accumulate
faster than they're cleaned up, pinning every source signal (and its
listeners, and anything those listeners close over) until the parent
signal itself is GC'd or aborts.
This is a long-standing Node issue with multiple open reports:
- [nodejs/node#54614](https://github.com/nodejs/node/issues/54614) —
original report, still open. A [follow-up from
ChainSafe](https://github.com/nodejs/node/issues/54614#issuecomment-4055656572)
describes the exact same shape in a Lodestar production workload (req +
timeout signals composed per request accumulating in long-running
worker) and the same mitigation: drop `AbortSignal.any`, compose
manually.
- [nodejs/node#55351](https://github.com/nodejs/node/issues/55351) —
mechanism confirmed by Node member @jasnell: *"the set of dependent
signals known to the AbortSignal are kept in an internal Set using
WeakRefs. The AbortSignals are being properly gc'd but the Set is never
cleaned out of the WeakRefs making those leak."* Partially fixed by [PR
#55354](https://github.com/nodejs/node/pull/55354), shipped in Node
22.12.0 — but only covers the tight-loop case, not long-lived parent
signals.
- [nodejs/node#57584](https://github.com/nodejs/node/issues/57584) —
circular-dependency variant, still open.
- [nodejs/node#62363](https://github.com/nodejs/node/issues/62363) —
regression in Node 24/25 from an unrelated V8 change ("Don't pretenure
WeakCells"). Different root cause, same symptom.
A separate issue in `apps/webapp/app/entry.server.tsx` —
`setTimeout(abort, ABORT_DELAY)` with no `clearTimeout` on success paths
— kept the React render tree + `remixContext` alive for 30s per
successful HTML request. Same pattern fixed upstream in React Router
templates
([react-router#14200](https://github.com/remix-run/react-router/pull/14200)),
never backported to Remix v2.
## What changed
- **`apps/webapp/app/utils/sse.ts`** — single-signal abort chain.
`AbortSignal.any` removed; `AbortSignal.timeout` replaced by a plain
`setTimeout` cleared when the controller aborts; named sentinel
constants used as stackless abort reasons; request-abort handler
explicitly removed on cleanup.
- **`apps/webapp/app/entry.server.tsx`** — clears the `setTimeout(abort,
ABORT_DELAY)` timer in `onShellReady` / `onAllReady` / `onShellError`.
- **`apps/webapp/app/v3/tracer.server.ts` + `env.server.ts`** — gates
OpenTelemetry `HttpInstrumentation` and `ExpressInstrumentation` behind
`DISABLE_HTTP_INSTRUMENTATION=true` as an escape hatch for future
OTel-listener retention patterns. Defaults to enabled.
- **`apps/webapp/app/presenters/v3/RunStreamPresenter.server.ts`** —
uses the shared `ABORT_REASON_SEND_ERROR` sentinel.
## Verification
### Full-app reproduction (memlab)
Isolated local harness, 500 abrupt SSE disconnects against a
dev-presence route, GC between passes, heap snapshot diff with
[memlab](https://facebook.github.io/memlab/):
| Run | Heap delta after 500 conns + GC | memlab retained leaks |
| --- | --- | --- |
| Before | +16.0 MB (linear with request count) | 158 clusters; 250
`ServerResponse`, 1000 `AbortController`, 250 `SpanImpl` retained |
| After | **+3.3 MB (noise)** | **0 app-code leaks** |
### Standalone mechanism isolation
To confirm *which* axis of the change is load-bearing, a separate
standalone Node script (`/tmp/abort-leak-test.mjs`) ran 2000 requests ×
200 KB payload per variant:
| Variant | Heap delta after GC |
| --- | --- |
| baseline (no signal machinery) | 0 MB |
| V1: `AbortSignal.any` + string abort reason | **+9.1 MB** |
| V2: `AbortSignal.any` only (no reason) | **+10.8 MB** |
| V3: string reason only (no `AbortSignal.any`) | 0 MB |
| V4: neither (the fix) | 0 MB |
| V5: `AbortSignal.any` with no listener on the composite | **+10.2 MB**
|
This proves `AbortSignal.any` is the sole mechanism. The reason type
(`.abort()` vs `.abort("string")`) is irrelevant for retention — V3 is
clean, V5 leaks even without a listener on the composite.
## Risk
- `sse.ts` is used by the dev-presence routes. Behaviour is equivalent —
timeouts and client disconnects still abort the stream. `signal.reason`
is now a named string sentinel (`"timeout"`, `"request_aborted"`, etc.)
instead of the previous string arg or default `AbortError`. No in-tree
reader of `signal.reason` exists.
- `entry.server.tsx` change is a standard cleanup of an abort timer,
matches upstream React Router guidance.
- `tracer.server.ts` change is env-gated and defaults to current
behaviour.
- Three other webapp `AbortSignal.timeout()` callsites (alert delivery,
remote-build status) are fire-and-forget passed directly to `fetch` —
not composed with anything long-lived, no retention risk, untouched.
## Test plan
- [ ] Existing SSE integration tests pass
- [ ] Dev-presence SSE behaves normally across tab open/close cycles
- [ ] No heap growth under sustained aborted-connection traffic (heap
snapshot diff)
## Follow-up
The same `AbortSignal.any([userSignal, internalSignal])` pattern exists
in several SDK/core callsites that ship to customers
(`packages/core/src/v3/realtimeStreams/manager.ts`,
`packages/trigger-sdk/src/v3/{ai,chat,chat-client,sessions}.ts`,
`packages/core/src/v3/workers/warmStartClient.ts`). Whether those leak
in practice depends on the user passing a long-lived signal. Tracked
separately.
Mirrors the existing `supervisor.serviceAccount` pattern onto webapp so
operators can annotate the SA (IRSA `eks.amazonaws.com/role-arn`,
Workload Identity, etc.) or bring their own SA. Without this,
`webapp.serviceAccount.annotations` isn't exposed and operators have to
patch the SA out-of-band.
```yaml
webapp:
serviceAccount:
create: true
name: ""
annotations:
eks.amazonaws.com/role-arn: arn:aws:iam::123456789012:role/trigger-webapp
```
Three pieces, same as supervisor:
- `webapp.serviceAccount.create` toggle on the SA block
- `webapp.serviceAccount.annotations` + `name` values
- `trigger-v4.webappServiceAccountName` helper, used by the SA, the
token-syncer RoleBinding subject, and the Deployment's
`serviceAccountName`
Role + RoleBinding are left unguarded (matching supervisor's shape where
`rbac.create` is a separate toggle from `serviceAccount.create`) -
BYO-SA users take on the responsibility of ensuring the SA they supply
has the permissions the RoleBinding grants.
Verified with `helm template` against default values, an IRSA annotation
override, and `create: false` with a custom name.
## Problem
When `batchTrigger()` is called with large payloads, each item's payload
is uploaded to R2 server-side during the streaming loop before being
enqueued. This makes the loop slow — around 3 seconds per item. Workers
pick up and execute each item as it's enqueued, running concurrently
with the ongoing stream.
For the last item in the batch, a race exists between the streaming loop
finishing and the batch completion cleanup:
1. The loop enqueues the last item and returns from `enqueueBatchItem()`
2. A waiting worker picks up the item almost instantly and executes it
3. `recordSuccess()` fires, `processedCount` hits the expected total,
`finalizeBatch()` runs
4. `cleanup()` deletes all Redis keys for the batch, including
`enqueuedItemsKey`
5. The streaming loop exits and calls `getBatchEnqueuedCount()` — reads
the now-deleted key — returns 0
The count check finds `enqueuedCount (0) !== batch.runCount`, falls
through to a Postgres fallback, but the fallback only checked `sealed`.
The BatchQueue completion path sets `status = COMPLETED` in Postgres
without setting `sealed = true` (that's the streaming endpoint's job),
so the fallback misses it too.
This causes the endpoint to return `sealed: false`. The SDK treats this
as retryable and retries up to 5 times with exponential backoff. Each
retry calls `enqueueBatchItem()`, which reads the batch meta key from
Redis — also deleted by `cleanup()` — and throws "Batch not found or not
initialized" (500). The final retry gets a 422 because the batch is
already COMPLETED, which the SDK does not retry, causing an `ApiError`
to be thrown from `await batchTrigger()` in the parent run — even though
all child runs completed successfully.
## Fix
In the Postgres fallback inside `StreamBatchItemsService`, also check
`status === "COMPLETED"` alongside `sealed`. This covers the
fast-completion path where the BatchQueue finishes all runs before the
streaming endpoint gets to seal the batch normally.
Also switches `findUnique` to `findFirst` per webapp convention.
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
## Summary
Adds `RUN_ENGINE_READ_REPLICA_SNAPSHOTS_SINCE_ENABLED` (default `"0"`).
When enabled, the Prisma reads inside `RunEngine.getSnapshotsSince` run
against the read-only replica client instead of the primary. Offloads
the snapshot-polling queries fired by every running task runner off the
writer.
## Why
`getSnapshotsSince` is called from the managed runner's
fetch-and-process loop (once per poll interval, plus on every
snapshot-change notification). It runs four sequential reads per call —
one `findFirst` by snapshot id, one `findMany` on snapshots with
`createdAt > X`, one raw SQL against `_completedWaitpoints`, and chunked
`findMany` on `waitpoint`. Per concurrent run, every few seconds. It's
read-only, tolerates a small amount of staleness, and is an obvious
candidate for the replica.
## Replica-lag considerations
- **Step 1 "since snapshot not found"**: if the runner just received a
snapshot id from the primary and asks the replica before it replicates,
the function throws and the caller treats the response as an error
(runner falls back to a metadata refresh). Self-correcting, not silent.
- **Step 2 missing newly-created snapshots**: the next poll's `createdAt
> sinceSnapshot.createdAt` filter still picks them up once the replica
catches up.
- **Waitpoint junction race**: the riskiest path — if a latest snapshot
is replicated but its `_completedWaitpoints` join rows aren't yet, the
runner could advance past that snapshot with `completedWaitpoints: []`.
WAL/storage-level replication replays commits in order, so in practice
both should appear atomically on the reader, but the race window is why
the flag ships disabled.
Aurora reader shrinks all three windows to single-digit ms in typical
conditions, and its storage-level replication gives atomic visibility of
committed transactions on the reader.
## Test plan
- [ ] Flip the flag on in a non-prod environment, confirm snapshot
polling behaves normally and `getSnapshotsSince` errors in Sentry stay
flat.
- [ ] Verify writer query volume drops and reader query volume rises on
the snapshot-polling queries.
- [ ] Keep an eye on `AuroraReplicaLag` (or equivalent) during rollout.
## Summary
Stamp every Prisma span with `db.datasource: "writer" | "replica"` so
traces can distinguish which client the query went through.
Both `PrismaClient` instances share the same global
`@prisma/instrumentation`, so their spans come out with identical names
and attributes today. This makes them trivially filterable.
## How
Two pieces in `apps/webapp/app/`:
1. **`v3/tracer.server.ts`** — a `DatasourceAttributeSpanProcessor`
reads an OTel context key in `onStart` and calls
`span.setAttribute("db.datasource", value)`. Registered as the first
span processor.
2. **`db.server.ts`** — `tagDatasource(datasource, client)` wraps each
`PrismaClient` with `$extends({ query: { $allOperations } })`. The
middleware sets the context key around the query and directly tags the
active span (to catch `prisma:client:operation`, which Prisma creates
before the middleware fires).
### Context-propagation gotcha
`PrismaPromise` is lazy — `query(args)` returns a thenable that only
starts when someone `.then()`s it. The naive `context.with(ctx, () =>
query(args))` restores ALS synchronously, so when Prisma's internal code
awaits the thenable later, the engine spans fire with the original ALS.
Wrapping as `async () => await query(args)` forces the `.then()` inside
the `context.with` callback, so ALS stays on our context for the engine
spans.
### Coverage
- **Tagged**: all `prisma:engine:*` (`connection`, `db_query`,
`serialize`, `query`, etc.), `prisma:client:operation`,
`prisma:client:serialize`, `prisma:client:connect`
- **Not tagged**: `prisma:client:load_engine` — one-time startup, fires
before any query
Concurrent `Promise.all([writer.x, replica.y])` correctly tags each pool
separately (ALS isolates per-Promise chain).
### Performance
One `context.with` (~200ns) and one `setAttribute` per span (effectively
free per OTel JS benchmarks) per Prisma op. Negligible against a query
path measured in milliseconds.
## Test plan
- [ ] Verify `db.datasource` appears on `prisma:engine:connection` spans
after the webapp is restarted
- [ ] Spot-check a handful of real traces carry the attribute
The cancel button was missing from the runs list for runs in `DEQUEUED`
status. The runs list gates the button on `run.isCancellable`, which
goes through `isCancellableRunStatus` -> `CANCELLABLE_RUN_STATUSES` =
`NON_FINAL_RUN_STATUSES`. `DEQUEUED` was never added to that list when
it was introduced in the run engine.
The single run page uses a separate check (`!run.isFinished`, i.e. the
inverse of `FINAL_RUN_STATUSES`), so cancellation already worked there -
only the list was affected.
Adding `DEQUEUED` to `NON_FINAL_RUN_STATUSES` also flips
`isCrashableRunStatus` and `isFailableRunStatus`, but:
- The crash path is the right behaviour - a `DEQUEUED` run (worker has
claimed but not yet executing) can legitimately crash before
`EXECUTING`, same as `PENDING`/`DELAYED` already do.
- The fail path (`failedTaskRun.server.ts`) is only reached from V1 code
paths (marqs consumers, v1 heartbeat handler). `DEQUEUED` is a
V2-engine-only status, so V1 consumers never see it.
When cancelling a `DEQUEUED` run the execution snapshot goes to
`PENDING_CANCEL` (worker must ack) but `TaskRun.status` flips to
`CANCELED` immediately - the UI reflects cancellation without waiting
for the worker. Added an integration test in
`run-engine/src/engine/tests/cancelling.test.ts` covering the full
trigger -> dequeue -> cancel -> worker-ack flow.
## Stall safety
The stall recovery path (PENDING_EXECUTING heartbeat miss ->
nack-and-requeue -> back to QUEUED) lives entirely inside
`@internal/run-engine` and never touches the webapp's `taskStatus.ts`
helpers - the engine has zero imports from `~/v3/taskStatus` and doesn't
know `CrashTaskRunService` / `FailedTaskRunService` exist. A stalled
DEQUEUED run still goes back to the queue for retry; this change cannot
cause stalls to crash or fail.
The only realistic impact is the intended UI fix - the theoretical V1
crash/fail branches for DEQUEUED are unreachable in practice because V1
runs never have DEQUEUED status.
## Summary
Regenerating a RuntimeEnvironment API key no longer immediately
invalidates the previous one. Rotation is now overlap-based: the old key
keeps working for 24 hours so customers can roll it out in their env
vars without downtime, then stops working.
## Design
- **New `RevokedApiKey` table** (one row per revocation). Holds the
archived `apiKey`, a FK to the env, an `expiresAt`, and a `createdAt`.
Indexed on `apiKey` (high-cardinality equality — single-row hits) and on
`runtimeEnvironmentId`.
- **`regenerateApiKey` wraps both writes in a single `$transaction`:**
insert a `RevokedApiKey` with `expiresAt = now + 24h`, update the env
with the new `apiKey`/`pkApiKey`.
- **`findEnvironmentByApiKey` does a two-step lookup:** primary
unique-index hit on `RuntimeEnvironment.apiKey` first; on miss,
`RevokedApiKey.findFirst({ apiKey, expiresAt: { gt: now } })` with an
`include: { runtimeEnvironment }`. Two-step (not `OR`-join) keeps the
hot path identical to today and puts the fallback cost only on invalid
keys. Both lookups use `$replica`.
- **Admin endpoint** `POST /admin/api/v1/revoked-api-keys/:id` accepts
`{ expiresAt }` and updates the row. Setting to `now` ends the grace
window immediately; setting to the future extends it.
- **Modal copy** on the regenerate dialog updated — previously warned of
downtime, now explains the 24h overlap.
## Why a separate table instead of columns on `RuntimeEnvironment`
- Keeps the hot auth path's primary lookup unchanged — no
OR/nullable-apiKey semantics to reason about.
- Naturally supports multiple in-flight grace windows (regenerate twice
in a day → two old keys valid until their independent expiries).
- FK + cascade cleans up correctly when an env is deleted; nothing to
backfill.
## Test plan
Verified locally against hello-world with dev and prod env keys:
- [x] baseline — current key authenticates (`GET /api/v1/runs`) → `200`
- [x] regenerate via UI — DB shows old key in `RevokedApiKey` with
`expiresAt ≈ now+24h`, env has new key
- [x] grace window — both old and new keys → `200`; bogus key → `401`
- [x] admin endpoint: `expiresAt = now` → old key `401`
- [x] admin endpoint: `expiresAt = +1h` (after early-expire) → old key
`200` again
- [x] admin endpoint: `expiresAt = past` → old key `401`
- [x] admin 400 (invalid body), 404 (unknown id), 401 (missing/non-admin
PAT)
- [x] same flow exercised end-to-end on a PROD-typed env — behavior
identical
- [x] `pnpm run typecheck --filter webapp` passes
## ✅ Checklist
- [x] I have followed every step in the [contributing
guide](https://github.com/triggerdotdev/trigger.dev/blob/main/CONTRIBUTING.md)
- [x] The PR title follows the convention.
- [x] I ran and tested the code works
---
## Summary
Adds a server-side gate that detects deploy attempts from v3 CLI
versions (i.e. `trigger.dev@3.x`) at the `POST /api/v1/deployments`
entry point and, when enabled, rejects them with a clear upgrade
message. v4 CLI deploys are completely unaffected.
The last 3.x CLI release was `3.3.7`, which we can't update. This
approach short-circuits the deploy before any DB writes, image-ref
generation, S2 stream creation, or queue enqueue — no side effects in
either mode.
## How v3 vs v4 are distinguished
I pulled the published CLI tarballs for `trigger.dev@3.3.7`, `4.0.0`,
`4.0.1`, `4.0.5`, `4.1.0`, `4.2.0`, and the current `4.4.4` in the repo.
The cleanest, most reliable signal is the request body to `POST
/api/v1/deployments`:
| Field on initialize | v3.3.7 CLI | v4.x CLI |
|---|---|---|
| `type` | **never sent** | always sent — `"MANAGED"` (run_engine_v2) or
`"V1"` |
| `isNativeBuild` / `gitMeta` / `triggeredVia` / `runtime` | not sent |
sent |
| `registryHost` / `namespace` | sent (v3-only; stripped by current Zod
schema) | not sent |
Every v4 call site I inspected sets `type: features.run_engine_v2 ?
"MANAGED" : "V1"` unconditionally. `payload.type` is `undefined` if and
only if the client is a 3.x CLI.
## Behavior
- Detection always runs and emits `logger.warn("Detected deploy from
deprecated v3 CLI", { environmentId, projectId, organizationId, enforced
})`, which lets us watch how many v3 deploys are still happening before
enforcement is flipped.
- Enforcement is gated behind `DEPRECATE_V3_CLI_DEPLOYS_ENABLED`
(default `"0"`, off). When `"1"`, the server returns `400` with:
> The trigger.dev CLI v3 is no longer supported for deployments. Please
upgrade your project to v4: https://trigger.dev/docs/migrating-from-v3
The v3 CLI surfaces this verbatim as `Failed to start deployment:
<message>` because `zodfetch` throws `ApiError` for non-retryable 4xx
(400/422) and `deploy.js` in 3.3.7 prints `error.message`.
## Out of scope (intentionally)
- `api.v1.deployments.$deploymentId.finalize.ts` /
`FinalizeDeploymentService` /
`createDeploymentBackgroundWorkerV3.server.ts` are V1-engine paths, not
the v3 CLI gate. Leaving them alone per review.
- Container-side `createDeploymentBackgroundWorker` call in
`managed-index-controller.ts` is still used by v4's in-image indexer.
Not touched.
- v3 `trigger dev` flow (different code path) — separate deprecation
if/when needed.
## Testing
- Ran `pnpm run typecheck --filter webapp` locally — passes.
- Verified v4 tarballs (4.0.0, 4.0.1, 4.0.5, 4.1.0, 4.2.0, 4.4.4) all
include `type:` in the `initializeDeployment` call site, so none will be
accidentally blocked.
- Verified v3.3.7 tarball's `initializeDeployment` payload has no `type`
field.
Rollout plan after merge:
1. Deploy with `DEPRECATE_V3_CLI_DEPLOYS_ENABLED` unset → watch
`Detected deploy from deprecated v3 CLI` log volume.
2. When comfortable, set `DEPRECATE_V3_CLI_DEPLOYS_ENABLED=1` to
enforce.
---
## Changelog
Detect v3 CLI deploys on `/api/v1/deployments` and, when
`DEPRECATE_V3_CLI_DEPLOYS_ENABLED=1`, reject them with an upgrade
message pointing at https://trigger.dev/docs/migrating-from-v3. v4 CLI
deploys are unaffected.
Link to Devin session:
https://app.devin.ai/sessions/b242c11bd86e4099aeec8b59bab62143
Requested by: @ericallam
Example cURL call using an admin user PAT (replace with a real one):
```sh
curl -X PUT https://cloud.trigger.dev/admin/api/v1/environments/<environmentId>/burst-factor \
-H "Authorization: Bearer tr_pat_1234" \
-H "Content-Type: application/json" \
-d '{"burstFactor": 1.5}'
```
Adds a `MicroVM` badge next to the region name on the regions page. Uses
the existing `small` badge variant for visual consistency with the
`Default` badge already on this page.
## Summary
Large error stacks and messages can OOM the worker process when
serialized into OTel spans or `TaskRunError` objects. This was reported
when throwing an error with a massive `.stack` property from a chat
agent hook.
This adds frame-based stack truncation (similar to Sentry's approach)
plus message length limits, applied consistently across all error
serialization paths.
### What changed
**`packages/core/src/v3/errors.ts`**
- `truncateStack()` — parses `error.stack` into message lines + frame
lines, caps at 50 frames (keep top 5 closest to throw + bottom 45 entry
points, with "... N frames omitted ..." in between). Individual lines
capped at 1024 chars.
- `truncateMessage()` — caps error messages at 1000 chars
- Applied in `parseError()` and `sanitizeError()`
**`packages/core/src/v3/otel/utils.ts`**
- `sanitizeSpanError()` now uses `truncateStack` and `truncateMessage`
from `errors.ts` instead of duplicating truncation logic
- Non-Error values (strings, JSON) capped at 5000 chars
**`packages/core/src/v3/tracer.ts`**
- `startActiveSpan` catch block now delegates to `recordSpanException()`
instead of calling `span.recordException()` directly
### Limits
| What | Limit | Rationale |
|------|-------|-----------|
| Stack frames | 50 | Matches Sentry's `STACKTRACE_FRAME_LIMIT` |
| Top frames kept | 5 | Closest to throw site |
| Bottom frames kept | 45 | Entry points / framework frames |
| Per-line length | 1024 | Matches Sentry, prevents regex DoS |
| Message length | 1000 | Bounded but generous |
| Generic string (non-Error) | 5000 | Fallback for JSON/string errors in
spans |
## Test plan
- [x] 17 unit tests in `packages/core/test/errors.test.ts`
- [x] E2E: threw a 300-frame / 5000-char-message error in the ai-chat
reference app, verified truncated stack and message in span via
`get_span_details`
- [x] Verified the run survived the error (no OOM, continued waiting for
next message)
Two changes to cut error volume from logs that represent handled
conditions, not real errors (combined ~1600/hr in prod):
1. api.v1.waitpoints.tokens.$waitpointFriendlyId.complete.ts
The route throws `json(..., { status: 404 })` when a waitpoint
isn't found, but the generic catch block caught that Response,
logged it as an error (with an empty {} body because Error fields
are non-enumerable), and rethrew as a 500 — so clients saw a 500
instead of the intended 404, and every stale-waitpoint request
produced a Sentry event.
Fix: re-throw Response objects unchanged so the correct status
propagates and we don't log user 404s as errors. Also serialize
remaining Error instances explicitly (name/message/stack) so the
logs are actionable when we do hit a real error.
2. v3/marqs/sharedQueueConsumer.server.ts:603
"Task run has invalid status for execution. Going to ack" — the
message itself says we're handling it gracefully. Benign race
between dequeue and completion/cancellation. Demote to warn.
## Summary
Nests the `TaskRunExecutionSnapshot` creation inside the
`taskRun.update()` Prisma call in the dequeue flow, reducing **2 DB
commits → 1** per dequeue operation. This is the highest-volume of the
five unmerged flows identified in TRI-8450 (~9,200 commits/sec on the
engine service).
**Pattern**: Follows the same nested-write approach already used in the
completion path (`runAttemptSystem.ts:735`) and trigger path
(`engine/index.ts:674`).
**Changes**:
- `dequeueSystem.ts`: Moved snapshot creation into `executionSnapshots:
{ create: {...} }` within the existing `taskRun.update()`. Pre-generates
the snapshot ID via `generateInternalId()` (plain cuid, matching what
Prisma's `@default(cuid())` produces) so the event emission, heartbeat
enqueue, and return value can all be constructed from data already in
scope — **no extra DB read needed** after the merged write.
`SnapshotId.toFriendlyId()` is used only for the return value's
`friendlyId` field, matching the original `createExecutionSnapshot`
behavior.
- `executionSnapshotSystem.ts`: Added public
`enqueueHeartbeatIfNeeded()` method that exposes the heartbeat
scheduling logic (previously only available internally via
`createExecutionSnapshot`). This is needed because `PENDING_EXECUTING`
requires a heartbeat, unlike the `FINISHED` status in the completion
reference pattern. This method is reusable by future merge targets
(retry-immediate, checkpoint, cancel, requeue).
**Net DB change per dequeue**: eliminates 1 write transaction (the
separate `TaskRunExecutionSnapshot.create`). No extra reads added — the
snapshot ID is pre-generated and the `executionSnapshotCreated` event
payload is constructed inline from values already available in the
closure.
## Review & Testing Checklist for Human
- [ ] **Verify manually-constructed event payload matches DB state**:
The `executionSnapshotCreated` event is now built inline (not read back
from DB). Confirm the field values (`runStatus: "PENDING"`,
`attemptNumber`, `checkpointId`, `workerId`, `runnerId`,
`completedWaitpointIds`) match what Prisma actually writes. A mismatch
here would be silent — event consumers would get stale/wrong data.
- [ ] **Verify `attemptNumber` source is equivalent**: Old code used
`lockedTaskRun.attemptNumber` (post-update result). New code uses
`result.run.attemptNumber` (pre-update). The `taskRun.update()` data
payload does NOT include `attemptNumber`, so they should be identical —
but confirm this assumption holds for all dequeue scenarios (e.g.
retried runs).
- [ ] **Verify `isValid` defaults to `true` in schema**: The old
`createExecutionSnapshot` explicitly set `isValid: error ? false :
true`. The nested create omits `isValid` (no error in the dequeue happy
path). Confirm the Prisma schema default for
`TaskRunExecutionSnapshot.isValid` is `true`.
- [ ] **Verify `runStatus: "PENDING"` hardcoding matches the mapping**:
The old code passed `lockedTaskRun.status` ("DEQUEUED") to
`createExecutionSnapshot`, which mapped it to "PENDING" via `run.status
=== "DEQUEUED" ? "PENDING" : run.status`. The new code hardcodes
`"PENDING"` directly. This is correct but brittle if `status` ever
changes from "DEQUEUED" to something else upstream.
- [ ] **Spot-check `completedWaitpoints` connect + order logic**: The
nested create replicates the connect/order logic from
`createExecutionSnapshot` (lines 387-393). Verify the
`snapshot.completedWaitpoints` type provides `id` and `index` fields
compatible with this usage.
- [ ] **Verify `checkpoint` in return value**: The return now uses
`snapshot.checkpoint` (from the *previous* snapshot) instead of reading
the newly-created snapshot's checkpoint relation. Since `checkpointId`
is passed through unchanged, they should be identical — but worth a
sanity check.
**Recommended test plan**: deploy to staging, run the
`sample_pg_activity.py` sampler for a 5-minute window, and verify the
COMMIT count drop on the engine service + proportional `IO:XactSync`
reduction.
### Notes
- This only covers the **dequeue** flow (flow #1 from TRI-8450). The
remaining four flows (retry-immediate, checkpoint, requeue, cancel) are
separate follow-ups.
- The new `enqueueHeartbeatIfNeeded` method is deliberately designed for
reuse by those follow-up PRs.
- CI note: the `priority.test.ts` failure in shard 7 is a flaky ordering
assertion unrelated to this change (it compares `friendlyId` values in
dequeue order). The `audit` check is also pre-existing/unrelated.
Link to Devin session:
https://app.devin.ai/sessions/034fe0e7224f49278a2de260203e1377
Requested by: @ericallam
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: Eric Allam <eallam@icloud.com>
Wraps getEntitlement in platform.v3.server.ts with the existing
platformCache (LRU memory + Redis) under a new `entitlement` namespace.
Eliminates a synchronous billing-service HTTP round trip on every
trigger.
Cache config: 60s fresh / 60s stale SWR. Cache key is the
organization id. Errors are caught inside the loader and return the
existing permissive { hasAccess: true } fallback, which is also
cached to prevent thundering-herd on billing outages.
Trade-off: plan upgrade/downgrade is now visible after up to ~120s
worst-case (60s fresh + 60s stale revalidation). Acceptable since
the existing limits and usage namespaces use 5min/10min, and the
defensive hasAccess: true fallback already exists.
Replace the expensive DISTINCT query for task filter dropdowns with a
dedicated TaskIdentifier registry table backed by Redis. Environments
migrate automatically on their next deploy, with a transparent fallback
to the legacy query for unmigrated environments. Also fixes duplicate
dropdown entries when a task changes trigger source, and adds
active/archived grouping for removed tasks. Moves BackgroundWorkerTask
reads in the trigger hot path to the read replica.
Pool Redis connections for non-blocking ops (ingestData, appendPart,
getLastChunkIndex)
using a shared singleton instead of new Redis() per request. Use
redis.disconnect()
for immediate teardown in streamResponse cleanup. Add 15s inactivity
timeout fallback.
Fix broken request.signal in Remix/Express by wiring Express
res.on('close') to an
AbortController via httpAsyncStorage. All SSE/streaming routes now use
getRequestAbortSignal() which fires reliably on client disconnect,
bypassing the
Node.js undici GC bug (nodejs/node#55428) that severs the signal chain.
Extends the admin worker groups endpoint with a GET loader and more
fields on POST (type, hidden, workloadType, cloudProvider, location,
staticIPs, enableFastPath), and pulls the PAT + admin check that was
inlined or locally duplicated across every admin.api route into a shared
helper in personalAccessToken.server.ts. The generic
authenticateAdminRequest returns a discriminated result;
requireAdminApiRequest is the thin Remix loader/action wrapper that
throws. The neverthrow-style route (platform-notifications.ts) now
composes the generic helper instead of duplicating the check. Verified
locally against GET (listing) and POST (new fields, invalid enum,
minimal backwards-compat).
## Summary
Upgrades all `@remix-run/*` packages in `apps/webapp` from **2.1.0 →
2.17.4** to address security vulnerabilities. Recreation of #2951 on a
fresh checkout of `main`.
**Updated packages (`apps/webapp/package.json`):**
- `@remix-run/express`, `@remix-run/node`, `@remix-run/react`,
`@remix-run/serve`, `@remix-run/server-runtime`: 2.1.0 → 2.17.4
- `@remix-run/router`: ^1.15.3 → ^1.23.2
- `@remix-run/dev`, `@remix-run/eslint-config`, `@remix-run/testing`:
2.1.0 → 2.17.4
**Root `package.json` overrides:**
- `@remix-run/dev@2.17.4>tar-fs`: 2.1.3 → 2.1.4
- `testcontainers@10.28.0>tar-fs`: 3.0.9 → 3.1.1
**Documentation:** Updated Remix version references in `CLAUDE.md`,
`apps/webapp/CLAUDE.md`, and `.cursor/rules/webapp.mdc`.
**Server changes:** Added `.server-changes/upgrade-remix-security.md`
for release tracking per `CONTRIBUTING.md`.
No application code changes — only `package.json` files, documentation,
a server-changes entry, and the regenerated `pnpm-lock.yaml`.
### Updates since last revision
Addressed all 3 Devin Review findings:
1. **Missing `.server-changes/` file** — added
`.server-changes/upgrade-remix-security.md` (commit ce22a0bd4)
2. **Sentry Remix patch (`@sentry/remix@9.46.0`)** — verified the patch
at `patches/@sentry__remix@9.46.0.patch` applies cleanly against 2.17.4.
The patch modifies Sentry's own `RemixInstrumentation` wrapper (removing
`request.clone()` and form data attributes), not Remix internals. The
underlying Remix APIs it hooks into (`callRouteAction`,
`callRouteLoader`) are stable across 2.1→2.17.
3. **`remix-typedjson@0.3.1` compatibility** — peer deps declare
`@remix-run/react: ^1.16.0 || ^2.0`, covering 2.17.4. Confirmed working
at runtime across all 22 tested pages that use it (root.tsx, hooks,
route loaders).
### Verification performed during this session
- **Runtime:** Express+Remix integration, magic link login, client-side
routing, MetaFunction rendering
- **Operational:** hello-world task triggered via API, runs list, run
detail, tasks page
- **Comprehensive UI:** 22 pages, 11 filter types, environment/project
switchers, interactive elements
- **Docker:** Production Dockerfile (`docker/webapp/Dockerfile`) builds
successfully
- **Changelog audit:** All 16 minor versions reviewed — every breaking
change is behind opt-in future flags the webapp doesn't enable
## Review & Testing Checklist for Human
- [ ] **Verify auth flows in staging** — `remix-auth`,
`remix-auth-email-link`, and `remix-auth-github` declare peer deps on
`@remix-run/server-runtime@^1.x`, which is now 2.17.4. Login (magic link
+ OAuth) should be tested in a staging environment since local dev
testing may not exercise all auth code paths.
- [ ] **Verify tar-fs override versions** resolve the targeted security
advisories (2.1.4 and 3.1.1)
- [ ] **Review new transitive dependencies** added by the upgrade:
`turbo-stream@2.4.1`, `undici@6.25.0`, `valibot@1.3.1`, `ws@7.5.10`
Recommended test plan: deploy to staging and exercise core webapp flows
— login (email magic link + GitHub OAuth), dashboard navigation, task
triggering/viewing, and API endpoints — to catch runtime regressions not
covered by local testing.
### Notes
- Peer dependency warnings for `remix-auth-*` packages (expecting
`@remix-run/server-runtime@^1.x`) were present in the original PR #2951
as well and appear to be pre-existing
- The lockfile diff is large (~1200 lines) but mechanical — driven by
the Remix version bump cascading through transitive dependencies
- CI failures (`audit`, `units/internal/1-of-8`) are unrelated: `audit`
is a `claude-code-action` bot permissions issue; the internal test
failure is a ClickHouse testcontainers `Failed to connect to Reaper`
flake
Link to Devin session:
https://app.devin.ai/sessions/d9fa9953b9bf40e5a8d12b8f5ba5b86b
Requested by: @ericallam
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: Eric Allam <eallam@icloud.com>
Adds missing list deployments API page, fixes defaultMachine → machine
in config docs, and clarifies browser CORS usage for wait token
completion with corrected warning placement
The TaskRun.runTags string array already stores tag names, making the
TaskRunTag M2M relation redundant write overhead. Remove createTags
calls, connect: tags, and join table writes from both V1 and V2 trigger
paths. Simplify the add-tags API to just push to runTags directly.
## Summary
12 new features, 59 improvements, 17 bug fixes.
## Highlights
- Add support for setting TTL (time-to-live) defaults at the task level
and globally in trigger.config.ts, with per-trigger overrides still
taking precedence
([#3196](https://github.com/triggerdotdev/trigger.dev/pull/3196))
- Large run outputs can use the new API which allows switching object
storage providers.
([#3275](https://github.com/triggerdotdev/trigger.dev/pull/3275))
## Improvements
- Add platform notifications support to the CLI. The `trigger dev` and
`trigger login` commands now fetch and display platform notifications
(info, warn, error, success) from the server. Includes discovery-based
filtering to conditionally show notifications based on project file
patterns, color markup rendering for styled terminal output, and a
non-blocking display flow with a spinner fallback for slow fetches. Use
`--skip-platform-notifications` flag with `trigger dev` to disable the
notification check.
([#3254](https://github.com/triggerdotdev/trigger.dev/pull/3254))
- Add `get_span_details` MCP tool for inspecting individual spans within
a run trace.
([#3255](https://github.com/triggerdotdev/trigger.dev/pull/3255))
- New `get_span_details` tool returns full span attributes, timing,
events, and AI enrichment (model, tokens, cost, speed)
- Span IDs now shown in `get_run_details` trace output for easy
discovery
- New API endpoint `GET /api/v1/runs/:runId/spans/:spanId`
- New `retrieveSpan()` method on the API client
- `get_query_schema` — discover available TRQL tables and columns
- `query` — execute TRQL queries against your data
- `list_dashboards` — list built-in dashboards and their widgets
- `run_dashboard_query` — execute a single dashboard widget query
- `whoami` — show current profile, user, and API URL
- `list_profiles` — list all configured CLI profiles
- `switch_profile` — switch active profile for the MCP session
- `start_dev_server` — start `trigger dev` in the background and stream
output
- `stop_dev_server` — stop the running dev server
- `dev_server_status` — check dev server status and view recent logs
- `GET /api/v1/query/schema` — query table schema discovery
- `GET /api/v1/query/dashboards` — list built-in dashboards
- `--readonly` flag hides write tools (`deploy`, `trigger_task`,
`cancel_run`) so the AI cannot make changes
- `read:query` JWT scope for query endpoint authorization
- `get_run_details` trace output is now paginated with cursor support
- MCP tool annotations (`readOnlyHint`, `destructiveHint`) for all tools
- `get_query_schema` now requires a table name and returns only one
table's schema (was returning all tables)
- `get_current_worker` no longer inlines payload schemas; use new
`get_task_schema` tool instead
- Query results formatted as text tables instead of JSON (~50% fewer
tokens)
- `cancel_run`, `list_deploys`, `list_preview_branches` formatted as
text instead of raw JSON
- Schema and dashboard API responses cached to avoid redundant fetches
- Adapted the CLI API client to propagate the trigger source via http
headers.
([#3241](https://github.com/triggerdotdev/trigger.dev/pull/3241))
- Propagate run tags to span attributes so they can be extracted
server-side for LLM cost attribution metadata.
([#3213](https://github.com/triggerdotdev/trigger.dev/pull/3213))
- New `get_span_details` tool returns full span attributes, timing,
events, and AI enrichment (model, tokens, cost, speed)
- Span IDs now shown in `get_run_details` trace output for easy
discovery
- New API endpoint `GET /api/v1/runs/:runId/spans/:spanId`
- New `retrieveSpan()` method on the API client
- `get_query_schema` — discover available TRQL tables and columns
- `query` — execute TRQL queries against your data
- `list_dashboards` — list built-in dashboards and their widgets
- `run_dashboard_query` — execute a single dashboard widget query
- `whoami` — show current profile, user, and API URL
- `list_profiles` — list all configured CLI profiles
- `switch_profile` — switch active profile for the MCP session
- `start_dev_server` — start `trigger dev` in the background and stream
output
- `stop_dev_server` — stop the running dev server
- `dev_server_status` — check dev server status and view recent logs
- `GET /api/v1/query/schema` — query table schema discovery
- `GET /api/v1/query/dashboards` — list built-in dashboards
- `--readonly` flag hides write tools (`deploy`, `trigger_task`,
`cancel_run`) so the AI cannot make changes
- `read:query` JWT scope for query endpoint authorization
- `get_run_details` trace output is now paginated with cursor support
- MCP tool annotations (`readOnlyHint`, `destructiveHint`) for all tools
- `get_query_schema` now requires a table name and returns only one
table's schema (was returning all tables)
- `get_current_worker` no longer inlines payload schemas; use new
`get_task_schema` tool instead
- Query results formatted as text tables instead of JSON (~50% fewer
tokens)
- `cancel_run`, `list_deploys`, `list_preview_branches` formatted as
text instead of raw JSON
- Schema and dashboard API responses cached to avoid redundant fetches
- Add optional `hasPrivateLink` field to the dequeue message
organization object for private networking support
([#3264](https://github.com/triggerdotdev/trigger.dev/pull/3264))
- Define and manage AI prompts with `prompts.define()`. Create typesafe
prompt templates with variables, resolve them at runtime, and manage
versions and overrides from the dashboard without redeploying.
([#3244](https://github.com/triggerdotdev/trigger.dev/pull/3244))
## Bug fixes
- Fix dev CLI leaking build directories on rebuild, causing disk space
accumulation. Deprecated workers are now pruned (capped at 2 retained)
when no active runs reference them. The watchdog process also cleans up
`.trigger/tmp/` when the dev CLI is killed ungracefully (e.g. SIGKILL
from pnpm).
([#3224](https://github.com/triggerdotdev/trigger.dev/pull/3224))
- Fix `--load` flag being silently ignored on local/self-hosted builds.
([#3114](https://github.com/triggerdotdev/trigger.dev/pull/3114))
- Fixed `search_docs` tool failing due to renamed upstream Mintlify tool
(`SearchTriggerDev` → `search_trigger_dev`)
- Fixed `list_deploys` failing when deployments have null
`runtime`/`runtimeVersion` fields (#3139)
- Fixed `list_preview_branches` crashing due to incorrect response shape
access
- Fixed `metrics` table column documented as `value` instead of
`metric_value` in query docs
- Fixed dev CLI leaking build directories on rebuild — deprecated
workers now clean up their build dirs when their last run completes
- Fixed `search_docs` tool failing due to renamed upstream Mintlify tool
(`SearchTriggerDev` → `search_trigger_dev`)
- Fixed `list_deploys` failing when deployments have null
`runtime`/`runtimeVersion` fields (#3139)
- Fixed `list_preview_branches` crashing due to incorrect response shape
access
- Fixed `metrics` table column documented as `value` instead of
`metric_value` in query docs
- Fixed dev CLI leaking build directories on rebuild — deprecated
workers now clean up their build dirs when their last run completes
## Server changes
These changes affect the self-hosted Docker image and Trigger.dev Cloud:
- Add admin UI for viewing and editing feature flags (org-level
overrides and global defaults).
([#3291](https://github.com/triggerdotdev/trigger.dev/pull/3291))
- AI prompt management dashboard and enhanced span inspectors.
**Prompt management:**
- Prompts list page with version status, model, override indicators, and
24h usage sparklines
- Prompt detail page with template viewer, variable preview, version
history timeline, and override editor
- Create, edit, and remove overrides to change prompt content or model
without redeploying
- Promote any code-deployed version to current
- Generations tab with infinite scroll, live polling, and inline span
inspector
- Per-prompt metrics: total generations, avg tokens, avg cost, latency,
with version-level breakdowns
**AI span inspectors:**
- Custom inspectors for `ai.generateText`, `ai.streamText`,
`ai.generateObject`, `ai.streamObject` parent spans
- `ai.toolCall` inspector showing tool name, call ID, and input
arguments
- `ai.embed` inspector showing model, provider, and input text
- Prompt tab on AI spans linking to prompt version with template and
input variables
- Compact timestamp and duration header on all AI span inspectors
**AI metrics dashboard:**
- Operations, Providers, and Prompts filters on the AI Metrics dashboard
- Cost by prompt widget
- "AI" section in the sidebar with Prompts and AI Metrics links
**Other improvements:**
- Resizable panel sizes now persist across page refreshes
- Fixed `<div>` inside `<p>` DOM nesting warnings in span titles and
chat messages
([#3244](https://github.com/triggerdotdev/trigger.dev/pull/3244))
- Add allowRollbacks query param to the promote deployment API to enable
version downgrades
([#3214](https://github.com/triggerdotdev/trigger.dev/pull/3214))
- Pre-warm compute templates on deploy for orgs with compute access.
Required for projects using a compute region, background-only for
others.
([#3114](https://github.com/triggerdotdev/trigger.dev/pull/3114))
- Add automatic LLM cost calculation for spans with GenAI semantic
conventions. When a span arrives with `gen_ai.response.model` and token
usage data, costs are calculated from an in-memory pricing registry
backed by Postgres and dual-written to both span attributes
(`trigger.llm.*`) and a new `llm_metrics_v1` ClickHouse table that
captures usage, cost, performance (TTFC, tokens/sec), and behavioral
(finish reason, operation type) metrics.
([#3213](https://github.com/triggerdotdev/trigger.dev/pull/3213))
- Add API endpoint `GET /api/v1/runs/:runId/spans/:spanId` that returns
detailed span information including properties, events, AI enrichment
(model, tokens, cost), and triggered child runs.
([#3255](https://github.com/triggerdotdev/trigger.dev/pull/3255))
- Multi-provider object storage with protocol-based routing for
zero-downtime migration
([#3275](https://github.com/triggerdotdev/trigger.dev/pull/3275))
- Add IAM role-based auth support for object stores (no access keys
required).
([#3275](https://github.com/triggerdotdev/trigger.dev/pull/3275))
- Add platform notifications to inform users about new features,
changelogs, and platform events directly in the dashboard.
([#3254](https://github.com/triggerdotdev/trigger.dev/pull/3254))
- Add private networking support via AWS PrivateLink. Includes
BillingClient methods for managing private connections, org settings UI
pages for connection management, and supervisor changes to apply
`privatelink` pod labels for CiliumNetworkPolicy matching.
([#3264](https://github.com/triggerdotdev/trigger.dev/pull/3264))
- Reduce run start latency by skipping the intermediate queue when
concurrency is available. This optimization is rolled out per-region and
enabled automatically for development environments.
([#3299](https://github.com/triggerdotdev/trigger.dev/pull/3299))
- Extended the search filter on the environment variables page to match
on environment type (production, staging, development, preview) and
branch name, not just variable name and value.
([#3302](https://github.com/triggerdotdev/trigger.dev/pull/3302))
- Set `application_name` on Prisma connections from SERVICE_NAME so DB
load can be attributed by service
([#3348](https://github.com/triggerdotdev/trigger.dev/pull/3348))
- Fix transient R2/object store upload failures during batchTrigger()
item streaming.
- Added p-retry (3 attempts, 500ms–2s exponential backoff) around
`uploadPacketToObjectStore` in `BatchPayloadProcessor.process()` so
transient network errors self-heal server-side rather than aborting the
entire batch stream.
- Removed `x-should-retry: false` from the 500 response on the batch
items route so the SDK's existing 5xx retry path can recover if
server-side retries are exhausted. Item deduplication by index makes
full-stream retries safe.
([#3331](https://github.com/triggerdotdev/trigger.dev/pull/3331))
- Concurrency-keyed queues now use a single master queue entry per base
queue instead of one entry per key. Prevents high-CK-count tenants from
consuming the entire parentQueueLimit window and starving other tenants
on the same shard.
([#3219](https://github.com/triggerdotdev/trigger.dev/pull/3219))
- Reduce lock contention when processing large `batchTriggerAndWait`
batches. Previously, each batch item acquired a Redis lock on the parent
run to insert a `TaskRunWaitpoint` row, causing
`LockAcquisitionTimeoutError` with high concurrency (880 errors/24h in
prod). Since `blockRunWithCreatedBatch` already transitions the parent
to `EXECUTING_WITH_WAITPOINTS` before items are processed, the per-item
lock is unnecessary. The new `blockRunWithWaitpointLockless` method
performs only the idempotent CTE insert without acquiring the lock.
([#3232](https://github.com/triggerdotdev/trigger.dev/pull/3232))
- Strip `secure` query parameter from QUERY_CLICKHOUSE_URL before
passing to ClickHouse client. This was already done for the main and
logs ClickHouse clients but was missing for the query client, causing a
startup crash with `Error: Unknown URL parameters: secure`.
([#3204](https://github.com/triggerdotdev/trigger.dev/pull/3204))
- Fix `OrganizationsPresenter.#getEnvironment` matching the wrong
development environment on teams with multiple members. All dev
environments share the slug `"dev"`, so the previous `find` by slug
alone could return another member's environment. Now filters DEVELOPMENT
environments by `orgMember.userId` to ensure the logged-in user's dev
environment is selected.
([#3273](https://github.com/triggerdotdev/trigger.dev/pull/3273))
<details>
<summary>Raw changeset output</summary>
# Releases
## @trigger.dev/build@4.4.4
### Patch Changes
- Updated dependencies:
- `@trigger.dev/core@4.4.4`
## trigger.dev@4.4.4
### Patch Changes
- Add platform notifications support to the CLI. The `trigger dev` and
`trigger login` commands now fetch and display platform notifications
(info, warn, error, success) from the server. Includes discovery-based
filtering to conditionally show notifications based on project file
patterns, color markup rendering for styled terminal output, and a
non-blocking display flow with a spinner fallback for slow fetches. Use
`--skip-platform-notifications` flag with `trigger dev` to disable the
notification check.
([#3254](https://github.com/triggerdotdev/trigger.dev/pull/3254))
- Fix dev CLI leaking build directories on rebuild, causing disk space
accumulation. Deprecated workers are now pruned (capped at 2 retained)
when no active runs reference them. The watchdog process also cleans up
`.trigger/tmp/` when the dev CLI is killed ungracefully (e.g. SIGKILL
from pnpm).
([#3224](https://github.com/triggerdotdev/trigger.dev/pull/3224))
- Fix `--load` flag being silently ignored on local/self-hosted builds.
([#3114](https://github.com/triggerdotdev/trigger.dev/pull/3114))
- Add `get_span_details` MCP tool for inspecting individual spans within
a run trace.
([#3255](https://github.com/triggerdotdev/trigger.dev/pull/3255))
- New `get_span_details` tool returns full span attributes, timing,
events, and AI enrichment (model, tokens, cost, speed)
- Span IDs now shown in `get_run_details` trace output for easy
discovery
- New API endpoint `GET /api/v1/runs/:runId/spans/:spanId`
- New `retrieveSpan()` method on the API client
- MCP server improvements: new tools, bug fixes, and new flags.
([#3224](https://github.com/triggerdotdev/trigger.dev/pull/3224))
**New tools:**
- `get_query_schema` — discover available TRQL tables and columns
- `query` — execute TRQL queries against your data
- `list_dashboards` — list built-in dashboards and their widgets
- `run_dashboard_query` — execute a single dashboard widget query
- `whoami` — show current profile, user, and API URL
- `list_profiles` — list all configured CLI profiles
- `switch_profile` — switch active profile for the MCP session
- `start_dev_server` — start `trigger dev` in the background and stream
output
- `stop_dev_server` — stop the running dev server
- `dev_server_status` — check dev server status and view recent logs
**New API endpoints:**
- `GET /api/v1/query/schema` — query table schema discovery
- `GET /api/v1/query/dashboards` — list built-in dashboards
**New features:**
- `--readonly` flag hides write tools (`deploy`, `trigger_task`,
`cancel_run`) so the AI cannot make changes
- `read:query` JWT scope for query endpoint authorization
- `get_run_details` trace output is now paginated with cursor support
- MCP tool annotations (`readOnlyHint`, `destructiveHint`) for all tools
**Bug fixes:**
- Fixed `search_docs` tool failing due to renamed upstream Mintlify tool
(`SearchTriggerDev` → `search_trigger_dev`)
- Fixed `list_deploys` failing when deployments have null
`runtime`/`runtimeVersion` fields (#3139)
- Fixed `list_preview_branches` crashing due to incorrect response shape
access
- Fixed `metrics` table column documented as `value` instead of
`metric_value` in query docs
- Fixed dev CLI leaking build directories on rebuild — deprecated
workers now clean up their build dirs when their last run completes
**Context optimizations:**
- `get_query_schema` now requires a table name and returns only one
table's schema (was returning all tables)
- `get_current_worker` no longer inlines payload schemas; use new
`get_task_schema` tool instead
- Query results formatted as text tables instead of JSON (~50% fewer
tokens)
- `cancel_run`, `list_deploys`, `list_preview_branches` formatted as
text instead of raw JSON
- Schema and dashboard API responses cached to avoid redundant fetches
- Add support for setting TTL (time-to-live) defaults at the task level
and globally in trigger.config.ts, with per-trigger overrides still
taking precedence
([#3196](https://github.com/triggerdotdev/trigger.dev/pull/3196))
- Adapted the CLI API client to propagate the trigger source via http
headers.
([#3241](https://github.com/triggerdotdev/trigger.dev/pull/3241))
- Updated dependencies:
- `@trigger.dev/core@4.4.4`
- `@trigger.dev/build@4.4.4`
- `@trigger.dev/schema-to-json@4.4.4`
## @trigger.dev/core@4.4.4
### Patch Changes
- Fix `list_deploys` MCP tool failing when deployments have null
`runtime` or `runtimeVersion` fields.
([#3224](https://github.com/triggerdotdev/trigger.dev/pull/3224))
- Propagate run tags to span attributes so they can be extracted
server-side for LLM cost attribution metadata.
([#3213](https://github.com/triggerdotdev/trigger.dev/pull/3213))
- Add `get_span_details` MCP tool for inspecting individual spans within
a run trace.
([#3255](https://github.com/triggerdotdev/trigger.dev/pull/3255))
- New `get_span_details` tool returns full span attributes, timing,
events, and AI enrichment (model, tokens, cost, speed)
- Span IDs now shown in `get_run_details` trace output for easy
discovery
- New API endpoint `GET /api/v1/runs/:runId/spans/:spanId`
- New `retrieveSpan()` method on the API client
- MCP server improvements: new tools, bug fixes, and new flags.
([#3224](https://github.com/triggerdotdev/trigger.dev/pull/3224))
**New tools:**
- `get_query_schema` — discover available TRQL tables and columns
- `query` — execute TRQL queries against your data
- `list_dashboards` — list built-in dashboards and their widgets
- `run_dashboard_query` — execute a single dashboard widget query
- `whoami` — show current profile, user, and API URL
- `list_profiles` — list all configured CLI profiles
- `switch_profile` — switch active profile for the MCP session
- `start_dev_server` — start `trigger dev` in the background and stream
output
- `stop_dev_server` — stop the running dev server
- `dev_server_status` — check dev server status and view recent logs
**New API endpoints:**
- `GET /api/v1/query/schema` — query table schema discovery
- `GET /api/v1/query/dashboards` — list built-in dashboards
**New features:**
- `--readonly` flag hides write tools (`deploy`, `trigger_task`,
`cancel_run`) so the AI cannot make changes
- `read:query` JWT scope for query endpoint authorization
- `get_run_details` trace output is now paginated with cursor support
- MCP tool annotations (`readOnlyHint`, `destructiveHint`) for all tools
**Bug fixes:**
- Fixed `search_docs` tool failing due to renamed upstream Mintlify tool
(`SearchTriggerDev` → `search_trigger_dev`)
- Fixed `list_deploys` failing when deployments have null
`runtime`/`runtimeVersion` fields (#3139)
- Fixed `list_preview_branches` crashing due to incorrect response shape
access
- Fixed `metrics` table column documented as `value` instead of
`metric_value` in query docs
- Fixed dev CLI leaking build directories on rebuild — deprecated
workers now clean up their build dirs when their last run completes
**Context optimizations:**
- `get_query_schema` now requires a table name and returns only one
table's schema (was returning all tables)
- `get_current_worker` no longer inlines payload schemas; use new
`get_task_schema` tool instead
- Query results formatted as text tables instead of JSON (~50% fewer
tokens)
- `cancel_run`, `list_deploys`, `list_preview_branches` formatted as
text instead of raw JSON
- Schema and dashboard API responses cached to avoid redundant fetches
- Large run outputs can use the new API which allows switching object
storage providers.
([#3275](https://github.com/triggerdotdev/trigger.dev/pull/3275))
- Add optional `hasPrivateLink` field to the dequeue message
organization object for private networking support
([#3264](https://github.com/triggerdotdev/trigger.dev/pull/3264))
- Add support for setting TTL (time-to-live) defaults at the task level
and globally in trigger.config.ts, with per-trigger overrides still
taking precedence
([#3196](https://github.com/triggerdotdev/trigger.dev/pull/3196))
- Adapted the CLI API client to propagate the trigger source via http
headers.
([#3241](https://github.com/triggerdotdev/trigger.dev/pull/3241))
## @trigger.dev/python@4.4.4
### Patch Changes
- Updated dependencies:
- `@trigger.dev/sdk@4.4.4`
- `@trigger.dev/core@4.4.4`
- `@trigger.dev/build@4.4.4`
## @trigger.dev/react-hooks@4.4.4
### Patch Changes
- Updated dependencies:
- `@trigger.dev/core@4.4.4`
## @trigger.dev/redis-worker@4.4.4
### Patch Changes
- Adapted the CLI API client to propagate the trigger source via http
headers.
([#3241](https://github.com/triggerdotdev/trigger.dev/pull/3241))
- Updated dependencies:
- `@trigger.dev/core@4.4.4`
## @trigger.dev/rsc@4.4.4
### Patch Changes
- Updated dependencies:
- `@trigger.dev/core@4.4.4`
## @trigger.dev/schema-to-json@4.4.4
### Patch Changes
- Updated dependencies:
- `@trigger.dev/core@4.4.4`
## @trigger.dev/sdk@4.4.4
### Patch Changes
- Define and manage AI prompts with `prompts.define()`. Create typesafe
prompt templates with variables, resolve them at runtime, and manage
versions and overrides from the dashboard without redeploying.
([#3244](https://github.com/triggerdotdev/trigger.dev/pull/3244))
- Add support for setting TTL (time-to-live) defaults at the task level
and globally in trigger.config.ts, with per-trigger overrides still
taking precedence
([#3196](https://github.com/triggerdotdev/trigger.dev/pull/3196))
- Adapted the CLI API client to propagate the trigger source via http
headers.
([#3241](https://github.com/triggerdotdev/trigger.dev/pull/3241))
- Updated dependencies:
- `@trigger.dev/core@4.4.4`
</details>
---------
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Adds region-level gating so MICROVM regions are only visible and usable
by orgs with the `hasComputeAccess` feature flag. Admins and explicit
allowlist behavior unchanged.
- New shared helper (`regionAccess.server.ts`) with
`resolveComputeAccess`, `defaultVisibilityFilter`, and
`isComputeRegionAccessible`
- `RegionsPresenter` filters out MICROVM regions for non-compute orgs
- `SetDefaultRegionService` blocks setting a MICROVM region as default
without compute access
- `WorkerGroupService` blocks triggering runs in MICROVM regions without
compute access
- `computeTemplateCreation` refactored to use shared
`resolveComputeAccess`
- Updated snapshot callback schema
Sets `application_name` on the Prisma writer and replica connection
strings using the existing `SERVICE_NAME` env var, so DB load can be
attributed by service.
A single "fetch failed" from the object store was aborting the entire
batch stream with no retry. Added p-retry (3 attempts, 500ms-2s backoff)
around ploadPacketToObjectStore so transient network errors self-heal
server-side instead of propagating to the SDK.
### Text wrapping fix
- Fixes message text not wrapping on the run inspector if there were no
spaces in the text
- Fixes inspector title truncation
- Adds a copy text button for the Message property
<img width="468" height="740" alt="CleanShot 2026-04-04 at 10 19 02@2x"
src="https://github.com/user-attachments/assets/71e42bf3-d103-44a2-b3b4-937c0b60a4bc"
/>
This is a small improvement mainly with the UI Skills file:
- Animate open and close the Resizable panels
- Uses the built in animation hooks from react-window-splitter
- Includes a global variable for the animation easing and timing for
consistency
https://github.com/user-attachments/assets/50ed0019-ed12-4e08-b95c-7c6d1fe5bac0
## Summary
- Drop all 8 foreign key constraints on TaskRun. The run listing path is
now fully ClickHouse-backed so we no longer need Postgres to enforce
referential integrity on this table. The FK constraints add write
overhead on every insert/update with no remaining benefit. Prisma
queries are unaffected.
- Remove PostgresRunsRepository and its associated feature flag
(runsListRepository), which was the last remaining code path querying
TaskRun directly for list/count operations.
- Drop three indexes that were only useful for the Postgres run list
path and have no remaining query consumers:
- TaskRun_runtimeEnvironmentId_id_idx — was the cursor pagination index
for PostgresRunsRepository; superseded by the (runtimeEnvironmentId,
createdAt DESC) composite index
- TaskRun_scheduleId_idx — redundant with the (scheduleId, createdAt
DESC) composite index; no direct Postgres queries filter by scheduleId
alone
- TaskRun_rootTaskRunId_idx — no queries filter TaskRun by rootTaskRunId
as a WHERE clause anywhere in the codebase
All index drops use CONCURRENTLY IF EXISTS to avoid table locks in
production.
## Test plan
- pnpm run db:migrate:deploy applies all migrations cleanly
- pnpm run typecheck --filter webapp passes
- Run list pages load correctly in the dashboard (ClickHouse path)
- Scheduled task runs still trigger and appear correctly
This allows seamless migration to different object storage.
Existing runs that have offloaded payloads/outputs will continue to use
the default object store (configured using `OBJECT_STORE_*` env vars).
You can add additional stores by setting new env vars:
- `OBJECT_STORE_DEFAULT_PROTOCOL` this determines where new run large
payloads will get stored.
- If you set that you need to set new env vars for that protocol.
Example:
```
OBJECT_STORE_DEFAULT_PROTOCOL=“s3"
OBJECT_STORE_S3_BASE_URL=https://s3.us-east-1.amazonaws.com
OBJECT_STORE_S3_ACCESS_KEY_ID=<val>
OBJECT_STORE_S3_SECRET_ACCESS_KEY=<val>
OBJECT_STORE_S3_REGION=us-east-1
OBJECT_STORE_S3_SERVICE=s3
```
---------
Co-authored-by: nicktrn <55853254+nicktrn@users.noreply.github.com>
Adds a migration reference for users moving from n8n to Trigger.dev.
Includes a concept map, four common patterns covering the
migration-specific gaps, and a full customer onboarding example. The
onboarding workflow highlights the 3-day wait pattern, an area where
n8n's execution model has known reliability issues at production scale
that Trigger.dev handles natively
- Added versions filtering on the Errors list and page
- Added errors stacked bars to the graph on the individual error page
---------
Co-authored-by: James Ritchie <james@trigger.dev>
The @internal/compute package had its main/types pointing to
./src/index.ts with no build step. This works in dev (tsc resolves .ts
at compile time) but fails at runtime in Docker because Node.js can't
load .ts files directly.
Added tsconfig.build.json and build/clean/dev scripts matching the
pattern used by schedule-engine and other internal packages. Exports now
point to dist/.
Temporary workaround that enables filtering by environment in the
envvars page, without changing any UI.
---------
Co-authored-by: Claude <noreply@anthropic.com>
## Summary
Currently, every triggered run follows a two-step path through Redis:
1. **Enqueue** — A Lua script atomically adds the message to a queue
sorted set (ordered by priority-adjusted timestamp)
2. **Dequeue** — A debounced `processQueueForWorkerQueue` job fires
~500ms later, checks concurrency limits, removes the message from the
sorted set, and pushes it to a worker queue (Redis list) where workers
pick it up via `BLPOP`
This means every run pays at least ~500ms of latency between being
triggered and being available for a worker to execute, even when the
queue is empty and concurrency is wide open.
### What changed
The enqueue Lua scripts now atomically decide whether to **skip the
queue sorted set entirely** and push directly to the worker queue. This
happens inside the same Lua script that handles normal enqueue, so the
decision is atomic with respect to concurrency bookkeeping.
A run takes the **fast path** when all of these are true:
- **Fast path is enabled** for this worker queue (gated per
`WorkerInstanceGroup`)
- **No available messages** in the queue (`ZRANGEBYSCORE` finds nothing
with score ≤ now) — this respects priority ordering and allows fast path
even when the queue has future-scored messages (e.g. nacked retries with
delay)
- **Environment concurrency** has capacity
- **Queue concurrency** has capacity (including per-concurrency-key
limits for CK queues)
When the fast path is taken:
- The message is stored and pushed directly to the worker queue
(`RPUSH`)
- Concurrency slots are claimed (`SADD` to the same sets used by the
normal dequeue path)
- The `processQueueForWorkerQueue` job is **not scheduled** (no work to
do)
- TTL sorted set is skipped (the `expireRun` worker job handles TTL
independently)
When any condition fails, the existing slow path runs unchanged.
### Rollout gating
- **Development environments**: Fast path is always enabled
- **Production environments**: Gated by a new `enableFastPath` boolean
on `WorkerInstanceGroup` (defaults to `false`), allowing
region-by-region rollout
### Rolling deploy safety
Each process registers its own Lua scripts via `defineCommand`
(identified by SHA hash). Old and new processes never share scripts. The
Redis data structures are fully compatible in both directions — ack,
nack, and release operations work identically regardless of which path a
message took.
## Test plan
- [x] Fast path taken when queue is empty and concurrency available
- [x] Slow path when `enableFastPath` is false
- [x] Slow path when queue has available messages (respects priority
ordering)
- [x] Fast path when queue only has future-scored messages
- [x] Slow path when env concurrency is full
- [x] Fast-path message can be acknowledged correctly
- [x] Fast-path message can be nacked and re-enqueued to the queue
sorted set
- [x] Run all existing run-queue tests (ack, nack, CK, concurrency
sweeper, dequeue) to verify no regressions
- [x] Typecheck passes for run-engine and webapp
Add TTL (time-to-live) defaults at task-level and config-level, with
precedence: per-trigger > task > config > dev default (10m).
Docs PR: #3200 (merge after packages are released)
- Rebuild llm_pricing_tiers and llm_prices in syncLlmCatalog for
source=default
- Add vitest config, sync regression tests, and pin vitest 3.1.4
- Update pnpm-lock.yaml for the new devDependency
Adds a dialog to the admin orgs page for viewing and editing per-org
feature flag overrides. Flags are introspected from the catalog so the
UI stays in sync with available flags automatically. Also adds a new tab
for global flags.
Refactors featureFlags.server.ts to split catalog definition (shared)
from server-only runtime (flags(), makeSetMultipleFlags). The shared
module exports flag metadata and validation so both the UI and API
routes can use it without pulling in server dependencies.
Adds support for taint tolerations for scheduled runs. Useful for
selectively tolerating taints on dedicated node pools.
The new `KUBERNETES_SCHEDULED_RUN_TOLERATIONS` env variable accepts a
comma-separated list in the format key=value:effect (or key:effect for
the Exists operator).
Drive-by: renames all `KUBERNETES_SCHEDULE_*` affinity env vars to
KUBERNETES_SCHEDULED_RUN_* for clarity — this feature isn't used in
production yet or published in a tagged image; the name change is fine.
For human reviewer:
- Check if Redis connection + code makes sense
- Check CLI methods (it's on a hotpath)
- Check DB Migrations and new tables
## ✅ Checklist
- [x] I have followed every step in the [contributing
guide](https://github.com/triggerdotdev/trigger.dev/blob/main/CONTRIBUTING.md)
- [x] The PR title follows the convention.
- [x] I ran and tested the code works
---
## Testing
Spawning new CLI / Dashboard notifications, check MVP, check if failures
not produce any problems with CLI/Dashboard
---
## Changelog
Added notifications mechanism for Dashboard and CLI
---
## Screenshots
💯
Adds three new top-level columns to the ClickHouse task_runs_v2 table
primarily for analytics:
- `trigger_source` / `root_trigger_source` - extracted from the existing
TaskRun.annotations JSON during WAL
replication
- `is_warm_start` - new nullable boolean on TaskRun in Postgres, set in
the existing taskRun.update() at attempt
start (no additional write). null until the first attempt starts.
Run region is already available via the existing `worker_queue` column
in ClickHouse.
Scheduled runs create predictable hourly spikes that compete with
on-demand runs for node capacity. Runs triggered "on-demand" via the
SDK, API, or dashboard, are more sensitive to cold start latency since
users are typically
waiting on the result. When a burst of scheduled runs lands at the top
of the hour, it can saturate the shared pool resources causing
contention, affecting cold starts across the board.
The idea in this change is to absorb these periodic spikes in a
dedicated pool without affecting the cold starts of on-demand runs.
Scheduled runs are inherently less sensitive to cold starts.
### Changes in this PR
Follows up on run annotations (#3241), which made trigger origin
available on every run in the tree. This PR exposes
annotations at dequeue time to the supervisor. This enables scheduling
decisions based on trigger source.
The affinities are soft preferences at schedule time, so runs fall back
gracefully if the target pool is out out of capacity.
Queue limit ServiceValidationErrors were being logged at error level.
These are
expected validation rejections, not bugs.
- Add logLevel property to ServiceValidationError (webapp + run-engine)
- Set logLevel: warn on all queue limit throws
- Schedule engine: detect queue limit failures and log as warn
- Redis-worker: respect logLevel on thrown errors
Adds an `annotations` JSONB column to task runs that captures where and
how each run was triggered.
This enables filtering and analyzing trigger origins without querying up
the run tree. Also enables making scheduling decisions based on the
trigger source, e.g., use separate affinities for scheduled runs.
Each run records:
- **triggerSource**: who initiated it (sdk, api, dashboard, cli, mcp,
schedule)
- **triggerAction**: what kind of action (trigger, replay, test)
- **rootTriggerSource**: the trigger source of the root ancestor,
propagated through the entire run
tree
- **rootScheduleId**: schedule id, in case the run tree was triggered
from a schedule
Currently the main motivation for annotations it to determine whether a
run is part of a schedule-originated tree without traversing ancestors.
### A couple of design considerations
- **Decoupled source from method**: triggerSource and triggerAction are
separate fields to avoid
combinatorial explosion (every new source × every new action)
- **Server-side first**: all annotation values are primarily determined
on the server, only a minor SDK change needed
- **Forward-compatible**: annotation fields use
`z.enum([...]).or(anyString)` so new values can be
added without breaking validation; we currently don't need an explicit
version field for annotations.
Note: `metadata` would have been a more fitting name for the db column,
as it is consistent with other tables where we store this type of
information. It is already in use to store user metadata though, so we
go with `annotations` instead.
- Full prompt management UI: list, detail, override, and version
management for AI prompts defined with `prompts.define()`
- Rich AI span inspectors for all AI SDK operations with token usage,
messages, and prompt context
- Real-time generation tracking with live polling and filtering
## Prompt management
Define prompts in your code with `prompts.define()`, then manage
versions and overrides from the dashboard without redeploying:
```typescript
import { task, prompts } from "@trigger.dev/sdk";
import { generateText } from "ai";
import { openai } from "@ai-sdk/openai";
import { z } from "zod";
const supportPrompt = prompts.define({
id: "customer-support",
model: "gpt-4o",
variables: z.object({
customerName: z.string(),
plan: z.string(),
issue: z.string(),
}),
content: `You are a support agent for Acme SaaS.
Customer: {{customerName}} ({{plan}} plan)
Issue: {{issue}}
Respond with empathy and precision.`,
});
export const supportTask = task({
id: "handle-support",
run: async (payload) => {
const resolved = await supportPrompt.resolve({
customerName: payload.name,
plan: payload.plan,
issue: payload.issue,
});
const result = await generateText({
model: openai(resolved.model ?? "gpt-4o"),
system: resolved.text,
prompt: payload.issue,
...resolved.toAISDKTelemetry(),
});
return { response: result.text };
},
});
```
The prompts list page shows each prompt with its current version, model,
override status, and a usage sparkline over the last 24 hours.
From the prompt detail page you can:
- **Create overrides** to change the prompt template or model without
redeploying. Overrides take priority over the deployed version when
`prompt.resolve()` is called.
- **Promote** any code-deployed version to be the current version
- **Browse generations** across all versions with infinite scroll and
live polling for new results
- **Filter** by version, model, operation type, and provider
- **View metrics** (total generations, avg tokens, avg cost, latency)
broken down by version
## AI span inspectors
Every AI SDK operation now gets a custom inspector in the run trace
view:
- **`ai.generateText` / `ai.streamText`** — Shows model, token usage,
cost, the full message thread (system prompt, user message, assistant
response), and linked prompt details
- **`ai.generateObject` / `ai.streamObject`** — Same as above plus the
JSON schema and structured output
- **`ai.toolCall`** — Shows tool name, call ID, and input arguments
- **`ai.embed`** — Shows model and the text being embedded
For generation spans linked to a prompt, a "Prompt" tab shows the prompt
metadata, the input variables passed to `resolve()`, and the template
content from the prompt version.
All AI span inspectors include a compact timestamp and duration header.
## Other improvements
- Resizable panel sizes now persist across page refreshes (patched
`@window-splitter/state` to fix snapshot restoration)
- Run page panels also persist their sizes
- Fixed `<div>` inside `<p>` DOM nesting warnings in span titles and
chat messages
- Added Operations and Providers filters to the AI metrics dashboard
## Screenshots
<img width="3680" height="2392" alt="CleanShot 2026-03-21 at 10 14
17@2x"
src="https://github.com/user-attachments/assets/f3e59989-a2fa-4990-a9d0-3cacda431868"
/>
<img width="3680" height="2392" alt="CleanShot 2026-03-21 at 10 15
37@2x"
src="https://github.com/user-attachments/assets/2f2d02df-2d2b-44fb-ac6f-9153f6a6c387"
/>
<img width="3680" height="2392" alt="CleanShot 2026-03-21 at 10 15
54@2x"
src="https://github.com/user-attachments/assets/baa161e0-ef91-4fa4-a55f-986b71cccdf0"
/>
- Automatic LLM cost enrichment for AI SDK spans (streamText,
generateText, generateObject) or any other spans that use semantic
gen_ai attributes with support for 145+ models
- New AI span inspector sidebar showing model, tokens, cost, messages,
tool calls, and response text
- LLM metrics dual-write to ClickHouse `llm_metrics_v1` table for
analytics
- LLM metrics built-in dashboard (unlinked at the moment)
- Provider cost fallback — uses gateway/OpenRouter reported costs from
`providerMetadata` when registry pricing is unavailable
- Prefix-stripping for gateway/OpenRouter model names (e.g.
`mistral/mistral-large-3` matches `mistral-large-3` pricing)
- Admin dashboard for managing LLM model pricing (list, create, edit,
delete, search, test pattern matching)
- Missing models detection page — queries ClickHouse for unpriced models
with sample spans and Claude Code-ready prompts for adding pricing
- AI span seed script (`pnpm run db:seed:ai-spans`) with 51 spans across
12 provider systems for local dev testing
- UI fixes: `completionTokens`/`promptTokens` aliases,
`ai.response.object` display for generateObject, cache read/write token
breakdown
## Screenshots:
<img width="1030" height="104" alt="CleanShot 2026-03-17 at 16 48 54@2x"
src="https://github.com/user-attachments/assets/bc8fccda-e48b-4d0c-bfb1-e620064e5979"
/>
<img width="1094" height="1512" alt="CleanShot 2026-03-17 at 16 49
23@2x"
src="https://github.com/user-attachments/assets/c2424569-d07e-4d67-a436-e8250043a1ee"
/>
<img width="1074" height="1412" alt="CleanShot 2026-03-17 at 16 49
18@2x"
src="https://github.com/user-attachments/assets/22342ac4-4769-45d1-a328-a24fb9a82a50"
/>
<img width="1012" height="2292" alt="CleanShot 2026-03-17 at 16 39
01@2x"
src="https://github.com/user-attachments/assets/59e327d1-6652-4293-8be0-bb8326e5fbc5"
/>
<img width="3680" height="2392" alt="CleanShot 2026-03-15 at 08 29
38@2x"
src="https://github.com/user-attachments/assets/1f77beb8-de67-495b-b890-bcdb8d7f1fe8"
/>
---------
Co-authored-by: James Ritchie <james@trigger.dev>
When processing batchTriggerAndWait items, each batch item was acquiring
a
Redis lock on the parent run to insert a TaskRunWaitpoint row. With high
concurrency (processingConcurrency=50), this caused
LockAcquisitionTimeoutError
(880 errors/24h in prod), orphaned runs, and stuck parent runs.
Since blockRunWithCreatedBatch already transitions the parent to
EXECUTING_WITH_WAITPOINTS before items are processed, the per-item lock
is
unnecessary. The new blockRunWithWaitpointLockless method performs only
the
idempotent CTE insert and timeout scheduling without acquiring the lock.
## Summary
Major expansion of the MCP server (14 → 25 tools), context efficiency
optimizations, new API endpoints, and a fix for the dev CLI leaking
build directories on disk.
### New MCP tools
- **Query & analytics**: `get_query_schema`, `query`, `list_dashboards`,
`run_dashboard_query` — query your data using TRQL directly from AI
assistants
- **Profile management**: `whoami`, `list_profiles`, `switch_profile` —
see and switch CLI profiles per-project (persisted to
`.trigger/mcp.json`)
- **Dev server control**: `start_dev_server`, `stop_dev_server`,
`dev_server_status` — start/stop `trigger dev` and stream build output
- **Task introspection**: `get_task_schema` — get payload schema for a
specific task (split out from `get_current_worker` to reduce context)
### New API endpoints
- `GET /api/v1/query/schema` — discover TRQL tables and columns
(server-driven, multi-table)
- `GET /api/v1/query/dashboards` — list built-in dashboard widgets and
their queries
### New features
- **`--readonly` flag** — hides write tools (`deploy`, `trigger_task`,
`cancel_run`) so agents can't make changes
- **`read:query` JWT scope** — new authorization scope for query
endpoints, with per-table granularity (`read:query:runs`,
`read:query:llm_metrics`, etc.)
- **Paginated trace output** — `get_run_details` now paginates trace
events via cursor, caching the full trace in a temp file so subsequent
pages don't re-fetch
- **MCP tool annotations** — all tools now have
`readOnlyHint`/`destructiveHint` annotations for clients that support
them
- **Project-scoped profile persistence** — `switch_profile` saves to
`.trigger/mcp.json` (gitignored), automatically loaded on next MCP
server start
### Context optimizations
- `get_query_schema` requires a table name — returns one table's schema
instead of all tables (60-80% fewer tokens)
- `get_current_worker` no longer inlines payload schemas — use
`get_task_schema` for specific tasks
- Query results formatted as text tables instead of JSON (~50% fewer
tokens for flat data)
- `cancel_run`, `list_deploys`, `list_preview_branches` formatted as
text instead of raw `JSON.stringify()`
- Schema and dashboard API responses cached (1hr and 5min respectively)
### Bug fixes
- Fixed `search_docs` failing due to renamed upstream Mintlify tool
(`SearchTriggerDev` → `search_trigger_dev`)
- Fixed `list_deploys` failing when deployments have null
`runtime`/`runtimeVersion` fields (fixes#3139)
- Fixed `list_preview_branches` crashing due to incorrect response shape
access
- Fixed `metrics` table column documented as `value` instead of
`metric_value` in query docs
- Fixed `/api/v1/query` not accepting JWT auth (added `allowJWT: true`)
### Dev CLI build directory fix
The dev CLI was leaking `build-*` directories in `.trigger/tmp/` on
every rebuild, accumulating hundreds of MB over time (842MB observed).
Three layers of protection added:
1. **During session**: deprecated workers are pruned (capped at 2
retained) when no active runs reference them, preventing unbounded
accumulation
2. **On SIGKILL/crash**: the watchdog process now cleans up
`.trigger/tmp/` when it detects the parent CLI was killed
3. **On next startup**: existing `clearTmpDirs()` wipes any remaining
orphans
## Test plan
- [ ] `pnpm run mcp:smoke` — 17 automated smoke tests for all read-only
MCP tools
- [ ] `pnpm run mcp:test list` — verify 25 tools registered (21 in
`--readonly` mode)
- [ ] `pnpm run mcp:test --readonly list` — verify write tools hidden
- [ ] Manual: start dev server, trigger task, rebuild multiple times,
verify build dirs stay capped at 4
- [ ] Manual: SIGKILL the dev CLI, verify watchdog cleans up
`.trigger/tmp/`
- [ ] Verify new API endpoints return correct data: `GET
/api/v1/query/schema`, `GET /api/v1/query/dashboards`
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Queues with concurrency keys now appear as a single entry in the master
queue instead of one entry per key. This prevents high-CK-count tenants
from consuming the entire `parentQueueLimit` window and starving other
tenants on the same shard.
A new per-queue **CK index** (sorted set) tracks active concurrency key
sub-queues. The master queue gets one `:ck:*` wildcard entry per base
queue. Dequeuing from that entry round-robins across sub-queues,
maintaining per-CK concurrency tracking and fairness.
All existing operations (enqueue, dequeue, ack, nack, DLQ, TTL expiry)
are CK-index-aware and keep the index consistent. Old-format entries
drain naturally during rollout — no migration step needed, single
deploy.
## Adds 2 self serve features
### 1. self serve preview branches
- Copies the patterns of the self serve concurrency
- Self serve only available on Pro plan (otherwise you are linked to the
billing plans page)
- Global self serve branches limit: 180 (+20 for the Pro plan). It can
be overridden per Org
- You need to archive branches before reducing the number of extra
branches you're paying for
- Branches are removed immediately but remain billed until the end of
the billing cycle like extra concurrency
### 2. self serve team members
- Copies the patterns of the self serve concurrency
- Self serve only available on Pro plan (otherwise you are linked to the
billing plans page)
- Global self serve members is unlimited but can be limited with the
same env var quota and overridden per org if needed
- You need to remove team members before reducing the number of members
you pay for
- Team members are removed immediately but remain billed until the end
of the billing cycle like extra concurrency
Deprecates the syncVercelEnvVars build extension and adds warnings in
both the Vercel integration docs and the extension's own page to prevent
conflicts with the native env var sync
The `trigger.dev deploy` and `trigger.dev dev` commands now warn (with the suggested fix) when your code loads a package through `createRequire()` that won't be available in the deployed image. Previously it would fail at runtime in production to load the package. Deploys also now show bundler warnings for your code instead of discarding them.
Repo-specific signal for anyone (human or agent) reviewing a PR in this codebase. Calibrates what counts as critical, what to always check, and what to skip.
## What makes a 🔴 Important finding here
Reserve 🔴 for things that would page someone or block a rollback. In this codebase, that means:
- **Rolling-deploy breakage.** Old and new versions of the webapp/supervisor run side-by-side during deploys. A change is broken if:
- A Lua script's behavior changes for a given key set without versioning (rename the script with a behavior-descriptive suffix like `Tracked` rather than `V2` — both versions must coexist safely).
- A Redis data shape used by both versions changes in place. New shapes need a new key namespace.
- A migration is not backward-compatible with the prior image.
- **Schema / migration safety.** Prisma migrations must be backward-compatible with the prior deploy. Adding NOT NULL without a default, dropping a column an old image still reads, renaming a column — all 🔴.
- **ClickHouse migration ordering + idempotency.** Goose runs in strict mode in the deploy pipeline and refuses to apply a missing version below the current version — slotting a new file in below the latest already-applied version blocks the deploy. New ClickHouse migration files MUST use the next available number (`max(files in internal-packages/clickhouse/schema/) + 1`); if main has added migrations while you've been on a branch, renumber yours. DDL must also be idempotent (`ADD COLUMN IF NOT EXISTS`, `DROP COLUMN IF EXISTS`, `CREATE TABLE IF NOT EXISTS`, `ADD INDEX IF NOT EXISTS`) so a partial / `--allow-missing` apply elsewhere doesn't fail on retry. Either fault is 🔴 — both break test/prod deploys. Rules live in `internal-packages/clickhouse/CLAUDE.md`.
- **Queue / concurrency correctness.** RunQueue, MarQS (V1, legacy), redis-worker — any change to enqueue / dequeue / locking semantics. Re-derive the invariant on paper before flagging or accepting.
- **Missing index on a hot table.** New Prisma queries against `TaskRun`, `TaskRunExecutionSnapshot`, `JobRun`, `Project`, etc. must use an existing index. Check `internal-packages/database/prisma/schema.prisma` for the relevant `@@index` lines — don't guess and don't propose `EXPLAIN`.
- **Recovery-path queries.** Any `TaskRun.findFirst` / `findMany` added to a schedule, run-recovery, or restart loop. Recovery fan-outs (Redis crash, restart storms) turn "rare indexed query" into a DB incident. 🔴 even if indexed.
- **Aggregations on hot tables.** No `COUNT` / `GROUP BY` on `TaskRun` or other tables that can reach billions of rows. Use Redis or ClickHouse for counts.
- **Prod Redis blast-radius.** New code paths that `SCAN` with broad patterns (`*foo*`) on prod-shaped Redis, or `EVAL` Lua with `SCAN` loops inside. Both are 🔴.
- **`@trigger.dev/core` direct import** from anywhere outside the SDK package. Always import from `@trigger.dev/sdk`. Core direct imports are 🔴 — they break the public API contract.
- **Heavy execute-deps imported into request-handler bundles.** Specifically `chat.handover` and similar split-bundle entry points must not transitively import the agent task's execute path. Watch for new imports added at module top-level of route files.
- **V1 engine code modified in a "V2 only" PR.** The `apps/webapp/app/v3/` directory contains both. If the PR description says V2-only but it touches `triggerTaskV1`, `cancelTaskRunV1`, `MarQS`, etc. — 🔴.
## Performance (always review)
Every PR gets a performance pass — not just the ones that look perf-sensitive. For each new query or unit of work, weigh three things: (a) the size of the table it hits, (b) whether it sits on a hot path, (c) whether the data it walks can be deep or wide (run trees, batches). The 🔴 bullets above on indexes, recovery-path queries, aggregations, and Redis `SCAN` are part of this pass — the rest below extends it.
**Treat these tables as large — no scans, no `COUNT` / `GROUP BY`, no unbounded fetch:**
- **Postgres — the `TaskRun` family:** `TaskRun`, `TaskRunExecutionSnapshot`, `Waitpoint`, `BatchTaskRun` and their join tables. Assume billions of rows.
- **ClickHouse — `task_events_v1` / `task_events_v2`.** Partitioned by `toDate(inserted_at)`; `ORDER BY (environment_id, toUnixTimestamp(start_time), trace_id)`. Note `span_id` / `parent_span_id` are NOT in the sort key — span-id lookups can't skip granules, only `environment_id` + a `start_time` window can.
**Hot paths — extra scrutiny on any added query or work:**
- **Trigger + batch trigger** (`triggerTask.server.ts`, `batchTriggerV3.server.ts`) — see `apps/webapp/CLAUDE.md`; do not add DB queries to these.
- **Dequeue / RunQueue** (`dequeueSystem.ts`, run-queue read/lock paths) — runs on every execution.
- **Execution-snapshot creation in the run engine** — any engine function that writes a `TaskRunExecutionSnapshot` runs per state transition; a new query there multiplies by run volume.
- Batch + parent/child fan-out (one run triggers thousands of children).
- Waitpoint / run-dependency chains.
- Tag / attribute many-to-many joins against the run/event tables.
**Anti-patterns (severity):**
- **Per-level fan-out that re-scans a large table once per tree depth** → 🔴. A BFS issuing one query per level (e.g. `parent_span_id IN {thisLevel}`) re-reads the same granules D times for a depth-D tree. Prefer one windowed query + an in-memory tree build.
- **Dropping the partition-pruning predicate** — `inserted_at` for ClickHouse, the `createdAt` window for partitioned Postgres — to "widen" a lookup → 🔴. Without it the query scans every partition. Keep a bounded window even for ancestor / backfill lookups.
- **Unbounded `IN (...)` built from a result set** (a BFS frontier, a batch's child ids) → 🟡. It can reach the row cap (`MAXIMUM_TRACE_SUMMARY_VIEW_COUNT` defaults to 25k). Cap or chunk to ≤1–2k ids per query.
- **Sequential per-level round-trips** where one recursive or windowed query would do → 🟡. N levels = N round-trip latencies stacked.
- **Replacing a single bounded query with a multi-query walk for _every_ call** (not just a rare fallback) → 🔴 on a hot read path, 🟡 elsewhere. Keep the cheap single-query path; branch into the expensive walk only when the cheap one comes up short.
## Always check
- **Tests use testcontainers, not mocks.** Vitest with `redisTest` / `postgresTest` / `containerTest` from `@internal/testcontainers`. Any new `vi.mock(...)` on Redis, Postgres, BullMQ, or other infra is wrong here — 🔴 if added in production-path tests, 🟡 if isolated unit test.
- **User-facing public-package changes have a changeset.** `pnpm run changeset:add` produces `.changeset/*.md`. Changesets are user-facing release notes, not a catalog of every change: required when a `packages/*` or `integrations/*` change is something a user would notice or act on, skipped for internal-only changes, refactors, chores, and packages not consumed independently (e.g. `@trigger.dev/redis-worker`). Missing on a user-facing change → 🟡; missing on a breaking change → 🔴. Do not flag a missing note when the change is not user-facing.
- **User-facing server-only changes have `.server-changes/*.md`.** Required for user-facing `apps/webapp/`, `apps/supervisor/` edits in a PR with no package or integration change that requires a changeset; skip internal-only or admin-only changes, refactors, and chores. Body should be 1-2 sentences (it has to fit as one bullet in a future changelog). Missing on a user-facing change → 🟡.
- **Lua script naming.** Coexisting scripts use behavior-descriptive suffixes (`Tracked`), never `V2`. Old name must keep working until the next deploy clears it.
- **RunQueue payload shape.** V2 run-queue payload's `projectId` is consumed by `workerQueueResolver` for override matching. If a PR drops it from the payload, 🔴.
- **`safeSend` scope.** Defensive IPC wrappers belong on loop / interval / handler contexts, not one-shot terminal sends. If the PR adds `safeSend` to a single terminal call for consistency, 🟡 with a "remove this" suggestion.
- **Zod version.** Pinned to `3.25.76` monorepo-wide. New package adding zod with a different version or range — 🔴.
## Skip (do NOT flag)
- Anything oxfmt / oxlint catches. CI enforces both via the `code-quality` check.
- TypeScript style preferences (`type` vs `interface`) — already covered by repo standards.
- Test coverage exhortations as a generic suggestion. Only flag missing tests when a specific code path is genuinely untested and the path has prior incidents.
-`agentcrumbs` markers (`// @crumbs`, `// #region @crumbs`) and `agentcrumbs` imports — these are temporary debug instrumentation stripped before merge.
-`// removed comments for removed code`, renamed `_unused` vars, re-exported types as "backwards compatibility shims" — also covered by repo standards.
- Suggestions to "add error handling" without naming a specific scenario that breaks.
The `apps/webapp/app/v3/` directory name is misleading — most code there is V2. Only specific files are V1-only legacy: `MarQS` queue, `triggerTaskV1`, `cancelTaskRunV1`, and a handful of others (see `apps/webapp/CLAUDE.md` for the exact list). Don't flag "you should refactor this to use V2" on those — they're frozen.
## Confidence calibration for this repo
The most common false-positive pattern: speculating about race conditions in code paths the agent doesn't have runtime visibility into. If the only evidence is "this *could* race", drop it. If you can point to a specific interleaving with file:line for each step, surface it.
description: Adversarially verifies one landed packet against its requirement; read-only.
model: opus
---
You are an adversarial code reviewer for one landed packet. READ-ONLY: never modify code, never commit, never push, never post to GitHub.
- Try to refute that the change answers its stated requirement; look for the failure scenario, not confirmation.
- Check the diff for unrelated drift, dead code, broken semantics of neighbors, and whether tests prove the actual invariant (would the test fail if the fix were subtly wrong?).
- Check the change landed in the correct PR/branch of the stack.
- Distinguish fact from inference; cite exact file:line evidence.
- Return: verdict (approve / needs-changes) with evidence per concern, and the exact minimal correction when needs-changes.
description: Implements exactly one work packet — minimal diff, targeted checks, own-paths-only commits.
model: opus
---
You are a code writer. Implement exactly the one work packet in your prompt.
- Minimal diff; match surrounding style and idiom.
- Prefer no comment at all; comment only a non-obvious constraint, max 2 short lines. All texts (comments, commit messages) short, clear, simple.
- Verify the packet's own diagnosis against the code before applying; if it is wrong, STOP without committing and report why.
- Run only the targeted checks for your packet: the relevant vitest files, `pnpm run typecheck --filter <pkg>` when the change warrants it. Never full suites unless asked.
-`pnpm run format` on touched files before committing.
- Stage and commit ONLY your packet's files. Conventional commit message. NO Claude attribution, no Co-Authored-By.
- Push only if the packet explicitly says to.
- Return: what changed, evidence (test output), commit SHA, and anything contradicting the diagnosis.
The `v3/` directory name is misleading - most code here is actively used by the current V2 engine. Only the specific files below are legacy V1-only code.
The v3 engine (RunEngineVersion `V1`: MarQS queue + Graphile worker) is end-of-life and its execution code has been removed from the webapp. The `app/v3/` directory name is historical: everything under it now serves the current V2 engine (`@internal/run-engine` + `@trigger.dev/redis-worker`).
## V1-Only Files - Never Modify
There is no `V1` execution path anymore. If you find a `RunEngineVersion` branch, the `V1` arm should only reject or finalize gracefully (for example, mark a historical run cancelled in the DB), never run V1 work. Do not reintroduce MarQS, the graphile worker, or the v3 socket.io namespaces.
-`authenticatedSocketConnection.server.ts` (V1 dev WebSocket using DevQueueConsumer)
-`sharedSocketConnection.ts` (V1 shared queue socket using SharedQueueConsumer)
## The deprecation boundary (keep this)
## V1/V2 Branching Pattern
Requests from clients still on v3 (old SDK/CLI) or historical V1 runs must return a clean 4xx, never a 5xx. The boundary lives in:
Some services act as routers that branch on `RunEngineVersion`:
-`services/cancelTaskRun.server.ts` - calls V1 service or `engine.cancelRun()` for V2
-`services/batchTriggerV3.server.ts` - uses marqs for V1 path, run-engine for V2
-`engineDeprecation.server.ts` - the `V3_TRIGGER_DEPRECATION_MESSAGE` / `V3_DEV_DEPRECATION_MESSAGE` / `V3_MIGRATION_URL` upgrade messages.
-`engineVersion.server.ts` - `determineEngineVersion()` still detects a V1 project/run so callers can reject it.
-`services/triggerTask.server.ts`, `services/cancelTaskRun.server.ts`, `services/rescheduleTaskRun.server.ts` - the `V1` arm rejects or finalizes gracefully instead of executing.
-`services/initializeDeployment.server.ts` - the `DEPRECATE_V3_CLI_DEPLOYS_ENABLED`-gated v3 CLI deploy rejection.
-`handleWebsockets.server.ts` - the legacy `trigger dev` websocket closes with the upgrade message.
When editing these shared services, only modify V2 code paths.
- **Background jobs**: `@trigger.dev/redis-worker` (`commonWorker.server.ts`, `alertsWorker.server.ts`, `batchTriggerWorker.server.ts`; `legacyRunEngineWorker.server.ts` still hosts the live batch-completion jobs)
- **Queue operations**: RunQueue inside run-engine (`runQueue.server.ts`), not MarQS
When adding a new dependency to any package.json in the monorepo:
1.**Look up the latest version** on npm before adding:
```bash
pnpm view <package-name> version
```
If unsure which version to use (e.g. major version compatibility), confirm with the user.
2. **Edit the package.json directly** — do NOT use `pnpm add` as it can cause issues in the monorepo. Add the dependency with the correct version range (typically `^x.y.z`).
3. **Run `pnpm i` from the repo root** after editing to install and update the lockfile:
```bash
pnpm i
```
Always run from the repo root, not from the package directory.
When modifying server apps (webapp, supervisor, coordinator, etc.) with **no package changes**, add a `.server-changes/` file instead of a changeset:
`.server-changes/` files are user-facing release notes, not a catalog of every change. When a user-facing server app change (webapp, supervisor, etc.) is in a PR with **no package or integration change that requires a changeset**, add a `.server-changes/` file instead of a changeset. Skip it for internal-only or admin-only changes, refactors, and chores:
- If the PR also touches `packages/`, just the changeset is sufficient (no `.server-changes/` needed).
- If the PR also touches `packages/` or `integrations/` and that change needs a changeset, the changeset covers it (no `.server-changes/` needed). If the package or integration change is internal and needs no changeset, still add a `.server-changes/` file for the user-facing server change.
The body ships **verbatim in user-facing release notes**. Keep it to 1–2 short sentences, non-technical, written for a dashboard user: describe what changed for them, never the implementation (no header names, endpoints, middleware, storage mechanisms, internal tools). See `.server-changes/README.md` for full guidance.
description: Use this skill when writing or modifying Drizzle ORM schemas, queries, or migrations in this repo — specifically the `@internal/dashboard-agent-db` package (the dashboard agent's conversation datastore). Covers pg-core schema definition, the postgres-js driver, drizzle-kit migrations, and this repo's conventions: a dedicated Postgres schema, foreign-key-free cross-database design, pooler-safe connections, and the access-pattern query layer. Drizzle is NOT the main database — that's Prisma.
Drizzle is used in exactly one place: **`internal-packages/dashboard-agent-db`** (`@internal/dashboard-agent-db`), the in-dashboard agent's conversation store. Everything else in the monorepo is **Prisma** (`@trigger.dev/database`). Keep them separate.
Pinned versions: **`drizzle-orm` ^0.45**, **`drizzle-kit` ^0.31** (dev), **`postgres` ^3.4** (postgres.js driver). drizzle-orm and drizzle-kit are intentionally on different version lines — 0.31.x is the correct companion for 0.45.x, there is no peer dependency between them.
## Critical rules
1.**Drizzle is only the agent's own datastore.** The agent (and its task bundle) must have **no access to the main Prisma database or ClickHouse**. Never import the Prisma client into the agent task or into `@internal/dashboard-agent-db`. Main data is reached via the API, not Drizzle.
2.**Foreign-key-free.** In cloud this DB is a *separate* PlanetScale database, so it can't FK into the main DB. Reference main entities (`organizationId`, `userId`, …) **by id only — never `.references()`**. Joins happen in app code; tenant scoping is enforced in the query layer.
3.**One dedicated Postgres schema.** All tables live under `pgSchema("trigger_dashboard_agent")` so they're schema-qualified and isolated from Prisma's `public` schema (this is what makes the OSS single-database fallback safe).
4.**Pooler-safe connections.** Connections go through a transaction-mode pooler (PlanetScale / PgBouncer-style), so postgres.js must run with **`prepare: false`** — prepared statements don't survive a connection being handed to another client between checkouts.
5.**Node16 module resolution.** Relative imports need explicit **`.js`** extensions (`import { chats } from "./schema.js"`), even though the source is `.ts`.
6.**Scope every user query.** All queries that touch user data go through `src/queries.ts` and are scoped by `organizationId` / `userId`, so callers can't forget the `where`. Don't write ad-hoc cross-tenant queries elsewhere.
## Package layout
```text
internal-packages/dashboard-agent-db/
drizzle.config.ts # drizzle-kit config (schema path, out dir, schemaFilter)
`package.json` points `main`/`types` at `./src/index.ts` (consumed as source, no build step) — same as other simple internal packages.
## Schema (pg-core)
Use `pgSchema(...).table(...)`, not the bare `pgTable`, so tables land in the dedicated schema. ([schemas](https://orm.drizzle.team/docs/schemas), [pg column types](https://orm.drizzle.team/docs/column-types/pg), [indexes](https://orm.drizzle.team/docs/indexes-constraints))
// Inferred row types for the query layer + consumers.
exporttypeChat=typeofchats.$inferSelect;
exporttypeNewChat=typeofchats.$inferInsert;
```
Notes:
-`timestamp(..., { withTimezone: true })` → `timestamp with time zone`. Use `.defaultNow()` for `DEFAULT now()`.
- For a "newest first, nulls last" sort the partial index uses `.desc()`; the *query* uses raw `sql` for `NULLS LAST` (see below).
- Don't add `.references()` — see critical rule 2.
## Client (postgres.js + drizzle)
([connect overview](https://orm.drizzle.team/docs/connect-overview)) One small pool, `prepare: false`. In the agent task create it once in `onBoot` (per-process); in the webapp wrap it in the `singleton(...)` helper.
`drizzle.config.ts` must set **`schemaFilter`** so drizzle-kit only ever manages our schema — never Prisma's `public` (critical in the OSS single-DB fallback):
pnpm run db:generate # diff schema.ts → emit SQL into drizzle/. OFFLINE (no DB needed).
# review the generated drizzle/000N_*.sql before committing
pnpm run db:migrate # apply pending migrations. Needs a real DATABASE URL.
```
- `db:generate` is **offline** — it only reads `schema.ts`, so you can verify a schema change compiles to valid DDL with no database. Use it as a fast check.
- drizzle-kit names migration files with a **random suffix** (`0000_magenta_lilandra.sql`). Don't regenerate a committed migration just to "refresh" it — that churns the filename. After the first migration is committed, schema changes produce a **new** `000N_*.sql`; commit that.
- Generated DDL for a new schema is one `CREATE SCHEMA` + schema-qualified `CREATE TABLE`s + indexes, **no foreign keys** (by design here).
## Common gotchas
- **`prepare: false`** is not optional with a pooler — without it you'll get prepared-statement errors under load.
- **Missing `.js` extension** on a relative import → TS2835 under Node16 resolution.
- **Extra-config callback returns an array** `(t) => [ ... ]` in drizzle-orm 0.36+. The old object form `(t) => ({ ... })` is deprecated.
- **`NULLS LAST` / `NULLS FIRST`** aren't on the `desc()` helper — use raw `sql\`col desc nulls last\`` in `orderBy`.
- **Don't `SELECT *` into list views** — explicitly pick columns so you never ship a megabyte `messages` blob or a session token to a list query.
- **Adding a dependency**: edit `package.json`, then `pnpm i` from the repo root (never `pnpm add`). Mind the repo's `minimumReleaseAge` (3 days) — pin with a caret range and let pnpm resolve an old-enough version.
description: End-to-end smoke test for the public Errors HTTP API (error groups). Seeds failed runs into ClickHouse so the error materialized views populate, then drives the real endpoints against the running webapp — list (with filters + pagination), retrieve, resolve/ignore/unresolve, the `filter[error]` runs filter, user attribution via the `trigger.dev mint-token` -> JWT exchange, and the 401/403/404 negatives. Use for "smoke test the errors API", "test the errors API e2e", "prove the errors endpoints work", or to re-verify after changes.
allowed-tools: Read, Bash
---
# Errors API — end-to-end smoke test
Proves the public Errors API against the **running** webapp with real HTTP. No
mocks. The error data plane is ClickHouse (`errors_v1` + `error_occurrences_v1`,
both materialized-view-fed from `task_runs_v2`) plus Postgres `ErrorGroupState`
for lifecycle status; this skill seeds straight into `task_runs_v2` and lets the
- Attribution: `api.v1.projects.$projectRef.$env.jwt.ts` stamps `act:{sub}` for PAT **and** UAT exchanges; `@trigger.dev/rbac` surfaces `act.sub` through bearer auth; the action handlers read `authentication.actor?.sub`.
`errorId` is `error_<fingerprint>` (round-trips via `ErrorId` in `@trigger.dev/core/v3/isomorphic`).
## Prerequisites
- Webapp running on http://localhost:3030 (`pnpm run dev --filter webapp`). Confirm `curl -s http://localhost:3030/healthcheck`.
- DB seeded (`pnpm run db:seed`), and a local ClickHouse reachable at `CLICKHOUSE_URL` (the `pnpm run docker` stack).
- The CLI built + logged in to localhost:3030 (`pnpm run build --filter trigger.dev`; profile `default` points at localhost:3030). Needed only for the attribution leg.
> Important wiring facts the seed relies on (verified):
> - The MVs read the error type/message from `error.data.*`, so the seeded
> - The MVs only fire for failed statuses: `SYSTEM_FAILURE | CRASHED | INTERRUPTED | COMPLETED_WITH_ERRORS | TIMED_OUT`, and require a non-empty `error_fingerprint`.
> - `GET /api/v1/runs` lists run **ids** from ClickHouse but **hydrates from Postgres** `TaskRun`. So the error-list/detail/action legs work from a ClickHouse-only seed, but the `filter[error]` leg needs a **paired** Postgres `TaskRun` row whose `id` equals the ClickHouse `run_id`.
Run everything from the repo root in one shell. Invoke the built CLI via a
function (a `CLI="node …"` variable won't word-split under zsh):
Q=$(python3 -c "import urllib.parse;print(urllib.parse.quote('INSERT INTO trigger_dev.task_runs_v2 FORMAT JSONEachRow'))")
mkrow(){# status fingerprint errorType message runId
echo"{\"environment_id\":\"$ENV\",\"organization_id\":\"$ORG\",\"project_id\":\"$PROJ\",\"run_id\":\"$5\",\"friendly_id\":\"run_$5\",\"status\":\"$1\",\"environment_type\":\"DEVELOPMENT\",\"engine\":\"V2\",\"task_identifier\":\"$TASK\",\"created_at\":\"$NOW_CH\",\"updated_at\":\"$NOW_CH\",\"error\":{\"data\":{\"type\":\"$3\",\"message\":\"$4\",\"stack\":\"at x (a.ts:1:1)\"}},\"error_fingerprint\":\"$2\",\"task_version\":\"20240101.1\",\"_version\":\"$NOW_MS\",\"_is_deleted\":0}"
# Poll until both fingerprints appear in errors_v1 (the MV is near-instant locally).
for i in $(seq 1 10);do
N=$(curl -s "$CHURL" --data-binary "SELECT count() FROM (SELECT 1 FROM trigger_dev.errors_v1 WHERE environment_id='$ENV' AND error_fingerprint IN ('$FP_A','$FP_B') GROUP BY error_fingerprint)")
["$N"="2"]&& break; sleep 1
done
echo"seeded fingerprints in errors_v1: $N (want 2)"
curl -s "$B/api/v1/errors?filter%5BtaskIdentifier%5D=$TASK&filter%5Bsearch%5D=AlphaBoom&filter%5Bperiod%5D=1d" -H "$H"| python3 -c "import sys,json;print('search:',[e['errorType'] for e in json.load(sys.stdin)['data']])"# ['AlphaBoom']
curl -s "$B/api/v1/errors/$ERRID_B" -H "$H"| st # status ignored | ignoredUntil True | reason known flake
curl -s -X POST "$B/api/v1/errors/$ERRID_A/unresolve" -H "$H" >/dev/null
curl -s "$B/api/v1/errors/$ERRID_A" -H "$H"| st # status unresolved
```
PASS: each transition reflected; `filter[status]=ignored` returns only beta:
```bash
curl -s "$B/api/v1/errors?filter%5BtaskIdentifier%5D=$TASK&filter%5Bstatus%5D=ignored&filter%5Bperiod%5D=1d" -H "$H"| python3 -c "import sys,json;print([e['id'] for e in json.load(sys.stdin)['data']])"# [error_<fpB>]
```
### 5. `filter[error]` on the runs list (paired PG + CH seed)
The runs list hydrates from Postgres, so seed a matching `TaskRun` row + a CH row
curl -s "$B/api/v1/runs?filter%5Berror%5D=error_$FP_R" -H "$H"| python3 -c "import sys,json;d=json.load(sys.stdin);print('runs:',[r['id'] for r in d['data']])"
```
PASS: one run, `run_<RID>` (status maps to `FAILED`). Proves `filter[error]` -> fingerprint -> CH -> PG hydration.
### 6. Attribution — `mint-token` -> JWT exchange records the acting user
PASS: `act.sub` is the user id (matches `cli whoami`), and `detail.resolvedBy` equals that user id (not null). A plain env key leaves it null (step 4). A **PAT** exchanged the same way also stamps `act` — repeat with the stored PAT to confirm `ignoredByUserId` attribution.
unresolve flip status (and `filter[status]` follows); step 5's `filter[error]`
returns the paired run; step 6 records `resolvedBy` = the acting user via the
JWT exchange (null with a plain env key); and step 7 returns 404/401/401/403/200.
A red leg is a bug or a missing prereq — report the exact status + body and file
a Linear issue, don't tune around it.
## Notes / gotchas
- Run files use a unique `$RUN` suffix per invocation, so reruns don't collide and seeded rows stay isolated by their unique task identifier. They are local-dev test rows (90-day ClickHouse TTL); no cleanup required.
- After **adding** the route files, the classic Remix dev compiler may not register them until a dev-server restart (a stale manifest returns Remix's HTML 404 on the new paths). If `POST …/resolve` returns a 404 HTML page rather than 401/200, restart `pnpm run dev --filter webapp`.
- The rbac `act` extraction lives in `@trigger.dev/rbac` (a built dep). After editing it, `pnpm run build --filter @trigger.dev/rbac` and restart the webapp so the attribution leg (step 6) reflects the change.
description: Use when adding, modifying, or debugging OTel span timeline events in the trace view. Covers event structure, ClickHouse storage constraints, rendering in SpanTimeline component, admin visibility, and the step-by-step process for adding new events.
The trace view's right panel shows a timeline of events for the selected span. These are OTel span events rendered by `app/utils/timelineSpanEvents.ts` and the `SpanTimeline` component.
## How They Work
1.**Span events** in OTel are attached to a parent span. In ClickHouse, they're stored as separate rows with `kind: "SPAN_EVENT"` sharing the parent span's `span_id`. The `#mergeRecordsIntoSpanDetail` method reassembles them into the span's `events` array at query time.
2. The timeline only renders events whose `name` starts with `trigger.dev/` - all others are silently filtered out.
3. The **display name** comes from `properties.event` (not the span event name), mapped through `getFriendlyNameForEvent()`.
4. Events are shown on the **span they belong to** - events on one span don't appear in another span's timeline.
## ClickHouse Storage Constraint
When events are written to ClickHouse, `spanEventsToTaskEventV1Input()` filters out events whose `start_time` is not greater than the parent span's `startTime`. Events at or before the span start are silently dropped. This means span events must have timestamps strictly after the span's own `startTimeUnixNano`.
## Timeline Rendering (SpanTimeline component)
The `SpanTimeline` component in `app/components/run/RunTimeline.tsx` renders:
1.**Events** (thin 1px line with hollow dots) - all events from `createTimelineSpanEventsFromSpanEvents()`
2.**"Started"** marker (thick cap) - at the span's `startTime`
3.**Duration bar** (thick 7px line) - from "Started" to "Finished"
4.**"Finished"** marker (thick cap) - at `startTime + duration`
The thin line before "Started" only appears when there are events with timestamps between the span start and the first child span. For the Attempt span this works well (Dequeued -> Pod scheduled -> Launched -> etc. all happen before execution starts). Events all get `lineVariant: "light"` (thin) while the execution bar gets `variant: "normal"` (thick).
## Trace View Sort Order
Sibling spans (same parent) are sorted by `start_time ASC` from the ClickHouse query. The `createTreeFromFlatItems` function preserves this order. Event timestamps don't affect sort order - only the span's own `start_time`.
## Event Structure
```typescript
// OTel span event format
{
name:"trigger.dev/run",// Must start with "trigger.dev/" to render
timeUnixNano:"1711200000000000000",
attributes:[
{key:"event",value:{stringValue:"dequeue"}},// The actual event type
{key:"duration",value:{intValue: 150}},// Optional: duration in ms
]
}
```
## Admin-Only Events
`getAdminOnlyForEvent()` controls visibility. Events default to **admin-only** (`true`).
| Event | Admin-only | Friendly name |
|-------|-----------|---------------|
| `dequeue` | No | Dequeued |
| `fork` | No | Launched |
| `import` | No (if no fork event) | Importing task file |
The main trigger.dev webapp, which powers it's API and dashboard and makes up the docker image that is produced as an OSS image, is a Remix 2.1.0 app that uses an express server, written in TypeScript. The following subsystems are either included in the webapp or are used by the webapp in another part of the monorepo:
The main trigger.dev webapp, which powers it's API and dashboard and makes up the docker image that is produced as an OSS image, is a Remix 2.17.4 app that uses an express server, written in TypeScript. The following subsystems are either included in the webapp or are used by the webapp in another part of the monorepo:
- `@trigger.dev/database` exports a Prisma 6.14.0 client that is used extensively in the webapp to access a PostgreSQL instance. The schema file is [schema.prisma](mdc:internal-packages/database/prisma/schema.prisma)
- `@trigger.dev/core` is a published package and is used to share code between the `@trigger.dev/sdk` and the webapp. It includes functionality but also a load of Zod schemas for data validation. When importing from `@trigger.dev/core` in the webapp, we never import the root `@trigger.dev/core` path, instead we favor one of the subpath exports that you can find in [package.json](mdc:packages/core/package.json)
ENCRYPTION_KEY=ae13021afef0819c3a307ad487071c06 # Must be a random 16 byte hex string. You can generate an encryption key by running `openssl rand -hex 16` in your terminal
MANAGED_WORKER_SECRET=abcdef1234 # Must match the supervisor's MANAGED_WORKER_SECRET
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0 # zizmor:ignore[artipacked] changesets/action pushes the release branch; no artifact upload here so no leak path
You are auditing this PR for drift against `.claude/REVIEW.md`.
## Context
`.claude/REVIEW.md` is the repo's source of truth for what AI / agent code reviewers should treat as critical findings (rolling-deploy safety, hot-table indexes, recovery-path queries, testcontainers usage, Lua versioning, etc.). It is consumed by review agents to calibrate severity. If REVIEW.md goes stale, every future agent review degrades.
## Strategy — read this first
You have a hard turn budget. Spend it on signal, not coverage. The audit is allowed to miss things; it is NOT allowed to time out.
1. Read `.claude/REVIEW.md` once, in full.
2. Run `git diff origin/main...HEAD --name-only` to get the list of changed files. Do NOT read the diff content yet.
3. Scan the file-list for relevance to REVIEW.md scope. Relevance signals: changes to Prisma schema, Redis / queue / Lua code, hot tables, recovery / restart loops, new packages, deletions of paths REVIEW.md cites. Skim everything else.
4. Open at most **5 files** total — only the ones most likely to surface a real signal. If nothing in the file-list looks relevant to any REVIEW.md rule, do NOT read any files; go straight to the verdict.
5. Form a verdict and stop. Do not exhaust the turn budget exploring.
Large PRs (>50 files changed) are a strong signal to be MORE selective, not more thorough. Pick 3-5 files at most.
## What to look for
- **Stale references** — does any REVIEW.md rule cite a file, directory, function, table, Prisma model, or package name that has been removed or renamed in this PR (or is already gone from `main`)?
- **Contradictions** — does code in this PR clearly violate a current REVIEW.md rule? (Don't re-review the PR. Only flag if REVIEW.md and the PR plainly disagree.)
- **Missing rules** — does this PR introduce a new pattern future reviewers should know about? Examples: a new hot table, a new Lua-script versioning convention, a new safety wrapper, a new "must always check" invariant.
- **Obsolete rules** — has the repo moved past a constraint REVIEW.md still asserts? (e.g. a deprecated path is gone, a pattern is now linted, V1 code is deleted.)
## Response format
If nothing needs changing:
✅ REVIEW.md looks current for this PR.
Otherwise:
📝 **REVIEW.md updates suggested:**
- **[stale]** `<rule excerpt>` — <what's stale and why>
- **[contradiction]** `<rule excerpt>` — <what in this PR disagrees>
- **[missing]** under `## <section>` — <one-sentence draft rule>
- **[obsolete]** `<rule excerpt>` — <why this rule no longer applies>
## Rules
- Maximum 3 suggestions per audit. Pick the highest-signal ones.
- Only flag things that would actually mislead a future reviewer. Style and wording do not count.
- Do NOT review the PR itself. Do NOT propose rules outside REVIEW.md's existing sections.
- Do NOT propose rules for one-off PR specifics that don't generalize to future PRs.
- If REVIEW.md does not exist in the repo, respond with `(skip)` and stop.
- When in doubt between "one more file read" and "finish now" — finish now.
You are reviewing a PR to check whether any CLAUDE.md files or .claude/rules/ files need updating.
You are reviewing a PR to check whether any agent instruction files need updating.
In this repo:
- Root shared agent guidance lives in `AGENTS.md`.
- Root `CLAUDE.md` is only a Claude Code adapter that imports `AGENTS.md`.
- Subdirectories may still have scoped `CLAUDE.md` files.
- `.claude/rules/` contains additional Claude Code guidance.
## Your task
1. Run `git diff origin/main...HEAD --name-only` to see which files changed in this PR.
2. For each changed directory, check if there's a CLAUDE.md in that directory or a parent directory.
3. Determine if any CLAUDE.md or .claude/rules/ file should be updated based on the changes. Consider:
2. For each changed directory, check the applicable instruction files: root `AGENTS.md`, any `CLAUDE.md` in that directory or a parent directory, and relevant `.claude/rules/` files.
3. Determine if any instruction file should be updated based on the changes. Consider:
- New files/directories that aren't covered by existing documentation
- Changed architecture or patterns that contradict current CLAUDE.md guidance
- New dependencies, services, or infrastructure that Claude should know about
- Renamed or moved files that are referenced in CLAUDE.md
- Changed architecture or patterns that contradict current agent guidance
- New dependencies, services, or infrastructure that agents should know about
- Renamed or moved files that are referenced in an instruction file
- Changes to build commands, test patterns, or development workflows
## Response format
If NO updates are needed, respond with exactly:
✅ CLAUDE.md files look current for this PR.
✅ Agent instruction files look current for this PR.
If updates ARE needed, respond with a short list:
📝 **CLAUDE.md updates suggested:**
📝 **Agent instruction updates suggested:**
- `AGENTS.md`: [what should be added/changed]
- `path/to/CLAUDE.md`: [what should be added/changed]
- `.claude/rules/file.md`: [what should be added/changed]
Keep suggestions specific and brief. Only flag things that would actually mislead Claude in future sessions.
Keep suggestions specific and brief. Only flag things that would actually mislead agents in future sessions.
Do NOT suggest updates for trivial changes (bug fixes, small refactors within existing patterns).
Do NOT suggest creating new CLAUDE.md files - only updates to existing ones.
Do NOT suggest creating new CLAUDE.md files - only updates to existing instruction files.
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0 # zizmor:ignore[artipacked] needs persisted git creds for tag push; no artifact upload here so no leak path
pull_request_target: # zizmor:ignore[dangerous-triggers] needed to comment/close fork PRs; safe because we never check out PR HEAD ref so no fork-controlled code runs
gh pr close ${{ github.event.pull_request.number }} \
--repo ${{ github.repository }} \
--comment "Thanks for your contribution! We require all external PRs to be opened in **draft** status first so you can address CodeRabbit review comments and ensure CI passes before requesting a review. Please re-open this PR as a draft. See [CONTRIBUTING.md](https://github.com/${{ github.repository }}/blob/main/CONTRIBUTING.md#pr-workflow) for details."
This directory tracks changes to server-only components (webapp, supervisor, coordinator, etc.) that are not captured by changesets. Changesets only track published npm packages — server changes would otherwise go undocumented.
This directory tracks changes to server-only components (webapp, supervisor, etc.) that are not captured by changesets. Changesets only track published npm packages — server changes would otherwise go undocumented.
## When to add a file
**Server-only PRs**: If your PR only changes `apps/webapp/`, `apps/supervisor/`, `apps/coordinator/`, or other server components (and does NOT change anything in `packages/`), add a `.server-changes/` file.
These entries are **user-facing release notes**, not a catalog of every change. The test is "would a user or customer care about this change?", not "did I touch a server app?". Add one only when a server-only change is something a user would notice, act on, or want to hear about (a feature, a bug fix they could have hit, a behavior or performance change they would feel). Skip it for internal-only or admin-only changes, refactors, test-only changes, chores, and performance tuning with no user-visible effect. Anyone who wants the exact history reads the commits. When in doubt, ask a maintainer rather than adding a note by default.
**Mixed PRs** (both packages and server): Just add a changeset as usual. No `.server-changes/` file needed — the changeset covers it.
**Server-only PRs**: If your PR only changes `apps/webapp/`, `apps/supervisor/`, or other server components (and does NOT change anything in `packages/`) AND the change is user-facing, add a `.server-changes/` file.
**Package-only PRs**: Just add a changeset as usual.
**Mixed PRs** (both packages and server): the changeset covers it, so no `.server-changes/` file is needed. If the package change is internal and needs no changeset but the server change is user-facing, add a `.server-changes/` file for it.
**Package-only PRs**: Just add a changeset as usual, when the change is user-facing.
## File format
@@ -31,13 +33,27 @@ Speed up batch queue processing by removing stalls and fixing retry race
The body text (below the frontmatter) is a one-line description of the change. Keep it concise — it will appear in release notes.
### Writing guidance
These entries are public-facing - they ship verbatim in user-visible release notes. A few rules to keep them clean:
- **Write for the user, not the reviewer.** Lead with what the user notices or has to do. If a reader who doesn't know the codebase can't tell what changed for them, rewrite it.
- **One sentence is usually enough.** The body is the bullet in the changelog. If you need a paragraph, you're probably describing the implementation rather than the change.
- **Describe behavior, not implementation.** Skip internal scopes, middleware names, library specifics, framework internals. Users care about what's different for them, not how it's wired.
- **Never name internal tools or infra.** Observability stacks, internal services, infra components, monitoring backends, CI surfaces, AWS specifics - none of these belong in user-facing notes.
Before / after:
- ❌ _"The image verification step now parses the manifest's layer media types and returns a new result the finalizer rejects."_ (describes the wiring; a user can't act on it)
- ✅ _"Deploying with an outdated CLI could produce an image that fails to start on every run. These deploys are now stopped before going live, with a message asking you to upgrade the CLI and re-deploy."_ (what the user sees and does)
## Lifecycle
1. Engineer adds a `.server-changes/` file in their PR
Added `/engine/v1/dev/disconnect` endpoint to auto-cancel runs when the CLI disconnects. Maximum of 500 runs can be cancelled. Uses the bulk action system when there are more than 25 runs to cancel.
A durable guard improves reliability for runs waiting on triggerAndWait or batchTriggerAndWait if there's a database error that interrupts a child run finishing.
This repository is a pnpm monorepo managed with Turbo. It contains multiple apps and packages that make up the Trigger.dev platform and SDK.
This file provides guidance to Claude Code when working with this repository. Subdirectory CLAUDE.md files provide deeper context when you navigate into specific areas.
## Repository layout
-`apps/webapp`– Remix application that serves as the main API and dashboard.
-`apps/supervisor`– Node application for executing built tasks.
-`packages/*`– Published packages such as `@trigger.dev/sdk`, the CLI (`trigger.dev`), and shared libraries.
-`internal-packages/*`– Internal-only packages used by the webapp and other apps.
-`references/*`– Example projects for manual testing and development of new features.
-`ai/references`– Contains additional documentation including an overview (`repo.md`) and testing guidelines (`tests.md`).
## Build and Development Commands
See `ai/references/repo.md` for a more complete explanation of the workspaces.
This is a pnpm 10.33.2 monorepo using Turborepo. Run commands from root with `pnpm run`.
## Development setup
1. Install dependencies with `pnpm i` (pnpm `10.23.0` and Node.js `20.20.0` are required).
2. Copy `.env.example` to `.env` and generate a random 16 byte hex string for `ENCRYPTION_KEY` (`openssl rand -hex 16`). Update other secrets if needed.
3. Start the local services with Docker:
```bash
pnpm run docker
```
4. Run database migrations:
```bash
pnpm run db:migrate
```
5. Build the webapp, CLI and SDK packages:
```bash
pnpm run build --filter webapp && pnpm run build --filter trigger.dev && pnpm run build --filter @trigger.dev/sdk
```
6. Launch the development server:
```bash
pnpm run dev --filter webapp
```
The webapp runs on <http://localhost:3030>.
**Adding dependencies:** Edit `package.json` directly instead of using `pnpm add`, then run `pnpm i` from the repo root. See `.claude/rules/package-installation.md` for the full process.
For full setup instructions see `CONTRIBUTING.md`.
```bash
pnpm run docker # Core dev services (Postgres, Redis, Electric, MinIO, ClickHouse, s2-lite)
# pnpm run docker:full # Same + observability stack (Prometheus, Grafana, OTEL) and chaos tooling
pnpm run db:migrate # Run database migrations
pnpm run db:seed # Seed the database (required for reference projects)
## Running tests
- Unit tests use **vitest**. Run all tests:
```bash
pnpm run test
```
- Run tests for a specific workspace (example for `webapp`):
```bash
pnpm run test --filter webapp
```
- Prefer running a single test file from within its directory:
```bash
cd apps/webapp
pnpm run test ./src/components/Button.test.ts
```
If packages in that workspace need to be built first, run `pnpm run build --filter webapp`.
# Build packages (required before running)
pnpm run build --filter webapp && pnpm run build --filter trigger.dev && pnpm run build --filter @trigger.dev/sdk
Refer to `ai/references/tests.md` for details on writing tests. Tests should avoid mocks or stubs and use the helpers from `@internal/testcontainers` when Redis or Postgres are needed.
pnpm run dev --filter webapp # Run webapp (http://localhost:3030)
pnpm run dev --filter trigger.dev --filter "@trigger.dev/*"# Watch CLI and packages
```
## Coding style
- Formatting is enforced using Prettier. Run `pnpm run format` before committing.
- Follow the existing project conventions. Test files live beside the files under test and use descriptive `describe` and `it` blocks.
- Do not commit directly to the `main` branch. All changes should be made in a separate branch and go through a pull request.
### Verifying Changes
## Additional docs
- The root `README.md` describes Trigger.dev and links to documentation.
- The `docs` workspace contains our documentation site, which can be run locally with:
```bash
pnpm run dev --filter docs
```
- `references/README.md` explains how to create new reference projects for manual testing.
The verification command depends on where the change lives:
- **Apps and internal packages** (`apps/*`, `internal-packages/*`): Use `typecheck`. **Never use `build`** for these — building proves almost nothing about correctness.
- **Public packages** (`packages/*`): Use `build`.
```bash
# Apps and internal packages — use typecheck
pnpm run typecheck --filter webapp # ~1-2 minutes
pnpm run typecheck --filter @internal/run-engine
# Public packages — use build
pnpm run build --filter @trigger.dev/sdk
pnpm run build --filter @trigger.dev/core
```
Only run typecheck/build after major changes (new files, significant refactors, schema changes). For small edits, trust the types and let CI catch issues.
## Testing
We use vitest exclusively. **Never mock anything** - use testcontainers instead.
```bash
pnpm run test --filter webapp # All tests for a package
cd internal-packages/run-engine
pnpm run test ./src/engine/tests/ttl.test.ts --run # Single test file
pnpm run build --filter @internal/run-engine # May need to build deps first
```
Test files go next to source files (e.g., `MyService.ts` -> `MyService.test.ts`).
redisTest("should use redis",async({redisOptions})=>{
/* ... */
});
postgresTest("should use postgres",async({prisma})=>{
/* ... */
});
containerTest("should use both",async({prisma,redisOptions})=>{
/* ... */
});
```
## Code Style
### Formatting and linting
Format and lint are enforced by CI (`code-quality` check). Run before committing:
```bash
pnpm run format # oxfmt — auto-fixes formatting
pnpm run lint:fix # oxlint — auto-fixes lint violations
pnpm run lint # oxlint — check only (no fixes)
```
### Dead code
We use knip to control unused dependencies and code. It is enforced by CI `code-quality`.
Scan your code before pushing with:
```bash
pnpm run knip
```
If there are false positives, edit ./knip.json so that it passes.
### Imports
**Prefer static imports over dynamic imports.** Only use dynamic `import()` when:
- Circular dependencies cannot be resolved otherwise
- Code splitting is genuinely needed for performance
- The module must be loaded conditionally at runtime
Dynamic imports add unnecessary overhead in hot paths and make code harder to analyze. If you find yourself using `await import()`, ask if a regular `import` statement would work instead.
## Changesets and Server Changes
Changesets and `.server-changes/` files are **user-facing release notes**. They ship verbatim into the changelog that customers read to decide what to upgrade for or pay attention to. They are not a catalog of every change: anyone who wants the exact history reads the commits. So the question is not "did I touch a public package or a server app?" but **"would a user or customer care about this change?"**
**Add one** when the change is something a user would notice, act on, or want to hear about: a new feature, a bug fix they could have hit, a behavior or performance change they would feel, a breaking change.
**Skip it** (no changeset, no `.server-changes/` file) when the change is not worth communicating to users, even if it touches a public package or a server app. For example:
- internal-only or admin-only changes, refactors, test-only changes, chores
- performance or query tuning with no user-visible behavior change
- changes to a public package that is not consumed independently (e.g. `@trigger.dev/redis-worker`), where a version bump means nothing to a user
When in doubt, ask a maintainer rather than adding a note by default. An unnecessary entry is noise in the changelog, not a safe default.
### How to add one
When a **public package** (`packages/*` or `integrations/*`) change is user-facing, add a changeset:
```bash
pnpm run changeset:add
```
- Default to **patch** for bug fixes and minor changes
- Confirm with maintainers before selecting **minor** (new features)
- **Never** select major without explicit approval
When a **server-only** change (`apps/webapp/`, `apps/supervisor/`, etc., with no package changes) is user-facing, add a `.server-changes/` file instead. See `.server-changes/README.md` for format and documentation.
**Write the description for users, not maintainers.** Both changesets and `.server-changes/` notes ship verbatim in user-visible release notes. Lead with what changed *for the user*: one plain sentence describing behavior, not implementation, and never naming internal tools or infra. The full writing guidance in `.server-changes/README.md` applies to changesets too.
## Dependency Pinning
Zod is pinned to a single version across the entire monorepo (currently `3.25.76`). When adding zod to a new or existing package, use the **exact same version** as the rest of the repo - never a different version or a range. Mismatched zod versions cause runtime type incompatibilities (e.g., schemas from one package can't be used as body validators in another).
## Architecture Overview
### Request Flow
User API call -> Webapp routes -> Services -> RunEngine -> Redis Queue -> Supervisor -> Container execution -> Results back through RunEngine -> ClickHouse (analytics) + PostgreSQL (state)
v3 (engine V1: MarQS + Graphile worker) is end-of-life and its execution code has been removed. The `apps/webapp/app/v3/` directory name is historical - everything there now serves V2 (Run Engine 2.0, `@internal/run-engine` + redis-worker). There is no V1 execution path: a `RunEngineVersion``V1` branch only rejects or finalizes gracefully so v3 clients get a clean 4xx, never a 5xx. Do not reintroduce V1. See `apps/webapp/CLAUDE.md` and `.claude/rules/legacy-v3-code.md`.
### Documentation
Docs live in `docs/` as a Mintlify site (MDX format). See `docs/CLAUDE.md` for conventions.
### Reference Projects
Reference/example projects for testing SDK and platform features live in a separate repo: [`triggerdotdev/references`](https://github.com/triggerdotdev/references). Clone it alongside this repo and use its `projects/hello-world` to manually test changes before submitting PRs. See that repo's README for setup and linking to a local monorepo build.
## Docker Image Guidelines
When updating Docker image references:
- **Always use multiplatform/index digests**, not architecture-specific digests
- Architecture-specific digests cause CI failures on different build environments
- Use the digest from the main Docker Hub page, not from a specific OS/ARCH variant
## Writing Trigger.dev Tasks
Always import from `@trigger.dev/sdk`. Never use `@trigger.dev/sdk/v3` or deprecated `client.defineJob`.
```typescript
import{task}from"@trigger.dev/sdk";
exportconstmyTask=task({
id:"my-task",
run: async(payload:{message: string})=>{
// Task logic
},
});
```
### SDK Documentation Rules
The `rules/` directory contains versioned SDK documentation distributed via the SDK installer. Current version: `rules/manifest.json`. Do NOT update `rules/` or `.claude/skills/trigger-dev-tasks/` unless explicitly asked - these are maintained in separate dedicated passes.
## Testing with the hello-world Reference Project
The reference projects live in the separate [`triggerdotdev/references`](https://github.com/triggerdotdev/references) repo - clone it alongside this repo.
First-time setup:
1.`pnpm run db:seed` to seed the database (creates the References org + hello-world project)
2. Build the CLI/packages you want to test: `pnpm run build --filter trigger.dev`
3. In your `references` clone, follow its README to link to your local monorepo build, then authorize: `cd projects/hello-world && pnpm exec trigger login -a http://localhost:3030`
The preferred way to query for crumbs is to use `pnpm exec agentcrumbs query --app trigger` with the `--limit` option and cursor pagination, and clear existing crumbs before reproducing a bug via `pnpm exec agentcrumbs clear --app trigger`.
@@ -4,11 +4,13 @@ Trigger.dev uses [changesets](https://github.com/changesets/changesets) to manag
## Adding a changeset (package changes)
To add a changeset, use `pnpm run changeset:add` and follow the instructions [here](https://github.com/changesets/changesets/blob/main/docs/adding-a-changeset.md). Please only ever select one of our public packages when adding a changeset.
Changesets and `.server-changes/` files are user-facing release notes, not a catalog of every change. Add one only when the change is something a user would notice or acton. Skip it for internal-only changes, refactors, chores, and packages that are not consumed independently (e.g. `@trigger.dev/redis-worker`). Anyone who wants the exact history reads the commits.
To add a changeset, use `pnpm run changeset:add` and follow the [Changesets adding-a-changeset guide](https://github.com/changesets/changesets/blob/main/docs/adding-a-changeset.md). Please only ever select one of our public packages when adding a changeset.
## Adding a server change (server-only changes)
If your PR only changes server components (`apps/webapp/`, `apps/supervisor/`, etc.) and does NOT change any published packages, add a `.server-changes/` file instead of a changeset:
If your PR only changes server components (`apps/webapp/`, `apps/supervisor/`, etc.), does NOT change any published packages, AND the change is user-facing, add a `.server-changes/` file instead of a changeset:
For **mixed PRs** (both packages and server): just add a changeset. No `.server-changes/` file needed.
For **mixed PRs** (both packages and server): the changeset covers it, so no `.server-changes/` file is needed. If the package change is internal and needs no changeset but the server change is user-facing, add a `.server-changes/` file for it.
See `.server-changes/README.md` for full documentation.
## When to add which
Only for user-facing changes. Skip the note entirely for internal-only or admin-only changes, refactors, and chores.
| PR changes | What to add |
|---|---|
| Only packages (`packages/`) | Changeset (`pnpm run changeset:add`) |
| Only server (`apps/`) | `.server-changes/` file |
| Both packages and server | Just the changeset |
| Only packages (`packages/` or `integrations/`) | Changeset (`pnpm run changeset:add`), if the package change is user-facing |
| Only server (`apps/`) | `.server-changes/` file, if the server change is user-facing |
| Both packages and server | The changeset covers it; if the package change needs no changeset but the server change is user-facing, add a `.server-changes/` file |
This file provides guidance to Claude Code when working with this repository. Subdirectory CLAUDE.md files provide deeper context when you navigate into specific areas.
## Build and Development Commands
This is a pnpm 10.23.0 monorepo using Turborepo. Run commands from root with `pnpm run`.
```bash
pnpm run docker # Start Docker services (PostgreSQL, Redis, Electric)
pnpm run db:migrate # Run database migrations
pnpm run db:seed # Seed the database (required for reference projects)
# Build packages (required before running)
pnpm run build --filter webapp && pnpm run build --filter trigger.dev && pnpm run build --filter @trigger.dev/sdk
pnpm run dev --filter webapp # Run webapp (http://localhost:3030)
pnpm run dev --filter trigger.dev --filter "@trigger.dev/*"# Watch CLI and packages
```
## Testing
We use vitest exclusively. **Never mock anything** - use testcontainers instead.
```bash
pnpm run test --filter webapp # All tests for a package
cd internal-packages/run-engine
pnpm run test ./src/engine/tests/ttl.test.ts --run # Single test file
pnpm run build --filter @internal/run-engine # May need to build deps first
```
Test files go next to source files (e.g., `MyService.ts` -> `MyService.test.ts`).
redisTest("should use redis",async({redisOptions})=>{/* ... */});
postgresTest("should use postgres",async({prisma})=>{/* ... */});
containerTest("should use both",async({prisma,redisOptions})=>{/* ... */});
```
## Changesets and Server Changes
When modifying any public package (`packages/*` or `integrations/*`), add a changeset:
```bash
pnpm run changeset:add
```
- Default to **patch** for bug fixes and minor changes
- Confirm with maintainers before selecting **minor** (new features)
- **Never** select major without explicit approval
When modifying only server components (`apps/webapp/`, `apps/supervisor/`, etc.) with no package changes, add a `.server-changes/` file instead. See `.server-changes/README.md` for format and documentation.
## Architecture Overview
### Request Flow
User API call -> Webapp routes -> Services -> RunEngine -> Redis Queue -> Supervisor -> Container execution -> Results back through RunEngine -> ClickHouse (analytics) + PostgreSQL (state)
- **internal-packages/zodworker**: Graphile-worker wrapper (DEPRECATED - use redis-worker)
### Legacy V1 Engine Code
The `apps/webapp/app/v3/` directory name is misleading - most code there is actively used by V2. Only specific files are V1-only legacy (MarQS queue, triggerTaskV1, cancelTaskRunV1, etc.). See `apps/webapp/CLAUDE.md` for the exact list. When you encounter V1/V2 branching in services, only modify V2 code paths. All new work uses Run Engine 2.0 (`@internal/run-engine`) and redis-worker.
### Documentation
Docs live in `docs/` as a Mintlify site (MDX format). See `docs/CLAUDE.md` for conventions.
### Reference Projects
The `references/` directory contains test workspaces for testing SDK and platform features. Use `references/hello-world` to manually test changes before submitting PRs.
## Docker Image Guidelines
When updating Docker image references:
- **Always use multiplatform/index digests**, not architecture-specific digests
- Architecture-specific digests cause CI failures on different build environments
- Use the digest from the main Docker Hub page, not from a specific OS/ARCH variant
## Writing Trigger.dev Tasks
Always import from `@trigger.dev/sdk`. Never use `@trigger.dev/sdk/v3` or deprecated `client.defineJob`.
```typescript
import{task}from"@trigger.dev/sdk";
exportconstmyTask=task({
id:"my-task",
run: async(payload:{message: string})=>{
// Task logic
},
});
```
### SDK Documentation Rules
The `rules/` directory contains versioned SDK documentation distributed via the SDK installer. Current version: `rules/manifest.json`. Do NOT update `rules/` or `.claude/skills/trigger-dev-tasks/` unless explicitly asked - these are maintained in separate dedicated passes.
@@ -49,9 +49,9 @@ branch are tagged into a release periodically.
```
cd trigger.dev
```
3. Ensure you are on the correct version of Node.js (20.20.0). If you are using `nvm`, there is an `.nvmrc` file that will automatically select the correct version of Node.js when you navigate to the repository.
3. Ensure you are on the correct version of Node.js (24.18.0). If you are using `nvm`, there is an `.nvmrc` file that will automatically select the correct version of Node.js when you navigate to the repository.
4. Run `corepack enable` to use the correct version of pnpm (`10.23.0`) as specified in the root `package.json` file.
4. Run `corepack enable` to use the correct version of pnpm (`10.33.2`) as specified in the root `package.json` file.
5. Install the required packages using pnpm.
```
@@ -71,21 +71,27 @@ branch are tagged into a release periodically.
Feel free to update `SESSION_SECRET` and `MAGIC_LINK_SECRET` as well using the same method.
8. Start Docker. This starts the required services like Postgres & Redis. If this is your first time using Docker, consider going through this [guide](DOCKER_INSTALLATION.md)
8. Start Docker. This starts the core dev services (Postgres, Redis, Electric, MinIO, ClickHouse, s2-lite) and runs the ClickHouse migrator once on first start. If this is your first time using Docker, consider going through this [guide](DOCKER_INSTALLATION.md).
```
pnpm run docker
```
For the observability stack (Prometheus, Grafana, OTEL collector) and other optional tooling (Toxiproxy, nginx-h2, ch-ui, extra electric shard), use `pnpm run docker:full` instead. See `docker/docker-compose.extras.yml` for the full list.
9. Migrate the database
```
pnpm run db:migrate
```
10. Build everything
10. Build the webapp, CLI, and SDK
```
pnpm run build --filter webapp && pnpm run build --filter trigger.dev && pnpm run build --filter @trigger.dev/sdk
pnpm run build --filter webapp --filter trigger.dev --filter @trigger.dev/sdk
```
11. Run the app. See the section below.
11. Seed the database. This creates a local user, a `References` org, and the reference projects (including `hello-world`) with stable IDs.
```
pnpm run db:seed
```
12. Run the app. See the section below.
## Running
@@ -101,29 +107,32 @@ branch are tagged into a release periodically.
## Manual testing using hello-world
We use the `<root>/references/hello-world` subdirectory as a staging ground for testing changes to the SDK (`@trigger.dev/sdk` at `<root>/packages/trigger-sdk`), the Core package (`@trigger.dev/core` at `<root>packages/core`), the CLI (`trigger.dev` at `<root>/packages/cli-v3`) and the platform (The remix app at `<root>/apps/webapp`). The instructions below will get you started on using the `hello-world` for local development of Trigger.dev.
The `hello-world` reference project (and the others) live in a separate repo:
[`triggerdotdev/references`](https://github.com/triggerdotdev/references). Clone it
alongside this repo. It's the staging ground for testing changes to the SDK
(`@trigger.dev/sdk` at `<root>/packages/trigger-sdk`), the Core package
(`@trigger.dev/core` at `<root>/packages/core`), the CLI (`trigger.dev` at
`<root>/packages/cli-v3`) and the platform (the Remix app at `<root>/apps/webapp`).
To exercise your local monorepo changes, the reference project links to your local
build — see the references repo's README for the `pnpm run link` flow.
> Paths below such as `projects/hello-world` are relative to your `references`
> clone, not this repo.
### First-time setup
First, make sure you are running the webapp according to the instructions above. Then:
First, make sure you are running the webapp according to the instructions above. The seed step from setup already created a `hello-world` project under the `References` org with the stable ref `proj_rrkpdguyagvsoktglnod` — log in at http://localhost:3030 with any email to access it. Then:
1. Visit http://localhost:3030 in your browser and create a new project called "hello-world".
2. In Postgres go to the "Projects" table and for the project you create change the `externalRef` to `proj_rrkpdguyagvsoktglnod`.
3. Build the CLI
1. Build the CLI and packages (skip if you already ran the build step in setup)
```sh
# Build the CLI
pnpm run build --filter trigger.dev
# Make it accessible to `pnpm exec`
pnpm i
pnpm run build --filter trigger.dev --filter "@trigger.dev/*"
```
4. Change into the `<root>/references/hello-world` directory and authorize the CLI to the local server:
2. In your `references` clone, link to your local monorepo build (see its README), then change into `projects/hello-world` and authorize the CLI to the local server:
```sh
cd references/hello-world
cd projects/hello-world
cp .env.example .env
pnpm exec trigger login -a http://localhost:3030
```
@@ -133,7 +142,7 @@ This will open a new browser window and authorize the CLI against your local use
You can optionally pass a `--profile` flag to the `login` command, which will allow you to use the CLI with separate accounts/servers. We suggest using a profile called `local` for your local development:
```sh
cd references/hello-world
cd projects/hello-world
pnpm exec trigger login -a http://localhost:3030 --profile local
# later when you run the dev or deploy command:
pnpm exec trigger dev --profile local
@@ -146,46 +155,46 @@ The following steps should be followed any time you start working on a new featu
1. Make sure the webapp is running on localhost:3030
2. Open a terminal window and build the CLI and packages and watch for changes
2. In this repo, open a terminal window and build the CLI and packages and watch for changes (the reference project links against this build)
```sh
pnpm run dev --filter trigger.dev --filter "@trigger.dev/*"
```
3. Open another terminal window, and change into the `<root>/references/hello-world` directory.
3. Open another terminal window, and change into `projects/hello-world` in your `references` clone.
4. Run the `dev` command, which will register all the local tasks with the platform and allow you to start testing task execution:
```sh
# in <root>/references/hello-world
# in <references-clone>/projects/hello-world
pnpm exec trigger dev
```
If you want additional debug logging, you can use the `--log-level debug` flag:
```sh
# in <root>/references/hello-world
# in <references-clone>/projects/hello-world
pnpm exec trigger dev --log-level debug
```
6. If you make any changes in the CLI/Core/SDK, you'll need to `CTRL+C` to exit the `dev` command and restart it to pickup changes. Any changes to the files inside of the `hello-world/src/trigger` dir will automatically be rebuilt by the `dev` command.
5. If you make any changes in the CLI/Core/SDK, you'll need to `CTRL+C` to exit the `dev` command and restart it to pickup changes. Any changes to the files inside the reference project's `src/trigger` dir will automatically be rebuilt by the `dev` command.
7. Navigate to the `hello-world` project in your local dashboard at localhost:3030 and you should see the list of tasks.
6. Navigate to the `hello-world` project in your local dashboard at localhost:3030 and you should see the list of tasks.
8. Go to the "Test" page in the sidebar and select a task. Then enter a payload and click "Run test". You can tell what the payloads should be by looking at the relevant task file inside the `/references/hello-world/src/trigger` folder. Many of them accept an empty payload.
7. On the Tasks page, open a task and press the "Test" button to open its test page. Then enter a payload and click "Run test". You can tell what the payloads should be by looking at the relevant task file inside the reference project's `src/trigger` folder. Many of them accept an empty payload.
9. Feel free to add additional files in `hello-world/src/trigger` to test out specific aspects of the system, or add in edge cases.
8. Feel free to add additional files in the reference project's `src/trigger` dir to test out specific aspects of the system, or add in edge cases.
@@ -193,65 +202,66 @@ pnpm exec trigger dev --log-level debug
This creates a migration file. Check the migration file does only what you want. If you're adding any database indexes they must use `CONCURRENTLY`, otherwise they'll lock the table when executed.
4. Run the migration.
4. Run the migration:
```
pnpm run db:migrate:deploy
pnpm run generate
```
```
pnpm run db:migrate:deploy
pnpm run generate
```
This executes the migrations against your database and applies changes to the database schema(s), and then regenerates the Prisma client.
This executes the migrations against your database and applies changes to the database schema(s), and then regenerates the Prisma client.
4. Commit generated migrations as well as changes to the schema.prisma file
5. If you're using VSCode you may need to restart the Typescript server in the webapp to get updated type inference. Open a TypeScript file, then open the Command Palette (View > Command Palette) and run `TypeScript: Restart TS server`.
5. Commit the generated migration files as well as the changes to `schema.prisma`.
6. If you're using VSCode you may need to restart the TypeScript server in the webapp to get updated type inference. Open a TypeScript file, then open the Command Palette (View > Command Palette) and run `TypeScript: Restart TS server`.
## Add sample jobs
## Git hooks (lefthook)
The [references/job-catalog](./references/job-catalog/) project defines simple jobs you can get started with.
We use [lefthook](https://lefthook.dev) for local git hooks, configured in `lefthook.yml` (the source of truth for what runs and when). Today that's a pre-push hook mirroring the CI `code-quality` checks; the set may grow, so check `lefthook.yml` rather than this guide.
1. `cd` into `references/job-catalog`
2. Create a `.env` file with the following content,
replacing `<TRIGGER_DEV_API_KEY>` with an actual key:
Hooks install automatically on `pnpm install`. A failing hook prints exactly what to run to fix it.
```env
TRIGGER_API_KEY=[TRIGGER_DEV_API_KEY]
TRIGGER_API_URL=http://localhost:3030
```
**Opting out**
`TRIGGER_API_URL` is used to configure the URL for your Trigger.dev instance,
where the jobs will be registered.
- GitButler skips hooks on `but push` unless you enable **Run hooks** in the project settings (off by default).
- Plain git: `LEFTHOOK=0 git push` / `--no-verify` to skip once; `pnpm exec lefthook uninstall` to remove.
3. Run one of the the `job-catalog` files:
```sh
pnpm run events
```
This will open up a local server using `express` on port 8080. Then in a new terminal window you can run the trigger-cli dev command:
```sh
pnpm run dev:trigger
```
See the [Job Catalog](./references/job-catalog/README.md) file for more.
4. Navigate to your trigger.dev instance ([http://localhost:3030](http://localhost:3030/)), to see the jobs.
You can use the test feature to trigger them.
This never affects correctness — CI enforces the same checks on every PR; the hooks just give you faster feedback.
## Making a pull request
**If you get errors, be sure to fix them before committing.**
- Be sure to [check the "Allow edits from maintainers" option](https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/allowing-changes-to-a-pull-request-branch-created-from-a-fork) while creating you PR.
- If your PR refers to or fixes an issue, be sure to add `refs #XXX` or `fixes #XXX` to the PR description. Replacing `XXX` with the respective issue number. See more about [Linking a pull request to an issue
> **Note:** We may close PRs if we decide that the cost of integrating the change outweighs the benefits. To improve the chances of your PR getting accepted, follow the guidelines below.
### PR workflow
1. **Always open your PR in draft status first.** Do not mark it as "Ready for Review" until the steps below are complete.
2. **Run format and lint locally before pushing:**
```bash
pnpm run format
pnpm run lint
pnpm run knip
```
These are enforced by CI — the `code-quality` check will fail if either produces a diff or errors.
3. **Address all CodeRabbit code review comments.** Our CI runs an automated code review via CodeRabbit. Go through each comment and either fix the issue or resolve it with a comment explaining why no change is needed.
4. **Wait for all CI checks to pass.** Do not mark the PR as "Ready for Review" until every check is green.
5. **Then mark the PR as "Ready for Review"** so a maintainer can take a look.
### Cost/benefit analysis for risky changes
If your change touches core infrastructure, modifies widely-used code paths, or could introduce regressions, consider doing a brief cost/benefit analysis and including it in the PR description. Explain what the benefit is to users and why the risk is worth it. This goes a long way toward helping maintainers evaluate your contribution.
### General guidelines
- Be sure to [check the "Allow edits from maintainers" option](https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/allowing-changes-to-a-pull-request-branch-created-from-a-fork) while creating your PR.
- If your PR refers to or fixes an issue, be sure to add `refs #XXX` or `fixes #XXX` to the PR description. Replacing `XXX` with the respective issue number. See more about [Linking a pull request to an issue](https://docs.github.com/en/issues/tracking-your-work-with-issues/linking-a-pull-request-to-an-issue).
- Be sure to fill the PR Template accordingly.
## Adding changesets
We use [changesets](https://github.com/changesets/changesets) to manage our package versions and changelogs. If you've never used changesets before, first read [their guide here](https://github.com/changesets/changesets/blob/main/docs/adding-a-changeset.md).
If you are contributing a change to any packages in this monorepo (anything in either the `/packages` or `/integrations` directories), then you will need to add a changeset to your Pull Requests before they can be merged.
Changesets are user-facing release notes, not a catalog of every change. If you are contributing a **user-facing** change to a package in this monorepo (anything in `/packages` or `/integrations` that a user would notice or act on), add a changeset to your Pull Request before it can be merged. Skip the changeset for internal-only changes, refactors, chores, and packages that are not consumed independently (e.g. `@trigger.dev/redis-worker`), where a version bump means nothing to a user.
To add a changeset, run the following command in the root of the repo
@@ -269,7 +279,7 @@ Most of the time the changes you'll make are likely to be categorized as patch r
## Adding server changes
Changesets only track published npm packages. If your PR only changes server components (`apps/webapp/`, `apps/supervisor/`, `apps/coordinator/`, etc.) with no package changes, add a `.server-changes/` file so the change appears in release notes.
Changesets only track published npm packages. If your PR only changes server components (`apps/webapp/`, `apps/supervisor/`, etc.) with no package changes AND the change is user-facing, add a `.server-changes/` file so the change appears in release notes. Skip it for internal-only or admin-only changes, refactors, and chores.
The body text (below the frontmatter) is a one-line description of the change. Keep it concise — it will appear in release notes.
**When to add which:**
**When to add which** (only for user-facing changes; skip the note entirely for internal-only or admin-only changes, refactors, and chores):
| PR changes | What to add |
|---|---|
| Only packages (`packages/`) | Changeset |
| Only server (`apps/`) | `.server-changes/` file |
| Both packages and server | Just the changeset |
| Only packages (`packages/` or `integrations/`) | Changeset (if the package change is user-facing) |
| Only server (`apps/`) | `.server-changes/` file (if the server change is user-facing) |
| Both packages and server | The changeset covers it; if the package change needs no changeset but the server change is user-facing, add a `.server-changes/` file |
See `.server-changes/README.md` for more details.
@@ -320,3 +330,7 @@ The process running on port `3030` should be destroyed.
```sh
sudo kill -9 <PID>
```
### Running two clones side by side (worktree, branch experiment)
The default `pnpm run docker` uses the project name `triggerdotdev-docker` and the standard host ports (5432, 6379, 3060, 4566, 8123, 9000, 9005, 9006). To stand up a second instance in another clone without clashing, set a different `COMPOSE_PROJECT_NAME` and the offset host ports in that clone's `.env`. The "Running multiple instances side by side" block in `.env.example` lists every overridable env var with its default for reference; uncomment the lines you need and update `DATABASE_URL` / `CLICKHOUSE_URL` / `REDIS_PORT` / `APP_ORIGIN` / `LOGIN_ORIGIN` / `ELECTRIC_ORIGIN` / `REALTIME_STREAMS_S2_ENDPOINT` to match.
# Onboarding: taking over the hosted webhooks PR (#4344) for design
You are picking up **PR #4344 "hosted webhooks, agent channels, and human-in-the-loop"** to own the UX and front-end. This doc gets you from a clean machine to a running dashboard with realistic webhook data you can screenshot, restyle, and iterate on.
The feature is built and green (all backend plumbing, all four dashboard surfaces, the in-app test console). Your job is the visual and interaction design of the dashboard surfaces, not the backend. Everything below is oriented around that.
---
## 1. What you are designing
Hosted webhooks let a Trigger.dev user receive and verify a provider's webhooks (Stripe, GitHub, and so on) as a task, with no ingress or verification code of their own. A `webhook()` handler in their project gets a hosted URL; deliveries to that URL are verified, recorded, and routed to their `onEvent` handler.
The dashboard has **four surfaces you own**, all under the "Webhooks" nav section (teal icon):
| Surface | Route (under `/orgs/:org/projects/:project/env/:env`) | What it shows |
| --- | --- | --- |
| **Deliveries list** | `/webhooks` | Every delivery across all endpoints in the environment. Runs-style filter bar (Status, Webhook, Created, plus a More-filters menu for Delivery ID / Run ID), applied-filter pills, a Webhook column linking to the handler. This is the main screen. |
| **Delivery detail** | `/webhooks/deliveries/:deliveryParam` | One delivery. Main panel is a tabbed view (Event payload / Request headers) rendered as JSON. Sidebar property table (status badge, webhook + run links, external delivery id, idempotency key, timestamps, computed duration, error). Also has a friendly "not available / retained for N days" empty state for expired or bogus links. |
| **Handler detail + Console** | `/webhooks/:webhookParam` | The handler (the `webhook()` in the user's code). Tabs: Deliveries, Runs, Endpoints. This page also hosts the **Webhook Console / Composer** (see section 5), the tool you will lean on for data. |
| **Endpoint detail** | `/webhooks/endpoints/:endpointParam` | One endpoint. Left: scoped deliveries. Right: a **Connect** card (webhook URL, signing secret set/rotate/generate, provider setup rendered from the verifier config), Routing, Scope, Metadata. |
The status vocabulary, badges, and colors live in `components/webhookDeliveries/v1/DeliveryStatus.tsx` and `components/webhookEndpoints/v1/EndpointStatus.tsx`. The nav accent color is a Tailwind token `--color-webhooks` (teal), used via `text-webhooks`.
---
## 2. Get the code
You need the PR branch, `feat/hosted-webhook-ingress`.
gh pr checkout 4344# lands you on feat/hosted-webhook-ingress
```
If you plan to push design changes back to this branch, coordinate with Eric first: the branch is rebased and force-pushed periodically, so agree on timing or work on a child branch and open a follow-up.
Toolchain: pnpm 10.33.2 via corepack, Node 22+. Use `corepack pnpm` (a bare `pnpm` can be an old global that wipes `node_modules`).
```bash
corepack enable
corepack pnpm install
```
---
## 3. Bring the stack up
Four services and the webapp. Run from the repo root.
Now edit `.env` and add the two webhook-delivery replication lines (they are NOT in `.env.example`, and without them the Deliveries list looks empty even after you send webhooks, see section 5):
```bash
# webhook deliveries replication (required for the Deliveries list/detail to populate)
**Log in (dev):** open http://localhost:3030, submit the email `local@trigger.dev`. Dev auto-verifies the magic link (watch the webapp log for `/magic?token=`). That seeded user is an org admin, which matters for the next step.
---
## 4. Turn the feature on
The dashboard is gated by a feature flag, `hasWebhooksAccess` (default off).
- The seeded dev user `local@trigger.dev` is an **admin**, and admins bypass the flag, so on a fresh seed the Webhooks nav section is already visible to you. Nothing to do.
- If you use a non-admin user, set `featureFlags.hasWebhooksAccess = true` on the `Organization` row to reveal the nav section. (A global `FeatureFlag` row with key `hasWebhooksAccess` makes the pages reachable by URL, but the left nav reads only org-level flags, so the section stays hidden for non-admins.)
If the "Webhooks" section is missing from the left nav, this flag is why.
---
## 5. Get nice data (the part that matters)
Delivery rows are what make these screens interesting: a spread of providers, statuses, payloads, timestamps. Here is how the data flows and how to produce it.
### The pipeline (why an empty list is usually a setup issue, not a bug)
`ingest -> engine (verify, filter, route) -> Postgres WebhookDelivery rows -> replication -> ClickHouse`. The Deliveries **list orders and paginates from ClickHouse**, then hydrates every visible field from Postgres. So if replication is off (section 3), you can create deliveries and still see an empty list. Enable the two replication env vars and restart the webapp.
One caveat baked into the design: replication starts streaming from the moment it is enabled, so deliveries written **before** you turned it on will not appear. Turn replication on first, then generate data.
### Fastest path: the seed script
There is a seed script that inserts a full, stable dataset directly into both stores (Postgres and ClickHouse), so you get realistic screens on a fresh DB with no workers, no `trigger dev`, and no signing secrets to set:
```bash
corepack pnpm --filter webapp run db:seed:webhooks
# optional: deliveries per endpoint (default 45)
corepack pnpm --filter webapp run db:seed:webhooks -- 60
```
It creates six endpoints across different providers and verifier schemes (Stripe, GitHub, Slack, Svix, Discord, and a custom shared-secret one, with a mix of active/inactive and secret-set/not-set), then a spread of deliveries over the last two weeks covering **every** delivery status (SUCCEEDED, FAILED, FILTERED, PENDING, PROCESSING), realistic per-provider payloads and headers, and a mix of test and live. It attaches to the first DEVELOPMENT environment your local user can see (set `WEBHOOK_SEED_PROJECT="<project name>"` to target a specific one), and prints the exact Deliveries URL when it finishes. Re-running clears and reseeds that environment, so you always get the same clean dataset. The script is `apps/webapp/seed-webhook-deliveries.ts`; edit the `ENDPOINTS` array or the status weights to shape the data to whatever you are designing.
Because it writes the ClickHouse rows directly, seeded data shows up **without** the replication setup in section 3. That replication env is only needed for the live and Composer paths below. (The seed uses `WEBHOOK_DELIVERIES_REPLICATION_CLICKHOUSE_URL` if set, otherwise `CLICKHOUSE_URL`, which is already in `.env.example`.)
This is the recommended way to get data. The interactive paths below are for exercising the live pipeline (real verification, real routed runs) or the in-app test console.
### Interactive: create an endpoint (one-time)
The Composer sends to an endpoint, and endpoints only exist once a Trigger project that declares a `webhook()` has been dev-run or deployed. Quickest path: a tiny demo project.
// event is the parsed body, headers is a Web Headers object
},
});
// A real provider, for realistic payloads:
exportconststripeWebhook=webhook({
id:"stripe-webhook",
source: webhooks.stripe(),
onEvent: async({event})=>{},
});
```
Link that demo project to your local build and run `trigger dev` (see `AGENTS.md` "Testing with the hello-world Reference Project" for linking; the `triggerdotdev/references` repo has ready-made projects). Running `trigger dev` registers the `webhook()` handlers, which creates their endpoints. Set each endpoint's signing secret from the **endpoint detail Connect card** (Generate or paste).
### Interactive: fire deliveries with the Webhook Console
Open the handler detail page (`/webhooks/:webhookParam`). It hosts the **Composer** (`components/webhookConsole/WebhookComposer.tsx`). It has four source tabs and four signature modes, and it injects the delivery straight through the engine in-process, so it is fast and does not consume any real rate budget:
- **Sample tab**: pick a real provider event from the built-in catalog (`@internal/webhook-sources`, six first-class providers plus a large sample manifest). This is the fastest way to get realistic Stripe / GitHub / Svix / Square / Discord payloads with correct-looking headers.
- **Body tab**: hand-write any JSON.
- **Replay tab**: re-send a prior delivery.
- **AI tab**: generate a payload with a prompt.
- **Signature modes** `signed | unsigned | tampered | simulate`: this is how you produce a **spread of delivery statuses**. `signed` (with a secret set) verifies and routes to a SUCCEEDED delivery; `unsigned` and `tampered` produce failed/rejected deliveries. Send a mix to populate every status badge you need to design.
To get SUCCEEDED deliveries whose **runs** also complete (nicest end-to-end data), keep the demo project's `trigger dev` running so the routed task actually executes.
### Interactive: a real provider (most realistic)
For genuine payloads and headers, point the Stripe CLI at an endpoint: `stripe listen --forward-to http://localhost:3030/webhooks/v1/ingest/<opaqueId>`, set that endpoint's `whsec` via the Connect card, then `stripe trigger payment_intent.succeeded`.
| Data (presenters, read-only from your side) | `apps/webapp/app/presenters/v3/WebhookDeliveriesListPresenter.server.ts`, `WebhookDeliveryDetailPresenter.server.ts`, `WebhookDetailPresenter.server.ts`, `webhookComposerEndpoints.server.ts` |
| Nav entry | `apps/webapp/app/components/navigation/SideMenu.tsx` (the `staticSections` "webhooks" push) |
| Accent color token | `apps/webapp/app/tailwind.css` (`--color-webhooks`, used as `text-webhooks`) |
| Data seed script | `apps/webapp/seed-webhook-deliveries.ts` (run via `db:seed:webhooks`) |
**Styling:** the webapp is on Tailwind v4 (CSS-first `@theme` in `apps/webapp/app/tailwind.css`, there is no `tailwind.config.js`). Add or change design tokens there.
**Design language to match:** these screens deliberately reuse the Runs page primitives (the filter bar is built from `RunFilters` / `SharedFilters`, the tables mirror the Runs table cells). Match the Runs and Sessions pages, not a new visual system.
---
## 7. Iterating
- **HMR vs restart:** editing a component (`.tsx`) hot-reloads. Editing a `.server.ts` file makes the Remix dev server restart the app (a brief connection refused, then it comes back). Editing Tailwind tokens hot-reloads.
- **Screenshots:** capture from the running dashboard at http://localhost:3030. Save shots outside the repo or to a scratch folder so they do not get committed.
- **Typecheck after non-trivial changes:** `corepack pnpm run typecheck --filter webapp` (about 1 to 2 minutes). For small style tweaks, trust it and let CI catch anything.
- **One boundary gotcha that the dev server will NOT catch:** route files must not leak server-only imports into the client bundle. The dev server tolerates it, but the production build fails. If you touch a route file and import anything server-only, run `corepack pnpm --filter webapp run build:remix` before pushing. Pure component and style edits are unaffected.
---
## 8. Shipping your changes
Follow the repo PR workflow:
- Format and lint before committing: `corepack pnpm run format` (oxfmt) and `corepack pnpm run lint:fix` (oxlint). CI enforces both.
- Commit style is Conventional Commits, for example `feat(webapp): redesign webhook deliveries table`. No emoji, no attribution footer.
- The PR is a **draft** awaiting an AI review pass, then a human review, before it flips to ready. Do not flip it to ready yourself; push your commits and let Eric coordinate the review and any rebase onto `main`.
- CI to expect: `code-quality` (oxfmt + oxlint), `typecheck`, webapp unit shards, and the Playwright `e2e-webapp` job. Style-only changes usually only risk `code-quality`.
---
## 9. Quick reference
- **Webapp:** http://localhost:3030 (port comes from `REMIX_APP_PORT`, falling back to `PORT`/3030).
- **Seed data:** `corepack pnpm --filter webapp run db:seed:webhooks` (append `-- <n>` for deliveries per endpoint).
- **Must-set env for data to show:** `WEBHOOK_DELIVERIES_REPLICATION_ENABLED=1` and `WEBHOOK_DELIVERIES_REPLICATION_CLICKHOUSE_URL=http://default:password@localhost:8123`.
- **Login:** `local@trigger.dev`, magic link auto-verifies in dev.
- **Feature docs:** `docs/webhooks/` (overview, sources, connect, deliveries, channels, human-in-the-loop). Read `overview.mdx` and `deliveries.mdx` first for the mental model behind the screens.
A user writes a `webhook()` in their project. On deploy (or `trigger dev`) that handler gets one or more hosted endpoints, each with a signing secret. A provider POSTs to the endpoint's URL; the engine verifies the signature, optionally filters, records a `WebhookDelivery`, and triggers the routed task run. The dashboard reads those deliveries: the list orders them out of ClickHouse and hydrates the rest from Postgres, the detail page reads Postgres directly (it holds the only copy of the event payload and headers). Everything you design sits on top of that delivery record and the endpoint that produced it.
We take the security of Trigger.dev seriously — for both our Cloud service and self-hosted deployments. This document explains how to report a vulnerability and what to expect from us.
## Reporting a vulnerability
**Please do not report security vulnerabilities through public GitHub issues, pull requests, or our Discord.**
Use one of these private channels instead:
1.**GitHub (preferred):** Open a private report from the repository's **Security** tab — click **"Report a vulnerability"** ([direct link](https://github.com/triggerdotdev/trigger.dev/security/advisories/new)).
2.**Email:**`security@trigger.dev`
Please include as much of the following as you can:
- A description of the vulnerability and its impact
- Steps to reproduce, ideally with a proof of concept
- Affected version(s) and component(s)
- Any suggested remediation
If you report by email, we will open a private GitHub Security Advisory to track the issue. All reports — however they reach us — are tracked there.
## What to expect
| Stage | Target |
| --- | --- |
| Acknowledgement of your report | within 3 business days |
| Validation and severity assessment (CVSS 3.1) | within 1 week |
We assess severity using CVSS 3.1 and prioritise remediation accordingly:
| Severity (CVSS 3.1) | Target time to resolve |
| --- | --- |
| Critical (9.0–10.0) | 7 days |
| High (7.0–8.9) | 30 days |
| Medium (4.0–6.9) | 90 days |
| Low (0.1–3.9) | As needed |
These are best-effort targets, measured from the point we validate and accept a report — not guarantees. Real-world exploitability may lead us to escalate an issue beyond its base score.
## Coordinated disclosure
We follow coordinated disclosure. Please give us a reasonable opportunity to investigate and ship a fix before any public disclosure. Our default disclosure window is 90 days from acceptance, though we aim to resolve issues sooner.
Once a fix is released we publish a GitHub Security Advisory (and request a CVE where applicable), and we credit reporters unless you ask to remain anonymous.
## Supported versions
We patch the **latest released version line** only. Self-hosters should run the latest version-tagged release to receive security fixes. See the [self-hosting documentation](https://trigger.dev/docs/self-hosting/overview).
This is a pnpm 10.23.0 monorepo that uses turborepo @turbo.json. The following workspaces are relevant
This is a pnpm 10.33.2 monorepo that uses turborepo @turbo.json. The following workspaces are relevant
## Apps
@@ -23,7 +23,6 @@ This is a pnpm 10.23.0 monorepo that uses turborepo @turbo.json. The following w
- <root>/internal-packages/run-engine is the `@internal/run-engine` package that is "Run Engine 2.0" and handles moving a run all the way through it's lifecycle
- <root>/internal-packages/redis is the `@internal/redis` package that exports Redis types and the `createRedisClient` function to unify how we create redis clients in the repo. It's not used everywhere yet, but it's the preferred way to create redis clients from now on.
- <root>/internal-packages/testcontainers is the `@internal/testcontainers` package that exports a few useful functions for spinning up local testcontainers when writing vitest tests. See our [tests.md](./tests.md) file for more information.
- <root>/internal-packages/zodworker is the `@internal/zodworker` package that implements a wrapper around graphile-worker that allows us to use zod to validate our background jobs. We are moving away from using graphile-worker as our background job system, replacing it with our own redis-worker package.
The `docker-provider` allows the platform to be orchestrator-agnostic. The platform can perform actions such as `INDEX_TASKS` or `INVOKE_TASK` which the provider translates into Docker actions.
Some files were not shown because too many files have changed in this diff
Show More
Reference in New Issue
Block a user
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
Saadi Myftija
