Switches the native build server from opt-in to opt-out in project build
settings.
- It's now enabled by default, stored as a new
\`disableNativeBuildServer\` opt-out key so previously-saved
\`useNativeBuildServer: false\` values aren't treated as deliberate
opt-outs.
- The "Use native build server" checkbox is checked by default;
unchecking it persists the opt-out.
- Brief wording: clarifies build settings apply to GitHub-triggered and
native build server deployments, and the native build server hint no
longer says "in the future".
## Summary
Under high OTLP ingest volume, the whole decode, transform, and enrich
pipeline runs on the request event loop, so a single CPU core becomes
the ceiling while the rest sit idle. This adds an opt-in worker pool
that moves decode, transform, and LLM-cost enrichment onto worker
threads, keeping the main thread free for I/O. It is off by default
(`OTEL_TRANSFORM_WORKER_POOL_ENABLED`), so behavior is unchanged unless
enabled.
## Design
Workers do decode, filter, convert, and enrich (including LLM pricing
match). The main thread stays the single database reader: it loads the
pricing registry and broadcasts the compiled model rows to the workers
(re-broadcasting on every reload), so workers never touch the database.
The pure transform is extracted into a dependency-light module (no
Prisma/Redis/ClickHouse imports) so it can run inside a worker.
Importantly, the main thread keeps the existing single consolidated
insert path, so ClickHouse insert batching and part count are unchanged.
The parallelism buys CPU headroom, not more insert streams (which would
add merge pressure).
The worker is bundled as a standalone file at build time and ships in
the existing image with no Dockerfile change. In local load testing the
pool sustained roughly 2.6x the throughput of the single-thread path and
kept the main thread responsive under load.
## Summary
Sending a chat message immediately after an action (for example an undo)
could make the message's response vanish from the UI. The transport
opened a response stream that closed on the *earlier* turn's completion
instead of waiting for the send's own turn. The agent still produced and
persisted the answer, so it reappeared on refresh. Same "disappearing
message" class as
[#4176](https://github.com/triggerdotdev/trigger.dev/pull/4176),
different cause.
## Fix
A send's response stream had no way to tell whether a `turn-complete`
belonged to its turn. `POST /realtime/v1/sessions/:id/in/append` now
returns the appended record's sequence number, and the transport skips
any turn-complete whose `session-in-event-id` (the agent's committed
`.in` cursor) is below that seq, closing only on its own turn. Older
webapps omit the seq, in which case the transport falls back to the
previous behavior, so the SDK and server can ship independently.
Because the fix spans the SDK and the server, both a webapp deploy and
an SDK release are needed for the full effect.
Verified end to end with the ai-chat reference app:
undo-then-immediate-send loses the follow-up's answer before the fix and
streams it inline after, with a revert-the-guard run reproducing the
loss on the same script. Unit tests cover the skip and the no-seq
fallback.
## Summary
The default realtime backend was hardcoded to Electric. This adds a
`REALTIME_BACKEND_DEFAULT` env var (`electric` | `native` | `shadow`,
default `electric`) that chooses the backend for any environment whose
org has no `realtimeBackend` override. Behavior is unchanged unless you
set it; per-org overrides still win.
The default is applied at every point where the per-org flag falls
through: the initial value, the flag lookup default, and the error
fallback.
## Problem
Several display/grouping issues in the **Errors** feature, all rooted in
how the ClickHouse error materialized views (`errors_mv_v1`,
`error_occurrences_mv_v1`) read the stored error JSON produced by
`parseError`:
1. **Messageless errors show "Unknown error".** An empty message falls
straight through `coalesce(nullIf(message,''), 'Unknown error')` to the
literal, even though the error's class `name` is available (e.g. an
Effect tagged error `ListMessagesError` with no message).
2. **Unrelated errors collapse into one group.**
`calculateErrorFingerprint` keys on `type : message : stack`, where
`type` is always the union tag (`BUILT_IN_ERROR`, …), `message` is
empty, and the stack isn't read — so every messageless built-in error
(and every string/custom error) hashes to the same constant input → one
fingerprint.
3. **error_type shows the internal tag.** `coalesce(type, name, …)`
always resolves to `type` (always present), so the column shows
`BUILT_IN_ERROR` instead of the real class name.
4. **Stack traces never populate.** The MVs read `error.data.stack`, but
the serializer stores the trace under `stackTrace` — so the column is
always empty.
## Fix
All display changes are `ALTER TABLE … MODIFY QUERY` on the two views
(migration `035`); the fingerprint change is in the webapp.
- **Fingerprint** (`errorFingerprinting.ts`): fall back **message → name
→ raw**. Messageless errors now group by class name (or raw value for
non-Error throws); message-bearing errors are **unchanged**
(short-circuits at `message`), so existing groups don't split — only
currently-messageless errors get their own group going forward.
- **error_message**: same `message → name → raw` fallback before
`'Unknown error'`.
- **error_type**: coalesce `name → code → 'Error'` (drops the reliance
on the union tag). Built-in → class name, internal → `code`,
string/custom → `Error`.
- **stack trace**: read `error.data.stackTrace`. Bounded as before
(serializer caps 50 frames / 1024 chars per line; MV clips to 2000
chars).
## Migration notes
- `MODIFY QUERY` swaps the view query in place (no drop/recreate gap);
Down restores the previous query.
- **Existing rows are left unchanged** — changes apply only to rows
inserted after the migration. No backfill.
## Tests
`errorFingerprinting.test.ts` — 57 pass, incl. new cases for messageless
class names, string/custom raw values, and stability of message-bearing
fingerprints.
Fixes the display-derivation half of TRI-11938 (error_type + stack
trace); relates to TRI-9254 and TRI-9250.
Applying a directory-sync effect that would demote the org's last Owner
(a
group remap, or a provision) previously threw and 500'd the settings
save. Now
rbac.setUserRole reports code:"last_owner" and applyEffect skips just
that
member (they keep Owner) while the rest of the batch applies.
Adds the machine-readable RoleAssignmentResult.code to the plugin
contract so
callers can tell the last-owner guard apart from a real failure.
## Summary
The RBAC and SSO auth plugins can own their own database client, but
they could only read `DATABASE_URL`, so every connection they opened
landed on the primary. The host webapp now resolves writer and
read-replica URLs from its env (the same fallback chain its own Prisma
clients use: control-plane URL first, then the default) and passes them
to the plugins at create time via a shared `PluginDatabaseConfig`, along
with separate connection limits for writes (default 2) and reads
(default 5, tunable via `RBAC_DATABASE_*_CONNECTION_LIMIT` and
`SSO_DATABASE_*_CONNECTION_LIMIT`).
A plugin can then route hot-path reads (per-request auth checks, login
routing) to the read replica and keep only rare mutations on the
primary. With no replica configured, or no plugin installed, nothing
changes: the OSS fallback ignores the new option and keeps reading
through the Prisma clients it is already given.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
## Summary
This adds an always-visible info tooltip on the Location column and a
note in the "set default region" confirmation dialog making it explicit.
It also removes the obsolete "V4" badge from the Regions page title.
## What & why
Signup promo credits. A new logged-out `/promo?code=<code>` landing page
validates the code and carries it through signup via a cookie. When the
new organization is activated by selecting a plan, the code is redeemed
and its credits are applied; the usage page then shows the remaining
promo credits and their expiry.
## Notes
- The code is redeemed at **plan selection**, not org creation: the
credit grant targets the org's usage allowance, which only exists once a
plan is selected — applying at creation would have nothing to grant
onto. Redemption is best-effort and never blocks plan selection.
- Pairs with the corresponding billing-service change (promo code
validate/apply/credits + grant issuance); the two are released together.
## Testing
Verified locally end to end: `/promo` shows the offer, a new account
carries the code through signup, selecting the Free plan redeems it, and
the usage page shows the remaining credits.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
## What
Adds per-variable secret support to the `syncEnvVars` build extension.
Return `{ name, value, isSecret: true }` and the variable is stored as a
secret (redacted in the dashboard, value non-revealable), just like a
manually created secret env var. Secret and non-secret variables can be
mixed in one callback.
```ts
syncEnvVars(async () => [
{ name: "PUBLIC_API_URL", value: "https://api.example.com" },
{ name: "DATABASE_URL", value: "postgres://...", isSecret: true },
]);
```
## How
Env vars flow through the build pipeline as a flat name→value map, and
the import API's `isSecret` is per-call. So secret vars are carried
through the layer + manifest in parallel `secretEnv` / `secretParentEnv`
maps, and at deploy time they go up in a second `importEnvVars` call
with `isSecret: true` (the plain vars in the first call). The record
form (`{ KEY: "value" }`) is unchanged and stays non-secret.
## Commits
- `feat(core)`: carry secret env vars through the build layer + manifest
schema
- `feat(build)`: partition `isSecret` vars in `syncEnvVars`
- `feat(cli)`: merge secret layers and import them with `isSecret: true`
at deploy
- `test(build)`: cover the partitioning + document `isSecret`
## Testing
- vitest covers the partitioning (secret/non-secret × child/parent) and
that the record form stays non-secret.
- Verified against a local webapp that the deploy's import contract
stores the secret var redacted (`isSecret: true`) and the plain var
visible.
Closes TRI-11099
Creating a Prisma migration now formats its schema first, keeping
migration-related schema edits consistently formatted without adding
work to the repository-wide format command. Run `pnpm run format:prisma`
to format either schema on demand.
## Summary
A Query page (TRQL) query that pulls fields out of a run's `output` with
JSON functions (`JSONExtractString`, `JSONExtractInt`, `JSONHas`, and
the rest of the family) failed with "The first argument of function ...
should be a string containing JSON, illegal type: JSON". Those queries
now work.
## Root cause and fix
`output` is a native ClickHouse `JSON` column, but `JSONExtract*`,
`JSONHas`, `JSONLength`, and `JSONType` all expect a String containing
JSON text. The compiler already swaps in the column's String companion
(`output_text`) when a JSON column is selected or compared, but not
inside function-call arguments, so it emitted `JSONExtractInt(output,
'x')` against the native column.
The fix prints the companion column for the first argument of these
functions when it resolves to a bare JSON field, keeping the table alias
when qualified (so it works in JOINs):
JSONExtractInt(output, 'x') -> JSONExtractInt(output_text, 'x')
JSONExtractArrayRaw(assumeNotNull(output), 'y') ->
JSONExtractArrayRaw(assumeNotNull(output_text), 'y')
It also reaches through value-preserving passthrough wrappers like
`assumeNotNull(...)`, while leaving value-changing wrappers like
`toJSONString(output)` on the native column (that argument is already a
String). The swap is also semantically correct, not just a type fix:
`output_text` is the unwrapped data JSON that the TRQL `output` model
already represents, so field paths line up.
Covered by printer unit tests and a ClickHouse integration test that
runs the whole family (plus the wrapped and `toJSONString` cases)
against a real native-JSON column. Both new cases fail with the exact
"illegal type: JSON" error without the fix.
## ✅ Checklist
- [x] I have followed every step in the [contributing
guide](https://github.com/triggerdotdev/trigger.dev/blob/main/CONTRIBUTING.md)
- [x] The PR title follows the convention.
- [x] I ran and tested the code works
---
## Testing
- Verified the rule emits exactly five errors for un-awaited throws of
the known
async redirect helpers while ignoring awaited throws, returned promises,
and
synchronous `redirect(...)`.
- Verified `--fix` inserts `await` in async functions and produces a
clean
second lint run.
- Verified synchronous functions remain diagnostic-only so autofix
cannot
introduce invalid syntax.
- Ran `pnpm run format`, `pnpm run lint`,
`pnpm run typecheck --filter webapp`, and `git diff --check`.
---
## Changelog
Adds an Oxlint rule that prevents async redirect helpers from being
thrown
without awaiting their `Response`. Existing violations are fixed, the
autofix
is limited to async functions, and the plugin uses an explicit ESM
extension.
---
## Screenshots
See the test-results comment for CLI evidence.
💯
Link to Devin session:
https://app.devin.ai/sessions/e60ad7610773401da3d3040cf1252337
Requested by: @ericallam
---------
Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: Eric Allam <eric@trigger.dev>
- Combine SDK and dashboard bulk actions docs
- Fix API reference pages for bulk actions
- Fix weird rendering on bulk actions page
## Todo
- [ ] not sure about having the SDK+dashboard combined and under "Using
the dashboard"... need to find the right place
## Summary
The limits page didn't spell out that large payloads offload to object
storage automatically, and its single "512KB" note conflated two
different thresholds. This clarifies the behaviour.
On the way in, the SDK uploads any trigger or batch-item payload over
128KB to object storage before sending, so large triggers and batches
don't hit the request body limit (`trigger` / `triggerAndWait` since
4.5.0, `batchTrigger` / `batchTriggerAndWait` since 4.5.2). On
retrieval, payloads and outputs over 512KB are stored in object storage
and returned as a presigned URL from `runs.retrieve`.
## Summary
Magic link login could appear completely broken: submitting your email
on the login page showed a stale "This email is unauthorized" error
instead of the "we've sent you a magic link" confirmation, even when the
address was fine.
This PR reverts
[#4215](https://github.com/triggerdotdev/trigger.dev/pull/4215) (whose
diagnosis and fix turned out to be wrong) and fixes the actual bug,
which was in how login errors are stored and consumed.
## Root cause
Two session bugs compounded on the login page:
- The `/login` loader read the flashed `auth:error` without committing
the session. A Remix flash is only consumed when the session is
committed after the read, so once any attempt flashed an error (for
example an address rejected on an instance with `WHITELISTED_EMAILS`
set), it stayed in the session cookie and reappeared on every later
`/login` visit, making successful attempts look like failures.
- The `/login/magic` action stored its validation and rate limit errors
with `session.set`, which survives every later read and commit, so those
errors stuck permanently.
[#4215](https://github.com/triggerdotdev/trigger.dev/pull/4215) had
instead diagnosed a server-only module leaking into the client bundle
and crashing navigation. Checking the shipped images' client bundles via
their sourcemaps shows `.server` modules were always stubbed out, so
that change fixed nothing and is reverted here.
## Fix
- `/login` reads the flashed error and commits the session when one was
present, so an error renders once and clears. The `redirectTo` branch
now surfaces the error too instead of leaving it in the cookie.
- `/login/magic` flashes its errors instead of `set`ting them.
Verified end-to-end on a live preview environment: a rejected address
shows the error once and a reload clears it; a valid address lands on
the confirmation screen with the address named; GitHub, Google, and SSO
login paths are untouched by this diff.
## Summary
1 improvement, 2 bug fixes.
## Breaking changes
- Removed support for the end-of-life v3 `trigger dev` CLI. Starting a
dev session with an old v3 CLI now returns an upgrade message instead of
connecting - upgrade to the v4 CLI to continue using `trigger dev`.
([#4198](https://github.com/triggerdotdev/trigger.dev/pull/4198))
## Bug fixes
- Fix TS2742 ("inferred type cannot be named") when exporting a
`chat.agent` from a project with declaration emit: `ChatTaskWirePayload`
and `ChatInputChunk` are now declared in the public
`@trigger.dev/sdk/chat` subpath, so inferred agent types emit portable
declarations and the wire types are directly importable.
([#4218](https://github.com/triggerdotdev/trigger.dev/pull/4218))
## Server changes
These changes affect the self-hosted Docker image and Trigger.dev Cloud:
- Reduce primary database load on the runs page by serving its
empty-state check from ClickHouse instead of Postgres.
([#4202](https://github.com/triggerdotdev/trigger.dev/pull/4202))
- Fixed submitting your email on the login page reloading back to an
empty form instead of showing the magic link confirmation screen.
([#4215](https://github.com/triggerdotdev/trigger.dev/pull/4215))
<details>
<summary>Raw changeset output</summary>
# Releases
## @trigger.dev/build@4.5.3
### Patch Changes
- Updated dependencies:
- `@trigger.dev/core@4.5.3`
## trigger.dev@4.5.3
### Patch Changes
- Updated dependencies:
- `@trigger.dev/build@4.5.3`
- `@trigger.dev/core@4.5.3`
- `@trigger.dev/schema-to-json@4.5.3`
## @trigger.dev/python@4.5.3
### Patch Changes
- Updated dependencies:
- `@trigger.dev/sdk@4.5.3`
- `@trigger.dev/build@4.5.3`
- `@trigger.dev/core@4.5.3`
## @trigger.dev/react-hooks@4.5.3
### Patch Changes
- Updated dependencies:
- `@trigger.dev/core@4.5.3`
## @trigger.dev/redis-worker@4.5.3
### Patch Changes
- Updated dependencies:
- `@trigger.dev/core@4.5.3`
## @trigger.dev/rsc@4.5.3
### Patch Changes
- Updated dependencies:
- `@trigger.dev/core@4.5.3`
## @trigger.dev/schema-to-json@4.5.3
### Patch Changes
- Updated dependencies:
- `@trigger.dev/core@4.5.3`
## @trigger.dev/sdk@4.5.3
### Patch Changes
- Fix TS2742 ("inferred type cannot be named") when exporting a
`chat.agent` from a project with declaration emit: `ChatTaskWirePayload`
and `ChatInputChunk` are now declared in the public
`@trigger.dev/sdk/chat` subpath, so inferred agent types emit portable
declarations and the wire types are directly importable.
([#4218](https://github.com/triggerdotdev/trigger.dev/pull/4218))
- Updated dependencies:
- `@trigger.dev/core@4.5.3`
## @trigger.dev/core@4.5.3
</details>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
## Summary
Exporting a `chat.agent` from a project with `declaration: true` failed
with TS2742: the inferred type of the agent references
`ChatTaskWirePayload`, which was declared in an internal module not
reachable through the package exports map, so tsc could only name it via
a file path into `node_modules` and refused to emit. Consumers had to
hand-mirror the wire type and annotate their export.
## Fix
`ChatTaskWirePayload` and `ChatInputChunk` are now declared in
`@trigger.dev/sdk/chat` (a public subpath) and re-exported type-only
from the internal shared module, so every internal import is unchanged
and the browser/server module split is untouched. Declaration emit for
an inferred agent type now produces a portable specifier:
```ts
export declare const chatAgent: Task<"chat-agent", import("@trigger.dev/sdk/chat").ChatTaskWirePayload<MyUIMessage, MyClientData>, unknown>;
```
As a side effect the wire types are now directly importable, which is
what affected users were reconstructing by hand.
## Verification
Reproduced against the built 4.5.2-equivalent package: a consumer
fixture with declaration emit produced `import("<file
path>/ai-shared.js")` in its declaration (the TS2742 trigger); after the
fix the same fixture emits the public specifier with zero diagnostics. A
regression test now builds that consumer simulation in a temp directory
on every test run: it copies the built package into a fake node_modules
(copied, not symlinked, because tsc only applies exports-map naming to
real node_modules paths), compiles the fixture with the TypeScript API,
and asserts no errors, no relative-path imports, and no internal module
references in the emit.
## Summary
Submitting your email on the login page could reload back to an empty
login form instead of showing the "we've sent you a magic link"
confirmation. The magic link email was still sent, so it looked like
nothing happened.
## Root cause
The `/login/magic` route imported a server-only cookie module
(`magicLinkEmailCookie.server.ts`) whose top-level `env.NODE_ENV` read
got bundled into the route's client JS. On the client `env` is
undefined, so the module threw a `TypeError` at module eval, which
aborted Remix's client-side navigation to the confirmation and
hard-reloaded back to `/login`. It only surfaced in production builds
(local dev auto-logs-in, and local prod builds happen to tree-shake the
module out), which is why it slipped through.
## Fix
The email-link strategy already stores the submitted address in the
session (`auth:email`), so the separate cookie was redundant. Deleted
the cookie module and read the address from the session in the loader.
With the module gone, nothing server-only can leak into the client
bundle regardless of tree-shaking.
Verified the confirmation renders with the email address, the SSO
domain-policy redirect (with the email prefilled) still works, and a
production build no longer bundles the module.
Condense the kept rationale comments (logLevel/warn, last-Owner dedup,
role
overwrite) and drop the obvious function-header comments that just
restated
the code. No behavior change.
## Summary
Excludes the non-published plugins workspace from Changesets release
planning so it cannot drive public package version bumps.
## Verification
Ran `pnpm run changeset:version` with temporary changesets for
`@trigger.dev/plugins` and `@trigger.dev/core`; the ignored workspace
produced no release-driver updates, and the public package changeset
versioned normally.
Directory-sync effects are idempotent and the accounts-webhook worker
retries
the whole event, so a single failed attempt (typically a role assignment
losing a serializable race during a backfill burst) is self-healing
rather
than alert-worthy. Tag those thrown errors with logLevel "warn" so the
worker
logs at warn instead of error, keeping them visible for triage without
paging.
## Summary
The runs page's empty-state check (whether an environment has ever had a
run, which decides between the "getting started" and "no runs match your
filters" states) ran a `findFirst` against the Postgres `TaskRun` table.
This moves it to ClickHouse, the same store the runs list itself reads
from, so the check no longer queries `TaskRun`.
## Design
Only the runs list triggers the check now (via an `includeHasAnyRuns`
flag); the other presenters that reuse `NextRunListPresenter` (API,
schedule detail, waitpoint detail, error group) no longer issue it. When
the list is empty it runs `SELECT 1 FROM task_runs_v2 ... LIMIT 1`
filtered on the full `(organization_id, project_id, environment_id)`
sort-key prefix with a configurable `created_at` lower bound
(`RUN_LIST_HAS_RUNS_LOOKBACK_DAYS`, default 30), so it hits the primary
index and reads minimal granules.
Results are cached in a tiered memory + Redis SWR cache. Only positive
("has runs") results are cached, so an environment with no runs is
always re-checked and its first run shows up immediately.
Follow-up to #4194 (v3 execution app + core-helper removal). The v3
(engine V1) is end-of-lifed and enforced off in prod, so this removes a
self-contained slice of the remaining dead v3 code while **keeping every
user-facing deprecation message** - a user still on v3 must still be
told to upgrade.
## Legacy dev websocket
`app/v3/handleWebsockets.server.ts` backs the `/ws` transport used
**only** by the legacy v3 `trigger dev` CLI (v4 dev uses a different
transport). It's now authenticate-then-close with
`V3_DEV_DEPRECATION_MESSAGE`, so an old CLI is still told what to do -
only the legacy `AuthenticatedSocketConnection` / `DevQueueConsumer`
execution behind it (which can no longer run) is removed.
- Deleted `app/v3/authenticatedSocketConnection.server.ts` (its only
consumer).
- `engineDeprecation.server.ts` and the deprecation message constants
are untouched.
## Docs
Deleted the intentionally-legacy "Docker (legacy)" self-hosting page
(`open-source-self-hosting.mdx`) and redirected
`/open-source-self-hosting` (+ the existing
`/v3/open-source-self-hosting` alias) to `/self-hosting/overview`;
repointed the two inbound links. The current `self-hosting/*` docs
already describe the v4 (single supervisor) setup.
## Deliberately out of scope
Despite the branch name, this PR does **not** touch MarQS or the
socket.io coordinator/provider namespaces. Investigation found MarQS is
entangled with **live v2** queue/metrics/concurrency/project-cleanup
code (`runQueue`, `queueSizeLimits`, `taskRunConcurrencyTracker`,
`EnvironmentQueuePresenter`, `registerProjectMetrics`, `deleteProject`),
so it needs a per-file reviewed pass, not a bulk delete. That remainder
stays on TRI-11883.
refs TRI-11883
## Summary
4 improvements, 5 bug fixes.
## Improvements
- Add SDK and API client helpers for run bulk actions.
([#4105](https://github.com/triggerdotdev/trigger.dev/pull/4105))
- Large batch payloads now offload to object storage instead of riding
inline in the trigger request. `batchTrigger` and `batchTriggerAndWait`
(and the by-id and by-task variants) offload any per-item payload over
128KB before sending, the same way single `trigger` and `triggerAndWait`
already do, so a big batch no longer blows past the API body limit.
([#4165](https://github.com/triggerdotdev/trigger.dev/pull/4165))
- Removed internal helpers that were only used by the end-of-life v3
self-hosted compute providers.
([#4194](https://github.com/triggerdotdev/trigger.dev/pull/4194))
- Add an `onEvent` callback to `TriggerChatTransport` /
`useTriggerChatTransport` that emits typed lifecycle events for sends,
stream connects, first chunk, and turn completion. Send-success metrics,
time-to-first-token, and "sent but never answered" watchdogs become a
few lines of client code.
([#4187](https://github.com/triggerdotdev/trigger.dev/pull/4187))
```ts
onEvent: (event) => {
if (event.type === "message-sent") metrics.timing("chat.send_ms",
event.durationMs);
if (event.type === "first-chunk") metrics.timing("chat.ttft_ms",
event.sinceSendMs ?? 0);
},
```
## Bug fixes
- fix(cli): honor the MCP server's `--dev-only` flag
([#4199](https://github.com/triggerdotdev/trigger.dev/pull/4199))
- Fix chat turns that throw (for example from an `onTurnStart` hook)
leaking their message listener, which lost or duplicated messages sent
during later turns.
([#4176](https://github.com/triggerdotdev/trigger.dev/pull/4176))
- Fix `chat.agent` and `chat.createSession` permanently dropping user
messages when several arrived during a single turn: every buffered
message is now dispatched as its own turn instead of only the first.
([#4176](https://github.com/triggerdotdev/trigger.dev/pull/4176))
- Fix chat continuation runs replaying already-answered messages: turns
delivered while the run was suspended now advance the session.in resume
cursor, so a new run picks up exactly where the previous one left off.
([#4176](https://github.com/triggerdotdev/trigger.dev/pull/4176))
- Fix `chat.createSession` swallowing a message sent shortly after
stopping a turn: the turn's message listener now detaches when the
stream settles, so those messages run as the next turn.
([#4176](https://github.com/triggerdotdev/trigger.dev/pull/4176))
<details>
<summary>Raw changeset output</summary>
# Releases
## @trigger.dev/build@4.5.2
### Patch Changes
- Updated dependencies:
- `@trigger.dev/core@4.5.2`
## trigger.dev@4.5.2
### Patch Changes
- fix(cli): honor the MCP server's `--dev-only` flag
([#4199](https://github.com/triggerdotdev/trigger.dev/pull/4199))
- Updated dependencies:
- `@trigger.dev/core@4.5.2`
- `@trigger.dev/build@4.5.2`
- `@trigger.dev/schema-to-json@4.5.2`
## @trigger.dev/core@4.5.2
### Patch Changes
- Add SDK and API client helpers for run bulk actions.
([#4105](https://github.com/triggerdotdev/trigger.dev/pull/4105))
- Large batch payloads now offload to object storage instead of riding
inline in the trigger request. `batchTrigger` and `batchTriggerAndWait`
(and the by-id and by-task variants) offload any per-item payload over
128KB before sending, the same way single `trigger` and `triggerAndWait`
already do, so a big batch no longer blows past the API body limit.
([#4165](https://github.com/triggerdotdev/trigger.dev/pull/4165))
- Removed internal helpers that were only used by the end-of-life v3
self-hosted compute providers.
([#4194](https://github.com/triggerdotdev/trigger.dev/pull/4194))
## @trigger.dev/python@4.5.2
### Patch Changes
- Updated dependencies:
- `@trigger.dev/core@4.5.2`
- `@trigger.dev/sdk@4.5.2`
- `@trigger.dev/build@4.5.2`
## @trigger.dev/react-hooks@4.5.2
### Patch Changes
- Updated dependencies:
- `@trigger.dev/core@4.5.2`
## @trigger.dev/redis-worker@4.5.2
### Patch Changes
- Updated dependencies:
- `@trigger.dev/core@4.5.2`
## @trigger.dev/rsc@4.5.2
### Patch Changes
- Updated dependencies:
- `@trigger.dev/core@4.5.2`
## @trigger.dev/schema-to-json@4.5.2
### Patch Changes
- Updated dependencies:
- `@trigger.dev/core@4.5.2`
## @trigger.dev/sdk@4.5.2
### Patch Changes
- Add SDK and API client helpers for run bulk actions.
([#4105](https://github.com/triggerdotdev/trigger.dev/pull/4105))
- Fix chat turns that throw (for example from an `onTurnStart` hook)
leaking their message listener, which lost or duplicated messages sent
during later turns.
([#4176](https://github.com/triggerdotdev/trigger.dev/pull/4176))
- Fix `chat.agent` and `chat.createSession` permanently dropping user
messages when several arrived during a single turn: every buffered
message is now dispatched as its own turn instead of only the first.
([#4176](https://github.com/triggerdotdev/trigger.dev/pull/4176))
- Fix chat continuation runs replaying already-answered messages: turns
delivered while the run was suspended now advance the session.in resume
cursor, so a new run picks up exactly where the previous one left off.
([#4176](https://github.com/triggerdotdev/trigger.dev/pull/4176))
- Fix `chat.createSession` swallowing a message sent shortly after
stopping a turn: the turn's message listener now detaches when the
stream settles, so those messages run as the next turn.
([#4176](https://github.com/triggerdotdev/trigger.dev/pull/4176))
- Add an `onEvent` callback to `TriggerChatTransport` /
`useTriggerChatTransport` that emits typed lifecycle events for sends,
stream connects, first chunk, and turn completion. Send-success metrics,
time-to-first-token, and "sent but never answered" watchdogs become a
few lines of client code.
([#4187](https://github.com/triggerdotdev/trigger.dev/pull/4187))
```ts
onEvent: (event) => {
if (event.type === "message-sent") metrics.timing("chat.send_ms",
event.durationMs);
if (event.type === "first-chunk") metrics.timing("chat.ttft_ms",
event.sinceSendMs ?? 0);
},
```
- Large batch payloads now offload to object storage instead of riding
inline in the trigger request. `batchTrigger` and `batchTriggerAndWait`
(and the by-id and by-task variants) offload any per-item payload over
128KB before sending, the same way single `trigger` and `triggerAndWait`
already do, so a big batch no longer blows past the API body limit.
([#4165](https://github.com/triggerdotdev/trigger.dev/pull/4165))
- Updated dependencies:
- `@trigger.dev/core@4.5.2`
## @trigger.dev/plugins@4.5.2
### Patch Changes
- Updated dependencies:
- `@trigger.dev/core@4.5.2`
</details>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
## Summary
UI/layout/copy pass over the org **SSO & Directory Sync** settings page
(formerly "Identity & Access"). No logic, gates, flags, or data flow
changed — server-side auth (`manage:sso`), Enterprise entitlement,
action validation, and data loading are all untouched.
- Renamed the nav item, page title, and meta from "Identity & Access" to
"SSO & Directory Sync".
- Added a reusable `SettingsLayout` component system (container,
section, header, row, block, actions) modeled on `/account/security`,
and refactored the SSO page onto it (section titles, dividers, left
title/subtitle + right action rows).
- Tightened all UI copy: concise, active voice, consistent labels, no
em-dashes.
- `Select` primitive: additive `wrap`, `popoverClassName`, and
`placement` props (all default to prior behavior) so role options show a
bright title with a wrapping description, right-aligned popover, and no
horizontal overflow.
- Removed the external-link arrow icon from buttons that open a modal;
kept it only on genuinely external actions (Contact us, Open in new
tab).
- Polished the admin portal link dialog: smaller description, tighter
spacing, `ClipboardField` with a permanent copy button, removed the
redundant Copy link button, and a provider-aware Open label (e.g. "Open
in WorkOS") derived from the link host with a safe fallback.
### SSO page UI
<img width="3568" height="2550" alt="CleanShot 2026-07-08 at 18 52
11@2x"
src="https://github.com/user-attachments/assets/009d2437-7552-4ff0-a457-64744a9fcd88"
/>
### Login with SSO and normal email test (local)
https://github.com/user-attachments/assets/b33a4ce9-c1fa-45c9-bd3c-077cb6fc9473
## Test plan
- [ ] Non-Enterprise org: SSO page shows the upsell state
- [ ] Enterprise org, non-Owner without `manage:sso`: 403
- [ ] Enterprise Owner: verify domains, configure SSO, connect
directory, JIT/default/group role selects, and enforcement toggle all
work
- [ ] Role select popovers: bright title + wrapping description,
right-aligned, no horizontal scroll
- [ ] Admin portal dialog: copy button works, "Open in WorkOS" opens the
portal in a new tab
---------
Co-authored-by: Cursor <cursoragent@cursor.com>
## What
Adds a new Example projects page: **ClickHouse chat agent** — a
`chat.agent()` that answers questions about your data by writing and
running SQL against ClickHouse Cloud via the official Node.js ClickHouse
client.
The page follows the existing example-project format (overview,
features, GitHub repo card, how-it-works with code excerpts, relevant
code links) and is registered in `docs.json` in alphabetical order.
## Note
The GitHub repo card links to
`triggerdotdev/examples/tree/main/clickhouse-chat-agent`, which lands in
a companion examples PR — merge that one first.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
v3 (engine V1) is end-of-lifed and the v3 clusters are gone, so this
removes the dead v3 execution code from the monorepo. It's the first
pass of TRI-11824 - the webapp v3 code paths are deliberately left
untouched and gated for a follow-up.
## Apps
Deletes the three v3-only execution apps and their build wiring:
- `apps/coordinator`, `apps/kubernetes-provider`, `apps/docker-provider`
- `.github/workflows/publish-worker.yml` - it built only those three;
the v4 worker publish is a separate workflow
- Their references in `.changeset/config.json`, `.cursorignore`,
`CHANGESETS.md`, `CONTRIBUTING.md`, `.server-changes/README.md`
- `pnpm-lock.yaml` regenerated to prune the apps and their app-only
dependencies (`socket.io`, `@kubernetes/client-node`, `p-queue`,
`execa`, `prom-client`, `tinyexec`)
## Core
Removes the helpers in `@trigger.dev/core` that only those apps used -
`ProviderShell`, `SimpleLogger`, the `Exec`/process helpers,
`isExecaChildProcess`, `getTextBody`, and `testDockerCheckpoint`. Each
was verified to have no remaining consumers anywhere in the repo.
Kept the helpers still used elsewhere: `ExponentialBackoff` (warm-start
client), `HttpReply`/`getJsonBody` (serverOnly http server),
`SimpleStructuredLogger` (widely used), and
`ZodNamespace`/`ZodSocketConnection` (still referenced by legacy v3
webapp code, hence the follow-up pass).
The `./v3/apps` and `./v3/serverOnly` export subpaths remain - only dead
members were trimmed from their barrels, so no `package.json` exports
changed.
## Verification
`@trigger.dev/core` builds, and `typecheck` passes for core, supervisor,
cli-v3, run-engine, redis-worker, and webapp.
refs TRI-11824
### Problem
The agent playground chat builds its realtime transport baseURL from
apiOrigin, but points it at a same-origin /resources/... dashboard
route. When API_ORIGIN differs from APP_ORIGIN, the in/append POST goes
cross-origin, fails the CORS preflight, and messages never reach the
agent ("Failed to fetch").
It only reproduces where the two origins differ — not locally, where
both default to localhost:3030.
Fixes#4149.
### Fix
Build the base URL from window.location.origin (falling back to
apiOrigin on SSR), so realtime traffic stays same-origin — the same
approach AgentView.tsx already uses.
### Testing
Typecheck passes. The CORS path only manifests when API_ORIGIN !=
APP_ORIGIN, so verify on test-cloud (can't reproduce locally).
## Summary
Deleting a project triggered an unbounded database cleanup that scanned
the project's entire run history, so deleting a project with many runs
could be very slow. Project deletion is a soft delete again: run data is
retained and the deletion completes quickly.
## Fix
Project deletion ran a cascade hard-delete whose `BulkActionItem` step
filtered through a relation to `TaskRun` scoped by `projectId`. Prisma
compiles that to an `EXISTS`-join over the project's entire `TaskRun`
set (a large, hot table with no `projectId` index), and it ran on every
project deletion unconditionally.
Removing the cascade-cleanup call restores the prior soft-delete
behaviour: queues are removed, the project is marked deleted, and run
data is retained. The cascade-cleanup service (added in
[#4117](https://github.com/triggerdotdev/trigger.dev/pull/4117)) had no
other callers, so it and its test are deleted.
## Summary
Realtime streams (AI-agent token streaming and run streams) now default
to v2 for self-hosters, backed by a bundled [s2-lite](https://s2.dev)
service. Self-hosting previously shipped no S2 configuration, so streams
ran on the Redis-backed v1 path and there were no docs for wiring up v2.
Both the Docker Compose stack and the Helm chart now provision s2-lite
with persistent storage and set the stream env vars out of the box.
## What's included
- **Docker Compose**: a persistent `s2` service (s2-lite), a basin init
spec, and the `REALTIME_STREAMS_S2_*` plus
`REALTIME_STREAMS_DEFAULT_VERSION=v2` env on the webapp. `.env.example`
documents the v1 fallback and hosted-S2 options.
- **Helm**: an `s2` StatefulSet, PVC, Service and ConfigMap (runs as the
non-root image user via `fsGroup`), an `s2` values block, and webapp env
wiring with an existing-secret path for hosted S2.
- **Docs**: the `REALTIME_STREAMS_S2_*` and
`REALTIME_STREAMS_DEFAULT_VERSION` vars in the webapp env reference,
plus a "Realtime streams" section in the Docker and Kubernetes
self-hosting guides.
## Notes
- The OSS code default stays `v1`; v2 becomes the default purely through
the self-hosting artifacts, so non-self-host deployments are unaffected.
Disabling s2, or setting the version back to `v1`, cleanly reverts to
Redis-backed v1.
- With v2 enabled, the bundled s2 service is a required dependency for
streaming: if it is down, streams error while the task itself still
runs. That is the intended trade for the better v2 path.
- You can point at a hosted S2 at s2.dev instead of the bundled server.
New `/billing-limits` page covering the full [billing limits
feature](https://trigger.dev/changelog/billing-limits) : the three limit
options (plan / custom / no limit), billing alerts (% of limit or dollar
thresholds), what happens when the limit is reached, the recovery flow,
the soft-limits caveat, and the billing limit marker on the Usage page.
## Summary
posthog-js sent product analytics to PostHog Cloud directly from the
browser. This points `api_host` at a same-origin `/ph` path that
forwards to PostHog Cloud EU server-side, following PostHog's standard
first-party reverse-proxy setup.
## How it works
A resource route forwards each request server-side, splitting by path:
`/ph/static/*` and `/ph/array/*` go to the asset host, everything else
(analytics events, feature flags) goes to the ingest host. It rewrites
the `Host` header, strips the `/ph` prefix, and streams the response
back. Only PostHog's own cookies are forwarded, so the app session
cookie stays first-party. Upstream hosts default to PostHog Cloud EU,
overridable via `POSTHOG_INGEST_HOST` / `POSTHOG_ASSETS_HOST`.
It also sets `cross_subdomain_cookie` so a single PostHog session is
shared across the marketing site and app.
Verified locally: static assets return 200 from the EU asset host, and
analytics events return 200 through the ingest host.
## Summary
`sendMessage` from `useChat` gives no feedback about whether a message
actually reached the backend, and the `fetch` override is wire-level: it
requires knowing endpoint semantics, cannot attribute requests to
messages, and misses the headStart first-turn POST entirely. This adds a
typed `onEvent` observability callback to `TriggerChatTransport` /
`useTriggerChatTransport` so send-success metrics, time-to-first-token,
and "sent but never answered" watchdogs become a few lines of client
code.
## Example
```ts
const transport = useTriggerChatTransport({
task: "my-chat",
accessToken: ({ chatId }) => mintChatAccessToken(chatId),
onEvent: (event) => {
switch (event.type) {
case "message-sent":
// Durably acknowledged by the session's input stream, not just "request accepted".
metrics.increment("chat.message_sent", { source: event.source });
metrics.timing("chat.send_duration_ms", event.durationMs);
break;
case "message-send-failed":
metrics.increment("chat.message_send_failed", { status: event.status });
break;
case "first-chunk":
metrics.timing("chat.ttft_ms", event.sinceSendMs ?? 0);
break;
case "turn-completed":
metrics.timing("chat.turn_duration_ms", event.sinceSendMs ?? 0);
break;
}
},
});
```
## Design
One callback, one discriminated union (`ChatTransportEvent`):
- `message-sent` / `message-send-failed`: terminal send outcomes with
`messageId`, a `source` discriminator (submit, regenerate, steer,
action, stop, head-start), `durationMs`, `bodyBytes`, the append's
idempotency key (`partId`, also stored on the server-side record), and
error + HTTP status on failure. `message-sent` means the append was
durably acknowledged, after any internal token-refresh retries.
- `stream-connected` (with a `resumed` flag and the cursor it connected
from), `first-chunk` (chunk type plus `sinceSendMs` for
time-to-first-token), `turn-completed` (`sinceSendMs` full-turn latency
and the agent's committed input cursor), and `stream-error` follow the
response side, so a send can be paired with the answer that should
follow it. `messageId` on response events is client-side attribution
from the last turn-producing send on that chat.
Emissions sit at the transport's existing choke points, covering every
send path uniformly (including steering and headStart, which the fetch
override cannot observe). Exceptions thrown by the callback are
swallowed: observability can never break the chat. The React hook keeps
the callback live across renders instead of freezing the first-render
closure.
## Verification
Unit tests drive the transport directly with the `fetch` override as the
network stub (send success/failure per source, stream lifecycle, resumed
flag, field enrichment, callback exceptions swallowed). Verified
end-to-end against a realistic metrics setup in the ai-chat reference
app (counters, send-duration and TTFT histograms, and both watchdogs
built purely on these events): a healthy two-turn chat produces exactly
the expected event sequence and TTFT values; an oversized append records
`message_send_failed` with status 413; and killing the worker after a
durable send fires both `sent_but_no_stream` and `sent_but_unanswered`,
reproducing and detecting the "message disappeared" failure mode that
motivated this feature.
## Summary
The run page could show an AI generation cost well above what the
provider actually charged, most visibly for OpenRouter and Vercel AI
Gateway requests where a heavily cache-read prompt was priced at the
full input rate. When the provider reports an exact per-request cost, we
now use that instead of catalog pricing.
## Fix
Gateway and OpenRouter include the exact per-request cost in
`ai.response.providerMetadata` (`openrouter.usage.cost` /
`gateway.cost`). That figure already reflects the cache-read discount
and the real per-provider rate, which the catalog cannot reconstruct:
cache-read counts do not arrive in `gen_ai.usage.*`, and per-model
catalog prices drift from what the provider billed, in either direction.
So provider-reported cost is now preferred, and the catalog is used only
when no provider cost is present.
Fallback routing is covered by the same change: when OpenRouter routes
to a different model, `gen_ai.response.model` already carries the served
model, so the cost follows the served model and the provider's own
figure makes it exact.
`extractProviderCost` now runs on every AI span, so it gets a cheap
`"cost"` substring guard to skip the JSON parse on reasoning-model spans
whose provider metadata carries large reasoning text and no cost field.
Regression tests cover the cache-discount overcharge, fallback
served-model pricing, gateway cost, and the catalog fallback path.
## Summary
`batchTrigger` and `batchTriggerAndWait` (and the by-id and by-task
variants) now offload any per-item payload over 128KB to object storage
before sending, the same way single `trigger`/`triggerAndWait` already
do since
[#3785](https://github.com/triggerdotdev/trigger.dev/pull/3785). A batch
of large items no longer inflates the request body past the API limit.
## Demo
A live local run: `batchTriggerAndWait` of 5 items × 300KB (1.5MB
total). Each item offloads to object storage, so the receiver run rows
hold a 65-byte `application/store` pointer instead of the 300KB body,
and every item round-trips (received == sent).
<img width="1000" height="494" alt="batch large-payload offload demo"
src="https://github.com/user-attachments/assets/77ae3958-97d6-4b5c-ab25-39b217caefbc"
/>
## Design
Both the array and streaming batch paths funnel through
`executeBatchTwoPhase`, so offloading happens once there: each item is
measured, then offloaded through the existing
`conditionallyExportPacket` when it crosses 128KB, with bounded
concurrency so a big batch doesn't fire an unbounded number of presigned
PUTs.
Because items are offloaded before the request, SDK batches arrive as
small `application/store` references, so the server-side inline offload
during item ingest (parallelised in
[#3777](https://github.com/triggerdotdev/trigger.dev/pull/3777)) mostly
no longer fires for them.
Every trigger and item also carries its pre-offload serialised size as
`options.payloadSize`. The trigger span records that value, so an
offloaded payload shows its real size instead of the size of the small
object-store reference (previously the span measured the reference).
Container runtimes (cri-o / containerd / podman) can't pull
zstd-compressed layers carried in a Docker v2s2 manifest
(`application/vnd.docker.image.rootfs.diff.tar.zstd`). A deploy built
with an outdated CLI can produce exactly that combination - and today
it's promoted to current and then fails every run at image-pull time.
This extends the pre-promotion image check (#4049) to also inspect the
manifest's layer media types. If any layer uses the unpullable zstd/v2s2
media type, the deploy is rejected at finalize with a clear message to
upgrade the CLI and re-deploy, instead of silently shipping a version
that can't start.
The manifest is already returned by the existing ECR `BatchGetImage`
call, so there's no extra registry request for single-arch images.
Parsing is a lenient Zod schema and **fails open** - a manifest we can't
read never blocks a deploy. Manifest lists / OCI indexes (no top-level
`layers[]`) and OCI zstd (`...tar+zstd`, which runtimes support) pass
unaffected.
Also clarifies in the contributor docs that changesets and
`.server-changes/` notes are user-facing and should be written for
users, not maintainers.
refs TRI-11702
<!-- ccr-slack-attribution -->
_Requested by **Matt Aitken** · [Slack
thread](https://triggerdotdev.slack.com/archives/C032WA2S43F/p1783430373189849?thread_ts=1783430373.189849&cid=C032WA2S43F)_
## ✅ Checklist
- [x] The PR title follows the convention.
- [x] I ran and tested the code works (typecheck of the edited files is
clean; see Testing)
---
## Testing
**Before:** the webapp run-replication test shard failed on nearly every
PR because assertions waited a fixed 1s for rows to replicate from
Postgres → ClickHouse and intermittently checked before the row arrived
under CI load.
**After:** those assertions poll (up to 30s, 250ms interval) until the
rows land, so they pass as soon as replication completes and stop
flaking, without slowing the happy path.
These tests are testcontainers-backed (need Docker + Postgres +
ClickHouse), so the full suite is exercised in CI. Locally I confirmed
the edited `runsReplicationService.part1..part8.test.ts` files
type-check with no new errors.
---
## Changelog
**How:** wrapped the ~21 present-row assertions across
`runsReplicationService.part1..part8.test.ts` in `vi.waitFor`, matching
the existing poll pattern in `part9.test.ts`. Left absence assertions
(expecting 0 rows / no spans) on a fixed settle delay since there is
nothing to poll for. Tests only — no production code changed.
Note: this does NOT touch the `subscribe()` startup race in
`internal-packages/replication/src/client.ts` (a riskier, separate
follow-up).
💯
---
_Generated by [Claude
Code](https://claude.ai/code/session_01KtUdSLKrK17eFVuRYXT6uj)_
---------
Co-authored-by: Claude <noreply@anthropic.com>
## Summary
Adds SDK and API support for run bulk actions. You can now create bulk
cancel or replay actions from `@trigger.dev/sdk` using run IDs or the
same filters as `runs.list()`, then retrieve, list, poll, or abort the
action by its `bulk_` handle.
Tests, docs, changesets added.
## Design
The dashboard bulk action service now accepts structured filters instead
of reading directly from a dashboard request, so the dashboard and API
share the same creation path. Replay actions created through the API are
attributed with the existing `api` trigger source, while
dashboard-created actions keep `dashboard`.
The SDK exposes the new surface under `runs.bulk.*`, including
`targetRegion` for replay region overrides and cursor pagination for
listing bulk actions.
## Filters and runIds
Nuance on filters. If `filter` is provided, it MUST have at least one
key. This is to remove the footgun of passing no filter and selecting
all runs.
```typescript
{ action: "cancel", runIds: ["run_1"] } // valid
{ action: "cancel", runIds: [] } // invalid, min(1)
{ action: "cancel", filter: { status: "FAILED" } } // valid
{ action: "cancel", filter: {} } // invalid
{ action: "cancel", filter: {}, runIds: ["run_1"] } // invalid
```
## Summary
Three fixes to the run-ops database split (the Cloud-only mode where
run-lifecycle rows live on a dedicated Postgres). All are inert in the
default single-database deployment.
The main fix: on the batch trigger paths, a parentless batch's item runs
chose their physical store from a fresh per-org mint-flag read at
processing time, so flipping an org's flag mid-batch could land an item
in a different store than its batch, breaking the `TaskRun.batchId`
foreign key (or silently orphaning the item). The other two harden the
split's safety nets: the schema-parity test now actually compares
columns, and the read fan-out gate now signals when it has been silently
disabled.
## Batch item residency
`RunEngineBatchTriggerService` (api.v2) and the BatchQueue item callback
(api.v3) now anchor each item's id mint on the batch's own friendlyId,
mirroring the already-safe `BatchTriggerV3Service`. Residency is a pure
id-shape check, so an item can no longer diverge from its batch across a
mid-batch flag flip. The pre-failed-run fallback is anchored the same
way (it also sets `batchId`), and the shared mint branch is consolidated
into one helper so every mint path stays in lockstep. No new database
queries; single-database mode is unchanged (a cuid-shaped batch
friendlyId yields a cuid item).
## Schema parity test
The parity test previously read only the dedicated schema and matched
model headers with regexes, so it never compared columns and could not
catch a run-subgraph column that diverged between the two physical
schemas. It now parses both schemas and asserts bidirectional
scalar-column parity (type, nullability, array-ness, default) across the
run-subgraph models, and fails on any field line it can't parse. Scoped
to the run-subgraph models so unrelated control-plane edits don't break
it.
## Read fan-out signal
The split read fan-out gate is decided by the object identity of the NEW
vs control-plane clients. It now warns when both run-ops URLs are set
but the NEW client isn't a distinct instance (fan-out silently off), and
a new test exercises the real topology-into-gate wiring so a future
refactor that aliases the clients can't disable fan-out unnoticed.
## Verification
New unit and glue tests cover all three changes; the DB-backed
residency, store-routing, and topology suites pass against real
Postgres; `typecheck` is clean for both packages.
## Summary
Sending a message to a chat whose run had ended could make the message
vanish: the continuation run replayed already-answered messages, never
processed the new one, and a page refresh lost it entirely. Chasing that
report surfaced four composing message-loss bugs in the chat session
runtime; this PR fixes all of them, each with a regression test.
## The fixes
1. **Stale resume cursor.** Records delivered while a run was suspended
(the waitpoint path) advanced the SSE resume counter but not the
committed-consume cursor, so the `session-in-event-id` header stamped on
turn-completes went stale by one record per suspended turn. Continuation
boots seed from that header, which is what made them replay
already-processed messages. `session.in.wait()` now advances both
cursors.
2. **Only the first buffered message dispatched.** Messages arriving
during a turn are consumed into a buffer whose end-of-turn pickup
dispatched only the first entry; the buffer was recreated each turn, so
the rest were discarded, and since consuming a record commits the cursor
the loss was permanent. A continuation boot's replay delivers several
records back-to-back, which put the user's new message at index 1 or
later. The buffer now outlives the turn and drains one message per turn
in both `chat.agent` and `chat.createSession` (whose equivalent buffer
was never read at all).
3. **Post-stop window in `chat.createSession`.** The turn's message
listener stayed attached through the stopped turn's post-stream work, so
a message sent shortly after stopping a turn was consumed into the dead
steering queue and lost. The listener now detaches when the stream
settles, matching the `chat.agent` loop.
4. **Handler leak on errored turns.** A turn that threw outside the
streaming section (for example from an `onTurnStart` hook) leaked its
message listener. Previously that silently lost mid-turn messages; with
the loop-level buffer it would have duplicated them instead. The
subscription handle is now detached by the turn's catch/finally, and
`chat.createSession` defensively detaches its prior turn's listener when
user code exits a turn without `complete()`/`done()`.
## Verification
Reproduced end-to-end with the ai-chat reference project before the fix
(message consumed but never answered, two replayed turns, gone on
refresh) and verified after (single clean turn, survives refresh,
turn-complete cursors strictly advancing). Regression tests in
`packages/trigger-sdk/test/pending-message-drain.test.ts` cover all
four, each verified red against the unfixed behavior. A smoke sweep of
the standard chat scenarios (basic send, multi-turn, suspend/resume,
mid-stream refresh, stop, steering, cancel + continue, and the
`createSession` variant) passes on the final branch state.
## Summary
When the platform database is briefly unreachable while a run is
resuming from a wait, the run no longer fails with
`TASK_EXECUTION_ABORTED`. The worker now retries the resume through the
outage instead of aborting on the first blip.
## Root cause
Resuming a run calls the engine's `continue` worker-action endpoint.
That route caught every error and returned a `422`, which the worker's
HTTP client treats as non-retryable. So a transient Prisma
infrastructure error (for example `P1001` "Can't reach database server")
was flattened into a permanent failure: the worker gave up, force-killed
the run process, and completed it with `TASK_EXECUTION_ABORTED`.
## Fix
- The `continue` route now lets infrastructure errors propagate to the
generic 500 handler (message scrubbed, and retryable by the worker's
HTTP client), the same treatment the trigger path already gives them via
`isInfrastructureError`. Genuine validation errors (snapshot mismatch,
invalid state) still return `422`, so a stale retry stays non-retryable.
Resuming is idempotent server-side (guarded by the snapshot id), so
retrying is safe.
- The worker's `continueRunExecution` calls (both the
runner-to-supervisor and supervisor-to-engine hops) retry with a longer,
jittered backoff so they can ride out an outage lasting tens of seconds,
and the jitter keeps a fleet of resuming runs from stampeding the
database the moment it recovers.
Builds on #3960, which scrubbed the leaked message on these routes but
left the status non-retryable.
No changeset: this is a server-side behaviour fix recorded via
`.server-changes`. The `@trigger.dev/core` edits are internal run-engine
worker plumbing, not a public API change.
Saadi Myftija