github-featured/triggerdotdev--trigger.dev
2
0
Fork 0
Code Issues Pull Requests Deployments Models Agent Notes Packages Projects Releases Wiki Activity
7,290 Commits 300 Branches 4,167 Tags
e0681d23943441bb5f0294868502950d5bf374c5
Commit Graph

7290 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
github-actions[bot] e0681d2394 chore: release v4.5.0-rc.4 (#3788)
🚀 Publish Trigger.dev Docker / units (push) Failing after 3s
Details
🚀 Publish Trigger.dev Docker / typecheck (push) Failing after 19s
Details
🚀 Publish Trigger.dev Docker / publish-webapp (push) Has been skipped
Details
🚀 Publish Trigger.dev Docker / publish-worker (push) Has been skipped
Details
🚀 Publish Trigger.dev Docker / publish-worker-v4 (push) Has been skipped
Details
🧭 Helm Chart Release / lint-and-test (push) Has been cancelled
Details
🧭 Helm Chart Release / release (push) Has been cancelled
Details 
## Summary
1 new feature, 5 improvements.

## Highlights

- Mollifier `mutateSnapshot` now enforces a tag cap: an `append_tags`
patch carrying `maxTags` returns `"limit_exceeded"` (writing nothing)
when the deduped tag count would exceed the limit, so a buffered run
can't accumulate more tags via the tags API than the trigger validator
allows at creation.
([#3756](https://github.com/triggerdotdev/trigger.dev/pull/3756))

## Improvements
- Mollifier buffer extensions: idempotency dedup, an atomic
`mutateSnapshot` API, metadata CAS, claim primitives, and a
`MollifierSnapshot` type. The buffer's Redis client now reconnects with
jittered backoff so a fleet of clients doesn't stampede Redis in
lockstep after a blip.
([#3752](https://github.com/triggerdotdev/trigger.dev/pull/3752))
- Add `onTerminalFailure` callback to `MollifierDrainerOptions` so the
customer's run lands a SYSTEM_FAILURE PG row even when the drainer
exhausts `maxAttempts` on a retryable PG error. Previously,
retryable-error exhaustion called `buffer.fail()` directly, which
atomically marks FAILED + DELs the entry hash with no PG write — silent
data loss when PG was unreachable across the full retry budget. The
callback fires before `buffer.fail()` on any terminal path (`cause:
"non-retryable"` or `"max-attempts-exhausted"`); throwing a retryable
error from the callback causes the drainer to requeue rather than fail.
([#3754](https://github.com/triggerdotdev/trigger.dev/pull/3754))
- Bump `@s2-dev/streamstore` to `0.22.10` to fix a
`TASK_RUN_UNCAUGHT_EXCEPTION` ("Invalid state: Unable to enqueue") when
a `chat.agent` turn is aborted mid-stream.
([#3792](https://github.com/triggerdotdev/trigger.dev/pull/3792))
- Coerce numeric `concurrencyKey` values to string at the API boundary
across `tasks.trigger`, `tasks.batchTrigger`, and the Phase-2 streaming
batch endpoint.
([#3789](https://github.com/triggerdotdev/trigger.dev/pull/3789))
- Add a `tools` option to `chat.agent`. Declaring your tools here
threads them into the SDK's internal `convertToModelMessages`, so each
tool's `toModelOutput` is re-applied when prior-turn history is
re-converted.
([#3790](https://github.com/triggerdotdev/trigger.dev/pull/3790))

<details>
<summary>Raw changeset output</summary>

⚠️⚠️⚠️⚠️⚠️⚠️

`main` is currently in **pre mode** so this branch has prereleases
rather than normal releases. If you want to exit prereleases, run
`changeset pre exit` on `main`.

⚠️⚠️⚠️⚠️⚠️⚠️

# Releases
## @trigger.dev/redis-worker@4.5.0-rc.4

### Minor Changes

- Mollifier buffer extensions: idempotency dedup, an atomic
`mutateSnapshot` API, metadata CAS, claim primitives, and a
`MollifierSnapshot` type. The buffer's Redis client now reconnects with
jittered backoff so a fleet of clients doesn't stampede Redis in
lockstep after a blip.
([#3752](https://github.com/triggerdotdev/trigger.dev/pull/3752))
- Add `onTerminalFailure` callback to `MollifierDrainerOptions` so the
customer's run lands a SYSTEM_FAILURE PG row even when the drainer
exhausts `maxAttempts` on a retryable PG error. Previously,
retryable-error exhaustion called `buffer.fail()` directly, which
atomically marks FAILED + DELs the entry hash with no PG write — silent
data loss when PG was unreachable across the full retry budget. The
callback fires before `buffer.fail()` on any terminal path (`cause:
"non-retryable"` or `"max-attempts-exhausted"`); throwing a retryable
error from the callback causes the drainer to requeue rather than fail.
([#3754](https://github.com/triggerdotdev/trigger.dev/pull/3754))

### Patch Changes

- Pipeline the per-entry `HGETALL` fetches in
`MollifierBuffer.listEntriesForEnv`. The previous serial implementation
issued one Redis round-trip per runId returned by `LRANGE`, which
dominated stale-sweep wall-time at any meaningful backlog (at the
sweep's default maxCount=1000, this is ~1000 RTTs per env per pass).
Behaviour is unchanged — entries are still skipped when the entry hash
has been torn down by a concurrent drainer ack/fail between the LRANGE
and the HGETALL.
([#3752](https://github.com/triggerdotdev/trigger.dev/pull/3752))
- Mollifier `mutateSnapshot` now enforces a tag cap: an `append_tags`
patch carrying `maxTags` returns `"limit_exceeded"` (writing nothing)
when the deduped tag count would exceed the limit, so a buffered run
can't accumulate more tags via the tags API than the trigger validator
allows at creation.
([#3756](https://github.com/triggerdotdev/trigger.dev/pull/3756))
-   Updated dependencies:
    -   `@trigger.dev/core@4.5.0-rc.4`

## @trigger.dev/build@4.5.0-rc.4

### Patch Changes

-   Updated dependencies:
    -   `@trigger.dev/core@4.5.0-rc.4`

## trigger.dev@4.5.0-rc.4

### Patch Changes

- Bump `@s2-dev/streamstore` to `0.22.10` to fix a
`TASK_RUN_UNCAUGHT_EXCEPTION` ("Invalid state: Unable to enqueue") when
a `chat.agent` turn is aborted mid-stream.
([#3792](https://github.com/triggerdotdev/trigger.dev/pull/3792))
-   Updated dependencies:
    -   `@trigger.dev/core@4.5.0-rc.4`
    -   `@trigger.dev/build@4.5.0-rc.4`
    -   `@trigger.dev/schema-to-json@4.5.0-rc.4`

## @trigger.dev/core@4.5.0-rc.4

### Patch Changes

- Coerce numeric `concurrencyKey` values to string at the API boundary
across `tasks.trigger`, `tasks.batchTrigger`, and the Phase-2 streaming
batch endpoint.
([#3789](https://github.com/triggerdotdev/trigger.dev/pull/3789))
- Bump `@s2-dev/streamstore` to `0.22.10` to fix a
`TASK_RUN_UNCAUGHT_EXCEPTION` ("Invalid state: Unable to enqueue") when
a `chat.agent` turn is aborted mid-stream.
([#3792](https://github.com/triggerdotdev/trigger.dev/pull/3792))

## @trigger.dev/plugins@4.5.0-rc.4

### Patch Changes

-   Updated dependencies:
    -   `@trigger.dev/core@4.5.0-rc.4`

## @trigger.dev/python@4.5.0-rc.4

### Patch Changes

-   Updated dependencies:
    -   `@trigger.dev/sdk@4.5.0-rc.4`
    -   `@trigger.dev/core@4.5.0-rc.4`
    -   `@trigger.dev/build@4.5.0-rc.4`

## @trigger.dev/react-hooks@4.5.0-rc.4

### Patch Changes

-   Updated dependencies:
    -   `@trigger.dev/core@4.5.0-rc.4`

## @trigger.dev/rsc@4.5.0-rc.4

### Patch Changes

-   Updated dependencies:
    -   `@trigger.dev/core@4.5.0-rc.4`

## @trigger.dev/schema-to-json@4.5.0-rc.4

### Patch Changes

-   Updated dependencies:
    -   `@trigger.dev/core@4.5.0-rc.4`

## @trigger.dev/sdk@4.5.0-rc.4

### Patch Changes

- Add a `tools` option to `chat.agent`. Declaring your tools here
threads them into the SDK's internal `convertToModelMessages`, so each
tool's `toModelOutput` is re-applied when prior-turn history is
re-converted.
([#3790](https://github.com/triggerdotdev/trigger.dev/pull/3790))

    ```ts
    chat.agent({
      tools: { readFile, search },
      run: async ({ messages, tools, signal }) =>
        streamText({ model, messages, tools, abortSignal: signal }),
    });
    ```

Also exports `InferChatUIMessageFromTools<typeof tools>` to derive the
chat `UIMessage` type (typed tool parts) directly from a tool set.

-   Updated dependencies:
    -   `@trigger.dev/core@4.5.0-rc.4`

</details>

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
helm-v4.5.0-rc.4 v.docker.4.5.0-rc.4 v4.5.0-rc.4 		2026-06-02 09:35:14 +01:00
nicktrn e35f5742d4 fix(webapp): upgrade posthog-node to v5, drop axios + stale override (#3801) 
Follow-up to #3796, which bumped the slack-client axios paths but left
posthog-node's transitive `axios@1.15.1` in place.

`posthog-node` 4.17.1 → 5.35.6. v5 drops the axios dependency entirely
(it's now fetch-based via `@posthog/core`), so posthog's old axios path
disappears. With #3796 already on main (webapp + d3 references on
`@slack/web-api@7.16.0`), nothing else pins the old line, so the
now-dead `axios@>=1.0.0 <1.15.0` override is removed and axios resolves
to a single patched `1.16.1` repo-wide. This closes the remaining axios
advisories.

Compat: the webapp's usage in `telemetry.server.ts` (`new PostHog(key, {
host })`, `.identify`, `.groupIdentify`, `.capture`) is all object-form
API that v5 preserves; `pnpm run typecheck --filter webapp` passes.

Node: posthog-node v5 requires Node `^20.20.0 || >=22.22.0`. We run
20.20.0 in dev (`.nvmrc`), CI, and the published Docker image
(`node:20.20-bullseye-slim`), so we're compliant.
 2026-06-02 10:33:44 +02:00
nicktrn 952139d492 fix(webapp): bump @slack/web-api to 7.16.0 for patched axios (#3796) 
Bumps `@slack/web-api` 7.9.1 → 7.16.0 in the webapp and the two
`references` examples (d3-chat, d3-openai-agents). 7.16.0 depends on
`axios@^1.16.0`, so every slack-client axios path resolves to 1.16.1
instead of 1.15.1.

This clears the slack and references axios paths. `posthog-node`'s
transitive axios still resolves the older line - that's handled in a
follow-up that upgrades posthog-node to v5 (which drops the axios
dependency entirely and lets us retire the now-stale axios override).
The dependabot axios advisories fully close once both land.
 2026-06-02 08:18:28 +01:00
Eric Allam 2dd9f37b40 fix(core,cli): stop chat.agent uncaught exception on mid-stream abort (#3792) 
## Summary

A `chat.agent` turn that gets aborted mid-stream (stop generation, idle
suspend, cancellation) could surface a `TASK_RUN_UNCAUGHT_EXCEPTION` —
`TypeError: Invalid state: Unable to enqueue` — in the dashboard. The
run kept working, but the exceptions were loud and confusing.

## Root cause

The realtime stream writer batches chunks through
`@s2-dev/streamstore`'s `BatchTransform`, which holds them for a short
linger window before flushing. When the turn's abort signal fires while
a record is still buffered, the stream's writable is aborted and the
transform's readable controller is closed — but the pending linger
`setTimeout` still fires and calls `controller.enqueue()` on the dead
controller, throwing from a timer callback where nothing can catch it.

Fixed upstream in `@s2-dev/streamstore@0.22.10`, which wraps the linger
flush in a try/catch that discards the closed-controller error. This
bumps the dependency across core, the CLI, and the webapp, and adds a
regression test against the real `BatchTransform`.

Verified end-to-end: a mid-stream stop that previously failed the run
with `TASK_RUN_UNCAUGHT_EXCEPTION` now leaves the run healthy.
 2026-06-01 17:12:48 +00:00
Eric Allam e9e2ec1cfc fix(sdk): re-apply tool toModelOutput across chat.agent turns (#3790) 
## Summary

`chat.agent` now takes a `tools` option. Until now tools only went to
`streamText` inside `run()`, so the SDK had no tools when it
re-converted the persisted `UIMessage` history at the start of each
turn. Any tool with a `toModelOutput` (raw image bytes into an image
content part, or a sub-agent transcript compressed to a summary) had its
transform applied on turn 1 and skipped from turn 2 onward, so the raw
output got JSON-stringified back into the prompt and the model lost the
transformed view.

Declaring `tools` on the config threads them into that conversion, so
`toModelOutput` runs on every turn. The resolved set is handed back,
typed, on the `run()` payload as `tools`:

```ts
const tools = { searchDocs, renderChart };

export const myChat = chat.agent({
  tools,
  run: async ({ messages, tools, signal }) =>
    streamText({ ...chat.toStreamTextOptions({ tools }), messages, abortSignal: signal }),
});
```

`tools` also accepts a per-turn function for tools that depend on the
user or a feature flag. Only `inputSchema` and `toModelOutput` are read
during conversion, never `execute`. Also exports
`InferChatUIMessageFromTools<typeof tools>` to derive the chat
`UIMessage` type from a tool set. No behavior change for agents that
don't declare `tools`.
 2026-06-01 17:26:27 +01:00
Daniel Sutton e21b68cc5f feat(webapp): dashboard parity for mollifier-buffered runs (#3757) 
## Summary

Dashboard surfaces handle buffered runs by falling back to the mollifier
snapshot:

- Run detail, span detail, streams view (`_app.../runs.\$runParam`,
`resources.../spans.\$spanParam`, `resources.../streams.\$streamKey`).
- Redirect routes (`@.runs.\$runParam`, `runs.\$runParam`,
`projects.v3.\$projectRef.runs.\$runParam`).
- Action routes — cancel / replay / idempotency-reset / debug — under
`resources.taskruns/...` and `resources.../idempotencyKey.reset`.
- Logs download.
- Realtime subscription route + per-run resource
(`realtime.v1.runs.\$runId`, `resources.../realtime.v1.*`).
- `CancelRunDialog` gains an `onCancelSubmitted` callback so submit
isn't raced by the Radix `DialogClose` wrapper.

Stacked on the mutations PR.

## Test plan

- [x] \`pnpm run typecheck --filter webapp\` passes
- [x] \`pnpm run test --filter webapp
test/mollifierRealtimeRunResource.test.ts\` passes
- [x] \`pnpm run test --filter webapp
test/mollifierRealtimeRunResourceBuffer.test.ts\` passes
- [x] \`pnpm run test --filter webapp
test/mollifierRealtimeSubscription.test.ts\` passes
- [x] Manual smoke: trigger a buffered run, open it in the dashboard,
replay/cancel from the UI

---

## Ship-gate follow-up fixes

- **Auto-redirect to root span on direct nav** — loader sets `?span=`
from root span (PG) or buffered snapshot spanId before 302'ing, so
bookmark/share-link/direct-nav doesn't leave the panel collapsed.
- **RunPresenter switches from `findFirstOrThrow` to `findFirst` + typed
`RunNotInPgError`** — kills the per-poll `PrismaClient error` log spam
for buffered runs without changing the route-loader's fallback flow.
- **Span detail panel renders for buffered runs** — `SpanPresenter.call`
now falls back to `findRunByIdWithMollifierFallback` +
`buildSyntheticSpanRun` instead of returning undefined and triggering
the "Event not found" toast loop.
- **Logs download for buffered runs returns a gzipped placeholder line**
— replaces the 404 with a content-encoded line explaining the run is
queued. Same org-membership gate as the PG path.
- **Admin Debug-Run button hidden for buffered runs + SpanRun circular
type alias broken** (squashed) — buttons gate on a new `isBuffered` flag
on the synthetic SpanRun. Required grounding SpanRun in
`SpanPresenter.getRun` to break a circular type alias TS no longer
tolerates once `isBuffered` is a literal field on the shape.
- **Replay action requires user auth + org-membership** (🚩 Devin
finding) — `action` was unauthenticated and the PG `findFirst` had no
org filter, so any caller with a valid `runParam` could replay any run.
Buffered fallback inherited the same gap. Fixed to mirror the cancel
route.

---------

Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-06-01 16:50:31 +01:00
Daniel Sutton e1950778e2 feat(webapp): mollifier API mutations on buffered runs (#3756) 
## Summary

Cancel, replay, reschedule, metadata, tags, and idempotency-key-reset
now succeed against a run that's still in the mollifier buffer.
Mutations are applied to the buffered snapshot via Lua CAS; the drainer
carries the mutation forward when it replays.

Primitives added:

- `mutateWithFallback` — PG-first / buffer-fallback resolver with
bounded-wait safety net for entries that transition mid-mutation.
- `applyMetadataMutation` — buffered metadata PUT mirroring the PG-side
retry loop with CAS atomicity.
- `resolveRunForMutation` — discriminated-union resolver used by route
`findResource` so the route builder's pre-action 404 check sees buffered
runs.

Routes wired (whole files, no GET/POST splits):
- `api.v2.runs.\$runParam.cancel.ts`
- `api.v1.runs.\$runParam.replay.ts`
- `api.v1.runs.\$runParam.reschedule.ts`
- `api.v1.runs.\$runId.metadata.ts`
- `api.v1.runs.\$runId.tags.ts`
- `resetIdempotencyKey.server.ts`

Stacked on the reads PR.

## Test plan

- [x] \`pnpm run typecheck --filter webapp\` passes
- [x] \`pnpm run test --filter webapp
test/mollifierMutateWithFallback.test.ts\` passes
- [x] \`pnpm run test --filter webapp
test/mollifierApplyMetadataMutation.test.ts\` passes
- [x] \`pnpm run test --filter webapp
test/mollifierResolveRunForMutation.test.ts\` passes

---------

Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-06-01 16:24:10 +01:00
Daniel Sutton a1dc3c5ae1 feat(webapp): mollifier API GET read-fallback — synthetic primitives + route wiring (#3755) 
## Summary

Synthesise QUEUED/FAILED responses from the mollifier buffer when a
TaskRun row hasn't landed in Postgres yet. Wires the synthesis into:

- `ApiRetrieveRunPresenter`
- v1 trace GET route
- v1 spans GET route
- attempts route gains a GET loader (fixes pre-existing Remix "no
loader" 400)

The `readFallback` infra itself lives on the trigger PR (consumed by
`IdempotencyKeyConcern`); this PR adds the route-level
synthetic-rendering primitives.

Stacked on the replay PR.

## Test plan

- [x] \`pnpm run typecheck --filter webapp\` passes
- [x] \`pnpm run test --filter webapp
test/mollifierSyntheticRedirectInfo.test.ts\` passes
- [x] \`pnpm run test --filter webapp
test/mollifierSyntheticSpanRun.test.ts\` passes

---------

Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-06-01 14:53:58 +01:00
Daniel Sutton 4745754a7a feat(webapp,run-engine): mollifier drainer replay + stale sweep + cancelled-run engine API (#3754) 
## Summary

The replay side of the mollifier:

- `DrainerHandler`: reads buffered snapshots and replays them through
`engine.trigger` to materialise PG rows.
- `RunEngine.createCancelledRun`: new public method the handler uses to
write CANCELED rows directly from snapshots (bypass queue + waitpoint,
emit `runCancelled`). Tolerates the cjson empty-table tags edge case
found during validation.
- Drainer fairness: org → env rotation so a heavy env doesn't starve
light ones in the same org.
- Stale-entry sweep + telemetry + alertable gauge so a stuck/offline
drainer surfaces in alerts.

Both the drainer and sweep default-off; nothing fires unless flagged on
(`TRIGGER_MOLLIFIER_DRAINER_ENABLED`,
`TRIGGER_MOLLIFIER_STALE_SWEEP_ENABLED`).

Stacked on the trigger-time decisions PR.

## Test plan

- [x] \`pnpm run typecheck --filter webapp\` passes
- [x] \`pnpm run test --filter webapp
test/mollifierDrainerHandler.test.ts\` passes
- [x] \`pnpm run test --filter webapp test/mollifierStaleSweep.test.ts\`
passes
- [x] \`pnpm run test --filter @internal/run-engine
src/engine/tests/createCancelledRun.test.ts\` passes
- [x] \`pnpm run test --filter @trigger.dev/redis-worker
packages/redis-worker/src/mollifier/drainer.test.ts\` passes

---

## Ship-gate follow-up fix

**Drainer writes SYSTEM_FAILURE on max-attempts exhaustion.** Adds an
`onTerminalFailure` callback on `MollifierDrainerOptions` so the
customer's run lands a SYSTEM_FAILURE PG row even when the drainer
exhausts `MAX_ATTEMPTS` on a retryable PG error (previously
`buffer.fail()` was called with no row written → silent data loss). The
callback runs before `buffer.fail()` on every terminal path
(non-retryable AND max-attempts-exhausted), and re-throwing a retryable
error from the callback causes the drainer to requeue rather than fail.

Bumps `@trigger.dev/redis-worker` to a **minor** changeset (additive
option + new exported types). Includes 5 unit tests covering both
terminal causes plus the requeue-on-retryable-callback-failure path and
no-callback back-compat.

---------

Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-06-01 13:20:05 +01:00
Daniel Sutton 20a676d2ed fix(core,webapp): coerce numeric concurrencyKey to string (#3789) 
## Summary

`concurrencyKey` validation accepted only `z.string().optional()` on the
single-trigger and V2/V3 batch endpoints, and the Phase-2 streaming
NDJSON endpoint accepted `z.record(z.unknown()).optional()` for the
entire `options` field. Callers passing `concurrencyKey: someNumericId`
(e.g. `payload.userId`) either failed schema validation on the first two
paths or sailed through on Phase-2 and then failed downstream at
`prisma.taskRun.create` with `Argument concurrencyKey: Expected String
or Null, provided Int`.

The schema now accepts `string | number` for `concurrencyKey` and
stringifies on the way in, across all three paths. The Phase-2 NDJSON
`options` is tightened to reuse the strict
`BatchTriggerTaskItem.options` shape so it validates identically to the
V2/V3 batch endpoints.

A defensive `typeof === "number"` coercion at the `engine.trigger` call
site in `RunEngineTriggerTaskService` covers in-flight Redis-stored
batch items enqueued before the schema fix — those items are rebuilt
from a `Record<string, unknown>` shape that bypasses the new schema and
would otherwise continue failing for up to their TTL.

## Test plan

- [x] `packages/core/src/v3/schemas/batchItemNDJSON.test.ts` — unit
tests covering numeric→string coercion, string passthrough, no-options,
and rejection of non-string/non-number shapes across
`TriggerTaskRequestBody`, `BatchTriggerTaskItem`, and `BatchItemNDJSON`.
- [x] `apps/webapp/test/engine/triggerTask.test.ts` — `containerTest`
simulating the in-flight Redis batch-item shape (numeric
`concurrencyKey` via `Record<string, unknown>`), verifies the run is
created with `concurrencyKey: "51262"`. Without the worker coercion, the
test reproduces the production stack at `prisma.taskRun.create`.
- [x] `pnpm run typecheck --filter webapp` clean.
- [x] `pnpm run build --filter @trigger.dev/core --filter
@trigger.dev/sdk --filter trigger.dev` clean.

Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-06-01 12:34:29 +01:00
Daniel Sutton 577f35eebe feat(webapp): mollifier trigger-time decisions — mollify, claim, read fallback (#3753) 
## Summary

The trigger hot path's mollifier integration:

- `mollifyTrigger`: when the gate trips, write the engine.trigger
snapshot to the buffer and return a synthesised QUEUED response.
Postgres write is deferred to drainer-replay (next PR in the stack).
- Pre-gate idempotency-key claim: same-key triggers serialise through
Redis so a burst lands in PG / buffer exactly once.
- Read-fallback extensions: `findRunByIdWithMollifierFallback` for the
trigger-time idempotency lookup that must see buffered runs.
- Gate bypasses: `debounce`, `oneTimeUseToken`,
`parentTaskRunId`/`triggerAndWait` skip the mollify path entirely.
- `triggerTask` + `IdempotencyKeyConcern` wired to the above.

All behaviour gated by the master `TRIGGER_MOLLIFIER_ENABLED` switch;
off-state hot path is unchanged (the gate is not even consulted).

Stacked on the buffer extensions PR.

## Test plan

- [x] \`pnpm run typecheck --filter webapp\` passes
- [x] \`pnpm run test --filter webapp test/mollifierMollify.test.ts\`
passes
- [x] \`pnpm run test --filter webapp
test/mollifierIdempotencyClaim.test.ts\` passes
- [x] \`pnpm run test --filter webapp
test/mollifierReadFallback.test.ts\` passes
- [x] \`pnpm run test --filter webapp test/mollifierGate.test.ts\`
passes
- [x] \`pnpm run test --filter webapp test/engine/triggerTask.test.ts\`
passes

---

## Ship-gate follow-up fixes

- **Batch items bypass the mollifier gate** — fixes
`BatchTaskRunItem_taskRunId_fkey` FK violation on batch triggers when
the gate trips. End-state is a drainer-side `BatchTaskRunItem`
create-on-materialise; batch traffic passes through the gate until that
lands.
- **IdempotencyKeyConcern honours buffered-run TTL on expiry** —
buffered path now clears expired idempotency claims (read-side) and
resets the buffer's `mollifier:idempotency:*` SETNX binding (write-side)
so a re-trigger past the customer's TTL lands as a fresh run instead of
echoing the stale buffered runId.

---------

Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-06-01 12:00:24 +01:00
Daniel Sutton 3924aa4adb feat(redis-worker,webapp): mollifier buffer extensions + snapshot type (#3752) 
## Summary

Buffer-side data layer used by the rest of the mollifier phase-3 stack.

- `buffer.ts` gains entry inspection (`getEntry`), idempotency lookup
(`lookupIdempotency`), in-place snapshot mutation (`mutateSnapshot`),
and dwell tracking. All atomic via Lua.
- `mollifierSnapshot.server.ts`: shared `MollifierSnapshot` type plus
(de)serialise helpers.
- Drops the entry-TTL config and its env var. The drainer is the
recovery mechanism; an entry that survives the drainer should surface as
a stale-sweep alert, not silently TTL away.

Adds methods to the buffer interface; nothing consumes them yet.
Subsequent PRs in the stack wire trigger-time mollify, read-fallback,
and mutation paths against this surface.

## Test plan

- [x] \`pnpm run typecheck --filter webapp\` passes
- [x] \`pnpm run test --filter @trigger.dev/redis-worker
packages/redis-worker/src/mollifier/buffer.test.ts\` passes

---------

Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-06-01 11:42:55 +01:00
Saadi Myftija 181d9ba541 feat: preview environment dispatch workflow (#3786) 
Publishes a `repository_dispatch` `preview-deploy` event that enables
preview deployments. Opt-in only currently.
 2026-06-01 11:57:40 +02:00
github-actions[bot] 8f066ac389 chore: release v4.5.0-rc.3 (#3763)
🚀 Publish Trigger.dev Docker / typecheck (push) Failing after 0s
Details
🚀 Publish Trigger.dev Docker / publish-webapp (push) Has been skipped
Details
🚀 Publish Trigger.dev Docker / publish-worker (push) Has been skipped
Details
🚀 Publish Trigger.dev Docker / publish-worker-v4 (push) Has been skipped
Details
🚀 Publish Trigger.dev Docker / units (push) Failing after 0s
Details
🧭 Helm Chart Release / lint-and-test (push) Has been cancelled
Details
🧭 Helm Chart Release / release (push) Has been cancelled
Details 
## Summary
1 improvement, 2 bug fixes.

## Improvements
- Retry `TASK_MIDDLEWARE_ERROR` under the task's retry policy instead of
failing the run on the first attempt. The error was already classified
as retryable by `shouldRetryError`, but `shouldLookupRetrySettings` did
not include it, so the retry flow fell through to `fail_run`. Fixes
#3231. ([#3676](https://github.com/triggerdotdev/trigger.dev/pull/3676))

## Bug fixes
- Fix `TypeError` in `unflattenAttributes` when the input attribute map
contains conflicting dotted key paths (e.g. both `a.b` set to a scalar
and `a.b.c` set to a value). The path-walk loop now applies
last-write-wins when a prior key wrote a primitive, null, or array at an
intermediate slot, matching the existing precedent in
`AttributeFlattener.addAttribute`. Callers no longer crash when handed
malformed external attribute inputs.
([#3762](https://github.com/triggerdotdev/trigger.dev/pull/3762))
- Fix external trace context leaking across runs on warm-started workers
with `processKeepAlive` enabled. Every subsequent run's attempt span was
being exported with the first run's `traceId` and `parentSpanId`,
breaking causal-chain navigation in external APM tools. Runs without an
external trace context are unaffected.
([#3768](https://github.com/triggerdotdev/trigger.dev/pull/3768))

<details>
<summary>Raw changeset output</summary>

⚠️⚠️⚠️⚠️⚠️⚠️

`main` is currently in **pre mode** so this branch has prereleases
rather than normal releases. If you want to exit prereleases, run
`changeset pre exit` on `main`.

⚠️⚠️⚠️⚠️⚠️⚠️

# Releases
## @trigger.dev/build@4.5.0-rc.3

### Patch Changes

-   Updated dependencies:
    -   `@trigger.dev/core@4.5.0-rc.3`

## trigger.dev@4.5.0-rc.3

### Patch Changes

-   Updated dependencies:
    -   `@trigger.dev/core@4.5.0-rc.3`
    -   `@trigger.dev/build@4.5.0-rc.3`
    -   `@trigger.dev/schema-to-json@4.5.0-rc.3`

## @trigger.dev/core@4.5.0-rc.3

### Patch Changes

- Retry `TASK_MIDDLEWARE_ERROR` under the task's retry policy instead of
failing the run on the first attempt. The error was already classified
as retryable by `shouldRetryError`, but `shouldLookupRetrySettings` did
not include it, so the retry flow fell through to `fail_run`. Fixes
#3231. ([#3676](https://github.com/triggerdotdev/trigger.dev/pull/3676))
- Fix `TypeError` in `unflattenAttributes` when the input attribute map
contains conflicting dotted key paths (e.g. both `a.b` set to a scalar
and `a.b.c` set to a value). The path-walk loop now applies
last-write-wins when a prior key wrote a primitive, null, or array at an
intermediate slot, matching the existing precedent in
`AttributeFlattener.addAttribute`. Callers no longer crash when handed
malformed external attribute inputs.
([#3762](https://github.com/triggerdotdev/trigger.dev/pull/3762))
- Fix external trace context leaking across runs on warm-started workers
with `processKeepAlive` enabled. Every subsequent run's attempt span was
being exported with the first run's `traceId` and `parentSpanId`,
breaking causal-chain navigation in external APM tools. Runs without an
external trace context are unaffected.
([#3768](https://github.com/triggerdotdev/trigger.dev/pull/3768))

## @trigger.dev/plugins@4.5.0-rc.3

### Patch Changes

-   Updated dependencies:
    -   `@trigger.dev/core@4.5.0-rc.3`

## @trigger.dev/python@4.5.0-rc.3

### Patch Changes

-   Updated dependencies:
    -   `@trigger.dev/core@4.5.0-rc.3`
    -   `@trigger.dev/build@4.5.0-rc.3`
    -   `@trigger.dev/sdk@4.5.0-rc.3`

## @trigger.dev/react-hooks@4.5.0-rc.3

### Patch Changes

-   Updated dependencies:
    -   `@trigger.dev/core@4.5.0-rc.3`

## @trigger.dev/redis-worker@4.5.0-rc.3

### Patch Changes

-   Updated dependencies:
    -   `@trigger.dev/core@4.5.0-rc.3`

## @trigger.dev/rsc@4.5.0-rc.3

### Patch Changes

-   Updated dependencies:
    -   `@trigger.dev/core@4.5.0-rc.3`

## @trigger.dev/schema-to-json@4.5.0-rc.3

### Patch Changes

-   Updated dependencies:
    -   `@trigger.dev/core@4.5.0-rc.3`

## @trigger.dev/sdk@4.5.0-rc.3

### Patch Changes

-   Updated dependencies:
    -   `@trigger.dev/core@4.5.0-rc.3`

</details>

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
helm-v4.5.0-rc.3 v.docker.4.5.0-rc.3 v4.5.0-rc.3 		2026-06-01 10:34:45 +01:00
Eric Allam 9211032733 chore(database): drop unused TaskRun status composite index (#3743) 
## Summary

Drops the `TaskRun_status_runtimeEnvironmentId_createdAt_id_idx` index
from the `TaskRun` table. After #3742 gated the legacy
`WAITING_FOR_DEPLOY` drain to V1-engine workers only, this index sees
zero scans on both writer and reader replicas. Removing it cuts index
maintenance on every `TaskRun` INSERT/UPDATE.

## Why

The index existed to support `WHERE status = X AND runtimeEnvironmentId
= Y` queries from `ExecuteTasksWaitingForDeployService`, which is
V1-only and no longer triggered on V2 deployments. A code grep across
`apps/webapp` and `internal-packages/run-engine` confirmed no V2
production query uses this access pattern — every other `status:` filter
on `TaskRun` is paired with `id`/`friendlyId`/`parentSpanId` and uses a
different index.

Dropping it also unlocks HOT updates on the dequeue path. The dequeue
`UPDATE` modifies `status` (`QUEUED` -> `DEQUEUED`), and `status` is the
leading column of this index — its presence blocked HOT eligibility for
every `TaskRun` UPDATE. With the index gone, dequeue UPDATEs can become
HOT, reducing WAL bytes and removing the B-tree page contention on this
index's right-edge leaves.

Uses `DROP INDEX CONCURRENTLY` to avoid blocking writes during the drop.

## Sequencing

Should only ship once #3742 has soaked long enough to confirm the index
is genuinely cold (24h+ of zero scans on `pg_stat_user_indexes`).
 2026-06-01 09:37:37 +01:00
Saadi Myftija 9cb6fd17c3 fix(webapp): idempotent DeploymentBackgroundWorker creation (#3772) 
Make `POST /api/v1/deployments/:deploymentId/background-workers`
idempotent so client-side retries no longer collide on the
`BackgroundWorker` `(project, env, version)` unique index. Helps make
deployments more resilient against the class of indexing failures that
surfaces in the dashboard as "Indexing timed out", e.g. during transient
database issues.
 2026-05-29 10:03:55 +02:00
nicktrn c043c4a6ad fix(core): external trace context leaks across warm-started runs (#3768) 
On warm-started workers with `processKeepAlive` enabled, every run's
attempt span was exported with the first run's `traceId` and
`parentSpanId`. `ExternalSpanExporterWrapper` and
`ExternalLogRecordExporterWrapper` captured `externalTraceContext` at
`TracingSDK` construction, and the SDK is memoized for the worker's
lifetime - so the per-run reset of
`StandardTraceContextManager.traceContext` never reached the wrappers.

Reported by a customer running v4.4.x: 33 distinct runs on the same
host/pid showed up in their APM as siblings of one parent span.

Fix: the wrappers now read external context live from the trace context
manager per export. Runs without an external trace context fall through
to the unchanged `externalTraceId` fallback - no behaviour change for
them.

Regression test in
`packages/core/test/externalSpanExporterWrapper.test.ts` asserts that
reassigning the manager between exports produces correctly-parented
spans.
 2026-05-28 10:43:39 +00:00
Matt Aitken 816986d44e fix(webapp): treat Phase 2 batch-stream retries as idempotent (TRI-9944) (#3766) 
Returns sealed:true when a fast-completing batchTrigger races
the stream finalisation, instead of throwing 422/BatchTriggerError.
 2026-05-28 09:28:18 +00:00
Iss 5083d161b2 docs: adds relevant env vars to self hosting docs (#3148) 2026-05-27 12:25:18 -04:00
Iss 8399aa2052 fix(core): retry TASK_MIDDLEWARE_ERROR under the task's retry policy (#3676) 
Retries `TASK_MIDDLEWARE_ERROR` under the task's retry policy.
`shouldRetryError` already classed it as retryable, but
`shouldLookupRetrySettings` did not, so the run fell through to
`fail_run` on attempt 1 instead of using the task's `retry` config.
Fixes #3231.
 2026-05-27 17:17:50 +01:00
Iss 7324a3307f chore(webapp): rename support tier to "Priority" on plan cards and li… (#3692) 2026-05-27 12:17:40 -04:00
Iss df96a937db docs: troubleshoot "Stream is being deleted" during long waits (#3704) 2026-05-27 12:17:21 -04:00
Matt Aitken 90bbbd1c44 fix(webapp): recover from ClickHouse JSON parse failures on out-of-range integers (#3759) 
## Summary

Second class of poisoned-row failure in the runs replication path. PR
#3708 plugged lone UTF-16 surrogates; this one handles bare JSON integer
literals outside ClickHouse's `Int64`..`UInt64` range. Recovery stays
purely reactive — the existing `sanitizeRows` walker just gains an extra
branch, so the hot replication path pays nothing on healthy rows.

Fixes the still-firing customer-facing symptom from
[TRI-9755](https://linear.app/triggerdotdev/issue/TRI-9755):
`scan-social-profiles` runs continued to be stranded in `EXECUTING` on
the Tasks page after #3708 deployed. CloudWatch showed `Dropped batch —
ClickHouse JSON parse error but sanitizer found nothing to fix` firing
**8/8 times** since the previous deploy (zero successful sanitizations).
Root cause: upstream JS Number precision loss on a 21-digit Google Plus
ID (`117039831458782873093` → `117039831458782870000`) — the
precision-lossy value still serialises as a bare integer that exceeds
`UInt64.MAX`, which ClickHouse rejects with `INCORRECT_DATA`.

## How the bug ships

The customer task emits an output containing a Poshmark profile's
`spec_format`:

```json
{"key":"gp_id","proper_key":"Gp Id","value":117039831458782870000,"type":"int"}
```

That value is `1.17e20` — comfortably above `UInt64.MAX` (`1.84e19`) but
comfortably below `1e21`. `Number.prototype.toString` only switches to
exponential form at `|value| >= 1e21`, so `JSON.stringify` emits the
bare token `117039831458782870000` and the ClickHouse
`JSON(max_dynamic_paths)` column fails with:

```
Code: 117. DB::Exception: Cannot parse JSON object here: {…}: (while reading the value of key output): (at row 1)
: While executing ParallelParsingBlockInputFormat. (INCORRECT_DATA) (version 25.12.x)
```

Same error verbatim as prod. The same number quoted
(`"117039831458782870000"`) inserts fine — ClickHouse's dynamic JSON
column accepts a `String` subtype on the same path.

## What changed

`apps/webapp/app/v3/eventRepository/sanitizeRowsOnParseError.server.ts`:

- New private `isUnsafeJsonInteger(value)` helper — true iff `value` is
a finite integer-valued JS Number where `|value| < 1e21` (so
`JSON.stringify` emits integer form, not exponent) **and** `value` falls
outside `[Int64.MIN, UInt64.MAX]`.
- `sanitizeUnknownInPlace` gains a number-branch: when the predicate
holds, replace the Number with `String(value)`. The downstream JSON
column dynamic-types the path as String for that row — fine, since the
value was already precision-lossy upstream (no JS Number above 2^53 is
numerically meaningful anyway).
- Float-valued numbers, large floats (>= 1e21), NaN and Infinity are
left alone — `JSON.stringify` emits them with exponents or as `null`,
both of which ClickHouse accepts.

`apps/webapp/test/sanitizeRowsOnParseError.test.ts`: four new unit tests
+ an extension to `sanitizeRows` covering surrogate + integer fixes
counted together across rows. The unit suite now covers:

- Positive value above `UInt64.MAX` (`117039831458782870000` — the
actual prod value)
- Negative value below `Int64.MIN`
- Boundary values pass through (`42`, `Number.MAX_SAFE_INTEGER`, `2^63`)
- Non-integer numbers untouched (floats, `1e25`, NaN, Infinity)
- The actual `scan-social-profiles` nested shape — finds the offending
`gp_id` deep inside
`output.data.profiles[].spec_format[].platform_variables[].value`

`.server-changes/runs-replication-bigint-recovery.md` — release notes
entry.

## Why reactive, not pre-flight

`#prepareJson` runs millions of times per day on the replication hot
path. Walking every JSON tree to look for oversized integers would add
bounded-but-real CPU on every healthy row. `sanitizeRows` only fires
after a ClickHouse parse-error rejection, which is a few times a day
platform-wide. Extending it costs effectively zero on healthy traffic
and gains us recovery on the rare poisoned row.

## Verification

- Reproduced 1:1 in a throwaway Docker
`clickhouse/clickhouse-server:25.12.11.4` (closest available to the prod
`25.12.1.1579` build). Pre-sanitize JSON fails with the exact prod
error; post-sanitize JSON inserts cleanly and the row is readable with
`gp_id` stored as a String subtype.
- `pnpm --filter webapp exec vitest run
test/sanitizeRowsOnParseError.test.ts` — 22/22 passing (18 existing + 4
new).
- `pnpm run typecheck --filter webapp` — clean.

## Test plan

- [x] `pnpm run typecheck --filter webapp`
- [x] Unit tests pass against new + existing cases
- [x] End-to-end Docker ClickHouse repro confirms recovery
- [ ] Post-deploy: confirm `Sanitizing batch after ClickHouse JSON parse
error` warns fire instead of `Dropped batch …` errors when
`scan-social-profiles` outputs trip CH again
- [ ] Post-deploy: confirm `permanentlyDroppedBatches` counter stops
climbing in
`/stp/trigger-app-prod/ecs/replication/service-container/process-logs`

## What this does NOT do

- Doesn't backfill the ~120k+ existing stranded `EXECUTING` rows in
production. Same as #3708 — that needs a reconciliation/backfill sweep
(separate ticket — TRI-9755 fix #3).
- Doesn't address the upstream root cause (the customer task emitting a
JS-Number-precision-lossy big int). That's a customer-task concern; our
replication path needs to be robust to whatever shape arrives.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
 2026-05-27 09:00:53 -07:00
Oskar Otwinowski 5ba1b32d53 fix(core): handle conflicting dotted paths in unflattenAttributes (#3762) 2026-05-27 16:18:50 +02:00
Eric Allam 37eeaa3690 fix(webapp): skip V1 WAITING_FOR_DEPLOY drain on V2 promotes (#3742) 
## Summary

Stops the legacy V1 `WAITING_FOR_DEPLOY` drain from running on every V2
deployment promotion. The drain queries `TaskRun` by
`status='WAITING_FOR_DEPLOY'`, which only V1-engine runs ever have — V2
runs use `PENDING_VERSION` and are handled out of band. Calling the
drain on V2 promotes produced empty queries against the status index and
unnecessary reader-DB load.

## Fix

Two layers:

1. Gate the enqueue at the call site in `ChangeCurrentDeploymentService`
so it only fires when the deployment's worker is on engine V1.
2. Add a `LEGACY_RUN_ENGINE_WAITING_FOR_DEPLOY_DISABLED` env var
(default `0`). When set to `1`, the service returns immediately from
`call()` — neuters any jobs already sitting in the worker queue from
before the deploy lands.

V1 customers see no change; V2 promotes no longer trigger the drain.
 2026-05-25 07:10:13 +01:00
Matt Aitken 596a9bb90c fix(webapp): retain sessions-replication singleton import via globalThis assignment (#3738) 
Linear: [TRI-9864](https://linear.app/triggerdotdev/issue/TRI-9864)
(Urgent)
Production incident:
[TRI-9863](https://linear.app/triggerdotdev/issue/TRI-9863) (mitigated
by image revert in cloud#910)

## Bug

`apps/webapp/package.json` declares `"sideEffects": false`. PR #3333
(`71d98b4e`) replaced the previous real method-call retention idiom at
the two `sessionsReplicationInstance` import sites with:

```ts
import { sessionsReplicationInstance } from "...";
void sessionsReplicationInstance;
```

esbuild treats `void <identifier>;` as a pure expression statement under
`sideEffects: false` and **tree-shakes the entire import** — including
the `singleton(...)` call inside `sessionsReplicationInstance.server.ts`
which is the only thing that fires
`initializeSessionsReplicationInstance()`. The sessions→ClickHouse
logical replication worker never starts, the slot is unconsumed, lag
grows.

### How it manifested in production

cloud#907's image bump rolled the `SessionReplicationService` ECS task
on prod at 14:32 UTC. The new container's startup log emitted `🗃️ Runs
replication service enabled` but **not** `🗃️ Sessions replication
service enabled` or `🗃️ Sessions replication service started`.
CloudWatch `OldestReplicationSlotLag` grew at ~220 MB/min and the `High
replication lag` alarm fired at 14:37 UTC. Prod was reverted to the
previous image (cloud#910) to stop the bleed.

### Verification

`grep` of the built bundle `apps/webapp/build/index.js` (built from
`c0365d36`):

- **3** occurrences of `Runs replication` / `runsReplicationInstance`
strings ✅
- **0** occurrences of `Sessions replication` /
`sessionsReplicationInstance` / `SessionsReplicationService` ❌

The runs path survives tree-shaking because `adminWorker.server.ts` and
`admin.api.v1.runs-replication.*` routes have real method calls
(`.start()`, `.teardown()`, `.backfill()`) — observable uses the
tree-shaker must preserve. The sessions singleton has no real callers,
only the `void` no-ops, hence its complete elimination from the bundle.

## Fix

Replace `void sessionsReplicationInstance;` with an assignment to
`globalThis`, an unambiguous observable side effect the bundler cannot
eliminate:

```ts
(globalThis as Record<string, unknown>).__sessionsReplicationInstance =
  sessionsReplicationInstance;
```

Applied at both call sites: `apps/webapp/app/entry.server.tsx` and
`apps/webapp/app/v3/services/adminWorker.server.ts`.

Surrounding comments updated to document the bundler interaction so the
next maintainer doesn't reintroduce `void`.

## Out of scope (follow-ups)

- **Robustness improvement**: change `apps/webapp/package.json` from
`"sideEffects": false` to an allowlist that includes
`*Instance.server.ts` files. Prevents the same regression shape via any
future `*Instance` singleton.
- **Build-time check**: add a `grep` post-build step in `publish.yml`
requiring `"Sessions replication"` to appear in
`apps/webapp/build/index.js`. Catches this exact regression at CI time.

## Test plan

- [x] `pnpm run typecheck --filter webapp` clean
- [ ] After merge + publish: confirm new image's
`SessionReplicationService` container logs `🗃️ Sessions replication
service enabled` and `🗃️ Sessions replication service started` at
startup
- [ ] After re-deploying to prod: confirm `OldestReplicationSlotLag`
stops growing and drains
 2026-05-24 16:21:15 +01:00
Matt Aitken eefb96c87d fix(webapp): recover from ClickHouse JSON parse failures in runs replication (#3708) 
## Summary

On a ClickHouse `Cannot parse JSON object` rejection,
`RunsReplicationService` now sanitizes lone UTF-16 surrogates across the
failing batch via the existing `sanitizeRows` helper and retries once.
If the sanitizer found nothing or the retry also fails, the batch is
dropped loudly with a counter increment, so the surrounding
`#insertWithRetry` layer doesn't spin three more times on a
deterministic failure. Non-parse errors propagate unchanged.

Mirrors the pattern from #3659 (for `ClickhouseEventRepository`) — same
root cause (lone UTF-16 surrogates in user-provided JSON), same recovery
shape, **reusing the same shared helpers** (`sanitizeRows`,
`isClickHouseJsonParseError`, `parseRowNumberFromError`).

Fixes the customer-facing symptom from
[TRI-9755](https://linear.app/triggerdotdev/issue/TRI-9755): a single
row's poisoned `output` JSON used to take down the
`COMPLETED_SUCCESSFULLY` UPDATE events for its 50+ batch-mates,
stranding them in `EXECUTING` in ClickHouse forever and inflating
"Running" counts on the Tasks page. Confirmed in production this is
ongoing — ~120k stale rows accumulated in a single 5-hour burst on
2026-05-18; smaller continuous leak before and after.

## What changed

`apps/webapp/app/services/runsReplicationService.server.ts`:

- Imports the three helpers from
`~/v3/eventRepository/sanitizeRowsOnParseError.server` (no duplication;
no move).
- New private `#insertWithJsonParseRecovery<T>(rows, doInsert,
contextLabel, attempt)` — generic over `TaskRunInsertArray[]` and
`PayloadInsertArray[]`, structurally identical to
`ClickhouseEventRepository.#insertWithJsonParseRecovery`. Try → on parse
error sanitize the whole batch (the `at row N` hint is logged but not
used to slice — semantics under `input_format_parallel_parsing` aren't
stable) → retry once → drop with loud log if sanitizer found nothing OR
retry still fails.
- `#insertTaskRunInserts` and `#insertPayloadInserts` extract a
`doInsert` closure and hand it to the wrapper. Existing error logging,
span recording, and `recordSpanError` are preserved inside the closure.
- New `private _permanentlyDroppedBatches = 0` counter with a public
getter, for ops dashboards and tests (matches the events-repo
convention). One shared counter for both insert sites — granularity
comes from the `contextLabel` (`task_runs_v2` /
`raw_task_runs_payload_v1`) on every log line.

`.server-changes/runs-replication-utf16-recovery.md` — release notes
entry.

## Why no new tests

The shared helpers already have full unit + real-ClickHouse contract
coverage from #3659
(`apps/webapp/test/sanitizeRowsOnParseError.test.ts`,
`apps/webapp/test/otlpUtf16Sanitization.integration.test.ts`). The new
wrapper is a line-for-line structural port. Adding a parallel
integration test would require synthesizing bad data that *escapes* the
preemptive `detectBadJsonStrings` check in `#prepareJson` but still
trips ClickHouse — non-trivial without hand-crafted fixtures and
wouldn't cover any new logic.

## What this does NOT do

- Doesn't touch the ~120k existing stale `EXECUTING` rows in production.
That needs a reconciliation/backfill sweep (separate ticket — TRI-9755
fix #3).
- Doesn't sanitize the `error` column path
(`runsReplicationService.server.ts:932 const errorData = { data:
run.error };`). Reactive recovery will catch it if it ever poisons a
batch, but feeding it through `#prepareJson` like `output` is a cheap
follow-up.

## Test plan

- [x] `pnpm run typecheck --filter webapp` — clean
- [ ] Post-deploy: confirm `permanentlyDroppedBatches` counter stays at
zero (or near-zero) in
`/stp/trigger-app-prod/ecs/replication/service-container/process-logs`,
and watch for `Sanitizing batch after ClickHouse JSON parse error` warns
to confirm recovery is firing on real traffic
- [ ] Post-deploy: confirm the rate of new
"EXECUTING-but-actually-COMPLETED" zombies in ClickHouse flattens
(current rate ≈ tens-to-hundreds per hour platform-wide)

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
 2026-05-23 20:20:09 +01:00
Eric Allam 9f64bf404b docs(ai-chat): slim-wire HITL continuations + field-level merge contract (#3721) 
## Summary

Updates the AI chat docs to match the slim-wire + field-level merge
behavior shipped in #3719 and the precise `.in/append` cap +
CORS-readable 413 shipped in #3720. No behavior changes here — code is
correct in `main`; the docs were lagging on three patterns customers
copy out of the page.

## What changed

- **`hydrateMessages` examples upsert by id** (in `lifecycle-hooks.mdx`,
`patterns/database-persistence.mdx`, and
`patterns/persistence-and-replay.mdx`). The previous
`stored.push(newMsg)` pattern duplicated the assistant id on HITL
continuations and caused the LLM to receive a tool call with no
`arguments`. The new examples include the rationale inline.
- **`onValidateMessages` example filters to user messages**
(`lifecycle-hooks.mdx`). The previous example called
`validateUIMessages({ messages, tools })` directly, which now throws on
HITL slim wires (the AI SDK schema requires `input` on resolved tool
parts). New example shows the filter pattern, with a Warning callout
explaining why.
- **Merge contract description updated** (`lifecycle-hooks.mdx`). The
old wording said incoming messages are "auto-merged" / "replaced"; the
new description explains the actual field-level overlay (state advances
only).
- **Approval-responded wire example slimmed** (`client-protocol.mdx`).
Shows the minimum shape the agent reads — `state` + `approval` (or
`output` / `errorText` for HITL). Notes that the built-in transports
ship this slim shape by default and that fuller shapes are still
accepted.
- **`/in/append` 413 row and FAQ updated** (`client-protocol.mdx`,
`patterns/trusted-edge-signals.mdx`). Reflects the new precise S2 cap
and the CORS-readable 413.
- **New changelog entry** at the top of `changelog.mdx` covering all of
the above.

The historical `## 512 KiB ceiling removed` entry further down the
changelog is left as-is (it's a snapshot of the prior transition), and
the v4.5 upgrade-guide section is skipped — the merge contract is
backwards compatible.

## Test plan

- Mintlify dev preview renders cleanly with no broken anchors
- Linked references resolve (`/ai-chat/lifecycle-hooks#hydratemessages`,
`/ai-chat/lifecycle-hooks#onvalidatemessages`,
`/ai-chat/patterns/database-persistence#alternative-hydratemessages`,
`/ai-chat/client-protocol#step-3-send-messages-stops-and-actions`,
`/ai-chat/patterns/large-payloads`)
 2026-05-23 17:27:15 +01:00
github-actions[bot] d34014ddcc chore: release v4.5.0-rc.2 (#3702)
🚀 Publish Trigger.dev Docker / typecheck (push) Failing after 0s
Details
🚀 Publish Trigger.dev Docker / publish-webapp (push) Has been skipped
Details
🚀 Publish Trigger.dev Docker / publish-worker (push) Has been skipped
Details
🚀 Publish Trigger.dev Docker / publish-worker-v4 (push) Has been skipped
Details
🚀 Publish Trigger.dev Docker / units (push) Failing after 4s
Details 
## Summary
3 improvements, 1 bug fix.

## Improvements
- The per-turn merge now overlays the wire copy's tool-part state
advancement onto the agent's existing chain — `state` + the matching
resolution field (`output` / `errorText` / `approval`) come from the
wire, everything else (text, reasoning, tool `input`, provider metadata)
stays whatever the snapshot or `hydrateMessages` returned. Previously a
full-message replace overwrote those fields with whatever the client
shipped, so a slimmed wire copy landed a tool call with no `arguments`
on the next LLM call. Covers `output-available` / `output-error` (HITL
`addToolOutput`) and `approval-responded` / `output-denied` (approval
flow).
- `TriggerChatTransport.sendMessages` and `AgentChat.sendRaw` now slim
assistant messages that carry advanced tool parts. The wire payload is
just `{ id, role, parts: [<state + resolution field>] }` for
`submit-message` continuations; everything else passes through.
Reasoning blobs and full tool inputs no longer ride the wire on every
`addToolOutput` / `addToolApproveResponse`, so continuation payloads
stay well under the `.in/append` cap on long agent loops.
- Add `TriggerClient` for running multiple SDK clients side-by-side,
each with its own auth, preview branch, and baseURL. Useful when a
single process needs to trigger tasks or read runs across multiple
projects, environments, or preview branches without mutating shared
global state.
([#3683](https://github.com/triggerdotdev/trigger.dev/pull/3683))

## Bug fixes
- Fix `chat.agent` HITL continuations on reasoning-heavy turns. Two
changes that work together:
([#3719](https://github.com/triggerdotdev/trigger.dev/pull/3719))

<details>
<summary>Raw changeset output</summary>

⚠️⚠️⚠️⚠️⚠️⚠️

`main` is currently in **pre mode** so this branch has prereleases
rather than normal releases. If you want to exit prereleases, run
`changeset pre exit` on `main`.

⚠️⚠️⚠️⚠️⚠️⚠️

# Releases
## @trigger.dev/build@4.5.0-rc.2

### Patch Changes

-   Updated dependencies:
    -   `@trigger.dev/core@4.5.0-rc.2`

## trigger.dev@4.5.0-rc.2

### Patch Changes

-   Updated dependencies:
    -   `@trigger.dev/build@4.5.0-rc.2`
    -   `@trigger.dev/core@4.5.0-rc.2`
    -   `@trigger.dev/schema-to-json@4.5.0-rc.2`

## @trigger.dev/plugins@4.5.0-rc.2

### Patch Changes

-   Updated dependencies:
    -   `@trigger.dev/core@4.5.0-rc.2`

## @trigger.dev/python@4.5.0-rc.2

### Patch Changes

-   Updated dependencies:
    -   `@trigger.dev/sdk@4.5.0-rc.2`
    -   `@trigger.dev/build@4.5.0-rc.2`
    -   `@trigger.dev/core@4.5.0-rc.2`

## @trigger.dev/react-hooks@4.5.0-rc.2

### Patch Changes

-   Updated dependencies:
    -   `@trigger.dev/core@4.5.0-rc.2`

## @trigger.dev/redis-worker@4.5.0-rc.2

### Patch Changes

-   Updated dependencies:
    -   `@trigger.dev/core@4.5.0-rc.2`

## @trigger.dev/rsc@4.5.0-rc.2

### Patch Changes

-   Updated dependencies:
    -   `@trigger.dev/core@4.5.0-rc.2`

## @trigger.dev/schema-to-json@4.5.0-rc.2

### Patch Changes

-   Updated dependencies:
    -   `@trigger.dev/core@4.5.0-rc.2`

## @trigger.dev/sdk@4.5.0-rc.2

### Patch Changes

- Fix `chat.agent` HITL continuations on reasoning-heavy turns. Two
changes that work together:
([#3719](https://github.com/triggerdotdev/trigger.dev/pull/3719))

- The per-turn merge now overlays the wire copy's tool-part state
advancement onto the agent's existing chain — `state` + the matching
resolution field (`output` / `errorText` / `approval`) come from the
wire, everything else (text, reasoning, tool `input`, provider metadata)
stays whatever the snapshot or `hydrateMessages` returned. Previously a
full-message replace overwrote those fields with whatever the client
shipped, so a slimmed wire copy landed a tool call with no `arguments`
on the next LLM call. Covers `output-available` / `output-error` (HITL
`addToolOutput`) and `approval-responded` / `output-denied` (approval
flow).
- `TriggerChatTransport.sendMessages` and `AgentChat.sendRaw` now slim
assistant messages that carry advanced tool parts. The wire payload is
just `{ id, role, parts: [<state + resolution field>] }` for
`submit-message` continuations; everything else passes through.
Reasoning blobs and full tool inputs no longer ride the wire on every
`addToolOutput` / `addToolApproveResponse`, so continuation payloads
stay well under the `.in/append` cap on long agent loops.

Note: `onValidateMessages` receives the slim wire on HITL turns. If you
call `validateUIMessages` from `ai` against the full `messages` array it
will reject the slim assistant; filter to user messages (or skip on HITL
turns) — see the updated docstring on `onValidateMessages` for the
recommended pattern.

For `hydrateMessages` hooks that persist the chain, this release also
adds a small helper to the `@trigger.dev/sdk/ai` surface:

    ```ts
    import { chat, upsertIncomingMessage } from "@trigger.dev/sdk/ai";

    chat.agent({
hydrateMessages: async ({ chatId, trigger, incomingMessages }) => {
const record = await db.chat.findUnique({ where: { id: chatId } });
        const stored = record?.messages ?? [];
if (upsertIncomingMessage(stored, { trigger, incomingMessages })) {
await db.chat.update({ where: { id: chatId }, data: { messages: stored }
});
        }
        return stored;
      },
    });
    ```

It pushes fresh user messages by id, no-ops on HITL continuations (the
incoming shares an id with the existing assistant — the runtime overlays
the new tool-state advance), and skips on non-`submit-message` triggers.
Returns `true` if it mutated `stored` so the caller knows whether to
persist.

Net effect: `chat.addToolOutput(...)` /
`chat.addToolApproveResponse(...)` on multi-step reasoning agents
(OpenAI Responses with `store: false`, Anthropic extended thinking,
etc.) no longer blows the cap and no longer corrupts the LLM input.

- Add `TriggerClient` for running multiple SDK clients side-by-side,
each with its own auth, preview branch, and baseURL. Useful when a
single process needs to trigger tasks or read runs across multiple
projects, environments, or preview branches without mutating shared
global state.
([#3683](https://github.com/triggerdotdev/trigger.dev/pull/3683))

    ```ts
    import { TriggerClient } from "@trigger.dev/sdk";

const prod = new TriggerClient({ accessToken:
process.env.TRIGGER_PROD_KEY });
    const preview = new TriggerClient({
      accessToken: process.env.TRIGGER_PREVIEW_KEY,
      previewBranch: "signup-flow",
    });

    await prod.tasks.trigger("send-email", payload);
    await preview.runs.list({ status: ["COMPLETED"] });
    ```

-   Updated dependencies:
    -   `@trigger.dev/core@4.5.0-rc.2`

## @trigger.dev/core@4.5.0-rc.2

</details>

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
v.docker.4.5.0-rc.2 v4.5.0-rc.2 		2026-05-23 17:15:33 +01:00
Eric Allam 75679c7518 fix(sdk): chat HITL continuations no longer break the next LLM call (#3719) 
## Summary

Multi-step reasoning agents with HITL tools (OpenAI Responses with
`store: false`, Anthropic extended thinking, etc.) failed on
`chat.addToolOutput(...)` continuations — either the wire payload blew
the `.in/append` cap (reasoning blobs + tool inputs routinely > 512
KiB), or app-side slimming workarounds got overwritten server-side and
the next LLM call landed a tool call with no `arguments`. Both modes are
fixed.

## Design

The per-turn merge in `chat.agent` now overlays only the tool-part state
advances (`output-available` / `output-error` / `approval-responded` /
`output-denied`) from the wire copy onto the hydrated/snapshot chain.
Previously it replaced the entire message, which dropped `input`,
reasoning, and text from the LLM's view whenever the wire was slim.

In parallel, `TriggerChatTransport.sendMessages` and `AgentChat.sendRaw`
now slim the assistant message themselves on `submit-message`
continuations: ship `{ id, role, parts: [<resolved tool part only>] }`,
everything else reconstructed server-side from `hydrateMessages` or the
durable snapshot. Continuation payloads drop from 600 KiB – 1 MiB to ~1
KiB.

`references/ai-chat` `aiChatHydrated.hydrateMessages` now upserts by id
instead of pushing. With slim continuations, a blind push duplicates the
assistant id in the returned chain — the merge updates the first match,
the slim duplicate goes straight to `toModelMessages` with no `input`,
and the LLM 4xx's. This is the canonical pattern customers should mirror
in their own hydrate implementations.

## Test plan

- 11 new tests (slim helper unit + slim+merge integration for HITL,
approval, default no-hydrate branch)
- Full SDK suite: 239 tests pass across 19 files
- End-to-end sweep against `references/ai-chat`: 19 customer-side smoke
tests green; HITL wire bodies confirmed at ~1 KiB (was 600 KiB+); no
provider 4xx errors across OpenAI Responses or Anthropic
 2026-05-23 17:09:39 +01:00
Eric Allam c0365d36fb fix(webapp): precise S2 record cap + CORS 413 on session append (#3720) 
## Summary

Two improvements to session `.in/append`:

- Oversize-body 413 responses now carry CORS headers, so browser fetches
see a readable status instead of an opaque `TypeError: Failed to fetch`.
App-side retry-on-disconnect loops no longer spin forever on a
permanently-rejected payload.
- The per-record cap is now computed precisely against S2's actual
ceiling instead of the conservative 512 KiB floor. Legitimate ~600-900
KiB tool outputs (search results, file content) now succeed;
pathological all-quote content that would double under JSON escape still
rejects cleanly.

## Design

S2 enforces a per-record metered size of `8 + 2*H + Σ(header name +
value) + body ≤ 1048576` bytes. With no record headers (our case), the
budget reduces to `body ≤ 1048568`. Verified empirically against cloud
S2 — append succeeds at metered=1048576 and 422s at 1048577 with `record
must have metered size less than 1 MiB`.

The old `MAX_APPEND_BODY_BYTES = 512 KiB` was derived by assuming
worst-case JSON escape doubling (every byte becomes `\"` or `\\`),
giving `(1 MiB - overhead) / 2`. Safe, but rejects ~half the legitimate
input space.

The new flow:

1. Pre-cap the HTTP body at 1 MiB (DoS guard against reading arbitrary
garbage before we can compute the wrap).
2. After reading, `S2RealtimeStreams.#appendPartByName` computes
`Buffer.byteLength(JSON.stringify({data: part, id: partId}), "utf8") +
8` and throws `S2RecordTooLargeError` (a `ServiceValidationError` with
status 413) if it would exceed S2's ceiling. The route's existing error
branch maps the throw to a 413 with a descriptive message.

The 413 CORS fix is a single-line change in `apiBuilder.server.ts` —
`wrapResponse` was being skipped on the body-too-large branch; every
other error branch wraps; the 413 was the exception.

## Test plan

- Empirically verified against cloud S2 with a boundary scan across
`[1048568, 1048569, ..., 1048576]` and across H ∈ {0, 1×5 hdr bytes,
1×14 hdr bytes} — the formula matches exactly
- Browser-side fetch on a 700 KiB POST now resolves with a readable
`status: 413` (no `TypeError: Failed to fetch`)
- A 900 KiB ASCII tool output now passes (would have 413'd at 512 KiB
pre-fix)
 2026-05-23 16:08:10 +01:00
Matt Aitken 2fcc484a2b fix(webapp): route OrganizationDataStoresRegistry writes through the writer prisma (#3722) 
## Bug

The `OrganizationDataStoresRegistry` singleton in
`apps/webapp/app/services/dataStores/organizationDataStoresRegistryInstance.server.ts`
was constructed with `$replica`. That client was then used by both the
polling read path *and* by `addDataStore` / `updateDataStore` /
`deleteDataStore` (and their backing `SecretStore.setSecret` upserts).
The write methods route through the read replica, which Postgres rejects
with **error code 25006**:

```
Invalid prisma.secretStore.upsert() invocation:
ConnectorError(ConnectorError { user_facing_error: None, kind: QueryError(PostgresError {
  code: "25006",
  message: "cannot execute INSERT in a read-only transaction",
  ...
}), transient: false })
```

User-visible symptom: the admin `/admin/data-stores` "Add data store"
form returns a 400 with this error wrapped, so no
`OrganizationDataStore` row can ever be created via the UI.

The read path (`loadFromDatabase` polling + `SecretStore.getSecret`) is
unaffected because `findMany` + secret read are read-only.

## Fix

Change the registry constructor to take both a writer and a replica:

```ts
constructor(writer: PrismaClient, replica: PrismaClient | PrismaReplicaClient)
```

- `loadFromDatabase()` keeps using `_replica` (and its
`SecretStore.getSecret` calls) — these are background cache-fillers, not
on user-latency-sensitive paths.
- `addDataStore` / `updateDataStore` / `deleteDataStore` (and their
`SecretStore.setSecret` / `deleteSecret` calls) now use `_writer`.

`organizationDataStoresRegistryInstance.server.ts` passes `(prisma,
$replica)` from `~/db.server`. Test sites that constructed with
`(prisma)` now pass `(prisma, prisma)` — the testcontainer exposes a
single client, so the writer/replica split collapses to one connection.

## Files

-
`apps/webapp/app/services/dataStores/organizationDataStoresRegistry.server.ts`
— constructor + read/write split
-
`apps/webapp/app/services/dataStores/organizationDataStoresRegistryInstance.server.ts`
— pass `prisma` alongside `$replica`
- `apps/webapp/test/organizationDataStoresRegistry.test.ts` — 14 call
sites bumped
- `apps/webapp/test/clickhouseFactory.test.ts` — 5 call sites bumped

## Test plan

- [x] Existing `organizationDataStoresRegistry.test.ts` +
`clickhouseFactory.test.ts` still pass (constructor sites updated;
behavior unchanged for tests).
- [ ] After deploy to test cloud, retry `/admin/data-stores` "Add data
store" form for the HIPAA org — should now succeed and the row should
appear.
- [ ] Verify the registry's polling reload picks up the new row within
`ORGANIZATION_DATA_STORES_RELOAD_INTERVAL_MS` (60s default) and the
factory starts routing to the org-scoped instance.
 2026-05-23 12:00:31 +01:00
Eric Allam 61ca40b4b1 perf(run-engine,webapp): look up PENDING_VERSION runs via ClickHouse (#3707) 
## Summary

When a background worker registers, the engine resolves runs that were
queued before the worker was ready (status `PENDING_VERSION`). That
lookup used to scan a Postgres status index on `TaskRun`. Move it to
ClickHouse: query candidate run ids from `task_runs_v2`, then refetch
the actual rows from Postgres by primary key with a `status =
'PENDING_VERSION'` guard for idempotency.

## Design

The lookup is a pluggable interface on the run engine
(`PendingVersionRunIdLookup`). The webapp wires a ClickHouse-backed
implementation through the org-scoped `clickhouseFactory` using a new
`"engine"` client type, configured by `RUN_ENGINE_CLICKHOUSE_*` env
vars. The URL falls back to `CLICKHOUSE_URL` when unset, so self-hosted
deployments don't need new config to keep working.

When the lookup returns no candidates, one bounded retry is scheduled
~5s later to cover ClickHouse replication lag against `task_runs_v2`.
The Postgres status guard on both the candidate refetch and the inner
`updateMany` prevents double-promotion when a retry races with a
concurrent deploy.

Tests cover three existing PENDING_VERSION cases via a small
Postgres-backed test adapter; new ClickHouse-backed integration tests
will follow.
 2026-05-22 17:18:41 +01:00
Eric Allam 0d4891a5f2 perf(database): drop unused TaskRun(scheduleId, createdAt) index (#3706) 
## Summary

Drops the unused composite Postgres index
`TaskRun_scheduleId_createdAt_idx`. The schedule list view reads from
ClickHouse, so this index served no Prisma query while still being
maintained on every `TaskRun` INSERT/UPDATE. Removing it reduces write
amplification on the primary database.

Sibling to the prior drop of `TaskRun_scheduleId_idx` and the earlier
removal of the `TaskRun.scheduleId` foreign key — all stemming from
migrating schedule-aware reads to ClickHouse.

## Verification

- Sampled `pg_stat_user_indexes` for `TaskRun` over multiple hours —
zero scans against this index.
- Grepped the codebase for any Prisma query filtering
`TaskRun.scheduleId` — none found. All schedule-aware listing routes
through `clickhouseRunsRepository`.
 2026-05-22 16:34:00 +01:00
Saadi Myftija 2cbfcfac3d chore(webapp): make HTTP keep-alive timeout configurable (#3705) 
Make the Express server's `keepAliveTimeout` configurable via
`HTTP_KEEPALIVE_TIMEOUT_MS`. Default preserved at 65000 ms — no behavior
change if unset.
 2026-05-22 16:28:08 +01:00
nicktrn ddad9700d7 fix(supervisor): compat shim for COMPUTE checkpoint type (#3703) 
Workloads bundled with CLI versions before v4.4.4 use a strict zod enum
for `checkpoint.type` that only allows DOCKER and KUBERNETES. When a
customer's runs are routed via the compute path, those old runners
receive `type: "COMPUTE"` on `/snapshots/since/...` and `/dequeue`
responses and fail validation - blocking silent migration of existing
deployments.

The workload never reads the field - only validates the shape. Rewriting
COMPUTE -> KUBERNETES on the way out lets older runners keep parsing
while the database and internal services keep the real value. Limited to
the two workload-facing endpoints whose response includes a checkpoint;
`/continue`, `/attempts/start`, `/attempts/complete` all return shapes
without one.

Followup to #3114.
 2026-05-22 15:41:30 +01:00
Daniel Sutton 1015876b98 feat(webapp): user-based Sentry attribution with tenant tags (#3678) 
## Summary

Stamp every Sentry event with the signed-in user and the tenant (org /
project / env) the request belongs to, so "Users Impacted" counts
distinct humans and events become filterable per tenant.

**Design after review (current):**

- `user.id = real user cuid` (from `requireUser`). "Users Impacted"
counts humans, not tenants.
- Tenant context (org / project / env slugs, IDs, env type) moves
entirely onto tags: `org_slug`, `project_slug`, `env_slug`, `org_id`,
`project_id`, `project_ref`, `environment_id`, `env_type`, plus
`impersonating` when set.
- Backed by an `AsyncLocalStorage` scope established at the HTTP entry.
Each entry point fills what it knows; loaders enrich the same scope with
what they already have.

**Zero new database queries.** The middleware does a regex match only.
Dashboard loaders that already query Prisma gain a couple of extra
selected columns; nothing new round-trips.

## How it's wired

- **Express middleware (`tenantContextResolver.server.ts`)** — parses
the URL with a regex and always opens an ALS scope. Populates whatever
subset of slugs is present: `/orgs/:o` → just `orgSlug`;
`/orgs/:o/projects/:p` adds `projectSlug`; the full triple adds
`envSlug`. Non-tenant paths get an empty scope so loaders can still
enrich.
- **`_app/route.tsx`** — already calls `requireUser`. Adds
`tenantContext.enrich({ userId: user.id })` for every authenticated
dashboard request. No new query.
- **Env layout loader (`_app.orgs.$o.projects.$p.env.$e/route.tsx`)** —
its existing `prisma.project.findFirst` gains two columns in `select`
(`externalRef`, `organization.id`). After it picks an env, calls
`tenantContext.enrich({ orgId, projectId, projectRef, envId, envType
})`. Same query, +2 columns.
- **API path (`apiBuilder.server.ts`)** — wraps every handler in
`tenantContext.run(tenantContextFromAuthEnvironment(authenticationResult.environment),
…)`. The mapper pulls `userId` from `env.orgMember?.userId` (already
selected by `authIncludeBase` — no schema change). Covers
`createLoaderApiRoute`, `createActionApiRoute`, and
`createMultiMethodApiRoute`.
- **Event processor (`sentryTenantContext.server.ts`)** — registered in
`entry.server.tsx` so it lives in the Remix bundle and shares the same
`tenantContext` ALS instance as the middleware and loaders. Stamps
whatever's present; nothing forced.

## Example events from local verification

| URL | `user.id` | Tags |
|-----|-----------|------|
| `/orgs/:o/projects/:p/env/:e/...` | real user cuid | `org_slug`,
`project_slug`, `env_slug`, `org_id`, `project_id`, `project_ref`,
`environment_id`, `env_type` |
| `/orgs/:o/settings` (non-env-scoped) | real user cuid | `org_slug`
only |
| API request with `orgMember` | `orgMember.userId` | full tenant set |
| API request without `orgMember` | (unset) | full tenant set |

## Trade-offs

1. On env-scoped pages, errors that fire before the env layout loader's
enrich callback runs get slugs + `user.id` but not the tenant IDs /
`env_type`. Realistic errors deep in async work get the full set. (Same
race as before, narrower window now that slugs/`user.id` are populated
up-front by the middleware and `_app` enrich.)
2. API requests where the environment has no `orgMember` get tenant tags
but no `user.id`. Those events still show in the issue but don't
contribute to "Users Impacted".

## Out of scope (deferred)

Background workers (`redis-worker`, `schedule-engine`) and socket
handlers. Those entry points don't set `tenantContext.run` yet — their
events ship without tenant attribution until each is wired in a
follow-up.

## Tests

31 unit tests across 4 files. New tests notably cover:

- `parseTenantPath`: org-only, org+project, and full-triple URL
variants.
- `tenantContext.enrich`: in-place patch, no-op outside `run()`,
concurrent-scope isolation, empty-scope + enrich pattern (for non-tenant
pages).
- `tenantContextFromAuthEnvironment`: with and without `orgMember` —
verifies the API path's `user.id` mapping.
- `addTenantContextToEvent`: empty scope, userId-only, slugs-only, full
enrichment, conditional tag emission, preservation of prior `event.user`
fields.

## Test plan

- [ ] `pnpm run typecheck --filter webapp`
- [ ] `pnpm run test --filter webapp -- test/tenantContext.test.ts
test/sentryTenantContext.test.ts test/tenantContextResolver.test.ts
test/tenantContextFromAuthEnvironment.test.ts`
- [ ] Local manual: with `SENTRY_DSN` set, hit a dashboard URL and an
API route, confirm the captured events carry `user.id` + the expected
tag set in Sentry.
- [ ] After ship: confirm "Users Impacted" on a real Sentry issue
reflects distinct users (not tenants).

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-05-22 15:10:41 +01:00
Matt Aitken 71d98b4e6b Support for org-scoped ClickHouse (#3333) 
Added `OrganizationDataStore` which allows orgs to have data stored in
specific separate services.

For now this is just used for ClickHouse. When using ClickHouse we get a
client for the factory and pass in the org id.

Particular care has to be made with two hot-insert paths:
1. RunReplicationService
2. OTLPExporter

---------

Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: Claude <noreply@anthropic.com>
 2026-05-22 14:43:03 +01:00
Eric Allam 832cf7220b feat(sdk,core): add TriggerClient for per-instance SDK configuration (#3683) 
## Summary

`new TriggerClient({...})` exposes the management API (tasks, runs,
schedules, envvars, batch, queues, deployments, prompts, auth) as an
explicit instance with its own auth, preview branch, and baseURL.
Multiple clients can coexist in one process without mutating shared
global state — useful when a single service triggers across multiple
projects, environments, or preview branches.

```ts
import { TriggerClient } from "@trigger.dev/sdk";

const prod = new TriggerClient({ accessToken: process.env.TRIGGER_PROD_KEY });
const preview = new TriggerClient({
  accessToken: process.env.TRIGGER_PREVIEW_KEY,
  previewBranch: "signup-flow",
});

await prod.tasks.trigger("send-email", payload);
await preview.runs.list({ status: ["COMPLETED"] });
```

The existing global `configure()` API keeps working unchanged.

## Design

Instance methods enter an `AsyncLocalStorage`-backed scope (`sdkScope`)
before delegating to the existing module-level functions. The four
"pollution" points that previously read globals now consult the scope
first:

- `apiClientManager.{baseURL, accessToken, branchName}` and
`clientOrThrow` — identity fields are scope-only when scoped; `baseURL`
still falls back to `TRIGGER_API_URL` because plumbing (where the API
lives) is not identity.
- `taskContext.{ctx, worker, isWarmStart, isInsideTask}` — masked inside
an isolated scope so a `client.tasks.trigger(...)` from inside a task
doesn't leak the parent's `parentRunId` / `lockToVersion` / `isTest`
into a trigger that hits a different project.
- Inline `getEnvVar("TRIGGER_VERSION")` reads in `shared.ts` go through
a `scopedEnvVar` helper that returns `undefined` inside an isolated
scope.

The `TriggerClient` class itself is a thin wrapper that captures the
scope in its constructor and proxies each namespace method to enter that
scope before calling the existing impl. Generic inference (e.g.
`client.tasks.trigger<typeof t>(...)`) is preserved via `Pick<typeof ns,
keyof curatedSubset>` typings.

Two correctness fixes uncovered along the way are folded in:

- `apiClientManager.setGlobalAPIClientConfiguration` no longer silently
no-ops on the second call. `configure()` now actually overrides as users
expect (this is the root cause behind some "I changed the config but
nothing happened" reports).
- `apiClientManager.runWithConfig` (and therefore `auth.withAuth`) is
now backed by `sdkScope.withScope` instead of "mutate the global and
restore in finally". Two parallel `withAuth` calls with different
configs no longer stomp each other.

Surface curation: instance namespaces drop methods that don't make sense
per-instance — `batch.*AndWait` (runtime-dependent), `schedules.task` /
`schedules.timezones` (definition-time / stateless), `prompts.define`
(definition-time), `auth.configure` / `auth.withAuth` (global-only).

## Test plan

- [x] 9 runtime unit tests in `triggerClient.test.ts` cover: required
accessToken, instance auth + branch headers, no env fallback for
identity fields, no leakage between global and instance, four parallel
calls across two clients stay isolated, taskContext masking +
`inheritContext: true` override, `configure()` second-call override,
parallel `auth.withAuth` isolation.
- [x] 10 type-level assertions in `triggerClient.types.test.ts` using
`expectTypeOf` + `@ts-expect-error` lock in generic inference, return
type passthrough, overload preservation, and curated-surface drift.
- [x] Full SDK suite (219 tests) and core suite (530 tests) pass.
- [x] Webapp typecheck clean.
- [x] End-to-end smoke test against local webapp and a
freshly-provisioned cloud project — six concurrent multi-client triggers
all returned 200 with run IDs, headers per-client as expected.
- [ ] Reviewer: run `references/multi-client` per its `README.md` to
reproduce the smoke test locally.

## Try it

`references/multi-client` is a new reference workspace that exercises
this end-to-end:

- `src/trigger/echo.ts` — trivial target task
- `src/trigger/fanOut.ts` — opens two `TriggerClient`s from inside a
task, fires `echo` through each in parallel
- `src/external/main.ts` — external Node script with two clients
triggering `echo` sequentially and concurrently; logs every outgoing
request's `authorization` + `x-trigger-branch`
- `src/external/isolation.ts` — interleaves global `configure()` and an
instance call, asserts the captured fetch sequence shows no leakage
either way
 2026-05-22 12:29:42 +01:00
nicktrn 18d7144e74 ci: daily dependabot critical-severity slack alerts (#3701) 
Sibling to the weekly summary, focused on critical alerts only. Pings
Slack daily while any critical alerts are open; skips the post entirely
when zero, so no daily "all clear" noise.

- Daily 08:00 UTC cron + `workflow_dispatch` with `severity` input
(default `critical`, override to `high`/`medium`/`low` for manual
checks)
- Reuses the existing `dependabot-summary` environment (token, channel,
bot)
- Alerts link at the end is severity-filtered
 2026-05-22 10:32:08 +01:00
Eric Allam c0b9fdfce9 docs(ai-chat): clarify lastEventId is sessionId-keyed across run boundaries (#3700) 
## Summary

Two docs edits that close a footgun customers persisting transport state
can hit. Clearing `lastEventId` on `chat.endRun()` looks intuitive — the
Run ended, the cursor must be stale — but the cursor is sessionId-keyed,
not runId-keyed. Clearing it forces the next `sendMessages` to subscribe
from `seq_num=0`, which may hit the prior turn's still-durable
`turn-complete` record and close the SSE empty before the new Run's
chunks arrive.

Spells out the invariant in the frontend transport persistence table and
adds a Warning in the `chat.endRun()` reference.

## Test plan

- [x] Mintlify preview renders
- [x] No callout stacking
 2026-05-22 10:26:06 +01:00
Eric Allam c80b85e2cf docs(ai-chat): atomic onTurnComplete writes + Anthropic prose (#3693) 
## Summary

Three post-merge fixes for the AI Agents docs (#3226), all caught by
review after merge.

## Fixes

- **`onTurnComplete` examples now use `db.$transaction`** — both the
Database persistence "Complete example" and the Lifecycle hooks
reference example were doing two separate `await` calls
(`db.chat.update` then `db.chatSession.upsert`). That's the exact
non-atomic pattern the warning earlier on the persistence page calls out
as ❌: a refresh between the two writes reads a stale `lastEventId` and
duplicates the assistant message on resume. Both examples now use the
recommended atomic form.

- **Background injection self-review prose aligned with the code** — the
prose said "gpt-4o-mini" but the example above it had been swapped to
`claude-haiku-4-5`. The Anthropic-sweep script only touched code blocks;
this prose line wasn't picked up.

## Test plan

- [x] Both updated examples use `db.$transaction([...])`
- [x] Prose matches the model used in the code block
- [ ] Mintlify deployment passes
 2026-05-21 17:04:13 +01:00
Eric Allam 80bb600cd5 docs(ai-chat): AI Agents documentation for v4.5 (#3226) 
## Summary

Lands the full AI Agents documentation surface alongside the v4.5
release candidate of `@trigger.dev/sdk`. Covers `chat.agent` end to end
— defining agents, lifecycle hooks, the frontend transport, sub-agents,
recovery from cancel/crash/OOM, AI Prompts integration — and the
Sessions primitive that backs it.

## Coverage

- **Conceptual**: Overview, Quick Start, How it works.
- **Building agents**: Backend (`chat.agent` / `chat.createSession` /
raw primitives), Lifecycle hooks, Frontend transport, Server-side
`AgentChat`, Sessions reference, `chat.local` state primitive,
TypeScript types.
- **Features**: AI Prompts integration, Fast starts (Preload + Head
Start), Compaction, Pending Messages (steering), Background Injection
(`chat.inject` + `chat.defer`), Actions (undo / regenerate / edit),
Error handling.
- **Patterns (13)**: Sub-agents, Branching conversations, Code sandbox,
Database persistence, Persistence and replay, HITL, Tool result
auditing, Large payloads, Agent skills, OOM resilience, Recovery boot,
Trusted edge signals, Version upgrades.
- **Reference**: API Reference, Client Protocol (wire format), Testing
harness (`mockChatAgent`), MCP server tools, Upgrade guide, Changelog.

## Structure changes

- Top-level nav: AI → **Agents**, with sub-groups for *Building agents /
Features / Patterns / Reference*.
- New RC banner snippet on every page links to the supported AI SDK
versions table on the API Reference.
- All examples use Anthropic with `stopWhen: stepCountIs(15)`.

---------

Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-21 16:50:59 +01:00
github-actions[bot] 0b5b817a49 chore: release v4.5.0-rc.1 (#3691)
🚀 Publish Trigger.dev Docker / typecheck (push) Failing after 0s
Details
🚀 Publish Trigger.dev Docker / publish-webapp (push) Has been skipped
Details
🚀 Publish Trigger.dev Docker / publish-worker (push) Has been skipped
Details
🚀 Publish Trigger.dev Docker / publish-worker-v4 (push) Has been skipped
Details
🚀 Publish Trigger.dev Docker / units (push) Failing after 0s
Details
🧭 Helm Chart Release / lint-and-test (push) Has been cancelled
Details
🧭 Helm Chart Release / release (push) Has been cancelled
Details 
## Summary
2 bug fixes.

## Bug fixes
- Fix `chat.agent` skills silently missing in `trigger dev` for projects
whose task files read `process.env` at module top level (e.g. a
third-party SDK client initialized at import). Skill folders now bundle
into `.trigger/skills/` reliably regardless of which env vars are set
when the CLI launches.
([#3690](https://github.com/triggerdotdev/trigger.dev/pull/3690))
- Fix `COULD_NOT_FIND_EXECUTOR` when a task's definition is loaded via
`await import(...)` from inside another task's `run()`. The runtime
workers now register such tasks with a sentinel file context, and the
catalog logs a one-time warning per task id.
([#3688](https://github.com/triggerdotdev/trigger.dev/pull/3688))

<details>
<summary>Raw changeset output</summary>

⚠️⚠️⚠️⚠️⚠️⚠️

`main` is currently in **pre mode** so this branch has prereleases
rather than normal releases. If you want to exit prereleases, run
`changeset pre exit` on `main`.

⚠️⚠️⚠️⚠️⚠️⚠️

# Releases
## @trigger.dev/build@4.5.0-rc.1

### Patch Changes

-   Updated dependencies:
    -   `@trigger.dev/core@4.5.0-rc.1`

## trigger.dev@4.5.0-rc.1

### Patch Changes

- Fix `chat.agent` skills silently missing in `trigger dev` for projects
whose task files read `process.env` at module top level (e.g. a
third-party SDK client initialized at import). Skill folders now bundle
into `.trigger/skills/` reliably regardless of which env vars are set
when the CLI launches.
([#3690](https://github.com/triggerdotdev/trigger.dev/pull/3690))
- Fix `COULD_NOT_FIND_EXECUTOR` when a task's definition is loaded via
`await import(...)` from inside another task's `run()`. The runtime
workers now register such tasks with a sentinel file context, and the
catalog logs a one-time warning per task id.
([#3688](https://github.com/triggerdotdev/trigger.dev/pull/3688))
-   Updated dependencies:
    -   `@trigger.dev/core@4.5.0-rc.1`
    -   `@trigger.dev/build@4.5.0-rc.1`
    -   `@trigger.dev/schema-to-json@4.5.0-rc.1`

## @trigger.dev/core@4.5.0-rc.1

### Patch Changes

- Fix `COULD_NOT_FIND_EXECUTOR` when a task's definition is loaded via
`await import(...)` from inside another task's `run()`. The runtime
workers now register such tasks with a sentinel file context, and the
catalog logs a one-time warning per task id.
([#3688](https://github.com/triggerdotdev/trigger.dev/pull/3688))

## @trigger.dev/plugins@4.5.0-rc.1

### Patch Changes

-   Updated dependencies:
    -   `@trigger.dev/core@4.5.0-rc.1`

## @trigger.dev/python@4.5.0-rc.1

### Patch Changes

-   Updated dependencies:
    -   `@trigger.dev/core@4.5.0-rc.1`
    -   `@trigger.dev/build@4.5.0-rc.1`
    -   `@trigger.dev/sdk@4.5.0-rc.1`

## @trigger.dev/react-hooks@4.5.0-rc.1

### Patch Changes

-   Updated dependencies:
    -   `@trigger.dev/core@4.5.0-rc.1`

## @trigger.dev/redis-worker@4.5.0-rc.1

### Patch Changes

-   Updated dependencies:
    -   `@trigger.dev/core@4.5.0-rc.1`

## @trigger.dev/rsc@4.5.0-rc.1

### Patch Changes

-   Updated dependencies:
    -   `@trigger.dev/core@4.5.0-rc.1`

## @trigger.dev/schema-to-json@4.5.0-rc.1

### Patch Changes

-   Updated dependencies:
    -   `@trigger.dev/core@4.5.0-rc.1`

## @trigger.dev/sdk@4.5.0-rc.1

### Patch Changes

-   Updated dependencies:
    -   `@trigger.dev/core@4.5.0-rc.1`

</details>

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
helm-v4.5.0-rc.1 v.docker.4.5.0-rc.1 v4.5.0-rc.1 		2026-05-21 16:40:21 +01:00
Eric Allam 5ddb81ac8d fix(cli): stop chat.agent skills silently disappearing from trigger dev (#3690) 
## Summary

`trigger.dev dev` was silently dropping registered `chat.agent` skills
for any project whose task files read `process.env` at module top level
— e.g. a third-party SDK client initialized at import. The agent would
boot fine, but `skill.local()` failed at runtime with `ENOENT` because
the skill folder was never copied into `.trigger/skills/`.

## Design

The CLI ran two indexer passes in dev: the worker's own indexer (with
the full env it eventually executes tasks in), and a separate
skill-discovery indexer with only the CLI process's env. Top-level reads
of vars like `TRIGGER_API_URL` imported cleanly in the worker pass and
threw in the skill pass — the latter caught the error, warned, and
skipped skill copying. Failure was silent enough that `skill.local()`
only surfaced it at task runtime.

The skill registry is already part of the worker manifest. This PR drops
the duplicate pass and copies skill folders from that manifest after the
worker initializes. One indexer instead of two; a bad `SKILL.md` now
surfaces as a startup error instead of silently disappearing skills.

Deploy is unaffected — its skill discovery uses the project's
environment variables (fetched via the API, which fills in
`TRIGGER_API_URL` etc.), so the dev failure mode doesn't reach there.

## Test plan

- [x] New `references/agent-skills` reference project with
`skills.define` + a task that calls `skill.local()` and runs a bundled
script
- [x] On `main`, adding a top-level
`process.env.TRIGGER_API_URL!.includes(...)` read in any task file
reproduces the symptom: warning at dev startup, no `.trigger/skills/`
folder, `skill.local()` fails with ENOENT
- [x] On this branch, same project boots clean and `skill.local()` works
end-to-end
- [x] Deploy still works end-to-end with the new reference project
 2026-05-21 16:30:24 +01:00
nicktrn dbf9b4e7e5 fix(core,cli): register tasks loaded via dynamic import during run (#3688) 
On a warm worker process, a task whose `task()` definition is loaded via
`await import(...)` from inside another task's `run()` could end up
permanently missing from the catalog: the `task()` call fired with no
`_currentFileContext` set, `registerTaskMetadata` silently returned, and
Node's ESM module cache then blocked the worker's setContext + re-import
recovery from ever firing the call again. Subsequent runs of that task
on the same warm process failed with `COULD_NOT_FIND_EXECUTOR` until the
process hit `maxExecutionsPerProcess` and exited.

All five of these had to coincide on the same worker for the bug to
surface:

1. `processKeepAlive` enabled (so catalog state survives across runs).
2. A `run()` function (or lifecycle hook) does `await import(...)`.
3. The import's transitive static graph reaches a `task()` /
`schemaTask()` call.
4. The task containing the dynamic import is the **first** task to run
on a given warm worker process — so the dropped `task()` calls fire on
this process for the first time, are silently dropped, and Node's module
cache locks the wrong outcome in.
5. A subsequent run for one of the dropped task ids lands on the same
warm worker before it recycles.

The runtime workers now set a sentinel file context (`<no-context>`)
around the `executor.execute(...)` call, so `task()` invocations firing
during a run register normally. The catalog detects the sentinel and
emits a one-time `console.warn` per task id so the pattern stays visible
without spamming. The indexer never sets this context, so deploy-time
behavior is unchanged.

Repro is `references/hello-world/src/trigger/dynamicImportRepro*.ts`.
Verified end-to-end against a deployed image with firestarter
warm-starts on: pre-fix saw `COULD_NOT_FIND_EXECUTOR` on children that
landed on the parent-poisoned worker; post-fix all 23/23 runs succeeded
and the warning surfaces in the parent's run trace.
 2026-05-21 16:21:49 +01:00
github-actions[bot] acfba02409 chore: release v4.5.0-rc.0 (#3563)
🚀 Publish Trigger.dev Docker / typecheck (push) Failing after 4s
Details
🚀 Publish Trigger.dev Docker / units (push) Failing after 0s
Details
🚀 Publish Trigger.dev Docker / publish-webapp (push) Has been skipped
Details
🚀 Publish Trigger.dev Docker / publish-worker (push) Has been skipped
Details
🚀 Publish Trigger.dev Docker / publish-worker-v4 (push) Has been skipped
Details
🧭 Helm Chart Release / lint-and-test (push) Has been cancelled
Details
🧭 Helm Chart Release / release (push) Has been cancelled
Details 
## Summary
44 improvements, 1 bug fix.

## Improvements
- **AI Prompts** — define prompt templates as code alongside your tasks,
version them on deploy, and override the text or model from the
dashboard without redeploying. Prompts integrate with the Vercel AI SDK
via `toAISDKTelemetry()` (links every generation span back to the
prompt) and with `chat.agent` via `chat.prompt.set()` +
`chat.toStreamTextOptions()`.
([#3629](https://github.com/triggerdotdev/trigger.dev/pull/3629))
- **Code-defined, deploy-versioned templates** — define with
`prompts.define({ id, model, config, variables, content })`. Every
deploy creates a new version visible in the dashboard. Mustache-style
placeholders (`{{var}}`, `{{#cond}}...{{/cond}}`) with Zod / ArkType /
Valibot-typed variables.
- **Dashboard overrides** — change a prompt's text or model from the
dashboard without redeploying. Overrides take priority over the deployed
"current" version and are environment-scoped (dev / staging / production
independent).
- **Resolve API** — `prompt.resolve(vars, { version?, label? })` returns
the compiled `text`, resolved `model`, `version`, and labels. Standalone
`prompts.resolve<typeof handle>(slug, vars)` for cross-file resolution
with full type inference on slug and variable shape.
- **AI SDK integration** — spread `resolved.toAISDKTelemetry({ ...extra
})` into any `generateText` / `streamText` call and every generation
span links to the prompt in the dashboard alongside its input variables,
model, tokens, and cost.
- **`chat.agent` integration** — `chat.prompt.set(resolved)` stores the
resolved prompt run-scoped; `chat.toStreamTextOptions({ registry })`
pulls `system`, `model` (resolved via the AI SDK provider registry),
`temperature` / `maxTokens` / etc., and telemetry into a single spread
for `streamText`.
- **Management SDK** — `prompts.list()`, `prompts.versions(slug)`,
`prompts.promote(slug, version)`, `prompts.createOverride(slug, body)`,
`prompts.updateOverride(slug, body)`, `prompts.removeOverride(slug)`,
`prompts.reactivateOverride(slug, version)`.
- **Dashboard** — prompts list with per-prompt usage sparklines;
per-prompt detail with Template / Details / Versions / Generations /
Metrics tabs. AI generation spans get a custom inspector showing the
linked prompt's metadata, input variables, and template content
alongside model, tokens, cost, and the message thread.
- Adds `onBoot` to `chat.agent` — a lifecycle hook that fires once per
worker process picking up the chat. Runs for the initial run, preloaded
runs, AND reactive continuation runs (post-cancel, crash, `endRun`,
`requestUpgrade`, OOM retry), before any other hook. Use it to
initialize `chat.local`, open per-process resources, or re-hydrate state
from your DB on continuation — anywhere the SAME run picking up after
suspend/resume isn't enough.
([#3543](https://github.com/triggerdotdev/trigger.dev/pull/3543))
- **AI SDK `useChat` integration** — a custom
[`ChatTransport`](https://sdk.vercel.ai/docs/ai-sdk-ui/transport)
(`useTriggerChatTransport`) plugs straight into Vercel AI SDK's
`useChat` hook. Text streaming, tool calls, reasoning, and `data-*`
parts all work natively over Trigger.dev's realtime streams. No custom
API routes needed.
- **First-turn fast path (`chat.headStart`)** — opt-in handler that runs
the first turn's `streamText` step in your warm server process while the
agent run boots in parallel, cutting cold-start TTFC by roughly half
(measured 2801ms → 1218ms on `claude-sonnet-4-6`). The agent owns step
2+ (tool execution, persistence, hooks) so heavy deps stay where they
belong. Web Fetch handler works natively in Next.js, Hono, SvelteKit,
Remix, Workers, etc.; bridge to Express/Fastify/Koa via
`chat.toNodeListener`. New `@trigger.dev/sdk/chat-server` subpath.
- **Multi-turn durability via Sessions** — every chat is backed by a
durable Session that outlives any individual run. Conversations resume
across page refreshes, idle timeout, crashes, and deploys; `resume:
true` reconnects via `lastEventId` so clients only see new chunks.
`sessions.list` enumerates chats for inbox-style UIs.
- **Auto-accumulated history, delta-only wire** — the backend
accumulates the full conversation across turns; clients only ship the
new message each turn. Long chats never hit the 512 KiB body cap.
Register `hydrateMessages` to be the source of truth yourself.
- **Lifecycle hooks** — `onPreload`, `onChatStart`,
`onValidateMessages`, `hydrateMessages`, `onTurnStart`,
`onBeforeTurnComplete`, `onTurnComplete`, `onChatSuspend`,
`onChatResume` — for persistence, validation, and post-turn work.
- **Stop generation** — client-driven `transport.stopGeneration(chatId)`
aborts mid-stream; the run stays alive for the next message, partial
response is captured, and aborted parts (stuck `partial-call` tools,
in-progress reasoning) are auto-cleaned.
- **Tool approvals (HITL)** — tools with `needsApproval: true` pause
until the user approves or denies via `addToolApprovalResponse`. The
runtime reconciles the updated assistant message by ID and continues
`streamText`.
- **Steering and background injection** — `pendingMessages` injects user
messages between tool-call steps so users can steer the agent
mid-execution; `chat.inject()` + `chat.defer()` adds context from
background work (self-review, RAG, safety checks) between turns.
- **Actions** — non-turn frontend commands (undo, rollback, regenerate,
edit) sent via `transport.sendAction`. Fire `hydrateMessages` +
`onAction` only — no turn hooks, no `run()`. `onAction` can return a
`StreamTextResult` for a model response, or `void` for side-effect-only.
- **Typed state primitives** — `chat.local<T>` for per-run state
accessible from hooks, `run()`, tools, and subtasks (auto-serialized
through `ai.toolExecute`); `chat.store` for typed shared data between
agent and client; `chat.history` for reading and mutating the message
chain; `clientDataSchema` for typed `clientData` in every hook.
- **`chat.toStreamTextOptions()`** — one spread into `streamText` wires
up versioned system [Prompts](https://trigger.dev/docs/ai/prompts),
model resolution, telemetry metadata, compaction, steering, and
background injection.
- **Multi-tab coordination** — `multiTab: true` + `useMultiTabChat`
prevents duplicate sends and syncs state across browser tabs via
`BroadcastChannel`. Non-active tabs go read-only with live updates.
- **Network resilience** — built-in indefinite retry with bounded
backoff, reconnect on `online` / tab refocus / bfcache restore,
`Last-Event-ID` mid-stream resume. No app code needed.
- **Sessions** — a durable, run-aware stream channel keyed on a stable
`externalId`. A Session is the unit of state that owns a multi-run
conversation: messages flow through `.in`, responses through `.out`,
both survive run boundaries. Sessions back the new `chat.agent` runtime,
and you can build on them directly for any pattern that needs durable
bi-directional streaming across runs.
([#3542](https://github.com/triggerdotdev/trigger.dev/pull/3542))
- Add `ai.toolExecute(task)` so you can wire a Trigger subtask in as the
`execute` handler of an AI SDK `tool()` while defining `description` and
`inputSchema` yourself — useful when you want full control over the tool
surface and just need Trigger's subtask machinery for the body.
([#3546](https://github.com/triggerdotdev/trigger.dev/pull/3546))
- Type `chat.createStartSessionAction` against your chat agent so
`clientData` is typed end-to-end on the first turn:
([#3684](https://github.com/triggerdotdev/trigger.dev/pull/3684))
- Add `region` to the runs list / retrieve API: filter runs by region
(`runs.list({ region: "..." })` / `filter[region]=<masterQueue>`) and
read each run's executing region from the new `region` field on the
response.
([#3612](https://github.com/triggerdotdev/trigger.dev/pull/3612))
- Add `TRIGGER_BUILD_SKIP_REWRITE_TIMESTAMP=1` escape hatch for local
self-hosted builds whose buildx driver doesn't support
`rewrite-timestamp` alongside push (e.g. orbstack's default `docker`
driver).
([#3618](https://github.com/triggerdotdev/trigger.dev/pull/3618))
- Reject overlong `idempotencyKey` values at the API boundary so they no
longer trip an internal size limit on the underlying unique index and
surface as a generic 500. Inputs are capped at 2048 characters — well
above what `idempotencyKeys.create()` produces (a 64-character hash) and
above any realistic raw key. Applies to `tasks.trigger`,
`tasks.batchTrigger`, `batch.create` (Phase 1 streaming batches),
`wait.createToken`, `wait.forDuration`, and the input/session stream
waitpoint endpoints. Over-limit requests now return a structured 400
instead.
([#3560](https://github.com/triggerdotdev/trigger.dev/pull/3560))
- **AI SDK `useChat` integration** — a custom
[`ChatTransport`](https://sdk.vercel.ai/docs/ai-sdk-ui/transport)
(`useTriggerChatTransport`) plugs straight into Vercel AI SDK's
`useChat` hook. Text streaming, tool calls, reasoning, and `data-*`
parts all work natively over Trigger.dev's realtime streams. No custom
API routes needed.
- **First-turn fast path (`chat.headStart`)** — opt-in handler that runs
the first turn's `streamText` step in your warm server process while the
agent run boots in parallel, cutting cold-start TTFC by roughly half
(measured 2801ms → 1218ms on `claude-sonnet-4-6`). The agent owns step
2+ (tool execution, persistence, hooks) so heavy deps stay where they
belong. Web Fetch handler works natively in Next.js, Hono, SvelteKit,
Remix, Workers, etc.; bridge to Express/Fastify/Koa via
`chat.toNodeListener`. New `@trigger.dev/sdk/chat-server` subpath.
- **Multi-turn durability via Sessions** — every chat is backed by a
durable Session that outlives any individual run. Conversations resume
across page refreshes, idle timeout, crashes, and deploys; `resume:
true` reconnects via `lastEventId` so clients only see new chunks.
`sessions.list` enumerates chats for inbox-style UIs.
- **Auto-accumulated history, delta-only wire** — the backend
accumulates the full conversation across turns; clients only ship the
new message each turn. Long chats never hit the 512 KiB body cap.
Register `hydrateMessages` to be the source of truth yourself.
- **Lifecycle hooks** — `onPreload`, `onChatStart`,
`onValidateMessages`, `hydrateMessages`, `onTurnStart`,
`onBeforeTurnComplete`, `onTurnComplete`, `onChatSuspend`,
`onChatResume` — for persistence, validation, and post-turn work.
- **Stop generation** — client-driven `transport.stopGeneration(chatId)`
aborts mid-stream; the run stays alive for the next message, partial
response is captured, and aborted parts (stuck `partial-call` tools,
in-progress reasoning) are auto-cleaned.
- **Tool approvals (HITL)** — tools with `needsApproval: true` pause
until the user approves or denies via `addToolApprovalResponse`. The
runtime reconciles the updated assistant message by ID and continues
`streamText`.
- **Steering and background injection** — `pendingMessages` injects user
messages between tool-call steps so users can steer the agent
mid-execution; `chat.inject()` + `chat.defer()` adds context from
background work (self-review, RAG, safety checks) between turns.
- **Actions** — non-turn frontend commands (undo, rollback, regenerate,
edit) sent via `transport.sendAction`. Fire `hydrateMessages` +
`onAction` only — no turn hooks, no `run()`. `onAction` can return a
`StreamTextResult` for a model response, or `void` for side-effect-only.
- **Typed state primitives** — `chat.local<T>` for per-run state
accessible from hooks, `run()`, tools, and subtasks (auto-serialized
through `ai.toolExecute`); `chat.store` for typed shared data between
agent and client; `chat.history` for reading and mutating the message
chain; `clientDataSchema` for typed `clientData` in every hook.
- **`chat.toStreamTextOptions()`** — one spread into `streamText` wires
up versioned system [Prompts](https://trigger.dev/docs/ai/prompts),
model resolution, telemetry metadata, compaction, steering, and
background injection.
- **Multi-tab coordination** — `multiTab: true` + `useMultiTabChat`
prevents duplicate sends and syncs state across browser tabs via
`BroadcastChannel`. Non-active tabs go read-only with live updates.
- **Network resilience** — built-in indefinite retry with bounded
backoff, reconnect on `online` / tab refocus / bfcache restore,
`Last-Event-ID` mid-stream resume. No app code needed.
- Retry `TASK_PROCESS_SIGSEGV` task crashes under the user's retry
policy instead of failing the run on the first segfault. SIGSEGV in Node
tasks is frequently non-deterministic (native addon races, JIT/GC
interaction, near-OOM in native code, host issues), so retrying on a
fresh process often succeeds. The retry is gated by the task's existing
`retry` config + `maxAttempts` — same path `TASK_PROCESS_SIGTERM` and
uncaught exceptions already use — so tasks without a retry policy still
fail fast.
([#3552](https://github.com/triggerdotdev/trigger.dev/pull/3552))
- The public interfaces for a plugin system. Initially consolidated
authentication and authorization interfaces.
([#3499](https://github.com/triggerdotdev/trigger.dev/pull/3499))
- Add MollifierBuffer and MollifierDrainer primitives for trigger burst
smoothing.
([#3614](https://github.com/triggerdotdev/trigger.dev/pull/3614))

## Bug fixes
- Fix `LocalsKey<T>` type incompatibility across dual-package builds.
The phantom value-type brand no longer uses a module-level `unique
symbol`, so a single TypeScript compilation that resolves the type from
both the ESM and CJS outputs (which can happen under certain pnpm
hoisting layouts) no longer sees two structurally-incompatible variants
of the same type.
([#3626](https://github.com/triggerdotdev/trigger.dev/pull/3626))

<details>
<summary>Raw changeset output</summary>

⚠️⚠️⚠️⚠️⚠️⚠️

`main` is currently in **pre mode** so this branch has prereleases
rather than normal releases. If you want to exit prereleases, run
`changeset pre exit` on `main`.

⚠️⚠️⚠️⚠️⚠️⚠️

# Releases
## @trigger.dev/sdk@4.5.0-rc.0

### Minor Changes

- **AI Prompts** — define prompt templates as code alongside your tasks,
version them on deploy, and override the text or model from the
dashboard without redeploying. Prompts integrate with the Vercel AI SDK
via `toAISDKTelemetry()` (links every generation span back to the
prompt) and with `chat.agent` via `chat.prompt.set()` +
`chat.toStreamTextOptions()`.
([#3629](https://github.com/triggerdotdev/trigger.dev/pull/3629))

    ```ts
    import { prompts } from "@trigger.dev/sdk";
    import { generateText } from "ai";
    import { openai } from "@ai-sdk/openai";
    import { z } from "zod";

    export const supportPrompt = prompts.define({
      id: "customer-support",
      model: "gpt-4o",
      config: { temperature: 0.7 },
      variables: z.object({
        customerName: z.string(),
        plan: z.string(),
        issue: z.string(),
      }),
      content: `You are a support agent for Acme.

    Customer: {{customerName}} ({{plan}} plan)
    Issue: {{issue}}`,
    });

    const resolved = await supportPrompt.resolve({
      customerName: "Alice",
      plan: "Pro",
      issue: "Can't access billing",
    });

    const result = await generateText({
      model: openai(resolved.model ?? "gpt-4o"),
      system: resolved.text,
      prompt: "Can't access billing",
      ...resolved.toAISDKTelemetry(),
    });
    ```

    **What you get:**

- **Code-defined, deploy-versioned templates** — define with
`prompts.define({ id, model, config, variables, content })`. Every
deploy creates a new version visible in the dashboard. Mustache-style
placeholders (`{{var}}`, `{{#cond}}...{{/cond}}`) with Zod / ArkType /
Valibot-typed variables.
- **Dashboard overrides** — change a prompt's text or model from the
dashboard without redeploying. Overrides take priority over the deployed
"current" version and are environment-scoped (dev / staging / production
independent).
- **Resolve API** — `prompt.resolve(vars, { version?, label? })` returns
the compiled `text`, resolved `model`, `version`, and labels. Standalone
`prompts.resolve<typeof handle>(slug, vars)` for cross-file resolution
with full type inference on slug and variable shape.
- **AI SDK integration** — spread `resolved.toAISDKTelemetry({ ...extra
})` into any `generateText` / `streamText` call and every generation
span links to the prompt in the dashboard alongside its input variables,
model, tokens, and cost.
- **`chat.agent` integration** — `chat.prompt.set(resolved)` stores the
resolved prompt run-scoped; `chat.toStreamTextOptions({ registry })`
pulls `system`, `model` (resolved via the AI SDK provider registry),
`temperature` / `maxTokens` / etc., and telemetry into a single spread
for `streamText`.
- **Management SDK** — `prompts.list()`, `prompts.versions(slug)`,
`prompts.promote(slug, version)`, `prompts.createOverride(slug, body)`,
`prompts.updateOverride(slug, body)`, `prompts.removeOverride(slug)`,
`prompts.reactivateOverride(slug, version)`.
- **Dashboard** — prompts list with per-prompt usage sparklines;
per-prompt detail with Template / Details / Versions / Generations /
Metrics tabs. AI generation spans get a custom inspector showing the
linked prompt's metadata, input variables, and template content
alongside model, tokens, cost, and the message thread.

See [/docs/ai/prompts](https://trigger.dev/docs/ai/prompts) for the full
reference — template syntax, version resolution order, override
workflow, and type utilities (`PromptHandle`, `PromptIdentifier`,
`PromptVariables`).

- Adds `onBoot` to `chat.agent` — a lifecycle hook that fires once per
worker process picking up the chat. Runs for the initial run, preloaded
runs, AND reactive continuation runs (post-cancel, crash, `endRun`,
`requestUpgrade`, OOM retry), before any other hook. Use it to
initialize `chat.local`, open per-process resources, or re-hydrate state
from your DB on continuation — anywhere the SAME run picking up after
suspend/resume isn't enough.
([#3543](https://github.com/triggerdotdev/trigger.dev/pull/3543))

    ```ts
const userContext = chat.local<{ name: string; plan: string }>({ id:
"userContext" });

    export const myChat = chat.agent({
      id: "my-chat",
      onBoot: async ({ clientData, continuation }) => {
const user = await db.user.findUnique({ where: { id: clientData.userId }
});
        userContext.init({ name: user.name, plan: user.plan });
      },
      run: async ({ messages, signal }) =>
streamText({ model: openai("gpt-4o"), messages, abortSignal: signal }),
    });
    ```

Use `onBoot` (not `onChatStart`) for state setup that must run every
time a worker picks up the chat — `onChatStart` fires once per chat and
won't run on continuation, leaving `chat.local` uninitialized when
`run()` tries to use it.

- **AI Agents** — run AI SDK chat completions as durable Trigger.dev
agents instead of fragile API routes. Define an agent in one function,
point `useChat` at it from React, and the conversation survives page
refreshes, network blips, and process restarts.
([#3543](https://github.com/triggerdotdev/trigger.dev/pull/3543))

    ```ts
    import { chat } from "@trigger.dev/sdk/ai";
    import { streamText } from "ai";
    import { openai } from "@ai-sdk/openai";

    export const myChat = chat.agent({
      id: "my-chat",
      run: async ({ messages, signal }) =>
streamText({ model: openai("gpt-4o"), messages, abortSignal: signal }),
    });
    ```

    ```tsx
    import { useChat } from "@ai-sdk/react";
import { useTriggerChatTransport } from "@trigger.dev/sdk/chat/react";

const transport = useTriggerChatTransport({ task: "my-chat",
accessToken, startSession });
    const { messages, sendMessage } = useChat({ transport });
    ```

    **What you get:**

- **AI SDK `useChat` integration** — a custom
[`ChatTransport`](https://sdk.vercel.ai/docs/ai-sdk-ui/transport)
(`useTriggerChatTransport`) plugs straight into Vercel AI SDK's
`useChat` hook. Text streaming, tool calls, reasoning, and `data-*`
parts all work natively over Trigger.dev's realtime streams. No custom
API routes needed.
- **First-turn fast path (`chat.headStart`)** — opt-in handler that runs
the first turn's `streamText` step in your warm server process while the
agent run boots in parallel, cutting cold-start TTFC by roughly half
(measured 2801ms → 1218ms on `claude-sonnet-4-6`). The agent owns step
2+ (tool execution, persistence, hooks) so heavy deps stay where they
belong. Web Fetch handler works natively in Next.js, Hono, SvelteKit,
Remix, Workers, etc.; bridge to Express/Fastify/Koa via
`chat.toNodeListener`. New `@trigger.dev/sdk/chat-server` subpath.
- **Multi-turn durability via Sessions** — every chat is backed by a
durable Session that outlives any individual run. Conversations resume
across page refreshes, idle timeout, crashes, and deploys; `resume:
true` reconnects via `lastEventId` so clients only see new chunks.
`sessions.list` enumerates chats for inbox-style UIs.
- **Auto-accumulated history, delta-only wire** — the backend
accumulates the full conversation across turns; clients only ship the
new message each turn. Long chats never hit the 512 KiB body cap.
Register `hydrateMessages` to be the source of truth yourself.
- **Lifecycle hooks** — `onPreload`, `onChatStart`,
`onValidateMessages`, `hydrateMessages`, `onTurnStart`,
`onBeforeTurnComplete`, `onTurnComplete`, `onChatSuspend`,
`onChatResume` — for persistence, validation, and post-turn work.
- **Stop generation** — client-driven `transport.stopGeneration(chatId)`
aborts mid-stream; the run stays alive for the next message, partial
response is captured, and aborted parts (stuck `partial-call` tools,
in-progress reasoning) are auto-cleaned.
- **Tool approvals (HITL)** — tools with `needsApproval: true` pause
until the user approves or denies via `addToolApprovalResponse`. The
runtime reconciles the updated assistant message by ID and continues
`streamText`.
- **Steering and background injection** — `pendingMessages` injects user
messages between tool-call steps so users can steer the agent
mid-execution; `chat.inject()` + `chat.defer()` adds context from
background work (self-review, RAG, safety checks) between turns.
- **Actions** — non-turn frontend commands (undo, rollback, regenerate,
edit) sent via `transport.sendAction`. Fire `hydrateMessages` +
`onAction` only — no turn hooks, no `run()`. `onAction` can return a
`StreamTextResult` for a model response, or `void` for side-effect-only.
- **Typed state primitives** — `chat.local<T>` for per-run state
accessible from hooks, `run()`, tools, and subtasks (auto-serialized
through `ai.toolExecute`); `chat.store` for typed shared data between
agent and client; `chat.history` for reading and mutating the message
chain; `clientDataSchema` for typed `clientData` in every hook.
- **`chat.toStreamTextOptions()`** — one spread into `streamText` wires
up versioned system [Prompts](https://trigger.dev/docs/ai/prompts),
model resolution, telemetry metadata, compaction, steering, and
background injection.
- **Multi-tab coordination** — `multiTab: true` + `useMultiTabChat`
prevents duplicate sends and syncs state across browser tabs via
`BroadcastChannel`. Non-active tabs go read-only with live updates.
- **Network resilience** — built-in indefinite retry with bounded
backoff, reconnect on `online` / tab refocus / bfcache restore,
`Last-Event-ID` mid-stream resume. No app code needed.

See [/docs/ai-chat](https://trigger.dev/docs/ai-chat/overview) for the
full surface — quick start, three backend approaches (`chat.agent`,
`chat.createSession`, raw task), persistence and code-sandbox patterns,
type-level guides, and API reference.

- Add read primitives to `chat.history` for HITL flows:
`getPendingToolCalls()`, `getResolvedToolCalls()`,
`extractNewToolResults(message)`, `getChain()`, and
`findMessage(messageId)`. These lift the accumulator-walking logic that
customers building human-in-the-loop tools were re-implementing into the
SDK. ([#3543](https://github.com/triggerdotdev/trigger.dev/pull/3543))

Use `getPendingToolCalls()` to gate fresh user turns while a tool call
is awaiting an answer. Use `extractNewToolResults(message)` to dedup
tool results when persisting to your own store — the helper returns only
the parts whose `toolCallId` is not already resolved on the chain.

    ```ts
    const pending = chat.history.getPendingToolCalls();
    if (pending.length > 0) {
      // an addToolOutput is expected before a new user message
    }

    onTurnComplete: async ({ responseMessage }) => {
const newResults = chat.history.extractNewToolResults(responseMessage);
      for (const r of newResults) {
await db.toolResults.upsert({ id: r.toolCallId, output: r.output,
errorText: r.errorText });
      }
    };
    ```

- **Sessions** — a durable, run-aware stream channel keyed on a stable
`externalId`. A Session is the unit of state that owns a multi-run
conversation: messages flow through `.in`, responses through `.out`,
both survive run boundaries. Sessions back the new `chat.agent` runtime,
and you can build on them directly for any pattern that needs durable
bi-directional streaming across runs.
([#3542](https://github.com/triggerdotdev/trigger.dev/pull/3542))

    ```ts
    import { sessions, tasks } from "@trigger.dev/sdk";

    // Trigger a task and subscribe to its session output in one call
const { runId, stream } = await tasks.triggerAndSubscribe("my-task",
payload, {
      externalId: "user-456",
    });

    for await (const chunk of stream) {
      // ...
    }

// Enumerate existing sessions (powers inbox-style UIs without a
separate index)
for await (const s of sessions.list({ type: "chat.agent", tag:
"user:user-456" })) {
      console.log(s.id, s.externalId, s.createdAt, s.closedAt);
    }
    ```

See [/docs/ai-chat/overview](https://trigger.dev/docs/ai-chat/overview)
for the full surface — Sessions powers the durable, resumable chat
runtime described there.

### Patch Changes

- Add Agent Skills for `chat.agent`. Drop a folder with a `SKILL.md` and
any helper scripts/references next to your task code, register it with
`skills.define({ id, path })`, and the CLI bundles it into the deploy
image automatically — no `trigger.config.ts` changes. The agent gets a
one-line summary in its system prompt and discovers full instructions on
demand via `loadSkill`, with `bash` and `readFile` tools scoped
per-skill (path-traversal guards, output caps, abort-signal
propagation).
([#3543](https://github.com/triggerdotdev/trigger.dev/pull/3543))

    ```ts
const pdfSkill = skills.define({ id: "pdf-extract", path:
"./skills/pdf-extract" });

    chat.skills.set([await pdfSkill.local()]);
    ```

Built on the [AI SDK cookbook
pattern](https://ai-sdk.dev/cookbook/guides/agent-skills) — portable
across providers. SDK + CLI only for now; dashboard-editable `SKILL.md`
text is on the roadmap.

- Add `ai.toolExecute(task)` so you can wire a Trigger subtask in as the
`execute` handler of an AI SDK `tool()` while defining `description` and
`inputSchema` yourself — useful when you want full control over the tool
surface and just need Trigger's subtask machinery for the body.
([#3546](https://github.com/triggerdotdev/trigger.dev/pull/3546))

    ```ts
    const myTool = tool({
      description: "...",
      inputSchema: z.object({ ... }),
      execute: ai.toolExecute(mySubtask),
    });
    ```

`ai.tool(task)` (`toolFromTask`) keeps doing the all-in-one wrap and now
aligns its return type with AI SDK's `ToolSet`. Minimum `ai` peer raised
to `^6.0.116` to avoid cross-version `ToolSet` mismatches in monorepos.

- Stamp `gen_ai.conversation.id` (the chat id) on every span and metric
emitted from inside a `chat.task` or `chat.agent` run. Lets you filter
dashboard spans, runs, and metrics by the chat conversation that
produced them — independent of the run boundary, so multi-run chats
correlate cleanly. No code changes required on the user side.
([#3543](https://github.com/triggerdotdev/trigger.dev/pull/3543))

- Type `chat.createStartSessionAction` against your chat agent so
`clientData` is typed end-to-end on the first turn:
([#3684](https://github.com/triggerdotdev/trigger.dev/pull/3684))

    ```ts
    import { chat } from "@trigger.dev/sdk/ai";
    import type { myChat } from "@/trigger/chat";

export const startChatSession = chat.createStartSessionAction<typeof
myChat>("my-chat");

// In the browser, threaded from the transport's typed startSession
callback:
    const transport = useTriggerChatTransport<typeof myChat>({
      task: "my-chat",
startSession: ({ chatId, clientData }) => startChatSession({ chatId,
clientData }),
      // ...
    });
    ```

`ChatStartSessionParams` gains a typed `clientData` field — folded into
the first run's `payload.metadata` so `onPreload` / `onChatStart` see
the same shape per-turn `metadata` carries via the transport. The opaque
session-level `metadata` field is unchanged.

- Unit-test `chat.agent` definitions offline with `mockChatAgent` from
`@trigger.dev/sdk/ai/test`. Drives a real agent's turn loop in-process —
no network, no task runtime — so you can send messages, actions, and
stop signals via driver methods, inspect captured output chunks, and
verify hooks fire. Pairs with `MockLanguageModelV3` from `ai/test` for
model mocking. `setupLocals` lets you pre-seed `locals` (DB clients,
service stubs) before `run()` starts.
([#3543](https://github.com/triggerdotdev/trigger.dev/pull/3543))

The broader `runInMockTaskContext` harness it's built on lives at
`@trigger.dev/core/v3/test` — useful for unit-testing any task code, not
just chat.

- Add `region` to the runs list / retrieve API: filter runs by region
(`runs.list({ region: "..." })` / `filter[region]=<masterQueue>`) and
read each run's executing region from the new `region` field on the
response.
([#3612](https://github.com/triggerdotdev/trigger.dev/pull/3612))

-   Updated dependencies:
    -   `@trigger.dev/core@4.5.0-rc.0`

## @trigger.dev/build@4.5.0-rc.0

### Patch Changes

- Add Agent Skills for `chat.agent`. Drop a folder with a `SKILL.md` and
any helper scripts/references next to your task code, register it with
`skills.define({ id, path })`, and the CLI bundles it into the deploy
image automatically — no `trigger.config.ts` changes. The agent gets a
one-line summary in its system prompt and discovers full instructions on
demand via `loadSkill`, with `bash` and `readFile` tools scoped
per-skill (path-traversal guards, output caps, abort-signal
propagation).
([#3543](https://github.com/triggerdotdev/trigger.dev/pull/3543))

    ```ts
const pdfSkill = skills.define({ id: "pdf-extract", path:
"./skills/pdf-extract" });

    chat.skills.set([await pdfSkill.local()]);
    ```

Built on the [AI SDK cookbook
pattern](https://ai-sdk.dev/cookbook/guides/agent-skills) — portable
across providers. SDK + CLI only for now; dashboard-editable `SKILL.md`
text is on the roadmap.

-   Updated dependencies:
    -   `@trigger.dev/core@4.5.0-rc.0`

## trigger.dev@4.5.0-rc.0

### Patch Changes

- Add Agent Skills for `chat.agent`. Drop a folder with a `SKILL.md` and
any helper scripts/references next to your task code, register it with
`skills.define({ id, path })`, and the CLI bundles it into the deploy
image automatically — no `trigger.config.ts` changes. The agent gets a
one-line summary in its system prompt and discovers full instructions on
demand via `loadSkill`, with `bash` and `readFile` tools scoped
per-skill (path-traversal guards, output caps, abort-signal
propagation).
([#3543](https://github.com/triggerdotdev/trigger.dev/pull/3543))

    ```ts
const pdfSkill = skills.define({ id: "pdf-extract", path:
"./skills/pdf-extract" });

    chat.skills.set([await pdfSkill.local()]);
    ```

Built on the [AI SDK cookbook
pattern](https://ai-sdk.dev/cookbook/guides/agent-skills) — portable
across providers. SDK + CLI only for now; dashboard-editable `SKILL.md`
text is on the roadmap.

- Add `TRIGGER_BUILD_SKIP_REWRITE_TIMESTAMP=1` escape hatch for local
self-hosted builds whose buildx driver doesn't support
`rewrite-timestamp` alongside push (e.g. orbstack's default `docker`
driver).
([#3618](https://github.com/triggerdotdev/trigger.dev/pull/3618))

- The CLI MCP server's agent-chat tools (`start_agent_chat`,
`send_agent_message`, `close_agent_chat`) now run on the new Sessions
primitive, so AI assistants driving a `chat.agent` get the same
idempotent-by-`chatId`, durable-across-runs behavior the browser
transport gets. Required PAT scopes go from `write:inputStreams` to
`read:sessions` + `write:sessions`.
([#3546](https://github.com/triggerdotdev/trigger.dev/pull/3546))

- MCP `list_runs` tool: add a `region` filter input and surface each
run's executing region in the formatted summary.
([#3612](https://github.com/triggerdotdev/trigger.dev/pull/3612))

-   Updated dependencies:
    -   `@trigger.dev/core@4.5.0-rc.0`
    -   `@trigger.dev/build@4.5.0-rc.0`
    -   `@trigger.dev/schema-to-json@4.5.0-rc.0`

## @trigger.dev/core@4.5.0-rc.0

### Patch Changes

- Add Agent Skills for `chat.agent`. Drop a folder with a `SKILL.md` and
any helper scripts/references next to your task code, register it with
`skills.define({ id, path })`, and the CLI bundles it into the deploy
image automatically — no `trigger.config.ts` changes. The agent gets a
one-line summary in its system prompt and discovers full instructions on
demand via `loadSkill`, with `bash` and `readFile` tools scoped
per-skill (path-traversal guards, output caps, abort-signal
propagation).
([#3543](https://github.com/triggerdotdev/trigger.dev/pull/3543))

    ```ts
const pdfSkill = skills.define({ id: "pdf-extract", path:
"./skills/pdf-extract" });

    chat.skills.set([await pdfSkill.local()]);
    ```

Built on the [AI SDK cookbook
pattern](https://ai-sdk.dev/cookbook/guides/agent-skills) — portable
across providers. SDK + CLI only for now; dashboard-editable `SKILL.md`
text is on the roadmap.

- Reject overlong `idempotencyKey` values at the API boundary so they no
longer trip an internal size limit on the underlying unique index and
surface as a generic 500. Inputs are capped at 2048 characters — well
above what `idempotencyKeys.create()` produces (a 64-character hash) and
above any realistic raw key. Applies to `tasks.trigger`,
`tasks.batchTrigger`, `batch.create` (Phase 1 streaming batches),
`wait.createToken`, `wait.forDuration`, and the input/session stream
waitpoint endpoints. Over-limit requests now return a structured 400
instead.
([#3560](https://github.com/triggerdotdev/trigger.dev/pull/3560))

- **AI Agents** — run AI SDK chat completions as durable Trigger.dev
agents instead of fragile API routes. Define an agent in one function,
point `useChat` at it from React, and the conversation survives page
refreshes, network blips, and process restarts.
([#3543](https://github.com/triggerdotdev/trigger.dev/pull/3543))

    ```ts
    import { chat } from "@trigger.dev/sdk/ai";
    import { streamText } from "ai";
    import { openai } from "@ai-sdk/openai";

    export const myChat = chat.agent({
      id: "my-chat",
      run: async ({ messages, signal }) =>
streamText({ model: openai("gpt-4o"), messages, abortSignal: signal }),
    });
    ```

    ```tsx
    import { useChat } from "@ai-sdk/react";
import { useTriggerChatTransport } from "@trigger.dev/sdk/chat/react";

const transport = useTriggerChatTransport({ task: "my-chat",
accessToken, startSession });
    const { messages, sendMessage } = useChat({ transport });
    ```

    **What you get:**

- **AI SDK `useChat` integration** — a custom
[`ChatTransport`](https://sdk.vercel.ai/docs/ai-sdk-ui/transport)
(`useTriggerChatTransport`) plugs straight into Vercel AI SDK's
`useChat` hook. Text streaming, tool calls, reasoning, and `data-*`
parts all work natively over Trigger.dev's realtime streams. No custom
API routes needed.
- **First-turn fast path (`chat.headStart`)** — opt-in handler that runs
the first turn's `streamText` step in your warm server process while the
agent run boots in parallel, cutting cold-start TTFC by roughly half
(measured 2801ms → 1218ms on `claude-sonnet-4-6`). The agent owns step
2+ (tool execution, persistence, hooks) so heavy deps stay where they
belong. Web Fetch handler works natively in Next.js, Hono, SvelteKit,
Remix, Workers, etc.; bridge to Express/Fastify/Koa via
`chat.toNodeListener`. New `@trigger.dev/sdk/chat-server` subpath.
- **Multi-turn durability via Sessions** — every chat is backed by a
durable Session that outlives any individual run. Conversations resume
across page refreshes, idle timeout, crashes, and deploys; `resume:
true` reconnects via `lastEventId` so clients only see new chunks.
`sessions.list` enumerates chats for inbox-style UIs.
- **Auto-accumulated history, delta-only wire** — the backend
accumulates the full conversation across turns; clients only ship the
new message each turn. Long chats never hit the 512 KiB body cap.
Register `hydrateMessages` to be the source of truth yourself.
- **Lifecycle hooks** — `onPreload`, `onChatStart`,
`onValidateMessages`, `hydrateMessages`, `onTurnStart`,
`onBeforeTurnComplete`, `onTurnComplete`, `onChatSuspend`,
`onChatResume` — for persistence, validation, and post-turn work.
- **Stop generation** — client-driven `transport.stopGeneration(chatId)`
aborts mid-stream; the run stays alive for the next message, partial
response is captured, and aborted parts (stuck `partial-call` tools,
in-progress reasoning) are auto-cleaned.
- **Tool approvals (HITL)** — tools with `needsApproval: true` pause
until the user approves or denies via `addToolApprovalResponse`. The
runtime reconciles the updated assistant message by ID and continues
`streamText`.
- **Steering and background injection** — `pendingMessages` injects user
messages between tool-call steps so users can steer the agent
mid-execution; `chat.inject()` + `chat.defer()` adds context from
background work (self-review, RAG, safety checks) between turns.
- **Actions** — non-turn frontend commands (undo, rollback, regenerate,
edit) sent via `transport.sendAction`. Fire `hydrateMessages` +
`onAction` only — no turn hooks, no `run()`. `onAction` can return a
`StreamTextResult` for a model response, or `void` for side-effect-only.
- **Typed state primitives** — `chat.local<T>` for per-run state
accessible from hooks, `run()`, tools, and subtasks (auto-serialized
through `ai.toolExecute`); `chat.store` for typed shared data between
agent and client; `chat.history` for reading and mutating the message
chain; `clientDataSchema` for typed `clientData` in every hook.
- **`chat.toStreamTextOptions()`** — one spread into `streamText` wires
up versioned system [Prompts](https://trigger.dev/docs/ai/prompts),
model resolution, telemetry metadata, compaction, steering, and
background injection.
- **Multi-tab coordination** — `multiTab: true` + `useMultiTabChat`
prevents duplicate sends and syncs state across browser tabs via
`BroadcastChannel`. Non-active tabs go read-only with live updates.
- **Network resilience** — built-in indefinite retry with bounded
backoff, reconnect on `online` / tab refocus / bfcache restore,
`Last-Event-ID` mid-stream resume. No app code needed.

See [/docs/ai-chat](https://trigger.dev/docs/ai-chat/overview) for the
full surface — quick start, three backend approaches (`chat.agent`,
`chat.createSession`, raw task), persistence and code-sandbox patterns,
type-level guides, and API reference.

- Stamp `gen_ai.conversation.id` (the chat id) on every span and metric
emitted from inside a `chat.task` or `chat.agent` run. Lets you filter
dashboard spans, runs, and metrics by the chat conversation that
produced them — independent of the run boundary, so multi-run chats
correlate cleanly. No code changes required on the user side.
([#3543](https://github.com/triggerdotdev/trigger.dev/pull/3543))

- Fix `LocalsKey<T>` type incompatibility across dual-package builds.
The phantom value-type brand no longer uses a module-level `unique
symbol`, so a single TypeScript compilation that resolves the type from
both the ESM and CJS outputs (which can happen under certain pnpm
hoisting layouts) no longer sees two structurally-incompatible variants
of the same type.
([#3626](https://github.com/triggerdotdev/trigger.dev/pull/3626))

- Unit-test `chat.agent` definitions offline with `mockChatAgent` from
`@trigger.dev/sdk/ai/test`. Drives a real agent's turn loop in-process —
no network, no task runtime — so you can send messages, actions, and
stop signals via driver methods, inspect captured output chunks, and
verify hooks fire. Pairs with `MockLanguageModelV3` from `ai/test` for
model mocking. `setupLocals` lets you pre-seed `locals` (DB clients,
service stubs) before `run()` starts.
([#3543](https://github.com/triggerdotdev/trigger.dev/pull/3543))

The broader `runInMockTaskContext` harness it's built on lives at
`@trigger.dev/core/v3/test` — useful for unit-testing any task code, not
just chat.

- Retry `TASK_PROCESS_SIGSEGV` task crashes under the user's retry
policy instead of failing the run on the first segfault. SIGSEGV in Node
tasks is frequently non-deterministic (native addon races, JIT/GC
interaction, near-OOM in native code, host issues), so retrying on a
fresh process often succeeds. The retry is gated by the task's existing
`retry` config + `maxAttempts` — same path `TASK_PROCESS_SIGTERM` and
uncaught exceptions already use — so tasks without a retry policy still
fail fast.
([#3552](https://github.com/triggerdotdev/trigger.dev/pull/3552))

- Add `region` to the runs list / retrieve API: filter runs by region
(`runs.list({ region: "..." })` / `filter[region]=<masterQueue>`) and
read each run's executing region from the new `region` field on the
response.
([#3612](https://github.com/triggerdotdev/trigger.dev/pull/3612))

- **Sessions** — a durable, run-aware stream channel keyed on a stable
`externalId`. A Session is the unit of state that owns a multi-run
conversation: messages flow through `.in`, responses through `.out`,
both survive run boundaries. Sessions back the new `chat.agent` runtime,
and you can build on them directly for any pattern that needs durable
bi-directional streaming across runs.
([#3542](https://github.com/triggerdotdev/trigger.dev/pull/3542))

    ```ts
    import { sessions, tasks } from "@trigger.dev/sdk";

    // Trigger a task and subscribe to its session output in one call
const { runId, stream } = await tasks.triggerAndSubscribe("my-task",
payload, {
      externalId: "user-456",
    });

    for await (const chunk of stream) {
      // ...
    }

// Enumerate existing sessions (powers inbox-style UIs without a
separate index)
for await (const s of sessions.list({ type: "chat.agent", tag:
"user:user-456" })) {
      console.log(s.id, s.externalId, s.createdAt, s.closedAt);
    }
    ```

See [/docs/ai-chat/overview](https://trigger.dev/docs/ai-chat/overview)
for the full surface — Sessions powers the durable, resumable chat
runtime described there.

## @trigger.dev/plugins@4.5.0-rc.0

### Patch Changes

- The public interfaces for a plugin system. Initially consolidated
authentication and authorization interfaces.
([#3499](https://github.com/triggerdotdev/trigger.dev/pull/3499))
-   Updated dependencies:
    -   `@trigger.dev/core@4.5.0-rc.0`

## @trigger.dev/python@4.5.0-rc.0

### Patch Changes

-   Updated dependencies:
    -   `@trigger.dev/sdk@4.5.0-rc.0`
    -   `@trigger.dev/core@4.5.0-rc.0`
    -   `@trigger.dev/build@4.5.0-rc.0`

## @trigger.dev/react-hooks@4.5.0-rc.0

### Patch Changes

-   Updated dependencies:
    -   `@trigger.dev/core@4.5.0-rc.0`

## @trigger.dev/redis-worker@4.5.0-rc.0

### Patch Changes

- Add MollifierBuffer and MollifierDrainer primitives for trigger burst
smoothing.
([#3614](https://github.com/triggerdotdev/trigger.dev/pull/3614))

MollifierBuffer (`accept`, `pop`, `ack`, `requeue`, `fail`,
`evaluateTrip`) is a per-env FIFO over Redis with atomic Lua transitions
for status tracking. `evaluateTrip` is a sliding-window trip evaluator
the webapp gate uses to detect per-env trigger bursts.

MollifierDrainer pops entries through a polling loop with a
user-supplied handler. The loop survives transient Redis errors via
capped exponential backoff (up to 5s), and per-env pop failures don't
poison the rest of the batch — one env's blip is logged and counted as
failed for that tick. Rotation is two-level: orgs at the top, envs
within each org. The buffer maintains `mollifier:orgs` and
`mollifier:org-envs:${orgId}` atomically with per-env queues, so the
drainer walks orgs → envs directly without an in-memory cache. The
`maxOrgsPerTick` option (default 500) caps how many orgs are scheduled
per tick; for each picked org, one env is popped (rotating round-robin
within the org). An org with N envs gets the same per-tick scheduling
slot as an org with 1 env, so tenant-level drainage throughput is
determined by org count rather than env count.

-   Updated dependencies:
    -   `@trigger.dev/core@4.5.0-rc.0`

## @trigger.dev/rsc@4.5.0-rc.0

### Patch Changes

-   Updated dependencies:
    -   `@trigger.dev/core@4.5.0-rc.0`

## @trigger.dev/schema-to-json@4.5.0-rc.0

### Patch Changes

-   Updated dependencies:
    -   `@trigger.dev/core@4.5.0-rc.0`

</details>

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
v.docker.4.5.0-rc.0 v4.5.0-rc.0 helm-v4.5.0-rc.0 		2026-05-21 14:39:02 +01:00
Eric Allam 422f9f0bb2 ci: unblock changesets release PRs (#3687) 
## Summary

Two CI workflows were blocking the v4.5.0-rc.0 release PR (#3563) and
would block every future changeset release PR.

### 1. `changesets-pr.yml` — self-report `All PR Checks`

The changesets bot pushes commits authored by `GITHUB_TOKEN`. By GitHub
design, `GITHUB_TOKEN`-authored pushes can't trigger downstream
workflows (loop-prevention). That means `pr_checks.yml` never fires on
release-PR commits, leaving the required `All PR Checks` status
permanently `Expected — Waiting for status to be reported`. The PR can't
merge.

The fix: after `changesets/action` creates the PR, post a `success`
check with the exact `All PR Checks` context onto the PR's head SHA.
GitHub's required-check evaluation is satisfied by any check with the
right context name — the source doesn't have to be `pr_checks.yml`.

**Why this is safe:** the release PR only mechanically bumps
`package.json`, `pnpm-lock.yaml`, and `CHANGELOG.md` from changesets
that were already on `main` (and already ran full CI when they merged).
If a human ever pushes a commit to `changeset-release/main`,
`pr_checks.yml` fires on that push (real user, not `GITHUB_TOKEN`) and
posts its own `All PR Checks` status — last write wins for the same
context on the same SHA, so the human-push result overrides the
auto-success.

### 2. `vouch-check-pr.yml` — exempt `github-actions[bot]`

The `require-draft` job auto-closes any non-draft PR whose author is not
a `MEMBER`/`OWNER`/`COLLABORATOR`, with an explicit allowlist for
`devin-ai-integration[bot]` and `dependabot[bot]`. The changesets bot
publishes as `github-actions[bot]` with `author_association:
CONTRIBUTOR`, so every release PR was getting auto-closed on open with a
"please re-open as draft" comment. Add `github-actions[bot]` to the
exemption list.

## Test plan
- [ ] After merge, the next changeset bot push to
`changeset-release/main` should post `All PR Checks: success` on the
release PR's head SHA, and the PR should not get auto-closed by `Vouch -
Check PR`.
- [ ] Confirm `pr_checks.yml` still fires + gates normal
(human-authored) PRs to `main`.
 2026-05-21 14:22:52 +01:00
Eric Allam 89d085a496 fix(references): repair ai-chat typecheck against current wire shape (#3685) 
## Summary

Pre-existing typecheck errors in `references/ai-chat` against the
current SDK shape. Unblocks `pnpm exec tsc --noEmit` in the reference
project.

## What changed

Three categories of fixes inside `references/ai-chat`. No SDK changes.

### 1. `payload.messages` → `payload.message`

The wire payload is now delta-only — one new message per trigger,
optional. Old code in two raw-task files reads `payload.messages`
(plural array) which no longer exists.

```ts
// before
const messages = await conversation.addIncoming(currentPayload.messages, ...);

// after
const messages = await conversation.addIncoming(
  currentPayload.message ? [currentPayload.message] : [],
  ...
);
```

Same fix to the `chat.messages.on` handler, reading `msg.message`
(singular) instead of `msg.messages[length - 1]`.

### 2. `clientData` non-null assertion in `cf-trust-test`

`ChatTurnContext.clientData` is typed as `?: TClientData` on
`onTurnStart` / `run` event objects even when the agent declares a
`clientDataSchema`. The runtime validates against the schema before the
hook fires, so it's structurally non-null — but TypeScript can't know
that. Non-null assert for now.

Follow-up worth filing: narrow `ChatTurnContext.clientData` to
non-optional when the agent has a `clientDataSchema`. Same friction the
docs friction-test subagent flagged.

### 3. `stress-emit.parseConfig` retyped against `ModelMessage[]`

The `run` callback hands `messages: ModelMessage[]`, not `UIMessage[]`.
Update `parseConfig` to accept `ModelMessage[]` and pull text from
`content` (string or array-of-parts).

## Test plan

- [x] `pnpm exec tsc --noEmit` in `references/ai-chat` passes (was 8
errors, now 0)
 2026-05-21 12:41:18 +01:00
Eric Allam 9ff410bfa4 feat(sdk): type chat.createStartSessionAction against your chat agent (#3684) 
## Summary

Type `chat.createStartSessionAction` against the chat agent so
`clientData` is typed end-to-end on the first turn. Closes the gap where
`useTriggerChatTransport`'s `startSession` callback already hands you a
typed `clientData` (via the transport generic) but the server-side
action couldn't accept it without untyped routing through the `metadata`
field.

## Design

`ChatStartSessionParams` gains a typed `clientData` field via the new
generic:

```ts
export type ChatStartSessionParams<TChat extends AnyTask = AnyTask> = {
  chatId: string;
  clientData?: InferChatClientData<TChat>;
  triggerConfig?: Partial<SessionTriggerConfig>;
  metadata?: Record<string, unknown>;
};

function createChatStartSessionAction<TChat extends AnyTask = AnyTask>(
  taskId: string,
  options?: CreateChatStartSessionActionOptions
): (params: ChatStartSessionParams<TChat>) => Promise<ChatStartSessionResult>
```

When provided, `clientData` is folded into the first run's
`triggerConfig.basePayload.metadata`, so `onPreload` / `onChatStart` see
the same shape per-turn `metadata` carries via the transport. The opaque
session-level `metadata` field stays exactly as before — it lands on the
Session row, not the run payload.

## Usage

```ts
// actions.ts
import { chat } from "@trigger.dev/sdk/ai";
import type { myChat } from "@/trigger/chat";

export const startChatSession = chat.createStartSessionAction<typeof myChat>("my-chat");
```

```tsx
// Chat.tsx
const transport = useTriggerChatTransport<typeof myChat>({
  task: "my-chat",
  accessToken: ({ chatId }) => mintChatAccessToken(chatId),
  startSession: ({ chatId, clientData }) =>
    startChatSession({ chatId, clientData }),
});
```

## Test plan

- [x] `pnpm run build --filter @trigger.dev/sdk` passes
- [ ] Verify a `chat.agent` with `clientDataSchema` reads the typed
clientData from `onPreload` payload metadata on the first turn
 2026-05-21 11:51:35 +01:00