github-featured/triggerdotdev--trigger.dev
2
0
Fork 0
Code Issues Pull Requests Deployments Models Agent Notes Packages Projects Releases Wiki Activity
7,274 Commits 300 Branches 4,167 Tags
c043c4a6ad36e1bfb5257efd54ca54605ad8a37f
Commit Graph

7274 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
nicktrn c043c4a6ad fix(core): external trace context leaks across warm-started runs (#3768) 
On warm-started workers with `processKeepAlive` enabled, every run's
attempt span was exported with the first run's `traceId` and
`parentSpanId`. `ExternalSpanExporterWrapper` and
`ExternalLogRecordExporterWrapper` captured `externalTraceContext` at
`TracingSDK` construction, and the SDK is memoized for the worker's
lifetime - so the per-run reset of
`StandardTraceContextManager.traceContext` never reached the wrappers.

Reported by a customer running v4.4.x: 33 distinct runs on the same
host/pid showed up in their APM as siblings of one parent span.

Fix: the wrappers now read external context live from the trace context
manager per export. Runs without an external trace context fall through
to the unchanged `externalTraceId` fallback - no behaviour change for
them.

Regression test in
`packages/core/test/externalSpanExporterWrapper.test.ts` asserts that
reassigning the manager between exports produces correctly-parented
spans.
 2026-05-28 10:43:39 +00:00
Matt Aitken 816986d44e fix(webapp): treat Phase 2 batch-stream retries as idempotent (TRI-9944) (#3766) 
Returns sealed:true when a fast-completing batchTrigger races
the stream finalisation, instead of throwing 422/BatchTriggerError.
 2026-05-28 09:28:18 +00:00
Iss 5083d161b2 docs: adds relevant env vars to self hosting docs (#3148) 2026-05-27 12:25:18 -04:00
Iss 8399aa2052 fix(core): retry TASK_MIDDLEWARE_ERROR under the task's retry policy (#3676) 
Retries `TASK_MIDDLEWARE_ERROR` under the task's retry policy.
`shouldRetryError` already classed it as retryable, but
`shouldLookupRetrySettings` did not, so the run fell through to
`fail_run` on attempt 1 instead of using the task's `retry` config.
Fixes #3231.
 2026-05-27 17:17:50 +01:00
Iss 7324a3307f chore(webapp): rename support tier to "Priority" on plan cards and li… (#3692) 2026-05-27 12:17:40 -04:00
Iss df96a937db docs: troubleshoot "Stream is being deleted" during long waits (#3704) 2026-05-27 12:17:21 -04:00
Matt Aitken 90bbbd1c44 fix(webapp): recover from ClickHouse JSON parse failures on out-of-range integers (#3759) 
## Summary

Second class of poisoned-row failure in the runs replication path. PR
#3708 plugged lone UTF-16 surrogates; this one handles bare JSON integer
literals outside ClickHouse's `Int64`..`UInt64` range. Recovery stays
purely reactive — the existing `sanitizeRows` walker just gains an extra
branch, so the hot replication path pays nothing on healthy rows.

Fixes the still-firing customer-facing symptom from
[TRI-9755](https://linear.app/triggerdotdev/issue/TRI-9755):
`scan-social-profiles` runs continued to be stranded in `EXECUTING` on
the Tasks page after #3708 deployed. CloudWatch showed `Dropped batch —
ClickHouse JSON parse error but sanitizer found nothing to fix` firing
**8/8 times** since the previous deploy (zero successful sanitizations).
Root cause: upstream JS Number precision loss on a 21-digit Google Plus
ID (`117039831458782873093` → `117039831458782870000`) — the
precision-lossy value still serialises as a bare integer that exceeds
`UInt64.MAX`, which ClickHouse rejects with `INCORRECT_DATA`.

## How the bug ships

The customer task emits an output containing a Poshmark profile's
`spec_format`:

```json
{"key":"gp_id","proper_key":"Gp Id","value":117039831458782870000,"type":"int"}
```

That value is `1.17e20` — comfortably above `UInt64.MAX` (`1.84e19`) but
comfortably below `1e21`. `Number.prototype.toString` only switches to
exponential form at `|value| >= 1e21`, so `JSON.stringify` emits the
bare token `117039831458782870000` and the ClickHouse
`JSON(max_dynamic_paths)` column fails with:

```
Code: 117. DB::Exception: Cannot parse JSON object here: {…}: (while reading the value of key output): (at row 1)
: While executing ParallelParsingBlockInputFormat. (INCORRECT_DATA) (version 25.12.x)
```

Same error verbatim as prod. The same number quoted
(`"117039831458782870000"`) inserts fine — ClickHouse's dynamic JSON
column accepts a `String` subtype on the same path.

## What changed

`apps/webapp/app/v3/eventRepository/sanitizeRowsOnParseError.server.ts`:

- New private `isUnsafeJsonInteger(value)` helper — true iff `value` is
a finite integer-valued JS Number where `|value| < 1e21` (so
`JSON.stringify` emits integer form, not exponent) **and** `value` falls
outside `[Int64.MIN, UInt64.MAX]`.
- `sanitizeUnknownInPlace` gains a number-branch: when the predicate
holds, replace the Number with `String(value)`. The downstream JSON
column dynamic-types the path as String for that row — fine, since the
value was already precision-lossy upstream (no JS Number above 2^53 is
numerically meaningful anyway).
- Float-valued numbers, large floats (>= 1e21), NaN and Infinity are
left alone — `JSON.stringify` emits them with exponents or as `null`,
both of which ClickHouse accepts.

`apps/webapp/test/sanitizeRowsOnParseError.test.ts`: four new unit tests
+ an extension to `sanitizeRows` covering surrogate + integer fixes
counted together across rows. The unit suite now covers:

- Positive value above `UInt64.MAX` (`117039831458782870000` — the
actual prod value)
- Negative value below `Int64.MIN`
- Boundary values pass through (`42`, `Number.MAX_SAFE_INTEGER`, `2^63`)
- Non-integer numbers untouched (floats, `1e25`, NaN, Infinity)
- The actual `scan-social-profiles` nested shape — finds the offending
`gp_id` deep inside
`output.data.profiles[].spec_format[].platform_variables[].value`

`.server-changes/runs-replication-bigint-recovery.md` — release notes
entry.

## Why reactive, not pre-flight

`#prepareJson` runs millions of times per day on the replication hot
path. Walking every JSON tree to look for oversized integers would add
bounded-but-real CPU on every healthy row. `sanitizeRows` only fires
after a ClickHouse parse-error rejection, which is a few times a day
platform-wide. Extending it costs effectively zero on healthy traffic
and gains us recovery on the rare poisoned row.

## Verification

- Reproduced 1:1 in a throwaway Docker
`clickhouse/clickhouse-server:25.12.11.4` (closest available to the prod
`25.12.1.1579` build). Pre-sanitize JSON fails with the exact prod
error; post-sanitize JSON inserts cleanly and the row is readable with
`gp_id` stored as a String subtype.
- `pnpm --filter webapp exec vitest run
test/sanitizeRowsOnParseError.test.ts` — 22/22 passing (18 existing + 4
new).
- `pnpm run typecheck --filter webapp` — clean.

## Test plan

- [x] `pnpm run typecheck --filter webapp`
- [x] Unit tests pass against new + existing cases
- [x] End-to-end Docker ClickHouse repro confirms recovery
- [ ] Post-deploy: confirm `Sanitizing batch after ClickHouse JSON parse
error` warns fire instead of `Dropped batch …` errors when
`scan-social-profiles` outputs trip CH again
- [ ] Post-deploy: confirm `permanentlyDroppedBatches` counter stops
climbing in
`/stp/trigger-app-prod/ecs/replication/service-container/process-logs`

## What this does NOT do

- Doesn't backfill the ~120k+ existing stranded `EXECUTING` rows in
production. Same as #3708 — that needs a reconciliation/backfill sweep
(separate ticket — TRI-9755 fix #3).
- Doesn't address the upstream root cause (the customer task emitting a
JS-Number-precision-lossy big int). That's a customer-task concern; our
replication path needs to be robust to whatever shape arrives.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
 2026-05-27 09:00:53 -07:00
Oskar Otwinowski 5ba1b32d53 fix(core): handle conflicting dotted paths in unflattenAttributes (#3762) 2026-05-27 16:18:50 +02:00
Eric Allam 37eeaa3690 fix(webapp): skip V1 WAITING_FOR_DEPLOY drain on V2 promotes (#3742) 
## Summary

Stops the legacy V1 `WAITING_FOR_DEPLOY` drain from running on every V2
deployment promotion. The drain queries `TaskRun` by
`status='WAITING_FOR_DEPLOY'`, which only V1-engine runs ever have — V2
runs use `PENDING_VERSION` and are handled out of band. Calling the
drain on V2 promotes produced empty queries against the status index and
unnecessary reader-DB load.

## Fix

Two layers:

1. Gate the enqueue at the call site in `ChangeCurrentDeploymentService`
so it only fires when the deployment's worker is on engine V1.
2. Add a `LEGACY_RUN_ENGINE_WAITING_FOR_DEPLOY_DISABLED` env var
(default `0`). When set to `1`, the service returns immediately from
`call()` — neuters any jobs already sitting in the worker queue from
before the deploy lands.

V1 customers see no change; V2 promotes no longer trigger the drain.
 2026-05-25 07:10:13 +01:00
Matt Aitken 596a9bb90c fix(webapp): retain sessions-replication singleton import via globalThis assignment (#3738) 
Linear: [TRI-9864](https://linear.app/triggerdotdev/issue/TRI-9864)
(Urgent)
Production incident:
[TRI-9863](https://linear.app/triggerdotdev/issue/TRI-9863) (mitigated
by image revert in cloud#910)

## Bug

`apps/webapp/package.json` declares `"sideEffects": false`. PR #3333
(`71d98b4e`) replaced the previous real method-call retention idiom at
the two `sessionsReplicationInstance` import sites with:

```ts
import { sessionsReplicationInstance } from "...";
void sessionsReplicationInstance;
```

esbuild treats `void <identifier>;` as a pure expression statement under
`sideEffects: false` and **tree-shakes the entire import** — including
the `singleton(...)` call inside `sessionsReplicationInstance.server.ts`
which is the only thing that fires
`initializeSessionsReplicationInstance()`. The sessions→ClickHouse
logical replication worker never starts, the slot is unconsumed, lag
grows.

### How it manifested in production

cloud#907's image bump rolled the `SessionReplicationService` ECS task
on prod at 14:32 UTC. The new container's startup log emitted `🗃️ Runs
replication service enabled` but **not** `🗃️ Sessions replication
service enabled` or `🗃️ Sessions replication service started`.
CloudWatch `OldestReplicationSlotLag` grew at ~220 MB/min and the `High
replication lag` alarm fired at 14:37 UTC. Prod was reverted to the
previous image (cloud#910) to stop the bleed.

### Verification

`grep` of the built bundle `apps/webapp/build/index.js` (built from
`c0365d36`):

- **3** occurrences of `Runs replication` / `runsReplicationInstance`
strings ✅
- **0** occurrences of `Sessions replication` /
`sessionsReplicationInstance` / `SessionsReplicationService` ❌

The runs path survives tree-shaking because `adminWorker.server.ts` and
`admin.api.v1.runs-replication.*` routes have real method calls
(`.start()`, `.teardown()`, `.backfill()`) — observable uses the
tree-shaker must preserve. The sessions singleton has no real callers,
only the `void` no-ops, hence its complete elimination from the bundle.

## Fix

Replace `void sessionsReplicationInstance;` with an assignment to
`globalThis`, an unambiguous observable side effect the bundler cannot
eliminate:

```ts
(globalThis as Record<string, unknown>).__sessionsReplicationInstance =
  sessionsReplicationInstance;
```

Applied at both call sites: `apps/webapp/app/entry.server.tsx` and
`apps/webapp/app/v3/services/adminWorker.server.ts`.

Surrounding comments updated to document the bundler interaction so the
next maintainer doesn't reintroduce `void`.

## Out of scope (follow-ups)

- **Robustness improvement**: change `apps/webapp/package.json` from
`"sideEffects": false` to an allowlist that includes
`*Instance.server.ts` files. Prevents the same regression shape via any
future `*Instance` singleton.
- **Build-time check**: add a `grep` post-build step in `publish.yml`
requiring `"Sessions replication"` to appear in
`apps/webapp/build/index.js`. Catches this exact regression at CI time.

## Test plan

- [x] `pnpm run typecheck --filter webapp` clean
- [ ] After merge + publish: confirm new image's
`SessionReplicationService` container logs `🗃️ Sessions replication
service enabled` and `🗃️ Sessions replication service started` at
startup
- [ ] After re-deploying to prod: confirm `OldestReplicationSlotLag`
stops growing and drains
 2026-05-24 16:21:15 +01:00
Matt Aitken eefb96c87d fix(webapp): recover from ClickHouse JSON parse failures in runs replication (#3708) 
## Summary

On a ClickHouse `Cannot parse JSON object` rejection,
`RunsReplicationService` now sanitizes lone UTF-16 surrogates across the
failing batch via the existing `sanitizeRows` helper and retries once.
If the sanitizer found nothing or the retry also fails, the batch is
dropped loudly with a counter increment, so the surrounding
`#insertWithRetry` layer doesn't spin three more times on a
deterministic failure. Non-parse errors propagate unchanged.

Mirrors the pattern from #3659 (for `ClickhouseEventRepository`) — same
root cause (lone UTF-16 surrogates in user-provided JSON), same recovery
shape, **reusing the same shared helpers** (`sanitizeRows`,
`isClickHouseJsonParseError`, `parseRowNumberFromError`).

Fixes the customer-facing symptom from
[TRI-9755](https://linear.app/triggerdotdev/issue/TRI-9755): a single
row's poisoned `output` JSON used to take down the
`COMPLETED_SUCCESSFULLY` UPDATE events for its 50+ batch-mates,
stranding them in `EXECUTING` in ClickHouse forever and inflating
"Running" counts on the Tasks page. Confirmed in production this is
ongoing — ~120k stale rows accumulated in a single 5-hour burst on
2026-05-18; smaller continuous leak before and after.

## What changed

`apps/webapp/app/services/runsReplicationService.server.ts`:

- Imports the three helpers from
`~/v3/eventRepository/sanitizeRowsOnParseError.server` (no duplication;
no move).
- New private `#insertWithJsonParseRecovery<T>(rows, doInsert,
contextLabel, attempt)` — generic over `TaskRunInsertArray[]` and
`PayloadInsertArray[]`, structurally identical to
`ClickhouseEventRepository.#insertWithJsonParseRecovery`. Try → on parse
error sanitize the whole batch (the `at row N` hint is logged but not
used to slice — semantics under `input_format_parallel_parsing` aren't
stable) → retry once → drop with loud log if sanitizer found nothing OR
retry still fails.
- `#insertTaskRunInserts` and `#insertPayloadInserts` extract a
`doInsert` closure and hand it to the wrapper. Existing error logging,
span recording, and `recordSpanError` are preserved inside the closure.
- New `private _permanentlyDroppedBatches = 0` counter with a public
getter, for ops dashboards and tests (matches the events-repo
convention). One shared counter for both insert sites — granularity
comes from the `contextLabel` (`task_runs_v2` /
`raw_task_runs_payload_v1`) on every log line.

`.server-changes/runs-replication-utf16-recovery.md` — release notes
entry.

## Why no new tests

The shared helpers already have full unit + real-ClickHouse contract
coverage from #3659
(`apps/webapp/test/sanitizeRowsOnParseError.test.ts`,
`apps/webapp/test/otlpUtf16Sanitization.integration.test.ts`). The new
wrapper is a line-for-line structural port. Adding a parallel
integration test would require synthesizing bad data that *escapes* the
preemptive `detectBadJsonStrings` check in `#prepareJson` but still
trips ClickHouse — non-trivial without hand-crafted fixtures and
wouldn't cover any new logic.

## What this does NOT do

- Doesn't touch the ~120k existing stale `EXECUTING` rows in production.
That needs a reconciliation/backfill sweep (separate ticket — TRI-9755
fix #3).
- Doesn't sanitize the `error` column path
(`runsReplicationService.server.ts:932 const errorData = { data:
run.error };`). Reactive recovery will catch it if it ever poisons a
batch, but feeding it through `#prepareJson` like `output` is a cheap
follow-up.

## Test plan

- [x] `pnpm run typecheck --filter webapp` — clean
- [ ] Post-deploy: confirm `permanentlyDroppedBatches` counter stays at
zero (or near-zero) in
`/stp/trigger-app-prod/ecs/replication/service-container/process-logs`,
and watch for `Sanitizing batch after ClickHouse JSON parse error` warns
to confirm recovery is firing on real traffic
- [ ] Post-deploy: confirm the rate of new
"EXECUTING-but-actually-COMPLETED" zombies in ClickHouse flattens
(current rate ≈ tens-to-hundreds per hour platform-wide)

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
 2026-05-23 20:20:09 +01:00
Eric Allam 9f64bf404b docs(ai-chat): slim-wire HITL continuations + field-level merge contract (#3721) 
## Summary

Updates the AI chat docs to match the slim-wire + field-level merge
behavior shipped in #3719 and the precise `.in/append` cap +
CORS-readable 413 shipped in #3720. No behavior changes here — code is
correct in `main`; the docs were lagging on three patterns customers
copy out of the page.

## What changed

- **`hydrateMessages` examples upsert by id** (in `lifecycle-hooks.mdx`,
`patterns/database-persistence.mdx`, and
`patterns/persistence-and-replay.mdx`). The previous
`stored.push(newMsg)` pattern duplicated the assistant id on HITL
continuations and caused the LLM to receive a tool call with no
`arguments`. The new examples include the rationale inline.
- **`onValidateMessages` example filters to user messages**
(`lifecycle-hooks.mdx`). The previous example called
`validateUIMessages({ messages, tools })` directly, which now throws on
HITL slim wires (the AI SDK schema requires `input` on resolved tool
parts). New example shows the filter pattern, with a Warning callout
explaining why.
- **Merge contract description updated** (`lifecycle-hooks.mdx`). The
old wording said incoming messages are "auto-merged" / "replaced"; the
new description explains the actual field-level overlay (state advances
only).
- **Approval-responded wire example slimmed** (`client-protocol.mdx`).
Shows the minimum shape the agent reads — `state` + `approval` (or
`output` / `errorText` for HITL). Notes that the built-in transports
ship this slim shape by default and that fuller shapes are still
accepted.
- **`/in/append` 413 row and FAQ updated** (`client-protocol.mdx`,
`patterns/trusted-edge-signals.mdx`). Reflects the new precise S2 cap
and the CORS-readable 413.
- **New changelog entry** at the top of `changelog.mdx` covering all of
the above.

The historical `## 512 KiB ceiling removed` entry further down the
changelog is left as-is (it's a snapshot of the prior transition), and
the v4.5 upgrade-guide section is skipped — the merge contract is
backwards compatible.

## Test plan

- Mintlify dev preview renders cleanly with no broken anchors
- Linked references resolve (`/ai-chat/lifecycle-hooks#hydratemessages`,
`/ai-chat/lifecycle-hooks#onvalidatemessages`,
`/ai-chat/patterns/database-persistence#alternative-hydratemessages`,
`/ai-chat/client-protocol#step-3-send-messages-stops-and-actions`,
`/ai-chat/patterns/large-payloads`)
 2026-05-23 17:27:15 +01:00
github-actions[bot] d34014ddcc chore: release v4.5.0-rc.2 (#3702)
🚀 Publish Trigger.dev Docker / typecheck (push) Failing after 0s
Details
🚀 Publish Trigger.dev Docker / publish-webapp (push) Has been skipped
Details
🚀 Publish Trigger.dev Docker / publish-worker (push) Has been skipped
Details
🚀 Publish Trigger.dev Docker / publish-worker-v4 (push) Has been skipped
Details
🚀 Publish Trigger.dev Docker / units (push) Failing after 4s
Details 
## Summary
3 improvements, 1 bug fix.

## Improvements
- The per-turn merge now overlays the wire copy's tool-part state
advancement onto the agent's existing chain — `state` + the matching
resolution field (`output` / `errorText` / `approval`) come from the
wire, everything else (text, reasoning, tool `input`, provider metadata)
stays whatever the snapshot or `hydrateMessages` returned. Previously a
full-message replace overwrote those fields with whatever the client
shipped, so a slimmed wire copy landed a tool call with no `arguments`
on the next LLM call. Covers `output-available` / `output-error` (HITL
`addToolOutput`) and `approval-responded` / `output-denied` (approval
flow).
- `TriggerChatTransport.sendMessages` and `AgentChat.sendRaw` now slim
assistant messages that carry advanced tool parts. The wire payload is
just `{ id, role, parts: [<state + resolution field>] }` for
`submit-message` continuations; everything else passes through.
Reasoning blobs and full tool inputs no longer ride the wire on every
`addToolOutput` / `addToolApproveResponse`, so continuation payloads
stay well under the `.in/append` cap on long agent loops.
- Add `TriggerClient` for running multiple SDK clients side-by-side,
each with its own auth, preview branch, and baseURL. Useful when a
single process needs to trigger tasks or read runs across multiple
projects, environments, or preview branches without mutating shared
global state.
([#3683](https://github.com/triggerdotdev/trigger.dev/pull/3683))

## Bug fixes
- Fix `chat.agent` HITL continuations on reasoning-heavy turns. Two
changes that work together:
([#3719](https://github.com/triggerdotdev/trigger.dev/pull/3719))

<details>
<summary>Raw changeset output</summary>

⚠️⚠️⚠️⚠️⚠️⚠️

`main` is currently in **pre mode** so this branch has prereleases
rather than normal releases. If you want to exit prereleases, run
`changeset pre exit` on `main`.

⚠️⚠️⚠️⚠️⚠️⚠️

# Releases
## @trigger.dev/build@4.5.0-rc.2

### Patch Changes

-   Updated dependencies:
    -   `@trigger.dev/core@4.5.0-rc.2`

## trigger.dev@4.5.0-rc.2

### Patch Changes

-   Updated dependencies:
    -   `@trigger.dev/build@4.5.0-rc.2`
    -   `@trigger.dev/core@4.5.0-rc.2`
    -   `@trigger.dev/schema-to-json@4.5.0-rc.2`

## @trigger.dev/plugins@4.5.0-rc.2

### Patch Changes

-   Updated dependencies:
    -   `@trigger.dev/core@4.5.0-rc.2`

## @trigger.dev/python@4.5.0-rc.2

### Patch Changes

-   Updated dependencies:
    -   `@trigger.dev/sdk@4.5.0-rc.2`
    -   `@trigger.dev/build@4.5.0-rc.2`
    -   `@trigger.dev/core@4.5.0-rc.2`

## @trigger.dev/react-hooks@4.5.0-rc.2

### Patch Changes

-   Updated dependencies:
    -   `@trigger.dev/core@4.5.0-rc.2`

## @trigger.dev/redis-worker@4.5.0-rc.2

### Patch Changes

-   Updated dependencies:
    -   `@trigger.dev/core@4.5.0-rc.2`

## @trigger.dev/rsc@4.5.0-rc.2

### Patch Changes

-   Updated dependencies:
    -   `@trigger.dev/core@4.5.0-rc.2`

## @trigger.dev/schema-to-json@4.5.0-rc.2

### Patch Changes

-   Updated dependencies:
    -   `@trigger.dev/core@4.5.0-rc.2`

## @trigger.dev/sdk@4.5.0-rc.2

### Patch Changes

- Fix `chat.agent` HITL continuations on reasoning-heavy turns. Two
changes that work together:
([#3719](https://github.com/triggerdotdev/trigger.dev/pull/3719))

- The per-turn merge now overlays the wire copy's tool-part state
advancement onto the agent's existing chain — `state` + the matching
resolution field (`output` / `errorText` / `approval`) come from the
wire, everything else (text, reasoning, tool `input`, provider metadata)
stays whatever the snapshot or `hydrateMessages` returned. Previously a
full-message replace overwrote those fields with whatever the client
shipped, so a slimmed wire copy landed a tool call with no `arguments`
on the next LLM call. Covers `output-available` / `output-error` (HITL
`addToolOutput`) and `approval-responded` / `output-denied` (approval
flow).
- `TriggerChatTransport.sendMessages` and `AgentChat.sendRaw` now slim
assistant messages that carry advanced tool parts. The wire payload is
just `{ id, role, parts: [<state + resolution field>] }` for
`submit-message` continuations; everything else passes through.
Reasoning blobs and full tool inputs no longer ride the wire on every
`addToolOutput` / `addToolApproveResponse`, so continuation payloads
stay well under the `.in/append` cap on long agent loops.

Note: `onValidateMessages` receives the slim wire on HITL turns. If you
call `validateUIMessages` from `ai` against the full `messages` array it
will reject the slim assistant; filter to user messages (or skip on HITL
turns) — see the updated docstring on `onValidateMessages` for the
recommended pattern.

For `hydrateMessages` hooks that persist the chain, this release also
adds a small helper to the `@trigger.dev/sdk/ai` surface:

    ```ts
    import { chat, upsertIncomingMessage } from "@trigger.dev/sdk/ai";

    chat.agent({
hydrateMessages: async ({ chatId, trigger, incomingMessages }) => {
const record = await db.chat.findUnique({ where: { id: chatId } });
        const stored = record?.messages ?? [];
if (upsertIncomingMessage(stored, { trigger, incomingMessages })) {
await db.chat.update({ where: { id: chatId }, data: { messages: stored }
});
        }
        return stored;
      },
    });
    ```

It pushes fresh user messages by id, no-ops on HITL continuations (the
incoming shares an id with the existing assistant — the runtime overlays
the new tool-state advance), and skips on non-`submit-message` triggers.
Returns `true` if it mutated `stored` so the caller knows whether to
persist.

Net effect: `chat.addToolOutput(...)` /
`chat.addToolApproveResponse(...)` on multi-step reasoning agents
(OpenAI Responses with `store: false`, Anthropic extended thinking,
etc.) no longer blows the cap and no longer corrupts the LLM input.

- Add `TriggerClient` for running multiple SDK clients side-by-side,
each with its own auth, preview branch, and baseURL. Useful when a
single process needs to trigger tasks or read runs across multiple
projects, environments, or preview branches without mutating shared
global state.
([#3683](https://github.com/triggerdotdev/trigger.dev/pull/3683))

    ```ts
    import { TriggerClient } from "@trigger.dev/sdk";

const prod = new TriggerClient({ accessToken:
process.env.TRIGGER_PROD_KEY });
    const preview = new TriggerClient({
      accessToken: process.env.TRIGGER_PREVIEW_KEY,
      previewBranch: "signup-flow",
    });

    await prod.tasks.trigger("send-email", payload);
    await preview.runs.list({ status: ["COMPLETED"] });
    ```

-   Updated dependencies:
    -   `@trigger.dev/core@4.5.0-rc.2`

## @trigger.dev/core@4.5.0-rc.2

</details>

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
v.docker.4.5.0-rc.2 v4.5.0-rc.2 		2026-05-23 17:15:33 +01:00
Eric Allam 75679c7518 fix(sdk): chat HITL continuations no longer break the next LLM call (#3719) 
## Summary

Multi-step reasoning agents with HITL tools (OpenAI Responses with
`store: false`, Anthropic extended thinking, etc.) failed on
`chat.addToolOutput(...)` continuations — either the wire payload blew
the `.in/append` cap (reasoning blobs + tool inputs routinely > 512
KiB), or app-side slimming workarounds got overwritten server-side and
the next LLM call landed a tool call with no `arguments`. Both modes are
fixed.

## Design

The per-turn merge in `chat.agent` now overlays only the tool-part state
advances (`output-available` / `output-error` / `approval-responded` /
`output-denied`) from the wire copy onto the hydrated/snapshot chain.
Previously it replaced the entire message, which dropped `input`,
reasoning, and text from the LLM's view whenever the wire was slim.

In parallel, `TriggerChatTransport.sendMessages` and `AgentChat.sendRaw`
now slim the assistant message themselves on `submit-message`
continuations: ship `{ id, role, parts: [<resolved tool part only>] }`,
everything else reconstructed server-side from `hydrateMessages` or the
durable snapshot. Continuation payloads drop from 600 KiB – 1 MiB to ~1
KiB.

`references/ai-chat` `aiChatHydrated.hydrateMessages` now upserts by id
instead of pushing. With slim continuations, a blind push duplicates the
assistant id in the returned chain — the merge updates the first match,
the slim duplicate goes straight to `toModelMessages` with no `input`,
and the LLM 4xx's. This is the canonical pattern customers should mirror
in their own hydrate implementations.

## Test plan

- 11 new tests (slim helper unit + slim+merge integration for HITL,
approval, default no-hydrate branch)
- Full SDK suite: 239 tests pass across 19 files
- End-to-end sweep against `references/ai-chat`: 19 customer-side smoke
tests green; HITL wire bodies confirmed at ~1 KiB (was 600 KiB+); no
provider 4xx errors across OpenAI Responses or Anthropic
 2026-05-23 17:09:39 +01:00
Eric Allam c0365d36fb fix(webapp): precise S2 record cap + CORS 413 on session append (#3720) 
## Summary

Two improvements to session `.in/append`:

- Oversize-body 413 responses now carry CORS headers, so browser fetches
see a readable status instead of an opaque `TypeError: Failed to fetch`.
App-side retry-on-disconnect loops no longer spin forever on a
permanently-rejected payload.
- The per-record cap is now computed precisely against S2's actual
ceiling instead of the conservative 512 KiB floor. Legitimate ~600-900
KiB tool outputs (search results, file content) now succeed;
pathological all-quote content that would double under JSON escape still
rejects cleanly.

## Design

S2 enforces a per-record metered size of `8 + 2*H + Σ(header name +
value) + body ≤ 1048576` bytes. With no record headers (our case), the
budget reduces to `body ≤ 1048568`. Verified empirically against cloud
S2 — append succeeds at metered=1048576 and 422s at 1048577 with `record
must have metered size less than 1 MiB`.

The old `MAX_APPEND_BODY_BYTES = 512 KiB` was derived by assuming
worst-case JSON escape doubling (every byte becomes `\"` or `\\`),
giving `(1 MiB - overhead) / 2`. Safe, but rejects ~half the legitimate
input space.

The new flow:

1. Pre-cap the HTTP body at 1 MiB (DoS guard against reading arbitrary
garbage before we can compute the wrap).
2. After reading, `S2RealtimeStreams.#appendPartByName` computes
`Buffer.byteLength(JSON.stringify({data: part, id: partId}), "utf8") +
8` and throws `S2RecordTooLargeError` (a `ServiceValidationError` with
status 413) if it would exceed S2's ceiling. The route's existing error
branch maps the throw to a 413 with a descriptive message.

The 413 CORS fix is a single-line change in `apiBuilder.server.ts` —
`wrapResponse` was being skipped on the body-too-large branch; every
other error branch wraps; the 413 was the exception.

## Test plan

- Empirically verified against cloud S2 with a boundary scan across
`[1048568, 1048569, ..., 1048576]` and across H ∈ {0, 1×5 hdr bytes,
1×14 hdr bytes} — the formula matches exactly
- Browser-side fetch on a 700 KiB POST now resolves with a readable
`status: 413` (no `TypeError: Failed to fetch`)
- A 900 KiB ASCII tool output now passes (would have 413'd at 512 KiB
pre-fix)
 2026-05-23 16:08:10 +01:00
Matt Aitken 2fcc484a2b fix(webapp): route OrganizationDataStoresRegistry writes through the writer prisma (#3722) 
## Bug

The `OrganizationDataStoresRegistry` singleton in
`apps/webapp/app/services/dataStores/organizationDataStoresRegistryInstance.server.ts`
was constructed with `$replica`. That client was then used by both the
polling read path *and* by `addDataStore` / `updateDataStore` /
`deleteDataStore` (and their backing `SecretStore.setSecret` upserts).
The write methods route through the read replica, which Postgres rejects
with **error code 25006**:

```
Invalid prisma.secretStore.upsert() invocation:
ConnectorError(ConnectorError { user_facing_error: None, kind: QueryError(PostgresError {
  code: "25006",
  message: "cannot execute INSERT in a read-only transaction",
  ...
}), transient: false })
```

User-visible symptom: the admin `/admin/data-stores` "Add data store"
form returns a 400 with this error wrapped, so no
`OrganizationDataStore` row can ever be created via the UI.

The read path (`loadFromDatabase` polling + `SecretStore.getSecret`) is
unaffected because `findMany` + secret read are read-only.

## Fix

Change the registry constructor to take both a writer and a replica:

```ts
constructor(writer: PrismaClient, replica: PrismaClient | PrismaReplicaClient)
```

- `loadFromDatabase()` keeps using `_replica` (and its
`SecretStore.getSecret` calls) — these are background cache-fillers, not
on user-latency-sensitive paths.
- `addDataStore` / `updateDataStore` / `deleteDataStore` (and their
`SecretStore.setSecret` / `deleteSecret` calls) now use `_writer`.

`organizationDataStoresRegistryInstance.server.ts` passes `(prisma,
$replica)` from `~/db.server`. Test sites that constructed with
`(prisma)` now pass `(prisma, prisma)` — the testcontainer exposes a
single client, so the writer/replica split collapses to one connection.

## Files

-
`apps/webapp/app/services/dataStores/organizationDataStoresRegistry.server.ts`
— constructor + read/write split
-
`apps/webapp/app/services/dataStores/organizationDataStoresRegistryInstance.server.ts`
— pass `prisma` alongside `$replica`
- `apps/webapp/test/organizationDataStoresRegistry.test.ts` — 14 call
sites bumped
- `apps/webapp/test/clickhouseFactory.test.ts` — 5 call sites bumped

## Test plan

- [x] Existing `organizationDataStoresRegistry.test.ts` +
`clickhouseFactory.test.ts` still pass (constructor sites updated;
behavior unchanged for tests).
- [ ] After deploy to test cloud, retry `/admin/data-stores` "Add data
store" form for the HIPAA org — should now succeed and the row should
appear.
- [ ] Verify the registry's polling reload picks up the new row within
`ORGANIZATION_DATA_STORES_RELOAD_INTERVAL_MS` (60s default) and the
factory starts routing to the org-scoped instance.
 2026-05-23 12:00:31 +01:00
Eric Allam 61ca40b4b1 perf(run-engine,webapp): look up PENDING_VERSION runs via ClickHouse (#3707) 
## Summary

When a background worker registers, the engine resolves runs that were
queued before the worker was ready (status `PENDING_VERSION`). That
lookup used to scan a Postgres status index on `TaskRun`. Move it to
ClickHouse: query candidate run ids from `task_runs_v2`, then refetch
the actual rows from Postgres by primary key with a `status =
'PENDING_VERSION'` guard for idempotency.

## Design

The lookup is a pluggable interface on the run engine
(`PendingVersionRunIdLookup`). The webapp wires a ClickHouse-backed
implementation through the org-scoped `clickhouseFactory` using a new
`"engine"` client type, configured by `RUN_ENGINE_CLICKHOUSE_*` env
vars. The URL falls back to `CLICKHOUSE_URL` when unset, so self-hosted
deployments don't need new config to keep working.

When the lookup returns no candidates, one bounded retry is scheduled
~5s later to cover ClickHouse replication lag against `task_runs_v2`.
The Postgres status guard on both the candidate refetch and the inner
`updateMany` prevents double-promotion when a retry races with a
concurrent deploy.

Tests cover three existing PENDING_VERSION cases via a small
Postgres-backed test adapter; new ClickHouse-backed integration tests
will follow.
 2026-05-22 17:18:41 +01:00
Eric Allam 0d4891a5f2 perf(database): drop unused TaskRun(scheduleId, createdAt) index (#3706) 
## Summary

Drops the unused composite Postgres index
`TaskRun_scheduleId_createdAt_idx`. The schedule list view reads from
ClickHouse, so this index served no Prisma query while still being
maintained on every `TaskRun` INSERT/UPDATE. Removing it reduces write
amplification on the primary database.

Sibling to the prior drop of `TaskRun_scheduleId_idx` and the earlier
removal of the `TaskRun.scheduleId` foreign key — all stemming from
migrating schedule-aware reads to ClickHouse.

## Verification

- Sampled `pg_stat_user_indexes` for `TaskRun` over multiple hours —
zero scans against this index.
- Grepped the codebase for any Prisma query filtering
`TaskRun.scheduleId` — none found. All schedule-aware listing routes
through `clickhouseRunsRepository`.
 2026-05-22 16:34:00 +01:00
Saadi Myftija 2cbfcfac3d chore(webapp): make HTTP keep-alive timeout configurable (#3705) 
Make the Express server's `keepAliveTimeout` configurable via
`HTTP_KEEPALIVE_TIMEOUT_MS`. Default preserved at 65000 ms — no behavior
change if unset.
 2026-05-22 16:28:08 +01:00
nicktrn ddad9700d7 fix(supervisor): compat shim for COMPUTE checkpoint type (#3703) 
Workloads bundled with CLI versions before v4.4.4 use a strict zod enum
for `checkpoint.type` that only allows DOCKER and KUBERNETES. When a
customer's runs are routed via the compute path, those old runners
receive `type: "COMPUTE"` on `/snapshots/since/...` and `/dequeue`
responses and fail validation - blocking silent migration of existing
deployments.

The workload never reads the field - only validates the shape. Rewriting
COMPUTE -> KUBERNETES on the way out lets older runners keep parsing
while the database and internal services keep the real value. Limited to
the two workload-facing endpoints whose response includes a checkpoint;
`/continue`, `/attempts/start`, `/attempts/complete` all return shapes
without one.

Followup to #3114.
 2026-05-22 15:41:30 +01:00
Daniel Sutton 1015876b98 feat(webapp): user-based Sentry attribution with tenant tags (#3678) 
## Summary

Stamp every Sentry event with the signed-in user and the tenant (org /
project / env) the request belongs to, so "Users Impacted" counts
distinct humans and events become filterable per tenant.

**Design after review (current):**

- `user.id = real user cuid` (from `requireUser`). "Users Impacted"
counts humans, not tenants.
- Tenant context (org / project / env slugs, IDs, env type) moves
entirely onto tags: `org_slug`, `project_slug`, `env_slug`, `org_id`,
`project_id`, `project_ref`, `environment_id`, `env_type`, plus
`impersonating` when set.
- Backed by an `AsyncLocalStorage` scope established at the HTTP entry.
Each entry point fills what it knows; loaders enrich the same scope with
what they already have.

**Zero new database queries.** The middleware does a regex match only.
Dashboard loaders that already query Prisma gain a couple of extra
selected columns; nothing new round-trips.

## How it's wired

- **Express middleware (`tenantContextResolver.server.ts`)** — parses
the URL with a regex and always opens an ALS scope. Populates whatever
subset of slugs is present: `/orgs/:o` → just `orgSlug`;
`/orgs/:o/projects/:p` adds `projectSlug`; the full triple adds
`envSlug`. Non-tenant paths get an empty scope so loaders can still
enrich.
- **`_app/route.tsx`** — already calls `requireUser`. Adds
`tenantContext.enrich({ userId: user.id })` for every authenticated
dashboard request. No new query.
- **Env layout loader (`_app.orgs.$o.projects.$p.env.$e/route.tsx`)** —
its existing `prisma.project.findFirst` gains two columns in `select`
(`externalRef`, `organization.id`). After it picks an env, calls
`tenantContext.enrich({ orgId, projectId, projectRef, envId, envType
})`. Same query, +2 columns.
- **API path (`apiBuilder.server.ts`)** — wraps every handler in
`tenantContext.run(tenantContextFromAuthEnvironment(authenticationResult.environment),
…)`. The mapper pulls `userId` from `env.orgMember?.userId` (already
selected by `authIncludeBase` — no schema change). Covers
`createLoaderApiRoute`, `createActionApiRoute`, and
`createMultiMethodApiRoute`.
- **Event processor (`sentryTenantContext.server.ts`)** — registered in
`entry.server.tsx` so it lives in the Remix bundle and shares the same
`tenantContext` ALS instance as the middleware and loaders. Stamps
whatever's present; nothing forced.

## Example events from local verification

| URL | `user.id` | Tags |
|-----|-----------|------|
| `/orgs/:o/projects/:p/env/:e/...` | real user cuid | `org_slug`,
`project_slug`, `env_slug`, `org_id`, `project_id`, `project_ref`,
`environment_id`, `env_type` |
| `/orgs/:o/settings` (non-env-scoped) | real user cuid | `org_slug`
only |
| API request with `orgMember` | `orgMember.userId` | full tenant set |
| API request without `orgMember` | (unset) | full tenant set |

## Trade-offs

1. On env-scoped pages, errors that fire before the env layout loader's
enrich callback runs get slugs + `user.id` but not the tenant IDs /
`env_type`. Realistic errors deep in async work get the full set. (Same
race as before, narrower window now that slugs/`user.id` are populated
up-front by the middleware and `_app` enrich.)
2. API requests where the environment has no `orgMember` get tenant tags
but no `user.id`. Those events still show in the issue but don't
contribute to "Users Impacted".

## Out of scope (deferred)

Background workers (`redis-worker`, `schedule-engine`) and socket
handlers. Those entry points don't set `tenantContext.run` yet — their
events ship without tenant attribution until each is wired in a
follow-up.

## Tests

31 unit tests across 4 files. New tests notably cover:

- `parseTenantPath`: org-only, org+project, and full-triple URL
variants.
- `tenantContext.enrich`: in-place patch, no-op outside `run()`,
concurrent-scope isolation, empty-scope + enrich pattern (for non-tenant
pages).
- `tenantContextFromAuthEnvironment`: with and without `orgMember` —
verifies the API path's `user.id` mapping.
- `addTenantContextToEvent`: empty scope, userId-only, slugs-only, full
enrichment, conditional tag emission, preservation of prior `event.user`
fields.

## Test plan

- [ ] `pnpm run typecheck --filter webapp`
- [ ] `pnpm run test --filter webapp -- test/tenantContext.test.ts
test/sentryTenantContext.test.ts test/tenantContextResolver.test.ts
test/tenantContextFromAuthEnvironment.test.ts`
- [ ] Local manual: with `SENTRY_DSN` set, hit a dashboard URL and an
API route, confirm the captured events carry `user.id` + the expected
tag set in Sentry.
- [ ] After ship: confirm "Users Impacted" on a real Sentry issue
reflects distinct users (not tenants).

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-05-22 15:10:41 +01:00
Matt Aitken 71d98b4e6b Support for org-scoped ClickHouse (#3333) 
Added `OrganizationDataStore` which allows orgs to have data stored in
specific separate services.

For now this is just used for ClickHouse. When using ClickHouse we get a
client for the factory and pass in the org id.

Particular care has to be made with two hot-insert paths:
1. RunReplicationService
2. OTLPExporter

---------

Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: Claude <noreply@anthropic.com>
 2026-05-22 14:43:03 +01:00
Eric Allam 832cf7220b feat(sdk,core): add TriggerClient for per-instance SDK configuration (#3683) 
## Summary

`new TriggerClient({...})` exposes the management API (tasks, runs,
schedules, envvars, batch, queues, deployments, prompts, auth) as an
explicit instance with its own auth, preview branch, and baseURL.
Multiple clients can coexist in one process without mutating shared
global state — useful when a single service triggers across multiple
projects, environments, or preview branches.

```ts
import { TriggerClient } from "@trigger.dev/sdk";

const prod = new TriggerClient({ accessToken: process.env.TRIGGER_PROD_KEY });
const preview = new TriggerClient({
  accessToken: process.env.TRIGGER_PREVIEW_KEY,
  previewBranch: "signup-flow",
});

await prod.tasks.trigger("send-email", payload);
await preview.runs.list({ status: ["COMPLETED"] });
```

The existing global `configure()` API keeps working unchanged.

## Design

Instance methods enter an `AsyncLocalStorage`-backed scope (`sdkScope`)
before delegating to the existing module-level functions. The four
"pollution" points that previously read globals now consult the scope
first:

- `apiClientManager.{baseURL, accessToken, branchName}` and
`clientOrThrow` — identity fields are scope-only when scoped; `baseURL`
still falls back to `TRIGGER_API_URL` because plumbing (where the API
lives) is not identity.
- `taskContext.{ctx, worker, isWarmStart, isInsideTask}` — masked inside
an isolated scope so a `client.tasks.trigger(...)` from inside a task
doesn't leak the parent's `parentRunId` / `lockToVersion` / `isTest`
into a trigger that hits a different project.
- Inline `getEnvVar("TRIGGER_VERSION")` reads in `shared.ts` go through
a `scopedEnvVar` helper that returns `undefined` inside an isolated
scope.

The `TriggerClient` class itself is a thin wrapper that captures the
scope in its constructor and proxies each namespace method to enter that
scope before calling the existing impl. Generic inference (e.g.
`client.tasks.trigger<typeof t>(...)`) is preserved via `Pick<typeof ns,
keyof curatedSubset>` typings.

Two correctness fixes uncovered along the way are folded in:

- `apiClientManager.setGlobalAPIClientConfiguration` no longer silently
no-ops on the second call. `configure()` now actually overrides as users
expect (this is the root cause behind some "I changed the config but
nothing happened" reports).
- `apiClientManager.runWithConfig` (and therefore `auth.withAuth`) is
now backed by `sdkScope.withScope` instead of "mutate the global and
restore in finally". Two parallel `withAuth` calls with different
configs no longer stomp each other.

Surface curation: instance namespaces drop methods that don't make sense
per-instance — `batch.*AndWait` (runtime-dependent), `schedules.task` /
`schedules.timezones` (definition-time / stateless), `prompts.define`
(definition-time), `auth.configure` / `auth.withAuth` (global-only).

## Test plan

- [x] 9 runtime unit tests in `triggerClient.test.ts` cover: required
accessToken, instance auth + branch headers, no env fallback for
identity fields, no leakage between global and instance, four parallel
calls across two clients stay isolated, taskContext masking +
`inheritContext: true` override, `configure()` second-call override,
parallel `auth.withAuth` isolation.
- [x] 10 type-level assertions in `triggerClient.types.test.ts` using
`expectTypeOf` + `@ts-expect-error` lock in generic inference, return
type passthrough, overload preservation, and curated-surface drift.
- [x] Full SDK suite (219 tests) and core suite (530 tests) pass.
- [x] Webapp typecheck clean.
- [x] End-to-end smoke test against local webapp and a
freshly-provisioned cloud project — six concurrent multi-client triggers
all returned 200 with run IDs, headers per-client as expected.
- [ ] Reviewer: run `references/multi-client` per its `README.md` to
reproduce the smoke test locally.

## Try it

`references/multi-client` is a new reference workspace that exercises
this end-to-end:

- `src/trigger/echo.ts` — trivial target task
- `src/trigger/fanOut.ts` — opens two `TriggerClient`s from inside a
task, fires `echo` through each in parallel
- `src/external/main.ts` — external Node script with two clients
triggering `echo` sequentially and concurrently; logs every outgoing
request's `authorization` + `x-trigger-branch`
- `src/external/isolation.ts` — interleaves global `configure()` and an
instance call, asserts the captured fetch sequence shows no leakage
either way
 2026-05-22 12:29:42 +01:00
nicktrn 18d7144e74 ci: daily dependabot critical-severity slack alerts (#3701) 
Sibling to the weekly summary, focused on critical alerts only. Pings
Slack daily while any critical alerts are open; skips the post entirely
when zero, so no daily "all clear" noise.

- Daily 08:00 UTC cron + `workflow_dispatch` with `severity` input
(default `critical`, override to `high`/`medium`/`low` for manual
checks)
- Reuses the existing `dependabot-summary` environment (token, channel,
bot)
- Alerts link at the end is severity-filtered
 2026-05-22 10:32:08 +01:00
Eric Allam c0b9fdfce9 docs(ai-chat): clarify lastEventId is sessionId-keyed across run boundaries (#3700) 
## Summary

Two docs edits that close a footgun customers persisting transport state
can hit. Clearing `lastEventId` on `chat.endRun()` looks intuitive — the
Run ended, the cursor must be stale — but the cursor is sessionId-keyed,
not runId-keyed. Clearing it forces the next `sendMessages` to subscribe
from `seq_num=0`, which may hit the prior turn's still-durable
`turn-complete` record and close the SSE empty before the new Run's
chunks arrive.

Spells out the invariant in the frontend transport persistence table and
adds a Warning in the `chat.endRun()` reference.

## Test plan

- [x] Mintlify preview renders
- [x] No callout stacking
 2026-05-22 10:26:06 +01:00
Eric Allam c80b85e2cf docs(ai-chat): atomic onTurnComplete writes + Anthropic prose (#3693) 
## Summary

Three post-merge fixes for the AI Agents docs (#3226), all caught by
review after merge.

## Fixes

- **`onTurnComplete` examples now use `db.$transaction`** — both the
Database persistence "Complete example" and the Lifecycle hooks
reference example were doing two separate `await` calls
(`db.chat.update` then `db.chatSession.upsert`). That's the exact
non-atomic pattern the warning earlier on the persistence page calls out
as ❌: a refresh between the two writes reads a stale `lastEventId` and
duplicates the assistant message on resume. Both examples now use the
recommended atomic form.

- **Background injection self-review prose aligned with the code** — the
prose said "gpt-4o-mini" but the example above it had been swapped to
`claude-haiku-4-5`. The Anthropic-sweep script only touched code blocks;
this prose line wasn't picked up.

## Test plan

- [x] Both updated examples use `db.$transaction([...])`
- [x] Prose matches the model used in the code block
- [ ] Mintlify deployment passes
 2026-05-21 17:04:13 +01:00
Eric Allam 80bb600cd5 docs(ai-chat): AI Agents documentation for v4.5 (#3226) 
## Summary

Lands the full AI Agents documentation surface alongside the v4.5
release candidate of `@trigger.dev/sdk`. Covers `chat.agent` end to end
— defining agents, lifecycle hooks, the frontend transport, sub-agents,
recovery from cancel/crash/OOM, AI Prompts integration — and the
Sessions primitive that backs it.

## Coverage

- **Conceptual**: Overview, Quick Start, How it works.
- **Building agents**: Backend (`chat.agent` / `chat.createSession` /
raw primitives), Lifecycle hooks, Frontend transport, Server-side
`AgentChat`, Sessions reference, `chat.local` state primitive,
TypeScript types.
- **Features**: AI Prompts integration, Fast starts (Preload + Head
Start), Compaction, Pending Messages (steering), Background Injection
(`chat.inject` + `chat.defer`), Actions (undo / regenerate / edit),
Error handling.
- **Patterns (13)**: Sub-agents, Branching conversations, Code sandbox,
Database persistence, Persistence and replay, HITL, Tool result
auditing, Large payloads, Agent skills, OOM resilience, Recovery boot,
Trusted edge signals, Version upgrades.
- **Reference**: API Reference, Client Protocol (wire format), Testing
harness (`mockChatAgent`), MCP server tools, Upgrade guide, Changelog.

## Structure changes

- Top-level nav: AI → **Agents**, with sub-groups for *Building agents /
Features / Patterns / Reference*.
- New RC banner snippet on every page links to the supported AI SDK
versions table on the API Reference.
- All examples use Anthropic with `stopWhen: stepCountIs(15)`.

---------

Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-21 16:50:59 +01:00
github-actions[bot] 0b5b817a49 chore: release v4.5.0-rc.1 (#3691)
🚀 Publish Trigger.dev Docker / typecheck (push) Failing after 0s
Details
🚀 Publish Trigger.dev Docker / publish-webapp (push) Has been skipped
Details
🚀 Publish Trigger.dev Docker / publish-worker (push) Has been skipped
Details
🚀 Publish Trigger.dev Docker / publish-worker-v4 (push) Has been skipped
Details
🚀 Publish Trigger.dev Docker / units (push) Failing after 0s
Details
🧭 Helm Chart Release / lint-and-test (push) Has been cancelled
Details
🧭 Helm Chart Release / release (push) Has been cancelled
Details 
## Summary
2 bug fixes.

## Bug fixes
- Fix `chat.agent` skills silently missing in `trigger dev` for projects
whose task files read `process.env` at module top level (e.g. a
third-party SDK client initialized at import). Skill folders now bundle
into `.trigger/skills/` reliably regardless of which env vars are set
when the CLI launches.
([#3690](https://github.com/triggerdotdev/trigger.dev/pull/3690))
- Fix `COULD_NOT_FIND_EXECUTOR` when a task's definition is loaded via
`await import(...)` from inside another task's `run()`. The runtime
workers now register such tasks with a sentinel file context, and the
catalog logs a one-time warning per task id.
([#3688](https://github.com/triggerdotdev/trigger.dev/pull/3688))

<details>
<summary>Raw changeset output</summary>

⚠️⚠️⚠️⚠️⚠️⚠️

`main` is currently in **pre mode** so this branch has prereleases
rather than normal releases. If you want to exit prereleases, run
`changeset pre exit` on `main`.

⚠️⚠️⚠️⚠️⚠️⚠️

# Releases
## @trigger.dev/build@4.5.0-rc.1

### Patch Changes

-   Updated dependencies:
    -   `@trigger.dev/core@4.5.0-rc.1`

## trigger.dev@4.5.0-rc.1

### Patch Changes

- Fix `chat.agent` skills silently missing in `trigger dev` for projects
whose task files read `process.env` at module top level (e.g. a
third-party SDK client initialized at import). Skill folders now bundle
into `.trigger/skills/` reliably regardless of which env vars are set
when the CLI launches.
([#3690](https://github.com/triggerdotdev/trigger.dev/pull/3690))
- Fix `COULD_NOT_FIND_EXECUTOR` when a task's definition is loaded via
`await import(...)` from inside another task's `run()`. The runtime
workers now register such tasks with a sentinel file context, and the
catalog logs a one-time warning per task id.
([#3688](https://github.com/triggerdotdev/trigger.dev/pull/3688))
-   Updated dependencies:
    -   `@trigger.dev/core@4.5.0-rc.1`
    -   `@trigger.dev/build@4.5.0-rc.1`
    -   `@trigger.dev/schema-to-json@4.5.0-rc.1`

## @trigger.dev/core@4.5.0-rc.1

### Patch Changes

- Fix `COULD_NOT_FIND_EXECUTOR` when a task's definition is loaded via
`await import(...)` from inside another task's `run()`. The runtime
workers now register such tasks with a sentinel file context, and the
catalog logs a one-time warning per task id.
([#3688](https://github.com/triggerdotdev/trigger.dev/pull/3688))

## @trigger.dev/plugins@4.5.0-rc.1

### Patch Changes

-   Updated dependencies:
    -   `@trigger.dev/core@4.5.0-rc.1`

## @trigger.dev/python@4.5.0-rc.1

### Patch Changes

-   Updated dependencies:
    -   `@trigger.dev/core@4.5.0-rc.1`
    -   `@trigger.dev/build@4.5.0-rc.1`
    -   `@trigger.dev/sdk@4.5.0-rc.1`

## @trigger.dev/react-hooks@4.5.0-rc.1

### Patch Changes

-   Updated dependencies:
    -   `@trigger.dev/core@4.5.0-rc.1`

## @trigger.dev/redis-worker@4.5.0-rc.1

### Patch Changes

-   Updated dependencies:
    -   `@trigger.dev/core@4.5.0-rc.1`

## @trigger.dev/rsc@4.5.0-rc.1

### Patch Changes

-   Updated dependencies:
    -   `@trigger.dev/core@4.5.0-rc.1`

## @trigger.dev/schema-to-json@4.5.0-rc.1

### Patch Changes

-   Updated dependencies:
    -   `@trigger.dev/core@4.5.0-rc.1`

## @trigger.dev/sdk@4.5.0-rc.1

### Patch Changes

-   Updated dependencies:
    -   `@trigger.dev/core@4.5.0-rc.1`

</details>

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
helm-v4.5.0-rc.1 v.docker.4.5.0-rc.1 v4.5.0-rc.1 		2026-05-21 16:40:21 +01:00
Eric Allam 5ddb81ac8d fix(cli): stop chat.agent skills silently disappearing from trigger dev (#3690) 
## Summary

`trigger.dev dev` was silently dropping registered `chat.agent` skills
for any project whose task files read `process.env` at module top level
— e.g. a third-party SDK client initialized at import. The agent would
boot fine, but `skill.local()` failed at runtime with `ENOENT` because
the skill folder was never copied into `.trigger/skills/`.

## Design

The CLI ran two indexer passes in dev: the worker's own indexer (with
the full env it eventually executes tasks in), and a separate
skill-discovery indexer with only the CLI process's env. Top-level reads
of vars like `TRIGGER_API_URL` imported cleanly in the worker pass and
threw in the skill pass — the latter caught the error, warned, and
skipped skill copying. Failure was silent enough that `skill.local()`
only surfaced it at task runtime.

The skill registry is already part of the worker manifest. This PR drops
the duplicate pass and copies skill folders from that manifest after the
worker initializes. One indexer instead of two; a bad `SKILL.md` now
surfaces as a startup error instead of silently disappearing skills.

Deploy is unaffected — its skill discovery uses the project's
environment variables (fetched via the API, which fills in
`TRIGGER_API_URL` etc.), so the dev failure mode doesn't reach there.

## Test plan

- [x] New `references/agent-skills` reference project with
`skills.define` + a task that calls `skill.local()` and runs a bundled
script
- [x] On `main`, adding a top-level
`process.env.TRIGGER_API_URL!.includes(...)` read in any task file
reproduces the symptom: warning at dev startup, no `.trigger/skills/`
folder, `skill.local()` fails with ENOENT
- [x] On this branch, same project boots clean and `skill.local()` works
end-to-end
- [x] Deploy still works end-to-end with the new reference project
 2026-05-21 16:30:24 +01:00
nicktrn dbf9b4e7e5 fix(core,cli): register tasks loaded via dynamic import during run (#3688) 
On a warm worker process, a task whose `task()` definition is loaded via
`await import(...)` from inside another task's `run()` could end up
permanently missing from the catalog: the `task()` call fired with no
`_currentFileContext` set, `registerTaskMetadata` silently returned, and
Node's ESM module cache then blocked the worker's setContext + re-import
recovery from ever firing the call again. Subsequent runs of that task
on the same warm process failed with `COULD_NOT_FIND_EXECUTOR` until the
process hit `maxExecutionsPerProcess` and exited.

All five of these had to coincide on the same worker for the bug to
surface:

1. `processKeepAlive` enabled (so catalog state survives across runs).
2. A `run()` function (or lifecycle hook) does `await import(...)`.
3. The import's transitive static graph reaches a `task()` /
`schemaTask()` call.
4. The task containing the dynamic import is the **first** task to run
on a given warm worker process — so the dropped `task()` calls fire on
this process for the first time, are silently dropped, and Node's module
cache locks the wrong outcome in.
5. A subsequent run for one of the dropped task ids lands on the same
warm worker before it recycles.

The runtime workers now set a sentinel file context (`<no-context>`)
around the `executor.execute(...)` call, so `task()` invocations firing
during a run register normally. The catalog detects the sentinel and
emits a one-time `console.warn` per task id so the pattern stays visible
without spamming. The indexer never sets this context, so deploy-time
behavior is unchanged.

Repro is `references/hello-world/src/trigger/dynamicImportRepro*.ts`.
Verified end-to-end against a deployed image with firestarter
warm-starts on: pre-fix saw `COULD_NOT_FIND_EXECUTOR` on children that
landed on the parent-poisoned worker; post-fix all 23/23 runs succeeded
and the warning surfaces in the parent's run trace.
 2026-05-21 16:21:49 +01:00
github-actions[bot] acfba02409 chore: release v4.5.0-rc.0 (#3563)
🚀 Publish Trigger.dev Docker / typecheck (push) Failing after 4s
Details
🚀 Publish Trigger.dev Docker / units (push) Failing after 0s
Details
🚀 Publish Trigger.dev Docker / publish-webapp (push) Has been skipped
Details
🚀 Publish Trigger.dev Docker / publish-worker (push) Has been skipped
Details
🚀 Publish Trigger.dev Docker / publish-worker-v4 (push) Has been skipped
Details
🧭 Helm Chart Release / lint-and-test (push) Has been cancelled
Details
🧭 Helm Chart Release / release (push) Has been cancelled
Details 
## Summary
44 improvements, 1 bug fix.

## Improvements
- **AI Prompts** — define prompt templates as code alongside your tasks,
version them on deploy, and override the text or model from the
dashboard without redeploying. Prompts integrate with the Vercel AI SDK
via `toAISDKTelemetry()` (links every generation span back to the
prompt) and with `chat.agent` via `chat.prompt.set()` +
`chat.toStreamTextOptions()`.
([#3629](https://github.com/triggerdotdev/trigger.dev/pull/3629))
- **Code-defined, deploy-versioned templates** — define with
`prompts.define({ id, model, config, variables, content })`. Every
deploy creates a new version visible in the dashboard. Mustache-style
placeholders (`{{var}}`, `{{#cond}}...{{/cond}}`) with Zod / ArkType /
Valibot-typed variables.
- **Dashboard overrides** — change a prompt's text or model from the
dashboard without redeploying. Overrides take priority over the deployed
"current" version and are environment-scoped (dev / staging / production
independent).
- **Resolve API** — `prompt.resolve(vars, { version?, label? })` returns
the compiled `text`, resolved `model`, `version`, and labels. Standalone
`prompts.resolve<typeof handle>(slug, vars)` for cross-file resolution
with full type inference on slug and variable shape.
- **AI SDK integration** — spread `resolved.toAISDKTelemetry({ ...extra
})` into any `generateText` / `streamText` call and every generation
span links to the prompt in the dashboard alongside its input variables,
model, tokens, and cost.
- **`chat.agent` integration** — `chat.prompt.set(resolved)` stores the
resolved prompt run-scoped; `chat.toStreamTextOptions({ registry })`
pulls `system`, `model` (resolved via the AI SDK provider registry),
`temperature` / `maxTokens` / etc., and telemetry into a single spread
for `streamText`.
- **Management SDK** — `prompts.list()`, `prompts.versions(slug)`,
`prompts.promote(slug, version)`, `prompts.createOverride(slug, body)`,
`prompts.updateOverride(slug, body)`, `prompts.removeOverride(slug)`,
`prompts.reactivateOverride(slug, version)`.
- **Dashboard** — prompts list with per-prompt usage sparklines;
per-prompt detail with Template / Details / Versions / Generations /
Metrics tabs. AI generation spans get a custom inspector showing the
linked prompt's metadata, input variables, and template content
alongside model, tokens, cost, and the message thread.
- Adds `onBoot` to `chat.agent` — a lifecycle hook that fires once per
worker process picking up the chat. Runs for the initial run, preloaded
runs, AND reactive continuation runs (post-cancel, crash, `endRun`,
`requestUpgrade`, OOM retry), before any other hook. Use it to
initialize `chat.local`, open per-process resources, or re-hydrate state
from your DB on continuation — anywhere the SAME run picking up after
suspend/resume isn't enough.
([#3543](https://github.com/triggerdotdev/trigger.dev/pull/3543))
- **AI SDK `useChat` integration** — a custom
[`ChatTransport`](https://sdk.vercel.ai/docs/ai-sdk-ui/transport)
(`useTriggerChatTransport`) plugs straight into Vercel AI SDK's
`useChat` hook. Text streaming, tool calls, reasoning, and `data-*`
parts all work natively over Trigger.dev's realtime streams. No custom
API routes needed.
- **First-turn fast path (`chat.headStart`)** — opt-in handler that runs
the first turn's `streamText` step in your warm server process while the
agent run boots in parallel, cutting cold-start TTFC by roughly half
(measured 2801ms → 1218ms on `claude-sonnet-4-6`). The agent owns step
2+ (tool execution, persistence, hooks) so heavy deps stay where they
belong. Web Fetch handler works natively in Next.js, Hono, SvelteKit,
Remix, Workers, etc.; bridge to Express/Fastify/Koa via
`chat.toNodeListener`. New `@trigger.dev/sdk/chat-server` subpath.
- **Multi-turn durability via Sessions** — every chat is backed by a
durable Session that outlives any individual run. Conversations resume
across page refreshes, idle timeout, crashes, and deploys; `resume:
true` reconnects via `lastEventId` so clients only see new chunks.
`sessions.list` enumerates chats for inbox-style UIs.
- **Auto-accumulated history, delta-only wire** — the backend
accumulates the full conversation across turns; clients only ship the
new message each turn. Long chats never hit the 512 KiB body cap.
Register `hydrateMessages` to be the source of truth yourself.
- **Lifecycle hooks** — `onPreload`, `onChatStart`,
`onValidateMessages`, `hydrateMessages`, `onTurnStart`,
`onBeforeTurnComplete`, `onTurnComplete`, `onChatSuspend`,
`onChatResume` — for persistence, validation, and post-turn work.
- **Stop generation** — client-driven `transport.stopGeneration(chatId)`
aborts mid-stream; the run stays alive for the next message, partial
response is captured, and aborted parts (stuck `partial-call` tools,
in-progress reasoning) are auto-cleaned.
- **Tool approvals (HITL)** — tools with `needsApproval: true` pause
until the user approves or denies via `addToolApprovalResponse`. The
runtime reconciles the updated assistant message by ID and continues
`streamText`.
- **Steering and background injection** — `pendingMessages` injects user
messages between tool-call steps so users can steer the agent
mid-execution; `chat.inject()` + `chat.defer()` adds context from
background work (self-review, RAG, safety checks) between turns.
- **Actions** — non-turn frontend commands (undo, rollback, regenerate,
edit) sent via `transport.sendAction`. Fire `hydrateMessages` +
`onAction` only — no turn hooks, no `run()`. `onAction` can return a
`StreamTextResult` for a model response, or `void` for side-effect-only.
- **Typed state primitives** — `chat.local<T>` for per-run state
accessible from hooks, `run()`, tools, and subtasks (auto-serialized
through `ai.toolExecute`); `chat.store` for typed shared data between
agent and client; `chat.history` for reading and mutating the message
chain; `clientDataSchema` for typed `clientData` in every hook.
- **`chat.toStreamTextOptions()`** — one spread into `streamText` wires
up versioned system [Prompts](https://trigger.dev/docs/ai/prompts),
model resolution, telemetry metadata, compaction, steering, and
background injection.
- **Multi-tab coordination** — `multiTab: true` + `useMultiTabChat`
prevents duplicate sends and syncs state across browser tabs via
`BroadcastChannel`. Non-active tabs go read-only with live updates.
- **Network resilience** — built-in indefinite retry with bounded
backoff, reconnect on `online` / tab refocus / bfcache restore,
`Last-Event-ID` mid-stream resume. No app code needed.
- **Sessions** — a durable, run-aware stream channel keyed on a stable
`externalId`. A Session is the unit of state that owns a multi-run
conversation: messages flow through `.in`, responses through `.out`,
both survive run boundaries. Sessions back the new `chat.agent` runtime,
and you can build on them directly for any pattern that needs durable
bi-directional streaming across runs.
([#3542](https://github.com/triggerdotdev/trigger.dev/pull/3542))
- Add `ai.toolExecute(task)` so you can wire a Trigger subtask in as the
`execute` handler of an AI SDK `tool()` while defining `description` and
`inputSchema` yourself — useful when you want full control over the tool
surface and just need Trigger's subtask machinery for the body.
([#3546](https://github.com/triggerdotdev/trigger.dev/pull/3546))
- Type `chat.createStartSessionAction` against your chat agent so
`clientData` is typed end-to-end on the first turn:
([#3684](https://github.com/triggerdotdev/trigger.dev/pull/3684))
- Add `region` to the runs list / retrieve API: filter runs by region
(`runs.list({ region: "..." })` / `filter[region]=<masterQueue>`) and
read each run's executing region from the new `region` field on the
response.
([#3612](https://github.com/triggerdotdev/trigger.dev/pull/3612))
- Add `TRIGGER_BUILD_SKIP_REWRITE_TIMESTAMP=1` escape hatch for local
self-hosted builds whose buildx driver doesn't support
`rewrite-timestamp` alongside push (e.g. orbstack's default `docker`
driver).
([#3618](https://github.com/triggerdotdev/trigger.dev/pull/3618))
- Reject overlong `idempotencyKey` values at the API boundary so they no
longer trip an internal size limit on the underlying unique index and
surface as a generic 500. Inputs are capped at 2048 characters — well
above what `idempotencyKeys.create()` produces (a 64-character hash) and
above any realistic raw key. Applies to `tasks.trigger`,
`tasks.batchTrigger`, `batch.create` (Phase 1 streaming batches),
`wait.createToken`, `wait.forDuration`, and the input/session stream
waitpoint endpoints. Over-limit requests now return a structured 400
instead.
([#3560](https://github.com/triggerdotdev/trigger.dev/pull/3560))
- **AI SDK `useChat` integration** — a custom
[`ChatTransport`](https://sdk.vercel.ai/docs/ai-sdk-ui/transport)
(`useTriggerChatTransport`) plugs straight into Vercel AI SDK's
`useChat` hook. Text streaming, tool calls, reasoning, and `data-*`
parts all work natively over Trigger.dev's realtime streams. No custom
API routes needed.
- **First-turn fast path (`chat.headStart`)** — opt-in handler that runs
the first turn's `streamText` step in your warm server process while the
agent run boots in parallel, cutting cold-start TTFC by roughly half
(measured 2801ms → 1218ms on `claude-sonnet-4-6`). The agent owns step
2+ (tool execution, persistence, hooks) so heavy deps stay where they
belong. Web Fetch handler works natively in Next.js, Hono, SvelteKit,
Remix, Workers, etc.; bridge to Express/Fastify/Koa via
`chat.toNodeListener`. New `@trigger.dev/sdk/chat-server` subpath.
- **Multi-turn durability via Sessions** — every chat is backed by a
durable Session that outlives any individual run. Conversations resume
across page refreshes, idle timeout, crashes, and deploys; `resume:
true` reconnects via `lastEventId` so clients only see new chunks.
`sessions.list` enumerates chats for inbox-style UIs.
- **Auto-accumulated history, delta-only wire** — the backend
accumulates the full conversation across turns; clients only ship the
new message each turn. Long chats never hit the 512 KiB body cap.
Register `hydrateMessages` to be the source of truth yourself.
- **Lifecycle hooks** — `onPreload`, `onChatStart`,
`onValidateMessages`, `hydrateMessages`, `onTurnStart`,
`onBeforeTurnComplete`, `onTurnComplete`, `onChatSuspend`,
`onChatResume` — for persistence, validation, and post-turn work.
- **Stop generation** — client-driven `transport.stopGeneration(chatId)`
aborts mid-stream; the run stays alive for the next message, partial
response is captured, and aborted parts (stuck `partial-call` tools,
in-progress reasoning) are auto-cleaned.
- **Tool approvals (HITL)** — tools with `needsApproval: true` pause
until the user approves or denies via `addToolApprovalResponse`. The
runtime reconciles the updated assistant message by ID and continues
`streamText`.
- **Steering and background injection** — `pendingMessages` injects user
messages between tool-call steps so users can steer the agent
mid-execution; `chat.inject()` + `chat.defer()` adds context from
background work (self-review, RAG, safety checks) between turns.
- **Actions** — non-turn frontend commands (undo, rollback, regenerate,
edit) sent via `transport.sendAction`. Fire `hydrateMessages` +
`onAction` only — no turn hooks, no `run()`. `onAction` can return a
`StreamTextResult` for a model response, or `void` for side-effect-only.
- **Typed state primitives** — `chat.local<T>` for per-run state
accessible from hooks, `run()`, tools, and subtasks (auto-serialized
through `ai.toolExecute`); `chat.store` for typed shared data between
agent and client; `chat.history` for reading and mutating the message
chain; `clientDataSchema` for typed `clientData` in every hook.
- **`chat.toStreamTextOptions()`** — one spread into `streamText` wires
up versioned system [Prompts](https://trigger.dev/docs/ai/prompts),
model resolution, telemetry metadata, compaction, steering, and
background injection.
- **Multi-tab coordination** — `multiTab: true` + `useMultiTabChat`
prevents duplicate sends and syncs state across browser tabs via
`BroadcastChannel`. Non-active tabs go read-only with live updates.
- **Network resilience** — built-in indefinite retry with bounded
backoff, reconnect on `online` / tab refocus / bfcache restore,
`Last-Event-ID` mid-stream resume. No app code needed.
- Retry `TASK_PROCESS_SIGSEGV` task crashes under the user's retry
policy instead of failing the run on the first segfault. SIGSEGV in Node
tasks is frequently non-deterministic (native addon races, JIT/GC
interaction, near-OOM in native code, host issues), so retrying on a
fresh process often succeeds. The retry is gated by the task's existing
`retry` config + `maxAttempts` — same path `TASK_PROCESS_SIGTERM` and
uncaught exceptions already use — so tasks without a retry policy still
fail fast.
([#3552](https://github.com/triggerdotdev/trigger.dev/pull/3552))
- The public interfaces for a plugin system. Initially consolidated
authentication and authorization interfaces.
([#3499](https://github.com/triggerdotdev/trigger.dev/pull/3499))
- Add MollifierBuffer and MollifierDrainer primitives for trigger burst
smoothing.
([#3614](https://github.com/triggerdotdev/trigger.dev/pull/3614))

## Bug fixes
- Fix `LocalsKey<T>` type incompatibility across dual-package builds.
The phantom value-type brand no longer uses a module-level `unique
symbol`, so a single TypeScript compilation that resolves the type from
both the ESM and CJS outputs (which can happen under certain pnpm
hoisting layouts) no longer sees two structurally-incompatible variants
of the same type.
([#3626](https://github.com/triggerdotdev/trigger.dev/pull/3626))

<details>
<summary>Raw changeset output</summary>

⚠️⚠️⚠️⚠️⚠️⚠️

`main` is currently in **pre mode** so this branch has prereleases
rather than normal releases. If you want to exit prereleases, run
`changeset pre exit` on `main`.

⚠️⚠️⚠️⚠️⚠️⚠️

# Releases
## @trigger.dev/sdk@4.5.0-rc.0

### Minor Changes

- **AI Prompts** — define prompt templates as code alongside your tasks,
version them on deploy, and override the text or model from the
dashboard without redeploying. Prompts integrate with the Vercel AI SDK
via `toAISDKTelemetry()` (links every generation span back to the
prompt) and with `chat.agent` via `chat.prompt.set()` +
`chat.toStreamTextOptions()`.
([#3629](https://github.com/triggerdotdev/trigger.dev/pull/3629))

    ```ts
    import { prompts } from "@trigger.dev/sdk";
    import { generateText } from "ai";
    import { openai } from "@ai-sdk/openai";
    import { z } from "zod";

    export const supportPrompt = prompts.define({
      id: "customer-support",
      model: "gpt-4o",
      config: { temperature: 0.7 },
      variables: z.object({
        customerName: z.string(),
        plan: z.string(),
        issue: z.string(),
      }),
      content: `You are a support agent for Acme.

    Customer: {{customerName}} ({{plan}} plan)
    Issue: {{issue}}`,
    });

    const resolved = await supportPrompt.resolve({
      customerName: "Alice",
      plan: "Pro",
      issue: "Can't access billing",
    });

    const result = await generateText({
      model: openai(resolved.model ?? "gpt-4o"),
      system: resolved.text,
      prompt: "Can't access billing",
      ...resolved.toAISDKTelemetry(),
    });
    ```

    **What you get:**

- **Code-defined, deploy-versioned templates** — define with
`prompts.define({ id, model, config, variables, content })`. Every
deploy creates a new version visible in the dashboard. Mustache-style
placeholders (`{{var}}`, `{{#cond}}...{{/cond}}`) with Zod / ArkType /
Valibot-typed variables.
- **Dashboard overrides** — change a prompt's text or model from the
dashboard without redeploying. Overrides take priority over the deployed
"current" version and are environment-scoped (dev / staging / production
independent).
- **Resolve API** — `prompt.resolve(vars, { version?, label? })` returns
the compiled `text`, resolved `model`, `version`, and labels. Standalone
`prompts.resolve<typeof handle>(slug, vars)` for cross-file resolution
with full type inference on slug and variable shape.
- **AI SDK integration** — spread `resolved.toAISDKTelemetry({ ...extra
})` into any `generateText` / `streamText` call and every generation
span links to the prompt in the dashboard alongside its input variables,
model, tokens, and cost.
- **`chat.agent` integration** — `chat.prompt.set(resolved)` stores the
resolved prompt run-scoped; `chat.toStreamTextOptions({ registry })`
pulls `system`, `model` (resolved via the AI SDK provider registry),
`temperature` / `maxTokens` / etc., and telemetry into a single spread
for `streamText`.
- **Management SDK** — `prompts.list()`, `prompts.versions(slug)`,
`prompts.promote(slug, version)`, `prompts.createOverride(slug, body)`,
`prompts.updateOverride(slug, body)`, `prompts.removeOverride(slug)`,
`prompts.reactivateOverride(slug, version)`.
- **Dashboard** — prompts list with per-prompt usage sparklines;
per-prompt detail with Template / Details / Versions / Generations /
Metrics tabs. AI generation spans get a custom inspector showing the
linked prompt's metadata, input variables, and template content
alongside model, tokens, cost, and the message thread.

See [/docs/ai/prompts](https://trigger.dev/docs/ai/prompts) for the full
reference — template syntax, version resolution order, override
workflow, and type utilities (`PromptHandle`, `PromptIdentifier`,
`PromptVariables`).

- Adds `onBoot` to `chat.agent` — a lifecycle hook that fires once per
worker process picking up the chat. Runs for the initial run, preloaded
runs, AND reactive continuation runs (post-cancel, crash, `endRun`,
`requestUpgrade`, OOM retry), before any other hook. Use it to
initialize `chat.local`, open per-process resources, or re-hydrate state
from your DB on continuation — anywhere the SAME run picking up after
suspend/resume isn't enough.
([#3543](https://github.com/triggerdotdev/trigger.dev/pull/3543))

    ```ts
const userContext = chat.local<{ name: string; plan: string }>({ id:
"userContext" });

    export const myChat = chat.agent({
      id: "my-chat",
      onBoot: async ({ clientData, continuation }) => {
const user = await db.user.findUnique({ where: { id: clientData.userId }
});
        userContext.init({ name: user.name, plan: user.plan });
      },
      run: async ({ messages, signal }) =>
streamText({ model: openai("gpt-4o"), messages, abortSignal: signal }),
    });
    ```

Use `onBoot` (not `onChatStart`) for state setup that must run every
time a worker picks up the chat — `onChatStart` fires once per chat and
won't run on continuation, leaving `chat.local` uninitialized when
`run()` tries to use it.

- **AI Agents** — run AI SDK chat completions as durable Trigger.dev
agents instead of fragile API routes. Define an agent in one function,
point `useChat` at it from React, and the conversation survives page
refreshes, network blips, and process restarts.
([#3543](https://github.com/triggerdotdev/trigger.dev/pull/3543))

    ```ts
    import { chat } from "@trigger.dev/sdk/ai";
    import { streamText } from "ai";
    import { openai } from "@ai-sdk/openai";

    export const myChat = chat.agent({
      id: "my-chat",
      run: async ({ messages, signal }) =>
streamText({ model: openai("gpt-4o"), messages, abortSignal: signal }),
    });
    ```

    ```tsx
    import { useChat } from "@ai-sdk/react";
import { useTriggerChatTransport } from "@trigger.dev/sdk/chat/react";

const transport = useTriggerChatTransport({ task: "my-chat",
accessToken, startSession });
    const { messages, sendMessage } = useChat({ transport });
    ```

    **What you get:**

- **AI SDK `useChat` integration** — a custom
[`ChatTransport`](https://sdk.vercel.ai/docs/ai-sdk-ui/transport)
(`useTriggerChatTransport`) plugs straight into Vercel AI SDK's
`useChat` hook. Text streaming, tool calls, reasoning, and `data-*`
parts all work natively over Trigger.dev's realtime streams. No custom
API routes needed.
- **First-turn fast path (`chat.headStart`)** — opt-in handler that runs
the first turn's `streamText` step in your warm server process while the
agent run boots in parallel, cutting cold-start TTFC by roughly half
(measured 2801ms → 1218ms on `claude-sonnet-4-6`). The agent owns step
2+ (tool execution, persistence, hooks) so heavy deps stay where they
belong. Web Fetch handler works natively in Next.js, Hono, SvelteKit,
Remix, Workers, etc.; bridge to Express/Fastify/Koa via
`chat.toNodeListener`. New `@trigger.dev/sdk/chat-server` subpath.
- **Multi-turn durability via Sessions** — every chat is backed by a
durable Session that outlives any individual run. Conversations resume
across page refreshes, idle timeout, crashes, and deploys; `resume:
true` reconnects via `lastEventId` so clients only see new chunks.
`sessions.list` enumerates chats for inbox-style UIs.
- **Auto-accumulated history, delta-only wire** — the backend
accumulates the full conversation across turns; clients only ship the
new message each turn. Long chats never hit the 512 KiB body cap.
Register `hydrateMessages` to be the source of truth yourself.
- **Lifecycle hooks** — `onPreload`, `onChatStart`,
`onValidateMessages`, `hydrateMessages`, `onTurnStart`,
`onBeforeTurnComplete`, `onTurnComplete`, `onChatSuspend`,
`onChatResume` — for persistence, validation, and post-turn work.
- **Stop generation** — client-driven `transport.stopGeneration(chatId)`
aborts mid-stream; the run stays alive for the next message, partial
response is captured, and aborted parts (stuck `partial-call` tools,
in-progress reasoning) are auto-cleaned.
- **Tool approvals (HITL)** — tools with `needsApproval: true` pause
until the user approves or denies via `addToolApprovalResponse`. The
runtime reconciles the updated assistant message by ID and continues
`streamText`.
- **Steering and background injection** — `pendingMessages` injects user
messages between tool-call steps so users can steer the agent
mid-execution; `chat.inject()` + `chat.defer()` adds context from
background work (self-review, RAG, safety checks) between turns.
- **Actions** — non-turn frontend commands (undo, rollback, regenerate,
edit) sent via `transport.sendAction`. Fire `hydrateMessages` +
`onAction` only — no turn hooks, no `run()`. `onAction` can return a
`StreamTextResult` for a model response, or `void` for side-effect-only.
- **Typed state primitives** — `chat.local<T>` for per-run state
accessible from hooks, `run()`, tools, and subtasks (auto-serialized
through `ai.toolExecute`); `chat.store` for typed shared data between
agent and client; `chat.history` for reading and mutating the message
chain; `clientDataSchema` for typed `clientData` in every hook.
- **`chat.toStreamTextOptions()`** — one spread into `streamText` wires
up versioned system [Prompts](https://trigger.dev/docs/ai/prompts),
model resolution, telemetry metadata, compaction, steering, and
background injection.
- **Multi-tab coordination** — `multiTab: true` + `useMultiTabChat`
prevents duplicate sends and syncs state across browser tabs via
`BroadcastChannel`. Non-active tabs go read-only with live updates.
- **Network resilience** — built-in indefinite retry with bounded
backoff, reconnect on `online` / tab refocus / bfcache restore,
`Last-Event-ID` mid-stream resume. No app code needed.

See [/docs/ai-chat](https://trigger.dev/docs/ai-chat/overview) for the
full surface — quick start, three backend approaches (`chat.agent`,
`chat.createSession`, raw task), persistence and code-sandbox patterns,
type-level guides, and API reference.

- Add read primitives to `chat.history` for HITL flows:
`getPendingToolCalls()`, `getResolvedToolCalls()`,
`extractNewToolResults(message)`, `getChain()`, and
`findMessage(messageId)`. These lift the accumulator-walking logic that
customers building human-in-the-loop tools were re-implementing into the
SDK. ([#3543](https://github.com/triggerdotdev/trigger.dev/pull/3543))

Use `getPendingToolCalls()` to gate fresh user turns while a tool call
is awaiting an answer. Use `extractNewToolResults(message)` to dedup
tool results when persisting to your own store — the helper returns only
the parts whose `toolCallId` is not already resolved on the chain.

    ```ts
    const pending = chat.history.getPendingToolCalls();
    if (pending.length > 0) {
      // an addToolOutput is expected before a new user message
    }

    onTurnComplete: async ({ responseMessage }) => {
const newResults = chat.history.extractNewToolResults(responseMessage);
      for (const r of newResults) {
await db.toolResults.upsert({ id: r.toolCallId, output: r.output,
errorText: r.errorText });
      }
    };
    ```

- **Sessions** — a durable, run-aware stream channel keyed on a stable
`externalId`. A Session is the unit of state that owns a multi-run
conversation: messages flow through `.in`, responses through `.out`,
both survive run boundaries. Sessions back the new `chat.agent` runtime,
and you can build on them directly for any pattern that needs durable
bi-directional streaming across runs.
([#3542](https://github.com/triggerdotdev/trigger.dev/pull/3542))

    ```ts
    import { sessions, tasks } from "@trigger.dev/sdk";

    // Trigger a task and subscribe to its session output in one call
const { runId, stream } = await tasks.triggerAndSubscribe("my-task",
payload, {
      externalId: "user-456",
    });

    for await (const chunk of stream) {
      // ...
    }

// Enumerate existing sessions (powers inbox-style UIs without a
separate index)
for await (const s of sessions.list({ type: "chat.agent", tag:
"user:user-456" })) {
      console.log(s.id, s.externalId, s.createdAt, s.closedAt);
    }
    ```

See [/docs/ai-chat/overview](https://trigger.dev/docs/ai-chat/overview)
for the full surface — Sessions powers the durable, resumable chat
runtime described there.

### Patch Changes

- Add Agent Skills for `chat.agent`. Drop a folder with a `SKILL.md` and
any helper scripts/references next to your task code, register it with
`skills.define({ id, path })`, and the CLI bundles it into the deploy
image automatically — no `trigger.config.ts` changes. The agent gets a
one-line summary in its system prompt and discovers full instructions on
demand via `loadSkill`, with `bash` and `readFile` tools scoped
per-skill (path-traversal guards, output caps, abort-signal
propagation).
([#3543](https://github.com/triggerdotdev/trigger.dev/pull/3543))

    ```ts
const pdfSkill = skills.define({ id: "pdf-extract", path:
"./skills/pdf-extract" });

    chat.skills.set([await pdfSkill.local()]);
    ```

Built on the [AI SDK cookbook
pattern](https://ai-sdk.dev/cookbook/guides/agent-skills) — portable
across providers. SDK + CLI only for now; dashboard-editable `SKILL.md`
text is on the roadmap.

- Add `ai.toolExecute(task)` so you can wire a Trigger subtask in as the
`execute` handler of an AI SDK `tool()` while defining `description` and
`inputSchema` yourself — useful when you want full control over the tool
surface and just need Trigger's subtask machinery for the body.
([#3546](https://github.com/triggerdotdev/trigger.dev/pull/3546))

    ```ts
    const myTool = tool({
      description: "...",
      inputSchema: z.object({ ... }),
      execute: ai.toolExecute(mySubtask),
    });
    ```

`ai.tool(task)` (`toolFromTask`) keeps doing the all-in-one wrap and now
aligns its return type with AI SDK's `ToolSet`. Minimum `ai` peer raised
to `^6.0.116` to avoid cross-version `ToolSet` mismatches in monorepos.

- Stamp `gen_ai.conversation.id` (the chat id) on every span and metric
emitted from inside a `chat.task` or `chat.agent` run. Lets you filter
dashboard spans, runs, and metrics by the chat conversation that
produced them — independent of the run boundary, so multi-run chats
correlate cleanly. No code changes required on the user side.
([#3543](https://github.com/triggerdotdev/trigger.dev/pull/3543))

- Type `chat.createStartSessionAction` against your chat agent so
`clientData` is typed end-to-end on the first turn:
([#3684](https://github.com/triggerdotdev/trigger.dev/pull/3684))

    ```ts
    import { chat } from "@trigger.dev/sdk/ai";
    import type { myChat } from "@/trigger/chat";

export const startChatSession = chat.createStartSessionAction<typeof
myChat>("my-chat");

// In the browser, threaded from the transport's typed startSession
callback:
    const transport = useTriggerChatTransport<typeof myChat>({
      task: "my-chat",
startSession: ({ chatId, clientData }) => startChatSession({ chatId,
clientData }),
      // ...
    });
    ```

`ChatStartSessionParams` gains a typed `clientData` field — folded into
the first run's `payload.metadata` so `onPreload` / `onChatStart` see
the same shape per-turn `metadata` carries via the transport. The opaque
session-level `metadata` field is unchanged.

- Unit-test `chat.agent` definitions offline with `mockChatAgent` from
`@trigger.dev/sdk/ai/test`. Drives a real agent's turn loop in-process —
no network, no task runtime — so you can send messages, actions, and
stop signals via driver methods, inspect captured output chunks, and
verify hooks fire. Pairs with `MockLanguageModelV3` from `ai/test` for
model mocking. `setupLocals` lets you pre-seed `locals` (DB clients,
service stubs) before `run()` starts.
([#3543](https://github.com/triggerdotdev/trigger.dev/pull/3543))

The broader `runInMockTaskContext` harness it's built on lives at
`@trigger.dev/core/v3/test` — useful for unit-testing any task code, not
just chat.

- Add `region` to the runs list / retrieve API: filter runs by region
(`runs.list({ region: "..." })` / `filter[region]=<masterQueue>`) and
read each run's executing region from the new `region` field on the
response.
([#3612](https://github.com/triggerdotdev/trigger.dev/pull/3612))

-   Updated dependencies:
    -   `@trigger.dev/core@4.5.0-rc.0`

## @trigger.dev/build@4.5.0-rc.0

### Patch Changes

- Add Agent Skills for `chat.agent`. Drop a folder with a `SKILL.md` and
any helper scripts/references next to your task code, register it with
`skills.define({ id, path })`, and the CLI bundles it into the deploy
image automatically — no `trigger.config.ts` changes. The agent gets a
one-line summary in its system prompt and discovers full instructions on
demand via `loadSkill`, with `bash` and `readFile` tools scoped
per-skill (path-traversal guards, output caps, abort-signal
propagation).
([#3543](https://github.com/triggerdotdev/trigger.dev/pull/3543))

    ```ts
const pdfSkill = skills.define({ id: "pdf-extract", path:
"./skills/pdf-extract" });

    chat.skills.set([await pdfSkill.local()]);
    ```

Built on the [AI SDK cookbook
pattern](https://ai-sdk.dev/cookbook/guides/agent-skills) — portable
across providers. SDK + CLI only for now; dashboard-editable `SKILL.md`
text is on the roadmap.

-   Updated dependencies:
    -   `@trigger.dev/core@4.5.0-rc.0`

## trigger.dev@4.5.0-rc.0

### Patch Changes

- Add Agent Skills for `chat.agent`. Drop a folder with a `SKILL.md` and
any helper scripts/references next to your task code, register it with
`skills.define({ id, path })`, and the CLI bundles it into the deploy
image automatically — no `trigger.config.ts` changes. The agent gets a
one-line summary in its system prompt and discovers full instructions on
demand via `loadSkill`, with `bash` and `readFile` tools scoped
per-skill (path-traversal guards, output caps, abort-signal
propagation).
([#3543](https://github.com/triggerdotdev/trigger.dev/pull/3543))

    ```ts
const pdfSkill = skills.define({ id: "pdf-extract", path:
"./skills/pdf-extract" });

    chat.skills.set([await pdfSkill.local()]);
    ```

Built on the [AI SDK cookbook
pattern](https://ai-sdk.dev/cookbook/guides/agent-skills) — portable
across providers. SDK + CLI only for now; dashboard-editable `SKILL.md`
text is on the roadmap.

- Add `TRIGGER_BUILD_SKIP_REWRITE_TIMESTAMP=1` escape hatch for local
self-hosted builds whose buildx driver doesn't support
`rewrite-timestamp` alongside push (e.g. orbstack's default `docker`
driver).
([#3618](https://github.com/triggerdotdev/trigger.dev/pull/3618))

- The CLI MCP server's agent-chat tools (`start_agent_chat`,
`send_agent_message`, `close_agent_chat`) now run on the new Sessions
primitive, so AI assistants driving a `chat.agent` get the same
idempotent-by-`chatId`, durable-across-runs behavior the browser
transport gets. Required PAT scopes go from `write:inputStreams` to
`read:sessions` + `write:sessions`.
([#3546](https://github.com/triggerdotdev/trigger.dev/pull/3546))

- MCP `list_runs` tool: add a `region` filter input and surface each
run's executing region in the formatted summary.
([#3612](https://github.com/triggerdotdev/trigger.dev/pull/3612))

-   Updated dependencies:
    -   `@trigger.dev/core@4.5.0-rc.0`
    -   `@trigger.dev/build@4.5.0-rc.0`
    -   `@trigger.dev/schema-to-json@4.5.0-rc.0`

## @trigger.dev/core@4.5.0-rc.0

### Patch Changes

- Add Agent Skills for `chat.agent`. Drop a folder with a `SKILL.md` and
any helper scripts/references next to your task code, register it with
`skills.define({ id, path })`, and the CLI bundles it into the deploy
image automatically — no `trigger.config.ts` changes. The agent gets a
one-line summary in its system prompt and discovers full instructions on
demand via `loadSkill`, with `bash` and `readFile` tools scoped
per-skill (path-traversal guards, output caps, abort-signal
propagation).
([#3543](https://github.com/triggerdotdev/trigger.dev/pull/3543))

    ```ts
const pdfSkill = skills.define({ id: "pdf-extract", path:
"./skills/pdf-extract" });

    chat.skills.set([await pdfSkill.local()]);
    ```

Built on the [AI SDK cookbook
pattern](https://ai-sdk.dev/cookbook/guides/agent-skills) — portable
across providers. SDK + CLI only for now; dashboard-editable `SKILL.md`
text is on the roadmap.

- Reject overlong `idempotencyKey` values at the API boundary so they no
longer trip an internal size limit on the underlying unique index and
surface as a generic 500. Inputs are capped at 2048 characters — well
above what `idempotencyKeys.create()` produces (a 64-character hash) and
above any realistic raw key. Applies to `tasks.trigger`,
`tasks.batchTrigger`, `batch.create` (Phase 1 streaming batches),
`wait.createToken`, `wait.forDuration`, and the input/session stream
waitpoint endpoints. Over-limit requests now return a structured 400
instead.
([#3560](https://github.com/triggerdotdev/trigger.dev/pull/3560))

- **AI Agents** — run AI SDK chat completions as durable Trigger.dev
agents instead of fragile API routes. Define an agent in one function,
point `useChat` at it from React, and the conversation survives page
refreshes, network blips, and process restarts.
([#3543](https://github.com/triggerdotdev/trigger.dev/pull/3543))

    ```ts
    import { chat } from "@trigger.dev/sdk/ai";
    import { streamText } from "ai";
    import { openai } from "@ai-sdk/openai";

    export const myChat = chat.agent({
      id: "my-chat",
      run: async ({ messages, signal }) =>
streamText({ model: openai("gpt-4o"), messages, abortSignal: signal }),
    });
    ```

    ```tsx
    import { useChat } from "@ai-sdk/react";
import { useTriggerChatTransport } from "@trigger.dev/sdk/chat/react";

const transport = useTriggerChatTransport({ task: "my-chat",
accessToken, startSession });
    const { messages, sendMessage } = useChat({ transport });
    ```

    **What you get:**

- **AI SDK `useChat` integration** — a custom
[`ChatTransport`](https://sdk.vercel.ai/docs/ai-sdk-ui/transport)
(`useTriggerChatTransport`) plugs straight into Vercel AI SDK's
`useChat` hook. Text streaming, tool calls, reasoning, and `data-*`
parts all work natively over Trigger.dev's realtime streams. No custom
API routes needed.
- **First-turn fast path (`chat.headStart`)** — opt-in handler that runs
the first turn's `streamText` step in your warm server process while the
agent run boots in parallel, cutting cold-start TTFC by roughly half
(measured 2801ms → 1218ms on `claude-sonnet-4-6`). The agent owns step
2+ (tool execution, persistence, hooks) so heavy deps stay where they
belong. Web Fetch handler works natively in Next.js, Hono, SvelteKit,
Remix, Workers, etc.; bridge to Express/Fastify/Koa via
`chat.toNodeListener`. New `@trigger.dev/sdk/chat-server` subpath.
- **Multi-turn durability via Sessions** — every chat is backed by a
durable Session that outlives any individual run. Conversations resume
across page refreshes, idle timeout, crashes, and deploys; `resume:
true` reconnects via `lastEventId` so clients only see new chunks.
`sessions.list` enumerates chats for inbox-style UIs.
- **Auto-accumulated history, delta-only wire** — the backend
accumulates the full conversation across turns; clients only ship the
new message each turn. Long chats never hit the 512 KiB body cap.
Register `hydrateMessages` to be the source of truth yourself.
- **Lifecycle hooks** — `onPreload`, `onChatStart`,
`onValidateMessages`, `hydrateMessages`, `onTurnStart`,
`onBeforeTurnComplete`, `onTurnComplete`, `onChatSuspend`,
`onChatResume` — for persistence, validation, and post-turn work.
- **Stop generation** — client-driven `transport.stopGeneration(chatId)`
aborts mid-stream; the run stays alive for the next message, partial
response is captured, and aborted parts (stuck `partial-call` tools,
in-progress reasoning) are auto-cleaned.
- **Tool approvals (HITL)** — tools with `needsApproval: true` pause
until the user approves or denies via `addToolApprovalResponse`. The
runtime reconciles the updated assistant message by ID and continues
`streamText`.
- **Steering and background injection** — `pendingMessages` injects user
messages between tool-call steps so users can steer the agent
mid-execution; `chat.inject()` + `chat.defer()` adds context from
background work (self-review, RAG, safety checks) between turns.
- **Actions** — non-turn frontend commands (undo, rollback, regenerate,
edit) sent via `transport.sendAction`. Fire `hydrateMessages` +
`onAction` only — no turn hooks, no `run()`. `onAction` can return a
`StreamTextResult` for a model response, or `void` for side-effect-only.
- **Typed state primitives** — `chat.local<T>` for per-run state
accessible from hooks, `run()`, tools, and subtasks (auto-serialized
through `ai.toolExecute`); `chat.store` for typed shared data between
agent and client; `chat.history` for reading and mutating the message
chain; `clientDataSchema` for typed `clientData` in every hook.
- **`chat.toStreamTextOptions()`** — one spread into `streamText` wires
up versioned system [Prompts](https://trigger.dev/docs/ai/prompts),
model resolution, telemetry metadata, compaction, steering, and
background injection.
- **Multi-tab coordination** — `multiTab: true` + `useMultiTabChat`
prevents duplicate sends and syncs state across browser tabs via
`BroadcastChannel`. Non-active tabs go read-only with live updates.
- **Network resilience** — built-in indefinite retry with bounded
backoff, reconnect on `online` / tab refocus / bfcache restore,
`Last-Event-ID` mid-stream resume. No app code needed.

See [/docs/ai-chat](https://trigger.dev/docs/ai-chat/overview) for the
full surface — quick start, three backend approaches (`chat.agent`,
`chat.createSession`, raw task), persistence and code-sandbox patterns,
type-level guides, and API reference.

- Stamp `gen_ai.conversation.id` (the chat id) on every span and metric
emitted from inside a `chat.task` or `chat.agent` run. Lets you filter
dashboard spans, runs, and metrics by the chat conversation that
produced them — independent of the run boundary, so multi-run chats
correlate cleanly. No code changes required on the user side.
([#3543](https://github.com/triggerdotdev/trigger.dev/pull/3543))

- Fix `LocalsKey<T>` type incompatibility across dual-package builds.
The phantom value-type brand no longer uses a module-level `unique
symbol`, so a single TypeScript compilation that resolves the type from
both the ESM and CJS outputs (which can happen under certain pnpm
hoisting layouts) no longer sees two structurally-incompatible variants
of the same type.
([#3626](https://github.com/triggerdotdev/trigger.dev/pull/3626))

- Unit-test `chat.agent` definitions offline with `mockChatAgent` from
`@trigger.dev/sdk/ai/test`. Drives a real agent's turn loop in-process —
no network, no task runtime — so you can send messages, actions, and
stop signals via driver methods, inspect captured output chunks, and
verify hooks fire. Pairs with `MockLanguageModelV3` from `ai/test` for
model mocking. `setupLocals` lets you pre-seed `locals` (DB clients,
service stubs) before `run()` starts.
([#3543](https://github.com/triggerdotdev/trigger.dev/pull/3543))

The broader `runInMockTaskContext` harness it's built on lives at
`@trigger.dev/core/v3/test` — useful for unit-testing any task code, not
just chat.

- Retry `TASK_PROCESS_SIGSEGV` task crashes under the user's retry
policy instead of failing the run on the first segfault. SIGSEGV in Node
tasks is frequently non-deterministic (native addon races, JIT/GC
interaction, near-OOM in native code, host issues), so retrying on a
fresh process often succeeds. The retry is gated by the task's existing
`retry` config + `maxAttempts` — same path `TASK_PROCESS_SIGTERM` and
uncaught exceptions already use — so tasks without a retry policy still
fail fast.
([#3552](https://github.com/triggerdotdev/trigger.dev/pull/3552))

- Add `region` to the runs list / retrieve API: filter runs by region
(`runs.list({ region: "..." })` / `filter[region]=<masterQueue>`) and
read each run's executing region from the new `region` field on the
response.
([#3612](https://github.com/triggerdotdev/trigger.dev/pull/3612))

- **Sessions** — a durable, run-aware stream channel keyed on a stable
`externalId`. A Session is the unit of state that owns a multi-run
conversation: messages flow through `.in`, responses through `.out`,
both survive run boundaries. Sessions back the new `chat.agent` runtime,
and you can build on them directly for any pattern that needs durable
bi-directional streaming across runs.
([#3542](https://github.com/triggerdotdev/trigger.dev/pull/3542))

    ```ts
    import { sessions, tasks } from "@trigger.dev/sdk";

    // Trigger a task and subscribe to its session output in one call
const { runId, stream } = await tasks.triggerAndSubscribe("my-task",
payload, {
      externalId: "user-456",
    });

    for await (const chunk of stream) {
      // ...
    }

// Enumerate existing sessions (powers inbox-style UIs without a
separate index)
for await (const s of sessions.list({ type: "chat.agent", tag:
"user:user-456" })) {
      console.log(s.id, s.externalId, s.createdAt, s.closedAt);
    }
    ```

See [/docs/ai-chat/overview](https://trigger.dev/docs/ai-chat/overview)
for the full surface — Sessions powers the durable, resumable chat
runtime described there.

## @trigger.dev/plugins@4.5.0-rc.0

### Patch Changes

- The public interfaces for a plugin system. Initially consolidated
authentication and authorization interfaces.
([#3499](https://github.com/triggerdotdev/trigger.dev/pull/3499))
-   Updated dependencies:
    -   `@trigger.dev/core@4.5.0-rc.0`

## @trigger.dev/python@4.5.0-rc.0

### Patch Changes

-   Updated dependencies:
    -   `@trigger.dev/sdk@4.5.0-rc.0`
    -   `@trigger.dev/core@4.5.0-rc.0`
    -   `@trigger.dev/build@4.5.0-rc.0`

## @trigger.dev/react-hooks@4.5.0-rc.0

### Patch Changes

-   Updated dependencies:
    -   `@trigger.dev/core@4.5.0-rc.0`

## @trigger.dev/redis-worker@4.5.0-rc.0

### Patch Changes

- Add MollifierBuffer and MollifierDrainer primitives for trigger burst
smoothing.
([#3614](https://github.com/triggerdotdev/trigger.dev/pull/3614))

MollifierBuffer (`accept`, `pop`, `ack`, `requeue`, `fail`,
`evaluateTrip`) is a per-env FIFO over Redis with atomic Lua transitions
for status tracking. `evaluateTrip` is a sliding-window trip evaluator
the webapp gate uses to detect per-env trigger bursts.

MollifierDrainer pops entries through a polling loop with a
user-supplied handler. The loop survives transient Redis errors via
capped exponential backoff (up to 5s), and per-env pop failures don't
poison the rest of the batch — one env's blip is logged and counted as
failed for that tick. Rotation is two-level: orgs at the top, envs
within each org. The buffer maintains `mollifier:orgs` and
`mollifier:org-envs:${orgId}` atomically with per-env queues, so the
drainer walks orgs → envs directly without an in-memory cache. The
`maxOrgsPerTick` option (default 500) caps how many orgs are scheduled
per tick; for each picked org, one env is popped (rotating round-robin
within the org). An org with N envs gets the same per-tick scheduling
slot as an org with 1 env, so tenant-level drainage throughput is
determined by org count rather than env count.

-   Updated dependencies:
    -   `@trigger.dev/core@4.5.0-rc.0`

## @trigger.dev/rsc@4.5.0-rc.0

### Patch Changes

-   Updated dependencies:
    -   `@trigger.dev/core@4.5.0-rc.0`

## @trigger.dev/schema-to-json@4.5.0-rc.0

### Patch Changes

-   Updated dependencies:
    -   `@trigger.dev/core@4.5.0-rc.0`

</details>

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
v.docker.4.5.0-rc.0 v4.5.0-rc.0 helm-v4.5.0-rc.0 		2026-05-21 14:39:02 +01:00
Eric Allam 422f9f0bb2 ci: unblock changesets release PRs (#3687) 
## Summary

Two CI workflows were blocking the v4.5.0-rc.0 release PR (#3563) and
would block every future changeset release PR.

### 1. `changesets-pr.yml` — self-report `All PR Checks`

The changesets bot pushes commits authored by `GITHUB_TOKEN`. By GitHub
design, `GITHUB_TOKEN`-authored pushes can't trigger downstream
workflows (loop-prevention). That means `pr_checks.yml` never fires on
release-PR commits, leaving the required `All PR Checks` status
permanently `Expected — Waiting for status to be reported`. The PR can't
merge.

The fix: after `changesets/action` creates the PR, post a `success`
check with the exact `All PR Checks` context onto the PR's head SHA.
GitHub's required-check evaluation is satisfied by any check with the
right context name — the source doesn't have to be `pr_checks.yml`.

**Why this is safe:** the release PR only mechanically bumps
`package.json`, `pnpm-lock.yaml`, and `CHANGELOG.md` from changesets
that were already on `main` (and already ran full CI when they merged).
If a human ever pushes a commit to `changeset-release/main`,
`pr_checks.yml` fires on that push (real user, not `GITHUB_TOKEN`) and
posts its own `All PR Checks` status — last write wins for the same
context on the same SHA, so the human-push result overrides the
auto-success.

### 2. `vouch-check-pr.yml` — exempt `github-actions[bot]`

The `require-draft` job auto-closes any non-draft PR whose author is not
a `MEMBER`/`OWNER`/`COLLABORATOR`, with an explicit allowlist for
`devin-ai-integration[bot]` and `dependabot[bot]`. The changesets bot
publishes as `github-actions[bot]` with `author_association:
CONTRIBUTOR`, so every release PR was getting auto-closed on open with a
"please re-open as draft" comment. Add `github-actions[bot]` to the
exemption list.

## Test plan
- [ ] After merge, the next changeset bot push to
`changeset-release/main` should post `All PR Checks: success` on the
release PR's head SHA, and the PR should not get auto-closed by `Vouch -
Check PR`.
- [ ] Confirm `pr_checks.yml` still fires + gates normal
(human-authored) PRs to `main`.
 2026-05-21 14:22:52 +01:00
Eric Allam 89d085a496 fix(references): repair ai-chat typecheck against current wire shape (#3685) 
## Summary

Pre-existing typecheck errors in `references/ai-chat` against the
current SDK shape. Unblocks `pnpm exec tsc --noEmit` in the reference
project.

## What changed

Three categories of fixes inside `references/ai-chat`. No SDK changes.

### 1. `payload.messages` → `payload.message`

The wire payload is now delta-only — one new message per trigger,
optional. Old code in two raw-task files reads `payload.messages`
(plural array) which no longer exists.

```ts
// before
const messages = await conversation.addIncoming(currentPayload.messages, ...);

// after
const messages = await conversation.addIncoming(
  currentPayload.message ? [currentPayload.message] : [],
  ...
);
```

Same fix to the `chat.messages.on` handler, reading `msg.message`
(singular) instead of `msg.messages[length - 1]`.

### 2. `clientData` non-null assertion in `cf-trust-test`

`ChatTurnContext.clientData` is typed as `?: TClientData` on
`onTurnStart` / `run` event objects even when the agent declares a
`clientDataSchema`. The runtime validates against the schema before the
hook fires, so it's structurally non-null — but TypeScript can't know
that. Non-null assert for now.

Follow-up worth filing: narrow `ChatTurnContext.clientData` to
non-optional when the agent has a `clientDataSchema`. Same friction the
docs friction-test subagent flagged.

### 3. `stress-emit.parseConfig` retyped against `ModelMessage[]`

The `run` callback hands `messages: ModelMessage[]`, not `UIMessage[]`.
Update `parseConfig` to accept `ModelMessage[]` and pull text from
`content` (string or array-of-parts).

## Test plan

- [x] `pnpm exec tsc --noEmit` in `references/ai-chat` passes (was 8
errors, now 0)
 2026-05-21 12:41:18 +01:00
Eric Allam 9ff410bfa4 feat(sdk): type chat.createStartSessionAction against your chat agent (#3684) 
## Summary

Type `chat.createStartSessionAction` against the chat agent so
`clientData` is typed end-to-end on the first turn. Closes the gap where
`useTriggerChatTransport`'s `startSession` callback already hands you a
typed `clientData` (via the transport generic) but the server-side
action couldn't accept it without untyped routing through the `metadata`
field.

## Design

`ChatStartSessionParams` gains a typed `clientData` field via the new
generic:

```ts
export type ChatStartSessionParams<TChat extends AnyTask = AnyTask> = {
  chatId: string;
  clientData?: InferChatClientData<TChat>;
  triggerConfig?: Partial<SessionTriggerConfig>;
  metadata?: Record<string, unknown>;
};

function createChatStartSessionAction<TChat extends AnyTask = AnyTask>(
  taskId: string,
  options?: CreateChatStartSessionActionOptions
): (params: ChatStartSessionParams<TChat>) => Promise<ChatStartSessionResult>
```

When provided, `clientData` is folded into the first run's
`triggerConfig.basePayload.metadata`, so `onPreload` / `onChatStart` see
the same shape per-turn `metadata` carries via the transport. The opaque
session-level `metadata` field stays exactly as before — it lands on the
Session row, not the run payload.

## Usage

```ts
// actions.ts
import { chat } from "@trigger.dev/sdk/ai";
import type { myChat } from "@/trigger/chat";

export const startChatSession = chat.createStartSessionAction<typeof myChat>("my-chat");
```

```tsx
// Chat.tsx
const transport = useTriggerChatTransport<typeof myChat>({
  task: "my-chat",
  accessToken: ({ chatId }) => mintChatAccessToken(chatId),
  startSession: ({ chatId, clientData }) =>
    startChatSession({ chatId, clientData }),
});
```

## Test plan

- [x] `pnpm run build --filter @trigger.dev/sdk` passes
- [ ] Verify a `chat.agent` with `clientDataSchema` reads the typed
clientData from `onPreload` payload metadata on the first turn
 2026-05-21 11:51:35 +01:00
Eric Allam d343727021 fix(webapp,sdk): keep chat.agent snapshots on one object store (#3679) 
(`OBJECT_STORE_BASE_URL`) and a named protocol provider
(`OBJECT_STORE_DEFAULT_PROTOCOL=s3`), chat.agent session snapshot writes
landed in the named provider but reads fell through to the default — so
the recovery boot couldn't find the snapshot it had just written.

After a mid-stream cancel, the missing snapshot triggered a fallback
replay path that dropped the user's follow-up message, leaving the chat
stuck in `submitted` indefinitely.

Fix:
- New `/api/v1/sessions/:id/snapshot-url` route handles PUT + GET
  symmetrically — both prefix unprefixed keys with
  `OBJECT_STORE_DEFAULT_PROTOCOL` so they always round-trip through the
  same store.
- `Session.chatSnapshotStoragePath` persists the resolved URI on first
  write so future protocol changes don't strand existing snapshots.
  Reads prefer the stored URI and fall back to the computed default for
  pre-column sessions.
- SDK calls `createChatSnapshotUploadUrl` / `getChatSnapshotUrl`; the
  generic v1/v2 packets endpoints are unchanged.

## Test plan
- [x] Configure local with two providers (R2 default + MinIO `s3` named)
      and `OBJECT_STORE_DEFAULT_PROTOCOL=s3`.
- [x] Reproduce hang: send a message, cancel mid-stream, send another —
      without the fix it hangs in `submitted`; with the fix it streams.
- [x] Snapshot lands in the `s3`-protocol bucket and
      `Session.chatSnapshotStoragePath` is set after first write.
- [x] SDK unit tests pass; webapp typecheck passes.
 2026-05-20 20:22:59 +01:00
Eric Allam aec7e0a93d perf(webapp): index EnvironmentVariableValue.environmentId (#3675) 
Env-var lookups via `GET
/api/v1/projects/:projectRef/envvars/:slug/:name` run a Prisma
`findMany` on `EnvironmentVariableValue` filtered by `environmentId` +
`isSecret`. The only existing indexes are the primary key and a unique
on `(variableId, environmentId)`, so `environmentId` is never the
leading column — the planner falls back to a Parallel Seq Scan over the
whole table to find what is, in practice, a handful of rows per
environment.

Two changes:

- Add a btree index on `EnvironmentVariableValue(environmentId)` so the
planner switches to an index scan. The composite `(variableId,
environmentId)` unique stays in place; the new index is purely additive.
- Route the `findMany` inside `getEnvironmentWithRedactedSecrets`
through the read replica via a new `replicaClient` constructor param on
the repository (defaulting to `$replica`, mirroring how `prismaClient`
defaults to `prisma`). Writes and read-after-write methods stay on the
primary.

## Test plan

- [ ] `pnpm run typecheck --filter webapp`
- [ ] Confirm `EXPLAIN` plan flips from Parallel Seq Scan to an index
scan
- [ ] Existing env-var route tests still pass
 2026-05-20 13:43:44 +01:00
Daniel Sutton 6b46a34c46 fix(webapp): return 404 instead of 500 for missing env/project/schedule loaders (#3663) 
## Summary

- Dashboard loaders for runs / sessions / batches / schedule-detail
threw bare `Error("X not found")` when a slug didn't resolve. Remix
surfaces this as a 500 and Sentry captures it via auto-instrumentation,
producing ongoing noise from real users following stale preview-branch
or deleted-resource links (the URLs in those Sentry events all carry
`?_data=routes/...`, i.e. client-side revalidation, not full-page
navigation).
- Added a `throwNotFound(statusText)` helper in
`app/utils/httpErrors.ts` that throws a Response with status 404,
matching the established pattern in sibling routes (agents, alerts,
bulk-actions, etc.).
- Migrated 5 loader sites to `throwNotFound` (4× "Environment not
found", 1× "Schedule not found").
- Migrated 1 loader site (`runs._index` project branch) to
`redirectWithErrorMessage("/", request, "Project not found")` to match
the pre-existing convention used by every other dashboard route's
project-not-found branch.
- Intentionally **not** touched: bare `throw new Error("X not found")`
inside `resources.*` action routes (sit inside try/catch blocks that
already redirect with a flash message), the invariant assertion in
`vercel.connect.tsx`, and the admin config check in
`admin.api.v1.runs-replication.backfill.ts`.

## Where the fix is visible

Normal browser navigation to these URLs doesn't reach the buggy loaders
— the parent env-layout
(`_app.orgs.$organizationSlug.projects.$projectParam.env.$envParam/route.tsx`)
already filters missing envs/projects and redirects/404s before the
child loader runs. The bug fires exclusively when Remix calls a single
child loader via `?_data=routes/...`, which happens during client-side
navigation or `useRevalidator`. That matches every Sentry event URL.

## Test plan

- [x] Unit test for the new helper —
`apps/webapp/test/httpErrors.test.ts`
- [x] `pnpm run typecheck --filter webapp` clean
- [x] Manual verification via Playwright on `main` vs this branch (6
cases): main returns 500 for each defective `_data` URL; branch returns
404 or 204 + `X-Remix-Redirect` as designed
- [x] Verified user-visible 404 catch boundary on `schedules/<missing>`
(the one case reachable via normal nav)

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-20 09:46:44 +01:00
Eric Allam f91b96efb7 feat(sdk,core): preserve chat.agent context after cancel / OOM / crash (#3671) 2026-05-20 07:27:05 +01:00
Eric Allam 12d2125148 fix(sdk,core,build): SDK hardening pass (#3670) 
## Summary

Five hardening fixes across `@trigger.dev/sdk`, `@trigger.dev/core`, and
`@trigger.dev/build`.

- `tasks.triggerAndSubscribe` now forwards caller `requestOptions`
(custom API keys, per-request overrides) to the underlying
`apiClient.triggerTask` call instead of silently dropping them.
- `SSEStreamSubscription` no longer retries permanent client errors
forever. The default `nonRetryableStatuses` widens from `[404, 410]` to
`[400, 404, 409, 410, 422]`, so a malformed session-stream request fails
fast instead of busy-looping under bounded backoff.
- Session writer falls back to manually wiring the caller's
`AbortSignal` on Node 18, where `AbortSignal.any` is unavailable.
Caller-driven cancellation now propagates on every supported runtime.
- `TriggerChatTransport` throws immediately when a `chat.handover`
response is missing `X-Trigger-Chat-Access-Token`, instead of silently
downgrading every subsequent turn back to the handover path. `dispose()`
aborts every active `session.out` subscription before tearing the
coordinator down, so unmount/navigation no longer leaves SSE readers in
flight.
- Removed the experimental `@trigger.dev/build/extensions/secureExec`
build extension. It will return alongside the sandbox feature it was
built to support.

## Test plan

- [ ] `pnpm run build --filter @trigger.dev/sdk --filter
@trigger.dev/core --filter @trigger.dev/build`
- [ ] `pnpm --filter @trigger.dev/sdk test --run` (183 tests, including
chat / chat-server / sessions / handover)
- [ ] `pnpm --filter @trigger.dev/core test --run`
- [ ] Manually trigger a `chat.handover` whose response strips
`X-Trigger-Chat-Access-Token`, and confirm the transport throws
synchronously rather than degrading.
- [ ] Unmount a chat UI mid-stream and confirm the active `session.out`
SSE connection closes immediately.
 2026-05-19 16:10:32 +01:00
Iss 204a766bb1 feat(webapp): expose is_warm_start in TRQL runs schema (#3667) 
Add is_warm_start to TRQL runs schema so warm vs cold start data is
queryable
 2026-05-19 09:05:31 -04:00
Eric Allam 436b7a9ea1 fix(webapp): fold S2 token scope into access-token cache key (#3668) 
## Summary

The S2 access-token cache key was `${basin}:${streamPrefix}` — purely
server-derived but blind to the **scope/ops list** hardcoded one method
away. When the ops list changes in code (e.g. #3644 added `trim` so
`chat.agent`'s per-turn trim chain can issue `AppendRecord.trim()`),
pre-deploy tokens still in cache get returned to SDK callers for up to
the token's TTL (24h default), surfacing as `Operation not permitted`
403s on any op outside the old scope.

## Fix

Lift the ops list to a module constant and fold its sorted-join
fingerprint into the cache key:

```ts
const S2_TOKEN_OPS = ["append", "create-stream", "trim"] as const;
const S2_TOKEN_OPS_FINGERPRINT = [...S2_TOKEN_OPS].sort().join(",");

// in getS2AccessToken
const cacheKey = `${this.basin}:${this.streamPrefix}:${S2_TOKEN_OPS_FINGERPRINT}`;

// in s2IssueAccessToken
scope: { /* ... */ ops: [...S2_TOKEN_OPS], /* ... */ }
```

The fingerprint is derived from the single source of truth, so any
future scope change auto-invalidates without anyone remembering to bump
a literal version. The Unkey L1 (in-memory LRU) and L2 (Redis) layers
share the same key derivation, so both reset together on the next deploy
with no manual cache busting.

## Test plan

- [ ] `pnpm run typecheck --filter webapp`
- [ ] Run a multi-turn `chat.agent` chat via `references/ai-chat` and
confirm no `chat.agent: trim failed; will retry next turn` warn span
fires across turn-completes.
 2026-05-19 13:33:03 +01:00
Oskar Otwinowski 2fbac48e0d feat(webapp): prompt to clear TRIGGER_VERSION on disabling Vercel atomic deployments (#3666) 
- Ask user if they want to remove TRIGGER_VERSION when they disable
atomic deployments, and explain what is the situation if they leave it
as it is
- Install TRIGGER_SECRET keys as sensitive values in Vercel
<img width="1136" height="714" alt="image"
src="https://github.com/user-attachments/assets/a7351da1-5b2a-44e5-acdd-d30c9359f3ed"
/>
<img width="1136" height="714" alt="image"
src="https://github.com/user-attachments/assets/e773ede2-74cb-438e-811c-338f678d2f7d"
/>
<img width="1136" height="714" alt="image"
src="https://github.com/user-attachments/assets/c7b235a8-e06d-48d3-ac28-c5c9aacc6069"
/>
 2026-05-19 13:53:51 +02:00
Eric Allam e825409f0e ci(release): exit changeset pre mode before snapshot prerelease (#3665) 
## Summary

The prerelease (snapshot) path of the release workflow fails immediately
whenever `main` carries an active `.changeset/pre.json` (i.e. during an
in-progress RC cycle, like the current v4 RC):

```
🦋 error Snapshot release is not allowed in pre mode
🦋 To resolve this exit the pre mode by running `changeset pre exit`
```

This blocks `chat-prerelease` snapshots from main even though the
snapshots are unrelated to the RC cycle.

Adds a conditional `changeset pre exit` step right before `Snapshot
version` in the prerelease job. The job runs on a checkout with
`persist-credentials: false`, so the `pre.json` deletion stays on the
runner's working tree — main's persisted pre-mode state is untouched,
and v4 RC publishes keep working normally.

## Test plan

- [ ] Re-run the `🦋 Changesets Release` workflow with `type=prerelease`,
`ref=main`, `prerelease_tag=chat-prerelease` and confirm it gets past
the snapshot step and publishes.
- [ ] Confirm `.changeset/pre.json` on `main` is unchanged after the
run.
 2026-05-19 10:11:44 +01:00
Daniel Sutton 2f261e5e69 fix(webapp): catch loader/action throws before Remix serializes them (#3664) 
## Summary

Companion to #3536, which patched routes that already had a leaking
`catch (e) { return json({error: e.message}, 500) }`. That pattern can't
reach routes which have no catch in the first place — when those throw,
Remix's default error path serializes `error.message` into the response
body, and the SDK then wraps the leaked string as `TriggerApiError`.

Across 28 raw api.v1 loaders/actions plus one dashboard polling
endpoint, each handler now:

- Wraps its body in `try { ... } catch (error) { ... }`.
- Re-throws `Response` instances so auth helpers' `throw json(...)` /
`throw redirect(...)` pass through unchanged.
- Logs non-Response errors via `logger.error` so server-side visibility
is preserved.
- Returns a generic body — `{"error": "Internal Server Error"}` 500 for
raw API routes, or `{ changelogs: [] }` 200 for the polling widget
(degrade silently across transient blips; the consumer hook already
coped with empty payloads).

For six routes where #3536 left an inner try/catch covering only a
service call (`alertChannels`, `batches.results`,
`deployments.finalize`, `deployments.background-workers`,
`deployments.promote`, `projects.background-workers`): an outer
try/catch is added so auth/parsing failures are also sanitized. Inner
typed-error handling (`ServiceValidationError` → 422 with message, etc.)
is preserved exactly.

For two routes whose existing catch returned 400 + `error.message`
(`api.v1.authorization-code`, `api.v1.orgs.\$orgParam.projects` action):
the body is sanitized to a generic per-route string. **Status code stays
400** — clients that key on the 4xx/5xx distinction (and the SDK's
no-retry-on-4xx behavior) are unaffected.

## Test plan

- [x] \`pnpm run typecheck --filter webapp\`
- [x] Per-route synthetic-throw probe: inject \`throw new
Error("SYNTHETIC ...")\` at the top of each catch'd try, curl the route
with a dummy bearer, confirm the response body is the generic shape and
that the synthetic message lands server-side via \`logger.error\`. 29
routes verified.
- [x] Real-P1001 probe on the envvars loader: \`docker stop database\`
mid-flight, confirm response is generic 500 (not the leaked Prisma
message).
- [x] Sampled legitimate 4xx/2xx paths across each pattern variant
(naked-wrap, partial-expanded, 400-preserved) to confirm the wraps don't
interfere with normal control flow.
 2026-05-19 09:32:35 +01:00
nicktrn 5dacab0c72 fix: validate email format on magic link login (#3660) 
Reject non-email strings at the magic link form instead of accepting any
string and proceeding through rate-limit / authenticator steps.
 2026-05-18 16:18:58 +00:00
Oskar Otwinowski 02d61afc1c fix(webapp): sanitize OTel attributes on ClickHouse JSON parse rejection (#3659) 
Before fix:
<img width="1264" height="987" alt="image"
src="https://github.com/user-attachments/assets/24b8b85c-b89f-4109-9004-8d6af61d2849"
/>
After fix:
<img width="1264" height="987" alt="image"
src="https://github.com/user-attachments/assets/89bbc587-c50a-45ab-b203-dbe91028e918"
/>
 2026-05-18 17:42:31 +02:00
Eric Allam 427d9e078a feat(sdk): functional baseURL and fetch override on chat transports (#3655) 
## Summary

`TriggerChatTransport`, `AgentChat`, and `chat.createStartSessionAction`
now accept a string-or-function `baseURL` so callers can route per
endpoint — e.g. `.in/append` through a trusted edge proxy while keeping
`.out` SSE direct. The same surfaces add a `fetch` override for header
injection, custom retries, or proxy rewrites that go beyond URL routing.
SSE GETs are covered too via a new `fetchClient` option on
`SSEStreamSubscription`.

```ts
// TriggerChatTransport / AgentChat — endpoints: "in" | "out"
baseURL: ({ endpoint }) =>
  endpoint === "out" ? DIRECT : PROXY,

fetch: (url, init, ctx) => {
  init.headers = new Headers(init.headers);
  init.headers.set("traceparent", currentTraceparent());
  return globalThis.fetch(url, init);
},

// chat.createStartSessionAction — endpoints: "sessions" | "auth"
chat.createStartSessionAction("my-agent", {
  baseURL: ({ endpoint }) => (endpoint === "sessions" ? PROXY : DIRECT),
});
```

`streamBaseURL` on `TriggerChatTransport` is kept as a backwards-compat
alias and continues to win for the `"out"` endpoint when set.
Plain-string `baseURL` still applies to every endpoint, matching prior
behavior.
 2026-05-18 16:42:18 +01:00
Eric Allam 8b98e21b4b fix(sdk,core): cache realtime-stream credentials per slot with refresh on writer failure (#3658) 
## Summary

Hot-loop writers — `streams.writer` / `streams.pipe` on the run-scoped
side, `chat.response.write` / `chat.stream.*` on the session side — were
issuing a fresh `PUT` to mint S2 credentials for every chunk. On run
streams, each PUT also pushed the streamId onto
`TaskRun.realtimeStreams`,
so a chat-agent turn writing N chunks produced N PUTs and N duplicate
array pushes against the same row.

The SDK now caches the initialize response per cache slot: `(runId,
key)`
for run streams, the session id for session streams. First call PUTs as
before; subsequent calls reuse the cached promise. Hot-loop writers do
one PUT per slot for the lifetime of the cache.

S2 access tokens have a 1-day TTL. If a writer's `wait()` rejects (auth
error, expired token, network blip), the cache evicts the matching slot
so the next call re-PUTs and mints fresh credentials, identity-checked
so a concurrent caller's fresh promise isn't accidentally cleared.

## chat.agent guardrail

`streams.pipe / writer / append / read` called inside a `chat.agent` run
now logs a one-time warning pointing at `chat.response.write` /
`chat.stream.*` — `streams.*` is run-scoped and isn't visible on the
chat session. The ai-chat docs are updated to drop the old guidance
toward run-scoped streams.
 2026-05-18 16:41:58 +01:00
Daniel Sutton 906d5fafb6 feat(mollifier): trigger burst smoothing — Phase 1 (monitoring) (#3614) 
## Summary
- Introduce the Mollifier: a Redis-backed buffer for `trigger()` API
calls during traffic spikes, with a per-env trip evaluator and a drainer
ack-loop.
- Phase 1 is dual-write monitoring — every mollified trigger is buffered
to Redis AND continues to `engine.trigger`. No customer-facing behaviour
change.
- Telemetry events: `mollifier.would_mollify`, `mollifier.buffered`,
`mollifier.drained`, plus the `mollifier.decisions` counter.
  - Gated behind a feature flag (default off).

  ## Test plan
  - [x] `pnpm run test --filter @trigger.dev/redis-worker`
  - [x] `pnpm run test --filter webapp -- mollifier`
  - [x] Manual: with flag off, no behaviour change vs main
- [x] Manual: with flag on + threshold lowered, observe
`mollifier.buffered` + `mollifier.drained` log pairs with matching
`runId`

---------

Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-18 14:58:38 +01:00
Eric Allam 6c9f1f197e chore: parameterize docker host ports and wire s2-lite by default (#3642) 
## Summary

Two papercuts new contributors hit running this repo locally:

1. Fresh clones default to v1 (Redis-only) realtime streams, so Sessions
and `chat.agent` error with `"S2 configuration is missing"`, even though
the `s2` service is already in `docker/docker-compose.yml` and pre-seeds
a `trigger-local` basin. Wire `REALTIME_STREAMS_S2_*` to it in
`.env.example` so the new-contributor flow just works. (Also drop the s2
healthcheck: the image is distroless, so the `wget` check always reports
unhealthy.)

2. Two clones can't both run `pnpm run docker` because ports, project
name, and container names are all hardcoded. Parameterize every host
port as `${VAR:-default}`, drive the project name via
`COMPOSE_PROJECT_NAME` (with a top-level `name:` field as the default),
prefix container names with `${CONTAINER_PREFIX:-}`, and pass
`--env-file .env` so compose reads the same root `.env` the webapp does.
The "Running multiple instances side by side" block in `.env.example`
lists every overridable knob.

Also split the optional services (`electric-shard-1`, `ch-ui`,
`toxiproxy`, `nginx-h2`, `otel-collector`, `prometheus`, `grafana`) into
`docker-compose.extras.yml` behind a new `pnpm run docker:full` script.
The core stack keeps everything the webapp actually needs to boot:
postgres, redis, electric, minio, clickhouse + migrator, s2-lite.

Defaults match every previous hardcoded value, so existing setups keep
working without touching `.env`.

## Test plan

- [x] `pnpm run docker` on a clean clone brings up the core services on
the standard ports under the `triggerdotdev-docker` project name.
- [x] Setting `COMPOSE_PROJECT_NAME=triggerdotdev-docker-alt` + the
`*_HOST_PORT` overrides in `.env` brings up a second stack alongside the
default one with no port or container-name clashes.
- [x] Webapp boots cleanly against the default `.env.example` values;
`/healthcheck` returns 200, no S2 errors.
- [x] s2-lite basin `trigger-local` accepts an append + read via the
same REST endpoints the webapp uses.
- [x] `pnpm run docker:full` brings up the optional services alongside
the core ones in the same project.
 2026-05-18 09:28:58 +00:00