Commit Graph

1855 Commits

Author SHA1 Message Date
Matt Aitken f4a96bdf84 Fail dev and deploy on duplicate task ids (#3865)
## What

`dev` and `deploy` now fail with a clear error when two tasks are
defined with the same id — including across task types (e.g. a scheduled
task and a regular task sharing an id).

## Why

Tasks are registered into the resource catalog keyed by id, so a second
definition with the same id silently overwrote the first. One of the
tasks would just vanish from the worker with no warning — easy to miss,
hard to debug. (Any earlier duplicate-id check ran against the
post-registration task list, which is already de-duplicated, so it never
actually fired.)

## How

- **Detect at registration** (`@trigger.dev/core`):
`StandardResourceCatalog` records a collision when a task id is
registered more than once, capturing the files involved — the only point
where both definitions are visible before the id-keyed map collapses
them. Exposed via `listTaskIdCollisions()`.
- **Fail indexing** (`trigger.dev` CLI): both index workers report
collisions via a new `TASKS_FAILED_TO_INDEX` message;
`indexWorkerManifest` rejects with a new `DuplicateTaskIdsError`. `dev`
renders a dedicated error (offending ids + files + docs link); `deploy`
fails with the same message. Runtime worker boot is unaffected — it
never reads the collisions.
- **Server-side backstop** (webapp): background-worker registration also
rejects duplicate ids with a clear `ServiceValidationError`, so
duplicates are caught even from an older CLI.

## Testing

- Unit tests for collision collection in the catalog and for the
error-message formatting (standard, same-file, and 3+-definition cases).
- Verified end to end against a local webapp: a project with a regular
task and a scheduled task sharing an id now fails `dev` with the
dedicated error; a project with distinct ids still starts normally.

## Changeset

Patch for `@trigger.dev/core` and `trigger.dev`.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-09 12:10:49 +01:00
Eric Allam 18b90285b2 feat(cli): set up AI tooling in trigger init and add getting-started skill (#3872)
## Summary

`trigger init` now sets up your AI coding assistant as part of project
setup. Instead of the old either/or "MCP or CLI" prompt, it offers the
MCP server and agent skills together, then asks whether to scaffold with
the CLI or let your assistant do it.

A new `getting-started` agent skill backs that hand-off: it teaches the
assistant the bootstrap recipe (install the SDK, write
`trigger.config.ts`, scaffold a first task, wire tsconfig/gitignore, run
`trigger dev`) and is explicit about the two steps that genuinely need a
human (`trigger login` and copying the DEV secret key from the
dashboard). It ships in the CLI alongside the existing skills,
version-matched to your SDK.

Prompt-once gating is shared, so opting in or out during `init` means
`trigger dev` won't ask about skills again.
2026-06-09 11:41:32 +01:00
Eric Allam 8b85da1b26 feat(cli): install Trigger.dev agent skills into your coding agent (#3868)
## Summary

`trigger skills` installs Trigger.dev agent skills into your coding
agent so it knows how to write Trigger.dev code: tasks, schedules,
realtime, and `chat.agent` AI agents. The skills are `SKILL.md` files
(the open Agent Skills format) bundled with the CLI and copied into each
tool's native skills directory (Claude Code, Cursor, GitHub Copilot, and
Codex / `AGENTS.md`), version-matched to the CLI you run. `trigger dev`
offers to install them on first run, and a one-line always-on pointer is
written into your `CLAUDE.md` / Cursor rules / etc. so the agent always
knows which skills are available and loads the right one on demand.

This replaces the old `install-rules` command, which stays as an alias.
Four skills ship to start: `authoring-tasks`, `realtime-and-frontend`,
`authoring-chat-agent`, and `chat-agent-advanced`.
2026-06-08 17:44:02 +01:00
nicktrn fa15438e42 perf(ci): speed up unit tests with LPT sharding + container scoping (#3855)
Speeds up and de-flakes the unit-test suite: testcontainers booted once
per vitest worker (per-test isolation kept only where a test runs
background redis work that outlives it), a duration-weighted shard
sequencer so each shard does roughly equal work, the slowest suites
split, two genuine flakes fixed (`streamBatchItems` shared-redis leak;
run-engine waits that relied on fixed sleeps), and transient DockerHub
pulls retried.

**Timings (CI, per-shard wall):** worst unit-test shard ~771s → ~294s;
packages/webapp shards ~250-270s, most internal ~190-240s. All 25 shards
green.

A shard breaks down as ~70s fixed setup (install / image-pull /
generate) + ~70s cold `^build` + the actual container tests. So the
remaining cost is mostly the tests themselves plus that fixed setup.

**Next (separate, timings):**
- **typecheck (~6m24s)** — the slowest check overall; bound by
full-graph `tsc`, not the TS version (a TS6 branch is still ~6m17s). The
real lever is **tsgo** (the Go compiler).
- Possible later: turbo CI caching could trim the ~70s cold build on
*warm* runs, but it's conditional (cold runs rebuild anyway) and doesn't
touch setup or test time — secondary.

`cli-v3` e2e and `sdk-compat` are path-gated (don't run on test-infra
changes) and already comfortably fast.
2026-06-07 12:00:32 +01:00
Eric Allam fa4804e6a7 chore(core,sdk): move the AI SDK v7 forward-compat typecheck out of CI (#3854) 2026-06-06 18:34:27 +01:00
nicktrn 707bf1adb4 ci: reduce unit test flakiness and shard re-run cost (#3844)
A unit-test shard recently failed on a timing race rather than a real
regression - a run-engine waitpoint test sleeps 1250ms waiting on a
1000ms timeout that's processed by a ~1000ms worker poll, so on a
CPU-starved shard the margin evaporates and the whole matrix goes red.
Because `fail-fast` defaults on, that one flake cancels the sibling
shards, and the only recovery is re-running the entire matrix "just to
be sure" - which is itself slow.

This is the low-risk first pass at that pain:

- `fail-fast: false` on the webapp and internal shard matrices, so one
flaky shard no longer cancels its siblings. "Re-run failed jobs" now
re-runs just the failed shard instead of the whole matrix.
- CI-scoped `retry: process.env.CI ? 2 : 0` on the timing-sensitive
packages (`run-engine`, `redis-worker`, `schedule-engine`). Flakes
self-heal in CI; local runs stay at `retry: 0` so they still surface in
dev. A stopgap until the timing tests are made deterministic.
- `fetch-depth: 1` on the unit-test checkouts - they don't use git
history, so the full clone was wasted setup time across ~20 jobs.
- Reconcile the pre-pull image tags with what testcontainers actually
pulls (`redis:7-alpine` -> `redis:7.2`, `ryuk:0.11.0` -> `ryuk:0.14.0`)
and add `minio/minio:latest` to the webapp pre-pull. Otherwise those
images pull unauthenticated at test time and risk Docker Hub rate-limit
flakes (worst on fork PRs, where the authenticated pre-pull is skipped
entirely).

Deeper follow-ups - bigger runners, turbo remote cache, runtime-weighted
sharding, and the real root-cause fix (container reuse / template-DB
isolation + deterministic timing tests) - are tracked under TRI-10484.
2026-06-05 17:59:11 +01:00
Eric Allam 96f4c1bf2c chore(core,sdk): make the ai-v7 typecheck pass deterministic (#3847)
## Summary

The SDK and core packages run a second, forward-compat typecheck pass
(`tsc --noEmit -p tsconfig.ai-v7.json`) that remaps the `"ai"` import to
the ESM-only AI SDK 7 canary, so we catch source that only compiles
against one major. That pass inherited `composite: true` from the base
tsconfig, which makes `tsc` write a `.tsbuildinfo` even under
`--noEmit`.

Incremental buildinfo caches each file's resolved module format (CJS vs
ESM) and module resolution. When that state goes stale or is replayed,
the v7 pass can report spurious `TS1479` ("CommonJS module ... cannot
`require` an ECMAScript module") errors on the `"ai"` import even though
the source is fine in a clean checkout. Because this pass shares the
typecheck job that gates the Docker image publish, a spurious failure
there blocks publishing.

## Fix

Set `composite: false` and `incremental: false` on both
`tsconfig.ai-v7.json` files. The pass is `--noEmit` only, so it never
needed incremental state. Now each run is a clean, full check that
writes no buildinfo and can't replay stale resolution.

Verified: both `@trigger.dev/sdk` and `@trigger.dev/core` typecheck
green, and neither writes an ai-v7 `.tsbuildinfo` anymore.
2026-06-05 14:13:47 +00:00
github-actions[bot] a730faadfe chore: release v4.5.0-rc.5 (#3808)
🚀 Publish Trigger.dev Docker / typecheck (push) Failing after 3s
🚀 Publish Trigger.dev Docker / publish-webapp (push) Has been skipped
🚀 Publish Trigger.dev Docker / publish-worker (push) Has been skipped
🚀 Publish Trigger.dev Docker / publish-worker-v4 (push) Has been skipped
🚀 Publish Trigger.dev Docker / units (push) Failing after 4s
🧭 Helm Chart Release / lint-and-test (push) Has been cancelled
🧭 Helm Chart Release / release (push) Has been cancelled
## Summary
1 new feature, 8 improvements, 1 bug fix.

## Highlights

- Add optional `shouldPauseScaling` to the supervisor consumer pool
scaling options to freeze scale-up while it returns true (scale-down
stays allowed).
([#3836](https://github.com/triggerdotdev/trigger.dev/pull/3836))

## Improvements
- The MCP server no longer tells the AI agent to wait for a run to
complete after every `trigger_task` call. Waiting is now opt-in: the
agent only waits when you ask it to (for example "trigger and then wait
for it to finish"). This avoids burning tokens polling runs you didn't
need to block on and keeps responses clearer.
([#3838](https://github.com/triggerdotdev/trigger.dev/pull/3838))
- Update the bundled OpenTelemetry packages to their latest releases
(`@opentelemetry/sdk-node` 0.218.0, `@opentelemetry/core` 2.7.1,
`@opentelemetry/host-metrics` 0.38.3).
([#3810](https://github.com/triggerdotdev/trigger.dev/pull/3810))
- `envvars.upload` now accepts an optional `isSecret` flag, letting you
create the imported variables as secret (redacted) environment
variables. When omitted, variables default to non-secret.
([#3809](https://github.com/triggerdotdev/trigger.dev/pull/3809))
- Offload large trigger payloads to object storage before sending the
trigger API request. The SDK uploads packets at or above the existing
128KB limit and sends an `application/store` pointer instead of
embedding large JSON in the request body. `TriggerTaskRequestBody` now
validates that `application/store` payloads are non-empty storage paths.
([#3785](https://github.com/triggerdotdev/trigger.dev/pull/3785))
- Make mollifier buffer and drainer internals configurable.
`MollifierBuffer` now accepts `ackGraceTtlSeconds`,
`maxRetriesPerRequest`, `reconnectStepMs`, and `reconnectMaxMs` options,
and `MollifierDrainer` accepts `maxBackoffMs` and `backoffFloorMs`. All
default to their previous hardcoded values, so existing behaviour is
unchanged.
([#3822](https://github.com/triggerdotdev/trigger.dev/pull/3822))
- `MollifierDrainer` accepts a `drainBatchSize` option (default 1) that
controls how many entries are popped per env per tick — in-flight
handlers remain capped by the global `concurrency`. `MollifierBuffer`
also gains `getDrainingCount()` / `listStaleDraining()`, backed by a new
`mollifier:draining` ZSET maintained atomically with
pop/ack/fail/requeue (observability-only).
([#3797](https://github.com/triggerdotdev/trigger.dev/pull/3797))
- Adds AI SDK 7 support. The `ai` peer range now includes v7, and the
`chat.agent` / chat surfaces work against v7's ESM-only build. On v7,
install `@ai-sdk/otel` alongside `ai` and the SDK registers it for you
so `experimental_telemetry` spans keep flowing into your run traces (v7
stopped emitting them from `ai` core). v5 and v6 keep working unchanged.
([#3833](https://github.com/triggerdotdev/trigger.dev/pull/3833))
- `useTriggerChatTransport` now recovers when restored session state
points at a session that no longer exists in the current environment
([#3816](https://github.com/triggerdotdev/trigger.dev/pull/3816))

## Bug fixes
- Fix `@trigger.dev/core` build: cast the underlying log record exporter
when calling `forceFlush` so it typechecks against the updated
OpenTelemetry `LogRecordExporter` type (which no longer declares
`forceFlush`).
([#3829](https://github.com/triggerdotdev/trigger.dev/pull/3829))

<details>
<summary>Raw changeset output</summary>

⚠️⚠️⚠️⚠️⚠️⚠️

`main` is currently in **pre mode** so this branch has prereleases
rather than normal releases. If you want to exit prereleases, run
`changeset pre exit` on `main`.

⚠️⚠️⚠️⚠️⚠️⚠️

# Releases
## @trigger.dev/build@4.5.0-rc.5

### Patch Changes

-   Updated dependencies:
    -   `@trigger.dev/core@4.5.0-rc.5`

## trigger.dev@4.5.0-rc.5

### Patch Changes

- The MCP server no longer tells the AI agent to wait for a run to
complete after every `trigger_task` call. Waiting is now opt-in: the
agent only waits when you ask it to (for example "trigger and then wait
for it to finish"). This avoids burning tokens polling runs you didn't
need to block on and keeps responses clearer.
([#3838](https://github.com/triggerdotdev/trigger.dev/pull/3838))
- Update the bundled OpenTelemetry packages to their latest releases
(`@opentelemetry/sdk-node` 0.218.0, `@opentelemetry/core` 2.7.1,
`@opentelemetry/host-metrics` 0.38.3).
([#3810](https://github.com/triggerdotdev/trigger.dev/pull/3810))
-   Updated dependencies:
    -   `@trigger.dev/core@4.5.0-rc.5`
    -   `@trigger.dev/build@4.5.0-rc.5`
    -   `@trigger.dev/schema-to-json@4.5.0-rc.5`

## @trigger.dev/core@4.5.0-rc.5

### Patch Changes

- Add optional `shouldPauseScaling` to the supervisor consumer pool
scaling options to freeze scale-up while it returns true (scale-down
stays allowed).
([#3836](https://github.com/triggerdotdev/trigger.dev/pull/3836))

- Fix `@trigger.dev/core` build: cast the underlying log record exporter
when calling `forceFlush` so it typechecks against the updated
OpenTelemetry `LogRecordExporter` type (which no longer declares
`forceFlush`).
([#3829](https://github.com/triggerdotdev/trigger.dev/pull/3829))

- `envvars.upload` now accepts an optional `isSecret` flag, letting you
create the imported variables as secret (redacted) environment
variables. When omitted, variables default to non-secret.
([#3809](https://github.com/triggerdotdev/trigger.dev/pull/3809))

    ```ts
    await envvars.upload("proj_1234", "prod", {
      variables: { STRIPE_SECRET_KEY: "sk_live_..." },
      isSecret: true,
    });
    ```

- Offload large trigger payloads to object storage before sending the
trigger API request. The SDK uploads packets at or above the existing
128KB limit and sends an `application/store` pointer instead of
embedding large JSON in the request body. `TriggerTaskRequestBody` now
validates that `application/store` payloads are non-empty storage paths.
([#3785](https://github.com/triggerdotdev/trigger.dev/pull/3785))

Payload uploads use the same resolved `ApiClient` as the trigger call
(including `requestOptions.clientConfig`), not only the global
`apiClientManager.client` — so custom `baseURL`, access token, and
preview branch apply to both presign and trigger.

- Update the bundled OpenTelemetry packages to their latest releases
(`@opentelemetry/sdk-node` 0.218.0, `@opentelemetry/core` 2.7.1,
`@opentelemetry/host-metrics` 0.38.3).
([#3810](https://github.com/triggerdotdev/trigger.dev/pull/3810))

## @trigger.dev/plugins@4.5.0-rc.5

### Patch Changes

-   Updated dependencies:
    -   `@trigger.dev/core@4.5.0-rc.5`

## @trigger.dev/python@4.5.0-rc.5

### Patch Changes

-   Updated dependencies:
    -   `@trigger.dev/sdk@4.5.0-rc.5`
    -   `@trigger.dev/core@4.5.0-rc.5`
    -   `@trigger.dev/build@4.5.0-rc.5`

## @trigger.dev/react-hooks@4.5.0-rc.5

### Patch Changes

-   Updated dependencies:
    -   `@trigger.dev/core@4.5.0-rc.5`

## @trigger.dev/redis-worker@4.5.0-rc.5

### Patch Changes

- Make mollifier buffer and drainer internals configurable.
`MollifierBuffer` now accepts `ackGraceTtlSeconds`,
`maxRetriesPerRequest`, `reconnectStepMs`, and `reconnectMaxMs` options,
and `MollifierDrainer` accepts `maxBackoffMs` and `backoffFloorMs`. All
default to their previous hardcoded values, so existing behaviour is
unchanged.
([#3822](https://github.com/triggerdotdev/trigger.dev/pull/3822))
- `MollifierDrainer` accepts a `drainBatchSize` option (default 1) that
controls how many entries are popped per env per tick — in-flight
handlers remain capped by the global `concurrency`. `MollifierBuffer`
also gains `getDrainingCount()` / `listStaleDraining()`, backed by a new
`mollifier:draining` ZSET maintained atomically with
pop/ack/fail/requeue (observability-only).
([#3797](https://github.com/triggerdotdev/trigger.dev/pull/3797))
-   Updated dependencies:
    -   `@trigger.dev/core@4.5.0-rc.5`

## @trigger.dev/rsc@4.5.0-rc.5

### Patch Changes

-   Updated dependencies:
    -   `@trigger.dev/core@4.5.0-rc.5`

## @trigger.dev/schema-to-json@4.5.0-rc.5

### Patch Changes

-   Updated dependencies:
    -   `@trigger.dev/core@4.5.0-rc.5`

## @trigger.dev/sdk@4.5.0-rc.5

### Patch Changes

- Adds AI SDK 7 support. The `ai` peer range now includes v7, and the
`chat.agent` / chat surfaces work against v7's ESM-only build. On v7,
install `@ai-sdk/otel` alongside `ai` and the SDK registers it for you
so `experimental_telemetry` spans keep flowing into your run traces (v7
stopped emitting them from `ai` core). v5 and v6 keep working unchanged.
([#3833](https://github.com/triggerdotdev/trigger.dev/pull/3833))

- `useTriggerChatTransport` now recovers when restored session state
points at a session that no longer exists in the current environment
([#3816](https://github.com/triggerdotdev/trigger.dev/pull/3816))

- Offload large trigger payloads to object storage before sending the
trigger API request. The SDK uploads packets at or above the existing
128KB limit and sends an `application/store` pointer instead of
embedding large JSON in the request body. `TriggerTaskRequestBody` now
validates that `application/store` payloads are non-empty storage paths.
([#3785](https://github.com/triggerdotdev/trigger.dev/pull/3785))

Payload uploads use the same resolved `ApiClient` as the trigger call
(including `requestOptions.clientConfig`), not only the global
`apiClientManager.client` — so custom `baseURL`, access token, and
preview branch apply to both presign and trigger.

- Update the bundled OpenTelemetry packages to their latest releases
(`@opentelemetry/sdk-node` 0.218.0, `@opentelemetry/core` 2.7.1,
`@opentelemetry/host-metrics` 0.38.3).
([#3810](https://github.com/triggerdotdev/trigger.dev/pull/3810))

-   Updated dependencies:
    -   `@trigger.dev/core@4.5.0-rc.5`

</details>

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2026-06-05 14:09:26 +01:00
nicktrn 35c56f1d09 feat(supervisor): add opt-in dequeue backpressure (#3836)
The supervisor can now pause dequeuing - and freeze consumer-pool
scale-up - when a backpressure signal says the cluster can't place more
work, then ramp dequeuing back up gradually once it clears. The signal
is a verdict published to a Redis key by a cluster-side component; the
supervisor reads it on a short refresh and gates `preDequeue` on it.

Off by default (`TRIGGER_DEQUEUE_BACKPRESSURE_ENABLED`). Everything
fails open: a missing, stale, or unreadable verdict never pins the
brake, and the hot-path read is a synchronous cached lookup with no I/O.
The scale-up freeze leaves scale-down untouched, and on release the
resume is ramped so a deep queue isn't hammered all at once.

Dry-run is on by default (`TRIGGER_DEQUEUE_BACKPRESSURE_DRY_RUN`): even
once enabled it only logs what it would have done, and surfaces the
computed state through metrics, until explicitly set to act. Prometheus:
`supervisor_backpressure_engaged`, `_dry_run`,
`_skipped_dequeues_total`.

Refs TRI-5354
2026-06-05 13:58:19 +01:00
Eric Allam 85886b96da feat(webapp,supervisor): isolate scheduled runs on a dedicated worker queue (#3839)
## Summary

Scheduled runs and their descendants can now be routed to a dedicated
per-region worker queue, processed by a separate worker fleet, so a
burst of scheduled crons no longer competes with standard and agent runs
for the same queue and inflates their startup latency. It is off by
default and enabled per organization via a feature flag (with a global
default), so nothing changes until it is turned on.

## Design

At trigger time, any run whose lineage originates from a schedule
(`rootTriggerSource === "schedule"`, which already propagates from a
scheduled run down to all of its children) gets its worker queue
suffixed with `:scheduled`. The worker queue name is an opaque string
persisted on the run and used verbatim by enqueue and dequeue, so this
needs no Lua, message-envelope, or concurrency changes. Concurrency
stays keyed by environment and queue, not by worker queue.

On the consumer side, the dequeue endpoint gains an optional
`queueClass` selector. A supervisor sends `queueClass: "scheduled"` and
the server derives the actual queue from the worker's own group, so a
token can only ever reach its own region's queues. A fleet picks its
class with the `TRIGGER_WORKER_QUEUE_CLASS` env var (`default` or
`scheduled`), so a dedicated scheduled fleet can run alongside the
standard one.

Verified end to end against a local managed-worker setup: scheduled runs
route to the dedicated queue, are drained only by the scheduled fleet,
and standard runs are left untouched.
2026-06-05 09:41:57 +01:00
Eric Allam 884bea6ada fix(cli): stop the MCP waiting for every triggered run by default (#3838)
## Summary

The Trigger.dev MCP server told the AI agent to wait for the run to
complete after every `trigger_task` call. The agent followed that
instruction even when the user only wanted to fire-and-forget, which
burned tokens polling runs nobody needed to block on and made responses
less clear.

Waiting is now opt-in. After triggering, the response tells the agent
the run is executing in the background and to only wait if the user
asked it to (for example "trigger and then wait for it to finish"). The
`trigger_task` tool description is updated to match. The
`wait_for_run_to_complete` tool itself is unchanged, so explicit waits
still work.
2026-06-05 09:34:22 +01:00
Eric Allam 8c9fee3933 feat(sdk): add AI SDK 7 support (#3833)
## Summary

Adds support for Vercel AI SDK 7. The `ai` peer range now includes v7,
and the `chat.agent` / chat surfaces work against v7's ESM-only build.
v5 and v6 keep working unchanged, so this is additive.

## Telemetry on v7

On v7, model-call spans moved out of `ai` core into the separate
`@ai-sdk/otel` adapter, so `experimental_telemetry` alone produces
nothing until an integration is registered. Install `@ai-sdk/otel`
alongside `ai@7` and the SDK registers it once per worker at chat agent
boot, so spans keep flowing into run traces with no extra setup.

If you (or a library you import) already register `@ai-sdk/otel`, the
SDK detects the existing integration and skips its own registration, so
you won't get duplicate spans. Set `TRIGGER_AI_SDK_OTEL_AUTOREGISTER=0`
to disable auto-registration entirely.

## Notes

`ai@7` is ESM-only, which tripped TS1479 in the SDK's CommonJS build.
Runtime value imports from `ai` are isolated behind a paired ESM/CJS
shim so both module formats resolve the right form; type-only imports
stay as direct `import type` at their use sites.
2026-06-05 08:51:40 +01:00
Katia Bulatova cae3dcb7dd Env vars page performance fix (#3829)
## Summary

This PR improves performance across the Environment Variables page.

## Changes

### Targeted value loading

- load only the non-secret (environmentId, key) pairs required by the
page. Secret values continue to be redacted in the UI.

### SSR windowing + virtualization

- SSR-render only the first 50 rows
- hydrate those rows
- virtualize the remaining dataset client-side
- search is now URL-driven during SSR, ensuring deep links such as
`?search=DATABASE_URL`

### Lightweight 'Create' flow

- 'Create' page no longer loads the full Environment Variables dataset.

## Results

Large projects no longer render thousands of rows during SSR.
Example (~11k rendered rows):

Metric | Before | After
-- | -- | --
Document size | ~150 MB | ~5 MB
SSR rows | ~11k | 50
Browser DOM rows | Thousands | ~26–38
2026-06-04 16:28:55 +02:00
Daniel Sutton 4ea3ef138f chore(webapp,redis-worker): make mollifier constants configurable (#3822)
## Summary

The mollifier had ~21 behavioural constants baked in as hardcoded values
— the buffer's ack-grace TTL and Redis retry/reconnect tuning, the
drainer's poll interval and backoff envelope, the pre-gate idempotency
claim TTL/wait/poll, the buffered-run mutate-with-fallback wait loop,
the metadata CAS retry budget and backoff, the stale-sweep scan bounds,
and the draining-gauge interval. None could be adjusted without a code
change, which makes tuning the system under production load impossible.

This exposes all of them as `TRIGGER_MOLLIFIER_*` environment variables,
each defaulting to its previous hardcoded value. Behaviour is identical
unless an operator sets a var, so it's a safe no-op deploy.

## Design

The package-level classes (`MollifierBuffer`, `MollifierDrainer` in
`@trigger.dev/redis-worker`) gain optional constructor options
defaulting to the old constants — backward compatible, hence a patch
changeset. The webapp factories and worker bootstraps read the env and
pass them through. The route- and concern-level pure helpers
(mutate-with-fallback, metadata mutation, idempotency claim, stale-sweep
state) keep their existing `?? DEFAULT` option fallbacks and are fed env
values at their call sites, so they stay unit-testable without importing
`env.server`.

## Test plan

- [x] `@trigger.dev/redis-worker` builds
- [x] webapp typecheck passes
- [x] mollifier buffer + drainer testcontainer suites pass (modulo a
couple of pre-existing flaky timing tests)
- [x] Reviewer: confirm the `TRIGGER_MOLLIFIER_*` env var names match
ops conventions

---------

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>
2026-06-04 09:36:34 +00:00
Katia Bulatova bb7d7dc7d1 feat(sdk,core): offload large trigger payloads via object storage (#3785)
Adds backward-compatible support for large trigger payloads by reusing
the existing object-storage packet flow.

Large payloads are uploaded to object storage before the trigger request
is sent. The trigger API receives a small application/store pointer
payload instead of embedding large JSON bodies in the request.

Small payload behavior is unchanged.
2026-06-04 11:29:18 +02:00
Eric Allam 9818ad5240 fix(sdk): recover chat transport when a restored session no longer exists (#3816)
## Summary

When a chat's restored session state points at a session that no longer
exists in the current environment — for example a `sessions` entry that
was persisted against a different trigger environment —
`useTriggerChatTransport` assumed the session was live and never created
a real one. The next message then failed with a 404 and the chat
couldn't send.

## Fix

`callWithAuthRetry` now treats a 404 from a session-PAT-authed call as
"this session doesn't exist here". After the existing 401/403 token
refresh, a 404 recreates the session via `startSession`, drops the stale
`lastEventId` resume cursor (it pointed at another environment's
stream), and retries the send once. When `startSession` isn't configured
the transport throws a clear message instead of a bare 404.
2026-06-03 15:54:45 +01:00
nicktrn a9f756b260 chore: move reference projects to their own repo (#3812)
The reference/example projects (`references/`) now live in their own
repo, https://github.com/triggerdotdev/references, so their heavy,
frequently-changing dependencies are no longer part of this repo's
lockfile and tooling. This removes them here and repoints everything
that referenced them.

- Deletes `references/`; updates the pnpm workspace + lockfile.
- Clears the references-only CI rules and `.vscode` configs.
- Repoints the docs (contributor/agent + one public page) to the new
repo.
- `seed.mts` keeps the local-dev projects (hello-world, d3-chat,
realtime-streams).
2026-06-02 22:19:28 +01:00
nicktrn a4d8c9f65f chore(deps): update OpenTelemetry suite to 0.218.0 / 2.7.1 (#3810)
Brings the OpenTelemetry packages up to the latest coherent release
across the webapp and the published packages (`@trigger.dev/core`, the
CLI, `@trigger.dev/sdk`) plus
`internal-packages/{tracing,testcontainers}`:

- `@opentelemetry/sdk-node` 0.218.0
- `@opentelemetry/core` 2.7.1
- `@opentelemetry/host-metrics` 0.38.3

We were already on the otel 2.x line, so this is a same-major minor move
- the versions are pinned to `@opentelemetry/sdk-node@0.218.0`'s own
declared dependency set so the experimental (0.2xx) and stable (2.x)
packages stay coherent (mixing them is the usual otel breakage).

**One code change:** otel 0.215 made `forceFlush()` a required method on
`LogRecordExporter`, so `ExternalLogRecordExporterWrapper` (core's
tracing SDK) gains a `forceFlush()` that delegates to the underlying
exporter.

**Notable upgrades along the way:** OTLP exporters can take a custom
HTTP agent (connection pooling/keepAlive on the export path), HTTP
request headers are captured at span creation, and core hot-path perf
improvements in 2.6.1/2.7. `host-metrics` 0.37→0.38 is a clean upgrade.

Patch changeset added for the three published packages. References
projects are intentionally untouched.

Verified: `@trigger.dev/core` / CLI / `@trigger.dev/sdk` build, webapp +
`@internal/tracing` typecheck - all green.
2026-06-02 21:25:38 +01:00
Eric Allam 4b78d7e84e feat(core,webapp): support isSecret on environment variable imports (#3809)
## Summary

The environment variables import API now accepts an optional `isSecret`
flag, so imported variables can be created as secret (redacted)
environment variables instead of plaintext. When the flag is omitted,
variables default to non-secret, preserving existing behavior for CLI
deploys and dashboard imports.

This is useful for tools that push secrets into Trigger.dev (for
example, syncing from a secrets manager) and want them stored as secrets
rather than plain environment variables.

It's available through `envvars.import` in the SDK and the `POST
/api/v1/projects/{projectRef}/envvars/{slug}/import` endpoint, and is
honored for both regular and preview-branch environments.

```ts
await envvars.import("proj_1234", "prod", {
  variables: { STRIPE_SECRET_KEY: "sk_live_..." },
  isSecret: true,
});
```
2026-06-02 14:57:06 +00:00
Daniel Sutton 139eede41c feat(redis-worker): batched pop in MollifierDrainer for fast single-env drains (#3797)
## Summary

Adds `drainBatchSize` to `MollifierDrainer` (default `1` — preserves
existing behaviour) and wires `TRIGGER_MOLLIFIER_DRAIN_BATCH_SIZE`
through the webapp (default `50`). Each tick the drainer now pops up to
`drainBatchSize` from each chosen env, then dispatches every popped
entry through the shared `concurrency`-bounded `pLimit`. Per-org/per-env
fairness is unchanged — only the in-env pop count grows.

Pre-existing behaviour was one pop per env per tick. For a single-env
burst that single-flighted the drain at the per-tick floor of `pop +
engine.trigger ≈ 50–60 ms`. With buffer entries piling up under a
real-world tenant burst that's tens of minutes of tail latency to fully
materialise — even though PG itself could comfortably sustain the
writes.

## Why this matters — heavy-tail illustration

Scenario: 100 customers in one window — 94 fire 20 triggers each, 5 fire
100, 1 fires 1000. Gate at `THRESHOLD=10/s`, `HOLD_MS=500`. First 10 of
each burst hit PG directly; the rest buffer.

| Customers | Triggers each | PG direct | Buffered each | Total buffered
|
|---|---|---|---|---|
| 94 small  | 20    | 10  | 10  | 940 |
| 5 medium  | 100   | 10  | 90  | 450 |
| 1 heavy   | 1000  | 10  | 990 | 990 |

**With `DRAIN_BATCH_SIZE=50`, `DRAIN_CONCURRENCY=50`, ~50 ms
`engine.trigger`:**

| Tick | Pops | Dispatch waves | Wall-clock |
|---|---|---|---|
| 1 | 94×10 + 5×50 + 1×50 = 1 240 | 25 × 50 ms | ~1 300 ms (94 smalls
done) |
| 2 | 5×40 + 1×50 = 250 | 5 × 50 ms | ~300 ms (5 mediums done) |
| 3–20 | heavy alone, 50/tick | 1 × 50 ms | ~100 ms each |

| Customer class | Buffered fully drained |
|---|---|
| 94 small  | **~1.3 s** |
| 5 medium  | **~1.6 s** |
| 1 heavy   | **~3.4 s** |

**Without batching (one pop per env per tick — current behaviour):**

| Customer class | Buffered fully drained |
|---|---|
| 94 small  | ~500 ms |
| 5 medium  | ~4.5 s |
| 1 heavy   | **~49 s** |

So the heavy single-tenant tail drops from ~49 s to ~3.4 s (~14× faster)
without changing PG load characteristics. Smalls go up slightly in this
scenario (500 ms → 1.3 s) because all 100 envs share one tick's dispatch
queue — that's the trade we accept for the heavy tail; the worst-case
small wait is still inside one tick. PG load is identical either way (50
concurrent inserts at a time, capped by `DRAIN_CONCURRENCY`).

## What changed

**`packages/redis-worker`**
- New `drainBatchSize` option (default 1 — full backward compat).
- `runOnce()` refactored to pop per-env batches in parallel, then
dispatch all popped entries through the existing global `pLimit`.
Mid-batch pop failure aborts only that env's batch and counts as one
failure (same semantic as the old per-env path).
- Removed the now-unused `processOneFromEnv` helper.

**`apps/webapp`**
- `TRIGGER_MOLLIFIER_DRAIN_BATCH_SIZE` env var (default 50, matching
`DRAIN_CONCURRENCY`).
- Wired into `mollifierDrainer.server.ts`.

**Test cloud config** (separate cloud PR):
`TRIGGER_MOLLIFIER_DRAIN_BATCH_SIZE="50"` on the worker service.
Production rollout deferred until we've watched it on test cloud.

## Test plan

- [x] All 25 stub-based drainer tests pass (18 pre-existing + 7 new). 7
new tests under `MollifierDrainer.drainBatchSize`:
  - pops up to `drainBatchSize` across ticks
  - global `concurrency` cap still holds when batch > concurrency
  - mid-batch pop failure isolation
  - multi-env batch fan-out in one tick
- **hierarchical org fairness preserved at `drainBatchSize > 1`**
(load-bearing — guards against future regressions to
per-env-instead-of-per-org rotation)
  - mixed success/failure accounting in a batched tick
  - bounded pops on empty queue (no Lua spam past `drainBatchSize`)
- [x] All pre-existing tests still pass unchanged at default
`drainBatchSize=1` → backward-compat locked.
- [x] `pnpm run build --filter @trigger.dev/redis-worker` clean.
- [x] `pnpm run typecheck --filter webapp` clean.
- [x] `redisTest` block (real Redis via testcontainers) — couldn't run
locally on this branch due to testcontainers runtime discovery; will
validate in CI.
- [ ] Test-cloud smoke after cloud PR lands: fire `burst 50` against a
flagged env and confirm the 50th entry's drain time drops from ~2.5 s to
<200 ms.

## Notes

- Per-tick memory bound: `maxOrgsPerTick × drainBatchSize` entries can
sit in the JS pLimit queue between pop and dispatch. At defaults that's
`500 × 50 = 25 000` × ~5 KB snapshot ≈ ~125 MB worst case per worker —
well within headroom.
- The pre-batch model's strict per-env throughput cap of `1/tick` is
documented as the fairness baseline elsewhere. Org-level fairness is
what callers actually rely on; this change does not weaken that.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-06-02 14:21:37 +01:00
Dan cd252801eb feat: dashboard agent - package upgrades (#3793)
1. in webapp folder update ai-sdk to 6.x.x
2. update vitest to 4.xx
2026-06-02 10:46:34 +01:00
github-actions[bot] e0681d2394 chore: release v4.5.0-rc.4 (#3788)
🚀 Publish Trigger.dev Docker / units (push) Failing after 3s
🚀 Publish Trigger.dev Docker / typecheck (push) Failing after 19s
🚀 Publish Trigger.dev Docker / publish-webapp (push) Has been skipped
🚀 Publish Trigger.dev Docker / publish-worker (push) Has been skipped
🚀 Publish Trigger.dev Docker / publish-worker-v4 (push) Has been skipped
🧭 Helm Chart Release / lint-and-test (push) Has been cancelled
🧭 Helm Chart Release / release (push) Has been cancelled
## Summary
1 new feature, 5 improvements.

## Highlights

- Mollifier `mutateSnapshot` now enforces a tag cap: an `append_tags`
patch carrying `maxTags` returns `"limit_exceeded"` (writing nothing)
when the deduped tag count would exceed the limit, so a buffered run
can't accumulate more tags via the tags API than the trigger validator
allows at creation.
([#3756](https://github.com/triggerdotdev/trigger.dev/pull/3756))

## Improvements
- Mollifier buffer extensions: idempotency dedup, an atomic
`mutateSnapshot` API, metadata CAS, claim primitives, and a
`MollifierSnapshot` type. The buffer's Redis client now reconnects with
jittered backoff so a fleet of clients doesn't stampede Redis in
lockstep after a blip.
([#3752](https://github.com/triggerdotdev/trigger.dev/pull/3752))
- Add `onTerminalFailure` callback to `MollifierDrainerOptions` so the
customer's run lands a SYSTEM_FAILURE PG row even when the drainer
exhausts `maxAttempts` on a retryable PG error. Previously,
retryable-error exhaustion called `buffer.fail()` directly, which
atomically marks FAILED + DELs the entry hash with no PG write — silent
data loss when PG was unreachable across the full retry budget. The
callback fires before `buffer.fail()` on any terminal path (`cause:
"non-retryable"` or `"max-attempts-exhausted"`); throwing a retryable
error from the callback causes the drainer to requeue rather than fail.
([#3754](https://github.com/triggerdotdev/trigger.dev/pull/3754))
- Bump `@s2-dev/streamstore` to `0.22.10` to fix a
`TASK_RUN_UNCAUGHT_EXCEPTION` ("Invalid state: Unable to enqueue") when
a `chat.agent` turn is aborted mid-stream.
([#3792](https://github.com/triggerdotdev/trigger.dev/pull/3792))
- Coerce numeric `concurrencyKey` values to string at the API boundary
across `tasks.trigger`, `tasks.batchTrigger`, and the Phase-2 streaming
batch endpoint.
([#3789](https://github.com/triggerdotdev/trigger.dev/pull/3789))
- Add a `tools` option to `chat.agent`. Declaring your tools here
threads them into the SDK's internal `convertToModelMessages`, so each
tool's `toModelOutput` is re-applied when prior-turn history is
re-converted.
([#3790](https://github.com/triggerdotdev/trigger.dev/pull/3790))

<details>
<summary>Raw changeset output</summary>

⚠️⚠️⚠️⚠️⚠️⚠️

`main` is currently in **pre mode** so this branch has prereleases
rather than normal releases. If you want to exit prereleases, run
`changeset pre exit` on `main`.

⚠️⚠️⚠️⚠️⚠️⚠️

# Releases
## @trigger.dev/redis-worker@4.5.0-rc.4

### Minor Changes

- Mollifier buffer extensions: idempotency dedup, an atomic
`mutateSnapshot` API, metadata CAS, claim primitives, and a
`MollifierSnapshot` type. The buffer's Redis client now reconnects with
jittered backoff so a fleet of clients doesn't stampede Redis in
lockstep after a blip.
([#3752](https://github.com/triggerdotdev/trigger.dev/pull/3752))
- Add `onTerminalFailure` callback to `MollifierDrainerOptions` so the
customer's run lands a SYSTEM_FAILURE PG row even when the drainer
exhausts `maxAttempts` on a retryable PG error. Previously,
retryable-error exhaustion called `buffer.fail()` directly, which
atomically marks FAILED + DELs the entry hash with no PG write — silent
data loss when PG was unreachable across the full retry budget. The
callback fires before `buffer.fail()` on any terminal path (`cause:
"non-retryable"` or `"max-attempts-exhausted"`); throwing a retryable
error from the callback causes the drainer to requeue rather than fail.
([#3754](https://github.com/triggerdotdev/trigger.dev/pull/3754))

### Patch Changes

- Pipeline the per-entry `HGETALL` fetches in
`MollifierBuffer.listEntriesForEnv`. The previous serial implementation
issued one Redis round-trip per runId returned by `LRANGE`, which
dominated stale-sweep wall-time at any meaningful backlog (at the
sweep's default maxCount=1000, this is ~1000 RTTs per env per pass).
Behaviour is unchanged — entries are still skipped when the entry hash
has been torn down by a concurrent drainer ack/fail between the LRANGE
and the HGETALL.
([#3752](https://github.com/triggerdotdev/trigger.dev/pull/3752))
- Mollifier `mutateSnapshot` now enforces a tag cap: an `append_tags`
patch carrying `maxTags` returns `"limit_exceeded"` (writing nothing)
when the deduped tag count would exceed the limit, so a buffered run
can't accumulate more tags via the tags API than the trigger validator
allows at creation.
([#3756](https://github.com/triggerdotdev/trigger.dev/pull/3756))
-   Updated dependencies:
    -   `@trigger.dev/core@4.5.0-rc.4`

## @trigger.dev/build@4.5.0-rc.4

### Patch Changes

-   Updated dependencies:
    -   `@trigger.dev/core@4.5.0-rc.4`

## trigger.dev@4.5.0-rc.4

### Patch Changes

- Bump `@s2-dev/streamstore` to `0.22.10` to fix a
`TASK_RUN_UNCAUGHT_EXCEPTION` ("Invalid state: Unable to enqueue") when
a `chat.agent` turn is aborted mid-stream.
([#3792](https://github.com/triggerdotdev/trigger.dev/pull/3792))
-   Updated dependencies:
    -   `@trigger.dev/core@4.5.0-rc.4`
    -   `@trigger.dev/build@4.5.0-rc.4`
    -   `@trigger.dev/schema-to-json@4.5.0-rc.4`

## @trigger.dev/core@4.5.0-rc.4

### Patch Changes

- Coerce numeric `concurrencyKey` values to string at the API boundary
across `tasks.trigger`, `tasks.batchTrigger`, and the Phase-2 streaming
batch endpoint.
([#3789](https://github.com/triggerdotdev/trigger.dev/pull/3789))
- Bump `@s2-dev/streamstore` to `0.22.10` to fix a
`TASK_RUN_UNCAUGHT_EXCEPTION` ("Invalid state: Unable to enqueue") when
a `chat.agent` turn is aborted mid-stream.
([#3792](https://github.com/triggerdotdev/trigger.dev/pull/3792))

## @trigger.dev/plugins@4.5.0-rc.4

### Patch Changes

-   Updated dependencies:
    -   `@trigger.dev/core@4.5.0-rc.4`

## @trigger.dev/python@4.5.0-rc.4

### Patch Changes

-   Updated dependencies:
    -   `@trigger.dev/sdk@4.5.0-rc.4`
    -   `@trigger.dev/core@4.5.0-rc.4`
    -   `@trigger.dev/build@4.5.0-rc.4`

## @trigger.dev/react-hooks@4.5.0-rc.4

### Patch Changes

-   Updated dependencies:
    -   `@trigger.dev/core@4.5.0-rc.4`

## @trigger.dev/rsc@4.5.0-rc.4

### Patch Changes

-   Updated dependencies:
    -   `@trigger.dev/core@4.5.0-rc.4`

## @trigger.dev/schema-to-json@4.5.0-rc.4

### Patch Changes

-   Updated dependencies:
    -   `@trigger.dev/core@4.5.0-rc.4`

## @trigger.dev/sdk@4.5.0-rc.4

### Patch Changes

- Add a `tools` option to `chat.agent`. Declaring your tools here
threads them into the SDK's internal `convertToModelMessages`, so each
tool's `toModelOutput` is re-applied when prior-turn history is
re-converted.
([#3790](https://github.com/triggerdotdev/trigger.dev/pull/3790))

    ```ts
    chat.agent({
      tools: { readFile, search },
      run: async ({ messages, tools, signal }) =>
        streamText({ model, messages, tools, abortSignal: signal }),
    });
    ```

Also exports `InferChatUIMessageFromTools<typeof tools>` to derive the
chat `UIMessage` type (typed tool parts) directly from a tool set.

-   Updated dependencies:
    -   `@trigger.dev/core@4.5.0-rc.4`

</details>

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2026-06-02 09:35:14 +01:00
Eric Allam 2dd9f37b40 fix(core,cli): stop chat.agent uncaught exception on mid-stream abort (#3792)
## Summary

A `chat.agent` turn that gets aborted mid-stream (stop generation, idle
suspend, cancellation) could surface a `TASK_RUN_UNCAUGHT_EXCEPTION` —
`TypeError: Invalid state: Unable to enqueue` — in the dashboard. The
run kept working, but the exceptions were loud and confusing.

## Root cause

The realtime stream writer batches chunks through
`@s2-dev/streamstore`'s `BatchTransform`, which holds them for a short
linger window before flushing. When the turn's abort signal fires while
a record is still buffered, the stream's writable is aborted and the
transform's readable controller is closed — but the pending linger
`setTimeout` still fires and calls `controller.enqueue()` on the dead
controller, throwing from a timer callback where nothing can catch it.

Fixed upstream in `@s2-dev/streamstore@0.22.10`, which wraps the linger
flush in a try/catch that discards the closed-controller error. This
bumps the dependency across core, the CLI, and the webapp, and adds a
regression test against the real `BatchTransform`.

Verified end-to-end: a mid-stream stop that previously failed the run
with `TASK_RUN_UNCAUGHT_EXCEPTION` now leaves the run healthy.
2026-06-01 17:12:48 +00:00
Eric Allam e9e2ec1cfc fix(sdk): re-apply tool toModelOutput across chat.agent turns (#3790)
## Summary

`chat.agent` now takes a `tools` option. Until now tools only went to
`streamText` inside `run()`, so the SDK had no tools when it
re-converted the persisted `UIMessage` history at the start of each
turn. Any tool with a `toModelOutput` (raw image bytes into an image
content part, or a sub-agent transcript compressed to a summary) had its
transform applied on turn 1 and skipped from turn 2 onward, so the raw
output got JSON-stringified back into the prompt and the model lost the
transformed view.

Declaring `tools` on the config threads them into that conversion, so
`toModelOutput` runs on every turn. The resolved set is handed back,
typed, on the `run()` payload as `tools`:

```ts
const tools = { searchDocs, renderChart };

export const myChat = chat.agent({
  tools,
  run: async ({ messages, tools, signal }) =>
    streamText({ ...chat.toStreamTextOptions({ tools }), messages, abortSignal: signal }),
});
```

`tools` also accepts a per-turn function for tools that depend on the
user or a feature flag. Only `inputSchema` and `toModelOutput` are read
during conversion, never `execute`. Also exports
`InferChatUIMessageFromTools<typeof tools>` to derive the chat
`UIMessage` type from a tool set. No behavior change for agents that
don't declare `tools`.
2026-06-01 17:26:27 +01:00
Daniel Sutton e1950778e2 feat(webapp): mollifier API mutations on buffered runs (#3756)
## Summary

Cancel, replay, reschedule, metadata, tags, and idempotency-key-reset
now succeed against a run that's still in the mollifier buffer.
Mutations are applied to the buffered snapshot via Lua CAS; the drainer
carries the mutation forward when it replays.

Primitives added:

- `mutateWithFallback` — PG-first / buffer-fallback resolver with
bounded-wait safety net for entries that transition mid-mutation.
- `applyMetadataMutation` — buffered metadata PUT mirroring the PG-side
retry loop with CAS atomicity.
- `resolveRunForMutation` — discriminated-union resolver used by route
`findResource` so the route builder's pre-action 404 check sees buffered
runs.

Routes wired (whole files, no GET/POST splits):
- `api.v2.runs.\$runParam.cancel.ts`
- `api.v1.runs.\$runParam.replay.ts`
- `api.v1.runs.\$runParam.reschedule.ts`
- `api.v1.runs.\$runId.metadata.ts`
- `api.v1.runs.\$runId.tags.ts`
- `resetIdempotencyKey.server.ts`

Stacked on the reads PR.

## Test plan

- [x] \`pnpm run typecheck --filter webapp\` passes
- [x] \`pnpm run test --filter webapp
test/mollifierMutateWithFallback.test.ts\` passes
- [x] \`pnpm run test --filter webapp
test/mollifierApplyMetadataMutation.test.ts\` passes
- [x] \`pnpm run test --filter webapp
test/mollifierResolveRunForMutation.test.ts\` passes

---------

Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-06-01 16:24:10 +01:00
Daniel Sutton 4745754a7a feat(webapp,run-engine): mollifier drainer replay + stale sweep + cancelled-run engine API (#3754)
## Summary

The replay side of the mollifier:

- `DrainerHandler`: reads buffered snapshots and replays them through
`engine.trigger` to materialise PG rows.
- `RunEngine.createCancelledRun`: new public method the handler uses to
write CANCELED rows directly from snapshots (bypass queue + waitpoint,
emit `runCancelled`). Tolerates the cjson empty-table tags edge case
found during validation.
- Drainer fairness: org → env rotation so a heavy env doesn't starve
light ones in the same org.
- Stale-entry sweep + telemetry + alertable gauge so a stuck/offline
drainer surfaces in alerts.

Both the drainer and sweep default-off; nothing fires unless flagged on
(`TRIGGER_MOLLIFIER_DRAINER_ENABLED`,
`TRIGGER_MOLLIFIER_STALE_SWEEP_ENABLED`).

Stacked on the trigger-time decisions PR.

## Test plan

- [x] \`pnpm run typecheck --filter webapp\` passes
- [x] \`pnpm run test --filter webapp
test/mollifierDrainerHandler.test.ts\` passes
- [x] \`pnpm run test --filter webapp test/mollifierStaleSweep.test.ts\`
passes
- [x] \`pnpm run test --filter @internal/run-engine
src/engine/tests/createCancelledRun.test.ts\` passes
- [x] \`pnpm run test --filter @trigger.dev/redis-worker
packages/redis-worker/src/mollifier/drainer.test.ts\` passes

---

## Ship-gate follow-up fix

**Drainer writes SYSTEM_FAILURE on max-attempts exhaustion.** Adds an
`onTerminalFailure` callback on `MollifierDrainerOptions` so the
customer's run lands a SYSTEM_FAILURE PG row even when the drainer
exhausts `MAX_ATTEMPTS` on a retryable PG error (previously
`buffer.fail()` was called with no row written → silent data loss). The
callback runs before `buffer.fail()` on every terminal path
(non-retryable AND max-attempts-exhausted), and re-throwing a retryable
error from the callback causes the drainer to requeue rather than fail.

Bumps `@trigger.dev/redis-worker` to a **minor** changeset (additive
option + new exported types). Includes 5 unit tests covering both
terminal causes plus the requeue-on-retryable-callback-failure path and
no-callback back-compat.

---------

Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-06-01 13:20:05 +01:00
Daniel Sutton 20a676d2ed fix(core,webapp): coerce numeric concurrencyKey to string (#3789)
## Summary

`concurrencyKey` validation accepted only `z.string().optional()` on the
single-trigger and V2/V3 batch endpoints, and the Phase-2 streaming
NDJSON endpoint accepted `z.record(z.unknown()).optional()` for the
entire `options` field. Callers passing `concurrencyKey: someNumericId`
(e.g. `payload.userId`) either failed schema validation on the first two
paths or sailed through on Phase-2 and then failed downstream at
`prisma.taskRun.create` with `Argument concurrencyKey: Expected String
or Null, provided Int`.

The schema now accepts `string | number` for `concurrencyKey` and
stringifies on the way in, across all three paths. The Phase-2 NDJSON
`options` is tightened to reuse the strict
`BatchTriggerTaskItem.options` shape so it validates identically to the
V2/V3 batch endpoints.

A defensive `typeof === "number"` coercion at the `engine.trigger` call
site in `RunEngineTriggerTaskService` covers in-flight Redis-stored
batch items enqueued before the schema fix — those items are rebuilt
from a `Record<string, unknown>` shape that bypasses the new schema and
would otherwise continue failing for up to their TTL.

## Test plan

- [x] `packages/core/src/v3/schemas/batchItemNDJSON.test.ts` — unit
tests covering numeric→string coercion, string passthrough, no-options,
and rejection of non-string/non-number shapes across
`TriggerTaskRequestBody`, `BatchTriggerTaskItem`, and `BatchItemNDJSON`.
- [x] `apps/webapp/test/engine/triggerTask.test.ts` — `containerTest`
simulating the in-flight Redis batch-item shape (numeric
`concurrencyKey` via `Record<string, unknown>`), verifies the run is
created with `concurrencyKey: "51262"`. Without the worker coercion, the
test reproduces the production stack at `prisma.taskRun.create`.
- [x] `pnpm run typecheck --filter webapp` clean.
- [x] `pnpm run build --filter @trigger.dev/core --filter
@trigger.dev/sdk --filter trigger.dev` clean.

Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-06-01 12:34:29 +01:00
Daniel Sutton 3924aa4adb feat(redis-worker,webapp): mollifier buffer extensions + snapshot type (#3752)
## Summary

Buffer-side data layer used by the rest of the mollifier phase-3 stack.

- `buffer.ts` gains entry inspection (`getEntry`), idempotency lookup
(`lookupIdempotency`), in-place snapshot mutation (`mutateSnapshot`),
and dwell tracking. All atomic via Lua.
- `mollifierSnapshot.server.ts`: shared `MollifierSnapshot` type plus
(de)serialise helpers.
- Drops the entry-TTL config and its env var. The drainer is the
recovery mechanism; an entry that survives the drainer should surface as
a stale-sweep alert, not silently TTL away.

Adds methods to the buffer interface; nothing consumes them yet.
Subsequent PRs in the stack wire trigger-time mollify, read-fallback,
and mutation paths against this surface.

## Test plan

- [x] \`pnpm run typecheck --filter webapp\` passes
- [x] \`pnpm run test --filter @trigger.dev/redis-worker
packages/redis-worker/src/mollifier/buffer.test.ts\` passes

---------

Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-06-01 11:42:55 +01:00
github-actions[bot] 8f066ac389 chore: release v4.5.0-rc.3 (#3763)
🚀 Publish Trigger.dev Docker / typecheck (push) Failing after 0s
🚀 Publish Trigger.dev Docker / publish-webapp (push) Has been skipped
🚀 Publish Trigger.dev Docker / publish-worker (push) Has been skipped
🚀 Publish Trigger.dev Docker / publish-worker-v4 (push) Has been skipped
🚀 Publish Trigger.dev Docker / units (push) Failing after 0s
🧭 Helm Chart Release / lint-and-test (push) Has been cancelled
🧭 Helm Chart Release / release (push) Has been cancelled
## Summary
1 improvement, 2 bug fixes.

## Improvements
- Retry `TASK_MIDDLEWARE_ERROR` under the task's retry policy instead of
failing the run on the first attempt. The error was already classified
as retryable by `shouldRetryError`, but `shouldLookupRetrySettings` did
not include it, so the retry flow fell through to `fail_run`. Fixes
#3231. ([#3676](https://github.com/triggerdotdev/trigger.dev/pull/3676))

## Bug fixes
- Fix `TypeError` in `unflattenAttributes` when the input attribute map
contains conflicting dotted key paths (e.g. both `a.b` set to a scalar
and `a.b.c` set to a value). The path-walk loop now applies
last-write-wins when a prior key wrote a primitive, null, or array at an
intermediate slot, matching the existing precedent in
`AttributeFlattener.addAttribute`. Callers no longer crash when handed
malformed external attribute inputs.
([#3762](https://github.com/triggerdotdev/trigger.dev/pull/3762))
- Fix external trace context leaking across runs on warm-started workers
with `processKeepAlive` enabled. Every subsequent run's attempt span was
being exported with the first run's `traceId` and `parentSpanId`,
breaking causal-chain navigation in external APM tools. Runs without an
external trace context are unaffected.
([#3768](https://github.com/triggerdotdev/trigger.dev/pull/3768))

<details>
<summary>Raw changeset output</summary>

⚠️⚠️⚠️⚠️⚠️⚠️

`main` is currently in **pre mode** so this branch has prereleases
rather than normal releases. If you want to exit prereleases, run
`changeset pre exit` on `main`.

⚠️⚠️⚠️⚠️⚠️⚠️

# Releases
## @trigger.dev/build@4.5.0-rc.3

### Patch Changes

-   Updated dependencies:
    -   `@trigger.dev/core@4.5.0-rc.3`

## trigger.dev@4.5.0-rc.3

### Patch Changes

-   Updated dependencies:
    -   `@trigger.dev/core@4.5.0-rc.3`
    -   `@trigger.dev/build@4.5.0-rc.3`
    -   `@trigger.dev/schema-to-json@4.5.0-rc.3`

## @trigger.dev/core@4.5.0-rc.3

### Patch Changes

- Retry `TASK_MIDDLEWARE_ERROR` under the task's retry policy instead of
failing the run on the first attempt. The error was already classified
as retryable by `shouldRetryError`, but `shouldLookupRetrySettings` did
not include it, so the retry flow fell through to `fail_run`. Fixes
#3231. ([#3676](https://github.com/triggerdotdev/trigger.dev/pull/3676))
- Fix `TypeError` in `unflattenAttributes` when the input attribute map
contains conflicting dotted key paths (e.g. both `a.b` set to a scalar
and `a.b.c` set to a value). The path-walk loop now applies
last-write-wins when a prior key wrote a primitive, null, or array at an
intermediate slot, matching the existing precedent in
`AttributeFlattener.addAttribute`. Callers no longer crash when handed
malformed external attribute inputs.
([#3762](https://github.com/triggerdotdev/trigger.dev/pull/3762))
- Fix external trace context leaking across runs on warm-started workers
with `processKeepAlive` enabled. Every subsequent run's attempt span was
being exported with the first run's `traceId` and `parentSpanId`,
breaking causal-chain navigation in external APM tools. Runs without an
external trace context are unaffected.
([#3768](https://github.com/triggerdotdev/trigger.dev/pull/3768))

## @trigger.dev/plugins@4.5.0-rc.3

### Patch Changes

-   Updated dependencies:
    -   `@trigger.dev/core@4.5.0-rc.3`

## @trigger.dev/python@4.5.0-rc.3

### Patch Changes

-   Updated dependencies:
    -   `@trigger.dev/core@4.5.0-rc.3`
    -   `@trigger.dev/build@4.5.0-rc.3`
    -   `@trigger.dev/sdk@4.5.0-rc.3`

## @trigger.dev/react-hooks@4.5.0-rc.3

### Patch Changes

-   Updated dependencies:
    -   `@trigger.dev/core@4.5.0-rc.3`

## @trigger.dev/redis-worker@4.5.0-rc.3

### Patch Changes

-   Updated dependencies:
    -   `@trigger.dev/core@4.5.0-rc.3`

## @trigger.dev/rsc@4.5.0-rc.3

### Patch Changes

-   Updated dependencies:
    -   `@trigger.dev/core@4.5.0-rc.3`

## @trigger.dev/schema-to-json@4.5.0-rc.3

### Patch Changes

-   Updated dependencies:
    -   `@trigger.dev/core@4.5.0-rc.3`

## @trigger.dev/sdk@4.5.0-rc.3

### Patch Changes

-   Updated dependencies:
    -   `@trigger.dev/core@4.5.0-rc.3`

</details>

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2026-06-01 10:34:45 +01:00
nicktrn c043c4a6ad fix(core): external trace context leaks across warm-started runs (#3768)
On warm-started workers with `processKeepAlive` enabled, every run's
attempt span was exported with the first run's `traceId` and
`parentSpanId`. `ExternalSpanExporterWrapper` and
`ExternalLogRecordExporterWrapper` captured `externalTraceContext` at
`TracingSDK` construction, and the SDK is memoized for the worker's
lifetime - so the per-run reset of
`StandardTraceContextManager.traceContext` never reached the wrappers.

Reported by a customer running v4.4.x: 33 distinct runs on the same
host/pid showed up in their APM as siblings of one parent span.

Fix: the wrappers now read external context live from the trace context
manager per export. Runs without an external trace context fall through
to the unchanged `externalTraceId` fallback - no behaviour change for
them.

Regression test in
`packages/core/test/externalSpanExporterWrapper.test.ts` asserts that
reassigning the manager between exports produces correctly-parented
spans.
2026-05-28 10:43:39 +00:00
Iss 8399aa2052 fix(core): retry TASK_MIDDLEWARE_ERROR under the task's retry policy (#3676)
Retries `TASK_MIDDLEWARE_ERROR` under the task's retry policy.
`shouldRetryError` already classed it as retryable, but
`shouldLookupRetrySettings` did not, so the run fell through to
`fail_run` on attempt 1 instead of using the task's `retry` config.
Fixes #3231.
2026-05-27 17:17:50 +01:00
Oskar Otwinowski 5ba1b32d53 fix(core): handle conflicting dotted paths in unflattenAttributes (#3762) 2026-05-27 16:18:50 +02:00
github-actions[bot] d34014ddcc chore: release v4.5.0-rc.2 (#3702)
🚀 Publish Trigger.dev Docker / typecheck (push) Failing after 0s
🚀 Publish Trigger.dev Docker / publish-webapp (push) Has been skipped
🚀 Publish Trigger.dev Docker / publish-worker (push) Has been skipped
🚀 Publish Trigger.dev Docker / publish-worker-v4 (push) Has been skipped
🚀 Publish Trigger.dev Docker / units (push) Failing after 4s
## Summary
3 improvements, 1 bug fix.

## Improvements
- The per-turn merge now overlays the wire copy's tool-part state
advancement onto the agent's existing chain — `state` + the matching
resolution field (`output` / `errorText` / `approval`) come from the
wire, everything else (text, reasoning, tool `input`, provider metadata)
stays whatever the snapshot or `hydrateMessages` returned. Previously a
full-message replace overwrote those fields with whatever the client
shipped, so a slimmed wire copy landed a tool call with no `arguments`
on the next LLM call. Covers `output-available` / `output-error` (HITL
`addToolOutput`) and `approval-responded` / `output-denied` (approval
flow).
- `TriggerChatTransport.sendMessages` and `AgentChat.sendRaw` now slim
assistant messages that carry advanced tool parts. The wire payload is
just `{ id, role, parts: [<state + resolution field>] }` for
`submit-message` continuations; everything else passes through.
Reasoning blobs and full tool inputs no longer ride the wire on every
`addToolOutput` / `addToolApproveResponse`, so continuation payloads
stay well under the `.in/append` cap on long agent loops.
- Add `TriggerClient` for running multiple SDK clients side-by-side,
each with its own auth, preview branch, and baseURL. Useful when a
single process needs to trigger tasks or read runs across multiple
projects, environments, or preview branches without mutating shared
global state.
([#3683](https://github.com/triggerdotdev/trigger.dev/pull/3683))

## Bug fixes
- Fix `chat.agent` HITL continuations on reasoning-heavy turns. Two
changes that work together:
([#3719](https://github.com/triggerdotdev/trigger.dev/pull/3719))

<details>
<summary>Raw changeset output</summary>

⚠️⚠️⚠️⚠️⚠️⚠️

`main` is currently in **pre mode** so this branch has prereleases
rather than normal releases. If you want to exit prereleases, run
`changeset pre exit` on `main`.

⚠️⚠️⚠️⚠️⚠️⚠️

# Releases
## @trigger.dev/build@4.5.0-rc.2

### Patch Changes

-   Updated dependencies:
    -   `@trigger.dev/core@4.5.0-rc.2`

## trigger.dev@4.5.0-rc.2

### Patch Changes

-   Updated dependencies:
    -   `@trigger.dev/build@4.5.0-rc.2`
    -   `@trigger.dev/core@4.5.0-rc.2`
    -   `@trigger.dev/schema-to-json@4.5.0-rc.2`

## @trigger.dev/plugins@4.5.0-rc.2

### Patch Changes

-   Updated dependencies:
    -   `@trigger.dev/core@4.5.0-rc.2`

## @trigger.dev/python@4.5.0-rc.2

### Patch Changes

-   Updated dependencies:
    -   `@trigger.dev/sdk@4.5.0-rc.2`
    -   `@trigger.dev/build@4.5.0-rc.2`
    -   `@trigger.dev/core@4.5.0-rc.2`

## @trigger.dev/react-hooks@4.5.0-rc.2

### Patch Changes

-   Updated dependencies:
    -   `@trigger.dev/core@4.5.0-rc.2`

## @trigger.dev/redis-worker@4.5.0-rc.2

### Patch Changes

-   Updated dependencies:
    -   `@trigger.dev/core@4.5.0-rc.2`

## @trigger.dev/rsc@4.5.0-rc.2

### Patch Changes

-   Updated dependencies:
    -   `@trigger.dev/core@4.5.0-rc.2`

## @trigger.dev/schema-to-json@4.5.0-rc.2

### Patch Changes

-   Updated dependencies:
    -   `@trigger.dev/core@4.5.0-rc.2`

## @trigger.dev/sdk@4.5.0-rc.2

### Patch Changes

- Fix `chat.agent` HITL continuations on reasoning-heavy turns. Two
changes that work together:
([#3719](https://github.com/triggerdotdev/trigger.dev/pull/3719))

- The per-turn merge now overlays the wire copy's tool-part state
advancement onto the agent's existing chain — `state` + the matching
resolution field (`output` / `errorText` / `approval`) come from the
wire, everything else (text, reasoning, tool `input`, provider metadata)
stays whatever the snapshot or `hydrateMessages` returned. Previously a
full-message replace overwrote those fields with whatever the client
shipped, so a slimmed wire copy landed a tool call with no `arguments`
on the next LLM call. Covers `output-available` / `output-error` (HITL
`addToolOutput`) and `approval-responded` / `output-denied` (approval
flow).
- `TriggerChatTransport.sendMessages` and `AgentChat.sendRaw` now slim
assistant messages that carry advanced tool parts. The wire payload is
just `{ id, role, parts: [<state + resolution field>] }` for
`submit-message` continuations; everything else passes through.
Reasoning blobs and full tool inputs no longer ride the wire on every
`addToolOutput` / `addToolApproveResponse`, so continuation payloads
stay well under the `.in/append` cap on long agent loops.

Note: `onValidateMessages` receives the slim wire on HITL turns. If you
call `validateUIMessages` from `ai` against the full `messages` array it
will reject the slim assistant; filter to user messages (or skip on HITL
turns) — see the updated docstring on `onValidateMessages` for the
recommended pattern.

For `hydrateMessages` hooks that persist the chain, this release also
adds a small helper to the `@trigger.dev/sdk/ai` surface:

    ```ts
    import { chat, upsertIncomingMessage } from "@trigger.dev/sdk/ai";

    chat.agent({
hydrateMessages: async ({ chatId, trigger, incomingMessages }) => {
const record = await db.chat.findUnique({ where: { id: chatId } });
        const stored = record?.messages ?? [];
if (upsertIncomingMessage(stored, { trigger, incomingMessages })) {
await db.chat.update({ where: { id: chatId }, data: { messages: stored }
});
        }
        return stored;
      },
    });
    ```

It pushes fresh user messages by id, no-ops on HITL continuations (the
incoming shares an id with the existing assistant — the runtime overlays
the new tool-state advance), and skips on non-`submit-message` triggers.
Returns `true` if it mutated `stored` so the caller knows whether to
persist.

Net effect: `chat.addToolOutput(...)` /
`chat.addToolApproveResponse(...)` on multi-step reasoning agents
(OpenAI Responses with `store: false`, Anthropic extended thinking,
etc.) no longer blows the cap and no longer corrupts the LLM input.

- Add `TriggerClient` for running multiple SDK clients side-by-side,
each with its own auth, preview branch, and baseURL. Useful when a
single process needs to trigger tasks or read runs across multiple
projects, environments, or preview branches without mutating shared
global state.
([#3683](https://github.com/triggerdotdev/trigger.dev/pull/3683))

    ```ts
    import { TriggerClient } from "@trigger.dev/sdk";

const prod = new TriggerClient({ accessToken:
process.env.TRIGGER_PROD_KEY });
    const preview = new TriggerClient({
      accessToken: process.env.TRIGGER_PREVIEW_KEY,
      previewBranch: "signup-flow",
    });

    await prod.tasks.trigger("send-email", payload);
    await preview.runs.list({ status: ["COMPLETED"] });
    ```

-   Updated dependencies:
    -   `@trigger.dev/core@4.5.0-rc.2`

## @trigger.dev/core@4.5.0-rc.2

</details>

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2026-05-23 17:15:33 +01:00
Eric Allam 75679c7518 fix(sdk): chat HITL continuations no longer break the next LLM call (#3719)
## Summary

Multi-step reasoning agents with HITL tools (OpenAI Responses with
`store: false`, Anthropic extended thinking, etc.) failed on
`chat.addToolOutput(...)` continuations — either the wire payload blew
the `.in/append` cap (reasoning blobs + tool inputs routinely > 512
KiB), or app-side slimming workarounds got overwritten server-side and
the next LLM call landed a tool call with no `arguments`. Both modes are
fixed.

## Design

The per-turn merge in `chat.agent` now overlays only the tool-part state
advances (`output-available` / `output-error` / `approval-responded` /
`output-denied`) from the wire copy onto the hydrated/snapshot chain.
Previously it replaced the entire message, which dropped `input`,
reasoning, and text from the LLM's view whenever the wire was slim.

In parallel, `TriggerChatTransport.sendMessages` and `AgentChat.sendRaw`
now slim the assistant message themselves on `submit-message`
continuations: ship `{ id, role, parts: [<resolved tool part only>] }`,
everything else reconstructed server-side from `hydrateMessages` or the
durable snapshot. Continuation payloads drop from 600 KiB – 1 MiB to ~1
KiB.

`references/ai-chat` `aiChatHydrated.hydrateMessages` now upserts by id
instead of pushing. With slim continuations, a blind push duplicates the
assistant id in the returned chain — the merge updates the first match,
the slim duplicate goes straight to `toModelMessages` with no `input`,
and the LLM 4xx's. This is the canonical pattern customers should mirror
in their own hydrate implementations.

## Test plan

- 11 new tests (slim helper unit + slim+merge integration for HITL,
approval, default no-hydrate branch)
- Full SDK suite: 239 tests pass across 19 files
- End-to-end sweep against `references/ai-chat`: 19 customer-side smoke
tests green; HITL wire bodies confirmed at ~1 KiB (was 600 KiB+); no
provider 4xx errors across OpenAI Responses or Anthropic
2026-05-23 17:09:39 +01:00
Eric Allam 832cf7220b feat(sdk,core): add TriggerClient for per-instance SDK configuration (#3683)
## Summary

`new TriggerClient({...})` exposes the management API (tasks, runs,
schedules, envvars, batch, queues, deployments, prompts, auth) as an
explicit instance with its own auth, preview branch, and baseURL.
Multiple clients can coexist in one process without mutating shared
global state — useful when a single service triggers across multiple
projects, environments, or preview branches.

```ts
import { TriggerClient } from "@trigger.dev/sdk";

const prod = new TriggerClient({ accessToken: process.env.TRIGGER_PROD_KEY });
const preview = new TriggerClient({
  accessToken: process.env.TRIGGER_PREVIEW_KEY,
  previewBranch: "signup-flow",
});

await prod.tasks.trigger("send-email", payload);
await preview.runs.list({ status: ["COMPLETED"] });
```

The existing global `configure()` API keeps working unchanged.

## Design

Instance methods enter an `AsyncLocalStorage`-backed scope (`sdkScope`)
before delegating to the existing module-level functions. The four
"pollution" points that previously read globals now consult the scope
first:

- `apiClientManager.{baseURL, accessToken, branchName}` and
`clientOrThrow` — identity fields are scope-only when scoped; `baseURL`
still falls back to `TRIGGER_API_URL` because plumbing (where the API
lives) is not identity.
- `taskContext.{ctx, worker, isWarmStart, isInsideTask}` — masked inside
an isolated scope so a `client.tasks.trigger(...)` from inside a task
doesn't leak the parent's `parentRunId` / `lockToVersion` / `isTest`
into a trigger that hits a different project.
- Inline `getEnvVar("TRIGGER_VERSION")` reads in `shared.ts` go through
a `scopedEnvVar` helper that returns `undefined` inside an isolated
scope.

The `TriggerClient` class itself is a thin wrapper that captures the
scope in its constructor and proxies each namespace method to enter that
scope before calling the existing impl. Generic inference (e.g.
`client.tasks.trigger<typeof t>(...)`) is preserved via `Pick<typeof ns,
keyof curatedSubset>` typings.

Two correctness fixes uncovered along the way are folded in:

- `apiClientManager.setGlobalAPIClientConfiguration` no longer silently
no-ops on the second call. `configure()` now actually overrides as users
expect (this is the root cause behind some "I changed the config but
nothing happened" reports).
- `apiClientManager.runWithConfig` (and therefore `auth.withAuth`) is
now backed by `sdkScope.withScope` instead of "mutate the global and
restore in finally". Two parallel `withAuth` calls with different
configs no longer stomp each other.

Surface curation: instance namespaces drop methods that don't make sense
per-instance — `batch.*AndWait` (runtime-dependent), `schedules.task` /
`schedules.timezones` (definition-time / stateless), `prompts.define`
(definition-time), `auth.configure` / `auth.withAuth` (global-only).

## Test plan

- [x] 9 runtime unit tests in `triggerClient.test.ts` cover: required
accessToken, instance auth + branch headers, no env fallback for
identity fields, no leakage between global and instance, four parallel
calls across two clients stay isolated, taskContext masking +
`inheritContext: true` override, `configure()` second-call override,
parallel `auth.withAuth` isolation.
- [x] 10 type-level assertions in `triggerClient.types.test.ts` using
`expectTypeOf` + `@ts-expect-error` lock in generic inference, return
type passthrough, overload preservation, and curated-surface drift.
- [x] Full SDK suite (219 tests) and core suite (530 tests) pass.
- [x] Webapp typecheck clean.
- [x] End-to-end smoke test against local webapp and a
freshly-provisioned cloud project — six concurrent multi-client triggers
all returned 200 with run IDs, headers per-client as expected.
- [ ] Reviewer: run `references/multi-client` per its `README.md` to
reproduce the smoke test locally.

## Try it

`references/multi-client` is a new reference workspace that exercises
this end-to-end:

- `src/trigger/echo.ts` — trivial target task
- `src/trigger/fanOut.ts` — opens two `TriggerClient`s from inside a
task, fires `echo` through each in parallel
- `src/external/main.ts` — external Node script with two clients
triggering `echo` sequentially and concurrently; logs every outgoing
request's `authorization` + `x-trigger-branch`
- `src/external/isolation.ts` — interleaves global `configure()` and an
instance call, asserts the captured fetch sequence shows no leakage
either way
2026-05-22 12:29:42 +01:00
github-actions[bot] 0b5b817a49 chore: release v4.5.0-rc.1 (#3691)
🚀 Publish Trigger.dev Docker / typecheck (push) Failing after 0s
🚀 Publish Trigger.dev Docker / publish-webapp (push) Has been skipped
🚀 Publish Trigger.dev Docker / publish-worker (push) Has been skipped
🚀 Publish Trigger.dev Docker / publish-worker-v4 (push) Has been skipped
🚀 Publish Trigger.dev Docker / units (push) Failing after 0s
🧭 Helm Chart Release / lint-and-test (push) Has been cancelled
🧭 Helm Chart Release / release (push) Has been cancelled
## Summary
2 bug fixes.

## Bug fixes
- Fix `chat.agent` skills silently missing in `trigger dev` for projects
whose task files read `process.env` at module top level (e.g. a
third-party SDK client initialized at import). Skill folders now bundle
into `.trigger/skills/` reliably regardless of which env vars are set
when the CLI launches.
([#3690](https://github.com/triggerdotdev/trigger.dev/pull/3690))
- Fix `COULD_NOT_FIND_EXECUTOR` when a task's definition is loaded via
`await import(...)` from inside another task's `run()`. The runtime
workers now register such tasks with a sentinel file context, and the
catalog logs a one-time warning per task id.
([#3688](https://github.com/triggerdotdev/trigger.dev/pull/3688))

<details>
<summary>Raw changeset output</summary>

⚠️⚠️⚠️⚠️⚠️⚠️

`main` is currently in **pre mode** so this branch has prereleases
rather than normal releases. If you want to exit prereleases, run
`changeset pre exit` on `main`.

⚠️⚠️⚠️⚠️⚠️⚠️

# Releases
## @trigger.dev/build@4.5.0-rc.1

### Patch Changes

-   Updated dependencies:
    -   `@trigger.dev/core@4.5.0-rc.1`

## trigger.dev@4.5.0-rc.1

### Patch Changes

- Fix `chat.agent` skills silently missing in `trigger dev` for projects
whose task files read `process.env` at module top level (e.g. a
third-party SDK client initialized at import). Skill folders now bundle
into `.trigger/skills/` reliably regardless of which env vars are set
when the CLI launches.
([#3690](https://github.com/triggerdotdev/trigger.dev/pull/3690))
- Fix `COULD_NOT_FIND_EXECUTOR` when a task's definition is loaded via
`await import(...)` from inside another task's `run()`. The runtime
workers now register such tasks with a sentinel file context, and the
catalog logs a one-time warning per task id.
([#3688](https://github.com/triggerdotdev/trigger.dev/pull/3688))
-   Updated dependencies:
    -   `@trigger.dev/core@4.5.0-rc.1`
    -   `@trigger.dev/build@4.5.0-rc.1`
    -   `@trigger.dev/schema-to-json@4.5.0-rc.1`

## @trigger.dev/core@4.5.0-rc.1

### Patch Changes

- Fix `COULD_NOT_FIND_EXECUTOR` when a task's definition is loaded via
`await import(...)` from inside another task's `run()`. The runtime
workers now register such tasks with a sentinel file context, and the
catalog logs a one-time warning per task id.
([#3688](https://github.com/triggerdotdev/trigger.dev/pull/3688))

## @trigger.dev/plugins@4.5.0-rc.1

### Patch Changes

-   Updated dependencies:
    -   `@trigger.dev/core@4.5.0-rc.1`

## @trigger.dev/python@4.5.0-rc.1

### Patch Changes

-   Updated dependencies:
    -   `@trigger.dev/core@4.5.0-rc.1`
    -   `@trigger.dev/build@4.5.0-rc.1`
    -   `@trigger.dev/sdk@4.5.0-rc.1`

## @trigger.dev/react-hooks@4.5.0-rc.1

### Patch Changes

-   Updated dependencies:
    -   `@trigger.dev/core@4.5.0-rc.1`

## @trigger.dev/redis-worker@4.5.0-rc.1

### Patch Changes

-   Updated dependencies:
    -   `@trigger.dev/core@4.5.0-rc.1`

## @trigger.dev/rsc@4.5.0-rc.1

### Patch Changes

-   Updated dependencies:
    -   `@trigger.dev/core@4.5.0-rc.1`

## @trigger.dev/schema-to-json@4.5.0-rc.1

### Patch Changes

-   Updated dependencies:
    -   `@trigger.dev/core@4.5.0-rc.1`

## @trigger.dev/sdk@4.5.0-rc.1

### Patch Changes

-   Updated dependencies:
    -   `@trigger.dev/core@4.5.0-rc.1`

</details>

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2026-05-21 16:40:21 +01:00
Eric Allam 5ddb81ac8d fix(cli): stop chat.agent skills silently disappearing from trigger dev (#3690)
## Summary

`trigger.dev dev` was silently dropping registered `chat.agent` skills
for any project whose task files read `process.env` at module top level
— e.g. a third-party SDK client initialized at import. The agent would
boot fine, but `skill.local()` failed at runtime with `ENOENT` because
the skill folder was never copied into `.trigger/skills/`.

## Design

The CLI ran two indexer passes in dev: the worker's own indexer (with
the full env it eventually executes tasks in), and a separate
skill-discovery indexer with only the CLI process's env. Top-level reads
of vars like `TRIGGER_API_URL` imported cleanly in the worker pass and
threw in the skill pass — the latter caught the error, warned, and
skipped skill copying. Failure was silent enough that `skill.local()`
only surfaced it at task runtime.

The skill registry is already part of the worker manifest. This PR drops
the duplicate pass and copies skill folders from that manifest after the
worker initializes. One indexer instead of two; a bad `SKILL.md` now
surfaces as a startup error instead of silently disappearing skills.

Deploy is unaffected — its skill discovery uses the project's
environment variables (fetched via the API, which fills in
`TRIGGER_API_URL` etc.), so the dev failure mode doesn't reach there.

## Test plan

- [x] New `references/agent-skills` reference project with
`skills.define` + a task that calls `skill.local()` and runs a bundled
script
- [x] On `main`, adding a top-level
`process.env.TRIGGER_API_URL!.includes(...)` read in any task file
reproduces the symptom: warning at dev startup, no `.trigger/skills/`
folder, `skill.local()` fails with ENOENT
- [x] On this branch, same project boots clean and `skill.local()` works
end-to-end
- [x] Deploy still works end-to-end with the new reference project
2026-05-21 16:30:24 +01:00
nicktrn dbf9b4e7e5 fix(core,cli): register tasks loaded via dynamic import during run (#3688)
On a warm worker process, a task whose `task()` definition is loaded via
`await import(...)` from inside another task's `run()` could end up
permanently missing from the catalog: the `task()` call fired with no
`_currentFileContext` set, `registerTaskMetadata` silently returned, and
Node's ESM module cache then blocked the worker's setContext + re-import
recovery from ever firing the call again. Subsequent runs of that task
on the same warm process failed with `COULD_NOT_FIND_EXECUTOR` until the
process hit `maxExecutionsPerProcess` and exited.

All five of these had to coincide on the same worker for the bug to
surface:

1. `processKeepAlive` enabled (so catalog state survives across runs).
2. A `run()` function (or lifecycle hook) does `await import(...)`.
3. The import's transitive static graph reaches a `task()` /
`schemaTask()` call.
4. The task containing the dynamic import is the **first** task to run
on a given warm worker process — so the dropped `task()` calls fire on
this process for the first time, are silently dropped, and Node's module
cache locks the wrong outcome in.
5. A subsequent run for one of the dropped task ids lands on the same
warm worker before it recycles.

The runtime workers now set a sentinel file context (`<no-context>`)
around the `executor.execute(...)` call, so `task()` invocations firing
during a run register normally. The catalog detects the sentinel and
emits a one-time `console.warn` per task id so the pattern stays visible
without spamming. The indexer never sets this context, so deploy-time
behavior is unchanged.

Repro is `references/hello-world/src/trigger/dynamicImportRepro*.ts`.
Verified end-to-end against a deployed image with firestarter
warm-starts on: pre-fix saw `COULD_NOT_FIND_EXECUTOR` on children that
landed on the parent-poisoned worker; post-fix all 23/23 runs succeeded
and the warning surfaces in the parent's run trace.
2026-05-21 16:21:49 +01:00
github-actions[bot] acfba02409 chore: release v4.5.0-rc.0 (#3563)
🚀 Publish Trigger.dev Docker / typecheck (push) Failing after 4s
🚀 Publish Trigger.dev Docker / units (push) Failing after 0s
🚀 Publish Trigger.dev Docker / publish-webapp (push) Has been skipped
🚀 Publish Trigger.dev Docker / publish-worker (push) Has been skipped
🚀 Publish Trigger.dev Docker / publish-worker-v4 (push) Has been skipped
🧭 Helm Chart Release / lint-and-test (push) Has been cancelled
🧭 Helm Chart Release / release (push) Has been cancelled
## Summary
44 improvements, 1 bug fix.

## Improvements
- **AI Prompts** — define prompt templates as code alongside your tasks,
version them on deploy, and override the text or model from the
dashboard without redeploying. Prompts integrate with the Vercel AI SDK
via `toAISDKTelemetry()` (links every generation span back to the
prompt) and with `chat.agent` via `chat.prompt.set()` +
`chat.toStreamTextOptions()`.
([#3629](https://github.com/triggerdotdev/trigger.dev/pull/3629))
- **Code-defined, deploy-versioned templates** — define with
`prompts.define({ id, model, config, variables, content })`. Every
deploy creates a new version visible in the dashboard. Mustache-style
placeholders (`{{var}}`, `{{#cond}}...{{/cond}}`) with Zod / ArkType /
Valibot-typed variables.
- **Dashboard overrides** — change a prompt's text or model from the
dashboard without redeploying. Overrides take priority over the deployed
"current" version and are environment-scoped (dev / staging / production
independent).
- **Resolve API** — `prompt.resolve(vars, { version?, label? })` returns
the compiled `text`, resolved `model`, `version`, and labels. Standalone
`prompts.resolve<typeof handle>(slug, vars)` for cross-file resolution
with full type inference on slug and variable shape.
- **AI SDK integration** — spread `resolved.toAISDKTelemetry({ ...extra
})` into any `generateText` / `streamText` call and every generation
span links to the prompt in the dashboard alongside its input variables,
model, tokens, and cost.
- **`chat.agent` integration** — `chat.prompt.set(resolved)` stores the
resolved prompt run-scoped; `chat.toStreamTextOptions({ registry })`
pulls `system`, `model` (resolved via the AI SDK provider registry),
`temperature` / `maxTokens` / etc., and telemetry into a single spread
for `streamText`.
- **Management SDK** — `prompts.list()`, `prompts.versions(slug)`,
`prompts.promote(slug, version)`, `prompts.createOverride(slug, body)`,
`prompts.updateOverride(slug, body)`, `prompts.removeOverride(slug)`,
`prompts.reactivateOverride(slug, version)`.
- **Dashboard** — prompts list with per-prompt usage sparklines;
per-prompt detail with Template / Details / Versions / Generations /
Metrics tabs. AI generation spans get a custom inspector showing the
linked prompt's metadata, input variables, and template content
alongside model, tokens, cost, and the message thread.
- Adds `onBoot` to `chat.agent` — a lifecycle hook that fires once per
worker process picking up the chat. Runs for the initial run, preloaded
runs, AND reactive continuation runs (post-cancel, crash, `endRun`,
`requestUpgrade`, OOM retry), before any other hook. Use it to
initialize `chat.local`, open per-process resources, or re-hydrate state
from your DB on continuation — anywhere the SAME run picking up after
suspend/resume isn't enough.
([#3543](https://github.com/triggerdotdev/trigger.dev/pull/3543))
- **AI SDK `useChat` integration** — a custom
[`ChatTransport`](https://sdk.vercel.ai/docs/ai-sdk-ui/transport)
(`useTriggerChatTransport`) plugs straight into Vercel AI SDK's
`useChat` hook. Text streaming, tool calls, reasoning, and `data-*`
parts all work natively over Trigger.dev's realtime streams. No custom
API routes needed.
- **First-turn fast path (`chat.headStart`)** — opt-in handler that runs
the first turn's `streamText` step in your warm server process while the
agent run boots in parallel, cutting cold-start TTFC by roughly half
(measured 2801ms → 1218ms on `claude-sonnet-4-6`). The agent owns step
2+ (tool execution, persistence, hooks) so heavy deps stay where they
belong. Web Fetch handler works natively in Next.js, Hono, SvelteKit,
Remix, Workers, etc.; bridge to Express/Fastify/Koa via
`chat.toNodeListener`. New `@trigger.dev/sdk/chat-server` subpath.
- **Multi-turn durability via Sessions** — every chat is backed by a
durable Session that outlives any individual run. Conversations resume
across page refreshes, idle timeout, crashes, and deploys; `resume:
true` reconnects via `lastEventId` so clients only see new chunks.
`sessions.list` enumerates chats for inbox-style UIs.
- **Auto-accumulated history, delta-only wire** — the backend
accumulates the full conversation across turns; clients only ship the
new message each turn. Long chats never hit the 512 KiB body cap.
Register `hydrateMessages` to be the source of truth yourself.
- **Lifecycle hooks** — `onPreload`, `onChatStart`,
`onValidateMessages`, `hydrateMessages`, `onTurnStart`,
`onBeforeTurnComplete`, `onTurnComplete`, `onChatSuspend`,
`onChatResume` — for persistence, validation, and post-turn work.
- **Stop generation** — client-driven `transport.stopGeneration(chatId)`
aborts mid-stream; the run stays alive for the next message, partial
response is captured, and aborted parts (stuck `partial-call` tools,
in-progress reasoning) are auto-cleaned.
- **Tool approvals (HITL)** — tools with `needsApproval: true` pause
until the user approves or denies via `addToolApprovalResponse`. The
runtime reconciles the updated assistant message by ID and continues
`streamText`.
- **Steering and background injection** — `pendingMessages` injects user
messages between tool-call steps so users can steer the agent
mid-execution; `chat.inject()` + `chat.defer()` adds context from
background work (self-review, RAG, safety checks) between turns.
- **Actions** — non-turn frontend commands (undo, rollback, regenerate,
edit) sent via `transport.sendAction`. Fire `hydrateMessages` +
`onAction` only — no turn hooks, no `run()`. `onAction` can return a
`StreamTextResult` for a model response, or `void` for side-effect-only.
- **Typed state primitives** — `chat.local<T>` for per-run state
accessible from hooks, `run()`, tools, and subtasks (auto-serialized
through `ai.toolExecute`); `chat.store` for typed shared data between
agent and client; `chat.history` for reading and mutating the message
chain; `clientDataSchema` for typed `clientData` in every hook.
- **`chat.toStreamTextOptions()`** — one spread into `streamText` wires
up versioned system [Prompts](https://trigger.dev/docs/ai/prompts),
model resolution, telemetry metadata, compaction, steering, and
background injection.
- **Multi-tab coordination** — `multiTab: true` + `useMultiTabChat`
prevents duplicate sends and syncs state across browser tabs via
`BroadcastChannel`. Non-active tabs go read-only with live updates.
- **Network resilience** — built-in indefinite retry with bounded
backoff, reconnect on `online` / tab refocus / bfcache restore,
`Last-Event-ID` mid-stream resume. No app code needed.
- **Sessions** — a durable, run-aware stream channel keyed on a stable
`externalId`. A Session is the unit of state that owns a multi-run
conversation: messages flow through `.in`, responses through `.out`,
both survive run boundaries. Sessions back the new `chat.agent` runtime,
and you can build on them directly for any pattern that needs durable
bi-directional streaming across runs.
([#3542](https://github.com/triggerdotdev/trigger.dev/pull/3542))
- Add `ai.toolExecute(task)` so you can wire a Trigger subtask in as the
`execute` handler of an AI SDK `tool()` while defining `description` and
`inputSchema` yourself — useful when you want full control over the tool
surface and just need Trigger's subtask machinery for the body.
([#3546](https://github.com/triggerdotdev/trigger.dev/pull/3546))
- Type `chat.createStartSessionAction` against your chat agent so
`clientData` is typed end-to-end on the first turn:
([#3684](https://github.com/triggerdotdev/trigger.dev/pull/3684))
- Add `region` to the runs list / retrieve API: filter runs by region
(`runs.list({ region: "..." })` / `filter[region]=<masterQueue>`) and
read each run's executing region from the new `region` field on the
response.
([#3612](https://github.com/triggerdotdev/trigger.dev/pull/3612))
- Add `TRIGGER_BUILD_SKIP_REWRITE_TIMESTAMP=1` escape hatch for local
self-hosted builds whose buildx driver doesn't support
`rewrite-timestamp` alongside push (e.g. orbstack's default `docker`
driver).
([#3618](https://github.com/triggerdotdev/trigger.dev/pull/3618))
- Reject overlong `idempotencyKey` values at the API boundary so they no
longer trip an internal size limit on the underlying unique index and
surface as a generic 500. Inputs are capped at 2048 characters — well
above what `idempotencyKeys.create()` produces (a 64-character hash) and
above any realistic raw key. Applies to `tasks.trigger`,
`tasks.batchTrigger`, `batch.create` (Phase 1 streaming batches),
`wait.createToken`, `wait.forDuration`, and the input/session stream
waitpoint endpoints. Over-limit requests now return a structured 400
instead.
([#3560](https://github.com/triggerdotdev/trigger.dev/pull/3560))
- **AI SDK `useChat` integration** — a custom
[`ChatTransport`](https://sdk.vercel.ai/docs/ai-sdk-ui/transport)
(`useTriggerChatTransport`) plugs straight into Vercel AI SDK's
`useChat` hook. Text streaming, tool calls, reasoning, and `data-*`
parts all work natively over Trigger.dev's realtime streams. No custom
API routes needed.
- **First-turn fast path (`chat.headStart`)** — opt-in handler that runs
the first turn's `streamText` step in your warm server process while the
agent run boots in parallel, cutting cold-start TTFC by roughly half
(measured 2801ms → 1218ms on `claude-sonnet-4-6`). The agent owns step
2+ (tool execution, persistence, hooks) so heavy deps stay where they
belong. Web Fetch handler works natively in Next.js, Hono, SvelteKit,
Remix, Workers, etc.; bridge to Express/Fastify/Koa via
`chat.toNodeListener`. New `@trigger.dev/sdk/chat-server` subpath.
- **Multi-turn durability via Sessions** — every chat is backed by a
durable Session that outlives any individual run. Conversations resume
across page refreshes, idle timeout, crashes, and deploys; `resume:
true` reconnects via `lastEventId` so clients only see new chunks.
`sessions.list` enumerates chats for inbox-style UIs.
- **Auto-accumulated history, delta-only wire** — the backend
accumulates the full conversation across turns; clients only ship the
new message each turn. Long chats never hit the 512 KiB body cap.
Register `hydrateMessages` to be the source of truth yourself.
- **Lifecycle hooks** — `onPreload`, `onChatStart`,
`onValidateMessages`, `hydrateMessages`, `onTurnStart`,
`onBeforeTurnComplete`, `onTurnComplete`, `onChatSuspend`,
`onChatResume` — for persistence, validation, and post-turn work.
- **Stop generation** — client-driven `transport.stopGeneration(chatId)`
aborts mid-stream; the run stays alive for the next message, partial
response is captured, and aborted parts (stuck `partial-call` tools,
in-progress reasoning) are auto-cleaned.
- **Tool approvals (HITL)** — tools with `needsApproval: true` pause
until the user approves or denies via `addToolApprovalResponse`. The
runtime reconciles the updated assistant message by ID and continues
`streamText`.
- **Steering and background injection** — `pendingMessages` injects user
messages between tool-call steps so users can steer the agent
mid-execution; `chat.inject()` + `chat.defer()` adds context from
background work (self-review, RAG, safety checks) between turns.
- **Actions** — non-turn frontend commands (undo, rollback, regenerate,
edit) sent via `transport.sendAction`. Fire `hydrateMessages` +
`onAction` only — no turn hooks, no `run()`. `onAction` can return a
`StreamTextResult` for a model response, or `void` for side-effect-only.
- **Typed state primitives** — `chat.local<T>` for per-run state
accessible from hooks, `run()`, tools, and subtasks (auto-serialized
through `ai.toolExecute`); `chat.store` for typed shared data between
agent and client; `chat.history` for reading and mutating the message
chain; `clientDataSchema` for typed `clientData` in every hook.
- **`chat.toStreamTextOptions()`** — one spread into `streamText` wires
up versioned system [Prompts](https://trigger.dev/docs/ai/prompts),
model resolution, telemetry metadata, compaction, steering, and
background injection.
- **Multi-tab coordination** — `multiTab: true` + `useMultiTabChat`
prevents duplicate sends and syncs state across browser tabs via
`BroadcastChannel`. Non-active tabs go read-only with live updates.
- **Network resilience** — built-in indefinite retry with bounded
backoff, reconnect on `online` / tab refocus / bfcache restore,
`Last-Event-ID` mid-stream resume. No app code needed.
- Retry `TASK_PROCESS_SIGSEGV` task crashes under the user's retry
policy instead of failing the run on the first segfault. SIGSEGV in Node
tasks is frequently non-deterministic (native addon races, JIT/GC
interaction, near-OOM in native code, host issues), so retrying on a
fresh process often succeeds. The retry is gated by the task's existing
`retry` config + `maxAttempts` — same path `TASK_PROCESS_SIGTERM` and
uncaught exceptions already use — so tasks without a retry policy still
fail fast.
([#3552](https://github.com/triggerdotdev/trigger.dev/pull/3552))
- The public interfaces for a plugin system. Initially consolidated
authentication and authorization interfaces.
([#3499](https://github.com/triggerdotdev/trigger.dev/pull/3499))
- Add MollifierBuffer and MollifierDrainer primitives for trigger burst
smoothing.
([#3614](https://github.com/triggerdotdev/trigger.dev/pull/3614))

## Bug fixes
- Fix `LocalsKey<T>` type incompatibility across dual-package builds.
The phantom value-type brand no longer uses a module-level `unique
symbol`, so a single TypeScript compilation that resolves the type from
both the ESM and CJS outputs (which can happen under certain pnpm
hoisting layouts) no longer sees two structurally-incompatible variants
of the same type.
([#3626](https://github.com/triggerdotdev/trigger.dev/pull/3626))

<details>
<summary>Raw changeset output</summary>

⚠️⚠️⚠️⚠️⚠️⚠️

`main` is currently in **pre mode** so this branch has prereleases
rather than normal releases. If you want to exit prereleases, run
`changeset pre exit` on `main`.

⚠️⚠️⚠️⚠️⚠️⚠️

# Releases
## @trigger.dev/sdk@4.5.0-rc.0

### Minor Changes

- **AI Prompts** — define prompt templates as code alongside your tasks,
version them on deploy, and override the text or model from the
dashboard without redeploying. Prompts integrate with the Vercel AI SDK
via `toAISDKTelemetry()` (links every generation span back to the
prompt) and with `chat.agent` via `chat.prompt.set()` +
`chat.toStreamTextOptions()`.
([#3629](https://github.com/triggerdotdev/trigger.dev/pull/3629))

    ```ts
    import { prompts } from "@trigger.dev/sdk";
    import { generateText } from "ai";
    import { openai } from "@ai-sdk/openai";
    import { z } from "zod";

    export const supportPrompt = prompts.define({
      id: "customer-support",
      model: "gpt-4o",
      config: { temperature: 0.7 },
      variables: z.object({
        customerName: z.string(),
        plan: z.string(),
        issue: z.string(),
      }),
      content: `You are a support agent for Acme.

    Customer: {{customerName}} ({{plan}} plan)
    Issue: {{issue}}`,
    });

    const resolved = await supportPrompt.resolve({
      customerName: "Alice",
      plan: "Pro",
      issue: "Can't access billing",
    });

    const result = await generateText({
      model: openai(resolved.model ?? "gpt-4o"),
      system: resolved.text,
      prompt: "Can't access billing",
      ...resolved.toAISDKTelemetry(),
    });
    ```

    **What you get:**

- **Code-defined, deploy-versioned templates** — define with
`prompts.define({ id, model, config, variables, content })`. Every
deploy creates a new version visible in the dashboard. Mustache-style
placeholders (`{{var}}`, `{{#cond}}...{{/cond}}`) with Zod / ArkType /
Valibot-typed variables.
- **Dashboard overrides** — change a prompt's text or model from the
dashboard without redeploying. Overrides take priority over the deployed
"current" version and are environment-scoped (dev / staging / production
independent).
- **Resolve API** — `prompt.resolve(vars, { version?, label? })` returns
the compiled `text`, resolved `model`, `version`, and labels. Standalone
`prompts.resolve<typeof handle>(slug, vars)` for cross-file resolution
with full type inference on slug and variable shape.
- **AI SDK integration** — spread `resolved.toAISDKTelemetry({ ...extra
})` into any `generateText` / `streamText` call and every generation
span links to the prompt in the dashboard alongside its input variables,
model, tokens, and cost.
- **`chat.agent` integration** — `chat.prompt.set(resolved)` stores the
resolved prompt run-scoped; `chat.toStreamTextOptions({ registry })`
pulls `system`, `model` (resolved via the AI SDK provider registry),
`temperature` / `maxTokens` / etc., and telemetry into a single spread
for `streamText`.
- **Management SDK** — `prompts.list()`, `prompts.versions(slug)`,
`prompts.promote(slug, version)`, `prompts.createOverride(slug, body)`,
`prompts.updateOverride(slug, body)`, `prompts.removeOverride(slug)`,
`prompts.reactivateOverride(slug, version)`.
- **Dashboard** — prompts list with per-prompt usage sparklines;
per-prompt detail with Template / Details / Versions / Generations /
Metrics tabs. AI generation spans get a custom inspector showing the
linked prompt's metadata, input variables, and template content
alongside model, tokens, cost, and the message thread.

See [/docs/ai/prompts](https://trigger.dev/docs/ai/prompts) for the full
reference — template syntax, version resolution order, override
workflow, and type utilities (`PromptHandle`, `PromptIdentifier`,
`PromptVariables`).

- Adds `onBoot` to `chat.agent` — a lifecycle hook that fires once per
worker process picking up the chat. Runs for the initial run, preloaded
runs, AND reactive continuation runs (post-cancel, crash, `endRun`,
`requestUpgrade`, OOM retry), before any other hook. Use it to
initialize `chat.local`, open per-process resources, or re-hydrate state
from your DB on continuation — anywhere the SAME run picking up after
suspend/resume isn't enough.
([#3543](https://github.com/triggerdotdev/trigger.dev/pull/3543))

    ```ts
const userContext = chat.local<{ name: string; plan: string }>({ id:
"userContext" });

    export const myChat = chat.agent({
      id: "my-chat",
      onBoot: async ({ clientData, continuation }) => {
const user = await db.user.findUnique({ where: { id: clientData.userId }
});
        userContext.init({ name: user.name, plan: user.plan });
      },
      run: async ({ messages, signal }) =>
streamText({ model: openai("gpt-4o"), messages, abortSignal: signal }),
    });
    ```

Use `onBoot` (not `onChatStart`) for state setup that must run every
time a worker picks up the chat — `onChatStart` fires once per chat and
won't run on continuation, leaving `chat.local` uninitialized when
`run()` tries to use it.

- **AI Agents** — run AI SDK chat completions as durable Trigger.dev
agents instead of fragile API routes. Define an agent in one function,
point `useChat` at it from React, and the conversation survives page
refreshes, network blips, and process restarts.
([#3543](https://github.com/triggerdotdev/trigger.dev/pull/3543))

    ```ts
    import { chat } from "@trigger.dev/sdk/ai";
    import { streamText } from "ai";
    import { openai } from "@ai-sdk/openai";

    export const myChat = chat.agent({
      id: "my-chat",
      run: async ({ messages, signal }) =>
streamText({ model: openai("gpt-4o"), messages, abortSignal: signal }),
    });
    ```

    ```tsx
    import { useChat } from "@ai-sdk/react";
import { useTriggerChatTransport } from "@trigger.dev/sdk/chat/react";

const transport = useTriggerChatTransport({ task: "my-chat",
accessToken, startSession });
    const { messages, sendMessage } = useChat({ transport });
    ```

    **What you get:**

- **AI SDK `useChat` integration** — a custom
[`ChatTransport`](https://sdk.vercel.ai/docs/ai-sdk-ui/transport)
(`useTriggerChatTransport`) plugs straight into Vercel AI SDK's
`useChat` hook. Text streaming, tool calls, reasoning, and `data-*`
parts all work natively over Trigger.dev's realtime streams. No custom
API routes needed.
- **First-turn fast path (`chat.headStart`)** — opt-in handler that runs
the first turn's `streamText` step in your warm server process while the
agent run boots in parallel, cutting cold-start TTFC by roughly half
(measured 2801ms → 1218ms on `claude-sonnet-4-6`). The agent owns step
2+ (tool execution, persistence, hooks) so heavy deps stay where they
belong. Web Fetch handler works natively in Next.js, Hono, SvelteKit,
Remix, Workers, etc.; bridge to Express/Fastify/Koa via
`chat.toNodeListener`. New `@trigger.dev/sdk/chat-server` subpath.
- **Multi-turn durability via Sessions** — every chat is backed by a
durable Session that outlives any individual run. Conversations resume
across page refreshes, idle timeout, crashes, and deploys; `resume:
true` reconnects via `lastEventId` so clients only see new chunks.
`sessions.list` enumerates chats for inbox-style UIs.
- **Auto-accumulated history, delta-only wire** — the backend
accumulates the full conversation across turns; clients only ship the
new message each turn. Long chats never hit the 512 KiB body cap.
Register `hydrateMessages` to be the source of truth yourself.
- **Lifecycle hooks** — `onPreload`, `onChatStart`,
`onValidateMessages`, `hydrateMessages`, `onTurnStart`,
`onBeforeTurnComplete`, `onTurnComplete`, `onChatSuspend`,
`onChatResume` — for persistence, validation, and post-turn work.
- **Stop generation** — client-driven `transport.stopGeneration(chatId)`
aborts mid-stream; the run stays alive for the next message, partial
response is captured, and aborted parts (stuck `partial-call` tools,
in-progress reasoning) are auto-cleaned.
- **Tool approvals (HITL)** — tools with `needsApproval: true` pause
until the user approves or denies via `addToolApprovalResponse`. The
runtime reconciles the updated assistant message by ID and continues
`streamText`.
- **Steering and background injection** — `pendingMessages` injects user
messages between tool-call steps so users can steer the agent
mid-execution; `chat.inject()` + `chat.defer()` adds context from
background work (self-review, RAG, safety checks) between turns.
- **Actions** — non-turn frontend commands (undo, rollback, regenerate,
edit) sent via `transport.sendAction`. Fire `hydrateMessages` +
`onAction` only — no turn hooks, no `run()`. `onAction` can return a
`StreamTextResult` for a model response, or `void` for side-effect-only.
- **Typed state primitives** — `chat.local<T>` for per-run state
accessible from hooks, `run()`, tools, and subtasks (auto-serialized
through `ai.toolExecute`); `chat.store` for typed shared data between
agent and client; `chat.history` for reading and mutating the message
chain; `clientDataSchema` for typed `clientData` in every hook.
- **`chat.toStreamTextOptions()`** — one spread into `streamText` wires
up versioned system [Prompts](https://trigger.dev/docs/ai/prompts),
model resolution, telemetry metadata, compaction, steering, and
background injection.
- **Multi-tab coordination** — `multiTab: true` + `useMultiTabChat`
prevents duplicate sends and syncs state across browser tabs via
`BroadcastChannel`. Non-active tabs go read-only with live updates.
- **Network resilience** — built-in indefinite retry with bounded
backoff, reconnect on `online` / tab refocus / bfcache restore,
`Last-Event-ID` mid-stream resume. No app code needed.

See [/docs/ai-chat](https://trigger.dev/docs/ai-chat/overview) for the
full surface — quick start, three backend approaches (`chat.agent`,
`chat.createSession`, raw task), persistence and code-sandbox patterns,
type-level guides, and API reference.

- Add read primitives to `chat.history` for HITL flows:
`getPendingToolCalls()`, `getResolvedToolCalls()`,
`extractNewToolResults(message)`, `getChain()`, and
`findMessage(messageId)`. These lift the accumulator-walking logic that
customers building human-in-the-loop tools were re-implementing into the
SDK. ([#3543](https://github.com/triggerdotdev/trigger.dev/pull/3543))

Use `getPendingToolCalls()` to gate fresh user turns while a tool call
is awaiting an answer. Use `extractNewToolResults(message)` to dedup
tool results when persisting to your own store — the helper returns only
the parts whose `toolCallId` is not already resolved on the chain.

    ```ts
    const pending = chat.history.getPendingToolCalls();
    if (pending.length > 0) {
      // an addToolOutput is expected before a new user message
    }

    onTurnComplete: async ({ responseMessage }) => {
const newResults = chat.history.extractNewToolResults(responseMessage);
      for (const r of newResults) {
await db.toolResults.upsert({ id: r.toolCallId, output: r.output,
errorText: r.errorText });
      }
    };
    ```

- **Sessions** — a durable, run-aware stream channel keyed on a stable
`externalId`. A Session is the unit of state that owns a multi-run
conversation: messages flow through `.in`, responses through `.out`,
both survive run boundaries. Sessions back the new `chat.agent` runtime,
and you can build on them directly for any pattern that needs durable
bi-directional streaming across runs.
([#3542](https://github.com/triggerdotdev/trigger.dev/pull/3542))

    ```ts
    import { sessions, tasks } from "@trigger.dev/sdk";

    // Trigger a task and subscribe to its session output in one call
const { runId, stream } = await tasks.triggerAndSubscribe("my-task",
payload, {
      externalId: "user-456",
    });

    for await (const chunk of stream) {
      // ...
    }

// Enumerate existing sessions (powers inbox-style UIs without a
separate index)
for await (const s of sessions.list({ type: "chat.agent", tag:
"user:user-456" })) {
      console.log(s.id, s.externalId, s.createdAt, s.closedAt);
    }
    ```

See [/docs/ai-chat/overview](https://trigger.dev/docs/ai-chat/overview)
for the full surface — Sessions powers the durable, resumable chat
runtime described there.

### Patch Changes

- Add Agent Skills for `chat.agent`. Drop a folder with a `SKILL.md` and
any helper scripts/references next to your task code, register it with
`skills.define({ id, path })`, and the CLI bundles it into the deploy
image automatically — no `trigger.config.ts` changes. The agent gets a
one-line summary in its system prompt and discovers full instructions on
demand via `loadSkill`, with `bash` and `readFile` tools scoped
per-skill (path-traversal guards, output caps, abort-signal
propagation).
([#3543](https://github.com/triggerdotdev/trigger.dev/pull/3543))

    ```ts
const pdfSkill = skills.define({ id: "pdf-extract", path:
"./skills/pdf-extract" });

    chat.skills.set([await pdfSkill.local()]);
    ```

Built on the [AI SDK cookbook
pattern](https://ai-sdk.dev/cookbook/guides/agent-skills) — portable
across providers. SDK + CLI only for now; dashboard-editable `SKILL.md`
text is on the roadmap.

- Add `ai.toolExecute(task)` so you can wire a Trigger subtask in as the
`execute` handler of an AI SDK `tool()` while defining `description` and
`inputSchema` yourself — useful when you want full control over the tool
surface and just need Trigger's subtask machinery for the body.
([#3546](https://github.com/triggerdotdev/trigger.dev/pull/3546))

    ```ts
    const myTool = tool({
      description: "...",
      inputSchema: z.object({ ... }),
      execute: ai.toolExecute(mySubtask),
    });
    ```

`ai.tool(task)` (`toolFromTask`) keeps doing the all-in-one wrap and now
aligns its return type with AI SDK's `ToolSet`. Minimum `ai` peer raised
to `^6.0.116` to avoid cross-version `ToolSet` mismatches in monorepos.

- Stamp `gen_ai.conversation.id` (the chat id) on every span and metric
emitted from inside a `chat.task` or `chat.agent` run. Lets you filter
dashboard spans, runs, and metrics by the chat conversation that
produced them — independent of the run boundary, so multi-run chats
correlate cleanly. No code changes required on the user side.
([#3543](https://github.com/triggerdotdev/trigger.dev/pull/3543))

- Type `chat.createStartSessionAction` against your chat agent so
`clientData` is typed end-to-end on the first turn:
([#3684](https://github.com/triggerdotdev/trigger.dev/pull/3684))

    ```ts
    import { chat } from "@trigger.dev/sdk/ai";
    import type { myChat } from "@/trigger/chat";

export const startChatSession = chat.createStartSessionAction<typeof
myChat>("my-chat");

// In the browser, threaded from the transport's typed startSession
callback:
    const transport = useTriggerChatTransport<typeof myChat>({
      task: "my-chat",
startSession: ({ chatId, clientData }) => startChatSession({ chatId,
clientData }),
      // ...
    });
    ```

`ChatStartSessionParams` gains a typed `clientData` field — folded into
the first run's `payload.metadata` so `onPreload` / `onChatStart` see
the same shape per-turn `metadata` carries via the transport. The opaque
session-level `metadata` field is unchanged.

- Unit-test `chat.agent` definitions offline with `mockChatAgent` from
`@trigger.dev/sdk/ai/test`. Drives a real agent's turn loop in-process —
no network, no task runtime — so you can send messages, actions, and
stop signals via driver methods, inspect captured output chunks, and
verify hooks fire. Pairs with `MockLanguageModelV3` from `ai/test` for
model mocking. `setupLocals` lets you pre-seed `locals` (DB clients,
service stubs) before `run()` starts.
([#3543](https://github.com/triggerdotdev/trigger.dev/pull/3543))

The broader `runInMockTaskContext` harness it's built on lives at
`@trigger.dev/core/v3/test` — useful for unit-testing any task code, not
just chat.

- Add `region` to the runs list / retrieve API: filter runs by region
(`runs.list({ region: "..." })` / `filter[region]=<masterQueue>`) and
read each run's executing region from the new `region` field on the
response.
([#3612](https://github.com/triggerdotdev/trigger.dev/pull/3612))

-   Updated dependencies:
    -   `@trigger.dev/core@4.5.0-rc.0`

## @trigger.dev/build@4.5.0-rc.0

### Patch Changes

- Add Agent Skills for `chat.agent`. Drop a folder with a `SKILL.md` and
any helper scripts/references next to your task code, register it with
`skills.define({ id, path })`, and the CLI bundles it into the deploy
image automatically — no `trigger.config.ts` changes. The agent gets a
one-line summary in its system prompt and discovers full instructions on
demand via `loadSkill`, with `bash` and `readFile` tools scoped
per-skill (path-traversal guards, output caps, abort-signal
propagation).
([#3543](https://github.com/triggerdotdev/trigger.dev/pull/3543))

    ```ts
const pdfSkill = skills.define({ id: "pdf-extract", path:
"./skills/pdf-extract" });

    chat.skills.set([await pdfSkill.local()]);
    ```

Built on the [AI SDK cookbook
pattern](https://ai-sdk.dev/cookbook/guides/agent-skills) — portable
across providers. SDK + CLI only for now; dashboard-editable `SKILL.md`
text is on the roadmap.

-   Updated dependencies:
    -   `@trigger.dev/core@4.5.0-rc.0`

## trigger.dev@4.5.0-rc.0

### Patch Changes

- Add Agent Skills for `chat.agent`. Drop a folder with a `SKILL.md` and
any helper scripts/references next to your task code, register it with
`skills.define({ id, path })`, and the CLI bundles it into the deploy
image automatically — no `trigger.config.ts` changes. The agent gets a
one-line summary in its system prompt and discovers full instructions on
demand via `loadSkill`, with `bash` and `readFile` tools scoped
per-skill (path-traversal guards, output caps, abort-signal
propagation).
([#3543](https://github.com/triggerdotdev/trigger.dev/pull/3543))

    ```ts
const pdfSkill = skills.define({ id: "pdf-extract", path:
"./skills/pdf-extract" });

    chat.skills.set([await pdfSkill.local()]);
    ```

Built on the [AI SDK cookbook
pattern](https://ai-sdk.dev/cookbook/guides/agent-skills) — portable
across providers. SDK + CLI only for now; dashboard-editable `SKILL.md`
text is on the roadmap.

- Add `TRIGGER_BUILD_SKIP_REWRITE_TIMESTAMP=1` escape hatch for local
self-hosted builds whose buildx driver doesn't support
`rewrite-timestamp` alongside push (e.g. orbstack's default `docker`
driver).
([#3618](https://github.com/triggerdotdev/trigger.dev/pull/3618))

- The CLI MCP server's agent-chat tools (`start_agent_chat`,
`send_agent_message`, `close_agent_chat`) now run on the new Sessions
primitive, so AI assistants driving a `chat.agent` get the same
idempotent-by-`chatId`, durable-across-runs behavior the browser
transport gets. Required PAT scopes go from `write:inputStreams` to
`read:sessions` + `write:sessions`.
([#3546](https://github.com/triggerdotdev/trigger.dev/pull/3546))

- MCP `list_runs` tool: add a `region` filter input and surface each
run's executing region in the formatted summary.
([#3612](https://github.com/triggerdotdev/trigger.dev/pull/3612))

-   Updated dependencies:
    -   `@trigger.dev/core@4.5.0-rc.0`
    -   `@trigger.dev/build@4.5.0-rc.0`
    -   `@trigger.dev/schema-to-json@4.5.0-rc.0`

## @trigger.dev/core@4.5.0-rc.0

### Patch Changes

- Add Agent Skills for `chat.agent`. Drop a folder with a `SKILL.md` and
any helper scripts/references next to your task code, register it with
`skills.define({ id, path })`, and the CLI bundles it into the deploy
image automatically — no `trigger.config.ts` changes. The agent gets a
one-line summary in its system prompt and discovers full instructions on
demand via `loadSkill`, with `bash` and `readFile` tools scoped
per-skill (path-traversal guards, output caps, abort-signal
propagation).
([#3543](https://github.com/triggerdotdev/trigger.dev/pull/3543))

    ```ts
const pdfSkill = skills.define({ id: "pdf-extract", path:
"./skills/pdf-extract" });

    chat.skills.set([await pdfSkill.local()]);
    ```

Built on the [AI SDK cookbook
pattern](https://ai-sdk.dev/cookbook/guides/agent-skills) — portable
across providers. SDK + CLI only for now; dashboard-editable `SKILL.md`
text is on the roadmap.

- Reject overlong `idempotencyKey` values at the API boundary so they no
longer trip an internal size limit on the underlying unique index and
surface as a generic 500. Inputs are capped at 2048 characters — well
above what `idempotencyKeys.create()` produces (a 64-character hash) and
above any realistic raw key. Applies to `tasks.trigger`,
`tasks.batchTrigger`, `batch.create` (Phase 1 streaming batches),
`wait.createToken`, `wait.forDuration`, and the input/session stream
waitpoint endpoints. Over-limit requests now return a structured 400
instead.
([#3560](https://github.com/triggerdotdev/trigger.dev/pull/3560))

- **AI Agents** — run AI SDK chat completions as durable Trigger.dev
agents instead of fragile API routes. Define an agent in one function,
point `useChat` at it from React, and the conversation survives page
refreshes, network blips, and process restarts.
([#3543](https://github.com/triggerdotdev/trigger.dev/pull/3543))

    ```ts
    import { chat } from "@trigger.dev/sdk/ai";
    import { streamText } from "ai";
    import { openai } from "@ai-sdk/openai";

    export const myChat = chat.agent({
      id: "my-chat",
      run: async ({ messages, signal }) =>
streamText({ model: openai("gpt-4o"), messages, abortSignal: signal }),
    });
    ```

    ```tsx
    import { useChat } from "@ai-sdk/react";
import { useTriggerChatTransport } from "@trigger.dev/sdk/chat/react";

const transport = useTriggerChatTransport({ task: "my-chat",
accessToken, startSession });
    const { messages, sendMessage } = useChat({ transport });
    ```

    **What you get:**

- **AI SDK `useChat` integration** — a custom
[`ChatTransport`](https://sdk.vercel.ai/docs/ai-sdk-ui/transport)
(`useTriggerChatTransport`) plugs straight into Vercel AI SDK's
`useChat` hook. Text streaming, tool calls, reasoning, and `data-*`
parts all work natively over Trigger.dev's realtime streams. No custom
API routes needed.
- **First-turn fast path (`chat.headStart`)** — opt-in handler that runs
the first turn's `streamText` step in your warm server process while the
agent run boots in parallel, cutting cold-start TTFC by roughly half
(measured 2801ms → 1218ms on `claude-sonnet-4-6`). The agent owns step
2+ (tool execution, persistence, hooks) so heavy deps stay where they
belong. Web Fetch handler works natively in Next.js, Hono, SvelteKit,
Remix, Workers, etc.; bridge to Express/Fastify/Koa via
`chat.toNodeListener`. New `@trigger.dev/sdk/chat-server` subpath.
- **Multi-turn durability via Sessions** — every chat is backed by a
durable Session that outlives any individual run. Conversations resume
across page refreshes, idle timeout, crashes, and deploys; `resume:
true` reconnects via `lastEventId` so clients only see new chunks.
`sessions.list` enumerates chats for inbox-style UIs.
- **Auto-accumulated history, delta-only wire** — the backend
accumulates the full conversation across turns; clients only ship the
new message each turn. Long chats never hit the 512 KiB body cap.
Register `hydrateMessages` to be the source of truth yourself.
- **Lifecycle hooks** — `onPreload`, `onChatStart`,
`onValidateMessages`, `hydrateMessages`, `onTurnStart`,
`onBeforeTurnComplete`, `onTurnComplete`, `onChatSuspend`,
`onChatResume` — for persistence, validation, and post-turn work.
- **Stop generation** — client-driven `transport.stopGeneration(chatId)`
aborts mid-stream; the run stays alive for the next message, partial
response is captured, and aborted parts (stuck `partial-call` tools,
in-progress reasoning) are auto-cleaned.
- **Tool approvals (HITL)** — tools with `needsApproval: true` pause
until the user approves or denies via `addToolApprovalResponse`. The
runtime reconciles the updated assistant message by ID and continues
`streamText`.
- **Steering and background injection** — `pendingMessages` injects user
messages between tool-call steps so users can steer the agent
mid-execution; `chat.inject()` + `chat.defer()` adds context from
background work (self-review, RAG, safety checks) between turns.
- **Actions** — non-turn frontend commands (undo, rollback, regenerate,
edit) sent via `transport.sendAction`. Fire `hydrateMessages` +
`onAction` only — no turn hooks, no `run()`. `onAction` can return a
`StreamTextResult` for a model response, or `void` for side-effect-only.
- **Typed state primitives** — `chat.local<T>` for per-run state
accessible from hooks, `run()`, tools, and subtasks (auto-serialized
through `ai.toolExecute`); `chat.store` for typed shared data between
agent and client; `chat.history` for reading and mutating the message
chain; `clientDataSchema` for typed `clientData` in every hook.
- **`chat.toStreamTextOptions()`** — one spread into `streamText` wires
up versioned system [Prompts](https://trigger.dev/docs/ai/prompts),
model resolution, telemetry metadata, compaction, steering, and
background injection.
- **Multi-tab coordination** — `multiTab: true` + `useMultiTabChat`
prevents duplicate sends and syncs state across browser tabs via
`BroadcastChannel`. Non-active tabs go read-only with live updates.
- **Network resilience** — built-in indefinite retry with bounded
backoff, reconnect on `online` / tab refocus / bfcache restore,
`Last-Event-ID` mid-stream resume. No app code needed.

See [/docs/ai-chat](https://trigger.dev/docs/ai-chat/overview) for the
full surface — quick start, three backend approaches (`chat.agent`,
`chat.createSession`, raw task), persistence and code-sandbox patterns,
type-level guides, and API reference.

- Stamp `gen_ai.conversation.id` (the chat id) on every span and metric
emitted from inside a `chat.task` or `chat.agent` run. Lets you filter
dashboard spans, runs, and metrics by the chat conversation that
produced them — independent of the run boundary, so multi-run chats
correlate cleanly. No code changes required on the user side.
([#3543](https://github.com/triggerdotdev/trigger.dev/pull/3543))

- Fix `LocalsKey<T>` type incompatibility across dual-package builds.
The phantom value-type brand no longer uses a module-level `unique
symbol`, so a single TypeScript compilation that resolves the type from
both the ESM and CJS outputs (which can happen under certain pnpm
hoisting layouts) no longer sees two structurally-incompatible variants
of the same type.
([#3626](https://github.com/triggerdotdev/trigger.dev/pull/3626))

- Unit-test `chat.agent` definitions offline with `mockChatAgent` from
`@trigger.dev/sdk/ai/test`. Drives a real agent's turn loop in-process —
no network, no task runtime — so you can send messages, actions, and
stop signals via driver methods, inspect captured output chunks, and
verify hooks fire. Pairs with `MockLanguageModelV3` from `ai/test` for
model mocking. `setupLocals` lets you pre-seed `locals` (DB clients,
service stubs) before `run()` starts.
([#3543](https://github.com/triggerdotdev/trigger.dev/pull/3543))

The broader `runInMockTaskContext` harness it's built on lives at
`@trigger.dev/core/v3/test` — useful for unit-testing any task code, not
just chat.

- Retry `TASK_PROCESS_SIGSEGV` task crashes under the user's retry
policy instead of failing the run on the first segfault. SIGSEGV in Node
tasks is frequently non-deterministic (native addon races, JIT/GC
interaction, near-OOM in native code, host issues), so retrying on a
fresh process often succeeds. The retry is gated by the task's existing
`retry` config + `maxAttempts` — same path `TASK_PROCESS_SIGTERM` and
uncaught exceptions already use — so tasks without a retry policy still
fail fast.
([#3552](https://github.com/triggerdotdev/trigger.dev/pull/3552))

- Add `region` to the runs list / retrieve API: filter runs by region
(`runs.list({ region: "..." })` / `filter[region]=<masterQueue>`) and
read each run's executing region from the new `region` field on the
response.
([#3612](https://github.com/triggerdotdev/trigger.dev/pull/3612))

- **Sessions** — a durable, run-aware stream channel keyed on a stable
`externalId`. A Session is the unit of state that owns a multi-run
conversation: messages flow through `.in`, responses through `.out`,
both survive run boundaries. Sessions back the new `chat.agent` runtime,
and you can build on them directly for any pattern that needs durable
bi-directional streaming across runs.
([#3542](https://github.com/triggerdotdev/trigger.dev/pull/3542))

    ```ts
    import { sessions, tasks } from "@trigger.dev/sdk";

    // Trigger a task and subscribe to its session output in one call
const { runId, stream } = await tasks.triggerAndSubscribe("my-task",
payload, {
      externalId: "user-456",
    });

    for await (const chunk of stream) {
      // ...
    }

// Enumerate existing sessions (powers inbox-style UIs without a
separate index)
for await (const s of sessions.list({ type: "chat.agent", tag:
"user:user-456" })) {
      console.log(s.id, s.externalId, s.createdAt, s.closedAt);
    }
    ```

See [/docs/ai-chat/overview](https://trigger.dev/docs/ai-chat/overview)
for the full surface — Sessions powers the durable, resumable chat
runtime described there.

## @trigger.dev/plugins@4.5.0-rc.0

### Patch Changes

- The public interfaces for a plugin system. Initially consolidated
authentication and authorization interfaces.
([#3499](https://github.com/triggerdotdev/trigger.dev/pull/3499))
-   Updated dependencies:
    -   `@trigger.dev/core@4.5.0-rc.0`

## @trigger.dev/python@4.5.0-rc.0

### Patch Changes

-   Updated dependencies:
    -   `@trigger.dev/sdk@4.5.0-rc.0`
    -   `@trigger.dev/core@4.5.0-rc.0`
    -   `@trigger.dev/build@4.5.0-rc.0`

## @trigger.dev/react-hooks@4.5.0-rc.0

### Patch Changes

-   Updated dependencies:
    -   `@trigger.dev/core@4.5.0-rc.0`

## @trigger.dev/redis-worker@4.5.0-rc.0

### Patch Changes

- Add MollifierBuffer and MollifierDrainer primitives for trigger burst
smoothing.
([#3614](https://github.com/triggerdotdev/trigger.dev/pull/3614))

MollifierBuffer (`accept`, `pop`, `ack`, `requeue`, `fail`,
`evaluateTrip`) is a per-env FIFO over Redis with atomic Lua transitions
for status tracking. `evaluateTrip` is a sliding-window trip evaluator
the webapp gate uses to detect per-env trigger bursts.

MollifierDrainer pops entries through a polling loop with a
user-supplied handler. The loop survives transient Redis errors via
capped exponential backoff (up to 5s), and per-env pop failures don't
poison the rest of the batch — one env's blip is logged and counted as
failed for that tick. Rotation is two-level: orgs at the top, envs
within each org. The buffer maintains `mollifier:orgs` and
`mollifier:org-envs:${orgId}` atomically with per-env queues, so the
drainer walks orgs → envs directly without an in-memory cache. The
`maxOrgsPerTick` option (default 500) caps how many orgs are scheduled
per tick; for each picked org, one env is popped (rotating round-robin
within the org). An org with N envs gets the same per-tick scheduling
slot as an org with 1 env, so tenant-level drainage throughput is
determined by org count rather than env count.

-   Updated dependencies:
    -   `@trigger.dev/core@4.5.0-rc.0`

## @trigger.dev/rsc@4.5.0-rc.0

### Patch Changes

-   Updated dependencies:
    -   `@trigger.dev/core@4.5.0-rc.0`

## @trigger.dev/schema-to-json@4.5.0-rc.0

### Patch Changes

-   Updated dependencies:
    -   `@trigger.dev/core@4.5.0-rc.0`

</details>

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2026-05-21 14:39:02 +01:00
Eric Allam 9ff410bfa4 feat(sdk): type chat.createStartSessionAction against your chat agent (#3684)
## Summary

Type `chat.createStartSessionAction` against the chat agent so
`clientData` is typed end-to-end on the first turn. Closes the gap where
`useTriggerChatTransport`'s `startSession` callback already hands you a
typed `clientData` (via the transport generic) but the server-side
action couldn't accept it without untyped routing through the `metadata`
field.

## Design

`ChatStartSessionParams` gains a typed `clientData` field via the new
generic:

```ts
export type ChatStartSessionParams<TChat extends AnyTask = AnyTask> = {
  chatId: string;
  clientData?: InferChatClientData<TChat>;
  triggerConfig?: Partial<SessionTriggerConfig>;
  metadata?: Record<string, unknown>;
};

function createChatStartSessionAction<TChat extends AnyTask = AnyTask>(
  taskId: string,
  options?: CreateChatStartSessionActionOptions
): (params: ChatStartSessionParams<TChat>) => Promise<ChatStartSessionResult>
```

When provided, `clientData` is folded into the first run's
`triggerConfig.basePayload.metadata`, so `onPreload` / `onChatStart` see
the same shape per-turn `metadata` carries via the transport. The opaque
session-level `metadata` field stays exactly as before — it lands on the
Session row, not the run payload.

## Usage

```ts
// actions.ts
import { chat } from "@trigger.dev/sdk/ai";
import type { myChat } from "@/trigger/chat";

export const startChatSession = chat.createStartSessionAction<typeof myChat>("my-chat");
```

```tsx
// Chat.tsx
const transport = useTriggerChatTransport<typeof myChat>({
  task: "my-chat",
  accessToken: ({ chatId }) => mintChatAccessToken(chatId),
  startSession: ({ chatId, clientData }) =>
    startChatSession({ chatId, clientData }),
});
```

## Test plan

- [x] `pnpm run build --filter @trigger.dev/sdk` passes
- [ ] Verify a `chat.agent` with `clientDataSchema` reads the typed
clientData from `onPreload` payload metadata on the first turn
2026-05-21 11:51:35 +01:00
Eric Allam d343727021 fix(webapp,sdk): keep chat.agent snapshots on one object store (#3679)
(`OBJECT_STORE_BASE_URL`) and a named protocol provider
(`OBJECT_STORE_DEFAULT_PROTOCOL=s3`), chat.agent session snapshot writes
landed in the named provider but reads fell through to the default — so
the recovery boot couldn't find the snapshot it had just written.

After a mid-stream cancel, the missing snapshot triggered a fallback
replay path that dropped the user's follow-up message, leaving the chat
stuck in `submitted` indefinitely.

Fix:
- New `/api/v1/sessions/:id/snapshot-url` route handles PUT + GET
  symmetrically — both prefix unprefixed keys with
  `OBJECT_STORE_DEFAULT_PROTOCOL` so they always round-trip through the
  same store.
- `Session.chatSnapshotStoragePath` persists the resolved URI on first
  write so future protocol changes don't strand existing snapshots.
  Reads prefer the stored URI and fall back to the computed default for
  pre-column sessions.
- SDK calls `createChatSnapshotUploadUrl` / `getChatSnapshotUrl`; the
  generic v1/v2 packets endpoints are unchanged.

## Test plan
- [x] Configure local with two providers (R2 default + MinIO `s3` named)
      and `OBJECT_STORE_DEFAULT_PROTOCOL=s3`.
- [x] Reproduce hang: send a message, cancel mid-stream, send another —
      without the fix it hangs in `submitted`; with the fix it streams.
- [x] Snapshot lands in the `s3`-protocol bucket and
      `Session.chatSnapshotStoragePath` is set after first write.
- [x] SDK unit tests pass; webapp typecheck passes.
2026-05-20 20:22:59 +01:00
Eric Allam f91b96efb7 feat(sdk,core): preserve chat.agent context after cancel / OOM / crash (#3671) 2026-05-20 07:27:05 +01:00
Eric Allam 12d2125148 fix(sdk,core,build): SDK hardening pass (#3670)
## Summary

Five hardening fixes across `@trigger.dev/sdk`, `@trigger.dev/core`, and
`@trigger.dev/build`.

- `tasks.triggerAndSubscribe` now forwards caller `requestOptions`
(custom API keys, per-request overrides) to the underlying
`apiClient.triggerTask` call instead of silently dropping them.
- `SSEStreamSubscription` no longer retries permanent client errors
forever. The default `nonRetryableStatuses` widens from `[404, 410]` to
`[400, 404, 409, 410, 422]`, so a malformed session-stream request fails
fast instead of busy-looping under bounded backoff.
- Session writer falls back to manually wiring the caller's
`AbortSignal` on Node 18, where `AbortSignal.any` is unavailable.
Caller-driven cancellation now propagates on every supported runtime.
- `TriggerChatTransport` throws immediately when a `chat.handover`
response is missing `X-Trigger-Chat-Access-Token`, instead of silently
downgrading every subsequent turn back to the handover path. `dispose()`
aborts every active `session.out` subscription before tearing the
coordinator down, so unmount/navigation no longer leaves SSE readers in
flight.
- Removed the experimental `@trigger.dev/build/extensions/secureExec`
build extension. It will return alongside the sandbox feature it was
built to support.

## Test plan

- [ ] `pnpm run build --filter @trigger.dev/sdk --filter
@trigger.dev/core --filter @trigger.dev/build`
- [ ] `pnpm --filter @trigger.dev/sdk test --run` (183 tests, including
chat / chat-server / sessions / handover)
- [ ] `pnpm --filter @trigger.dev/core test --run`
- [ ] Manually trigger a `chat.handover` whose response strips
`X-Trigger-Chat-Access-Token`, and confirm the transport throws
synchronously rather than degrading.
- [ ] Unmount a chat UI mid-stream and confirm the active `session.out`
SSE connection closes immediately.
2026-05-19 16:10:32 +01:00
Eric Allam 427d9e078a feat(sdk): functional baseURL and fetch override on chat transports (#3655)
## Summary

`TriggerChatTransport`, `AgentChat`, and `chat.createStartSessionAction`
now accept a string-or-function `baseURL` so callers can route per
endpoint — e.g. `.in/append` through a trusted edge proxy while keeping
`.out` SSE direct. The same surfaces add a `fetch` override for header
injection, custom retries, or proxy rewrites that go beyond URL routing.
SSE GETs are covered too via a new `fetchClient` option on
`SSEStreamSubscription`.

```ts
// TriggerChatTransport / AgentChat — endpoints: "in" | "out"
baseURL: ({ endpoint }) =>
  endpoint === "out" ? DIRECT : PROXY,

fetch: (url, init, ctx) => {
  init.headers = new Headers(init.headers);
  init.headers.set("traceparent", currentTraceparent());
  return globalThis.fetch(url, init);
},

// chat.createStartSessionAction — endpoints: "sessions" | "auth"
chat.createStartSessionAction("my-agent", {
  baseURL: ({ endpoint }) => (endpoint === "sessions" ? PROXY : DIRECT),
});
```

`streamBaseURL` on `TriggerChatTransport` is kept as a backwards-compat
alias and continues to win for the `"out"` endpoint when set.
Plain-string `baseURL` still applies to every endpoint, matching prior
behavior.
2026-05-18 16:42:18 +01:00
Eric Allam 8b98e21b4b fix(sdk,core): cache realtime-stream credentials per slot with refresh on writer failure (#3658)
## Summary

Hot-loop writers — `streams.writer` / `streams.pipe` on the run-scoped
side, `chat.response.write` / `chat.stream.*` on the session side — were
issuing a fresh `PUT` to mint S2 credentials for every chunk. On run
streams, each PUT also pushed the streamId onto
`TaskRun.realtimeStreams`,
so a chat-agent turn writing N chunks produced N PUTs and N duplicate
array pushes against the same row.

The SDK now caches the initialize response per cache slot: `(runId,
key)`
for run streams, the session id for session streams. First call PUTs as
before; subsequent calls reuse the cached promise. Hot-loop writers do
one PUT per slot for the lifetime of the cache.

S2 access tokens have a 1-day TTL. If a writer's `wait()` rejects (auth
error, expired token, network blip), the cache evicts the matching slot
so the next call re-PUTs and mints fresh credentials, identity-checked
so a concurrent caller's fresh promise isn't accidentally cleared.

## chat.agent guardrail

`streams.pipe / writer / append / read` called inside a `chat.agent` run
now logs a one-time warning pointing at `chat.response.write` /
`chat.stream.*` — `streams.*` is run-scoped and isn't visible on the
chat session. The ai-chat docs are updated to drop the old guidance
toward run-scoped streams.
2026-05-18 16:41:58 +01:00
Daniel Sutton 906d5fafb6 feat(mollifier): trigger burst smoothing — Phase 1 (monitoring) (#3614)
## Summary
- Introduce the Mollifier: a Redis-backed buffer for `trigger()` API
calls during traffic spikes, with a per-env trip evaluator and a drainer
ack-loop.
- Phase 1 is dual-write monitoring — every mollified trigger is buffered
to Redis AND continues to `engine.trigger`. No customer-facing behaviour
change.
- Telemetry events: `mollifier.would_mollify`, `mollifier.buffered`,
`mollifier.drained`, plus the `mollifier.decisions` counter.
  - Gated behind a feature flag (default off).

  ## Test plan
  - [x] `pnpm run test --filter @trigger.dev/redis-worker`
  - [x] `pnpm run test --filter webapp -- mollifier`
  - [x] Manual: with flag off, no behaviour change vs main
- [x] Manual: with flag on + threshold lowered, observe
`mollifier.buffered` + `mollifier.drained` log pairs with matching
`runId`

---------

Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-18 14:58:38 +01:00
Eric Allam 82853debea feat(webapp,core,sdk,cli): bound session.out via per-turn trim (#3644)
## Summary

Long-running chat agents were filling `session.out` forever — every
`chat.agent` turn appended to the same S2 stream with no trim, and the
Sessions dashboard re-streamed the entire history from `seq_num=0` on
every page load. After this change the agent appends an S2 `trim`
command record after each `trigger:turn-complete`, pointing back at the
previous turn-complete's seq_num. `session.out` stays roughly one turn
long at steady state, regardless of session age.

`trigger:turn-complete` and `trigger:upgrade-required` move from
`chunk.type`-shaped data records into header-form control records under
a uniform `trigger-control` namespace. Built-in transports
(`TriggerChatTransport`, `AgentChat`, the dashboard's `AgentView`)
handle the new shape transparently. Custom transports need a one-line
filter on the `trigger-control` header — see the rewritten "Records on
session.out" section in the client-protocol docs.

The Sessions detail page in the dashboard fetches the agent's per-turn
S3 snapshot via a presigned URL and seeds the transcript view, then
SSE-tails from the snapshot's `lastOutEventId`. Bandwidth and
time-to-first-render scale with unread turns instead of session
lifetime.

Resume contract is now explicit: single-turn-boundary resume always
works (the prior turn-complete is still on the stream), the S2 trim is
eventually consistent over 10-60s, and multi-turn-away resume falls back
to a snapshot reload.
2026-05-18 10:21:56 +01:00
Eric Allam 55fa2d4967 fix(cli): TRIGGER_BUILD_SKIP_REWRITE_TIMESTAMP escape hatch for local self-hosted builds (#3618)
## Summary

Local self-hosted deploys (`trigger deploy --local-build --push
--builder orbstack` or any other buildx setup using the **docker**
driver) fail at the push step with:

```
ERROR: failed to build: failed to solve:
  exporter option "rewrite-timestamp" conflicts with "unpack"
```

The docker driver auto-enables `unpack=true` when pushing, and that's
incompatible with `rewrite-timestamp` (which the CLI sets for
reproducible-build hashing).

Adds a simple env-var opt-out so contributors can keep using their
default builder. The flag is only read by the local-build code path;
remote/cloud builds are unaffected.

```bash
TRIGGER_BUILD_SKIP_REWRITE_TIMESTAMP=1 \
  pnpm exec trigger deploy --profile default --local-build --push --builder orbstack
```

The trade-off: skipping `rewrite-timestamp` means layer timestamps
reflect actual build time, so two identical builds produce different
layer hashes. Fine for a local-dev registry; the only real consumer of
timestamp-stability is registry-layer cache hit rates.

## Test plan

- [x] Manual: ran `trigger deploy --profile default --local-build --push
--builder orbstack` against the localhost webapp + a local Docker
registry on port 5001 — first failed with the rewrite-timestamp/unpack
error, then succeeded after setting
`TRIGGER_BUILD_SKIP_REWRITE_TIMESTAMP=1`.
- [x] Full chat.agent smoke sweep (15 tests, including suspend/resume,
deepResearch subtask, AgentChat orchestrator) against the deployed image
— all pass.

🤖 Generated with [Claude Code](https://claude.com/claude-code)
2026-05-17 09:28:57 +01:00
Eric Allam 627e059298 feat(trigger-sdk): add streamBaseURL to TriggerChatTransport (#3641)
`TriggerChatTransport` had a single `baseURL` option covering both the
`.in/append` POSTs and the long-lived `.out` SSE subscription. Customers
wanting to route the SSE through a proxy (e.g. a Cloudflare worker
capturing JA4 fingerprints for bot detection) had to send every append
through the proxy too, adding a hop to every user message.

New optional `streamBaseURL` overrides the SSE base URL only; appends
keep using `baseURL`. Falls back to `baseURL` when unset, so existing
transports are unchanged.

```ts
const transport = new TriggerChatTransport({
  task: "ai-chat",
  baseURL: "https://api.trigger.dev",
  streamBaseURL: "https://chat-proxy.example.com",
  accessToken,
  startSession,
});
```

Verified with a new test in `chat.test.ts` that asserts `.in/append`
routes through `baseURL` and `.out` SSE routes through `streamBaseURL`.
All existing tests still pass.
2026-05-17 07:55:12 +03:00
Eric Allam dfa3ede209 feat(ci): support release candidates via changesets pre mode (#3628)
## Summary

Enables shipping `X.Y.Z-rc.N` prereleases of `@trigger.dev/*` via
changesets pre mode. RCs publish under the `rc` npm dist-tag, never
claim `latest`, and don't trigger marketing-site changelog PRs. The
plumbing is hyphen-in-version detection in `release.yml` — no separate
workflow, no opt-in flag at publish time.

Validated end-to-end against a sandbox repo (real npm publishes, Docker
builds, Helm chart pushes, GitHub releases) before porting back. Full RC
lifecycle tested: pre enter → rc.0 → iterate to rc.1 → pre exit →
stable. Plus interaction with the existing release-branch hotfix flow.

## What changes

### `release.yml`
- New `is_prerelease` output (hyphen-in-version)
- GitHub release adds `--prerelease` flag for RC publishes (Pre-release
badge, not Latest)
- `dispatch-changelog` job gated on `is_prerelease != 'true'` — no
marketing-site PR per RC

### Docker workflows
- Removes the `:v4-beta` floating tag entirely from `publish-webapp.yml`
and `publish-worker-v4.yml`. v4 is GA; the tag is a misnomer and is
already inconsistent with the npm side (npm `v4-beta` dist-tag was
frozen at 4.0.4 months ago while Docker `:v4-beta` kept bumping).
Self-hosters should pin to a versioned tag going forward — the last
value of `:v4-beta` stays frozen wherever it currently points.

### CLI version-check fix
(`packages/cli-v3/src/utilities/initialBanner.ts`)
Switches the "new version available" comparison from JavaScript
`localeCompare` to `semver.lt`. The old comparison handled `X.Y.Z-rc.N`
vs `X.Y.Z` incorrectly — a user on `4.5.0-rc.0` would never be prompted
to upgrade once `4.5.0` stable shipped (lex order put the prerelease
ahead of the bare version). Real semver gets this right.

Stable users were never affected: the check queries the `@latest`
dist-tag, which by convention never points at a prerelease.

## How an RC actually publishes after this

1. `pnpm exec changeset pre enter rc` on main, push the `pre.json`
2. Bot regenerates the release PR as `chore: release v<X.Y.Z>-rc.0`
3. Merge → `release.yml` runs `changeset publish` which reads
`pre.json.tag` and publishes under `--tag rc`. GitHub release marked
Pre-release. No marketing-site dispatch.
4. Iterate by adding changesets normally; bot bumps to `rc.1`, `rc.2`, …
5. When ready: `pnpm exec changeset pre exit`, push, merge regenerated
PR → stable ships under `latest` and the marketing-site dispatch fires.
2026-05-15 16:43:26 +01:00