re2-test-supervisor-backpressure
1844 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 nicktrn
|
8190215de2 |
feat(core): freeze consumer-pool scale-up under backpressure
Add optional shouldPauseScaling to ScalingOptions; when it returns true the pool stops scaling up (scale-down still allowed), so it won't add consumers to drain a queue backpressure is deliberately holding. |
||
|
Katia Bulatova
|
cae3dcb7dd |
Env vars page performance fix (#3829)
## Summary This PR improves performance across the Environment Variables page. ## Changes ### Targeted value loading - load only the non-secret (environmentId, key) pairs required by the page. Secret values continue to be redacted in the UI. ### SSR windowing + virtualization - SSR-render only the first 50 rows - hydrate those rows - virtualize the remaining dataset client-side - search is now URL-driven during SSR, ensuring deep links such as `?search=DATABASE_URL` ### Lightweight 'Create' flow - 'Create' page no longer loads the full Environment Variables dataset. ## Results Large projects no longer render thousands of rows during SSR. Example (~11k rendered rows): Metric | Before | After -- | -- | -- Document size | ~150 MB | ~5 MB SSR rows | ~11k | 50 Browser DOM rows | Thousands | ~26–38 |
||
|
Daniel Sutton
|
4ea3ef138f |
chore(webapp,redis-worker): make mollifier constants configurable (#3822)
## Summary The mollifier had ~21 behavioural constants baked in as hardcoded values — the buffer's ack-grace TTL and Redis retry/reconnect tuning, the drainer's poll interval and backoff envelope, the pre-gate idempotency claim TTL/wait/poll, the buffered-run mutate-with-fallback wait loop, the metadata CAS retry budget and backoff, the stale-sweep scan bounds, and the draining-gauge interval. None could be adjusted without a code change, which makes tuning the system under production load impossible. This exposes all of them as `TRIGGER_MOLLIFIER_*` environment variables, each defaulting to its previous hardcoded value. Behaviour is identical unless an operator sets a var, so it's a safe no-op deploy. ## Design The package-level classes (`MollifierBuffer`, `MollifierDrainer` in `@trigger.dev/redis-worker`) gain optional constructor options defaulting to the old constants — backward compatible, hence a patch changeset. The webapp factories and worker bootstraps read the env and pass them through. The route- and concern-level pure helpers (mutate-with-fallback, metadata mutation, idempotency claim, stale-sweep state) keep their existing `?? DEFAULT` option fallbacks and are fed env values at their call sites, so they stay unit-testable without importing `env.server`. ## Test plan - [x] `@trigger.dev/redis-worker` builds - [x] webapp typecheck passes - [x] mollifier buffer + drainer testcontainer suites pass (modulo a couple of pre-existing flaky timing tests) - [x] Reviewer: confirm the `TRIGGER_MOLLIFIER_*` env var names match ops conventions --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com> |
||
|
Katia Bulatova
|
bb7d7dc7d1 |
feat(sdk,core): offload large trigger payloads via object storage (#3785)
Adds backward-compatible support for large trigger payloads by reusing the existing object-storage packet flow. Large payloads are uploaded to object storage before the trigger request is sent. The trigger API receives a small application/store pointer payload instead of embedding large JSON bodies in the request. Small payload behavior is unchanged. |
||
|
Eric Allam
|
9818ad5240 |
fix(sdk): recover chat transport when a restored session no longer exists (#3816)
## Summary When a chat's restored session state points at a session that no longer exists in the current environment — for example a `sessions` entry that was persisted against a different trigger environment — `useTriggerChatTransport` assumed the session was live and never created a real one. The next message then failed with a 404 and the chat couldn't send. ## Fix `callWithAuthRetry` now treats a 404 from a session-PAT-authed call as "this session doesn't exist here". After the existing 401/403 token refresh, a 404 recreates the session via `startSession`, drops the stale `lastEventId` resume cursor (it pointed at another environment's stream), and retries the send once. When `startSession` isn't configured the transport throws a clear message instead of a bare 404. |
||
|
nicktrn
|
a9f756b260 |
chore: move reference projects to their own repo (#3812)
The reference/example projects (`references/`) now live in their own repo, https://github.com/triggerdotdev/references, so their heavy, frequently-changing dependencies are no longer part of this repo's lockfile and tooling. This removes them here and repoints everything that referenced them. - Deletes `references/`; updates the pnpm workspace + lockfile. - Clears the references-only CI rules and `.vscode` configs. - Repoints the docs (contributor/agent + one public page) to the new repo. - `seed.mts` keeps the local-dev projects (hello-world, d3-chat, realtime-streams). |
||
|
nicktrn
|
a4d8c9f65f |
chore(deps): update OpenTelemetry suite to 0.218.0 / 2.7.1 (#3810)
Brings the OpenTelemetry packages up to the latest coherent release
across the webapp and the published packages (`@trigger.dev/core`, the
CLI, `@trigger.dev/sdk`) plus
`internal-packages/{tracing,testcontainers}`:
- `@opentelemetry/sdk-node` 0.218.0
- `@opentelemetry/core` 2.7.1
- `@opentelemetry/host-metrics` 0.38.3
We were already on the otel 2.x line, so this is a same-major minor move
- the versions are pinned to `@opentelemetry/sdk-node@0.218.0`'s own
declared dependency set so the experimental (0.2xx) and stable (2.x)
packages stay coherent (mixing them is the usual otel breakage).
**One code change:** otel 0.215 made `forceFlush()` a required method on
`LogRecordExporter`, so `ExternalLogRecordExporterWrapper` (core's
tracing SDK) gains a `forceFlush()` that delegates to the underlying
exporter.
**Notable upgrades along the way:** OTLP exporters can take a custom
HTTP agent (connection pooling/keepAlive on the export path), HTTP
request headers are captured at span creation, and core hot-path perf
improvements in 2.6.1/2.7. `host-metrics` 0.37→0.38 is a clean upgrade.
Patch changeset added for the three published packages. References
projects are intentionally untouched.
Verified: `@trigger.dev/core` / CLI / `@trigger.dev/sdk` build, webapp +
`@internal/tracing` typecheck - all green.
|
||
|
Eric Allam
|
4b78d7e84e |
feat(core,webapp): support isSecret on environment variable imports (#3809)
## Summary
The environment variables import API now accepts an optional `isSecret`
flag, so imported variables can be created as secret (redacted)
environment variables instead of plaintext. When the flag is omitted,
variables default to non-secret, preserving existing behavior for CLI
deploys and dashboard imports.
This is useful for tools that push secrets into Trigger.dev (for
example, syncing from a secrets manager) and want them stored as secrets
rather than plain environment variables.
It's available through `envvars.import` in the SDK and the `POST
/api/v1/projects/{projectRef}/envvars/{slug}/import` endpoint, and is
honored for both regular and preview-branch environments.
```ts
await envvars.import("proj_1234", "prod", {
variables: { STRIPE_SECRET_KEY: "sk_live_..." },
isSecret: true,
});
```
|
||
|
Daniel Sutton
|
139eede41c |
feat(redis-worker): batched pop in MollifierDrainer for fast single-env drains (#3797)
## Summary Adds `drainBatchSize` to `MollifierDrainer` (default `1` — preserves existing behaviour) and wires `TRIGGER_MOLLIFIER_DRAIN_BATCH_SIZE` through the webapp (default `50`). Each tick the drainer now pops up to `drainBatchSize` from each chosen env, then dispatches every popped entry through the shared `concurrency`-bounded `pLimit`. Per-org/per-env fairness is unchanged — only the in-env pop count grows. Pre-existing behaviour was one pop per env per tick. For a single-env burst that single-flighted the drain at the per-tick floor of `pop + engine.trigger ≈ 50–60 ms`. With buffer entries piling up under a real-world tenant burst that's tens of minutes of tail latency to fully materialise — even though PG itself could comfortably sustain the writes. ## Why this matters — heavy-tail illustration Scenario: 100 customers in one window — 94 fire 20 triggers each, 5 fire 100, 1 fires 1000. Gate at `THRESHOLD=10/s`, `HOLD_MS=500`. First 10 of each burst hit PG directly; the rest buffer. | Customers | Triggers each | PG direct | Buffered each | Total buffered | |---|---|---|---|---| | 94 small | 20 | 10 | 10 | 940 | | 5 medium | 100 | 10 | 90 | 450 | | 1 heavy | 1000 | 10 | 990 | 990 | **With `DRAIN_BATCH_SIZE=50`, `DRAIN_CONCURRENCY=50`, ~50 ms `engine.trigger`:** | Tick | Pops | Dispatch waves | Wall-clock | |---|---|---|---| | 1 | 94×10 + 5×50 + 1×50 = 1 240 | 25 × 50 ms | ~1 300 ms (94 smalls done) | | 2 | 5×40 + 1×50 = 250 | 5 × 50 ms | ~300 ms (5 mediums done) | | 3–20 | heavy alone, 50/tick | 1 × 50 ms | ~100 ms each | | Customer class | Buffered fully drained | |---|---| | 94 small | **~1.3 s** | | 5 medium | **~1.6 s** | | 1 heavy | **~3.4 s** | **Without batching (one pop per env per tick — current behaviour):** | Customer class | Buffered fully drained | |---|---| | 94 small | ~500 ms | | 5 medium | ~4.5 s | | 1 heavy | **~49 s** | So the heavy single-tenant tail drops from ~49 s to ~3.4 s (~14× faster) without changing PG load characteristics. Smalls go up slightly in this scenario (500 ms → 1.3 s) because all 100 envs share one tick's dispatch queue — that's the trade we accept for the heavy tail; the worst-case small wait is still inside one tick. PG load is identical either way (50 concurrent inserts at a time, capped by `DRAIN_CONCURRENCY`). ## What changed **`packages/redis-worker`** - New `drainBatchSize` option (default 1 — full backward compat). - `runOnce()` refactored to pop per-env batches in parallel, then dispatch all popped entries through the existing global `pLimit`. Mid-batch pop failure aborts only that env's batch and counts as one failure (same semantic as the old per-env path). - Removed the now-unused `processOneFromEnv` helper. **`apps/webapp`** - `TRIGGER_MOLLIFIER_DRAIN_BATCH_SIZE` env var (default 50, matching `DRAIN_CONCURRENCY`). - Wired into `mollifierDrainer.server.ts`. **Test cloud config** (separate cloud PR): `TRIGGER_MOLLIFIER_DRAIN_BATCH_SIZE="50"` on the worker service. Production rollout deferred until we've watched it on test cloud. ## Test plan - [x] All 25 stub-based drainer tests pass (18 pre-existing + 7 new). 7 new tests under `MollifierDrainer.drainBatchSize`: - pops up to `drainBatchSize` across ticks - global `concurrency` cap still holds when batch > concurrency - mid-batch pop failure isolation - multi-env batch fan-out in one tick - **hierarchical org fairness preserved at `drainBatchSize > 1`** (load-bearing — guards against future regressions to per-env-instead-of-per-org rotation) - mixed success/failure accounting in a batched tick - bounded pops on empty queue (no Lua spam past `drainBatchSize`) - [x] All pre-existing tests still pass unchanged at default `drainBatchSize=1` → backward-compat locked. - [x] `pnpm run build --filter @trigger.dev/redis-worker` clean. - [x] `pnpm run typecheck --filter webapp` clean. - [x] `redisTest` block (real Redis via testcontainers) — couldn't run locally on this branch due to testcontainers runtime discovery; will validate in CI. - [ ] Test-cloud smoke after cloud PR lands: fire `burst 50` against a flagged env and confirm the 50th entry's drain time drops from ~2.5 s to <200 ms. ## Notes - Per-tick memory bound: `maxOrgsPerTick × drainBatchSize` entries can sit in the JS pLimit queue between pop and dispatch. At defaults that's `500 × 50 = 25 000` × ~5 KB snapshot ≈ ~125 MB worst case per worker — well within headroom. - The pre-batch model's strict per-env throughput cap of `1/tick` is documented as the fairness baseline elsewhere. Org-level fairness is what callers actually rely on; this change does not weaken that. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
Dan
|
cd252801eb |
feat: dashboard agent - package upgrades (#3793)
1. in webapp folder update ai-sdk to 6.x.x 2. update vitest to 4.xx |
||
|
github-actions[bot]
|
e0681d2394 |
chore: release v4.5.0-rc.4 (#3788)
🚀 Publish Trigger.dev Docker / units (push) Failing after 3s
🚀 Publish Trigger.dev Docker / typecheck (push) Failing after 19s
🚀 Publish Trigger.dev Docker / publish-webapp (push) Has been skipped
🚀 Publish Trigger.dev Docker / publish-worker (push) Has been skipped
🚀 Publish Trigger.dev Docker / publish-worker-v4 (push) Has been skipped
🧭 Helm Chart Release / lint-and-test (push) Has been cancelled
🧭 Helm Chart Release / release (push) Has been cancelled
## Summary 1 new feature, 5 improvements. ## Highlights - Mollifier `mutateSnapshot` now enforces a tag cap: an `append_tags` patch carrying `maxTags` returns `"limit_exceeded"` (writing nothing) when the deduped tag count would exceed the limit, so a buffered run can't accumulate more tags via the tags API than the trigger validator allows at creation. ([#3756](https://github.com/triggerdotdev/trigger.dev/pull/3756)) ## Improvements - Mollifier buffer extensions: idempotency dedup, an atomic `mutateSnapshot` API, metadata CAS, claim primitives, and a `MollifierSnapshot` type. The buffer's Redis client now reconnects with jittered backoff so a fleet of clients doesn't stampede Redis in lockstep after a blip. ([#3752](https://github.com/triggerdotdev/trigger.dev/pull/3752)) - Add `onTerminalFailure` callback to `MollifierDrainerOptions` so the customer's run lands a SYSTEM_FAILURE PG row even when the drainer exhausts `maxAttempts` on a retryable PG error. Previously, retryable-error exhaustion called `buffer.fail()` directly, which atomically marks FAILED + DELs the entry hash with no PG write — silent data loss when PG was unreachable across the full retry budget. The callback fires before `buffer.fail()` on any terminal path (`cause: "non-retryable"` or `"max-attempts-exhausted"`); throwing a retryable error from the callback causes the drainer to requeue rather than fail. ([#3754](https://github.com/triggerdotdev/trigger.dev/pull/3754)) - Bump `@s2-dev/streamstore` to `0.22.10` to fix a `TASK_RUN_UNCAUGHT_EXCEPTION` ("Invalid state: Unable to enqueue") when a `chat.agent` turn is aborted mid-stream. ([#3792](https://github.com/triggerdotdev/trigger.dev/pull/3792)) - Coerce numeric `concurrencyKey` values to string at the API boundary across `tasks.trigger`, `tasks.batchTrigger`, and the Phase-2 streaming batch endpoint. ([#3789](https://github.com/triggerdotdev/trigger.dev/pull/3789)) - Add a `tools` option to `chat.agent`. Declaring your tools here threads them into the SDK's internal `convertToModelMessages`, so each tool's `toModelOutput` is re-applied when prior-turn history is re-converted. ([#3790](https://github.com/triggerdotdev/trigger.dev/pull/3790)) <details> <summary>Raw changeset output</summary> ⚠️⚠️⚠️⚠️⚠️⚠️ `main` is currently in **pre mode** so this branch has prereleases rather than normal releases. If you want to exit prereleases, run `changeset pre exit` on `main`. ⚠️⚠️⚠️⚠️⚠️⚠️ # Releases ## @trigger.dev/redis-worker@4.5.0-rc.4 ### Minor Changes - Mollifier buffer extensions: idempotency dedup, an atomic `mutateSnapshot` API, metadata CAS, claim primitives, and a `MollifierSnapshot` type. The buffer's Redis client now reconnects with jittered backoff so a fleet of clients doesn't stampede Redis in lockstep after a blip. ([#3752](https://github.com/triggerdotdev/trigger.dev/pull/3752)) - Add `onTerminalFailure` callback to `MollifierDrainerOptions` so the customer's run lands a SYSTEM_FAILURE PG row even when the drainer exhausts `maxAttempts` on a retryable PG error. Previously, retryable-error exhaustion called `buffer.fail()` directly, which atomically marks FAILED + DELs the entry hash with no PG write — silent data loss when PG was unreachable across the full retry budget. The callback fires before `buffer.fail()` on any terminal path (`cause: "non-retryable"` or `"max-attempts-exhausted"`); throwing a retryable error from the callback causes the drainer to requeue rather than fail. ([#3754](https://github.com/triggerdotdev/trigger.dev/pull/3754)) ### Patch Changes - Pipeline the per-entry `HGETALL` fetches in `MollifierBuffer.listEntriesForEnv`. The previous serial implementation issued one Redis round-trip per runId returned by `LRANGE`, which dominated stale-sweep wall-time at any meaningful backlog (at the sweep's default maxCount=1000, this is ~1000 RTTs per env per pass). Behaviour is unchanged — entries are still skipped when the entry hash has been torn down by a concurrent drainer ack/fail between the LRANGE and the HGETALL. ([#3752](https://github.com/triggerdotdev/trigger.dev/pull/3752)) - Mollifier `mutateSnapshot` now enforces a tag cap: an `append_tags` patch carrying `maxTags` returns `"limit_exceeded"` (writing nothing) when the deduped tag count would exceed the limit, so a buffered run can't accumulate more tags via the tags API than the trigger validator allows at creation. ([#3756](https://github.com/triggerdotdev/trigger.dev/pull/3756)) - Updated dependencies: - `@trigger.dev/core@4.5.0-rc.4` ## @trigger.dev/build@4.5.0-rc.4 ### Patch Changes - Updated dependencies: - `@trigger.dev/core@4.5.0-rc.4` ## trigger.dev@4.5.0-rc.4 ### Patch Changes - Bump `@s2-dev/streamstore` to `0.22.10` to fix a `TASK_RUN_UNCAUGHT_EXCEPTION` ("Invalid state: Unable to enqueue") when a `chat.agent` turn is aborted mid-stream. ([#3792](https://github.com/triggerdotdev/trigger.dev/pull/3792)) - Updated dependencies: - `@trigger.dev/core@4.5.0-rc.4` - `@trigger.dev/build@4.5.0-rc.4` - `@trigger.dev/schema-to-json@4.5.0-rc.4` ## @trigger.dev/core@4.5.0-rc.4 ### Patch Changes - Coerce numeric `concurrencyKey` values to string at the API boundary across `tasks.trigger`, `tasks.batchTrigger`, and the Phase-2 streaming batch endpoint. ([#3789](https://github.com/triggerdotdev/trigger.dev/pull/3789)) - Bump `@s2-dev/streamstore` to `0.22.10` to fix a `TASK_RUN_UNCAUGHT_EXCEPTION` ("Invalid state: Unable to enqueue") when a `chat.agent` turn is aborted mid-stream. ([#3792](https://github.com/triggerdotdev/trigger.dev/pull/3792)) ## @trigger.dev/plugins@4.5.0-rc.4 ### Patch Changes - Updated dependencies: - `@trigger.dev/core@4.5.0-rc.4` ## @trigger.dev/python@4.5.0-rc.4 ### Patch Changes - Updated dependencies: - `@trigger.dev/sdk@4.5.0-rc.4` - `@trigger.dev/core@4.5.0-rc.4` - `@trigger.dev/build@4.5.0-rc.4` ## @trigger.dev/react-hooks@4.5.0-rc.4 ### Patch Changes - Updated dependencies: - `@trigger.dev/core@4.5.0-rc.4` ## @trigger.dev/rsc@4.5.0-rc.4 ### Patch Changes - Updated dependencies: - `@trigger.dev/core@4.5.0-rc.4` ## @trigger.dev/schema-to-json@4.5.0-rc.4 ### Patch Changes - Updated dependencies: - `@trigger.dev/core@4.5.0-rc.4` ## @trigger.dev/sdk@4.5.0-rc.4 ### Patch Changes - Add a `tools` option to `chat.agent`. Declaring your tools here threads them into the SDK's internal `convertToModelMessages`, so each tool's `toModelOutput` is re-applied when prior-turn history is re-converted. ([#3790](https://github.com/triggerdotdev/trigger.dev/pull/3790)) ```ts chat.agent({ tools: { readFile, search }, run: async ({ messages, tools, signal }) => streamText({ model, messages, tools, abortSignal: signal }), }); ``` Also exports `InferChatUIMessageFromTools<typeof tools>` to derive the chat `UIMessage` type (typed tool parts) directly from a tool set. - Updated dependencies: - `@trigger.dev/core@4.5.0-rc.4` </details> Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> |
||
|
Eric Allam
|
2dd9f37b40 |
fix(core,cli): stop chat.agent uncaught exception on mid-stream abort (#3792)
## Summary A `chat.agent` turn that gets aborted mid-stream (stop generation, idle suspend, cancellation) could surface a `TASK_RUN_UNCAUGHT_EXCEPTION` — `TypeError: Invalid state: Unable to enqueue` — in the dashboard. The run kept working, but the exceptions were loud and confusing. ## Root cause The realtime stream writer batches chunks through `@s2-dev/streamstore`'s `BatchTransform`, which holds them for a short linger window before flushing. When the turn's abort signal fires while a record is still buffered, the stream's writable is aborted and the transform's readable controller is closed — but the pending linger `setTimeout` still fires and calls `controller.enqueue()` on the dead controller, throwing from a timer callback where nothing can catch it. Fixed upstream in `@s2-dev/streamstore@0.22.10`, which wraps the linger flush in a try/catch that discards the closed-controller error. This bumps the dependency across core, the CLI, and the webapp, and adds a regression test against the real `BatchTransform`. Verified end-to-end: a mid-stream stop that previously failed the run with `TASK_RUN_UNCAUGHT_EXCEPTION` now leaves the run healthy. |
||
|
Eric Allam
|
e9e2ec1cfc |
fix(sdk): re-apply tool toModelOutput across chat.agent turns (#3790)
## Summary
`chat.agent` now takes a `tools` option. Until now tools only went to
`streamText` inside `run()`, so the SDK had no tools when it
re-converted the persisted `UIMessage` history at the start of each
turn. Any tool with a `toModelOutput` (raw image bytes into an image
content part, or a sub-agent transcript compressed to a summary) had its
transform applied on turn 1 and skipped from turn 2 onward, so the raw
output got JSON-stringified back into the prompt and the model lost the
transformed view.
Declaring `tools` on the config threads them into that conversion, so
`toModelOutput` runs on every turn. The resolved set is handed back,
typed, on the `run()` payload as `tools`:
```ts
const tools = { searchDocs, renderChart };
export const myChat = chat.agent({
tools,
run: async ({ messages, tools, signal }) =>
streamText({ ...chat.toStreamTextOptions({ tools }), messages, abortSignal: signal }),
});
```
`tools` also accepts a per-turn function for tools that depend on the
user or a feature flag. Only `inputSchema` and `toModelOutput` are read
during conversion, never `execute`. Also exports
`InferChatUIMessageFromTools<typeof tools>` to derive the chat
`UIMessage` type from a tool set. No behavior change for agents that
don't declare `tools`.
|
||
|
Daniel Sutton
|
e1950778e2 |
feat(webapp): mollifier API mutations on buffered runs (#3756)
## Summary Cancel, replay, reschedule, metadata, tags, and idempotency-key-reset now succeed against a run that's still in the mollifier buffer. Mutations are applied to the buffered snapshot via Lua CAS; the drainer carries the mutation forward when it replays. Primitives added: - `mutateWithFallback` — PG-first / buffer-fallback resolver with bounded-wait safety net for entries that transition mid-mutation. - `applyMetadataMutation` — buffered metadata PUT mirroring the PG-side retry loop with CAS atomicity. - `resolveRunForMutation` — discriminated-union resolver used by route `findResource` so the route builder's pre-action 404 check sees buffered runs. Routes wired (whole files, no GET/POST splits): - `api.v2.runs.\$runParam.cancel.ts` - `api.v1.runs.\$runParam.replay.ts` - `api.v1.runs.\$runParam.reschedule.ts` - `api.v1.runs.\$runId.metadata.ts` - `api.v1.runs.\$runId.tags.ts` - `resetIdempotencyKey.server.ts` Stacked on the reads PR. ## Test plan - [x] \`pnpm run typecheck --filter webapp\` passes - [x] \`pnpm run test --filter webapp test/mollifierMutateWithFallback.test.ts\` passes - [x] \`pnpm run test --filter webapp test/mollifierApplyMetadataMutation.test.ts\` passes - [x] \`pnpm run test --filter webapp test/mollifierResolveRunForMutation.test.ts\` passes --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
Daniel Sutton
|
4745754a7a |
feat(webapp,run-engine): mollifier drainer replay + stale sweep + cancelled-run engine API (#3754)
## Summary The replay side of the mollifier: - `DrainerHandler`: reads buffered snapshots and replays them through `engine.trigger` to materialise PG rows. - `RunEngine.createCancelledRun`: new public method the handler uses to write CANCELED rows directly from snapshots (bypass queue + waitpoint, emit `runCancelled`). Tolerates the cjson empty-table tags edge case found during validation. - Drainer fairness: org → env rotation so a heavy env doesn't starve light ones in the same org. - Stale-entry sweep + telemetry + alertable gauge so a stuck/offline drainer surfaces in alerts. Both the drainer and sweep default-off; nothing fires unless flagged on (`TRIGGER_MOLLIFIER_DRAINER_ENABLED`, `TRIGGER_MOLLIFIER_STALE_SWEEP_ENABLED`). Stacked on the trigger-time decisions PR. ## Test plan - [x] \`pnpm run typecheck --filter webapp\` passes - [x] \`pnpm run test --filter webapp test/mollifierDrainerHandler.test.ts\` passes - [x] \`pnpm run test --filter webapp test/mollifierStaleSweep.test.ts\` passes - [x] \`pnpm run test --filter @internal/run-engine src/engine/tests/createCancelledRun.test.ts\` passes - [x] \`pnpm run test --filter @trigger.dev/redis-worker packages/redis-worker/src/mollifier/drainer.test.ts\` passes --- ## Ship-gate follow-up fix **Drainer writes SYSTEM_FAILURE on max-attempts exhaustion.** Adds an `onTerminalFailure` callback on `MollifierDrainerOptions` so the customer's run lands a SYSTEM_FAILURE PG row even when the drainer exhausts `MAX_ATTEMPTS` on a retryable PG error (previously `buffer.fail()` was called with no row written → silent data loss). The callback runs before `buffer.fail()` on every terminal path (non-retryable AND max-attempts-exhausted), and re-throwing a retryable error from the callback causes the drainer to requeue rather than fail. Bumps `@trigger.dev/redis-worker` to a **minor** changeset (additive option + new exported types). Includes 5 unit tests covering both terminal causes plus the requeue-on-retryable-callback-failure path and no-callback back-compat. --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
Daniel Sutton
|
20a676d2ed |
fix(core,webapp): coerce numeric concurrencyKey to string (#3789)
## Summary `concurrencyKey` validation accepted only `z.string().optional()` on the single-trigger and V2/V3 batch endpoints, and the Phase-2 streaming NDJSON endpoint accepted `z.record(z.unknown()).optional()` for the entire `options` field. Callers passing `concurrencyKey: someNumericId` (e.g. `payload.userId`) either failed schema validation on the first two paths or sailed through on Phase-2 and then failed downstream at `prisma.taskRun.create` with `Argument concurrencyKey: Expected String or Null, provided Int`. The schema now accepts `string | number` for `concurrencyKey` and stringifies on the way in, across all three paths. The Phase-2 NDJSON `options` is tightened to reuse the strict `BatchTriggerTaskItem.options` shape so it validates identically to the V2/V3 batch endpoints. A defensive `typeof === "number"` coercion at the `engine.trigger` call site in `RunEngineTriggerTaskService` covers in-flight Redis-stored batch items enqueued before the schema fix — those items are rebuilt from a `Record<string, unknown>` shape that bypasses the new schema and would otherwise continue failing for up to their TTL. ## Test plan - [x] `packages/core/src/v3/schemas/batchItemNDJSON.test.ts` — unit tests covering numeric→string coercion, string passthrough, no-options, and rejection of non-string/non-number shapes across `TriggerTaskRequestBody`, `BatchTriggerTaskItem`, and `BatchItemNDJSON`. - [x] `apps/webapp/test/engine/triggerTask.test.ts` — `containerTest` simulating the in-flight Redis batch-item shape (numeric `concurrencyKey` via `Record<string, unknown>`), verifies the run is created with `concurrencyKey: "51262"`. Without the worker coercion, the test reproduces the production stack at `prisma.taskRun.create`. - [x] `pnpm run typecheck --filter webapp` clean. - [x] `pnpm run build --filter @trigger.dev/core --filter @trigger.dev/sdk --filter trigger.dev` clean. Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
Daniel Sutton
|
3924aa4adb |
feat(redis-worker,webapp): mollifier buffer extensions + snapshot type (#3752)
## Summary Buffer-side data layer used by the rest of the mollifier phase-3 stack. - `buffer.ts` gains entry inspection (`getEntry`), idempotency lookup (`lookupIdempotency`), in-place snapshot mutation (`mutateSnapshot`), and dwell tracking. All atomic via Lua. - `mollifierSnapshot.server.ts`: shared `MollifierSnapshot` type plus (de)serialise helpers. - Drops the entry-TTL config and its env var. The drainer is the recovery mechanism; an entry that survives the drainer should surface as a stale-sweep alert, not silently TTL away. Adds methods to the buffer interface; nothing consumes them yet. Subsequent PRs in the stack wire trigger-time mollify, read-fallback, and mutation paths against this surface. ## Test plan - [x] \`pnpm run typecheck --filter webapp\` passes - [x] \`pnpm run test --filter @trigger.dev/redis-worker packages/redis-worker/src/mollifier/buffer.test.ts\` passes --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
github-actions[bot]
|
8f066ac389 |
chore: release v4.5.0-rc.3 (#3763)
🚀 Publish Trigger.dev Docker / typecheck (push) Failing after 0s
🚀 Publish Trigger.dev Docker / publish-webapp (push) Has been skipped
🚀 Publish Trigger.dev Docker / publish-worker (push) Has been skipped
🚀 Publish Trigger.dev Docker / publish-worker-v4 (push) Has been skipped
🚀 Publish Trigger.dev Docker / units (push) Failing after 0s
🧭 Helm Chart Release / lint-and-test (push) Has been cancelled
🧭 Helm Chart Release / release (push) Has been cancelled
## Summary 1 improvement, 2 bug fixes. ## Improvements - Retry `TASK_MIDDLEWARE_ERROR` under the task's retry policy instead of failing the run on the first attempt. The error was already classified as retryable by `shouldRetryError`, but `shouldLookupRetrySettings` did not include it, so the retry flow fell through to `fail_run`. Fixes #3231. ([#3676](https://github.com/triggerdotdev/trigger.dev/pull/3676)) ## Bug fixes - Fix `TypeError` in `unflattenAttributes` when the input attribute map contains conflicting dotted key paths (e.g. both `a.b` set to a scalar and `a.b.c` set to a value). The path-walk loop now applies last-write-wins when a prior key wrote a primitive, null, or array at an intermediate slot, matching the existing precedent in `AttributeFlattener.addAttribute`. Callers no longer crash when handed malformed external attribute inputs. ([#3762](https://github.com/triggerdotdev/trigger.dev/pull/3762)) - Fix external trace context leaking across runs on warm-started workers with `processKeepAlive` enabled. Every subsequent run's attempt span was being exported with the first run's `traceId` and `parentSpanId`, breaking causal-chain navigation in external APM tools. Runs without an external trace context are unaffected. ([#3768](https://github.com/triggerdotdev/trigger.dev/pull/3768)) <details> <summary>Raw changeset output</summary> ⚠️⚠️⚠️⚠️⚠️⚠️ `main` is currently in **pre mode** so this branch has prereleases rather than normal releases. If you want to exit prereleases, run `changeset pre exit` on `main`. ⚠️⚠️⚠️⚠️⚠️⚠️ # Releases ## @trigger.dev/build@4.5.0-rc.3 ### Patch Changes - Updated dependencies: - `@trigger.dev/core@4.5.0-rc.3` ## trigger.dev@4.5.0-rc.3 ### Patch Changes - Updated dependencies: - `@trigger.dev/core@4.5.0-rc.3` - `@trigger.dev/build@4.5.0-rc.3` - `@trigger.dev/schema-to-json@4.5.0-rc.3` ## @trigger.dev/core@4.5.0-rc.3 ### Patch Changes - Retry `TASK_MIDDLEWARE_ERROR` under the task's retry policy instead of failing the run on the first attempt. The error was already classified as retryable by `shouldRetryError`, but `shouldLookupRetrySettings` did not include it, so the retry flow fell through to `fail_run`. Fixes #3231. ([#3676](https://github.com/triggerdotdev/trigger.dev/pull/3676)) - Fix `TypeError` in `unflattenAttributes` when the input attribute map contains conflicting dotted key paths (e.g. both `a.b` set to a scalar and `a.b.c` set to a value). The path-walk loop now applies last-write-wins when a prior key wrote a primitive, null, or array at an intermediate slot, matching the existing precedent in `AttributeFlattener.addAttribute`. Callers no longer crash when handed malformed external attribute inputs. ([#3762](https://github.com/triggerdotdev/trigger.dev/pull/3762)) - Fix external trace context leaking across runs on warm-started workers with `processKeepAlive` enabled. Every subsequent run's attempt span was being exported with the first run's `traceId` and `parentSpanId`, breaking causal-chain navigation in external APM tools. Runs without an external trace context are unaffected. ([#3768](https://github.com/triggerdotdev/trigger.dev/pull/3768)) ## @trigger.dev/plugins@4.5.0-rc.3 ### Patch Changes - Updated dependencies: - `@trigger.dev/core@4.5.0-rc.3` ## @trigger.dev/python@4.5.0-rc.3 ### Patch Changes - Updated dependencies: - `@trigger.dev/core@4.5.0-rc.3` - `@trigger.dev/build@4.5.0-rc.3` - `@trigger.dev/sdk@4.5.0-rc.3` ## @trigger.dev/react-hooks@4.5.0-rc.3 ### Patch Changes - Updated dependencies: - `@trigger.dev/core@4.5.0-rc.3` ## @trigger.dev/redis-worker@4.5.0-rc.3 ### Patch Changes - Updated dependencies: - `@trigger.dev/core@4.5.0-rc.3` ## @trigger.dev/rsc@4.5.0-rc.3 ### Patch Changes - Updated dependencies: - `@trigger.dev/core@4.5.0-rc.3` ## @trigger.dev/schema-to-json@4.5.0-rc.3 ### Patch Changes - Updated dependencies: - `@trigger.dev/core@4.5.0-rc.3` ## @trigger.dev/sdk@4.5.0-rc.3 ### Patch Changes - Updated dependencies: - `@trigger.dev/core@4.5.0-rc.3` </details> Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> |
||
|
nicktrn
|
c043c4a6ad |
fix(core): external trace context leaks across warm-started runs (#3768)
On warm-started workers with `processKeepAlive` enabled, every run's attempt span was exported with the first run's `traceId` and `parentSpanId`. `ExternalSpanExporterWrapper` and `ExternalLogRecordExporterWrapper` captured `externalTraceContext` at `TracingSDK` construction, and the SDK is memoized for the worker's lifetime - so the per-run reset of `StandardTraceContextManager.traceContext` never reached the wrappers. Reported by a customer running v4.4.x: 33 distinct runs on the same host/pid showed up in their APM as siblings of one parent span. Fix: the wrappers now read external context live from the trace context manager per export. Runs without an external trace context fall through to the unchanged `externalTraceId` fallback - no behaviour change for them. Regression test in `packages/core/test/externalSpanExporterWrapper.test.ts` asserts that reassigning the manager between exports produces correctly-parented spans. |
||
|
Iss
|
8399aa2052 |
fix(core): retry TASK_MIDDLEWARE_ERROR under the task's retry policy (#3676)
Retries `TASK_MIDDLEWARE_ERROR` under the task's retry policy. `shouldRetryError` already classed it as retryable, but `shouldLookupRetrySettings` did not, so the run fell through to `fail_run` on attempt 1 instead of using the task's `retry` config. Fixes #3231. |
||
|
Oskar Otwinowski
|
5ba1b32d53 | fix(core): handle conflicting dotted paths in unflattenAttributes (#3762) | ||
|
github-actions[bot]
|
d34014ddcc |
chore: release v4.5.0-rc.2 (#3702)
🚀 Publish Trigger.dev Docker / typecheck (push) Failing after 0s
🚀 Publish Trigger.dev Docker / publish-webapp (push) Has been skipped
🚀 Publish Trigger.dev Docker / publish-worker (push) Has been skipped
🚀 Publish Trigger.dev Docker / publish-worker-v4 (push) Has been skipped
🚀 Publish Trigger.dev Docker / units (push) Failing after 4s
## Summary
3 improvements, 1 bug fix.
## Improvements
- The per-turn merge now overlays the wire copy's tool-part state
advancement onto the agent's existing chain — `state` + the matching
resolution field (`output` / `errorText` / `approval`) come from the
wire, everything else (text, reasoning, tool `input`, provider metadata)
stays whatever the snapshot or `hydrateMessages` returned. Previously a
full-message replace overwrote those fields with whatever the client
shipped, so a slimmed wire copy landed a tool call with no `arguments`
on the next LLM call. Covers `output-available` / `output-error` (HITL
`addToolOutput`) and `approval-responded` / `output-denied` (approval
flow).
- `TriggerChatTransport.sendMessages` and `AgentChat.sendRaw` now slim
assistant messages that carry advanced tool parts. The wire payload is
just `{ id, role, parts: [<state + resolution field>] }` for
`submit-message` continuations; everything else passes through.
Reasoning blobs and full tool inputs no longer ride the wire on every
`addToolOutput` / `addToolApproveResponse`, so continuation payloads
stay well under the `.in/append` cap on long agent loops.
- Add `TriggerClient` for running multiple SDK clients side-by-side,
each with its own auth, preview branch, and baseURL. Useful when a
single process needs to trigger tasks or read runs across multiple
projects, environments, or preview branches without mutating shared
global state.
([#3683](https://github.com/triggerdotdev/trigger.dev/pull/3683))
## Bug fixes
- Fix `chat.agent` HITL continuations on reasoning-heavy turns. Two
changes that work together:
([#3719](https://github.com/triggerdotdev/trigger.dev/pull/3719))
<details>
<summary>Raw changeset output</summary>
⚠️⚠️⚠️⚠️⚠️⚠️
`main` is currently in **pre mode** so this branch has prereleases
rather than normal releases. If you want to exit prereleases, run
`changeset pre exit` on `main`.
⚠️⚠️⚠️⚠️⚠️⚠️
# Releases
## @trigger.dev/build@4.5.0-rc.2
### Patch Changes
- Updated dependencies:
- `@trigger.dev/core@4.5.0-rc.2`
## trigger.dev@4.5.0-rc.2
### Patch Changes
- Updated dependencies:
- `@trigger.dev/build@4.5.0-rc.2`
- `@trigger.dev/core@4.5.0-rc.2`
- `@trigger.dev/schema-to-json@4.5.0-rc.2`
## @trigger.dev/plugins@4.5.0-rc.2
### Patch Changes
- Updated dependencies:
- `@trigger.dev/core@4.5.0-rc.2`
## @trigger.dev/python@4.5.0-rc.2
### Patch Changes
- Updated dependencies:
- `@trigger.dev/sdk@4.5.0-rc.2`
- `@trigger.dev/build@4.5.0-rc.2`
- `@trigger.dev/core@4.5.0-rc.2`
## @trigger.dev/react-hooks@4.5.0-rc.2
### Patch Changes
- Updated dependencies:
- `@trigger.dev/core@4.5.0-rc.2`
## @trigger.dev/redis-worker@4.5.0-rc.2
### Patch Changes
- Updated dependencies:
- `@trigger.dev/core@4.5.0-rc.2`
## @trigger.dev/rsc@4.5.0-rc.2
### Patch Changes
- Updated dependencies:
- `@trigger.dev/core@4.5.0-rc.2`
## @trigger.dev/schema-to-json@4.5.0-rc.2
### Patch Changes
- Updated dependencies:
- `@trigger.dev/core@4.5.0-rc.2`
## @trigger.dev/sdk@4.5.0-rc.2
### Patch Changes
- Fix `chat.agent` HITL continuations on reasoning-heavy turns. Two
changes that work together:
([#3719](https://github.com/triggerdotdev/trigger.dev/pull/3719))
- The per-turn merge now overlays the wire copy's tool-part state
advancement onto the agent's existing chain — `state` + the matching
resolution field (`output` / `errorText` / `approval`) come from the
wire, everything else (text, reasoning, tool `input`, provider metadata)
stays whatever the snapshot or `hydrateMessages` returned. Previously a
full-message replace overwrote those fields with whatever the client
shipped, so a slimmed wire copy landed a tool call with no `arguments`
on the next LLM call. Covers `output-available` / `output-error` (HITL
`addToolOutput`) and `approval-responded` / `output-denied` (approval
flow).
- `TriggerChatTransport.sendMessages` and `AgentChat.sendRaw` now slim
assistant messages that carry advanced tool parts. The wire payload is
just `{ id, role, parts: [<state + resolution field>] }` for
`submit-message` continuations; everything else passes through.
Reasoning blobs and full tool inputs no longer ride the wire on every
`addToolOutput` / `addToolApproveResponse`, so continuation payloads
stay well under the `.in/append` cap on long agent loops.
Note: `onValidateMessages` receives the slim wire on HITL turns. If you
call `validateUIMessages` from `ai` against the full `messages` array it
will reject the slim assistant; filter to user messages (or skip on HITL
turns) — see the updated docstring on `onValidateMessages` for the
recommended pattern.
For `hydrateMessages` hooks that persist the chain, this release also
adds a small helper to the `@trigger.dev/sdk/ai` surface:
```ts
import { chat, upsertIncomingMessage } from "@trigger.dev/sdk/ai";
chat.agent({
hydrateMessages: async ({ chatId, trigger, incomingMessages }) => {
const record = await db.chat.findUnique({ where: { id: chatId } });
const stored = record?.messages ?? [];
if (upsertIncomingMessage(stored, { trigger, incomingMessages })) {
await db.chat.update({ where: { id: chatId }, data: { messages: stored }
});
}
return stored;
},
});
```
It pushes fresh user messages by id, no-ops on HITL continuations (the
incoming shares an id with the existing assistant — the runtime overlays
the new tool-state advance), and skips on non-`submit-message` triggers.
Returns `true` if it mutated `stored` so the caller knows whether to
persist.
Net effect: `chat.addToolOutput(...)` /
`chat.addToolApproveResponse(...)` on multi-step reasoning agents
(OpenAI Responses with `store: false`, Anthropic extended thinking,
etc.) no longer blows the cap and no longer corrupts the LLM input.
- Add `TriggerClient` for running multiple SDK clients side-by-side,
each with its own auth, preview branch, and baseURL. Useful when a
single process needs to trigger tasks or read runs across multiple
projects, environments, or preview branches without mutating shared
global state.
([#3683](https://github.com/triggerdotdev/trigger.dev/pull/3683))
```ts
import { TriggerClient } from "@trigger.dev/sdk";
const prod = new TriggerClient({ accessToken:
process.env.TRIGGER_PROD_KEY });
const preview = new TriggerClient({
accessToken: process.env.TRIGGER_PREVIEW_KEY,
previewBranch: "signup-flow",
});
await prod.tasks.trigger("send-email", payload);
await preview.runs.list({ status: ["COMPLETED"] });
```
- Updated dependencies:
- `@trigger.dev/core@4.5.0-rc.2`
## @trigger.dev/core@4.5.0-rc.2
</details>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
|
||
|
Eric Allam
|
75679c7518 |
fix(sdk): chat HITL continuations no longer break the next LLM call (#3719)
## Summary
Multi-step reasoning agents with HITL tools (OpenAI Responses with
`store: false`, Anthropic extended thinking, etc.) failed on
`chat.addToolOutput(...)` continuations — either the wire payload blew
the `.in/append` cap (reasoning blobs + tool inputs routinely > 512
KiB), or app-side slimming workarounds got overwritten server-side and
the next LLM call landed a tool call with no `arguments`. Both modes are
fixed.
## Design
The per-turn merge in `chat.agent` now overlays only the tool-part state
advances (`output-available` / `output-error` / `approval-responded` /
`output-denied`) from the wire copy onto the hydrated/snapshot chain.
Previously it replaced the entire message, which dropped `input`,
reasoning, and text from the LLM's view whenever the wire was slim.
In parallel, `TriggerChatTransport.sendMessages` and `AgentChat.sendRaw`
now slim the assistant message themselves on `submit-message`
continuations: ship `{ id, role, parts: [<resolved tool part only>] }`,
everything else reconstructed server-side from `hydrateMessages` or the
durable snapshot. Continuation payloads drop from 600 KiB – 1 MiB to ~1
KiB.
`references/ai-chat` `aiChatHydrated.hydrateMessages` now upserts by id
instead of pushing. With slim continuations, a blind push duplicates the
assistant id in the returned chain — the merge updates the first match,
the slim duplicate goes straight to `toModelMessages` with no `input`,
and the LLM 4xx's. This is the canonical pattern customers should mirror
in their own hydrate implementations.
## Test plan
- 11 new tests (slim helper unit + slim+merge integration for HITL,
approval, default no-hydrate branch)
- Full SDK suite: 239 tests pass across 19 files
- End-to-end sweep against `references/ai-chat`: 19 customer-side smoke
tests green; HITL wire bodies confirmed at ~1 KiB (was 600 KiB+); no
provider 4xx errors across OpenAI Responses or Anthropic
|
||
|
Eric Allam
|
832cf7220b |
feat(sdk,core): add TriggerClient for per-instance SDK configuration (#3683)
## Summary
`new TriggerClient({...})` exposes the management API (tasks, runs,
schedules, envvars, batch, queues, deployments, prompts, auth) as an
explicit instance with its own auth, preview branch, and baseURL.
Multiple clients can coexist in one process without mutating shared
global state — useful when a single service triggers across multiple
projects, environments, or preview branches.
```ts
import { TriggerClient } from "@trigger.dev/sdk";
const prod = new TriggerClient({ accessToken: process.env.TRIGGER_PROD_KEY });
const preview = new TriggerClient({
accessToken: process.env.TRIGGER_PREVIEW_KEY,
previewBranch: "signup-flow",
});
await prod.tasks.trigger("send-email", payload);
await preview.runs.list({ status: ["COMPLETED"] });
```
The existing global `configure()` API keeps working unchanged.
## Design
Instance methods enter an `AsyncLocalStorage`-backed scope (`sdkScope`)
before delegating to the existing module-level functions. The four
"pollution" points that previously read globals now consult the scope
first:
- `apiClientManager.{baseURL, accessToken, branchName}` and
`clientOrThrow` — identity fields are scope-only when scoped; `baseURL`
still falls back to `TRIGGER_API_URL` because plumbing (where the API
lives) is not identity.
- `taskContext.{ctx, worker, isWarmStart, isInsideTask}` — masked inside
an isolated scope so a `client.tasks.trigger(...)` from inside a task
doesn't leak the parent's `parentRunId` / `lockToVersion` / `isTest`
into a trigger that hits a different project.
- Inline `getEnvVar("TRIGGER_VERSION")` reads in `shared.ts` go through
a `scopedEnvVar` helper that returns `undefined` inside an isolated
scope.
The `TriggerClient` class itself is a thin wrapper that captures the
scope in its constructor and proxies each namespace method to enter that
scope before calling the existing impl. Generic inference (e.g.
`client.tasks.trigger<typeof t>(...)`) is preserved via `Pick<typeof ns,
keyof curatedSubset>` typings.
Two correctness fixes uncovered along the way are folded in:
- `apiClientManager.setGlobalAPIClientConfiguration` no longer silently
no-ops on the second call. `configure()` now actually overrides as users
expect (this is the root cause behind some "I changed the config but
nothing happened" reports).
- `apiClientManager.runWithConfig` (and therefore `auth.withAuth`) is
now backed by `sdkScope.withScope` instead of "mutate the global and
restore in finally". Two parallel `withAuth` calls with different
configs no longer stomp each other.
Surface curation: instance namespaces drop methods that don't make sense
per-instance — `batch.*AndWait` (runtime-dependent), `schedules.task` /
`schedules.timezones` (definition-time / stateless), `prompts.define`
(definition-time), `auth.configure` / `auth.withAuth` (global-only).
## Test plan
- [x] 9 runtime unit tests in `triggerClient.test.ts` cover: required
accessToken, instance auth + branch headers, no env fallback for
identity fields, no leakage between global and instance, four parallel
calls across two clients stay isolated, taskContext masking +
`inheritContext: true` override, `configure()` second-call override,
parallel `auth.withAuth` isolation.
- [x] 10 type-level assertions in `triggerClient.types.test.ts` using
`expectTypeOf` + `@ts-expect-error` lock in generic inference, return
type passthrough, overload preservation, and curated-surface drift.
- [x] Full SDK suite (219 tests) and core suite (530 tests) pass.
- [x] Webapp typecheck clean.
- [x] End-to-end smoke test against local webapp and a
freshly-provisioned cloud project — six concurrent multi-client triggers
all returned 200 with run IDs, headers per-client as expected.
- [ ] Reviewer: run `references/multi-client` per its `README.md` to
reproduce the smoke test locally.
## Try it
`references/multi-client` is a new reference workspace that exercises
this end-to-end:
- `src/trigger/echo.ts` — trivial target task
- `src/trigger/fanOut.ts` — opens two `TriggerClient`s from inside a
task, fires `echo` through each in parallel
- `src/external/main.ts` — external Node script with two clients
triggering `echo` sequentially and concurrently; logs every outgoing
request's `authorization` + `x-trigger-branch`
- `src/external/isolation.ts` — interleaves global `configure()` and an
instance call, asserts the captured fetch sequence shows no leakage
either way
|
||
|
github-actions[bot]
|
0b5b817a49 |
chore: release v4.5.0-rc.1 (#3691)
🚀 Publish Trigger.dev Docker / typecheck (push) Failing after 0s
🚀 Publish Trigger.dev Docker / publish-webapp (push) Has been skipped
🚀 Publish Trigger.dev Docker / publish-worker (push) Has been skipped
🚀 Publish Trigger.dev Docker / publish-worker-v4 (push) Has been skipped
🚀 Publish Trigger.dev Docker / units (push) Failing after 0s
🧭 Helm Chart Release / lint-and-test (push) Has been cancelled
🧭 Helm Chart Release / release (push) Has been cancelled
## Summary 2 bug fixes. ## Bug fixes - Fix `chat.agent` skills silently missing in `trigger dev` for projects whose task files read `process.env` at module top level (e.g. a third-party SDK client initialized at import). Skill folders now bundle into `.trigger/skills/` reliably regardless of which env vars are set when the CLI launches. ([#3690](https://github.com/triggerdotdev/trigger.dev/pull/3690)) - Fix `COULD_NOT_FIND_EXECUTOR` when a task's definition is loaded via `await import(...)` from inside another task's `run()`. The runtime workers now register such tasks with a sentinel file context, and the catalog logs a one-time warning per task id. ([#3688](https://github.com/triggerdotdev/trigger.dev/pull/3688)) <details> <summary>Raw changeset output</summary> ⚠️⚠️⚠️⚠️⚠️⚠️ `main` is currently in **pre mode** so this branch has prereleases rather than normal releases. If you want to exit prereleases, run `changeset pre exit` on `main`. ⚠️⚠️⚠️⚠️⚠️⚠️ # Releases ## @trigger.dev/build@4.5.0-rc.1 ### Patch Changes - Updated dependencies: - `@trigger.dev/core@4.5.0-rc.1` ## trigger.dev@4.5.0-rc.1 ### Patch Changes - Fix `chat.agent` skills silently missing in `trigger dev` for projects whose task files read `process.env` at module top level (e.g. a third-party SDK client initialized at import). Skill folders now bundle into `.trigger/skills/` reliably regardless of which env vars are set when the CLI launches. ([#3690](https://github.com/triggerdotdev/trigger.dev/pull/3690)) - Fix `COULD_NOT_FIND_EXECUTOR` when a task's definition is loaded via `await import(...)` from inside another task's `run()`. The runtime workers now register such tasks with a sentinel file context, and the catalog logs a one-time warning per task id. ([#3688](https://github.com/triggerdotdev/trigger.dev/pull/3688)) - Updated dependencies: - `@trigger.dev/core@4.5.0-rc.1` - `@trigger.dev/build@4.5.0-rc.1` - `@trigger.dev/schema-to-json@4.5.0-rc.1` ## @trigger.dev/core@4.5.0-rc.1 ### Patch Changes - Fix `COULD_NOT_FIND_EXECUTOR` when a task's definition is loaded via `await import(...)` from inside another task's `run()`. The runtime workers now register such tasks with a sentinel file context, and the catalog logs a one-time warning per task id. ([#3688](https://github.com/triggerdotdev/trigger.dev/pull/3688)) ## @trigger.dev/plugins@4.5.0-rc.1 ### Patch Changes - Updated dependencies: - `@trigger.dev/core@4.5.0-rc.1` ## @trigger.dev/python@4.5.0-rc.1 ### Patch Changes - Updated dependencies: - `@trigger.dev/core@4.5.0-rc.1` - `@trigger.dev/build@4.5.0-rc.1` - `@trigger.dev/sdk@4.5.0-rc.1` ## @trigger.dev/react-hooks@4.5.0-rc.1 ### Patch Changes - Updated dependencies: - `@trigger.dev/core@4.5.0-rc.1` ## @trigger.dev/redis-worker@4.5.0-rc.1 ### Patch Changes - Updated dependencies: - `@trigger.dev/core@4.5.0-rc.1` ## @trigger.dev/rsc@4.5.0-rc.1 ### Patch Changes - Updated dependencies: - `@trigger.dev/core@4.5.0-rc.1` ## @trigger.dev/schema-to-json@4.5.0-rc.1 ### Patch Changes - Updated dependencies: - `@trigger.dev/core@4.5.0-rc.1` ## @trigger.dev/sdk@4.5.0-rc.1 ### Patch Changes - Updated dependencies: - `@trigger.dev/core@4.5.0-rc.1` </details> Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> |
||
|
Eric Allam
|
5ddb81ac8d |
fix(cli): stop chat.agent skills silently disappearing from trigger dev (#3690)
## Summary `trigger.dev dev` was silently dropping registered `chat.agent` skills for any project whose task files read `process.env` at module top level — e.g. a third-party SDK client initialized at import. The agent would boot fine, but `skill.local()` failed at runtime with `ENOENT` because the skill folder was never copied into `.trigger/skills/`. ## Design The CLI ran two indexer passes in dev: the worker's own indexer (with the full env it eventually executes tasks in), and a separate skill-discovery indexer with only the CLI process's env. Top-level reads of vars like `TRIGGER_API_URL` imported cleanly in the worker pass and threw in the skill pass — the latter caught the error, warned, and skipped skill copying. Failure was silent enough that `skill.local()` only surfaced it at task runtime. The skill registry is already part of the worker manifest. This PR drops the duplicate pass and copies skill folders from that manifest after the worker initializes. One indexer instead of two; a bad `SKILL.md` now surfaces as a startup error instead of silently disappearing skills. Deploy is unaffected — its skill discovery uses the project's environment variables (fetched via the API, which fills in `TRIGGER_API_URL` etc.), so the dev failure mode doesn't reach there. ## Test plan - [x] New `references/agent-skills` reference project with `skills.define` + a task that calls `skill.local()` and runs a bundled script - [x] On `main`, adding a top-level `process.env.TRIGGER_API_URL!.includes(...)` read in any task file reproduces the symptom: warning at dev startup, no `.trigger/skills/` folder, `skill.local()` fails with ENOENT - [x] On this branch, same project boots clean and `skill.local()` works end-to-end - [x] Deploy still works end-to-end with the new reference project |
||
|
nicktrn
|
dbf9b4e7e5 |
fix(core,cli): register tasks loaded via dynamic import during run (#3688)
On a warm worker process, a task whose `task()` definition is loaded via `await import(...)` from inside another task's `run()` could end up permanently missing from the catalog: the `task()` call fired with no `_currentFileContext` set, `registerTaskMetadata` silently returned, and Node's ESM module cache then blocked the worker's setContext + re-import recovery from ever firing the call again. Subsequent runs of that task on the same warm process failed with `COULD_NOT_FIND_EXECUTOR` until the process hit `maxExecutionsPerProcess` and exited. All five of these had to coincide on the same worker for the bug to surface: 1. `processKeepAlive` enabled (so catalog state survives across runs). 2. A `run()` function (or lifecycle hook) does `await import(...)`. 3. The import's transitive static graph reaches a `task()` / `schemaTask()` call. 4. The task containing the dynamic import is the **first** task to run on a given warm worker process — so the dropped `task()` calls fire on this process for the first time, are silently dropped, and Node's module cache locks the wrong outcome in. 5. A subsequent run for one of the dropped task ids lands on the same warm worker before it recycles. The runtime workers now set a sentinel file context (`<no-context>`) around the `executor.execute(...)` call, so `task()` invocations firing during a run register normally. The catalog detects the sentinel and emits a one-time `console.warn` per task id so the pattern stays visible without spamming. The indexer never sets this context, so deploy-time behavior is unchanged. Repro is `references/hello-world/src/trigger/dynamicImportRepro*.ts`. Verified end-to-end against a deployed image with firestarter warm-starts on: pre-fix saw `COULD_NOT_FIND_EXECUTOR` on children that landed on the parent-poisoned worker; post-fix all 23/23 runs succeeded and the warning surfaces in the parent's run trace. |
||
|
github-actions[bot]
|
acfba02409 |
chore: release v4.5.0-rc.0 (#3563)
🚀 Publish Trigger.dev Docker / typecheck (push) Failing after 4s
🚀 Publish Trigger.dev Docker / units (push) Failing after 0s
🚀 Publish Trigger.dev Docker / publish-webapp (push) Has been skipped
🚀 Publish Trigger.dev Docker / publish-worker (push) Has been skipped
🚀 Publish Trigger.dev Docker / publish-worker-v4 (push) Has been skipped
🧭 Helm Chart Release / lint-and-test (push) Has been cancelled
🧭 Helm Chart Release / release (push) Has been cancelled
## Summary 44 improvements, 1 bug fix. ## Improvements - **AI Prompts** — define prompt templates as code alongside your tasks, version them on deploy, and override the text or model from the dashboard without redeploying. Prompts integrate with the Vercel AI SDK via `toAISDKTelemetry()` (links every generation span back to the prompt) and with `chat.agent` via `chat.prompt.set()` + `chat.toStreamTextOptions()`. ([#3629](https://github.com/triggerdotdev/trigger.dev/pull/3629)) - **Code-defined, deploy-versioned templates** — define with `prompts.define({ id, model, config, variables, content })`. Every deploy creates a new version visible in the dashboard. Mustache-style placeholders (`{{var}}`, `{{#cond}}...{{/cond}}`) with Zod / ArkType / Valibot-typed variables. - **Dashboard overrides** — change a prompt's text or model from the dashboard without redeploying. Overrides take priority over the deployed "current" version and are environment-scoped (dev / staging / production independent). - **Resolve API** — `prompt.resolve(vars, { version?, label? })` returns the compiled `text`, resolved `model`, `version`, and labels. Standalone `prompts.resolve<typeof handle>(slug, vars)` for cross-file resolution with full type inference on slug and variable shape. - **AI SDK integration** — spread `resolved.toAISDKTelemetry({ ...extra })` into any `generateText` / `streamText` call and every generation span links to the prompt in the dashboard alongside its input variables, model, tokens, and cost. - **`chat.agent` integration** — `chat.prompt.set(resolved)` stores the resolved prompt run-scoped; `chat.toStreamTextOptions({ registry })` pulls `system`, `model` (resolved via the AI SDK provider registry), `temperature` / `maxTokens` / etc., and telemetry into a single spread for `streamText`. - **Management SDK** — `prompts.list()`, `prompts.versions(slug)`, `prompts.promote(slug, version)`, `prompts.createOverride(slug, body)`, `prompts.updateOverride(slug, body)`, `prompts.removeOverride(slug)`, `prompts.reactivateOverride(slug, version)`. - **Dashboard** — prompts list with per-prompt usage sparklines; per-prompt detail with Template / Details / Versions / Generations / Metrics tabs. AI generation spans get a custom inspector showing the linked prompt's metadata, input variables, and template content alongside model, tokens, cost, and the message thread. - Adds `onBoot` to `chat.agent` — a lifecycle hook that fires once per worker process picking up the chat. Runs for the initial run, preloaded runs, AND reactive continuation runs (post-cancel, crash, `endRun`, `requestUpgrade`, OOM retry), before any other hook. Use it to initialize `chat.local`, open per-process resources, or re-hydrate state from your DB on continuation — anywhere the SAME run picking up after suspend/resume isn't enough. ([#3543](https://github.com/triggerdotdev/trigger.dev/pull/3543)) - **AI SDK `useChat` integration** — a custom [`ChatTransport`](https://sdk.vercel.ai/docs/ai-sdk-ui/transport) (`useTriggerChatTransport`) plugs straight into Vercel AI SDK's `useChat` hook. Text streaming, tool calls, reasoning, and `data-*` parts all work natively over Trigger.dev's realtime streams. No custom API routes needed. - **First-turn fast path (`chat.headStart`)** — opt-in handler that runs the first turn's `streamText` step in your warm server process while the agent run boots in parallel, cutting cold-start TTFC by roughly half (measured 2801ms → 1218ms on `claude-sonnet-4-6`). The agent owns step 2+ (tool execution, persistence, hooks) so heavy deps stay where they belong. Web Fetch handler works natively in Next.js, Hono, SvelteKit, Remix, Workers, etc.; bridge to Express/Fastify/Koa via `chat.toNodeListener`. New `@trigger.dev/sdk/chat-server` subpath. - **Multi-turn durability via Sessions** — every chat is backed by a durable Session that outlives any individual run. Conversations resume across page refreshes, idle timeout, crashes, and deploys; `resume: true` reconnects via `lastEventId` so clients only see new chunks. `sessions.list` enumerates chats for inbox-style UIs. - **Auto-accumulated history, delta-only wire** — the backend accumulates the full conversation across turns; clients only ship the new message each turn. Long chats never hit the 512 KiB body cap. Register `hydrateMessages` to be the source of truth yourself. - **Lifecycle hooks** — `onPreload`, `onChatStart`, `onValidateMessages`, `hydrateMessages`, `onTurnStart`, `onBeforeTurnComplete`, `onTurnComplete`, `onChatSuspend`, `onChatResume` — for persistence, validation, and post-turn work. - **Stop generation** — client-driven `transport.stopGeneration(chatId)` aborts mid-stream; the run stays alive for the next message, partial response is captured, and aborted parts (stuck `partial-call` tools, in-progress reasoning) are auto-cleaned. - **Tool approvals (HITL)** — tools with `needsApproval: true` pause until the user approves or denies via `addToolApprovalResponse`. The runtime reconciles the updated assistant message by ID and continues `streamText`. - **Steering and background injection** — `pendingMessages` injects user messages between tool-call steps so users can steer the agent mid-execution; `chat.inject()` + `chat.defer()` adds context from background work (self-review, RAG, safety checks) between turns. - **Actions** — non-turn frontend commands (undo, rollback, regenerate, edit) sent via `transport.sendAction`. Fire `hydrateMessages` + `onAction` only — no turn hooks, no `run()`. `onAction` can return a `StreamTextResult` for a model response, or `void` for side-effect-only. - **Typed state primitives** — `chat.local<T>` for per-run state accessible from hooks, `run()`, tools, and subtasks (auto-serialized through `ai.toolExecute`); `chat.store` for typed shared data between agent and client; `chat.history` for reading and mutating the message chain; `clientDataSchema` for typed `clientData` in every hook. - **`chat.toStreamTextOptions()`** — one spread into `streamText` wires up versioned system [Prompts](https://trigger.dev/docs/ai/prompts), model resolution, telemetry metadata, compaction, steering, and background injection. - **Multi-tab coordination** — `multiTab: true` + `useMultiTabChat` prevents duplicate sends and syncs state across browser tabs via `BroadcastChannel`. Non-active tabs go read-only with live updates. - **Network resilience** — built-in indefinite retry with bounded backoff, reconnect on `online` / tab refocus / bfcache restore, `Last-Event-ID` mid-stream resume. No app code needed. - **Sessions** — a durable, run-aware stream channel keyed on a stable `externalId`. A Session is the unit of state that owns a multi-run conversation: messages flow through `.in`, responses through `.out`, both survive run boundaries. Sessions back the new `chat.agent` runtime, and you can build on them directly for any pattern that needs durable bi-directional streaming across runs. ([#3542](https://github.com/triggerdotdev/trigger.dev/pull/3542)) - Add `ai.toolExecute(task)` so you can wire a Trigger subtask in as the `execute` handler of an AI SDK `tool()` while defining `description` and `inputSchema` yourself — useful when you want full control over the tool surface and just need Trigger's subtask machinery for the body. ([#3546](https://github.com/triggerdotdev/trigger.dev/pull/3546)) - Type `chat.createStartSessionAction` against your chat agent so `clientData` is typed end-to-end on the first turn: ([#3684](https://github.com/triggerdotdev/trigger.dev/pull/3684)) - Add `region` to the runs list / retrieve API: filter runs by region (`runs.list({ region: "..." })` / `filter[region]=<masterQueue>`) and read each run's executing region from the new `region` field on the response. ([#3612](https://github.com/triggerdotdev/trigger.dev/pull/3612)) - Add `TRIGGER_BUILD_SKIP_REWRITE_TIMESTAMP=1` escape hatch for local self-hosted builds whose buildx driver doesn't support `rewrite-timestamp` alongside push (e.g. orbstack's default `docker` driver). ([#3618](https://github.com/triggerdotdev/trigger.dev/pull/3618)) - Reject overlong `idempotencyKey` values at the API boundary so they no longer trip an internal size limit on the underlying unique index and surface as a generic 500. Inputs are capped at 2048 characters — well above what `idempotencyKeys.create()` produces (a 64-character hash) and above any realistic raw key. Applies to `tasks.trigger`, `tasks.batchTrigger`, `batch.create` (Phase 1 streaming batches), `wait.createToken`, `wait.forDuration`, and the input/session stream waitpoint endpoints. Over-limit requests now return a structured 400 instead. ([#3560](https://github.com/triggerdotdev/trigger.dev/pull/3560)) - **AI SDK `useChat` integration** — a custom [`ChatTransport`](https://sdk.vercel.ai/docs/ai-sdk-ui/transport) (`useTriggerChatTransport`) plugs straight into Vercel AI SDK's `useChat` hook. Text streaming, tool calls, reasoning, and `data-*` parts all work natively over Trigger.dev's realtime streams. No custom API routes needed. - **First-turn fast path (`chat.headStart`)** — opt-in handler that runs the first turn's `streamText` step in your warm server process while the agent run boots in parallel, cutting cold-start TTFC by roughly half (measured 2801ms → 1218ms on `claude-sonnet-4-6`). The agent owns step 2+ (tool execution, persistence, hooks) so heavy deps stay where they belong. Web Fetch handler works natively in Next.js, Hono, SvelteKit, Remix, Workers, etc.; bridge to Express/Fastify/Koa via `chat.toNodeListener`. New `@trigger.dev/sdk/chat-server` subpath. - **Multi-turn durability via Sessions** — every chat is backed by a durable Session that outlives any individual run. Conversations resume across page refreshes, idle timeout, crashes, and deploys; `resume: true` reconnects via `lastEventId` so clients only see new chunks. `sessions.list` enumerates chats for inbox-style UIs. - **Auto-accumulated history, delta-only wire** — the backend accumulates the full conversation across turns; clients only ship the new message each turn. Long chats never hit the 512 KiB body cap. Register `hydrateMessages` to be the source of truth yourself. - **Lifecycle hooks** — `onPreload`, `onChatStart`, `onValidateMessages`, `hydrateMessages`, `onTurnStart`, `onBeforeTurnComplete`, `onTurnComplete`, `onChatSuspend`, `onChatResume` — for persistence, validation, and post-turn work. - **Stop generation** — client-driven `transport.stopGeneration(chatId)` aborts mid-stream; the run stays alive for the next message, partial response is captured, and aborted parts (stuck `partial-call` tools, in-progress reasoning) are auto-cleaned. - **Tool approvals (HITL)** — tools with `needsApproval: true` pause until the user approves or denies via `addToolApprovalResponse`. The runtime reconciles the updated assistant message by ID and continues `streamText`. - **Steering and background injection** — `pendingMessages` injects user messages between tool-call steps so users can steer the agent mid-execution; `chat.inject()` + `chat.defer()` adds context from background work (self-review, RAG, safety checks) between turns. - **Actions** — non-turn frontend commands (undo, rollback, regenerate, edit) sent via `transport.sendAction`. Fire `hydrateMessages` + `onAction` only — no turn hooks, no `run()`. `onAction` can return a `StreamTextResult` for a model response, or `void` for side-effect-only. - **Typed state primitives** — `chat.local<T>` for per-run state accessible from hooks, `run()`, tools, and subtasks (auto-serialized through `ai.toolExecute`); `chat.store` for typed shared data between agent and client; `chat.history` for reading and mutating the message chain; `clientDataSchema` for typed `clientData` in every hook. - **`chat.toStreamTextOptions()`** — one spread into `streamText` wires up versioned system [Prompts](https://trigger.dev/docs/ai/prompts), model resolution, telemetry metadata, compaction, steering, and background injection. - **Multi-tab coordination** — `multiTab: true` + `useMultiTabChat` prevents duplicate sends and syncs state across browser tabs via `BroadcastChannel`. Non-active tabs go read-only with live updates. - **Network resilience** — built-in indefinite retry with bounded backoff, reconnect on `online` / tab refocus / bfcache restore, `Last-Event-ID` mid-stream resume. No app code needed. - Retry `TASK_PROCESS_SIGSEGV` task crashes under the user's retry policy instead of failing the run on the first segfault. SIGSEGV in Node tasks is frequently non-deterministic (native addon races, JIT/GC interaction, near-OOM in native code, host issues), so retrying on a fresh process often succeeds. The retry is gated by the task's existing `retry` config + `maxAttempts` — same path `TASK_PROCESS_SIGTERM` and uncaught exceptions already use — so tasks without a retry policy still fail fast. ([#3552](https://github.com/triggerdotdev/trigger.dev/pull/3552)) - The public interfaces for a plugin system. Initially consolidated authentication and authorization interfaces. ([#3499](https://github.com/triggerdotdev/trigger.dev/pull/3499)) - Add MollifierBuffer and MollifierDrainer primitives for trigger burst smoothing. ([#3614](https://github.com/triggerdotdev/trigger.dev/pull/3614)) ## Bug fixes - Fix `LocalsKey<T>` type incompatibility across dual-package builds. The phantom value-type brand no longer uses a module-level `unique symbol`, so a single TypeScript compilation that resolves the type from both the ESM and CJS outputs (which can happen under certain pnpm hoisting layouts) no longer sees two structurally-incompatible variants of the same type. ([#3626](https://github.com/triggerdotdev/trigger.dev/pull/3626)) <details> <summary>Raw changeset output</summary> ⚠️⚠️⚠️⚠️⚠️⚠️ `main` is currently in **pre mode** so this branch has prereleases rather than normal releases. If you want to exit prereleases, run `changeset pre exit` on `main`. ⚠️⚠️⚠️⚠️⚠️⚠️ # Releases ## @trigger.dev/sdk@4.5.0-rc.0 ### Minor Changes - **AI Prompts** — define prompt templates as code alongside your tasks, version them on deploy, and override the text or model from the dashboard without redeploying. Prompts integrate with the Vercel AI SDK via `toAISDKTelemetry()` (links every generation span back to the prompt) and with `chat.agent` via `chat.prompt.set()` + `chat.toStreamTextOptions()`. ([#3629](https://github.com/triggerdotdev/trigger.dev/pull/3629)) ```ts import { prompts } from "@trigger.dev/sdk"; import { generateText } from "ai"; import { openai } from "@ai-sdk/openai"; import { z } from "zod"; export const supportPrompt = prompts.define({ id: "customer-support", model: "gpt-4o", config: { temperature: 0.7 }, variables: z.object({ customerName: z.string(), plan: z.string(), issue: z.string(), }), content: `You are a support agent for Acme. Customer: {{customerName}} ({{plan}} plan) Issue: {{issue}}`, }); const resolved = await supportPrompt.resolve({ customerName: "Alice", plan: "Pro", issue: "Can't access billing", }); const result = await generateText({ model: openai(resolved.model ?? "gpt-4o"), system: resolved.text, prompt: "Can't access billing", ...resolved.toAISDKTelemetry(), }); ``` **What you get:** - **Code-defined, deploy-versioned templates** — define with `prompts.define({ id, model, config, variables, content })`. Every deploy creates a new version visible in the dashboard. Mustache-style placeholders (`{{var}}`, `{{#cond}}...{{/cond}}`) with Zod / ArkType / Valibot-typed variables. - **Dashboard overrides** — change a prompt's text or model from the dashboard without redeploying. Overrides take priority over the deployed "current" version and are environment-scoped (dev / staging / production independent). - **Resolve API** — `prompt.resolve(vars, { version?, label? })` returns the compiled `text`, resolved `model`, `version`, and labels. Standalone `prompts.resolve<typeof handle>(slug, vars)` for cross-file resolution with full type inference on slug and variable shape. - **AI SDK integration** — spread `resolved.toAISDKTelemetry({ ...extra })` into any `generateText` / `streamText` call and every generation span links to the prompt in the dashboard alongside its input variables, model, tokens, and cost. - **`chat.agent` integration** — `chat.prompt.set(resolved)` stores the resolved prompt run-scoped; `chat.toStreamTextOptions({ registry })` pulls `system`, `model` (resolved via the AI SDK provider registry), `temperature` / `maxTokens` / etc., and telemetry into a single spread for `streamText`. - **Management SDK** — `prompts.list()`, `prompts.versions(slug)`, `prompts.promote(slug, version)`, `prompts.createOverride(slug, body)`, `prompts.updateOverride(slug, body)`, `prompts.removeOverride(slug)`, `prompts.reactivateOverride(slug, version)`. - **Dashboard** — prompts list with per-prompt usage sparklines; per-prompt detail with Template / Details / Versions / Generations / Metrics tabs. AI generation spans get a custom inspector showing the linked prompt's metadata, input variables, and template content alongside model, tokens, cost, and the message thread. See [/docs/ai/prompts](https://trigger.dev/docs/ai/prompts) for the full reference — template syntax, version resolution order, override workflow, and type utilities (`PromptHandle`, `PromptIdentifier`, `PromptVariables`). - Adds `onBoot` to `chat.agent` — a lifecycle hook that fires once per worker process picking up the chat. Runs for the initial run, preloaded runs, AND reactive continuation runs (post-cancel, crash, `endRun`, `requestUpgrade`, OOM retry), before any other hook. Use it to initialize `chat.local`, open per-process resources, or re-hydrate state from your DB on continuation — anywhere the SAME run picking up after suspend/resume isn't enough. ([#3543](https://github.com/triggerdotdev/trigger.dev/pull/3543)) ```ts const userContext = chat.local<{ name: string; plan: string }>({ id: "userContext" }); export const myChat = chat.agent({ id: "my-chat", onBoot: async ({ clientData, continuation }) => { const user = await db.user.findUnique({ where: { id: clientData.userId } }); userContext.init({ name: user.name, plan: user.plan }); }, run: async ({ messages, signal }) => streamText({ model: openai("gpt-4o"), messages, abortSignal: signal }), }); ``` Use `onBoot` (not `onChatStart`) for state setup that must run every time a worker picks up the chat — `onChatStart` fires once per chat and won't run on continuation, leaving `chat.local` uninitialized when `run()` tries to use it. - **AI Agents** — run AI SDK chat completions as durable Trigger.dev agents instead of fragile API routes. Define an agent in one function, point `useChat` at it from React, and the conversation survives page refreshes, network blips, and process restarts. ([#3543](https://github.com/triggerdotdev/trigger.dev/pull/3543)) ```ts import { chat } from "@trigger.dev/sdk/ai"; import { streamText } from "ai"; import { openai } from "@ai-sdk/openai"; export const myChat = chat.agent({ id: "my-chat", run: async ({ messages, signal }) => streamText({ model: openai("gpt-4o"), messages, abortSignal: signal }), }); ``` ```tsx import { useChat } from "@ai-sdk/react"; import { useTriggerChatTransport } from "@trigger.dev/sdk/chat/react"; const transport = useTriggerChatTransport({ task: "my-chat", accessToken, startSession }); const { messages, sendMessage } = useChat({ transport }); ``` **What you get:** - **AI SDK `useChat` integration** — a custom [`ChatTransport`](https://sdk.vercel.ai/docs/ai-sdk-ui/transport) (`useTriggerChatTransport`) plugs straight into Vercel AI SDK's `useChat` hook. Text streaming, tool calls, reasoning, and `data-*` parts all work natively over Trigger.dev's realtime streams. No custom API routes needed. - **First-turn fast path (`chat.headStart`)** — opt-in handler that runs the first turn's `streamText` step in your warm server process while the agent run boots in parallel, cutting cold-start TTFC by roughly half (measured 2801ms → 1218ms on `claude-sonnet-4-6`). The agent owns step 2+ (tool execution, persistence, hooks) so heavy deps stay where they belong. Web Fetch handler works natively in Next.js, Hono, SvelteKit, Remix, Workers, etc.; bridge to Express/Fastify/Koa via `chat.toNodeListener`. New `@trigger.dev/sdk/chat-server` subpath. - **Multi-turn durability via Sessions** — every chat is backed by a durable Session that outlives any individual run. Conversations resume across page refreshes, idle timeout, crashes, and deploys; `resume: true` reconnects via `lastEventId` so clients only see new chunks. `sessions.list` enumerates chats for inbox-style UIs. - **Auto-accumulated history, delta-only wire** — the backend accumulates the full conversation across turns; clients only ship the new message each turn. Long chats never hit the 512 KiB body cap. Register `hydrateMessages` to be the source of truth yourself. - **Lifecycle hooks** — `onPreload`, `onChatStart`, `onValidateMessages`, `hydrateMessages`, `onTurnStart`, `onBeforeTurnComplete`, `onTurnComplete`, `onChatSuspend`, `onChatResume` — for persistence, validation, and post-turn work. - **Stop generation** — client-driven `transport.stopGeneration(chatId)` aborts mid-stream; the run stays alive for the next message, partial response is captured, and aborted parts (stuck `partial-call` tools, in-progress reasoning) are auto-cleaned. - **Tool approvals (HITL)** — tools with `needsApproval: true` pause until the user approves or denies via `addToolApprovalResponse`. The runtime reconciles the updated assistant message by ID and continues `streamText`. - **Steering and background injection** — `pendingMessages` injects user messages between tool-call steps so users can steer the agent mid-execution; `chat.inject()` + `chat.defer()` adds context from background work (self-review, RAG, safety checks) between turns. - **Actions** — non-turn frontend commands (undo, rollback, regenerate, edit) sent via `transport.sendAction`. Fire `hydrateMessages` + `onAction` only — no turn hooks, no `run()`. `onAction` can return a `StreamTextResult` for a model response, or `void` for side-effect-only. - **Typed state primitives** — `chat.local<T>` for per-run state accessible from hooks, `run()`, tools, and subtasks (auto-serialized through `ai.toolExecute`); `chat.store` for typed shared data between agent and client; `chat.history` for reading and mutating the message chain; `clientDataSchema` for typed `clientData` in every hook. - **`chat.toStreamTextOptions()`** — one spread into `streamText` wires up versioned system [Prompts](https://trigger.dev/docs/ai/prompts), model resolution, telemetry metadata, compaction, steering, and background injection. - **Multi-tab coordination** — `multiTab: true` + `useMultiTabChat` prevents duplicate sends and syncs state across browser tabs via `BroadcastChannel`. Non-active tabs go read-only with live updates. - **Network resilience** — built-in indefinite retry with bounded backoff, reconnect on `online` / tab refocus / bfcache restore, `Last-Event-ID` mid-stream resume. No app code needed. See [/docs/ai-chat](https://trigger.dev/docs/ai-chat/overview) for the full surface — quick start, three backend approaches (`chat.agent`, `chat.createSession`, raw task), persistence and code-sandbox patterns, type-level guides, and API reference. - Add read primitives to `chat.history` for HITL flows: `getPendingToolCalls()`, `getResolvedToolCalls()`, `extractNewToolResults(message)`, `getChain()`, and `findMessage(messageId)`. These lift the accumulator-walking logic that customers building human-in-the-loop tools were re-implementing into the SDK. ([#3543](https://github.com/triggerdotdev/trigger.dev/pull/3543)) Use `getPendingToolCalls()` to gate fresh user turns while a tool call is awaiting an answer. Use `extractNewToolResults(message)` to dedup tool results when persisting to your own store — the helper returns only the parts whose `toolCallId` is not already resolved on the chain. ```ts const pending = chat.history.getPendingToolCalls(); if (pending.length > 0) { // an addToolOutput is expected before a new user message } onTurnComplete: async ({ responseMessage }) => { const newResults = chat.history.extractNewToolResults(responseMessage); for (const r of newResults) { await db.toolResults.upsert({ id: r.toolCallId, output: r.output, errorText: r.errorText }); } }; ``` - **Sessions** — a durable, run-aware stream channel keyed on a stable `externalId`. A Session is the unit of state that owns a multi-run conversation: messages flow through `.in`, responses through `.out`, both survive run boundaries. Sessions back the new `chat.agent` runtime, and you can build on them directly for any pattern that needs durable bi-directional streaming across runs. ([#3542](https://github.com/triggerdotdev/trigger.dev/pull/3542)) ```ts import { sessions, tasks } from "@trigger.dev/sdk"; // Trigger a task and subscribe to its session output in one call const { runId, stream } = await tasks.triggerAndSubscribe("my-task", payload, { externalId: "user-456", }); for await (const chunk of stream) { // ... } // Enumerate existing sessions (powers inbox-style UIs without a separate index) for await (const s of sessions.list({ type: "chat.agent", tag: "user:user-456" })) { console.log(s.id, s.externalId, s.createdAt, s.closedAt); } ``` See [/docs/ai-chat/overview](https://trigger.dev/docs/ai-chat/overview) for the full surface — Sessions powers the durable, resumable chat runtime described there. ### Patch Changes - Add Agent Skills for `chat.agent`. Drop a folder with a `SKILL.md` and any helper scripts/references next to your task code, register it with `skills.define({ id, path })`, and the CLI bundles it into the deploy image automatically — no `trigger.config.ts` changes. The agent gets a one-line summary in its system prompt and discovers full instructions on demand via `loadSkill`, with `bash` and `readFile` tools scoped per-skill (path-traversal guards, output caps, abort-signal propagation). ([#3543](https://github.com/triggerdotdev/trigger.dev/pull/3543)) ```ts const pdfSkill = skills.define({ id: "pdf-extract", path: "./skills/pdf-extract" }); chat.skills.set([await pdfSkill.local()]); ``` Built on the [AI SDK cookbook pattern](https://ai-sdk.dev/cookbook/guides/agent-skills) — portable across providers. SDK + CLI only for now; dashboard-editable `SKILL.md` text is on the roadmap. - Add `ai.toolExecute(task)` so you can wire a Trigger subtask in as the `execute` handler of an AI SDK `tool()` while defining `description` and `inputSchema` yourself — useful when you want full control over the tool surface and just need Trigger's subtask machinery for the body. ([#3546](https://github.com/triggerdotdev/trigger.dev/pull/3546)) ```ts const myTool = tool({ description: "...", inputSchema: z.object({ ... }), execute: ai.toolExecute(mySubtask), }); ``` `ai.tool(task)` (`toolFromTask`) keeps doing the all-in-one wrap and now aligns its return type with AI SDK's `ToolSet`. Minimum `ai` peer raised to `^6.0.116` to avoid cross-version `ToolSet` mismatches in monorepos. - Stamp `gen_ai.conversation.id` (the chat id) on every span and metric emitted from inside a `chat.task` or `chat.agent` run. Lets you filter dashboard spans, runs, and metrics by the chat conversation that produced them — independent of the run boundary, so multi-run chats correlate cleanly. No code changes required on the user side. ([#3543](https://github.com/triggerdotdev/trigger.dev/pull/3543)) - Type `chat.createStartSessionAction` against your chat agent so `clientData` is typed end-to-end on the first turn: ([#3684](https://github.com/triggerdotdev/trigger.dev/pull/3684)) ```ts import { chat } from "@trigger.dev/sdk/ai"; import type { myChat } from "@/trigger/chat"; export const startChatSession = chat.createStartSessionAction<typeof myChat>("my-chat"); // In the browser, threaded from the transport's typed startSession callback: const transport = useTriggerChatTransport<typeof myChat>({ task: "my-chat", startSession: ({ chatId, clientData }) => startChatSession({ chatId, clientData }), // ... }); ``` `ChatStartSessionParams` gains a typed `clientData` field — folded into the first run's `payload.metadata` so `onPreload` / `onChatStart` see the same shape per-turn `metadata` carries via the transport. The opaque session-level `metadata` field is unchanged. - Unit-test `chat.agent` definitions offline with `mockChatAgent` from `@trigger.dev/sdk/ai/test`. Drives a real agent's turn loop in-process — no network, no task runtime — so you can send messages, actions, and stop signals via driver methods, inspect captured output chunks, and verify hooks fire. Pairs with `MockLanguageModelV3` from `ai/test` for model mocking. `setupLocals` lets you pre-seed `locals` (DB clients, service stubs) before `run()` starts. ([#3543](https://github.com/triggerdotdev/trigger.dev/pull/3543)) The broader `runInMockTaskContext` harness it's built on lives at `@trigger.dev/core/v3/test` — useful for unit-testing any task code, not just chat. - Add `region` to the runs list / retrieve API: filter runs by region (`runs.list({ region: "..." })` / `filter[region]=<masterQueue>`) and read each run's executing region from the new `region` field on the response. ([#3612](https://github.com/triggerdotdev/trigger.dev/pull/3612)) - Updated dependencies: - `@trigger.dev/core@4.5.0-rc.0` ## @trigger.dev/build@4.5.0-rc.0 ### Patch Changes - Add Agent Skills for `chat.agent`. Drop a folder with a `SKILL.md` and any helper scripts/references next to your task code, register it with `skills.define({ id, path })`, and the CLI bundles it into the deploy image automatically — no `trigger.config.ts` changes. The agent gets a one-line summary in its system prompt and discovers full instructions on demand via `loadSkill`, with `bash` and `readFile` tools scoped per-skill (path-traversal guards, output caps, abort-signal propagation). ([#3543](https://github.com/triggerdotdev/trigger.dev/pull/3543)) ```ts const pdfSkill = skills.define({ id: "pdf-extract", path: "./skills/pdf-extract" }); chat.skills.set([await pdfSkill.local()]); ``` Built on the [AI SDK cookbook pattern](https://ai-sdk.dev/cookbook/guides/agent-skills) — portable across providers. SDK + CLI only for now; dashboard-editable `SKILL.md` text is on the roadmap. - Updated dependencies: - `@trigger.dev/core@4.5.0-rc.0` ## trigger.dev@4.5.0-rc.0 ### Patch Changes - Add Agent Skills for `chat.agent`. Drop a folder with a `SKILL.md` and any helper scripts/references next to your task code, register it with `skills.define({ id, path })`, and the CLI bundles it into the deploy image automatically — no `trigger.config.ts` changes. The agent gets a one-line summary in its system prompt and discovers full instructions on demand via `loadSkill`, with `bash` and `readFile` tools scoped per-skill (path-traversal guards, output caps, abort-signal propagation). ([#3543](https://github.com/triggerdotdev/trigger.dev/pull/3543)) ```ts const pdfSkill = skills.define({ id: "pdf-extract", path: "./skills/pdf-extract" }); chat.skills.set([await pdfSkill.local()]); ``` Built on the [AI SDK cookbook pattern](https://ai-sdk.dev/cookbook/guides/agent-skills) — portable across providers. SDK + CLI only for now; dashboard-editable `SKILL.md` text is on the roadmap. - Add `TRIGGER_BUILD_SKIP_REWRITE_TIMESTAMP=1` escape hatch for local self-hosted builds whose buildx driver doesn't support `rewrite-timestamp` alongside push (e.g. orbstack's default `docker` driver). ([#3618](https://github.com/triggerdotdev/trigger.dev/pull/3618)) - The CLI MCP server's agent-chat tools (`start_agent_chat`, `send_agent_message`, `close_agent_chat`) now run on the new Sessions primitive, so AI assistants driving a `chat.agent` get the same idempotent-by-`chatId`, durable-across-runs behavior the browser transport gets. Required PAT scopes go from `write:inputStreams` to `read:sessions` + `write:sessions`. ([#3546](https://github.com/triggerdotdev/trigger.dev/pull/3546)) - MCP `list_runs` tool: add a `region` filter input and surface each run's executing region in the formatted summary. ([#3612](https://github.com/triggerdotdev/trigger.dev/pull/3612)) - Updated dependencies: - `@trigger.dev/core@4.5.0-rc.0` - `@trigger.dev/build@4.5.0-rc.0` - `@trigger.dev/schema-to-json@4.5.0-rc.0` ## @trigger.dev/core@4.5.0-rc.0 ### Patch Changes - Add Agent Skills for `chat.agent`. Drop a folder with a `SKILL.md` and any helper scripts/references next to your task code, register it with `skills.define({ id, path })`, and the CLI bundles it into the deploy image automatically — no `trigger.config.ts` changes. The agent gets a one-line summary in its system prompt and discovers full instructions on demand via `loadSkill`, with `bash` and `readFile` tools scoped per-skill (path-traversal guards, output caps, abort-signal propagation). ([#3543](https://github.com/triggerdotdev/trigger.dev/pull/3543)) ```ts const pdfSkill = skills.define({ id: "pdf-extract", path: "./skills/pdf-extract" }); chat.skills.set([await pdfSkill.local()]); ``` Built on the [AI SDK cookbook pattern](https://ai-sdk.dev/cookbook/guides/agent-skills) — portable across providers. SDK + CLI only for now; dashboard-editable `SKILL.md` text is on the roadmap. - Reject overlong `idempotencyKey` values at the API boundary so they no longer trip an internal size limit on the underlying unique index and surface as a generic 500. Inputs are capped at 2048 characters — well above what `idempotencyKeys.create()` produces (a 64-character hash) and above any realistic raw key. Applies to `tasks.trigger`, `tasks.batchTrigger`, `batch.create` (Phase 1 streaming batches), `wait.createToken`, `wait.forDuration`, and the input/session stream waitpoint endpoints. Over-limit requests now return a structured 400 instead. ([#3560](https://github.com/triggerdotdev/trigger.dev/pull/3560)) - **AI Agents** — run AI SDK chat completions as durable Trigger.dev agents instead of fragile API routes. Define an agent in one function, point `useChat` at it from React, and the conversation survives page refreshes, network blips, and process restarts. ([#3543](https://github.com/triggerdotdev/trigger.dev/pull/3543)) ```ts import { chat } from "@trigger.dev/sdk/ai"; import { streamText } from "ai"; import { openai } from "@ai-sdk/openai"; export const myChat = chat.agent({ id: "my-chat", run: async ({ messages, signal }) => streamText({ model: openai("gpt-4o"), messages, abortSignal: signal }), }); ``` ```tsx import { useChat } from "@ai-sdk/react"; import { useTriggerChatTransport } from "@trigger.dev/sdk/chat/react"; const transport = useTriggerChatTransport({ task: "my-chat", accessToken, startSession }); const { messages, sendMessage } = useChat({ transport }); ``` **What you get:** - **AI SDK `useChat` integration** — a custom [`ChatTransport`](https://sdk.vercel.ai/docs/ai-sdk-ui/transport) (`useTriggerChatTransport`) plugs straight into Vercel AI SDK's `useChat` hook. Text streaming, tool calls, reasoning, and `data-*` parts all work natively over Trigger.dev's realtime streams. No custom API routes needed. - **First-turn fast path (`chat.headStart`)** — opt-in handler that runs the first turn's `streamText` step in your warm server process while the agent run boots in parallel, cutting cold-start TTFC by roughly half (measured 2801ms → 1218ms on `claude-sonnet-4-6`). The agent owns step 2+ (tool execution, persistence, hooks) so heavy deps stay where they belong. Web Fetch handler works natively in Next.js, Hono, SvelteKit, Remix, Workers, etc.; bridge to Express/Fastify/Koa via `chat.toNodeListener`. New `@trigger.dev/sdk/chat-server` subpath. - **Multi-turn durability via Sessions** — every chat is backed by a durable Session that outlives any individual run. Conversations resume across page refreshes, idle timeout, crashes, and deploys; `resume: true` reconnects via `lastEventId` so clients only see new chunks. `sessions.list` enumerates chats for inbox-style UIs. - **Auto-accumulated history, delta-only wire** — the backend accumulates the full conversation across turns; clients only ship the new message each turn. Long chats never hit the 512 KiB body cap. Register `hydrateMessages` to be the source of truth yourself. - **Lifecycle hooks** — `onPreload`, `onChatStart`, `onValidateMessages`, `hydrateMessages`, `onTurnStart`, `onBeforeTurnComplete`, `onTurnComplete`, `onChatSuspend`, `onChatResume` — for persistence, validation, and post-turn work. - **Stop generation** — client-driven `transport.stopGeneration(chatId)` aborts mid-stream; the run stays alive for the next message, partial response is captured, and aborted parts (stuck `partial-call` tools, in-progress reasoning) are auto-cleaned. - **Tool approvals (HITL)** — tools with `needsApproval: true` pause until the user approves or denies via `addToolApprovalResponse`. The runtime reconciles the updated assistant message by ID and continues `streamText`. - **Steering and background injection** — `pendingMessages` injects user messages between tool-call steps so users can steer the agent mid-execution; `chat.inject()` + `chat.defer()` adds context from background work (self-review, RAG, safety checks) between turns. - **Actions** — non-turn frontend commands (undo, rollback, regenerate, edit) sent via `transport.sendAction`. Fire `hydrateMessages` + `onAction` only — no turn hooks, no `run()`. `onAction` can return a `StreamTextResult` for a model response, or `void` for side-effect-only. - **Typed state primitives** — `chat.local<T>` for per-run state accessible from hooks, `run()`, tools, and subtasks (auto-serialized through `ai.toolExecute`); `chat.store` for typed shared data between agent and client; `chat.history` for reading and mutating the message chain; `clientDataSchema` for typed `clientData` in every hook. - **`chat.toStreamTextOptions()`** — one spread into `streamText` wires up versioned system [Prompts](https://trigger.dev/docs/ai/prompts), model resolution, telemetry metadata, compaction, steering, and background injection. - **Multi-tab coordination** — `multiTab: true` + `useMultiTabChat` prevents duplicate sends and syncs state across browser tabs via `BroadcastChannel`. Non-active tabs go read-only with live updates. - **Network resilience** — built-in indefinite retry with bounded backoff, reconnect on `online` / tab refocus / bfcache restore, `Last-Event-ID` mid-stream resume. No app code needed. See [/docs/ai-chat](https://trigger.dev/docs/ai-chat/overview) for the full surface — quick start, three backend approaches (`chat.agent`, `chat.createSession`, raw task), persistence and code-sandbox patterns, type-level guides, and API reference. - Stamp `gen_ai.conversation.id` (the chat id) on every span and metric emitted from inside a `chat.task` or `chat.agent` run. Lets you filter dashboard spans, runs, and metrics by the chat conversation that produced them — independent of the run boundary, so multi-run chats correlate cleanly. No code changes required on the user side. ([#3543](https://github.com/triggerdotdev/trigger.dev/pull/3543)) - Fix `LocalsKey<T>` type incompatibility across dual-package builds. The phantom value-type brand no longer uses a module-level `unique symbol`, so a single TypeScript compilation that resolves the type from both the ESM and CJS outputs (which can happen under certain pnpm hoisting layouts) no longer sees two structurally-incompatible variants of the same type. ([#3626](https://github.com/triggerdotdev/trigger.dev/pull/3626)) - Unit-test `chat.agent` definitions offline with `mockChatAgent` from `@trigger.dev/sdk/ai/test`. Drives a real agent's turn loop in-process — no network, no task runtime — so you can send messages, actions, and stop signals via driver methods, inspect captured output chunks, and verify hooks fire. Pairs with `MockLanguageModelV3` from `ai/test` for model mocking. `setupLocals` lets you pre-seed `locals` (DB clients, service stubs) before `run()` starts. ([#3543](https://github.com/triggerdotdev/trigger.dev/pull/3543)) The broader `runInMockTaskContext` harness it's built on lives at `@trigger.dev/core/v3/test` — useful for unit-testing any task code, not just chat. - Retry `TASK_PROCESS_SIGSEGV` task crashes under the user's retry policy instead of failing the run on the first segfault. SIGSEGV in Node tasks is frequently non-deterministic (native addon races, JIT/GC interaction, near-OOM in native code, host issues), so retrying on a fresh process often succeeds. The retry is gated by the task's existing `retry` config + `maxAttempts` — same path `TASK_PROCESS_SIGTERM` and uncaught exceptions already use — so tasks without a retry policy still fail fast. ([#3552](https://github.com/triggerdotdev/trigger.dev/pull/3552)) - Add `region` to the runs list / retrieve API: filter runs by region (`runs.list({ region: "..." })` / `filter[region]=<masterQueue>`) and read each run's executing region from the new `region` field on the response. ([#3612](https://github.com/triggerdotdev/trigger.dev/pull/3612)) - **Sessions** — a durable, run-aware stream channel keyed on a stable `externalId`. A Session is the unit of state that owns a multi-run conversation: messages flow through `.in`, responses through `.out`, both survive run boundaries. Sessions back the new `chat.agent` runtime, and you can build on them directly for any pattern that needs durable bi-directional streaming across runs. ([#3542](https://github.com/triggerdotdev/trigger.dev/pull/3542)) ```ts import { sessions, tasks } from "@trigger.dev/sdk"; // Trigger a task and subscribe to its session output in one call const { runId, stream } = await tasks.triggerAndSubscribe("my-task", payload, { externalId: "user-456", }); for await (const chunk of stream) { // ... } // Enumerate existing sessions (powers inbox-style UIs without a separate index) for await (const s of sessions.list({ type: "chat.agent", tag: "user:user-456" })) { console.log(s.id, s.externalId, s.createdAt, s.closedAt); } ``` See [/docs/ai-chat/overview](https://trigger.dev/docs/ai-chat/overview) for the full surface — Sessions powers the durable, resumable chat runtime described there. ## @trigger.dev/plugins@4.5.0-rc.0 ### Patch Changes - The public interfaces for a plugin system. Initially consolidated authentication and authorization interfaces. ([#3499](https://github.com/triggerdotdev/trigger.dev/pull/3499)) - Updated dependencies: - `@trigger.dev/core@4.5.0-rc.0` ## @trigger.dev/python@4.5.0-rc.0 ### Patch Changes - Updated dependencies: - `@trigger.dev/sdk@4.5.0-rc.0` - `@trigger.dev/core@4.5.0-rc.0` - `@trigger.dev/build@4.5.0-rc.0` ## @trigger.dev/react-hooks@4.5.0-rc.0 ### Patch Changes - Updated dependencies: - `@trigger.dev/core@4.5.0-rc.0` ## @trigger.dev/redis-worker@4.5.0-rc.0 ### Patch Changes - Add MollifierBuffer and MollifierDrainer primitives for trigger burst smoothing. ([#3614](https://github.com/triggerdotdev/trigger.dev/pull/3614)) MollifierBuffer (`accept`, `pop`, `ack`, `requeue`, `fail`, `evaluateTrip`) is a per-env FIFO over Redis with atomic Lua transitions for status tracking. `evaluateTrip` is a sliding-window trip evaluator the webapp gate uses to detect per-env trigger bursts. MollifierDrainer pops entries through a polling loop with a user-supplied handler. The loop survives transient Redis errors via capped exponential backoff (up to 5s), and per-env pop failures don't poison the rest of the batch — one env's blip is logged and counted as failed for that tick. Rotation is two-level: orgs at the top, envs within each org. The buffer maintains `mollifier:orgs` and `mollifier:org-envs:${orgId}` atomically with per-env queues, so the drainer walks orgs → envs directly without an in-memory cache. The `maxOrgsPerTick` option (default 500) caps how many orgs are scheduled per tick; for each picked org, one env is popped (rotating round-robin within the org). An org with N envs gets the same per-tick scheduling slot as an org with 1 env, so tenant-level drainage throughput is determined by org count rather than env count. - Updated dependencies: - `@trigger.dev/core@4.5.0-rc.0` ## @trigger.dev/rsc@4.5.0-rc.0 ### Patch Changes - Updated dependencies: - `@trigger.dev/core@4.5.0-rc.0` ## @trigger.dev/schema-to-json@4.5.0-rc.0 ### Patch Changes - Updated dependencies: - `@trigger.dev/core@4.5.0-rc.0` </details> Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> |
||
|
Eric Allam
|
9ff410bfa4 |
feat(sdk): type chat.createStartSessionAction against your chat agent (#3684)
## Summary
Type `chat.createStartSessionAction` against the chat agent so
`clientData` is typed end-to-end on the first turn. Closes the gap where
`useTriggerChatTransport`'s `startSession` callback already hands you a
typed `clientData` (via the transport generic) but the server-side
action couldn't accept it without untyped routing through the `metadata`
field.
## Design
`ChatStartSessionParams` gains a typed `clientData` field via the new
generic:
```ts
export type ChatStartSessionParams<TChat extends AnyTask = AnyTask> = {
chatId: string;
clientData?: InferChatClientData<TChat>;
triggerConfig?: Partial<SessionTriggerConfig>;
metadata?: Record<string, unknown>;
};
function createChatStartSessionAction<TChat extends AnyTask = AnyTask>(
taskId: string,
options?: CreateChatStartSessionActionOptions
): (params: ChatStartSessionParams<TChat>) => Promise<ChatStartSessionResult>
```
When provided, `clientData` is folded into the first run's
`triggerConfig.basePayload.metadata`, so `onPreload` / `onChatStart` see
the same shape per-turn `metadata` carries via the transport. The opaque
session-level `metadata` field stays exactly as before — it lands on the
Session row, not the run payload.
## Usage
```ts
// actions.ts
import { chat } from "@trigger.dev/sdk/ai";
import type { myChat } from "@/trigger/chat";
export const startChatSession = chat.createStartSessionAction<typeof myChat>("my-chat");
```
```tsx
// Chat.tsx
const transport = useTriggerChatTransport<typeof myChat>({
task: "my-chat",
accessToken: ({ chatId }) => mintChatAccessToken(chatId),
startSession: ({ chatId, clientData }) =>
startChatSession({ chatId, clientData }),
});
```
## Test plan
- [x] `pnpm run build --filter @trigger.dev/sdk` passes
- [ ] Verify a `chat.agent` with `clientDataSchema` reads the typed
clientData from `onPreload` payload metadata on the first turn
|
||
|
Eric Allam
|
d343727021 |
fix(webapp,sdk): keep chat.agent snapshots on one object store (#3679)
(`OBJECT_STORE_BASE_URL`) and a named protocol provider
(`OBJECT_STORE_DEFAULT_PROTOCOL=s3`), chat.agent session snapshot writes
landed in the named provider but reads fell through to the default — so
the recovery boot couldn't find the snapshot it had just written.
After a mid-stream cancel, the missing snapshot triggered a fallback
replay path that dropped the user's follow-up message, leaving the chat
stuck in `submitted` indefinitely.
Fix:
- New `/api/v1/sessions/:id/snapshot-url` route handles PUT + GET
symmetrically — both prefix unprefixed keys with
`OBJECT_STORE_DEFAULT_PROTOCOL` so they always round-trip through the
same store.
- `Session.chatSnapshotStoragePath` persists the resolved URI on first
write so future protocol changes don't strand existing snapshots.
Reads prefer the stored URI and fall back to the computed default for
pre-column sessions.
- SDK calls `createChatSnapshotUploadUrl` / `getChatSnapshotUrl`; the
generic v1/v2 packets endpoints are unchanged.
## Test plan
- [x] Configure local with two providers (R2 default + MinIO `s3` named)
and `OBJECT_STORE_DEFAULT_PROTOCOL=s3`.
- [x] Reproduce hang: send a message, cancel mid-stream, send another —
without the fix it hangs in `submitted`; with the fix it streams.
- [x] Snapshot lands in the `s3`-protocol bucket and
`Session.chatSnapshotStoragePath` is set after first write.
- [x] SDK unit tests pass; webapp typecheck passes.
|
||
|
Eric Allam
|
f91b96efb7 | feat(sdk,core): preserve chat.agent context after cancel / OOM / crash (#3671) | ||
|
Eric Allam
|
12d2125148 |
fix(sdk,core,build): SDK hardening pass (#3670)
## Summary Five hardening fixes across `@trigger.dev/sdk`, `@trigger.dev/core`, and `@trigger.dev/build`. - `tasks.triggerAndSubscribe` now forwards caller `requestOptions` (custom API keys, per-request overrides) to the underlying `apiClient.triggerTask` call instead of silently dropping them. - `SSEStreamSubscription` no longer retries permanent client errors forever. The default `nonRetryableStatuses` widens from `[404, 410]` to `[400, 404, 409, 410, 422]`, so a malformed session-stream request fails fast instead of busy-looping under bounded backoff. - Session writer falls back to manually wiring the caller's `AbortSignal` on Node 18, where `AbortSignal.any` is unavailable. Caller-driven cancellation now propagates on every supported runtime. - `TriggerChatTransport` throws immediately when a `chat.handover` response is missing `X-Trigger-Chat-Access-Token`, instead of silently downgrading every subsequent turn back to the handover path. `dispose()` aborts every active `session.out` subscription before tearing the coordinator down, so unmount/navigation no longer leaves SSE readers in flight. - Removed the experimental `@trigger.dev/build/extensions/secureExec` build extension. It will return alongside the sandbox feature it was built to support. ## Test plan - [ ] `pnpm run build --filter @trigger.dev/sdk --filter @trigger.dev/core --filter @trigger.dev/build` - [ ] `pnpm --filter @trigger.dev/sdk test --run` (183 tests, including chat / chat-server / sessions / handover) - [ ] `pnpm --filter @trigger.dev/core test --run` - [ ] Manually trigger a `chat.handover` whose response strips `X-Trigger-Chat-Access-Token`, and confirm the transport throws synchronously rather than degrading. - [ ] Unmount a chat UI mid-stream and confirm the active `session.out` SSE connection closes immediately. |
||
|
Eric Allam
|
427d9e078a |
feat(sdk): functional baseURL and fetch override on chat transports (#3655)
## Summary
`TriggerChatTransport`, `AgentChat`, and `chat.createStartSessionAction`
now accept a string-or-function `baseURL` so callers can route per
endpoint — e.g. `.in/append` through a trusted edge proxy while keeping
`.out` SSE direct. The same surfaces add a `fetch` override for header
injection, custom retries, or proxy rewrites that go beyond URL routing.
SSE GETs are covered too via a new `fetchClient` option on
`SSEStreamSubscription`.
```ts
// TriggerChatTransport / AgentChat — endpoints: "in" | "out"
baseURL: ({ endpoint }) =>
endpoint === "out" ? DIRECT : PROXY,
fetch: (url, init, ctx) => {
init.headers = new Headers(init.headers);
init.headers.set("traceparent", currentTraceparent());
return globalThis.fetch(url, init);
},
// chat.createStartSessionAction — endpoints: "sessions" | "auth"
chat.createStartSessionAction("my-agent", {
baseURL: ({ endpoint }) => (endpoint === "sessions" ? PROXY : DIRECT),
});
```
`streamBaseURL` on `TriggerChatTransport` is kept as a backwards-compat
alias and continues to win for the `"out"` endpoint when set.
Plain-string `baseURL` still applies to every endpoint, matching prior
behavior.
|
||
|
Eric Allam
|
8b98e21b4b |
fix(sdk,core): cache realtime-stream credentials per slot with refresh on writer failure (#3658)
## Summary Hot-loop writers — `streams.writer` / `streams.pipe` on the run-scoped side, `chat.response.write` / `chat.stream.*` on the session side — were issuing a fresh `PUT` to mint S2 credentials for every chunk. On run streams, each PUT also pushed the streamId onto `TaskRun.realtimeStreams`, so a chat-agent turn writing N chunks produced N PUTs and N duplicate array pushes against the same row. The SDK now caches the initialize response per cache slot: `(runId, key)` for run streams, the session id for session streams. First call PUTs as before; subsequent calls reuse the cached promise. Hot-loop writers do one PUT per slot for the lifetime of the cache. S2 access tokens have a 1-day TTL. If a writer's `wait()` rejects (auth error, expired token, network blip), the cache evicts the matching slot so the next call re-PUTs and mints fresh credentials, identity-checked so a concurrent caller's fresh promise isn't accidentally cleared. ## chat.agent guardrail `streams.pipe / writer / append / read` called inside a `chat.agent` run now logs a one-time warning pointing at `chat.response.write` / `chat.stream.*` — `streams.*` is run-scoped and isn't visible on the chat session. The ai-chat docs are updated to drop the old guidance toward run-scoped streams. |
||
|
Daniel Sutton
|
906d5fafb6 |
feat(mollifier): trigger burst smoothing — Phase 1 (monitoring) (#3614)
## Summary - Introduce the Mollifier: a Redis-backed buffer for `trigger()` API calls during traffic spikes, with a per-env trip evaluator and a drainer ack-loop. - Phase 1 is dual-write monitoring — every mollified trigger is buffered to Redis AND continues to `engine.trigger`. No customer-facing behaviour change. - Telemetry events: `mollifier.would_mollify`, `mollifier.buffered`, `mollifier.drained`, plus the `mollifier.decisions` counter. - Gated behind a feature flag (default off). ## Test plan - [x] `pnpm run test --filter @trigger.dev/redis-worker` - [x] `pnpm run test --filter webapp -- mollifier` - [x] Manual: with flag off, no behaviour change vs main - [x] Manual: with flag on + threshold lowered, observe `mollifier.buffered` + `mollifier.drained` log pairs with matching `runId` --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
Eric Allam
|
82853debea |
feat(webapp,core,sdk,cli): bound session.out via per-turn trim (#3644)
## Summary Long-running chat agents were filling `session.out` forever — every `chat.agent` turn appended to the same S2 stream with no trim, and the Sessions dashboard re-streamed the entire history from `seq_num=0` on every page load. After this change the agent appends an S2 `trim` command record after each `trigger:turn-complete`, pointing back at the previous turn-complete's seq_num. `session.out` stays roughly one turn long at steady state, regardless of session age. `trigger:turn-complete` and `trigger:upgrade-required` move from `chunk.type`-shaped data records into header-form control records under a uniform `trigger-control` namespace. Built-in transports (`TriggerChatTransport`, `AgentChat`, the dashboard's `AgentView`) handle the new shape transparently. Custom transports need a one-line filter on the `trigger-control` header — see the rewritten "Records on session.out" section in the client-protocol docs. The Sessions detail page in the dashboard fetches the agent's per-turn S3 snapshot via a presigned URL and seeds the transcript view, then SSE-tails from the snapshot's `lastOutEventId`. Bandwidth and time-to-first-render scale with unread turns instead of session lifetime. Resume contract is now explicit: single-turn-boundary resume always works (the prior turn-complete is still on the stream), the S2 trim is eventually consistent over 10-60s, and multi-turn-away resume falls back to a snapshot reload. |
||
|
Eric Allam
|
55fa2d4967 |
fix(cli): TRIGGER_BUILD_SKIP_REWRITE_TIMESTAMP escape hatch for local self-hosted builds (#3618)
## Summary Local self-hosted deploys (`trigger deploy --local-build --push --builder orbstack` or any other buildx setup using the **docker** driver) fail at the push step with: ``` ERROR: failed to build: failed to solve: exporter option "rewrite-timestamp" conflicts with "unpack" ``` The docker driver auto-enables `unpack=true` when pushing, and that's incompatible with `rewrite-timestamp` (which the CLI sets for reproducible-build hashing). Adds a simple env-var opt-out so contributors can keep using their default builder. The flag is only read by the local-build code path; remote/cloud builds are unaffected. ```bash TRIGGER_BUILD_SKIP_REWRITE_TIMESTAMP=1 \ pnpm exec trigger deploy --profile default --local-build --push --builder orbstack ``` The trade-off: skipping `rewrite-timestamp` means layer timestamps reflect actual build time, so two identical builds produce different layer hashes. Fine for a local-dev registry; the only real consumer of timestamp-stability is registry-layer cache hit rates. ## Test plan - [x] Manual: ran `trigger deploy --profile default --local-build --push --builder orbstack` against the localhost webapp + a local Docker registry on port 5001 — first failed with the rewrite-timestamp/unpack error, then succeeded after setting `TRIGGER_BUILD_SKIP_REWRITE_TIMESTAMP=1`. - [x] Full chat.agent smoke sweep (15 tests, including suspend/resume, deepResearch subtask, AgentChat orchestrator) against the deployed image — all pass. 🤖 Generated with [Claude Code](https://claude.com/claude-code) |
||
|
Eric Allam
|
627e059298 |
feat(trigger-sdk): add streamBaseURL to TriggerChatTransport (#3641)
`TriggerChatTransport` had a single `baseURL` option covering both the
`.in/append` POSTs and the long-lived `.out` SSE subscription. Customers
wanting to route the SSE through a proxy (e.g. a Cloudflare worker
capturing JA4 fingerprints for bot detection) had to send every append
through the proxy too, adding a hop to every user message.
New optional `streamBaseURL` overrides the SSE base URL only; appends
keep using `baseURL`. Falls back to `baseURL` when unset, so existing
transports are unchanged.
```ts
const transport = new TriggerChatTransport({
task: "ai-chat",
baseURL: "https://api.trigger.dev",
streamBaseURL: "https://chat-proxy.example.com",
accessToken,
startSession,
});
```
Verified with a new test in `chat.test.ts` that asserts `.in/append`
routes through `baseURL` and `.out` SSE routes through `streamBaseURL`.
All existing tests still pass.
|
||
|
Eric Allam
|
dfa3ede209 |
feat(ci): support release candidates via changesets pre mode (#3628)
## Summary Enables shipping `X.Y.Z-rc.N` prereleases of `@trigger.dev/*` via changesets pre mode. RCs publish under the `rc` npm dist-tag, never claim `latest`, and don't trigger marketing-site changelog PRs. The plumbing is hyphen-in-version detection in `release.yml` — no separate workflow, no opt-in flag at publish time. Validated end-to-end against a sandbox repo (real npm publishes, Docker builds, Helm chart pushes, GitHub releases) before porting back. Full RC lifecycle tested: pre enter → rc.0 → iterate to rc.1 → pre exit → stable. Plus interaction with the existing release-branch hotfix flow. ## What changes ### `release.yml` - New `is_prerelease` output (hyphen-in-version) - GitHub release adds `--prerelease` flag for RC publishes (Pre-release badge, not Latest) - `dispatch-changelog` job gated on `is_prerelease != 'true'` — no marketing-site PR per RC ### Docker workflows - Removes the `:v4-beta` floating tag entirely from `publish-webapp.yml` and `publish-worker-v4.yml`. v4 is GA; the tag is a misnomer and is already inconsistent with the npm side (npm `v4-beta` dist-tag was frozen at 4.0.4 months ago while Docker `:v4-beta` kept bumping). Self-hosters should pin to a versioned tag going forward — the last value of `:v4-beta` stays frozen wherever it currently points. ### CLI version-check fix (`packages/cli-v3/src/utilities/initialBanner.ts`) Switches the "new version available" comparison from JavaScript `localeCompare` to `semver.lt`. The old comparison handled `X.Y.Z-rc.N` vs `X.Y.Z` incorrectly — a user on `4.5.0-rc.0` would never be prompted to upgrade once `4.5.0` stable shipped (lex order put the prerelease ahead of the bare version). Real semver gets this right. Stable users were never affected: the check queries the `@latest` dist-tag, which by convention never points at a prerelease. ## How an RC actually publishes after this 1. `pnpm exec changeset pre enter rc` on main, push the `pre.json` 2. Bot regenerates the release PR as `chore: release v<X.Y.Z>-rc.0` 3. Merge → `release.yml` runs `changeset publish` which reads `pre.json.tag` and publishes under `--tag rc`. GitHub release marked Pre-release. No marketing-site dispatch. 4. Iterate by adding changesets normally; bot bumps to `rc.1`, `rc.2`, … 5. When ready: `pnpm exec changeset pre exit`, push, merge regenerated PR → stable ships under `latest` and the marketing-site dispatch fires. |
||
|
Eric Allam
|
4c42f6cc0b |
feat(webapp,core,cli): filter runs by region in dashboard, API, and MCP (#3612)
## Summary
Adds a Region column and Region filter (under More filters) to the runs
list dashboard, the same filter on the public runs list API
(`filter[region]`), and a matching `region` input on the MCP `list_runs`
tool. Each run's executing region is also surfaced as a new optional
`region` field on the runs list and run retrieve responses, populated
from the worker instance group's `masterQueue` identifier.
Useful when you run tasks across multiple regions and want to slice the
runs list — or your existing run-querying scripts — by where the run
actually executed.
## Design
The filter value in the URL / API is the `masterQueue` identifier (the
same string already persisted on `TaskRun` and replicated to ClickHouse
as `worker_queue`), so the query just becomes `worker_queue IN (...)`
with no server-side translation. The Region dropdown options come from a
new resource loader backed by `RegionsPresenter`, which now also exposes
`masterQueue` alongside the existing region metadata.
```ts
// public API
const runs = await runs.list({ region: ["us-east-1", "eu-west-1"] });
// each item: { id, status, ..., region?: "us-east-1" }
```
```ts
// MCP
list_runs({ environment: "prod", region: "us-east-1" })
```
|
||
|
Eric Allam
|
ac02c0f709 |
fix(core): drop unique-symbol brand on LocalsKey to fix dual-package builds (#3626)
## Summary `LocalsKey<T>` (the type returned by `locals.create()`) was branded with a module-level `declare const __local: unique symbol`. Each such declaration is its own nominal type, and `tshy` emits separate `.d.ts` files for the ESM and CJS outputs — each gets its own `__local` symbol. Under certain pnpm hoisting layouts a single TypeScript compilation can resolve `LocalsKey` from both the ESM source path and the CJS dist path within the same call site, producing two structurally-incompatible variants of the same type. TS surfaces this as the misleading error: ``` Argument of type 'LocalsKey<X>' is not assignable to parameter of type 'LocalsKey<X>'. Property '[__local]' is missing in type 'LocalsKey<X>' but required in type 'BrandLocal<X>'. ``` The error has been hitting CI on PRs opened since the chat.agent stack landed (e.g. #3625 typecheck job), but doesn't reproduce on developer machines where the pnpm node_modules layout was built up incrementally. ## Fix Replace the `unique symbol` brand with an optional phantom field that carries `T` at the type level: ```ts // before declare const __local: unique symbol; type BrandLocal<T> = { [__local]: T }; export type LocalsKey<T> = BrandLocal<T> & { readonly id: string; readonly __type: unique symbol; }; // after export type LocalsKey<T> = { readonly id: string; readonly __type: symbol; /** Phantom carrier for the value type — never read at runtime. */ readonly __valueType?: T; }; ``` The ESM and CJS `.d.ts` outputs now produce structurally identical types, so cross-output resolution no longer produces a mismatch. `T` is still carried at the type level via the optional phantom field. The runtime shape is unchanged — `manager.ts` was already casting via `as unknown`, which is no longer needed. ## Test plan - [ ] `pnpm run typecheck --filter @trigger.dev/core --filter @trigger.dev/sdk` - [ ] `pnpm run build --filter @trigger.dev/core --filter @trigger.dev/sdk` (clean rebuild) — confirms the ESM and CJS dist `.d.ts` outputs no longer carry distinct `unique symbol` declarations - [ ] `pnpm --filter @trigger.dev/core test test/mockTaskContext.test.ts --run` - [ ] `pnpm --filter @trigger.dev/sdk test test/mockChatAgent.test.ts --run` |
||
|
Eric Allam
|
8673d42c80 |
feat: ai-chat reference project + MCP agent-chat tooling
Top of the chat.agent stack: a full Next.js reference project that exercises chat.agent end-to-end, plus the CLI MCP tools that drive agent runs from Claude Code / Cursor / etc. references/ai-chat: - Full Next.js app with prisma persistence, multi-chat sidebar, per-chat model picker, debug panel, tool examples, smoke tests - Reference tools: getCurrentTime, searchHackerNews, createGithubIssue, PR review helpers, code sandbox - chat-client-test orchestrator for concurrent-send stress - references/hello-world chatAgent + triggerAndSubscribe examples CLI MCP tooling for chat.agent: - mcp/tools/agentChat.ts (start_agent_chat, send_agent_message, close_agent_chat) - mcp/tools/agents.ts + tasks.ts (list agents, agent run details) - dev-run-controller OOM kill + taskRunProcessPool tweaks - dev/managed entry-point hooks for skills bundling - buildWorker + bundleSkills (agent skills support) Includes ai-tool-helpers + mcp-agent-chat-sessions changesets, plus the streamdown@2 patch and pnpm-lock reconciliation. (Will be renamed to feature/ai-chat-reference-and-cli before push.) fix(cli): preserve lastEventId after sendMessage fallback to avoid stale turn-complete replay |
||
|
Eric Allam
|
16720a5e62 |
feat(sdk): chat.agent — runtime + browser transport
Adds the chat.agent({...}) task definition (server runtime) and the
browser-side TriggerChatTransport + AgentChat that drives it from a
React or Next.js app. The runtime sits on top of the Sessions primitive
and handles the durable conversational task lifecycle.
Server runtime:
- chat.agent({...}) — session-aware task definition
- Lifecycle hooks: onChatStart, onTurnStart, onTurnComplete, onAction,
onValidateMessages, hydrateMessages
- chat.history read primitives for HITL flows
- chat.local, chat.headStart, chat.handover, oomMachine
- Delta-only wire + S3 snapshot reconstruction at run boot
- Actions are no longer turns
Browser transport:
- TriggerChatTransport (ai-sdk Transport): delta-only wire sends,
SSE reconnection with lastEventId resume, stop/abort cleanup,
dynamic accessToken refresh
- AgentChat: direct programmatic API
- useTriggerChatTransport (React hook)
- chat-tab-coordinator: cross-tab leader election
Includes the chat-agent, chat-agent-delta-wire-snapshots,
chat-history-read-primitives, chat-head-start, chat-actions-no-turn,
chat-session-attributes, agent-skills, and mock-chat-agent-test-harness
changesets.
|
||
|
Eric Allam
|
979655c281 |
feat: Sessions dashboard, task_kind, and chat-ready hardening (1/4) (#3542)
## Summary A `/sessions` dashboard for inspecting durable Sessions, an `AGENT` / `SCHEDULED` task-kind filter for the runs list, and the server-side hardening (rate-limit exemption for packets, retry-with-backoff on stream appends, typed too-large-chunk error) that the `chat.agent` runtime in #3543 needs. Builds on the Sessions primitive shipped in #3417. ## Design The Sessions list + detail routes mirror the run inspector pattern. `TaskTriggerSource` gains `AGENT` and `SCHEDULED` values, persisted on `BackgroundWorker.taskKind` and `TaskRun.taskKind` (plus a matching Clickhouse column), so the runs list can filter by kind. New `@trigger.dev/core` modules — `sessionStreams`, `inputStreams`, a `sessionStreamInstance` for realtime streams, and the `realtime-streams-api` / `session-streams-api` surfaces — expose the typed shapes that chat.agent will use to drive `session.out`. `ChatChunkTooLargeError` lets the runtime drop oversized chunks with a typed surface instead of failing the run. `s2Append` retries transient failures with exponential backoff. `/api/v[12]/packets/*` is exempt from customer rate limits so chat snapshot reads and writes don't get throttled under load. ## Stack Part of a 4-PR stack. Merge bottom-up. 1. **This PR** (#3542) → `main` 2. #3543 → #3542 — `chat.agent` runtime + browser transport 3. #3545 → #3543 — agent-view dashboard 4. #3546 → #3545 — ai-chat reference + MCP tooling Replaces #3173 (closed). <!-- GitButler Footer Boundary Top --> --- This is **part 5 of 5 in a stack** made with GitButler: - <kbd> 5 </kbd> #3612 - <kbd> 4 </kbd> #3546 - <kbd> 3 </kbd> #3545 - <kbd> 2 </kbd> #3543 - <kbd> 1 </kbd> #3542 👈 <!-- GitButler Footer Boundary Bottom --> |
||
|
Daniel Sutton
|
09f5354a03 |
fix(core): cap idempotencyKey length at the API boundary (#3560)
`tasks.trigger`, `tasks.batchTrigger`, `batch.create`, `wait.createToken`, `wait.forDuration`, and the input/session stream waitpoint endpoints all accept a caller-supplied `idempotencyKey` and store it verbatim against a composite-unique index on `TaskRun`, `BatchTaskRun`, or `Waitpoint`. The schemas had no length cap, so a sufficiently long high-entropy key produced an index row larger than the underlying storage layer can hold. The insert failed at the database, and the caller saw a generic 500 from `RunEngineTriggerTaskService.call()` / `CreateBatchService` / waitpoint creation, depending on the endpoint. Keys produced by `idempotencyKeys.create()` are 64-character SHA-256 hashes and never trip this — it only manifests for direct REST callers (or SDK callers passing a raw string they generated themselves). Low-entropy keys also sail through, because the storage layer compresses repeated bytes before they reach the index, which is why the failure mode is intermittent and tied to caller-side key shape. ## Fix Add `.max(2048, "<field> must be 2048 characters or less")` to the seven schemas that feed an indexed `idempotencyKey` column: - `TriggerTaskRequestBody.options.idempotencyKey` - `BatchTriggerTaskItem.options.idempotencyKey` - `CreateBatchRequestBody.idempotencyKey` - `CreateWaitpointTokenRequestBody.idempotencyKey` - `CreateInputStreamWaitpointRequestBody.idempotencyKey` - `CreateSessionStreamWaitpointRequestBody.idempotencyKey` - `WaitForDurationRequestBody.idempotencyKey` Plus the `idempotency-key` HTTP header on the trigger route (and the three batch routes that re-export `HeadersSchema`). The header schema is lifted out of `api.v1.tasks.$taskId.trigger.ts` into `apps/webapp/app/v3/triggerHeaders.server.ts` so it can be exercised in tests without dragging the route's import-time side effects. The 2048 character ceiling is chosen to sit safely under the per-row index limit while staying generous against existing callers — keys that fit before still fit. Oversized keys now return a structured Zod 400 instead of a generic 500. Limit is documented under `Idempotency key` in `docs/limits.mdx` and as a `<Note>` on `docs/idempotency.mdx`. ## Test plan - [x] 15 schema unit tests added (`packages/core/src/v3/schemas/idempotencyKey.test.ts`, `apps/webapp/test/routes/triggerHeaders.test.ts`) — rejection-with-message + boundary acceptance for each capped schema. The webapp test exercises the extracted `TriggerHeadersSchema` directly with no mocks. - [x] `pnpm run build --filter @trigger.dev/core` - [x] `pnpm run typecheck --filter webapp` - [x] End-to-end verified locally: baseline (small key) → 200; 3000-char high-entropy header → 400 with the expected Zod error; same key at the 2048 boundary → 200; same key with the cap reverted → the database rejected the insert and the route returned 500 to the caller. Cap restored. Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
Eric Allam
|
be1a6cf8de |
feat: Sessions primitive — durable run-aware streams + dashboard
Adds Sessions, a durable, run-aware stream primitive that scopes session.in / session.out records to a session (not a single run). Records survive run boundaries; reconnect-from-last-event-id is built in. Server foundation: - New /realtime/v1/sessions/:session/:io/append + /records routes - sessionRunManager + sessionsRepository + clickhouseSessionsRepository - mintRunToken for short-lived per-session tokens - s2Append retry-with-backoff + undici cause diagnostics - /api/v[12]/packets/* exempt from customer rate limits - BackgroundWorker schema gains taskKind enum (TASK, AGENT, SCHEDULED) - TaskRun.taskKind column + clickhouse 029_add_task_kind_to_task_runs_v2 Core types: - new sessionStreams, inputStreams, realtimeStreams packages in @trigger.dev/core - session-streams-api / realtime-streams-api surface Sessions dashboard UI (the primitive's own viewer): - /sessions index + detail routes - SessionsTable, SessionFilters, SessionStatus, CloseSessionDialog - AGENT/SCHEDULED filter in RunFilters + TaskTriggerSource Includes the sessions-primitive changeset. |
||
|
Matt Aitken
|
e4981d1b11 |
feat(webapp): consolidate auth path + add comprehensive auth tests (#3499)
## Summary
Consolidates the webapp's authentication and authorization into a small
set of route helpers, replacing the ad-hoc `requireUser` /
`requireUserId` / `authenticatedEnvironmentForAuthentication` calls
scattered across routes. Same security model, but the per-request flow
(authenticate → authorize → load) now lives in one place per route
family.
Introduces a plugin seam (`@trigger.dev/plugins`) that lets the cloud
build install a richer RBAC implementation without touching webapp code.
The OSS fallback keeps the pre-RBAC permissive behaviour intact, so
self-hosted deployments work unchanged.
Adds a comprehensive end-to-end auth test suite that didn't exist before
— 193 `it()` blocks (vitest reports ~199 after `it.each` expansion)
covering API key, PAT and JWT auth across the public API surface, plus
dashboard session auth for admin pages.
## Changes
### Plugin contract — `@trigger.dev/plugins`
`RoleBaseAccessController` interface authoritative for both OSS
(fallback) and cloud (enterprise plugin):
- `authenticateBearer(request, { allowJWT? })` — API-key / public-JWT
auth, returns env + ability
- `authenticateSession(request, { userId, organizationId?, projectId?
})` — dashboard auth, caller resolves `userId` from the session cookie
and passes it in (no `helpers.getSessionUserId` callback — decouples the
plugin host from session-cookie code)
- `authenticatePat(request, { organizationId?, projectId? })` — PAT
auth, returns identity + `lastAccessedAt` so the host can throttle the
per-request update
- `authenticateAuthorize*` variants for the auth-and-check-in-one-call
cases
- `isUsingPlugin(): Promise<boolean>` — capability flag for UI /
branching where plugin-present-ness matters; replaces the
sentinel-string coupling that had `personalAccessToken.server` matching
`"RBAC plugin not installed"` literally
### Dashboard auth (started, partial rollout)
Admin and settings pages migrated to a unified `dashboardLoader` /
`dashboardAction` helper that authenticates the session, runs an
authorization check, and exposes the result to the route. Other
dashboard routes still on the old pattern; remaining migration tracked
in TRI-8730.
Migrated routes:
- `admin.*` (14 admin / back-office / feature-flags / LLM-models /
notifications / orgs / concurrency pages)
- `_app.orgs.$organizationSlug.settings.team`
- `_app.orgs.$organizationSlug.settings.roles`
### API / realtime / engine auth (complete for the migrated families)
71 routes migrated to a unified `apiBuilder` that centralizes Bearer /
PAT / Public-JWT authentication and applies the per-route authorization
check before the handler runs. Includes:
- `api.v1.*` and `api.v2.*` and `api.v3.*` — tasks, runs, batches,
queues, prompts, deployments, query, sessions, waitpoints, packets,
workers, idempotency keys
- `realtime.v1.*` — runs, batches, sessions, streams
- `engine.v1.*` — dev / worker-action protocols
29 routes still on the legacy `authenticateApiRequest*` helpers —
tracked as a post-deploy follow-up in TRI-9228.
Multi-resource auth direction is now explicit at the call site via
`anyResource(...)` (OR) and `everyResource(...)` (AND). Bare arrays no
longer typecheck — fixes a class of bug where a JWT scoped to one
resource could implicitly access others under OR semantics.
PAT auth path consolidated: was three DB queries per request (legacy
`authenticateApiRequestWithPersonalAccessToken` findFirst +
`rbac.authenticatePat` join + `lastAccessedAt` update). Now one query in
the steady state — plugin returns `lastAccessedAt`, host smart-skips the
update via JS-side throttle when fresh.
Side effect: action aliases preserved historic JWT scope semantics where
the new model is stricter (e.g. a `write:tasks` JWT now also satisfies
`trigger` / `batchTrigger` / `update` actions on the same resource —
matched at the auth boundary, not in the route handler).
### Backwards-compat fixes
The strict-match model regressed several real-world JWT shapes. Each
preserved via explicit `anyResource(...)` entries in the route's authz
block:
- **Batch retrieve routes** (`api.v1.batches.$batchId`, `api.v2.*`,
`realtime.v1.batches.*`) accept `read:runs` JWTs again (pre-RBAC
literal-match superScope behaviour)
- **Runs list routes** (`api.v1.runs`, `realtime.v1.runs`) accept
type-level `read:tasks` / `read:tags` on unfiltered queries (matched the
legacy `Object.keys` iteration semantic)
- **PAT/OAT auth shape** normalized through `toAuthenticated` so all
auth methods return the same slim `AuthenticatedEnvironment` (was:
API-key returned the slim shape but PAT/OAT returned raw Prisma
`Decimal` / no `orgMember`)
- **Scope `:` preservation** in resource ids — `read:tags:env:staging`
now correctly identifies the tag id as `env:staging`, not `env`
### Slim `AuthenticatedEnvironment`
Extracted to `@trigger.dev/core/v3/auth/environment` — a structural
shape independent of `@trigger.dev/database`. The plugin contract
returns this; webapp consumers import from there; the cloud plugin
(Drizzle) returns the same shape without Prisma's `Decimal` class
leaking into the public surface. Lets internal-packages (run-engine,
etc.) refer to `AuthenticatedEnvironment` without pulling Prisma in.
### Auth test suite (new — `*.e2e.full.test.ts`)
193 e2e tests run against a real spawned webapp + Postgres (no mocks).
Coverage matrix:
- **API key auth** — read / write / trigger / batchTrigger / deploy
actions across runs, batches, deployments, prompts, queues, query,
sessions, input-streams, waitpoints, tasks, idempotency keys; multi-key
resources (a run carries batch / tag / task identifiers — auth must
accept any matching scope)
- **Personal Access Token auth** — comprehensive matrix: scope match,
scope mismatch, missing scope, expired token, malformed token
- **Public JWT auth** — sub-vs-URL environment resolution, expired JWTs,
signature verification, scope checking, otu (one-time-use) token
semantics, branch-environment signing-key fallback
- **Dashboard session auth** — admin-only pages reject non-admins;
per-action gating
- **Cross-cutting edge cases** — revoked API key grace window, JWT
cross-environment isolation, MissingResource branch behaviour
### Hygiene cleanups
- Deleted dead `app/services/authorization.server.ts` (legacy
`checkAuthorization` + types — no live consumers post-migration) and its
orphaned test
- Dropped the never-populated `scopes` field from
`ApiAuthenticationResultSuccess`
- `scheduleEmail` moved out of `email.server.ts` into its own module —
breaks a `commonWorker → marqs/V1` import chain that was poisoning the
auth test graph
- OSS Roles page shows a deployment-aware empty state ("Roles aren't
available in this self-hosted deployment" vs the plan-upsell copy) via
`rbac.isUsingPlugin()`
- Team action handler: explicit per-intent ability gates
(`manage:billing` for purchase-seats, `manage:members` for set-role +
remove-member with self-leave carve-out)
### Cross-repo coordination
All public-package contract changes paired in `triggerdotdev/cloud#763`
(rbac-packages branch) — the enterprise plugin implements the same
`RoleBaseAccessController` interface against Drizzle.
## Test plan
- [x] `pnpm run typecheck --filter webapp` clean
- [x] `pnpm --filter webapp exec vitest run --config
vitest.e2e.full.config.ts` — 193/193 pass (requires Docker for
testcontainers)
- [x] Spot-check an authed API endpoint with a valid + invalid API key
against a local stack
- [x] Spot-check the migrated admin pages render and gate non-admins
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|
Matt Aitken
|
8e675a4e93 |
fix(core): retry TASK_PROCESS_SIGSEGV under the user's retry policy (#3552)
Closes [TRI-9234](https://linear.app/triggerdotdev/issue/TRI-9234/retry-task-process-sigsegv-errors-respecting-user-retry-config) ## What this changes SIGSEGV crashes (`TASK_PROCESS_SIGSEGV`) will now be **retried when an attempt fails**, in line with the task's configured retry settings (`retry.maxAttempts` etc.) — the same path SIGTERM and uncaught exceptions already use. Previously SIGSEGV was hard-classified as non-retriable and failed the run on the first segfault, ignoring the user's retry policy. Tasks without a retry policy still fail fast on the first SIGSEGV. Behaviour is unchanged for OOM kills (separate machine-bump retry path) and SIGKILL_TIMEOUT. ## Deploy **Only the webapp needs to ship.** The retry decision lives entirely in the webapp: - V2 path: `internal-packages/run-engine` (bundled into the webapp) - V1 path: `apps/webapp/app/v3/services/completeAttempt.server.ts` No supervisor, CLI, SDK, or customer-task-image changes required. Customers do not need to redeploy. The `@trigger.dev/core` changeset is just keeping the public package in sync — the published npm version isn't what makes the fix work. ## Why retry SIGSEGV in Node tasks is frequently non-deterministic across processes: - **Native addon races** (`sharp`, `canvas`, `better-sqlite3`, `node-rdkafka`, `bcrypt`, …) — libuv thread-pool work stepping on V8 handles. Different heap layout / thread schedule on a fresh process → retry often succeeds. - **JIT / GC interaction** — V8 turbofan deopt or GC during a native callback. Timing-dependent. - **Near-OOM in native code** — when RSS approaches the cgroup limit, native allocations fail and poorly-written addons dereference NULL → SIGSEGV instead of clean OOM-kill. - **Host / hardware issues** — bit flips, kernel quirks. Retry lands on a different host. The genuinely deterministic case (a user-code bug always tripping the same addon) is real, but a subset — and `maxAttempts` bounds the damage. ## Pre-existing inconsistency this resolves - `shouldRetryError` returned `false` for `TASK_PROCESS_SIGSEGV` → `fail_run`. - `shouldLookupRetrySettings` already listed `TASK_PROCESS_SIGSEGV` as retry-config-aware — but that branch was unreachable because `shouldRetryError` short-circuited first in `retrying.ts:86-90`. - We already retry `TASK_RUN_UNCAUGHT_EXCEPTION` (clearly a user-code bug) under the user's retry policy; refusing to retry SIGSEGV was the odd one out. ## Test plan - [x] `pnpm exec vitest run test/errors.test.ts` in `packages/core` — 26/26 pass (4 new) - [x] `pnpm run build --filter @trigger.dev/core` - [ ] CI green on PR 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
github-actions[bot]
|
41a486ea7e |
chore: release v4.4.6 (#3501)
🚀 Publish Trigger.dev Docker / typecheck (push) Failing after 0s
🚀 Publish Trigger.dev Docker / publish-webapp (push) Has been skipped
🚀 Publish Trigger.dev Docker / publish-worker (push) Has been skipped
🚀 Publish Trigger.dev Docker / publish-worker-v4 (push) Has been skipped
🚀 Publish Trigger.dev Docker / units (push) Failing after 5s
🧭 Helm Chart Release / lint-and-test (push) Has been cancelled
🧭 Helm Chart Release / release (push) Has been cancelled
## Summary
1 improvement, 1 bug fix.
## Improvements
- Fail attempts on uncaught exceptions instead of hanging to
`MAX_DURATION_EXCEEDED`. A Node `EventEmitter` (e.g. `node-redis`)
emitting `"error"` with no `.on("error", ...)` listener escalates to
`uncaughtException`, which the worker previously reported but did not
act on — runs drifted to maxDuration with empty attempts. They now fail
fast with the original error and status `FAILED`, and respect the task's
normal retry policy. You should still attach `.on("error", ...)`
listeners to long-lived clients to handle errors gracefully.
([#3529](https://github.com/triggerdotdev/trigger.dev/pull/3529))
## Bug fixes
- Fix dev workers spinning at 100% CPU after the parent CLI disconnects.
Orphaned `trigger-dev-run-worker` (and indexer) processes were caught in
an `uncaughtException` feedback loop: a periodic IPC send via
`process.send` would throw `ERR_IPC_CHANNEL_CLOSED` once the parent
closed the channel, which re-entered the same handler that itself called
`process.send`, scheduled via `setImmediate` and amplified by
source-map-support's `prepareStackTrace`. Fixed by (1) silently dropping
packets in `ZodIpcConnection` when the channel is disconnected, (2)
adding a `process.on("disconnect", ...)` handler in dev workers so they
exit cleanly when the CLI closes the IPC channel, and (3) wrapping all
`uncaughtException`-path `process.send` calls in a `safeSend` guard that
checks `process.connected` and swallows synchronous throws.
([#3491](https://github.com/triggerdotdev/trigger.dev/pull/3491))
<details>
<summary>Raw changeset output</summary>
# Releases
## @trigger.dev/build@4.4.6
### Patch Changes
- Updated dependencies:
- `@trigger.dev/core@4.4.6`
## trigger.dev@4.4.6
### Patch Changes
- Fix dev workers spinning at 100% CPU after the parent CLI disconnects.
Orphaned `trigger-dev-run-worker` (and indexer) processes were caught in
an `uncaughtException` feedback loop: a periodic IPC send via
`process.send` would throw `ERR_IPC_CHANNEL_CLOSED` once the parent
closed the channel, which re-entered the same handler that itself called
`process.send`, scheduled via `setImmediate` and amplified by
source-map-support's `prepareStackTrace`. Fixed by (1) silently dropping
packets in `ZodIpcConnection` when the channel is disconnected, (2)
adding a `process.on("disconnect", ...)` handler in dev workers so they
exit cleanly when the CLI closes the IPC channel, and (3) wrapping all
`uncaughtException`-path `process.send` calls in a `safeSend` guard that
checks `process.connected` and swallows synchronous throws.
([#3491](https://github.com/triggerdotdev/trigger.dev/pull/3491))
- Fail attempts on uncaught exceptions instead of hanging to
`MAX_DURATION_EXCEEDED`. A Node `EventEmitter` (e.g. `node-redis`)
emitting `"error"` with no `.on("error", ...)` listener escalates to
`uncaughtException`, which the worker previously reported but did not
act on — runs drifted to maxDuration with empty attempts. They now fail
fast with the original error and status `FAILED`, and respect the task's
normal retry policy. You should still attach `.on("error", ...)`
listeners to long-lived clients to handle errors gracefully.
([#3529](https://github.com/triggerdotdev/trigger.dev/pull/3529))
- Updated dependencies:
- `@trigger.dev/core@4.4.6`
- `@trigger.dev/build@4.4.6`
- `@trigger.dev/schema-to-json@4.4.6`
## @trigger.dev/core@4.4.6
### Patch Changes
- Fix dev workers spinning at 100% CPU after the parent CLI disconnects.
Orphaned `trigger-dev-run-worker` (and indexer) processes were caught in
an `uncaughtException` feedback loop: a periodic IPC send via
`process.send` would throw `ERR_IPC_CHANNEL_CLOSED` once the parent
closed the channel, which re-entered the same handler that itself called
`process.send`, scheduled via `setImmediate` and amplified by
source-map-support's `prepareStackTrace`. Fixed by (1) silently dropping
packets in `ZodIpcConnection` when the channel is disconnected, (2)
adding a `process.on("disconnect", ...)` handler in dev workers so they
exit cleanly when the CLI closes the IPC channel, and (3) wrapping all
`uncaughtException`-path `process.send` calls in a `safeSend` guard that
checks `process.connected` and swallows synchronous throws.
([#3491](https://github.com/triggerdotdev/trigger.dev/pull/3491))
- Fail attempts on uncaught exceptions instead of hanging to
`MAX_DURATION_EXCEEDED`. A Node `EventEmitter` (e.g. `node-redis`)
emitting `"error"` with no `.on("error", ...)` listener escalates to
`uncaughtException`, which the worker previously reported but did not
act on — runs drifted to maxDuration with empty attempts. They now fail
fast with the original error and status `FAILED`, and respect the task's
normal retry policy. You should still attach `.on("error", ...)`
listeners to long-lived clients to handle errors gracefully.
([#3529](https://github.com/triggerdotdev/trigger.dev/pull/3529))
## @trigger.dev/python@4.4.6
### Patch Changes
- Updated dependencies:
- `@trigger.dev/core@4.4.6`
- `@trigger.dev/build@4.4.6`
- `@trigger.dev/sdk@4.4.6`
## @trigger.dev/react-hooks@4.4.6
### Patch Changes
- Updated dependencies:
- `@trigger.dev/core@4.4.6`
## @trigger.dev/redis-worker@4.4.6
### Patch Changes
- Updated dependencies:
- `@trigger.dev/core@4.4.6`
## @trigger.dev/rsc@4.4.6
### Patch Changes
- Updated dependencies:
- `@trigger.dev/core@4.4.6`
## @trigger.dev/schema-to-json@4.4.6
### Patch Changes
- Updated dependencies:
- `@trigger.dev/core@4.4.6`
## @trigger.dev/sdk@4.4.6
### Patch Changes
- Updated dependencies:
- `@trigger.dev/core@4.4.6`
</details>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
|
||
|
Matt Aitken
|
62e006617e |
fix(cli): fail attempt on uncaught exception instead of hanging to maxDuration (TRI-9117) (#3529)
⚒️ Publish Worker (v4) / build (supervisor) (push) Has been cancelled
When a Node EventEmitter (e.g. node-redis) emits an "error" event with
no
listener attached, Node escalates it to process.on("uncaughtException")
in
the task worker. The worker reported the error via the
UNCAUGHT_EXCEPTION
IPC event but did not exit, and the supervisor-side handler in
taskRunProcess only logged the message at debug level — leaving the
run()
promise orphaned until maxDuration fired and producing empty attempts
(durationMs=0, costInCents=0).
The supervisor now rejects the in-flight attempt with an
UncaughtExceptionError and gracefully terminates the worker (preserving
the OTEL flush window) on UNCAUGHT_EXCEPTION. The attempt fails fast
with
TASK_EXECUTION_FAILED, surfacing the original error name, message, and
stack trace, and falls under the normal retry policy. This mirrors the
existing indexing-side behavior in indexWorkerManifest. Apply the same
handling to unhandled promise rejections, which Node already routes
through uncaughtException by default.
|