github-featured/triggerdotdev--trigger.dev
2
0
Fork 0
Code Issues Pull Requests Deployments Models Agent Notes Packages Projects Releases Wiki Activity
7,418 Commits 300 Branches 4,167 Tags
mollifier-phase-3
Commit Graph

7418 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Dan Sutton 3a9bca28ff chore(mollifier): drop docs/realtime change from PR 
Revert docs/realtime/react-hooks/subscribe.mdx to match main — the
realtime-burst Note belongs in a separate docs pass, not in the
mollifier feature PR.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-22 17:58:26 +01:00
Dan Sutton 3a1494d814 chore(mollifier): untrack scripts/mollifier-api-parity.sh 
Local working script.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-22 17:56:55 +01:00
Dan Sutton b5325ed719 chore(mollifier): restore changeset content from main (do not edit historical changelogs) 
I should not have touched this file. Restoring exactly from
origin/main — zero diff against main.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-22 17:55:37 +01:00
Dan Sutton 4c0c969f47 chore(mollifier): restore historical changeset; add buffer-extensions 
Restore .changeset/mollifier-redis-worker-primitives.md to its
original content (it was already on main from the Phase 1 scaffolding
PR — should never have been touched during consolidation).

Add .changeset/mollifier-buffer-extensions.md covering the
post-scaffolding additions: idempotency lookup, snapshot mutation API,
metadata CAS, watermark listing, claim primitives, ZSET-backed queue,
ack grace TTL, drop-entry-TTL, and the @trigger.dev/core notice field.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-22 17:54:05 +01:00
Dan Sutton 8e371009ae chore(mollifier): rename changeset; untrack stress-tasks, challenge scripts, ops doc 
Single redis-worker changeset under its original filename.
references/stress-tasks, scripts/mollifier-challenge, _ops/ come out
of the tree — kept locally as working artefacts.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-22 17:47:08 +01:00
Dan Sutton e23c6ee4d4 chore(mollifier): consolidate changesets + server-changes; untrack _plans 
Single changeset and server-changes entry for the mollifier feature
instead of one per commit. _plans/ files come out of the tree (they
stay on disk as untracked working notes, per convention).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-22 17:41:51 +01:00
Dan Sutton 92e7d2f02e Merge remote-tracking branch 'origin/main' into mollifier-phase-3 
# Conflicts:
#	apps/webapp/app/routes/_app.orgs.$organizationSlug.projects.$projectParam.env.$envParam.runs.$runParam/route.tsx
#	apps/webapp/app/routes/api.v1.runs.$runId.spans.$spanId.ts
#	apps/webapp/app/routes/api.v1.runs.$runId.trace.ts
#	apps/webapp/app/routes/resources.runs.$runParam.logs.download.ts
 2026-05-22 17:33:02 +01:00
Eric Allam 61ca40b4b1 perf(run-engine,webapp): look up PENDING_VERSION runs via ClickHouse (#3707) 
## Summary

When a background worker registers, the engine resolves runs that were
queued before the worker was ready (status `PENDING_VERSION`). That
lookup used to scan a Postgres status index on `TaskRun`. Move it to
ClickHouse: query candidate run ids from `task_runs_v2`, then refetch
the actual rows from Postgres by primary key with a `status =
'PENDING_VERSION'` guard for idempotency.

## Design

The lookup is a pluggable interface on the run engine
(`PendingVersionRunIdLookup`). The webapp wires a ClickHouse-backed
implementation through the org-scoped `clickhouseFactory` using a new
`"engine"` client type, configured by `RUN_ENGINE_CLICKHOUSE_*` env
vars. The URL falls back to `CLICKHOUSE_URL` when unset, so self-hosted
deployments don't need new config to keep working.

When the lookup returns no candidates, one bounded retry is scheduled
~5s later to cover ClickHouse replication lag against `task_runs_v2`.
The Postgres status guard on both the candidate refetch and the inner
`updateMany` prevents double-promotion when a retry races with a
concurrent deploy.

Tests cover three existing PENDING_VERSION cases via a small
Postgres-backed test adapter; new ClickHouse-backed integration tests
will follow.
 2026-05-22 17:18:41 +01:00
Dan Sutton 3fa9984de2 test(scripts): mollifier SDK response shape audit (5.6) 
Phase 4's audit found two Zod drifts reactively (idempotencyKey: null
and parentId: undefined). This script proactively sweeps every public
SDK method with a buffered branch by calling them through the real
@trigger.dev/core apiClient — zodfetch's schemas execute against each
response, so any drift now fails the audit. The existing
mollifier-challenge shell scripts only do jq structural checks, which
miss schema-level drift like null-vs-undefined or
optional-vs-nullable mismatches.

Covers nine methods against a fresh buffered run each (separate runs
for destructive ones so they don't interfere): retrieveRun,
retrieveRunTrace, retrieveSpan, listRunEvents, addTags,
updateRunMetadata, replayRun, rescheduleRun, cancelRun.

Manually verified against the live local webapp — all nine pass with
no drift surfaced. The audit is reusable as a smoke-check before each
prod rollout.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-22 16:50:47 +01:00
Eric Allam 0d4891a5f2 perf(database): drop unused TaskRun(scheduleId, createdAt) index (#3706) 
## Summary

Drops the unused composite Postgres index
`TaskRun_scheduleId_createdAt_idx`. The schedule list view reads from
ClickHouse, so this index served no Prisma query while still being
maintained on every `TaskRun` INSERT/UPDATE. Removing it reduces write
amplification on the primary database.

Sibling to the prior drop of `TaskRun_scheduleId_idx` and the earlier
removal of the `TaskRun.scheduleId` foreign key — all stemming from
migrating schedule-aware reads to ClickHouse.

## Verification

- Sampled `pg_stat_user_indexes` for `TaskRun` over multiple hours —
zero scans against this index.
- Grepped the codebase for any Prisma query filtering
`TaskRun.scheduleId` — none found. All schedule-aware listing routes
through `clickhouseRunsRepository`.
 2026-05-22 16:34:00 +01:00
Saadi Myftija 2cbfcfac3d chore(webapp): make HTTP keep-alive timeout configurable (#3705) 
Make the Express server's `keepAliveTimeout` configurable via
`HTTP_KEEPALIVE_TIMEOUT_MS`. Default preserved at 65000 ms — no behavior
change if unset.
 2026-05-22 16:28:08 +01:00
Dan Sutton 97018b1e65 fix(run-engine): emit runFailed from createFailedTaskRun 
The mollifier drainer's terminal-failure path (Phase 4G) and the
batch-trigger's "queue size limit exceeded" path both call
createFailedTaskRun to write a SYSTEM_FAILURE PG row for runs that
never actually executed. Neither path emitted runFailed afterwards,
so the runEngineHandlers' `runFailed` listener never fired — which
means PerformTaskRunAlertsService never enqueued an alert delivery
job, and customers' configured TASK_RUN alert channels missed the
failure entirely. The row was visible in the dashboard list but
silent for alerting purposes.

Emit runFailed from createFailedTaskRun with `attemptNumber: 0` as
the marker that the run never executed (distinguishes synthesised
terminal failures from runs that exhausted their retries).
PerformTaskRunAlertsService doesn't filter on attemptNumber or
status, so the existing pipeline picks the event up without further
changes. DeliverAlertService dispatches via the channel type
(email/webhook/etc) the same way it does for any other terminal
failure.

Test: a containerTest subscribes to runFailed before calling
createFailedTaskRun, asserts exactly one event fires with the
expected payload shape. The existing batchTrigger tests still pass
(they didn't assert the negative).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-22 15:53:14 +01:00
nicktrn ddad9700d7 fix(supervisor): compat shim for COMPUTE checkpoint type (#3703) 
Workloads bundled with CLI versions before v4.4.4 use a strict zod enum
for `checkpoint.type` that only allows DOCKER and KUBERNETES. When a
customer's runs are routed via the compute path, those old runners
receive `type: "COMPUTE"` on `/snapshots/since/...` and `/dequeue`
responses and fail validation - blocking silent migration of existing
deployments.

The workload never reads the field - only validates the shape. Rewriting
COMPUTE -> KUBERNETES on the way out lets older runners keep parsing
while the database and internal services keep the real value. Limited to
the two workload-facing endpoints whose response includes a checkpoint;
`/continue`, `/attempts/start`, `/attempts/complete` all return shapes
without one.

Followup to #3114.
 2026-05-22 15:41:30 +01:00
Dan Sutton 8dc878e96e feat(redis-worker,webapp): drop mollifier entry TTL — drainer is the recovery mechanism 
Buffer entries used to EXPIRE after entryTtlSeconds (600s dev / 1h
prod). Once that window elapsed without the drainer ack'ing, the
entry just vanished — no PG row, no log, no customer signal. The
stale-entry sweep was added in the previous commit so ops gets paged
on dwell-too-long; with that signal in place, the TTL itself is now
the cause of the failure mode it was meant to mitigate.

Remove it. Buffer entries persist until the drainer ACKs (with the
existing 30s post-materialise grace TTL) or FAILs them. Idempotency
lookup keys also lose their TTL — keeping them paired to the entry
hash prevents the dedup-drift bug where a TTL'd lookup would let the
same idempotency key spawn a second buffered run while the first
still existed. `failMollifierEntry` now DELs the entry hash + lookup
because the SYSTEM_FAILURE PG row written by the drainer is the
canonical record; the buffer entry is no longer load-bearing.

Knock-on changes:
- `MollifierBufferOptions`: `entryTtlSeconds` removed (no consumers
  outside this repo).
- `TRIGGER_MOLLIFIER_ENTRY_TTL_S`: removed from env.server.ts and the
  example .env. The stale-sweep threshold now has its own explicit
  default (5min) instead of "half of TTL".
- `MollifierBuffer.getEntryTtlSeconds`: retained — it returns the
  Redis-side TTL, which is now -1 in steady state and ~30s after ack.
  Used by the ack-grace-TTL test.
- Existing tests updated: TTL-related cases inverted to assert no TTL;
  FAILED-state cases inverted to assert teardown; runId-reuse-after-
  fail now succeeds (slot is reclaimable).

Operational alert: Redis memory pressure if the drainer is offline.
That's the same failure mode as Redis OOM in any other context, with
existing infra-level alerts. The mollifier.stale_entries.current
gauge fires first; ops should be on it long before memory becomes a
problem. See _ops/mollifier-ops.md.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-22 15:37:01 +01:00
Daniel Sutton 1015876b98 feat(webapp): user-based Sentry attribution with tenant tags (#3678) 
## Summary

Stamp every Sentry event with the signed-in user and the tenant (org /
project / env) the request belongs to, so "Users Impacted" counts
distinct humans and events become filterable per tenant.

**Design after review (current):**

- `user.id = real user cuid` (from `requireUser`). "Users Impacted"
counts humans, not tenants.
- Tenant context (org / project / env slugs, IDs, env type) moves
entirely onto tags: `org_slug`, `project_slug`, `env_slug`, `org_id`,
`project_id`, `project_ref`, `environment_id`, `env_type`, plus
`impersonating` when set.
- Backed by an `AsyncLocalStorage` scope established at the HTTP entry.
Each entry point fills what it knows; loaders enrich the same scope with
what they already have.

**Zero new database queries.** The middleware does a regex match only.
Dashboard loaders that already query Prisma gain a couple of extra
selected columns; nothing new round-trips.

## How it's wired

- **Express middleware (`tenantContextResolver.server.ts`)** — parses
the URL with a regex and always opens an ALS scope. Populates whatever
subset of slugs is present: `/orgs/:o` → just `orgSlug`;
`/orgs/:o/projects/:p` adds `projectSlug`; the full triple adds
`envSlug`. Non-tenant paths get an empty scope so loaders can still
enrich.
- **`_app/route.tsx`** — already calls `requireUser`. Adds
`tenantContext.enrich({ userId: user.id })` for every authenticated
dashboard request. No new query.
- **Env layout loader (`_app.orgs.$o.projects.$p.env.$e/route.tsx`)** —
its existing `prisma.project.findFirst` gains two columns in `select`
(`externalRef`, `organization.id`). After it picks an env, calls
`tenantContext.enrich({ orgId, projectId, projectRef, envId, envType
})`. Same query, +2 columns.
- **API path (`apiBuilder.server.ts`)** — wraps every handler in
`tenantContext.run(tenantContextFromAuthEnvironment(authenticationResult.environment),
…)`. The mapper pulls `userId` from `env.orgMember?.userId` (already
selected by `authIncludeBase` — no schema change). Covers
`createLoaderApiRoute`, `createActionApiRoute`, and
`createMultiMethodApiRoute`.
- **Event processor (`sentryTenantContext.server.ts`)** — registered in
`entry.server.tsx` so it lives in the Remix bundle and shares the same
`tenantContext` ALS instance as the middleware and loaders. Stamps
whatever's present; nothing forced.

## Example events from local verification

| URL | `user.id` | Tags |
|-----|-----------|------|
| `/orgs/:o/projects/:p/env/:e/...` | real user cuid | `org_slug`,
`project_slug`, `env_slug`, `org_id`, `project_id`, `project_ref`,
`environment_id`, `env_type` |
| `/orgs/:o/settings` (non-env-scoped) | real user cuid | `org_slug`
only |
| API request with `orgMember` | `orgMember.userId` | full tenant set |
| API request without `orgMember` | (unset) | full tenant set |

## Trade-offs

1. On env-scoped pages, errors that fire before the env layout loader's
enrich callback runs get slugs + `user.id` but not the tenant IDs /
`env_type`. Realistic errors deep in async work get the full set. (Same
race as before, narrower window now that slugs/`user.id` are populated
up-front by the middleware and `_app` enrich.)
2. API requests where the environment has no `orgMember` get tenant tags
but no `user.id`. Those events still show in the issue but don't
contribute to "Users Impacted".

## Out of scope (deferred)

Background workers (`redis-worker`, `schedule-engine`) and socket
handlers. Those entry points don't set `tenantContext.run` yet — their
events ship without tenant attribution until each is wired in a
follow-up.

## Tests

31 unit tests across 4 files. New tests notably cover:

- `parseTenantPath`: org-only, org+project, and full-triple URL
variants.
- `tenantContext.enrich`: in-place patch, no-op outside `run()`,
concurrent-scope isolation, empty-scope + enrich pattern (for non-tenant
pages).
- `tenantContextFromAuthEnvironment`: with and without `orgMember` —
verifies the API path's `user.id` mapping.
- `addTenantContextToEvent`: empty scope, userId-only, slugs-only, full
enrichment, conditional tag emission, preservation of prior `event.user`
fields.

## Test plan

- [ ] `pnpm run typecheck --filter webapp`
- [ ] `pnpm run test --filter webapp -- test/tenantContext.test.ts
test/sentryTenantContext.test.ts test/tenantContextResolver.test.ts
test/tenantContextFromAuthEnvironment.test.ts`
- [ ] Local manual: with `SENTRY_DSN` set, hit a dashboard URL and an
API route, confirm the captured events carry `user.id` + the expected
tag set in Sentry.
- [ ] After ship: confirm "Users Impacted" on a real Sentry issue
reflects distinct users (not tenants).

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-05-22 15:10:41 +01:00
Matt Aitken 71d98b4e6b Support for org-scoped ClickHouse (#3333) 
Added `OrganizationDataStore` which allows orgs to have data stored in
specific separate services.

For now this is just used for ClickHouse. When using ClickHouse we get a
client for the factory and pass in the org id.

Particular care has to be made with two hot-insert paths:
1. RunReplicationService
2. OTLPExporter

---------

Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: Claude <noreply@anthropic.com>
 2026-05-22 14:43:03 +01:00
Dan Sutton 449ded00af docs(mollifier): move ops manual to _ops/ 
The docs/ directory is the public Mintlify customer docs site; internal
operational runbooks belong elsewhere. Move the mollifier ops manual
to _ops/ alongside the _plans/ working-doc convention.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-22 14:38:29 +01:00
Dan Sutton 3549563cf1 docs(mollifier): ops manual with alertable signals and recovery flows 
Captures what the metrics mean, which signal is alertable (the
`stale_entries.current` gauge, not the counter), each named failure
mode and its recovery flow, and Redis debug commands for poking at
the buffer by hand. Mirrors the `batch-queue-metrics.md` internal-doc
style.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-22 14:36:15 +01:00
Dan Sutton af85cdaea4 feat(webapp): alertable gauge for mollifier stale-entry signal 
The mollifier.stale_entries counter from the previous commit reflects
sweep-tick events, not stable state. A single stuck entry observed
across N ticks contributes N events, so a rate() query is
proportional to (stuck-entry-count × scan-frequency), not "how many
entries are stale right now". Useful for historical views but the
wrong shape for ops alerts.

Add a companion observable gauge `mollifier.stale_entries.current`
with `{envId}` attribute. The sweep emits a per-env snapshot on each
pass (including zero counts for envs whose stale entries cleared),
and an OTel batch-observable callback exposes the latest snapshot to
the metric exporter on every scrape. Recommended alert:

  mollifier_stale_entries_current{envId=...} > 0 for 5m

The snapshot replaces (not merges) so an env that paged on a
previous sweep clears when the drainer catches up, instead of
staying latched at the last stale count. Test seam captures the
snapshot to verify per-env counts and the clear-on-drain behaviour.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-22 14:33:22 +01:00
Dan Sutton 14253dc212 feat(webapp): mollifier stale-entry sweep + OTel signal 
Without an external signal that the drainer is falling behind, a stuck
or offline drainer drives the buffer toward the entry-hash TTL line
and runs vanish silently — no PG row, no log, no dashboard
indication. Add a periodic read-only sweep over the buffer's queue
ZSETs that emits a `mollifier.stale_entries` OTel counter and a
structured `mollifier.stale_entry` warning log for each entry whose
dwell exceeds the configured threshold.

Independent of the drainer (its own gate +
`TRIGGER_MOLLIFIER_STALE_SWEEP_ENABLED`) so an entirely offline
drainer is exactly when the sweep is most useful. Defaults: interval
5min, threshold half of `entryTtlSeconds`, hard cap of 1000 entries
per env per pass.

Sweep is strictly read-only — does not remove or salvage entries.
The retention-policy question (drop the entry TTL entirely vs raise
it vs pre-TTL salvage) is intentionally deferred to a separate
change; this commit gets the signal in place first.

Tested with a real `MollifierBuffer` (testcontainers): stale entries
flagged, fresh entries left alone, multi-org scan walks every queue.
Manually verified end-to-end: with a 10s interval + 2s threshold,
each tick logs the buffered run with growing dwellMs as expected.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-22 14:20:45 +01:00
Dan Sutton 69e8535106 test(webapp): pin realtime buffered-resource resolution 
The previous commit's regression coverage was thin: only the log-dedup
gate was unit-tested. The load-bearing logic — synthesise a resource
when PG misses but the buffer has the run, with an `id` matching what
the drainer will eventually write — had no regression test, so a
future change that removed the buffered fallback would put the silent-
hang back into prod without anything failing in CI.

Extract the resource-resolution rules from the route's findResource
into `resolveRealtimeRunResource`, a pure function. Cover the
branching with unit tests (PG hit, PG hit during drain race, PG miss
+ buffer hit, missing taskIdentifier default, both miss) and pin the
full chain with a container-backed test that uses a real
MollifierBuffer + the real readFallback helper and asserts the
synthesised `id` matches `RunId.fromFriendlyId(friendlyId)`. That
identity is what Electric's `WHERE id='<id>'` clause depends on when
the drainer eventually INSERTs the row.

12 tests total across the three Phase-5.2 suites; one empirical probe
run after the refactor confirmed end-to-end behaviour unchanged.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-22 13:55:13 +01:00
Dan Sutton 50106dc892 fix(webapp): keep useRealtimeRun stream open across the buffered window 
Customers subscribing to a freshly-triggered run via useRealtimeRun
silently hung when the gate diverted the run into the mollifier buffer.
The route's findResource looked up the PG TaskRun by friendlyId, found
nothing, and returned 404. Electric SQL's ShapeStream treats the
initial 404 as terminal — no retry, no error surfaced to the hook,
and crucially no recovery after the drainer eventually INSERTed the
PG row. The customer's component shows the empty state indefinitely
even though the run is alive and progressing.

When the PG lookup misses but the buffer has the run, return a
synthetic resource whose `id` is derived from the friendlyId — the
same value engine.trigger will write when the drainer materialises
this run. The route then opens the Electric subscription against
`WHERE id='<id>'`, Electric streams an empty initial snapshot, and
the SDK long-polls until the drainer's INSERT propagates through.
Empirically validated end-to-end: trigger a buffered run, open the
subscription, simulate the drainer's PG INSERT + UPDATE, and the
SDK iterator yields the QUEUED and EXECUTING events in real time.

Adds a `mollifier.realtime_subscriptions.buffered` counter and a
structured log line. The observability gate fires once per cold
subscription (Electric's `handle` query param is the dedup signal),
not on every ~20s long-poll reconnect; that gate is unit-tested.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-22 13:43:53 +01:00
Dan Sutton 1ece983e95 revert(webapp): drop buffered scan from bulk-action service 
The bulk-action confirmation count is sourced from ClickHouse, so PG
rows not yet replicated to ClickHouse are silently excluded from both
the count and the processing pass. Phase 4's first-batch mollifier-
buffer scan broke that symmetry — buffered runs were processed without
being counted, so a customer confirming "Replay ~0 runs" could see N
buffered runs replayed without seeing them anywhere in the UI.

Restore the eventually-consistent contract: bulk actions only target
runs visible to ClickHouse. Buffered runs are picked up by subsequent
bulk actions once they drain into PG → ClickHouse, mirroring how
PG-not-yet-CH runs already work today.

Removes `bulkActionBuffer.server.ts` (helper) and its container-backed
test. Will reimplement once the buffered-runs UX (global status
indicator) gives the customer a way to see and confirm against the
buffered set.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-22 13:01:13 +01:00
Eric Allam 832cf7220b feat(sdk,core): add TriggerClient for per-instance SDK configuration (#3683) 
## Summary

`new TriggerClient({...})` exposes the management API (tasks, runs,
schedules, envvars, batch, queues, deployments, prompts, auth) as an
explicit instance with its own auth, preview branch, and baseURL.
Multiple clients can coexist in one process without mutating shared
global state — useful when a single service triggers across multiple
projects, environments, or preview branches.

```ts
import { TriggerClient } from "@trigger.dev/sdk";

const prod = new TriggerClient({ accessToken: process.env.TRIGGER_PROD_KEY });
const preview = new TriggerClient({
  accessToken: process.env.TRIGGER_PREVIEW_KEY,
  previewBranch: "signup-flow",
});

await prod.tasks.trigger("send-email", payload);
await preview.runs.list({ status: ["COMPLETED"] });
```

The existing global `configure()` API keeps working unchanged.

## Design

Instance methods enter an `AsyncLocalStorage`-backed scope (`sdkScope`)
before delegating to the existing module-level functions. The four
"pollution" points that previously read globals now consult the scope
first:

- `apiClientManager.{baseURL, accessToken, branchName}` and
`clientOrThrow` — identity fields are scope-only when scoped; `baseURL`
still falls back to `TRIGGER_API_URL` because plumbing (where the API
lives) is not identity.
- `taskContext.{ctx, worker, isWarmStart, isInsideTask}` — masked inside
an isolated scope so a `client.tasks.trigger(...)` from inside a task
doesn't leak the parent's `parentRunId` / `lockToVersion` / `isTest`
into a trigger that hits a different project.
- Inline `getEnvVar("TRIGGER_VERSION")` reads in `shared.ts` go through
a `scopedEnvVar` helper that returns `undefined` inside an isolated
scope.

The `TriggerClient` class itself is a thin wrapper that captures the
scope in its constructor and proxies each namespace method to enter that
scope before calling the existing impl. Generic inference (e.g.
`client.tasks.trigger<typeof t>(...)`) is preserved via `Pick<typeof ns,
keyof curatedSubset>` typings.

Two correctness fixes uncovered along the way are folded in:

- `apiClientManager.setGlobalAPIClientConfiguration` no longer silently
no-ops on the second call. `configure()` now actually overrides as users
expect (this is the root cause behind some "I changed the config but
nothing happened" reports).
- `apiClientManager.runWithConfig` (and therefore `auth.withAuth`) is
now backed by `sdkScope.withScope` instead of "mutate the global and
restore in finally". Two parallel `withAuth` calls with different
configs no longer stomp each other.

Surface curation: instance namespaces drop methods that don't make sense
per-instance — `batch.*AndWait` (runtime-dependent), `schedules.task` /
`schedules.timezones` (definition-time / stateless), `prompts.define`
(definition-time), `auth.configure` / `auth.withAuth` (global-only).

## Test plan

- [x] 9 runtime unit tests in `triggerClient.test.ts` cover: required
accessToken, instance auth + branch headers, no env fallback for
identity fields, no leakage between global and instance, four parallel
calls across two clients stay isolated, taskContext masking +
`inheritContext: true` override, `configure()` second-call override,
parallel `auth.withAuth` isolation.
- [x] 10 type-level assertions in `triggerClient.types.test.ts` using
`expectTypeOf` + `@ts-expect-error` lock in generic inference, return
type passthrough, overload preservation, and curated-surface drift.
- [x] Full SDK suite (219 tests) and core suite (530 tests) pass.
- [x] Webapp typecheck clean.
- [x] End-to-end smoke test against local webapp and a
freshly-provisioned cloud project — six concurrent multi-client triggers
all returned 200 with run IDs, headers per-client as expected.
- [ ] Reviewer: run `references/multi-client` per its `README.md` to
reproduce the smoke test locally.

## Try it

`references/multi-client` is a new reference workspace that exercises
this end-to-end:

- `src/trigger/echo.ts` — trivial target task
- `src/trigger/fanOut.ts` — opens two `TriggerClient`s from inside a
task, fires `echo` through each in parallel
- `src/external/main.ts` — external Node script with two clients
triggering `echo` sequentially and concurrently; logs every outgoing
request's `authorization` + `x-trigger-branch`
- `src/external/isolation.ts` — interleaves global `configure()` and an
instance call, asserts the captured fetch sequence shows no leakage
either way
 2026-05-22 12:29:42 +01:00
Dan Sutton 4408743570 feat(webapp): open run span before mollifier gate 
Today the run span (the SERVER trace event keyed by runId) is created
inside `traceEventConcern.traceRun`, which sits *after* the mollifier
gate. When the gate diverts a trigger into the Redis buffer, the run
span is therefore not written to the event store until the drainer
replays the snapshot — buffered runs are invisible in the trace view,
parents' trace trees miss the child until drain, and alerting
pipelines can't reference the run.

Hoist the gate evaluation and mollify branch inside `traceRun` so both
paths open the run span. The mollify branch records mollifier
attributes on the same event, captures `event.traceId`/`event.spanId`
into the buffer snapshot (replacing the separately-allocated
`mollifier.queued` OTel span), and returns the synthesised result.
`traceRun` flushes the PARTIAL event to the store on callback return.

Extend the existing call-site test to assert (a) traceRun fires before
buffer.accept and (b) the snapshot's traceId/spanId match the run
span's IDs. The MockTraceEventConcern now mirrors the production
ClickhouseEventRepository shape so the `traceContext.traceparent`
assertion exercises the seeding path.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-22 11:48:07 +01:00
Dan Sutton fbd5d2f9dc revert(webapp): drop mollifier listing-merge from runs list 
The runs list (API and dashboard) is eventually consistent — buffered
runs were creating a sandwich problem where the head of the list could
include buffered rows while in-transit rows between PG replication and
ClickHouse went missing. Drop the merge so the list returns PG/
ClickHouse rows only; buffered visibility will return via a separate
global status indicator.

Reverts the merge wiring in api.v1.runs, api.v1.projects.$projectRef
.runs, and the dashboard runs index, and deletes listingMerge.server
and dashboardListingMerge.server. The MCP list_runs tool rides through
the API and inherits the same behaviour.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-22 11:16:01 +01:00
nicktrn 18d7144e74 ci: daily dependabot critical-severity slack alerts (#3701) 
Sibling to the weekly summary, focused on critical alerts only. Pings
Slack daily while any critical alerts are open; skips the post entirely
when zero, so no daily "all clear" noise.

- Daily 08:00 UTC cron + `workflow_dispatch` with `severity` input
(default `critical`, override to `high`/`medium`/`low` for manual
checks)
- Reuses the existing `dependabot-summary` environment (token, channel,
bot)
- Alerts link at the end is severity-filtered
 2026-05-22 10:32:08 +01:00
Eric Allam c0b9fdfce9 docs(ai-chat): clarify lastEventId is sessionId-keyed across run boundaries (#3700) 
## Summary

Two docs edits that close a footgun customers persisting transport state
can hit. Clearing `lastEventId` on `chat.endRun()` looks intuitive — the
Run ended, the cursor must be stale — but the cursor is sessionId-keyed,
not runId-keyed. Clearing it forces the next `sendMessages` to subscribe
from `seq_num=0`, which may hit the prior turn's still-durable
`turn-complete` record and close the SSE empty before the new Run's
chunks arrive.

Spells out the invariant in the frontend transport persistence table and
adds a Warning in the `chat.endRun()` reference.

## Test plan

- [x] Mintlify preview renders
- [x] No callout stacking
 2026-05-22 10:26:06 +01:00
Dan Sutton d4b55c1893 feat(webapp): write SYSTEM_FAILURE PG row when drainer hits a non-retryable error 
Previously, a non-retryable engine.trigger failure during drain left
the buffer entry as `status: "FAILED"` in Redis with no PG row. The
customer saw the run in their SDK / dashboard listing for ~10 min
(buffer TTL) then it vanished entirely — no audit trail of the
failure. Billing was unaffected (no attempts ever ran) but
observability was zero.

Reuse the engine's existing `createFailedTaskRun` helper (the same one
batch-trigger calls when an item fails to start) — writes a terminal
SYSTEM_FAILURE TaskRun row with the engine.trigger error stored on
`error`, no attempts, P2002-idempotent on the unique constraint.

Drainer handler classifies the failure:
- Retryable PG error → rethrow so MollifierDrainer.drainOne requeues
- Non-retryable → createFailedTaskRun, swallow original error so the
  buffer entry is ack'd (PG now has the audit row)
- createFailedTaskRun also fails (PG truly unreachable) → rethrow
  original so drainer falls through to its existing buffer.fail
  terminal-marker path
- Snapshot too malformed to construct the environment block → rethrow
  (defensive — drainer falls through to buffer.fail)

Tests cover each path.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-22 10:07:54 +01:00
Dan Sutton 34536182f8 fix(webapp): align buffered API responses with existing SDK schemas 
A broader audit of every public API route's buffered branch found a
handful of schema-drift bugs the SDK would reject on existing clients:

- /api/v1/runs/{id}/spans/{spanId} returned `parentId: undefined`
  (omitted in JSON). Schema declares `parentId: z.string().nullable()`
  — present-but-null is required. Send `null` explicitly. Also reflect
  the snapshot's cancelled state in `isPartial` / `isCancelled`.

- /api/v1/runs/{id}/reschedule's buffered branch returned a stripped
  `{ id, delayUntil }`. The SDK's `rescheduleRun` validates against the
  full `RetrieveRunResponse` shape. Route the buffered response through
  the same ApiRetrieveRunPresenter the PG branch uses (which falls back
  to the buffer for synthetic runs). Allows `synthesisedResponse` in
  `mutateWithFallback` to be async.

- ApiRetrieveRunPresenter.synthesiseFoundRunFromBuffer ignored the
  snapshot's `cancelledAt` and `delayUntil`. Status was hardcoded to
  `PENDING` regardless of cancellation; `completedAt` and `delayUntil`
  were always `null`. SDK callers (and the MCP cancel_run helper)
  reported status as Queued after a successful cancel. Map the synthetic
  status through a small switch so CANCELED, SYSTEM_FAILURE and PENDING
  all surface correctly.

- Add `delayUntil` to SyntheticRun so set_delay reschedule patches
  survive the next retrieve. Mirror it onto the dashboard SpanRun
  synthesiser too.

Verified end-to-end by replaying every public-API method against a
buffered run.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-22 09:24:29 +01:00
Dan Sutton 49b9d0053b fix(webapp): make buffered API responses match SDK response shapes 
Two SDK schemas were drifting from what the mollifier paths emitted:

1. ListRunResponseItem declares `idempotencyKey: z.string().optional()`
   (omit-or-string). The listing-merge synthesiser was emitting
   `idempotencyKey: null` for buffered runs, which old SDK versions
   reject with a validation error before surfacing the row.

2. RetrieveRunTraceResponseBody declares a non-nullable `rootSpan`
   matching the recursive RetrieveRunTraceSpan shape. The buffered
   branch of /api/v1/runs/{id}/trace returned `rootSpan: null` plus an
   `events: []` field that isn't in the schema. Synthesise a real
   partial span (task identifier as message, no children, isPartial:
   true) from the buffer snapshot so the response satisfies the schema
   the SDK validates against.

Verified end-to-end by calling the MCP server's list_runs and
get_run_details against a buffered run; both now succeed.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-22 09:08:12 +01:00
Dan Sutton a15566ddce fix(webapp): control cancel-run Dialog state so submit isn't raced by Radix DialogClose 
Previous attempt wrapped the form's submit button in <DialogClose
asChild> so the dialog closed on click. That race-condition'd with
Remix's <Form>: Radix's Slot-attached onClick triggered
onOpenChange(false), the Dialog and its child Form unmounted mid-cycle,
and the button's name=value pair (carrying `redirectUrl`) was dropped
from the submitted FormData. The action then read
`submission.value.redirectUrl` as undefined and the resulting redirect
landed on `/env/dev` instead of the run-detail page.

Switch to a ControlledCancelRunDialog at the call site that owns the
Radix `open` state. The inner CancelRunDialog watches the navigation
state transitions and signals the parent to close the dialog once the
submission has captured its submitter cleanly. Submit-button name=value
is preserved; redirect resolves to the run-detail page; modal still
dismisses after submit.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-22 08:55:24 +01:00
Dan Sutton 75c6e2b414 fix(webapp): replay dialog falls back to the mollifier buffer 
The replay form loader hit `taskRun.findFirst` and threw 404 when the
run was buffered, which dumps the user back to the task list. Wire a
buffer fallback that synthesises the same loader return shape from the
snapshot, including a project-and-environments lookup scoped by the
buffer entry's orgId so the env selector renders identically. The
replay action itself already supports buffered runs via the
ReplayTaskRunService synthetic-run cast — only the form's preflight
load was broken.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-21 18:33:59 +01:00
Dan Sutton a1fe5f7419 feat(webapp): merge mollifier-buffered runs into the dashboard runs list 
Buffered runs are prepended to the runs table on the runs list page so
customers see freshly-triggered work even while the gate is diverting.
The merge uses a compound base64 cursor that wraps the PG presenter's
own cursor — page 1 can be entirely buffered (top of the list), page 2
takes the buffered overflow and transitions into the PG content, and
later pages drop the buffer scan entirely once it's been exhausted.

Filter predicates (tasks, statuses, tags, period, from/to, isTest,
runId) are evaluated against the buffer snapshot so the list reflects
the same filter scope as the PG-side query.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-21 17:33:28 +01:00
Dan Sutton 2052d3eecd fix(webapp): dismiss cancel dialog on submit; reflect cancelled state in synthetic SpanRun 
The cancel dialog stayed open after a successful submit because it was
uncontrolled Radix state and the action redirects to the same URL —
revalidation didn't trigger a re-mount. Wrap the submit button in
DialogClose so the click closes the dialog at the same time the form
posts.

The SyntheticRun synthesised for the run-detail page hardcoded status
PENDING regardless of whether the buffer snapshot had cancelledAt set.
Customers cancelling a buffered run saw their run still labelled Queued
until the drainer materialised it. Surface cancelledAt + cancelReason
on SyntheticRun, switch the synthesised SpanRun status to CANCELED, and
mirror the cancelled flag onto the single-span trace so the timeline
matches PG behaviour.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-21 17:10:30 +01:00
Eric Allam c80b85e2cf docs(ai-chat): atomic onTurnComplete writes + Anthropic prose (#3693) 
## Summary

Three post-merge fixes for the AI Agents docs (#3226), all caught by
review after merge.

## Fixes

- **`onTurnComplete` examples now use `db.$transaction`** — both the
Database persistence "Complete example" and the Lifecycle hooks
reference example were doing two separate `await` calls
(`db.chat.update` then `db.chatSession.upsert`). That's the exact
non-atomic pattern the warning earlier on the persistence page calls out
as ❌: a refresh between the two writes reads a stale `lastEventId` and
duplicates the assistant message on resume. Both examples now use the
recommended atomic form.

- **Background injection self-review prose aligned with the code** — the
prose said "gpt-4o-mini" but the example above it had been swapped to
`claude-haiku-4-5`. The Anthropic-sweep script only touched code blocks;
this prose line wasn't picked up.

## Test plan

- [x] Both updated examples use `db.$transaction([...])`
- [x] Prose matches the model used in the code block
- [ ] Mintlify deployment passes
 2026-05-21 17:04:13 +01:00
Dan Sutton 213185bbfe feat(webapp): dashboard parity for mollifier-buffered runs 
Synthesise the SpanRun shape from buffer snapshots so the run-detail
page's inspector panel renders identically to a PG-resident run. SSE
log stream, realtime stream resources, logs-download and debug resource
fall back to the buffer instead of 404-ing. Short-URL redirects resolve
buffered runs to the canonical dashboard URL. Bulk-cancel scans the
buffer alongside the ClickHouse selection so runs queued mid-burst are
included. Trigger response now carries the snapshot's spanId so the
dashboard's Run Test redirect opens the details panel without an extra
click.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-21 16:53:21 +01:00
Eric Allam 80bb600cd5 docs(ai-chat): AI Agents documentation for v4.5 (#3226) 
## Summary

Lands the full AI Agents documentation surface alongside the v4.5
release candidate of `@trigger.dev/sdk`. Covers `chat.agent` end to end
— defining agents, lifecycle hooks, the frontend transport, sub-agents,
recovery from cancel/crash/OOM, AI Prompts integration — and the
Sessions primitive that backs it.

## Coverage

- **Conceptual**: Overview, Quick Start, How it works.
- **Building agents**: Backend (`chat.agent` / `chat.createSession` /
raw primitives), Lifecycle hooks, Frontend transport, Server-side
`AgentChat`, Sessions reference, `chat.local` state primitive,
TypeScript types.
- **Features**: AI Prompts integration, Fast starts (Preload + Head
Start), Compaction, Pending Messages (steering), Background Injection
(`chat.inject` + `chat.defer`), Actions (undo / regenerate / edit),
Error handling.
- **Patterns (13)**: Sub-agents, Branching conversations, Code sandbox,
Database persistence, Persistence and replay, HITL, Tool result
auditing, Large payloads, Agent skills, OOM resilience, Recovery boot,
Trusted edge signals, Version upgrades.
- **Reference**: API Reference, Client Protocol (wire format), Testing
harness (`mockChatAgent`), MCP server tools, Upgrade guide, Changelog.

## Structure changes

- Top-level nav: AI → **Agents**, with sub-groups for *Building agents /
Features / Patterns / Reference*.
- New RC banner snippet on every page links to the supported AI SDK
versions table on the API Reference.
- All examples use Anthropic with `stopWhen: stepCountIs(15)`.

---------

Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-21 16:50:59 +01:00
github-actions[bot] 0b5b817a49 chore: release v4.5.0-rc.1 (#3691)
🚀 Publish Trigger.dev Docker / typecheck (push) Failing after 0s
Details
🚀 Publish Trigger.dev Docker / publish-webapp (push) Has been skipped
Details
🚀 Publish Trigger.dev Docker / publish-worker (push) Has been skipped
Details
🚀 Publish Trigger.dev Docker / publish-worker-v4 (push) Has been skipped
Details
🚀 Publish Trigger.dev Docker / units (push) Failing after 0s
Details
🧭 Helm Chart Release / lint-and-test (push) Has been cancelled
Details
🧭 Helm Chart Release / release (push) Has been cancelled
Details 
## Summary
2 bug fixes.

## Bug fixes
- Fix `chat.agent` skills silently missing in `trigger dev` for projects
whose task files read `process.env` at module top level (e.g. a
third-party SDK client initialized at import). Skill folders now bundle
into `.trigger/skills/` reliably regardless of which env vars are set
when the CLI launches.
([#3690](https://github.com/triggerdotdev/trigger.dev/pull/3690))
- Fix `COULD_NOT_FIND_EXECUTOR` when a task's definition is loaded via
`await import(...)` from inside another task's `run()`. The runtime
workers now register such tasks with a sentinel file context, and the
catalog logs a one-time warning per task id.
([#3688](https://github.com/triggerdotdev/trigger.dev/pull/3688))

<details>
<summary>Raw changeset output</summary>

⚠️⚠️⚠️⚠️⚠️⚠️

`main` is currently in **pre mode** so this branch has prereleases
rather than normal releases. If you want to exit prereleases, run
`changeset pre exit` on `main`.

⚠️⚠️⚠️⚠️⚠️⚠️

# Releases
## @trigger.dev/build@4.5.0-rc.1

### Patch Changes

-   Updated dependencies:
    -   `@trigger.dev/core@4.5.0-rc.1`

## trigger.dev@4.5.0-rc.1

### Patch Changes

- Fix `chat.agent` skills silently missing in `trigger dev` for projects
whose task files read `process.env` at module top level (e.g. a
third-party SDK client initialized at import). Skill folders now bundle
into `.trigger/skills/` reliably regardless of which env vars are set
when the CLI launches.
([#3690](https://github.com/triggerdotdev/trigger.dev/pull/3690))
- Fix `COULD_NOT_FIND_EXECUTOR` when a task's definition is loaded via
`await import(...)` from inside another task's `run()`. The runtime
workers now register such tasks with a sentinel file context, and the
catalog logs a one-time warning per task id.
([#3688](https://github.com/triggerdotdev/trigger.dev/pull/3688))
-   Updated dependencies:
    -   `@trigger.dev/core@4.5.0-rc.1`
    -   `@trigger.dev/build@4.5.0-rc.1`
    -   `@trigger.dev/schema-to-json@4.5.0-rc.1`

## @trigger.dev/core@4.5.0-rc.1

### Patch Changes

- Fix `COULD_NOT_FIND_EXECUTOR` when a task's definition is loaded via
`await import(...)` from inside another task's `run()`. The runtime
workers now register such tasks with a sentinel file context, and the
catalog logs a one-time warning per task id.
([#3688](https://github.com/triggerdotdev/trigger.dev/pull/3688))

## @trigger.dev/plugins@4.5.0-rc.1

### Patch Changes

-   Updated dependencies:
    -   `@trigger.dev/core@4.5.0-rc.1`

## @trigger.dev/python@4.5.0-rc.1

### Patch Changes

-   Updated dependencies:
    -   `@trigger.dev/core@4.5.0-rc.1`
    -   `@trigger.dev/build@4.5.0-rc.1`
    -   `@trigger.dev/sdk@4.5.0-rc.1`

## @trigger.dev/react-hooks@4.5.0-rc.1

### Patch Changes

-   Updated dependencies:
    -   `@trigger.dev/core@4.5.0-rc.1`

## @trigger.dev/redis-worker@4.5.0-rc.1

### Patch Changes

-   Updated dependencies:
    -   `@trigger.dev/core@4.5.0-rc.1`

## @trigger.dev/rsc@4.5.0-rc.1

### Patch Changes

-   Updated dependencies:
    -   `@trigger.dev/core@4.5.0-rc.1`

## @trigger.dev/schema-to-json@4.5.0-rc.1

### Patch Changes

-   Updated dependencies:
    -   `@trigger.dev/core@4.5.0-rc.1`

## @trigger.dev/sdk@4.5.0-rc.1

### Patch Changes

-   Updated dependencies:
    -   `@trigger.dev/core@4.5.0-rc.1`

</details>

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
helm-v4.5.0-rc.1 v.docker.4.5.0-rc.1 v4.5.0-rc.1 		2026-05-21 16:40:21 +01:00
Eric Allam 5ddb81ac8d fix(cli): stop chat.agent skills silently disappearing from trigger dev (#3690) 
## Summary

`trigger.dev dev` was silently dropping registered `chat.agent` skills
for any project whose task files read `process.env` at module top level
— e.g. a third-party SDK client initialized at import. The agent would
boot fine, but `skill.local()` failed at runtime with `ENOENT` because
the skill folder was never copied into `.trigger/skills/`.

## Design

The CLI ran two indexer passes in dev: the worker's own indexer (with
the full env it eventually executes tasks in), and a separate
skill-discovery indexer with only the CLI process's env. Top-level reads
of vars like `TRIGGER_API_URL` imported cleanly in the worker pass and
threw in the skill pass — the latter caught the error, warned, and
skipped skill copying. Failure was silent enough that `skill.local()`
only surfaced it at task runtime.

The skill registry is already part of the worker manifest. This PR drops
the duplicate pass and copies skill folders from that manifest after the
worker initializes. One indexer instead of two; a bad `SKILL.md` now
surfaces as a startup error instead of silently disappearing skills.

Deploy is unaffected — its skill discovery uses the project's
environment variables (fetched via the API, which fills in
`TRIGGER_API_URL` etc.), so the dev failure mode doesn't reach there.

## Test plan

- [x] New `references/agent-skills` reference project with
`skills.define` + a task that calls `skill.local()` and runs a bundled
script
- [x] On `main`, adding a top-level
`process.env.TRIGGER_API_URL!.includes(...)` read in any task file
reproduces the symptom: warning at dev startup, no `.trigger/skills/`
folder, `skill.local()` fails with ENOENT
- [x] On this branch, same project boots clean and `skill.local()` works
end-to-end
- [x] Deploy still works end-to-end with the new reference project
 2026-05-21 16:30:24 +01:00
nicktrn dbf9b4e7e5 fix(core,cli): register tasks loaded via dynamic import during run (#3688) 
On a warm worker process, a task whose `task()` definition is loaded via
`await import(...)` from inside another task's `run()` could end up
permanently missing from the catalog: the `task()` call fired with no
`_currentFileContext` set, `registerTaskMetadata` silently returned, and
Node's ESM module cache then blocked the worker's setContext + re-import
recovery from ever firing the call again. Subsequent runs of that task
on the same warm process failed with `COULD_NOT_FIND_EXECUTOR` until the
process hit `maxExecutionsPerProcess` and exited.

All five of these had to coincide on the same worker for the bug to
surface:

1. `processKeepAlive` enabled (so catalog state survives across runs).
2. A `run()` function (or lifecycle hook) does `await import(...)`.
3. The import's transitive static graph reaches a `task()` /
`schemaTask()` call.
4. The task containing the dynamic import is the **first** task to run
on a given warm worker process — so the dropped `task()` calls fire on
this process for the first time, are silently dropped, and Node's module
cache locks the wrong outcome in.
5. A subsequent run for one of the dropped task ids lands on the same
warm worker before it recycles.

The runtime workers now set a sentinel file context (`<no-context>`)
around the `executor.execute(...)` call, so `task()` invocations firing
during a run register normally. The catalog detects the sentinel and
emits a one-time `console.warn` per task id so the pattern stays visible
without spamming. The indexer never sets this context, so deploy-time
behavior is unchanged.

Repro is `references/hello-world/src/trigger/dynamicImportRepro*.ts`.
Verified end-to-end against a deployed image with firestarter
warm-starts on: pre-fix saw `COULD_NOT_FIND_EXECUTOR` on children that
landed on the parent-poisoned worker; post-fix all 23/23 runs succeeded
and the warning surfaces in the parent's run trace.
 2026-05-21 16:21:49 +01:00
github-actions[bot] acfba02409 chore: release v4.5.0-rc.0 (#3563)
🚀 Publish Trigger.dev Docker / typecheck (push) Failing after 4s
Details
🚀 Publish Trigger.dev Docker / units (push) Failing after 0s
Details
🚀 Publish Trigger.dev Docker / publish-webapp (push) Has been skipped
Details
🚀 Publish Trigger.dev Docker / publish-worker (push) Has been skipped
Details
🚀 Publish Trigger.dev Docker / publish-worker-v4 (push) Has been skipped
Details
🧭 Helm Chart Release / lint-and-test (push) Has been cancelled
Details
🧭 Helm Chart Release / release (push) Has been cancelled
Details 
## Summary
44 improvements, 1 bug fix.

## Improvements
- **AI Prompts** — define prompt templates as code alongside your tasks,
version them on deploy, and override the text or model from the
dashboard without redeploying. Prompts integrate with the Vercel AI SDK
via `toAISDKTelemetry()` (links every generation span back to the
prompt) and with `chat.agent` via `chat.prompt.set()` +
`chat.toStreamTextOptions()`.
([#3629](https://github.com/triggerdotdev/trigger.dev/pull/3629))
- **Code-defined, deploy-versioned templates** — define with
`prompts.define({ id, model, config, variables, content })`. Every
deploy creates a new version visible in the dashboard. Mustache-style
placeholders (`{{var}}`, `{{#cond}}...{{/cond}}`) with Zod / ArkType /
Valibot-typed variables.
- **Dashboard overrides** — change a prompt's text or model from the
dashboard without redeploying. Overrides take priority over the deployed
"current" version and are environment-scoped (dev / staging / production
independent).
- **Resolve API** — `prompt.resolve(vars, { version?, label? })` returns
the compiled `text`, resolved `model`, `version`, and labels. Standalone
`prompts.resolve<typeof handle>(slug, vars)` for cross-file resolution
with full type inference on slug and variable shape.
- **AI SDK integration** — spread `resolved.toAISDKTelemetry({ ...extra
})` into any `generateText` / `streamText` call and every generation
span links to the prompt in the dashboard alongside its input variables,
model, tokens, and cost.
- **`chat.agent` integration** — `chat.prompt.set(resolved)` stores the
resolved prompt run-scoped; `chat.toStreamTextOptions({ registry })`
pulls `system`, `model` (resolved via the AI SDK provider registry),
`temperature` / `maxTokens` / etc., and telemetry into a single spread
for `streamText`.
- **Management SDK** — `prompts.list()`, `prompts.versions(slug)`,
`prompts.promote(slug, version)`, `prompts.createOverride(slug, body)`,
`prompts.updateOverride(slug, body)`, `prompts.removeOverride(slug)`,
`prompts.reactivateOverride(slug, version)`.
- **Dashboard** — prompts list with per-prompt usage sparklines;
per-prompt detail with Template / Details / Versions / Generations /
Metrics tabs. AI generation spans get a custom inspector showing the
linked prompt's metadata, input variables, and template content
alongside model, tokens, cost, and the message thread.
- Adds `onBoot` to `chat.agent` — a lifecycle hook that fires once per
worker process picking up the chat. Runs for the initial run, preloaded
runs, AND reactive continuation runs (post-cancel, crash, `endRun`,
`requestUpgrade`, OOM retry), before any other hook. Use it to
initialize `chat.local`, open per-process resources, or re-hydrate state
from your DB on continuation — anywhere the SAME run picking up after
suspend/resume isn't enough.
([#3543](https://github.com/triggerdotdev/trigger.dev/pull/3543))
- **AI SDK `useChat` integration** — a custom
[`ChatTransport`](https://sdk.vercel.ai/docs/ai-sdk-ui/transport)
(`useTriggerChatTransport`) plugs straight into Vercel AI SDK's
`useChat` hook. Text streaming, tool calls, reasoning, and `data-*`
parts all work natively over Trigger.dev's realtime streams. No custom
API routes needed.
- **First-turn fast path (`chat.headStart`)** — opt-in handler that runs
the first turn's `streamText` step in your warm server process while the
agent run boots in parallel, cutting cold-start TTFC by roughly half
(measured 2801ms → 1218ms on `claude-sonnet-4-6`). The agent owns step
2+ (tool execution, persistence, hooks) so heavy deps stay where they
belong. Web Fetch handler works natively in Next.js, Hono, SvelteKit,
Remix, Workers, etc.; bridge to Express/Fastify/Koa via
`chat.toNodeListener`. New `@trigger.dev/sdk/chat-server` subpath.
- **Multi-turn durability via Sessions** — every chat is backed by a
durable Session that outlives any individual run. Conversations resume
across page refreshes, idle timeout, crashes, and deploys; `resume:
true` reconnects via `lastEventId` so clients only see new chunks.
`sessions.list` enumerates chats for inbox-style UIs.
- **Auto-accumulated history, delta-only wire** — the backend
accumulates the full conversation across turns; clients only ship the
new message each turn. Long chats never hit the 512 KiB body cap.
Register `hydrateMessages` to be the source of truth yourself.
- **Lifecycle hooks** — `onPreload`, `onChatStart`,
`onValidateMessages`, `hydrateMessages`, `onTurnStart`,
`onBeforeTurnComplete`, `onTurnComplete`, `onChatSuspend`,
`onChatResume` — for persistence, validation, and post-turn work.
- **Stop generation** — client-driven `transport.stopGeneration(chatId)`
aborts mid-stream; the run stays alive for the next message, partial
response is captured, and aborted parts (stuck `partial-call` tools,
in-progress reasoning) are auto-cleaned.
- **Tool approvals (HITL)** — tools with `needsApproval: true` pause
until the user approves or denies via `addToolApprovalResponse`. The
runtime reconciles the updated assistant message by ID and continues
`streamText`.
- **Steering and background injection** — `pendingMessages` injects user
messages between tool-call steps so users can steer the agent
mid-execution; `chat.inject()` + `chat.defer()` adds context from
background work (self-review, RAG, safety checks) between turns.
- **Actions** — non-turn frontend commands (undo, rollback, regenerate,
edit) sent via `transport.sendAction`. Fire `hydrateMessages` +
`onAction` only — no turn hooks, no `run()`. `onAction` can return a
`StreamTextResult` for a model response, or `void` for side-effect-only.
- **Typed state primitives** — `chat.local<T>` for per-run state
accessible from hooks, `run()`, tools, and subtasks (auto-serialized
through `ai.toolExecute`); `chat.store` for typed shared data between
agent and client; `chat.history` for reading and mutating the message
chain; `clientDataSchema` for typed `clientData` in every hook.
- **`chat.toStreamTextOptions()`** — one spread into `streamText` wires
up versioned system [Prompts](https://trigger.dev/docs/ai/prompts),
model resolution, telemetry metadata, compaction, steering, and
background injection.
- **Multi-tab coordination** — `multiTab: true` + `useMultiTabChat`
prevents duplicate sends and syncs state across browser tabs via
`BroadcastChannel`. Non-active tabs go read-only with live updates.
- **Network resilience** — built-in indefinite retry with bounded
backoff, reconnect on `online` / tab refocus / bfcache restore,
`Last-Event-ID` mid-stream resume. No app code needed.
- **Sessions** — a durable, run-aware stream channel keyed on a stable
`externalId`. A Session is the unit of state that owns a multi-run
conversation: messages flow through `.in`, responses through `.out`,
both survive run boundaries. Sessions back the new `chat.agent` runtime,
and you can build on them directly for any pattern that needs durable
bi-directional streaming across runs.
([#3542](https://github.com/triggerdotdev/trigger.dev/pull/3542))
- Add `ai.toolExecute(task)` so you can wire a Trigger subtask in as the
`execute` handler of an AI SDK `tool()` while defining `description` and
`inputSchema` yourself — useful when you want full control over the tool
surface and just need Trigger's subtask machinery for the body.
([#3546](https://github.com/triggerdotdev/trigger.dev/pull/3546))
- Type `chat.createStartSessionAction` against your chat agent so
`clientData` is typed end-to-end on the first turn:
([#3684](https://github.com/triggerdotdev/trigger.dev/pull/3684))
- Add `region` to the runs list / retrieve API: filter runs by region
(`runs.list({ region: "..." })` / `filter[region]=<masterQueue>`) and
read each run's executing region from the new `region` field on the
response.
([#3612](https://github.com/triggerdotdev/trigger.dev/pull/3612))
- Add `TRIGGER_BUILD_SKIP_REWRITE_TIMESTAMP=1` escape hatch for local
self-hosted builds whose buildx driver doesn't support
`rewrite-timestamp` alongside push (e.g. orbstack's default `docker`
driver).
([#3618](https://github.com/triggerdotdev/trigger.dev/pull/3618))
- Reject overlong `idempotencyKey` values at the API boundary so they no
longer trip an internal size limit on the underlying unique index and
surface as a generic 500. Inputs are capped at 2048 characters — well
above what `idempotencyKeys.create()` produces (a 64-character hash) and
above any realistic raw key. Applies to `tasks.trigger`,
`tasks.batchTrigger`, `batch.create` (Phase 1 streaming batches),
`wait.createToken`, `wait.forDuration`, and the input/session stream
waitpoint endpoints. Over-limit requests now return a structured 400
instead.
([#3560](https://github.com/triggerdotdev/trigger.dev/pull/3560))
- **AI SDK `useChat` integration** — a custom
[`ChatTransport`](https://sdk.vercel.ai/docs/ai-sdk-ui/transport)
(`useTriggerChatTransport`) plugs straight into Vercel AI SDK's
`useChat` hook. Text streaming, tool calls, reasoning, and `data-*`
parts all work natively over Trigger.dev's realtime streams. No custom
API routes needed.
- **First-turn fast path (`chat.headStart`)** — opt-in handler that runs
the first turn's `streamText` step in your warm server process while the
agent run boots in parallel, cutting cold-start TTFC by roughly half
(measured 2801ms → 1218ms on `claude-sonnet-4-6`). The agent owns step
2+ (tool execution, persistence, hooks) so heavy deps stay where they
belong. Web Fetch handler works natively in Next.js, Hono, SvelteKit,
Remix, Workers, etc.; bridge to Express/Fastify/Koa via
`chat.toNodeListener`. New `@trigger.dev/sdk/chat-server` subpath.
- **Multi-turn durability via Sessions** — every chat is backed by a
durable Session that outlives any individual run. Conversations resume
across page refreshes, idle timeout, crashes, and deploys; `resume:
true` reconnects via `lastEventId` so clients only see new chunks.
`sessions.list` enumerates chats for inbox-style UIs.
- **Auto-accumulated history, delta-only wire** — the backend
accumulates the full conversation across turns; clients only ship the
new message each turn. Long chats never hit the 512 KiB body cap.
Register `hydrateMessages` to be the source of truth yourself.
- **Lifecycle hooks** — `onPreload`, `onChatStart`,
`onValidateMessages`, `hydrateMessages`, `onTurnStart`,
`onBeforeTurnComplete`, `onTurnComplete`, `onChatSuspend`,
`onChatResume` — for persistence, validation, and post-turn work.
- **Stop generation** — client-driven `transport.stopGeneration(chatId)`
aborts mid-stream; the run stays alive for the next message, partial
response is captured, and aborted parts (stuck `partial-call` tools,
in-progress reasoning) are auto-cleaned.
- **Tool approvals (HITL)** — tools with `needsApproval: true` pause
until the user approves or denies via `addToolApprovalResponse`. The
runtime reconciles the updated assistant message by ID and continues
`streamText`.
- **Steering and background injection** — `pendingMessages` injects user
messages between tool-call steps so users can steer the agent
mid-execution; `chat.inject()` + `chat.defer()` adds context from
background work (self-review, RAG, safety checks) between turns.
- **Actions** — non-turn frontend commands (undo, rollback, regenerate,
edit) sent via `transport.sendAction`. Fire `hydrateMessages` +
`onAction` only — no turn hooks, no `run()`. `onAction` can return a
`StreamTextResult` for a model response, or `void` for side-effect-only.
- **Typed state primitives** — `chat.local<T>` for per-run state
accessible from hooks, `run()`, tools, and subtasks (auto-serialized
through `ai.toolExecute`); `chat.store` for typed shared data between
agent and client; `chat.history` for reading and mutating the message
chain; `clientDataSchema` for typed `clientData` in every hook.
- **`chat.toStreamTextOptions()`** — one spread into `streamText` wires
up versioned system [Prompts](https://trigger.dev/docs/ai/prompts),
model resolution, telemetry metadata, compaction, steering, and
background injection.
- **Multi-tab coordination** — `multiTab: true` + `useMultiTabChat`
prevents duplicate sends and syncs state across browser tabs via
`BroadcastChannel`. Non-active tabs go read-only with live updates.
- **Network resilience** — built-in indefinite retry with bounded
backoff, reconnect on `online` / tab refocus / bfcache restore,
`Last-Event-ID` mid-stream resume. No app code needed.
- Retry `TASK_PROCESS_SIGSEGV` task crashes under the user's retry
policy instead of failing the run on the first segfault. SIGSEGV in Node
tasks is frequently non-deterministic (native addon races, JIT/GC
interaction, near-OOM in native code, host issues), so retrying on a
fresh process often succeeds. The retry is gated by the task's existing
`retry` config + `maxAttempts` — same path `TASK_PROCESS_SIGTERM` and
uncaught exceptions already use — so tasks without a retry policy still
fail fast.
([#3552](https://github.com/triggerdotdev/trigger.dev/pull/3552))
- The public interfaces for a plugin system. Initially consolidated
authentication and authorization interfaces.
([#3499](https://github.com/triggerdotdev/trigger.dev/pull/3499))
- Add MollifierBuffer and MollifierDrainer primitives for trigger burst
smoothing.
([#3614](https://github.com/triggerdotdev/trigger.dev/pull/3614))

## Bug fixes
- Fix `LocalsKey<T>` type incompatibility across dual-package builds.
The phantom value-type brand no longer uses a module-level `unique
symbol`, so a single TypeScript compilation that resolves the type from
both the ESM and CJS outputs (which can happen under certain pnpm
hoisting layouts) no longer sees two structurally-incompatible variants
of the same type.
([#3626](https://github.com/triggerdotdev/trigger.dev/pull/3626))

<details>
<summary>Raw changeset output</summary>

⚠️⚠️⚠️⚠️⚠️⚠️

`main` is currently in **pre mode** so this branch has prereleases
rather than normal releases. If you want to exit prereleases, run
`changeset pre exit` on `main`.

⚠️⚠️⚠️⚠️⚠️⚠️

# Releases
## @trigger.dev/sdk@4.5.0-rc.0

### Minor Changes

- **AI Prompts** — define prompt templates as code alongside your tasks,
version them on deploy, and override the text or model from the
dashboard without redeploying. Prompts integrate with the Vercel AI SDK
via `toAISDKTelemetry()` (links every generation span back to the
prompt) and with `chat.agent` via `chat.prompt.set()` +
`chat.toStreamTextOptions()`.
([#3629](https://github.com/triggerdotdev/trigger.dev/pull/3629))

    ```ts
    import { prompts } from "@trigger.dev/sdk";
    import { generateText } from "ai";
    import { openai } from "@ai-sdk/openai";
    import { z } from "zod";

    export const supportPrompt = prompts.define({
      id: "customer-support",
      model: "gpt-4o",
      config: { temperature: 0.7 },
      variables: z.object({
        customerName: z.string(),
        plan: z.string(),
        issue: z.string(),
      }),
      content: `You are a support agent for Acme.

    Customer: {{customerName}} ({{plan}} plan)
    Issue: {{issue}}`,
    });

    const resolved = await supportPrompt.resolve({
      customerName: "Alice",
      plan: "Pro",
      issue: "Can't access billing",
    });

    const result = await generateText({
      model: openai(resolved.model ?? "gpt-4o"),
      system: resolved.text,
      prompt: "Can't access billing",
      ...resolved.toAISDKTelemetry(),
    });
    ```

    **What you get:**

- **Code-defined, deploy-versioned templates** — define with
`prompts.define({ id, model, config, variables, content })`. Every
deploy creates a new version visible in the dashboard. Mustache-style
placeholders (`{{var}}`, `{{#cond}}...{{/cond}}`) with Zod / ArkType /
Valibot-typed variables.
- **Dashboard overrides** — change a prompt's text or model from the
dashboard without redeploying. Overrides take priority over the deployed
"current" version and are environment-scoped (dev / staging / production
independent).
- **Resolve API** — `prompt.resolve(vars, { version?, label? })` returns
the compiled `text`, resolved `model`, `version`, and labels. Standalone
`prompts.resolve<typeof handle>(slug, vars)` for cross-file resolution
with full type inference on slug and variable shape.
- **AI SDK integration** — spread `resolved.toAISDKTelemetry({ ...extra
})` into any `generateText` / `streamText` call and every generation
span links to the prompt in the dashboard alongside its input variables,
model, tokens, and cost.
- **`chat.agent` integration** — `chat.prompt.set(resolved)` stores the
resolved prompt run-scoped; `chat.toStreamTextOptions({ registry })`
pulls `system`, `model` (resolved via the AI SDK provider registry),
`temperature` / `maxTokens` / etc., and telemetry into a single spread
for `streamText`.
- **Management SDK** — `prompts.list()`, `prompts.versions(slug)`,
`prompts.promote(slug, version)`, `prompts.createOverride(slug, body)`,
`prompts.updateOverride(slug, body)`, `prompts.removeOverride(slug)`,
`prompts.reactivateOverride(slug, version)`.
- **Dashboard** — prompts list with per-prompt usage sparklines;
per-prompt detail with Template / Details / Versions / Generations /
Metrics tabs. AI generation spans get a custom inspector showing the
linked prompt's metadata, input variables, and template content
alongside model, tokens, cost, and the message thread.

See [/docs/ai/prompts](https://trigger.dev/docs/ai/prompts) for the full
reference — template syntax, version resolution order, override
workflow, and type utilities (`PromptHandle`, `PromptIdentifier`,
`PromptVariables`).

- Adds `onBoot` to `chat.agent` — a lifecycle hook that fires once per
worker process picking up the chat. Runs for the initial run, preloaded
runs, AND reactive continuation runs (post-cancel, crash, `endRun`,
`requestUpgrade`, OOM retry), before any other hook. Use it to
initialize `chat.local`, open per-process resources, or re-hydrate state
from your DB on continuation — anywhere the SAME run picking up after
suspend/resume isn't enough.
([#3543](https://github.com/triggerdotdev/trigger.dev/pull/3543))

    ```ts
const userContext = chat.local<{ name: string; plan: string }>({ id:
"userContext" });

    export const myChat = chat.agent({
      id: "my-chat",
      onBoot: async ({ clientData, continuation }) => {
const user = await db.user.findUnique({ where: { id: clientData.userId }
});
        userContext.init({ name: user.name, plan: user.plan });
      },
      run: async ({ messages, signal }) =>
streamText({ model: openai("gpt-4o"), messages, abortSignal: signal }),
    });
    ```

Use `onBoot` (not `onChatStart`) for state setup that must run every
time a worker picks up the chat — `onChatStart` fires once per chat and
won't run on continuation, leaving `chat.local` uninitialized when
`run()` tries to use it.

- **AI Agents** — run AI SDK chat completions as durable Trigger.dev
agents instead of fragile API routes. Define an agent in one function,
point `useChat` at it from React, and the conversation survives page
refreshes, network blips, and process restarts.
([#3543](https://github.com/triggerdotdev/trigger.dev/pull/3543))

    ```ts
    import { chat } from "@trigger.dev/sdk/ai";
    import { streamText } from "ai";
    import { openai } from "@ai-sdk/openai";

    export const myChat = chat.agent({
      id: "my-chat",
      run: async ({ messages, signal }) =>
streamText({ model: openai("gpt-4o"), messages, abortSignal: signal }),
    });
    ```

    ```tsx
    import { useChat } from "@ai-sdk/react";
import { useTriggerChatTransport } from "@trigger.dev/sdk/chat/react";

const transport = useTriggerChatTransport({ task: "my-chat",
accessToken, startSession });
    const { messages, sendMessage } = useChat({ transport });
    ```

    **What you get:**

- **AI SDK `useChat` integration** — a custom
[`ChatTransport`](https://sdk.vercel.ai/docs/ai-sdk-ui/transport)
(`useTriggerChatTransport`) plugs straight into Vercel AI SDK's
`useChat` hook. Text streaming, tool calls, reasoning, and `data-*`
parts all work natively over Trigger.dev's realtime streams. No custom
API routes needed.
- **First-turn fast path (`chat.headStart`)** — opt-in handler that runs
the first turn's `streamText` step in your warm server process while the
agent run boots in parallel, cutting cold-start TTFC by roughly half
(measured 2801ms → 1218ms on `claude-sonnet-4-6`). The agent owns step
2+ (tool execution, persistence, hooks) so heavy deps stay where they
belong. Web Fetch handler works natively in Next.js, Hono, SvelteKit,
Remix, Workers, etc.; bridge to Express/Fastify/Koa via
`chat.toNodeListener`. New `@trigger.dev/sdk/chat-server` subpath.
- **Multi-turn durability via Sessions** — every chat is backed by a
durable Session that outlives any individual run. Conversations resume
across page refreshes, idle timeout, crashes, and deploys; `resume:
true` reconnects via `lastEventId` so clients only see new chunks.
`sessions.list` enumerates chats for inbox-style UIs.
- **Auto-accumulated history, delta-only wire** — the backend
accumulates the full conversation across turns; clients only ship the
new message each turn. Long chats never hit the 512 KiB body cap.
Register `hydrateMessages` to be the source of truth yourself.
- **Lifecycle hooks** — `onPreload`, `onChatStart`,
`onValidateMessages`, `hydrateMessages`, `onTurnStart`,
`onBeforeTurnComplete`, `onTurnComplete`, `onChatSuspend`,
`onChatResume` — for persistence, validation, and post-turn work.
- **Stop generation** — client-driven `transport.stopGeneration(chatId)`
aborts mid-stream; the run stays alive for the next message, partial
response is captured, and aborted parts (stuck `partial-call` tools,
in-progress reasoning) are auto-cleaned.
- **Tool approvals (HITL)** — tools with `needsApproval: true` pause
until the user approves or denies via `addToolApprovalResponse`. The
runtime reconciles the updated assistant message by ID and continues
`streamText`.
- **Steering and background injection** — `pendingMessages` injects user
messages between tool-call steps so users can steer the agent
mid-execution; `chat.inject()` + `chat.defer()` adds context from
background work (self-review, RAG, safety checks) between turns.
- **Actions** — non-turn frontend commands (undo, rollback, regenerate,
edit) sent via `transport.sendAction`. Fire `hydrateMessages` +
`onAction` only — no turn hooks, no `run()`. `onAction` can return a
`StreamTextResult` for a model response, or `void` for side-effect-only.
- **Typed state primitives** — `chat.local<T>` for per-run state
accessible from hooks, `run()`, tools, and subtasks (auto-serialized
through `ai.toolExecute`); `chat.store` for typed shared data between
agent and client; `chat.history` for reading and mutating the message
chain; `clientDataSchema` for typed `clientData` in every hook.
- **`chat.toStreamTextOptions()`** — one spread into `streamText` wires
up versioned system [Prompts](https://trigger.dev/docs/ai/prompts),
model resolution, telemetry metadata, compaction, steering, and
background injection.
- **Multi-tab coordination** — `multiTab: true` + `useMultiTabChat`
prevents duplicate sends and syncs state across browser tabs via
`BroadcastChannel`. Non-active tabs go read-only with live updates.
- **Network resilience** — built-in indefinite retry with bounded
backoff, reconnect on `online` / tab refocus / bfcache restore,
`Last-Event-ID` mid-stream resume. No app code needed.

See [/docs/ai-chat](https://trigger.dev/docs/ai-chat/overview) for the
full surface — quick start, three backend approaches (`chat.agent`,
`chat.createSession`, raw task), persistence and code-sandbox patterns,
type-level guides, and API reference.

- Add read primitives to `chat.history` for HITL flows:
`getPendingToolCalls()`, `getResolvedToolCalls()`,
`extractNewToolResults(message)`, `getChain()`, and
`findMessage(messageId)`. These lift the accumulator-walking logic that
customers building human-in-the-loop tools were re-implementing into the
SDK. ([#3543](https://github.com/triggerdotdev/trigger.dev/pull/3543))

Use `getPendingToolCalls()` to gate fresh user turns while a tool call
is awaiting an answer. Use `extractNewToolResults(message)` to dedup
tool results when persisting to your own store — the helper returns only
the parts whose `toolCallId` is not already resolved on the chain.

    ```ts
    const pending = chat.history.getPendingToolCalls();
    if (pending.length > 0) {
      // an addToolOutput is expected before a new user message
    }

    onTurnComplete: async ({ responseMessage }) => {
const newResults = chat.history.extractNewToolResults(responseMessage);
      for (const r of newResults) {
await db.toolResults.upsert({ id: r.toolCallId, output: r.output,
errorText: r.errorText });
      }
    };
    ```

- **Sessions** — a durable, run-aware stream channel keyed on a stable
`externalId`. A Session is the unit of state that owns a multi-run
conversation: messages flow through `.in`, responses through `.out`,
both survive run boundaries. Sessions back the new `chat.agent` runtime,
and you can build on them directly for any pattern that needs durable
bi-directional streaming across runs.
([#3542](https://github.com/triggerdotdev/trigger.dev/pull/3542))

    ```ts
    import { sessions, tasks } from "@trigger.dev/sdk";

    // Trigger a task and subscribe to its session output in one call
const { runId, stream } = await tasks.triggerAndSubscribe("my-task",
payload, {
      externalId: "user-456",
    });

    for await (const chunk of stream) {
      // ...
    }

// Enumerate existing sessions (powers inbox-style UIs without a
separate index)
for await (const s of sessions.list({ type: "chat.agent", tag:
"user:user-456" })) {
      console.log(s.id, s.externalId, s.createdAt, s.closedAt);
    }
    ```

See [/docs/ai-chat/overview](https://trigger.dev/docs/ai-chat/overview)
for the full surface — Sessions powers the durable, resumable chat
runtime described there.

### Patch Changes

- Add Agent Skills for `chat.agent`. Drop a folder with a `SKILL.md` and
any helper scripts/references next to your task code, register it with
`skills.define({ id, path })`, and the CLI bundles it into the deploy
image automatically — no `trigger.config.ts` changes. The agent gets a
one-line summary in its system prompt and discovers full instructions on
demand via `loadSkill`, with `bash` and `readFile` tools scoped
per-skill (path-traversal guards, output caps, abort-signal
propagation).
([#3543](https://github.com/triggerdotdev/trigger.dev/pull/3543))

    ```ts
const pdfSkill = skills.define({ id: "pdf-extract", path:
"./skills/pdf-extract" });

    chat.skills.set([await pdfSkill.local()]);
    ```

Built on the [AI SDK cookbook
pattern](https://ai-sdk.dev/cookbook/guides/agent-skills) — portable
across providers. SDK + CLI only for now; dashboard-editable `SKILL.md`
text is on the roadmap.

- Add `ai.toolExecute(task)` so you can wire a Trigger subtask in as the
`execute` handler of an AI SDK `tool()` while defining `description` and
`inputSchema` yourself — useful when you want full control over the tool
surface and just need Trigger's subtask machinery for the body.
([#3546](https://github.com/triggerdotdev/trigger.dev/pull/3546))

    ```ts
    const myTool = tool({
      description: "...",
      inputSchema: z.object({ ... }),
      execute: ai.toolExecute(mySubtask),
    });
    ```

`ai.tool(task)` (`toolFromTask`) keeps doing the all-in-one wrap and now
aligns its return type with AI SDK's `ToolSet`. Minimum `ai` peer raised
to `^6.0.116` to avoid cross-version `ToolSet` mismatches in monorepos.

- Stamp `gen_ai.conversation.id` (the chat id) on every span and metric
emitted from inside a `chat.task` or `chat.agent` run. Lets you filter
dashboard spans, runs, and metrics by the chat conversation that
produced them — independent of the run boundary, so multi-run chats
correlate cleanly. No code changes required on the user side.
([#3543](https://github.com/triggerdotdev/trigger.dev/pull/3543))

- Type `chat.createStartSessionAction` against your chat agent so
`clientData` is typed end-to-end on the first turn:
([#3684](https://github.com/triggerdotdev/trigger.dev/pull/3684))

    ```ts
    import { chat } from "@trigger.dev/sdk/ai";
    import type { myChat } from "@/trigger/chat";

export const startChatSession = chat.createStartSessionAction<typeof
myChat>("my-chat");

// In the browser, threaded from the transport's typed startSession
callback:
    const transport = useTriggerChatTransport<typeof myChat>({
      task: "my-chat",
startSession: ({ chatId, clientData }) => startChatSession({ chatId,
clientData }),
      // ...
    });
    ```

`ChatStartSessionParams` gains a typed `clientData` field — folded into
the first run's `payload.metadata` so `onPreload` / `onChatStart` see
the same shape per-turn `metadata` carries via the transport. The opaque
session-level `metadata` field is unchanged.

- Unit-test `chat.agent` definitions offline with `mockChatAgent` from
`@trigger.dev/sdk/ai/test`. Drives a real agent's turn loop in-process —
no network, no task runtime — so you can send messages, actions, and
stop signals via driver methods, inspect captured output chunks, and
verify hooks fire. Pairs with `MockLanguageModelV3` from `ai/test` for
model mocking. `setupLocals` lets you pre-seed `locals` (DB clients,
service stubs) before `run()` starts.
([#3543](https://github.com/triggerdotdev/trigger.dev/pull/3543))

The broader `runInMockTaskContext` harness it's built on lives at
`@trigger.dev/core/v3/test` — useful for unit-testing any task code, not
just chat.

- Add `region` to the runs list / retrieve API: filter runs by region
(`runs.list({ region: "..." })` / `filter[region]=<masterQueue>`) and
read each run's executing region from the new `region` field on the
response.
([#3612](https://github.com/triggerdotdev/trigger.dev/pull/3612))

-   Updated dependencies:
    -   `@trigger.dev/core@4.5.0-rc.0`

## @trigger.dev/build@4.5.0-rc.0

### Patch Changes

- Add Agent Skills for `chat.agent`. Drop a folder with a `SKILL.md` and
any helper scripts/references next to your task code, register it with
`skills.define({ id, path })`, and the CLI bundles it into the deploy
image automatically — no `trigger.config.ts` changes. The agent gets a
one-line summary in its system prompt and discovers full instructions on
demand via `loadSkill`, with `bash` and `readFile` tools scoped
per-skill (path-traversal guards, output caps, abort-signal
propagation).
([#3543](https://github.com/triggerdotdev/trigger.dev/pull/3543))

    ```ts
const pdfSkill = skills.define({ id: "pdf-extract", path:
"./skills/pdf-extract" });

    chat.skills.set([await pdfSkill.local()]);
    ```

Built on the [AI SDK cookbook
pattern](https://ai-sdk.dev/cookbook/guides/agent-skills) — portable
across providers. SDK + CLI only for now; dashboard-editable `SKILL.md`
text is on the roadmap.

-   Updated dependencies:
    -   `@trigger.dev/core@4.5.0-rc.0`

## trigger.dev@4.5.0-rc.0

### Patch Changes

- Add Agent Skills for `chat.agent`. Drop a folder with a `SKILL.md` and
any helper scripts/references next to your task code, register it with
`skills.define({ id, path })`, and the CLI bundles it into the deploy
image automatically — no `trigger.config.ts` changes. The agent gets a
one-line summary in its system prompt and discovers full instructions on
demand via `loadSkill`, with `bash` and `readFile` tools scoped
per-skill (path-traversal guards, output caps, abort-signal
propagation).
([#3543](https://github.com/triggerdotdev/trigger.dev/pull/3543))

    ```ts
const pdfSkill = skills.define({ id: "pdf-extract", path:
"./skills/pdf-extract" });

    chat.skills.set([await pdfSkill.local()]);
    ```

Built on the [AI SDK cookbook
pattern](https://ai-sdk.dev/cookbook/guides/agent-skills) — portable
across providers. SDK + CLI only for now; dashboard-editable `SKILL.md`
text is on the roadmap.

- Add `TRIGGER_BUILD_SKIP_REWRITE_TIMESTAMP=1` escape hatch for local
self-hosted builds whose buildx driver doesn't support
`rewrite-timestamp` alongside push (e.g. orbstack's default `docker`
driver).
([#3618](https://github.com/triggerdotdev/trigger.dev/pull/3618))

- The CLI MCP server's agent-chat tools (`start_agent_chat`,
`send_agent_message`, `close_agent_chat`) now run on the new Sessions
primitive, so AI assistants driving a `chat.agent` get the same
idempotent-by-`chatId`, durable-across-runs behavior the browser
transport gets. Required PAT scopes go from `write:inputStreams` to
`read:sessions` + `write:sessions`.
([#3546](https://github.com/triggerdotdev/trigger.dev/pull/3546))

- MCP `list_runs` tool: add a `region` filter input and surface each
run's executing region in the formatted summary.
([#3612](https://github.com/triggerdotdev/trigger.dev/pull/3612))

-   Updated dependencies:
    -   `@trigger.dev/core@4.5.0-rc.0`
    -   `@trigger.dev/build@4.5.0-rc.0`
    -   `@trigger.dev/schema-to-json@4.5.0-rc.0`

## @trigger.dev/core@4.5.0-rc.0

### Patch Changes

- Add Agent Skills for `chat.agent`. Drop a folder with a `SKILL.md` and
any helper scripts/references next to your task code, register it with
`skills.define({ id, path })`, and the CLI bundles it into the deploy
image automatically — no `trigger.config.ts` changes. The agent gets a
one-line summary in its system prompt and discovers full instructions on
demand via `loadSkill`, with `bash` and `readFile` tools scoped
per-skill (path-traversal guards, output caps, abort-signal
propagation).
([#3543](https://github.com/triggerdotdev/trigger.dev/pull/3543))

    ```ts
const pdfSkill = skills.define({ id: "pdf-extract", path:
"./skills/pdf-extract" });

    chat.skills.set([await pdfSkill.local()]);
    ```

Built on the [AI SDK cookbook
pattern](https://ai-sdk.dev/cookbook/guides/agent-skills) — portable
across providers. SDK + CLI only for now; dashboard-editable `SKILL.md`
text is on the roadmap.

- Reject overlong `idempotencyKey` values at the API boundary so they no
longer trip an internal size limit on the underlying unique index and
surface as a generic 500. Inputs are capped at 2048 characters — well
above what `idempotencyKeys.create()` produces (a 64-character hash) and
above any realistic raw key. Applies to `tasks.trigger`,
`tasks.batchTrigger`, `batch.create` (Phase 1 streaming batches),
`wait.createToken`, `wait.forDuration`, and the input/session stream
waitpoint endpoints. Over-limit requests now return a structured 400
instead.
([#3560](https://github.com/triggerdotdev/trigger.dev/pull/3560))

- **AI Agents** — run AI SDK chat completions as durable Trigger.dev
agents instead of fragile API routes. Define an agent in one function,
point `useChat` at it from React, and the conversation survives page
refreshes, network blips, and process restarts.
([#3543](https://github.com/triggerdotdev/trigger.dev/pull/3543))

    ```ts
    import { chat } from "@trigger.dev/sdk/ai";
    import { streamText } from "ai";
    import { openai } from "@ai-sdk/openai";

    export const myChat = chat.agent({
      id: "my-chat",
      run: async ({ messages, signal }) =>
streamText({ model: openai("gpt-4o"), messages, abortSignal: signal }),
    });
    ```

    ```tsx
    import { useChat } from "@ai-sdk/react";
import { useTriggerChatTransport } from "@trigger.dev/sdk/chat/react";

const transport = useTriggerChatTransport({ task: "my-chat",
accessToken, startSession });
    const { messages, sendMessage } = useChat({ transport });
    ```

    **What you get:**

- **AI SDK `useChat` integration** — a custom
[`ChatTransport`](https://sdk.vercel.ai/docs/ai-sdk-ui/transport)
(`useTriggerChatTransport`) plugs straight into Vercel AI SDK's
`useChat` hook. Text streaming, tool calls, reasoning, and `data-*`
parts all work natively over Trigger.dev's realtime streams. No custom
API routes needed.
- **First-turn fast path (`chat.headStart`)** — opt-in handler that runs
the first turn's `streamText` step in your warm server process while the
agent run boots in parallel, cutting cold-start TTFC by roughly half
(measured 2801ms → 1218ms on `claude-sonnet-4-6`). The agent owns step
2+ (tool execution, persistence, hooks) so heavy deps stay where they
belong. Web Fetch handler works natively in Next.js, Hono, SvelteKit,
Remix, Workers, etc.; bridge to Express/Fastify/Koa via
`chat.toNodeListener`. New `@trigger.dev/sdk/chat-server` subpath.
- **Multi-turn durability via Sessions** — every chat is backed by a
durable Session that outlives any individual run. Conversations resume
across page refreshes, idle timeout, crashes, and deploys; `resume:
true` reconnects via `lastEventId` so clients only see new chunks.
`sessions.list` enumerates chats for inbox-style UIs.
- **Auto-accumulated history, delta-only wire** — the backend
accumulates the full conversation across turns; clients only ship the
new message each turn. Long chats never hit the 512 KiB body cap.
Register `hydrateMessages` to be the source of truth yourself.
- **Lifecycle hooks** — `onPreload`, `onChatStart`,
`onValidateMessages`, `hydrateMessages`, `onTurnStart`,
`onBeforeTurnComplete`, `onTurnComplete`, `onChatSuspend`,
`onChatResume` — for persistence, validation, and post-turn work.
- **Stop generation** — client-driven `transport.stopGeneration(chatId)`
aborts mid-stream; the run stays alive for the next message, partial
response is captured, and aborted parts (stuck `partial-call` tools,
in-progress reasoning) are auto-cleaned.
- **Tool approvals (HITL)** — tools with `needsApproval: true` pause
until the user approves or denies via `addToolApprovalResponse`. The
runtime reconciles the updated assistant message by ID and continues
`streamText`.
- **Steering and background injection** — `pendingMessages` injects user
messages between tool-call steps so users can steer the agent
mid-execution; `chat.inject()` + `chat.defer()` adds context from
background work (self-review, RAG, safety checks) between turns.
- **Actions** — non-turn frontend commands (undo, rollback, regenerate,
edit) sent via `transport.sendAction`. Fire `hydrateMessages` +
`onAction` only — no turn hooks, no `run()`. `onAction` can return a
`StreamTextResult` for a model response, or `void` for side-effect-only.
- **Typed state primitives** — `chat.local<T>` for per-run state
accessible from hooks, `run()`, tools, and subtasks (auto-serialized
through `ai.toolExecute`); `chat.store` for typed shared data between
agent and client; `chat.history` for reading and mutating the message
chain; `clientDataSchema` for typed `clientData` in every hook.
- **`chat.toStreamTextOptions()`** — one spread into `streamText` wires
up versioned system [Prompts](https://trigger.dev/docs/ai/prompts),
model resolution, telemetry metadata, compaction, steering, and
background injection.
- **Multi-tab coordination** — `multiTab: true` + `useMultiTabChat`
prevents duplicate sends and syncs state across browser tabs via
`BroadcastChannel`. Non-active tabs go read-only with live updates.
- **Network resilience** — built-in indefinite retry with bounded
backoff, reconnect on `online` / tab refocus / bfcache restore,
`Last-Event-ID` mid-stream resume. No app code needed.

See [/docs/ai-chat](https://trigger.dev/docs/ai-chat/overview) for the
full surface — quick start, three backend approaches (`chat.agent`,
`chat.createSession`, raw task), persistence and code-sandbox patterns,
type-level guides, and API reference.

- Stamp `gen_ai.conversation.id` (the chat id) on every span and metric
emitted from inside a `chat.task` or `chat.agent` run. Lets you filter
dashboard spans, runs, and metrics by the chat conversation that
produced them — independent of the run boundary, so multi-run chats
correlate cleanly. No code changes required on the user side.
([#3543](https://github.com/triggerdotdev/trigger.dev/pull/3543))

- Fix `LocalsKey<T>` type incompatibility across dual-package builds.
The phantom value-type brand no longer uses a module-level `unique
symbol`, so a single TypeScript compilation that resolves the type from
both the ESM and CJS outputs (which can happen under certain pnpm
hoisting layouts) no longer sees two structurally-incompatible variants
of the same type.
([#3626](https://github.com/triggerdotdev/trigger.dev/pull/3626))

- Unit-test `chat.agent` definitions offline with `mockChatAgent` from
`@trigger.dev/sdk/ai/test`. Drives a real agent's turn loop in-process —
no network, no task runtime — so you can send messages, actions, and
stop signals via driver methods, inspect captured output chunks, and
verify hooks fire. Pairs with `MockLanguageModelV3` from `ai/test` for
model mocking. `setupLocals` lets you pre-seed `locals` (DB clients,
service stubs) before `run()` starts.
([#3543](https://github.com/triggerdotdev/trigger.dev/pull/3543))

The broader `runInMockTaskContext` harness it's built on lives at
`@trigger.dev/core/v3/test` — useful for unit-testing any task code, not
just chat.

- Retry `TASK_PROCESS_SIGSEGV` task crashes under the user's retry
policy instead of failing the run on the first segfault. SIGSEGV in Node
tasks is frequently non-deterministic (native addon races, JIT/GC
interaction, near-OOM in native code, host issues), so retrying on a
fresh process often succeeds. The retry is gated by the task's existing
`retry` config + `maxAttempts` — same path `TASK_PROCESS_SIGTERM` and
uncaught exceptions already use — so tasks without a retry policy still
fail fast.
([#3552](https://github.com/triggerdotdev/trigger.dev/pull/3552))

- Add `region` to the runs list / retrieve API: filter runs by region
(`runs.list({ region: "..." })` / `filter[region]=<masterQueue>`) and
read each run's executing region from the new `region` field on the
response.
([#3612](https://github.com/triggerdotdev/trigger.dev/pull/3612))

- **Sessions** — a durable, run-aware stream channel keyed on a stable
`externalId`. A Session is the unit of state that owns a multi-run
conversation: messages flow through `.in`, responses through `.out`,
both survive run boundaries. Sessions back the new `chat.agent` runtime,
and you can build on them directly for any pattern that needs durable
bi-directional streaming across runs.
([#3542](https://github.com/triggerdotdev/trigger.dev/pull/3542))

    ```ts
    import { sessions, tasks } from "@trigger.dev/sdk";

    // Trigger a task and subscribe to its session output in one call
const { runId, stream } = await tasks.triggerAndSubscribe("my-task",
payload, {
      externalId: "user-456",
    });

    for await (const chunk of stream) {
      // ...
    }

// Enumerate existing sessions (powers inbox-style UIs without a
separate index)
for await (const s of sessions.list({ type: "chat.agent", tag:
"user:user-456" })) {
      console.log(s.id, s.externalId, s.createdAt, s.closedAt);
    }
    ```

See [/docs/ai-chat/overview](https://trigger.dev/docs/ai-chat/overview)
for the full surface — Sessions powers the durable, resumable chat
runtime described there.

## @trigger.dev/plugins@4.5.0-rc.0

### Patch Changes

- The public interfaces for a plugin system. Initially consolidated
authentication and authorization interfaces.
([#3499](https://github.com/triggerdotdev/trigger.dev/pull/3499))
-   Updated dependencies:
    -   `@trigger.dev/core@4.5.0-rc.0`

## @trigger.dev/python@4.5.0-rc.0

### Patch Changes

-   Updated dependencies:
    -   `@trigger.dev/sdk@4.5.0-rc.0`
    -   `@trigger.dev/core@4.5.0-rc.0`
    -   `@trigger.dev/build@4.5.0-rc.0`

## @trigger.dev/react-hooks@4.5.0-rc.0

### Patch Changes

-   Updated dependencies:
    -   `@trigger.dev/core@4.5.0-rc.0`

## @trigger.dev/redis-worker@4.5.0-rc.0

### Patch Changes

- Add MollifierBuffer and MollifierDrainer primitives for trigger burst
smoothing.
([#3614](https://github.com/triggerdotdev/trigger.dev/pull/3614))

MollifierBuffer (`accept`, `pop`, `ack`, `requeue`, `fail`,
`evaluateTrip`) is a per-env FIFO over Redis with atomic Lua transitions
for status tracking. `evaluateTrip` is a sliding-window trip evaluator
the webapp gate uses to detect per-env trigger bursts.

MollifierDrainer pops entries through a polling loop with a
user-supplied handler. The loop survives transient Redis errors via
capped exponential backoff (up to 5s), and per-env pop failures don't
poison the rest of the batch — one env's blip is logged and counted as
failed for that tick. Rotation is two-level: orgs at the top, envs
within each org. The buffer maintains `mollifier:orgs` and
`mollifier:org-envs:${orgId}` atomically with per-env queues, so the
drainer walks orgs → envs directly without an in-memory cache. The
`maxOrgsPerTick` option (default 500) caps how many orgs are scheduled
per tick; for each picked org, one env is popped (rotating round-robin
within the org). An org with N envs gets the same per-tick scheduling
slot as an org with 1 env, so tenant-level drainage throughput is
determined by org count rather than env count.

-   Updated dependencies:
    -   `@trigger.dev/core@4.5.0-rc.0`

## @trigger.dev/rsc@4.5.0-rc.0

### Patch Changes

-   Updated dependencies:
    -   `@trigger.dev/core@4.5.0-rc.0`

## @trigger.dev/schema-to-json@4.5.0-rc.0

### Patch Changes

-   Updated dependencies:
    -   `@trigger.dev/core@4.5.0-rc.0`

</details>

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
v.docker.4.5.0-rc.0 v4.5.0-rc.0 helm-v4.5.0-rc.0 		2026-05-21 14:39:02 +01:00
Eric Allam 422f9f0bb2 ci: unblock changesets release PRs (#3687) 
## Summary

Two CI workflows were blocking the v4.5.0-rc.0 release PR (#3563) and
would block every future changeset release PR.

### 1. `changesets-pr.yml` — self-report `All PR Checks`

The changesets bot pushes commits authored by `GITHUB_TOKEN`. By GitHub
design, `GITHUB_TOKEN`-authored pushes can't trigger downstream
workflows (loop-prevention). That means `pr_checks.yml` never fires on
release-PR commits, leaving the required `All PR Checks` status
permanently `Expected — Waiting for status to be reported`. The PR can't
merge.

The fix: after `changesets/action` creates the PR, post a `success`
check with the exact `All PR Checks` context onto the PR's head SHA.
GitHub's required-check evaluation is satisfied by any check with the
right context name — the source doesn't have to be `pr_checks.yml`.

**Why this is safe:** the release PR only mechanically bumps
`package.json`, `pnpm-lock.yaml`, and `CHANGELOG.md` from changesets
that were already on `main` (and already ran full CI when they merged).
If a human ever pushes a commit to `changeset-release/main`,
`pr_checks.yml` fires on that push (real user, not `GITHUB_TOKEN`) and
posts its own `All PR Checks` status — last write wins for the same
context on the same SHA, so the human-push result overrides the
auto-success.

### 2. `vouch-check-pr.yml` — exempt `github-actions[bot]`

The `require-draft` job auto-closes any non-draft PR whose author is not
a `MEMBER`/`OWNER`/`COLLABORATOR`, with an explicit allowlist for
`devin-ai-integration[bot]` and `dependabot[bot]`. The changesets bot
publishes as `github-actions[bot]` with `author_association:
CONTRIBUTOR`, so every release PR was getting auto-closed on open with a
"please re-open as draft" comment. Add `github-actions[bot]` to the
exemption list.

## Test plan
- [ ] After merge, the next changeset bot push to
`changeset-release/main` should post `All PR Checks: success` on the
release PR's head SHA, and the PR should not get auto-closed by `Vouch -
Check PR`.
- [ ] Confirm `pr_checks.yml` still fires + gates normal
(human-authored) PRs to `main`.
 2026-05-21 14:22:52 +01:00
Dan Sutton d7bdfb5640 chore: gitignore .playwright-mcp/ runtime cache 
Playwright MCP writes per-session console logs and page snapshots into
.playwright-mcp/ when used for testing. Local debug artefacts only —
no source value, shouldn't appear as untracked noise after running the
dashboard mollifier challenge scripts.
 2026-05-21 12:41:23 +01:00
Eric Allam 89d085a496 fix(references): repair ai-chat typecheck against current wire shape (#3685) 
## Summary

Pre-existing typecheck errors in `references/ai-chat` against the
current SDK shape. Unblocks `pnpm exec tsc --noEmit` in the reference
project.

## What changed

Three categories of fixes inside `references/ai-chat`. No SDK changes.

### 1. `payload.messages` → `payload.message`

The wire payload is now delta-only — one new message per trigger,
optional. Old code in two raw-task files reads `payload.messages`
(plural array) which no longer exists.

```ts
// before
const messages = await conversation.addIncoming(currentPayload.messages, ...);

// after
const messages = await conversation.addIncoming(
  currentPayload.message ? [currentPayload.message] : [],
  ...
);
```

Same fix to the `chat.messages.on` handler, reading `msg.message`
(singular) instead of `msg.messages[length - 1]`.

### 2. `clientData` non-null assertion in `cf-trust-test`

`ChatTurnContext.clientData` is typed as `?: TClientData` on
`onTurnStart` / `run` event objects even when the agent declares a
`clientDataSchema`. The runtime validates against the schema before the
hook fires, so it's structurally non-null — but TypeScript can't know
that. Non-null assert for now.

Follow-up worth filing: narrow `ChatTurnContext.clientData` to
non-optional when the agent has a `clientDataSchema`. Same friction the
docs friction-test subagent flagged.

### 3. `stress-emit.parseConfig` retyped against `ModelMessage[]`

The `run` callback hands `messages: ModelMessage[]`, not `UIMessage[]`.
Update `parseConfig` to accept `ModelMessage[]` and pull text from
`content` (string or array-of-parts).

## Test plan

- [x] `pnpm exec tsc --noEmit` in `references/ai-chat` passes (was 8
errors, now 0)
 2026-05-21 12:41:18 +01:00
Dan Sutton 1f90a00566 test(scripts): stress tests for pre-gate idempotency claim 
Four scenarios that the unit-test stubs and the cold-gate burst (04)
don't exercise. All green against a live webapp with the claim system
wired in.

16 — claimant-crash recovery. Planted "pending" claim externally, fired
5 same-key triggers (all polling), DEL'd the claim mid-poll. Verifies
the retry-SETNX path: 1 waiter wins, 4 polling losers resolve to the
same runId.

17 — stale-runId recovery. Claim resolves to a runId that exists in
neither PG nor the buffer. IdempotencyKeyConcern logs a warn and falls
through; the trigger creates a fresh run. Validates the
"resolved-but-not-findable" branch.

18 — claim safety-net timeout. Long-lived "pending" claim with no
publisher; same-key trigger polls until safetyNetMs elapses, returns
503. Validates the wait/poll budget caps.

19 — burst → drain → re-burst with the same key. First burst converges
via the claim (drainer ON, materialises post-burst); second burst
resolves via PG-findFirst (existing IdempotencyKeyConcern behaviour),
bypassing the claim entirely. Validates that the new claim path
doesn't break the existing PG-cache resolution that takes over once
the run is in PG.
 2026-05-21 12:39:07 +01:00
Dan Sutton 0b85126ce9 feat: pre-gate idempotency-key claim serialises same-key triggers 
Closes the PG+buffer race during the mollifier gate-transition window.
Plan: _plans/2026-05-21-mollifier-idempotency-claim.md

redis-worker:
- New MollifierBuffer methods + atomic Lua: claimIdempotency
  (SETNX-with-TTL returning claimed/pending/resolved), publishClaim,
  releaseClaim, readClaim. Separate key namespace mollifier:claim:*
  to keep isolated from the B6a buffered-side mollifier:idempotency:*
  lookup.

webapp:
- New apps/webapp/app/v3/mollifier/idempotencyClaim.server.ts wraps
  the buffer primitives with a wait/poll loop. Returns claimed /
  resolved / timed_out. Fail-open on buffer outage so a transient
  Redis blip doesn't 500 the trigger hot path.
- IdempotencyKeyConcern.handleTriggerRequest now consults the claim
  after the existing PG-findFirst + buffer.lookupIdempotency cache
  checks miss. Skipped for resumeParentOnCompletion (triggerAndWait
  bypasses the mollifier gate via F4 and is PG-canonical anyway). When
  we own the claim, the result's new `claim` field signals the caller
  to publish on success / release on failure.
- RunEngineTriggerTaskService.callV2 wraps the trigger pipeline in a
  try/catch that publishes the winning runId or releases the claim
  depending on outcome. The publish updates the claim key so waiters
  polling for our key resolve to our runId.

Validated end-to-end:
- scripts/mollifier-challenge/04-idempotency-collision.sh runs
  cold-gate (no pre-warm) with 30 concurrent same-key triggers and
  converges on 1 runId / 1 isCached:false. Before this fix the same
  test produced 2 race-winners.
- 13 unit tests covering claimed/resolved/pending/timed_out paths,
  fail-open behaviour, abort signal, publishClaim, releaseClaim.
- All 94 webapp mollifier tests still green.
 2026-05-21 12:24:07 +01:00
Dan Sutton d499aa5348 docs(_plans): pre-gate idempotency-key claim design 2026-05-21 12:06:26 +01:00
Eric Allam 9ff410bfa4 feat(sdk): type chat.createStartSessionAction against your chat agent (#3684) 
## Summary

Type `chat.createStartSessionAction` against the chat agent so
`clientData` is typed end-to-end on the first turn. Closes the gap where
`useTriggerChatTransport`'s `startSession` callback already hands you a
typed `clientData` (via the transport generic) but the server-side
action couldn't accept it without untyped routing through the `metadata`
field.

## Design

`ChatStartSessionParams` gains a typed `clientData` field via the new
generic:

```ts
export type ChatStartSessionParams<TChat extends AnyTask = AnyTask> = {
  chatId: string;
  clientData?: InferChatClientData<TChat>;
  triggerConfig?: Partial<SessionTriggerConfig>;
  metadata?: Record<string, unknown>;
};

function createChatStartSessionAction<TChat extends AnyTask = AnyTask>(
  taskId: string,
  options?: CreateChatStartSessionActionOptions
): (params: ChatStartSessionParams<TChat>) => Promise<ChatStartSessionResult>
```

When provided, `clientData` is folded into the first run's
`triggerConfig.basePayload.metadata`, so `onPreload` / `onChatStart` see
the same shape per-turn `metadata` carries via the transport. The opaque
session-level `metadata` field stays exactly as before — it lands on the
Session row, not the run payload.

## Usage

```ts
// actions.ts
import { chat } from "@trigger.dev/sdk/ai";
import type { myChat } from "@/trigger/chat";

export const startChatSession = chat.createStartSessionAction<typeof myChat>("my-chat");
```

```tsx
// Chat.tsx
const transport = useTriggerChatTransport<typeof myChat>({
  task: "my-chat",
  accessToken: ({ chatId }) => mintChatAccessToken(chatId),
  startSession: ({ chatId, clientData }) =>
    startChatSession({ chatId, clientData }),
});
```

## Test plan

- [x] `pnpm run build --filter @trigger.dev/sdk` passes
- [ ] Verify a `chat.agent` with `clientDataSchema` reads the typed
clientData from `onPreload` payload metadata on the first turn
 2026-05-21 11:51:35 +01:00
Dan Sutton 4e4925d992 test: regression coverage for the 3 fixes found by Phase F validation 
Each fix lands a focused test that fails without the fix and passes
with it.

1. Cancel route findResource (b490afe23) — extracted the PG-or-buffer
   lookup into apps/webapp/app/v3/mollifier/resolveRunForMutation.server.ts
   so it's unit-testable independently of the route builder.
   apps/webapp/test/mollifierResolveRunForMutation.test.ts covers all
   three paths: PG hit, buffer hit, both miss, plus env/org mismatch
   and PG-hit-short-circuits-before-buffer.

2. createCancelledRun empty-tags (eef33e5bd) — added a containerTest
   case to internal-packages/run-engine/src/engine/tests/createCancelledRun.test.ts
   passing `tags: {} as unknown as string[]` (mimics the cjson decode
   shape for an empty Lua table) and asserting the PG row is created
   with runTags=[]. Without the defensive Array.isArray check the
   Prisma create rejects with `Argument 'set' is missing`.

3. applyMetadataMutation retry budget (4e7d5d8a2) — new file
   apps/webapp/test/mollifierApplyMetadataMutation.test.ts with a
   stub MollifierBuffer that simulates Lua-CAS semantics in memory.
   Covers: zero contention, 5/11 simulated conflicts within budget,
   99 conflicts exhausting, and a 30-way concurrent-write convergence
   test. Includes a regression assertion that maxRetries=3 (pre-fix
   default) exhausts under 8 conflicts — confirming the regression
   actually existed.
 2026-05-21 11:27:28 +01:00