docs/mcp-readonly-runtime-flag
7310 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Iss
|
b2f002b0fe | Merge branch 'main' into docs/mcp-readonly-runtime-flag | ||
|
nicktrn
|
e47ba19a8f |
chore(deps): bump transitive dependency overrides (#3818)
Routine maintenance pass on a few transitive `pnpm.overrides`. - `fast-uri` / `fast-xml-builder`: add overrides pinning to current releases (`3.1.2` / `1.1.7`). - `protobufjs` / `qs`: bump existing override pins that had fallen a patch behind (`7.5.6` / `6.15.2`). Overrides-only - no first-party code changes; lockfile regenerated to match. Verified the affected transitives resolve to the pinned releases via `pnpm why -r`. |
||
|
nicktrn
|
55d85d0b23 |
chore(emails): upgrade react-email to latest (#3819)
## ✅ Checklist - [x] I have followed every step in the [contributing guide](https://github.com/triggerdotdev/trigger.dev/blob/main/CONTRIBUTING.md) - [x] The PR title follows the convention. - [x] I ran and tested the code works --- ## Testing Upgraded the email packages in `internal-packages/emails`: | Package | Before | After | | --- | --- | --- | | `@react-email/components` | `0.0.16` | `1.0.12` | | `@react-email/render` | `^0.0.12` | `^2.0.8` | | `react-email` (CLI) | `^2.1.1` | `^6.5.0` | | `react-dom` | _(missing)_ | `^18.2.0` (now a required peer of render) | **Breaking change handled:** `render()` is now async (`Promise<string>`) in `@react-email/render` v1+. Added `await` in the `aws-ses`, `smtp` and `null` transports. `EmailClient` and the webapp callers were already async and needed no changes. Verification: - `pnpm run typecheck --filter emails` ✅ - `pnpm run typecheck --filter webapp` (consumer of the `emails` package) ✅ - **Before/after render comparison**: rendered all 11 templates (magic-link, invite, welcome, alert-attempt/run/error-group, deployment-failure/success, mfa-enabled/disabled, bulk-action-complete) to HTML with both the old and new packages and compared them visually + via HTML diff. Output is visually identical. The only HTML changes come from upstream improvements: `<Body>` now wraps content in a `<table>`/`<td>` for better email-client compatibility, an `x-apple-disable-message-reformatting` meta tag was added, and CSS shorthand (e.g. `margin`) is now also emitted as longhand. No visual regressions; the `CodeBlock`/dracula theme, buttons, and row/column layouts all render correctly. No changeset or `.server-changes/` entry is added: `emails` is a private internal package (not under `packages/`), and there is no user-facing behavior change. --- ## Changelog Upgrade `react-email` and `@react-email/{components,render}` in `internal-packages/emails` to their latest versions and adapt the mail transports to the now-async `render()` API. --- ## Screenshots Rendered email templates before vs after the upgrade (visually identical): **Before** (`@react-email/components@0.0.16`, `render@0.0.12`) |
||
|
Eric Allam
|
8ab5b484bf |
chore: default local dev event store to ClickHouse in .env.example (#3817)
## Summary A fresh clone running `trigger dev` against the local webapp shows the run in the dashboard but an empty trace view. `EVENT_REPOSITORY_DEFAULT_STORE` was unset in `.env.example`, so span ingestion fell back to the `postgres` store while the rest of the local stack (run replication, `CLICKHOUSE_URL`) is ClickHouse-backed. Defaulting it to `clickhouse_v2` makes dev run traces show up out of the box. |
||
|
nicktrn
|
a9f756b260 |
chore: move reference projects to their own repo (#3812)
The reference/example projects (`references/`) now live in their own repo, https://github.com/triggerdotdev/references, so their heavy, frequently-changing dependencies are no longer part of this repo's lockfile and tooling. This removes them here and repoints everything that referenced them. - Deletes `references/`; updates the pnpm workspace + lockfile. - Clears the references-only CI rules and `.vscode` configs. - Repoints the docs (contributor/agent + one public page) to the new repo. - `seed.mts` keeps the local-dev projects (hello-world, d3-chat, realtime-streams). |
||
|
nicktrn
|
a4d8c9f65f |
chore(deps): update OpenTelemetry suite to 0.218.0 / 2.7.1 (#3810)
Brings the OpenTelemetry packages up to the latest coherent release
across the webapp and the published packages (`@trigger.dev/core`, the
CLI, `@trigger.dev/sdk`) plus
`internal-packages/{tracing,testcontainers}`:
- `@opentelemetry/sdk-node` 0.218.0
- `@opentelemetry/core` 2.7.1
- `@opentelemetry/host-metrics` 0.38.3
We were already on the otel 2.x line, so this is a same-major minor move
- the versions are pinned to `@opentelemetry/sdk-node@0.218.0`'s own
declared dependency set so the experimental (0.2xx) and stable (2.x)
packages stay coherent (mixing them is the usual otel breakage).
**One code change:** otel 0.215 made `forceFlush()` a required method on
`LogRecordExporter`, so `ExternalLogRecordExporterWrapper` (core's
tracing SDK) gains a `forceFlush()` that delegates to the underlying
exporter.
**Notable upgrades along the way:** OTLP exporters can take a custom
HTTP agent (connection pooling/keepAlive on the export path), HTTP
request headers are captured at span creation, and core hot-path perf
improvements in 2.6.1/2.7. `host-metrics` 0.37→0.38 is a clean upgrade.
Patch changeset added for the three published packages. References
projects are intentionally untouched.
Verified: `@trigger.dev/core` / CLI / `@trigger.dev/sdk` build, webapp +
`@internal/tracing` typecheck - all green.
|
||
|
nicktrn
|
d541caeb5e |
feat(supervisor): wide events + warm-start trace propagation (#3669)
Adds wide-event observability for the supervisor: one flat-keyed JSON line per dequeue iteration, workload-server route, and run socket lifecycle event. Events carry `trace_id` sourced from the inbound W3C traceparent plus `meta.run_id` and related identifiers, so they join across services by run. The outbound warm-start POST also forwards the inbound traceparent so the upstream receiver continues the same trace instead of minting a new one. Off by default behind `TRIGGER_WIDE_EVENTS_ENABLED`. With the flag off, no events are emitted, no ALS state is allocated, and the outbound warm-start request is unchanged — every call site was audited to confirm the off path is byte-identical to current behavior. Dequeue-path phase timings recorded under `phase.<name>.duration_ms`: `restore`, `warm_start`, `workload_create`. A `path_taken` extra distinguishes `restore` / `warm_start` / `cold_create` / `skipped_no_image`. Refs TRI-9480. |
||
|
Katia Bulatova
|
4f8cf4cc63 |
feat(webapp): runs live updating
## Summary The Runs list now updates live without requiring a page refresh. Status changes and other run fields are updated in place while runs are executing. When new runs matching the current filters are created, a "New runs created" refresh button appears above the list. Root runs now show a live child-run status breakdown directly in the status tooltip. ### List live update - Visible runs update in place while they are still running. - A "New runs created" refresh button appears when new matching runs are detected. - Polling stops when all visible runs have finished and a refresh button is already shown. - Polling pauses when the browser tab is not visible. - Runs list status updates and new-run detection share a single runs/live polling path. ### Child-status tooltip - Root run tooltips now display a breakdown of child run statuses. - Child statuses are loaded when the tooltip opens (after a 400ms hover delay). - The tooltip stays up to date while child runs are still changing state. - Handles cases where child runs continue running after their parent run has completed, or have not yet been created. ### Supporting changes - Added hidden-tab awareness to polling. - Added safeguards around polling inputs (`runIds` deduping and limits). ## Test plan - [x] pnpm run typecheck --filter webapp passes - [x] cd apps/webapp && pnpm run test ./test/presenters/mapRunToLiveFields.test.ts --run passes - [x] cd apps/webapp && pnpm run test ./test/runsRepository.part2.test.ts --run -t "hasNewRuns" passes ### Manual smoke: - [x] Active runs update without a page refresh. - [x] A new matching run shows the refresh banner and the banner actions work as expected. - [x] Root run tooltips show live child-status updates and stop polling once child runs settle. --------- Co-authored-by: Ekaterina Bulatova <kathiekiwi@Ekaterinas-MacBook-Pro.local> |
||
|
Matt Aitken
|
6961004a73 |
fix(webapp): restore Postgres fallback for non-ClickHouse OTLP spans (#3803)
## Problem On environments where runs carry a Postgres-backed `taskEventStore` value (`taskEvent` or `taskEventPartitioned`), OTLP ingest endpoints (`POST /otel/v1/traces` and `/otel/v1/logs`) were returning HTTP 500. **Root cause:** The org-scoped ClickHouse factory introduced in a recent PR routes all OTLP spans through `getEventRepositoryForOrganizationSync` → `buildEventRepository`. That function only handles `"clickhouse"` and `"clickhouse_v2"` store values and throws `Unknown ClickHouse event repository store: <value>` for anything else. The throw occurred inside the grouping loop of `#exportEvents`, unwinding the entire method and returning 500 for the whole batch. The OpenTelemetry collector's `otlphttp` exporter treats HTTP 500 as non-retryable and drops the batch — causing real span loss. **Fix:** Guard the `getEventRepositoryForOrganizationSync` call in `#exportEvents` so it is only invoked for `clickhouse` / `clickhouse_v2` store values. All other values are routed directly to the Postgres `eventRepository`, matching the guard pattern already present in `resolveEventRepositoryForStore` and `getEventRepositoryForStore` in `eventRepository/index.server.ts`. The ClickHouse factory call is also wrapped in a try/catch that falls back to Postgres so any unexpected store value in a future OTLP batch degrades gracefully instead of failing the whole request. ## Changes - `apps/webapp/app/v3/otlpExporter.server.ts` — add Postgres routing guard and try/catch fallback in `#exportEvents` ## Testing The `eventRepository/index.server.ts` module already has the same guard pattern thoroughly covered. The fix brings `#exportEvents` into alignment with that existing, tested pattern. Manual verification: confirm OTLP batches containing Postgres-store spans return 200 and route to the correct repository. |
||
|
Eric Allam
|
4b78d7e84e |
feat(core,webapp): support isSecret on environment variable imports (#3809)
## Summary
The environment variables import API now accepts an optional `isSecret`
flag, so imported variables can be created as secret (redacted)
environment variables instead of plaintext. When the flag is omitted,
variables default to non-secret, preserving existing behavior for CLI
deploys and dashboard imports.
This is useful for tools that push secrets into Trigger.dev (for
example, syncing from a secrets manager) and want them stored as secrets
rather than plain environment variables.
It's available through `envvars.import` in the SDK and the `POST
/api/v1/projects/{projectRef}/envvars/{slug}/import` endpoint, and is
honored for both regular and preview-branch environments.
```ts
await envvars.import("proj_1234", "prod", {
variables: { STRIPE_SECRET_KEY: "sk_live_..." },
isSecret: true,
});
```
|
||
|
Eric Allam
|
005d7e0a3f |
chore: add pkg.pr.new preview package releases (#3806)
## What Adds [pkg.pr.new](https://pkg.pr.new) continuous preview releases. Every push to a branch builds the public `@trigger.dev/*` packages and publishes installable preview builds keyed by commit SHA — **without touching the npm registry**. pkg.pr.new drops install instructions on the associated PR: ``` npm i https://pkg.pr.new/@trigger.dev/sdk@<sha> ``` This lets reviewers and users try a branch (SDK, CLI, core, etc.) before anything is released, separate from the changesets release, the manual `--snapshot` prerelease, and the chat-prerelease flow. ## How `.github/workflows/preview-packages.yml` (push trigger) → install → generate Prisma → **stamp preview version** → build → `pkg-pr-new publish`. ### The version stamp (the important part) pkg.pr.new serves previews by SHA but does **not** rewrite the package.json `version` field. If a preview shipped as `4.5.0-rc.4`, a consumer who installed it would pin `4.5.0-rc.4` to the preview tarball in their lockfile/cache — and a later `npm i @trigger.dev/sdk@4.5.0-rc.4` from npm could resolve to the stale preview. This is a known, by-design gap in the tool (stackblitz-labs/pkg.pr.new#250, #390). `scripts/stamp-preview-version.mjs` runs **before the build** and rewrites every public package to a unique `0.0.0-preview-<sha>`. The `0.0.0-` prefix can never satisfy a real semver range, so the collision is structurally impossible (same convention React/Next canaries use). Running before the build also means `scripts/updateVersion.ts` bakes the preview version into the runtime `VERSION` constant, so previews are self-identifying (`trigger --version`, the `x-trigger-cli-version` header, the MCP server version) instead of all reporting the RC version. Sibling `workspace:` specifiers are relaxed to `workspace:*` so `pnpm pack` resolves them against the rewritten versions — `packages/python` pins peerDependencies as `workspace:^4.5.0-rc.4`, which would otherwise be unsatisfiable once the version changes. Non-public deps (`@trigger.dev/database`, `@internal/*`) are left untouched. All mutations happen on the ephemeral CI checkout; nothing is committed. ## GitHub App The pkg.pr.new GitHub App is **already installed** on `triggerdotdev/trigger.dev` (has been for a while), so no setup is needed. Confirmed live — this branch's pushes published all 10 public packages, e.g. ``` pnpm add https://pkg.pr.new/@trigger.dev/sdk@e4dfc59 ``` ## Fork limitation pkg.pr.new authenticates with a GitHub Actions OIDC token, which GitHub does not issue to `pull_request` workflows from forks. The `push` trigger therefore covers branches pushed to this repo (core team), not external fork PRs. Fork coverage would need a `workflow_run` two-stage setup; left out for now. ## Notes - Pinned `pkg-pr-new@0.0.75` (no Node engine constraint; Node 20 CI is fine). - pkg.pr.new [#525](https://github.com/stackblitz-labs/pkg.pr.new/pull/525) adds a built-in `--previewVersion` flag (still open). If it lands we can drop the version-rewrite half of the script, but we'd keep a pre-build stamp anyway so `updateVersion.ts` picks up the preview version (the flag rewrites at pack time, too late for the baked `VERSION`). |
||
|
Daniel Sutton
|
139eede41c |
feat(redis-worker): batched pop in MollifierDrainer for fast single-env drains (#3797)
## Summary Adds `drainBatchSize` to `MollifierDrainer` (default `1` — preserves existing behaviour) and wires `TRIGGER_MOLLIFIER_DRAIN_BATCH_SIZE` through the webapp (default `50`). Each tick the drainer now pops up to `drainBatchSize` from each chosen env, then dispatches every popped entry through the shared `concurrency`-bounded `pLimit`. Per-org/per-env fairness is unchanged — only the in-env pop count grows. Pre-existing behaviour was one pop per env per tick. For a single-env burst that single-flighted the drain at the per-tick floor of `pop + engine.trigger ≈ 50–60 ms`. With buffer entries piling up under a real-world tenant burst that's tens of minutes of tail latency to fully materialise — even though PG itself could comfortably sustain the writes. ## Why this matters — heavy-tail illustration Scenario: 100 customers in one window — 94 fire 20 triggers each, 5 fire 100, 1 fires 1000. Gate at `THRESHOLD=10/s`, `HOLD_MS=500`. First 10 of each burst hit PG directly; the rest buffer. | Customers | Triggers each | PG direct | Buffered each | Total buffered | |---|---|---|---|---| | 94 small | 20 | 10 | 10 | 940 | | 5 medium | 100 | 10 | 90 | 450 | | 1 heavy | 1000 | 10 | 990 | 990 | **With `DRAIN_BATCH_SIZE=50`, `DRAIN_CONCURRENCY=50`, ~50 ms `engine.trigger`:** | Tick | Pops | Dispatch waves | Wall-clock | |---|---|---|---| | 1 | 94×10 + 5×50 + 1×50 = 1 240 | 25 × 50 ms | ~1 300 ms (94 smalls done) | | 2 | 5×40 + 1×50 = 250 | 5 × 50 ms | ~300 ms (5 mediums done) | | 3–20 | heavy alone, 50/tick | 1 × 50 ms | ~100 ms each | | Customer class | Buffered fully drained | |---|---| | 94 small | **~1.3 s** | | 5 medium | **~1.6 s** | | 1 heavy | **~3.4 s** | **Without batching (one pop per env per tick — current behaviour):** | Customer class | Buffered fully drained | |---|---| | 94 small | ~500 ms | | 5 medium | ~4.5 s | | 1 heavy | **~49 s** | So the heavy single-tenant tail drops from ~49 s to ~3.4 s (~14× faster) without changing PG load characteristics. Smalls go up slightly in this scenario (500 ms → 1.3 s) because all 100 envs share one tick's dispatch queue — that's the trade we accept for the heavy tail; the worst-case small wait is still inside one tick. PG load is identical either way (50 concurrent inserts at a time, capped by `DRAIN_CONCURRENCY`). ## What changed **`packages/redis-worker`** - New `drainBatchSize` option (default 1 — full backward compat). - `runOnce()` refactored to pop per-env batches in parallel, then dispatch all popped entries through the existing global `pLimit`. Mid-batch pop failure aborts only that env's batch and counts as one failure (same semantic as the old per-env path). - Removed the now-unused `processOneFromEnv` helper. **`apps/webapp`** - `TRIGGER_MOLLIFIER_DRAIN_BATCH_SIZE` env var (default 50, matching `DRAIN_CONCURRENCY`). - Wired into `mollifierDrainer.server.ts`. **Test cloud config** (separate cloud PR): `TRIGGER_MOLLIFIER_DRAIN_BATCH_SIZE="50"` on the worker service. Production rollout deferred until we've watched it on test cloud. ## Test plan - [x] All 25 stub-based drainer tests pass (18 pre-existing + 7 new). 7 new tests under `MollifierDrainer.drainBatchSize`: - pops up to `drainBatchSize` across ticks - global `concurrency` cap still holds when batch > concurrency - mid-batch pop failure isolation - multi-env batch fan-out in one tick - **hierarchical org fairness preserved at `drainBatchSize > 1`** (load-bearing — guards against future regressions to per-env-instead-of-per-org rotation) - mixed success/failure accounting in a batched tick - bounded pops on empty queue (no Lua spam past `drainBatchSize`) - [x] All pre-existing tests still pass unchanged at default `drainBatchSize=1` → backward-compat locked. - [x] `pnpm run build --filter @trigger.dev/redis-worker` clean. - [x] `pnpm run typecheck --filter webapp` clean. - [x] `redisTest` block (real Redis via testcontainers) — couldn't run locally on this branch due to testcontainers runtime discovery; will validate in CI. - [ ] Test-cloud smoke after cloud PR lands: fire `burst 50` against a flagged env and confirm the 50th entry's drain time drops from ~2.5 s to <200 ms. ## Notes - Per-tick memory bound: `maxOrgsPerTick × drainBatchSize` entries can sit in the JS pLimit queue between pop and dispatch. At defaults that's `500 × 50 = 25 000` × ~5 KB snapshot ≈ ~125 MB worst case per worker — well within headroom. - The pre-batch model's strict per-env throughput cap of `1/tick` is documented as the fairness baseline elsewhere. Org-level fairness is what callers actually rely on; this change does not weaken that. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
Eric Allam
|
0bbb4f13c6 | chore: add ConProgramming to vouch list (#3807) | ||
|
DKP
|
fa5fc6a9c7 | chore(webapp): remove Vercel install Loops event (#3805) | ||
|
nicktrn
|
11631c19e1 |
chore: bump node to latest patch release (#3802)
Bumps Node to the latest 20.x patch. |
||
|
Matt Aitken
|
b23ec26d16 |
chore: vouch ConProgramming (#3804)
Vouches GitHub user [`ConProgramming`](https://github.com/ConProgramming) (Conner) as an outside contributor by adding them to `.github/VOUCHED.td`, so their PRs aren't auto-closed by the vouch check. Done as a direct edit rather than via the issue flow because they have no open Vouch Request issue to comment `vouch` on. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
Dan
|
cd252801eb |
feat: dashboard agent - package upgrades (#3793)
1. in webapp folder update ai-sdk to 6.x.x 2. update vitest to 4.xx |
||
|
Eric Allam
|
4c4ed22e82 |
docs(ai-chat): document the chat.agent tools option (#3791)
## Summary Documents the new `tools` option on `chat.agent` (companion to #3790). Adds a dedicated [Tools](/ai-chat/tools) guide: the three places tools show up (config, `toStreamTextOptions`, `streamText`), why declaring them on the config matters for `toModelOutput` across turns, static vs per-turn tools, the typed `run()` payload, `InferChatUIMessageFromTools`, the relationship to skills, and the manual `convertToModelMessages` path for `customAgent` loops. Threads the option through the rest of the guide: the reference tables, a happy-path section on the backend page, the types page, and the HITL / skills / tool-result-auditing patterns. Corrects the sub-agents guide, where the `toModelOutput` compression was implied to work across turns but silently degraded from turn 2 without config tools. Also unstacks the three callouts that were piled under the `chat.agent()` header on the backend page, and adds a changelog entry. |
||
|
github-actions[bot]
|
e0681d2394 |
chore: release v4.5.0-rc.4 (#3788)
🚀 Publish Trigger.dev Docker / units (push) Failing after 3s
🚀 Publish Trigger.dev Docker / typecheck (push) Failing after 19s
🚀 Publish Trigger.dev Docker / publish-webapp (push) Has been skipped
🚀 Publish Trigger.dev Docker / publish-worker (push) Has been skipped
🚀 Publish Trigger.dev Docker / publish-worker-v4 (push) Has been skipped
🧭 Helm Chart Release / lint-and-test (push) Has been cancelled
🧭 Helm Chart Release / release (push) Has been cancelled
## Summary 1 new feature, 5 improvements. ## Highlights - Mollifier `mutateSnapshot` now enforces a tag cap: an `append_tags` patch carrying `maxTags` returns `"limit_exceeded"` (writing nothing) when the deduped tag count would exceed the limit, so a buffered run can't accumulate more tags via the tags API than the trigger validator allows at creation. ([#3756](https://github.com/triggerdotdev/trigger.dev/pull/3756)) ## Improvements - Mollifier buffer extensions: idempotency dedup, an atomic `mutateSnapshot` API, metadata CAS, claim primitives, and a `MollifierSnapshot` type. The buffer's Redis client now reconnects with jittered backoff so a fleet of clients doesn't stampede Redis in lockstep after a blip. ([#3752](https://github.com/triggerdotdev/trigger.dev/pull/3752)) - Add `onTerminalFailure` callback to `MollifierDrainerOptions` so the customer's run lands a SYSTEM_FAILURE PG row even when the drainer exhausts `maxAttempts` on a retryable PG error. Previously, retryable-error exhaustion called `buffer.fail()` directly, which atomically marks FAILED + DELs the entry hash with no PG write — silent data loss when PG was unreachable across the full retry budget. The callback fires before `buffer.fail()` on any terminal path (`cause: "non-retryable"` or `"max-attempts-exhausted"`); throwing a retryable error from the callback causes the drainer to requeue rather than fail. ([#3754](https://github.com/triggerdotdev/trigger.dev/pull/3754)) - Bump `@s2-dev/streamstore` to `0.22.10` to fix a `TASK_RUN_UNCAUGHT_EXCEPTION` ("Invalid state: Unable to enqueue") when a `chat.agent` turn is aborted mid-stream. ([#3792](https://github.com/triggerdotdev/trigger.dev/pull/3792)) - Coerce numeric `concurrencyKey` values to string at the API boundary across `tasks.trigger`, `tasks.batchTrigger`, and the Phase-2 streaming batch endpoint. ([#3789](https://github.com/triggerdotdev/trigger.dev/pull/3789)) - Add a `tools` option to `chat.agent`. Declaring your tools here threads them into the SDK's internal `convertToModelMessages`, so each tool's `toModelOutput` is re-applied when prior-turn history is re-converted. ([#3790](https://github.com/triggerdotdev/trigger.dev/pull/3790)) <details> <summary>Raw changeset output</summary> ⚠️⚠️⚠️⚠️⚠️⚠️ `main` is currently in **pre mode** so this branch has prereleases rather than normal releases. If you want to exit prereleases, run `changeset pre exit` on `main`. ⚠️⚠️⚠️⚠️⚠️⚠️ # Releases ## @trigger.dev/redis-worker@4.5.0-rc.4 ### Minor Changes - Mollifier buffer extensions: idempotency dedup, an atomic `mutateSnapshot` API, metadata CAS, claim primitives, and a `MollifierSnapshot` type. The buffer's Redis client now reconnects with jittered backoff so a fleet of clients doesn't stampede Redis in lockstep after a blip. ([#3752](https://github.com/triggerdotdev/trigger.dev/pull/3752)) - Add `onTerminalFailure` callback to `MollifierDrainerOptions` so the customer's run lands a SYSTEM_FAILURE PG row even when the drainer exhausts `maxAttempts` on a retryable PG error. Previously, retryable-error exhaustion called `buffer.fail()` directly, which atomically marks FAILED + DELs the entry hash with no PG write — silent data loss when PG was unreachable across the full retry budget. The callback fires before `buffer.fail()` on any terminal path (`cause: "non-retryable"` or `"max-attempts-exhausted"`); throwing a retryable error from the callback causes the drainer to requeue rather than fail. ([#3754](https://github.com/triggerdotdev/trigger.dev/pull/3754)) ### Patch Changes - Pipeline the per-entry `HGETALL` fetches in `MollifierBuffer.listEntriesForEnv`. The previous serial implementation issued one Redis round-trip per runId returned by `LRANGE`, which dominated stale-sweep wall-time at any meaningful backlog (at the sweep's default maxCount=1000, this is ~1000 RTTs per env per pass). Behaviour is unchanged — entries are still skipped when the entry hash has been torn down by a concurrent drainer ack/fail between the LRANGE and the HGETALL. ([#3752](https://github.com/triggerdotdev/trigger.dev/pull/3752)) - Mollifier `mutateSnapshot` now enforces a tag cap: an `append_tags` patch carrying `maxTags` returns `"limit_exceeded"` (writing nothing) when the deduped tag count would exceed the limit, so a buffered run can't accumulate more tags via the tags API than the trigger validator allows at creation. ([#3756](https://github.com/triggerdotdev/trigger.dev/pull/3756)) - Updated dependencies: - `@trigger.dev/core@4.5.0-rc.4` ## @trigger.dev/build@4.5.0-rc.4 ### Patch Changes - Updated dependencies: - `@trigger.dev/core@4.5.0-rc.4` ## trigger.dev@4.5.0-rc.4 ### Patch Changes - Bump `@s2-dev/streamstore` to `0.22.10` to fix a `TASK_RUN_UNCAUGHT_EXCEPTION` ("Invalid state: Unable to enqueue") when a `chat.agent` turn is aborted mid-stream. ([#3792](https://github.com/triggerdotdev/trigger.dev/pull/3792)) - Updated dependencies: - `@trigger.dev/core@4.5.0-rc.4` - `@trigger.dev/build@4.5.0-rc.4` - `@trigger.dev/schema-to-json@4.5.0-rc.4` ## @trigger.dev/core@4.5.0-rc.4 ### Patch Changes - Coerce numeric `concurrencyKey` values to string at the API boundary across `tasks.trigger`, `tasks.batchTrigger`, and the Phase-2 streaming batch endpoint. ([#3789](https://github.com/triggerdotdev/trigger.dev/pull/3789)) - Bump `@s2-dev/streamstore` to `0.22.10` to fix a `TASK_RUN_UNCAUGHT_EXCEPTION` ("Invalid state: Unable to enqueue") when a `chat.agent` turn is aborted mid-stream. ([#3792](https://github.com/triggerdotdev/trigger.dev/pull/3792)) ## @trigger.dev/plugins@4.5.0-rc.4 ### Patch Changes - Updated dependencies: - `@trigger.dev/core@4.5.0-rc.4` ## @trigger.dev/python@4.5.0-rc.4 ### Patch Changes - Updated dependencies: - `@trigger.dev/sdk@4.5.0-rc.4` - `@trigger.dev/core@4.5.0-rc.4` - `@trigger.dev/build@4.5.0-rc.4` ## @trigger.dev/react-hooks@4.5.0-rc.4 ### Patch Changes - Updated dependencies: - `@trigger.dev/core@4.5.0-rc.4` ## @trigger.dev/rsc@4.5.0-rc.4 ### Patch Changes - Updated dependencies: - `@trigger.dev/core@4.5.0-rc.4` ## @trigger.dev/schema-to-json@4.5.0-rc.4 ### Patch Changes - Updated dependencies: - `@trigger.dev/core@4.5.0-rc.4` ## @trigger.dev/sdk@4.5.0-rc.4 ### Patch Changes - Add a `tools` option to `chat.agent`. Declaring your tools here threads them into the SDK's internal `convertToModelMessages`, so each tool's `toModelOutput` is re-applied when prior-turn history is re-converted. ([#3790](https://github.com/triggerdotdev/trigger.dev/pull/3790)) ```ts chat.agent({ tools: { readFile, search }, run: async ({ messages, tools, signal }) => streamText({ model, messages, tools, abortSignal: signal }), }); ``` Also exports `InferChatUIMessageFromTools<typeof tools>` to derive the chat `UIMessage` type (typed tool parts) directly from a tool set. - Updated dependencies: - `@trigger.dev/core@4.5.0-rc.4` </details> Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>helm-v4.5.0-rc.4 v.docker.4.5.0-rc.4 v4.5.0-rc.4 |
||
|
nicktrn
|
e35f5742d4 |
fix(webapp): upgrade posthog-node to v5, drop axios + stale override (#3801)
Follow-up to #3796, which bumped the slack-client axios paths but left posthog-node's transitive `axios@1.15.1` in place. `posthog-node` 4.17.1 → 5.35.6. v5 drops the axios dependency entirely (it's now fetch-based via `@posthog/core`), so posthog's old axios path disappears. With #3796 already on main (webapp + d3 references on `@slack/web-api@7.16.0`), nothing else pins the old line, so the now-dead `axios@>=1.0.0 <1.15.0` override is removed and axios resolves to a single patched `1.16.1` repo-wide. This closes the remaining axios advisories. Compat: the webapp's usage in `telemetry.server.ts` (`new PostHog(key, { host })`, `.identify`, `.groupIdentify`, `.capture`) is all object-form API that v5 preserves; `pnpm run typecheck --filter webapp` passes. Node: posthog-node v5 requires Node `^20.20.0 || >=22.22.0`. We run 20.20.0 in dev (`.nvmrc`), CI, and the published Docker image (`node:20.20-bullseye-slim`), so we're compliant. |
||
|
nicktrn
|
952139d492 |
fix(webapp): bump @slack/web-api to 7.16.0 for patched axios (#3796)
Bumps `@slack/web-api` 7.9.1 → 7.16.0 in the webapp and the two `references` examples (d3-chat, d3-openai-agents). 7.16.0 depends on `axios@^1.16.0`, so every slack-client axios path resolves to 1.16.1 instead of 1.15.1. This clears the slack and references axios paths. `posthog-node`'s transitive axios still resolves the older line - that's handled in a follow-up that upgrades posthog-node to v5 (which drops the axios dependency entirely and lets us retire the now-stale axios override). The dependabot axios advisories fully close once both land. |
||
|
Eric Allam
|
2dd9f37b40 |
fix(core,cli): stop chat.agent uncaught exception on mid-stream abort (#3792)
## Summary A `chat.agent` turn that gets aborted mid-stream (stop generation, idle suspend, cancellation) could surface a `TASK_RUN_UNCAUGHT_EXCEPTION` — `TypeError: Invalid state: Unable to enqueue` — in the dashboard. The run kept working, but the exceptions were loud and confusing. ## Root cause The realtime stream writer batches chunks through `@s2-dev/streamstore`'s `BatchTransform`, which holds them for a short linger window before flushing. When the turn's abort signal fires while a record is still buffered, the stream's writable is aborted and the transform's readable controller is closed — but the pending linger `setTimeout` still fires and calls `controller.enqueue()` on the dead controller, throwing from a timer callback where nothing can catch it. Fixed upstream in `@s2-dev/streamstore@0.22.10`, which wraps the linger flush in a try/catch that discards the closed-controller error. This bumps the dependency across core, the CLI, and the webapp, and adds a regression test against the real `BatchTransform`. Verified end-to-end: a mid-stream stop that previously failed the run with `TASK_RUN_UNCAUGHT_EXCEPTION` now leaves the run healthy. |
||
|
Eric Allam
|
e9e2ec1cfc |
fix(sdk): re-apply tool toModelOutput across chat.agent turns (#3790)
## Summary
`chat.agent` now takes a `tools` option. Until now tools only went to
`streamText` inside `run()`, so the SDK had no tools when it
re-converted the persisted `UIMessage` history at the start of each
turn. Any tool with a `toModelOutput` (raw image bytes into an image
content part, or a sub-agent transcript compressed to a summary) had its
transform applied on turn 1 and skipped from turn 2 onward, so the raw
output got JSON-stringified back into the prompt and the model lost the
transformed view.
Declaring `tools` on the config threads them into that conversion, so
`toModelOutput` runs on every turn. The resolved set is handed back,
typed, on the `run()` payload as `tools`:
```ts
const tools = { searchDocs, renderChart };
export const myChat = chat.agent({
tools,
run: async ({ messages, tools, signal }) =>
streamText({ ...chat.toStreamTextOptions({ tools }), messages, abortSignal: signal }),
});
```
`tools` also accepts a per-turn function for tools that depend on the
user or a feature flag. Only `inputSchema` and `toModelOutput` are read
during conversion, never `execute`. Also exports
`InferChatUIMessageFromTools<typeof tools>` to derive the chat
`UIMessage` type from a tool set. No behavior change for agents that
don't declare `tools`.
|
||
|
Daniel Sutton
|
e21b68cc5f |
feat(webapp): dashboard parity for mollifier-buffered runs (#3757)
## Summary Dashboard surfaces handle buffered runs by falling back to the mollifier snapshot: - Run detail, span detail, streams view (`_app.../runs.\$runParam`, `resources.../spans.\$spanParam`, `resources.../streams.\$streamKey`). - Redirect routes (`@.runs.\$runParam`, `runs.\$runParam`, `projects.v3.\$projectRef.runs.\$runParam`). - Action routes — cancel / replay / idempotency-reset / debug — under `resources.taskruns/...` and `resources.../idempotencyKey.reset`. - Logs download. - Realtime subscription route + per-run resource (`realtime.v1.runs.\$runId`, `resources.../realtime.v1.*`). - `CancelRunDialog` gains an `onCancelSubmitted` callback so submit isn't raced by the Radix `DialogClose` wrapper. Stacked on the mutations PR. ## Test plan - [x] \`pnpm run typecheck --filter webapp\` passes - [x] \`pnpm run test --filter webapp test/mollifierRealtimeRunResource.test.ts\` passes - [x] \`pnpm run test --filter webapp test/mollifierRealtimeRunResourceBuffer.test.ts\` passes - [x] \`pnpm run test --filter webapp test/mollifierRealtimeSubscription.test.ts\` passes - [x] Manual smoke: trigger a buffered run, open it in the dashboard, replay/cancel from the UI --- ## Ship-gate follow-up fixes - **Auto-redirect to root span on direct nav** — loader sets `?span=` from root span (PG) or buffered snapshot spanId before 302'ing, so bookmark/share-link/direct-nav doesn't leave the panel collapsed. - **RunPresenter switches from `findFirstOrThrow` to `findFirst` + typed `RunNotInPgError`** — kills the per-poll `PrismaClient error` log spam for buffered runs without changing the route-loader's fallback flow. - **Span detail panel renders for buffered runs** — `SpanPresenter.call` now falls back to `findRunByIdWithMollifierFallback` + `buildSyntheticSpanRun` instead of returning undefined and triggering the "Event not found" toast loop. - **Logs download for buffered runs returns a gzipped placeholder line** — replaces the 404 with a content-encoded line explaining the run is queued. Same org-membership gate as the PG path. - **Admin Debug-Run button hidden for buffered runs + SpanRun circular type alias broken** (squashed) — buttons gate on a new `isBuffered` flag on the synthetic SpanRun. Required grounding SpanRun in `SpanPresenter.getRun` to break a circular type alias TS no longer tolerates once `isBuffered` is a literal field on the shape. - **Replay action requires user auth + org-membership** (🚩 Devin finding) — `action` was unauthenticated and the PG `findFirst` had no org filter, so any caller with a valid `runParam` could replay any run. Buffered fallback inherited the same gap. Fixed to mirror the cancel route. --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
Daniel Sutton
|
e1950778e2 |
feat(webapp): mollifier API mutations on buffered runs (#3756)
## Summary Cancel, replay, reschedule, metadata, tags, and idempotency-key-reset now succeed against a run that's still in the mollifier buffer. Mutations are applied to the buffered snapshot via Lua CAS; the drainer carries the mutation forward when it replays. Primitives added: - `mutateWithFallback` — PG-first / buffer-fallback resolver with bounded-wait safety net for entries that transition mid-mutation. - `applyMetadataMutation` — buffered metadata PUT mirroring the PG-side retry loop with CAS atomicity. - `resolveRunForMutation` — discriminated-union resolver used by route `findResource` so the route builder's pre-action 404 check sees buffered runs. Routes wired (whole files, no GET/POST splits): - `api.v2.runs.\$runParam.cancel.ts` - `api.v1.runs.\$runParam.replay.ts` - `api.v1.runs.\$runParam.reschedule.ts` - `api.v1.runs.\$runId.metadata.ts` - `api.v1.runs.\$runId.tags.ts` - `resetIdempotencyKey.server.ts` Stacked on the reads PR. ## Test plan - [x] \`pnpm run typecheck --filter webapp\` passes - [x] \`pnpm run test --filter webapp test/mollifierMutateWithFallback.test.ts\` passes - [x] \`pnpm run test --filter webapp test/mollifierApplyMetadataMutation.test.ts\` passes - [x] \`pnpm run test --filter webapp test/mollifierResolveRunForMutation.test.ts\` passes --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
Daniel Sutton
|
a1dc3c5ae1 |
feat(webapp): mollifier API GET read-fallback — synthetic primitives + route wiring (#3755)
## Summary Synthesise QUEUED/FAILED responses from the mollifier buffer when a TaskRun row hasn't landed in Postgres yet. Wires the synthesis into: - `ApiRetrieveRunPresenter` - v1 trace GET route - v1 spans GET route - attempts route gains a GET loader (fixes pre-existing Remix "no loader" 400) The `readFallback` infra itself lives on the trigger PR (consumed by `IdempotencyKeyConcern`); this PR adds the route-level synthetic-rendering primitives. Stacked on the replay PR. ## Test plan - [x] \`pnpm run typecheck --filter webapp\` passes - [x] \`pnpm run test --filter webapp test/mollifierSyntheticRedirectInfo.test.ts\` passes - [x] \`pnpm run test --filter webapp test/mollifierSyntheticSpanRun.test.ts\` passes --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
Daniel Sutton
|
4745754a7a |
feat(webapp,run-engine): mollifier drainer replay + stale sweep + cancelled-run engine API (#3754)
## Summary The replay side of the mollifier: - `DrainerHandler`: reads buffered snapshots and replays them through `engine.trigger` to materialise PG rows. - `RunEngine.createCancelledRun`: new public method the handler uses to write CANCELED rows directly from snapshots (bypass queue + waitpoint, emit `runCancelled`). Tolerates the cjson empty-table tags edge case found during validation. - Drainer fairness: org → env rotation so a heavy env doesn't starve light ones in the same org. - Stale-entry sweep + telemetry + alertable gauge so a stuck/offline drainer surfaces in alerts. Both the drainer and sweep default-off; nothing fires unless flagged on (`TRIGGER_MOLLIFIER_DRAINER_ENABLED`, `TRIGGER_MOLLIFIER_STALE_SWEEP_ENABLED`). Stacked on the trigger-time decisions PR. ## Test plan - [x] \`pnpm run typecheck --filter webapp\` passes - [x] \`pnpm run test --filter webapp test/mollifierDrainerHandler.test.ts\` passes - [x] \`pnpm run test --filter webapp test/mollifierStaleSweep.test.ts\` passes - [x] \`pnpm run test --filter @internal/run-engine src/engine/tests/createCancelledRun.test.ts\` passes - [x] \`pnpm run test --filter @trigger.dev/redis-worker packages/redis-worker/src/mollifier/drainer.test.ts\` passes --- ## Ship-gate follow-up fix **Drainer writes SYSTEM_FAILURE on max-attempts exhaustion.** Adds an `onTerminalFailure` callback on `MollifierDrainerOptions` so the customer's run lands a SYSTEM_FAILURE PG row even when the drainer exhausts `MAX_ATTEMPTS` on a retryable PG error (previously `buffer.fail()` was called with no row written → silent data loss). The callback runs before `buffer.fail()` on every terminal path (non-retryable AND max-attempts-exhausted), and re-throwing a retryable error from the callback causes the drainer to requeue rather than fail. Bumps `@trigger.dev/redis-worker` to a **minor** changeset (additive option + new exported types). Includes 5 unit tests covering both terminal causes plus the requeue-on-retryable-callback-failure path and no-callback back-compat. --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
Daniel Sutton
|
20a676d2ed |
fix(core,webapp): coerce numeric concurrencyKey to string (#3789)
## Summary `concurrencyKey` validation accepted only `z.string().optional()` on the single-trigger and V2/V3 batch endpoints, and the Phase-2 streaming NDJSON endpoint accepted `z.record(z.unknown()).optional()` for the entire `options` field. Callers passing `concurrencyKey: someNumericId` (e.g. `payload.userId`) either failed schema validation on the first two paths or sailed through on Phase-2 and then failed downstream at `prisma.taskRun.create` with `Argument concurrencyKey: Expected String or Null, provided Int`. The schema now accepts `string | number` for `concurrencyKey` and stringifies on the way in, across all three paths. The Phase-2 NDJSON `options` is tightened to reuse the strict `BatchTriggerTaskItem.options` shape so it validates identically to the V2/V3 batch endpoints. A defensive `typeof === "number"` coercion at the `engine.trigger` call site in `RunEngineTriggerTaskService` covers in-flight Redis-stored batch items enqueued before the schema fix — those items are rebuilt from a `Record<string, unknown>` shape that bypasses the new schema and would otherwise continue failing for up to their TTL. ## Test plan - [x] `packages/core/src/v3/schemas/batchItemNDJSON.test.ts` — unit tests covering numeric→string coercion, string passthrough, no-options, and rejection of non-string/non-number shapes across `TriggerTaskRequestBody`, `BatchTriggerTaskItem`, and `BatchItemNDJSON`. - [x] `apps/webapp/test/engine/triggerTask.test.ts` — `containerTest` simulating the in-flight Redis batch-item shape (numeric `concurrencyKey` via `Record<string, unknown>`), verifies the run is created with `concurrencyKey: "51262"`. Without the worker coercion, the test reproduces the production stack at `prisma.taskRun.create`. - [x] `pnpm run typecheck --filter webapp` clean. - [x] `pnpm run build --filter @trigger.dev/core --filter @trigger.dev/sdk --filter trigger.dev` clean. Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
Daniel Sutton
|
577f35eebe |
feat(webapp): mollifier trigger-time decisions — mollify, claim, read fallback (#3753)
## Summary The trigger hot path's mollifier integration: - `mollifyTrigger`: when the gate trips, write the engine.trigger snapshot to the buffer and return a synthesised QUEUED response. Postgres write is deferred to drainer-replay (next PR in the stack). - Pre-gate idempotency-key claim: same-key triggers serialise through Redis so a burst lands in PG / buffer exactly once. - Read-fallback extensions: `findRunByIdWithMollifierFallback` for the trigger-time idempotency lookup that must see buffered runs. - Gate bypasses: `debounce`, `oneTimeUseToken`, `parentTaskRunId`/`triggerAndWait` skip the mollify path entirely. - `triggerTask` + `IdempotencyKeyConcern` wired to the above. All behaviour gated by the master `TRIGGER_MOLLIFIER_ENABLED` switch; off-state hot path is unchanged (the gate is not even consulted). Stacked on the buffer extensions PR. ## Test plan - [x] \`pnpm run typecheck --filter webapp\` passes - [x] \`pnpm run test --filter webapp test/mollifierMollify.test.ts\` passes - [x] \`pnpm run test --filter webapp test/mollifierIdempotencyClaim.test.ts\` passes - [x] \`pnpm run test --filter webapp test/mollifierReadFallback.test.ts\` passes - [x] \`pnpm run test --filter webapp test/mollifierGate.test.ts\` passes - [x] \`pnpm run test --filter webapp test/engine/triggerTask.test.ts\` passes --- ## Ship-gate follow-up fixes - **Batch items bypass the mollifier gate** — fixes `BatchTaskRunItem_taskRunId_fkey` FK violation on batch triggers when the gate trips. End-state is a drainer-side `BatchTaskRunItem` create-on-materialise; batch traffic passes through the gate until that lands. - **IdempotencyKeyConcern honours buffered-run TTL on expiry** — buffered path now clears expired idempotency claims (read-side) and resets the buffer's `mollifier:idempotency:*` SETNX binding (write-side) so a re-trigger past the customer's TTL lands as a fresh run instead of echoing the stale buffered runId. --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
Daniel Sutton
|
3924aa4adb |
feat(redis-worker,webapp): mollifier buffer extensions + snapshot type (#3752)
## Summary Buffer-side data layer used by the rest of the mollifier phase-3 stack. - `buffer.ts` gains entry inspection (`getEntry`), idempotency lookup (`lookupIdempotency`), in-place snapshot mutation (`mutateSnapshot`), and dwell tracking. All atomic via Lua. - `mollifierSnapshot.server.ts`: shared `MollifierSnapshot` type plus (de)serialise helpers. - Drops the entry-TTL config and its env var. The drainer is the recovery mechanism; an entry that survives the drainer should surface as a stale-sweep alert, not silently TTL away. Adds methods to the buffer interface; nothing consumes them yet. Subsequent PRs in the stack wire trigger-time mollify, read-fallback, and mutation paths against this surface. ## Test plan - [x] \`pnpm run typecheck --filter webapp\` passes - [x] \`pnpm run test --filter @trigger.dev/redis-worker packages/redis-worker/src/mollifier/buffer.test.ts\` passes --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
Saadi Myftija
|
181d9ba541 |
feat: preview environment dispatch workflow (#3786)
Publishes a `repository_dispatch` `preview-deploy` event that enables preview deployments. Opt-in only currently. |
||
|
github-actions[bot]
|
8f066ac389 |
chore: release v4.5.0-rc.3 (#3763)
🚀 Publish Trigger.dev Docker / typecheck (push) Failing after 0s
🚀 Publish Trigger.dev Docker / publish-webapp (push) Has been skipped
🚀 Publish Trigger.dev Docker / publish-worker (push) Has been skipped
🚀 Publish Trigger.dev Docker / publish-worker-v4 (push) Has been skipped
🚀 Publish Trigger.dev Docker / units (push) Failing after 0s
🧭 Helm Chart Release / lint-and-test (push) Has been cancelled
🧭 Helm Chart Release / release (push) Has been cancelled
## Summary 1 improvement, 2 bug fixes. ## Improvements - Retry `TASK_MIDDLEWARE_ERROR` under the task's retry policy instead of failing the run on the first attempt. The error was already classified as retryable by `shouldRetryError`, but `shouldLookupRetrySettings` did not include it, so the retry flow fell through to `fail_run`. Fixes #3231. ([#3676](https://github.com/triggerdotdev/trigger.dev/pull/3676)) ## Bug fixes - Fix `TypeError` in `unflattenAttributes` when the input attribute map contains conflicting dotted key paths (e.g. both `a.b` set to a scalar and `a.b.c` set to a value). The path-walk loop now applies last-write-wins when a prior key wrote a primitive, null, or array at an intermediate slot, matching the existing precedent in `AttributeFlattener.addAttribute`. Callers no longer crash when handed malformed external attribute inputs. ([#3762](https://github.com/triggerdotdev/trigger.dev/pull/3762)) - Fix external trace context leaking across runs on warm-started workers with `processKeepAlive` enabled. Every subsequent run's attempt span was being exported with the first run's `traceId` and `parentSpanId`, breaking causal-chain navigation in external APM tools. Runs without an external trace context are unaffected. ([#3768](https://github.com/triggerdotdev/trigger.dev/pull/3768)) <details> <summary>Raw changeset output</summary> ⚠️⚠️⚠️⚠️⚠️⚠️ `main` is currently in **pre mode** so this branch has prereleases rather than normal releases. If you want to exit prereleases, run `changeset pre exit` on `main`. ⚠️⚠️⚠️⚠️⚠️⚠️ # Releases ## @trigger.dev/build@4.5.0-rc.3 ### Patch Changes - Updated dependencies: - `@trigger.dev/core@4.5.0-rc.3` ## trigger.dev@4.5.0-rc.3 ### Patch Changes - Updated dependencies: - `@trigger.dev/core@4.5.0-rc.3` - `@trigger.dev/build@4.5.0-rc.3` - `@trigger.dev/schema-to-json@4.5.0-rc.3` ## @trigger.dev/core@4.5.0-rc.3 ### Patch Changes - Retry `TASK_MIDDLEWARE_ERROR` under the task's retry policy instead of failing the run on the first attempt. The error was already classified as retryable by `shouldRetryError`, but `shouldLookupRetrySettings` did not include it, so the retry flow fell through to `fail_run`. Fixes #3231. ([#3676](https://github.com/triggerdotdev/trigger.dev/pull/3676)) - Fix `TypeError` in `unflattenAttributes` when the input attribute map contains conflicting dotted key paths (e.g. both `a.b` set to a scalar and `a.b.c` set to a value). The path-walk loop now applies last-write-wins when a prior key wrote a primitive, null, or array at an intermediate slot, matching the existing precedent in `AttributeFlattener.addAttribute`. Callers no longer crash when handed malformed external attribute inputs. ([#3762](https://github.com/triggerdotdev/trigger.dev/pull/3762)) - Fix external trace context leaking across runs on warm-started workers with `processKeepAlive` enabled. Every subsequent run's attempt span was being exported with the first run's `traceId` and `parentSpanId`, breaking causal-chain navigation in external APM tools. Runs without an external trace context are unaffected. ([#3768](https://github.com/triggerdotdev/trigger.dev/pull/3768)) ## @trigger.dev/plugins@4.5.0-rc.3 ### Patch Changes - Updated dependencies: - `@trigger.dev/core@4.5.0-rc.3` ## @trigger.dev/python@4.5.0-rc.3 ### Patch Changes - Updated dependencies: - `@trigger.dev/core@4.5.0-rc.3` - `@trigger.dev/build@4.5.0-rc.3` - `@trigger.dev/sdk@4.5.0-rc.3` ## @trigger.dev/react-hooks@4.5.0-rc.3 ### Patch Changes - Updated dependencies: - `@trigger.dev/core@4.5.0-rc.3` ## @trigger.dev/redis-worker@4.5.0-rc.3 ### Patch Changes - Updated dependencies: - `@trigger.dev/core@4.5.0-rc.3` ## @trigger.dev/rsc@4.5.0-rc.3 ### Patch Changes - Updated dependencies: - `@trigger.dev/core@4.5.0-rc.3` ## @trigger.dev/schema-to-json@4.5.0-rc.3 ### Patch Changes - Updated dependencies: - `@trigger.dev/core@4.5.0-rc.3` ## @trigger.dev/sdk@4.5.0-rc.3 ### Patch Changes - Updated dependencies: - `@trigger.dev/core@4.5.0-rc.3` </details> Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>helm-v4.5.0-rc.3 v4.5.0-rc.3 v.docker.4.5.0-rc.3 |
||
|
Eric Allam
|
9211032733 |
chore(database): drop unused TaskRun status composite index (#3743)
## Summary Drops the `TaskRun_status_runtimeEnvironmentId_createdAt_id_idx` index from the `TaskRun` table. After #3742 gated the legacy `WAITING_FOR_DEPLOY` drain to V1-engine workers only, this index sees zero scans on both writer and reader replicas. Removing it cuts index maintenance on every `TaskRun` INSERT/UPDATE. ## Why The index existed to support `WHERE status = X AND runtimeEnvironmentId = Y` queries from `ExecuteTasksWaitingForDeployService`, which is V1-only and no longer triggered on V2 deployments. A code grep across `apps/webapp` and `internal-packages/run-engine` confirmed no V2 production query uses this access pattern — every other `status:` filter on `TaskRun` is paired with `id`/`friendlyId`/`parentSpanId` and uses a different index. Dropping it also unlocks HOT updates on the dequeue path. The dequeue `UPDATE` modifies `status` (`QUEUED` -> `DEQUEUED`), and `status` is the leading column of this index — its presence blocked HOT eligibility for every `TaskRun` UPDATE. With the index gone, dequeue UPDATEs can become HOT, reducing WAL bytes and removing the B-tree page contention on this index's right-edge leaves. Uses `DROP INDEX CONCURRENTLY` to avoid blocking writes during the drop. ## Sequencing Should only ship once #3742 has soaked long enough to confirm the index is genuinely cold (24h+ of zero scans on `pg_stat_user_indexes`). |
||
|
Saadi Myftija
|
9cb6fd17c3 |
fix(webapp): idempotent DeploymentBackgroundWorker creation (#3772)
Make `POST /api/v1/deployments/:deploymentId/background-workers` idempotent so client-side retries no longer collide on the `BackgroundWorker` `(project, env, version)` unique index. Helps make deployments more resilient against the class of indexing failures that surfaces in the dashboard as "Indexing timed out", e.g. during transient database issues. |
||
|
nicktrn
|
c043c4a6ad |
fix(core): external trace context leaks across warm-started runs (#3768)
On warm-started workers with `processKeepAlive` enabled, every run's attempt span was exported with the first run's `traceId` and `parentSpanId`. `ExternalSpanExporterWrapper` and `ExternalLogRecordExporterWrapper` captured `externalTraceContext` at `TracingSDK` construction, and the SDK is memoized for the worker's lifetime - so the per-run reset of `StandardTraceContextManager.traceContext` never reached the wrappers. Reported by a customer running v4.4.x: 33 distinct runs on the same host/pid showed up in their APM as siblings of one parent span. Fix: the wrappers now read external context live from the trace context manager per export. Runs without an external trace context fall through to the unchanged `externalTraceId` fallback - no behaviour change for them. Regression test in `packages/core/test/externalSpanExporterWrapper.test.ts` asserts that reassigning the manager between exports produces correctly-parented spans. |
||
|
Matt Aitken
|
816986d44e |
fix(webapp): treat Phase 2 batch-stream retries as idempotent (TRI-9944) (#3766)
Returns sealed:true when a fast-completing batchTrigger races the stream finalisation, instead of throwing 422/BatchTriggerError. |
||
|
Iss
|
5083d161b2 | docs: adds relevant env vars to self hosting docs (#3148) | ||
|
Iss
|
8399aa2052 |
fix(core): retry TASK_MIDDLEWARE_ERROR under the task's retry policy (#3676)
Retries `TASK_MIDDLEWARE_ERROR` under the task's retry policy. `shouldRetryError` already classed it as retryable, but `shouldLookupRetrySettings` did not, so the run fell through to `fail_run` on attempt 1 instead of using the task's `retry` config. Fixes #3231. |
||
|
Iss
|
7324a3307f | chore(webapp): rename support tier to "Priority" on plan cards and li… (#3692) | ||
|
Iss
|
df96a937db | docs: troubleshoot "Stream is being deleted" during long waits (#3704) | ||
|
Matt Aitken
|
90bbbd1c44 |
fix(webapp): recover from ClickHouse JSON parse failures on out-of-range integers (#3759)
## Summary Second class of poisoned-row failure in the runs replication path. PR #3708 plugged lone UTF-16 surrogates; this one handles bare JSON integer literals outside ClickHouse's `Int64`..`UInt64` range. Recovery stays purely reactive — the existing `sanitizeRows` walker just gains an extra branch, so the hot replication path pays nothing on healthy rows. Fixes the still-firing customer-facing symptom from [TRI-9755](https://linear.app/triggerdotdev/issue/TRI-9755): `scan-social-profiles` runs continued to be stranded in `EXECUTING` on the Tasks page after #3708 deployed. CloudWatch showed `Dropped batch — ClickHouse JSON parse error but sanitizer found nothing to fix` firing **8/8 times** since the previous deploy (zero successful sanitizations). Root cause: upstream JS Number precision loss on a 21-digit Google Plus ID (`117039831458782873093` → `117039831458782870000`) — the precision-lossy value still serialises as a bare integer that exceeds `UInt64.MAX`, which ClickHouse rejects with `INCORRECT_DATA`. ## How the bug ships The customer task emits an output containing a Poshmark profile's `spec_format`: ```json {"key":"gp_id","proper_key":"Gp Id","value":117039831458782870000,"type":"int"} ``` That value is `1.17e20` — comfortably above `UInt64.MAX` (`1.84e19`) but comfortably below `1e21`. `Number.prototype.toString` only switches to exponential form at `|value| >= 1e21`, so `JSON.stringify` emits the bare token `117039831458782870000` and the ClickHouse `JSON(max_dynamic_paths)` column fails with: ``` Code: 117. DB::Exception: Cannot parse JSON object here: {…}: (while reading the value of key output): (at row 1) : While executing ParallelParsingBlockInputFormat. (INCORRECT_DATA) (version 25.12.x) ``` Same error verbatim as prod. The same number quoted (`"117039831458782870000"`) inserts fine — ClickHouse's dynamic JSON column accepts a `String` subtype on the same path. ## What changed `apps/webapp/app/v3/eventRepository/sanitizeRowsOnParseError.server.ts`: - New private `isUnsafeJsonInteger(value)` helper — true iff `value` is a finite integer-valued JS Number where `|value| < 1e21` (so `JSON.stringify` emits integer form, not exponent) **and** `value` falls outside `[Int64.MIN, UInt64.MAX]`. - `sanitizeUnknownInPlace` gains a number-branch: when the predicate holds, replace the Number with `String(value)`. The downstream JSON column dynamic-types the path as String for that row — fine, since the value was already precision-lossy upstream (no JS Number above 2^53 is numerically meaningful anyway). - Float-valued numbers, large floats (>= 1e21), NaN and Infinity are left alone — `JSON.stringify` emits them with exponents or as `null`, both of which ClickHouse accepts. `apps/webapp/test/sanitizeRowsOnParseError.test.ts`: four new unit tests + an extension to `sanitizeRows` covering surrogate + integer fixes counted together across rows. The unit suite now covers: - Positive value above `UInt64.MAX` (`117039831458782870000` — the actual prod value) - Negative value below `Int64.MIN` - Boundary values pass through (`42`, `Number.MAX_SAFE_INTEGER`, `2^63`) - Non-integer numbers untouched (floats, `1e25`, NaN, Infinity) - The actual `scan-social-profiles` nested shape — finds the offending `gp_id` deep inside `output.data.profiles[].spec_format[].platform_variables[].value` `.server-changes/runs-replication-bigint-recovery.md` — release notes entry. ## Why reactive, not pre-flight `#prepareJson` runs millions of times per day on the replication hot path. Walking every JSON tree to look for oversized integers would add bounded-but-real CPU on every healthy row. `sanitizeRows` only fires after a ClickHouse parse-error rejection, which is a few times a day platform-wide. Extending it costs effectively zero on healthy traffic and gains us recovery on the rare poisoned row. ## Verification - Reproduced 1:1 in a throwaway Docker `clickhouse/clickhouse-server:25.12.11.4` (closest available to the prod `25.12.1.1579` build). Pre-sanitize JSON fails with the exact prod error; post-sanitize JSON inserts cleanly and the row is readable with `gp_id` stored as a String subtype. - `pnpm --filter webapp exec vitest run test/sanitizeRowsOnParseError.test.ts` — 22/22 passing (18 existing + 4 new). - `pnpm run typecheck --filter webapp` — clean. ## Test plan - [x] `pnpm run typecheck --filter webapp` - [x] Unit tests pass against new + existing cases - [x] End-to-end Docker ClickHouse repro confirms recovery - [ ] Post-deploy: confirm `Sanitizing batch after ClickHouse JSON parse error` warns fire instead of `Dropped batch …` errors when `scan-social-profiles` outputs trip CH again - [ ] Post-deploy: confirm `permanentlyDroppedBatches` counter stops climbing in `/stp/trigger-app-prod/ecs/replication/service-container/process-logs` ## What this does NOT do - Doesn't backfill the ~120k+ existing stranded `EXECUTING` rows in production. Same as #3708 — that needs a reconciliation/backfill sweep (separate ticket — TRI-9755 fix #3). - Doesn't address the upstream root cause (the customer task emitting a JS-Number-precision-lossy big int). That's a customer-task concern; our replication path needs to be robust to whatever shape arrives. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com> |
||
|
Oskar Otwinowski
|
5ba1b32d53 | fix(core): handle conflicting dotted paths in unflattenAttributes (#3762) | ||
|
Eric Allam
|
37eeaa3690 |
fix(webapp): skip V1 WAITING_FOR_DEPLOY drain on V2 promotes (#3742)
## Summary Stops the legacy V1 `WAITING_FOR_DEPLOY` drain from running on every V2 deployment promotion. The drain queries `TaskRun` by `status='WAITING_FOR_DEPLOY'`, which only V1-engine runs ever have — V2 runs use `PENDING_VERSION` and are handled out of band. Calling the drain on V2 promotes produced empty queries against the status index and unnecessary reader-DB load. ## Fix Two layers: 1. Gate the enqueue at the call site in `ChangeCurrentDeploymentService` so it only fires when the deployment's worker is on engine V1. 2. Add a `LEGACY_RUN_ENGINE_WAITING_FOR_DEPLOY_DISABLED` env var (default `0`). When set to `1`, the service returns immediately from `call()` — neuters any jobs already sitting in the worker queue from before the deploy lands. V1 customers see no change; V2 promotes no longer trigger the drain. |
||
|
Matt Aitken
|
596a9bb90c |
fix(webapp): retain sessions-replication singleton import via globalThis assignment (#3738)
Linear: [TRI-9864](https://linear.app/triggerdotdev/issue/TRI-9864) (Urgent) Production incident: [TRI-9863](https://linear.app/triggerdotdev/issue/TRI-9863) (mitigated by image revert in cloud#910) ## Bug `apps/webapp/package.json` declares `"sideEffects": false`. PR #3333 (`71d98b4e`) replaced the previous real method-call retention idiom at the two `sessionsReplicationInstance` import sites with: ```ts import { sessionsReplicationInstance } from "..."; void sessionsReplicationInstance; ``` esbuild treats `void <identifier>;` as a pure expression statement under `sideEffects: false` and **tree-shakes the entire import** — including the `singleton(...)` call inside `sessionsReplicationInstance.server.ts` which is the only thing that fires `initializeSessionsReplicationInstance()`. The sessions→ClickHouse logical replication worker never starts, the slot is unconsumed, lag grows. ### How it manifested in production cloud#907's image bump rolled the `SessionReplicationService` ECS task on prod at 14:32 UTC. The new container's startup log emitted `🗃️ Runs replication service enabled` but **not** `🗃️ Sessions replication service enabled` or `🗃️ Sessions replication service started`. CloudWatch `OldestReplicationSlotLag` grew at ~220 MB/min and the `High replication lag` alarm fired at 14:37 UTC. Prod was reverted to the previous image (cloud#910) to stop the bleed. ### Verification `grep` of the built bundle `apps/webapp/build/index.js` (built from `c0365d36`): - **3** occurrences of `Runs replication` / `runsReplicationInstance` strings ✅ - **0** occurrences of `Sessions replication` / `sessionsReplicationInstance` / `SessionsReplicationService` ❌ The runs path survives tree-shaking because `adminWorker.server.ts` and `admin.api.v1.runs-replication.*` routes have real method calls (`.start()`, `.teardown()`, `.backfill()`) — observable uses the tree-shaker must preserve. The sessions singleton has no real callers, only the `void` no-ops, hence its complete elimination from the bundle. ## Fix Replace `void sessionsReplicationInstance;` with an assignment to `globalThis`, an unambiguous observable side effect the bundler cannot eliminate: ```ts (globalThis as Record<string, unknown>).__sessionsReplicationInstance = sessionsReplicationInstance; ``` Applied at both call sites: `apps/webapp/app/entry.server.tsx` and `apps/webapp/app/v3/services/adminWorker.server.ts`. Surrounding comments updated to document the bundler interaction so the next maintainer doesn't reintroduce `void`. ## Out of scope (follow-ups) - **Robustness improvement**: change `apps/webapp/package.json` from `"sideEffects": false` to an allowlist that includes `*Instance.server.ts` files. Prevents the same regression shape via any future `*Instance` singleton. - **Build-time check**: add a `grep` post-build step in `publish.yml` requiring `"Sessions replication"` to appear in `apps/webapp/build/index.js`. Catches this exact regression at CI time. ## Test plan - [x] `pnpm run typecheck --filter webapp` clean - [ ] After merge + publish: confirm new image's `SessionReplicationService` container logs `🗃️ Sessions replication service enabled` and `🗃️ Sessions replication service started` at startup - [ ] After re-deploying to prod: confirm `OldestReplicationSlotLag` stops growing and drains |
||
|
Matt Aitken
|
eefb96c87d |
fix(webapp): recover from ClickHouse JSON parse failures in runs replication (#3708)
## Summary On a ClickHouse `Cannot parse JSON object` rejection, `RunsReplicationService` now sanitizes lone UTF-16 surrogates across the failing batch via the existing `sanitizeRows` helper and retries once. If the sanitizer found nothing or the retry also fails, the batch is dropped loudly with a counter increment, so the surrounding `#insertWithRetry` layer doesn't spin three more times on a deterministic failure. Non-parse errors propagate unchanged. Mirrors the pattern from #3659 (for `ClickhouseEventRepository`) — same root cause (lone UTF-16 surrogates in user-provided JSON), same recovery shape, **reusing the same shared helpers** (`sanitizeRows`, `isClickHouseJsonParseError`, `parseRowNumberFromError`). Fixes the customer-facing symptom from [TRI-9755](https://linear.app/triggerdotdev/issue/TRI-9755): a single row's poisoned `output` JSON used to take down the `COMPLETED_SUCCESSFULLY` UPDATE events for its 50+ batch-mates, stranding them in `EXECUTING` in ClickHouse forever and inflating "Running" counts on the Tasks page. Confirmed in production this is ongoing — ~120k stale rows accumulated in a single 5-hour burst on 2026-05-18; smaller continuous leak before and after. ## What changed `apps/webapp/app/services/runsReplicationService.server.ts`: - Imports the three helpers from `~/v3/eventRepository/sanitizeRowsOnParseError.server` (no duplication; no move). - New private `#insertWithJsonParseRecovery<T>(rows, doInsert, contextLabel, attempt)` — generic over `TaskRunInsertArray[]` and `PayloadInsertArray[]`, structurally identical to `ClickhouseEventRepository.#insertWithJsonParseRecovery`. Try → on parse error sanitize the whole batch (the `at row N` hint is logged but not used to slice — semantics under `input_format_parallel_parsing` aren't stable) → retry once → drop with loud log if sanitizer found nothing OR retry still fails. - `#insertTaskRunInserts` and `#insertPayloadInserts` extract a `doInsert` closure and hand it to the wrapper. Existing error logging, span recording, and `recordSpanError` are preserved inside the closure. - New `private _permanentlyDroppedBatches = 0` counter with a public getter, for ops dashboards and tests (matches the events-repo convention). One shared counter for both insert sites — granularity comes from the `contextLabel` (`task_runs_v2` / `raw_task_runs_payload_v1`) on every log line. `.server-changes/runs-replication-utf16-recovery.md` — release notes entry. ## Why no new tests The shared helpers already have full unit + real-ClickHouse contract coverage from #3659 (`apps/webapp/test/sanitizeRowsOnParseError.test.ts`, `apps/webapp/test/otlpUtf16Sanitization.integration.test.ts`). The new wrapper is a line-for-line structural port. Adding a parallel integration test would require synthesizing bad data that *escapes* the preemptive `detectBadJsonStrings` check in `#prepareJson` but still trips ClickHouse — non-trivial without hand-crafted fixtures and wouldn't cover any new logic. ## What this does NOT do - Doesn't touch the ~120k existing stale `EXECUTING` rows in production. That needs a reconciliation/backfill sweep (separate ticket — TRI-9755 fix #3). - Doesn't sanitize the `error` column path (`runsReplicationService.server.ts:932 const errorData = { data: run.error };`). Reactive recovery will catch it if it ever poisons a batch, but feeding it through `#prepareJson` like `output` is a cheap follow-up. ## Test plan - [x] `pnpm run typecheck --filter webapp` — clean - [ ] Post-deploy: confirm `permanentlyDroppedBatches` counter stays at zero (or near-zero) in `/stp/trigger-app-prod/ecs/replication/service-container/process-logs`, and watch for `Sanitizing batch after ClickHouse JSON parse error` warns to confirm recovery is firing on real traffic - [ ] Post-deploy: confirm the rate of new "EXECUTING-but-actually-COMPLETED" zombies in ClickHouse flattens (current rate ≈ tens-to-hundreds per hour platform-wide) 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com> |
||
|
Eric Allam
|
9f64bf404b |
docs(ai-chat): slim-wire HITL continuations + field-level merge contract (#3721)
## Summary Updates the AI chat docs to match the slim-wire + field-level merge behavior shipped in #3719 and the precise `.in/append` cap + CORS-readable 413 shipped in #3720. No behavior changes here — code is correct in `main`; the docs were lagging on three patterns customers copy out of the page. ## What changed - **`hydrateMessages` examples upsert by id** (in `lifecycle-hooks.mdx`, `patterns/database-persistence.mdx`, and `patterns/persistence-and-replay.mdx`). The previous `stored.push(newMsg)` pattern duplicated the assistant id on HITL continuations and caused the LLM to receive a tool call with no `arguments`. The new examples include the rationale inline. - **`onValidateMessages` example filters to user messages** (`lifecycle-hooks.mdx`). The previous example called `validateUIMessages({ messages, tools })` directly, which now throws on HITL slim wires (the AI SDK schema requires `input` on resolved tool parts). New example shows the filter pattern, with a Warning callout explaining why. - **Merge contract description updated** (`lifecycle-hooks.mdx`). The old wording said incoming messages are "auto-merged" / "replaced"; the new description explains the actual field-level overlay (state advances only). - **Approval-responded wire example slimmed** (`client-protocol.mdx`). Shows the minimum shape the agent reads — `state` + `approval` (or `output` / `errorText` for HITL). Notes that the built-in transports ship this slim shape by default and that fuller shapes are still accepted. - **`/in/append` 413 row and FAQ updated** (`client-protocol.mdx`, `patterns/trusted-edge-signals.mdx`). Reflects the new precise S2 cap and the CORS-readable 413. - **New changelog entry** at the top of `changelog.mdx` covering all of the above. The historical `## 512 KiB ceiling removed` entry further down the changelog is left as-is (it's a snapshot of the prior transition), and the v4.5 upgrade-guide section is skipped — the merge contract is backwards compatible. ## Test plan - Mintlify dev preview renders cleanly with no broken anchors - Linked references resolve (`/ai-chat/lifecycle-hooks#hydratemessages`, `/ai-chat/lifecycle-hooks#onvalidatemessages`, `/ai-chat/patterns/database-persistence#alternative-hydratemessages`, `/ai-chat/client-protocol#step-3-send-messages-stops-and-actions`, `/ai-chat/patterns/large-payloads`) |
||
|
github-actions[bot]
|
d34014ddcc |
chore: release v4.5.0-rc.2 (#3702)
🚀 Publish Trigger.dev Docker / typecheck (push) Failing after 0s
🚀 Publish Trigger.dev Docker / publish-webapp (push) Has been skipped
🚀 Publish Trigger.dev Docker / publish-worker (push) Has been skipped
🚀 Publish Trigger.dev Docker / publish-worker-v4 (push) Has been skipped
🚀 Publish Trigger.dev Docker / units (push) Failing after 4s
## Summary
3 improvements, 1 bug fix.
## Improvements
- The per-turn merge now overlays the wire copy's tool-part state
advancement onto the agent's existing chain — `state` + the matching
resolution field (`output` / `errorText` / `approval`) come from the
wire, everything else (text, reasoning, tool `input`, provider metadata)
stays whatever the snapshot or `hydrateMessages` returned. Previously a
full-message replace overwrote those fields with whatever the client
shipped, so a slimmed wire copy landed a tool call with no `arguments`
on the next LLM call. Covers `output-available` / `output-error` (HITL
`addToolOutput`) and `approval-responded` / `output-denied` (approval
flow).
- `TriggerChatTransport.sendMessages` and `AgentChat.sendRaw` now slim
assistant messages that carry advanced tool parts. The wire payload is
just `{ id, role, parts: [<state + resolution field>] }` for
`submit-message` continuations; everything else passes through.
Reasoning blobs and full tool inputs no longer ride the wire on every
`addToolOutput` / `addToolApproveResponse`, so continuation payloads
stay well under the `.in/append` cap on long agent loops.
- Add `TriggerClient` for running multiple SDK clients side-by-side,
each with its own auth, preview branch, and baseURL. Useful when a
single process needs to trigger tasks or read runs across multiple
projects, environments, or preview branches without mutating shared
global state.
([#3683](https://github.com/triggerdotdev/trigger.dev/pull/3683))
## Bug fixes
- Fix `chat.agent` HITL continuations on reasoning-heavy turns. Two
changes that work together:
([#3719](https://github.com/triggerdotdev/trigger.dev/pull/3719))
<details>
<summary>Raw changeset output</summary>
⚠️⚠️⚠️⚠️⚠️⚠️
`main` is currently in **pre mode** so this branch has prereleases
rather than normal releases. If you want to exit prereleases, run
`changeset pre exit` on `main`.
⚠️⚠️⚠️⚠️⚠️⚠️
# Releases
## @trigger.dev/build@4.5.0-rc.2
### Patch Changes
- Updated dependencies:
- `@trigger.dev/core@4.5.0-rc.2`
## trigger.dev@4.5.0-rc.2
### Patch Changes
- Updated dependencies:
- `@trigger.dev/build@4.5.0-rc.2`
- `@trigger.dev/core@4.5.0-rc.2`
- `@trigger.dev/schema-to-json@4.5.0-rc.2`
## @trigger.dev/plugins@4.5.0-rc.2
### Patch Changes
- Updated dependencies:
- `@trigger.dev/core@4.5.0-rc.2`
## @trigger.dev/python@4.5.0-rc.2
### Patch Changes
- Updated dependencies:
- `@trigger.dev/sdk@4.5.0-rc.2`
- `@trigger.dev/build@4.5.0-rc.2`
- `@trigger.dev/core@4.5.0-rc.2`
## @trigger.dev/react-hooks@4.5.0-rc.2
### Patch Changes
- Updated dependencies:
- `@trigger.dev/core@4.5.0-rc.2`
## @trigger.dev/redis-worker@4.5.0-rc.2
### Patch Changes
- Updated dependencies:
- `@trigger.dev/core@4.5.0-rc.2`
## @trigger.dev/rsc@4.5.0-rc.2
### Patch Changes
- Updated dependencies:
- `@trigger.dev/core@4.5.0-rc.2`
## @trigger.dev/schema-to-json@4.5.0-rc.2
### Patch Changes
- Updated dependencies:
- `@trigger.dev/core@4.5.0-rc.2`
## @trigger.dev/sdk@4.5.0-rc.2
### Patch Changes
- Fix `chat.agent` HITL continuations on reasoning-heavy turns. Two
changes that work together:
([#3719](https://github.com/triggerdotdev/trigger.dev/pull/3719))
- The per-turn merge now overlays the wire copy's tool-part state
advancement onto the agent's existing chain — `state` + the matching
resolution field (`output` / `errorText` / `approval`) come from the
wire, everything else (text, reasoning, tool `input`, provider metadata)
stays whatever the snapshot or `hydrateMessages` returned. Previously a
full-message replace overwrote those fields with whatever the client
shipped, so a slimmed wire copy landed a tool call with no `arguments`
on the next LLM call. Covers `output-available` / `output-error` (HITL
`addToolOutput`) and `approval-responded` / `output-denied` (approval
flow).
- `TriggerChatTransport.sendMessages` and `AgentChat.sendRaw` now slim
assistant messages that carry advanced tool parts. The wire payload is
just `{ id, role, parts: [<state + resolution field>] }` for
`submit-message` continuations; everything else passes through.
Reasoning blobs and full tool inputs no longer ride the wire on every
`addToolOutput` / `addToolApproveResponse`, so continuation payloads
stay well under the `.in/append` cap on long agent loops.
Note: `onValidateMessages` receives the slim wire on HITL turns. If you
call `validateUIMessages` from `ai` against the full `messages` array it
will reject the slim assistant; filter to user messages (or skip on HITL
turns) — see the updated docstring on `onValidateMessages` for the
recommended pattern.
For `hydrateMessages` hooks that persist the chain, this release also
adds a small helper to the `@trigger.dev/sdk/ai` surface:
```ts
import { chat, upsertIncomingMessage } from "@trigger.dev/sdk/ai";
chat.agent({
hydrateMessages: async ({ chatId, trigger, incomingMessages }) => {
const record = await db.chat.findUnique({ where: { id: chatId } });
const stored = record?.messages ?? [];
if (upsertIncomingMessage(stored, { trigger, incomingMessages })) {
await db.chat.update({ where: { id: chatId }, data: { messages: stored }
});
}
return stored;
},
});
```
It pushes fresh user messages by id, no-ops on HITL continuations (the
incoming shares an id with the existing assistant — the runtime overlays
the new tool-state advance), and skips on non-`submit-message` triggers.
Returns `true` if it mutated `stored` so the caller knows whether to
persist.
Net effect: `chat.addToolOutput(...)` /
`chat.addToolApproveResponse(...)` on multi-step reasoning agents
(OpenAI Responses with `store: false`, Anthropic extended thinking,
etc.) no longer blows the cap and no longer corrupts the LLM input.
- Add `TriggerClient` for running multiple SDK clients side-by-side,
each with its own auth, preview branch, and baseURL. Useful when a
single process needs to trigger tasks or read runs across multiple
projects, environments, or preview branches without mutating shared
global state.
([#3683](https://github.com/triggerdotdev/trigger.dev/pull/3683))
```ts
import { TriggerClient } from "@trigger.dev/sdk";
const prod = new TriggerClient({ accessToken:
process.env.TRIGGER_PROD_KEY });
const preview = new TriggerClient({
accessToken: process.env.TRIGGER_PREVIEW_KEY,
previewBranch: "signup-flow",
});
await prod.tasks.trigger("send-email", payload);
await preview.runs.list({ status: ["COMPLETED"] });
```
- Updated dependencies:
- `@trigger.dev/core@4.5.0-rc.2`
## @trigger.dev/core@4.5.0-rc.2
</details>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
v.docker.4.5.0-rc.2
v4.5.0-rc.2
|
||
|
Eric Allam
|
75679c7518 |
fix(sdk): chat HITL continuations no longer break the next LLM call (#3719)
## Summary
Multi-step reasoning agents with HITL tools (OpenAI Responses with
`store: false`, Anthropic extended thinking, etc.) failed on
`chat.addToolOutput(...)` continuations — either the wire payload blew
the `.in/append` cap (reasoning blobs + tool inputs routinely > 512
KiB), or app-side slimming workarounds got overwritten server-side and
the next LLM call landed a tool call with no `arguments`. Both modes are
fixed.
## Design
The per-turn merge in `chat.agent` now overlays only the tool-part state
advances (`output-available` / `output-error` / `approval-responded` /
`output-denied`) from the wire copy onto the hydrated/snapshot chain.
Previously it replaced the entire message, which dropped `input`,
reasoning, and text from the LLM's view whenever the wire was slim.
In parallel, `TriggerChatTransport.sendMessages` and `AgentChat.sendRaw`
now slim the assistant message themselves on `submit-message`
continuations: ship `{ id, role, parts: [<resolved tool part only>] }`,
everything else reconstructed server-side from `hydrateMessages` or the
durable snapshot. Continuation payloads drop from 600 KiB – 1 MiB to ~1
KiB.
`references/ai-chat` `aiChatHydrated.hydrateMessages` now upserts by id
instead of pushing. With slim continuations, a blind push duplicates the
assistant id in the returned chain — the merge updates the first match,
the slim duplicate goes straight to `toModelMessages` with no `input`,
and the LLM 4xx's. This is the canonical pattern customers should mirror
in their own hydrate implementations.
## Test plan
- 11 new tests (slim helper unit + slim+merge integration for HITL,
approval, default no-hydrate branch)
- Full SDK suite: 239 tests pass across 19 files
- End-to-end sweep against `references/ai-chat`: 19 customer-side smoke
tests green; HITL wire bodies confirmed at ~1 KiB (was 600 KiB+); no
provider 4xx errors across OpenAI Responses or Anthropic
|
||
|
Eric Allam
|
c0365d36fb |
fix(webapp): precise S2 record cap + CORS 413 on session append (#3720)
## Summary
Two improvements to session `.in/append`:
- Oversize-body 413 responses now carry CORS headers, so browser fetches
see a readable status instead of an opaque `TypeError: Failed to fetch`.
App-side retry-on-disconnect loops no longer spin forever on a
permanently-rejected payload.
- The per-record cap is now computed precisely against S2's actual
ceiling instead of the conservative 512 KiB floor. Legitimate ~600-900
KiB tool outputs (search results, file content) now succeed;
pathological all-quote content that would double under JSON escape still
rejects cleanly.
## Design
S2 enforces a per-record metered size of `8 + 2*H + Σ(header name +
value) + body ≤ 1048576` bytes. With no record headers (our case), the
budget reduces to `body ≤ 1048568`. Verified empirically against cloud
S2 — append succeeds at metered=1048576 and 422s at 1048577 with `record
must have metered size less than 1 MiB`.
The old `MAX_APPEND_BODY_BYTES = 512 KiB` was derived by assuming
worst-case JSON escape doubling (every byte becomes `\"` or `\\`),
giving `(1 MiB - overhead) / 2`. Safe, but rejects ~half the legitimate
input space.
The new flow:
1. Pre-cap the HTTP body at 1 MiB (DoS guard against reading arbitrary
garbage before we can compute the wrap).
2. After reading, `S2RealtimeStreams.#appendPartByName` computes
`Buffer.byteLength(JSON.stringify({data: part, id: partId}), "utf8") +
8` and throws `S2RecordTooLargeError` (a `ServiceValidationError` with
status 413) if it would exceed S2's ceiling. The route's existing error
branch maps the throw to a 413 with a descriptive message.
The 413 CORS fix is a single-line change in `apiBuilder.server.ts` —
`wrapResponse` was being skipped on the body-too-large branch; every
other error branch wraps; the 413 was the exception.
## Test plan
- Empirically verified against cloud S2 with a boundary scan across
`[1048568, 1048569, ..., 1048576]` and across H ∈ {0, 1×5 hdr bytes,
1×14 hdr bytes} — the formula matches exactly
- Browser-side fetch on a 700 KiB POST now resolves with a readable
`status: 413` (no `TypeError: Failed to fetch`)
- A 900 KiB ASCII tool output now passes (would have 413'd at 512 KiB
pre-fix)
|
||
|
Matt Aitken
|
2fcc484a2b |
fix(webapp): route OrganizationDataStoresRegistry writes through the writer prisma (#3722)
## Bug
The `OrganizationDataStoresRegistry` singleton in
`apps/webapp/app/services/dataStores/organizationDataStoresRegistryInstance.server.ts`
was constructed with `$replica`. That client was then used by both the
polling read path *and* by `addDataStore` / `updateDataStore` /
`deleteDataStore` (and their backing `SecretStore.setSecret` upserts).
The write methods route through the read replica, which Postgres rejects
with **error code 25006**:
```
Invalid prisma.secretStore.upsert() invocation:
ConnectorError(ConnectorError { user_facing_error: None, kind: QueryError(PostgresError {
code: "25006",
message: "cannot execute INSERT in a read-only transaction",
...
}), transient: false })
```
User-visible symptom: the admin `/admin/data-stores` "Add data store"
form returns a 400 with this error wrapped, so no
`OrganizationDataStore` row can ever be created via the UI.
The read path (`loadFromDatabase` polling + `SecretStore.getSecret`) is
unaffected because `findMany` + secret read are read-only.
## Fix
Change the registry constructor to take both a writer and a replica:
```ts
constructor(writer: PrismaClient, replica: PrismaClient | PrismaReplicaClient)
```
- `loadFromDatabase()` keeps using `_replica` (and its
`SecretStore.getSecret` calls) — these are background cache-fillers, not
on user-latency-sensitive paths.
- `addDataStore` / `updateDataStore` / `deleteDataStore` (and their
`SecretStore.setSecret` / `deleteSecret` calls) now use `_writer`.
`organizationDataStoresRegistryInstance.server.ts` passes `(prisma,
$replica)` from `~/db.server`. Test sites that constructed with
`(prisma)` now pass `(prisma, prisma)` — the testcontainer exposes a
single client, so the writer/replica split collapses to one connection.
## Files
-
`apps/webapp/app/services/dataStores/organizationDataStoresRegistry.server.ts`
— constructor + read/write split
-
`apps/webapp/app/services/dataStores/organizationDataStoresRegistryInstance.server.ts`
— pass `prisma` alongside `$replica`
- `apps/webapp/test/organizationDataStoresRegistry.test.ts` — 14 call
sites bumped
- `apps/webapp/test/clickhouseFactory.test.ts` — 5 call sites bumped
## Test plan
- [x] Existing `organizationDataStoresRegistry.test.ts` +
`clickhouseFactory.test.ts` still pass (constructor sites updated;
behavior unchanged for tests).
- [ ] After deploy to test cloud, retry `/admin/data-stores` "Add data
store" form for the HIPAA org — should now succeed and the row should
appear.
- [ ] Verify the registry's polling reload picks up the new row within
`ORGANIZATION_DATA_STORES_RELOAD_INTERVAL_MS` (60s default) and the
factory starts routing to the org-scoped instance.
|