4f8cf4cc634d02eae75debd27472ec30eebf70e2
7301 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Katia Bulatova
|
4f8cf4cc63 |
feat(webapp): runs live updating
## Summary The Runs list now updates live without requiring a page refresh. Status changes and other run fields are updated in place while runs are executing. When new runs matching the current filters are created, a "New runs created" refresh button appears above the list. Root runs now show a live child-run status breakdown directly in the status tooltip. ### List live update - Visible runs update in place while they are still running. - A "New runs created" refresh button appears when new matching runs are detected. - Polling stops when all visible runs have finished and a refresh button is already shown. - Polling pauses when the browser tab is not visible. - Runs list status updates and new-run detection share a single runs/live polling path. ### Child-status tooltip - Root run tooltips now display a breakdown of child run statuses. - Child statuses are loaded when the tooltip opens (after a 400ms hover delay). - The tooltip stays up to date while child runs are still changing state. - Handles cases where child runs continue running after their parent run has completed, or have not yet been created. ### Supporting changes - Added hidden-tab awareness to polling. - Added safeguards around polling inputs (`runIds` deduping and limits). ## Test plan - [x] pnpm run typecheck --filter webapp passes - [x] cd apps/webapp && pnpm run test ./test/presenters/mapRunToLiveFields.test.ts --run passes - [x] cd apps/webapp && pnpm run test ./test/runsRepository.part2.test.ts --run -t "hasNewRuns" passes ### Manual smoke: - [x] Active runs update without a page refresh. - [x] A new matching run shows the refresh banner and the banner actions work as expected. - [x] Root run tooltips show live child-status updates and stop polling once child runs settle. --------- Co-authored-by: Ekaterina Bulatova <kathiekiwi@Ekaterinas-MacBook-Pro.local> |
||
|
Matt Aitken
|
6961004a73 |
fix(webapp): restore Postgres fallback for non-ClickHouse OTLP spans (#3803)
## Problem On environments where runs carry a Postgres-backed `taskEventStore` value (`taskEvent` or `taskEventPartitioned`), OTLP ingest endpoints (`POST /otel/v1/traces` and `/otel/v1/logs`) were returning HTTP 500. **Root cause:** The org-scoped ClickHouse factory introduced in a recent PR routes all OTLP spans through `getEventRepositoryForOrganizationSync` → `buildEventRepository`. That function only handles `"clickhouse"` and `"clickhouse_v2"` store values and throws `Unknown ClickHouse event repository store: <value>` for anything else. The throw occurred inside the grouping loop of `#exportEvents`, unwinding the entire method and returning 500 for the whole batch. The OpenTelemetry collector's `otlphttp` exporter treats HTTP 500 as non-retryable and drops the batch — causing real span loss. **Fix:** Guard the `getEventRepositoryForOrganizationSync` call in `#exportEvents` so it is only invoked for `clickhouse` / `clickhouse_v2` store values. All other values are routed directly to the Postgres `eventRepository`, matching the guard pattern already present in `resolveEventRepositoryForStore` and `getEventRepositoryForStore` in `eventRepository/index.server.ts`. The ClickHouse factory call is also wrapped in a try/catch that falls back to Postgres so any unexpected store value in a future OTLP batch degrades gracefully instead of failing the whole request. ## Changes - `apps/webapp/app/v3/otlpExporter.server.ts` — add Postgres routing guard and try/catch fallback in `#exportEvents` ## Testing The `eventRepository/index.server.ts` module already has the same guard pattern thoroughly covered. The fix brings `#exportEvents` into alignment with that existing, tested pattern. Manual verification: confirm OTLP batches containing Postgres-store spans return 200 and route to the correct repository. |
||
|
Eric Allam
|
4b78d7e84e |
feat(core,webapp): support isSecret on environment variable imports (#3809)
## Summary
The environment variables import API now accepts an optional `isSecret`
flag, so imported variables can be created as secret (redacted)
environment variables instead of plaintext. When the flag is omitted,
variables default to non-secret, preserving existing behavior for CLI
deploys and dashboard imports.
This is useful for tools that push secrets into Trigger.dev (for
example, syncing from a secrets manager) and want them stored as secrets
rather than plain environment variables.
It's available through `envvars.import` in the SDK and the `POST
/api/v1/projects/{projectRef}/envvars/{slug}/import` endpoint, and is
honored for both regular and preview-branch environments.
```ts
await envvars.import("proj_1234", "prod", {
variables: { STRIPE_SECRET_KEY: "sk_live_..." },
isSecret: true,
});
```
|
||
|
Eric Allam
|
005d7e0a3f |
chore: add pkg.pr.new preview package releases (#3806)
## What Adds [pkg.pr.new](https://pkg.pr.new) continuous preview releases. Every push to a branch builds the public `@trigger.dev/*` packages and publishes installable preview builds keyed by commit SHA — **without touching the npm registry**. pkg.pr.new drops install instructions on the associated PR: ``` npm i https://pkg.pr.new/@trigger.dev/sdk@<sha> ``` This lets reviewers and users try a branch (SDK, CLI, core, etc.) before anything is released, separate from the changesets release, the manual `--snapshot` prerelease, and the chat-prerelease flow. ## How `.github/workflows/preview-packages.yml` (push trigger) → install → generate Prisma → **stamp preview version** → build → `pkg-pr-new publish`. ### The version stamp (the important part) pkg.pr.new serves previews by SHA but does **not** rewrite the package.json `version` field. If a preview shipped as `4.5.0-rc.4`, a consumer who installed it would pin `4.5.0-rc.4` to the preview tarball in their lockfile/cache — and a later `npm i @trigger.dev/sdk@4.5.0-rc.4` from npm could resolve to the stale preview. This is a known, by-design gap in the tool (stackblitz-labs/pkg.pr.new#250, #390). `scripts/stamp-preview-version.mjs` runs **before the build** and rewrites every public package to a unique `0.0.0-preview-<sha>`. The `0.0.0-` prefix can never satisfy a real semver range, so the collision is structurally impossible (same convention React/Next canaries use). Running before the build also means `scripts/updateVersion.ts` bakes the preview version into the runtime `VERSION` constant, so previews are self-identifying (`trigger --version`, the `x-trigger-cli-version` header, the MCP server version) instead of all reporting the RC version. Sibling `workspace:` specifiers are relaxed to `workspace:*` so `pnpm pack` resolves them against the rewritten versions — `packages/python` pins peerDependencies as `workspace:^4.5.0-rc.4`, which would otherwise be unsatisfiable once the version changes. Non-public deps (`@trigger.dev/database`, `@internal/*`) are left untouched. All mutations happen on the ephemeral CI checkout; nothing is committed. ## GitHub App The pkg.pr.new GitHub App is **already installed** on `triggerdotdev/trigger.dev` (has been for a while), so no setup is needed. Confirmed live — this branch's pushes published all 10 public packages, e.g. ``` pnpm add https://pkg.pr.new/@trigger.dev/sdk@e4dfc59 ``` ## Fork limitation pkg.pr.new authenticates with a GitHub Actions OIDC token, which GitHub does not issue to `pull_request` workflows from forks. The `push` trigger therefore covers branches pushed to this repo (core team), not external fork PRs. Fork coverage would need a `workflow_run` two-stage setup; left out for now. ## Notes - Pinned `pkg-pr-new@0.0.75` (no Node engine constraint; Node 20 CI is fine). - pkg.pr.new [#525](https://github.com/stackblitz-labs/pkg.pr.new/pull/525) adds a built-in `--previewVersion` flag (still open). If it lands we can drop the version-rewrite half of the script, but we'd keep a pre-build stamp anyway so `updateVersion.ts` picks up the preview version (the flag rewrites at pack time, too late for the baked `VERSION`). |
||
|
Daniel Sutton
|
139eede41c |
feat(redis-worker): batched pop in MollifierDrainer for fast single-env drains (#3797)
## Summary Adds `drainBatchSize` to `MollifierDrainer` (default `1` — preserves existing behaviour) and wires `TRIGGER_MOLLIFIER_DRAIN_BATCH_SIZE` through the webapp (default `50`). Each tick the drainer now pops up to `drainBatchSize` from each chosen env, then dispatches every popped entry through the shared `concurrency`-bounded `pLimit`. Per-org/per-env fairness is unchanged — only the in-env pop count grows. Pre-existing behaviour was one pop per env per tick. For a single-env burst that single-flighted the drain at the per-tick floor of `pop + engine.trigger ≈ 50–60 ms`. With buffer entries piling up under a real-world tenant burst that's tens of minutes of tail latency to fully materialise — even though PG itself could comfortably sustain the writes. ## Why this matters — heavy-tail illustration Scenario: 100 customers in one window — 94 fire 20 triggers each, 5 fire 100, 1 fires 1000. Gate at `THRESHOLD=10/s`, `HOLD_MS=500`. First 10 of each burst hit PG directly; the rest buffer. | Customers | Triggers each | PG direct | Buffered each | Total buffered | |---|---|---|---|---| | 94 small | 20 | 10 | 10 | 940 | | 5 medium | 100 | 10 | 90 | 450 | | 1 heavy | 1000 | 10 | 990 | 990 | **With `DRAIN_BATCH_SIZE=50`, `DRAIN_CONCURRENCY=50`, ~50 ms `engine.trigger`:** | Tick | Pops | Dispatch waves | Wall-clock | |---|---|---|---| | 1 | 94×10 + 5×50 + 1×50 = 1 240 | 25 × 50 ms | ~1 300 ms (94 smalls done) | | 2 | 5×40 + 1×50 = 250 | 5 × 50 ms | ~300 ms (5 mediums done) | | 3–20 | heavy alone, 50/tick | 1 × 50 ms | ~100 ms each | | Customer class | Buffered fully drained | |---|---| | 94 small | **~1.3 s** | | 5 medium | **~1.6 s** | | 1 heavy | **~3.4 s** | **Without batching (one pop per env per tick — current behaviour):** | Customer class | Buffered fully drained | |---|---| | 94 small | ~500 ms | | 5 medium | ~4.5 s | | 1 heavy | **~49 s** | So the heavy single-tenant tail drops from ~49 s to ~3.4 s (~14× faster) without changing PG load characteristics. Smalls go up slightly in this scenario (500 ms → 1.3 s) because all 100 envs share one tick's dispatch queue — that's the trade we accept for the heavy tail; the worst-case small wait is still inside one tick. PG load is identical either way (50 concurrent inserts at a time, capped by `DRAIN_CONCURRENCY`). ## What changed **`packages/redis-worker`** - New `drainBatchSize` option (default 1 — full backward compat). - `runOnce()` refactored to pop per-env batches in parallel, then dispatch all popped entries through the existing global `pLimit`. Mid-batch pop failure aborts only that env's batch and counts as one failure (same semantic as the old per-env path). - Removed the now-unused `processOneFromEnv` helper. **`apps/webapp`** - `TRIGGER_MOLLIFIER_DRAIN_BATCH_SIZE` env var (default 50, matching `DRAIN_CONCURRENCY`). - Wired into `mollifierDrainer.server.ts`. **Test cloud config** (separate cloud PR): `TRIGGER_MOLLIFIER_DRAIN_BATCH_SIZE="50"` on the worker service. Production rollout deferred until we've watched it on test cloud. ## Test plan - [x] All 25 stub-based drainer tests pass (18 pre-existing + 7 new). 7 new tests under `MollifierDrainer.drainBatchSize`: - pops up to `drainBatchSize` across ticks - global `concurrency` cap still holds when batch > concurrency - mid-batch pop failure isolation - multi-env batch fan-out in one tick - **hierarchical org fairness preserved at `drainBatchSize > 1`** (load-bearing — guards against future regressions to per-env-instead-of-per-org rotation) - mixed success/failure accounting in a batched tick - bounded pops on empty queue (no Lua spam past `drainBatchSize`) - [x] All pre-existing tests still pass unchanged at default `drainBatchSize=1` → backward-compat locked. - [x] `pnpm run build --filter @trigger.dev/redis-worker` clean. - [x] `pnpm run typecheck --filter webapp` clean. - [x] `redisTest` block (real Redis via testcontainers) — couldn't run locally on this branch due to testcontainers runtime discovery; will validate in CI. - [ ] Test-cloud smoke after cloud PR lands: fire `burst 50` against a flagged env and confirm the 50th entry's drain time drops from ~2.5 s to <200 ms. ## Notes - Per-tick memory bound: `maxOrgsPerTick × drainBatchSize` entries can sit in the JS pLimit queue between pop and dispatch. At defaults that's `500 × 50 = 25 000` × ~5 KB snapshot ≈ ~125 MB worst case per worker — well within headroom. - The pre-batch model's strict per-env throughput cap of `1/tick` is documented as the fairness baseline elsewhere. Org-level fairness is what callers actually rely on; this change does not weaken that. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
Eric Allam
|
0bbb4f13c6 | chore: add ConProgramming to vouch list (#3807) | ||
|
DKP
|
fa5fc6a9c7 | chore(webapp): remove Vercel install Loops event (#3805) | ||
|
nicktrn
|
11631c19e1 |
chore: bump node to latest patch release (#3802)
Bumps Node to the latest 20.x patch. |
||
|
Matt Aitken
|
b23ec26d16 |
chore: vouch ConProgramming (#3804)
Vouches GitHub user [`ConProgramming`](https://github.com/ConProgramming) (Conner) as an outside contributor by adding them to `.github/VOUCHED.td`, so their PRs aren't auto-closed by the vouch check. Done as a direct edit rather than via the issue flow because they have no open Vouch Request issue to comment `vouch` on. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
Dan
|
cd252801eb |
feat: dashboard agent - package upgrades (#3793)
1. in webapp folder update ai-sdk to 6.x.x 2. update vitest to 4.xx |
||
|
Eric Allam
|
4c4ed22e82 |
docs(ai-chat): document the chat.agent tools option (#3791)
## Summary Documents the new `tools` option on `chat.agent` (companion to #3790). Adds a dedicated [Tools](/ai-chat/tools) guide: the three places tools show up (config, `toStreamTextOptions`, `streamText`), why declaring them on the config matters for `toModelOutput` across turns, static vs per-turn tools, the typed `run()` payload, `InferChatUIMessageFromTools`, the relationship to skills, and the manual `convertToModelMessages` path for `customAgent` loops. Threads the option through the rest of the guide: the reference tables, a happy-path section on the backend page, the types page, and the HITL / skills / tool-result-auditing patterns. Corrects the sub-agents guide, where the `toModelOutput` compression was implied to work across turns but silently degraded from turn 2 without config tools. Also unstacks the three callouts that were piled under the `chat.agent()` header on the backend page, and adds a changelog entry. |
||
|
github-actions[bot]
|
e0681d2394 |
chore: release v4.5.0-rc.4 (#3788)
🚀 Publish Trigger.dev Docker / units (push) Failing after 3s
🚀 Publish Trigger.dev Docker / typecheck (push) Failing after 19s
🚀 Publish Trigger.dev Docker / publish-webapp (push) Has been skipped
🚀 Publish Trigger.dev Docker / publish-worker (push) Has been skipped
🚀 Publish Trigger.dev Docker / publish-worker-v4 (push) Has been skipped
🧭 Helm Chart Release / lint-and-test (push) Has been cancelled
🧭 Helm Chart Release / release (push) Has been cancelled
## Summary 1 new feature, 5 improvements. ## Highlights - Mollifier `mutateSnapshot` now enforces a tag cap: an `append_tags` patch carrying `maxTags` returns `"limit_exceeded"` (writing nothing) when the deduped tag count would exceed the limit, so a buffered run can't accumulate more tags via the tags API than the trigger validator allows at creation. ([#3756](https://github.com/triggerdotdev/trigger.dev/pull/3756)) ## Improvements - Mollifier buffer extensions: idempotency dedup, an atomic `mutateSnapshot` API, metadata CAS, claim primitives, and a `MollifierSnapshot` type. The buffer's Redis client now reconnects with jittered backoff so a fleet of clients doesn't stampede Redis in lockstep after a blip. ([#3752](https://github.com/triggerdotdev/trigger.dev/pull/3752)) - Add `onTerminalFailure` callback to `MollifierDrainerOptions` so the customer's run lands a SYSTEM_FAILURE PG row even when the drainer exhausts `maxAttempts` on a retryable PG error. Previously, retryable-error exhaustion called `buffer.fail()` directly, which atomically marks FAILED + DELs the entry hash with no PG write — silent data loss when PG was unreachable across the full retry budget. The callback fires before `buffer.fail()` on any terminal path (`cause: "non-retryable"` or `"max-attempts-exhausted"`); throwing a retryable error from the callback causes the drainer to requeue rather than fail. ([#3754](https://github.com/triggerdotdev/trigger.dev/pull/3754)) - Bump `@s2-dev/streamstore` to `0.22.10` to fix a `TASK_RUN_UNCAUGHT_EXCEPTION` ("Invalid state: Unable to enqueue") when a `chat.agent` turn is aborted mid-stream. ([#3792](https://github.com/triggerdotdev/trigger.dev/pull/3792)) - Coerce numeric `concurrencyKey` values to string at the API boundary across `tasks.trigger`, `tasks.batchTrigger`, and the Phase-2 streaming batch endpoint. ([#3789](https://github.com/triggerdotdev/trigger.dev/pull/3789)) - Add a `tools` option to `chat.agent`. Declaring your tools here threads them into the SDK's internal `convertToModelMessages`, so each tool's `toModelOutput` is re-applied when prior-turn history is re-converted. ([#3790](https://github.com/triggerdotdev/trigger.dev/pull/3790)) <details> <summary>Raw changeset output</summary> ⚠️⚠️⚠️⚠️⚠️⚠️ `main` is currently in **pre mode** so this branch has prereleases rather than normal releases. If you want to exit prereleases, run `changeset pre exit` on `main`. ⚠️⚠️⚠️⚠️⚠️⚠️ # Releases ## @trigger.dev/redis-worker@4.5.0-rc.4 ### Minor Changes - Mollifier buffer extensions: idempotency dedup, an atomic `mutateSnapshot` API, metadata CAS, claim primitives, and a `MollifierSnapshot` type. The buffer's Redis client now reconnects with jittered backoff so a fleet of clients doesn't stampede Redis in lockstep after a blip. ([#3752](https://github.com/triggerdotdev/trigger.dev/pull/3752)) - Add `onTerminalFailure` callback to `MollifierDrainerOptions` so the customer's run lands a SYSTEM_FAILURE PG row even when the drainer exhausts `maxAttempts` on a retryable PG error. Previously, retryable-error exhaustion called `buffer.fail()` directly, which atomically marks FAILED + DELs the entry hash with no PG write — silent data loss when PG was unreachable across the full retry budget. The callback fires before `buffer.fail()` on any terminal path (`cause: "non-retryable"` or `"max-attempts-exhausted"`); throwing a retryable error from the callback causes the drainer to requeue rather than fail. ([#3754](https://github.com/triggerdotdev/trigger.dev/pull/3754)) ### Patch Changes - Pipeline the per-entry `HGETALL` fetches in `MollifierBuffer.listEntriesForEnv`. The previous serial implementation issued one Redis round-trip per runId returned by `LRANGE`, which dominated stale-sweep wall-time at any meaningful backlog (at the sweep's default maxCount=1000, this is ~1000 RTTs per env per pass). Behaviour is unchanged — entries are still skipped when the entry hash has been torn down by a concurrent drainer ack/fail between the LRANGE and the HGETALL. ([#3752](https://github.com/triggerdotdev/trigger.dev/pull/3752)) - Mollifier `mutateSnapshot` now enforces a tag cap: an `append_tags` patch carrying `maxTags` returns `"limit_exceeded"` (writing nothing) when the deduped tag count would exceed the limit, so a buffered run can't accumulate more tags via the tags API than the trigger validator allows at creation. ([#3756](https://github.com/triggerdotdev/trigger.dev/pull/3756)) - Updated dependencies: - `@trigger.dev/core@4.5.0-rc.4` ## @trigger.dev/build@4.5.0-rc.4 ### Patch Changes - Updated dependencies: - `@trigger.dev/core@4.5.0-rc.4` ## trigger.dev@4.5.0-rc.4 ### Patch Changes - Bump `@s2-dev/streamstore` to `0.22.10` to fix a `TASK_RUN_UNCAUGHT_EXCEPTION` ("Invalid state: Unable to enqueue") when a `chat.agent` turn is aborted mid-stream. ([#3792](https://github.com/triggerdotdev/trigger.dev/pull/3792)) - Updated dependencies: - `@trigger.dev/core@4.5.0-rc.4` - `@trigger.dev/build@4.5.0-rc.4` - `@trigger.dev/schema-to-json@4.5.0-rc.4` ## @trigger.dev/core@4.5.0-rc.4 ### Patch Changes - Coerce numeric `concurrencyKey` values to string at the API boundary across `tasks.trigger`, `tasks.batchTrigger`, and the Phase-2 streaming batch endpoint. ([#3789](https://github.com/triggerdotdev/trigger.dev/pull/3789)) - Bump `@s2-dev/streamstore` to `0.22.10` to fix a `TASK_RUN_UNCAUGHT_EXCEPTION` ("Invalid state: Unable to enqueue") when a `chat.agent` turn is aborted mid-stream. ([#3792](https://github.com/triggerdotdev/trigger.dev/pull/3792)) ## @trigger.dev/plugins@4.5.0-rc.4 ### Patch Changes - Updated dependencies: - `@trigger.dev/core@4.5.0-rc.4` ## @trigger.dev/python@4.5.0-rc.4 ### Patch Changes - Updated dependencies: - `@trigger.dev/sdk@4.5.0-rc.4` - `@trigger.dev/core@4.5.0-rc.4` - `@trigger.dev/build@4.5.0-rc.4` ## @trigger.dev/react-hooks@4.5.0-rc.4 ### Patch Changes - Updated dependencies: - `@trigger.dev/core@4.5.0-rc.4` ## @trigger.dev/rsc@4.5.0-rc.4 ### Patch Changes - Updated dependencies: - `@trigger.dev/core@4.5.0-rc.4` ## @trigger.dev/schema-to-json@4.5.0-rc.4 ### Patch Changes - Updated dependencies: - `@trigger.dev/core@4.5.0-rc.4` ## @trigger.dev/sdk@4.5.0-rc.4 ### Patch Changes - Add a `tools` option to `chat.agent`. Declaring your tools here threads them into the SDK's internal `convertToModelMessages`, so each tool's `toModelOutput` is re-applied when prior-turn history is re-converted. ([#3790](https://github.com/triggerdotdev/trigger.dev/pull/3790)) ```ts chat.agent({ tools: { readFile, search }, run: async ({ messages, tools, signal }) => streamText({ model, messages, tools, abortSignal: signal }), }); ``` Also exports `InferChatUIMessageFromTools<typeof tools>` to derive the chat `UIMessage` type (typed tool parts) directly from a tool set. - Updated dependencies: - `@trigger.dev/core@4.5.0-rc.4` </details> Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>helm-v4.5.0-rc.4 v.docker.4.5.0-rc.4 v4.5.0-rc.4 |
||
|
nicktrn
|
e35f5742d4 |
fix(webapp): upgrade posthog-node to v5, drop axios + stale override (#3801)
Follow-up to #3796, which bumped the slack-client axios paths but left posthog-node's transitive `axios@1.15.1` in place. `posthog-node` 4.17.1 → 5.35.6. v5 drops the axios dependency entirely (it's now fetch-based via `@posthog/core`), so posthog's old axios path disappears. With #3796 already on main (webapp + d3 references on `@slack/web-api@7.16.0`), nothing else pins the old line, so the now-dead `axios@>=1.0.0 <1.15.0` override is removed and axios resolves to a single patched `1.16.1` repo-wide. This closes the remaining axios advisories. Compat: the webapp's usage in `telemetry.server.ts` (`new PostHog(key, { host })`, `.identify`, `.groupIdentify`, `.capture`) is all object-form API that v5 preserves; `pnpm run typecheck --filter webapp` passes. Node: posthog-node v5 requires Node `^20.20.0 || >=22.22.0`. We run 20.20.0 in dev (`.nvmrc`), CI, and the published Docker image (`node:20.20-bullseye-slim`), so we're compliant. |
||
|
nicktrn
|
952139d492 |
fix(webapp): bump @slack/web-api to 7.16.0 for patched axios (#3796)
Bumps `@slack/web-api` 7.9.1 → 7.16.0 in the webapp and the two `references` examples (d3-chat, d3-openai-agents). 7.16.0 depends on `axios@^1.16.0`, so every slack-client axios path resolves to 1.16.1 instead of 1.15.1. This clears the slack and references axios paths. `posthog-node`'s transitive axios still resolves the older line - that's handled in a follow-up that upgrades posthog-node to v5 (which drops the axios dependency entirely and lets us retire the now-stale axios override). The dependabot axios advisories fully close once both land. |
||
|
Eric Allam
|
2dd9f37b40 |
fix(core,cli): stop chat.agent uncaught exception on mid-stream abort (#3792)
## Summary A `chat.agent` turn that gets aborted mid-stream (stop generation, idle suspend, cancellation) could surface a `TASK_RUN_UNCAUGHT_EXCEPTION` — `TypeError: Invalid state: Unable to enqueue` — in the dashboard. The run kept working, but the exceptions were loud and confusing. ## Root cause The realtime stream writer batches chunks through `@s2-dev/streamstore`'s `BatchTransform`, which holds them for a short linger window before flushing. When the turn's abort signal fires while a record is still buffered, the stream's writable is aborted and the transform's readable controller is closed — but the pending linger `setTimeout` still fires and calls `controller.enqueue()` on the dead controller, throwing from a timer callback where nothing can catch it. Fixed upstream in `@s2-dev/streamstore@0.22.10`, which wraps the linger flush in a try/catch that discards the closed-controller error. This bumps the dependency across core, the CLI, and the webapp, and adds a regression test against the real `BatchTransform`. Verified end-to-end: a mid-stream stop that previously failed the run with `TASK_RUN_UNCAUGHT_EXCEPTION` now leaves the run healthy. |
||
|
Eric Allam
|
e9e2ec1cfc |
fix(sdk): re-apply tool toModelOutput across chat.agent turns (#3790)
## Summary
`chat.agent` now takes a `tools` option. Until now tools only went to
`streamText` inside `run()`, so the SDK had no tools when it
re-converted the persisted `UIMessage` history at the start of each
turn. Any tool with a `toModelOutput` (raw image bytes into an image
content part, or a sub-agent transcript compressed to a summary) had its
transform applied on turn 1 and skipped from turn 2 onward, so the raw
output got JSON-stringified back into the prompt and the model lost the
transformed view.
Declaring `tools` on the config threads them into that conversion, so
`toModelOutput` runs on every turn. The resolved set is handed back,
typed, on the `run()` payload as `tools`:
```ts
const tools = { searchDocs, renderChart };
export const myChat = chat.agent({
tools,
run: async ({ messages, tools, signal }) =>
streamText({ ...chat.toStreamTextOptions({ tools }), messages, abortSignal: signal }),
});
```
`tools` also accepts a per-turn function for tools that depend on the
user or a feature flag. Only `inputSchema` and `toModelOutput` are read
during conversion, never `execute`. Also exports
`InferChatUIMessageFromTools<typeof tools>` to derive the chat
`UIMessage` type from a tool set. No behavior change for agents that
don't declare `tools`.
|
||
|
Daniel Sutton
|
e21b68cc5f |
feat(webapp): dashboard parity for mollifier-buffered runs (#3757)
## Summary Dashboard surfaces handle buffered runs by falling back to the mollifier snapshot: - Run detail, span detail, streams view (`_app.../runs.\$runParam`, `resources.../spans.\$spanParam`, `resources.../streams.\$streamKey`). - Redirect routes (`@.runs.\$runParam`, `runs.\$runParam`, `projects.v3.\$projectRef.runs.\$runParam`). - Action routes — cancel / replay / idempotency-reset / debug — under `resources.taskruns/...` and `resources.../idempotencyKey.reset`. - Logs download. - Realtime subscription route + per-run resource (`realtime.v1.runs.\$runId`, `resources.../realtime.v1.*`). - `CancelRunDialog` gains an `onCancelSubmitted` callback so submit isn't raced by the Radix `DialogClose` wrapper. Stacked on the mutations PR. ## Test plan - [x] \`pnpm run typecheck --filter webapp\` passes - [x] \`pnpm run test --filter webapp test/mollifierRealtimeRunResource.test.ts\` passes - [x] \`pnpm run test --filter webapp test/mollifierRealtimeRunResourceBuffer.test.ts\` passes - [x] \`pnpm run test --filter webapp test/mollifierRealtimeSubscription.test.ts\` passes - [x] Manual smoke: trigger a buffered run, open it in the dashboard, replay/cancel from the UI --- ## Ship-gate follow-up fixes - **Auto-redirect to root span on direct nav** — loader sets `?span=` from root span (PG) or buffered snapshot spanId before 302'ing, so bookmark/share-link/direct-nav doesn't leave the panel collapsed. - **RunPresenter switches from `findFirstOrThrow` to `findFirst` + typed `RunNotInPgError`** — kills the per-poll `PrismaClient error` log spam for buffered runs without changing the route-loader's fallback flow. - **Span detail panel renders for buffered runs** — `SpanPresenter.call` now falls back to `findRunByIdWithMollifierFallback` + `buildSyntheticSpanRun` instead of returning undefined and triggering the "Event not found" toast loop. - **Logs download for buffered runs returns a gzipped placeholder line** — replaces the 404 with a content-encoded line explaining the run is queued. Same org-membership gate as the PG path. - **Admin Debug-Run button hidden for buffered runs + SpanRun circular type alias broken** (squashed) — buttons gate on a new `isBuffered` flag on the synthetic SpanRun. Required grounding SpanRun in `SpanPresenter.getRun` to break a circular type alias TS no longer tolerates once `isBuffered` is a literal field on the shape. - **Replay action requires user auth + org-membership** (🚩 Devin finding) — `action` was unauthenticated and the PG `findFirst` had no org filter, so any caller with a valid `runParam` could replay any run. Buffered fallback inherited the same gap. Fixed to mirror the cancel route. --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
Daniel Sutton
|
e1950778e2 |
feat(webapp): mollifier API mutations on buffered runs (#3756)
## Summary Cancel, replay, reschedule, metadata, tags, and idempotency-key-reset now succeed against a run that's still in the mollifier buffer. Mutations are applied to the buffered snapshot via Lua CAS; the drainer carries the mutation forward when it replays. Primitives added: - `mutateWithFallback` — PG-first / buffer-fallback resolver with bounded-wait safety net for entries that transition mid-mutation. - `applyMetadataMutation` — buffered metadata PUT mirroring the PG-side retry loop with CAS atomicity. - `resolveRunForMutation` — discriminated-union resolver used by route `findResource` so the route builder's pre-action 404 check sees buffered runs. Routes wired (whole files, no GET/POST splits): - `api.v2.runs.\$runParam.cancel.ts` - `api.v1.runs.\$runParam.replay.ts` - `api.v1.runs.\$runParam.reschedule.ts` - `api.v1.runs.\$runId.metadata.ts` - `api.v1.runs.\$runId.tags.ts` - `resetIdempotencyKey.server.ts` Stacked on the reads PR. ## Test plan - [x] \`pnpm run typecheck --filter webapp\` passes - [x] \`pnpm run test --filter webapp test/mollifierMutateWithFallback.test.ts\` passes - [x] \`pnpm run test --filter webapp test/mollifierApplyMetadataMutation.test.ts\` passes - [x] \`pnpm run test --filter webapp test/mollifierResolveRunForMutation.test.ts\` passes --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
Daniel Sutton
|
a1dc3c5ae1 |
feat(webapp): mollifier API GET read-fallback — synthetic primitives + route wiring (#3755)
## Summary Synthesise QUEUED/FAILED responses from the mollifier buffer when a TaskRun row hasn't landed in Postgres yet. Wires the synthesis into: - `ApiRetrieveRunPresenter` - v1 trace GET route - v1 spans GET route - attempts route gains a GET loader (fixes pre-existing Remix "no loader" 400) The `readFallback` infra itself lives on the trigger PR (consumed by `IdempotencyKeyConcern`); this PR adds the route-level synthetic-rendering primitives. Stacked on the replay PR. ## Test plan - [x] \`pnpm run typecheck --filter webapp\` passes - [x] \`pnpm run test --filter webapp test/mollifierSyntheticRedirectInfo.test.ts\` passes - [x] \`pnpm run test --filter webapp test/mollifierSyntheticSpanRun.test.ts\` passes --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
Daniel Sutton
|
4745754a7a |
feat(webapp,run-engine): mollifier drainer replay + stale sweep + cancelled-run engine API (#3754)
## Summary The replay side of the mollifier: - `DrainerHandler`: reads buffered snapshots and replays them through `engine.trigger` to materialise PG rows. - `RunEngine.createCancelledRun`: new public method the handler uses to write CANCELED rows directly from snapshots (bypass queue + waitpoint, emit `runCancelled`). Tolerates the cjson empty-table tags edge case found during validation. - Drainer fairness: org → env rotation so a heavy env doesn't starve light ones in the same org. - Stale-entry sweep + telemetry + alertable gauge so a stuck/offline drainer surfaces in alerts. Both the drainer and sweep default-off; nothing fires unless flagged on (`TRIGGER_MOLLIFIER_DRAINER_ENABLED`, `TRIGGER_MOLLIFIER_STALE_SWEEP_ENABLED`). Stacked on the trigger-time decisions PR. ## Test plan - [x] \`pnpm run typecheck --filter webapp\` passes - [x] \`pnpm run test --filter webapp test/mollifierDrainerHandler.test.ts\` passes - [x] \`pnpm run test --filter webapp test/mollifierStaleSweep.test.ts\` passes - [x] \`pnpm run test --filter @internal/run-engine src/engine/tests/createCancelledRun.test.ts\` passes - [x] \`pnpm run test --filter @trigger.dev/redis-worker packages/redis-worker/src/mollifier/drainer.test.ts\` passes --- ## Ship-gate follow-up fix **Drainer writes SYSTEM_FAILURE on max-attempts exhaustion.** Adds an `onTerminalFailure` callback on `MollifierDrainerOptions` so the customer's run lands a SYSTEM_FAILURE PG row even when the drainer exhausts `MAX_ATTEMPTS` on a retryable PG error (previously `buffer.fail()` was called with no row written → silent data loss). The callback runs before `buffer.fail()` on every terminal path (non-retryable AND max-attempts-exhausted), and re-throwing a retryable error from the callback causes the drainer to requeue rather than fail. Bumps `@trigger.dev/redis-worker` to a **minor** changeset (additive option + new exported types). Includes 5 unit tests covering both terminal causes plus the requeue-on-retryable-callback-failure path and no-callback back-compat. --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
Daniel Sutton
|
20a676d2ed |
fix(core,webapp): coerce numeric concurrencyKey to string (#3789)
## Summary `concurrencyKey` validation accepted only `z.string().optional()` on the single-trigger and V2/V3 batch endpoints, and the Phase-2 streaming NDJSON endpoint accepted `z.record(z.unknown()).optional()` for the entire `options` field. Callers passing `concurrencyKey: someNumericId` (e.g. `payload.userId`) either failed schema validation on the first two paths or sailed through on Phase-2 and then failed downstream at `prisma.taskRun.create` with `Argument concurrencyKey: Expected String or Null, provided Int`. The schema now accepts `string | number` for `concurrencyKey` and stringifies on the way in, across all three paths. The Phase-2 NDJSON `options` is tightened to reuse the strict `BatchTriggerTaskItem.options` shape so it validates identically to the V2/V3 batch endpoints. A defensive `typeof === "number"` coercion at the `engine.trigger` call site in `RunEngineTriggerTaskService` covers in-flight Redis-stored batch items enqueued before the schema fix — those items are rebuilt from a `Record<string, unknown>` shape that bypasses the new schema and would otherwise continue failing for up to their TTL. ## Test plan - [x] `packages/core/src/v3/schemas/batchItemNDJSON.test.ts` — unit tests covering numeric→string coercion, string passthrough, no-options, and rejection of non-string/non-number shapes across `TriggerTaskRequestBody`, `BatchTriggerTaskItem`, and `BatchItemNDJSON`. - [x] `apps/webapp/test/engine/triggerTask.test.ts` — `containerTest` simulating the in-flight Redis batch-item shape (numeric `concurrencyKey` via `Record<string, unknown>`), verifies the run is created with `concurrencyKey: "51262"`. Without the worker coercion, the test reproduces the production stack at `prisma.taskRun.create`. - [x] `pnpm run typecheck --filter webapp` clean. - [x] `pnpm run build --filter @trigger.dev/core --filter @trigger.dev/sdk --filter trigger.dev` clean. Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
Daniel Sutton
|
577f35eebe |
feat(webapp): mollifier trigger-time decisions — mollify, claim, read fallback (#3753)
## Summary The trigger hot path's mollifier integration: - `mollifyTrigger`: when the gate trips, write the engine.trigger snapshot to the buffer and return a synthesised QUEUED response. Postgres write is deferred to drainer-replay (next PR in the stack). - Pre-gate idempotency-key claim: same-key triggers serialise through Redis so a burst lands in PG / buffer exactly once. - Read-fallback extensions: `findRunByIdWithMollifierFallback` for the trigger-time idempotency lookup that must see buffered runs. - Gate bypasses: `debounce`, `oneTimeUseToken`, `parentTaskRunId`/`triggerAndWait` skip the mollify path entirely. - `triggerTask` + `IdempotencyKeyConcern` wired to the above. All behaviour gated by the master `TRIGGER_MOLLIFIER_ENABLED` switch; off-state hot path is unchanged (the gate is not even consulted). Stacked on the buffer extensions PR. ## Test plan - [x] \`pnpm run typecheck --filter webapp\` passes - [x] \`pnpm run test --filter webapp test/mollifierMollify.test.ts\` passes - [x] \`pnpm run test --filter webapp test/mollifierIdempotencyClaim.test.ts\` passes - [x] \`pnpm run test --filter webapp test/mollifierReadFallback.test.ts\` passes - [x] \`pnpm run test --filter webapp test/mollifierGate.test.ts\` passes - [x] \`pnpm run test --filter webapp test/engine/triggerTask.test.ts\` passes --- ## Ship-gate follow-up fixes - **Batch items bypass the mollifier gate** — fixes `BatchTaskRunItem_taskRunId_fkey` FK violation on batch triggers when the gate trips. End-state is a drainer-side `BatchTaskRunItem` create-on-materialise; batch traffic passes through the gate until that lands. - **IdempotencyKeyConcern honours buffered-run TTL on expiry** — buffered path now clears expired idempotency claims (read-side) and resets the buffer's `mollifier:idempotency:*` SETNX binding (write-side) so a re-trigger past the customer's TTL lands as a fresh run instead of echoing the stale buffered runId. --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
Daniel Sutton
|
3924aa4adb |
feat(redis-worker,webapp): mollifier buffer extensions + snapshot type (#3752)
## Summary Buffer-side data layer used by the rest of the mollifier phase-3 stack. - `buffer.ts` gains entry inspection (`getEntry`), idempotency lookup (`lookupIdempotency`), in-place snapshot mutation (`mutateSnapshot`), and dwell tracking. All atomic via Lua. - `mollifierSnapshot.server.ts`: shared `MollifierSnapshot` type plus (de)serialise helpers. - Drops the entry-TTL config and its env var. The drainer is the recovery mechanism; an entry that survives the drainer should surface as a stale-sweep alert, not silently TTL away. Adds methods to the buffer interface; nothing consumes them yet. Subsequent PRs in the stack wire trigger-time mollify, read-fallback, and mutation paths against this surface. ## Test plan - [x] \`pnpm run typecheck --filter webapp\` passes - [x] \`pnpm run test --filter @trigger.dev/redis-worker packages/redis-worker/src/mollifier/buffer.test.ts\` passes --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
Saadi Myftija
|
181d9ba541 |
feat: preview environment dispatch workflow (#3786)
Publishes a `repository_dispatch` `preview-deploy` event that enables preview deployments. Opt-in only currently. |
||
|
github-actions[bot]
|
8f066ac389 |
chore: release v4.5.0-rc.3 (#3763)
🚀 Publish Trigger.dev Docker / typecheck (push) Failing after 0s
🚀 Publish Trigger.dev Docker / publish-webapp (push) Has been skipped
🚀 Publish Trigger.dev Docker / publish-worker (push) Has been skipped
🚀 Publish Trigger.dev Docker / publish-worker-v4 (push) Has been skipped
🚀 Publish Trigger.dev Docker / units (push) Failing after 0s
🧭 Helm Chart Release / lint-and-test (push) Has been cancelled
🧭 Helm Chart Release / release (push) Has been cancelled
## Summary 1 improvement, 2 bug fixes. ## Improvements - Retry `TASK_MIDDLEWARE_ERROR` under the task's retry policy instead of failing the run on the first attempt. The error was already classified as retryable by `shouldRetryError`, but `shouldLookupRetrySettings` did not include it, so the retry flow fell through to `fail_run`. Fixes #3231. ([#3676](https://github.com/triggerdotdev/trigger.dev/pull/3676)) ## Bug fixes - Fix `TypeError` in `unflattenAttributes` when the input attribute map contains conflicting dotted key paths (e.g. both `a.b` set to a scalar and `a.b.c` set to a value). The path-walk loop now applies last-write-wins when a prior key wrote a primitive, null, or array at an intermediate slot, matching the existing precedent in `AttributeFlattener.addAttribute`. Callers no longer crash when handed malformed external attribute inputs. ([#3762](https://github.com/triggerdotdev/trigger.dev/pull/3762)) - Fix external trace context leaking across runs on warm-started workers with `processKeepAlive` enabled. Every subsequent run's attempt span was being exported with the first run's `traceId` and `parentSpanId`, breaking causal-chain navigation in external APM tools. Runs without an external trace context are unaffected. ([#3768](https://github.com/triggerdotdev/trigger.dev/pull/3768)) <details> <summary>Raw changeset output</summary> ⚠️⚠️⚠️⚠️⚠️⚠️ `main` is currently in **pre mode** so this branch has prereleases rather than normal releases. If you want to exit prereleases, run `changeset pre exit` on `main`. ⚠️⚠️⚠️⚠️⚠️⚠️ # Releases ## @trigger.dev/build@4.5.0-rc.3 ### Patch Changes - Updated dependencies: - `@trigger.dev/core@4.5.0-rc.3` ## trigger.dev@4.5.0-rc.3 ### Patch Changes - Updated dependencies: - `@trigger.dev/core@4.5.0-rc.3` - `@trigger.dev/build@4.5.0-rc.3` - `@trigger.dev/schema-to-json@4.5.0-rc.3` ## @trigger.dev/core@4.5.0-rc.3 ### Patch Changes - Retry `TASK_MIDDLEWARE_ERROR` under the task's retry policy instead of failing the run on the first attempt. The error was already classified as retryable by `shouldRetryError`, but `shouldLookupRetrySettings` did not include it, so the retry flow fell through to `fail_run`. Fixes #3231. ([#3676](https://github.com/triggerdotdev/trigger.dev/pull/3676)) - Fix `TypeError` in `unflattenAttributes` when the input attribute map contains conflicting dotted key paths (e.g. both `a.b` set to a scalar and `a.b.c` set to a value). The path-walk loop now applies last-write-wins when a prior key wrote a primitive, null, or array at an intermediate slot, matching the existing precedent in `AttributeFlattener.addAttribute`. Callers no longer crash when handed malformed external attribute inputs. ([#3762](https://github.com/triggerdotdev/trigger.dev/pull/3762)) - Fix external trace context leaking across runs on warm-started workers with `processKeepAlive` enabled. Every subsequent run's attempt span was being exported with the first run's `traceId` and `parentSpanId`, breaking causal-chain navigation in external APM tools. Runs without an external trace context are unaffected. ([#3768](https://github.com/triggerdotdev/trigger.dev/pull/3768)) ## @trigger.dev/plugins@4.5.0-rc.3 ### Patch Changes - Updated dependencies: - `@trigger.dev/core@4.5.0-rc.3` ## @trigger.dev/python@4.5.0-rc.3 ### Patch Changes - Updated dependencies: - `@trigger.dev/core@4.5.0-rc.3` - `@trigger.dev/build@4.5.0-rc.3` - `@trigger.dev/sdk@4.5.0-rc.3` ## @trigger.dev/react-hooks@4.5.0-rc.3 ### Patch Changes - Updated dependencies: - `@trigger.dev/core@4.5.0-rc.3` ## @trigger.dev/redis-worker@4.5.0-rc.3 ### Patch Changes - Updated dependencies: - `@trigger.dev/core@4.5.0-rc.3` ## @trigger.dev/rsc@4.5.0-rc.3 ### Patch Changes - Updated dependencies: - `@trigger.dev/core@4.5.0-rc.3` ## @trigger.dev/schema-to-json@4.5.0-rc.3 ### Patch Changes - Updated dependencies: - `@trigger.dev/core@4.5.0-rc.3` ## @trigger.dev/sdk@4.5.0-rc.3 ### Patch Changes - Updated dependencies: - `@trigger.dev/core@4.5.0-rc.3` </details> Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>helm-v4.5.0-rc.3 v4.5.0-rc.3 v.docker.4.5.0-rc.3 |
||
|
Eric Allam
|
9211032733 |
chore(database): drop unused TaskRun status composite index (#3743)
## Summary Drops the `TaskRun_status_runtimeEnvironmentId_createdAt_id_idx` index from the `TaskRun` table. After #3742 gated the legacy `WAITING_FOR_DEPLOY` drain to V1-engine workers only, this index sees zero scans on both writer and reader replicas. Removing it cuts index maintenance on every `TaskRun` INSERT/UPDATE. ## Why The index existed to support `WHERE status = X AND runtimeEnvironmentId = Y` queries from `ExecuteTasksWaitingForDeployService`, which is V1-only and no longer triggered on V2 deployments. A code grep across `apps/webapp` and `internal-packages/run-engine` confirmed no V2 production query uses this access pattern — every other `status:` filter on `TaskRun` is paired with `id`/`friendlyId`/`parentSpanId` and uses a different index. Dropping it also unlocks HOT updates on the dequeue path. The dequeue `UPDATE` modifies `status` (`QUEUED` -> `DEQUEUED`), and `status` is the leading column of this index — its presence blocked HOT eligibility for every `TaskRun` UPDATE. With the index gone, dequeue UPDATEs can become HOT, reducing WAL bytes and removing the B-tree page contention on this index's right-edge leaves. Uses `DROP INDEX CONCURRENTLY` to avoid blocking writes during the drop. ## Sequencing Should only ship once #3742 has soaked long enough to confirm the index is genuinely cold (24h+ of zero scans on `pg_stat_user_indexes`). |
||
|
Saadi Myftija
|
9cb6fd17c3 |
fix(webapp): idempotent DeploymentBackgroundWorker creation (#3772)
Make `POST /api/v1/deployments/:deploymentId/background-workers` idempotent so client-side retries no longer collide on the `BackgroundWorker` `(project, env, version)` unique index. Helps make deployments more resilient against the class of indexing failures that surfaces in the dashboard as "Indexing timed out", e.g. during transient database issues. |
||
|
nicktrn
|
c043c4a6ad |
fix(core): external trace context leaks across warm-started runs (#3768)
On warm-started workers with `processKeepAlive` enabled, every run's attempt span was exported with the first run's `traceId` and `parentSpanId`. `ExternalSpanExporterWrapper` and `ExternalLogRecordExporterWrapper` captured `externalTraceContext` at `TracingSDK` construction, and the SDK is memoized for the worker's lifetime - so the per-run reset of `StandardTraceContextManager.traceContext` never reached the wrappers. Reported by a customer running v4.4.x: 33 distinct runs on the same host/pid showed up in their APM as siblings of one parent span. Fix: the wrappers now read external context live from the trace context manager per export. Runs without an external trace context fall through to the unchanged `externalTraceId` fallback - no behaviour change for them. Regression test in `packages/core/test/externalSpanExporterWrapper.test.ts` asserts that reassigning the manager between exports produces correctly-parented spans. |
||
|
Matt Aitken
|
816986d44e |
fix(webapp): treat Phase 2 batch-stream retries as idempotent (TRI-9944) (#3766)
Returns sealed:true when a fast-completing batchTrigger races the stream finalisation, instead of throwing 422/BatchTriggerError. |
||
|
Iss
|
5083d161b2 | docs: adds relevant env vars to self hosting docs (#3148) | ||
|
Iss
|
8399aa2052 |
fix(core): retry TASK_MIDDLEWARE_ERROR under the task's retry policy (#3676)
Retries `TASK_MIDDLEWARE_ERROR` under the task's retry policy. `shouldRetryError` already classed it as retryable, but `shouldLookupRetrySettings` did not, so the run fell through to `fail_run` on attempt 1 instead of using the task's `retry` config. Fixes #3231. |
||
|
Iss
|
7324a3307f | chore(webapp): rename support tier to "Priority" on plan cards and li… (#3692) | ||
|
Iss
|
df96a937db | docs: troubleshoot "Stream is being deleted" during long waits (#3704) | ||
|
Matt Aitken
|
90bbbd1c44 |
fix(webapp): recover from ClickHouse JSON parse failures on out-of-range integers (#3759)
## Summary Second class of poisoned-row failure in the runs replication path. PR #3708 plugged lone UTF-16 surrogates; this one handles bare JSON integer literals outside ClickHouse's `Int64`..`UInt64` range. Recovery stays purely reactive — the existing `sanitizeRows` walker just gains an extra branch, so the hot replication path pays nothing on healthy rows. Fixes the still-firing customer-facing symptom from [TRI-9755](https://linear.app/triggerdotdev/issue/TRI-9755): `scan-social-profiles` runs continued to be stranded in `EXECUTING` on the Tasks page after #3708 deployed. CloudWatch showed `Dropped batch — ClickHouse JSON parse error but sanitizer found nothing to fix` firing **8/8 times** since the previous deploy (zero successful sanitizations). Root cause: upstream JS Number precision loss on a 21-digit Google Plus ID (`117039831458782873093` → `117039831458782870000`) — the precision-lossy value still serialises as a bare integer that exceeds `UInt64.MAX`, which ClickHouse rejects with `INCORRECT_DATA`. ## How the bug ships The customer task emits an output containing a Poshmark profile's `spec_format`: ```json {"key":"gp_id","proper_key":"Gp Id","value":117039831458782870000,"type":"int"} ``` That value is `1.17e20` — comfortably above `UInt64.MAX` (`1.84e19`) but comfortably below `1e21`. `Number.prototype.toString` only switches to exponential form at `|value| >= 1e21`, so `JSON.stringify` emits the bare token `117039831458782870000` and the ClickHouse `JSON(max_dynamic_paths)` column fails with: ``` Code: 117. DB::Exception: Cannot parse JSON object here: {…}: (while reading the value of key output): (at row 1) : While executing ParallelParsingBlockInputFormat. (INCORRECT_DATA) (version 25.12.x) ``` Same error verbatim as prod. The same number quoted (`"117039831458782870000"`) inserts fine — ClickHouse's dynamic JSON column accepts a `String` subtype on the same path. ## What changed `apps/webapp/app/v3/eventRepository/sanitizeRowsOnParseError.server.ts`: - New private `isUnsafeJsonInteger(value)` helper — true iff `value` is a finite integer-valued JS Number where `|value| < 1e21` (so `JSON.stringify` emits integer form, not exponent) **and** `value` falls outside `[Int64.MIN, UInt64.MAX]`. - `sanitizeUnknownInPlace` gains a number-branch: when the predicate holds, replace the Number with `String(value)`. The downstream JSON column dynamic-types the path as String for that row — fine, since the value was already precision-lossy upstream (no JS Number above 2^53 is numerically meaningful anyway). - Float-valued numbers, large floats (>= 1e21), NaN and Infinity are left alone — `JSON.stringify` emits them with exponents or as `null`, both of which ClickHouse accepts. `apps/webapp/test/sanitizeRowsOnParseError.test.ts`: four new unit tests + an extension to `sanitizeRows` covering surrogate + integer fixes counted together across rows. The unit suite now covers: - Positive value above `UInt64.MAX` (`117039831458782870000` — the actual prod value) - Negative value below `Int64.MIN` - Boundary values pass through (`42`, `Number.MAX_SAFE_INTEGER`, `2^63`) - Non-integer numbers untouched (floats, `1e25`, NaN, Infinity) - The actual `scan-social-profiles` nested shape — finds the offending `gp_id` deep inside `output.data.profiles[].spec_format[].platform_variables[].value` `.server-changes/runs-replication-bigint-recovery.md` — release notes entry. ## Why reactive, not pre-flight `#prepareJson` runs millions of times per day on the replication hot path. Walking every JSON tree to look for oversized integers would add bounded-but-real CPU on every healthy row. `sanitizeRows` only fires after a ClickHouse parse-error rejection, which is a few times a day platform-wide. Extending it costs effectively zero on healthy traffic and gains us recovery on the rare poisoned row. ## Verification - Reproduced 1:1 in a throwaway Docker `clickhouse/clickhouse-server:25.12.11.4` (closest available to the prod `25.12.1.1579` build). Pre-sanitize JSON fails with the exact prod error; post-sanitize JSON inserts cleanly and the row is readable with `gp_id` stored as a String subtype. - `pnpm --filter webapp exec vitest run test/sanitizeRowsOnParseError.test.ts` — 22/22 passing (18 existing + 4 new). - `pnpm run typecheck --filter webapp` — clean. ## Test plan - [x] `pnpm run typecheck --filter webapp` - [x] Unit tests pass against new + existing cases - [x] End-to-end Docker ClickHouse repro confirms recovery - [ ] Post-deploy: confirm `Sanitizing batch after ClickHouse JSON parse error` warns fire instead of `Dropped batch …` errors when `scan-social-profiles` outputs trip CH again - [ ] Post-deploy: confirm `permanentlyDroppedBatches` counter stops climbing in `/stp/trigger-app-prod/ecs/replication/service-container/process-logs` ## What this does NOT do - Doesn't backfill the ~120k+ existing stranded `EXECUTING` rows in production. Same as #3708 — that needs a reconciliation/backfill sweep (separate ticket — TRI-9755 fix #3). - Doesn't address the upstream root cause (the customer task emitting a JS-Number-precision-lossy big int). That's a customer-task concern; our replication path needs to be robust to whatever shape arrives. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com> |
||
|
Oskar Otwinowski
|
5ba1b32d53 | fix(core): handle conflicting dotted paths in unflattenAttributes (#3762) | ||
|
Eric Allam
|
37eeaa3690 |
fix(webapp): skip V1 WAITING_FOR_DEPLOY drain on V2 promotes (#3742)
## Summary Stops the legacy V1 `WAITING_FOR_DEPLOY` drain from running on every V2 deployment promotion. The drain queries `TaskRun` by `status='WAITING_FOR_DEPLOY'`, which only V1-engine runs ever have — V2 runs use `PENDING_VERSION` and are handled out of band. Calling the drain on V2 promotes produced empty queries against the status index and unnecessary reader-DB load. ## Fix Two layers: 1. Gate the enqueue at the call site in `ChangeCurrentDeploymentService` so it only fires when the deployment's worker is on engine V1. 2. Add a `LEGACY_RUN_ENGINE_WAITING_FOR_DEPLOY_DISABLED` env var (default `0`). When set to `1`, the service returns immediately from `call()` — neuters any jobs already sitting in the worker queue from before the deploy lands. V1 customers see no change; V2 promotes no longer trigger the drain. |
||
|
Matt Aitken
|
596a9bb90c |
fix(webapp): retain sessions-replication singleton import via globalThis assignment (#3738)
Linear: [TRI-9864](https://linear.app/triggerdotdev/issue/TRI-9864) (Urgent) Production incident: [TRI-9863](https://linear.app/triggerdotdev/issue/TRI-9863) (mitigated by image revert in cloud#910) ## Bug `apps/webapp/package.json` declares `"sideEffects": false`. PR #3333 (`71d98b4e`) replaced the previous real method-call retention idiom at the two `sessionsReplicationInstance` import sites with: ```ts import { sessionsReplicationInstance } from "..."; void sessionsReplicationInstance; ``` esbuild treats `void <identifier>;` as a pure expression statement under `sideEffects: false` and **tree-shakes the entire import** — including the `singleton(...)` call inside `sessionsReplicationInstance.server.ts` which is the only thing that fires `initializeSessionsReplicationInstance()`. The sessions→ClickHouse logical replication worker never starts, the slot is unconsumed, lag grows. ### How it manifested in production cloud#907's image bump rolled the `SessionReplicationService` ECS task on prod at 14:32 UTC. The new container's startup log emitted `🗃️ Runs replication service enabled` but **not** `🗃️ Sessions replication service enabled` or `🗃️ Sessions replication service started`. CloudWatch `OldestReplicationSlotLag` grew at ~220 MB/min and the `High replication lag` alarm fired at 14:37 UTC. Prod was reverted to the previous image (cloud#910) to stop the bleed. ### Verification `grep` of the built bundle `apps/webapp/build/index.js` (built from `c0365d36`): - **3** occurrences of `Runs replication` / `runsReplicationInstance` strings ✅ - **0** occurrences of `Sessions replication` / `sessionsReplicationInstance` / `SessionsReplicationService` ❌ The runs path survives tree-shaking because `adminWorker.server.ts` and `admin.api.v1.runs-replication.*` routes have real method calls (`.start()`, `.teardown()`, `.backfill()`) — observable uses the tree-shaker must preserve. The sessions singleton has no real callers, only the `void` no-ops, hence its complete elimination from the bundle. ## Fix Replace `void sessionsReplicationInstance;` with an assignment to `globalThis`, an unambiguous observable side effect the bundler cannot eliminate: ```ts (globalThis as Record<string, unknown>).__sessionsReplicationInstance = sessionsReplicationInstance; ``` Applied at both call sites: `apps/webapp/app/entry.server.tsx` and `apps/webapp/app/v3/services/adminWorker.server.ts`. Surrounding comments updated to document the bundler interaction so the next maintainer doesn't reintroduce `void`. ## Out of scope (follow-ups) - **Robustness improvement**: change `apps/webapp/package.json` from `"sideEffects": false` to an allowlist that includes `*Instance.server.ts` files. Prevents the same regression shape via any future `*Instance` singleton. - **Build-time check**: add a `grep` post-build step in `publish.yml` requiring `"Sessions replication"` to appear in `apps/webapp/build/index.js`. Catches this exact regression at CI time. ## Test plan - [x] `pnpm run typecheck --filter webapp` clean - [ ] After merge + publish: confirm new image's `SessionReplicationService` container logs `🗃️ Sessions replication service enabled` and `🗃️ Sessions replication service started` at startup - [ ] After re-deploying to prod: confirm `OldestReplicationSlotLag` stops growing and drains |
||
|
Matt Aitken
|
eefb96c87d |
fix(webapp): recover from ClickHouse JSON parse failures in runs replication (#3708)
## Summary On a ClickHouse `Cannot parse JSON object` rejection, `RunsReplicationService` now sanitizes lone UTF-16 surrogates across the failing batch via the existing `sanitizeRows` helper and retries once. If the sanitizer found nothing or the retry also fails, the batch is dropped loudly with a counter increment, so the surrounding `#insertWithRetry` layer doesn't spin three more times on a deterministic failure. Non-parse errors propagate unchanged. Mirrors the pattern from #3659 (for `ClickhouseEventRepository`) — same root cause (lone UTF-16 surrogates in user-provided JSON), same recovery shape, **reusing the same shared helpers** (`sanitizeRows`, `isClickHouseJsonParseError`, `parseRowNumberFromError`). Fixes the customer-facing symptom from [TRI-9755](https://linear.app/triggerdotdev/issue/TRI-9755): a single row's poisoned `output` JSON used to take down the `COMPLETED_SUCCESSFULLY` UPDATE events for its 50+ batch-mates, stranding them in `EXECUTING` in ClickHouse forever and inflating "Running" counts on the Tasks page. Confirmed in production this is ongoing — ~120k stale rows accumulated in a single 5-hour burst on 2026-05-18; smaller continuous leak before and after. ## What changed `apps/webapp/app/services/runsReplicationService.server.ts`: - Imports the three helpers from `~/v3/eventRepository/sanitizeRowsOnParseError.server` (no duplication; no move). - New private `#insertWithJsonParseRecovery<T>(rows, doInsert, contextLabel, attempt)` — generic over `TaskRunInsertArray[]` and `PayloadInsertArray[]`, structurally identical to `ClickhouseEventRepository.#insertWithJsonParseRecovery`. Try → on parse error sanitize the whole batch (the `at row N` hint is logged but not used to slice — semantics under `input_format_parallel_parsing` aren't stable) → retry once → drop with loud log if sanitizer found nothing OR retry still fails. - `#insertTaskRunInserts` and `#insertPayloadInserts` extract a `doInsert` closure and hand it to the wrapper. Existing error logging, span recording, and `recordSpanError` are preserved inside the closure. - New `private _permanentlyDroppedBatches = 0` counter with a public getter, for ops dashboards and tests (matches the events-repo convention). One shared counter for both insert sites — granularity comes from the `contextLabel` (`task_runs_v2` / `raw_task_runs_payload_v1`) on every log line. `.server-changes/runs-replication-utf16-recovery.md` — release notes entry. ## Why no new tests The shared helpers already have full unit + real-ClickHouse contract coverage from #3659 (`apps/webapp/test/sanitizeRowsOnParseError.test.ts`, `apps/webapp/test/otlpUtf16Sanitization.integration.test.ts`). The new wrapper is a line-for-line structural port. Adding a parallel integration test would require synthesizing bad data that *escapes* the preemptive `detectBadJsonStrings` check in `#prepareJson` but still trips ClickHouse — non-trivial without hand-crafted fixtures and wouldn't cover any new logic. ## What this does NOT do - Doesn't touch the ~120k existing stale `EXECUTING` rows in production. That needs a reconciliation/backfill sweep (separate ticket — TRI-9755 fix #3). - Doesn't sanitize the `error` column path (`runsReplicationService.server.ts:932 const errorData = { data: run.error };`). Reactive recovery will catch it if it ever poisons a batch, but feeding it through `#prepareJson` like `output` is a cheap follow-up. ## Test plan - [x] `pnpm run typecheck --filter webapp` — clean - [ ] Post-deploy: confirm `permanentlyDroppedBatches` counter stays at zero (or near-zero) in `/stp/trigger-app-prod/ecs/replication/service-container/process-logs`, and watch for `Sanitizing batch after ClickHouse JSON parse error` warns to confirm recovery is firing on real traffic - [ ] Post-deploy: confirm the rate of new "EXECUTING-but-actually-COMPLETED" zombies in ClickHouse flattens (current rate ≈ tens-to-hundreds per hour platform-wide) 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com> |
||
|
Eric Allam
|
9f64bf404b |
docs(ai-chat): slim-wire HITL continuations + field-level merge contract (#3721)
## Summary Updates the AI chat docs to match the slim-wire + field-level merge behavior shipped in #3719 and the precise `.in/append` cap + CORS-readable 413 shipped in #3720. No behavior changes here — code is correct in `main`; the docs were lagging on three patterns customers copy out of the page. ## What changed - **`hydrateMessages` examples upsert by id** (in `lifecycle-hooks.mdx`, `patterns/database-persistence.mdx`, and `patterns/persistence-and-replay.mdx`). The previous `stored.push(newMsg)` pattern duplicated the assistant id on HITL continuations and caused the LLM to receive a tool call with no `arguments`. The new examples include the rationale inline. - **`onValidateMessages` example filters to user messages** (`lifecycle-hooks.mdx`). The previous example called `validateUIMessages({ messages, tools })` directly, which now throws on HITL slim wires (the AI SDK schema requires `input` on resolved tool parts). New example shows the filter pattern, with a Warning callout explaining why. - **Merge contract description updated** (`lifecycle-hooks.mdx`). The old wording said incoming messages are "auto-merged" / "replaced"; the new description explains the actual field-level overlay (state advances only). - **Approval-responded wire example slimmed** (`client-protocol.mdx`). Shows the minimum shape the agent reads — `state` + `approval` (or `output` / `errorText` for HITL). Notes that the built-in transports ship this slim shape by default and that fuller shapes are still accepted. - **`/in/append` 413 row and FAQ updated** (`client-protocol.mdx`, `patterns/trusted-edge-signals.mdx`). Reflects the new precise S2 cap and the CORS-readable 413. - **New changelog entry** at the top of `changelog.mdx` covering all of the above. The historical `## 512 KiB ceiling removed` entry further down the changelog is left as-is (it's a snapshot of the prior transition), and the v4.5 upgrade-guide section is skipped — the merge contract is backwards compatible. ## Test plan - Mintlify dev preview renders cleanly with no broken anchors - Linked references resolve (`/ai-chat/lifecycle-hooks#hydratemessages`, `/ai-chat/lifecycle-hooks#onvalidatemessages`, `/ai-chat/patterns/database-persistence#alternative-hydratemessages`, `/ai-chat/client-protocol#step-3-send-messages-stops-and-actions`, `/ai-chat/patterns/large-payloads`) |
||
|
github-actions[bot]
|
d34014ddcc |
chore: release v4.5.0-rc.2 (#3702)
🚀 Publish Trigger.dev Docker / typecheck (push) Failing after 0s
🚀 Publish Trigger.dev Docker / publish-webapp (push) Has been skipped
🚀 Publish Trigger.dev Docker / publish-worker (push) Has been skipped
🚀 Publish Trigger.dev Docker / publish-worker-v4 (push) Has been skipped
🚀 Publish Trigger.dev Docker / units (push) Failing after 4s
## Summary
3 improvements, 1 bug fix.
## Improvements
- The per-turn merge now overlays the wire copy's tool-part state
advancement onto the agent's existing chain — `state` + the matching
resolution field (`output` / `errorText` / `approval`) come from the
wire, everything else (text, reasoning, tool `input`, provider metadata)
stays whatever the snapshot or `hydrateMessages` returned. Previously a
full-message replace overwrote those fields with whatever the client
shipped, so a slimmed wire copy landed a tool call with no `arguments`
on the next LLM call. Covers `output-available` / `output-error` (HITL
`addToolOutput`) and `approval-responded` / `output-denied` (approval
flow).
- `TriggerChatTransport.sendMessages` and `AgentChat.sendRaw` now slim
assistant messages that carry advanced tool parts. The wire payload is
just `{ id, role, parts: [<state + resolution field>] }` for
`submit-message` continuations; everything else passes through.
Reasoning blobs and full tool inputs no longer ride the wire on every
`addToolOutput` / `addToolApproveResponse`, so continuation payloads
stay well under the `.in/append` cap on long agent loops.
- Add `TriggerClient` for running multiple SDK clients side-by-side,
each with its own auth, preview branch, and baseURL. Useful when a
single process needs to trigger tasks or read runs across multiple
projects, environments, or preview branches without mutating shared
global state.
([#3683](https://github.com/triggerdotdev/trigger.dev/pull/3683))
## Bug fixes
- Fix `chat.agent` HITL continuations on reasoning-heavy turns. Two
changes that work together:
([#3719](https://github.com/triggerdotdev/trigger.dev/pull/3719))
<details>
<summary>Raw changeset output</summary>
⚠️⚠️⚠️⚠️⚠️⚠️
`main` is currently in **pre mode** so this branch has prereleases
rather than normal releases. If you want to exit prereleases, run
`changeset pre exit` on `main`.
⚠️⚠️⚠️⚠️⚠️⚠️
# Releases
## @trigger.dev/build@4.5.0-rc.2
### Patch Changes
- Updated dependencies:
- `@trigger.dev/core@4.5.0-rc.2`
## trigger.dev@4.5.0-rc.2
### Patch Changes
- Updated dependencies:
- `@trigger.dev/build@4.5.0-rc.2`
- `@trigger.dev/core@4.5.0-rc.2`
- `@trigger.dev/schema-to-json@4.5.0-rc.2`
## @trigger.dev/plugins@4.5.0-rc.2
### Patch Changes
- Updated dependencies:
- `@trigger.dev/core@4.5.0-rc.2`
## @trigger.dev/python@4.5.0-rc.2
### Patch Changes
- Updated dependencies:
- `@trigger.dev/sdk@4.5.0-rc.2`
- `@trigger.dev/build@4.5.0-rc.2`
- `@trigger.dev/core@4.5.0-rc.2`
## @trigger.dev/react-hooks@4.5.0-rc.2
### Patch Changes
- Updated dependencies:
- `@trigger.dev/core@4.5.0-rc.2`
## @trigger.dev/redis-worker@4.5.0-rc.2
### Patch Changes
- Updated dependencies:
- `@trigger.dev/core@4.5.0-rc.2`
## @trigger.dev/rsc@4.5.0-rc.2
### Patch Changes
- Updated dependencies:
- `@trigger.dev/core@4.5.0-rc.2`
## @trigger.dev/schema-to-json@4.5.0-rc.2
### Patch Changes
- Updated dependencies:
- `@trigger.dev/core@4.5.0-rc.2`
## @trigger.dev/sdk@4.5.0-rc.2
### Patch Changes
- Fix `chat.agent` HITL continuations on reasoning-heavy turns. Two
changes that work together:
([#3719](https://github.com/triggerdotdev/trigger.dev/pull/3719))
- The per-turn merge now overlays the wire copy's tool-part state
advancement onto the agent's existing chain — `state` + the matching
resolution field (`output` / `errorText` / `approval`) come from the
wire, everything else (text, reasoning, tool `input`, provider metadata)
stays whatever the snapshot or `hydrateMessages` returned. Previously a
full-message replace overwrote those fields with whatever the client
shipped, so a slimmed wire copy landed a tool call with no `arguments`
on the next LLM call. Covers `output-available` / `output-error` (HITL
`addToolOutput`) and `approval-responded` / `output-denied` (approval
flow).
- `TriggerChatTransport.sendMessages` and `AgentChat.sendRaw` now slim
assistant messages that carry advanced tool parts. The wire payload is
just `{ id, role, parts: [<state + resolution field>] }` for
`submit-message` continuations; everything else passes through.
Reasoning blobs and full tool inputs no longer ride the wire on every
`addToolOutput` / `addToolApproveResponse`, so continuation payloads
stay well under the `.in/append` cap on long agent loops.
Note: `onValidateMessages` receives the slim wire on HITL turns. If you
call `validateUIMessages` from `ai` against the full `messages` array it
will reject the slim assistant; filter to user messages (or skip on HITL
turns) — see the updated docstring on `onValidateMessages` for the
recommended pattern.
For `hydrateMessages` hooks that persist the chain, this release also
adds a small helper to the `@trigger.dev/sdk/ai` surface:
```ts
import { chat, upsertIncomingMessage } from "@trigger.dev/sdk/ai";
chat.agent({
hydrateMessages: async ({ chatId, trigger, incomingMessages }) => {
const record = await db.chat.findUnique({ where: { id: chatId } });
const stored = record?.messages ?? [];
if (upsertIncomingMessage(stored, { trigger, incomingMessages })) {
await db.chat.update({ where: { id: chatId }, data: { messages: stored }
});
}
return stored;
},
});
```
It pushes fresh user messages by id, no-ops on HITL continuations (the
incoming shares an id with the existing assistant — the runtime overlays
the new tool-state advance), and skips on non-`submit-message` triggers.
Returns `true` if it mutated `stored` so the caller knows whether to
persist.
Net effect: `chat.addToolOutput(...)` /
`chat.addToolApproveResponse(...)` on multi-step reasoning agents
(OpenAI Responses with `store: false`, Anthropic extended thinking,
etc.) no longer blows the cap and no longer corrupts the LLM input.
- Add `TriggerClient` for running multiple SDK clients side-by-side,
each with its own auth, preview branch, and baseURL. Useful when a
single process needs to trigger tasks or read runs across multiple
projects, environments, or preview branches without mutating shared
global state.
([#3683](https://github.com/triggerdotdev/trigger.dev/pull/3683))
```ts
import { TriggerClient } from "@trigger.dev/sdk";
const prod = new TriggerClient({ accessToken:
process.env.TRIGGER_PROD_KEY });
const preview = new TriggerClient({
accessToken: process.env.TRIGGER_PREVIEW_KEY,
previewBranch: "signup-flow",
});
await prod.tasks.trigger("send-email", payload);
await preview.runs.list({ status: ["COMPLETED"] });
```
- Updated dependencies:
- `@trigger.dev/core@4.5.0-rc.2`
## @trigger.dev/core@4.5.0-rc.2
</details>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
v.docker.4.5.0-rc.2
v4.5.0-rc.2
|
||
|
Eric Allam
|
75679c7518 |
fix(sdk): chat HITL continuations no longer break the next LLM call (#3719)
## Summary
Multi-step reasoning agents with HITL tools (OpenAI Responses with
`store: false`, Anthropic extended thinking, etc.) failed on
`chat.addToolOutput(...)` continuations — either the wire payload blew
the `.in/append` cap (reasoning blobs + tool inputs routinely > 512
KiB), or app-side slimming workarounds got overwritten server-side and
the next LLM call landed a tool call with no `arguments`. Both modes are
fixed.
## Design
The per-turn merge in `chat.agent` now overlays only the tool-part state
advances (`output-available` / `output-error` / `approval-responded` /
`output-denied`) from the wire copy onto the hydrated/snapshot chain.
Previously it replaced the entire message, which dropped `input`,
reasoning, and text from the LLM's view whenever the wire was slim.
In parallel, `TriggerChatTransport.sendMessages` and `AgentChat.sendRaw`
now slim the assistant message themselves on `submit-message`
continuations: ship `{ id, role, parts: [<resolved tool part only>] }`,
everything else reconstructed server-side from `hydrateMessages` or the
durable snapshot. Continuation payloads drop from 600 KiB – 1 MiB to ~1
KiB.
`references/ai-chat` `aiChatHydrated.hydrateMessages` now upserts by id
instead of pushing. With slim continuations, a blind push duplicates the
assistant id in the returned chain — the merge updates the first match,
the slim duplicate goes straight to `toModelMessages` with no `input`,
and the LLM 4xx's. This is the canonical pattern customers should mirror
in their own hydrate implementations.
## Test plan
- 11 new tests (slim helper unit + slim+merge integration for HITL,
approval, default no-hydrate branch)
- Full SDK suite: 239 tests pass across 19 files
- End-to-end sweep against `references/ai-chat`: 19 customer-side smoke
tests green; HITL wire bodies confirmed at ~1 KiB (was 600 KiB+); no
provider 4xx errors across OpenAI Responses or Anthropic
|
||
|
Eric Allam
|
c0365d36fb |
fix(webapp): precise S2 record cap + CORS 413 on session append (#3720)
## Summary
Two improvements to session `.in/append`:
- Oversize-body 413 responses now carry CORS headers, so browser fetches
see a readable status instead of an opaque `TypeError: Failed to fetch`.
App-side retry-on-disconnect loops no longer spin forever on a
permanently-rejected payload.
- The per-record cap is now computed precisely against S2's actual
ceiling instead of the conservative 512 KiB floor. Legitimate ~600-900
KiB tool outputs (search results, file content) now succeed;
pathological all-quote content that would double under JSON escape still
rejects cleanly.
## Design
S2 enforces a per-record metered size of `8 + 2*H + Σ(header name +
value) + body ≤ 1048576` bytes. With no record headers (our case), the
budget reduces to `body ≤ 1048568`. Verified empirically against cloud
S2 — append succeeds at metered=1048576 and 422s at 1048577 with `record
must have metered size less than 1 MiB`.
The old `MAX_APPEND_BODY_BYTES = 512 KiB` was derived by assuming
worst-case JSON escape doubling (every byte becomes `\"` or `\\`),
giving `(1 MiB - overhead) / 2`. Safe, but rejects ~half the legitimate
input space.
The new flow:
1. Pre-cap the HTTP body at 1 MiB (DoS guard against reading arbitrary
garbage before we can compute the wrap).
2. After reading, `S2RealtimeStreams.#appendPartByName` computes
`Buffer.byteLength(JSON.stringify({data: part, id: partId}), "utf8") +
8` and throws `S2RecordTooLargeError` (a `ServiceValidationError` with
status 413) if it would exceed S2's ceiling. The route's existing error
branch maps the throw to a 413 with a descriptive message.
The 413 CORS fix is a single-line change in `apiBuilder.server.ts` —
`wrapResponse` was being skipped on the body-too-large branch; every
other error branch wraps; the 413 was the exception.
## Test plan
- Empirically verified against cloud S2 with a boundary scan across
`[1048568, 1048569, ..., 1048576]` and across H ∈ {0, 1×5 hdr bytes,
1×14 hdr bytes} — the formula matches exactly
- Browser-side fetch on a 700 KiB POST now resolves with a readable
`status: 413` (no `TypeError: Failed to fetch`)
- A 900 KiB ASCII tool output now passes (would have 413'd at 512 KiB
pre-fix)
|
||
|
Matt Aitken
|
2fcc484a2b |
fix(webapp): route OrganizationDataStoresRegistry writes through the writer prisma (#3722)
## Bug
The `OrganizationDataStoresRegistry` singleton in
`apps/webapp/app/services/dataStores/organizationDataStoresRegistryInstance.server.ts`
was constructed with `$replica`. That client was then used by both the
polling read path *and* by `addDataStore` / `updateDataStore` /
`deleteDataStore` (and their backing `SecretStore.setSecret` upserts).
The write methods route through the read replica, which Postgres rejects
with **error code 25006**:
```
Invalid prisma.secretStore.upsert() invocation:
ConnectorError(ConnectorError { user_facing_error: None, kind: QueryError(PostgresError {
code: "25006",
message: "cannot execute INSERT in a read-only transaction",
...
}), transient: false })
```
User-visible symptom: the admin `/admin/data-stores` "Add data store"
form returns a 400 with this error wrapped, so no
`OrganizationDataStore` row can ever be created via the UI.
The read path (`loadFromDatabase` polling + `SecretStore.getSecret`) is
unaffected because `findMany` + secret read are read-only.
## Fix
Change the registry constructor to take both a writer and a replica:
```ts
constructor(writer: PrismaClient, replica: PrismaClient | PrismaReplicaClient)
```
- `loadFromDatabase()` keeps using `_replica` (and its
`SecretStore.getSecret` calls) — these are background cache-fillers, not
on user-latency-sensitive paths.
- `addDataStore` / `updateDataStore` / `deleteDataStore` (and their
`SecretStore.setSecret` / `deleteSecret` calls) now use `_writer`.
`organizationDataStoresRegistryInstance.server.ts` passes `(prisma,
$replica)` from `~/db.server`. Test sites that constructed with
`(prisma)` now pass `(prisma, prisma)` — the testcontainer exposes a
single client, so the writer/replica split collapses to one connection.
## Files
-
`apps/webapp/app/services/dataStores/organizationDataStoresRegistry.server.ts`
— constructor + read/write split
-
`apps/webapp/app/services/dataStores/organizationDataStoresRegistryInstance.server.ts`
— pass `prisma` alongside `$replica`
- `apps/webapp/test/organizationDataStoresRegistry.test.ts` — 14 call
sites bumped
- `apps/webapp/test/clickhouseFactory.test.ts` — 5 call sites bumped
## Test plan
- [x] Existing `organizationDataStoresRegistry.test.ts` +
`clickhouseFactory.test.ts` still pass (constructor sites updated;
behavior unchanged for tests).
- [ ] After deploy to test cloud, retry `/admin/data-stores` "Add data
store" form for the HIPAA org — should now succeed and the row should
appear.
- [ ] Verify the registry's polling reload picks up the new row within
`ORGANIZATION_DATA_STORES_RELOAD_INTERVAL_MS` (60s default) and the
factory starts routing to the org-scoped instance.
|
||
|
Eric Allam
|
61ca40b4b1 |
perf(run-engine,webapp): look up PENDING_VERSION runs via ClickHouse (#3707)
## Summary When a background worker registers, the engine resolves runs that were queued before the worker was ready (status `PENDING_VERSION`). That lookup used to scan a Postgres status index on `TaskRun`. Move it to ClickHouse: query candidate run ids from `task_runs_v2`, then refetch the actual rows from Postgres by primary key with a `status = 'PENDING_VERSION'` guard for idempotency. ## Design The lookup is a pluggable interface on the run engine (`PendingVersionRunIdLookup`). The webapp wires a ClickHouse-backed implementation through the org-scoped `clickhouseFactory` using a new `"engine"` client type, configured by `RUN_ENGINE_CLICKHOUSE_*` env vars. The URL falls back to `CLICKHOUSE_URL` when unset, so self-hosted deployments don't need new config to keep working. When the lookup returns no candidates, one bounded retry is scheduled ~5s later to cover ClickHouse replication lag against `task_runs_v2`. The Postgres status guard on both the candidate refetch and the inner `updateMany` prevents double-promotion when a retry races with a concurrent deploy. Tests cover three existing PENDING_VERSION cases via a small Postgres-backed test adapter; new ClickHouse-backed integration tests will follow. |
||
|
Eric Allam
|
0d4891a5f2 |
perf(database): drop unused TaskRun(scheduleId, createdAt) index (#3706)
## Summary Drops the unused composite Postgres index `TaskRun_scheduleId_createdAt_idx`. The schedule list view reads from ClickHouse, so this index served no Prisma query while still being maintained on every `TaskRun` INSERT/UPDATE. Removing it reduces write amplification on the primary database. Sibling to the prior drop of `TaskRun_scheduleId_idx` and the earlier removal of the `TaskRun.scheduleId` foreign key — all stemming from migrating schedule-aware reads to ClickHouse. ## Verification - Sampled `pg_stat_user_indexes` for `TaskRun` over multiple hours — zero scans against this index. - Grepped the codebase for any Prisma query filtering `TaskRun.scheduleId` — none found. All schedule-aware listing routes through `clickhouseRunsRepository`. |
||
|
Saadi Myftija
|
2cbfcfac3d |
chore(webapp): make HTTP keep-alive timeout configurable (#3705)
Make the Express server's `keepAliveTimeout` configurable via `HTTP_KEEPALIVE_TIMEOUT_MS`. Default preserved at 65000 ms — no behavior change if unset. |
||
|
nicktrn
|
ddad9700d7 |
fix(supervisor): compat shim for COMPUTE checkpoint type (#3703)
Workloads bundled with CLI versions before v4.4.4 use a strict zod enum for `checkpoint.type` that only allows DOCKER and KUBERNETES. When a customer's runs are routed via the compute path, those old runners receive `type: "COMPUTE"` on `/snapshots/since/...` and `/dequeue` responses and fail validation - blocking silent migration of existing deployments. The workload never reads the field - only validates the shape. Rewriting COMPUTE -> KUBERNETES on the way out lets older runners keep parsing while the database and internal services keep the real value. Limited to the two workload-facing endpoints whose response includes a checkpoint; `/continue`, `/attempts/start`, `/attempts/complete` all return shapes without one. Followup to #3114. |
||
|
Daniel Sutton
|
1015876b98 |
feat(webapp): user-based Sentry attribution with tenant tags (#3678)
## Summary
Stamp every Sentry event with the signed-in user and the tenant (org /
project / env) the request belongs to, so "Users Impacted" counts
distinct humans and events become filterable per tenant.
**Design after review (current):**
- `user.id = real user cuid` (from `requireUser`). "Users Impacted"
counts humans, not tenants.
- Tenant context (org / project / env slugs, IDs, env type) moves
entirely onto tags: `org_slug`, `project_slug`, `env_slug`, `org_id`,
`project_id`, `project_ref`, `environment_id`, `env_type`, plus
`impersonating` when set.
- Backed by an `AsyncLocalStorage` scope established at the HTTP entry.
Each entry point fills what it knows; loaders enrich the same scope with
what they already have.
**Zero new database queries.** The middleware does a regex match only.
Dashboard loaders that already query Prisma gain a couple of extra
selected columns; nothing new round-trips.
## How it's wired
- **Express middleware (`tenantContextResolver.server.ts`)** — parses
the URL with a regex and always opens an ALS scope. Populates whatever
subset of slugs is present: `/orgs/:o` → just `orgSlug`;
`/orgs/:o/projects/:p` adds `projectSlug`; the full triple adds
`envSlug`. Non-tenant paths get an empty scope so loaders can still
enrich.
- **`_app/route.tsx`** — already calls `requireUser`. Adds
`tenantContext.enrich({ userId: user.id })` for every authenticated
dashboard request. No new query.
- **Env layout loader (`_app.orgs.$o.projects.$p.env.$e/route.tsx`)** —
its existing `prisma.project.findFirst` gains two columns in `select`
(`externalRef`, `organization.id`). After it picks an env, calls
`tenantContext.enrich({ orgId, projectId, projectRef, envId, envType
})`. Same query, +2 columns.
- **API path (`apiBuilder.server.ts`)** — wraps every handler in
`tenantContext.run(tenantContextFromAuthEnvironment(authenticationResult.environment),
…)`. The mapper pulls `userId` from `env.orgMember?.userId` (already
selected by `authIncludeBase` — no schema change). Covers
`createLoaderApiRoute`, `createActionApiRoute`, and
`createMultiMethodApiRoute`.
- **Event processor (`sentryTenantContext.server.ts`)** — registered in
`entry.server.tsx` so it lives in the Remix bundle and shares the same
`tenantContext` ALS instance as the middleware and loaders. Stamps
whatever's present; nothing forced.
## Example events from local verification
| URL | `user.id` | Tags |
|-----|-----------|------|
| `/orgs/:o/projects/:p/env/:e/...` | real user cuid | `org_slug`,
`project_slug`, `env_slug`, `org_id`, `project_id`, `project_ref`,
`environment_id`, `env_type` |
| `/orgs/:o/settings` (non-env-scoped) | real user cuid | `org_slug`
only |
| API request with `orgMember` | `orgMember.userId` | full tenant set |
| API request without `orgMember` | (unset) | full tenant set |
## Trade-offs
1. On env-scoped pages, errors that fire before the env layout loader's
enrich callback runs get slugs + `user.id` but not the tenant IDs /
`env_type`. Realistic errors deep in async work get the full set. (Same
race as before, narrower window now that slugs/`user.id` are populated
up-front by the middleware and `_app` enrich.)
2. API requests where the environment has no `orgMember` get tenant tags
but no `user.id`. Those events still show in the issue but don't
contribute to "Users Impacted".
## Out of scope (deferred)
Background workers (`redis-worker`, `schedule-engine`) and socket
handlers. Those entry points don't set `tenantContext.run` yet — their
events ship without tenant attribution until each is wired in a
follow-up.
## Tests
31 unit tests across 4 files. New tests notably cover:
- `parseTenantPath`: org-only, org+project, and full-triple URL
variants.
- `tenantContext.enrich`: in-place patch, no-op outside `run()`,
concurrent-scope isolation, empty-scope + enrich pattern (for non-tenant
pages).
- `tenantContextFromAuthEnvironment`: with and without `orgMember` —
verifies the API path's `user.id` mapping.
- `addTenantContextToEvent`: empty scope, userId-only, slugs-only, full
enrichment, conditional tag emission, preservation of prior `event.user`
fields.
## Test plan
- [ ] `pnpm run typecheck --filter webapp`
- [ ] `pnpm run test --filter webapp -- test/tenantContext.test.ts
test/sentryTenantContext.test.ts test/tenantContextResolver.test.ts
test/tenantContextFromAuthEnvironment.test.ts`
- [ ] Local manual: with `SENTRY_DSN` set, hit a dashboard URL and an
API route, confirm the captured events carry `user.id` + the expected
tag set in Sentry.
- [ ] After ship: confirm "Users Impacted" on a real Sentry issue
reflects distinct users (not tenants).
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
|
||
|
Matt Aitken
|
71d98b4e6b |
Support for org-scoped ClickHouse (#3333)
Added `OrganizationDataStore` which allows orgs to have data stored in specific separate services. For now this is just used for ClickHouse. When using ClickHouse we get a client for the factory and pass in the org id. Particular care has to be made with two hot-insert paths: 1. RunReplicationService 2. OTLPExporter --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> Co-authored-by: Claude <noreply@anthropic.com> |
||
|
Eric Allam
|
832cf7220b |
feat(sdk,core): add TriggerClient for per-instance SDK configuration (#3683)
## Summary
`new TriggerClient({...})` exposes the management API (tasks, runs,
schedules, envvars, batch, queues, deployments, prompts, auth) as an
explicit instance with its own auth, preview branch, and baseURL.
Multiple clients can coexist in one process without mutating shared
global state — useful when a single service triggers across multiple
projects, environments, or preview branches.
```ts
import { TriggerClient } from "@trigger.dev/sdk";
const prod = new TriggerClient({ accessToken: process.env.TRIGGER_PROD_KEY });
const preview = new TriggerClient({
accessToken: process.env.TRIGGER_PREVIEW_KEY,
previewBranch: "signup-flow",
});
await prod.tasks.trigger("send-email", payload);
await preview.runs.list({ status: ["COMPLETED"] });
```
The existing global `configure()` API keeps working unchanged.
## Design
Instance methods enter an `AsyncLocalStorage`-backed scope (`sdkScope`)
before delegating to the existing module-level functions. The four
"pollution" points that previously read globals now consult the scope
first:
- `apiClientManager.{baseURL, accessToken, branchName}` and
`clientOrThrow` — identity fields are scope-only when scoped; `baseURL`
still falls back to `TRIGGER_API_URL` because plumbing (where the API
lives) is not identity.
- `taskContext.{ctx, worker, isWarmStart, isInsideTask}` — masked inside
an isolated scope so a `client.tasks.trigger(...)` from inside a task
doesn't leak the parent's `parentRunId` / `lockToVersion` / `isTest`
into a trigger that hits a different project.
- Inline `getEnvVar("TRIGGER_VERSION")` reads in `shared.ts` go through
a `scopedEnvVar` helper that returns `undefined` inside an isolated
scope.
The `TriggerClient` class itself is a thin wrapper that captures the
scope in its constructor and proxies each namespace method to enter that
scope before calling the existing impl. Generic inference (e.g.
`client.tasks.trigger<typeof t>(...)`) is preserved via `Pick<typeof ns,
keyof curatedSubset>` typings.
Two correctness fixes uncovered along the way are folded in:
- `apiClientManager.setGlobalAPIClientConfiguration` no longer silently
no-ops on the second call. `configure()` now actually overrides as users
expect (this is the root cause behind some "I changed the config but
nothing happened" reports).
- `apiClientManager.runWithConfig` (and therefore `auth.withAuth`) is
now backed by `sdkScope.withScope` instead of "mutate the global and
restore in finally". Two parallel `withAuth` calls with different
configs no longer stomp each other.
Surface curation: instance namespaces drop methods that don't make sense
per-instance — `batch.*AndWait` (runtime-dependent), `schedules.task` /
`schedules.timezones` (definition-time / stateless), `prompts.define`
(definition-time), `auth.configure` / `auth.withAuth` (global-only).
## Test plan
- [x] 9 runtime unit tests in `triggerClient.test.ts` cover: required
accessToken, instance auth + branch headers, no env fallback for
identity fields, no leakage between global and instance, four parallel
calls across two clients stay isolated, taskContext masking +
`inheritContext: true` override, `configure()` second-call override,
parallel `auth.withAuth` isolation.
- [x] 10 type-level assertions in `triggerClient.types.test.ts` using
`expectTypeOf` + `@ts-expect-error` lock in generic inference, return
type passthrough, overload preservation, and curated-surface drift.
- [x] Full SDK suite (219 tests) and core suite (530 tests) pass.
- [x] Webapp typecheck clean.
- [x] End-to-end smoke test against local webapp and a
freshly-provisioned cloud project — six concurrent multi-client triggers
all returned 200 with run IDs, headers per-client as expected.
- [ ] Reviewer: run `references/multi-client` per its `README.md` to
reproduce the smoke test locally.
## Try it
`references/multi-client` is a new reference workspace that exercises
this end-to-end:
- `src/trigger/echo.ts` — trivial target task
- `src/trigger/fanOut.ts` — opens two `TriggerClient`s from inside a
task, fires `echo` through each in parallel
- `src/external/main.ts` — external Node script with two clients
triggering `echo` sequentially and concurrently; logs every outgoing
request's `authorization` + `x-trigger-branch`
- `src/external/isolation.ts` — interleaves global `configure()` and an
instance call, asserts the captured fetch sequence shows no leakage
either way
|