webapp-dashboard-agent
7309 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Dan
|
ca434e546f |
feat(ai-assistant): introduce test task tools and enhance AIChat integration
- Added new tools for listing testable tasks, generating test payloads, and running test tasks, improving the AI assistant's capabilities for testing workflows. - Enhanced AIChatMessages and AIChatProvider to support the new test task functionalities, including handling generated payloads and navigating to test runs. - Updated tool schemas to include new test-related operations, ensuring a seamless user experience when interacting with testing features. - Improved the AI assistant's guidelines to reflect the new testing capabilities, providing clearer instructions for users. These changes aim to enrich the testing experience within the AI assistant, making it easier for users to manage and execute test tasks effectively. |
||
|
Dan
|
43ad5fcd2f |
feat(ai-assistant): enhance navigation and span detail tools
- Updated the navigateToPage tool to support deep-linking to specific runs and spans, improving user navigation capabilities. - Introduced a new getSpanDetails tool for retrieving detailed information about individual spans within a run, aiding in error investigation. - Enhanced AI assistant guidelines to reflect new navigation and inspection capabilities, ensuring users can effectively query run and span details. - Added spanId and runId parameters to relevant schemas and functions for better context handling. These changes aim to provide a more robust and informative experience for users interacting with the AI assistant, particularly in navigating and diagnosing run-related issues. |
||
|
Dan
|
13e63db468 | feat(ai-assistant): add ai-sparkle-hover animation and enhance AskAI and AIChatHeader components | ||
|
Dan
|
dfcadcaaf5 |
feat(ai-assistant): enhance AIChatMessages and introduce new tool result components
- Updated AIChatMessages to support structured output rendering for various tool results, improving user interaction. - Added ToolOutput component to handle different tool outputs, including FailureSummaryCard, FilterChips, and MiniTable for better data presentation. - Enhanced AIChatPanel to pass onSendMessage prop for improved message handling. - Introduced AIChatToolResults for better organization of tool result components. These changes aim to provide a richer and more informative user experience when interacting with the AI assistant. |
||
|
Dan
|
76360ea261 | feat(ai-assistant): enhance multi-step request handling and tool loading | ||
|
Dan
|
d6fa0d64ff |
feat(ai-assistant): enhance chat message handling, history and tool ux
- Updated AIChatMessages component to improve auto-navigation logic, ensuring navigation only occurs during active streaming. - Introduced ToolResultCard for displaying tool outputs with collapsible details, enhancing user interaction. - Added tool labels for better UI feedback on tool actions. - Refined AIChatPanel to manage chat state more effectively, including resuming previous chats. - Improved AIChatProvider to handle chat switching and state management with better reliability. These changes aim to provide a more intuitive and responsive experience for users interacting with the AI assistant. |
||
|
Dan
|
f63b7c67a3 | feat(ai-assistant): implement auto-navigation for dynamic tools in chat messages | ||
|
Dan
|
830b830154 | feat(ai-assistant): refine session management and error handling | ||
|
Dan
|
0772ab4106 | feat(ai-assistant): enhance analytics tools with ClickHouse integration | ||
|
Dan
|
0e204a2a9a | feat(ai-assistant): implement run tools | ||
|
Dan
|
e209bac792 | feat(ai-assistant): implement error classification and retrieval tools | ||
|
Dan
|
669d396f7c |
feat(ai-assistant): add scaffolding for expanding AI assistant capabilities with V1B tools and enhancements
- Introduced new tools for managing runs, errors, and analytics, including: - `listRuns`, `getRunDetails`, `getRunLogs`, `getRunGraph`, `applyRunFilters`, `queryRuns` - `listErrors`, `getErrorDetails`, `findSimilarErrors`, `classifyFailure` - `summarizeCurrentView`, `aggregateRuns`, `correlateRunsWithDeploy` - Updated AIChatToolCall component with new tool labels for improved user feedback. - Enhanced the AI assistant's system prompt to include capabilities for inspecting runs and errors. - Added new utility functions for handling ClickHouse queries and summarizing run data. These changes aim to provide a more comprehensive and user-friendly experience for users interacting with the AI assistant, enabling better insights into runs and error management. |
||
|
Dan
|
36628d6bd3 |
feat(ai-assistant): enhance chat UI with improved input handling and message display
- Updated AskAI component to hide when the chat drawer is open. - Modified AIChatContextBanner for better styling. - Refactored AIChatHeader to include scroll handling and improved chat history display. - Enhanced AIChatInput to support auto-growing text area and focus management. - Improved AIChatMessages to handle scrolling and display messages more effectively. - Added error handling and retry functionality in AIChatPanel. - Introduced new suggested prompts with animation effects. These changes aim to provide a more seamless and user-friendly chat experience in the AI assistant interface. |
||
|
Dan
|
f35a328e8a |
feat(ai-assistant): add chat panel UI and wire into dashboard nav
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
Dan
|
3e42603e03 |
feat(ai-assistant): dashboard agent task, tools, and session routes
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
Dan
|
94af0df33e |
feat(ai-assistant): add AiChat/AiChatSession persistence tables
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
Matt Aitken
|
b23ec26d16 |
chore: vouch ConProgramming (#3804)
Vouches GitHub user [`ConProgramming`](https://github.com/ConProgramming) (Conner) as an outside contributor by adding them to `.github/VOUCHED.td`, so their PRs aren't auto-closed by the vouch check. Done as a direct edit rather than via the issue flow because they have no open Vouch Request issue to comment `vouch` on. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
Dan
|
cd252801eb |
feat: dashboard agent - package upgrades (#3793)
1. in webapp folder update ai-sdk to 6.x.x 2. update vitest to 4.xx |
||
|
Eric Allam
|
4c4ed22e82 |
docs(ai-chat): document the chat.agent tools option (#3791)
## Summary Documents the new `tools` option on `chat.agent` (companion to #3790). Adds a dedicated [Tools](/ai-chat/tools) guide: the three places tools show up (config, `toStreamTextOptions`, `streamText`), why declaring them on the config matters for `toModelOutput` across turns, static vs per-turn tools, the typed `run()` payload, `InferChatUIMessageFromTools`, the relationship to skills, and the manual `convertToModelMessages` path for `customAgent` loops. Threads the option through the rest of the guide: the reference tables, a happy-path section on the backend page, the types page, and the HITL / skills / tool-result-auditing patterns. Corrects the sub-agents guide, where the `toModelOutput` compression was implied to work across turns but silently degraded from turn 2 without config tools. Also unstacks the three callouts that were piled under the `chat.agent()` header on the backend page, and adds a changelog entry. |
||
|
github-actions[bot]
|
e0681d2394 |
chore: release v4.5.0-rc.4 (#3788)
🚀 Publish Trigger.dev Docker / units (push) Failing after 3s
🚀 Publish Trigger.dev Docker / typecheck (push) Failing after 19s
🚀 Publish Trigger.dev Docker / publish-webapp (push) Has been skipped
🚀 Publish Trigger.dev Docker / publish-worker (push) Has been skipped
🚀 Publish Trigger.dev Docker / publish-worker-v4 (push) Has been skipped
🧭 Helm Chart Release / lint-and-test (push) Has been cancelled
🧭 Helm Chart Release / release (push) Has been cancelled
## Summary 1 new feature, 5 improvements. ## Highlights - Mollifier `mutateSnapshot` now enforces a tag cap: an `append_tags` patch carrying `maxTags` returns `"limit_exceeded"` (writing nothing) when the deduped tag count would exceed the limit, so a buffered run can't accumulate more tags via the tags API than the trigger validator allows at creation. ([#3756](https://github.com/triggerdotdev/trigger.dev/pull/3756)) ## Improvements - Mollifier buffer extensions: idempotency dedup, an atomic `mutateSnapshot` API, metadata CAS, claim primitives, and a `MollifierSnapshot` type. The buffer's Redis client now reconnects with jittered backoff so a fleet of clients doesn't stampede Redis in lockstep after a blip. ([#3752](https://github.com/triggerdotdev/trigger.dev/pull/3752)) - Add `onTerminalFailure` callback to `MollifierDrainerOptions` so the customer's run lands a SYSTEM_FAILURE PG row even when the drainer exhausts `maxAttempts` on a retryable PG error. Previously, retryable-error exhaustion called `buffer.fail()` directly, which atomically marks FAILED + DELs the entry hash with no PG write — silent data loss when PG was unreachable across the full retry budget. The callback fires before `buffer.fail()` on any terminal path (`cause: "non-retryable"` or `"max-attempts-exhausted"`); throwing a retryable error from the callback causes the drainer to requeue rather than fail. ([#3754](https://github.com/triggerdotdev/trigger.dev/pull/3754)) - Bump `@s2-dev/streamstore` to `0.22.10` to fix a `TASK_RUN_UNCAUGHT_EXCEPTION` ("Invalid state: Unable to enqueue") when a `chat.agent` turn is aborted mid-stream. ([#3792](https://github.com/triggerdotdev/trigger.dev/pull/3792)) - Coerce numeric `concurrencyKey` values to string at the API boundary across `tasks.trigger`, `tasks.batchTrigger`, and the Phase-2 streaming batch endpoint. ([#3789](https://github.com/triggerdotdev/trigger.dev/pull/3789)) - Add a `tools` option to `chat.agent`. Declaring your tools here threads them into the SDK's internal `convertToModelMessages`, so each tool's `toModelOutput` is re-applied when prior-turn history is re-converted. ([#3790](https://github.com/triggerdotdev/trigger.dev/pull/3790)) <details> <summary>Raw changeset output</summary> ⚠️⚠️⚠️⚠️⚠️⚠️ `main` is currently in **pre mode** so this branch has prereleases rather than normal releases. If you want to exit prereleases, run `changeset pre exit` on `main`. ⚠️⚠️⚠️⚠️⚠️⚠️ # Releases ## @trigger.dev/redis-worker@4.5.0-rc.4 ### Minor Changes - Mollifier buffer extensions: idempotency dedup, an atomic `mutateSnapshot` API, metadata CAS, claim primitives, and a `MollifierSnapshot` type. The buffer's Redis client now reconnects with jittered backoff so a fleet of clients doesn't stampede Redis in lockstep after a blip. ([#3752](https://github.com/triggerdotdev/trigger.dev/pull/3752)) - Add `onTerminalFailure` callback to `MollifierDrainerOptions` so the customer's run lands a SYSTEM_FAILURE PG row even when the drainer exhausts `maxAttempts` on a retryable PG error. Previously, retryable-error exhaustion called `buffer.fail()` directly, which atomically marks FAILED + DELs the entry hash with no PG write — silent data loss when PG was unreachable across the full retry budget. The callback fires before `buffer.fail()` on any terminal path (`cause: "non-retryable"` or `"max-attempts-exhausted"`); throwing a retryable error from the callback causes the drainer to requeue rather than fail. ([#3754](https://github.com/triggerdotdev/trigger.dev/pull/3754)) ### Patch Changes - Pipeline the per-entry `HGETALL` fetches in `MollifierBuffer.listEntriesForEnv`. The previous serial implementation issued one Redis round-trip per runId returned by `LRANGE`, which dominated stale-sweep wall-time at any meaningful backlog (at the sweep's default maxCount=1000, this is ~1000 RTTs per env per pass). Behaviour is unchanged — entries are still skipped when the entry hash has been torn down by a concurrent drainer ack/fail between the LRANGE and the HGETALL. ([#3752](https://github.com/triggerdotdev/trigger.dev/pull/3752)) - Mollifier `mutateSnapshot` now enforces a tag cap: an `append_tags` patch carrying `maxTags` returns `"limit_exceeded"` (writing nothing) when the deduped tag count would exceed the limit, so a buffered run can't accumulate more tags via the tags API than the trigger validator allows at creation. ([#3756](https://github.com/triggerdotdev/trigger.dev/pull/3756)) - Updated dependencies: - `@trigger.dev/core@4.5.0-rc.4` ## @trigger.dev/build@4.5.0-rc.4 ### Patch Changes - Updated dependencies: - `@trigger.dev/core@4.5.0-rc.4` ## trigger.dev@4.5.0-rc.4 ### Patch Changes - Bump `@s2-dev/streamstore` to `0.22.10` to fix a `TASK_RUN_UNCAUGHT_EXCEPTION` ("Invalid state: Unable to enqueue") when a `chat.agent` turn is aborted mid-stream. ([#3792](https://github.com/triggerdotdev/trigger.dev/pull/3792)) - Updated dependencies: - `@trigger.dev/core@4.5.0-rc.4` - `@trigger.dev/build@4.5.0-rc.4` - `@trigger.dev/schema-to-json@4.5.0-rc.4` ## @trigger.dev/core@4.5.0-rc.4 ### Patch Changes - Coerce numeric `concurrencyKey` values to string at the API boundary across `tasks.trigger`, `tasks.batchTrigger`, and the Phase-2 streaming batch endpoint. ([#3789](https://github.com/triggerdotdev/trigger.dev/pull/3789)) - Bump `@s2-dev/streamstore` to `0.22.10` to fix a `TASK_RUN_UNCAUGHT_EXCEPTION` ("Invalid state: Unable to enqueue") when a `chat.agent` turn is aborted mid-stream. ([#3792](https://github.com/triggerdotdev/trigger.dev/pull/3792)) ## @trigger.dev/plugins@4.5.0-rc.4 ### Patch Changes - Updated dependencies: - `@trigger.dev/core@4.5.0-rc.4` ## @trigger.dev/python@4.5.0-rc.4 ### Patch Changes - Updated dependencies: - `@trigger.dev/sdk@4.5.0-rc.4` - `@trigger.dev/core@4.5.0-rc.4` - `@trigger.dev/build@4.5.0-rc.4` ## @trigger.dev/react-hooks@4.5.0-rc.4 ### Patch Changes - Updated dependencies: - `@trigger.dev/core@4.5.0-rc.4` ## @trigger.dev/rsc@4.5.0-rc.4 ### Patch Changes - Updated dependencies: - `@trigger.dev/core@4.5.0-rc.4` ## @trigger.dev/schema-to-json@4.5.0-rc.4 ### Patch Changes - Updated dependencies: - `@trigger.dev/core@4.5.0-rc.4` ## @trigger.dev/sdk@4.5.0-rc.4 ### Patch Changes - Add a `tools` option to `chat.agent`. Declaring your tools here threads them into the SDK's internal `convertToModelMessages`, so each tool's `toModelOutput` is re-applied when prior-turn history is re-converted. ([#3790](https://github.com/triggerdotdev/trigger.dev/pull/3790)) ```ts chat.agent({ tools: { readFile, search }, run: async ({ messages, tools, signal }) => streamText({ model, messages, tools, abortSignal: signal }), }); ``` Also exports `InferChatUIMessageFromTools<typeof tools>` to derive the chat `UIMessage` type (typed tool parts) directly from a tool set. - Updated dependencies: - `@trigger.dev/core@4.5.0-rc.4` </details> Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>helm-v4.5.0-rc.4 v.docker.4.5.0-rc.4 v4.5.0-rc.4 |
||
|
nicktrn
|
e35f5742d4 |
fix(webapp): upgrade posthog-node to v5, drop axios + stale override (#3801)
Follow-up to #3796, which bumped the slack-client axios paths but left posthog-node's transitive `axios@1.15.1` in place. `posthog-node` 4.17.1 → 5.35.6. v5 drops the axios dependency entirely (it's now fetch-based via `@posthog/core`), so posthog's old axios path disappears. With #3796 already on main (webapp + d3 references on `@slack/web-api@7.16.0`), nothing else pins the old line, so the now-dead `axios@>=1.0.0 <1.15.0` override is removed and axios resolves to a single patched `1.16.1` repo-wide. This closes the remaining axios advisories. Compat: the webapp's usage in `telemetry.server.ts` (`new PostHog(key, { host })`, `.identify`, `.groupIdentify`, `.capture`) is all object-form API that v5 preserves; `pnpm run typecheck --filter webapp` passes. Node: posthog-node v5 requires Node `^20.20.0 || >=22.22.0`. We run 20.20.0 in dev (`.nvmrc`), CI, and the published Docker image (`node:20.20-bullseye-slim`), so we're compliant. |
||
|
nicktrn
|
952139d492 |
fix(webapp): bump @slack/web-api to 7.16.0 for patched axios (#3796)
Bumps `@slack/web-api` 7.9.1 → 7.16.0 in the webapp and the two `references` examples (d3-chat, d3-openai-agents). 7.16.0 depends on `axios@^1.16.0`, so every slack-client axios path resolves to 1.16.1 instead of 1.15.1. This clears the slack and references axios paths. `posthog-node`'s transitive axios still resolves the older line - that's handled in a follow-up that upgrades posthog-node to v5 (which drops the axios dependency entirely and lets us retire the now-stale axios override). The dependabot axios advisories fully close once both land. |
||
|
Eric Allam
|
2dd9f37b40 |
fix(core,cli): stop chat.agent uncaught exception on mid-stream abort (#3792)
## Summary A `chat.agent` turn that gets aborted mid-stream (stop generation, idle suspend, cancellation) could surface a `TASK_RUN_UNCAUGHT_EXCEPTION` — `TypeError: Invalid state: Unable to enqueue` — in the dashboard. The run kept working, but the exceptions were loud and confusing. ## Root cause The realtime stream writer batches chunks through `@s2-dev/streamstore`'s `BatchTransform`, which holds them for a short linger window before flushing. When the turn's abort signal fires while a record is still buffered, the stream's writable is aborted and the transform's readable controller is closed — but the pending linger `setTimeout` still fires and calls `controller.enqueue()` on the dead controller, throwing from a timer callback where nothing can catch it. Fixed upstream in `@s2-dev/streamstore@0.22.10`, which wraps the linger flush in a try/catch that discards the closed-controller error. This bumps the dependency across core, the CLI, and the webapp, and adds a regression test against the real `BatchTransform`. Verified end-to-end: a mid-stream stop that previously failed the run with `TASK_RUN_UNCAUGHT_EXCEPTION` now leaves the run healthy. |
||
|
Eric Allam
|
e9e2ec1cfc |
fix(sdk): re-apply tool toModelOutput across chat.agent turns (#3790)
## Summary
`chat.agent` now takes a `tools` option. Until now tools only went to
`streamText` inside `run()`, so the SDK had no tools when it
re-converted the persisted `UIMessage` history at the start of each
turn. Any tool with a `toModelOutput` (raw image bytes into an image
content part, or a sub-agent transcript compressed to a summary) had its
transform applied on turn 1 and skipped from turn 2 onward, so the raw
output got JSON-stringified back into the prompt and the model lost the
transformed view.
Declaring `tools` on the config threads them into that conversion, so
`toModelOutput` runs on every turn. The resolved set is handed back,
typed, on the `run()` payload as `tools`:
```ts
const tools = { searchDocs, renderChart };
export const myChat = chat.agent({
tools,
run: async ({ messages, tools, signal }) =>
streamText({ ...chat.toStreamTextOptions({ tools }), messages, abortSignal: signal }),
});
```
`tools` also accepts a per-turn function for tools that depend on the
user or a feature flag. Only `inputSchema` and `toModelOutput` are read
during conversion, never `execute`. Also exports
`InferChatUIMessageFromTools<typeof tools>` to derive the chat
`UIMessage` type from a tool set. No behavior change for agents that
don't declare `tools`.
|
||
|
Daniel Sutton
|
e21b68cc5f |
feat(webapp): dashboard parity for mollifier-buffered runs (#3757)
## Summary Dashboard surfaces handle buffered runs by falling back to the mollifier snapshot: - Run detail, span detail, streams view (`_app.../runs.\$runParam`, `resources.../spans.\$spanParam`, `resources.../streams.\$streamKey`). - Redirect routes (`@.runs.\$runParam`, `runs.\$runParam`, `projects.v3.\$projectRef.runs.\$runParam`). - Action routes — cancel / replay / idempotency-reset / debug — under `resources.taskruns/...` and `resources.../idempotencyKey.reset`. - Logs download. - Realtime subscription route + per-run resource (`realtime.v1.runs.\$runId`, `resources.../realtime.v1.*`). - `CancelRunDialog` gains an `onCancelSubmitted` callback so submit isn't raced by the Radix `DialogClose` wrapper. Stacked on the mutations PR. ## Test plan - [x] \`pnpm run typecheck --filter webapp\` passes - [x] \`pnpm run test --filter webapp test/mollifierRealtimeRunResource.test.ts\` passes - [x] \`pnpm run test --filter webapp test/mollifierRealtimeRunResourceBuffer.test.ts\` passes - [x] \`pnpm run test --filter webapp test/mollifierRealtimeSubscription.test.ts\` passes - [x] Manual smoke: trigger a buffered run, open it in the dashboard, replay/cancel from the UI --- ## Ship-gate follow-up fixes - **Auto-redirect to root span on direct nav** — loader sets `?span=` from root span (PG) or buffered snapshot spanId before 302'ing, so bookmark/share-link/direct-nav doesn't leave the panel collapsed. - **RunPresenter switches from `findFirstOrThrow` to `findFirst` + typed `RunNotInPgError`** — kills the per-poll `PrismaClient error` log spam for buffered runs without changing the route-loader's fallback flow. - **Span detail panel renders for buffered runs** — `SpanPresenter.call` now falls back to `findRunByIdWithMollifierFallback` + `buildSyntheticSpanRun` instead of returning undefined and triggering the "Event not found" toast loop. - **Logs download for buffered runs returns a gzipped placeholder line** — replaces the 404 with a content-encoded line explaining the run is queued. Same org-membership gate as the PG path. - **Admin Debug-Run button hidden for buffered runs + SpanRun circular type alias broken** (squashed) — buttons gate on a new `isBuffered` flag on the synthetic SpanRun. Required grounding SpanRun in `SpanPresenter.getRun` to break a circular type alias TS no longer tolerates once `isBuffered` is a literal field on the shape. - **Replay action requires user auth + org-membership** (🚩 Devin finding) — `action` was unauthenticated and the PG `findFirst` had no org filter, so any caller with a valid `runParam` could replay any run. Buffered fallback inherited the same gap. Fixed to mirror the cancel route. --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
Daniel Sutton
|
e1950778e2 |
feat(webapp): mollifier API mutations on buffered runs (#3756)
## Summary Cancel, replay, reschedule, metadata, tags, and idempotency-key-reset now succeed against a run that's still in the mollifier buffer. Mutations are applied to the buffered snapshot via Lua CAS; the drainer carries the mutation forward when it replays. Primitives added: - `mutateWithFallback` — PG-first / buffer-fallback resolver with bounded-wait safety net for entries that transition mid-mutation. - `applyMetadataMutation` — buffered metadata PUT mirroring the PG-side retry loop with CAS atomicity. - `resolveRunForMutation` — discriminated-union resolver used by route `findResource` so the route builder's pre-action 404 check sees buffered runs. Routes wired (whole files, no GET/POST splits): - `api.v2.runs.\$runParam.cancel.ts` - `api.v1.runs.\$runParam.replay.ts` - `api.v1.runs.\$runParam.reschedule.ts` - `api.v1.runs.\$runId.metadata.ts` - `api.v1.runs.\$runId.tags.ts` - `resetIdempotencyKey.server.ts` Stacked on the reads PR. ## Test plan - [x] \`pnpm run typecheck --filter webapp\` passes - [x] \`pnpm run test --filter webapp test/mollifierMutateWithFallback.test.ts\` passes - [x] \`pnpm run test --filter webapp test/mollifierApplyMetadataMutation.test.ts\` passes - [x] \`pnpm run test --filter webapp test/mollifierResolveRunForMutation.test.ts\` passes --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
Daniel Sutton
|
a1dc3c5ae1 |
feat(webapp): mollifier API GET read-fallback — synthetic primitives + route wiring (#3755)
## Summary Synthesise QUEUED/FAILED responses from the mollifier buffer when a TaskRun row hasn't landed in Postgres yet. Wires the synthesis into: - `ApiRetrieveRunPresenter` - v1 trace GET route - v1 spans GET route - attempts route gains a GET loader (fixes pre-existing Remix "no loader" 400) The `readFallback` infra itself lives on the trigger PR (consumed by `IdempotencyKeyConcern`); this PR adds the route-level synthetic-rendering primitives. Stacked on the replay PR. ## Test plan - [x] \`pnpm run typecheck --filter webapp\` passes - [x] \`pnpm run test --filter webapp test/mollifierSyntheticRedirectInfo.test.ts\` passes - [x] \`pnpm run test --filter webapp test/mollifierSyntheticSpanRun.test.ts\` passes --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
Daniel Sutton
|
4745754a7a |
feat(webapp,run-engine): mollifier drainer replay + stale sweep + cancelled-run engine API (#3754)
## Summary The replay side of the mollifier: - `DrainerHandler`: reads buffered snapshots and replays them through `engine.trigger` to materialise PG rows. - `RunEngine.createCancelledRun`: new public method the handler uses to write CANCELED rows directly from snapshots (bypass queue + waitpoint, emit `runCancelled`). Tolerates the cjson empty-table tags edge case found during validation. - Drainer fairness: org → env rotation so a heavy env doesn't starve light ones in the same org. - Stale-entry sweep + telemetry + alertable gauge so a stuck/offline drainer surfaces in alerts. Both the drainer and sweep default-off; nothing fires unless flagged on (`TRIGGER_MOLLIFIER_DRAINER_ENABLED`, `TRIGGER_MOLLIFIER_STALE_SWEEP_ENABLED`). Stacked on the trigger-time decisions PR. ## Test plan - [x] \`pnpm run typecheck --filter webapp\` passes - [x] \`pnpm run test --filter webapp test/mollifierDrainerHandler.test.ts\` passes - [x] \`pnpm run test --filter webapp test/mollifierStaleSweep.test.ts\` passes - [x] \`pnpm run test --filter @internal/run-engine src/engine/tests/createCancelledRun.test.ts\` passes - [x] \`pnpm run test --filter @trigger.dev/redis-worker packages/redis-worker/src/mollifier/drainer.test.ts\` passes --- ## Ship-gate follow-up fix **Drainer writes SYSTEM_FAILURE on max-attempts exhaustion.** Adds an `onTerminalFailure` callback on `MollifierDrainerOptions` so the customer's run lands a SYSTEM_FAILURE PG row even when the drainer exhausts `MAX_ATTEMPTS` on a retryable PG error (previously `buffer.fail()` was called with no row written → silent data loss). The callback runs before `buffer.fail()` on every terminal path (non-retryable AND max-attempts-exhausted), and re-throwing a retryable error from the callback causes the drainer to requeue rather than fail. Bumps `@trigger.dev/redis-worker` to a **minor** changeset (additive option + new exported types). Includes 5 unit tests covering both terminal causes plus the requeue-on-retryable-callback-failure path and no-callback back-compat. --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
Daniel Sutton
|
20a676d2ed |
fix(core,webapp): coerce numeric concurrencyKey to string (#3789)
## Summary `concurrencyKey` validation accepted only `z.string().optional()` on the single-trigger and V2/V3 batch endpoints, and the Phase-2 streaming NDJSON endpoint accepted `z.record(z.unknown()).optional()` for the entire `options` field. Callers passing `concurrencyKey: someNumericId` (e.g. `payload.userId`) either failed schema validation on the first two paths or sailed through on Phase-2 and then failed downstream at `prisma.taskRun.create` with `Argument concurrencyKey: Expected String or Null, provided Int`. The schema now accepts `string | number` for `concurrencyKey` and stringifies on the way in, across all three paths. The Phase-2 NDJSON `options` is tightened to reuse the strict `BatchTriggerTaskItem.options` shape so it validates identically to the V2/V3 batch endpoints. A defensive `typeof === "number"` coercion at the `engine.trigger` call site in `RunEngineTriggerTaskService` covers in-flight Redis-stored batch items enqueued before the schema fix — those items are rebuilt from a `Record<string, unknown>` shape that bypasses the new schema and would otherwise continue failing for up to their TTL. ## Test plan - [x] `packages/core/src/v3/schemas/batchItemNDJSON.test.ts` — unit tests covering numeric→string coercion, string passthrough, no-options, and rejection of non-string/non-number shapes across `TriggerTaskRequestBody`, `BatchTriggerTaskItem`, and `BatchItemNDJSON`. - [x] `apps/webapp/test/engine/triggerTask.test.ts` — `containerTest` simulating the in-flight Redis batch-item shape (numeric `concurrencyKey` via `Record<string, unknown>`), verifies the run is created with `concurrencyKey: "51262"`. Without the worker coercion, the test reproduces the production stack at `prisma.taskRun.create`. - [x] `pnpm run typecheck --filter webapp` clean. - [x] `pnpm run build --filter @trigger.dev/core --filter @trigger.dev/sdk --filter trigger.dev` clean. Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
Daniel Sutton
|
577f35eebe |
feat(webapp): mollifier trigger-time decisions — mollify, claim, read fallback (#3753)
## Summary The trigger hot path's mollifier integration: - `mollifyTrigger`: when the gate trips, write the engine.trigger snapshot to the buffer and return a synthesised QUEUED response. Postgres write is deferred to drainer-replay (next PR in the stack). - Pre-gate idempotency-key claim: same-key triggers serialise through Redis so a burst lands in PG / buffer exactly once. - Read-fallback extensions: `findRunByIdWithMollifierFallback` for the trigger-time idempotency lookup that must see buffered runs. - Gate bypasses: `debounce`, `oneTimeUseToken`, `parentTaskRunId`/`triggerAndWait` skip the mollify path entirely. - `triggerTask` + `IdempotencyKeyConcern` wired to the above. All behaviour gated by the master `TRIGGER_MOLLIFIER_ENABLED` switch; off-state hot path is unchanged (the gate is not even consulted). Stacked on the buffer extensions PR. ## Test plan - [x] \`pnpm run typecheck --filter webapp\` passes - [x] \`pnpm run test --filter webapp test/mollifierMollify.test.ts\` passes - [x] \`pnpm run test --filter webapp test/mollifierIdempotencyClaim.test.ts\` passes - [x] \`pnpm run test --filter webapp test/mollifierReadFallback.test.ts\` passes - [x] \`pnpm run test --filter webapp test/mollifierGate.test.ts\` passes - [x] \`pnpm run test --filter webapp test/engine/triggerTask.test.ts\` passes --- ## Ship-gate follow-up fixes - **Batch items bypass the mollifier gate** — fixes `BatchTaskRunItem_taskRunId_fkey` FK violation on batch triggers when the gate trips. End-state is a drainer-side `BatchTaskRunItem` create-on-materialise; batch traffic passes through the gate until that lands. - **IdempotencyKeyConcern honours buffered-run TTL on expiry** — buffered path now clears expired idempotency claims (read-side) and resets the buffer's `mollifier:idempotency:*` SETNX binding (write-side) so a re-trigger past the customer's TTL lands as a fresh run instead of echoing the stale buffered runId. --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
Daniel Sutton
|
3924aa4adb |
feat(redis-worker,webapp): mollifier buffer extensions + snapshot type (#3752)
## Summary Buffer-side data layer used by the rest of the mollifier phase-3 stack. - `buffer.ts` gains entry inspection (`getEntry`), idempotency lookup (`lookupIdempotency`), in-place snapshot mutation (`mutateSnapshot`), and dwell tracking. All atomic via Lua. - `mollifierSnapshot.server.ts`: shared `MollifierSnapshot` type plus (de)serialise helpers. - Drops the entry-TTL config and its env var. The drainer is the recovery mechanism; an entry that survives the drainer should surface as a stale-sweep alert, not silently TTL away. Adds methods to the buffer interface; nothing consumes them yet. Subsequent PRs in the stack wire trigger-time mollify, read-fallback, and mutation paths against this surface. ## Test plan - [x] \`pnpm run typecheck --filter webapp\` passes - [x] \`pnpm run test --filter @trigger.dev/redis-worker packages/redis-worker/src/mollifier/buffer.test.ts\` passes --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
Saadi Myftija
|
181d9ba541 |
feat: preview environment dispatch workflow (#3786)
Publishes a `repository_dispatch` `preview-deploy` event that enables preview deployments. Opt-in only currently. |
||
|
github-actions[bot]
|
8f066ac389 |
chore: release v4.5.0-rc.3 (#3763)
🚀 Publish Trigger.dev Docker / typecheck (push) Failing after 0s
🚀 Publish Trigger.dev Docker / publish-webapp (push) Has been skipped
🚀 Publish Trigger.dev Docker / publish-worker (push) Has been skipped
🚀 Publish Trigger.dev Docker / publish-worker-v4 (push) Has been skipped
🚀 Publish Trigger.dev Docker / units (push) Failing after 0s
🧭 Helm Chart Release / lint-and-test (push) Has been cancelled
🧭 Helm Chart Release / release (push) Has been cancelled
## Summary 1 improvement, 2 bug fixes. ## Improvements - Retry `TASK_MIDDLEWARE_ERROR` under the task's retry policy instead of failing the run on the first attempt. The error was already classified as retryable by `shouldRetryError`, but `shouldLookupRetrySettings` did not include it, so the retry flow fell through to `fail_run`. Fixes #3231. ([#3676](https://github.com/triggerdotdev/trigger.dev/pull/3676)) ## Bug fixes - Fix `TypeError` in `unflattenAttributes` when the input attribute map contains conflicting dotted key paths (e.g. both `a.b` set to a scalar and `a.b.c` set to a value). The path-walk loop now applies last-write-wins when a prior key wrote a primitive, null, or array at an intermediate slot, matching the existing precedent in `AttributeFlattener.addAttribute`. Callers no longer crash when handed malformed external attribute inputs. ([#3762](https://github.com/triggerdotdev/trigger.dev/pull/3762)) - Fix external trace context leaking across runs on warm-started workers with `processKeepAlive` enabled. Every subsequent run's attempt span was being exported with the first run's `traceId` and `parentSpanId`, breaking causal-chain navigation in external APM tools. Runs without an external trace context are unaffected. ([#3768](https://github.com/triggerdotdev/trigger.dev/pull/3768)) <details> <summary>Raw changeset output</summary> ⚠️⚠️⚠️⚠️⚠️⚠️ `main` is currently in **pre mode** so this branch has prereleases rather than normal releases. If you want to exit prereleases, run `changeset pre exit` on `main`. ⚠️⚠️⚠️⚠️⚠️⚠️ # Releases ## @trigger.dev/build@4.5.0-rc.3 ### Patch Changes - Updated dependencies: - `@trigger.dev/core@4.5.0-rc.3` ## trigger.dev@4.5.0-rc.3 ### Patch Changes - Updated dependencies: - `@trigger.dev/core@4.5.0-rc.3` - `@trigger.dev/build@4.5.0-rc.3` - `@trigger.dev/schema-to-json@4.5.0-rc.3` ## @trigger.dev/core@4.5.0-rc.3 ### Patch Changes - Retry `TASK_MIDDLEWARE_ERROR` under the task's retry policy instead of failing the run on the first attempt. The error was already classified as retryable by `shouldRetryError`, but `shouldLookupRetrySettings` did not include it, so the retry flow fell through to `fail_run`. Fixes #3231. ([#3676](https://github.com/triggerdotdev/trigger.dev/pull/3676)) - Fix `TypeError` in `unflattenAttributes` when the input attribute map contains conflicting dotted key paths (e.g. both `a.b` set to a scalar and `a.b.c` set to a value). The path-walk loop now applies last-write-wins when a prior key wrote a primitive, null, or array at an intermediate slot, matching the existing precedent in `AttributeFlattener.addAttribute`. Callers no longer crash when handed malformed external attribute inputs. ([#3762](https://github.com/triggerdotdev/trigger.dev/pull/3762)) - Fix external trace context leaking across runs on warm-started workers with `processKeepAlive` enabled. Every subsequent run's attempt span was being exported with the first run's `traceId` and `parentSpanId`, breaking causal-chain navigation in external APM tools. Runs without an external trace context are unaffected. ([#3768](https://github.com/triggerdotdev/trigger.dev/pull/3768)) ## @trigger.dev/plugins@4.5.0-rc.3 ### Patch Changes - Updated dependencies: - `@trigger.dev/core@4.5.0-rc.3` ## @trigger.dev/python@4.5.0-rc.3 ### Patch Changes - Updated dependencies: - `@trigger.dev/core@4.5.0-rc.3` - `@trigger.dev/build@4.5.0-rc.3` - `@trigger.dev/sdk@4.5.0-rc.3` ## @trigger.dev/react-hooks@4.5.0-rc.3 ### Patch Changes - Updated dependencies: - `@trigger.dev/core@4.5.0-rc.3` ## @trigger.dev/redis-worker@4.5.0-rc.3 ### Patch Changes - Updated dependencies: - `@trigger.dev/core@4.5.0-rc.3` ## @trigger.dev/rsc@4.5.0-rc.3 ### Patch Changes - Updated dependencies: - `@trigger.dev/core@4.5.0-rc.3` ## @trigger.dev/schema-to-json@4.5.0-rc.3 ### Patch Changes - Updated dependencies: - `@trigger.dev/core@4.5.0-rc.3` ## @trigger.dev/sdk@4.5.0-rc.3 ### Patch Changes - Updated dependencies: - `@trigger.dev/core@4.5.0-rc.3` </details> Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>helm-v4.5.0-rc.3 v4.5.0-rc.3 v.docker.4.5.0-rc.3 |
||
|
Eric Allam
|
9211032733 |
chore(database): drop unused TaskRun status composite index (#3743)
## Summary Drops the `TaskRun_status_runtimeEnvironmentId_createdAt_id_idx` index from the `TaskRun` table. After #3742 gated the legacy `WAITING_FOR_DEPLOY` drain to V1-engine workers only, this index sees zero scans on both writer and reader replicas. Removing it cuts index maintenance on every `TaskRun` INSERT/UPDATE. ## Why The index existed to support `WHERE status = X AND runtimeEnvironmentId = Y` queries from `ExecuteTasksWaitingForDeployService`, which is V1-only and no longer triggered on V2 deployments. A code grep across `apps/webapp` and `internal-packages/run-engine` confirmed no V2 production query uses this access pattern — every other `status:` filter on `TaskRun` is paired with `id`/`friendlyId`/`parentSpanId` and uses a different index. Dropping it also unlocks HOT updates on the dequeue path. The dequeue `UPDATE` modifies `status` (`QUEUED` -> `DEQUEUED`), and `status` is the leading column of this index — its presence blocked HOT eligibility for every `TaskRun` UPDATE. With the index gone, dequeue UPDATEs can become HOT, reducing WAL bytes and removing the B-tree page contention on this index's right-edge leaves. Uses `DROP INDEX CONCURRENTLY` to avoid blocking writes during the drop. ## Sequencing Should only ship once #3742 has soaked long enough to confirm the index is genuinely cold (24h+ of zero scans on `pg_stat_user_indexes`). |
||
|
Saadi Myftija
|
9cb6fd17c3 |
fix(webapp): idempotent DeploymentBackgroundWorker creation (#3772)
Make `POST /api/v1/deployments/:deploymentId/background-workers` idempotent so client-side retries no longer collide on the `BackgroundWorker` `(project, env, version)` unique index. Helps make deployments more resilient against the class of indexing failures that surfaces in the dashboard as "Indexing timed out", e.g. during transient database issues. |
||
|
nicktrn
|
c043c4a6ad |
fix(core): external trace context leaks across warm-started runs (#3768)
On warm-started workers with `processKeepAlive` enabled, every run's attempt span was exported with the first run's `traceId` and `parentSpanId`. `ExternalSpanExporterWrapper` and `ExternalLogRecordExporterWrapper` captured `externalTraceContext` at `TracingSDK` construction, and the SDK is memoized for the worker's lifetime - so the per-run reset of `StandardTraceContextManager.traceContext` never reached the wrappers. Reported by a customer running v4.4.x: 33 distinct runs on the same host/pid showed up in their APM as siblings of one parent span. Fix: the wrappers now read external context live from the trace context manager per export. Runs without an external trace context fall through to the unchanged `externalTraceId` fallback - no behaviour change for them. Regression test in `packages/core/test/externalSpanExporterWrapper.test.ts` asserts that reassigning the manager between exports produces correctly-parented spans. |
||
|
Matt Aitken
|
816986d44e |
fix(webapp): treat Phase 2 batch-stream retries as idempotent (TRI-9944) (#3766)
Returns sealed:true when a fast-completing batchTrigger races the stream finalisation, instead of throwing 422/BatchTriggerError. |
||
|
Iss
|
5083d161b2 | docs: adds relevant env vars to self hosting docs (#3148) | ||
|
Iss
|
8399aa2052 |
fix(core): retry TASK_MIDDLEWARE_ERROR under the task's retry policy (#3676)
Retries `TASK_MIDDLEWARE_ERROR` under the task's retry policy. `shouldRetryError` already classed it as retryable, but `shouldLookupRetrySettings` did not, so the run fell through to `fail_run` on attempt 1 instead of using the task's `retry` config. Fixes #3231. |
||
|
Iss
|
7324a3307f | chore(webapp): rename support tier to "Priority" on plan cards and li… (#3692) | ||
|
Iss
|
df96a937db | docs: troubleshoot "Stream is being deleted" during long waits (#3704) | ||
|
Matt Aitken
|
90bbbd1c44 |
fix(webapp): recover from ClickHouse JSON parse failures on out-of-range integers (#3759)
## Summary Second class of poisoned-row failure in the runs replication path. PR #3708 plugged lone UTF-16 surrogates; this one handles bare JSON integer literals outside ClickHouse's `Int64`..`UInt64` range. Recovery stays purely reactive — the existing `sanitizeRows` walker just gains an extra branch, so the hot replication path pays nothing on healthy rows. Fixes the still-firing customer-facing symptom from [TRI-9755](https://linear.app/triggerdotdev/issue/TRI-9755): `scan-social-profiles` runs continued to be stranded in `EXECUTING` on the Tasks page after #3708 deployed. CloudWatch showed `Dropped batch — ClickHouse JSON parse error but sanitizer found nothing to fix` firing **8/8 times** since the previous deploy (zero successful sanitizations). Root cause: upstream JS Number precision loss on a 21-digit Google Plus ID (`117039831458782873093` → `117039831458782870000`) — the precision-lossy value still serialises as a bare integer that exceeds `UInt64.MAX`, which ClickHouse rejects with `INCORRECT_DATA`. ## How the bug ships The customer task emits an output containing a Poshmark profile's `spec_format`: ```json {"key":"gp_id","proper_key":"Gp Id","value":117039831458782870000,"type":"int"} ``` That value is `1.17e20` — comfortably above `UInt64.MAX` (`1.84e19`) but comfortably below `1e21`. `Number.prototype.toString` only switches to exponential form at `|value| >= 1e21`, so `JSON.stringify` emits the bare token `117039831458782870000` and the ClickHouse `JSON(max_dynamic_paths)` column fails with: ``` Code: 117. DB::Exception: Cannot parse JSON object here: {…}: (while reading the value of key output): (at row 1) : While executing ParallelParsingBlockInputFormat. (INCORRECT_DATA) (version 25.12.x) ``` Same error verbatim as prod. The same number quoted (`"117039831458782870000"`) inserts fine — ClickHouse's dynamic JSON column accepts a `String` subtype on the same path. ## What changed `apps/webapp/app/v3/eventRepository/sanitizeRowsOnParseError.server.ts`: - New private `isUnsafeJsonInteger(value)` helper — true iff `value` is a finite integer-valued JS Number where `|value| < 1e21` (so `JSON.stringify` emits integer form, not exponent) **and** `value` falls outside `[Int64.MIN, UInt64.MAX]`. - `sanitizeUnknownInPlace` gains a number-branch: when the predicate holds, replace the Number with `String(value)`. The downstream JSON column dynamic-types the path as String for that row — fine, since the value was already precision-lossy upstream (no JS Number above 2^53 is numerically meaningful anyway). - Float-valued numbers, large floats (>= 1e21), NaN and Infinity are left alone — `JSON.stringify` emits them with exponents or as `null`, both of which ClickHouse accepts. `apps/webapp/test/sanitizeRowsOnParseError.test.ts`: four new unit tests + an extension to `sanitizeRows` covering surrogate + integer fixes counted together across rows. The unit suite now covers: - Positive value above `UInt64.MAX` (`117039831458782870000` — the actual prod value) - Negative value below `Int64.MIN` - Boundary values pass through (`42`, `Number.MAX_SAFE_INTEGER`, `2^63`) - Non-integer numbers untouched (floats, `1e25`, NaN, Infinity) - The actual `scan-social-profiles` nested shape — finds the offending `gp_id` deep inside `output.data.profiles[].spec_format[].platform_variables[].value` `.server-changes/runs-replication-bigint-recovery.md` — release notes entry. ## Why reactive, not pre-flight `#prepareJson` runs millions of times per day on the replication hot path. Walking every JSON tree to look for oversized integers would add bounded-but-real CPU on every healthy row. `sanitizeRows` only fires after a ClickHouse parse-error rejection, which is a few times a day platform-wide. Extending it costs effectively zero on healthy traffic and gains us recovery on the rare poisoned row. ## Verification - Reproduced 1:1 in a throwaway Docker `clickhouse/clickhouse-server:25.12.11.4` (closest available to the prod `25.12.1.1579` build). Pre-sanitize JSON fails with the exact prod error; post-sanitize JSON inserts cleanly and the row is readable with `gp_id` stored as a String subtype. - `pnpm --filter webapp exec vitest run test/sanitizeRowsOnParseError.test.ts` — 22/22 passing (18 existing + 4 new). - `pnpm run typecheck --filter webapp` — clean. ## Test plan - [x] `pnpm run typecheck --filter webapp` - [x] Unit tests pass against new + existing cases - [x] End-to-end Docker ClickHouse repro confirms recovery - [ ] Post-deploy: confirm `Sanitizing batch after ClickHouse JSON parse error` warns fire instead of `Dropped batch …` errors when `scan-social-profiles` outputs trip CH again - [ ] Post-deploy: confirm `permanentlyDroppedBatches` counter stops climbing in `/stp/trigger-app-prod/ecs/replication/service-container/process-logs` ## What this does NOT do - Doesn't backfill the ~120k+ existing stranded `EXECUTING` rows in production. Same as #3708 — that needs a reconciliation/backfill sweep (separate ticket — TRI-9755 fix #3). - Doesn't address the upstream root cause (the customer task emitting a JS-Number-precision-lossy big int). That's a customer-task concern; our replication path needs to be robust to whatever shape arrives. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com> |
||
|
Oskar Otwinowski
|
5ba1b32d53 | fix(core): handle conflicting dotted paths in unflattenAttributes (#3762) | ||
|
Eric Allam
|
37eeaa3690 |
fix(webapp): skip V1 WAITING_FOR_DEPLOY drain on V2 promotes (#3742)
## Summary Stops the legacy V1 `WAITING_FOR_DEPLOY` drain from running on every V2 deployment promotion. The drain queries `TaskRun` by `status='WAITING_FOR_DEPLOY'`, which only V1-engine runs ever have — V2 runs use `PENDING_VERSION` and are handled out of band. Calling the drain on V2 promotes produced empty queries against the status index and unnecessary reader-DB load. ## Fix Two layers: 1. Gate the enqueue at the call site in `ChangeCurrentDeploymentService` so it only fires when the deployment's worker is on engine V1. 2. Add a `LEGACY_RUN_ENGINE_WAITING_FOR_DEPLOY_DISABLED` env var (default `0`). When set to `1`, the service returns immediately from `call()` — neuters any jobs already sitting in the worker queue from before the deploy lands. V1 customers see no change; V2 promotes no longer trigger the drain. |
||
|
Matt Aitken
|
596a9bb90c |
fix(webapp): retain sessions-replication singleton import via globalThis assignment (#3738)
Linear: [TRI-9864](https://linear.app/triggerdotdev/issue/TRI-9864) (Urgent) Production incident: [TRI-9863](https://linear.app/triggerdotdev/issue/TRI-9863) (mitigated by image revert in cloud#910) ## Bug `apps/webapp/package.json` declares `"sideEffects": false`. PR #3333 (`71d98b4e`) replaced the previous real method-call retention idiom at the two `sessionsReplicationInstance` import sites with: ```ts import { sessionsReplicationInstance } from "..."; void sessionsReplicationInstance; ``` esbuild treats `void <identifier>;` as a pure expression statement under `sideEffects: false` and **tree-shakes the entire import** — including the `singleton(...)` call inside `sessionsReplicationInstance.server.ts` which is the only thing that fires `initializeSessionsReplicationInstance()`. The sessions→ClickHouse logical replication worker never starts, the slot is unconsumed, lag grows. ### How it manifested in production cloud#907's image bump rolled the `SessionReplicationService` ECS task on prod at 14:32 UTC. The new container's startup log emitted `🗃️ Runs replication service enabled` but **not** `🗃️ Sessions replication service enabled` or `🗃️ Sessions replication service started`. CloudWatch `OldestReplicationSlotLag` grew at ~220 MB/min and the `High replication lag` alarm fired at 14:37 UTC. Prod was reverted to the previous image (cloud#910) to stop the bleed. ### Verification `grep` of the built bundle `apps/webapp/build/index.js` (built from `c0365d36`): - **3** occurrences of `Runs replication` / `runsReplicationInstance` strings ✅ - **0** occurrences of `Sessions replication` / `sessionsReplicationInstance` / `SessionsReplicationService` ❌ The runs path survives tree-shaking because `adminWorker.server.ts` and `admin.api.v1.runs-replication.*` routes have real method calls (`.start()`, `.teardown()`, `.backfill()`) — observable uses the tree-shaker must preserve. The sessions singleton has no real callers, only the `void` no-ops, hence its complete elimination from the bundle. ## Fix Replace `void sessionsReplicationInstance;` with an assignment to `globalThis`, an unambiguous observable side effect the bundler cannot eliminate: ```ts (globalThis as Record<string, unknown>).__sessionsReplicationInstance = sessionsReplicationInstance; ``` Applied at both call sites: `apps/webapp/app/entry.server.tsx` and `apps/webapp/app/v3/services/adminWorker.server.ts`. Surrounding comments updated to document the bundler interaction so the next maintainer doesn't reintroduce `void`. ## Out of scope (follow-ups) - **Robustness improvement**: change `apps/webapp/package.json` from `"sideEffects": false` to an allowlist that includes `*Instance.server.ts` files. Prevents the same regression shape via any future `*Instance` singleton. - **Build-time check**: add a `grep` post-build step in `publish.yml` requiring `"Sessions replication"` to appear in `apps/webapp/build/index.js`. Catches this exact regression at CI time. ## Test plan - [x] `pnpm run typecheck --filter webapp` clean - [ ] After merge + publish: confirm new image's `SessionReplicationService` container logs `🗃️ Sessions replication service enabled` and `🗃️ Sessions replication service started` at startup - [ ] After re-deploying to prod: confirm `OldestReplicationSlotLag` stops growing and drains |
||
|
Matt Aitken
|
eefb96c87d |
fix(webapp): recover from ClickHouse JSON parse failures in runs replication (#3708)
## Summary On a ClickHouse `Cannot parse JSON object` rejection, `RunsReplicationService` now sanitizes lone UTF-16 surrogates across the failing batch via the existing `sanitizeRows` helper and retries once. If the sanitizer found nothing or the retry also fails, the batch is dropped loudly with a counter increment, so the surrounding `#insertWithRetry` layer doesn't spin three more times on a deterministic failure. Non-parse errors propagate unchanged. Mirrors the pattern from #3659 (for `ClickhouseEventRepository`) — same root cause (lone UTF-16 surrogates in user-provided JSON), same recovery shape, **reusing the same shared helpers** (`sanitizeRows`, `isClickHouseJsonParseError`, `parseRowNumberFromError`). Fixes the customer-facing symptom from [TRI-9755](https://linear.app/triggerdotdev/issue/TRI-9755): a single row's poisoned `output` JSON used to take down the `COMPLETED_SUCCESSFULLY` UPDATE events for its 50+ batch-mates, stranding them in `EXECUTING` in ClickHouse forever and inflating "Running" counts on the Tasks page. Confirmed in production this is ongoing — ~120k stale rows accumulated in a single 5-hour burst on 2026-05-18; smaller continuous leak before and after. ## What changed `apps/webapp/app/services/runsReplicationService.server.ts`: - Imports the three helpers from `~/v3/eventRepository/sanitizeRowsOnParseError.server` (no duplication; no move). - New private `#insertWithJsonParseRecovery<T>(rows, doInsert, contextLabel, attempt)` — generic over `TaskRunInsertArray[]` and `PayloadInsertArray[]`, structurally identical to `ClickhouseEventRepository.#insertWithJsonParseRecovery`. Try → on parse error sanitize the whole batch (the `at row N` hint is logged but not used to slice — semantics under `input_format_parallel_parsing` aren't stable) → retry once → drop with loud log if sanitizer found nothing OR retry still fails. - `#insertTaskRunInserts` and `#insertPayloadInserts` extract a `doInsert` closure and hand it to the wrapper. Existing error logging, span recording, and `recordSpanError` are preserved inside the closure. - New `private _permanentlyDroppedBatches = 0` counter with a public getter, for ops dashboards and tests (matches the events-repo convention). One shared counter for both insert sites — granularity comes from the `contextLabel` (`task_runs_v2` / `raw_task_runs_payload_v1`) on every log line. `.server-changes/runs-replication-utf16-recovery.md` — release notes entry. ## Why no new tests The shared helpers already have full unit + real-ClickHouse contract coverage from #3659 (`apps/webapp/test/sanitizeRowsOnParseError.test.ts`, `apps/webapp/test/otlpUtf16Sanitization.integration.test.ts`). The new wrapper is a line-for-line structural port. Adding a parallel integration test would require synthesizing bad data that *escapes* the preemptive `detectBadJsonStrings` check in `#prepareJson` but still trips ClickHouse — non-trivial without hand-crafted fixtures and wouldn't cover any new logic. ## What this does NOT do - Doesn't touch the ~120k existing stale `EXECUTING` rows in production. That needs a reconciliation/backfill sweep (separate ticket — TRI-9755 fix #3). - Doesn't sanitize the `error` column path (`runsReplicationService.server.ts:932 const errorData = { data: run.error };`). Reactive recovery will catch it if it ever poisons a batch, but feeding it through `#prepareJson` like `output` is a cheap follow-up. ## Test plan - [x] `pnpm run typecheck --filter webapp` — clean - [ ] Post-deploy: confirm `permanentlyDroppedBatches` counter stays at zero (or near-zero) in `/stp/trigger-app-prod/ecs/replication/service-container/process-logs`, and watch for `Sanitizing batch after ClickHouse JSON parse error` warns to confirm recovery is firing on real traffic - [ ] Post-deploy: confirm the rate of new "EXECUTING-but-actually-COMPLETED" zombies in ClickHouse flattens (current rate ≈ tens-to-hundreds per hour platform-wide) 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com> |
||
|
Eric Allam
|
9f64bf404b |
docs(ai-chat): slim-wire HITL continuations + field-level merge contract (#3721)
## Summary Updates the AI chat docs to match the slim-wire + field-level merge behavior shipped in #3719 and the precise `.in/append` cap + CORS-readable 413 shipped in #3720. No behavior changes here — code is correct in `main`; the docs were lagging on three patterns customers copy out of the page. ## What changed - **`hydrateMessages` examples upsert by id** (in `lifecycle-hooks.mdx`, `patterns/database-persistence.mdx`, and `patterns/persistence-and-replay.mdx`). The previous `stored.push(newMsg)` pattern duplicated the assistant id on HITL continuations and caused the LLM to receive a tool call with no `arguments`. The new examples include the rationale inline. - **`onValidateMessages` example filters to user messages** (`lifecycle-hooks.mdx`). The previous example called `validateUIMessages({ messages, tools })` directly, which now throws on HITL slim wires (the AI SDK schema requires `input` on resolved tool parts). New example shows the filter pattern, with a Warning callout explaining why. - **Merge contract description updated** (`lifecycle-hooks.mdx`). The old wording said incoming messages are "auto-merged" / "replaced"; the new description explains the actual field-level overlay (state advances only). - **Approval-responded wire example slimmed** (`client-protocol.mdx`). Shows the minimum shape the agent reads — `state` + `approval` (or `output` / `errorText` for HITL). Notes that the built-in transports ship this slim shape by default and that fuller shapes are still accepted. - **`/in/append` 413 row and FAQ updated** (`client-protocol.mdx`, `patterns/trusted-edge-signals.mdx`). Reflects the new precise S2 cap and the CORS-readable 413. - **New changelog entry** at the top of `changelog.mdx` covering all of the above. The historical `## 512 KiB ceiling removed` entry further down the changelog is left as-is (it's a snapshot of the prior transition), and the v4.5 upgrade-guide section is skipped — the merge contract is backwards compatible. ## Test plan - Mintlify dev preview renders cleanly with no broken anchors - Linked references resolve (`/ai-chat/lifecycle-hooks#hydratemessages`, `/ai-chat/lifecycle-hooks#onvalidatemessages`, `/ai-chat/patterns/database-persistence#alternative-hydratemessages`, `/ai-chat/client-protocol#step-3-send-messages-stops-and-actions`, `/ai-chat/patterns/large-payloads`) |
||
|
github-actions[bot]
|
d34014ddcc |
chore: release v4.5.0-rc.2 (#3702)
🚀 Publish Trigger.dev Docker / typecheck (push) Failing after 0s
🚀 Publish Trigger.dev Docker / publish-webapp (push) Has been skipped
🚀 Publish Trigger.dev Docker / publish-worker (push) Has been skipped
🚀 Publish Trigger.dev Docker / publish-worker-v4 (push) Has been skipped
🚀 Publish Trigger.dev Docker / units (push) Failing after 4s
## Summary
3 improvements, 1 bug fix.
## Improvements
- The per-turn merge now overlays the wire copy's tool-part state
advancement onto the agent's existing chain — `state` + the matching
resolution field (`output` / `errorText` / `approval`) come from the
wire, everything else (text, reasoning, tool `input`, provider metadata)
stays whatever the snapshot or `hydrateMessages` returned. Previously a
full-message replace overwrote those fields with whatever the client
shipped, so a slimmed wire copy landed a tool call with no `arguments`
on the next LLM call. Covers `output-available` / `output-error` (HITL
`addToolOutput`) and `approval-responded` / `output-denied` (approval
flow).
- `TriggerChatTransport.sendMessages` and `AgentChat.sendRaw` now slim
assistant messages that carry advanced tool parts. The wire payload is
just `{ id, role, parts: [<state + resolution field>] }` for
`submit-message` continuations; everything else passes through.
Reasoning blobs and full tool inputs no longer ride the wire on every
`addToolOutput` / `addToolApproveResponse`, so continuation payloads
stay well under the `.in/append` cap on long agent loops.
- Add `TriggerClient` for running multiple SDK clients side-by-side,
each with its own auth, preview branch, and baseURL. Useful when a
single process needs to trigger tasks or read runs across multiple
projects, environments, or preview branches without mutating shared
global state.
([#3683](https://github.com/triggerdotdev/trigger.dev/pull/3683))
## Bug fixes
- Fix `chat.agent` HITL continuations on reasoning-heavy turns. Two
changes that work together:
([#3719](https://github.com/triggerdotdev/trigger.dev/pull/3719))
<details>
<summary>Raw changeset output</summary>
⚠️⚠️⚠️⚠️⚠️⚠️
`main` is currently in **pre mode** so this branch has prereleases
rather than normal releases. If you want to exit prereleases, run
`changeset pre exit` on `main`.
⚠️⚠️⚠️⚠️⚠️⚠️
# Releases
## @trigger.dev/build@4.5.0-rc.2
### Patch Changes
- Updated dependencies:
- `@trigger.dev/core@4.5.0-rc.2`
## trigger.dev@4.5.0-rc.2
### Patch Changes
- Updated dependencies:
- `@trigger.dev/build@4.5.0-rc.2`
- `@trigger.dev/core@4.5.0-rc.2`
- `@trigger.dev/schema-to-json@4.5.0-rc.2`
## @trigger.dev/plugins@4.5.0-rc.2
### Patch Changes
- Updated dependencies:
- `@trigger.dev/core@4.5.0-rc.2`
## @trigger.dev/python@4.5.0-rc.2
### Patch Changes
- Updated dependencies:
- `@trigger.dev/sdk@4.5.0-rc.2`
- `@trigger.dev/build@4.5.0-rc.2`
- `@trigger.dev/core@4.5.0-rc.2`
## @trigger.dev/react-hooks@4.5.0-rc.2
### Patch Changes
- Updated dependencies:
- `@trigger.dev/core@4.5.0-rc.2`
## @trigger.dev/redis-worker@4.5.0-rc.2
### Patch Changes
- Updated dependencies:
- `@trigger.dev/core@4.5.0-rc.2`
## @trigger.dev/rsc@4.5.0-rc.2
### Patch Changes
- Updated dependencies:
- `@trigger.dev/core@4.5.0-rc.2`
## @trigger.dev/schema-to-json@4.5.0-rc.2
### Patch Changes
- Updated dependencies:
- `@trigger.dev/core@4.5.0-rc.2`
## @trigger.dev/sdk@4.5.0-rc.2
### Patch Changes
- Fix `chat.agent` HITL continuations on reasoning-heavy turns. Two
changes that work together:
([#3719](https://github.com/triggerdotdev/trigger.dev/pull/3719))
- The per-turn merge now overlays the wire copy's tool-part state
advancement onto the agent's existing chain — `state` + the matching
resolution field (`output` / `errorText` / `approval`) come from the
wire, everything else (text, reasoning, tool `input`, provider metadata)
stays whatever the snapshot or `hydrateMessages` returned. Previously a
full-message replace overwrote those fields with whatever the client
shipped, so a slimmed wire copy landed a tool call with no `arguments`
on the next LLM call. Covers `output-available` / `output-error` (HITL
`addToolOutput`) and `approval-responded` / `output-denied` (approval
flow).
- `TriggerChatTransport.sendMessages` and `AgentChat.sendRaw` now slim
assistant messages that carry advanced tool parts. The wire payload is
just `{ id, role, parts: [<state + resolution field>] }` for
`submit-message` continuations; everything else passes through.
Reasoning blobs and full tool inputs no longer ride the wire on every
`addToolOutput` / `addToolApproveResponse`, so continuation payloads
stay well under the `.in/append` cap on long agent loops.
Note: `onValidateMessages` receives the slim wire on HITL turns. If you
call `validateUIMessages` from `ai` against the full `messages` array it
will reject the slim assistant; filter to user messages (or skip on HITL
turns) — see the updated docstring on `onValidateMessages` for the
recommended pattern.
For `hydrateMessages` hooks that persist the chain, this release also
adds a small helper to the `@trigger.dev/sdk/ai` surface:
```ts
import { chat, upsertIncomingMessage } from "@trigger.dev/sdk/ai";
chat.agent({
hydrateMessages: async ({ chatId, trigger, incomingMessages }) => {
const record = await db.chat.findUnique({ where: { id: chatId } });
const stored = record?.messages ?? [];
if (upsertIncomingMessage(stored, { trigger, incomingMessages })) {
await db.chat.update({ where: { id: chatId }, data: { messages: stored }
});
}
return stored;
},
});
```
It pushes fresh user messages by id, no-ops on HITL continuations (the
incoming shares an id with the existing assistant — the runtime overlays
the new tool-state advance), and skips on non-`submit-message` triggers.
Returns `true` if it mutated `stored` so the caller knows whether to
persist.
Net effect: `chat.addToolOutput(...)` /
`chat.addToolApproveResponse(...)` on multi-step reasoning agents
(OpenAI Responses with `store: false`, Anthropic extended thinking,
etc.) no longer blows the cap and no longer corrupts the LLM input.
- Add `TriggerClient` for running multiple SDK clients side-by-side,
each with its own auth, preview branch, and baseURL. Useful when a
single process needs to trigger tasks or read runs across multiple
projects, environments, or preview branches without mutating shared
global state.
([#3683](https://github.com/triggerdotdev/trigger.dev/pull/3683))
```ts
import { TriggerClient } from "@trigger.dev/sdk";
const prod = new TriggerClient({ accessToken:
process.env.TRIGGER_PROD_KEY });
const preview = new TriggerClient({
accessToken: process.env.TRIGGER_PREVIEW_KEY,
previewBranch: "signup-flow",
});
await prod.tasks.trigger("send-email", payload);
await preview.runs.list({ status: ["COMPLETED"] });
```
- Updated dependencies:
- `@trigger.dev/core@4.5.0-rc.2`
## @trigger.dev/core@4.5.0-rc.2
</details>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
v.docker.4.5.0-rc.2
v4.5.0-rc.2
|
||
|
Eric Allam
|
75679c7518 |
fix(sdk): chat HITL continuations no longer break the next LLM call (#3719)
## Summary
Multi-step reasoning agents with HITL tools (OpenAI Responses with
`store: false`, Anthropic extended thinking, etc.) failed on
`chat.addToolOutput(...)` continuations — either the wire payload blew
the `.in/append` cap (reasoning blobs + tool inputs routinely > 512
KiB), or app-side slimming workarounds got overwritten server-side and
the next LLM call landed a tool call with no `arguments`. Both modes are
fixed.
## Design
The per-turn merge in `chat.agent` now overlays only the tool-part state
advances (`output-available` / `output-error` / `approval-responded` /
`output-denied`) from the wire copy onto the hydrated/snapshot chain.
Previously it replaced the entire message, which dropped `input`,
reasoning, and text from the LLM's view whenever the wire was slim.
In parallel, `TriggerChatTransport.sendMessages` and `AgentChat.sendRaw`
now slim the assistant message themselves on `submit-message`
continuations: ship `{ id, role, parts: [<resolved tool part only>] }`,
everything else reconstructed server-side from `hydrateMessages` or the
durable snapshot. Continuation payloads drop from 600 KiB – 1 MiB to ~1
KiB.
`references/ai-chat` `aiChatHydrated.hydrateMessages` now upserts by id
instead of pushing. With slim continuations, a blind push duplicates the
assistant id in the returned chain — the merge updates the first match,
the slim duplicate goes straight to `toModelMessages` with no `input`,
and the LLM 4xx's. This is the canonical pattern customers should mirror
in their own hydrate implementations.
## Test plan
- 11 new tests (slim helper unit + slim+merge integration for HITL,
approval, default no-hydrate branch)
- Full SDK suite: 239 tests pass across 19 files
- End-to-end sweep against `references/ai-chat`: 19 customer-side smoke
tests green; HITL wire bodies confirmed at ~1 KiB (was 600 KiB+); no
provider 4xx errors across OpenAI Responses or Anthropic
|
||
|
Eric Allam
|
c0365d36fb |
fix(webapp): precise S2 record cap + CORS 413 on session append (#3720)
## Summary
Two improvements to session `.in/append`:
- Oversize-body 413 responses now carry CORS headers, so browser fetches
see a readable status instead of an opaque `TypeError: Failed to fetch`.
App-side retry-on-disconnect loops no longer spin forever on a
permanently-rejected payload.
- The per-record cap is now computed precisely against S2's actual
ceiling instead of the conservative 512 KiB floor. Legitimate ~600-900
KiB tool outputs (search results, file content) now succeed;
pathological all-quote content that would double under JSON escape still
rejects cleanly.
## Design
S2 enforces a per-record metered size of `8 + 2*H + Σ(header name +
value) + body ≤ 1048576` bytes. With no record headers (our case), the
budget reduces to `body ≤ 1048568`. Verified empirically against cloud
S2 — append succeeds at metered=1048576 and 422s at 1048577 with `record
must have metered size less than 1 MiB`.
The old `MAX_APPEND_BODY_BYTES = 512 KiB` was derived by assuming
worst-case JSON escape doubling (every byte becomes `\"` or `\\`),
giving `(1 MiB - overhead) / 2`. Safe, but rejects ~half the legitimate
input space.
The new flow:
1. Pre-cap the HTTP body at 1 MiB (DoS guard against reading arbitrary
garbage before we can compute the wrap).
2. After reading, `S2RealtimeStreams.#appendPartByName` computes
`Buffer.byteLength(JSON.stringify({data: part, id: partId}), "utf8") +
8` and throws `S2RecordTooLargeError` (a `ServiceValidationError` with
status 413) if it would exceed S2's ceiling. The route's existing error
branch maps the throw to a 413 with a descriptive message.
The 413 CORS fix is a single-line change in `apiBuilder.server.ts` —
`wrapResponse` was being skipped on the body-too-large branch; every
other error branch wraps; the 413 was the exception.
## Test plan
- Empirically verified against cloud S2 with a boundary scan across
`[1048568, 1048569, ..., 1048576]` and across H ∈ {0, 1×5 hdr bytes,
1×14 hdr bytes} — the formula matches exactly
- Browser-side fetch on a 700 KiB POST now resolves with a readable
`status: 413` (no `TypeError: Failed to fetch`)
- A 900 KiB ASCII tool output now passes (would have 413'd at 512 KiB
pre-fix)
|