fix/locked-version-trigger-stale-replica
7381 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Eric Allam
|
8008ba7c4b |
feat(webapp): add task metadata cache resolution metrics
Emit a `task_meta_cache.resolve` counter on the trigger path, labeled by lookup path (locked / current) and the source that satisfied it (cache / replica / writer / miss). cache over total is the cache hit rate (its inverse is coldness); writer over total is how often the read replica returned empty for a row the primary had. Labels are bounded, with no env / worker / slug ids. |
||
|
Eric Allam
|
5cea365e45 |
fix(webapp): stop locked-version triggers failing on stale replica reads
A locked-version trigger such as triggerAndWait resolved the task's metadata from the read replica and, on a miss, threw a non-retryable "task not found on locked version" even though the task was registered. A read replica can return an empty result for a row that already exists on the primary, so this surfaced as intermittent, self-recovering trigger failures. The locked worker is already resolved on the primary in the same request, so the resolver now re-checks the primary when the replica returns no row, and only reports the task missing when the primary genuinely lacks it. This runs on the cache-miss path only and leaves the hot path unchanged. |
||
|
Eric Allam
|
43b493628c |
docs(ai-chat): add the 4.5.0-rc.6 changelog entry (#3927)
## Summary Adds the 4.5.0-rc.6 entry to the AI chat changelog, covering the chat-facing items shipping in [#3870](https://github.com/triggerdotdev/trigger.dev/pull/3870): the chat.agent reliability batch, the continuation boot latency fix, the chat.headStart hydration and reasoning fixes, the chat.createSession stop and continuation fixes, and the new trigger skills installer. Should merge alongside the release so the changelog matches the published version. |
||
|
Eric Allam
|
3bc3a1796f |
docs(ai-chat): custom agents page, backend decision table, and a building-agents anatomy entry (#3921)
## Summary Documents the two lower-level chat backend APIs and restructures the Building agents section so it has a sane reading order. **Custom agents page.** `chat.customAgent()` was effectively undocumented (one passing mention) and `chat.createSession()` was buried at the bottom of the Backend page, prompted by a customer asking whether dropping down a level was supported at all. Both now live on one dedicated page framed as a composition: register with `customAgent`, then drive turns with the managed `createSession` iterator or a hand-rolled primitives loop. The page covers the patterns the managed lifecycle otherwise handles for you, each verified against a running agent: seeding history on continuation runs (and why the seed must go through the turn-0 `addIncoming`, which replaces the accumulator), persisting the user message before streaming so a mid-stream reload keeps it, racing `totalUsage` after a stop so the loop cannot wedge, and the single-message wire shape. **Backend page.** Now leads with a decision table across the three abstraction levels and focuses on `chat.agent()`, routing to the new page. Stale examples that read a plural `messages` field off the wire payload are fixed (copy-pasting them broke turn accumulation), and the ChatSessionOptions / ChatTurn reference tables gain their missing rows (`compaction`, `pendingMessages`, usage fields, `setMessages`, `prepareStep`). **Anatomy page + reorder.** The Building agents group opened with the long How it works mechanics page, a wall right after the Quick Start. A short Anatomy page now leads the group: the three moving parts, one annotated example where each region names the page that covers it, and a routing table. How it works moves to the end of the group as the depth payoff, matching where peer docs put their internals pages. All pages visually verified against a local Mintlify build; cross-links and anchors updated across the section. |
||
|
Eric Allam
|
84809b02ca |
docs(ai-chat): head-start persistence contract and a clearer sessions page (#3908)
## Summary Two documentation improvements for the AI chat docs. **Head-start persistence contract.** The fast starts page now documents what your hooks can rely on across a head-start handover: one stable assistant `messageId` for the whole turn, `onTurnComplete` as the canonical persistence point, reasoning parts flowing into durable history, and how Head Start composes with `hydrateMessages` (the first-turn history arrives as `incomingMessages`, and the runtime splices the warm partial onto the hydrated chain, deduplicated by id). The hydrate examples on the lifecycle hooks and database persistence pages now upsert their conversation row, since head-start first turns run without a preload to create it. **Sessions page.** The page opened with "a durable, task-bound, bi-directional I/O channel pair", which reads as jargon and omitted run orchestration entirely. It now leads with the plain mental model (a pair of durable streams: input carries user messages, output carries everything the agent produces) plus the Session's role orchestrating runs, a diagram, a minimal runnable example, and a section on the one-session-many-runs lifecycle. Documents behavior shipping in [#3907](https://github.com/triggerdotdev/trigger.dev/pull/3907). |
||
|
Eric Allam
|
51af9ae14c |
docs(ai-chat): correct chat.agent reference drift (#3892)
## Summary Accuracy fixes across the AI chat docs: drop the non-existent per-call option from `transport.preload`, clarify that `onValidateMessages` only fires on turns carrying incoming messages, soften the turn-complete token-refresh wording (the header is optional), document the new `onTurnComplete` `error` field and `finishReason`, and correct the idle-timeout default to 30 seconds. |
||
|
Eric Allam
|
b8a576a348 |
docs: document the trigger skills installer (replaces agent rules) (#3871)
## Summary Updates the AI-tooling docs for the new `trigger skills` installer that shipped in #3868. The Skills page now documents `trigger skills` (skills bundled with the CLI, version-matched to your SDK) and the four bundled skills: `authoring-tasks`, `realtime-and-frontend`, `authoring-chat-agent`, `chat-agent-advanced`. The old Agent Rules page becomes a short "rules are now skills" redirect (kept because existing redirects and the CLI link point at it), and the Building with AI overview collapses the three-way Skills/Rules/MCP comparison into Skills vs MCP. Hold until the v4.5 CLI release ships, since `trigger skills` is not on npm until then. |
||
|
Eric Allam
|
97c12e2510 |
docs(management): document TriggerClient for multi-target SDK usage (#3694)
## Summary Docs follow-up for #3683 (`TriggerClient` for per-instance SDK configuration). Adds a dedicated reference page and threads the new pattern through the existing management + preview-branches docs. ## What's in **New page** `docs/management/multiple-clients.mdx` — when to use `TriggerClient` vs `configure()` vs `auth.withAuth`, env-var fallback rules, isolation contract, namespace surface, `inheritContext` opt-in, and a when-to-use-what table. **Updated pages** - `docs/management/authentication.mdx` — rewrote the `auth.withAuth` section to reflect the now-ALS-backed semantics (the prior version warned about concurrency races and pointed at issue #3298 as a tracked fix; that fix landed in #3683). Added `tr_preview_*` to the key prefix list. Reframed the multi-target use case to lead with `TriggerClient`, with `auth.withAuth` as the temporary-override helper. - `docs/management/overview.mdx` — added a `Multiple clients in one process` subsection. - `docs/deployment/preview-branches.mdx` — added a `Triggering across multiple branches from one process` example. - `docs/triggering.mdx` — one-liner pointing at the new page for cross-project triggering. - `docs/docs.json` — slotted `management/multiple-clients` into the Management API nav, right after authentication. Paired with #3683. ## Test plan - [ ] Mintlify preview renders cleanly - [ ] Code samples in each updated page run as documented - [ ] Cross-page links resolve (`/management/multiple-clients`, `/management/authentication`) |
||
|
github-actions[bot]
|
5fab8cafcf |
chore: release v4.5.0-rc.6 (#3870)
🚀 Publish Trigger.dev Docker / units (push) Failing after 4s
🚀 Publish Trigger.dev Docker / typecheck (push) Failing after 5s
🚀 Publish Trigger.dev Docker / publish-worker-v4 (push) Has been skipped
🚀 Publish Trigger.dev Docker / scan-webapp (push) Has been skipped
🚀 Publish Trigger.dev Docker / publish-webapp (push) Has been skipped
🚀 Publish Trigger.dev Docker / publish-worker (push) Has been skipped
🧭 Helm Chart Release / lint-and-test (push) Has been cancelled
🧭 Helm Chart Release / release (push) Has been cancelled
🚀 Publish Trigger.dev Docker / 📣 Dispatch main image (push) Has been cancelled
## Summary 7 improvements, 1 bug fix. ## Improvements - `trigger init` now sets up your AI coding assistant as part of project setup: pick the MCP server, the agent skills, or both, then scaffold with the CLI or hand off to your assistant. Adds a new `getting-started` agent skill that teaches assistants how to bootstrap Trigger.dev (install the SDK, write `trigger.config.ts`, create a first task, run `trigger dev`), so the AI-driven setup path works end to end. It ships in the CLI alongside the existing skills, version-matched to your SDK. ([#3872](https://github.com/triggerdotdev/trigger.dev/pull/3872)) - `dev` and `deploy` now fail with a clear error when two tasks are defined with the same id, including across different task types (e.g. a scheduled task and a regular task sharing an id). Previously the second definition silently overwrote the first, so one of the tasks would vanish with no warning. Task ids are detected as duplicates during indexing (naming each offending id and the files it was found in), and the same rule is enforced server-side when the background worker is registered. ([#3865](https://github.com/triggerdotdev/trigger.dev/pull/3865)) - `trigger skills` installs Trigger.dev agent skills into your coding agent so it knows how to write tasks, schedules, realtime, and chat.agent code. The skills ship with the CLI and are copied into each tool's native skills directory (Claude Code, Cursor, GitHub Copilot, and Codex / AGENTS.md), and `trigger dev` offers to install them on first run. ([#3868](https://github.com/triggerdotdev/trigger.dev/pull/3868)) - Reliability fixes for `chat.agent`. A user message sent while the agent is streaming is no longer delivered twice (which could run a duplicate turn), input appends now carry an idempotency key so a retried send can't duplicate a message, stopping a generation clears the streaming state so a page reload doesn't replay the stopped turn, and runs can now carry the full set of dashboard tags instead of being silently truncated. `onTurnComplete` now fires on errored turns (with the thrown error attached) and the failed turn's user message is persisted so it isn't lost on the next run. Custom agents and manual `chat.writeTurnComplete` callers now trim the output stream, sending a custom action no longer leaves a second stream reader running, and a long-lived `watch` subscription no longer grows its dedupe set without bound. ([#3891](https://github.com/triggerdotdev/trigger.dev/pull/3891)) - Continuation chat boots no longer stall for around 10 seconds before the first turn. The `session.in` resume cursor is now found with a non-blocking records read instead of draining an SSE long-poll (which always waited out its full 5 second inactivity window, twice per boot), the boot reads run concurrently, and chat snapshots carry the cursor so subsequent boots skip the scan entirely. ([#3907](https://github.com/triggerdotdev/trigger.dev/pull/3907)) - Record client-side dequeue API latency in the supervisor consumer pool as a Prometheus histogram (`queue_consumer_pool_dequeue_duration_seconds`, labelled by `outcome`: success/empty/error). ([#3887](https://github.com/triggerdotdev/trigger.dev/pull/3887)) - Add `GetProjectEnvironmentsResponseBody` and `ProjectEnvironment` schemas for the new `GET /api/v1/projects/{projectRef}/environments` endpoint, which lists the parent environments (dev, staging, preview, prod) a personal access token can access for a project. Dev is scoped to the token owner and branch (preview child) environments are excluded. ([#3880](https://github.com/triggerdotdev/trigger.dev/pull/3880)) ## Bug fixes - Fix two `chat.createSession()` bugs: stopping a generation no longer wedges the run (the turn loop raced a `totalUsage` promise that never settles after a stop-abort), and continuation runs now wait for the next message instead of invoking the model with an empty prompt. ([#3920](https://github.com/triggerdotdev/trigger.dev/pull/3920)) <details> <summary>Raw changeset output</summary> ⚠️⚠️⚠️⚠️⚠️⚠️ `main` is currently in **pre mode** so this branch has prereleases rather than normal releases. If you want to exit prereleases, run `changeset pre exit` on `main`. ⚠️⚠️⚠️⚠️⚠️⚠️ # Releases ## @trigger.dev/build@4.5.0-rc.6 ### Patch Changes - Updated dependencies: - `@trigger.dev/core@4.5.0-rc.6` ## trigger.dev@4.5.0-rc.6 ### Patch Changes - `trigger init` now sets up your AI coding assistant as part of project setup: pick the MCP server, the agent skills, or both, then scaffold with the CLI or hand off to your assistant. Adds a new `getting-started` agent skill that teaches assistants how to bootstrap Trigger.dev (install the SDK, write `trigger.config.ts`, create a first task, run `trigger dev`), so the AI-driven setup path works end to end. It ships in the CLI alongside the existing skills, version-matched to your SDK. ([#3872](https://github.com/triggerdotdev/trigger.dev/pull/3872)) - `dev` and `deploy` now fail with a clear error when two tasks are defined with the same id, including across different task types (e.g. a scheduled task and a regular task sharing an id). Previously the second definition silently overwrote the first, so one of the tasks would vanish with no warning. Task ids are detected as duplicates during indexing (naming each offending id and the files it was found in), and the same rule is enforced server-side when the background worker is registered. ([#3865](https://github.com/triggerdotdev/trigger.dev/pull/3865)) - `trigger skills` installs Trigger.dev agent skills into your coding agent so it knows how to write tasks, schedules, realtime, and chat.agent code. The skills ship with the CLI and are copied into each tool's native skills directory (Claude Code, Cursor, GitHub Copilot, and Codex / AGENTS.md), and `trigger dev` offers to install them on first run. ([#3868](https://github.com/triggerdotdev/trigger.dev/pull/3868)) ```bash trigger skills --target claude-code ``` Replaces the previous `install-rules` command, which stays as an alias. - Updated dependencies: - `@trigger.dev/core@4.5.0-rc.6` - `@trigger.dev/build@4.5.0-rc.6` - `@trigger.dev/schema-to-json@4.5.0-rc.6` ## @trigger.dev/core@4.5.0-rc.6 ### Patch Changes - Reliability fixes for `chat.agent`. A user message sent while the agent is streaming is no longer delivered twice (which could run a duplicate turn), input appends now carry an idempotency key so a retried send can't duplicate a message, stopping a generation clears the streaming state so a page reload doesn't replay the stopped turn, and runs can now carry the full set of dashboard tags instead of being silently truncated. `onTurnComplete` now fires on errored turns (with the thrown error attached) and the failed turn's user message is persisted so it isn't lost on the next run. Custom agents and manual `chat.writeTurnComplete` callers now trim the output stream, sending a custom action no longer leaves a second stream reader running, and a long-lived `watch` subscription no longer grows its dedupe set without bound. ([#3891](https://github.com/triggerdotdev/trigger.dev/pull/3891)) - Continuation chat boots no longer stall for around 10 seconds before the first turn. The `session.in` resume cursor is now found with a non-blocking records read instead of draining an SSE long-poll (which always waited out its full 5 second inactivity window, twice per boot), the boot reads run concurrently, and chat snapshots carry the cursor so subsequent boots skip the scan entirely. ([#3907](https://github.com/triggerdotdev/trigger.dev/pull/3907)) - Record client-side dequeue API latency in the supervisor consumer pool as a Prometheus histogram (`queue_consumer_pool_dequeue_duration_seconds`, labelled by `outcome`: success/empty/error). ([#3887](https://github.com/triggerdotdev/trigger.dev/pull/3887)) - `dev` and `deploy` now fail with a clear error when two tasks are defined with the same id, including across different task types (e.g. a scheduled task and a regular task sharing an id). Previously the second definition silently overwrote the first, so one of the tasks would vanish with no warning. Task ids are detected as duplicates during indexing (naming each offending id and the files it was found in), and the same rule is enforced server-side when the background worker is registered. ([#3865](https://github.com/triggerdotdev/trigger.dev/pull/3865)) - Add `GetProjectEnvironmentsResponseBody` and `ProjectEnvironment` schemas for the new `GET /api/v1/projects/{projectRef}/environments` endpoint, which lists the parent environments (dev, staging, preview, prod) a personal access token can access for a project. Dev is scoped to the token owner and branch (preview child) environments are excluded. ([#3880](https://github.com/triggerdotdev/trigger.dev/pull/3880)) ## @trigger.dev/python@4.5.0-rc.6 ### Patch Changes - Updated dependencies: - `@trigger.dev/sdk@4.5.0-rc.6` - `@trigger.dev/core@4.5.0-rc.6` - `@trigger.dev/build@4.5.0-rc.6` ## @trigger.dev/react-hooks@4.5.0-rc.6 ### Patch Changes - Updated dependencies: - `@trigger.dev/core@4.5.0-rc.6` ## @trigger.dev/redis-worker@4.5.0-rc.6 ### Patch Changes - Updated dependencies: - `@trigger.dev/core@4.5.0-rc.6` ## @trigger.dev/rsc@4.5.0-rc.6 ### Patch Changes - Updated dependencies: - `@trigger.dev/core@4.5.0-rc.6` ## @trigger.dev/schema-to-json@4.5.0-rc.6 ### Patch Changes - Updated dependencies: - `@trigger.dev/core@4.5.0-rc.6` ## @trigger.dev/sdk@4.5.0-rc.6 ### Patch Changes - Reliability fixes for `chat.agent`. A user message sent while the agent is streaming is no longer delivered twice (which could run a duplicate turn), input appends now carry an idempotency key so a retried send can't duplicate a message, stopping a generation clears the streaming state so a page reload doesn't replay the stopped turn, and runs can now carry the full set of dashboard tags instead of being silently truncated. `onTurnComplete` now fires on errored turns (with the thrown error attached) and the failed turn's user message is persisted so it isn't lost on the next run. Custom agents and manual `chat.writeTurnComplete` callers now trim the output stream, sending a custom action no longer leaves a second stream reader running, and a long-lived `watch` subscription no longer grows its dedupe set without bound. ([#3891](https://github.com/triggerdotdev/trigger.dev/pull/3891)) - Continuation chat boots no longer stall for around 10 seconds before the first turn. The `session.in` resume cursor is now found with a non-blocking records read instead of draining an SSE long-poll (which always waited out its full 5 second inactivity window, twice per boot), the boot reads run concurrently, and chat snapshots carry the cursor so subsequent boots skip the scan entirely. ([#3907](https://github.com/triggerdotdev/trigger.dev/pull/3907)) - Fix `chat.headStart` when `hydrateMessages` is registered. The warm route's step-1 partial now reaches the agent's accumulator on the hydrate path, so `onTurnComplete` carries the full first turn (the head-start user message included), tool-call handovers resume from step 2 instead of re-running step 1, and the assistant `messageId` stays stable across the handover. ([#3907](https://github.com/triggerdotdev/trigger.dev/pull/3907)) - Preserve reasoning parts across the `chat.headStart` handover. Extended-thinking models' step-1 reasoning now lands in the durable session history (and `onTurnComplete`) under the same assistant `messageId`, with provider metadata intact so Anthropic thinking signatures survive replays. ([#3907](https://github.com/triggerdotdev/trigger.dev/pull/3907)) - Fix two `chat.createSession()` bugs: stopping a generation no longer wedges the run (the turn loop raced a `totalUsage` promise that never settles after a stop-abort), and continuation runs now wait for the next message instead of invoking the model with an empty prompt. ([#3920](https://github.com/triggerdotdev/trigger.dev/pull/3920)) - Updated dependencies: - `@trigger.dev/core@4.5.0-rc.6` ## @trigger.dev/plugins@4.5.0-rc.6 ### Patch Changes - Updated dependencies: - `@trigger.dev/core@4.5.0-rc.6` </details> Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>helm-v4.5.0-rc.6 v.docker.4.5.0-rc.6 v4.5.0-rc.6 |
||
|
Iss
|
002c441f50 |
feat(webapp): self serve schedules add-on (#3811)
Adds the purchase UI for extra schedules, mirroring preview branches ## Changes - `setSchedulesAddOn` platform client + `SetSchedulesAddOnService` (purchase + quota-increase via Plain). - `ScheduleListPresenter` surfaces add-on / quota / pricing; `checkSchedule` counts purchased schedules toward the limit (`base + purchased`). - `PurchaseSchedulesModal` on the Schedules page — bought in **bundles of 1,000 ($10/mo each)**; bundle increments enforced client-side and in the action's zod schema. |
||
|
Eric Allam
|
47834198fc |
fix(sdk): stop chat.createSession wedging on stop and erroring on continuation boots (#3920)
## Summary
Two `chat.createSession()` bugs that break chats at its abstraction
level:
1. **Stopping a generation wedged the run forever.** `turn.complete()`
bare-awaited the AI SDK's `totalUsage` promise, which never settles
after a stop-abort. The run stayed stuck inside the stopped turn (trace
shows a permanently partial `ai.streamText` span and no further `waiting
for next message`), so the chat could never take another message. Fixed
with the same 2s `Promise.race` guard `chat.agent`'s turn loop already
uses.
2. **Continuation runs invoked the model with an empty prompt.** The
first turn only waited for a message on `preload` boots. A continuation
run (spawned after a cancel, crash, or version upgrade) arrives with the
boot payload stripped, so the loop ran a turn with zero messages and
errored with `AI_InvalidPromptError: messages must not be empty`.
Message-less continuation boots now wait for the next session input
("waiting for first message (continuation)"), and `turn.continuation` is
preserved across the wait so user code can seed stored history off it.
Both reproduced and verified end-to-end against a live environment (stop
followed by a next turn; cancel followed by a continuation turn with
seeded history), plus the existing unit suite.
|
||
|
Eric Allam
|
a04cdffda6 |
fix(webapp): stop replica lag from double-triggering session runs and 404ing fresh sessions (#3914)
## Summary Two read-replica races on the session APIs could break chats whose first activity lands inside the replication window (or any time the replica lags): 1. A session's first `.in` append or `.out` subscribe could fail with a 404 for a session that exists on the writer, because the route resolved the Session row on the replica only. 2. `ensureRunForSession` probed run liveness on the replica, so a probe miss on a run triggered moments earlier was judged "run is dead" and a second live run was spawned for the same session. Both runs then consumed the same input stream, producing duplicated turns and doubled responses (and doubled LLM cost). ## Fix Liveness now re-probes the writer before declaring the current run dead (the old code already fell back to the writer, but only to recover the friendlyId, after the wrong verdict was made). Session resolution on the append and subscribe/init routes goes through a new `resolveSessionWithWriterFallback`, which stays replica-first on the hot path and only touches the writer on a miss. Reproduced and verified against a local streaming replica with an artificial apply delay: pre-fix, a send immediately after session creation reliably produced either the 404 or two executing runs with a doubled response; post-fix, the same flow produces exactly one run and one response. Also rides along: the local docker replica's default apply delay drops from 150ms to a realistic 20ms (override via `REPLICA_APPLY_DELAY` when you want to deliberately widen the race window). |
||
|
Matt Aitken
|
eb498d137f |
fix(plugins): drop unused gitBranch re-export from the package entry (#3923)
`@trigger.dev/plugins` re-exported `sanitizeBranchName`/`isValidGitBranchName` from `@trigger.dev/core` as a convenience forwarder. Nothing actually imports them through this package — every consumer (webapp, `@trigger.dev/rbac`, …) imports them directly from `@trigger.dev/core/v3/utils/gitBranch`. Removing the forwarder keeps the package entry free of **runtime** core imports (only type re-exports + `buildJwtAbility` remain), so consumers that bundle `@trigger.dev/plugins` from source don't pull an unrelated core subpath into their build. No behavior change; the helpers remain available from `@trigger.dev/core` where they're defined. |
||
|
Matt Aitken
|
f48c89752c |
perf(webapp): parallelize streaming batch-item ingest (#3777)
## Problem
The item-streaming endpoint of the two-phase batch API (`POST
/api/v3/batches/:batchId/items`) processed streamed items strictly
sequentially. For a batch of many large payloads, each offloaded to
object storage inline, this serialized N object-store round-trips inside
a single request and could exceed Node's default `server.requestTimeout`
(300s). The webapp then returned `408`, which the SDK reads as `408
terminated` and retries up to 5 times, turning a slow ingest into a
failure that takes tens of minutes to surface.
## Fix
Ingest now runs through `p-map` over the NDJSON async iterable with
bounded concurrency (`STREAMING_BATCH_INGEST_CONCURRENCY`, default 10):
- `p-map` pulls lazily from the stream, so at most `concurrency` items
are read and in-flight at once. Peak memory stays bounded to roughly
`concurrency × STREAMING_BATCH_ITEM_MAXIMUM_SIZE` and request-body
backpressure is preserved.
- Set the env to `1` for fully sequential ingestion (escape hatch).
## Why this is safe (ordering and idempotency unchanged)
- Ordering derives from each item's index (enqueue `timestamp =
batch.createdAt + index`), not enqueue order.
- Dedup is atomic per index in `enqueueBatchItem`.
- The NDJSON parser now stamps oversized-item markers with their emit
position, removing the consumer's sequential `lastIndex` assumption (the
only order-dependent bit).
- The count-check and conditional-seal path is untouched.
## Scope
This speeds up every batch ingested through the streaming endpoint, not
just large-payload batches. Each item does a per-item Redis enqueue
regardless of size, and those now overlap. Large payloads benefit most
because they add an object-store offload round-trip on top of the
enqueue.
## Verification
Added an integration test (`streamBatchItems.test.ts`) that drives the
real service against Postgres + Redis + RunEngine and times a 150-item
batch at increasing concurrency. Object-store offload is modelled as a
fixed per-item latency (local round-trips are too small to compare
meaningfully):
```
runCount=150
large payloads (10ms/item offload):
concurrency=1 1739ms
concurrency=10 192ms (9.1x faster)
concurrency=50 57ms (30.7x faster)
small payloads (Redis enqueue only, no offload):
concurrency=1 90ms
concurrency=10 24ms (3.7x faster)
```
The test asserts correctness at every concurrency (all items accepted,
sealed, enqueued exactly once), that parallel ingest beats the
sequential floor, and that the small-payload case is strictly faster
than sequential, so the win is not specific to large payloads.
Also exercised end-to-end over real HTTP against a local server: a
20-item batch (12MB body) ingests and seals, a re-stream of the sealed
batch returns `sealed: true` with zero re-accepted items (idempotent
retry), and an oversized item still seals at its correct index.
Existing coverage stays green: concurrent ingest of a 100-item batch,
in-flight processing never exceeding the configured concurrency,
concurrent dedup on streaming retry, and emit-position marker indexing.
## Follow-ups (not in this PR)
- SDK pre-offload of large item payloads (send `application/store` refs
instead of raw blobs) to remove object-store work from the request hot
path and shrink the request body.
- Optional `server.requestTimeout` bump as a safety net.
## CI fix
Added `.github/workflows/codeql.yml` to replace GitHub's automatic
("dynamic") CodeQL scanning. The dynamic setup was failing to upload
SARIF results because the auto-generated `GITHUB_TOKEN` lacked the
`security-events: write` permission. The explicit workflow grants that
permission at the job level and pins all actions to commit SHAs,
consistent with the repo's security conventions.
## ✅ Checklist
- [ ] I have followed every step in the [contributing
guide](https://github.com/triggerdotdev/trigger.dev/blob/main/CONTRIBUTING.md)
- [ ] The PR title follows the convention.
- [ ] I ran and tested the code works
---
## Testing
- Integration test (`streamBatchItems.test.ts`) validates correctness
and performance at concurrency 1, 10, and 50 for both large and small
payloads.
- End-to-end verified over real HTTP: 20-item/12MB batch ingests and
seals, idempotent retry returns `sealed: true`, oversized item seals at
correct index.
---
## Changelog
Streaming batch ingest now processes items with bounded concurrency
instead of one at a time, so batches of many large payloads ingest far
faster and no longer time out. Concurrency is configurable via
`STREAMING_BATCH_INGEST_CONCURRENCY` (default 10); set it to 1 for fully
sequential ingestion.
---
## Screenshots
_[Screenshots]_
💯
---------
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
|
||
|
Matt Aitken
|
5d6ea33166 |
refactor: share the public-token JWT scope decoder; make @trigger.dev/plugins internal (#3919)
## What `buildJwtAbility` — the decoder for public-token scope strings (`read:tags:…`, `read:runs:run_abc`, `admin`, …) — now lives in `@trigger.dev/plugins` as the single source of truth. `@trigger.dev/rbac` re-exports it, so the built-in fallback and any auth plugin interpret a token identically. Scope strings are split on only the first **two** colons (`action:type:id`), so a resource id that itself contains colons — e.g. a tag like `user:123` — is matched in full rather than truncated to its first segment. (The fallback already did this; this makes it the one shared implementation.) `@trigger.dev/plugins` is now **private (unpublished)** and gains a `@triggerdotdev/source` export condition, so consumers bundle it from source per-commit like `@trigger.dev/core` instead of resolving a published version — no cross-version coordination. ## Why Two hand-maintained copies of the scope grammar drift, and the difference silently changes what a token grants. One shared decoder removes that class of bug. ## Notes - No changeset: `@trigger.dev/plugins` is now private and `@trigger.dev/rbac` is internal — neither is published. - Unit coverage for the colon-id path lives in `internal-packages/rbac/src/ability.test.ts` (now exercising the shared function). |
||
|
Eric Allam
|
78b7136bf7 |
chore: vouch saasjesus as a contributor (#3917)
Vouches `saasjesus` as a contributor (vouch request #3915) so their PRs clear the vouch check instead of being auto-closed. |
||
|
nicktrn
|
de8231cb9d |
chore: bump shell-quote to 1.8.4 (#3913)
Refreshes the locked `shell-quote` to 1.8.4 (transitive, lockfile-only). <!-- This is an auto-generated description by cubic. --> <a href="https://cubic.dev/pr/triggerdotdev/trigger.dev/pull/3913?utm_source=github" target="_blank" rel="noopener noreferrer" data-no-image-dialog="true"><picture><source media="(prefers-color-scheme: dark)" srcset="https://www.cubic.dev/buttons/review-in-cubic-dark.svg"><source media="(prefers-color-scheme: light)" srcset="https://www.cubic.dev/buttons/review-in-cubic-light.svg"><img alt="Review in cubic" src="https://www.cubic.dev/buttons/review-in-cubic-dark.svg"></picture></a> <!-- End of auto-generated description by cubic. --> |
||
|
Eric Allam
|
954ee5c572 |
fix(webapp): deliver realtime changes with current content when the read replica lags (#3910)
## Summary When the realtime runs feed (the backend behind the `realtimeBackend` feature flag) hydrates a change from a Postgres read replica, the read can race the replica's apply of the very write that triggered it. The delivered row then carries the previous change's content, and an isolated final change (for example a last `metadata.set` before a run goes quiet) is not corrected until the roughly 20 second backstop poll. Measured against a replica with deliberate apply delay, every delivery trailed exactly one change behind and a final change stranded for the full backstop interval. ## Fix Publishers stamp each change record with the committed row's `updatedAt`, taken from writes they already perform, so the stamp costs no extra queries. The router delays its wake hydrate until the replica's measured lag has passed, anchored to that timestamp: a record that has already spent longer than the lag in transit is hydrated immediately, so only the racing leading edge ever waits. After hydrating, a tripwire compares each row against its record's watermark. Still-stale rows are withheld and retried briefly, and each detection feeds the lag estimate. If retries run out, the rows are delivered anyway (liveness over freshness) and follow-up re-hydrates emit the fresh version through the normal working-set diff once the replica catches up, with the backstop as the terminal net. Replica lag is sampled reader-side only, and only while feeds are active. Aurora reports live lag via `aurora_replica_status()`; vanilla Postgres can only report "caught up or not" (mid-apply lag is not honestly measurable from a replica), so tripwire observations floor the estimate there. Deployments without a replica resolve to zero lag and skip the gate entirely. Tunables live under `REALTIME_BACKEND_NATIVE_REPLICA_LAG_*`, and `realtime_native.stale_hydrates` plus `realtime_native.replica_lag_estimate_ms` make replica health observable. Two adjacent fixes: a metadata update that writes nothing no longer publishes a change record, and buffered parent and root metadata operations now publish when the flusher writes them, so those changes wake live feeds instead of waiting for the backstop. For local testing, `docker-compose` gains an opt-in `database-replica` service (compose profile `replica`) with a configurable `recovery_min_apply_delay`, which reproduces replica-lag behavior deterministically. With the gate disabled this rig reproduces the one-change-behind delivery exactly; with it enabled, deliveries arrive with current content at roughly the true replica lag, across write rates faster and slower than the lag itself. |
||
|
Eric Allam
|
8dc77c0ccd |
fix(webapp): only load env var values for displayed environments (#3903)
## Summary The environment variables page loaded every variable value in the project, unfiltered by environment. Archiving a preview branch does not delete its environment variable value rows, so projects that churn preview branches accumulate values forever, and every page view loaded all of them. On large projects this made the page loader take many seconds and stalled the server while deserializing the oversized result. ## Fix The presenter now loads the displayed environments first and filters the `values` relation to those environment IDs. That matches the display semantics exactly (per-user dev environments and active branch environments included), and the lookup is covered by the existing unique index on `(variableId, environmentId)`. Values in archived branch environments are no longer fetched at all. Covered by a new testcontainers test asserting that values from active environments (including branch environments) are returned while archived branch environments are excluded. |
||
|
Eric Allam
|
cc9eabd14d |
test(webapp): use relative fixture dates in runs cursor pagination tests (#3912)
## Summary `test/runsRepositoryCursor.test.ts` pinned its fixture runs to `createdAt = 2026-06-04T16:55:07Z`. `listRuns` applies the default 7 day window when no time filter is given, so the fixtures aged out of the window at 16:55 UTC on 2026-06-11 and all five tests started failing for every branch, regardless of what the branch changed. The tests were green on their own CI two days earlier because the fixtures were only five days old at the time. This switches the fixture base to a relative timestamp (one hour ago), so the fixtures stay inside the default window permanently. Verified the suite goes 5/5 green with this change on the same environment where the pinned dates fail 5/5. |
||
|
Eric Allam
|
187c0476c3 |
perf(webapp): shrink run trace loader payload and add trace span cap controls (#3906)
## Summary The run trace page loader serialized every span's raw OTel events (with full properties) into the response, even though the tree UI only renders the derived `timelineEvents` and the span detail panel refetches what it needs. On event-heavy traces that inflated both the loader payload and the server-side heap copies built per request. This PR keeps raw span events server-side and pairs that with a few related trace-view improvements: - A new optional `TRACE_VIEW_EMERGENCY_SPAN_CAP` env var (unset by default) clamps the trace summary and detailed trace summary span limits on both event store paths, including the public run trace endpoint, so operators can bound trace query sizes in one place without retuning the per-store limits. - The TreeView virtualizer resolved every rendered row with a linear scan over the whole tree (and `getNodeProps` did the same via `findIndex`); rows now resolve through memoized id lookup maps, which matters once traces reach tens of thousands of spans. - The run stream SSE lookup now applies the same organization membership scoping as the rest of the run page presenters, for consistency. Behavior is unchanged by default: the trace tree renders from the same `timelineEvents` it always has, and the new cap only takes effect when set. |
||
|
Eric Allam
|
2b6d2492fe |
fix(sdk,core): head-start handover correctness and continuation boot latency (#3907)
## Summary Three related fixes for `chat.headStart` and continuation boots, found while investigating customer reports. **1. `chat.headStart` now works with `hydrateMessages`.** The turn-0 handover splice only ran on the default accumulation path, so agents registering `hydrateMessages` silently lost the warm route's step-1 response: pure-text turns fired `onTurnComplete` with no assistant message (and an empty durable write), tool-call turns re-ran step 1 from scratch under a fresh `messageId`, and the head-start user message never reached the hydrate hook at all. The first-turn history now reaches `hydrateMessages` as `incomingMessages`, and the splice runs after both accumulation branches, deduplicated by the handover `messageId`. **2. Reasoning parts survive the handover.** The synthesized partial only mapped text and tool-call parts, so an extended-thinking model's step-1 reasoning streamed to the browser but never reached durable history. Reasoning parts now map through with provider metadata, so Anthropic thinking signatures survive a UIMessage round trip on hydrate replays. **3. Continuation boots no longer stall for ~10 seconds.** The `.in` resume cursor was found by draining an SSE subscription that only closes after its 5 second inactivity window, and the scan ran twice per boot. It is now a non-blocking records read of the latest turn-complete header, runs at most once per boot, the boot reads run concurrently, and chat snapshots carry the cursor so subsequent boots skip the scan entirely. Measured locally on a cancel-then-continue repro: pre-turn continuation latency dropped from ~11s to ~0.5s. Every fix was verified red-green: new unit tests reproduced each failure before the fix, and end-to-end smoke tests against a live local stack covered both handover legs, reasoning persistence with extended thinking (including a follow-up turn that round-trips the persisted signed reasoning back to the provider), and the boot timing comparison. ## Rollout SDK-only; no server change required. A new SDK against a server that does not serialize record headers degrades to the existing no-cursor fallback. Old SDKs ignore the new snapshot field, and new SDKs fall back to the records scan on snapshots written before it existed. |
||
|
DKP
|
93b4715967 |
feat(webapp): hipaa baa add-on on paid pricing tiers (#3904)
## Summary HIPAA BAA is offered as a paid add-on on every paid plan. Each paid tier on the in-app pricing card now has a "HIPAA BAA add-on" row with a "Request a BAA" link that opens the existing contact dialog pre-filled with a new `hipaa` inquiry type, prompting the user for their company name and a brief description of the PHI workload. The contact form's `feedbackTypes` are restructured to match the marketing /contact form: every inquiry type carries a Plain label ID and a "Contact form: ..." thread title, so threads land in Plain identically whether they come from the dashboard or the marketing site. The included-compute line on each tier also picks up the credits wording from the marketing pricing page, and the Enterprise tier lifts its title above the features row. |
||
|
Saadi Myftija
|
d0b2d79b3b |
fix(supervisor): cancel pending delayed snapshots when the run completes or disconnects (#3894)
⚒️ Publish Worker (v4) / build (supervisor) (push) Has been cancelled
The compute suspend flow delays snapshots by `snapshotDelayMs` (~30s) so short-lived waitpoints skip the snapshot entirely, with the intent that a run continuing before the delay expires cancels the pending snapshot. But the only `cancel()` call site was the `/continue` action, which runners only invoke when restoring from an already-taken snapshot — so pending snapshots were never cancelled (zero `snapshot.canceled` events ever emitted in prod). When a run resumed and completed inside the window, the stale snapshot fired ~30s later anyway, pausing the VM 6–13s mid warm-start long-poll; the frozen guest couldn't fire its abort timer or send a FIN, causing stalls and run-engine driven retries. ### Change - Cancel the pending snapshot on `attempt.complete` — after the platform accepts the completion, before the HTTP reply (so it can't reorder with the runner's next `/suspend`). - Cancel on `runDisconnected` (crash, exit, or run replaced on the socket). - Both cancels are guarded by a runnerId match (new `TimerWheel.peek()`): a stale duplicate runner for a reassigned run must not cancel the fresh runner's pending snapshot. A missing runnerId falls through to an unconditional cancel (the pre-existing `/continue` behavior is unchanged). Waitpoint suspensions keep the runner socket connected and the attempt incomplete, so neither hook touches a snapshot that is still wanted. Known limitation (fail-safe direction): `socket.data.runnerId` is frozen at the websocket handshake, so after a same-supervisor restore the disconnect-path guard refuses the cancel. The `attempt.complete` path uses the runner's current header id and is unaffected.re2-prod-supervisor-tri-10293 re2-test-supervisor-tri-10293 |
||
|
Saadi Myftija
|
2397ca2999 |
fix(supervisor): retry transient instance create failures in compute workload manager (#3902)
`ComputeWorkloadManager.create` swallows gateway errors currently, so a cold start that fails placement (e.g. a netns slot with a busy tap, a full node disk) silently abandons the dequeued run until the run engine's `PENDING_EXECUTING` heartbeat timeout redrives it via stall detection. ### Changes - Retry `instances.create` with short backoff (default 3 attempts, 250ms backoff), recording `createAttempts` in the wide event. - **Only statuses where the create definitely did not commit are retried**: 500 (agent/fcrun create failed) and 503 (no placement). 502/504 are excluded — the gateway emits those when it fails to reach the node or read its response, which can happen *after* the agent committed the create; the gateway only records the instance name on a clean 201, so a same-name retry would miss the collision check and could double-create the VM on another node. Network-level fetch failures are retried (if the gateway processed the create, its name index is populated and the retry 409s harmlessly). Timeouts are not retried. - **Retry attempts after a 5xx use a deterministic `-rN` name suffix**: a failed create can leave its name registered until async cleanup runs. Attempt 1 keeps the unsuffixed name. |
||
|
Daniel Sutton
|
7b4443a437 |
test(webapp): stop streamBatchItems container tests timing out on cold start (#3900)
Fixes an intermittent `Test timed out in 30000ms` in the `streamBatchItems` suite. Not a logic hang — the 30s budget covers container setup, and each case boots its own per-test Redis container + a full `RunEngine`, so under CI Docker contention a cold boot can cross 30s (which is why the failure moved between tests). - New `containerTestWithIsolatedRedisNoClickhouse` fixture (Postgres clone + per-test Redis, no ClickHouse) — this suite never uses ClickHouse, but the old fixture's auto `resetClickhouse` forced a ClickHouse boot + migration onto the cold-start test. - Raised `testTimeout` 30s → 120s, matching the run-engine convention for this footprint. |
||
|
Oskar Otwinowski
|
1c7e64acde | feat(supervisor): stamp org identity label on compute microVMs (#3899) | ||
|
Eric Allam
|
f5f29ceb26 |
fix(sdk,core): chat.agent delivery, idempotency, and recovery fixes (#3891)
## Summary A batch of reliability fixes for `chat.agent`: - A user message sent while the agent is streaming is no longer delivered twice (which could run a duplicate turn). - Input appends carry an idempotency key (`X-Part-Id`) so a retried send can't duplicate a message. - `onTurnComplete` now fires on errored turns with the thrown error attached, and the failed turn's user message is persisted so it isn't lost on the next run. - Stopping a generation clears the streaming state, so a page reload doesn't replay the stopped turn. - Custom agents and manual `chat.writeTurnComplete` callers trim the output stream, sending a custom action no longer leaves a second stream reader running, a long-lived `watch` subscription no longer grows its dedupe set without bound, promoting a queued message to steering no longer risks a double-send, and runs keep the full set of dashboard tags. The `X-Part-Id` header is accepted by current servers (they just don't dedupe on it yet), so this is safe to ship ahead of the matching server change. |
||
|
Eric Allam
|
b82d100483 |
fix(webapp): harden the realtime session routes (#3890)
## Summary Reliability and authorization fixes for realtime chat sessions: - Session-stream waitpoint delivery is scoped to the environment, so two environments using the same session `externalId` can no longer complete each other's waitpoints. - The session snapshot-url routes now enforce per-session authorization, and appending to a session's `out` channel requires secret-key auth, so a session-scoped token can't read another session's snapshot or forge assistant output. - Appends that carry an `X-Part-Id` header are deduplicated on retry, so a retried send can't duplicate a message. - Session creation rejects expired sessions (instead of triggering a run that can never receive input), `externalId` is immutable after creation, and the sessions list endpoint returns friendly `run_*` ids to match the single-session routes. ## Rollout The waitpoint cache key gains an environment prefix. To keep waitpoints registered by the previous deploy working across the boundary, the drain reads both the new and the previous key for this release; the legacy read can be removed a release later once no pre-deploy waitpoints remain. |
||
|
Eric Allam
|
f9d57d3bd5 |
feat(webapp): add a new backend for the realtime runs feed (#3864)
## Summary Adds a second backend for the realtime runs feed (`useRealtimeRun`, `subscribeToRunsWithTag`, `subscribeToBatch`), built to stay healthy when a single busy environment has many subscribers watching many runs at once. It is gated behind a feature flag with the existing backend as the default, so nothing changes for users until it is enabled per environment. ## Design A run change is published once, as a small self-describing record, to a single per-environment channel. Every feed is then a predicate over that one stream rather than owning a channel: - A per-instance router indexes the currently-held feeds by run, tag, and batch. When a run changes it hydrates the affected rows once and serializes them once, then fans the result to every matching feed. One hot shared tag watched by many subscribers costs a single database query and serialize, not one per subscriber. - Feeds that don't match a change are never woken, wake delivery per environment is coalesced on a leading edge (250ms default) so a burst of changes costs one wake, and cold reads coalesce onto a single short-TTL-cached resolve. - An admission gate bounds how many cold ClickHouse resolves run concurrently, so a mass reconnect across many distinct filters queues instead of stampeding the database. - Changes that land while a client is between long-polls are delivered on its next poll instead of waiting for the periodic backstop: each environment buffers its recent change records, subscriptions linger briefly after the last feed closes, and a newly-armed poll replays exactly the connection's gap. - The per-connection replay cursors behind that are shared across instances via Redis (a single timestamp each), so a poll landing on a different instance behind the load balancer still reads the connection's true gap instead of falling back to a cold resolve. Cursor reads have a bounded deadline and degrade to the cold-read path on any Redis trouble. - Tag subscriptions with multiple tags match runs carrying all of the tags, mirroring the existing backend's filter semantics, and live long-polls hold for about 20 seconds to match its cadence. - The per-environment channel supports Redis Cluster sharded pub/sub, so the wake path scales horizontally across shards by environment. - The backend reports its health through OpenTelemetry metrics (delivery lag, poll resolution paths, backstop outcomes, replay and cursor-store activity), with a provisioned Grafana dashboard for local development. Everything is behind the feature flag and tunable via env vars; the existing backend remains the default. |
||
|
Daniel Sutton
|
6afc9bfa4c |
fix(run-engine): retry getSnapshotsSince on the replica then primary when the read replica lags (#3889)
## Summary
When `RUN_ENGINE_READ_REPLICA_SNAPSHOTS_SINCE_ENABLED` is on,
`RunEngine.getSnapshotsSince` reads from the read replica. During write
spikes the replica can briefly lag, so the snapshot id a runner just
learned from the writer isn't visible there yet: the lookup threw, the
worker route returned a 500, and the runner waited for its next poll —
turning sub-second snapshot notifications into poll-interval latency
exactly when things are busiest. This PR makes the flag safe to enable:
a replica miss of the since snapshot gets one jittered retry on the
replica (most lag windows are shorter than the ~50–200ms wait, so the
writer is never touched), then falls back to the primary, observed via a
new `run_engine.snapshots_since.replica_miss` counter with an `outcome`
attribute (`replica_retry` vs `primary`). Only genuine misses — absent
on the primary too — remain errors.
## Design
- `getExecutionSnapshotsSince` now throws a typed
`ExecutionSnapshotNotFoundError` so the engine can distinguish the
expected lag miss from real failures. The message string is unchanged
and the error never leaves the engine.
- The recovery path only engages when the flag is on, a distinct replica
client is configured, and no transaction client was passed. With the
flag off, the path is behaviorally identical to before.
- Retry delay bounds are configurable
(`RUN_ENGINE_SNAPSHOTS_SINCE_REPLICA_RETRY_MIN_MS`/`MAX_MS`, default
50/200; `MAX_MS=0` skips the replica retry and goes straight to the
primary).
- The warn log fires only when the primary serves the read (the writer
spill is the operationally interesting event); replica-retry recoveries
are counted but quiet. A permanently-missing snapshot id stays an
error-level failure with a `failedDuring` field, so lag metrics aren't
polluted by bogus ids.
- Stale-tail lag (replica has the since snapshot but not newer rows)
deliberately still returns the replica's view; the next poll catches up.
- The since-snapshot anchor lookup is now scoped to the polled run
(`where: { id, runId }`), so a snapshot id from a different run raises
not-found instead of silently anchoring a too-wide window of the run's
snapshots.
## Test plan
All vitest + testcontainers, no mocks. A new `schemaOnlyPrisma` fixture
(migrated-but-empty clone database) simulates a replica that hasn't
caught up, and a real in-memory OTel meter pins the counter semantics
per outcome.
- [x] Replica catches up during the jittered retry window → served by
the replica, `outcome=replica_retry` = 1, primary never consulted
- [x] Replica permanently missing the since snapshot → served by the
primary, `outcome=primary` = 1
- [x] Snapshot missing on both replica and primary → null, counter = 0
- [x] Replica has the since snapshot but lags by one → the replica's
view is served, no fallback (verified discriminating power: the test
fails if reads secretly hit the primary)
- [x] Flag off with a replica configured → primary serves the read
- [x] Transaction client provided → bypasses the replica entirely
- [x] Since snapshot belonging to a different run → null
- [x] Existing getSnapshotsSince + waitpoints suites green; run-engine,
testcontainers, and webapp typechecks pass
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
|
||
|
Saadi Myftija
|
081b6bac17 |
feat(supervisor): publish client-side dequeue API latency as a Prometheus histogram (#3887)
⚒️ Publish Worker (v4) / build (supervisor) (push) Has been cancelled
The supervisor's dequeue round-trip time (`POST /engine/v1/worker-actions/dequeue`) was measured but only flowed into wide events and OTel span attributes — there was no Prometheus series, so latency percentiles and error rates weren't queryable. This adds `queue_consumer_pool_dequeue_duration_seconds` (histogram, label `outcome=success|empty|error`) to the existing consumer-pool metrics, scraped automatically by the existing ServiceMonitors on queue-raider/schedule-raider/supervisor. - Records every dequeue call, including failed ones, which previously emitted no timing at all - The pool's shared `ConsumerPoolMetrics` instance is injected into each consumer (mirrors the `BackpressureMetrics` → `BackpressureMonitor` wiring) - Buckets extend to 30s because `wrapZodFetch` retries internally (5 attempts, ≥7.5s backoff before a retryable error surfaces) - Existing `dequeueResponseMs` wide-event/span behavior unchangedre2-prod-client-deqeueue-metrics |
||
|
Daniel Sutton
|
459dce2a97 |
ci: make the main-image dispatch repo and ref configurable (#3883)
The `dispatch-main-image` job was hard-gated to `triggerdotdev/trigger.dev` on the `main` ref. This makes it configurable via repository variables, all defaulting to the current values: - `MAIN_IMAGE_DISPATCH_REPO` — the repo allowed to dispatch (default `triggerdotdev/trigger.dev`). - `MAIN_IMAGE_DISPATCH_REF_PREFIX` — the ref-name prefix that dispatches, matched with `startsWith(github.ref_name, …)` (default `main`). - `MAIN_IMAGE_DISPATCH_TARGET` — the `repository_dispatch` target (default `triggerdotdev/cloud`). The job is additionally gated on `github.event_name == 'push'`. This is necessary, not just defensive: the gate now keys off `github.ref_name` rather than the computed image tag, and `ref_name` is still `main` when `release.yml` invokes this workflow via `workflow_call` during a release — so without the event guard the job would fire during every release and fail on the absent `CROSS_REPO_PAT`. A version-equality check can't replace it because `build-*` tags strip the prefix to the version output. Behaviour note: the intended dispatch paths — push to `main`, and push of a `<prefix>*` tag in a downstream repo — are `push` events and are unchanged. The one case that no longer dispatches is a manual `workflow_dispatch` run of `publish.yml` on `main` (it previously did, via the old `version == 'main'` check). That path is indistinguishable from a manual release by event name, so `push`-only is the clean discriminator. Dispatching still requires `CROSS_REPO_PAT`, so setting the variables alone doesn't enable anything. --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
Eric Allam
|
87448ccaf2 |
feat(webapp,core): add an endpoint to list a project's environments (#3880)
## Summary
Adds `GET /api/v1/projects/{projectRef}/environments` (personal access
token auth), which lists the base environments a user can access for a
project — their own dev environment plus the project's staging, preview,
and production environments.
## Details
- Built on the PAT route builder, so it inherits org-membership auth and
the per-resource ability check.
- `dev` is scoped to the token owner; archived environments are
excluded.
- Returns the branchable **parent** preview environment — preview branch
children are not included. A consumer targets the parent; branch-level
overrides are handled separately.
- Sorted to match the dashboard's environment switcher (dev → staging →
preview → prod), and never returns API keys.
Example response:
```json
[
{ "id": "...", "slug": "dev", "type": "DEVELOPMENT", "isBranchableEnvironment": false, "branchName": null, "paused": false },
{ "id": "...", "slug": "stg", "type": "STAGING", "isBranchableEnvironment": false, "branchName": null, "paused": false },
{ "id": "...", "slug": "preview", "type": "PREVIEW", "isBranchableEnvironment": true, "branchName": null, "paused": false },
{ "id": "...", "slug": "prod", "type": "PRODUCTION", "isBranchableEnvironment": false, "branchName": null, "paused": false }
]
```
|
||
|
Eric Allam
|
b28c6d0b90 |
fix(webapp): sanitize streamed agent URLs before rendering in the agent view (#3882)
## Summary The dashboard's Agent view rendered `source-url` and `file` message parts by putting their `url` straight into an `href`/`src`. Those URLs come from streamed agent and tool data, so a tool that emitted something like `javascript:alert(1)` produced a clickable XSS payload in the dashboard. ## Fix A `toSafeUrl` helper now gates every URL before it reaches an `href`/`src`: it allows only `http:`/`https:`/`blob:` (and `data:image/...` for inline images) and returns `null` for anything else. Unsafe values render as plain text instead of a link or image, so a hostile or malformed URL degrades gracefully rather than becoming clickable. Safe URLs render exactly as before. Covered by a unit test over the allow/deny list. |
||
|
Daniel Sutton
|
bc01f6ea3a |
fix(webapp): stop writer DB connectivity errors leaking to trigger() API clients (#3874)
## Summary
During `trigger()` worker-queue resolution, `getWorkerQueue` wrapped any
error from `getDefaultWorkerGroupForProject` into a client-facing
`ServiceValidationError` (HTTP 422) carrying `error.message`. That
method runs `project.findFirst` on the **writer**; when the writer is
unreachable Prisma throws a connection error (P1001) whose message
includes the database host, and that raw message was returned to the API
client and surfaced in the run view via the SDK's `TriggerApiError`.
It also mis-classifies a transient outage: a 422 is not retried by the
SDK, so triggers failed permanently instead of riding out a brief writer
blip.
## Design
This is the only place on the trigger path that folds a *caught* error's
message into a client-facing error — every other DB failure on the path
propagates to the route's generic 500 handler (scrubbed, and retried by
the SDK). So the fix is local:
- Add `isInfrastructureError()` — true for Prisma connection-level
failures (the DB-unreachable family: P1001/P1002/P1008/P1017, plus the
init/panic/unknown client error classes), false for query/validation
errors (e.g. P2002).
- At the wrap site, rethrow infrastructure errors so they reach the
generic 500 handler (no raw message, and retryable). Genuine domain
failures (e.g. "Project not found.") still become a 422.
Only P1001 ("can't reach database server") has been observed in
practice; the rest of the connection family is included as same-class
forward-proofing.
## Test plan
- [x] Unit: `isInfrastructureError` classifies a P1001 (incl. the Prisma
6.x `PrismaClientKnownRequestError` shape) and init errors as
infrastructure; P2002 and a plain `Error` as not
- [x] `getWorkerQueue` rethrows a P1001 unchanged instead of wrapping it
in a `ServiceValidationError`; still wraps a domain failure as a
`ServiceValidationError` — RED on current code, GREEN after
- [ ] (optional) toxiproxy e2e: trigger with the writer cut → HTTP 500
generic body, no DB host in the response
---------
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
|
||
|
Eric Allam
|
3bc88c453e |
perf(webapp): memoize react-router per-request route matching via pnpm patch (#3877)
## Summary Under high request load the webapp spends most of its CPU inside react-router's `matchRoutes`, not in application code. `@remix-run/router@1.23.2` (the React Router v6 / Remix 2 core) re-flattens, re-ranks, and recompiles the entire route table on every request, and with the webapp's ~436 routes that cost dominates once request rates climb. There is no `NODE_ENV` gate, so production pays it too. This adds a pnpm patch that memoizes the parts that depend only on the static route manifest: it caches the flattened/ranked branches per route tree, hoists the loop-invariant `decodePath` out of the match loop, and caches compiled path regexes. ## Benchmark CPU profile over the same load (100 concurrent tag feeds, ~425 req/s), `NODE_ENV=production`, before vs after the patch: | Metric | Before | After | | --- | --- | --- | | Active CPU (self-time over the window) | 28.3s | 18.5s (-34%) | | Route-matching self-time | 19.2s | 7.5s (-61%) | | Event-loop lag p99 | 322ms | 113ms (-65%) | | Idle headroom | 26% | 52% | Application/realtime code was ~0% of CPU in both profiles; the bottleneck was entirely generic per-request route matching. ## Why a patch instead of an upgrade The inefficiency is acknowledged upstream ([remix-run/react-router#8653](https://github.com/remix-run/react-router/issues/8653)). A contributor PR doing exactly this ([remix-run/react-router#14866](https://github.com/remix-run/react-router/pull/14866)) was closed in favor of a narrower fix ([remix-run/react-router#14967](https://github.com/remix-run/react-router/pull/14967), branch caching only, shipped in React Router v7), with the maintainer suggesting patch-package as the interim until the Remix 3 route-pattern rewrite (see [remix-run/remix#4786](https://github.com/remix-run/remix/discussions/4786)). We are on the v6-era core and cannot pick up even the partial fix without a framework migration, so this patch is the sanctioned stopgap, and it also includes the compiled-regex cache the merged PR left out. [`patches/README.md`](https://github.com/triggerdotdev/trigger.dev/blob/perf/react-router-route-matching/patches/README.md) documents the full rationale, the safety argument (deterministic, internal-only, bounded caches), and when to remove the patch. |
||
|
Eric Allam
|
d97335902b |
docs: add a database connections guide for tasks (#3881)
## Summary A new guide for connecting a database to your tasks: where to create the client, how to size the connection pool against your provider's limit, when to reach for a pooler, and how to release connections at waits so you don't hit "too many connections" or crash on resume. It covers node-postgres, Prisma, Drizzle, and MongoDB, with researched direct and pooled connection limits for the common Postgres providers (Supabase, Neon, RDS, PlanetScale) and MongoDB Atlas. The page lives under Documentation, Troubleshooting, and is linked from the chat agent docs (overview, lifecycle hooks, chat.local, and the database persistence pattern). |
||
|
Oskar Otwinowski
|
774a9793e3 |
fix(webapp): paginate shared Vercel env var fetch in onboarding + pull (#3879)
Closes #3850 <img width="1709" height="1115" alt="Screenshot 2026-06-09 at 18 44 40" src="https://github.com/user-attachments/assets/edc1f091-0937-403d-a6e9-9d9477e77ed0" /> |
||
|
Matt Aitken
|
df964ea4ee |
feat(ci): dispatch a repository event when the main webapp image is published (#3875)
## Summary On every `main` build, once the webapp image is pushed to the registry, the publish workflow emits a cross-repo `repository_dispatch` event (`main-image-published`) carrying a digest-pinned image ref. Other repositories in the org can subscribe to that event and build or deploy from the exact artifact, instead of chasing the moving `main` tag. ## Design `publish-webapp.yml` now exposes the pushed multi-arch index digest as a workflow output. `publish.yml` adds a `dispatch-main-image` job (after `publish-webapp`) that builds `<image_repo>@<digest>` and sends the dispatch via the same pinned `peter-evans/repository-dispatch` action already used elsewhere in this repo, authed with `CROSS_REPO_PAT`. It fires only when the published tag is `main`, so semver releases and other tag builds are excluded, and only from the canonical repo so forks never dispatch. The payload is JSON-escaped with `jq`. |
||
|
Matt Aitken
|
1b0f2c71dd |
fix(webapp): correct backward pagination slice in listRunIds (#3867)
## Problem Two backward-pagination bugs in `ClickHouseRunsRepository.listRunIds`, both pre-existing (they predate the composite-cursor work in #3852 and were spotted during/after it): **1. Wrong slice (straddled pages).** `listRunRows` fetches `page.size + 1` rows to detect `hasMore`. That extra row is the one *farthest from the cursor* in both directions (forward orders DESC; backward orders ASC), so it's always the *trailing* element. Forward correctly used `rows.slice(0, size)`, but backward+`hasMore` used `rows.slice(1, size + 1)` — dropping the row *closest* to the cursor and keeping the has-more sentinel. The page straddled two logical pages (one run from the correct previous page + one from the page before it), so paging "newer" across a boundary **repeated and skipped** runs. **2. Stranded forward cursor on a partial backward page.** In the backward `!hasMore` branch, `nextCursor` was `reversedRows.at(page.size - 1)`. On a partial page (fewer than `page.size` rows — reachable via `runs.list` by passing a forward page's cursor as `page[before]`), that index overshoots → `undefined` → `nextCursor` becomes `null`, leaving no way to page forward again. ## Fix - **Slice:** both directions now slice `rows.slice(0, size)` (the sentinel is the trailing element either way). - **Partial-page cursor:** the backward `!hasMore` branch takes the oldest row on the page, `rows.at(0)`, for `nextCursor` — equivalent to the old expression for full pages, correct for partial ones. Forward pagination, the cursor *values* for full pages, and the `hasMore === true` paths were already correct and are unchanged. ## Tests `runsRepositoryCursor.test.ts` gains two cases (both fail on `main`, pass here): - **multi-page backward walk:** forward across all pages, then backward from the last page — each backward page must *exactly* reproduce the corresponding forward page (no straddling: `main` returns `{b,c}` instead of `{c,d}`), and the full traversal covers every run once. - **partial backward page:** backward onto a partial first page must still expose a working forward cursor (and paging forward from it reaches the rest) — `main` returns a `null` nextCursor. The three existing cursor tests (forward completeness, backward round-trip, legacy cursor) still pass. --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
Matt Aitken
|
f4a96bdf84 |
Fail dev and deploy on duplicate task ids (#3865)
## What `dev` and `deploy` now fail with a clear error when two tasks are defined with the same id — including across task types (e.g. a scheduled task and a regular task sharing an id). ## Why Tasks are registered into the resource catalog keyed by id, so a second definition with the same id silently overwrote the first. One of the tasks would just vanish from the worker with no warning — easy to miss, hard to debug. (Any earlier duplicate-id check ran against the post-registration task list, which is already de-duplicated, so it never actually fired.) ## How - **Detect at registration** (`@trigger.dev/core`): `StandardResourceCatalog` records a collision when a task id is registered more than once, capturing the files involved — the only point where both definitions are visible before the id-keyed map collapses them. Exposed via `listTaskIdCollisions()`. - **Fail indexing** (`trigger.dev` CLI): both index workers report collisions via a new `TASKS_FAILED_TO_INDEX` message; `indexWorkerManifest` rejects with a new `DuplicateTaskIdsError`. `dev` renders a dedicated error (offending ids + files + docs link); `deploy` fails with the same message. Runtime worker boot is unaffected — it never reads the collisions. - **Server-side backstop** (webapp): background-worker registration also rejects duplicate ids with a clear `ServiceValidationError`, so duplicates are caught even from an older CLI. ## Testing - Unit tests for collision collection in the catalog and for the error-message formatting (standard, same-file, and 3+-definition cases). - Verified end to end against a local webapp: a project with a regular task and a scheduled task sharing an id now fails `dev` with the dedicated error; a project with distinct ids still starts normally. ## Changeset Patch for `@trigger.dev/core` and `trigger.dev`. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
Matt Aitken
|
6bcd369ea1 |
feat(webapp,rbac): REQUIRE_PLUGINS=1 fail-fast for required plugin loads [TRI-9852] (#3734)
## Summary - `internal-packages/rbac/src/index.ts` — in `LazyController.load()`'s catch block, throw an Error when `process.env.REQUIRE_PLUGINS === "1"` instead of silently falling back. The throw is captured into the lazy controller's init promise, so it surfaces on the first method call. - `apps/webapp/app/routes/healthcheck.tsx` — `await rbac.isUsingPlugin()` after the DB ping. With `REQUIRE_PLUGINS=1` and a failed plugin load, the throw surfaces here and the healthcheck returns 500 → readiness probe fails → rollout is rolled back. Noop for self-hosters. - `.server-changes/require-plugins-fail-fast.md` — server-changes entry. - `internal-packages/rbac/src/require-plugins.test.ts` — 4 unit tests covering loader branching: unset → fallback, `=1` → throw, `forceFallback: true` wins, only exactly `"1"` enforces. - `internal-packages/testcontainers/src/webapp.ts` — adds `requirePlugins?: boolean` to `StartWebappOptions`. Implies `forceRbacFallback: false`. - `apps/webapp/test/healthcheck-require-plugins.e2e.test.ts` — e2e closes the loop: spawns a real webapp, hits `/healthcheck` via HTTP, asserts 500 with `REQUIRE_PLUGINS=1` and 200 without. ## Motivation Today the RBAC plugin loader catches any plugin-load failure (missing module, broken transitive dep, init throw) and silently returns the default fallback implementation. This is the correct behaviour for self-hosters who don't ship the plugin — but it's dangerous in deployments where the plugin is expected to load: an accidentally-missing or broken plugin would silently disable enforcement. `REQUIRE_PLUGINS=1` makes the loader fail loudly in those deployments. The variable name is intentionally plural and generic — future plugin contracts (audit logs, SSO) can read the same flag without renaming. Closes [TRI-9852](https://linear.app/triggerdotdev/issue/TRI-9852/require-plugins1-fail-fast-for-required-plugin-loads). ## Test plan - [x] `pnpm run test --filter @trigger.dev/rbac` — 38/38 tests pass, including the 4 new loader tests - [x] `pnpm run typecheck --filter webapp` — passes - [x] `pnpm run typecheck --filter @trigger.dev/rbac --filter @internal/testcontainers` — passes - [x] e2e test added (`healthcheck-require-plugins.e2e.test.ts`) — CI runs it via `e2e-webapp.yml`. Couldn't run locally (no Docker daemon up); CI has Docker provisioned. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
Eric Allam
|
0ee1461d86 |
fix(webapp): show scheduled runs under their correct region (#3873)
## Summary Runs routed onto a dedicated scheduled worker queue were showing under a phantom region in the dashboard, run details, and the API, and slipped through region filters. They now resolve to their real region everywhere. ## Fix A worker queue can carry a `:scheduled` suffix that routes scheduled-lineage runs onto their own list. That suffix is an internal routing detail, but it was leaking anywhere the worker queue is read as a region. A `baseWorkerQueue` helper strips any `:<class>` suffix back to the base region (region names never contain a colon, so it's everything before the first colon) and is applied at every region read site: the runs list, run detail, the public API, and replay's region override. The runs-replication writer also stores the base region in ClickHouse so the region filter matches. |
||
|
Eric Allam
|
18b90285b2 |
feat(cli): set up AI tooling in trigger init and add getting-started skill (#3872)
## Summary `trigger init` now sets up your AI coding assistant as part of project setup. Instead of the old either/or "MCP or CLI" prompt, it offers the MCP server and agent skills together, then asks whether to scaffold with the CLI or let your assistant do it. A new `getting-started` agent skill backs that hand-off: it teaches the assistant the bootstrap recipe (install the SDK, write `trigger.config.ts`, scaffold a first task, wire tsconfig/gitignore, run `trigger dev`) and is explicit about the two steps that genuinely need a human (`trigger login` and copying the DEV secret key from the dashboard). It ships in the CLI alongside the existing skills, version-matched to your SDK. Prompt-once gating is shared, so opting in or out during `init` means `trigger dev` won't ask about skills again. |
||
|
Daniel Sutton
|
e9c459fb8d |
ci: allow forks to override published container image namespace (#3866)
## Summary
The container publish workflows hardcoded `ghcr.io/triggerdotdev/...` as
the image destination. As a result, a fork that builds on push-to-`main`
(or on the worker publish tags) would attempt to push to — and attest —
the upstream packages rather than its own, which fails on permissions
and is surprising besides.
This makes the image destination configurable via a single
`IMAGE_REGISTRY` repository variable, while leaving the upstream
defaults byte-identical:
- **Single source of truth** (`publish.yml`): a `resolve-registry` job
resolves the target registry namespace once — `IMAGE_REGISTRY`
repository variable, defaulting to `ghcr.io/${{ github.repository_owner
}}` — and passes it down to every publish job as an `image_registry`
input. So a fork publishes to its own namespace automatically with no
configuration.
- **Webapp** (`publish-webapp.yml`): the image now lives at
`<registry>/<repo-name>` (e.g. `ghcr.io/<owner>/trigger.dev`). The
provenance attestation and the downstream Trivy scan follow the same
computed repo via the `image_repo` workflow output.
- **Workers** (`publish-worker.yml`, `publish-worker-v4.yml`): build
under `<registry>/<worker-name>`. They keep a `vars.IMAGE_REGISTRY ||
ghcr.io/<owner>` fallback so they still resolve correctly on their
direct `infra-*` / `re2-*` push triggers (which bypass the parent
workflow).
A single `IMAGE_REGISTRY` namespace variable now governs both webapp and
workers (the earlier `WEBAPP_IMAGE_REPO` full-path override is dropped,
removing the full-path/namespace asymmetry). When `IMAGE_REGISTRY` is
unset, every resolved image name is exactly what it is today, so there
is no change for this repo.
## Test plan
- [x] `actionlint` passes on all four workflows
- [ ] On merge, confirm the webapp publish still pushes
`ghcr.io/triggerdotdev/trigger.dev:main` + the commit-SHA tag (defaults
unchanged)
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
||
|
Daniel Sutton
|
d7028e278e |
feat(webapp): label mollifier decisions by enrolled org (#3869)
## Summary
The `mollifier.decisions` metric only carried an `outcome` label, so for
an org that has the mollifier enabled there was no way to see how often
its triggers pass through the gate instead of being diverted — making it
hard to tell why the trip isn't firing for an opted-in org.
This adds two bounded labels: `enrolled` (`"true"`/`"false"`, the
per-org flag) and `org` (the org id, attached **only** when `enrolled`
is true). For an enrolled org you can now compare directly:
`mollifier.decisions{outcome="pass_through", enrolled="true",
org="<id>"}` vs `{outcome="mollify", ...}`.
## Design
`recordDecision` now takes an options object (`{ reason?, enrolled,
orgId? }`). The `org` label is restricted to the enrolled cohort to keep
cardinality bounded — the guard lives in a pure `decisionLabels` helper,
so a non-enrolled org id can never be attached even if one is passed.
The enrolled set is small and capped operationally.
The per-org flag is resolved once at the top of `evaluateGate`
(in-memory, no DB round-trip on the trigger hot path) so every decision
— including the debounce / one-time-use-token / triggerAndWait bypasses
— is labelled consistently.
## Test plan
- [x] `mollifierGate.test.ts` cascade asserts `enrolled`/`org` on every
gate branch
- [x] `mollifierDecisionLabels.test.ts` (new) proves `org` is dropped
for non-enrolled even when an id is passed (cardinality guard)
- [x] `vitest run mollifierGate mollifierDecisionLabels` — 34/34 pass
- [x] `pnpm run typecheck --filter webapp` clean
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
|
||
|
Oskar Otwinowski
|
93532cdb99 |
feat(supervisor): forward per-run labels to the compute provider (#3821)
Add an optional network_labels field to the internal compute client's create and restore request schemas and forward per-VM endpoint labels on both paths, so a restored VM keeps the same labels as a freshly-booted one. Mirrors the label the Kubernetes workload manager already sets on the run pod. --------- Co-authored-by: nicktrn <55853254+nicktrn@users.noreply.github.com> |
||
|
Eric Allam
|
8b85da1b26 |
feat(cli): install Trigger.dev agent skills into your coding agent (#3868)
## Summary `trigger skills` installs Trigger.dev agent skills into your coding agent so it knows how to write Trigger.dev code: tasks, schedules, realtime, and `chat.agent` AI agents. The skills are `SKILL.md` files (the open Agent Skills format) bundled with the CLI and copied into each tool's native skills directory (Claude Code, Cursor, GitHub Copilot, and Codex / `AGENTS.md`), version-matched to the CLI you run. `trigger dev` offers to install them on first run, and a one-line always-on pointer is written into your `CLAUDE.md` / Cursor rules / etc. so the agent always knows which skills are available and loads the right one on demand. This replaces the old `install-rules` command, which stays as an alias. Four skills ship to start: `authoring-tasks`, `realtime-and-frontend`, `authoring-chat-agent`, and `chat-agent-advanced`. |
||
|
Matt Aitken
|
ef04cc39ef |
fix(webapp): use composite keyset cursor for run pagination (#3852)
## Problem
`ClickHouseRunsRepository.listRunIds` / `listRuns` order results by the
composite key `(created_at, run_id)`, but the cursor predicate cut on
`run_id` **alone**:
```ts
.where("run_id < {runId: String}", { runId: cursor })
.orderBy("created_at DESC, run_id DESC")
```
This is only sound when `run_id` lexicographic order matches
`created_at` order. `run_id`s are cuids — only coarsely time-sortable —
so when a burst of runs is created within a sub-second window, the two
orders can diverge. When they do, the next-page predicate (`run_id <
cursor`, where `cursor` is the *last* page element = the smallest
`created_at`, not necessarily the smallest `run_id`):
- **re-includes** rows already returned on a previous page (duplicates),
and
- **skips** rows it should have returned (silent data loss).
For bulk **replay** this caused runs to be replayed more than once
(replay has no idempotency guard). For the dashboard and the `runs.list`
API it could silently repeat or skip runs at page boundaries.
## Fix
Make the cursor predicate match the composite ordering:
- Cursors now encode the full `(created_at, run_id)` key as an **opaque
URL-safe base64 token**
(`base64url({"c":<createdAtMs>,"r":"<runId>"})`), and the query cuts on
the matching tuple — `(created_at, run_id) < (…)` forward / `> (…)`
backward.
- The `ORDER BY` is unchanged, so the query stays aligned with the
table's primary key — no performance regression (the tuple range
predicate is actually more index-friendly than `run_id <` alone).
- Cursors are **server-issued opaque tokens** (the SDK only echoes
`pagination.next` / `pagination.previous` back), so this needs **no
client/SDK update**. Legacy cursors were the bare internal `run_id`;
they're detected by decode failure (a cuid isn't a valid base64-wrapped
JSON payload) and fall back to the old `run_id`-only predicate, so
in-flight cursors keep working and drain naturally. New cursors also no
longer expose a bare internal run id.
- `listRunIds` is now the single cursor-aware list primitive: it returns
`{ runIds, pagination: { nextCursor, previousCursor } }`, and `listRuns`
builds on it (one place constructs cursors). Bulk actions consume the
same method and advance by `pagination.nextCursor`, finishing when it's
`null`.
- `getTaskRunsQueryBuilder` now also selects
`toUnixTimestamp64Milli(created_at) AS created_at_ms`, using a dedicated
`TaskRunListQueryResult` schema. The shared `TaskRunV2QueryResult` stays
`run_id`-only so the run-engine pending-version lookup
(`getPendingVersionIdsQueryBuilder`, which selects only `run_id`)
doesn't fail validation on a column it doesn't query.
## Tests
New `runsRepositoryCursor.test.ts` (testcontainer-backed, real
Postgres→ClickHouse replication):
- **forward** pagination returns every run exactly once when `run_id`
order is the reverse of `created_at` order (reproduces the
duplicate/skip bug — fails on `main`; this
walk-until-`nextCursor`-null-and-assert-complete is exactly the bulk
action's iteration),
- **backward** pagination round-trips to the previous page across a
boundary,
- **legacy** bare-`run_id` cursor still uses the old predicate
(backwards compatibility).
The existing `runsRepository` suites (part1–4) still pass; `part4`'s
`count new runs with listRunIds` test was updated for the new `{ runIds,
pagination }` return shape, and the `clickhouse` `taskRuns`
query-builder snapshots were regenerated for the added `created_at_ms`
column.
## Notes
- Separate, pre-existing issue (out of scope, not introduced here):
`listRuns`' backward display-slicing (`rows.slice(1, size+1)` when
`hasMore`) has an off-by-one that can return a straddled page. Tracked
separately.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
