github-featured/triggerdotdev--trigger.dev
2
0
Fork 0
Code Issues Pull Requests Deployments Models Agent Notes Packages Projects Releases Wiki Activity
7,373 Commits 300 Branches 4,167 Tags
v.docker.4.5.0-rc.6
Commit Graph

7373 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
github-actions[bot] 5fab8cafcf chore: release v4.5.0-rc.6 (#3870)
🚀 Publish Trigger.dev Docker / units (push) Failing after 4s
Details
🚀 Publish Trigger.dev Docker / typecheck (push) Failing after 5s
Details
🚀 Publish Trigger.dev Docker / publish-worker-v4 (push) Has been skipped
Details
🚀 Publish Trigger.dev Docker / scan-webapp (push) Has been skipped
Details
🚀 Publish Trigger.dev Docker / publish-webapp (push) Has been skipped
Details
🚀 Publish Trigger.dev Docker / publish-worker (push) Has been skipped
Details
🧭 Helm Chart Release / lint-and-test (push) Has been cancelled
Details
🧭 Helm Chart Release / release (push) Has been cancelled
Details
🚀 Publish Trigger.dev Docker / 📣 Dispatch main image (push) Has been cancelled
Details 
## Summary
7 improvements, 1 bug fix.

## Improvements
- `trigger init` now sets up your AI coding assistant as part of project
setup: pick the MCP server, the agent skills, or both, then scaffold
with the CLI or hand off to your assistant. Adds a new `getting-started`
agent skill that teaches assistants how to bootstrap Trigger.dev
(install the SDK, write `trigger.config.ts`, create a first task, run
`trigger dev`), so the AI-driven setup path works end to end. It ships
in the CLI alongside the existing skills, version-matched to your SDK.
([#3872](https://github.com/triggerdotdev/trigger.dev/pull/3872))
- `dev` and `deploy` now fail with a clear error when two tasks are
defined with the same id, including across different task types (e.g. a
scheduled task and a regular task sharing an id). Previously the second
definition silently overwrote the first, so one of the tasks would
vanish with no warning. Task ids are detected as duplicates during
indexing (naming each offending id and the files it was found in), and
the same rule is enforced server-side when the background worker is
registered.
([#3865](https://github.com/triggerdotdev/trigger.dev/pull/3865))
- `trigger skills` installs Trigger.dev agent skills into your coding
agent so it knows how to write tasks, schedules, realtime, and
chat.agent code. The skills ship with the CLI and are copied into each
tool's native skills directory (Claude Code, Cursor, GitHub Copilot, and
Codex / AGENTS.md), and `trigger dev` offers to install them on first
run. ([#3868](https://github.com/triggerdotdev/trigger.dev/pull/3868))
- Reliability fixes for `chat.agent`. A user message sent while the
agent is streaming is no longer delivered twice (which could run a
duplicate turn), input appends now carry an idempotency key so a retried
send can't duplicate a message, stopping a generation clears the
streaming state so a page reload doesn't replay the stopped turn, and
runs can now carry the full set of dashboard tags instead of being
silently truncated. `onTurnComplete` now fires on errored turns (with
the thrown error attached) and the failed turn's user message is
persisted so it isn't lost on the next run. Custom agents and manual
`chat.writeTurnComplete` callers now trim the output stream, sending a
custom action no longer leaves a second stream reader running, and a
long-lived `watch` subscription no longer grows its dedupe set without
bound. ([#3891](https://github.com/triggerdotdev/trigger.dev/pull/3891))
- Continuation chat boots no longer stall for around 10 seconds before
the first turn. The `session.in` resume cursor is now found with a
non-blocking records read instead of draining an SSE long-poll (which
always waited out its full 5 second inactivity window, twice per boot),
the boot reads run concurrently, and chat snapshots carry the cursor so
subsequent boots skip the scan entirely.
([#3907](https://github.com/triggerdotdev/trigger.dev/pull/3907))
- Record client-side dequeue API latency in the supervisor consumer pool
as a Prometheus histogram
(`queue_consumer_pool_dequeue_duration_seconds`, labelled by `outcome`:
success/empty/error).
([#3887](https://github.com/triggerdotdev/trigger.dev/pull/3887))
- Add `GetProjectEnvironmentsResponseBody` and `ProjectEnvironment`
schemas for the new `GET /api/v1/projects/{projectRef}/environments`
endpoint, which lists the parent environments (dev, staging, preview,
prod) a personal access token can access for a project. Dev is scoped to
the token owner and branch (preview child) environments are excluded.
([#3880](https://github.com/triggerdotdev/trigger.dev/pull/3880))

## Bug fixes
- Fix two `chat.createSession()` bugs: stopping a generation no longer
wedges the run (the turn loop raced a `totalUsage` promise that never
settles after a stop-abort), and continuation runs now wait for the next
message instead of invoking the model with an empty prompt.
([#3920](https://github.com/triggerdotdev/trigger.dev/pull/3920))

<details>
<summary>Raw changeset output</summary>

⚠️⚠️⚠️⚠️⚠️⚠️

`main` is currently in **pre mode** so this branch has prereleases
rather than normal releases. If you want to exit prereleases, run
`changeset pre exit` on `main`.

⚠️⚠️⚠️⚠️⚠️⚠️

# Releases
## @trigger.dev/build@4.5.0-rc.6

### Patch Changes

-   Updated dependencies:
    -   `@trigger.dev/core@4.5.0-rc.6`

## trigger.dev@4.5.0-rc.6

### Patch Changes

- `trigger init` now sets up your AI coding assistant as part of project
setup: pick the MCP server, the agent skills, or both, then scaffold
with the CLI or hand off to your assistant. Adds a new `getting-started`
agent skill that teaches assistants how to bootstrap Trigger.dev
(install the SDK, write `trigger.config.ts`, create a first task, run
`trigger dev`), so the AI-driven setup path works end to end. It ships
in the CLI alongside the existing skills, version-matched to your SDK.
([#3872](https://github.com/triggerdotdev/trigger.dev/pull/3872))

- `dev` and `deploy` now fail with a clear error when two tasks are
defined with the same id, including across different task types (e.g. a
scheduled task and a regular task sharing an id). Previously the second
definition silently overwrote the first, so one of the tasks would
vanish with no warning. Task ids are detected as duplicates during
indexing (naming each offending id and the files it was found in), and
the same rule is enforced server-side when the background worker is
registered.
([#3865](https://github.com/triggerdotdev/trigger.dev/pull/3865))

- `trigger skills` installs Trigger.dev agent skills into your coding
agent so it knows how to write tasks, schedules, realtime, and
chat.agent code. The skills ship with the CLI and are copied into each
tool's native skills directory (Claude Code, Cursor, GitHub Copilot, and
Codex / AGENTS.md), and `trigger dev` offers to install them on first
run. ([#3868](https://github.com/triggerdotdev/trigger.dev/pull/3868))

    ```bash
    trigger skills --target claude-code
    ```

Replaces the previous `install-rules` command, which stays as an alias.

-   Updated dependencies:
    -   `@trigger.dev/core@4.5.0-rc.6`
    -   `@trigger.dev/build@4.5.0-rc.6`
    -   `@trigger.dev/schema-to-json@4.5.0-rc.6`

## @trigger.dev/core@4.5.0-rc.6

### Patch Changes

- Reliability fixes for `chat.agent`. A user message sent while the
agent is streaming is no longer delivered twice (which could run a
duplicate turn), input appends now carry an idempotency key so a retried
send can't duplicate a message, stopping a generation clears the
streaming state so a page reload doesn't replay the stopped turn, and
runs can now carry the full set of dashboard tags instead of being
silently truncated. `onTurnComplete` now fires on errored turns (with
the thrown error attached) and the failed turn's user message is
persisted so it isn't lost on the next run. Custom agents and manual
`chat.writeTurnComplete` callers now trim the output stream, sending a
custom action no longer leaves a second stream reader running, and a
long-lived `watch` subscription no longer grows its dedupe set without
bound. ([#3891](https://github.com/triggerdotdev/trigger.dev/pull/3891))
- Continuation chat boots no longer stall for around 10 seconds before
the first turn. The `session.in` resume cursor is now found with a
non-blocking records read instead of draining an SSE long-poll (which
always waited out its full 5 second inactivity window, twice per boot),
the boot reads run concurrently, and chat snapshots carry the cursor so
subsequent boots skip the scan entirely.
([#3907](https://github.com/triggerdotdev/trigger.dev/pull/3907))
- Record client-side dequeue API latency in the supervisor consumer pool
as a Prometheus histogram
(`queue_consumer_pool_dequeue_duration_seconds`, labelled by `outcome`:
success/empty/error).
([#3887](https://github.com/triggerdotdev/trigger.dev/pull/3887))
- `dev` and `deploy` now fail with a clear error when two tasks are
defined with the same id, including across different task types (e.g. a
scheduled task and a regular task sharing an id). Previously the second
definition silently overwrote the first, so one of the tasks would
vanish with no warning. Task ids are detected as duplicates during
indexing (naming each offending id and the files it was found in), and
the same rule is enforced server-side when the background worker is
registered.
([#3865](https://github.com/triggerdotdev/trigger.dev/pull/3865))
- Add `GetProjectEnvironmentsResponseBody` and `ProjectEnvironment`
schemas for the new `GET /api/v1/projects/{projectRef}/environments`
endpoint, which lists the parent environments (dev, staging, preview,
prod) a personal access token can access for a project. Dev is scoped to
the token owner and branch (preview child) environments are excluded.
([#3880](https://github.com/triggerdotdev/trigger.dev/pull/3880))

## @trigger.dev/python@4.5.0-rc.6

### Patch Changes

-   Updated dependencies:
    -   `@trigger.dev/sdk@4.5.0-rc.6`
    -   `@trigger.dev/core@4.5.0-rc.6`
    -   `@trigger.dev/build@4.5.0-rc.6`

## @trigger.dev/react-hooks@4.5.0-rc.6

### Patch Changes

-   Updated dependencies:
    -   `@trigger.dev/core@4.5.0-rc.6`

## @trigger.dev/redis-worker@4.5.0-rc.6

### Patch Changes

-   Updated dependencies:
    -   `@trigger.dev/core@4.5.0-rc.6`

## @trigger.dev/rsc@4.5.0-rc.6

### Patch Changes

-   Updated dependencies:
    -   `@trigger.dev/core@4.5.0-rc.6`

## @trigger.dev/schema-to-json@4.5.0-rc.6

### Patch Changes

-   Updated dependencies:
    -   `@trigger.dev/core@4.5.0-rc.6`

## @trigger.dev/sdk@4.5.0-rc.6

### Patch Changes

- Reliability fixes for `chat.agent`. A user message sent while the
agent is streaming is no longer delivered twice (which could run a
duplicate turn), input appends now carry an idempotency key so a retried
send can't duplicate a message, stopping a generation clears the
streaming state so a page reload doesn't replay the stopped turn, and
runs can now carry the full set of dashboard tags instead of being
silently truncated. `onTurnComplete` now fires on errored turns (with
the thrown error attached) and the failed turn's user message is
persisted so it isn't lost on the next run. Custom agents and manual
`chat.writeTurnComplete` callers now trim the output stream, sending a
custom action no longer leaves a second stream reader running, and a
long-lived `watch` subscription no longer grows its dedupe set without
bound. ([#3891](https://github.com/triggerdotdev/trigger.dev/pull/3891))
- Continuation chat boots no longer stall for around 10 seconds before
the first turn. The `session.in` resume cursor is now found with a
non-blocking records read instead of draining an SSE long-poll (which
always waited out its full 5 second inactivity window, twice per boot),
the boot reads run concurrently, and chat snapshots carry the cursor so
subsequent boots skip the scan entirely.
([#3907](https://github.com/triggerdotdev/trigger.dev/pull/3907))
- Fix `chat.headStart` when `hydrateMessages` is registered. The warm
route's step-1 partial now reaches the agent's accumulator on the
hydrate path, so `onTurnComplete` carries the full first turn (the
head-start user message included), tool-call handovers resume from step
2 instead of re-running step 1, and the assistant `messageId` stays
stable across the handover.
([#3907](https://github.com/triggerdotdev/trigger.dev/pull/3907))
- Preserve reasoning parts across the `chat.headStart` handover.
Extended-thinking models' step-1 reasoning now lands in the durable
session history (and `onTurnComplete`) under the same assistant
`messageId`, with provider metadata intact so Anthropic thinking
signatures survive replays.
([#3907](https://github.com/triggerdotdev/trigger.dev/pull/3907))
- Fix two `chat.createSession()` bugs: stopping a generation no longer
wedges the run (the turn loop raced a `totalUsage` promise that never
settles after a stop-abort), and continuation runs now wait for the next
message instead of invoking the model with an empty prompt.
([#3920](https://github.com/triggerdotdev/trigger.dev/pull/3920))
-   Updated dependencies:
    -   `@trigger.dev/core@4.5.0-rc.6`

## @trigger.dev/plugins@4.5.0-rc.6

### Patch Changes

-   Updated dependencies:
    -   `@trigger.dev/core@4.5.0-rc.6`

</details>

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
helm-v4.5.0-rc.6 v.docker.4.5.0-rc.6 v4.5.0-rc.6 		2026-06-12 16:47:00 +01:00
Iss 002c441f50 feat(webapp): self serve schedules add-on (#3811) 
Adds the purchase UI for extra schedules, mirroring preview branches

## Changes
- `setSchedulesAddOn` platform client + `SetSchedulesAddOnService`
(purchase + quota-increase via Plain).
- `ScheduleListPresenter` surfaces add-on / quota / pricing;
`checkSchedule` counts purchased schedules toward the limit (`base +
purchased`).
- `PurchaseSchedulesModal` on the Schedules page — bought in **bundles
of 1,000 ($10/mo each)**; bundle increments enforced client-side and in
the action's zod schema.
 2026-06-12 11:27:06 -04:00
Eric Allam 47834198fc fix(sdk): stop chat.createSession wedging on stop and erroring on continuation boots (#3920) 
## Summary

Two `chat.createSession()` bugs that break chats at its abstraction
level:

1. **Stopping a generation wedged the run forever.** `turn.complete()`
bare-awaited the AI SDK's `totalUsage` promise, which never settles
after a stop-abort. The run stayed stuck inside the stopped turn (trace
shows a permanently partial `ai.streamText` span and no further `waiting
for next message`), so the chat could never take another message. Fixed
with the same 2s `Promise.race` guard `chat.agent`'s turn loop already
uses.

2. **Continuation runs invoked the model with an empty prompt.** The
first turn only waited for a message on `preload` boots. A continuation
run (spawned after a cancel, crash, or version upgrade) arrives with the
boot payload stripped, so the loop ran a turn with zero messages and
errored with `AI_InvalidPromptError: messages must not be empty`.
Message-less continuation boots now wait for the next session input
("waiting for first message (continuation)"), and `turn.continuation` is
preserved across the wait so user code can seed stored history off it.

Both reproduced and verified end-to-end against a live environment (stop
followed by a next turn; cancel followed by a continuation turn with
seeded history), plus the existing unit suite.
 2026-06-12 14:07:45 +01:00
Eric Allam a04cdffda6 fix(webapp): stop replica lag from double-triggering session runs and 404ing fresh sessions (#3914) 
## Summary

Two read-replica races on the session APIs could break chats whose first
activity lands inside the replication window (or any time the replica
lags):

1. A session's first `.in` append or `.out` subscribe could fail with a
404 for a session that exists on the writer, because the route resolved
the Session row on the replica only.
2. `ensureRunForSession` probed run liveness on the replica, so a probe
miss on a run triggered moments earlier was judged "run is dead" and a
second live run was spawned for the same session. Both runs then
consumed the same input stream, producing duplicated turns and doubled
responses (and doubled LLM cost).

## Fix

Liveness now re-probes the writer before declaring the current run dead
(the old code already fell back to the writer, but only to recover the
friendlyId, after the wrong verdict was made). Session resolution on the
append and subscribe/init routes goes through a new
`resolveSessionWithWriterFallback`, which stays replica-first on the hot
path and only touches the writer on a miss.

Reproduced and verified against a local streaming replica with an
artificial apply delay: pre-fix, a send immediately after session
creation reliably produced either the 404 or two executing runs with a
doubled response; post-fix, the same flow produces exactly one run and
one response.

Also rides along: the local docker replica's default apply delay drops
from 150ms to a realistic 20ms (override via `REPLICA_APPLY_DELAY` when
you want to deliberately widen the race window).
 2026-06-12 14:07:36 +01:00
Matt Aitken eb498d137f fix(plugins): drop unused gitBranch re-export from the package entry (#3923) 
`@trigger.dev/plugins` re-exported
`sanitizeBranchName`/`isValidGitBranchName` from `@trigger.dev/core` as
a convenience forwarder. Nothing actually imports them through this
package — every consumer (webapp, `@trigger.dev/rbac`, …) imports them
directly from `@trigger.dev/core/v3/utils/gitBranch`.

Removing the forwarder keeps the package entry free of **runtime** core
imports (only type re-exports + `buildJwtAbility` remain), so consumers
that bundle `@trigger.dev/plugins` from source don't pull an unrelated
core subpath into their build.

No behavior change; the helpers remain available from
`@trigger.dev/core` where they're defined.
 2026-06-12 12:16:04 +00:00
Matt Aitken f48c89752c perf(webapp): parallelize streaming batch-item ingest (#3777) 
## Problem

The item-streaming endpoint of the two-phase batch API (`POST
/api/v3/batches/:batchId/items`) processed streamed items strictly
sequentially. For a batch of many large payloads, each offloaded to
object storage inline, this serialized N object-store round-trips inside
a single request and could exceed Node's default `server.requestTimeout`
(300s). The webapp then returned `408`, which the SDK reads as `408
terminated` and retries up to 5 times, turning a slow ingest into a
failure that takes tens of minutes to surface.

## Fix

Ingest now runs through `p-map` over the NDJSON async iterable with
bounded concurrency (`STREAMING_BATCH_INGEST_CONCURRENCY`, default 10):

- `p-map` pulls lazily from the stream, so at most `concurrency` items
are read and in-flight at once. Peak memory stays bounded to roughly
`concurrency × STREAMING_BATCH_ITEM_MAXIMUM_SIZE` and request-body
backpressure is preserved.
- Set the env to `1` for fully sequential ingestion (escape hatch).

## Why this is safe (ordering and idempotency unchanged)

- Ordering derives from each item's index (enqueue `timestamp =
batch.createdAt + index`), not enqueue order.
- Dedup is atomic per index in `enqueueBatchItem`.
- The NDJSON parser now stamps oversized-item markers with their emit
position, removing the consumer's sequential `lastIndex` assumption (the
only order-dependent bit).
- The count-check and conditional-seal path is untouched.

## Scope

This speeds up every batch ingested through the streaming endpoint, not
just large-payload batches. Each item does a per-item Redis enqueue
regardless of size, and those now overlap. Large payloads benefit most
because they add an object-store offload round-trip on top of the
enqueue.

## Verification

Added an integration test (`streamBatchItems.test.ts`) that drives the
real service against Postgres + Redis + RunEngine and times a 150-item
batch at increasing concurrency. Object-store offload is modelled as a
fixed per-item latency (local round-trips are too small to compare
meaningfully):

```
runCount=150
  large payloads (10ms/item offload):
    concurrency=1   1739ms
    concurrency=10  192ms  (9.1x faster)
    concurrency=50  57ms   (30.7x faster)
  small payloads (Redis enqueue only, no offload):
    concurrency=1   90ms
    concurrency=10  24ms   (3.7x faster)
```

The test asserts correctness at every concurrency (all items accepted,
sealed, enqueued exactly once), that parallel ingest beats the
sequential floor, and that the small-payload case is strictly faster
than sequential, so the win is not specific to large payloads.

Also exercised end-to-end over real HTTP against a local server: a
20-item batch (12MB body) ingests and seals, a re-stream of the sealed
batch returns `sealed: true` with zero re-accepted items (idempotent
retry), and an oversized item still seals at its correct index.

Existing coverage stays green: concurrent ingest of a 100-item batch,
in-flight processing never exceeding the configured concurrency,
concurrent dedup on streaming retry, and emit-position marker indexing.

## Follow-ups (not in this PR)

- SDK pre-offload of large item payloads (send `application/store` refs
instead of raw blobs) to remove object-store work from the request hot
path and shrink the request body.
- Optional `server.requestTimeout` bump as a safety net.

## CI fix

Added `.github/workflows/codeql.yml` to replace GitHub's automatic
("dynamic") CodeQL scanning. The dynamic setup was failing to upload
SARIF results because the auto-generated `GITHUB_TOKEN` lacked the
`security-events: write` permission. The explicit workflow grants that
permission at the job level and pins all actions to commit SHAs,
consistent with the repo's security conventions.

## ✅ Checklist

- [ ] I have followed every step in the [contributing
guide](https://github.com/triggerdotdev/trigger.dev/blob/main/CONTRIBUTING.md)
- [ ] The PR title follows the convention.
- [ ] I ran and tested the code works

---

## Testing

- Integration test (`streamBatchItems.test.ts`) validates correctness
and performance at concurrency 1, 10, and 50 for both large and small
payloads.
- End-to-end verified over real HTTP: 20-item/12MB batch ingests and
seals, idempotent retry returns `sealed: true`, oversized item seals at
correct index.

---

## Changelog

Streaming batch ingest now processes items with bounded concurrency
instead of one at a time, so batches of many large payloads ingest far
faster and no longer time out. Concurrency is configurable via
`STREAMING_BATCH_INGEST_CONCURRENCY` (default 10); set it to 1 for fully
sequential ingestion.

---

## Screenshots

_[Screenshots]_

💯

---------

Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
 2026-06-12 13:01:29 +01:00
Matt Aitken 5d6ea33166 refactor: share the public-token JWT scope decoder; make @trigger.dev/plugins internal (#3919) 
## What

`buildJwtAbility` — the decoder for public-token scope strings
(`read:tags:…`, `read:runs:run_abc`, `admin`, …) — now lives in
`@trigger.dev/plugins` as the single source of truth.
`@trigger.dev/rbac` re-exports it, so the built-in fallback and any auth
plugin interpret a token identically.

Scope strings are split on only the first **two** colons
(`action:type:id`), so a resource id that itself contains colons — e.g.
a tag like `user:123` — is matched in full rather than truncated to its
first segment. (The fallback already did this; this makes it the one
shared implementation.)

`@trigger.dev/plugins` is now **private (unpublished)** and gains a
`@triggerdotdev/source` export condition, so consumers bundle it from
source per-commit like `@trigger.dev/core` instead of resolving a
published version — no cross-version coordination.

## Why

Two hand-maintained copies of the scope grammar drift, and the
difference silently changes what a token grants. One shared decoder
removes that class of bug.

## Notes

- No changeset: `@trigger.dev/plugins` is now private and
`@trigger.dev/rbac` is internal — neither is published.
- Unit coverage for the colon-id path lives in
`internal-packages/rbac/src/ability.test.ts` (now exercising the shared
function).
 2026-06-12 12:44:32 +01:00
Eric Allam 78b7136bf7 chore: vouch saasjesus as a contributor (#3917) 
Vouches `saasjesus` as a contributor (vouch request #3915) so their PRs
clear the vouch check instead of being auto-closed.
 2026-06-12 11:56:45 +01:00
nicktrn de8231cb9d chore: bump shell-quote to 1.8.4 (#3913) 
Refreshes the locked `shell-quote` to 1.8.4 (transitive, lockfile-only).

<!-- This is an auto-generated description by cubic. -->
<a
href="https://cubic.dev/pr/triggerdotdev/trigger.dev/pull/3913?utm_source=github"
target="_blank" rel="noopener noreferrer"
data-no-image-dialog="true"><picture><source
media="(prefers-color-scheme: dark)"
srcset="https://www.cubic.dev/buttons/review-in-cubic-dark.svg"><source
media="(prefers-color-scheme: light)"
srcset="https://www.cubic.dev/buttons/review-in-cubic-light.svg"><img
alt="Review in cubic"
src="https://www.cubic.dev/buttons/review-in-cubic-dark.svg"></picture></a>
<!-- End of auto-generated description by cubic. -->
 2026-06-12 09:42:35 +00:00
Eric Allam 954ee5c572 fix(webapp): deliver realtime changes with current content when the read replica lags (#3910) 
## Summary

When the realtime runs feed (the backend behind the `realtimeBackend`
feature flag) hydrates a change from a Postgres read replica, the read
can race the replica's apply of the very write that triggered it. The
delivered row then carries the previous change's content, and an
isolated final change (for example a last `metadata.set` before a run
goes quiet) is not corrected until the roughly 20 second backstop poll.
Measured against a replica with deliberate apply delay, every delivery
trailed exactly one change behind and a final change stranded for the
full backstop interval.

## Fix

Publishers stamp each change record with the committed row's
`updatedAt`, taken from writes they already perform, so the stamp costs
no extra queries. The router delays its wake hydrate until the replica's
measured lag has passed, anchored to that timestamp: a record that has
already spent longer than the lag in transit is hydrated immediately, so
only the racing leading edge ever waits. After hydrating, a tripwire
compares each row against its record's watermark. Still-stale rows are
withheld and retried briefly, and each detection feeds the lag estimate.
If retries run out, the rows are delivered anyway (liveness over
freshness) and follow-up re-hydrates emit the fresh version through the
normal working-set diff once the replica catches up, with the backstop
as the terminal net.

Replica lag is sampled reader-side only, and only while feeds are
active. Aurora reports live lag via `aurora_replica_status()`; vanilla
Postgres can only report "caught up or not" (mid-apply lag is not
honestly measurable from a replica), so tripwire observations floor the
estimate there. Deployments without a replica resolve to zero lag and
skip the gate entirely. Tunables live under
`REALTIME_BACKEND_NATIVE_REPLICA_LAG_*`, and
`realtime_native.stale_hydrates` plus
`realtime_native.replica_lag_estimate_ms` make replica health
observable.

Two adjacent fixes: a metadata update that writes nothing no longer
publishes a change record, and buffered parent and root metadata
operations now publish when the flusher writes them, so those changes
wake live feeds instead of waiting for the backstop.

For local testing, `docker-compose` gains an opt-in `database-replica`
service (compose profile `replica`) with a configurable
`recovery_min_apply_delay`, which reproduces replica-lag behavior
deterministically. With the gate disabled this rig reproduces the
one-change-behind delivery exactly; with it enabled, deliveries arrive
with current content at roughly the true replica lag, across write rates
faster and slower than the lag itself.
 2026-06-12 07:34:50 +01:00
Eric Allam 8dc77c0ccd fix(webapp): only load env var values for displayed environments (#3903) 
## Summary

The environment variables page loaded every variable value in the
project, unfiltered by environment. Archiving a preview branch does not
delete its environment variable value rows, so projects that churn
preview branches accumulate values forever, and every page view loaded
all of them. On large projects this made the page loader take many
seconds and stalled the server while deserializing the oversized result.

## Fix

The presenter now loads the displayed environments first and filters the
`values` relation to those environment IDs. That matches the display
semantics exactly (per-user dev environments and active branch
environments included), and the lookup is covered by the existing unique
index on `(variableId, environmentId)`. Values in archived branch
environments are no longer fetched at all.

Covered by a new testcontainers test asserting that values from active
environments (including branch environments) are returned while archived
branch environments are excluded.
 2026-06-11 19:30:59 +01:00
Eric Allam cc9eabd14d test(webapp): use relative fixture dates in runs cursor pagination tests (#3912) 
## Summary

`test/runsRepositoryCursor.test.ts` pinned its fixture runs to
`createdAt = 2026-06-04T16:55:07Z`. `listRuns` applies the default 7 day
window when no time filter is given, so the fixtures aged out of the
window at 16:55 UTC on 2026-06-11 and all five tests started failing for
every branch, regardless of what the branch changed. The tests were
green on their own CI two days earlier because the fixtures were only
five days old at the time.

This switches the fixture base to a relative timestamp (one hour ago),
so the fixtures stay inside the default window permanently. Verified the
suite goes 5/5 green with this change on the same environment where the
pinned dates fail 5/5.
 2026-06-11 19:09:30 +01:00
Eric Allam 187c0476c3 perf(webapp): shrink run trace loader payload and add trace span cap controls (#3906) 
## Summary

The run trace page loader serialized every span's raw OTel events (with
full properties) into the response, even though the tree UI only renders
the derived `timelineEvents` and the span detail panel refetches what it
needs. On event-heavy traces that inflated both the loader payload and
the server-side heap copies built per request. This PR keeps raw span
events server-side and pairs that with a few related trace-view
improvements:

- A new optional `TRACE_VIEW_EMERGENCY_SPAN_CAP` env var (unset by
default) clamps the trace summary and detailed trace summary span limits
on both event store paths, including the public run trace endpoint, so
operators can bound trace query sizes in one place without retuning the
per-store limits.
- The TreeView virtualizer resolved every rendered row with a linear
scan over the whole tree (and `getNodeProps` did the same via
`findIndex`); rows now resolve through memoized id lookup maps, which
matters once traces reach tens of thousands of spans.
- The run stream SSE lookup now applies the same organization membership
scoping as the rest of the run page presenters, for consistency.

Behavior is unchanged by default: the trace tree renders from the same
`timelineEvents` it always has, and the new cap only takes effect when
set.
 2026-06-11 18:49:05 +01:00
Eric Allam 2b6d2492fe fix(sdk,core): head-start handover correctness and continuation boot latency (#3907) 
## Summary

Three related fixes for `chat.headStart` and continuation boots, found
while investigating customer reports.

**1. `chat.headStart` now works with `hydrateMessages`.** The turn-0
handover splice only ran on the default accumulation path, so agents
registering `hydrateMessages` silently lost the warm route's step-1
response: pure-text turns fired `onTurnComplete` with no assistant
message (and an empty durable write), tool-call turns re-ran step 1 from
scratch under a fresh `messageId`, and the head-start user message never
reached the hydrate hook at all. The first-turn history now reaches
`hydrateMessages` as `incomingMessages`, and the splice runs after both
accumulation branches, deduplicated by the handover `messageId`.

**2. Reasoning parts survive the handover.** The synthesized partial
only mapped text and tool-call parts, so an extended-thinking model's
step-1 reasoning streamed to the browser but never reached durable
history. Reasoning parts now map through with provider metadata, so
Anthropic thinking signatures survive a UIMessage round trip on hydrate
replays.

**3. Continuation boots no longer stall for ~10 seconds.** The `.in`
resume cursor was found by draining an SSE subscription that only closes
after its 5 second inactivity window, and the scan ran twice per boot.
It is now a non-blocking records read of the latest turn-complete
header, runs at most once per boot, the boot reads run concurrently, and
chat snapshots carry the cursor so subsequent boots skip the scan
entirely. Measured locally on a cancel-then-continue repro: pre-turn
continuation latency dropped from ~11s to ~0.5s.

Every fix was verified red-green: new unit tests reproduced each failure
before the fix, and end-to-end smoke tests against a live local stack
covered both handover legs, reasoning persistence with extended thinking
(including a follow-up turn that round-trips the persisted signed
reasoning back to the provider), and the boot timing comparison.

## Rollout

SDK-only; no server change required. A new SDK against a server that
does not serialize record headers degrades to the existing no-cursor
fallback. Old SDKs ignore the new snapshot field, and new SDKs fall back
to the records scan on snapshots written before it existed.
 2026-06-11 18:48:11 +01:00
DKP 93b4715967 feat(webapp): hipaa baa add-on on paid pricing tiers (#3904) 
## Summary

HIPAA BAA is offered as a paid add-on on every paid plan. Each paid tier
on the in-app pricing card now has a "HIPAA BAA add-on" row with a
"Request a BAA" link that opens the existing contact dialog pre-filled
with a new `hipaa` inquiry type, prompting the user for their company
name and a brief description of the PHI workload.

The contact form's `feedbackTypes` are restructured to match the
marketing /contact form: every inquiry type carries a Plain label ID and
a "Contact form: ..." thread title, so threads land in Plain identically
whether they come from the dashboard or the marketing site. The
included-compute line on each tier also picks up the credits wording
from the marketing pricing page, and the Enterprise tier lifts its title
above the features row.
 2026-06-11 17:32:06 +01:00
Saadi Myftija d0b2d79b3b fix(supervisor): cancel pending delayed snapshots when the run completes or disconnects (#3894)
⚒️ Publish Worker (v4) / build (supervisor) (push) Has been cancelled
Details 
The compute suspend flow delays snapshots by `snapshotDelayMs` (~30s) so
short-lived waitpoints skip the snapshot entirely, with the intent that
a run continuing before the delay expires cancels the pending snapshot.
But the only `cancel()` call site was the `/continue` action, which
runners only invoke when restoring from an already-taken snapshot — so
pending snapshots were never cancelled (zero `snapshot.canceled` events
ever emitted in prod). When a run resumed and completed inside the
window, the stale snapshot fired ~30s later anyway, pausing the VM 6–13s
mid warm-start long-poll; the frozen guest couldn't fire its abort timer
or send a FIN, causing stalls and run-engine driven retries.

### Change

- Cancel the pending snapshot on `attempt.complete` — after the platform
accepts the completion, before the HTTP reply (so it can't reorder with
the runner's next `/suspend`).
- Cancel on `runDisconnected` (crash, exit, or run replaced on the
socket).
- Both cancels are guarded by a runnerId match (new
`TimerWheel.peek()`): a stale duplicate runner for a reassigned run must
not cancel the fresh runner's pending snapshot. A missing runnerId falls
through to an unconditional cancel (the pre-existing `/continue`
behavior is unchanged).

Waitpoint suspensions keep the runner socket connected and the attempt
incomplete, so neither hook touches a snapshot that is still wanted.

Known limitation (fail-safe direction): `socket.data.runnerId` is frozen
at the websocket handshake, so after a same-supervisor restore the
disconnect-path guard refuses the cancel. The `attempt.complete` path
uses the runner's current header id and is unaffected.
re2-prod-supervisor-tri-10293 re2-test-supervisor-tri-10293 		2026-06-11 18:29:54 +02:00
Saadi Myftija 2397ca2999 fix(supervisor): retry transient instance create failures in compute workload manager (#3902) 
`ComputeWorkloadManager.create` swallows gateway errors currently, so a
cold start that fails placement (e.g. a netns slot with a busy tap, a
full node disk) silently abandons the dequeued run until the run
engine's `PENDING_EXECUTING` heartbeat timeout redrives it via stall
detection.

### Changes

- Retry `instances.create` with short backoff (default 3 attempts, 250ms
backoff), recording `createAttempts` in the wide event.
- **Only statuses where the create definitely did not commit are
retried**: 500 (agent/fcrun create failed) and 503 (no placement).
502/504 are excluded — the gateway emits those when it fails to reach
the node or read its response, which can happen *after* the agent
committed the create; the gateway only records the instance name on a
clean 201, so a same-name retry would miss the collision check and could
double-create the VM on another node. Network-level fetch failures are
retried (if the gateway processed the create, its name index is
populated and the retry 409s harmlessly). Timeouts are not retried.
- **Retry attempts after a 5xx use a deterministic `-rN` name suffix**:
a failed create can leave its name registered until async cleanup runs.
Attempt 1 keeps the unsuffixed name.
 2026-06-11 18:29:40 +02:00
Daniel Sutton 7b4443a437 test(webapp): stop streamBatchItems container tests timing out on cold start (#3900) 
Fixes an intermittent `Test timed out in 30000ms` in the
`streamBatchItems` suite. Not a logic hang — the 30s budget covers
container setup, and each case boots its own per-test Redis container +
a full `RunEngine`, so under CI Docker contention a cold boot can cross
30s (which is why the failure moved between tests).

- New `containerTestWithIsolatedRedisNoClickhouse` fixture (Postgres
clone + per-test Redis, no ClickHouse) — this suite never uses
ClickHouse, but the old fixture's auto `resetClickhouse` forced a
ClickHouse boot + migration onto the cold-start test.
- Raised `testTimeout` 30s → 120s, matching the run-engine convention
for this footprint.
 2026-06-11 13:24:21 +01:00
Oskar Otwinowski 1c7e64acde feat(supervisor): stamp org identity label on compute microVMs (#3899) 2026-06-11 11:49:56 +01:00
Eric Allam f5f29ceb26 fix(sdk,core): chat.agent delivery, idempotency, and recovery fixes (#3891) 
## Summary

A batch of reliability fixes for `chat.agent`:

- A user message sent while the agent is streaming is no longer
delivered twice (which could run a duplicate turn).
- Input appends carry an idempotency key (`X-Part-Id`) so a retried send
can't duplicate a message.
- `onTurnComplete` now fires on errored turns with the thrown error
attached, and the failed turn's user message is persisted so it isn't
lost on the next run.
- Stopping a generation clears the streaming state, so a page reload
doesn't replay the stopped turn.
- Custom agents and manual `chat.writeTurnComplete` callers trim the
output stream, sending a custom action no longer leaves a second stream
reader running, a long-lived `watch` subscription no longer grows its
dedupe set without bound, promoting a queued message to steering no
longer risks a double-send, and runs keep the full set of dashboard
tags.

The `X-Part-Id` header is accepted by current servers (they just don't
dedupe on it yet), so this is safe to ship ahead of the matching server
change.
 2026-06-11 10:35:46 +01:00
Eric Allam b82d100483 fix(webapp): harden the realtime session routes (#3890) 
## Summary

Reliability and authorization fixes for realtime chat sessions:

- Session-stream waitpoint delivery is scoped to the environment, so two
environments using the same session `externalId` can no longer complete
each other's waitpoints.
- The session snapshot-url routes now enforce per-session authorization,
and appending to a session's `out` channel requires secret-key auth, so
a session-scoped token can't read another session's snapshot or forge
assistant output.
- Appends that carry an `X-Part-Id` header are deduplicated on retry, so
a retried send can't duplicate a message.
- Session creation rejects expired sessions (instead of triggering a run
that can never receive input), `externalId` is immutable after creation,
and the sessions list endpoint returns friendly `run_*` ids to match the
single-session routes.

## Rollout

The waitpoint cache key gains an environment prefix. To keep waitpoints
registered by the previous deploy working across the boundary, the drain
reads both the new and the previous key for this release; the legacy
read can be removed a release later once no pre-deploy waitpoints
remain.
 2026-06-11 10:35:36 +01:00
Eric Allam f9d57d3bd5 feat(webapp): add a new backend for the realtime runs feed (#3864) 
## Summary

Adds a second backend for the realtime runs feed (`useRealtimeRun`,
`subscribeToRunsWithTag`, `subscribeToBatch`), built to stay healthy
when a single busy environment has many subscribers watching many runs
at once. It is gated behind a feature flag with the existing backend as
the default, so nothing changes for users until it is enabled per
environment.

## Design

A run change is published once, as a small self-describing record, to a
single per-environment channel. Every feed is then a predicate over that
one stream rather than owning a channel:

- A per-instance router indexes the currently-held feeds by run, tag,
and batch. When a run changes it hydrates the affected rows once and
serializes them once, then fans the result to every matching feed. One
hot shared tag watched by many subscribers costs a single database query
and serialize, not one per subscriber.
- Feeds that don't match a change are never woken, wake delivery per
environment is coalesced on a leading edge (250ms default) so a burst of
changes costs one wake, and cold reads coalesce onto a single
short-TTL-cached resolve.
- An admission gate bounds how many cold ClickHouse resolves run
concurrently, so a mass reconnect across many distinct filters queues
instead of stampeding the database.
- Changes that land while a client is between long-polls are delivered
on its next poll instead of waiting for the periodic backstop: each
environment buffers its recent change records, subscriptions linger
briefly after the last feed closes, and a newly-armed poll replays
exactly the connection's gap.
- The per-connection replay cursors behind that are shared across
instances via Redis (a single timestamp each), so a poll landing on a
different instance behind the load balancer still reads the connection's
true gap instead of falling back to a cold resolve. Cursor reads have a
bounded deadline and degrade to the cold-read path on any Redis trouble.
- Tag subscriptions with multiple tags match runs carrying all of the
tags, mirroring the existing backend's filter semantics, and live
long-polls hold for about 20 seconds to match its cadence.
- The per-environment channel supports Redis Cluster sharded pub/sub, so
the wake path scales horizontally across shards by environment.
- The backend reports its health through OpenTelemetry metrics (delivery
lag, poll resolution paths, backstop outcomes, replay and cursor-store
activity), with a provisioned Grafana dashboard for local development.

Everything is behind the feature flag and tunable via env vars; the
existing backend remains the default.
 2026-06-11 07:56:10 +01:00
Daniel Sutton 6afc9bfa4c fix(run-engine): retry getSnapshotsSince on the replica then primary when the read replica lags (#3889) 
## Summary

When `RUN_ENGINE_READ_REPLICA_SNAPSHOTS_SINCE_ENABLED` is on,
`RunEngine.getSnapshotsSince` reads from the read replica. During write
spikes the replica can briefly lag, so the snapshot id a runner just
learned from the writer isn't visible there yet: the lookup threw, the
worker route returned a 500, and the runner waited for its next poll —
turning sub-second snapshot notifications into poll-interval latency
exactly when things are busiest. This PR makes the flag safe to enable:
a replica miss of the since snapshot gets one jittered retry on the
replica (most lag windows are shorter than the ~50–200ms wait, so the
writer is never touched), then falls back to the primary, observed via a
new `run_engine.snapshots_since.replica_miss` counter with an `outcome`
attribute (`replica_retry` vs `primary`). Only genuine misses — absent
on the primary too — remain errors.

## Design

- `getExecutionSnapshotsSince` now throws a typed
`ExecutionSnapshotNotFoundError` so the engine can distinguish the
expected lag miss from real failures. The message string is unchanged
and the error never leaves the engine.
- The recovery path only engages when the flag is on, a distinct replica
client is configured, and no transaction client was passed. With the
flag off, the path is behaviorally identical to before.
- Retry delay bounds are configurable
(`RUN_ENGINE_SNAPSHOTS_SINCE_REPLICA_RETRY_MIN_MS`/`MAX_MS`, default
50/200; `MAX_MS=0` skips the replica retry and goes straight to the
primary).
- The warn log fires only when the primary serves the read (the writer
spill is the operationally interesting event); replica-retry recoveries
are counted but quiet. A permanently-missing snapshot id stays an
error-level failure with a `failedDuring` field, so lag metrics aren't
polluted by bogus ids.
- Stale-tail lag (replica has the since snapshot but not newer rows)
deliberately still returns the replica's view; the next poll catches up.
- The since-snapshot anchor lookup is now scoped to the polled run
(`where: { id, runId }`), so a snapshot id from a different run raises
not-found instead of silently anchoring a too-wide window of the run's
snapshots.

## Test plan

All vitest + testcontainers, no mocks. A new `schemaOnlyPrisma` fixture
(migrated-but-empty clone database) simulates a replica that hasn't
caught up, and a real in-memory OTel meter pins the counter semantics
per outcome.

- [x] Replica catches up during the jittered retry window → served by
the replica, `outcome=replica_retry` = 1, primary never consulted
- [x] Replica permanently missing the since snapshot → served by the
primary, `outcome=primary` = 1
- [x] Snapshot missing on both replica and primary → null, counter = 0
- [x] Replica has the since snapshot but lags by one → the replica's
view is served, no fallback (verified discriminating power: the test
fails if reads secretly hit the primary)
- [x] Flag off with a replica configured → primary serves the read
- [x] Transaction client provided → bypasses the replica entirely
- [x] Since snapshot belonging to a different run → null
- [x] Existing getSnapshotsSince + waitpoints suites green; run-engine,
testcontainers, and webapp typechecks pass

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
 2026-06-10 16:54:03 +00:00
Saadi Myftija 081b6bac17 feat(supervisor): publish client-side dequeue API latency as a Prometheus histogram (#3887)
⚒️ Publish Worker (v4) / build (supervisor) (push) Has been cancelled
Details 
The supervisor's dequeue round-trip time (`POST
/engine/v1/worker-actions/dequeue`) was measured but only flowed into
wide events and OTel span attributes — there was no Prometheus series,
so latency percentiles and error rates weren't queryable. This adds
`queue_consumer_pool_dequeue_duration_seconds` (histogram, label
`outcome=success|empty|error`) to the existing consumer-pool metrics,
scraped automatically by the existing ServiceMonitors on
queue-raider/schedule-raider/supervisor.

- Records every dequeue call, including failed ones, which previously
emitted no timing at all
- The pool's shared `ConsumerPoolMetrics` instance is injected into each
consumer (mirrors the `BackpressureMetrics` → `BackpressureMonitor`
wiring)
- Buckets extend to 30s because `wrapZodFetch` retries internally (5
attempts, ≥7.5s backoff before a retryable error surfaces)
- Existing `dequeueResponseMs` wide-event/span behavior unchanged
re2-prod-client-deqeueue-metrics 		2026-06-10 16:35:02 +02:00
Daniel Sutton 459dce2a97 ci: make the main-image dispatch repo and ref configurable (#3883) 
The `dispatch-main-image` job was hard-gated to
`triggerdotdev/trigger.dev` on the `main` ref. This makes it
configurable via repository variables, all defaulting to the current
values:

- `MAIN_IMAGE_DISPATCH_REPO` — the repo allowed to dispatch (default
`triggerdotdev/trigger.dev`).
- `MAIN_IMAGE_DISPATCH_REF_PREFIX` — the ref-name prefix that
dispatches, matched with `startsWith(github.ref_name, …)` (default
`main`).
- `MAIN_IMAGE_DISPATCH_TARGET` — the `repository_dispatch` target
(default `triggerdotdev/cloud`).

The job is additionally gated on `github.event_name == 'push'`. This is
necessary, not just defensive: the gate now keys off `github.ref_name`
rather than the computed image tag, and `ref_name` is still `main` when
`release.yml` invokes this workflow via `workflow_call` during a release
— so without the event guard the job would fire during every release and
fail on the absent `CROSS_REPO_PAT`. A version-equality check can't
replace it because `build-*` tags strip the prefix to the version
output.

Behaviour note: the intended dispatch paths — push to `main`, and push
of a `<prefix>*` tag in a downstream repo — are `push` events and are
unchanged. The one case that no longer dispatches is a manual
`workflow_dispatch` run of `publish.yml` on `main` (it previously did,
via the old `version == 'main'` check). That path is indistinguishable
from a manual release by event name, so `push`-only is the clean
discriminator.

Dispatching still requires `CROSS_REPO_PAT`, so setting the variables
alone doesn't enable anything.

---------

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
 2026-06-10 11:31:04 +01:00
Eric Allam 87448ccaf2 feat(webapp,core): add an endpoint to list a project's environments (#3880) 
## Summary

Adds `GET /api/v1/projects/{projectRef}/environments` (personal access
token auth), which lists the base environments a user can access for a
project — their own dev environment plus the project's staging, preview,
and production environments.

## Details

- Built on the PAT route builder, so it inherits org-membership auth and
the per-resource ability check.
- `dev` is scoped to the token owner; archived environments are
excluded.
- Returns the branchable **parent** preview environment — preview branch
children are not included. A consumer targets the parent; branch-level
overrides are handled separately.
- Sorted to match the dashboard's environment switcher (dev → staging →
preview → prod), and never returns API keys.

Example response:

```json
[
  { "id": "...", "slug": "dev",     "type": "DEVELOPMENT", "isBranchableEnvironment": false, "branchName": null, "paused": false },
  { "id": "...", "slug": "stg",     "type": "STAGING",     "isBranchableEnvironment": false, "branchName": null, "paused": false },
  { "id": "...", "slug": "preview", "type": "PREVIEW",     "isBranchableEnvironment": true,  "branchName": null, "paused": false },
  { "id": "...", "slug": "prod",    "type": "PRODUCTION",  "isBranchableEnvironment": false, "branchName": null, "paused": false }
]
```
 2026-06-10 10:13:16 +01:00
Eric Allam b28c6d0b90 fix(webapp): sanitize streamed agent URLs before rendering in the agent view (#3882) 
## Summary

The dashboard's Agent view rendered `source-url` and `file` message
parts by putting their `url` straight into an `href`/`src`. Those URLs
come from streamed agent and tool data, so a tool that emitted something
like `javascript:alert(1)` produced a clickable XSS payload in the
dashboard.

## Fix

A `toSafeUrl` helper now gates every URL before it reaches an
`href`/`src`: it allows only `http:`/`https:`/`blob:` (and
`data:image/...` for inline images) and returns `null` for anything
else. Unsafe values render as plain text instead of a link or image, so
a hostile or malformed URL degrades gracefully rather than becoming
clickable. Safe URLs render exactly as before. Covered by a unit test
over the allow/deny list.
 2026-06-10 09:52:24 +01:00
Daniel Sutton bc01f6ea3a fix(webapp): stop writer DB connectivity errors leaking to trigger() API clients (#3874) 
## Summary

During `trigger()` worker-queue resolution, `getWorkerQueue` wrapped any
error from `getDefaultWorkerGroupForProject` into a client-facing
`ServiceValidationError` (HTTP 422) carrying `error.message`. That
method runs `project.findFirst` on the **writer**; when the writer is
unreachable Prisma throws a connection error (P1001) whose message
includes the database host, and that raw message was returned to the API
client and surfaced in the run view via the SDK's `TriggerApiError`.

It also mis-classifies a transient outage: a 422 is not retried by the
SDK, so triggers failed permanently instead of riding out a brief writer
blip.

## Design

This is the only place on the trigger path that folds a *caught* error's
message into a client-facing error — every other DB failure on the path
propagates to the route's generic 500 handler (scrubbed, and retried by
the SDK). So the fix is local:

- Add `isInfrastructureError()` — true for Prisma connection-level
failures (the DB-unreachable family: P1001/P1002/P1008/P1017, plus the
init/panic/unknown client error classes), false for query/validation
errors (e.g. P2002).
- At the wrap site, rethrow infrastructure errors so they reach the
generic 500 handler (no raw message, and retryable). Genuine domain
failures (e.g. "Project not found.") still become a 422.

Only P1001 ("can't reach database server") has been observed in
practice; the rest of the connection family is included as same-class
forward-proofing.

## Test plan

- [x] Unit: `isInfrastructureError` classifies a P1001 (incl. the Prisma
6.x `PrismaClientKnownRequestError` shape) and init errors as
infrastructure; P2002 and a plain `Error` as not
- [x] `getWorkerQueue` rethrows a P1001 unchanged instead of wrapping it
in a `ServiceValidationError`; still wraps a domain failure as a
`ServiceValidationError` — RED on current code, GREEN after
- [ ] (optional) toxiproxy e2e: trigger with the writer cut → HTTP 500
generic body, no DB host in the response

---------

Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
 2026-06-10 08:53:57 +01:00
Eric Allam 3bc88c453e perf(webapp): memoize react-router per-request route matching via pnpm patch (#3877) 
## Summary

Under high request load the webapp spends most of its CPU inside
react-router's `matchRoutes`, not in application code.
`@remix-run/router@1.23.2` (the React Router v6 / Remix 2 core)
re-flattens, re-ranks, and recompiles the entire route table on every
request, and with the webapp's ~436 routes that cost dominates once
request rates climb. There is no `NODE_ENV` gate, so production pays it
too.

This adds a pnpm patch that memoizes the parts that depend only on the
static route manifest: it caches the flattened/ranked branches per route
tree, hoists the loop-invariant `decodePath` out of the match loop, and
caches compiled path regexes.

## Benchmark

CPU profile over the same load (100 concurrent tag feeds, ~425 req/s),
`NODE_ENV=production`, before vs after the patch:

| Metric | Before | After |
| --- | --- | --- |
| Active CPU (self-time over the window) | 28.3s | 18.5s (-34%) |
| Route-matching self-time | 19.2s | 7.5s (-61%) |
| Event-loop lag p99 | 322ms | 113ms (-65%) |
| Idle headroom | 26% | 52% |

Application/realtime code was ~0% of CPU in both profiles; the
bottleneck was entirely generic per-request route matching.

## Why a patch instead of an upgrade

The inefficiency is acknowledged upstream
([remix-run/react-router#8653](https://github.com/remix-run/react-router/issues/8653)).
A contributor PR doing exactly this
([remix-run/react-router#14866](https://github.com/remix-run/react-router/pull/14866))
was closed in favor of a narrower fix
([remix-run/react-router#14967](https://github.com/remix-run/react-router/pull/14967),
branch caching only, shipped in React Router v7), with the maintainer
suggesting patch-package as the interim until the Remix 3 route-pattern
rewrite (see
[remix-run/remix#4786](https://github.com/remix-run/remix/discussions/4786)).
We are on the v6-era core and cannot pick up even the partial fix
without a framework migration, so this patch is the sanctioned stopgap,
and it also includes the compiled-regex cache the merged PR left out.


[`patches/README.md`](https://github.com/triggerdotdev/trigger.dev/blob/perf/react-router-route-matching/patches/README.md)
documents the full rationale, the safety argument (deterministic,
internal-only, bounded caches), and when to remove the patch.
 2026-06-09 22:37:26 +01:00
Eric Allam d97335902b docs: add a database connections guide for tasks (#3881) 
## Summary

A new guide for connecting a database to your tasks: where to create the
client, how to size the connection pool against your provider's limit,
when to reach for a pooler, and how to release connections at waits so
you don't hit "too many connections" or crash on resume.

It covers node-postgres, Prisma, Drizzle, and MongoDB, with researched
direct and pooled connection limits for the common Postgres providers
(Supabase, Neon, RDS, PlanetScale) and MongoDB Atlas. The page lives
under Documentation, Troubleshooting, and is linked from the chat agent
docs (overview, lifecycle hooks, chat.local, and the database
persistence pattern).
 2026-06-09 22:24:51 +01:00
Oskar Otwinowski 774a9793e3 fix(webapp): paginate shared Vercel env var fetch in onboarding + pull (#3879) 
Closes #3850

<img width="1709" height="1115" alt="Screenshot 2026-06-09 at 18 44 40"
src="https://github.com/user-attachments/assets/edc1f091-0937-403d-a6e9-9d9477e77ed0"
/>
 2026-06-09 18:03:56 +00:00
Matt Aitken df964ea4ee feat(ci): dispatch a repository event when the main webapp image is published (#3875) 
## Summary

On every `main` build, once the webapp image is pushed to the registry,
the publish workflow emits a cross-repo `repository_dispatch` event
(`main-image-published`) carrying a digest-pinned image ref. Other
repositories in the org can subscribe to that event and build or deploy
from the exact artifact, instead of chasing the moving `main` tag.

## Design

`publish-webapp.yml` now exposes the pushed multi-arch index digest as a
workflow output. `publish.yml` adds a `dispatch-main-image` job (after
`publish-webapp`) that builds `<image_repo>@<digest>` and sends the
dispatch via the same pinned `peter-evans/repository-dispatch` action
already used elsewhere in this repo, authed with `CROSS_REPO_PAT`.

It fires only when the published tag is `main`, so semver releases and
other tag builds are excluded, and only from the canonical repo so forks
never dispatch. The payload is JSON-escaped with `jq`.
 2026-06-09 14:20:06 +01:00
Matt Aitken 1b0f2c71dd fix(webapp): correct backward pagination slice in listRunIds (#3867) 
## Problem

Two backward-pagination bugs in `ClickHouseRunsRepository.listRunIds`,
both pre-existing (they predate the composite-cursor work in #3852 and
were spotted during/after it):

**1. Wrong slice (straddled pages).** `listRunRows` fetches `page.size +
1` rows to detect `hasMore`. That extra row is the one *farthest from
the cursor* in both directions (forward orders DESC; backward orders
ASC), so it's always the *trailing* element. Forward correctly used
`rows.slice(0, size)`, but backward+`hasMore` used `rows.slice(1, size +
1)` — dropping the row *closest* to the cursor and keeping the has-more
sentinel. The page straddled two logical pages (one run from the correct
previous page + one from the page before it), so paging "newer" across a
boundary **repeated and skipped** runs.

**2. Stranded forward cursor on a partial backward page.** In the
backward `!hasMore` branch, `nextCursor` was `reversedRows.at(page.size
- 1)`. On a partial page (fewer than `page.size` rows — reachable via
`runs.list` by passing a forward page's cursor as `page[before]`), that
index overshoots → `undefined` → `nextCursor` becomes `null`, leaving no
way to page forward again.

## Fix

- **Slice:** both directions now slice `rows.slice(0, size)` (the
sentinel is the trailing element either way).
- **Partial-page cursor:** the backward `!hasMore` branch takes the
oldest row on the page, `rows.at(0)`, for `nextCursor` — equivalent to
the old expression for full pages, correct for partial ones.

Forward pagination, the cursor *values* for full pages, and the `hasMore
=== true` paths were already correct and are unchanged.

## Tests

`runsRepositoryCursor.test.ts` gains two cases (both fail on `main`,
pass here):

- **multi-page backward walk:** forward across all pages, then backward
from the last page — each backward page must *exactly* reproduce the
corresponding forward page (no straddling: `main` returns `{b,c}`
instead of `{c,d}`), and the full traversal covers every run once.
- **partial backward page:** backward onto a partial first page must
still expose a working forward cursor (and paging forward from it
reaches the rest) — `main` returns a `null` nextCursor.

The three existing cursor tests (forward completeness, backward
round-trip, legacy cursor) still pass.

---------

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
 2026-06-09 12:11:11 +01:00
Matt Aitken f4a96bdf84 Fail dev and deploy on duplicate task ids (#3865) 
## What

`dev` and `deploy` now fail with a clear error when two tasks are
defined with the same id — including across task types (e.g. a scheduled
task and a regular task sharing an id).

## Why

Tasks are registered into the resource catalog keyed by id, so a second
definition with the same id silently overwrote the first. One of the
tasks would just vanish from the worker with no warning — easy to miss,
hard to debug. (Any earlier duplicate-id check ran against the
post-registration task list, which is already de-duplicated, so it never
actually fired.)

## How

- **Detect at registration** (`@trigger.dev/core`):
`StandardResourceCatalog` records a collision when a task id is
registered more than once, capturing the files involved — the only point
where both definitions are visible before the id-keyed map collapses
them. Exposed via `listTaskIdCollisions()`.
- **Fail indexing** (`trigger.dev` CLI): both index workers report
collisions via a new `TASKS_FAILED_TO_INDEX` message;
`indexWorkerManifest` rejects with a new `DuplicateTaskIdsError`. `dev`
renders a dedicated error (offending ids + files + docs link); `deploy`
fails with the same message. Runtime worker boot is unaffected — it
never reads the collisions.
- **Server-side backstop** (webapp): background-worker registration also
rejects duplicate ids with a clear `ServiceValidationError`, so
duplicates are caught even from an older CLI.

## Testing

- Unit tests for collision collection in the catalog and for the
error-message formatting (standard, same-file, and 3+-definition cases).
- Verified end to end against a local webapp: a project with a regular
task and a scheduled task sharing an id now fails `dev` with the
dedicated error; a project with distinct ids still starts normally.

## Changeset

Patch for `@trigger.dev/core` and `trigger.dev`.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
 2026-06-09 12:10:49 +01:00
Matt Aitken 6bcd369ea1 feat(webapp,rbac): REQUIRE_PLUGINS=1 fail-fast for required plugin loads [TRI-9852] (#3734) 
## Summary

- `internal-packages/rbac/src/index.ts` — in `LazyController.load()`'s
catch block, throw an Error when `process.env.REQUIRE_PLUGINS === "1"`
instead of silently falling back. The throw is captured into the lazy
controller's init promise, so it surfaces on the first method call.
- `apps/webapp/app/routes/healthcheck.tsx` — `await
rbac.isUsingPlugin()` after the DB ping. With `REQUIRE_PLUGINS=1` and a
failed plugin load, the throw surfaces here and the healthcheck returns
500 → readiness probe fails → rollout is rolled back. Noop for
self-hosters.
- `.server-changes/require-plugins-fail-fast.md` — server-changes entry.
- `internal-packages/rbac/src/require-plugins.test.ts` — 4 unit tests
covering loader branching: unset → fallback, `=1` → throw,
`forceFallback: true` wins, only exactly `"1"` enforces.
- `internal-packages/testcontainers/src/webapp.ts` — adds
`requirePlugins?: boolean` to `StartWebappOptions`. Implies
`forceRbacFallback: false`.
- `apps/webapp/test/healthcheck-require-plugins.e2e.test.ts` — e2e
closes the loop: spawns a real webapp, hits `/healthcheck` via HTTP,
asserts 500 with `REQUIRE_PLUGINS=1` and 200 without.

## Motivation

Today the RBAC plugin loader catches any plugin-load failure (missing
module, broken transitive dep, init throw) and silently returns the
default fallback implementation. This is the correct behaviour for
self-hosters who don't ship the plugin — but it's dangerous in
deployments where the plugin is expected to load: an
accidentally-missing or broken plugin would silently disable
enforcement.

`REQUIRE_PLUGINS=1` makes the loader fail loudly in those deployments.
The variable name is intentionally plural and generic — future plugin
contracts (audit logs, SSO) can read the same flag without renaming.

Closes
[TRI-9852](https://linear.app/triggerdotdev/issue/TRI-9852/require-plugins1-fail-fast-for-required-plugin-loads).

## Test plan

- [x] `pnpm run test --filter @trigger.dev/rbac` — 38/38 tests pass,
including the 4 new loader tests
- [x] `pnpm run typecheck --filter webapp` — passes
- [x] `pnpm run typecheck --filter @trigger.dev/rbac --filter
@internal/testcontainers` — passes
- [x] e2e test added (`healthcheck-require-plugins.e2e.test.ts`) — CI
runs it via `e2e-webapp.yml`. Couldn't run locally (no Docker daemon
up); CI has Docker provisioned.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-06-09 12:08:20 +01:00
Eric Allam 0ee1461d86 fix(webapp): show scheduled runs under their correct region (#3873) 
## Summary

Runs routed onto a dedicated scheduled worker queue were showing under a
phantom region in the dashboard, run details, and the API, and slipped
through region filters. They now resolve to their real region
everywhere.

## Fix

A worker queue can carry a `:scheduled` suffix that routes
scheduled-lineage runs onto their own list. That suffix is an internal
routing detail, but it was leaking anywhere the worker queue is read as
a region. A `baseWorkerQueue` helper strips any `:<class>` suffix back
to the base region (region names never contain a colon, so it's
everything before the first colon) and is applied at every region read
site: the runs list, run detail, the public API, and replay's region
override. The runs-replication writer also stores the base region in
ClickHouse so the region filter matches.
 2026-06-09 11:46:44 +01:00
Eric Allam 18b90285b2 feat(cli): set up AI tooling in trigger init and add getting-started skill (#3872) 
## Summary

`trigger init` now sets up your AI coding assistant as part of project
setup. Instead of the old either/or "MCP or CLI" prompt, it offers the
MCP server and agent skills together, then asks whether to scaffold with
the CLI or let your assistant do it.

A new `getting-started` agent skill backs that hand-off: it teaches the
assistant the bootstrap recipe (install the SDK, write
`trigger.config.ts`, scaffold a first task, wire tsconfig/gitignore, run
`trigger dev`) and is explicit about the two steps that genuinely need a
human (`trigger login` and copying the DEV secret key from the
dashboard). It ships in the CLI alongside the existing skills,
version-matched to your SDK.

Prompt-once gating is shared, so opting in or out during `init` means
`trigger dev` won't ask about skills again.
 2026-06-09 11:41:32 +01:00
Daniel Sutton e9c459fb8d ci: allow forks to override published container image namespace (#3866) 
## Summary

The container publish workflows hardcoded `ghcr.io/triggerdotdev/...` as
the image destination. As a result, a fork that builds on push-to-`main`
(or on the worker publish tags) would attempt to push to — and attest —
the upstream packages rather than its own, which fails on permissions
and is surprising besides.

This makes the image destination configurable via a single
`IMAGE_REGISTRY` repository variable, while leaving the upstream
defaults byte-identical:

- **Single source of truth** (`publish.yml`): a `resolve-registry` job
resolves the target registry namespace once — `IMAGE_REGISTRY`
repository variable, defaulting to `ghcr.io/${{ github.repository_owner
}}` — and passes it down to every publish job as an `image_registry`
input. So a fork publishes to its own namespace automatically with no
configuration.
- **Webapp** (`publish-webapp.yml`): the image now lives at
`<registry>/<repo-name>` (e.g. `ghcr.io/<owner>/trigger.dev`). The
provenance attestation and the downstream Trivy scan follow the same
computed repo via the `image_repo` workflow output.
- **Workers** (`publish-worker.yml`, `publish-worker-v4.yml`): build
under `<registry>/<worker-name>`. They keep a `vars.IMAGE_REGISTRY ||
ghcr.io/<owner>` fallback so they still resolve correctly on their
direct `infra-*` / `re2-*` push triggers (which bypass the parent
workflow).

A single `IMAGE_REGISTRY` namespace variable now governs both webapp and
workers (the earlier `WEBAPP_IMAGE_REPO` full-path override is dropped,
removing the full-path/namespace asymmetry). When `IMAGE_REGISTRY` is
unset, every resolved image name is exactly what it is today, so there
is no change for this repo.

## Test plan

- [x] `actionlint` passes on all four workflows
- [ ] On merge, confirm the webapp publish still pushes
`ghcr.io/triggerdotdev/trigger.dev:main` + the commit-SHA tag (defaults
unchanged)

---------

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
 2026-06-09 09:37:39 +01:00
Daniel Sutton d7028e278e feat(webapp): label mollifier decisions by enrolled org (#3869) 
## Summary

The `mollifier.decisions` metric only carried an `outcome` label, so for
an org that has the mollifier enabled there was no way to see how often
its triggers pass through the gate instead of being diverted — making it
hard to tell why the trip isn't firing for an opted-in org.

This adds two bounded labels: `enrolled` (`"true"`/`"false"`, the
per-org flag) and `org` (the org id, attached **only** when `enrolled`
is true). For an enrolled org you can now compare directly:
`mollifier.decisions{outcome="pass_through", enrolled="true",
org="<id>"}` vs `{outcome="mollify", ...}`.

## Design

`recordDecision` now takes an options object (`{ reason?, enrolled,
orgId? }`). The `org` label is restricted to the enrolled cohort to keep
cardinality bounded — the guard lives in a pure `decisionLabels` helper,
so a non-enrolled org id can never be attached even if one is passed.
The enrolled set is small and capped operationally.

The per-org flag is resolved once at the top of `evaluateGate`
(in-memory, no DB round-trip on the trigger hot path) so every decision
— including the debounce / one-time-use-token / triggerAndWait bypasses
— is labelled consistently.

## Test plan

- [x] `mollifierGate.test.ts` cascade asserts `enrolled`/`org` on every
gate branch
- [x] `mollifierDecisionLabels.test.ts` (new) proves `org` is dropped
for non-enrolled even when an id is passed (cardinality guard)
- [x] `vitest run mollifierGate mollifierDecisionLabels` — 34/34 pass
- [x] `pnpm run typecheck --filter webapp` clean

Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
 2026-06-08 17:02:30 +00:00
Oskar Otwinowski 93532cdb99 feat(supervisor): forward per-run labels to the compute provider (#3821) 
Add an optional network_labels field to the internal compute client's
create and restore request schemas and forward per-VM endpoint labels on
both paths, so a restored VM keeps the same labels as a freshly-booted
one. Mirrors the label the Kubernetes workload manager already sets on
the run pod.

---------

Co-authored-by: nicktrn <55853254+nicktrn@users.noreply.github.com>
 2026-06-08 18:45:01 +02:00
Eric Allam 8b85da1b26 feat(cli): install Trigger.dev agent skills into your coding agent (#3868) 
## Summary

`trigger skills` installs Trigger.dev agent skills into your coding
agent so it knows how to write Trigger.dev code: tasks, schedules,
realtime, and `chat.agent` AI agents. The skills are `SKILL.md` files
(the open Agent Skills format) bundled with the CLI and copied into each
tool's native skills directory (Claude Code, Cursor, GitHub Copilot, and
Codex / `AGENTS.md`), version-matched to the CLI you run. `trigger dev`
offers to install them on first run, and a one-line always-on pointer is
written into your `CLAUDE.md` / Cursor rules / etc. so the agent always
knows which skills are available and loads the right one on demand.

This replaces the old `install-rules` command, which stays as an alias.
Four skills ship to start: `authoring-tasks`, `realtime-and-frontend`,
`authoring-chat-agent`, and `chat-agent-advanced`.
 2026-06-08 17:44:02 +01:00
Matt Aitken ef04cc39ef fix(webapp): use composite keyset cursor for run pagination (#3852) 
## Problem

`ClickHouseRunsRepository.listRunIds` / `listRuns` order results by the
composite key `(created_at, run_id)`, but the cursor predicate cut on
`run_id` **alone**:

```ts
.where("run_id < {runId: String}", { runId: cursor })
.orderBy("created_at DESC, run_id DESC")
```

This is only sound when `run_id` lexicographic order matches
`created_at` order. `run_id`s are cuids — only coarsely time-sortable —
so when a burst of runs is created within a sub-second window, the two
orders can diverge. When they do, the next-page predicate (`run_id <
cursor`, where `cursor` is the *last* page element = the smallest
`created_at`, not necessarily the smallest `run_id`):

- **re-includes** rows already returned on a previous page (duplicates),
and
- **skips** rows it should have returned (silent data loss).

For bulk **replay** this caused runs to be replayed more than once
(replay has no idempotency guard). For the dashboard and the `runs.list`
API it could silently repeat or skip runs at page boundaries.

## Fix

Make the cursor predicate match the composite ordering:

- Cursors now encode the full `(created_at, run_id)` key as an **opaque
URL-safe base64 token**
(`base64url({"c":<createdAtMs>,"r":"<runId>"})`), and the query cuts on
the matching tuple — `(created_at, run_id) < (…)` forward / `> (…)`
backward.
- The `ORDER BY` is unchanged, so the query stays aligned with the
table's primary key — no performance regression (the tuple range
predicate is actually more index-friendly than `run_id <` alone).
- Cursors are **server-issued opaque tokens** (the SDK only echoes
`pagination.next` / `pagination.previous` back), so this needs **no
client/SDK update**. Legacy cursors were the bare internal `run_id`;
they're detected by decode failure (a cuid isn't a valid base64-wrapped
JSON payload) and fall back to the old `run_id`-only predicate, so
in-flight cursors keep working and drain naturally. New cursors also no
longer expose a bare internal run id.
- `listRunIds` is now the single cursor-aware list primitive: it returns
`{ runIds, pagination: { nextCursor, previousCursor } }`, and `listRuns`
builds on it (one place constructs cursors). Bulk actions consume the
same method and advance by `pagination.nextCursor`, finishing when it's
`null`.
- `getTaskRunsQueryBuilder` now also selects
`toUnixTimestamp64Milli(created_at) AS created_at_ms`, using a dedicated
`TaskRunListQueryResult` schema. The shared `TaskRunV2QueryResult` stays
`run_id`-only so the run-engine pending-version lookup
(`getPendingVersionIdsQueryBuilder`, which selects only `run_id`)
doesn't fail validation on a column it doesn't query.

## Tests

New `runsRepositoryCursor.test.ts` (testcontainer-backed, real
Postgres→ClickHouse replication):

- **forward** pagination returns every run exactly once when `run_id`
order is the reverse of `created_at` order (reproduces the
duplicate/skip bug — fails on `main`; this
walk-until-`nextCursor`-null-and-assert-complete is exactly the bulk
action's iteration),
- **backward** pagination round-trips to the previous page across a
boundary,
- **legacy** bare-`run_id` cursor still uses the old predicate
(backwards compatibility).

The existing `runsRepository` suites (part1–4) still pass; `part4`'s
`count new runs with listRunIds` test was updated for the new `{ runIds,
pagination }` return shape, and the `clickhouse` `taskRuns`
query-builder snapshots were regenerated for the added `created_at_ms`
column.

## Notes

- Separate, pre-existing issue (out of scope, not introduced here):
`listRuns`' backward display-slicing (`rows.slice(1, size+1)` when
`hasMore`) has an off-by-one that can return a straddled page. Tracked
separately.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
 2026-06-08 12:57:47 +01:00
nicktrn f261ff2b85 chore(docker): tidy dev postgres + clickhouse images (#3859) 
Two small hygiene tweaks to **dev-only** images:

- `docker/Dockerfile.postgres`: add `--no-install-recommends` to the
partman install (leaner image, skips unneeded recommended packages).
- `internal-packages/clickhouse/Dockerfile`: run the migration helper as
a non-root user.

Both are local-dev images (the `pnpm run docker` stack) - no impact on
the published webapp image, prod, or self-hosting.
 2026-06-07 12:22:56 +01:00
nicktrn fa15438e42 perf(ci): speed up unit tests with LPT sharding + container scoping (#3855) 
Speeds up and de-flakes the unit-test suite: testcontainers booted once
per vitest worker (per-test isolation kept only where a test runs
background redis work that outlives it), a duration-weighted shard
sequencer so each shard does roughly equal work, the slowest suites
split, two genuine flakes fixed (`streamBatchItems` shared-redis leak;
run-engine waits that relied on fixed sleeps), and transient DockerHub
pulls retried.

**Timings (CI, per-shard wall):** worst unit-test shard ~771s → ~294s;
packages/webapp shards ~250-270s, most internal ~190-240s. All 25 shards
green.

A shard breaks down as ~70s fixed setup (install / image-pull /
generate) + ~70s cold `^build` + the actual container tests. So the
remaining cost is mostly the tests themselves plus that fixed setup.

**Next (separate, timings):**
- **typecheck (~6m24s)** — the slowest check overall; bound by
full-graph `tsc`, not the TS version (a TS6 branch is still ~6m17s). The
real lever is **tsgo** (the Go compiler).
- Possible later: turbo CI caching could trim the ~70s cold build on
*warm* runs, but it's conditional (cold runs rebuild anyway) and doesn't
touch setup or test time — secondary.

`cli-v3` e2e and `sdk-compat` are path-gated (don't run on test-infra
changes) and already comfortably fast.
 2026-06-07 12:00:32 +01:00
Eric Allam 97036fb741 feat(webapp,clickhouse): export run traces as log, markdown, or jsonl (#3851) 
## Summary

Adds a trace export to the run page. From the new **Export trace** menu
you can
copy a run's full trace to the clipboard as Markdown (for pasting into
an AI
assistant) or download it as a flat Log, a Markdown table, or JSON
Lines.
Internal engine-debug events are filtered out by default, and errors are
surfaced inline with their message.

## Design

The export streams events from the store to the gzipped response one at
a time
and never materialises the span tree, so a trace of any size exports
with
bounded memory and without stalling the server. Output is flat and
chronological: each line carries its own `spanId ← parentSpanId`, so the
hierarchy is reconstructable without nesting. Formats share a single
streaming
pipeline and are pluggable via `?format=log|jsonl|markdown`, so adding a
format
is an isolated change.

## Screenshots

**Export menu**

<img width="370" height="252" alt="trace-export-menu"
src="https://github.com/user-attachments/assets/3d10304a-8c49-4606-b15d-2859b137419f"
/>


**In context**

<img width="2400" height="1802" alt="trace-export-run-page"
src="https://github.com/user-attachments/assets/46c80b30-303b-47c6-9ace-a2fb06f6cb61"
/>
 2026-06-06 21:24:37 +01:00
Eric Allam fa4804e6a7 chore(core,sdk): move the AI SDK v7 forward-compat typecheck out of CI (#3854) 2026-06-06 18:34:27 +01:00
nicktrn 707bf1adb4 ci: reduce unit test flakiness and shard re-run cost (#3844) 
A unit-test shard recently failed on a timing race rather than a real
regression - a run-engine waitpoint test sleeps 1250ms waiting on a
1000ms timeout that's processed by a ~1000ms worker poll, so on a
CPU-starved shard the margin evaporates and the whole matrix goes red.
Because `fail-fast` defaults on, that one flake cancels the sibling
shards, and the only recovery is re-running the entire matrix "just to
be sure" - which is itself slow.

This is the low-risk first pass at that pain:

- `fail-fast: false` on the webapp and internal shard matrices, so one
flaky shard no longer cancels its siblings. "Re-run failed jobs" now
re-runs just the failed shard instead of the whole matrix.
- CI-scoped `retry: process.env.CI ? 2 : 0` on the timing-sensitive
packages (`run-engine`, `redis-worker`, `schedule-engine`). Flakes
self-heal in CI; local runs stay at `retry: 0` so they still surface in
dev. A stopgap until the timing tests are made deterministic.
- `fetch-depth: 1` on the unit-test checkouts - they don't use git
history, so the full clone was wasted setup time across ~20 jobs.
- Reconcile the pre-pull image tags with what testcontainers actually
pulls (`redis:7-alpine` -> `redis:7.2`, `ryuk:0.11.0` -> `ryuk:0.14.0`)
and add `minio/minio:latest` to the webapp pre-pull. Otherwise those
images pull unauthenticated at test time and risk Docker Hub rate-limit
flakes (worst on fork PRs, where the authenticated pre-pull is skipped
entirely).

Deeper follow-ups - bigger runners, turbo remote cache, runtime-weighted
sharding, and the real root-cause fix (container reuse / template-DB
isolation + deterministic timing tests) - are tracked under TRI-10484.
 2026-06-05 17:59:11 +01:00
nicktrn 16d59aa9e7 chore: harden webapp docker image (#3845) 
Hardens the webapp Docker image and adds a CVE scan of each published
image.

- Base image `bullseye-slim` → `bookworm-slim` (Debian 12), pinned by
digest. Adds `apt-get upgrade` + `--no-install-recommends` + apt-cache
cleanup across the build stages so OS packages are patched at build
time.
- Moves the `react-email` CLI to `devDependencies` in
`internal-packages/emails` — only the `email dev` preview script uses
it; the runtime render path is `@react-email/render` +
`@react-email/components`. This also drops the bundled `esbuild` binary
from the production image.
- Bumps `goose` v3.26.0 → v3.27.1 and its Go builder image 1.23 → 1.26.
- Adds a reusable Trivy image-scan workflow wired into `publish.yml`, so
every published image (main builds and releases) is scanned for
OS-package CVEs right after it's pushed to GHCR. Report-only (writes to
the run summary), runs alongside the worker publishes so it never blocks
a deploy.

Verified locally: the image builds clean on the new base, and
`@react-email/render` carries no `esbuild` dependency so email rendering
is unaffected.
 2026-06-05 17:52:42 +01:00
Eric Allam 96f4c1bf2c chore(core,sdk): make the ai-v7 typecheck pass deterministic (#3847) 
## Summary

The SDK and core packages run a second, forward-compat typecheck pass
(`tsc --noEmit -p tsconfig.ai-v7.json`) that remaps the `"ai"` import to
the ESM-only AI SDK 7 canary, so we catch source that only compiles
against one major. That pass inherited `composite: true` from the base
tsconfig, which makes `tsc` write a `.tsbuildinfo` even under
`--noEmit`.

Incremental buildinfo caches each file's resolved module format (CJS vs
ESM) and module resolution. When that state goes stale or is replayed,
the v7 pass can report spurious `TS1479` ("CommonJS module ... cannot
`require` an ECMAScript module") errors on the `"ai"` import even though
the source is fine in a clean checkout. Because this pass shares the
typecheck job that gates the Docker image publish, a spurious failure
there blocks publishing.

## Fix

Set `composite: false` and `incremental: false` on both
`tsconfig.ai-v7.json` files. The pass is `--noEmit` only, so it never
needed incremental state. Now each run is a clean, full check that
writes no buildinfo and can't replay stale resolution.

Verified: both `@trigger.dev/sdk` and `@trigger.dev/core` typecheck
green, and neither writes an ai-v7 `.tsbuildinfo` anymore.
 2026-06-05 14:13:47 +00:00
Katia Bulatova 4711adef84 fix(webapp): don't reload runs list when toggling bulk action inspector (#3841) 
## Summary

Opening or closing the Bulk action inspector should not affect the Runs
list, but it was still triggering refresh-related UI behavior. This PR
fixes that and smooths out a few related inspector interactions.

## Changelog

Stop reloading the runs list (and flashing its loading state) when
opening or closing the
Bulk action inspector. Filtering, pagination, and explicit refresh are
unaffected.
 2026-06-05 15:52:35 +02:00