Disable the execution workers and batch consumers in the worker queue
observation test. It only needs enqueue + processMasterQueue + the observer
gauge, and the extra workers add Redis connections and make engine.quit()
hang on worker shutdown when the shard's Redis is under pressure, timing the
test out in CI.
The runqueue.workerQueue.length gauge only observed worker queues that a
dequeue had registered, so a queue's depth stopped being reported once
dequeues stopped (or was never reported for a queue that backed up before
anything dequeued from it). A periodic observer now refreshes the observed
set from the WorkerInstanceGroup records instead, so every active worker
queue (and its scheduled split variant) keeps reporting its length
regardless of dequeue activity. Off by default; enable per-service via an
env var. Reads from the replica and skips configured cloud providers.
## Summary
When the realtime runs feed (the backend behind the `realtimeBackend`
feature flag) hydrates a change from a Postgres read replica, the read
can race the replica's apply of the very write that triggered it. The
delivered row then carries the previous change's content, and an
isolated final change (for example a last `metadata.set` before a run
goes quiet) is not corrected until the roughly 20 second backstop poll.
Measured against a replica with deliberate apply delay, every delivery
trailed exactly one change behind and a final change stranded for the
full backstop interval.
## Fix
Publishers stamp each change record with the committed row's
`updatedAt`, taken from writes they already perform, so the stamp costs
no extra queries. The router delays its wake hydrate until the replica's
measured lag has passed, anchored to that timestamp: a record that has
already spent longer than the lag in transit is hydrated immediately, so
only the racing leading edge ever waits. After hydrating, a tripwire
compares each row against its record's watermark. Still-stale rows are
withheld and retried briefly, and each detection feeds the lag estimate.
If retries run out, the rows are delivered anyway (liveness over
freshness) and follow-up re-hydrates emit the fresh version through the
normal working-set diff once the replica catches up, with the backstop
as the terminal net.
Replica lag is sampled reader-side only, and only while feeds are
active. Aurora reports live lag via `aurora_replica_status()`; vanilla
Postgres can only report "caught up or not" (mid-apply lag is not
honestly measurable from a replica), so tripwire observations floor the
estimate there. Deployments without a replica resolve to zero lag and
skip the gate entirely. Tunables live under
`REALTIME_BACKEND_NATIVE_REPLICA_LAG_*`, and
`realtime_native.stale_hydrates` plus
`realtime_native.replica_lag_estimate_ms` make replica health
observable.
Two adjacent fixes: a metadata update that writes nothing no longer
publishes a change record, and buffered parent and root metadata
operations now publish when the flusher writes them, so those changes
wake live feeds instead of waiting for the backstop.
For local testing, `docker-compose` gains an opt-in `database-replica`
service (compose profile `replica`) with a configurable
`recovery_min_apply_delay`, which reproduces replica-lag behavior
deterministically. With the gate disabled this rig reproduces the
one-change-behind delivery exactly; with it enabled, deliveries arrive
with current content at roughly the true replica lag, across write rates
faster and slower than the lag itself.
## Summary
The environment variables page loaded every variable value in the
project, unfiltered by environment. Archiving a preview branch does not
delete its environment variable value rows, so projects that churn
preview branches accumulate values forever, and every page view loaded
all of them. On large projects this made the page loader take many
seconds and stalled the server while deserializing the oversized result.
## Fix
The presenter now loads the displayed environments first and filters the
`values` relation to those environment IDs. That matches the display
semantics exactly (per-user dev environments and active branch
environments included), and the lookup is covered by the existing unique
index on `(variableId, environmentId)`. Values in archived branch
environments are no longer fetched at all.
Covered by a new testcontainers test asserting that values from active
environments (including branch environments) are returned while archived
branch environments are excluded.
## Summary
`test/runsRepositoryCursor.test.ts` pinned its fixture runs to
`createdAt = 2026-06-04T16:55:07Z`. `listRuns` applies the default 7 day
window when no time filter is given, so the fixtures aged out of the
window at 16:55 UTC on 2026-06-11 and all five tests started failing for
every branch, regardless of what the branch changed. The tests were
green on their own CI two days earlier because the fixtures were only
five days old at the time.
This switches the fixture base to a relative timestamp (one hour ago),
so the fixtures stay inside the default window permanently. Verified the
suite goes 5/5 green with this change on the same environment where the
pinned dates fail 5/5.
## Summary
The run trace page loader serialized every span's raw OTel events (with
full properties) into the response, even though the tree UI only renders
the derived `timelineEvents` and the span detail panel refetches what it
needs. On event-heavy traces that inflated both the loader payload and
the server-side heap copies built per request. This PR keeps raw span
events server-side and pairs that with a few related trace-view
improvements:
- A new optional `TRACE_VIEW_EMERGENCY_SPAN_CAP` env var (unset by
default) clamps the trace summary and detailed trace summary span limits
on both event store paths, including the public run trace endpoint, so
operators can bound trace query sizes in one place without retuning the
per-store limits.
- The TreeView virtualizer resolved every rendered row with a linear
scan over the whole tree (and `getNodeProps` did the same via
`findIndex`); rows now resolve through memoized id lookup maps, which
matters once traces reach tens of thousands of spans.
- The run stream SSE lookup now applies the same organization membership
scoping as the rest of the run page presenters, for consistency.
Behavior is unchanged by default: the trace tree renders from the same
`timelineEvents` it always has, and the new cap only takes effect when
set.
## Summary
Three related fixes for `chat.headStart` and continuation boots, found
while investigating customer reports.
**1. `chat.headStart` now works with `hydrateMessages`.** The turn-0
handover splice only ran on the default accumulation path, so agents
registering `hydrateMessages` silently lost the warm route's step-1
response: pure-text turns fired `onTurnComplete` with no assistant
message (and an empty durable write), tool-call turns re-ran step 1 from
scratch under a fresh `messageId`, and the head-start user message never
reached the hydrate hook at all. The first-turn history now reaches
`hydrateMessages` as `incomingMessages`, and the splice runs after both
accumulation branches, deduplicated by the handover `messageId`.
**2. Reasoning parts survive the handover.** The synthesized partial
only mapped text and tool-call parts, so an extended-thinking model's
step-1 reasoning streamed to the browser but never reached durable
history. Reasoning parts now map through with provider metadata, so
Anthropic thinking signatures survive a UIMessage round trip on hydrate
replays.
**3. Continuation boots no longer stall for ~10 seconds.** The `.in`
resume cursor was found by draining an SSE subscription that only closes
after its 5 second inactivity window, and the scan ran twice per boot.
It is now a non-blocking records read of the latest turn-complete
header, runs at most once per boot, the boot reads run concurrently, and
chat snapshots carry the cursor so subsequent boots skip the scan
entirely. Measured locally on a cancel-then-continue repro: pre-turn
continuation latency dropped from ~11s to ~0.5s.
Every fix was verified red-green: new unit tests reproduced each failure
before the fix, and end-to-end smoke tests against a live local stack
covered both handover legs, reasoning persistence with extended thinking
(including a follow-up turn that round-trips the persisted signed
reasoning back to the provider), and the boot timing comparison.
## Rollout
SDK-only; no server change required. A new SDK against a server that
does not serialize record headers degrades to the existing no-cursor
fallback. Old SDKs ignore the new snapshot field, and new SDKs fall back
to the records scan on snapshots written before it existed.
## Summary
HIPAA BAA is offered as a paid add-on on every paid plan. Each paid tier
on the in-app pricing card now has a "HIPAA BAA add-on" row with a
"Request a BAA" link that opens the existing contact dialog pre-filled
with a new `hipaa` inquiry type, prompting the user for their company
name and a brief description of the PHI workload.
The contact form's `feedbackTypes` are restructured to match the
marketing /contact form: every inquiry type carries a Plain label ID and
a "Contact form: ..." thread title, so threads land in Plain identically
whether they come from the dashboard or the marketing site. The
included-compute line on each tier also picks up the credits wording
from the marketing pricing page, and the Enterprise tier lifts its title
above the features row.
The compute suspend flow delays snapshots by `snapshotDelayMs` (~30s) so
short-lived waitpoints skip the snapshot entirely, with the intent that
a run continuing before the delay expires cancels the pending snapshot.
But the only `cancel()` call site was the `/continue` action, which
runners only invoke when restoring from an already-taken snapshot — so
pending snapshots were never cancelled (zero `snapshot.canceled` events
ever emitted in prod). When a run resumed and completed inside the
window, the stale snapshot fired ~30s later anyway, pausing the VM 6–13s
mid warm-start long-poll; the frozen guest couldn't fire its abort timer
or send a FIN, causing stalls and run-engine driven retries.
### Change
- Cancel the pending snapshot on `attempt.complete` — after the platform
accepts the completion, before the HTTP reply (so it can't reorder with
the runner's next `/suspend`).
- Cancel on `runDisconnected` (crash, exit, or run replaced on the
socket).
- Both cancels are guarded by a runnerId match (new
`TimerWheel.peek()`): a stale duplicate runner for a reassigned run must
not cancel the fresh runner's pending snapshot. A missing runnerId falls
through to an unconditional cancel (the pre-existing `/continue`
behavior is unchanged).
Waitpoint suspensions keep the runner socket connected and the attempt
incomplete, so neither hook touches a snapshot that is still wanted.
Known limitation (fail-safe direction): `socket.data.runnerId` is frozen
at the websocket handshake, so after a same-supervisor restore the
disconnect-path guard refuses the cancel. The `attempt.complete` path
uses the runner's current header id and is unaffected.
`ComputeWorkloadManager.create` swallows gateway errors currently, so a
cold start that fails placement (e.g. a netns slot with a busy tap, a
full node disk) silently abandons the dequeued run until the run
engine's `PENDING_EXECUTING` heartbeat timeout redrives it via stall
detection.
### Changes
- Retry `instances.create` with short backoff (default 3 attempts, 250ms
backoff), recording `createAttempts` in the wide event.
- **Only statuses where the create definitely did not commit are
retried**: 500 (agent/fcrun create failed) and 503 (no placement).
502/504 are excluded — the gateway emits those when it fails to reach
the node or read its response, which can happen *after* the agent
committed the create; the gateway only records the instance name on a
clean 201, so a same-name retry would miss the collision check and could
double-create the VM on another node. Network-level fetch failures are
retried (if the gateway processed the create, its name index is
populated and the retry 409s harmlessly). Timeouts are not retried.
- **Retry attempts after a 5xx use a deterministic `-rN` name suffix**:
a failed create can leave its name registered until async cleanup runs.
Attempt 1 keeps the unsuffixed name.
Fixes an intermittent `Test timed out in 30000ms` in the
`streamBatchItems` suite. Not a logic hang — the 30s budget covers
container setup, and each case boots its own per-test Redis container +
a full `RunEngine`, so under CI Docker contention a cold boot can cross
30s (which is why the failure moved between tests).
- New `containerTestWithIsolatedRedisNoClickhouse` fixture (Postgres
clone + per-test Redis, no ClickHouse) — this suite never uses
ClickHouse, but the old fixture's auto `resetClickhouse` forced a
ClickHouse boot + migration onto the cold-start test.
- Raised `testTimeout` 30s → 120s, matching the run-engine convention
for this footprint.
## Summary
A batch of reliability fixes for `chat.agent`:
- A user message sent while the agent is streaming is no longer
delivered twice (which could run a duplicate turn).
- Input appends carry an idempotency key (`X-Part-Id`) so a retried send
can't duplicate a message.
- `onTurnComplete` now fires on errored turns with the thrown error
attached, and the failed turn's user message is persisted so it isn't
lost on the next run.
- Stopping a generation clears the streaming state, so a page reload
doesn't replay the stopped turn.
- Custom agents and manual `chat.writeTurnComplete` callers trim the
output stream, sending a custom action no longer leaves a second stream
reader running, a long-lived `watch` subscription no longer grows its
dedupe set without bound, promoting a queued message to steering no
longer risks a double-send, and runs keep the full set of dashboard
tags.
The `X-Part-Id` header is accepted by current servers (they just don't
dedupe on it yet), so this is safe to ship ahead of the matching server
change.
## Summary
Reliability and authorization fixes for realtime chat sessions:
- Session-stream waitpoint delivery is scoped to the environment, so two
environments using the same session `externalId` can no longer complete
each other's waitpoints.
- The session snapshot-url routes now enforce per-session authorization,
and appending to a session's `out` channel requires secret-key auth, so
a session-scoped token can't read another session's snapshot or forge
assistant output.
- Appends that carry an `X-Part-Id` header are deduplicated on retry, so
a retried send can't duplicate a message.
- Session creation rejects expired sessions (instead of triggering a run
that can never receive input), `externalId` is immutable after creation,
and the sessions list endpoint returns friendly `run_*` ids to match the
single-session routes.
## Rollout
The waitpoint cache key gains an environment prefix. To keep waitpoints
registered by the previous deploy working across the boundary, the drain
reads both the new and the previous key for this release; the legacy
read can be removed a release later once no pre-deploy waitpoints
remain.
## Summary
Adds a second backend for the realtime runs feed (`useRealtimeRun`,
`subscribeToRunsWithTag`, `subscribeToBatch`), built to stay healthy
when a single busy environment has many subscribers watching many runs
at once. It is gated behind a feature flag with the existing backend as
the default, so nothing changes for users until it is enabled per
environment.
## Design
A run change is published once, as a small self-describing record, to a
single per-environment channel. Every feed is then a predicate over that
one stream rather than owning a channel:
- A per-instance router indexes the currently-held feeds by run, tag,
and batch. When a run changes it hydrates the affected rows once and
serializes them once, then fans the result to every matching feed. One
hot shared tag watched by many subscribers costs a single database query
and serialize, not one per subscriber.
- Feeds that don't match a change are never woken, wake delivery per
environment is coalesced on a leading edge (250ms default) so a burst of
changes costs one wake, and cold reads coalesce onto a single
short-TTL-cached resolve.
- An admission gate bounds how many cold ClickHouse resolves run
concurrently, so a mass reconnect across many distinct filters queues
instead of stampeding the database.
- Changes that land while a client is between long-polls are delivered
on its next poll instead of waiting for the periodic backstop: each
environment buffers its recent change records, subscriptions linger
briefly after the last feed closes, and a newly-armed poll replays
exactly the connection's gap.
- The per-connection replay cursors behind that are shared across
instances via Redis (a single timestamp each), so a poll landing on a
different instance behind the load balancer still reads the connection's
true gap instead of falling back to a cold resolve. Cursor reads have a
bounded deadline and degrade to the cold-read path on any Redis trouble.
- Tag subscriptions with multiple tags match runs carrying all of the
tags, mirroring the existing backend's filter semantics, and live
long-polls hold for about 20 seconds to match its cadence.
- The per-environment channel supports Redis Cluster sharded pub/sub, so
the wake path scales horizontally across shards by environment.
- The backend reports its health through OpenTelemetry metrics (delivery
lag, poll resolution paths, backstop outcomes, replay and cursor-store
activity), with a provisioned Grafana dashboard for local development.
Everything is behind the feature flag and tunable via env vars; the
existing backend remains the default.
## Summary
When `RUN_ENGINE_READ_REPLICA_SNAPSHOTS_SINCE_ENABLED` is on,
`RunEngine.getSnapshotsSince` reads from the read replica. During write
spikes the replica can briefly lag, so the snapshot id a runner just
learned from the writer isn't visible there yet: the lookup threw, the
worker route returned a 500, and the runner waited for its next poll —
turning sub-second snapshot notifications into poll-interval latency
exactly when things are busiest. This PR makes the flag safe to enable:
a replica miss of the since snapshot gets one jittered retry on the
replica (most lag windows are shorter than the ~50–200ms wait, so the
writer is never touched), then falls back to the primary, observed via a
new `run_engine.snapshots_since.replica_miss` counter with an `outcome`
attribute (`replica_retry` vs `primary`). Only genuine misses — absent
on the primary too — remain errors.
## Design
- `getExecutionSnapshotsSince` now throws a typed
`ExecutionSnapshotNotFoundError` so the engine can distinguish the
expected lag miss from real failures. The message string is unchanged
and the error never leaves the engine.
- The recovery path only engages when the flag is on, a distinct replica
client is configured, and no transaction client was passed. With the
flag off, the path is behaviorally identical to before.
- Retry delay bounds are configurable
(`RUN_ENGINE_SNAPSHOTS_SINCE_REPLICA_RETRY_MIN_MS`/`MAX_MS`, default
50/200; `MAX_MS=0` skips the replica retry and goes straight to the
primary).
- The warn log fires only when the primary serves the read (the writer
spill is the operationally interesting event); replica-retry recoveries
are counted but quiet. A permanently-missing snapshot id stays an
error-level failure with a `failedDuring` field, so lag metrics aren't
polluted by bogus ids.
- Stale-tail lag (replica has the since snapshot but not newer rows)
deliberately still returns the replica's view; the next poll catches up.
- The since-snapshot anchor lookup is now scoped to the polled run
(`where: { id, runId }`), so a snapshot id from a different run raises
not-found instead of silently anchoring a too-wide window of the run's
snapshots.
## Test plan
All vitest + testcontainers, no mocks. A new `schemaOnlyPrisma` fixture
(migrated-but-empty clone database) simulates a replica that hasn't
caught up, and a real in-memory OTel meter pins the counter semantics
per outcome.
- [x] Replica catches up during the jittered retry window → served by
the replica, `outcome=replica_retry` = 1, primary never consulted
- [x] Replica permanently missing the since snapshot → served by the
primary, `outcome=primary` = 1
- [x] Snapshot missing on both replica and primary → null, counter = 0
- [x] Replica has the since snapshot but lags by one → the replica's
view is served, no fallback (verified discriminating power: the test
fails if reads secretly hit the primary)
- [x] Flag off with a replica configured → primary serves the read
- [x] Transaction client provided → bypasses the replica entirely
- [x] Since snapshot belonging to a different run → null
- [x] Existing getSnapshotsSince + waitpoints suites green; run-engine,
testcontainers, and webapp typechecks pass
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
The supervisor's dequeue round-trip time (`POST
/engine/v1/worker-actions/dequeue`) was measured but only flowed into
wide events and OTel span attributes — there was no Prometheus series,
so latency percentiles and error rates weren't queryable. This adds
`queue_consumer_pool_dequeue_duration_seconds` (histogram, label
`outcome=success|empty|error`) to the existing consumer-pool metrics,
scraped automatically by the existing ServiceMonitors on
queue-raider/schedule-raider/supervisor.
- Records every dequeue call, including failed ones, which previously
emitted no timing at all
- The pool's shared `ConsumerPoolMetrics` instance is injected into each
consumer (mirrors the `BackpressureMetrics` → `BackpressureMonitor`
wiring)
- Buckets extend to 30s because `wrapZodFetch` retries internally (5
attempts, ≥7.5s backoff before a retryable error surfaces)
- Existing `dequeueResponseMs` wide-event/span behavior unchanged
The `dispatch-main-image` job was hard-gated to
`triggerdotdev/trigger.dev` on the `main` ref. This makes it
configurable via repository variables, all defaulting to the current
values:
- `MAIN_IMAGE_DISPATCH_REPO` — the repo allowed to dispatch (default
`triggerdotdev/trigger.dev`).
- `MAIN_IMAGE_DISPATCH_REF_PREFIX` — the ref-name prefix that
dispatches, matched with `startsWith(github.ref_name, …)` (default
`main`).
- `MAIN_IMAGE_DISPATCH_TARGET` — the `repository_dispatch` target
(default `triggerdotdev/cloud`).
The job is additionally gated on `github.event_name == 'push'`. This is
necessary, not just defensive: the gate now keys off `github.ref_name`
rather than the computed image tag, and `ref_name` is still `main` when
`release.yml` invokes this workflow via `workflow_call` during a release
— so without the event guard the job would fire during every release and
fail on the absent `CROSS_REPO_PAT`. A version-equality check can't
replace it because `build-*` tags strip the prefix to the version
output.
Behaviour note: the intended dispatch paths — push to `main`, and push
of a `<prefix>*` tag in a downstream repo — are `push` events and are
unchanged. The one case that no longer dispatches is a manual
`workflow_dispatch` run of `publish.yml` on `main` (it previously did,
via the old `version == 'main'` check). That path is indistinguishable
from a manual release by event name, so `push`-only is the clean
discriminator.
Dispatching still requires `CROSS_REPO_PAT`, so setting the variables
alone doesn't enable anything.
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
## Summary
Adds `GET /api/v1/projects/{projectRef}/environments` (personal access
token auth), which lists the base environments a user can access for a
project — their own dev environment plus the project's staging, preview,
and production environments.
## Details
- Built on the PAT route builder, so it inherits org-membership auth and
the per-resource ability check.
- `dev` is scoped to the token owner; archived environments are
excluded.
- Returns the branchable **parent** preview environment — preview branch
children are not included. A consumer targets the parent; branch-level
overrides are handled separately.
- Sorted to match the dashboard's environment switcher (dev → staging →
preview → prod), and never returns API keys.
Example response:
```json
[
{ "id": "...", "slug": "dev", "type": "DEVELOPMENT", "isBranchableEnvironment": false, "branchName": null, "paused": false },
{ "id": "...", "slug": "stg", "type": "STAGING", "isBranchableEnvironment": false, "branchName": null, "paused": false },
{ "id": "...", "slug": "preview", "type": "PREVIEW", "isBranchableEnvironment": true, "branchName": null, "paused": false },
{ "id": "...", "slug": "prod", "type": "PRODUCTION", "isBranchableEnvironment": false, "branchName": null, "paused": false }
]
```
## Summary
The dashboard's Agent view rendered `source-url` and `file` message
parts by putting their `url` straight into an `href`/`src`. Those URLs
come from streamed agent and tool data, so a tool that emitted something
like `javascript:alert(1)` produced a clickable XSS payload in the
dashboard.
## Fix
A `toSafeUrl` helper now gates every URL before it reaches an
`href`/`src`: it allows only `http:`/`https:`/`blob:` (and
`data:image/...` for inline images) and returns `null` for anything
else. Unsafe values render as plain text instead of a link or image, so
a hostile or malformed URL degrades gracefully rather than becoming
clickable. Safe URLs render exactly as before. Covered by a unit test
over the allow/deny list.
## Summary
During `trigger()` worker-queue resolution, `getWorkerQueue` wrapped any
error from `getDefaultWorkerGroupForProject` into a client-facing
`ServiceValidationError` (HTTP 422) carrying `error.message`. That
method runs `project.findFirst` on the **writer**; when the writer is
unreachable Prisma throws a connection error (P1001) whose message
includes the database host, and that raw message was returned to the API
client and surfaced in the run view via the SDK's `TriggerApiError`.
It also mis-classifies a transient outage: a 422 is not retried by the
SDK, so triggers failed permanently instead of riding out a brief writer
blip.
## Design
This is the only place on the trigger path that folds a *caught* error's
message into a client-facing error — every other DB failure on the path
propagates to the route's generic 500 handler (scrubbed, and retried by
the SDK). So the fix is local:
- Add `isInfrastructureError()` — true for Prisma connection-level
failures (the DB-unreachable family: P1001/P1002/P1008/P1017, plus the
init/panic/unknown client error classes), false for query/validation
errors (e.g. P2002).
- At the wrap site, rethrow infrastructure errors so they reach the
generic 500 handler (no raw message, and retryable). Genuine domain
failures (e.g. "Project not found.") still become a 422.
Only P1001 ("can't reach database server") has been observed in
practice; the rest of the connection family is included as same-class
forward-proofing.
## Test plan
- [x] Unit: `isInfrastructureError` classifies a P1001 (incl. the Prisma
6.x `PrismaClientKnownRequestError` shape) and init errors as
infrastructure; P2002 and a plain `Error` as not
- [x] `getWorkerQueue` rethrows a P1001 unchanged instead of wrapping it
in a `ServiceValidationError`; still wraps a domain failure as a
`ServiceValidationError` — RED on current code, GREEN after
- [ ] (optional) toxiproxy e2e: trigger with the writer cut → HTTP 500
generic body, no DB host in the response
---------
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
## Summary
Under high request load the webapp spends most of its CPU inside
react-router's `matchRoutes`, not in application code.
`@remix-run/router@1.23.2` (the React Router v6 / Remix 2 core)
re-flattens, re-ranks, and recompiles the entire route table on every
request, and with the webapp's ~436 routes that cost dominates once
request rates climb. There is no `NODE_ENV` gate, so production pays it
too.
This adds a pnpm patch that memoizes the parts that depend only on the
static route manifest: it caches the flattened/ranked branches per route
tree, hoists the loop-invariant `decodePath` out of the match loop, and
caches compiled path regexes.
## Benchmark
CPU profile over the same load (100 concurrent tag feeds, ~425 req/s),
`NODE_ENV=production`, before vs after the patch:
| Metric | Before | After |
| --- | --- | --- |
| Active CPU (self-time over the window) | 28.3s | 18.5s (-34%) |
| Route-matching self-time | 19.2s | 7.5s (-61%) |
| Event-loop lag p99 | 322ms | 113ms (-65%) |
| Idle headroom | 26% | 52% |
Application/realtime code was ~0% of CPU in both profiles; the
bottleneck was entirely generic per-request route matching.
## Why a patch instead of an upgrade
The inefficiency is acknowledged upstream
([remix-run/react-router#8653](https://github.com/remix-run/react-router/issues/8653)).
A contributor PR doing exactly this
([remix-run/react-router#14866](https://github.com/remix-run/react-router/pull/14866))
was closed in favor of a narrower fix
([remix-run/react-router#14967](https://github.com/remix-run/react-router/pull/14967),
branch caching only, shipped in React Router v7), with the maintainer
suggesting patch-package as the interim until the Remix 3 route-pattern
rewrite (see
[remix-run/remix#4786](https://github.com/remix-run/remix/discussions/4786)).
We are on the v6-era core and cannot pick up even the partial fix
without a framework migration, so this patch is the sanctioned stopgap,
and it also includes the compiled-regex cache the merged PR left out.
[`patches/README.md`](https://github.com/triggerdotdev/trigger.dev/blob/perf/react-router-route-matching/patches/README.md)
documents the full rationale, the safety argument (deterministic,
internal-only, bounded caches), and when to remove the patch.
## Summary
A new guide for connecting a database to your tasks: where to create the
client, how to size the connection pool against your provider's limit,
when to reach for a pooler, and how to release connections at waits so
you don't hit "too many connections" or crash on resume.
It covers node-postgres, Prisma, Drizzle, and MongoDB, with researched
direct and pooled connection limits for the common Postgres providers
(Supabase, Neon, RDS, PlanetScale) and MongoDB Atlas. The page lives
under Documentation, Troubleshooting, and is linked from the chat agent
docs (overview, lifecycle hooks, chat.local, and the database
persistence pattern).
## Summary
On every `main` build, once the webapp image is pushed to the registry,
the publish workflow emits a cross-repo `repository_dispatch` event
(`main-image-published`) carrying a digest-pinned image ref. Other
repositories in the org can subscribe to that event and build or deploy
from the exact artifact, instead of chasing the moving `main` tag.
## Design
`publish-webapp.yml` now exposes the pushed multi-arch index digest as a
workflow output. `publish.yml` adds a `dispatch-main-image` job (after
`publish-webapp`) that builds `<image_repo>@<digest>` and sends the
dispatch via the same pinned `peter-evans/repository-dispatch` action
already used elsewhere in this repo, authed with `CROSS_REPO_PAT`.
It fires only when the published tag is `main`, so semver releases and
other tag builds are excluded, and only from the canonical repo so forks
never dispatch. The payload is JSON-escaped with `jq`.
## Problem
Two backward-pagination bugs in `ClickHouseRunsRepository.listRunIds`,
both pre-existing (they predate the composite-cursor work in #3852 and
were spotted during/after it):
**1. Wrong slice (straddled pages).** `listRunRows` fetches `page.size +
1` rows to detect `hasMore`. That extra row is the one *farthest from
the cursor* in both directions (forward orders DESC; backward orders
ASC), so it's always the *trailing* element. Forward correctly used
`rows.slice(0, size)`, but backward+`hasMore` used `rows.slice(1, size +
1)` — dropping the row *closest* to the cursor and keeping the has-more
sentinel. The page straddled two logical pages (one run from the correct
previous page + one from the page before it), so paging "newer" across a
boundary **repeated and skipped** runs.
**2. Stranded forward cursor on a partial backward page.** In the
backward `!hasMore` branch, `nextCursor` was `reversedRows.at(page.size
- 1)`. On a partial page (fewer than `page.size` rows — reachable via
`runs.list` by passing a forward page's cursor as `page[before]`), that
index overshoots → `undefined` → `nextCursor` becomes `null`, leaving no
way to page forward again.
## Fix
- **Slice:** both directions now slice `rows.slice(0, size)` (the
sentinel is the trailing element either way).
- **Partial-page cursor:** the backward `!hasMore` branch takes the
oldest row on the page, `rows.at(0)`, for `nextCursor` — equivalent to
the old expression for full pages, correct for partial ones.
Forward pagination, the cursor *values* for full pages, and the `hasMore
=== true` paths were already correct and are unchanged.
## Tests
`runsRepositoryCursor.test.ts` gains two cases (both fail on `main`,
pass here):
- **multi-page backward walk:** forward across all pages, then backward
from the last page — each backward page must *exactly* reproduce the
corresponding forward page (no straddling: `main` returns `{b,c}`
instead of `{c,d}`), and the full traversal covers every run once.
- **partial backward page:** backward onto a partial first page must
still expose a working forward cursor (and paging forward from it
reaches the rest) — `main` returns a `null` nextCursor.
The three existing cursor tests (forward completeness, backward
round-trip, legacy cursor) still pass.
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
## What
`dev` and `deploy` now fail with a clear error when two tasks are
defined with the same id — including across task types (e.g. a scheduled
task and a regular task sharing an id).
## Why
Tasks are registered into the resource catalog keyed by id, so a second
definition with the same id silently overwrote the first. One of the
tasks would just vanish from the worker with no warning — easy to miss,
hard to debug. (Any earlier duplicate-id check ran against the
post-registration task list, which is already de-duplicated, so it never
actually fired.)
## How
- **Detect at registration** (`@trigger.dev/core`):
`StandardResourceCatalog` records a collision when a task id is
registered more than once, capturing the files involved — the only point
where both definitions are visible before the id-keyed map collapses
them. Exposed via `listTaskIdCollisions()`.
- **Fail indexing** (`trigger.dev` CLI): both index workers report
collisions via a new `TASKS_FAILED_TO_INDEX` message;
`indexWorkerManifest` rejects with a new `DuplicateTaskIdsError`. `dev`
renders a dedicated error (offending ids + files + docs link); `deploy`
fails with the same message. Runtime worker boot is unaffected — it
never reads the collisions.
- **Server-side backstop** (webapp): background-worker registration also
rejects duplicate ids with a clear `ServiceValidationError`, so
duplicates are caught even from an older CLI.
## Testing
- Unit tests for collision collection in the catalog and for the
error-message formatting (standard, same-file, and 3+-definition cases).
- Verified end to end against a local webapp: a project with a regular
task and a scheduled task sharing an id now fails `dev` with the
dedicated error; a project with distinct ids still starts normally.
## Changeset
Patch for `@trigger.dev/core` and `trigger.dev`.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
## Summary
- `internal-packages/rbac/src/index.ts` — in `LazyController.load()`'s
catch block, throw an Error when `process.env.REQUIRE_PLUGINS === "1"`
instead of silently falling back. The throw is captured into the lazy
controller's init promise, so it surfaces on the first method call.
- `apps/webapp/app/routes/healthcheck.tsx` — `await
rbac.isUsingPlugin()` after the DB ping. With `REQUIRE_PLUGINS=1` and a
failed plugin load, the throw surfaces here and the healthcheck returns
500 → readiness probe fails → rollout is rolled back. Noop for
self-hosters.
- `.server-changes/require-plugins-fail-fast.md` — server-changes entry.
- `internal-packages/rbac/src/require-plugins.test.ts` — 4 unit tests
covering loader branching: unset → fallback, `=1` → throw,
`forceFallback: true` wins, only exactly `"1"` enforces.
- `internal-packages/testcontainers/src/webapp.ts` — adds
`requirePlugins?: boolean` to `StartWebappOptions`. Implies
`forceRbacFallback: false`.
- `apps/webapp/test/healthcheck-require-plugins.e2e.test.ts` — e2e
closes the loop: spawns a real webapp, hits `/healthcheck` via HTTP,
asserts 500 with `REQUIRE_PLUGINS=1` and 200 without.
## Motivation
Today the RBAC plugin loader catches any plugin-load failure (missing
module, broken transitive dep, init throw) and silently returns the
default fallback implementation. This is the correct behaviour for
self-hosters who don't ship the plugin — but it's dangerous in
deployments where the plugin is expected to load: an
accidentally-missing or broken plugin would silently disable
enforcement.
`REQUIRE_PLUGINS=1` makes the loader fail loudly in those deployments.
The variable name is intentionally plural and generic — future plugin
contracts (audit logs, SSO) can read the same flag without renaming.
Closes
[TRI-9852](https://linear.app/triggerdotdev/issue/TRI-9852/require-plugins1-fail-fast-for-required-plugin-loads).
## Test plan
- [x] `pnpm run test --filter @trigger.dev/rbac` — 38/38 tests pass,
including the 4 new loader tests
- [x] `pnpm run typecheck --filter webapp` — passes
- [x] `pnpm run typecheck --filter @trigger.dev/rbac --filter
@internal/testcontainers` — passes
- [x] e2e test added (`healthcheck-require-plugins.e2e.test.ts`) — CI
runs it via `e2e-webapp.yml`. Couldn't run locally (no Docker daemon
up); CI has Docker provisioned.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
## Summary
Runs routed onto a dedicated scheduled worker queue were showing under a
phantom region in the dashboard, run details, and the API, and slipped
through region filters. They now resolve to their real region
everywhere.
## Fix
A worker queue can carry a `:scheduled` suffix that routes
scheduled-lineage runs onto their own list. That suffix is an internal
routing detail, but it was leaking anywhere the worker queue is read as
a region. A `baseWorkerQueue` helper strips any `:<class>` suffix back
to the base region (region names never contain a colon, so it's
everything before the first colon) and is applied at every region read
site: the runs list, run detail, the public API, and replay's region
override. The runs-replication writer also stores the base region in
ClickHouse so the region filter matches.
## Summary
`trigger init` now sets up your AI coding assistant as part of project
setup. Instead of the old either/or "MCP or CLI" prompt, it offers the
MCP server and agent skills together, then asks whether to scaffold with
the CLI or let your assistant do it.
A new `getting-started` agent skill backs that hand-off: it teaches the
assistant the bootstrap recipe (install the SDK, write
`trigger.config.ts`, scaffold a first task, wire tsconfig/gitignore, run
`trigger dev`) and is explicit about the two steps that genuinely need a
human (`trigger login` and copying the DEV secret key from the
dashboard). It ships in the CLI alongside the existing skills,
version-matched to your SDK.
Prompt-once gating is shared, so opting in or out during `init` means
`trigger dev` won't ask about skills again.
## Summary
The container publish workflows hardcoded `ghcr.io/triggerdotdev/...` as
the image destination. As a result, a fork that builds on push-to-`main`
(or on the worker publish tags) would attempt to push to — and attest —
the upstream packages rather than its own, which fails on permissions
and is surprising besides.
This makes the image destination configurable via a single
`IMAGE_REGISTRY` repository variable, while leaving the upstream
defaults byte-identical:
- **Single source of truth** (`publish.yml`): a `resolve-registry` job
resolves the target registry namespace once — `IMAGE_REGISTRY`
repository variable, defaulting to `ghcr.io/${{ github.repository_owner
}}` — and passes it down to every publish job as an `image_registry`
input. So a fork publishes to its own namespace automatically with no
configuration.
- **Webapp** (`publish-webapp.yml`): the image now lives at
`<registry>/<repo-name>` (e.g. `ghcr.io/<owner>/trigger.dev`). The
provenance attestation and the downstream Trivy scan follow the same
computed repo via the `image_repo` workflow output.
- **Workers** (`publish-worker.yml`, `publish-worker-v4.yml`): build
under `<registry>/<worker-name>`. They keep a `vars.IMAGE_REGISTRY ||
ghcr.io/<owner>` fallback so they still resolve correctly on their
direct `infra-*` / `re2-*` push triggers (which bypass the parent
workflow).
A single `IMAGE_REGISTRY` namespace variable now governs both webapp and
workers (the earlier `WEBAPP_IMAGE_REPO` full-path override is dropped,
removing the full-path/namespace asymmetry). When `IMAGE_REGISTRY` is
unset, every resolved image name is exactly what it is today, so there
is no change for this repo.
## Test plan
- [x] `actionlint` passes on all four workflows
- [ ] On merge, confirm the webapp publish still pushes
`ghcr.io/triggerdotdev/trigger.dev:main` + the commit-SHA tag (defaults
unchanged)
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
## Summary
The `mollifier.decisions` metric only carried an `outcome` label, so for
an org that has the mollifier enabled there was no way to see how often
its triggers pass through the gate instead of being diverted — making it
hard to tell why the trip isn't firing for an opted-in org.
This adds two bounded labels: `enrolled` (`"true"`/`"false"`, the
per-org flag) and `org` (the org id, attached **only** when `enrolled`
is true). For an enrolled org you can now compare directly:
`mollifier.decisions{outcome="pass_through", enrolled="true",
org="<id>"}` vs `{outcome="mollify", ...}`.
## Design
`recordDecision` now takes an options object (`{ reason?, enrolled,
orgId? }`). The `org` label is restricted to the enrolled cohort to keep
cardinality bounded — the guard lives in a pure `decisionLabels` helper,
so a non-enrolled org id can never be attached even if one is passed.
The enrolled set is small and capped operationally.
The per-org flag is resolved once at the top of `evaluateGate`
(in-memory, no DB round-trip on the trigger hot path) so every decision
— including the debounce / one-time-use-token / triggerAndWait bypasses
— is labelled consistently.
## Test plan
- [x] `mollifierGate.test.ts` cascade asserts `enrolled`/`org` on every
gate branch
- [x] `mollifierDecisionLabels.test.ts` (new) proves `org` is dropped
for non-enrolled even when an id is passed (cardinality guard)
- [x] `vitest run mollifierGate mollifierDecisionLabels` — 34/34 pass
- [x] `pnpm run typecheck --filter webapp` clean
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Add an optional network_labels field to the internal compute client's
create and restore request schemas and forward per-VM endpoint labels on
both paths, so a restored VM keeps the same labels as a freshly-booted
one. Mirrors the label the Kubernetes workload manager already sets on
the run pod.
---------
Co-authored-by: nicktrn <55853254+nicktrn@users.noreply.github.com>
## Summary
`trigger skills` installs Trigger.dev agent skills into your coding
agent so it knows how to write Trigger.dev code: tasks, schedules,
realtime, and `chat.agent` AI agents. The skills are `SKILL.md` files
(the open Agent Skills format) bundled with the CLI and copied into each
tool's native skills directory (Claude Code, Cursor, GitHub Copilot, and
Codex / `AGENTS.md`), version-matched to the CLI you run. `trigger dev`
offers to install them on first run, and a one-line always-on pointer is
written into your `CLAUDE.md` / Cursor rules / etc. so the agent always
knows which skills are available and loads the right one on demand.
This replaces the old `install-rules` command, which stays as an alias.
Four skills ship to start: `authoring-tasks`, `realtime-and-frontend`,
`authoring-chat-agent`, and `chat-agent-advanced`.
## Problem
`ClickHouseRunsRepository.listRunIds` / `listRuns` order results by the
composite key `(created_at, run_id)`, but the cursor predicate cut on
`run_id` **alone**:
```ts
.where("run_id < {runId: String}", { runId: cursor })
.orderBy("created_at DESC, run_id DESC")
```
This is only sound when `run_id` lexicographic order matches
`created_at` order. `run_id`s are cuids — only coarsely time-sortable —
so when a burst of runs is created within a sub-second window, the two
orders can diverge. When they do, the next-page predicate (`run_id <
cursor`, where `cursor` is the *last* page element = the smallest
`created_at`, not necessarily the smallest `run_id`):
- **re-includes** rows already returned on a previous page (duplicates),
and
- **skips** rows it should have returned (silent data loss).
For bulk **replay** this caused runs to be replayed more than once
(replay has no idempotency guard). For the dashboard and the `runs.list`
API it could silently repeat or skip runs at page boundaries.
## Fix
Make the cursor predicate match the composite ordering:
- Cursors now encode the full `(created_at, run_id)` key as an **opaque
URL-safe base64 token**
(`base64url({"c":<createdAtMs>,"r":"<runId>"})`), and the query cuts on
the matching tuple — `(created_at, run_id) < (…)` forward / `> (…)`
backward.
- The `ORDER BY` is unchanged, so the query stays aligned with the
table's primary key — no performance regression (the tuple range
predicate is actually more index-friendly than `run_id <` alone).
- Cursors are **server-issued opaque tokens** (the SDK only echoes
`pagination.next` / `pagination.previous` back), so this needs **no
client/SDK update**. Legacy cursors were the bare internal `run_id`;
they're detected by decode failure (a cuid isn't a valid base64-wrapped
JSON payload) and fall back to the old `run_id`-only predicate, so
in-flight cursors keep working and drain naturally. New cursors also no
longer expose a bare internal run id.
- `listRunIds` is now the single cursor-aware list primitive: it returns
`{ runIds, pagination: { nextCursor, previousCursor } }`, and `listRuns`
builds on it (one place constructs cursors). Bulk actions consume the
same method and advance by `pagination.nextCursor`, finishing when it's
`null`.
- `getTaskRunsQueryBuilder` now also selects
`toUnixTimestamp64Milli(created_at) AS created_at_ms`, using a dedicated
`TaskRunListQueryResult` schema. The shared `TaskRunV2QueryResult` stays
`run_id`-only so the run-engine pending-version lookup
(`getPendingVersionIdsQueryBuilder`, which selects only `run_id`)
doesn't fail validation on a column it doesn't query.
## Tests
New `runsRepositoryCursor.test.ts` (testcontainer-backed, real
Postgres→ClickHouse replication):
- **forward** pagination returns every run exactly once when `run_id`
order is the reverse of `created_at` order (reproduces the
duplicate/skip bug — fails on `main`; this
walk-until-`nextCursor`-null-and-assert-complete is exactly the bulk
action's iteration),
- **backward** pagination round-trips to the previous page across a
boundary,
- **legacy** bare-`run_id` cursor still uses the old predicate
(backwards compatibility).
The existing `runsRepository` suites (part1–4) still pass; `part4`'s
`count new runs with listRunIds` test was updated for the new `{ runIds,
pagination }` return shape, and the `clickhouse` `taskRuns`
query-builder snapshots were regenerated for the added `created_at_ms`
column.
## Notes
- Separate, pre-existing issue (out of scope, not introduced here):
`listRuns`' backward display-slicing (`rows.slice(1, size+1)` when
`hasMore`) has an off-by-one that can return a straddled page. Tracked
separately.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Two small hygiene tweaks to **dev-only** images:
- `docker/Dockerfile.postgres`: add `--no-install-recommends` to the
partman install (leaner image, skips unneeded recommended packages).
- `internal-packages/clickhouse/Dockerfile`: run the migration helper as
a non-root user.
Both are local-dev images (the `pnpm run docker` stack) - no impact on
the published webapp image, prod, or self-hosting.
Speeds up and de-flakes the unit-test suite: testcontainers booted once
per vitest worker (per-test isolation kept only where a test runs
background redis work that outlives it), a duration-weighted shard
sequencer so each shard does roughly equal work, the slowest suites
split, two genuine flakes fixed (`streamBatchItems` shared-redis leak;
run-engine waits that relied on fixed sleeps), and transient DockerHub
pulls retried.
**Timings (CI, per-shard wall):** worst unit-test shard ~771s → ~294s;
packages/webapp shards ~250-270s, most internal ~190-240s. All 25 shards
green.
A shard breaks down as ~70s fixed setup (install / image-pull /
generate) + ~70s cold `^build` + the actual container tests. So the
remaining cost is mostly the tests themselves plus that fixed setup.
**Next (separate, timings):**
- **typecheck (~6m24s)** — the slowest check overall; bound by
full-graph `tsc`, not the TS version (a TS6 branch is still ~6m17s). The
real lever is **tsgo** (the Go compiler).
- Possible later: turbo CI caching could trim the ~70s cold build on
*warm* runs, but it's conditional (cold runs rebuild anyway) and doesn't
touch setup or test time — secondary.
`cli-v3` e2e and `sdk-compat` are path-gated (don't run on test-infra
changes) and already comfortably fast.
## Summary
Adds a trace export to the run page. From the new **Export trace** menu
you can
copy a run's full trace to the clipboard as Markdown (for pasting into
an AI
assistant) or download it as a flat Log, a Markdown table, or JSON
Lines.
Internal engine-debug events are filtered out by default, and errors are
surfaced inline with their message.
## Design
The export streams events from the store to the gzipped response one at
a time
and never materialises the span tree, so a trace of any size exports
with
bounded memory and without stalling the server. Output is flat and
chronological: each line carries its own `spanId ← parentSpanId`, so the
hierarchy is reconstructable without nesting. Formats share a single
streaming
pipeline and are pluggable via `?format=log|jsonl|markdown`, so adding a
format
is an isolated change.
## Screenshots
**Export menu**
<img width="370" height="252" alt="trace-export-menu"
src="https://github.com/user-attachments/assets/3d10304a-8c49-4606-b15d-2859b137419f"
/>
**In context**
<img width="2400" height="1802" alt="trace-export-run-page"
src="https://github.com/user-attachments/assets/46c80b30-303b-47c6-9ace-a2fb06f6cb61"
/>
A unit-test shard recently failed on a timing race rather than a real
regression - a run-engine waitpoint test sleeps 1250ms waiting on a
1000ms timeout that's processed by a ~1000ms worker poll, so on a
CPU-starved shard the margin evaporates and the whole matrix goes red.
Because `fail-fast` defaults on, that one flake cancels the sibling
shards, and the only recovery is re-running the entire matrix "just to
be sure" - which is itself slow.
This is the low-risk first pass at that pain:
- `fail-fast: false` on the webapp and internal shard matrices, so one
flaky shard no longer cancels its siblings. "Re-run failed jobs" now
re-runs just the failed shard instead of the whole matrix.
- CI-scoped `retry: process.env.CI ? 2 : 0` on the timing-sensitive
packages (`run-engine`, `redis-worker`, `schedule-engine`). Flakes
self-heal in CI; local runs stay at `retry: 0` so they still surface in
dev. A stopgap until the timing tests are made deterministic.
- `fetch-depth: 1` on the unit-test checkouts - they don't use git
history, so the full clone was wasted setup time across ~20 jobs.
- Reconcile the pre-pull image tags with what testcontainers actually
pulls (`redis:7-alpine` -> `redis:7.2`, `ryuk:0.11.0` -> `ryuk:0.14.0`)
and add `minio/minio:latest` to the webapp pre-pull. Otherwise those
images pull unauthenticated at test time and risk Docker Hub rate-limit
flakes (worst on fork PRs, where the authenticated pre-pull is skipped
entirely).
Deeper follow-ups - bigger runners, turbo remote cache, runtime-weighted
sharding, and the real root-cause fix (container reuse / template-DB
isolation + deterministic timing tests) - are tracked under TRI-10484.
Hardens the webapp Docker image and adds a CVE scan of each published
image.
- Base image `bullseye-slim` → `bookworm-slim` (Debian 12), pinned by
digest. Adds `apt-get upgrade` + `--no-install-recommends` + apt-cache
cleanup across the build stages so OS packages are patched at build
time.
- Moves the `react-email` CLI to `devDependencies` in
`internal-packages/emails` — only the `email dev` preview script uses
it; the runtime render path is `@react-email/render` +
`@react-email/components`. This also drops the bundled `esbuild` binary
from the production image.
- Bumps `goose` v3.26.0 → v3.27.1 and its Go builder image 1.23 → 1.26.
- Adds a reusable Trivy image-scan workflow wired into `publish.yml`, so
every published image (main builds and releases) is scanned for
OS-package CVEs right after it's pushed to GHCR. Report-only (writes to
the run summary), runs alongside the worker publishes so it never blocks
a deploy.
Verified locally: the image builds clean on the new base, and
`@react-email/render` carries no `esbuild` dependency so email rendering
is unaffected.
## Summary
The SDK and core packages run a second, forward-compat typecheck pass
(`tsc --noEmit -p tsconfig.ai-v7.json`) that remaps the `"ai"` import to
the ESM-only AI SDK 7 canary, so we catch source that only compiles
against one major. That pass inherited `composite: true` from the base
tsconfig, which makes `tsc` write a `.tsbuildinfo` even under
`--noEmit`.
Incremental buildinfo caches each file's resolved module format (CJS vs
ESM) and module resolution. When that state goes stale or is replayed,
the v7 pass can report spurious `TS1479` ("CommonJS module ... cannot
`require` an ECMAScript module") errors on the `"ai"` import even though
the source is fine in a clean checkout. Because this pass shares the
typecheck job that gates the Docker image publish, a spurious failure
there blocks publishing.
## Fix
Set `composite: false` and `incremental: false` on both
`tsconfig.ai-v7.json` files. The pass is `--noEmit` only, so it never
needed incremental state. Now each run is a clean, full check that
writes no buildinfo and can't replay stale resolution.
Verified: both `@trigger.dev/sdk` and `@trigger.dev/core` typecheck
green, and neither writes an ai-v7 `.tsbuildinfo` anymore.
## Summary
Opening or closing the Bulk action inspector should not affect the Runs
list, but it was still triggering refresh-related UI behavior. This PR
fixes that and smooths out a few related inspector interactions.
## Changelog
Stop reloading the runs list (and flashing its loading state) when
opening or closing the
Bulk action inspector. Filtering, pagination, and explicit refresh are
unaffected.
## Summary
Documents AI SDK 7 support in the AI Chat docs. Pairs with the SDK
change in #3833.
- The reference compatibility matrix now lists the v7 peer range and
adds an `@ai-sdk/otel` row.
- A new "AI SDK 7 telemetry" section covers the `@ai-sdk/otel` install,
the automatic registration, and the `TRIGGER_AI_SDK_OTEL_AUTOREGISTER`
opt-out.
- The quick start surfaces the supported `ai` versions (v5/v6/v7) up
front, near where you install.
## Summary
1 new feature, 8 improvements, 1 bug fix.
## Highlights
- Add optional `shouldPauseScaling` to the supervisor consumer pool
scaling options to freeze scale-up while it returns true (scale-down
stays allowed).
([#3836](https://github.com/triggerdotdev/trigger.dev/pull/3836))
## Improvements
- The MCP server no longer tells the AI agent to wait for a run to
complete after every `trigger_task` call. Waiting is now opt-in: the
agent only waits when you ask it to (for example "trigger and then wait
for it to finish"). This avoids burning tokens polling runs you didn't
need to block on and keeps responses clearer.
([#3838](https://github.com/triggerdotdev/trigger.dev/pull/3838))
- Update the bundled OpenTelemetry packages to their latest releases
(`@opentelemetry/sdk-node` 0.218.0, `@opentelemetry/core` 2.7.1,
`@opentelemetry/host-metrics` 0.38.3).
([#3810](https://github.com/triggerdotdev/trigger.dev/pull/3810))
- `envvars.upload` now accepts an optional `isSecret` flag, letting you
create the imported variables as secret (redacted) environment
variables. When omitted, variables default to non-secret.
([#3809](https://github.com/triggerdotdev/trigger.dev/pull/3809))
- Offload large trigger payloads to object storage before sending the
trigger API request. The SDK uploads packets at or above the existing
128KB limit and sends an `application/store` pointer instead of
embedding large JSON in the request body. `TriggerTaskRequestBody` now
validates that `application/store` payloads are non-empty storage paths.
([#3785](https://github.com/triggerdotdev/trigger.dev/pull/3785))
- Make mollifier buffer and drainer internals configurable.
`MollifierBuffer` now accepts `ackGraceTtlSeconds`,
`maxRetriesPerRequest`, `reconnectStepMs`, and `reconnectMaxMs` options,
and `MollifierDrainer` accepts `maxBackoffMs` and `backoffFloorMs`. All
default to their previous hardcoded values, so existing behaviour is
unchanged.
([#3822](https://github.com/triggerdotdev/trigger.dev/pull/3822))
- `MollifierDrainer` accepts a `drainBatchSize` option (default 1) that
controls how many entries are popped per env per tick — in-flight
handlers remain capped by the global `concurrency`. `MollifierBuffer`
also gains `getDrainingCount()` / `listStaleDraining()`, backed by a new
`mollifier:draining` ZSET maintained atomically with
pop/ack/fail/requeue (observability-only).
([#3797](https://github.com/triggerdotdev/trigger.dev/pull/3797))
- Adds AI SDK 7 support. The `ai` peer range now includes v7, and the
`chat.agent` / chat surfaces work against v7's ESM-only build. On v7,
install `@ai-sdk/otel` alongside `ai` and the SDK registers it for you
so `experimental_telemetry` spans keep flowing into your run traces (v7
stopped emitting them from `ai` core). v5 and v6 keep working unchanged.
([#3833](https://github.com/triggerdotdev/trigger.dev/pull/3833))
- `useTriggerChatTransport` now recovers when restored session state
points at a session that no longer exists in the current environment
([#3816](https://github.com/triggerdotdev/trigger.dev/pull/3816))
## Bug fixes
- Fix `@trigger.dev/core` build: cast the underlying log record exporter
when calling `forceFlush` so it typechecks against the updated
OpenTelemetry `LogRecordExporter` type (which no longer declares
`forceFlush`).
([#3829](https://github.com/triggerdotdev/trigger.dev/pull/3829))
<details>
<summary>Raw changeset output</summary>
⚠️⚠️⚠️⚠️⚠️⚠️
`main` is currently in **pre mode** so this branch has prereleases
rather than normal releases. If you want to exit prereleases, run
`changeset pre exit` on `main`.
⚠️⚠️⚠️⚠️⚠️⚠️
# Releases
## @trigger.dev/build@4.5.0-rc.5
### Patch Changes
- Updated dependencies:
- `@trigger.dev/core@4.5.0-rc.5`
## trigger.dev@4.5.0-rc.5
### Patch Changes
- The MCP server no longer tells the AI agent to wait for a run to
complete after every `trigger_task` call. Waiting is now opt-in: the
agent only waits when you ask it to (for example "trigger and then wait
for it to finish"). This avoids burning tokens polling runs you didn't
need to block on and keeps responses clearer.
([#3838](https://github.com/triggerdotdev/trigger.dev/pull/3838))
- Update the bundled OpenTelemetry packages to their latest releases
(`@opentelemetry/sdk-node` 0.218.0, `@opentelemetry/core` 2.7.1,
`@opentelemetry/host-metrics` 0.38.3).
([#3810](https://github.com/triggerdotdev/trigger.dev/pull/3810))
- Updated dependencies:
- `@trigger.dev/core@4.5.0-rc.5`
- `@trigger.dev/build@4.5.0-rc.5`
- `@trigger.dev/schema-to-json@4.5.0-rc.5`
## @trigger.dev/core@4.5.0-rc.5
### Patch Changes
- Add optional `shouldPauseScaling` to the supervisor consumer pool
scaling options to freeze scale-up while it returns true (scale-down
stays allowed).
([#3836](https://github.com/triggerdotdev/trigger.dev/pull/3836))
- Fix `@trigger.dev/core` build: cast the underlying log record exporter
when calling `forceFlush` so it typechecks against the updated
OpenTelemetry `LogRecordExporter` type (which no longer declares
`forceFlush`).
([#3829](https://github.com/triggerdotdev/trigger.dev/pull/3829))
- `envvars.upload` now accepts an optional `isSecret` flag, letting you
create the imported variables as secret (redacted) environment
variables. When omitted, variables default to non-secret.
([#3809](https://github.com/triggerdotdev/trigger.dev/pull/3809))
```ts
await envvars.upload("proj_1234", "prod", {
variables: { STRIPE_SECRET_KEY: "sk_live_..." },
isSecret: true,
});
```
- Offload large trigger payloads to object storage before sending the
trigger API request. The SDK uploads packets at or above the existing
128KB limit and sends an `application/store` pointer instead of
embedding large JSON in the request body. `TriggerTaskRequestBody` now
validates that `application/store` payloads are non-empty storage paths.
([#3785](https://github.com/triggerdotdev/trigger.dev/pull/3785))
Payload uploads use the same resolved `ApiClient` as the trigger call
(including `requestOptions.clientConfig`), not only the global
`apiClientManager.client` — so custom `baseURL`, access token, and
preview branch apply to both presign and trigger.
- Update the bundled OpenTelemetry packages to their latest releases
(`@opentelemetry/sdk-node` 0.218.0, `@opentelemetry/core` 2.7.1,
`@opentelemetry/host-metrics` 0.38.3).
([#3810](https://github.com/triggerdotdev/trigger.dev/pull/3810))
## @trigger.dev/plugins@4.5.0-rc.5
### Patch Changes
- Updated dependencies:
- `@trigger.dev/core@4.5.0-rc.5`
## @trigger.dev/python@4.5.0-rc.5
### Patch Changes
- Updated dependencies:
- `@trigger.dev/sdk@4.5.0-rc.5`
- `@trigger.dev/core@4.5.0-rc.5`
- `@trigger.dev/build@4.5.0-rc.5`
## @trigger.dev/react-hooks@4.5.0-rc.5
### Patch Changes
- Updated dependencies:
- `@trigger.dev/core@4.5.0-rc.5`
## @trigger.dev/redis-worker@4.5.0-rc.5
### Patch Changes
- Make mollifier buffer and drainer internals configurable.
`MollifierBuffer` now accepts `ackGraceTtlSeconds`,
`maxRetriesPerRequest`, `reconnectStepMs`, and `reconnectMaxMs` options,
and `MollifierDrainer` accepts `maxBackoffMs` and `backoffFloorMs`. All
default to their previous hardcoded values, so existing behaviour is
unchanged.
([#3822](https://github.com/triggerdotdev/trigger.dev/pull/3822))
- `MollifierDrainer` accepts a `drainBatchSize` option (default 1) that
controls how many entries are popped per env per tick — in-flight
handlers remain capped by the global `concurrency`. `MollifierBuffer`
also gains `getDrainingCount()` / `listStaleDraining()`, backed by a new
`mollifier:draining` ZSET maintained atomically with
pop/ack/fail/requeue (observability-only).
([#3797](https://github.com/triggerdotdev/trigger.dev/pull/3797))
- Updated dependencies:
- `@trigger.dev/core@4.5.0-rc.5`
## @trigger.dev/rsc@4.5.0-rc.5
### Patch Changes
- Updated dependencies:
- `@trigger.dev/core@4.5.0-rc.5`
## @trigger.dev/schema-to-json@4.5.0-rc.5
### Patch Changes
- Updated dependencies:
- `@trigger.dev/core@4.5.0-rc.5`
## @trigger.dev/sdk@4.5.0-rc.5
### Patch Changes
- Adds AI SDK 7 support. The `ai` peer range now includes v7, and the
`chat.agent` / chat surfaces work against v7's ESM-only build. On v7,
install `@ai-sdk/otel` alongside `ai` and the SDK registers it for you
so `experimental_telemetry` spans keep flowing into your run traces (v7
stopped emitting them from `ai` core). v5 and v6 keep working unchanged.
([#3833](https://github.com/triggerdotdev/trigger.dev/pull/3833))
- `useTriggerChatTransport` now recovers when restored session state
points at a session that no longer exists in the current environment
([#3816](https://github.com/triggerdotdev/trigger.dev/pull/3816))
- Offload large trigger payloads to object storage before sending the
trigger API request. The SDK uploads packets at or above the existing
128KB limit and sends an `application/store` pointer instead of
embedding large JSON in the request body. `TriggerTaskRequestBody` now
validates that `application/store` payloads are non-empty storage paths.
([#3785](https://github.com/triggerdotdev/trigger.dev/pull/3785))
Payload uploads use the same resolved `ApiClient` as the trigger call
(including `requestOptions.clientConfig`), not only the global
`apiClientManager.client` — so custom `baseURL`, access token, and
preview branch apply to both presign and trigger.
- Update the bundled OpenTelemetry packages to their latest releases
(`@opentelemetry/sdk-node` 0.218.0, `@opentelemetry/core` 2.7.1,
`@opentelemetry/host-metrics` 0.38.3).
([#3810](https://github.com/triggerdotdev/trigger.dev/pull/3810))
- Updated dependencies:
- `@trigger.dev/core@4.5.0-rc.5`
</details>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
## Summary
The Prisma CLI was missing from production builds of the webapp image,
so anything that shells out to `prisma` at startup failed. The container
entrypoint and the standalone migration step both run `prisma migrate
deploy` / `prisma migrate status`, and those broke with `Command
"prisma" not found`.
## Fix
`prisma` was a `devDependency` of `@trigger.dev/database`. It had only
been landing in the pruned `--prod` install as a side effect of pnpm
auto-installing it as a peer of `@prisma/client`. A recent dependency
change shifted peer resolution so prisma stopped being materialized into
the production tree, and the CLI disappeared from the image.
Moving `prisma` into `dependencies` of `@trigger.dev/database` makes the
CLI an explicit part of production installs. It lands in the webapp
image only: the separately deployed supervisor, coordinator, and
provider images don't reach the database package in their production
trees (`core` only `devDepends` on it, so it isn't transitive), so
they're unaffected.
Verified against a locally built production image: `pnpm --filter
@trigger.dev/database exec prisma --version` now resolves the CLI and
the schema engine instead of failing.
The supervisor can now pause dequeuing - and freeze consumer-pool
scale-up - when a backpressure signal says the cluster can't place more
work, then ramp dequeuing back up gradually once it clears. The signal
is a verdict published to a Redis key by a cluster-side component; the
supervisor reads it on a short refresh and gates `preDequeue` on it.
Off by default (`TRIGGER_DEQUEUE_BACKPRESSURE_ENABLED`). Everything
fails open: a missing, stale, or unreadable verdict never pins the
brake, and the hot-path read is a synchronous cached lookup with no I/O.
The scale-up freeze leaves scale-down untouched, and on release the
resume is ramped so a deep queue isn't hammered all at once.
Dry-run is on by default (`TRIGGER_DEQUEUE_BACKPRESSURE_DRY_RUN`): even
once enabled it only logs what it would have done, and surfaces the
computed state through metrics, until explicitly set to act. Prometheus:
`supervisor_backpressure_engaged`, `_dry_run`,
`_skipped_dequeues_total`.
Refs TRI-5354
## Summary
Scheduled runs and their descendants can now be routed to a dedicated
per-region worker queue, processed by a separate worker fleet, so a
burst of scheduled crons no longer competes with standard and agent runs
for the same queue and inflates their startup latency. It is off by
default and enabled per organization via a feature flag (with a global
default), so nothing changes until it is turned on.
## Design
At trigger time, any run whose lineage originates from a schedule
(`rootTriggerSource === "schedule"`, which already propagates from a
scheduled run down to all of its children) gets its worker queue
suffixed with `:scheduled`. The worker queue name is an opaque string
persisted on the run and used verbatim by enqueue and dequeue, so this
needs no Lua, message-envelope, or concurrency changes. Concurrency
stays keyed by environment and queue, not by worker queue.
On the consumer side, the dequeue endpoint gains an optional
`queueClass` selector. A supervisor sends `queueClass: "scheduled"` and
the server derives the actual queue from the worker's own group, so a
token can only ever reach its own region's queues. A fleet picks its
class with the `TRIGGER_WORKER_QUEUE_CLASS` env var (`default` or
`scheduled`), so a dedicated scheduled fleet can run alongside the
standard one.
Verified end to end against a local managed-worker setup: scheduled runs
route to the dedicated queue, are drained only by the scheduled fleet,
and standard runs are left untouched.
## Summary
The Trigger.dev MCP server told the AI agent to wait for the run to
complete after every `trigger_task` call. The agent followed that
instruction even when the user only wanted to fire-and-forget, which
burned tokens polling runs nobody needed to block on and made responses
less clear.
Waiting is now opt-in. After triggering, the response tells the agent
the run is executing in the background and to only wait if the user
asked it to (for example "trigger and then wait for it to finish"). The
`trigger_task` tool description is updated to match. The
`wait_for_run_to_complete` tool itself is unchanged, so explicit waits
still work.
## Summary
This PR adds packet path validation before key construction and
presigning. Invalid paths are rejected before reaching either
object-store client implementation, ensuring consistent behavior
regardless of the underlying storage configuration.
## Summary
Adds support for Vercel AI SDK 7. The `ai` peer range now includes v7,
and the `chat.agent` / chat surfaces work against v7's ESM-only build.
v5 and v6 keep working unchanged, so this is additive.
## Telemetry on v7
On v7, model-call spans moved out of `ai` core into the separate
`@ai-sdk/otel` adapter, so `experimental_telemetry` alone produces
nothing until an integration is registered. Install `@ai-sdk/otel`
alongside `ai@7` and the SDK registers it once per worker at chat agent
boot, so spans keep flowing into run traces with no extra setup.
If you (or a library you import) already register `@ai-sdk/otel`, the
SDK detects the existing integration and skips its own registration, so
you won't get duplicate spans. Set `TRIGGER_AI_SDK_OTEL_AUTOREGISTER=0`
to disable auto-registration entirely.
## Notes
`ai@7` is ESM-only, which tripped TS1479 in the SDK's CommonJS build.
Runtime value imports from `ai` are isolated behind a paired ESM/CJS
shim so both module formats resolve the right form; type-only imports
stay as direct `import type` at their use sites.
## Summary
Three fixes to the AI agent surface in the dashboard, all surfaced while
verifying AI SDK 7 support (the SDK side is #3833):
- **AI SDK 7 telemetry rendering.** The generation-span inspector and
run metrics now read both the v6 (`ai.*`) and v7 (`gen_ai.*`) telemetry
attribute shapes. On v7 the Messages, Provider, and Tools views showed
empty/unknown before; now they render correctly.
- **HITL approvals in the conversation view.** The agent conversation
view renders human-in-the-loop tool approvals and denials (awaiting
approval, approved, denied with reason) instead of leaving the tool part
blank. This gap predated v7.
- **Chat snapshot store mismatch.** Chat session snapshots now resolve
through a single storage-key helper shared by the SDK write and the
dashboard read. Previously the write applied the default object-store
protocol to the key while the read fell back to a different store, so
the dashboard 404'd on the snapshot and showed only a partial
conversation.
## Summary
On a busy webapp the trace/log/metric ingestion path emits several
`info` logs per insert batch, which makes up the bulk of the service's
log output. This moves that per-batch chatter to `debug` and adds an
opt-in to drop successful HTTP access logs, cutting log volume with no
loss of error signal.
## Details
The per-batch ClickHouse insert logs, the flush scheduler's concurrency
adjustments, and the event-loop utilization sample (already exported as
a metric, so the log line was redundant) now log at `debug`. Error and
warning logs are untouched.
New `HTTP_ACCESS_LOG_DISABLED=1` env var: when set, the HTTP access
logger skips successful (2xx) requests while still logging non-2xx
responses. Defaults off, so existing deployments are unchanged.
## Summary
This PR improves performance across the Environment Variables page.
## Changes
### Targeted value loading
- load only the non-secret (environmentId, key) pairs required by the
page. Secret values continue to be redacted in the UI.
### SSR windowing + virtualization
- SSR-render only the first 50 rows
- hydrate those rows
- virtualize the remaining dataset client-side
- search is now URL-driven during SSR, ensuring deep links such as
`?search=DATABASE_URL`
### Lightweight 'Create' flow
- 'Create' page no longer loads the full Environment Variables dataset.
## Results
Large projects no longer render thousands of rows during SSR.
Example (~11k rendered rows):
Metric | Before | After
-- | -- | --
Document size | ~150 MB | ~5 MB
SSR rows | ~11k | 50
Browser DOM rows | Thousands | ~26–38
## Summary
The mollifier had ~21 behavioural constants baked in as hardcoded values
— the buffer's ack-grace TTL and Redis retry/reconnect tuning, the
drainer's poll interval and backoff envelope, the pre-gate idempotency
claim TTL/wait/poll, the buffered-run mutate-with-fallback wait loop,
the metadata CAS retry budget and backoff, the stale-sweep scan bounds,
and the draining-gauge interval. None could be adjusted without a code
change, which makes tuning the system under production load impossible.
This exposes all of them as `TRIGGER_MOLLIFIER_*` environment variables,
each defaulting to its previous hardcoded value. Behaviour is identical
unless an operator sets a var, so it's a safe no-op deploy.
## Design
The package-level classes (`MollifierBuffer`, `MollifierDrainer` in
`@trigger.dev/redis-worker`) gain optional constructor options
defaulting to the old constants — backward compatible, hence a patch
changeset. The webapp factories and worker bootstraps read the env and
pass them through. The route- and concern-level pure helpers
(mutate-with-fallback, metadata mutation, idempotency claim, stale-sweep
state) keep their existing `?? DEFAULT` option fallbacks and are fed env
values at their call sites, so they stay unit-testable without importing
`env.server`.
## Test plan
- [x] `@trigger.dev/redis-worker` builds
- [x] webapp typecheck passes
- [x] mollifier buffer + drainer testcontainer suites pass (modulo a
couple of pre-existing flaky timing tests)
- [x] Reviewer: confirm the `TRIGGER_MOLLIFIER_*` env var names match
ops conventions
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>
Adds backward-compatible support for large trigger payloads by reusing
the existing object-storage packet flow.
Large payloads are uploaded to object storage before the trigger request
is sent. The trigger API receives a small application/store pointer
payload instead of embedding large JSON bodies in the request.
Small payload behavior is unchanged.
## Summary
Adds a nullable `pricing_unit` column to the LLM model registry's
`llm_models` table, recording how each model is billed ("tokens",
"characters", "images", "minutes", "requests", "free", "not_findable").
It lets pricing-coverage reporting exclude models that aren't priced
per-token (image/video/audio models currently drag the "% priced" number
down even though they can never carry a per-token price), and lays the
groundwork for non-token pricing.
The default model catalog is entirely per-token, so `seed` and
`syncLlmCatalog` set `pricing_unit="tokens"` on those rows. The admin
LLM model form (create + edit) and the admin API get a pricing-unit
selector so admin-curated models can set it; existing rows can stay
unset.
Auto-discovered models get their unit from the model-registry pipeline,
which lands separately.
---------
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
## Summary
When a chat's restored session state points at a session that no longer
exists in the current environment — for example a `sessions` entry that
was persisted against a different trigger environment —
`useTriggerChatTransport` assumed the session was live and never created
a real one. The next message then failed with a 404 and the chat
couldn't send.
## Fix
`callWithAuthRetry` now treats a 404 from a session-PAT-authed call as
"this session doesn't exist here". After the existing 401/403 token
refresh, a 404 recreates the session via `startSession`, drops the stale
`lastEventId` resume cursor (it pointed at another environment's
stream), and retries the send once. When `startSession` isn't configured
the transport throws a clear message instead of a bare 404.
Routine maintenance pass on a few transitive `pnpm.overrides`.
- `fast-uri` / `fast-xml-builder`: add overrides pinning to current
releases (`3.1.2` / `1.1.7`).
- `protobufjs` / `qs`: bump existing override pins that had fallen a
patch behind (`7.5.6` / `6.15.2`).
Overrides-only - no first-party code changes; lockfile regenerated to
match. Verified the affected transitives resolve to the pinned releases
via `pnpm why -r`.
## ✅ Checklist
- [x] I have followed every step in the [contributing
guide](https://github.com/triggerdotdev/trigger.dev/blob/main/CONTRIBUTING.md)
- [x] The PR title follows the convention.
- [x] I ran and tested the code works
---
## Testing
Upgraded the email packages in `internal-packages/emails`:
| Package | Before | After |
| --- | --- | --- |
| `@react-email/components` | `0.0.16` | `1.0.12` |
| `@react-email/render` | `^0.0.12` | `^2.0.8` |
| `react-email` (CLI) | `^2.1.1` | `^6.5.0` |
| `react-dom` | _(missing)_ | `^18.2.0` (now a required peer of render)
|
**Breaking change handled:** `render()` is now async (`Promise<string>`)
in `@react-email/render` v1+. Added `await` in the `aws-ses`, `smtp` and
`null` transports. `EmailClient` and the webapp callers were already
async and needed no changes.
Verification:
- `pnpm run typecheck --filter emails` ✅
- `pnpm run typecheck --filter webapp` (consumer of the `emails`
package) ✅
- **Before/after render comparison**: rendered all 11 templates
(magic-link, invite, welcome, alert-attempt/run/error-group,
deployment-failure/success, mfa-enabled/disabled, bulk-action-complete)
to HTML with both the old and new packages and compared them visually +
via HTML diff. Output is visually identical. The only HTML changes come
from upstream improvements: `<Body>` now wraps content in a
`<table>`/`<td>` for better email-client compatibility, an
`x-apple-disable-message-reformatting` meta tag was added, and CSS
shorthand (e.g. `margin`) is now also emitted as longhand. No visual
regressions; the `CodeBlock`/dracula theme, buttons, and row/column
layouts all render correctly.
No changeset or `.server-changes/` entry is added: `emails` is a private
internal package (not under `packages/`), and there is no user-facing
behavior change.
---
## Changelog
Upgrade `react-email` and `@react-email/{components,render}` in
`internal-packages/emails` to their latest versions and adapt the mail
transports to the now-async `render()` API.
---
## Screenshots
Rendered email templates before vs after the upgrade (visually
identical):
**Before** (`@react-email/components@0.0.16`, `render@0.0.12`)
## Summary
A fresh clone running `trigger dev` against the local webapp shows the
run in the dashboard but an empty trace view.
`EVENT_REPOSITORY_DEFAULT_STORE` was unset in `.env.example`, so span
ingestion fell back to the `postgres` store while the rest of the local
stack (run replication, `CLICKHOUSE_URL`) is ClickHouse-backed.
Defaulting it to `clickhouse_v2` makes dev run traces show up out of the
box.
The reference/example projects (`references/`) now live in their own
repo, https://github.com/triggerdotdev/references, so their heavy,
frequently-changing dependencies are no longer part of this repo's
lockfile and tooling. This removes them here and repoints everything
that referenced them.
- Deletes `references/`; updates the pnpm workspace + lockfile.
- Clears the references-only CI rules and `.vscode` configs.
- Repoints the docs (contributor/agent + one public page) to the new
repo.
- `seed.mts` keeps the local-dev projects (hello-world, d3-chat,
realtime-streams).
Brings the OpenTelemetry packages up to the latest coherent release
across the webapp and the published packages (`@trigger.dev/core`, the
CLI, `@trigger.dev/sdk`) plus
`internal-packages/{tracing,testcontainers}`:
- `@opentelemetry/sdk-node` 0.218.0
- `@opentelemetry/core` 2.7.1
- `@opentelemetry/host-metrics` 0.38.3
We were already on the otel 2.x line, so this is a same-major minor move
- the versions are pinned to `@opentelemetry/sdk-node@0.218.0`'s own
declared dependency set so the experimental (0.2xx) and stable (2.x)
packages stay coherent (mixing them is the usual otel breakage).
**One code change:** otel 0.215 made `forceFlush()` a required method on
`LogRecordExporter`, so `ExternalLogRecordExporterWrapper` (core's
tracing SDK) gains a `forceFlush()` that delegates to the underlying
exporter.
**Notable upgrades along the way:** OTLP exporters can take a custom
HTTP agent (connection pooling/keepAlive on the export path), HTTP
request headers are captured at span creation, and core hot-path perf
improvements in 2.6.1/2.7. `host-metrics` 0.37→0.38 is a clean upgrade.
Patch changeset added for the three published packages. References
projects are intentionally untouched.
Verified: `@trigger.dev/core` / CLI / `@trigger.dev/sdk` build, webapp +
`@internal/tracing` typecheck - all green.
Adds wide-event observability for the supervisor: one flat-keyed JSON
line per dequeue iteration, workload-server route, and run socket
lifecycle event. Events carry `trace_id` sourced from the inbound W3C
traceparent plus `meta.run_id` and related identifiers, so they join
across services by run.
The outbound warm-start POST also forwards the inbound traceparent so
the upstream receiver continues the same trace instead of minting a new
one.
Off by default behind `TRIGGER_WIDE_EVENTS_ENABLED`. With the flag off,
no events are emitted, no ALS state is allocated, and the outbound
warm-start request is unchanged — every call site was audited to confirm
the off path is byte-identical to current behavior.
Dequeue-path phase timings recorded under `phase.<name>.duration_ms`:
`restore`, `warm_start`, `workload_create`. A `path_taken` extra
distinguishes `restore` / `warm_start` / `cold_create` /
`skipped_no_image`.
Refs TRI-9480.
## Summary
The Runs list now updates live without requiring a page refresh. Status
changes and other run fields are updated in place while runs are
executing.
When new runs matching the current filters are created, a "New runs
created" refresh button appears above the list.
Root runs now show a live child-run status breakdown directly in the
status tooltip.
### List live update
- Visible runs update in place while they are still running.
- A "New runs created" refresh button appears when new matching runs are
detected.
- Polling stops when all visible runs have finished and a refresh button
is already shown.
- Polling pauses when the browser tab is not visible.
- Runs list status updates and new-run detection share a single
runs/live polling path.
### Child-status tooltip
- Root run tooltips now display a breakdown of child run statuses.
- Child statuses are loaded when the tooltip opens (after a 400ms hover
delay).
- The tooltip stays up to date while child runs are still changing
state.
- Handles cases where child runs continue running after their parent run
has completed, or have not yet been created.
### Supporting changes
- Added hidden-tab awareness to polling.
- Added safeguards around polling inputs (`runIds` deduping and limits).
## Test plan
- [x] pnpm run typecheck --filter webapp passes
- [x] cd apps/webapp && pnpm run test
./test/presenters/mapRunToLiveFields.test.ts --run passes
- [x] cd apps/webapp && pnpm run test
./test/runsRepository.part2.test.ts --run -t "hasNewRuns" passes
### Manual smoke:
- [x] Active runs update without a page refresh.
- [x] A new matching run shows the refresh banner and the banner actions
work as expected.
- [x] Root run tooltips show live child-status updates and stop polling
once child runs settle.
---------
Co-authored-by: Ekaterina Bulatova <kathiekiwi@Ekaterinas-MacBook-Pro.local>
## Problem
On environments where runs carry a Postgres-backed `taskEventStore`
value (`taskEvent` or `taskEventPartitioned`), OTLP ingest endpoints
(`POST /otel/v1/traces` and `/otel/v1/logs`) were returning HTTP 500.
**Root cause:** The org-scoped ClickHouse factory introduced in a recent
PR routes all OTLP spans through `getEventRepositoryForOrganizationSync`
→ `buildEventRepository`. That function only handles `"clickhouse"` and
`"clickhouse_v2"` store values and throws `Unknown ClickHouse event
repository store: <value>` for anything else. The throw occurred inside
the grouping loop of `#exportEvents`, unwinding the entire method and
returning 500 for the whole batch.
The OpenTelemetry collector's `otlphttp` exporter treats HTTP 500 as
non-retryable and drops the batch — causing real span loss.
**Fix:** Guard the `getEventRepositoryForOrganizationSync` call in
`#exportEvents` so it is only invoked for `clickhouse` / `clickhouse_v2`
store values. All other values are routed directly to the Postgres
`eventRepository`, matching the guard pattern already present in
`resolveEventRepositoryForStore` and `getEventRepositoryForStore` in
`eventRepository/index.server.ts`.
The ClickHouse factory call is also wrapped in a try/catch that falls
back to Postgres so any unexpected store value in a future OTLP batch
degrades gracefully instead of failing the whole request.
## Changes
- `apps/webapp/app/v3/otlpExporter.server.ts` — add Postgres routing
guard and try/catch fallback in `#exportEvents`
## Testing
The `eventRepository/index.server.ts` module already has the same guard
pattern thoroughly covered. The fix brings `#exportEvents` into
alignment with that existing, tested pattern. Manual verification:
confirm OTLP batches containing Postgres-store spans return 200 and
route to the correct repository.
## Summary
The environment variables import API now accepts an optional `isSecret`
flag, so imported variables can be created as secret (redacted)
environment variables instead of plaintext. When the flag is omitted,
variables default to non-secret, preserving existing behavior for CLI
deploys and dashboard imports.
This is useful for tools that push secrets into Trigger.dev (for
example, syncing from a secrets manager) and want them stored as secrets
rather than plain environment variables.
It's available through `envvars.import` in the SDK and the `POST
/api/v1/projects/{projectRef}/envvars/{slug}/import` endpoint, and is
honored for both regular and preview-branch environments.
```ts
await envvars.import("proj_1234", "prod", {
variables: { STRIPE_SECRET_KEY: "sk_live_..." },
isSecret: true,
});
```
## What
Adds [pkg.pr.new](https://pkg.pr.new) continuous preview releases. Every
push to a branch builds the public `@trigger.dev/*` packages and
publishes installable preview builds keyed by commit SHA — **without
touching the npm registry**. pkg.pr.new drops install instructions on
the associated PR:
```
npm i https://pkg.pr.new/@trigger.dev/sdk@<sha>
```
This lets reviewers and users try a branch (SDK, CLI, core, etc.) before
anything is released, separate from the changesets release, the manual
`--snapshot` prerelease, and the chat-prerelease flow.
## How
`.github/workflows/preview-packages.yml` (push trigger) → install →
generate Prisma → **stamp preview version** → build → `pkg-pr-new
publish`.
### The version stamp (the important part)
pkg.pr.new serves previews by SHA but does **not** rewrite the
package.json `version` field. If a preview shipped as `4.5.0-rc.4`, a
consumer who installed it would pin `4.5.0-rc.4` to the preview tarball
in their lockfile/cache — and a later `npm i
@trigger.dev/sdk@4.5.0-rc.4` from npm could resolve to the stale
preview. This is a known, by-design gap in the tool
(stackblitz-labs/pkg.pr.new#250, #390).
`scripts/stamp-preview-version.mjs` runs **before the build** and
rewrites every public package to a unique `0.0.0-preview-<sha>`. The
`0.0.0-` prefix can never satisfy a real semver range, so the collision
is structurally impossible (same convention React/Next canaries use).
Running before the build also means `scripts/updateVersion.ts` bakes the
preview version into the runtime `VERSION` constant, so previews are
self-identifying (`trigger --version`, the `x-trigger-cli-version`
header, the MCP server version) instead of all reporting the RC version.
Sibling `workspace:` specifiers are relaxed to `workspace:*` so `pnpm
pack` resolves them against the rewritten versions — `packages/python`
pins peerDependencies as `workspace:^4.5.0-rc.4`, which would otherwise
be unsatisfiable once the version changes. Non-public deps
(`@trigger.dev/database`, `@internal/*`) are left untouched. All
mutations happen on the ephemeral CI checkout; nothing is committed.
## GitHub App
The pkg.pr.new GitHub App is **already installed** on
`triggerdotdev/trigger.dev` (has been for a while), so no setup is
needed. Confirmed live — this branch's pushes published all 10 public
packages, e.g.
```
pnpm add https://pkg.pr.new/@trigger.dev/sdk@e4dfc59
```
## Fork limitation
pkg.pr.new authenticates with a GitHub Actions OIDC token, which GitHub
does not issue to `pull_request` workflows from forks. The `push`
trigger therefore covers branches pushed to this repo (core team), not
external fork PRs. Fork coverage would need a `workflow_run` two-stage
setup; left out for now.
## Notes
- Pinned `pkg-pr-new@0.0.75` (no Node engine constraint; Node 20 CI is
fine).
- pkg.pr.new
[#525](https://github.com/stackblitz-labs/pkg.pr.new/pull/525) adds a
built-in `--previewVersion` flag (still open). If it lands we can drop
the version-rewrite half of the script, but we'd keep a pre-build stamp
anyway so `updateVersion.ts` picks up the preview version (the flag
rewrites at pack time, too late for the baked `VERSION`).
## Summary
Adds `drainBatchSize` to `MollifierDrainer` (default `1` — preserves
existing behaviour) and wires `TRIGGER_MOLLIFIER_DRAIN_BATCH_SIZE`
through the webapp (default `50`). Each tick the drainer now pops up to
`drainBatchSize` from each chosen env, then dispatches every popped
entry through the shared `concurrency`-bounded `pLimit`. Per-org/per-env
fairness is unchanged — only the in-env pop count grows.
Pre-existing behaviour was one pop per env per tick. For a single-env
burst that single-flighted the drain at the per-tick floor of `pop +
engine.trigger ≈ 50–60 ms`. With buffer entries piling up under a
real-world tenant burst that's tens of minutes of tail latency to fully
materialise — even though PG itself could comfortably sustain the
writes.
## Why this matters — heavy-tail illustration
Scenario: 100 customers in one window — 94 fire 20 triggers each, 5 fire
100, 1 fires 1000. Gate at `THRESHOLD=10/s`, `HOLD_MS=500`. First 10 of
each burst hit PG directly; the rest buffer.
| Customers | Triggers each | PG direct | Buffered each | Total buffered
|
|---|---|---|---|---|
| 94 small | 20 | 10 | 10 | 940 |
| 5 medium | 100 | 10 | 90 | 450 |
| 1 heavy | 1000 | 10 | 990 | 990 |
**With `DRAIN_BATCH_SIZE=50`, `DRAIN_CONCURRENCY=50`, ~50 ms
`engine.trigger`:**
| Tick | Pops | Dispatch waves | Wall-clock |
|---|---|---|---|
| 1 | 94×10 + 5×50 + 1×50 = 1 240 | 25 × 50 ms | ~1 300 ms (94 smalls
done) |
| 2 | 5×40 + 1×50 = 250 | 5 × 50 ms | ~300 ms (5 mediums done) |
| 3–20 | heavy alone, 50/tick | 1 × 50 ms | ~100 ms each |
| Customer class | Buffered fully drained |
|---|---|
| 94 small | **~1.3 s** |
| 5 medium | **~1.6 s** |
| 1 heavy | **~3.4 s** |
**Without batching (one pop per env per tick — current behaviour):**
| Customer class | Buffered fully drained |
|---|---|
| 94 small | ~500 ms |
| 5 medium | ~4.5 s |
| 1 heavy | **~49 s** |
So the heavy single-tenant tail drops from ~49 s to ~3.4 s (~14× faster)
without changing PG load characteristics. Smalls go up slightly in this
scenario (500 ms → 1.3 s) because all 100 envs share one tick's dispatch
queue — that's the trade we accept for the heavy tail; the worst-case
small wait is still inside one tick. PG load is identical either way (50
concurrent inserts at a time, capped by `DRAIN_CONCURRENCY`).
## What changed
**`packages/redis-worker`**
- New `drainBatchSize` option (default 1 — full backward compat).
- `runOnce()` refactored to pop per-env batches in parallel, then
dispatch all popped entries through the existing global `pLimit`.
Mid-batch pop failure aborts only that env's batch and counts as one
failure (same semantic as the old per-env path).
- Removed the now-unused `processOneFromEnv` helper.
**`apps/webapp`**
- `TRIGGER_MOLLIFIER_DRAIN_BATCH_SIZE` env var (default 50, matching
`DRAIN_CONCURRENCY`).
- Wired into `mollifierDrainer.server.ts`.
**Test cloud config** (separate cloud PR):
`TRIGGER_MOLLIFIER_DRAIN_BATCH_SIZE="50"` on the worker service.
Production rollout deferred until we've watched it on test cloud.
## Test plan
- [x] All 25 stub-based drainer tests pass (18 pre-existing + 7 new). 7
new tests under `MollifierDrainer.drainBatchSize`:
- pops up to `drainBatchSize` across ticks
- global `concurrency` cap still holds when batch > concurrency
- mid-batch pop failure isolation
- multi-env batch fan-out in one tick
- **hierarchical org fairness preserved at `drainBatchSize > 1`**
(load-bearing — guards against future regressions to
per-env-instead-of-per-org rotation)
- mixed success/failure accounting in a batched tick
- bounded pops on empty queue (no Lua spam past `drainBatchSize`)
- [x] All pre-existing tests still pass unchanged at default
`drainBatchSize=1` → backward-compat locked.
- [x] `pnpm run build --filter @trigger.dev/redis-worker` clean.
- [x] `pnpm run typecheck --filter webapp` clean.
- [x] `redisTest` block (real Redis via testcontainers) — couldn't run
locally on this branch due to testcontainers runtime discovery; will
validate in CI.
- [ ] Test-cloud smoke after cloud PR lands: fire `burst 50` against a
flagged env and confirm the 50th entry's drain time drops from ~2.5 s to
<200 ms.
## Notes
- Per-tick memory bound: `maxOrgsPerTick × drainBatchSize` entries can
sit in the JS pLimit queue between pop and dispatch. At defaults that's
`500 × 50 = 25 000` × ~5 KB snapshot ≈ ~125 MB worst case per worker —
well within headroom.
- The pre-batch model's strict per-env throughput cap of `1/tick` is
documented as the fairness baseline elsewhere. Org-level fairness is
what callers actually rely on; this change does not weaken that.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Vouches GitHub user
[`ConProgramming`](https://github.com/ConProgramming) (Conner) as an
outside contributor by adding them to `.github/VOUCHED.td`, so their PRs
aren't auto-closed by the vouch check.
Done as a direct edit rather than via the issue flow because they have
no open Vouch Request issue to comment `vouch` on.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
## Summary
Documents the new `tools` option on `chat.agent` (companion to #3790).
Adds a dedicated [Tools](/ai-chat/tools) guide: the three places tools
show up (config, `toStreamTextOptions`, `streamText`), why declaring
them on the config matters for `toModelOutput` across turns, static vs
per-turn tools, the typed `run()` payload,
`InferChatUIMessageFromTools`, the relationship to skills, and the
manual `convertToModelMessages` path for `customAgent` loops.
Threads the option through the rest of the guide: the reference tables,
a happy-path section on the backend page, the types page, and the HITL /
skills / tool-result-auditing patterns. Corrects the sub-agents guide,
where the `toModelOutput` compression was implied to work across turns
but silently degraded from turn 2 without config tools.
Also unstacks the three callouts that were piled under the
`chat.agent()` header on the backend page, and adds a changelog entry.
## Summary
1 new feature, 5 improvements.
## Highlights
- Mollifier `mutateSnapshot` now enforces a tag cap: an `append_tags`
patch carrying `maxTags` returns `"limit_exceeded"` (writing nothing)
when the deduped tag count would exceed the limit, so a buffered run
can't accumulate more tags via the tags API than the trigger validator
allows at creation.
([#3756](https://github.com/triggerdotdev/trigger.dev/pull/3756))
## Improvements
- Mollifier buffer extensions: idempotency dedup, an atomic
`mutateSnapshot` API, metadata CAS, claim primitives, and a
`MollifierSnapshot` type. The buffer's Redis client now reconnects with
jittered backoff so a fleet of clients doesn't stampede Redis in
lockstep after a blip.
([#3752](https://github.com/triggerdotdev/trigger.dev/pull/3752))
- Add `onTerminalFailure` callback to `MollifierDrainerOptions` so the
customer's run lands a SYSTEM_FAILURE PG row even when the drainer
exhausts `maxAttempts` on a retryable PG error. Previously,
retryable-error exhaustion called `buffer.fail()` directly, which
atomically marks FAILED + DELs the entry hash with no PG write — silent
data loss when PG was unreachable across the full retry budget. The
callback fires before `buffer.fail()` on any terminal path (`cause:
"non-retryable"` or `"max-attempts-exhausted"`); throwing a retryable
error from the callback causes the drainer to requeue rather than fail.
([#3754](https://github.com/triggerdotdev/trigger.dev/pull/3754))
- Bump `@s2-dev/streamstore` to `0.22.10` to fix a
`TASK_RUN_UNCAUGHT_EXCEPTION` ("Invalid state: Unable to enqueue") when
a `chat.agent` turn is aborted mid-stream.
([#3792](https://github.com/triggerdotdev/trigger.dev/pull/3792))
- Coerce numeric `concurrencyKey` values to string at the API boundary
across `tasks.trigger`, `tasks.batchTrigger`, and the Phase-2 streaming
batch endpoint.
([#3789](https://github.com/triggerdotdev/trigger.dev/pull/3789))
- Add a `tools` option to `chat.agent`. Declaring your tools here
threads them into the SDK's internal `convertToModelMessages`, so each
tool's `toModelOutput` is re-applied when prior-turn history is
re-converted.
([#3790](https://github.com/triggerdotdev/trigger.dev/pull/3790))
<details>
<summary>Raw changeset output</summary>
⚠️⚠️⚠️⚠️⚠️⚠️
`main` is currently in **pre mode** so this branch has prereleases
rather than normal releases. If you want to exit prereleases, run
`changeset pre exit` on `main`.
⚠️⚠️⚠️⚠️⚠️⚠️
# Releases
## @trigger.dev/redis-worker@4.5.0-rc.4
### Minor Changes
- Mollifier buffer extensions: idempotency dedup, an atomic
`mutateSnapshot` API, metadata CAS, claim primitives, and a
`MollifierSnapshot` type. The buffer's Redis client now reconnects with
jittered backoff so a fleet of clients doesn't stampede Redis in
lockstep after a blip.
([#3752](https://github.com/triggerdotdev/trigger.dev/pull/3752))
- Add `onTerminalFailure` callback to `MollifierDrainerOptions` so the
customer's run lands a SYSTEM_FAILURE PG row even when the drainer
exhausts `maxAttempts` on a retryable PG error. Previously,
retryable-error exhaustion called `buffer.fail()` directly, which
atomically marks FAILED + DELs the entry hash with no PG write — silent
data loss when PG was unreachable across the full retry budget. The
callback fires before `buffer.fail()` on any terminal path (`cause:
"non-retryable"` or `"max-attempts-exhausted"`); throwing a retryable
error from the callback causes the drainer to requeue rather than fail.
([#3754](https://github.com/triggerdotdev/trigger.dev/pull/3754))
### Patch Changes
- Pipeline the per-entry `HGETALL` fetches in
`MollifierBuffer.listEntriesForEnv`. The previous serial implementation
issued one Redis round-trip per runId returned by `LRANGE`, which
dominated stale-sweep wall-time at any meaningful backlog (at the
sweep's default maxCount=1000, this is ~1000 RTTs per env per pass).
Behaviour is unchanged — entries are still skipped when the entry hash
has been torn down by a concurrent drainer ack/fail between the LRANGE
and the HGETALL.
([#3752](https://github.com/triggerdotdev/trigger.dev/pull/3752))
- Mollifier `mutateSnapshot` now enforces a tag cap: an `append_tags`
patch carrying `maxTags` returns `"limit_exceeded"` (writing nothing)
when the deduped tag count would exceed the limit, so a buffered run
can't accumulate more tags via the tags API than the trigger validator
allows at creation.
([#3756](https://github.com/triggerdotdev/trigger.dev/pull/3756))
- Updated dependencies:
- `@trigger.dev/core@4.5.0-rc.4`
## @trigger.dev/build@4.5.0-rc.4
### Patch Changes
- Updated dependencies:
- `@trigger.dev/core@4.5.0-rc.4`
## trigger.dev@4.5.0-rc.4
### Patch Changes
- Bump `@s2-dev/streamstore` to `0.22.10` to fix a
`TASK_RUN_UNCAUGHT_EXCEPTION` ("Invalid state: Unable to enqueue") when
a `chat.agent` turn is aborted mid-stream.
([#3792](https://github.com/triggerdotdev/trigger.dev/pull/3792))
- Updated dependencies:
- `@trigger.dev/core@4.5.0-rc.4`
- `@trigger.dev/build@4.5.0-rc.4`
- `@trigger.dev/schema-to-json@4.5.0-rc.4`
## @trigger.dev/core@4.5.0-rc.4
### Patch Changes
- Coerce numeric `concurrencyKey` values to string at the API boundary
across `tasks.trigger`, `tasks.batchTrigger`, and the Phase-2 streaming
batch endpoint.
([#3789](https://github.com/triggerdotdev/trigger.dev/pull/3789))
- Bump `@s2-dev/streamstore` to `0.22.10` to fix a
`TASK_RUN_UNCAUGHT_EXCEPTION` ("Invalid state: Unable to enqueue") when
a `chat.agent` turn is aborted mid-stream.
([#3792](https://github.com/triggerdotdev/trigger.dev/pull/3792))
## @trigger.dev/plugins@4.5.0-rc.4
### Patch Changes
- Updated dependencies:
- `@trigger.dev/core@4.5.0-rc.4`
## @trigger.dev/python@4.5.0-rc.4
### Patch Changes
- Updated dependencies:
- `@trigger.dev/sdk@4.5.0-rc.4`
- `@trigger.dev/core@4.5.0-rc.4`
- `@trigger.dev/build@4.5.0-rc.4`
## @trigger.dev/react-hooks@4.5.0-rc.4
### Patch Changes
- Updated dependencies:
- `@trigger.dev/core@4.5.0-rc.4`
## @trigger.dev/rsc@4.5.0-rc.4
### Patch Changes
- Updated dependencies:
- `@trigger.dev/core@4.5.0-rc.4`
## @trigger.dev/schema-to-json@4.5.0-rc.4
### Patch Changes
- Updated dependencies:
- `@trigger.dev/core@4.5.0-rc.4`
## @trigger.dev/sdk@4.5.0-rc.4
### Patch Changes
- Add a `tools` option to `chat.agent`. Declaring your tools here
threads them into the SDK's internal `convertToModelMessages`, so each
tool's `toModelOutput` is re-applied when prior-turn history is
re-converted.
([#3790](https://github.com/triggerdotdev/trigger.dev/pull/3790))
```ts
chat.agent({
tools: { readFile, search },
run: async ({ messages, tools, signal }) =>
streamText({ model, messages, tools, abortSignal: signal }),
});
```
Also exports `InferChatUIMessageFromTools<typeof tools>` to derive the
chat `UIMessage` type (typed tool parts) directly from a tool set.
- Updated dependencies:
- `@trigger.dev/core@4.5.0-rc.4`
</details>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Follow-up to #3796, which bumped the slack-client axios paths but left
posthog-node's transitive `axios@1.15.1` in place.
`posthog-node` 4.17.1 → 5.35.6. v5 drops the axios dependency entirely
(it's now fetch-based via `@posthog/core`), so posthog's old axios path
disappears. With #3796 already on main (webapp + d3 references on
`@slack/web-api@7.16.0`), nothing else pins the old line, so the
now-dead `axios@>=1.0.0 <1.15.0` override is removed and axios resolves
to a single patched `1.16.1` repo-wide. This closes the remaining axios
advisories.
Compat: the webapp's usage in `telemetry.server.ts` (`new PostHog(key, {
host })`, `.identify`, `.groupIdentify`, `.capture`) is all object-form
API that v5 preserves; `pnpm run typecheck --filter webapp` passes.
Node: posthog-node v5 requires Node `^20.20.0 || >=22.22.0`. We run
20.20.0 in dev (`.nvmrc`), CI, and the published Docker image
(`node:20.20-bullseye-slim`), so we're compliant.
Bumps `@slack/web-api` 7.9.1 → 7.16.0 in the webapp and the two
`references` examples (d3-chat, d3-openai-agents). 7.16.0 depends on
`axios@^1.16.0`, so every slack-client axios path resolves to 1.16.1
instead of 1.15.1.
This clears the slack and references axios paths. `posthog-node`'s
transitive axios still resolves the older line - that's handled in a
follow-up that upgrades posthog-node to v5 (which drops the axios
dependency entirely and lets us retire the now-stale axios override).
The dependabot axios advisories fully close once both land.
## Summary
A `chat.agent` turn that gets aborted mid-stream (stop generation, idle
suspend, cancellation) could surface a `TASK_RUN_UNCAUGHT_EXCEPTION` —
`TypeError: Invalid state: Unable to enqueue` — in the dashboard. The
run kept working, but the exceptions were loud and confusing.
## Root cause
The realtime stream writer batches chunks through
`@s2-dev/streamstore`'s `BatchTransform`, which holds them for a short
linger window before flushing. When the turn's abort signal fires while
a record is still buffered, the stream's writable is aborted and the
transform's readable controller is closed — but the pending linger
`setTimeout` still fires and calls `controller.enqueue()` on the dead
controller, throwing from a timer callback where nothing can catch it.
Fixed upstream in `@s2-dev/streamstore@0.22.10`, which wraps the linger
flush in a try/catch that discards the closed-controller error. This
bumps the dependency across core, the CLI, and the webapp, and adds a
regression test against the real `BatchTransform`.
Verified end-to-end: a mid-stream stop that previously failed the run
with `TASK_RUN_UNCAUGHT_EXCEPTION` now leaves the run healthy.
## Summary
`chat.agent` now takes a `tools` option. Until now tools only went to
`streamText` inside `run()`, so the SDK had no tools when it
re-converted the persisted `UIMessage` history at the start of each
turn. Any tool with a `toModelOutput` (raw image bytes into an image
content part, or a sub-agent transcript compressed to a summary) had its
transform applied on turn 1 and skipped from turn 2 onward, so the raw
output got JSON-stringified back into the prompt and the model lost the
transformed view.
Declaring `tools` on the config threads them into that conversion, so
`toModelOutput` runs on every turn. The resolved set is handed back,
typed, on the `run()` payload as `tools`:
```ts
const tools = { searchDocs, renderChart };
export const myChat = chat.agent({
tools,
run: async ({ messages, tools, signal }) =>
streamText({ ...chat.toStreamTextOptions({ tools }), messages, abortSignal: signal }),
});
```
`tools` also accepts a per-turn function for tools that depend on the
user or a feature flag. Only `inputSchema` and `toModelOutput` are read
during conversion, never `execute`. Also exports
`InferChatUIMessageFromTools<typeof tools>` to derive the chat
`UIMessage` type from a tool set. No behavior change for agents that
don't declare `tools`.
## Summary
Dashboard surfaces handle buffered runs by falling back to the mollifier
snapshot:
- Run detail, span detail, streams view (`_app.../runs.\$runParam`,
`resources.../spans.\$spanParam`, `resources.../streams.\$streamKey`).
- Redirect routes (`@.runs.\$runParam`, `runs.\$runParam`,
`projects.v3.\$projectRef.runs.\$runParam`).
- Action routes — cancel / replay / idempotency-reset / debug — under
`resources.taskruns/...` and `resources.../idempotencyKey.reset`.
- Logs download.
- Realtime subscription route + per-run resource
(`realtime.v1.runs.\$runId`, `resources.../realtime.v1.*`).
- `CancelRunDialog` gains an `onCancelSubmitted` callback so submit
isn't raced by the Radix `DialogClose` wrapper.
Stacked on the mutations PR.
## Test plan
- [x] \`pnpm run typecheck --filter webapp\` passes
- [x] \`pnpm run test --filter webapp
test/mollifierRealtimeRunResource.test.ts\` passes
- [x] \`pnpm run test --filter webapp
test/mollifierRealtimeRunResourceBuffer.test.ts\` passes
- [x] \`pnpm run test --filter webapp
test/mollifierRealtimeSubscription.test.ts\` passes
- [x] Manual smoke: trigger a buffered run, open it in the dashboard,
replay/cancel from the UI
---
## Ship-gate follow-up fixes
- **Auto-redirect to root span on direct nav** — loader sets `?span=`
from root span (PG) or buffered snapshot spanId before 302'ing, so
bookmark/share-link/direct-nav doesn't leave the panel collapsed.
- **RunPresenter switches from `findFirstOrThrow` to `findFirst` + typed
`RunNotInPgError`** — kills the per-poll `PrismaClient error` log spam
for buffered runs without changing the route-loader's fallback flow.
- **Span detail panel renders for buffered runs** — `SpanPresenter.call`
now falls back to `findRunByIdWithMollifierFallback` +
`buildSyntheticSpanRun` instead of returning undefined and triggering
the "Event not found" toast loop.
- **Logs download for buffered runs returns a gzipped placeholder line**
— replaces the 404 with a content-encoded line explaining the run is
queued. Same org-membership gate as the PG path.
- **Admin Debug-Run button hidden for buffered runs + SpanRun circular
type alias broken** (squashed) — buttons gate on a new `isBuffered` flag
on the synthetic SpanRun. Required grounding SpanRun in
`SpanPresenter.getRun` to break a circular type alias TS no longer
tolerates once `isBuffered` is a literal field on the shape.
- **Replay action requires user auth + org-membership** (🚩 Devin
finding) — `action` was unauthenticated and the PG `findFirst` had no
org filter, so any caller with a valid `runParam` could replay any run.
Buffered fallback inherited the same gap. Fixed to mirror the cancel
route.
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
## Summary
Cancel, replay, reschedule, metadata, tags, and idempotency-key-reset
now succeed against a run that's still in the mollifier buffer.
Mutations are applied to the buffered snapshot via Lua CAS; the drainer
carries the mutation forward when it replays.
Primitives added:
- `mutateWithFallback` — PG-first / buffer-fallback resolver with
bounded-wait safety net for entries that transition mid-mutation.
- `applyMetadataMutation` — buffered metadata PUT mirroring the PG-side
retry loop with CAS atomicity.
- `resolveRunForMutation` — discriminated-union resolver used by route
`findResource` so the route builder's pre-action 404 check sees buffered
runs.
Routes wired (whole files, no GET/POST splits):
- `api.v2.runs.\$runParam.cancel.ts`
- `api.v1.runs.\$runParam.replay.ts`
- `api.v1.runs.\$runParam.reschedule.ts`
- `api.v1.runs.\$runId.metadata.ts`
- `api.v1.runs.\$runId.tags.ts`
- `resetIdempotencyKey.server.ts`
Stacked on the reads PR.
## Test plan
- [x] \`pnpm run typecheck --filter webapp\` passes
- [x] \`pnpm run test --filter webapp
test/mollifierMutateWithFallback.test.ts\` passes
- [x] \`pnpm run test --filter webapp
test/mollifierApplyMetadataMutation.test.ts\` passes
- [x] \`pnpm run test --filter webapp
test/mollifierResolveRunForMutation.test.ts\` passes
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
## Summary
Synthesise QUEUED/FAILED responses from the mollifier buffer when a
TaskRun row hasn't landed in Postgres yet. Wires the synthesis into:
- `ApiRetrieveRunPresenter`
- v1 trace GET route
- v1 spans GET route
- attempts route gains a GET loader (fixes pre-existing Remix "no
loader" 400)
The `readFallback` infra itself lives on the trigger PR (consumed by
`IdempotencyKeyConcern`); this PR adds the route-level
synthetic-rendering primitives.
Stacked on the replay PR.
## Test plan
- [x] \`pnpm run typecheck --filter webapp\` passes
- [x] \`pnpm run test --filter webapp
test/mollifierSyntheticRedirectInfo.test.ts\` passes
- [x] \`pnpm run test --filter webapp
test/mollifierSyntheticSpanRun.test.ts\` passes
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
## Summary
The replay side of the mollifier:
- `DrainerHandler`: reads buffered snapshots and replays them through
`engine.trigger` to materialise PG rows.
- `RunEngine.createCancelledRun`: new public method the handler uses to
write CANCELED rows directly from snapshots (bypass queue + waitpoint,
emit `runCancelled`). Tolerates the cjson empty-table tags edge case
found during validation.
- Drainer fairness: org → env rotation so a heavy env doesn't starve
light ones in the same org.
- Stale-entry sweep + telemetry + alertable gauge so a stuck/offline
drainer surfaces in alerts.
Both the drainer and sweep default-off; nothing fires unless flagged on
(`TRIGGER_MOLLIFIER_DRAINER_ENABLED`,
`TRIGGER_MOLLIFIER_STALE_SWEEP_ENABLED`).
Stacked on the trigger-time decisions PR.
## Test plan
- [x] \`pnpm run typecheck --filter webapp\` passes
- [x] \`pnpm run test --filter webapp
test/mollifierDrainerHandler.test.ts\` passes
- [x] \`pnpm run test --filter webapp test/mollifierStaleSweep.test.ts\`
passes
- [x] \`pnpm run test --filter @internal/run-engine
src/engine/tests/createCancelledRun.test.ts\` passes
- [x] \`pnpm run test --filter @trigger.dev/redis-worker
packages/redis-worker/src/mollifier/drainer.test.ts\` passes
---
## Ship-gate follow-up fix
**Drainer writes SYSTEM_FAILURE on max-attempts exhaustion.** Adds an
`onTerminalFailure` callback on `MollifierDrainerOptions` so the
customer's run lands a SYSTEM_FAILURE PG row even when the drainer
exhausts `MAX_ATTEMPTS` on a retryable PG error (previously
`buffer.fail()` was called with no row written → silent data loss). The
callback runs before `buffer.fail()` on every terminal path
(non-retryable AND max-attempts-exhausted), and re-throwing a retryable
error from the callback causes the drainer to requeue rather than fail.
Bumps `@trigger.dev/redis-worker` to a **minor** changeset (additive
option + new exported types). Includes 5 unit tests covering both
terminal causes plus the requeue-on-retryable-callback-failure path and
no-callback back-compat.
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
## Summary
`concurrencyKey` validation accepted only `z.string().optional()` on the
single-trigger and V2/V3 batch endpoints, and the Phase-2 streaming
NDJSON endpoint accepted `z.record(z.unknown()).optional()` for the
entire `options` field. Callers passing `concurrencyKey: someNumericId`
(e.g. `payload.userId`) either failed schema validation on the first two
paths or sailed through on Phase-2 and then failed downstream at
`prisma.taskRun.create` with `Argument concurrencyKey: Expected String
or Null, provided Int`.
The schema now accepts `string | number` for `concurrencyKey` and
stringifies on the way in, across all three paths. The Phase-2 NDJSON
`options` is tightened to reuse the strict
`BatchTriggerTaskItem.options` shape so it validates identically to the
V2/V3 batch endpoints.
A defensive `typeof === "number"` coercion at the `engine.trigger` call
site in `RunEngineTriggerTaskService` covers in-flight Redis-stored
batch items enqueued before the schema fix — those items are rebuilt
from a `Record<string, unknown>` shape that bypasses the new schema and
would otherwise continue failing for up to their TTL.
## Test plan
- [x] `packages/core/src/v3/schemas/batchItemNDJSON.test.ts` — unit
tests covering numeric→string coercion, string passthrough, no-options,
and rejection of non-string/non-number shapes across
`TriggerTaskRequestBody`, `BatchTriggerTaskItem`, and `BatchItemNDJSON`.
- [x] `apps/webapp/test/engine/triggerTask.test.ts` — `containerTest`
simulating the in-flight Redis batch-item shape (numeric
`concurrencyKey` via `Record<string, unknown>`), verifies the run is
created with `concurrencyKey: "51262"`. Without the worker coercion, the
test reproduces the production stack at `prisma.taskRun.create`.
- [x] `pnpm run typecheck --filter webapp` clean.
- [x] `pnpm run build --filter @trigger.dev/core --filter
@trigger.dev/sdk --filter trigger.dev` clean.
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
## Summary
The trigger hot path's mollifier integration:
- `mollifyTrigger`: when the gate trips, write the engine.trigger
snapshot to the buffer and return a synthesised QUEUED response.
Postgres write is deferred to drainer-replay (next PR in the stack).
- Pre-gate idempotency-key claim: same-key triggers serialise through
Redis so a burst lands in PG / buffer exactly once.
- Read-fallback extensions: `findRunByIdWithMollifierFallback` for the
trigger-time idempotency lookup that must see buffered runs.
- Gate bypasses: `debounce`, `oneTimeUseToken`,
`parentTaskRunId`/`triggerAndWait` skip the mollify path entirely.
- `triggerTask` + `IdempotencyKeyConcern` wired to the above.
All behaviour gated by the master `TRIGGER_MOLLIFIER_ENABLED` switch;
off-state hot path is unchanged (the gate is not even consulted).
Stacked on the buffer extensions PR.
## Test plan
- [x] \`pnpm run typecheck --filter webapp\` passes
- [x] \`pnpm run test --filter webapp test/mollifierMollify.test.ts\`
passes
- [x] \`pnpm run test --filter webapp
test/mollifierIdempotencyClaim.test.ts\` passes
- [x] \`pnpm run test --filter webapp
test/mollifierReadFallback.test.ts\` passes
- [x] \`pnpm run test --filter webapp test/mollifierGate.test.ts\`
passes
- [x] \`pnpm run test --filter webapp test/engine/triggerTask.test.ts\`
passes
---
## Ship-gate follow-up fixes
- **Batch items bypass the mollifier gate** — fixes
`BatchTaskRunItem_taskRunId_fkey` FK violation on batch triggers when
the gate trips. End-state is a drainer-side `BatchTaskRunItem`
create-on-materialise; batch traffic passes through the gate until that
lands.
- **IdempotencyKeyConcern honours buffered-run TTL on expiry** —
buffered path now clears expired idempotency claims (read-side) and
resets the buffer's `mollifier:idempotency:*` SETNX binding (write-side)
so a re-trigger past the customer's TTL lands as a fresh run instead of
echoing the stale buffered runId.
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
## Summary
Buffer-side data layer used by the rest of the mollifier phase-3 stack.
- `buffer.ts` gains entry inspection (`getEntry`), idempotency lookup
(`lookupIdempotency`), in-place snapshot mutation (`mutateSnapshot`),
and dwell tracking. All atomic via Lua.
- `mollifierSnapshot.server.ts`: shared `MollifierSnapshot` type plus
(de)serialise helpers.
- Drops the entry-TTL config and its env var. The drainer is the
recovery mechanism; an entry that survives the drainer should surface as
a stale-sweep alert, not silently TTL away.
Adds methods to the buffer interface; nothing consumes them yet.
Subsequent PRs in the stack wire trigger-time mollify, read-fallback,
and mutation paths against this surface.
## Test plan
- [x] \`pnpm run typecheck --filter webapp\` passes
- [x] \`pnpm run test --filter @trigger.dev/redis-worker
packages/redis-worker/src/mollifier/buffer.test.ts\` passes
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
## Summary
1 improvement, 2 bug fixes.
## Improvements
- Retry `TASK_MIDDLEWARE_ERROR` under the task's retry policy instead of
failing the run on the first attempt. The error was already classified
as retryable by `shouldRetryError`, but `shouldLookupRetrySettings` did
not include it, so the retry flow fell through to `fail_run`. Fixes
#3231. ([#3676](https://github.com/triggerdotdev/trigger.dev/pull/3676))
## Bug fixes
- Fix `TypeError` in `unflattenAttributes` when the input attribute map
contains conflicting dotted key paths (e.g. both `a.b` set to a scalar
and `a.b.c` set to a value). The path-walk loop now applies
last-write-wins when a prior key wrote a primitive, null, or array at an
intermediate slot, matching the existing precedent in
`AttributeFlattener.addAttribute`. Callers no longer crash when handed
malformed external attribute inputs.
([#3762](https://github.com/triggerdotdev/trigger.dev/pull/3762))
- Fix external trace context leaking across runs on warm-started workers
with `processKeepAlive` enabled. Every subsequent run's attempt span was
being exported with the first run's `traceId` and `parentSpanId`,
breaking causal-chain navigation in external APM tools. Runs without an
external trace context are unaffected.
([#3768](https://github.com/triggerdotdev/trigger.dev/pull/3768))
<details>
<summary>Raw changeset output</summary>
⚠️⚠️⚠️⚠️⚠️⚠️
`main` is currently in **pre mode** so this branch has prereleases
rather than normal releases. If you want to exit prereleases, run
`changeset pre exit` on `main`.
⚠️⚠️⚠️⚠️⚠️⚠️
# Releases
## @trigger.dev/build@4.5.0-rc.3
### Patch Changes
- Updated dependencies:
- `@trigger.dev/core@4.5.0-rc.3`
## trigger.dev@4.5.0-rc.3
### Patch Changes
- Updated dependencies:
- `@trigger.dev/core@4.5.0-rc.3`
- `@trigger.dev/build@4.5.0-rc.3`
- `@trigger.dev/schema-to-json@4.5.0-rc.3`
## @trigger.dev/core@4.5.0-rc.3
### Patch Changes
- Retry `TASK_MIDDLEWARE_ERROR` under the task's retry policy instead of
failing the run on the first attempt. The error was already classified
as retryable by `shouldRetryError`, but `shouldLookupRetrySettings` did
not include it, so the retry flow fell through to `fail_run`. Fixes
#3231. ([#3676](https://github.com/triggerdotdev/trigger.dev/pull/3676))
- Fix `TypeError` in `unflattenAttributes` when the input attribute map
contains conflicting dotted key paths (e.g. both `a.b` set to a scalar
and `a.b.c` set to a value). The path-walk loop now applies
last-write-wins when a prior key wrote a primitive, null, or array at an
intermediate slot, matching the existing precedent in
`AttributeFlattener.addAttribute`. Callers no longer crash when handed
malformed external attribute inputs.
([#3762](https://github.com/triggerdotdev/trigger.dev/pull/3762))
- Fix external trace context leaking across runs on warm-started workers
with `processKeepAlive` enabled. Every subsequent run's attempt span was
being exported with the first run's `traceId` and `parentSpanId`,
breaking causal-chain navigation in external APM tools. Runs without an
external trace context are unaffected.
([#3768](https://github.com/triggerdotdev/trigger.dev/pull/3768))
## @trigger.dev/plugins@4.5.0-rc.3
### Patch Changes
- Updated dependencies:
- `@trigger.dev/core@4.5.0-rc.3`
## @trigger.dev/python@4.5.0-rc.3
### Patch Changes
- Updated dependencies:
- `@trigger.dev/core@4.5.0-rc.3`
- `@trigger.dev/build@4.5.0-rc.3`
- `@trigger.dev/sdk@4.5.0-rc.3`
## @trigger.dev/react-hooks@4.5.0-rc.3
### Patch Changes
- Updated dependencies:
- `@trigger.dev/core@4.5.0-rc.3`
## @trigger.dev/redis-worker@4.5.0-rc.3
### Patch Changes
- Updated dependencies:
- `@trigger.dev/core@4.5.0-rc.3`
## @trigger.dev/rsc@4.5.0-rc.3
### Patch Changes
- Updated dependencies:
- `@trigger.dev/core@4.5.0-rc.3`
## @trigger.dev/schema-to-json@4.5.0-rc.3
### Patch Changes
- Updated dependencies:
- `@trigger.dev/core@4.5.0-rc.3`
## @trigger.dev/sdk@4.5.0-rc.3
### Patch Changes
- Updated dependencies:
- `@trigger.dev/core@4.5.0-rc.3`
</details>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
## Summary
Drops the `TaskRun_status_runtimeEnvironmentId_createdAt_id_idx` index
from the `TaskRun` table. After #3742 gated the legacy
`WAITING_FOR_DEPLOY` drain to V1-engine workers only, this index sees
zero scans on both writer and reader replicas. Removing it cuts index
maintenance on every `TaskRun` INSERT/UPDATE.
## Why
The index existed to support `WHERE status = X AND runtimeEnvironmentId
= Y` queries from `ExecuteTasksWaitingForDeployService`, which is
V1-only and no longer triggered on V2 deployments. A code grep across
`apps/webapp` and `internal-packages/run-engine` confirmed no V2
production query uses this access pattern — every other `status:` filter
on `TaskRun` is paired with `id`/`friendlyId`/`parentSpanId` and uses a
different index.
Dropping it also unlocks HOT updates on the dequeue path. The dequeue
`UPDATE` modifies `status` (`QUEUED` -> `DEQUEUED`), and `status` is the
leading column of this index — its presence blocked HOT eligibility for
every `TaskRun` UPDATE. With the index gone, dequeue UPDATEs can become
HOT, reducing WAL bytes and removing the B-tree page contention on this
index's right-edge leaves.
Uses `DROP INDEX CONCURRENTLY` to avoid blocking writes during the drop.
## Sequencing
Should only ship once #3742 has soaked long enough to confirm the index
is genuinely cold (24h+ of zero scans on `pg_stat_user_indexes`).
Make `POST /api/v1/deployments/:deploymentId/background-workers`
idempotent so client-side retries no longer collide on the
`BackgroundWorker` `(project, env, version)` unique index. Helps make
deployments more resilient against the class of indexing failures that
surfaces in the dashboard as "Indexing timed out", e.g. during transient
database issues.
On warm-started workers with `processKeepAlive` enabled, every run's
attempt span was exported with the first run's `traceId` and
`parentSpanId`. `ExternalSpanExporterWrapper` and
`ExternalLogRecordExporterWrapper` captured `externalTraceContext` at
`TracingSDK` construction, and the SDK is memoized for the worker's
lifetime - so the per-run reset of
`StandardTraceContextManager.traceContext` never reached the wrappers.
Reported by a customer running v4.4.x: 33 distinct runs on the same
host/pid showed up in their APM as siblings of one parent span.
Fix: the wrappers now read external context live from the trace context
manager per export. Runs without an external trace context fall through
to the unchanged `externalTraceId` fallback - no behaviour change for
them.
Regression test in
`packages/core/test/externalSpanExporterWrapper.test.ts` asserts that
reassigning the manager between exports produces correctly-parented
spans.
Retries `TASK_MIDDLEWARE_ERROR` under the task's retry policy.
`shouldRetryError` already classed it as retryable, but
`shouldLookupRetrySettings` did not, so the run fell through to
`fail_run` on attempt 1 instead of using the task's `retry` config.
Fixes#3231.
## Summary
Second class of poisoned-row failure in the runs replication path. PR
#3708 plugged lone UTF-16 surrogates; this one handles bare JSON integer
literals outside ClickHouse's `Int64`..`UInt64` range. Recovery stays
purely reactive — the existing `sanitizeRows` walker just gains an extra
branch, so the hot replication path pays nothing on healthy rows.
Fixes the still-firing customer-facing symptom from
[TRI-9755](https://linear.app/triggerdotdev/issue/TRI-9755):
`scan-social-profiles` runs continued to be stranded in `EXECUTING` on
the Tasks page after #3708 deployed. CloudWatch showed `Dropped batch —
ClickHouse JSON parse error but sanitizer found nothing to fix` firing
**8/8 times** since the previous deploy (zero successful sanitizations).
Root cause: upstream JS Number precision loss on a 21-digit Google Plus
ID (`117039831458782873093` → `117039831458782870000`) — the
precision-lossy value still serialises as a bare integer that exceeds
`UInt64.MAX`, which ClickHouse rejects with `INCORRECT_DATA`.
## How the bug ships
The customer task emits an output containing a Poshmark profile's
`spec_format`:
```json
{"key":"gp_id","proper_key":"Gp Id","value":117039831458782870000,"type":"int"}
```
That value is `1.17e20` — comfortably above `UInt64.MAX` (`1.84e19`) but
comfortably below `1e21`. `Number.prototype.toString` only switches to
exponential form at `|value| >= 1e21`, so `JSON.stringify` emits the
bare token `117039831458782870000` and the ClickHouse
`JSON(max_dynamic_paths)` column fails with:
```
Code: 117. DB::Exception: Cannot parse JSON object here: {…}: (while reading the value of key output): (at row 1)
: While executing ParallelParsingBlockInputFormat. (INCORRECT_DATA) (version 25.12.x)
```
Same error verbatim as prod. The same number quoted
(`"117039831458782870000"`) inserts fine — ClickHouse's dynamic JSON
column accepts a `String` subtype on the same path.
## What changed
`apps/webapp/app/v3/eventRepository/sanitizeRowsOnParseError.server.ts`:
- New private `isUnsafeJsonInteger(value)` helper — true iff `value` is
a finite integer-valued JS Number where `|value| < 1e21` (so
`JSON.stringify` emits integer form, not exponent) **and** `value` falls
outside `[Int64.MIN, UInt64.MAX]`.
- `sanitizeUnknownInPlace` gains a number-branch: when the predicate
holds, replace the Number with `String(value)`. The downstream JSON
column dynamic-types the path as String for that row — fine, since the
value was already precision-lossy upstream (no JS Number above 2^53 is
numerically meaningful anyway).
- Float-valued numbers, large floats (>= 1e21), NaN and Infinity are
left alone — `JSON.stringify` emits them with exponents or as `null`,
both of which ClickHouse accepts.
`apps/webapp/test/sanitizeRowsOnParseError.test.ts`: four new unit tests
+ an extension to `sanitizeRows` covering surrogate + integer fixes
counted together across rows. The unit suite now covers:
- Positive value above `UInt64.MAX` (`117039831458782870000` — the
actual prod value)
- Negative value below `Int64.MIN`
- Boundary values pass through (`42`, `Number.MAX_SAFE_INTEGER`, `2^63`)
- Non-integer numbers untouched (floats, `1e25`, NaN, Infinity)
- The actual `scan-social-profiles` nested shape — finds the offending
`gp_id` deep inside
`output.data.profiles[].spec_format[].platform_variables[].value`
`.server-changes/runs-replication-bigint-recovery.md` — release notes
entry.
## Why reactive, not pre-flight
`#prepareJson` runs millions of times per day on the replication hot
path. Walking every JSON tree to look for oversized integers would add
bounded-but-real CPU on every healthy row. `sanitizeRows` only fires
after a ClickHouse parse-error rejection, which is a few times a day
platform-wide. Extending it costs effectively zero on healthy traffic
and gains us recovery on the rare poisoned row.
## Verification
- Reproduced 1:1 in a throwaway Docker
`clickhouse/clickhouse-server:25.12.11.4` (closest available to the prod
`25.12.1.1579` build). Pre-sanitize JSON fails with the exact prod
error; post-sanitize JSON inserts cleanly and the row is readable with
`gp_id` stored as a String subtype.
- `pnpm --filter webapp exec vitest run
test/sanitizeRowsOnParseError.test.ts` — 22/22 passing (18 existing + 4
new).
- `pnpm run typecheck --filter webapp` — clean.
## Test plan
- [x] `pnpm run typecheck --filter webapp`
- [x] Unit tests pass against new + existing cases
- [x] End-to-end Docker ClickHouse repro confirms recovery
- [ ] Post-deploy: confirm `Sanitizing batch after ClickHouse JSON parse
error` warns fire instead of `Dropped batch …` errors when
`scan-social-profiles` outputs trip CH again
- [ ] Post-deploy: confirm `permanentlyDroppedBatches` counter stops
climbing in
`/stp/trigger-app-prod/ecs/replication/service-container/process-logs`
## What this does NOT do
- Doesn't backfill the ~120k+ existing stranded `EXECUTING` rows in
production. Same as #3708 — that needs a reconciliation/backfill sweep
(separate ticket — TRI-9755 fix#3).
- Doesn't address the upstream root cause (the customer task emitting a
JS-Number-precision-lossy big int). That's a customer-task concern; our
replication path needs to be robust to whatever shape arrives.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
## Summary
Stops the legacy V1 `WAITING_FOR_DEPLOY` drain from running on every V2
deployment promotion. The drain queries `TaskRun` by
`status='WAITING_FOR_DEPLOY'`, which only V1-engine runs ever have — V2
runs use `PENDING_VERSION` and are handled out of band. Calling the
drain on V2 promotes produced empty queries against the status index and
unnecessary reader-DB load.
## Fix
Two layers:
1. Gate the enqueue at the call site in `ChangeCurrentDeploymentService`
so it only fires when the deployment's worker is on engine V1.
2. Add a `LEGACY_RUN_ENGINE_WAITING_FOR_DEPLOY_DISABLED` env var
(default `0`). When set to `1`, the service returns immediately from
`call()` — neuters any jobs already sitting in the worker queue from
before the deploy lands.
V1 customers see no change; V2 promotes no longer trigger the drain.
Linear: [TRI-9864](https://linear.app/triggerdotdev/issue/TRI-9864)
(Urgent)
Production incident:
[TRI-9863](https://linear.app/triggerdotdev/issue/TRI-9863) (mitigated
by image revert in cloud#910)
## Bug
`apps/webapp/package.json` declares `"sideEffects": false`. PR #3333
(`71d98b4e`) replaced the previous real method-call retention idiom at
the two `sessionsReplicationInstance` import sites with:
```ts
import { sessionsReplicationInstance } from "...";
void sessionsReplicationInstance;
```
esbuild treats `void <identifier>;` as a pure expression statement under
`sideEffects: false` and **tree-shakes the entire import** — including
the `singleton(...)` call inside `sessionsReplicationInstance.server.ts`
which is the only thing that fires
`initializeSessionsReplicationInstance()`. The sessions→ClickHouse
logical replication worker never starts, the slot is unconsumed, lag
grows.
### How it manifested in production
cloud#907's image bump rolled the `SessionReplicationService` ECS task
on prod at 14:32 UTC. The new container's startup log emitted `🗃️ Runs
replication service enabled` but **not** `🗃️ Sessions replication
service enabled` or `🗃️ Sessions replication service started`.
CloudWatch `OldestReplicationSlotLag` grew at ~220 MB/min and the `High
replication lag` alarm fired at 14:37 UTC. Prod was reverted to the
previous image (cloud#910) to stop the bleed.
### Verification
`grep` of the built bundle `apps/webapp/build/index.js` (built from
`c0365d36`):
- **3** occurrences of `Runs replication` / `runsReplicationInstance`
strings ✅
- **0** occurrences of `Sessions replication` /
`sessionsReplicationInstance` / `SessionsReplicationService` ❌
The runs path survives tree-shaking because `adminWorker.server.ts` and
`admin.api.v1.runs-replication.*` routes have real method calls
(`.start()`, `.teardown()`, `.backfill()`) — observable uses the
tree-shaker must preserve. The sessions singleton has no real callers,
only the `void` no-ops, hence its complete elimination from the bundle.
## Fix
Replace `void sessionsReplicationInstance;` with an assignment to
`globalThis`, an unambiguous observable side effect the bundler cannot
eliminate:
```ts
(globalThis as Record<string, unknown>).__sessionsReplicationInstance =
sessionsReplicationInstance;
```
Applied at both call sites: `apps/webapp/app/entry.server.tsx` and
`apps/webapp/app/v3/services/adminWorker.server.ts`.
Surrounding comments updated to document the bundler interaction so the
next maintainer doesn't reintroduce `void`.
## Out of scope (follow-ups)
- **Robustness improvement**: change `apps/webapp/package.json` from
`"sideEffects": false` to an allowlist that includes
`*Instance.server.ts` files. Prevents the same regression shape via any
future `*Instance` singleton.
- **Build-time check**: add a `grep` post-build step in `publish.yml`
requiring `"Sessions replication"` to appear in
`apps/webapp/build/index.js`. Catches this exact regression at CI time.
## Test plan
- [x] `pnpm run typecheck --filter webapp` clean
- [ ] After merge + publish: confirm new image's
`SessionReplicationService` container logs `🗃️ Sessions replication
service enabled` and `🗃️ Sessions replication service started` at
startup
- [ ] After re-deploying to prod: confirm `OldestReplicationSlotLag`
stops growing and drains
## Summary
On a ClickHouse `Cannot parse JSON object` rejection,
`RunsReplicationService` now sanitizes lone UTF-16 surrogates across the
failing batch via the existing `sanitizeRows` helper and retries once.
If the sanitizer found nothing or the retry also fails, the batch is
dropped loudly with a counter increment, so the surrounding
`#insertWithRetry` layer doesn't spin three more times on a
deterministic failure. Non-parse errors propagate unchanged.
Mirrors the pattern from #3659 (for `ClickhouseEventRepository`) — same
root cause (lone UTF-16 surrogates in user-provided JSON), same recovery
shape, **reusing the same shared helpers** (`sanitizeRows`,
`isClickHouseJsonParseError`, `parseRowNumberFromError`).
Fixes the customer-facing symptom from
[TRI-9755](https://linear.app/triggerdotdev/issue/TRI-9755): a single
row's poisoned `output` JSON used to take down the
`COMPLETED_SUCCESSFULLY` UPDATE events for its 50+ batch-mates,
stranding them in `EXECUTING` in ClickHouse forever and inflating
"Running" counts on the Tasks page. Confirmed in production this is
ongoing — ~120k stale rows accumulated in a single 5-hour burst on
2026-05-18; smaller continuous leak before and after.
## What changed
`apps/webapp/app/services/runsReplicationService.server.ts`:
- Imports the three helpers from
`~/v3/eventRepository/sanitizeRowsOnParseError.server` (no duplication;
no move).
- New private `#insertWithJsonParseRecovery<T>(rows, doInsert,
contextLabel, attempt)` — generic over `TaskRunInsertArray[]` and
`PayloadInsertArray[]`, structurally identical to
`ClickhouseEventRepository.#insertWithJsonParseRecovery`. Try → on parse
error sanitize the whole batch (the `at row N` hint is logged but not
used to slice — semantics under `input_format_parallel_parsing` aren't
stable) → retry once → drop with loud log if sanitizer found nothing OR
retry still fails.
- `#insertTaskRunInserts` and `#insertPayloadInserts` extract a
`doInsert` closure and hand it to the wrapper. Existing error logging,
span recording, and `recordSpanError` are preserved inside the closure.
- New `private _permanentlyDroppedBatches = 0` counter with a public
getter, for ops dashboards and tests (matches the events-repo
convention). One shared counter for both insert sites — granularity
comes from the `contextLabel` (`task_runs_v2` /
`raw_task_runs_payload_v1`) on every log line.
`.server-changes/runs-replication-utf16-recovery.md` — release notes
entry.
## Why no new tests
The shared helpers already have full unit + real-ClickHouse contract
coverage from #3659
(`apps/webapp/test/sanitizeRowsOnParseError.test.ts`,
`apps/webapp/test/otlpUtf16Sanitization.integration.test.ts`). The new
wrapper is a line-for-line structural port. Adding a parallel
integration test would require synthesizing bad data that *escapes* the
preemptive `detectBadJsonStrings` check in `#prepareJson` but still
trips ClickHouse — non-trivial without hand-crafted fixtures and
wouldn't cover any new logic.
## What this does NOT do
- Doesn't touch the ~120k existing stale `EXECUTING` rows in production.
That needs a reconciliation/backfill sweep (separate ticket — TRI-9755
fix#3).
- Doesn't sanitize the `error` column path
(`runsReplicationService.server.ts:932 const errorData = { data:
run.error };`). Reactive recovery will catch it if it ever poisons a
batch, but feeding it through `#prepareJson` like `output` is a cheap
follow-up.
## Test plan
- [x] `pnpm run typecheck --filter webapp` — clean
- [ ] Post-deploy: confirm `permanentlyDroppedBatches` counter stays at
zero (or near-zero) in
`/stp/trigger-app-prod/ecs/replication/service-container/process-logs`,
and watch for `Sanitizing batch after ClickHouse JSON parse error` warns
to confirm recovery is firing on real traffic
- [ ] Post-deploy: confirm the rate of new
"EXECUTING-but-actually-COMPLETED" zombies in ClickHouse flattens
(current rate ≈ tens-to-hundreds per hour platform-wide)
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
## Summary
Updates the AI chat docs to match the slim-wire + field-level merge
behavior shipped in #3719 and the precise `.in/append` cap +
CORS-readable 413 shipped in #3720. No behavior changes here — code is
correct in `main`; the docs were lagging on three patterns customers
copy out of the page.
## What changed
- **`hydrateMessages` examples upsert by id** (in `lifecycle-hooks.mdx`,
`patterns/database-persistence.mdx`, and
`patterns/persistence-and-replay.mdx`). The previous
`stored.push(newMsg)` pattern duplicated the assistant id on HITL
continuations and caused the LLM to receive a tool call with no
`arguments`. The new examples include the rationale inline.
- **`onValidateMessages` example filters to user messages**
(`lifecycle-hooks.mdx`). The previous example called
`validateUIMessages({ messages, tools })` directly, which now throws on
HITL slim wires (the AI SDK schema requires `input` on resolved tool
parts). New example shows the filter pattern, with a Warning callout
explaining why.
- **Merge contract description updated** (`lifecycle-hooks.mdx`). The
old wording said incoming messages are "auto-merged" / "replaced"; the
new description explains the actual field-level overlay (state advances
only).
- **Approval-responded wire example slimmed** (`client-protocol.mdx`).
Shows the minimum shape the agent reads — `state` + `approval` (or
`output` / `errorText` for HITL). Notes that the built-in transports
ship this slim shape by default and that fuller shapes are still
accepted.
- **`/in/append` 413 row and FAQ updated** (`client-protocol.mdx`,
`patterns/trusted-edge-signals.mdx`). Reflects the new precise S2 cap
and the CORS-readable 413.
- **New changelog entry** at the top of `changelog.mdx` covering all of
the above.
The historical `## 512 KiB ceiling removed` entry further down the
changelog is left as-is (it's a snapshot of the prior transition), and
the v4.5 upgrade-guide section is skipped — the merge contract is
backwards compatible.
## Test plan
- Mintlify dev preview renders cleanly with no broken anchors
- Linked references resolve (`/ai-chat/lifecycle-hooks#hydratemessages`,
`/ai-chat/lifecycle-hooks#onvalidatemessages`,
`/ai-chat/patterns/database-persistence#alternative-hydratemessages`,
`/ai-chat/client-protocol#step-3-send-messages-stops-and-actions`,
`/ai-chat/patterns/large-payloads`)
## Summary
3 improvements, 1 bug fix.
## Improvements
- The per-turn merge now overlays the wire copy's tool-part state
advancement onto the agent's existing chain — `state` + the matching
resolution field (`output` / `errorText` / `approval`) come from the
wire, everything else (text, reasoning, tool `input`, provider metadata)
stays whatever the snapshot or `hydrateMessages` returned. Previously a
full-message replace overwrote those fields with whatever the client
shipped, so a slimmed wire copy landed a tool call with no `arguments`
on the next LLM call. Covers `output-available` / `output-error` (HITL
`addToolOutput`) and `approval-responded` / `output-denied` (approval
flow).
- `TriggerChatTransport.sendMessages` and `AgentChat.sendRaw` now slim
assistant messages that carry advanced tool parts. The wire payload is
just `{ id, role, parts: [<state + resolution field>] }` for
`submit-message` continuations; everything else passes through.
Reasoning blobs and full tool inputs no longer ride the wire on every
`addToolOutput` / `addToolApproveResponse`, so continuation payloads
stay well under the `.in/append` cap on long agent loops.
- Add `TriggerClient` for running multiple SDK clients side-by-side,
each with its own auth, preview branch, and baseURL. Useful when a
single process needs to trigger tasks or read runs across multiple
projects, environments, or preview branches without mutating shared
global state.
([#3683](https://github.com/triggerdotdev/trigger.dev/pull/3683))
## Bug fixes
- Fix `chat.agent` HITL continuations on reasoning-heavy turns. Two
changes that work together:
([#3719](https://github.com/triggerdotdev/trigger.dev/pull/3719))
<details>
<summary>Raw changeset output</summary>
⚠️⚠️⚠️⚠️⚠️⚠️
`main` is currently in **pre mode** so this branch has prereleases
rather than normal releases. If you want to exit prereleases, run
`changeset pre exit` on `main`.
⚠️⚠️⚠️⚠️⚠️⚠️
# Releases
## @trigger.dev/build@4.5.0-rc.2
### Patch Changes
- Updated dependencies:
- `@trigger.dev/core@4.5.0-rc.2`
## trigger.dev@4.5.0-rc.2
### Patch Changes
- Updated dependencies:
- `@trigger.dev/build@4.5.0-rc.2`
- `@trigger.dev/core@4.5.0-rc.2`
- `@trigger.dev/schema-to-json@4.5.0-rc.2`
## @trigger.dev/plugins@4.5.0-rc.2
### Patch Changes
- Updated dependencies:
- `@trigger.dev/core@4.5.0-rc.2`
## @trigger.dev/python@4.5.0-rc.2
### Patch Changes
- Updated dependencies:
- `@trigger.dev/sdk@4.5.0-rc.2`
- `@trigger.dev/build@4.5.0-rc.2`
- `@trigger.dev/core@4.5.0-rc.2`
## @trigger.dev/react-hooks@4.5.0-rc.2
### Patch Changes
- Updated dependencies:
- `@trigger.dev/core@4.5.0-rc.2`
## @trigger.dev/redis-worker@4.5.0-rc.2
### Patch Changes
- Updated dependencies:
- `@trigger.dev/core@4.5.0-rc.2`
## @trigger.dev/rsc@4.5.0-rc.2
### Patch Changes
- Updated dependencies:
- `@trigger.dev/core@4.5.0-rc.2`
## @trigger.dev/schema-to-json@4.5.0-rc.2
### Patch Changes
- Updated dependencies:
- `@trigger.dev/core@4.5.0-rc.2`
## @trigger.dev/sdk@4.5.0-rc.2
### Patch Changes
- Fix `chat.agent` HITL continuations on reasoning-heavy turns. Two
changes that work together:
([#3719](https://github.com/triggerdotdev/trigger.dev/pull/3719))
- The per-turn merge now overlays the wire copy's tool-part state
advancement onto the agent's existing chain — `state` + the matching
resolution field (`output` / `errorText` / `approval`) come from the
wire, everything else (text, reasoning, tool `input`, provider metadata)
stays whatever the snapshot or `hydrateMessages` returned. Previously a
full-message replace overwrote those fields with whatever the client
shipped, so a slimmed wire copy landed a tool call with no `arguments`
on the next LLM call. Covers `output-available` / `output-error` (HITL
`addToolOutput`) and `approval-responded` / `output-denied` (approval
flow).
- `TriggerChatTransport.sendMessages` and `AgentChat.sendRaw` now slim
assistant messages that carry advanced tool parts. The wire payload is
just `{ id, role, parts: [<state + resolution field>] }` for
`submit-message` continuations; everything else passes through.
Reasoning blobs and full tool inputs no longer ride the wire on every
`addToolOutput` / `addToolApproveResponse`, so continuation payloads
stay well under the `.in/append` cap on long agent loops.
Note: `onValidateMessages` receives the slim wire on HITL turns. If you
call `validateUIMessages` from `ai` against the full `messages` array it
will reject the slim assistant; filter to user messages (or skip on HITL
turns) — see the updated docstring on `onValidateMessages` for the
recommended pattern.
For `hydrateMessages` hooks that persist the chain, this release also
adds a small helper to the `@trigger.dev/sdk/ai` surface:
```ts
import { chat, upsertIncomingMessage } from "@trigger.dev/sdk/ai";
chat.agent({
hydrateMessages: async ({ chatId, trigger, incomingMessages }) => {
const record = await db.chat.findUnique({ where: { id: chatId } });
const stored = record?.messages ?? [];
if (upsertIncomingMessage(stored, { trigger, incomingMessages })) {
await db.chat.update({ where: { id: chatId }, data: { messages: stored }
});
}
return stored;
},
});
```
It pushes fresh user messages by id, no-ops on HITL continuations (the
incoming shares an id with the existing assistant — the runtime overlays
the new tool-state advance), and skips on non-`submit-message` triggers.
Returns `true` if it mutated `stored` so the caller knows whether to
persist.
Net effect: `chat.addToolOutput(...)` /
`chat.addToolApproveResponse(...)` on multi-step reasoning agents
(OpenAI Responses with `store: false`, Anthropic extended thinking,
etc.) no longer blows the cap and no longer corrupts the LLM input.
- Add `TriggerClient` for running multiple SDK clients side-by-side,
each with its own auth, preview branch, and baseURL. Useful when a
single process needs to trigger tasks or read runs across multiple
projects, environments, or preview branches without mutating shared
global state.
([#3683](https://github.com/triggerdotdev/trigger.dev/pull/3683))
```ts
import { TriggerClient } from "@trigger.dev/sdk";
const prod = new TriggerClient({ accessToken:
process.env.TRIGGER_PROD_KEY });
const preview = new TriggerClient({
accessToken: process.env.TRIGGER_PREVIEW_KEY,
previewBranch: "signup-flow",
});
await prod.tasks.trigger("send-email", payload);
await preview.runs.list({ status: ["COMPLETED"] });
```
- Updated dependencies:
- `@trigger.dev/core@4.5.0-rc.2`
## @trigger.dev/core@4.5.0-rc.2
</details>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
## Summary
Multi-step reasoning agents with HITL tools (OpenAI Responses with
`store: false`, Anthropic extended thinking, etc.) failed on
`chat.addToolOutput(...)` continuations — either the wire payload blew
the `.in/append` cap (reasoning blobs + tool inputs routinely > 512
KiB), or app-side slimming workarounds got overwritten server-side and
the next LLM call landed a tool call with no `arguments`. Both modes are
fixed.
## Design
The per-turn merge in `chat.agent` now overlays only the tool-part state
advances (`output-available` / `output-error` / `approval-responded` /
`output-denied`) from the wire copy onto the hydrated/snapshot chain.
Previously it replaced the entire message, which dropped `input`,
reasoning, and text from the LLM's view whenever the wire was slim.
In parallel, `TriggerChatTransport.sendMessages` and `AgentChat.sendRaw`
now slim the assistant message themselves on `submit-message`
continuations: ship `{ id, role, parts: [<resolved tool part only>] }`,
everything else reconstructed server-side from `hydrateMessages` or the
durable snapshot. Continuation payloads drop from 600 KiB – 1 MiB to ~1
KiB.
`references/ai-chat` `aiChatHydrated.hydrateMessages` now upserts by id
instead of pushing. With slim continuations, a blind push duplicates the
assistant id in the returned chain — the merge updates the first match,
the slim duplicate goes straight to `toModelMessages` with no `input`,
and the LLM 4xx's. This is the canonical pattern customers should mirror
in their own hydrate implementations.
## Test plan
- 11 new tests (slim helper unit + slim+merge integration for HITL,
approval, default no-hydrate branch)
- Full SDK suite: 239 tests pass across 19 files
- End-to-end sweep against `references/ai-chat`: 19 customer-side smoke
tests green; HITL wire bodies confirmed at ~1 KiB (was 600 KiB+); no
provider 4xx errors across OpenAI Responses or Anthropic
## Summary
Two improvements to session `.in/append`:
- Oversize-body 413 responses now carry CORS headers, so browser fetches
see a readable status instead of an opaque `TypeError: Failed to fetch`.
App-side retry-on-disconnect loops no longer spin forever on a
permanently-rejected payload.
- The per-record cap is now computed precisely against S2's actual
ceiling instead of the conservative 512 KiB floor. Legitimate ~600-900
KiB tool outputs (search results, file content) now succeed;
pathological all-quote content that would double under JSON escape still
rejects cleanly.
## Design
S2 enforces a per-record metered size of `8 + 2*H + Σ(header name +
value) + body ≤ 1048576` bytes. With no record headers (our case), the
budget reduces to `body ≤ 1048568`. Verified empirically against cloud
S2 — append succeeds at metered=1048576 and 422s at 1048577 with `record
must have metered size less than 1 MiB`.
The old `MAX_APPEND_BODY_BYTES = 512 KiB` was derived by assuming
worst-case JSON escape doubling (every byte becomes `\"` or `\\`),
giving `(1 MiB - overhead) / 2`. Safe, but rejects ~half the legitimate
input space.
The new flow:
1. Pre-cap the HTTP body at 1 MiB (DoS guard against reading arbitrary
garbage before we can compute the wrap).
2. After reading, `S2RealtimeStreams.#appendPartByName` computes
`Buffer.byteLength(JSON.stringify({data: part, id: partId}), "utf8") +
8` and throws `S2RecordTooLargeError` (a `ServiceValidationError` with
status 413) if it would exceed S2's ceiling. The route's existing error
branch maps the throw to a 413 with a descriptive message.
The 413 CORS fix is a single-line change in `apiBuilder.server.ts` —
`wrapResponse` was being skipped on the body-too-large branch; every
other error branch wraps; the 413 was the exception.
## Test plan
- Empirically verified against cloud S2 with a boundary scan across
`[1048568, 1048569, ..., 1048576]` and across H ∈ {0, 1×5 hdr bytes,
1×14 hdr bytes} — the formula matches exactly
- Browser-side fetch on a 700 KiB POST now resolves with a readable
`status: 413` (no `TypeError: Failed to fetch`)
- A 900 KiB ASCII tool output now passes (would have 413'd at 512 KiB
pre-fix)
## Bug
The `OrganizationDataStoresRegistry` singleton in
`apps/webapp/app/services/dataStores/organizationDataStoresRegistryInstance.server.ts`
was constructed with `$replica`. That client was then used by both the
polling read path *and* by `addDataStore` / `updateDataStore` /
`deleteDataStore` (and their backing `SecretStore.setSecret` upserts).
The write methods route through the read replica, which Postgres rejects
with **error code 25006**:
```
Invalid prisma.secretStore.upsert() invocation:
ConnectorError(ConnectorError { user_facing_error: None, kind: QueryError(PostgresError {
code: "25006",
message: "cannot execute INSERT in a read-only transaction",
...
}), transient: false })
```
User-visible symptom: the admin `/admin/data-stores` "Add data store"
form returns a 400 with this error wrapped, so no
`OrganizationDataStore` row can ever be created via the UI.
The read path (`loadFromDatabase` polling + `SecretStore.getSecret`) is
unaffected because `findMany` + secret read are read-only.
## Fix
Change the registry constructor to take both a writer and a replica:
```ts
constructor(writer: PrismaClient, replica: PrismaClient | PrismaReplicaClient)
```
- `loadFromDatabase()` keeps using `_replica` (and its
`SecretStore.getSecret` calls) — these are background cache-fillers, not
on user-latency-sensitive paths.
- `addDataStore` / `updateDataStore` / `deleteDataStore` (and their
`SecretStore.setSecret` / `deleteSecret` calls) now use `_writer`.
`organizationDataStoresRegistryInstance.server.ts` passes `(prisma,
$replica)` from `~/db.server`. Test sites that constructed with
`(prisma)` now pass `(prisma, prisma)` — the testcontainer exposes a
single client, so the writer/replica split collapses to one connection.
## Files
-
`apps/webapp/app/services/dataStores/organizationDataStoresRegistry.server.ts`
— constructor + read/write split
-
`apps/webapp/app/services/dataStores/organizationDataStoresRegistryInstance.server.ts`
— pass `prisma` alongside `$replica`
- `apps/webapp/test/organizationDataStoresRegistry.test.ts` — 14 call
sites bumped
- `apps/webapp/test/clickhouseFactory.test.ts` — 5 call sites bumped
## Test plan
- [x] Existing `organizationDataStoresRegistry.test.ts` +
`clickhouseFactory.test.ts` still pass (constructor sites updated;
behavior unchanged for tests).
- [ ] After deploy to test cloud, retry `/admin/data-stores` "Add data
store" form for the HIPAA org — should now succeed and the row should
appear.
- [ ] Verify the registry's polling reload picks up the new row within
`ORGANIZATION_DATA_STORES_RELOAD_INTERVAL_MS` (60s default) and the
factory starts routing to the org-scoped instance.
## Summary
When a background worker registers, the engine resolves runs that were
queued before the worker was ready (status `PENDING_VERSION`). That
lookup used to scan a Postgres status index on `TaskRun`. Move it to
ClickHouse: query candidate run ids from `task_runs_v2`, then refetch
the actual rows from Postgres by primary key with a `status =
'PENDING_VERSION'` guard for idempotency.
## Design
The lookup is a pluggable interface on the run engine
(`PendingVersionRunIdLookup`). The webapp wires a ClickHouse-backed
implementation through the org-scoped `clickhouseFactory` using a new
`"engine"` client type, configured by `RUN_ENGINE_CLICKHOUSE_*` env
vars. The URL falls back to `CLICKHOUSE_URL` when unset, so self-hosted
deployments don't need new config to keep working.
When the lookup returns no candidates, one bounded retry is scheduled
~5s later to cover ClickHouse replication lag against `task_runs_v2`.
The Postgres status guard on both the candidate refetch and the inner
`updateMany` prevents double-promotion when a retry races with a
concurrent deploy.
Tests cover three existing PENDING_VERSION cases via a small
Postgres-backed test adapter; new ClickHouse-backed integration tests
will follow.
## Summary
Drops the unused composite Postgres index
`TaskRun_scheduleId_createdAt_idx`. The schedule list view reads from
ClickHouse, so this index served no Prisma query while still being
maintained on every `TaskRun` INSERT/UPDATE. Removing it reduces write
amplification on the primary database.
Sibling to the prior drop of `TaskRun_scheduleId_idx` and the earlier
removal of the `TaskRun.scheduleId` foreign key — all stemming from
migrating schedule-aware reads to ClickHouse.
## Verification
- Sampled `pg_stat_user_indexes` for `TaskRun` over multiple hours —
zero scans against this index.
- Grepped the codebase for any Prisma query filtering
`TaskRun.scheduleId` — none found. All schedule-aware listing routes
through `clickhouseRunsRepository`.
Make the Express server's `keepAliveTimeout` configurable via
`HTTP_KEEPALIVE_TIMEOUT_MS`. Default preserved at 65000 ms — no behavior
change if unset.
Workloads bundled with CLI versions before v4.4.4 use a strict zod enum
for `checkpoint.type` that only allows DOCKER and KUBERNETES. When a
customer's runs are routed via the compute path, those old runners
receive `type: "COMPUTE"` on `/snapshots/since/...` and `/dequeue`
responses and fail validation - blocking silent migration of existing
deployments.
The workload never reads the field - only validates the shape. Rewriting
COMPUTE -> KUBERNETES on the way out lets older runners keep parsing
while the database and internal services keep the real value. Limited to
the two workload-facing endpoints whose response includes a checkpoint;
`/continue`, `/attempts/start`, `/attempts/complete` all return shapes
without one.
Followup to #3114.
## Summary
Stamp every Sentry event with the signed-in user and the tenant (org /
project / env) the request belongs to, so "Users Impacted" counts
distinct humans and events become filterable per tenant.
**Design after review (current):**
- `user.id = real user cuid` (from `requireUser`). "Users Impacted"
counts humans, not tenants.
- Tenant context (org / project / env slugs, IDs, env type) moves
entirely onto tags: `org_slug`, `project_slug`, `env_slug`, `org_id`,
`project_id`, `project_ref`, `environment_id`, `env_type`, plus
`impersonating` when set.
- Backed by an `AsyncLocalStorage` scope established at the HTTP entry.
Each entry point fills what it knows; loaders enrich the same scope with
what they already have.
**Zero new database queries.** The middleware does a regex match only.
Dashboard loaders that already query Prisma gain a couple of extra
selected columns; nothing new round-trips.
## How it's wired
- **Express middleware (`tenantContextResolver.server.ts`)** — parses
the URL with a regex and always opens an ALS scope. Populates whatever
subset of slugs is present: `/orgs/:o` → just `orgSlug`;
`/orgs/:o/projects/:p` adds `projectSlug`; the full triple adds
`envSlug`. Non-tenant paths get an empty scope so loaders can still
enrich.
- **`_app/route.tsx`** — already calls `requireUser`. Adds
`tenantContext.enrich({ userId: user.id })` for every authenticated
dashboard request. No new query.
- **Env layout loader (`_app.orgs.$o.projects.$p.env.$e/route.tsx`)** —
its existing `prisma.project.findFirst` gains two columns in `select`
(`externalRef`, `organization.id`). After it picks an env, calls
`tenantContext.enrich({ orgId, projectId, projectRef, envId, envType
})`. Same query, +2 columns.
- **API path (`apiBuilder.server.ts`)** — wraps every handler in
`tenantContext.run(tenantContextFromAuthEnvironment(authenticationResult.environment),
…)`. The mapper pulls `userId` from `env.orgMember?.userId` (already
selected by `authIncludeBase` — no schema change). Covers
`createLoaderApiRoute`, `createActionApiRoute`, and
`createMultiMethodApiRoute`.
- **Event processor (`sentryTenantContext.server.ts`)** — registered in
`entry.server.tsx` so it lives in the Remix bundle and shares the same
`tenantContext` ALS instance as the middleware and loaders. Stamps
whatever's present; nothing forced.
## Example events from local verification
| URL | `user.id` | Tags |
|-----|-----------|------|
| `/orgs/:o/projects/:p/env/:e/...` | real user cuid | `org_slug`,
`project_slug`, `env_slug`, `org_id`, `project_id`, `project_ref`,
`environment_id`, `env_type` |
| `/orgs/:o/settings` (non-env-scoped) | real user cuid | `org_slug`
only |
| API request with `orgMember` | `orgMember.userId` | full tenant set |
| API request without `orgMember` | (unset) | full tenant set |
## Trade-offs
1. On env-scoped pages, errors that fire before the env layout loader's
enrich callback runs get slugs + `user.id` but not the tenant IDs /
`env_type`. Realistic errors deep in async work get the full set. (Same
race as before, narrower window now that slugs/`user.id` are populated
up-front by the middleware and `_app` enrich.)
2. API requests where the environment has no `orgMember` get tenant tags
but no `user.id`. Those events still show in the issue but don't
contribute to "Users Impacted".
## Out of scope (deferred)
Background workers (`redis-worker`, `schedule-engine`) and socket
handlers. Those entry points don't set `tenantContext.run` yet — their
events ship without tenant attribution until each is wired in a
follow-up.
## Tests
31 unit tests across 4 files. New tests notably cover:
- `parseTenantPath`: org-only, org+project, and full-triple URL
variants.
- `tenantContext.enrich`: in-place patch, no-op outside `run()`,
concurrent-scope isolation, empty-scope + enrich pattern (for non-tenant
pages).
- `tenantContextFromAuthEnvironment`: with and without `orgMember` —
verifies the API path's `user.id` mapping.
- `addTenantContextToEvent`: empty scope, userId-only, slugs-only, full
enrichment, conditional tag emission, preservation of prior `event.user`
fields.
## Test plan
- [ ] `pnpm run typecheck --filter webapp`
- [ ] `pnpm run test --filter webapp -- test/tenantContext.test.ts
test/sentryTenantContext.test.ts test/tenantContextResolver.test.ts
test/tenantContextFromAuthEnvironment.test.ts`
- [ ] Local manual: with `SENTRY_DSN` set, hit a dashboard URL and an
API route, confirm the captured events carry `user.id` + the expected
tag set in Sentry.
- [ ] After ship: confirm "Users Impacted" on a real Sentry issue
reflects distinct users (not tenants).
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
Added `OrganizationDataStore` which allows orgs to have data stored in
specific separate services.
For now this is just used for ClickHouse. When using ClickHouse we get a
client for the factory and pass in the org id.
Particular care has to be made with two hot-insert paths:
1. RunReplicationService
2. OTLPExporter
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: Claude <noreply@anthropic.com>
## Summary
`new TriggerClient({...})` exposes the management API (tasks, runs,
schedules, envvars, batch, queues, deployments, prompts, auth) as an
explicit instance with its own auth, preview branch, and baseURL.
Multiple clients can coexist in one process without mutating shared
global state — useful when a single service triggers across multiple
projects, environments, or preview branches.
```ts
import { TriggerClient } from "@trigger.dev/sdk";
const prod = new TriggerClient({ accessToken: process.env.TRIGGER_PROD_KEY });
const preview = new TriggerClient({
accessToken: process.env.TRIGGER_PREVIEW_KEY,
previewBranch: "signup-flow",
});
await prod.tasks.trigger("send-email", payload);
await preview.runs.list({ status: ["COMPLETED"] });
```
The existing global `configure()` API keeps working unchanged.
## Design
Instance methods enter an `AsyncLocalStorage`-backed scope (`sdkScope`)
before delegating to the existing module-level functions. The four
"pollution" points that previously read globals now consult the scope
first:
- `apiClientManager.{baseURL, accessToken, branchName}` and
`clientOrThrow` — identity fields are scope-only when scoped; `baseURL`
still falls back to `TRIGGER_API_URL` because plumbing (where the API
lives) is not identity.
- `taskContext.{ctx, worker, isWarmStart, isInsideTask}` — masked inside
an isolated scope so a `client.tasks.trigger(...)` from inside a task
doesn't leak the parent's `parentRunId` / `lockToVersion` / `isTest`
into a trigger that hits a different project.
- Inline `getEnvVar("TRIGGER_VERSION")` reads in `shared.ts` go through
a `scopedEnvVar` helper that returns `undefined` inside an isolated
scope.
The `TriggerClient` class itself is a thin wrapper that captures the
scope in its constructor and proxies each namespace method to enter that
scope before calling the existing impl. Generic inference (e.g.
`client.tasks.trigger<typeof t>(...)`) is preserved via `Pick<typeof ns,
keyof curatedSubset>` typings.
Two correctness fixes uncovered along the way are folded in:
- `apiClientManager.setGlobalAPIClientConfiguration` no longer silently
no-ops on the second call. `configure()` now actually overrides as users
expect (this is the root cause behind some "I changed the config but
nothing happened" reports).
- `apiClientManager.runWithConfig` (and therefore `auth.withAuth`) is
now backed by `sdkScope.withScope` instead of "mutate the global and
restore in finally". Two parallel `withAuth` calls with different
configs no longer stomp each other.
Surface curation: instance namespaces drop methods that don't make sense
per-instance — `batch.*AndWait` (runtime-dependent), `schedules.task` /
`schedules.timezones` (definition-time / stateless), `prompts.define`
(definition-time), `auth.configure` / `auth.withAuth` (global-only).
## Test plan
- [x] 9 runtime unit tests in `triggerClient.test.ts` cover: required
accessToken, instance auth + branch headers, no env fallback for
identity fields, no leakage between global and instance, four parallel
calls across two clients stay isolated, taskContext masking +
`inheritContext: true` override, `configure()` second-call override,
parallel `auth.withAuth` isolation.
- [x] 10 type-level assertions in `triggerClient.types.test.ts` using
`expectTypeOf` + `@ts-expect-error` lock in generic inference, return
type passthrough, overload preservation, and curated-surface drift.
- [x] Full SDK suite (219 tests) and core suite (530 tests) pass.
- [x] Webapp typecheck clean.
- [x] End-to-end smoke test against local webapp and a
freshly-provisioned cloud project — six concurrent multi-client triggers
all returned 200 with run IDs, headers per-client as expected.
- [ ] Reviewer: run `references/multi-client` per its `README.md` to
reproduce the smoke test locally.
## Try it
`references/multi-client` is a new reference workspace that exercises
this end-to-end:
- `src/trigger/echo.ts` — trivial target task
- `src/trigger/fanOut.ts` — opens two `TriggerClient`s from inside a
task, fires `echo` through each in parallel
- `src/external/main.ts` — external Node script with two clients
triggering `echo` sequentially and concurrently; logs every outgoing
request's `authorization` + `x-trigger-branch`
- `src/external/isolation.ts` — interleaves global `configure()` and an
instance call, asserts the captured fetch sequence shows no leakage
either way
Sibling to the weekly summary, focused on critical alerts only. Pings
Slack daily while any critical alerts are open; skips the post entirely
when zero, so no daily "all clear" noise.
- Daily 08:00 UTC cron + `workflow_dispatch` with `severity` input
(default `critical`, override to `high`/`medium`/`low` for manual
checks)
- Reuses the existing `dependabot-summary` environment (token, channel,
bot)
- Alerts link at the end is severity-filtered
## Summary
Two docs edits that close a footgun customers persisting transport state
can hit. Clearing `lastEventId` on `chat.endRun()` looks intuitive — the
Run ended, the cursor must be stale — but the cursor is sessionId-keyed,
not runId-keyed. Clearing it forces the next `sendMessages` to subscribe
from `seq_num=0`, which may hit the prior turn's still-durable
`turn-complete` record and close the SSE empty before the new Run's
chunks arrive.
Spells out the invariant in the frontend transport persistence table and
adds a Warning in the `chat.endRun()` reference.
## Test plan
- [x] Mintlify preview renders
- [x] No callout stacking
## Summary
Three post-merge fixes for the AI Agents docs (#3226), all caught by
review after merge.
## Fixes
- **`onTurnComplete` examples now use `db.$transaction`** — both the
Database persistence "Complete example" and the Lifecycle hooks
reference example were doing two separate `await` calls
(`db.chat.update` then `db.chatSession.upsert`). That's the exact
non-atomic pattern the warning earlier on the persistence page calls out
as ❌: a refresh between the two writes reads a stale `lastEventId` and
duplicates the assistant message on resume. Both examples now use the
recommended atomic form.
- **Background injection self-review prose aligned with the code** — the
prose said "gpt-4o-mini" but the example above it had been swapped to
`claude-haiku-4-5`. The Anthropic-sweep script only touched code blocks;
this prose line wasn't picked up.
## Test plan
- [x] Both updated examples use `db.$transaction([...])`
- [x] Prose matches the model used in the code block
- [ ] Mintlify deployment passes
## Summary
Lands the full AI Agents documentation surface alongside the v4.5
release candidate of `@trigger.dev/sdk`. Covers `chat.agent` end to end
— defining agents, lifecycle hooks, the frontend transport, sub-agents,
recovery from cancel/crash/OOM, AI Prompts integration — and the
Sessions primitive that backs it.
## Coverage
- **Conceptual**: Overview, Quick Start, How it works.
- **Building agents**: Backend (`chat.agent` / `chat.createSession` /
raw primitives), Lifecycle hooks, Frontend transport, Server-side
`AgentChat`, Sessions reference, `chat.local` state primitive,
TypeScript types.
- **Features**: AI Prompts integration, Fast starts (Preload + Head
Start), Compaction, Pending Messages (steering), Background Injection
(`chat.inject` + `chat.defer`), Actions (undo / regenerate / edit),
Error handling.
- **Patterns (13)**: Sub-agents, Branching conversations, Code sandbox,
Database persistence, Persistence and replay, HITL, Tool result
auditing, Large payloads, Agent skills, OOM resilience, Recovery boot,
Trusted edge signals, Version upgrades.
- **Reference**: API Reference, Client Protocol (wire format), Testing
harness (`mockChatAgent`), MCP server tools, Upgrade guide, Changelog.
## Structure changes
- Top-level nav: AI → **Agents**, with sub-groups for *Building agents /
Features / Patterns / Reference*.
- New RC banner snippet on every page links to the supported AI SDK
versions table on the API Reference.
- All examples use Anthropic with `stopWhen: stepCountIs(15)`.
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
## Summary
2 bug fixes.
## Bug fixes
- Fix `chat.agent` skills silently missing in `trigger dev` for projects
whose task files read `process.env` at module top level (e.g. a
third-party SDK client initialized at import). Skill folders now bundle
into `.trigger/skills/` reliably regardless of which env vars are set
when the CLI launches.
([#3690](https://github.com/triggerdotdev/trigger.dev/pull/3690))
- Fix `COULD_NOT_FIND_EXECUTOR` when a task's definition is loaded via
`await import(...)` from inside another task's `run()`. The runtime
workers now register such tasks with a sentinel file context, and the
catalog logs a one-time warning per task id.
([#3688](https://github.com/triggerdotdev/trigger.dev/pull/3688))
<details>
<summary>Raw changeset output</summary>
⚠️⚠️⚠️⚠️⚠️⚠️
`main` is currently in **pre mode** so this branch has prereleases
rather than normal releases. If you want to exit prereleases, run
`changeset pre exit` on `main`.
⚠️⚠️⚠️⚠️⚠️⚠️
# Releases
## @trigger.dev/build@4.5.0-rc.1
### Patch Changes
- Updated dependencies:
- `@trigger.dev/core@4.5.0-rc.1`
## trigger.dev@4.5.0-rc.1
### Patch Changes
- Fix `chat.agent` skills silently missing in `trigger dev` for projects
whose task files read `process.env` at module top level (e.g. a
third-party SDK client initialized at import). Skill folders now bundle
into `.trigger/skills/` reliably regardless of which env vars are set
when the CLI launches.
([#3690](https://github.com/triggerdotdev/trigger.dev/pull/3690))
- Fix `COULD_NOT_FIND_EXECUTOR` when a task's definition is loaded via
`await import(...)` from inside another task's `run()`. The runtime
workers now register such tasks with a sentinel file context, and the
catalog logs a one-time warning per task id.
([#3688](https://github.com/triggerdotdev/trigger.dev/pull/3688))
- Updated dependencies:
- `@trigger.dev/core@4.5.0-rc.1`
- `@trigger.dev/build@4.5.0-rc.1`
- `@trigger.dev/schema-to-json@4.5.0-rc.1`
## @trigger.dev/core@4.5.0-rc.1
### Patch Changes
- Fix `COULD_NOT_FIND_EXECUTOR` when a task's definition is loaded via
`await import(...)` from inside another task's `run()`. The runtime
workers now register such tasks with a sentinel file context, and the
catalog logs a one-time warning per task id.
([#3688](https://github.com/triggerdotdev/trigger.dev/pull/3688))
## @trigger.dev/plugins@4.5.0-rc.1
### Patch Changes
- Updated dependencies:
- `@trigger.dev/core@4.5.0-rc.1`
## @trigger.dev/python@4.5.0-rc.1
### Patch Changes
- Updated dependencies:
- `@trigger.dev/core@4.5.0-rc.1`
- `@trigger.dev/build@4.5.0-rc.1`
- `@trigger.dev/sdk@4.5.0-rc.1`
## @trigger.dev/react-hooks@4.5.0-rc.1
### Patch Changes
- Updated dependencies:
- `@trigger.dev/core@4.5.0-rc.1`
## @trigger.dev/redis-worker@4.5.0-rc.1
### Patch Changes
- Updated dependencies:
- `@trigger.dev/core@4.5.0-rc.1`
## @trigger.dev/rsc@4.5.0-rc.1
### Patch Changes
- Updated dependencies:
- `@trigger.dev/core@4.5.0-rc.1`
## @trigger.dev/schema-to-json@4.5.0-rc.1
### Patch Changes
- Updated dependencies:
- `@trigger.dev/core@4.5.0-rc.1`
## @trigger.dev/sdk@4.5.0-rc.1
### Patch Changes
- Updated dependencies:
- `@trigger.dev/core@4.5.0-rc.1`
</details>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
## Summary
`trigger.dev dev` was silently dropping registered `chat.agent` skills
for any project whose task files read `process.env` at module top level
— e.g. a third-party SDK client initialized at import. The agent would
boot fine, but `skill.local()` failed at runtime with `ENOENT` because
the skill folder was never copied into `.trigger/skills/`.
## Design
The CLI ran two indexer passes in dev: the worker's own indexer (with
the full env it eventually executes tasks in), and a separate
skill-discovery indexer with only the CLI process's env. Top-level reads
of vars like `TRIGGER_API_URL` imported cleanly in the worker pass and
threw in the skill pass — the latter caught the error, warned, and
skipped skill copying. Failure was silent enough that `skill.local()`
only surfaced it at task runtime.
The skill registry is already part of the worker manifest. This PR drops
the duplicate pass and copies skill folders from that manifest after the
worker initializes. One indexer instead of two; a bad `SKILL.md` now
surfaces as a startup error instead of silently disappearing skills.
Deploy is unaffected — its skill discovery uses the project's
environment variables (fetched via the API, which fills in
`TRIGGER_API_URL` etc.), so the dev failure mode doesn't reach there.
## Test plan
- [x] New `references/agent-skills` reference project with
`skills.define` + a task that calls `skill.local()` and runs a bundled
script
- [x] On `main`, adding a top-level
`process.env.TRIGGER_API_URL!.includes(...)` read in any task file
reproduces the symptom: warning at dev startup, no `.trigger/skills/`
folder, `skill.local()` fails with ENOENT
- [x] On this branch, same project boots clean and `skill.local()` works
end-to-end
- [x] Deploy still works end-to-end with the new reference project
On a warm worker process, a task whose `task()` definition is loaded via
`await import(...)` from inside another task's `run()` could end up
permanently missing from the catalog: the `task()` call fired with no
`_currentFileContext` set, `registerTaskMetadata` silently returned, and
Node's ESM module cache then blocked the worker's setContext + re-import
recovery from ever firing the call again. Subsequent runs of that task
on the same warm process failed with `COULD_NOT_FIND_EXECUTOR` until the
process hit `maxExecutionsPerProcess` and exited.
All five of these had to coincide on the same worker for the bug to
surface:
1. `processKeepAlive` enabled (so catalog state survives across runs).
2. A `run()` function (or lifecycle hook) does `await import(...)`.
3. The import's transitive static graph reaches a `task()` /
`schemaTask()` call.
4. The task containing the dynamic import is the **first** task to run
on a given warm worker process — so the dropped `task()` calls fire on
this process for the first time, are silently dropped, and Node's module
cache locks the wrong outcome in.
5. A subsequent run for one of the dropped task ids lands on the same
warm worker before it recycles.
The runtime workers now set a sentinel file context (`<no-context>`)
around the `executor.execute(...)` call, so `task()` invocations firing
during a run register normally. The catalog detects the sentinel and
emits a one-time `console.warn` per task id so the pattern stays visible
without spamming. The indexer never sets this context, so deploy-time
behavior is unchanged.
Repro is `references/hello-world/src/trigger/dynamicImportRepro*.ts`.
Verified end-to-end against a deployed image with firestarter
warm-starts on: pre-fix saw `COULD_NOT_FIND_EXECUTOR` on children that
landed on the parent-poisoned worker; post-fix all 23/23 runs succeeded
and the warning surfaces in the parent's run trace.
## Summary
44 improvements, 1 bug fix.
## Improvements
- **AI Prompts** — define prompt templates as code alongside your tasks,
version them on deploy, and override the text or model from the
dashboard without redeploying. Prompts integrate with the Vercel AI SDK
via `toAISDKTelemetry()` (links every generation span back to the
prompt) and with `chat.agent` via `chat.prompt.set()` +
`chat.toStreamTextOptions()`.
([#3629](https://github.com/triggerdotdev/trigger.dev/pull/3629))
- **Code-defined, deploy-versioned templates** — define with
`prompts.define({ id, model, config, variables, content })`. Every
deploy creates a new version visible in the dashboard. Mustache-style
placeholders (`{{var}}`, `{{#cond}}...{{/cond}}`) with Zod / ArkType /
Valibot-typed variables.
- **Dashboard overrides** — change a prompt's text or model from the
dashboard without redeploying. Overrides take priority over the deployed
"current" version and are environment-scoped (dev / staging / production
independent).
- **Resolve API** — `prompt.resolve(vars, { version?, label? })` returns
the compiled `text`, resolved `model`, `version`, and labels. Standalone
`prompts.resolve<typeof handle>(slug, vars)` for cross-file resolution
with full type inference on slug and variable shape.
- **AI SDK integration** — spread `resolved.toAISDKTelemetry({ ...extra
})` into any `generateText` / `streamText` call and every generation
span links to the prompt in the dashboard alongside its input variables,
model, tokens, and cost.
- **`chat.agent` integration** — `chat.prompt.set(resolved)` stores the
resolved prompt run-scoped; `chat.toStreamTextOptions({ registry })`
pulls `system`, `model` (resolved via the AI SDK provider registry),
`temperature` / `maxTokens` / etc., and telemetry into a single spread
for `streamText`.
- **Management SDK** — `prompts.list()`, `prompts.versions(slug)`,
`prompts.promote(slug, version)`, `prompts.createOverride(slug, body)`,
`prompts.updateOverride(slug, body)`, `prompts.removeOverride(slug)`,
`prompts.reactivateOverride(slug, version)`.
- **Dashboard** — prompts list with per-prompt usage sparklines;
per-prompt detail with Template / Details / Versions / Generations /
Metrics tabs. AI generation spans get a custom inspector showing the
linked prompt's metadata, input variables, and template content
alongside model, tokens, cost, and the message thread.
- Adds `onBoot` to `chat.agent` — a lifecycle hook that fires once per
worker process picking up the chat. Runs for the initial run, preloaded
runs, AND reactive continuation runs (post-cancel, crash, `endRun`,
`requestUpgrade`, OOM retry), before any other hook. Use it to
initialize `chat.local`, open per-process resources, or re-hydrate state
from your DB on continuation — anywhere the SAME run picking up after
suspend/resume isn't enough.
([#3543](https://github.com/triggerdotdev/trigger.dev/pull/3543))
- **AI SDK `useChat` integration** — a custom
[`ChatTransport`](https://sdk.vercel.ai/docs/ai-sdk-ui/transport)
(`useTriggerChatTransport`) plugs straight into Vercel AI SDK's
`useChat` hook. Text streaming, tool calls, reasoning, and `data-*`
parts all work natively over Trigger.dev's realtime streams. No custom
API routes needed.
- **First-turn fast path (`chat.headStart`)** — opt-in handler that runs
the first turn's `streamText` step in your warm server process while the
agent run boots in parallel, cutting cold-start TTFC by roughly half
(measured 2801ms → 1218ms on `claude-sonnet-4-6`). The agent owns step
2+ (tool execution, persistence, hooks) so heavy deps stay where they
belong. Web Fetch handler works natively in Next.js, Hono, SvelteKit,
Remix, Workers, etc.; bridge to Express/Fastify/Koa via
`chat.toNodeListener`. New `@trigger.dev/sdk/chat-server` subpath.
- **Multi-turn durability via Sessions** — every chat is backed by a
durable Session that outlives any individual run. Conversations resume
across page refreshes, idle timeout, crashes, and deploys; `resume:
true` reconnects via `lastEventId` so clients only see new chunks.
`sessions.list` enumerates chats for inbox-style UIs.
- **Auto-accumulated history, delta-only wire** — the backend
accumulates the full conversation across turns; clients only ship the
new message each turn. Long chats never hit the 512 KiB body cap.
Register `hydrateMessages` to be the source of truth yourself.
- **Lifecycle hooks** — `onPreload`, `onChatStart`,
`onValidateMessages`, `hydrateMessages`, `onTurnStart`,
`onBeforeTurnComplete`, `onTurnComplete`, `onChatSuspend`,
`onChatResume` — for persistence, validation, and post-turn work.
- **Stop generation** — client-driven `transport.stopGeneration(chatId)`
aborts mid-stream; the run stays alive for the next message, partial
response is captured, and aborted parts (stuck `partial-call` tools,
in-progress reasoning) are auto-cleaned.
- **Tool approvals (HITL)** — tools with `needsApproval: true` pause
until the user approves or denies via `addToolApprovalResponse`. The
runtime reconciles the updated assistant message by ID and continues
`streamText`.
- **Steering and background injection** — `pendingMessages` injects user
messages between tool-call steps so users can steer the agent
mid-execution; `chat.inject()` + `chat.defer()` adds context from
background work (self-review, RAG, safety checks) between turns.
- **Actions** — non-turn frontend commands (undo, rollback, regenerate,
edit) sent via `transport.sendAction`. Fire `hydrateMessages` +
`onAction` only — no turn hooks, no `run()`. `onAction` can return a
`StreamTextResult` for a model response, or `void` for side-effect-only.
- **Typed state primitives** — `chat.local<T>` for per-run state
accessible from hooks, `run()`, tools, and subtasks (auto-serialized
through `ai.toolExecute`); `chat.store` for typed shared data between
agent and client; `chat.history` for reading and mutating the message
chain; `clientDataSchema` for typed `clientData` in every hook.
- **`chat.toStreamTextOptions()`** — one spread into `streamText` wires
up versioned system [Prompts](https://trigger.dev/docs/ai/prompts),
model resolution, telemetry metadata, compaction, steering, and
background injection.
- **Multi-tab coordination** — `multiTab: true` + `useMultiTabChat`
prevents duplicate sends and syncs state across browser tabs via
`BroadcastChannel`. Non-active tabs go read-only with live updates.
- **Network resilience** — built-in indefinite retry with bounded
backoff, reconnect on `online` / tab refocus / bfcache restore,
`Last-Event-ID` mid-stream resume. No app code needed.
- **Sessions** — a durable, run-aware stream channel keyed on a stable
`externalId`. A Session is the unit of state that owns a multi-run
conversation: messages flow through `.in`, responses through `.out`,
both survive run boundaries. Sessions back the new `chat.agent` runtime,
and you can build on them directly for any pattern that needs durable
bi-directional streaming across runs.
([#3542](https://github.com/triggerdotdev/trigger.dev/pull/3542))
- Add `ai.toolExecute(task)` so you can wire a Trigger subtask in as the
`execute` handler of an AI SDK `tool()` while defining `description` and
`inputSchema` yourself — useful when you want full control over the tool
surface and just need Trigger's subtask machinery for the body.
([#3546](https://github.com/triggerdotdev/trigger.dev/pull/3546))
- Type `chat.createStartSessionAction` against your chat agent so
`clientData` is typed end-to-end on the first turn:
([#3684](https://github.com/triggerdotdev/trigger.dev/pull/3684))
- Add `region` to the runs list / retrieve API: filter runs by region
(`runs.list({ region: "..." })` / `filter[region]=<masterQueue>`) and
read each run's executing region from the new `region` field on the
response.
([#3612](https://github.com/triggerdotdev/trigger.dev/pull/3612))
- Add `TRIGGER_BUILD_SKIP_REWRITE_TIMESTAMP=1` escape hatch for local
self-hosted builds whose buildx driver doesn't support
`rewrite-timestamp` alongside push (e.g. orbstack's default `docker`
driver).
([#3618](https://github.com/triggerdotdev/trigger.dev/pull/3618))
- Reject overlong `idempotencyKey` values at the API boundary so they no
longer trip an internal size limit on the underlying unique index and
surface as a generic 500. Inputs are capped at 2048 characters — well
above what `idempotencyKeys.create()` produces (a 64-character hash) and
above any realistic raw key. Applies to `tasks.trigger`,
`tasks.batchTrigger`, `batch.create` (Phase 1 streaming batches),
`wait.createToken`, `wait.forDuration`, and the input/session stream
waitpoint endpoints. Over-limit requests now return a structured 400
instead.
([#3560](https://github.com/triggerdotdev/trigger.dev/pull/3560))
- **AI SDK `useChat` integration** — a custom
[`ChatTransport`](https://sdk.vercel.ai/docs/ai-sdk-ui/transport)
(`useTriggerChatTransport`) plugs straight into Vercel AI SDK's
`useChat` hook. Text streaming, tool calls, reasoning, and `data-*`
parts all work natively over Trigger.dev's realtime streams. No custom
API routes needed.
- **First-turn fast path (`chat.headStart`)** — opt-in handler that runs
the first turn's `streamText` step in your warm server process while the
agent run boots in parallel, cutting cold-start TTFC by roughly half
(measured 2801ms → 1218ms on `claude-sonnet-4-6`). The agent owns step
2+ (tool execution, persistence, hooks) so heavy deps stay where they
belong. Web Fetch handler works natively in Next.js, Hono, SvelteKit,
Remix, Workers, etc.; bridge to Express/Fastify/Koa via
`chat.toNodeListener`. New `@trigger.dev/sdk/chat-server` subpath.
- **Multi-turn durability via Sessions** — every chat is backed by a
durable Session that outlives any individual run. Conversations resume
across page refreshes, idle timeout, crashes, and deploys; `resume:
true` reconnects via `lastEventId` so clients only see new chunks.
`sessions.list` enumerates chats for inbox-style UIs.
- **Auto-accumulated history, delta-only wire** — the backend
accumulates the full conversation across turns; clients only ship the
new message each turn. Long chats never hit the 512 KiB body cap.
Register `hydrateMessages` to be the source of truth yourself.
- **Lifecycle hooks** — `onPreload`, `onChatStart`,
`onValidateMessages`, `hydrateMessages`, `onTurnStart`,
`onBeforeTurnComplete`, `onTurnComplete`, `onChatSuspend`,
`onChatResume` — for persistence, validation, and post-turn work.
- **Stop generation** — client-driven `transport.stopGeneration(chatId)`
aborts mid-stream; the run stays alive for the next message, partial
response is captured, and aborted parts (stuck `partial-call` tools,
in-progress reasoning) are auto-cleaned.
- **Tool approvals (HITL)** — tools with `needsApproval: true` pause
until the user approves or denies via `addToolApprovalResponse`. The
runtime reconciles the updated assistant message by ID and continues
`streamText`.
- **Steering and background injection** — `pendingMessages` injects user
messages between tool-call steps so users can steer the agent
mid-execution; `chat.inject()` + `chat.defer()` adds context from
background work (self-review, RAG, safety checks) between turns.
- **Actions** — non-turn frontend commands (undo, rollback, regenerate,
edit) sent via `transport.sendAction`. Fire `hydrateMessages` +
`onAction` only — no turn hooks, no `run()`. `onAction` can return a
`StreamTextResult` for a model response, or `void` for side-effect-only.
- **Typed state primitives** — `chat.local<T>` for per-run state
accessible from hooks, `run()`, tools, and subtasks (auto-serialized
through `ai.toolExecute`); `chat.store` for typed shared data between
agent and client; `chat.history` for reading and mutating the message
chain; `clientDataSchema` for typed `clientData` in every hook.
- **`chat.toStreamTextOptions()`** — one spread into `streamText` wires
up versioned system [Prompts](https://trigger.dev/docs/ai/prompts),
model resolution, telemetry metadata, compaction, steering, and
background injection.
- **Multi-tab coordination** — `multiTab: true` + `useMultiTabChat`
prevents duplicate sends and syncs state across browser tabs via
`BroadcastChannel`. Non-active tabs go read-only with live updates.
- **Network resilience** — built-in indefinite retry with bounded
backoff, reconnect on `online` / tab refocus / bfcache restore,
`Last-Event-ID` mid-stream resume. No app code needed.
- Retry `TASK_PROCESS_SIGSEGV` task crashes under the user's retry
policy instead of failing the run on the first segfault. SIGSEGV in Node
tasks is frequently non-deterministic (native addon races, JIT/GC
interaction, near-OOM in native code, host issues), so retrying on a
fresh process often succeeds. The retry is gated by the task's existing
`retry` config + `maxAttempts` — same path `TASK_PROCESS_SIGTERM` and
uncaught exceptions already use — so tasks without a retry policy still
fail fast.
([#3552](https://github.com/triggerdotdev/trigger.dev/pull/3552))
- The public interfaces for a plugin system. Initially consolidated
authentication and authorization interfaces.
([#3499](https://github.com/triggerdotdev/trigger.dev/pull/3499))
- Add MollifierBuffer and MollifierDrainer primitives for trigger burst
smoothing.
([#3614](https://github.com/triggerdotdev/trigger.dev/pull/3614))
## Bug fixes
- Fix `LocalsKey<T>` type incompatibility across dual-package builds.
The phantom value-type brand no longer uses a module-level `unique
symbol`, so a single TypeScript compilation that resolves the type from
both the ESM and CJS outputs (which can happen under certain pnpm
hoisting layouts) no longer sees two structurally-incompatible variants
of the same type.
([#3626](https://github.com/triggerdotdev/trigger.dev/pull/3626))
<details>
<summary>Raw changeset output</summary>
⚠️⚠️⚠️⚠️⚠️⚠️
`main` is currently in **pre mode** so this branch has prereleases
rather than normal releases. If you want to exit prereleases, run
`changeset pre exit` on `main`.
⚠️⚠️⚠️⚠️⚠️⚠️
# Releases
## @trigger.dev/sdk@4.5.0-rc.0
### Minor Changes
- **AI Prompts** — define prompt templates as code alongside your tasks,
version them on deploy, and override the text or model from the
dashboard without redeploying. Prompts integrate with the Vercel AI SDK
via `toAISDKTelemetry()` (links every generation span back to the
prompt) and with `chat.agent` via `chat.prompt.set()` +
`chat.toStreamTextOptions()`.
([#3629](https://github.com/triggerdotdev/trigger.dev/pull/3629))
```ts
import { prompts } from "@trigger.dev/sdk";
import { generateText } from "ai";
import { openai } from "@ai-sdk/openai";
import { z } from "zod";
export const supportPrompt = prompts.define({
id: "customer-support",
model: "gpt-4o",
config: { temperature: 0.7 },
variables: z.object({
customerName: z.string(),
plan: z.string(),
issue: z.string(),
}),
content: `You are a support agent for Acme.
Customer: {{customerName}} ({{plan}} plan)
Issue: {{issue}}`,
});
const resolved = await supportPrompt.resolve({
customerName: "Alice",
plan: "Pro",
issue: "Can't access billing",
});
const result = await generateText({
model: openai(resolved.model ?? "gpt-4o"),
system: resolved.text,
prompt: "Can't access billing",
...resolved.toAISDKTelemetry(),
});
```
**What you get:**
- **Code-defined, deploy-versioned templates** — define with
`prompts.define({ id, model, config, variables, content })`. Every
deploy creates a new version visible in the dashboard. Mustache-style
placeholders (`{{var}}`, `{{#cond}}...{{/cond}}`) with Zod / ArkType /
Valibot-typed variables.
- **Dashboard overrides** — change a prompt's text or model from the
dashboard without redeploying. Overrides take priority over the deployed
"current" version and are environment-scoped (dev / staging / production
independent).
- **Resolve API** — `prompt.resolve(vars, { version?, label? })` returns
the compiled `text`, resolved `model`, `version`, and labels. Standalone
`prompts.resolve<typeof handle>(slug, vars)` for cross-file resolution
with full type inference on slug and variable shape.
- **AI SDK integration** — spread `resolved.toAISDKTelemetry({ ...extra
})` into any `generateText` / `streamText` call and every generation
span links to the prompt in the dashboard alongside its input variables,
model, tokens, and cost.
- **`chat.agent` integration** — `chat.prompt.set(resolved)` stores the
resolved prompt run-scoped; `chat.toStreamTextOptions({ registry })`
pulls `system`, `model` (resolved via the AI SDK provider registry),
`temperature` / `maxTokens` / etc., and telemetry into a single spread
for `streamText`.
- **Management SDK** — `prompts.list()`, `prompts.versions(slug)`,
`prompts.promote(slug, version)`, `prompts.createOverride(slug, body)`,
`prompts.updateOverride(slug, body)`, `prompts.removeOverride(slug)`,
`prompts.reactivateOverride(slug, version)`.
- **Dashboard** — prompts list with per-prompt usage sparklines;
per-prompt detail with Template / Details / Versions / Generations /
Metrics tabs. AI generation spans get a custom inspector showing the
linked prompt's metadata, input variables, and template content
alongside model, tokens, cost, and the message thread.
See [/docs/ai/prompts](https://trigger.dev/docs/ai/prompts) for the full
reference — template syntax, version resolution order, override
workflow, and type utilities (`PromptHandle`, `PromptIdentifier`,
`PromptVariables`).
- Adds `onBoot` to `chat.agent` — a lifecycle hook that fires once per
worker process picking up the chat. Runs for the initial run, preloaded
runs, AND reactive continuation runs (post-cancel, crash, `endRun`,
`requestUpgrade`, OOM retry), before any other hook. Use it to
initialize `chat.local`, open per-process resources, or re-hydrate state
from your DB on continuation — anywhere the SAME run picking up after
suspend/resume isn't enough.
([#3543](https://github.com/triggerdotdev/trigger.dev/pull/3543))
```ts
const userContext = chat.local<{ name: string; plan: string }>({ id:
"userContext" });
export const myChat = chat.agent({
id: "my-chat",
onBoot: async ({ clientData, continuation }) => {
const user = await db.user.findUnique({ where: { id: clientData.userId }
});
userContext.init({ name: user.name, plan: user.plan });
},
run: async ({ messages, signal }) =>
streamText({ model: openai("gpt-4o"), messages, abortSignal: signal }),
});
```
Use `onBoot` (not `onChatStart`) for state setup that must run every
time a worker picks up the chat — `onChatStart` fires once per chat and
won't run on continuation, leaving `chat.local` uninitialized when
`run()` tries to use it.
- **AI Agents** — run AI SDK chat completions as durable Trigger.dev
agents instead of fragile API routes. Define an agent in one function,
point `useChat` at it from React, and the conversation survives page
refreshes, network blips, and process restarts.
([#3543](https://github.com/triggerdotdev/trigger.dev/pull/3543))
```ts
import { chat } from "@trigger.dev/sdk/ai";
import { streamText } from "ai";
import { openai } from "@ai-sdk/openai";
export const myChat = chat.agent({
id: "my-chat",
run: async ({ messages, signal }) =>
streamText({ model: openai("gpt-4o"), messages, abortSignal: signal }),
});
```
```tsx
import { useChat } from "@ai-sdk/react";
import { useTriggerChatTransport } from "@trigger.dev/sdk/chat/react";
const transport = useTriggerChatTransport({ task: "my-chat",
accessToken, startSession });
const { messages, sendMessage } = useChat({ transport });
```
**What you get:**
- **AI SDK `useChat` integration** — a custom
[`ChatTransport`](https://sdk.vercel.ai/docs/ai-sdk-ui/transport)
(`useTriggerChatTransport`) plugs straight into Vercel AI SDK's
`useChat` hook. Text streaming, tool calls, reasoning, and `data-*`
parts all work natively over Trigger.dev's realtime streams. No custom
API routes needed.
- **First-turn fast path (`chat.headStart`)** — opt-in handler that runs
the first turn's `streamText` step in your warm server process while the
agent run boots in parallel, cutting cold-start TTFC by roughly half
(measured 2801ms → 1218ms on `claude-sonnet-4-6`). The agent owns step
2+ (tool execution, persistence, hooks) so heavy deps stay where they
belong. Web Fetch handler works natively in Next.js, Hono, SvelteKit,
Remix, Workers, etc.; bridge to Express/Fastify/Koa via
`chat.toNodeListener`. New `@trigger.dev/sdk/chat-server` subpath.
- **Multi-turn durability via Sessions** — every chat is backed by a
durable Session that outlives any individual run. Conversations resume
across page refreshes, idle timeout, crashes, and deploys; `resume:
true` reconnects via `lastEventId` so clients only see new chunks.
`sessions.list` enumerates chats for inbox-style UIs.
- **Auto-accumulated history, delta-only wire** — the backend
accumulates the full conversation across turns; clients only ship the
new message each turn. Long chats never hit the 512 KiB body cap.
Register `hydrateMessages` to be the source of truth yourself.
- **Lifecycle hooks** — `onPreload`, `onChatStart`,
`onValidateMessages`, `hydrateMessages`, `onTurnStart`,
`onBeforeTurnComplete`, `onTurnComplete`, `onChatSuspend`,
`onChatResume` — for persistence, validation, and post-turn work.
- **Stop generation** — client-driven `transport.stopGeneration(chatId)`
aborts mid-stream; the run stays alive for the next message, partial
response is captured, and aborted parts (stuck `partial-call` tools,
in-progress reasoning) are auto-cleaned.
- **Tool approvals (HITL)** — tools with `needsApproval: true` pause
until the user approves or denies via `addToolApprovalResponse`. The
runtime reconciles the updated assistant message by ID and continues
`streamText`.
- **Steering and background injection** — `pendingMessages` injects user
messages between tool-call steps so users can steer the agent
mid-execution; `chat.inject()` + `chat.defer()` adds context from
background work (self-review, RAG, safety checks) between turns.
- **Actions** — non-turn frontend commands (undo, rollback, regenerate,
edit) sent via `transport.sendAction`. Fire `hydrateMessages` +
`onAction` only — no turn hooks, no `run()`. `onAction` can return a
`StreamTextResult` for a model response, or `void` for side-effect-only.
- **Typed state primitives** — `chat.local<T>` for per-run state
accessible from hooks, `run()`, tools, and subtasks (auto-serialized
through `ai.toolExecute`); `chat.store` for typed shared data between
agent and client; `chat.history` for reading and mutating the message
chain; `clientDataSchema` for typed `clientData` in every hook.
- **`chat.toStreamTextOptions()`** — one spread into `streamText` wires
up versioned system [Prompts](https://trigger.dev/docs/ai/prompts),
model resolution, telemetry metadata, compaction, steering, and
background injection.
- **Multi-tab coordination** — `multiTab: true` + `useMultiTabChat`
prevents duplicate sends and syncs state across browser tabs via
`BroadcastChannel`. Non-active tabs go read-only with live updates.
- **Network resilience** — built-in indefinite retry with bounded
backoff, reconnect on `online` / tab refocus / bfcache restore,
`Last-Event-ID` mid-stream resume. No app code needed.
See [/docs/ai-chat](https://trigger.dev/docs/ai-chat/overview) for the
full surface — quick start, three backend approaches (`chat.agent`,
`chat.createSession`, raw task), persistence and code-sandbox patterns,
type-level guides, and API reference.
- Add read primitives to `chat.history` for HITL flows:
`getPendingToolCalls()`, `getResolvedToolCalls()`,
`extractNewToolResults(message)`, `getChain()`, and
`findMessage(messageId)`. These lift the accumulator-walking logic that
customers building human-in-the-loop tools were re-implementing into the
SDK. ([#3543](https://github.com/triggerdotdev/trigger.dev/pull/3543))
Use `getPendingToolCalls()` to gate fresh user turns while a tool call
is awaiting an answer. Use `extractNewToolResults(message)` to dedup
tool results when persisting to your own store — the helper returns only
the parts whose `toolCallId` is not already resolved on the chain.
```ts
const pending = chat.history.getPendingToolCalls();
if (pending.length > 0) {
// an addToolOutput is expected before a new user message
}
onTurnComplete: async ({ responseMessage }) => {
const newResults = chat.history.extractNewToolResults(responseMessage);
for (const r of newResults) {
await db.toolResults.upsert({ id: r.toolCallId, output: r.output,
errorText: r.errorText });
}
};
```
- **Sessions** — a durable, run-aware stream channel keyed on a stable
`externalId`. A Session is the unit of state that owns a multi-run
conversation: messages flow through `.in`, responses through `.out`,
both survive run boundaries. Sessions back the new `chat.agent` runtime,
and you can build on them directly for any pattern that needs durable
bi-directional streaming across runs.
([#3542](https://github.com/triggerdotdev/trigger.dev/pull/3542))
```ts
import { sessions, tasks } from "@trigger.dev/sdk";
// Trigger a task and subscribe to its session output in one call
const { runId, stream } = await tasks.triggerAndSubscribe("my-task",
payload, {
externalId: "user-456",
});
for await (const chunk of stream) {
// ...
}
// Enumerate existing sessions (powers inbox-style UIs without a
separate index)
for await (const s of sessions.list({ type: "chat.agent", tag:
"user:user-456" })) {
console.log(s.id, s.externalId, s.createdAt, s.closedAt);
}
```
See [/docs/ai-chat/overview](https://trigger.dev/docs/ai-chat/overview)
for the full surface — Sessions powers the durable, resumable chat
runtime described there.
### Patch Changes
- Add Agent Skills for `chat.agent`. Drop a folder with a `SKILL.md` and
any helper scripts/references next to your task code, register it with
`skills.define({ id, path })`, and the CLI bundles it into the deploy
image automatically — no `trigger.config.ts` changes. The agent gets a
one-line summary in its system prompt and discovers full instructions on
demand via `loadSkill`, with `bash` and `readFile` tools scoped
per-skill (path-traversal guards, output caps, abort-signal
propagation).
([#3543](https://github.com/triggerdotdev/trigger.dev/pull/3543))
```ts
const pdfSkill = skills.define({ id: "pdf-extract", path:
"./skills/pdf-extract" });
chat.skills.set([await pdfSkill.local()]);
```
Built on the [AI SDK cookbook
pattern](https://ai-sdk.dev/cookbook/guides/agent-skills) — portable
across providers. SDK + CLI only for now; dashboard-editable `SKILL.md`
text is on the roadmap.
- Add `ai.toolExecute(task)` so you can wire a Trigger subtask in as the
`execute` handler of an AI SDK `tool()` while defining `description` and
`inputSchema` yourself — useful when you want full control over the tool
surface and just need Trigger's subtask machinery for the body.
([#3546](https://github.com/triggerdotdev/trigger.dev/pull/3546))
```ts
const myTool = tool({
description: "...",
inputSchema: z.object({ ... }),
execute: ai.toolExecute(mySubtask),
});
```
`ai.tool(task)` (`toolFromTask`) keeps doing the all-in-one wrap and now
aligns its return type with AI SDK's `ToolSet`. Minimum `ai` peer raised
to `^6.0.116` to avoid cross-version `ToolSet` mismatches in monorepos.
- Stamp `gen_ai.conversation.id` (the chat id) on every span and metric
emitted from inside a `chat.task` or `chat.agent` run. Lets you filter
dashboard spans, runs, and metrics by the chat conversation that
produced them — independent of the run boundary, so multi-run chats
correlate cleanly. No code changes required on the user side.
([#3543](https://github.com/triggerdotdev/trigger.dev/pull/3543))
- Type `chat.createStartSessionAction` against your chat agent so
`clientData` is typed end-to-end on the first turn:
([#3684](https://github.com/triggerdotdev/trigger.dev/pull/3684))
```ts
import { chat } from "@trigger.dev/sdk/ai";
import type { myChat } from "@/trigger/chat";
export const startChatSession = chat.createStartSessionAction<typeof
myChat>("my-chat");
// In the browser, threaded from the transport's typed startSession
callback:
const transport = useTriggerChatTransport<typeof myChat>({
task: "my-chat",
startSession: ({ chatId, clientData }) => startChatSession({ chatId,
clientData }),
// ...
});
```
`ChatStartSessionParams` gains a typed `clientData` field — folded into
the first run's `payload.metadata` so `onPreload` / `onChatStart` see
the same shape per-turn `metadata` carries via the transport. The opaque
session-level `metadata` field is unchanged.
- Unit-test `chat.agent` definitions offline with `mockChatAgent` from
`@trigger.dev/sdk/ai/test`. Drives a real agent's turn loop in-process —
no network, no task runtime — so you can send messages, actions, and
stop signals via driver methods, inspect captured output chunks, and
verify hooks fire. Pairs with `MockLanguageModelV3` from `ai/test` for
model mocking. `setupLocals` lets you pre-seed `locals` (DB clients,
service stubs) before `run()` starts.
([#3543](https://github.com/triggerdotdev/trigger.dev/pull/3543))
The broader `runInMockTaskContext` harness it's built on lives at
`@trigger.dev/core/v3/test` — useful for unit-testing any task code, not
just chat.
- Add `region` to the runs list / retrieve API: filter runs by region
(`runs.list({ region: "..." })` / `filter[region]=<masterQueue>`) and
read each run's executing region from the new `region` field on the
response.
([#3612](https://github.com/triggerdotdev/trigger.dev/pull/3612))
- Updated dependencies:
- `@trigger.dev/core@4.5.0-rc.0`
## @trigger.dev/build@4.5.0-rc.0
### Patch Changes
- Add Agent Skills for `chat.agent`. Drop a folder with a `SKILL.md` and
any helper scripts/references next to your task code, register it with
`skills.define({ id, path })`, and the CLI bundles it into the deploy
image automatically — no `trigger.config.ts` changes. The agent gets a
one-line summary in its system prompt and discovers full instructions on
demand via `loadSkill`, with `bash` and `readFile` tools scoped
per-skill (path-traversal guards, output caps, abort-signal
propagation).
([#3543](https://github.com/triggerdotdev/trigger.dev/pull/3543))
```ts
const pdfSkill = skills.define({ id: "pdf-extract", path:
"./skills/pdf-extract" });
chat.skills.set([await pdfSkill.local()]);
```
Built on the [AI SDK cookbook
pattern](https://ai-sdk.dev/cookbook/guides/agent-skills) — portable
across providers. SDK + CLI only for now; dashboard-editable `SKILL.md`
text is on the roadmap.
- Updated dependencies:
- `@trigger.dev/core@4.5.0-rc.0`
## trigger.dev@4.5.0-rc.0
### Patch Changes
- Add Agent Skills for `chat.agent`. Drop a folder with a `SKILL.md` and
any helper scripts/references next to your task code, register it with
`skills.define({ id, path })`, and the CLI bundles it into the deploy
image automatically — no `trigger.config.ts` changes. The agent gets a
one-line summary in its system prompt and discovers full instructions on
demand via `loadSkill`, with `bash` and `readFile` tools scoped
per-skill (path-traversal guards, output caps, abort-signal
propagation).
([#3543](https://github.com/triggerdotdev/trigger.dev/pull/3543))
```ts
const pdfSkill = skills.define({ id: "pdf-extract", path:
"./skills/pdf-extract" });
chat.skills.set([await pdfSkill.local()]);
```
Built on the [AI SDK cookbook
pattern](https://ai-sdk.dev/cookbook/guides/agent-skills) — portable
across providers. SDK + CLI only for now; dashboard-editable `SKILL.md`
text is on the roadmap.
- Add `TRIGGER_BUILD_SKIP_REWRITE_TIMESTAMP=1` escape hatch for local
self-hosted builds whose buildx driver doesn't support
`rewrite-timestamp` alongside push (e.g. orbstack's default `docker`
driver).
([#3618](https://github.com/triggerdotdev/trigger.dev/pull/3618))
- The CLI MCP server's agent-chat tools (`start_agent_chat`,
`send_agent_message`, `close_agent_chat`) now run on the new Sessions
primitive, so AI assistants driving a `chat.agent` get the same
idempotent-by-`chatId`, durable-across-runs behavior the browser
transport gets. Required PAT scopes go from `write:inputStreams` to
`read:sessions` + `write:sessions`.
([#3546](https://github.com/triggerdotdev/trigger.dev/pull/3546))
- MCP `list_runs` tool: add a `region` filter input and surface each
run's executing region in the formatted summary.
([#3612](https://github.com/triggerdotdev/trigger.dev/pull/3612))
- Updated dependencies:
- `@trigger.dev/core@4.5.0-rc.0`
- `@trigger.dev/build@4.5.0-rc.0`
- `@trigger.dev/schema-to-json@4.5.0-rc.0`
## @trigger.dev/core@4.5.0-rc.0
### Patch Changes
- Add Agent Skills for `chat.agent`. Drop a folder with a `SKILL.md` and
any helper scripts/references next to your task code, register it with
`skills.define({ id, path })`, and the CLI bundles it into the deploy
image automatically — no `trigger.config.ts` changes. The agent gets a
one-line summary in its system prompt and discovers full instructions on
demand via `loadSkill`, with `bash` and `readFile` tools scoped
per-skill (path-traversal guards, output caps, abort-signal
propagation).
([#3543](https://github.com/triggerdotdev/trigger.dev/pull/3543))
```ts
const pdfSkill = skills.define({ id: "pdf-extract", path:
"./skills/pdf-extract" });
chat.skills.set([await pdfSkill.local()]);
```
Built on the [AI SDK cookbook
pattern](https://ai-sdk.dev/cookbook/guides/agent-skills) — portable
across providers. SDK + CLI only for now; dashboard-editable `SKILL.md`
text is on the roadmap.
- Reject overlong `idempotencyKey` values at the API boundary so they no
longer trip an internal size limit on the underlying unique index and
surface as a generic 500. Inputs are capped at 2048 characters — well
above what `idempotencyKeys.create()` produces (a 64-character hash) and
above any realistic raw key. Applies to `tasks.trigger`,
`tasks.batchTrigger`, `batch.create` (Phase 1 streaming batches),
`wait.createToken`, `wait.forDuration`, and the input/session stream
waitpoint endpoints. Over-limit requests now return a structured 400
instead.
([#3560](https://github.com/triggerdotdev/trigger.dev/pull/3560))
- **AI Agents** — run AI SDK chat completions as durable Trigger.dev
agents instead of fragile API routes. Define an agent in one function,
point `useChat` at it from React, and the conversation survives page
refreshes, network blips, and process restarts.
([#3543](https://github.com/triggerdotdev/trigger.dev/pull/3543))
```ts
import { chat } from "@trigger.dev/sdk/ai";
import { streamText } from "ai";
import { openai } from "@ai-sdk/openai";
export const myChat = chat.agent({
id: "my-chat",
run: async ({ messages, signal }) =>
streamText({ model: openai("gpt-4o"), messages, abortSignal: signal }),
});
```
```tsx
import { useChat } from "@ai-sdk/react";
import { useTriggerChatTransport } from "@trigger.dev/sdk/chat/react";
const transport = useTriggerChatTransport({ task: "my-chat",
accessToken, startSession });
const { messages, sendMessage } = useChat({ transport });
```
**What you get:**
- **AI SDK `useChat` integration** — a custom
[`ChatTransport`](https://sdk.vercel.ai/docs/ai-sdk-ui/transport)
(`useTriggerChatTransport`) plugs straight into Vercel AI SDK's
`useChat` hook. Text streaming, tool calls, reasoning, and `data-*`
parts all work natively over Trigger.dev's realtime streams. No custom
API routes needed.
- **First-turn fast path (`chat.headStart`)** — opt-in handler that runs
the first turn's `streamText` step in your warm server process while the
agent run boots in parallel, cutting cold-start TTFC by roughly half
(measured 2801ms → 1218ms on `claude-sonnet-4-6`). The agent owns step
2+ (tool execution, persistence, hooks) so heavy deps stay where they
belong. Web Fetch handler works natively in Next.js, Hono, SvelteKit,
Remix, Workers, etc.; bridge to Express/Fastify/Koa via
`chat.toNodeListener`. New `@trigger.dev/sdk/chat-server` subpath.
- **Multi-turn durability via Sessions** — every chat is backed by a
durable Session that outlives any individual run. Conversations resume
across page refreshes, idle timeout, crashes, and deploys; `resume:
true` reconnects via `lastEventId` so clients only see new chunks.
`sessions.list` enumerates chats for inbox-style UIs.
- **Auto-accumulated history, delta-only wire** — the backend
accumulates the full conversation across turns; clients only ship the
new message each turn. Long chats never hit the 512 KiB body cap.
Register `hydrateMessages` to be the source of truth yourself.
- **Lifecycle hooks** — `onPreload`, `onChatStart`,
`onValidateMessages`, `hydrateMessages`, `onTurnStart`,
`onBeforeTurnComplete`, `onTurnComplete`, `onChatSuspend`,
`onChatResume` — for persistence, validation, and post-turn work.
- **Stop generation** — client-driven `transport.stopGeneration(chatId)`
aborts mid-stream; the run stays alive for the next message, partial
response is captured, and aborted parts (stuck `partial-call` tools,
in-progress reasoning) are auto-cleaned.
- **Tool approvals (HITL)** — tools with `needsApproval: true` pause
until the user approves or denies via `addToolApprovalResponse`. The
runtime reconciles the updated assistant message by ID and continues
`streamText`.
- **Steering and background injection** — `pendingMessages` injects user
messages between tool-call steps so users can steer the agent
mid-execution; `chat.inject()` + `chat.defer()` adds context from
background work (self-review, RAG, safety checks) between turns.
- **Actions** — non-turn frontend commands (undo, rollback, regenerate,
edit) sent via `transport.sendAction`. Fire `hydrateMessages` +
`onAction` only — no turn hooks, no `run()`. `onAction` can return a
`StreamTextResult` for a model response, or `void` for side-effect-only.
- **Typed state primitives** — `chat.local<T>` for per-run state
accessible from hooks, `run()`, tools, and subtasks (auto-serialized
through `ai.toolExecute`); `chat.store` for typed shared data between
agent and client; `chat.history` for reading and mutating the message
chain; `clientDataSchema` for typed `clientData` in every hook.
- **`chat.toStreamTextOptions()`** — one spread into `streamText` wires
up versioned system [Prompts](https://trigger.dev/docs/ai/prompts),
model resolution, telemetry metadata, compaction, steering, and
background injection.
- **Multi-tab coordination** — `multiTab: true` + `useMultiTabChat`
prevents duplicate sends and syncs state across browser tabs via
`BroadcastChannel`. Non-active tabs go read-only with live updates.
- **Network resilience** — built-in indefinite retry with bounded
backoff, reconnect on `online` / tab refocus / bfcache restore,
`Last-Event-ID` mid-stream resume. No app code needed.
See [/docs/ai-chat](https://trigger.dev/docs/ai-chat/overview) for the
full surface — quick start, three backend approaches (`chat.agent`,
`chat.createSession`, raw task), persistence and code-sandbox patterns,
type-level guides, and API reference.
- Stamp `gen_ai.conversation.id` (the chat id) on every span and metric
emitted from inside a `chat.task` or `chat.agent` run. Lets you filter
dashboard spans, runs, and metrics by the chat conversation that
produced them — independent of the run boundary, so multi-run chats
correlate cleanly. No code changes required on the user side.
([#3543](https://github.com/triggerdotdev/trigger.dev/pull/3543))
- Fix `LocalsKey<T>` type incompatibility across dual-package builds.
The phantom value-type brand no longer uses a module-level `unique
symbol`, so a single TypeScript compilation that resolves the type from
both the ESM and CJS outputs (which can happen under certain pnpm
hoisting layouts) no longer sees two structurally-incompatible variants
of the same type.
([#3626](https://github.com/triggerdotdev/trigger.dev/pull/3626))
- Unit-test `chat.agent` definitions offline with `mockChatAgent` from
`@trigger.dev/sdk/ai/test`. Drives a real agent's turn loop in-process —
no network, no task runtime — so you can send messages, actions, and
stop signals via driver methods, inspect captured output chunks, and
verify hooks fire. Pairs with `MockLanguageModelV3` from `ai/test` for
model mocking. `setupLocals` lets you pre-seed `locals` (DB clients,
service stubs) before `run()` starts.
([#3543](https://github.com/triggerdotdev/trigger.dev/pull/3543))
The broader `runInMockTaskContext` harness it's built on lives at
`@trigger.dev/core/v3/test` — useful for unit-testing any task code, not
just chat.
- Retry `TASK_PROCESS_SIGSEGV` task crashes under the user's retry
policy instead of failing the run on the first segfault. SIGSEGV in Node
tasks is frequently non-deterministic (native addon races, JIT/GC
interaction, near-OOM in native code, host issues), so retrying on a
fresh process often succeeds. The retry is gated by the task's existing
`retry` config + `maxAttempts` — same path `TASK_PROCESS_SIGTERM` and
uncaught exceptions already use — so tasks without a retry policy still
fail fast.
([#3552](https://github.com/triggerdotdev/trigger.dev/pull/3552))
- Add `region` to the runs list / retrieve API: filter runs by region
(`runs.list({ region: "..." })` / `filter[region]=<masterQueue>`) and
read each run's executing region from the new `region` field on the
response.
([#3612](https://github.com/triggerdotdev/trigger.dev/pull/3612))
- **Sessions** — a durable, run-aware stream channel keyed on a stable
`externalId`. A Session is the unit of state that owns a multi-run
conversation: messages flow through `.in`, responses through `.out`,
both survive run boundaries. Sessions back the new `chat.agent` runtime,
and you can build on them directly for any pattern that needs durable
bi-directional streaming across runs.
([#3542](https://github.com/triggerdotdev/trigger.dev/pull/3542))
```ts
import { sessions, tasks } from "@trigger.dev/sdk";
// Trigger a task and subscribe to its session output in one call
const { runId, stream } = await tasks.triggerAndSubscribe("my-task",
payload, {
externalId: "user-456",
});
for await (const chunk of stream) {
// ...
}
// Enumerate existing sessions (powers inbox-style UIs without a
separate index)
for await (const s of sessions.list({ type: "chat.agent", tag:
"user:user-456" })) {
console.log(s.id, s.externalId, s.createdAt, s.closedAt);
}
```
See [/docs/ai-chat/overview](https://trigger.dev/docs/ai-chat/overview)
for the full surface — Sessions powers the durable, resumable chat
runtime described there.
## @trigger.dev/plugins@4.5.0-rc.0
### Patch Changes
- The public interfaces for a plugin system. Initially consolidated
authentication and authorization interfaces.
([#3499](https://github.com/triggerdotdev/trigger.dev/pull/3499))
- Updated dependencies:
- `@trigger.dev/core@4.5.0-rc.0`
## @trigger.dev/python@4.5.0-rc.0
### Patch Changes
- Updated dependencies:
- `@trigger.dev/sdk@4.5.0-rc.0`
- `@trigger.dev/core@4.5.0-rc.0`
- `@trigger.dev/build@4.5.0-rc.0`
## @trigger.dev/react-hooks@4.5.0-rc.0
### Patch Changes
- Updated dependencies:
- `@trigger.dev/core@4.5.0-rc.0`
## @trigger.dev/redis-worker@4.5.0-rc.0
### Patch Changes
- Add MollifierBuffer and MollifierDrainer primitives for trigger burst
smoothing.
([#3614](https://github.com/triggerdotdev/trigger.dev/pull/3614))
MollifierBuffer (`accept`, `pop`, `ack`, `requeue`, `fail`,
`evaluateTrip`) is a per-env FIFO over Redis with atomic Lua transitions
for status tracking. `evaluateTrip` is a sliding-window trip evaluator
the webapp gate uses to detect per-env trigger bursts.
MollifierDrainer pops entries through a polling loop with a
user-supplied handler. The loop survives transient Redis errors via
capped exponential backoff (up to 5s), and per-env pop failures don't
poison the rest of the batch — one env's blip is logged and counted as
failed for that tick. Rotation is two-level: orgs at the top, envs
within each org. The buffer maintains `mollifier:orgs` and
`mollifier:org-envs:${orgId}` atomically with per-env queues, so the
drainer walks orgs → envs directly without an in-memory cache. The
`maxOrgsPerTick` option (default 500) caps how many orgs are scheduled
per tick; for each picked org, one env is popped (rotating round-robin
within the org). An org with N envs gets the same per-tick scheduling
slot as an org with 1 env, so tenant-level drainage throughput is
determined by org count rather than env count.
- Updated dependencies:
- `@trigger.dev/core@4.5.0-rc.0`
## @trigger.dev/rsc@4.5.0-rc.0
### Patch Changes
- Updated dependencies:
- `@trigger.dev/core@4.5.0-rc.0`
## @trigger.dev/schema-to-json@4.5.0-rc.0
### Patch Changes
- Updated dependencies:
- `@trigger.dev/core@4.5.0-rc.0`
</details>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
## Summary
Two CI workflows were blocking the v4.5.0-rc.0 release PR (#3563) and
would block every future changeset release PR.
### 1. `changesets-pr.yml` — self-report `All PR Checks`
The changesets bot pushes commits authored by `GITHUB_TOKEN`. By GitHub
design, `GITHUB_TOKEN`-authored pushes can't trigger downstream
workflows (loop-prevention). That means `pr_checks.yml` never fires on
release-PR commits, leaving the required `All PR Checks` status
permanently `Expected — Waiting for status to be reported`. The PR can't
merge.
The fix: after `changesets/action` creates the PR, post a `success`
check with the exact `All PR Checks` context onto the PR's head SHA.
GitHub's required-check evaluation is satisfied by any check with the
right context name — the source doesn't have to be `pr_checks.yml`.
**Why this is safe:** the release PR only mechanically bumps
`package.json`, `pnpm-lock.yaml`, and `CHANGELOG.md` from changesets
that were already on `main` (and already ran full CI when they merged).
If a human ever pushes a commit to `changeset-release/main`,
`pr_checks.yml` fires on that push (real user, not `GITHUB_TOKEN`) and
posts its own `All PR Checks` status — last write wins for the same
context on the same SHA, so the human-push result overrides the
auto-success.
### 2. `vouch-check-pr.yml` — exempt `github-actions[bot]`
The `require-draft` job auto-closes any non-draft PR whose author is not
a `MEMBER`/`OWNER`/`COLLABORATOR`, with an explicit allowlist for
`devin-ai-integration[bot]` and `dependabot[bot]`. The changesets bot
publishes as `github-actions[bot]` with `author_association:
CONTRIBUTOR`, so every release PR was getting auto-closed on open with a
"please re-open as draft" comment. Add `github-actions[bot]` to the
exemption list.
## Test plan
- [ ] After merge, the next changeset bot push to
`changeset-release/main` should post `All PR Checks: success` on the
release PR's head SHA, and the PR should not get auto-closed by `Vouch -
Check PR`.
- [ ] Confirm `pr_checks.yml` still fires + gates normal
(human-authored) PRs to `main`.
## Summary
Pre-existing typecheck errors in `references/ai-chat` against the
current SDK shape. Unblocks `pnpm exec tsc --noEmit` in the reference
project.
## What changed
Three categories of fixes inside `references/ai-chat`. No SDK changes.
### 1. `payload.messages` → `payload.message`
The wire payload is now delta-only — one new message per trigger,
optional. Old code in two raw-task files reads `payload.messages`
(plural array) which no longer exists.
```ts
// before
const messages = await conversation.addIncoming(currentPayload.messages, ...);
// after
const messages = await conversation.addIncoming(
currentPayload.message ? [currentPayload.message] : [],
...
);
```
Same fix to the `chat.messages.on` handler, reading `msg.message`
(singular) instead of `msg.messages[length - 1]`.
### 2. `clientData` non-null assertion in `cf-trust-test`
`ChatTurnContext.clientData` is typed as `?: TClientData` on
`onTurnStart` / `run` event objects even when the agent declares a
`clientDataSchema`. The runtime validates against the schema before the
hook fires, so it's structurally non-null — but TypeScript can't know
that. Non-null assert for now.
Follow-up worth filing: narrow `ChatTurnContext.clientData` to
non-optional when the agent has a `clientDataSchema`. Same friction the
docs friction-test subagent flagged.
### 3. `stress-emit.parseConfig` retyped against `ModelMessage[]`
The `run` callback hands `messages: ModelMessage[]`, not `UIMessage[]`.
Update `parseConfig` to accept `ModelMessage[]` and pull text from
`content` (string or array-of-parts).
## Test plan
- [x] `pnpm exec tsc --noEmit` in `references/ai-chat` passes (was 8
errors, now 0)
## Summary
Type `chat.createStartSessionAction` against the chat agent so
`clientData` is typed end-to-end on the first turn. Closes the gap where
`useTriggerChatTransport`'s `startSession` callback already hands you a
typed `clientData` (via the transport generic) but the server-side
action couldn't accept it without untyped routing through the `metadata`
field.
## Design
`ChatStartSessionParams` gains a typed `clientData` field via the new
generic:
```ts
export type ChatStartSessionParams<TChat extends AnyTask = AnyTask> = {
chatId: string;
clientData?: InferChatClientData<TChat>;
triggerConfig?: Partial<SessionTriggerConfig>;
metadata?: Record<string, unknown>;
};
function createChatStartSessionAction<TChat extends AnyTask = AnyTask>(
taskId: string,
options?: CreateChatStartSessionActionOptions
): (params: ChatStartSessionParams<TChat>) => Promise<ChatStartSessionResult>
```
When provided, `clientData` is folded into the first run's
`triggerConfig.basePayload.metadata`, so `onPreload` / `onChatStart` see
the same shape per-turn `metadata` carries via the transport. The opaque
session-level `metadata` field stays exactly as before — it lands on the
Session row, not the run payload.
## Usage
```ts
// actions.ts
import { chat } from "@trigger.dev/sdk/ai";
import type { myChat } from "@/trigger/chat";
export const startChatSession = chat.createStartSessionAction<typeof myChat>("my-chat");
```
```tsx
// Chat.tsx
const transport = useTriggerChatTransport<typeof myChat>({
task: "my-chat",
accessToken: ({ chatId }) => mintChatAccessToken(chatId),
startSession: ({ chatId, clientData }) =>
startChatSession({ chatId, clientData }),
});
```
## Test plan
- [x] `pnpm run build --filter @trigger.dev/sdk` passes
- [ ] Verify a `chat.agent` with `clientDataSchema` reads the typed
clientData from `onPreload` payload metadata on the first turn
(`OBJECT_STORE_BASE_URL`) and a named protocol provider
(`OBJECT_STORE_DEFAULT_PROTOCOL=s3`), chat.agent session snapshot writes
landed in the named provider but reads fell through to the default — so
the recovery boot couldn't find the snapshot it had just written.
After a mid-stream cancel, the missing snapshot triggered a fallback
replay path that dropped the user's follow-up message, leaving the chat
stuck in `submitted` indefinitely.
Fix:
- New `/api/v1/sessions/:id/snapshot-url` route handles PUT + GET
symmetrically — both prefix unprefixed keys with
`OBJECT_STORE_DEFAULT_PROTOCOL` so they always round-trip through the
same store.
- `Session.chatSnapshotStoragePath` persists the resolved URI on first
write so future protocol changes don't strand existing snapshots.
Reads prefer the stored URI and fall back to the computed default for
pre-column sessions.
- SDK calls `createChatSnapshotUploadUrl` / `getChatSnapshotUrl`; the
generic v1/v2 packets endpoints are unchanged.
## Test plan
- [x] Configure local with two providers (R2 default + MinIO `s3` named)
and `OBJECT_STORE_DEFAULT_PROTOCOL=s3`.
- [x] Reproduce hang: send a message, cancel mid-stream, send another —
without the fix it hangs in `submitted`; with the fix it streams.
- [x] Snapshot lands in the `s3`-protocol bucket and
`Session.chatSnapshotStoragePath` is set after first write.
- [x] SDK unit tests pass; webapp typecheck passes.
Env-var lookups via `GET
/api/v1/projects/:projectRef/envvars/:slug/:name` run a Prisma
`findMany` on `EnvironmentVariableValue` filtered by `environmentId` +
`isSecret`. The only existing indexes are the primary key and a unique
on `(variableId, environmentId)`, so `environmentId` is never the
leading column — the planner falls back to a Parallel Seq Scan over the
whole table to find what is, in practice, a handful of rows per
environment.
Two changes:
- Add a btree index on `EnvironmentVariableValue(environmentId)` so the
planner switches to an index scan. The composite `(variableId,
environmentId)` unique stays in place; the new index is purely additive.
- Route the `findMany` inside `getEnvironmentWithRedactedSecrets`
through the read replica via a new `replicaClient` constructor param on
the repository (defaulting to `$replica`, mirroring how `prismaClient`
defaults to `prisma`). Writes and read-after-write methods stay on the
primary.
## Test plan
- [ ] `pnpm run typecheck --filter webapp`
- [ ] Confirm `EXPLAIN` plan flips from Parallel Seq Scan to an index
scan
- [ ] Existing env-var route tests still pass
## Summary
- Dashboard loaders for runs / sessions / batches / schedule-detail
threw bare `Error("X not found")` when a slug didn't resolve. Remix
surfaces this as a 500 and Sentry captures it via auto-instrumentation,
producing ongoing noise from real users following stale preview-branch
or deleted-resource links (the URLs in those Sentry events all carry
`?_data=routes/...`, i.e. client-side revalidation, not full-page
navigation).
- Added a `throwNotFound(statusText)` helper in
`app/utils/httpErrors.ts` that throws a Response with status 404,
matching the established pattern in sibling routes (agents, alerts,
bulk-actions, etc.).
- Migrated 5 loader sites to `throwNotFound` (4× "Environment not
found", 1× "Schedule not found").
- Migrated 1 loader site (`runs._index` project branch) to
`redirectWithErrorMessage("/", request, "Project not found")` to match
the pre-existing convention used by every other dashboard route's
project-not-found branch.
- Intentionally **not** touched: bare `throw new Error("X not found")`
inside `resources.*` action routes (sit inside try/catch blocks that
already redirect with a flash message), the invariant assertion in
`vercel.connect.tsx`, and the admin config check in
`admin.api.v1.runs-replication.backfill.ts`.
## Where the fix is visible
Normal browser navigation to these URLs doesn't reach the buggy loaders
— the parent env-layout
(`_app.orgs.$organizationSlug.projects.$projectParam.env.$envParam/route.tsx`)
already filters missing envs/projects and redirects/404s before the
child loader runs. The bug fires exclusively when Remix calls a single
child loader via `?_data=routes/...`, which happens during client-side
navigation or `useRevalidator`. That matches every Sentry event URL.
## Test plan
- [x] Unit test for the new helper —
`apps/webapp/test/httpErrors.test.ts`
- [x] `pnpm run typecheck --filter webapp` clean
- [x] Manual verification via Playwright on `main` vs this branch (6
cases): main returns 500 for each defective `_data` URL; branch returns
404 or 204 + `X-Remix-Redirect` as designed
- [x] Verified user-visible 404 catch boundary on `schedules/<missing>`
(the one case reachable via normal nav)
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
## Summary
Five hardening fixes across `@trigger.dev/sdk`, `@trigger.dev/core`, and
`@trigger.dev/build`.
- `tasks.triggerAndSubscribe` now forwards caller `requestOptions`
(custom API keys, per-request overrides) to the underlying
`apiClient.triggerTask` call instead of silently dropping them.
- `SSEStreamSubscription` no longer retries permanent client errors
forever. The default `nonRetryableStatuses` widens from `[404, 410]` to
`[400, 404, 409, 410, 422]`, so a malformed session-stream request fails
fast instead of busy-looping under bounded backoff.
- Session writer falls back to manually wiring the caller's
`AbortSignal` on Node 18, where `AbortSignal.any` is unavailable.
Caller-driven cancellation now propagates on every supported runtime.
- `TriggerChatTransport` throws immediately when a `chat.handover`
response is missing `X-Trigger-Chat-Access-Token`, instead of silently
downgrading every subsequent turn back to the handover path. `dispose()`
aborts every active `session.out` subscription before tearing the
coordinator down, so unmount/navigation no longer leaves SSE readers in
flight.
- Removed the experimental `@trigger.dev/build/extensions/secureExec`
build extension. It will return alongside the sandbox feature it was
built to support.
## Test plan
- [ ] `pnpm run build --filter @trigger.dev/sdk --filter
@trigger.dev/core --filter @trigger.dev/build`
- [ ] `pnpm --filter @trigger.dev/sdk test --run` (183 tests, including
chat / chat-server / sessions / handover)
- [ ] `pnpm --filter @trigger.dev/core test --run`
- [ ] Manually trigger a `chat.handover` whose response strips
`X-Trigger-Chat-Access-Token`, and confirm the transport throws
synchronously rather than degrading.
- [ ] Unmount a chat UI mid-stream and confirm the active `session.out`
SSE connection closes immediately.
## Summary
The S2 access-token cache key was `${basin}:${streamPrefix}` — purely
server-derived but blind to the **scope/ops list** hardcoded one method
away. When the ops list changes in code (e.g. #3644 added `trim` so
`chat.agent`'s per-turn trim chain can issue `AppendRecord.trim()`),
pre-deploy tokens still in cache get returned to SDK callers for up to
the token's TTL (24h default), surfacing as `Operation not permitted`
403s on any op outside the old scope.
## Fix
Lift the ops list to a module constant and fold its sorted-join
fingerprint into the cache key:
```ts
const S2_TOKEN_OPS = ["append", "create-stream", "trim"] as const;
const S2_TOKEN_OPS_FINGERPRINT = [...S2_TOKEN_OPS].sort().join(",");
// in getS2AccessToken
const cacheKey = `${this.basin}:${this.streamPrefix}:${S2_TOKEN_OPS_FINGERPRINT}`;
// in s2IssueAccessToken
scope: { /* ... */ ops: [...S2_TOKEN_OPS], /* ... */ }
```
The fingerprint is derived from the single source of truth, so any
future scope change auto-invalidates without anyone remembering to bump
a literal version. The Unkey L1 (in-memory LRU) and L2 (Redis) layers
share the same key derivation, so both reset together on the next deploy
with no manual cache busting.
## Test plan
- [ ] `pnpm run typecheck --filter webapp`
- [ ] Run a multi-turn `chat.agent` chat via `references/ai-chat` and
confirm no `chat.agent: trim failed; will retry next turn` warn span
fires across turn-completes.
## Summary
The prerelease (snapshot) path of the release workflow fails immediately
whenever `main` carries an active `.changeset/pre.json` (i.e. during an
in-progress RC cycle, like the current v4 RC):
```
🦋 error Snapshot release is not allowed in pre mode
🦋 To resolve this exit the pre mode by running `changeset pre exit`
```
This blocks `chat-prerelease` snapshots from main even though the
snapshots are unrelated to the RC cycle.
Adds a conditional `changeset pre exit` step right before `Snapshot
version` in the prerelease job. The job runs on a checkout with
`persist-credentials: false`, so the `pre.json` deletion stays on the
runner's working tree — main's persisted pre-mode state is untouched,
and v4 RC publishes keep working normally.
## Test plan
- [ ] Re-run the `🦋 Changesets Release` workflow with `type=prerelease`,
`ref=main`, `prerelease_tag=chat-prerelease` and confirm it gets past
the snapshot step and publishes.
- [ ] Confirm `.changeset/pre.json` on `main` is unchanged after the
run.
## Summary
Companion to #3536, which patched routes that already had a leaking
`catch (e) { return json({error: e.message}, 500) }`. That pattern can't
reach routes which have no catch in the first place — when those throw,
Remix's default error path serializes `error.message` into the response
body, and the SDK then wraps the leaked string as `TriggerApiError`.
Across 28 raw api.v1 loaders/actions plus one dashboard polling
endpoint, each handler now:
- Wraps its body in `try { ... } catch (error) { ... }`.
- Re-throws `Response` instances so auth helpers' `throw json(...)` /
`throw redirect(...)` pass through unchanged.
- Logs non-Response errors via `logger.error` so server-side visibility
is preserved.
- Returns a generic body — `{"error": "Internal Server Error"}` 500 for
raw API routes, or `{ changelogs: [] }` 200 for the polling widget
(degrade silently across transient blips; the consumer hook already
coped with empty payloads).
For six routes where #3536 left an inner try/catch covering only a
service call (`alertChannels`, `batches.results`,
`deployments.finalize`, `deployments.background-workers`,
`deployments.promote`, `projects.background-workers`): an outer
try/catch is added so auth/parsing failures are also sanitized. Inner
typed-error handling (`ServiceValidationError` → 422 with message, etc.)
is preserved exactly.
For two routes whose existing catch returned 400 + `error.message`
(`api.v1.authorization-code`, `api.v1.orgs.\$orgParam.projects` action):
the body is sanitized to a generic per-route string. **Status code stays
400** — clients that key on the 4xx/5xx distinction (and the SDK's
no-retry-on-4xx behavior) are unaffected.
## Test plan
- [x] \`pnpm run typecheck --filter webapp\`
- [x] Per-route synthetic-throw probe: inject \`throw new
Error("SYNTHETIC ...")\` at the top of each catch'd try, curl the route
with a dummy bearer, confirm the response body is the generic shape and
that the synthetic message lands server-side via \`logger.error\`. 29
routes verified.
- [x] Real-P1001 probe on the envvars loader: \`docker stop database\`
mid-flight, confirm response is generic 500 (not the leaked Prisma
message).
- [x] Sampled legitimate 4xx/2xx paths across each pattern variant
(naked-wrap, partial-expanded, 400-preserved) to confirm the wraps don't
interfere with normal control flow.
## Summary
`TriggerChatTransport`, `AgentChat`, and `chat.createStartSessionAction`
now accept a string-or-function `baseURL` so callers can route per
endpoint — e.g. `.in/append` through a trusted edge proxy while keeping
`.out` SSE direct. The same surfaces add a `fetch` override for header
injection, custom retries, or proxy rewrites that go beyond URL routing.
SSE GETs are covered too via a new `fetchClient` option on
`SSEStreamSubscription`.
```ts
// TriggerChatTransport / AgentChat — endpoints: "in" | "out"
baseURL: ({ endpoint }) =>
endpoint === "out" ? DIRECT : PROXY,
fetch: (url, init, ctx) => {
init.headers = new Headers(init.headers);
init.headers.set("traceparent", currentTraceparent());
return globalThis.fetch(url, init);
},
// chat.createStartSessionAction — endpoints: "sessions" | "auth"
chat.createStartSessionAction("my-agent", {
baseURL: ({ endpoint }) => (endpoint === "sessions" ? PROXY : DIRECT),
});
```
`streamBaseURL` on `TriggerChatTransport` is kept as a backwards-compat
alias and continues to win for the `"out"` endpoint when set.
Plain-string `baseURL` still applies to every endpoint, matching prior
behavior.
## Summary
Hot-loop writers — `streams.writer` / `streams.pipe` on the run-scoped
side, `chat.response.write` / `chat.stream.*` on the session side — were
issuing a fresh `PUT` to mint S2 credentials for every chunk. On run
streams, each PUT also pushed the streamId onto
`TaskRun.realtimeStreams`,
so a chat-agent turn writing N chunks produced N PUTs and N duplicate
array pushes against the same row.
The SDK now caches the initialize response per cache slot: `(runId,
key)`
for run streams, the session id for session streams. First call PUTs as
before; subsequent calls reuse the cached promise. Hot-loop writers do
one PUT per slot for the lifetime of the cache.
S2 access tokens have a 1-day TTL. If a writer's `wait()` rejects (auth
error, expired token, network blip), the cache evicts the matching slot
so the next call re-PUTs and mints fresh credentials, identity-checked
so a concurrent caller's fresh promise isn't accidentally cleared.
## chat.agent guardrail
`streams.pipe / writer / append / read` called inside a `chat.agent` run
now logs a one-time warning pointing at `chat.response.write` /
`chat.stream.*` — `streams.*` is run-scoped and isn't visible on the
chat session. The ai-chat docs are updated to drop the old guidance
toward run-scoped streams.
## Summary
- Introduce the Mollifier: a Redis-backed buffer for `trigger()` API
calls during traffic spikes, with a per-env trip evaluator and a drainer
ack-loop.
- Phase 1 is dual-write monitoring — every mollified trigger is buffered
to Redis AND continues to `engine.trigger`. No customer-facing behaviour
change.
- Telemetry events: `mollifier.would_mollify`, `mollifier.buffered`,
`mollifier.drained`, plus the `mollifier.decisions` counter.
- Gated behind a feature flag (default off).
## Test plan
- [x] `pnpm run test --filter @trigger.dev/redis-worker`
- [x] `pnpm run test --filter webapp -- mollifier`
- [x] Manual: with flag off, no behaviour change vs main
- [x] Manual: with flag on + threshold lowered, observe
`mollifier.buffered` + `mollifier.drained` log pairs with matching
`runId`
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
## Summary
Two papercuts new contributors hit running this repo locally:
1. Fresh clones default to v1 (Redis-only) realtime streams, so Sessions
and `chat.agent` error with `"S2 configuration is missing"`, even though
the `s2` service is already in `docker/docker-compose.yml` and pre-seeds
a `trigger-local` basin. Wire `REALTIME_STREAMS_S2_*` to it in
`.env.example` so the new-contributor flow just works. (Also drop the s2
healthcheck: the image is distroless, so the `wget` check always reports
unhealthy.)
2. Two clones can't both run `pnpm run docker` because ports, project
name, and container names are all hardcoded. Parameterize every host
port as `${VAR:-default}`, drive the project name via
`COMPOSE_PROJECT_NAME` (with a top-level `name:` field as the default),
prefix container names with `${CONTAINER_PREFIX:-}`, and pass
`--env-file .env` so compose reads the same root `.env` the webapp does.
The "Running multiple instances side by side" block in `.env.example`
lists every overridable knob.
Also split the optional services (`electric-shard-1`, `ch-ui`,
`toxiproxy`, `nginx-h2`, `otel-collector`, `prometheus`, `grafana`) into
`docker-compose.extras.yml` behind a new `pnpm run docker:full` script.
The core stack keeps everything the webapp actually needs to boot:
postgres, redis, electric, minio, clickhouse + migrator, s2-lite.
Defaults match every previous hardcoded value, so existing setups keep
working without touching `.env`.
## Test plan
- [x] `pnpm run docker` on a clean clone brings up the core services on
the standard ports under the `triggerdotdev-docker` project name.
- [x] Setting `COMPOSE_PROJECT_NAME=triggerdotdev-docker-alt` + the
`*_HOST_PORT` overrides in `.env` brings up a second stack alongside the
default one with no port or container-name clashes.
- [x] Webapp boots cleanly against the default `.env.example` values;
`/healthcheck` returns 200, no S2 errors.
- [x] s2-lite basin `trigger-local` accepts an append + read via the
same REST endpoints the webapp uses.
- [x] `pnpm run docker:full` brings up the optional services alongside
the core ones in the same project.
## Summary
Long-running chat agents were filling `session.out` forever — every
`chat.agent` turn appended to the same S2 stream with no trim, and the
Sessions dashboard re-streamed the entire history from `seq_num=0` on
every page load. After this change the agent appends an S2 `trim`
command record after each `trigger:turn-complete`, pointing back at the
previous turn-complete's seq_num. `session.out` stays roughly one turn
long at steady state, regardless of session age.
`trigger:turn-complete` and `trigger:upgrade-required` move from
`chunk.type`-shaped data records into header-form control records under
a uniform `trigger-control` namespace. Built-in transports
(`TriggerChatTransport`, `AgentChat`, the dashboard's `AgentView`)
handle the new shape transparently. Custom transports need a one-line
filter on the `trigger-control` header — see the rewritten "Records on
session.out" section in the client-protocol docs.
The Sessions detail page in the dashboard fetches the agent's per-turn
S3 snapshot via a presigned URL and seeds the transcript view, then
SSE-tails from the snapshot's `lastOutEventId`. Bandwidth and
time-to-first-render scale with unread turns instead of session
lifetime.
Resume contract is now explicit: single-turn-boundary resume always
works (the prior turn-complete is still on the stream), the S2 trim is
eventually consistent over 10-60s, and multi-turn-away resume falls back
to a snapshot reload.
## Summary
The PUT handler at `/realtime/v1/streams/:runId/:target/:streamId` ran
`taskRun.update({ realtimeStreams: { push: streamId } })` on every call,
even when the `streamId` was already present. SDK call patterns that
re-initialize the same stream key on every chunk produce a per-write row
UPDATE, duplicate entries pile up in the array, and the row-lock + TOAST
rewrite cost grows unbounded on long-running stream sessions.
## Fix
Mirror the sibling append handler: read the array first and only push
when the `streamId` isn't already present. Identical behavior for
first-time stream creation; repeat creates short-circuit to a single
indexed read. The dashboard's per-run stream listing keeps working
because the first create still records the entry.
## Test plan
- [ ] A fresh PUT for a new `(run, streamId)` adds the entry to the
array
- [ ] A repeat PUT for the same pair leaves the array unchanged
- [ ] 404 is returned when the run doesn't exist; 400 when the run is
completed
## Summary
- Adds a `beforeSend` rule in `apps/webapp/sentry.server.ts` that
collapses Prisma `P1001` ("Can't reach database server") errors into a
single Sentry issue regardless of which call site threw, by setting
`event.fingerprint = ["prisma-p1001-db-unreachable"]` and tagging
`db_unreachable:true`.
- Matches both `err.code === "P1001"` (Prisma's `KnownRequestError` when
a connection drops mid-query) and `err.errorCode === "P1001"`
(`InitializationError` when the client fails to connect at startup).
- Implemented as a small extensible `FINGERPRINT_RULES` table so further
fan-out errors can be added with one entry.
## Verification
End-to-end verified locally with `debug: true` on the SDK:
- Real Prisma `P1001` thrown from a loader (DB stopped mid-request) is
captured by Sentry's Remix auto-instrumentation
- `beforeSend` fires with `originalException.code === "P1001"`, rule
matches
- `event.fingerprint = ["prisma-p1001-db-unreachable"]` and
`tags.db_unreachable = "true"` applied
- Event lands in Sentry under the new fingerprint
## Test plan
- [ ] Deploy to staging; confirm P1001 events appear under a single
`prisma-p1001-db-unreachable` issue rather than fanning out
- [ ] Confirm `db_unreachable:true` tag is filterable in Sentry
- [ ] Verify non-P1001 errors are unaffected (event passes through
`beforeSend` untouched)
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
## Summary
Local self-hosted deploys (`trigger deploy --local-build --push
--builder orbstack` or any other buildx setup using the **docker**
driver) fail at the push step with:
```
ERROR: failed to build: failed to solve:
exporter option "rewrite-timestamp" conflicts with "unpack"
```
The docker driver auto-enables `unpack=true` when pushing, and that's
incompatible with `rewrite-timestamp` (which the CLI sets for
reproducible-build hashing).
Adds a simple env-var opt-out so contributors can keep using their
default builder. The flag is only read by the local-build code path;
remote/cloud builds are unaffected.
```bash
TRIGGER_BUILD_SKIP_REWRITE_TIMESTAMP=1 \
pnpm exec trigger deploy --profile default --local-build --push --builder orbstack
```
The trade-off: skipping `rewrite-timestamp` means layer timestamps
reflect actual build time, so two identical builds produce different
layer hashes. Fine for a local-dev registry; the only real consumer of
timestamp-stability is registry-layer cache hit rates.
## Test plan
- [x] Manual: ran `trigger deploy --profile default --local-build --push
--builder orbstack` against the localhost webapp + a local Docker
registry on port 5001 — first failed with the rewrite-timestamp/unpack
error, then succeeded after setting
`TRIGGER_BUILD_SKIP_REWRITE_TIMESTAMP=1`.
- [x] Full chat.agent smoke sweep (15 tests, including suspend/resume,
deepResearch subtask, AgentChat orchestrator) against the deployed image
— all pass.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
`TriggerChatTransport` had a single `baseURL` option covering both the
`.in/append` POSTs and the long-lived `.out` SSE subscription. Customers
wanting to route the SSE through a proxy (e.g. a Cloudflare worker
capturing JA4 fingerprints for bot detection) had to send every append
through the proxy too, adding a hop to every user message.
New optional `streamBaseURL` overrides the SSE base URL only; appends
keep using `baseURL`. Falls back to `baseURL` when unset, so existing
transports are unchanged.
```ts
const transport = new TriggerChatTransport({
task: "ai-chat",
baseURL: "https://api.trigger.dev",
streamBaseURL: "https://chat-proxy.example.com",
accessToken,
startSession,
});
```
Verified with a new test in `chat.test.ts` that asserts `.in/append`
routes through `baseURL` and `.out` SSE routes through `streamBaseURL`.
All existing tests still pass.
`dorny/paths-filter` defaults to OR semantics across the pattern array,
so the leading `**` matched every file and the `!...` excludes were
no-ops. The `code` filter has been returning `true` for every PR since
#3615.
Split into two filter steps: `code` moves into its own step with
`predicate-quantifier: every` so excludes actually subtract. The two
re-include workflow files become a separate `typecheck_self` filter that
the `typecheck` job ORs into its `if:`.
Side effect: workflow-file-only PRs that don't touch `pr_checks.yml` or
`typecheck.yml` no longer trigger typecheck. Previously they did because
the filter was broken-true.
## Summary
Codify two rules for ClickHouse migration authors that came out of the
029/030 ordering incident on the TRI-9367 test cloud deploy:
1. **Number files to `max(existing) + 1`, never slot in below the
latest.** Goose runs in strict mode in the cloud deploy pipeline and
refuses to apply a missing version below the current version — slotting
a file in below an already-applied number blocks the next deploy.
2. **DDL must be idempotent** (`ADD COLUMN IF NOT EXISTS`, `DROP COLUMN
IF EXISTS`, `CREATE TABLE IF NOT EXISTS`, etc.) so a retry or
out-of-order apply (`goose up --allow-missing` for local recovery,
manual fixups) is a no-op rather than an error.
## Where the rules live
- `internal-packages/clickhouse/CLAUDE.md` — full rules + example for
migration authors (and AI agents writing migrations).
- `.claude/REVIEW.md` — added a 🔴 finding under "What makes a 🔴
Important finding" so PR reviewers flag either fault as blocking.
The existing migration files are left untouched; the idempotency
requirement applies going forward.
## Test plan
- [ ] Next ClickHouse migration PR uses `IF NOT EXISTS` / `IF EXISTS`
forms
- [ ] No new migration files numbered below an already-applied version
on test/prod
## Summary
Renumber `029_add_task_kind_to_task_runs_v2.sql` →
`031_add_task_kind_to_task_runs_v2.sql` to fix a deploy-blocking
out-of-order migration, and make the DDL idempotent with `ADD COLUMN IF
NOT EXISTS` / `DROP COLUMN IF EXISTS`.
## Root cause
- Migration `030_create_sessions_v1.sql` landed on main on 2026-04-28
(PR #3417) and was applied to test cloud ClickHouse on a subsequent
deploy. Current goose version on test ClickHouse: **30**.
- Migration `029_add_task_kind_to_task_runs_v2.sql` was authored later
on 2026-05-10 as part of the Sessions primitive PR series (`be1a6cf8`).
- The next test cloud deploy failed because goose strict-mode refused to
apply a missing version *before* the current version:
```
goose run: error: found 1 missing migrations before current version 30:
version 29: 029_add_task_kind_to_task_runs_v2.sql
```
## Fix
1. **Rename to `031_*`** (next available number after 030). Goose now
treats it as a new migration after 030 and applies it cleanly on
test/prod where the column does not yet exist.
2. **Make the DDL idempotent** (`ADD COLUMN IF NOT EXISTS`). The
original 029 may have been applied in environments that ran goose with
`--allow-missing` (e.g. some local dev databases) — those would have the
column already, and the rename causes goose to see 031 as new and
re-attempt the ADD. Idempotent DDL keeps that path safe. The `Down`
mirrors with `DROP COLUMN IF EXISTS`.
## Test plan
- [ ] Test cloud deploy (after this lands) successfully runs the
ClickHouse migration step
- [ ] `task_kind` column shows up on `trigger_dev.task_runs_v2`
post-migration
- [ ] Local environments that had previously applied 029 do not error on
the next `goose up`
## Summary
Adds `.changeset/pre.json` to put the repo into changesets pre mode with
dist-tag `rc`. After this merges, the changesets bot regenerates the
existing release PR as `chore: release v4.5.0-rc.0`. Merging that PR
publishes the first release candidate of 4.5.0 to npm under `@rc`.
The pre-mode plumbing landed in #3628. The release content (chat.agent +
sessions + ai prompts + dashboard server-changes) landed in #3629.
## What ships when the bot PR merges
Under dist-tag `rc`:
-
`@trigger.dev/{sdk,core,build,react-hooks,redis-worker,plugins,python,rsc,schema-to-json}@4.5.0-rc.0`
- `trigger.dev@4.5.0-rc.0`
Plus:
- Docker image `ghcr.io/triggerdotdev/trigger.dev:v4.5.0-rc.0`
(immutable tag only — `:v4-beta` is not touched)
- Helm chart `oci://ghcr.io/triggerdotdev/charts/trigger.dev:4.5.0-rc.0`
- GitHub release `v4.5.0-rc.0` marked as Pre-release (no Latest badge)
What does NOT happen:
- npm `latest` stays at 4.4.6
- No marketing-site changelog PR (gated on `is_prerelease != 'true'`)
- Docker `:latest` not touched (we never push it anyway in this repo)
## Iteration
For subsequent rc.N: add a regular changeset to main, bot regenerates
the release PR as `v4.5.0-rc.N`. Merge to ship.
## Exiting pre mode
When ready to ship stable: `pnpm exec changeset pre exit`, push, merge
regenerated PR. That publishes `4.5.0` under `latest` and fires the
marketing-site dispatch.
## Summary
Refocuses the v4.5.0 changeset and server-changes content on the
public-facing AI features story, replacing the pre-release-internal diff
framing that had accumulated in `.changeset/` and `.server-changes/`.
Pairs with the RC support PR — the next bot regeneration will pick up
this content.
## What's in here
### Changeset rewrites
- **`chat-agent.md` rewritten as the headline AI Agents entry** —
written from the `docs/ai-chat/` surface (not from internal pre-release
diffs). Covers useChat integration, multi-turn durability via Sessions,
lifecycle hooks, stop generation, tool approvals (HITL), pending
messages + background injection, actions, typed state primitives,
`chat.toStreamTextOptions()`, multi-tab coordination, network
resilience, and the first-turn fast path (`chat.headStart`).
- **New `ai-prompts.md`** — announces the Prompts feature publicly for
the first time. Code-defined templates, deploy-versioning, dashboard
overrides, AI SDK telemetry integration, `chat.agent` integration via
`chat.prompt.set()` + `chat.toStreamTextOptions()`, full management SDK.
- **`sessions-primitive.md` expanded** — calls out
`tasks.triggerAndSubscribe()` and `sessions.list` as standalone
primitives (not just chat.agent infrastructure).
- **`chat-agent-on-boot-hook.md` trimmed** — drops "if you previously…"
pre-release migration framing.
- **Deletes 4 changesets** that described pre-release-internal
migrations or were circular ("groundwork for the upcoming chat.agent" —
chat.agent ships in the same release).
### Server-changes rewrites (`.server-changes/`)
Five new entries for the dashboard surface of the AI feature set:
- Agents list page
- Agent Playground
- Sessions dashboard
- Prompts dashboard (list with usage sparklines + detail with template /
Generations / Metrics / Versions tabs + override UI)
- Models registry (provider-grouped catalog with cross-tenant usage
metrics)
- AI generation span inspector on run traces
- Runs list Task source filter (Standard / Scheduled / Agent)
- Run-detail Agent view (segmented control)
Each entry is 1–2 sentences, no bullets, no implementation file paths —
fits as a single bullet in a future changelog.
Three older `.server-changes/` files were merged or split into the
cleaner taxonomy above and deleted.
## Out of scope
Non-AI-feature server-changes (admin-tabs, queue-length-cap fix,
worker-deployment race, streamdown upgrade, etc.) and changesets
(idempotency-key cap, sigsegv retry, locals-key fix, plugin auth, region
filters, etc.) are untouched.
## Summary
Enables shipping `X.Y.Z-rc.N` prereleases of `@trigger.dev/*` via
changesets pre mode. RCs publish under the `rc` npm dist-tag, never
claim `latest`, and don't trigger marketing-site changelog PRs. The
plumbing is hyphen-in-version detection in `release.yml` — no separate
workflow, no opt-in flag at publish time.
Validated end-to-end against a sandbox repo (real npm publishes, Docker
builds, Helm chart pushes, GitHub releases) before porting back. Full RC
lifecycle tested: pre enter → rc.0 → iterate to rc.1 → pre exit →
stable. Plus interaction with the existing release-branch hotfix flow.
## What changes
### `release.yml`
- New `is_prerelease` output (hyphen-in-version)
- GitHub release adds `--prerelease` flag for RC publishes (Pre-release
badge, not Latest)
- `dispatch-changelog` job gated on `is_prerelease != 'true'` — no
marketing-site PR per RC
### Docker workflows
- Removes the `:v4-beta` floating tag entirely from `publish-webapp.yml`
and `publish-worker-v4.yml`. v4 is GA; the tag is a misnomer and is
already inconsistent with the npm side (npm `v4-beta` dist-tag was
frozen at 4.0.4 months ago while Docker `:v4-beta` kept bumping).
Self-hosters should pin to a versioned tag going forward — the last
value of `:v4-beta` stays frozen wherever it currently points.
### CLI version-check fix
(`packages/cli-v3/src/utilities/initialBanner.ts`)
Switches the "new version available" comparison from JavaScript
`localeCompare` to `semver.lt`. The old comparison handled `X.Y.Z-rc.N`
vs `X.Y.Z` incorrectly — a user on `4.5.0-rc.0` would never be prompted
to upgrade once `4.5.0` stable shipped (lex order put the prerelease
ahead of the bare version). Real semver gets this right.
Stable users were never affected: the check queries the `@latest`
dist-tag, which by convention never points at a prerelease.
## How an RC actually publishes after this
1. `pnpm exec changeset pre enter rc` on main, push the `pre.json`
2. Bot regenerates the release PR as `chore: release v<X.Y.Z>-rc.0`
3. Merge → `release.yml` runs `changeset publish` which reads
`pre.json.tag` and publishes under `--tag rc`. GitHub release marked
Pre-release. No marketing-site dispatch.
4. Iterate by adding changesets normally; bot bumps to `rc.1`, `rc.2`, …
5. When ready: `pnpm exec changeset pre exit`, push, merge regenerated
PR → stable ships under `latest` and the marketing-site dispatch fires.
## Summary
Adds a Region column and Region filter (under More filters) to the runs
list dashboard, the same filter on the public runs list API
(`filter[region]`), and a matching `region` input on the MCP `list_runs`
tool. Each run's executing region is also surfaced as a new optional
`region` field on the runs list and run retrieve responses, populated
from the worker instance group's `masterQueue` identifier.
Useful when you run tasks across multiple regions and want to slice the
runs list — or your existing run-querying scripts — by where the run
actually executed.
## Design
The filter value in the URL / API is the `masterQueue` identifier (the
same string already persisted on `TaskRun` and replicated to ClickHouse
as `worker_queue`), so the query just becomes `worker_queue IN (...)`
with no server-side translation. The Region dropdown options come from a
new resource loader backed by `RegionsPresenter`, which now also exposes
`masterQueue` alongside the existing region metadata.
```ts
// public API
const runs = await runs.list({ region: ["us-east-1", "eu-west-1"] });
// each item: { id, status, ..., region?: "us-east-1" }
```
```ts
// MCP
list_runs({ environment: "prod", region: "us-east-1" })
```
## Summary
The trigger-task hotpath used to early-return without a DB query when a
caller passed both a queue override and a per-trigger TTL — the hottest
configuration on the trigger API. Adding `triggerSource` to the resolver
so the runs-list "Source" filter could distinguish STANDARD / SCHEDULED
/
AGENT runs removed those early-returns, costing +2 DB queries per
trigger
on non-locked calls and +1 on locked calls.
This change caches `BackgroundWorkerTask` metadata (`ttl`,
`triggerSource`,
`queueId`, `queueName`) in Redis so the resolver can satisfy every
caller
configuration with a single `HGET` on the warm path. PG fallback on miss
back-fills the cache.
Follow-up to #3542.
## Design
Two key spaces:
- `task-meta:env:{envId}` — the "current worker" view, refreshed at
every
deploy promotion. 24h safety TTL.
- `task-meta:by-worker:{workerId}` — used for `lockToVersion` triggers.
Immutable post-create. 30d sliding TTL so historical workers age out.
Cache writes use Lua scripts via `defineCommand` so `DEL` + `HSET` +
`EXPIRE` land atomically — concurrent readers never see the empty
intermediate state of a naive pipeline. Read-path back-fill uses
single-field upserts so concurrent back-fills don't wipe each other's
siblings.
The cache lives behind its own `TASK_META_CACHE_REDIS_*` env-var prefix
that falls back to the default `REDIS_*` set, so operators can route the
cache to a dedicated Redis instance if they want.
The service/instance file split (`taskMetadataCache.server.ts` for the
pure class, `taskMetadataCacheInstance.server.ts` for the env-wired
singleton) mirrors the existing `runsReplicationService` /
`runsReplicationInstance` pattern.
## Test plan
- [ ] `pnpm run typecheck --filter webapp`
- [ ] `pnpm run test ./test/engine/triggerTask.test.ts --run` — 8
existing tests untouched + 5 new tests covering warm cache, cold
miss with back-fill, queue + ttl path, by-worker vs env keyspace,
and the promotion cache write
- [ ] End-to-end against a dev worker: registering writes both keyspaces
with the expected TTLs, and `redis-cli HGETALL
"tr:task-meta:env:<envId>"`
returns the cached entries
## Benchmark
Measured `DefaultQueueManager.resolveQueueProperties` against a real
Postgres + Redis (vitest `containerTest`, single-host docker). 500
sequential calls and 2,000 parallel calls (concurrency=50) per scenario,
request shaped as `{ taskId, queue: "bench-queue", ttl: "5m" }` — the
hot path this PR restores.
```
sequential (one in flight at a time):
[noop cache (baseline)] n=500 mean=1.423ms p50=1.394ms p95=1.735ms p99=2.629ms max=11.100ms
[redis cache, cold ] n=500 mean=1.346ms p50=1.283ms p95=1.688ms p99=2.463ms max=5.058ms
[redis cache, warm ] n=500 mean=0.084ms p50=0.078ms p95=0.105ms p99=0.156ms max=1.129ms
speedup (warm vs baseline, sequential): 16.95x
parallel (concurrency=50):
[noop cache (baseline)] n=2000 mean=10.069ms p50=8.850ms p95=14.718ms p99=31.887ms total=405ms ops/s=4,940
[redis cache, warm ] n=2000 mean=0.614ms p50=0.568ms p95=1.189ms p99=1.432ms total=25ms ops/s=80,389
throughput speedup (warm vs baseline, parallel): 16.27x
```
Read:
- **Warm cache cuts resolver latency 17×** at p50 — from ~1.4 ms to ~78
µs per call.
- **Cold cache is on par with baseline** — the extra `HGET` miss adds
<50 µs against the two Postgres queries that follow, so the worst case
is not worse than today.
- **Under burst load (50 concurrent triggers)**, the baseline's p99
jumps to ~32 ms as Postgres connections queue up; warm stays at ~1.4 ms.
The cache moves the saturation point from ~5k ops/s (PG pool) to ~80k
ops/s (single-client Redis pipelining).
Caveats: single-host docker, local Postgres + Redis, resolver-only
measurement (excludes the rest of the trigger transaction). Prod adds
region-local Redis RTT (~0.3–0.8 ms) which shifts warm absolute numbers
up but keeps the ratio intact.
## Summary
A "Google auth conflict" Sentry alert fires whenever a user signs in via
Google whose Google account is linked to one user row but whose
Google-provided email is now on a *different* user row. The handler in
`apps/webapp/app/models/user.server.ts:236` already does the right thing
— it returns the existing auth-linked user and skips the update path so
neither row gets mutated — but it logs the situation with
`logger.error`, which routes to Sentry as an exception and pages the
on-call channel.
There's no exception to chase here: the branch is the intended outcome
for a known data shape (user changed their email on one account after
originally signing up via Google on another). Downgrading the call to
`logger.warn` keeps the diagnostic record in our logs (with all the same
context fields — email, both user IDs, authIdentifier) but stops it
firing the production error alert.
## Change
- `logger.error` → `logger.warn` for the conflict branch in
`findOrCreateGoogleUser`. Context payload is unchanged.
## Test plan
- [x] Typecheck only — there's no behavioural change to test, the log
level is the entire diff.
## Summary
`LocalsKey<T>` (the type returned by `locals.create()`) was branded with
a
module-level `declare const __local: unique symbol`. Each such
declaration
is its own nominal type, and `tshy` emits separate `.d.ts` files for the
ESM and CJS outputs — each gets its own `__local` symbol. Under certain
pnpm hoisting layouts a single TypeScript compilation can resolve
`LocalsKey` from both the ESM source path and the CJS dist path within
the same call site, producing two structurally-incompatible variants of
the same type. TS surfaces this as the misleading error:
```
Argument of type 'LocalsKey<X>' is not assignable to parameter of type
'LocalsKey<X>'. Property '[__local]' is missing in type 'LocalsKey<X>'
but required in type 'BrandLocal<X>'.
```
The error has been hitting CI on PRs opened since the chat.agent stack
landed (e.g. #3625 typecheck job), but doesn't reproduce on developer
machines where the pnpm node_modules layout was built up incrementally.
## Fix
Replace the `unique symbol` brand with an optional phantom field that
carries `T` at the type level:
```ts
// before
declare const __local: unique symbol;
type BrandLocal<T> = { [__local]: T };
export type LocalsKey<T> = BrandLocal<T> & {
readonly id: string;
readonly __type: unique symbol;
};
// after
export type LocalsKey<T> = {
readonly id: string;
readonly __type: symbol;
/** Phantom carrier for the value type — never read at runtime. */
readonly __valueType?: T;
};
```
The ESM and CJS `.d.ts` outputs now produce structurally identical
types,
so cross-output resolution no longer produces a mismatch. `T` is still
carried at the type level via the optional phantom field. The runtime
shape is unchanged — `manager.ts` was already casting via `as unknown`,
which is no longer needed.
## Test plan
- [ ] `pnpm run typecheck --filter @trigger.dev/core --filter
@trigger.dev/sdk`
- [ ] `pnpm run build --filter @trigger.dev/core --filter
@trigger.dev/sdk`
(clean rebuild) — confirms the ESM and CJS dist `.d.ts` outputs
no longer carry distinct `unique symbol` declarations
- [ ] `pnpm --filter @trigger.dev/core test test/mockTaskContext.test.ts
--run`
- [ ] `pnpm --filter @trigger.dev/sdk test test/mockChatAgent.test.ts
--run`
Follow-up to #3615. The `code` filter currently fires typecheck for any
change outside `docs/`, `.changeset/`, `hosting/`, or `.github/` - so a
docs-only PR like #3623 (touching `references/ai-chat/.env.example` +
`README.md`) triggered the typecheck job. None of the `references/*`
packages declare a `typecheck` script either, so even when a real code
change lands there, `turbo run typecheck` skips them. Running the job is
pure cost.
Tightens the filter to also exclude:
- `references/**` - playground projects, none of them contribute to
`turbo run typecheck` today
- `**/*.md` - markdown anywhere
- `**/.env.example` - example env files anywhere
Two known gaps left open:
- references/ have no real CI typecheck coverage. Separate question -
either add `typecheck` scripts to each (or top-level `tsc -p`), or
accept playground status.
- `changes` job still runs (it's a path-filter step) but the dependent
jobs all skip on irrelevant PRs.
## Summary
A complete Next.js reference project that exercises `chat.agent`
end-to-end, plus the CLI MCP tools that let Claude Code, Cursor, and
similar IDE agents drive a deployed `chat.agent` task from the editor.
Builds on #3545.
## Design
`references/ai-chat` is a full Next.js app: prisma-backed persistence,
multi-chat sidebar, per-chat model picker, debug panel, tool examples
(`getCurrentTime`, `searchHackerNews`, `createGithubIssue`, PR review
helpers, code sandbox), and smoke tests. It's intended both as a
copy-paste starting point and as a place to regression-test SDK changes.
The CLI gains MCP tools (`start_agent_chat`, `send_agent_message`,
`close_agent_chat`, `list_agents`) so an IDE agent can converse with a
deployed `chat.agent` task. The dev runtime adds one-shot OOM kill on
the run controller and skills bundling in the build pipeline.
Top of the chat.agent stack: a full Next.js reference project that
exercises chat.agent end-to-end, plus the CLI MCP tools that drive
agent runs from Claude Code / Cursor / etc.
references/ai-chat:
- Full Next.js app with prisma persistence, multi-chat sidebar,
per-chat model picker, debug panel, tool examples, smoke tests
- Reference tools: getCurrentTime, searchHackerNews, createGithubIssue,
PR review helpers, code sandbox
- chat-client-test orchestrator for concurrent-send stress
- references/hello-world chatAgent + triggerAndSubscribe examples
CLI MCP tooling for chat.agent:
- mcp/tools/agentChat.ts (start_agent_chat, send_agent_message,
close_agent_chat)
- mcp/tools/agents.ts + tasks.ts (list agents, agent run details)
- dev-run-controller OOM kill + taskRunProcessPool tweaks
- dev/managed entry-point hooks for skills bundling
- buildWorker + bundleSkills (agent skills support)
Includes ai-tool-helpers + mcp-agent-chat-sessions changesets, plus
the streamdown@2 patch and pnpm-lock reconciliation.
(Will be renamed to feature/ai-chat-reference-and-cli before push.)
fix(cli): preserve lastEventId after sendMessage fallback to avoid stale turn-complete replay
## Summary
A chat-aware run inspector and a `/playground` UI for testing
`chat.agent` tasks interactively. Builds on #3543's runtime.
## Design
The run inspector grows a new tab that renders the conversation chain
for any `chat.agent`-kind run. It subscribes to the run's session
streams, threads chat parts through a per-message renderer, and uses a
shared markdown + Shiki component for code highlighting (also used by
the test-payload panel).
The playground is a standalone `/playground` route that lets you drive a
deployed chat agent from the dashboard — pick a task, send messages,
watch tool calls render, and see span detail on every turn. The matching
`/agents` list view shows all deployed agents in the project.
Concurrent `POST /api/v1/deployments` requests for the same environment
race on the `WorkerDeployment(environmentId, version)` unique
constraint. Both requests read the same latest deployment via
`findFirst`, compute the same next version via
`calculateNextBuildVersion`, and both attempt
`prisma.workerDeployment.create()` — one wins, the other crashes with
Prisma `P2002`. The bug is a classic TOCTOU between the version read and
the version write; it's been latent since the version-assignment logic
was first added but only fires when two deploys land within milliseconds
of each other (CI matrices, retried CLI calls, webhook-triggered
redeploys).
## Approach
Extracts the version assignment + create into a small helper
`createDeploymentWithNextVersion`
(`apps/webapp/app/v3/services/initializeDeployment/createDeploymentWithNextVersion.server.ts`).
The helper retries on `P2002 (environmentId, version)` up to 5 times
with randomised 5–50ms jitter so N concurrent racers don't loop in
lockstep. Each attempt re-reads the latest version, recomputes via
`calculateNextBuildVersion`, and re-runs the caller's `buildData`
callback so version-dependent fields (image ref tag, friendlyId) are
always consistent with the version actually persisted. A `logger.warn`
fires per collision so the retry rate is observable in production logs.
When retries are exhausted, the helper throws a dedicated
`DeploymentVersionCollisionError` carrying `environmentId`, `attempts`,
and `lastAttemptedVersion`, with the original
`PrismaClientKnownRequestError` attached as `cause`. Sentry walks the
`cause` chain natively, so contention exhaustion shows up as a
distinguishable wrapper exception linked to the underlying `P2002`
rather than a generic unique-constraint violation that looks identical
to every other duplicate-key bug.
The behavioural change is limited to "catch P2002 and retry instead of
crashing." The image ref computation stays inside the builder callback
(same call site as before the refactor), so ECR / non-ECR behaviour, S2
stream creation order, and all downstream side effects are unchanged.
## Non-goals
- No new database migrations, no schema changes, no isolation-level /
locking changes. A serialisable transaction or advisory lock would also
fix this; retry-on-conflict is the smaller change that keeps the
existing version-allocation logic intact.
- Does not touch the analogous `calculateNextBuildVersion` call in
`createBackgroundWorker.server.ts`, which likely has the same race shape
against `BackgroundWorker`'s unique constraint — flagged as a follow-up.
## Test plan
- [x] `pnpm run typecheck --filter webapp` passes (no new errors in the
modified files).
- [x] Three real-Postgres tests in
`apps/webapp/test/createDeploymentWithNextVersion.test.ts` via
`containerTest`:
- 5 concurrent calls all produce distinct, persistable versions
(`Set(versions).size === concurrency`). The naive read-then-create
version of the helper fails this test with the exact same `P2002` seen
in production; the retry version passes.
- Non-`P2002` errors raised from the `buildData` callback propagate
immediately without retry, builder invoked exactly once.
- With `maxRetries: 0`, concurrent racers surface the wrapped
`DeploymentVersionCollisionError` (not a raw `P2002`); `environmentId`,
`attempts`, `lastAttemptedVersion` are populated and `error.cause.code
=== "P2002"`.
- [x] Existing `apps/webapp/test/getDeploymentImageRef.test.ts` still
green (the file was untouched in the final diff).
## Follow-ups (not in this PR)
- `createBackgroundWorker.server.ts` likely has the same TOCTOU shape
against its background-worker version unique constraint — should use the
same helper.
- Sentry visibility check: confirm `error.cause` chain renders as a
linked exception in the Sentry UI when the wrapped error fires (requires
a sandboxed triggering of the exhaustion path).
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
## Summary
Adds `chat.agent({...})`, a durable conversational task runtime, plus
the browser-side `TriggerChatTransport` + `AgentChat` that drive it from
a React or Next.js app. Conversations survive page refreshes, network
blips, idle suspend, and process restarts, with built-in tools, HITL
approvals, multi-turn state, and stop-mid-stream cancellation. Builds on
#3542.
## Design
Each `/in/append` request carries at most one new message. The agent
reconstructs prior history at run boot from an object-store snapshot
plus a `session.out` replay tail, so conversation context lives
server-side instead of bloating the wire. Awaited snapshot writes after
every `onTurnComplete` keep the chain durable across idle suspend.
Registering `hydrateMessages` short-circuits both paths for customers
who own their own conversation store.
Lifecycle hooks — `onChatStart`, `onTurnStart`, `onTurnComplete`,
`onAction`, `onValidateMessages`, `hydrateMessages` — cover validation,
persistence, and post-turn work. `chat.history` exposes read primitives
(`getPendingToolCalls`, `getResolvedToolCalls`, `extractNewToolResults`,
`findMessage`, `all`) for HITL flows. `chat.local` gives per-run typed
state with Proxy access and dirty tracking. `chat.headStart` bridges
first-turn TTFC via a customer HTTP handler. `oomMachine` opts a chat
into one-shot OOM-retry on a larger machine.
`TriggerChatTransport` is a `Transport` implementation for Vercel's
ai-sdk `useChat`: delta-only wire sends, SSE reconnection with
`lastEventId` resume, stop/abort cleanup, dynamic `accessToken` refresh,
`X-Peek-Settled` fast-close. `AgentChat` is the direct programmatic
equivalent. A cross-tab coordinator does leader election so multiple
open tabs share a single SSE.
```ts
import { chat } from "@trigger.dev/sdk/ai";
import { streamText } from "ai";
export const myChat = chat.agent({
id: "my-chat",
run: async ({ messages, signal }) =>
streamText({ model: openai("gpt-4o"), messages, abortSignal: signal }),
});
```
Adds the chat.agent({...}) task definition (server runtime) and the
browser-side TriggerChatTransport + AgentChat that drives it from a
React or Next.js app. The runtime sits on top of the Sessions primitive
and handles the durable conversational task lifecycle.
Server runtime:
- chat.agent({...}) — session-aware task definition
- Lifecycle hooks: onChatStart, onTurnStart, onTurnComplete, onAction,
onValidateMessages, hydrateMessages
- chat.history read primitives for HITL flows
- chat.local, chat.headStart, chat.handover, oomMachine
- Delta-only wire + S3 snapshot reconstruction at run boot
- Actions are no longer turns
Browser transport:
- TriggerChatTransport (ai-sdk Transport): delta-only wire sends,
SSE reconnection with lastEventId resume, stop/abort cleanup,
dynamic accessToken refresh
- AgentChat: direct programmatic API
- useTriggerChatTransport (React hook)
- chat-tab-coordinator: cross-tab leader election
Includes the chat-agent, chat-agent-delta-wire-snapshots,
chat-history-read-primitives, chat-head-start, chat-actions-no-turn,
chat-session-attributes, agent-skills, and mock-chat-agent-test-harness
changesets.
Adds a Mon 08:00 UTC workflow that posts a summary of open Dependabot
alerts and PRs to Slack. Uses env-scoped secrets so the alerts PAT and
Slack token are only available to this workflow.
The `code` paths filter currently matches `**` minus a tiny exclusion
list, so a PR that only touches `.github/workflows/*.yml` still flips
`code == true` and runs typecheck (~2 min on the runner).
Exclude `.github/**` from `code`, then re-include just `pr_checks.yml`
and `typecheck.yml` so a change to either of those still triggers the
full code check matrix.
Effect:
- workflow-only PRs (this one, future dependabot/codeql/etc.) skip
typecheck; `all-checks` treats the skipped job as non-failure so the
required status passes.
- modifying `pr_checks.yml` or `typecheck.yml` themselves still triggers
typecheck.
- the existing per-suite filters (`webapp`, `packages`, `internal`,
`cli`, `sdk`) already re-include the specific workflows that gate them,
so they're unaffected.
## Summary
A `/sessions` dashboard for inspecting durable Sessions, an `AGENT` /
`SCHEDULED` task-kind filter for the runs list, and the server-side
hardening (rate-limit exemption for packets, retry-with-backoff on
stream appends, typed too-large-chunk error) that the `chat.agent`
runtime in #3543 needs. Builds on the Sessions primitive shipped in
#3417.
## Design
The Sessions list + detail routes mirror the run inspector pattern.
`TaskTriggerSource` gains `AGENT` and `SCHEDULED` values, persisted on
`BackgroundWorker.taskKind` and `TaskRun.taskKind` (plus a matching
Clickhouse column), so the runs list can filter by kind.
New `@trigger.dev/core` modules — `sessionStreams`, `inputStreams`, a
`sessionStreamInstance` for realtime streams, and the
`realtime-streams-api` / `session-streams-api` surfaces — expose the
typed shapes that chat.agent will use to drive `session.out`.
`ChatChunkTooLargeError` lets the runtime drop oversized chunks with a
typed surface instead of failing the run. `s2Append` retries transient
failures with exponential backoff. `/api/v[12]/packets/*` is exempt from
customer rate limits so chat snapshot reads and writes don't get
throttled under load.
## Stack
Part of a 4-PR stack. Merge bottom-up.
1. **This PR** (#3542) → `main`
2. #3543 → #3542 — `chat.agent` runtime + browser transport
3. #3545 → #3543 — agent-view dashboard
4. #3546 → #3545 — ai-chat reference + MCP tooling
Replaces #3173 (closed).
<!-- GitButler Footer Boundary Top -->
---
This is **part 5 of 5 in a stack** made with GitButler:
- <kbd> 5 </kbd> #3612
- <kbd> 4 </kbd> #3546
- <kbd> 3 </kbd> #3545
- <kbd> 2 </kbd> #3543
- <kbd> 1 </kbd> #3542👈
<!-- GitButler Footer Boundary Bottom -->
`tasks.trigger`, `tasks.batchTrigger`, `batch.create`,
`wait.createToken`, `wait.forDuration`, and the input/session stream
waitpoint endpoints all accept a caller-supplied `idempotencyKey` and
store it verbatim against a composite-unique index on `TaskRun`,
`BatchTaskRun`, or `Waitpoint`. The schemas had no length cap, so a
sufficiently long high-entropy key produced an index row larger than the
underlying storage layer can hold. The insert failed at the database,
and the caller saw a generic 500 from
`RunEngineTriggerTaskService.call()` / `CreateBatchService` / waitpoint
creation, depending on the endpoint.
Keys produced by `idempotencyKeys.create()` are 64-character SHA-256
hashes and never trip this — it only manifests for direct REST callers
(or SDK callers passing a raw string they generated themselves).
Low-entropy keys also sail through, because the storage layer compresses
repeated bytes before they reach the index, which is why the failure
mode is intermittent and tied to caller-side key shape.
## Fix
Add `.max(2048, "<field> must be 2048 characters or less")` to the seven
schemas that feed an indexed `idempotencyKey` column:
- `TriggerTaskRequestBody.options.idempotencyKey`
- `BatchTriggerTaskItem.options.idempotencyKey`
- `CreateBatchRequestBody.idempotencyKey`
- `CreateWaitpointTokenRequestBody.idempotencyKey`
- `CreateInputStreamWaitpointRequestBody.idempotencyKey`
- `CreateSessionStreamWaitpointRequestBody.idempotencyKey`
- `WaitForDurationRequestBody.idempotencyKey`
Plus the `idempotency-key` HTTP header on the trigger route (and the
three batch routes that re-export `HeadersSchema`). The header schema is
lifted out of `api.v1.tasks.$taskId.trigger.ts` into
`apps/webapp/app/v3/triggerHeaders.server.ts` so it can be exercised in
tests without dragging the route's import-time side effects.
The 2048 character ceiling is chosen to sit safely under the per-row
index limit while staying generous against existing callers — keys that
fit before still fit. Oversized keys now return a structured Zod 400
instead of a generic 500.
Limit is documented under `Idempotency key` in `docs/limits.mdx` and as
a `<Note>` on `docs/idempotency.mdx`.
## Test plan
- [x] 15 schema unit tests added
(`packages/core/src/v3/schemas/idempotencyKey.test.ts`,
`apps/webapp/test/routes/triggerHeaders.test.ts`) —
rejection-with-message + boundary acceptance for each capped schema. The
webapp test exercises the extracted `TriggerHeadersSchema` directly with
no mocks.
- [x] `pnpm run build --filter @trigger.dev/core`
- [x] `pnpm run typecheck --filter webapp`
- [x] End-to-end verified locally: baseline (small key) → 200; 3000-char
high-entropy header → 400 with the expected Zod error; same key at the
2048 boundary → 200; same key with the cap reverted → the database
rejected the insert and the route returned 500 to the caller. Cap
restored.
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds Sessions, a durable, run-aware stream primitive that scopes
session.in / session.out records to a session (not a single run).
Records survive run boundaries; reconnect-from-last-event-id is built in.
Server foundation:
- New /realtime/v1/sessions/:session/:io/append + /records routes
- sessionRunManager + sessionsRepository + clickhouseSessionsRepository
- mintRunToken for short-lived per-session tokens
- s2Append retry-with-backoff + undici cause diagnostics
- /api/v[12]/packets/* exempt from customer rate limits
- BackgroundWorker schema gains taskKind enum (TASK, AGENT, SCHEDULED)
- TaskRun.taskKind column + clickhouse 029_add_task_kind_to_task_runs_v2
Core types:
- new sessionStreams, inputStreams, realtimeStreams packages in @trigger.dev/core
- session-streams-api / realtime-streams-api surface
Sessions dashboard UI (the primitive's own viewer):
- /sessions index + detail routes
- SessionsTable, SessionFilters, SessionStatus, CloseSessionDialog
- AGENT/SCHEDULED filter in RunFilters + TaskTriggerSource
Includes the sessions-primitive changeset.
Switching between the Users and Organizations tabs in the admin
dashboard now keeps the current `?search=` value, so you can flip
between the two without re-typing your filter. Other admin tabs don't
take `search` and so don't carry it.
`pr_checks` runs the full matrix on every PR. #3609 touched only
`apps/webapp/app/routes/admin.tsx` and still ran the 4-job CLI e2e
matrix and 5-job sdk-compat suite.
Adds a `changes` job using `dorny/paths-filter` and gates each tier:
- webapp + e2e-webapp: `apps/webapp/**`, `packages/**`,
`internal-packages/**`
- packages: `packages/**`
- internal: `internal-packages/**` + `packages/**` (cross-deps)
- e2e (cli-v3): `packages/{cli-v3,build,core,schema-to-json}/**`
- sdk-compat: `packages/{trigger-sdk,core}/**`
`.configs/**`, `package.json`, `pnpm-lock.yaml`, `pnpm-workspace.yaml`,
`turbo.json` are also included in every filter since they affect the
whole workspace.
Inlines the `units` reusable-workflow children so each can be gated
independently (status check names also flatten from `units / webapp /
...` to `webapp / ...`). `unit-tests.yml` is unaffected - still used by
`publish.yml`.
Adds an `all-checks` gate that always runs and short-circuits to success
when every dependent is success-or-skipped. With this in place a single
required status check (`All PR Checks`) is enough; before this,
`paths-ignore` would have left required checks Pending on docs/changeset
PRs ([gh
docs](https://docs.github.com/en/actions/managing-workflow-runs/skipping-workflow-runs)).
## Summary
- Recommend deploying NodeLocal DNS and lowering `ndots` to `1` in the
Kubernetes self-hosting guide.
- Recommend storing task events in ClickHouse
(`EVENT_REPOSITORY_DEFAULT_STORE=clickhouse_v2`) in both the Docker and
Kubernetes guides, plus a new row in the webapp env var reference.
## Summary
- Users on production are hitting `QuotaExceededError: Failed to execute
'setItem' on 'Storage'` when navigating runs, because their localStorage
is full of orphaned `panel-group-react-aria<n>-:<rid>:` entries.
- Each entry is a session-unique key written by the resizable panel
library; they accumulated to thousands per user over the last two months
and now block legitimate `setItem` calls (the run-view inspector can no
longer persist its layout, and the page crashes mid-render).
- This PR evicts the legacy entries once on client boot. The leak itself
is already plugged by the v1.1.3 upgrade in #XXXX — this is the cleanup
that recovers the wasted quota on existing users' machines.
## Root cause (already fixed, for context)
In v0.4.1 of the underlying library, `PanelGroupImpl` defaulted
`autosaveStrategy` to `"localStorage"` unconditionally — so *every*
`PanelGroup` wrote to localStorage on every autosave trigger, including
the four in `QueryEditor`, the one in `ReplayRunDialog`, the storybook
routes, etc. Without an `autosaveId`, the key fell back to
`panel-group-${useId()}`, and React Aria's `useId()` produces a new
session-unique prefix each visit. Result: entries accumulated without
bound across sessions.
The condition was introduced when
[#3282](https://github.com/triggerdotdev/trigger.dev/pull/3282) removed
the wrapper's explicit `autosaveStrategy="cookie"` override (to fix HTTP
431 cookie-size errors). That worked, but the library default that took
over silently caused this leak.
The v1.1.3 upgrade in the resizable-panel PR changed the default to
`autosaveStrategy = autosaveId ? "localStorage" : undefined`, so no new
entries are being written. Existing residue still needs to be removed
from users' browsers.
## Changes
- New file
[`apps/webapp/app/clientBeforeFirstRender.ts`](apps/webapp/app/clientBeforeFirstRender.ts)
— exports a `clientBeforeFirstRender()` function that runs
synchronously, before React hydrates. Encapsulates a small cleanup
helper that scans `localStorage` and removes:
- Every key starting with `panel-group-react-aria` (the legacy
auto-generated keys).
- The orphan `panel-run-parent-v2` key from before the autosaveId v2→v3
bump.
- [`apps/webapp/app/entry.client.tsx`](apps/webapp/app/entry.client.tsx)
— imports and invokes `clientBeforeFirstRender()` once, before
`hydrateRoot()`. This guarantees the cleanup completes before any
`ResizablePanelGroup` mounts and tries to write.
The cleanup is wrapped in `try/catch` so private-browsing /
disabled-storage scenarios fail silently. Idempotent: subsequent loads
find no matching keys and exit immediately.
## Test plan
- [x] Locally seed ~50 fake `panel-group-react-aria…` entries plus a
`panel-run-parent-v2` entry via DevTools console, hard reload → legacy
entries gone, real entries (`panel-run-parent-v3`, `panel-run-tree`)
preserved.
- [x] Idempotency: reload a second time, no errors, no state changes.
- [x] Add a control entry (`panel-run-parent-v3-but-different-suffix`) —
confirmed not over-matched.
- [x] Simulate broken `Storage.setItem` throwing — page still renders,
cleanup swallows the error.
- [x] Typecheck clean.
## Notes
- Customer report: `QuotaExceededError: Failed to execute 'setItem' on
'Storage': Setting the value of 'panel-run-parent-v3' exceeded the
quota.`
- The cleanup runs once per page load. Once a user has loaded the app
after this deploys, their localStorage is clean and the function becomes
a no-op forever.
## Summary
Follow-up to #3561. The drift-audit workflow timed out on PR #3542 (92
files, +5962 lines) by hitting `--max-turns 15` before reaching a
verdict, leaving a red ❌ on that PR with no sticky comment.
## Changes
- `--max-turns` bumped from 15 to 30.
- Prompt now opens with an explicit "Strategy" section: read REVIEW.md
once, scan the file-list only, open at most 5 files (3-5 on PRs >50
files), and bias toward finishing over exploring.
- Final rule: *"when in doubt between one more file read and finish now
— finish now."*
The audit is allowed to miss things. It is not allowed to time out and
leave a red X.
## Test plan
- [ ] Verify this PR's audit posts `✅ REVIEW.md looks current for this
PR.` (small diff)
- [ ] After merge, retry the audit on #3542 or a similarly large PR and
confirm it completes
## Summary
Local ClickHouse was burning ~325% CPU endlessly merging its own
telemetry tables (`metric_log`, `asynchronous_metric_log`, `part_log`,
`trace_log`) after the container had been running long enough to
accumulate hundreds of GB of system-log data. OrbStack Helper reflected
this on the host (~400% CPU).
These tables are not used by anything in the dev stack. They only exist
for ClickHouse to log itself, so disabling them eliminates the merge
churn entirely.
## Changes
- Adds `docker/config/clickhouse-disable-system-logs.xml`, mounted into
`/etc/clickhouse-server/config.d/`, that removes the noisy system log
tables via `<table remove="1"/>`.
- Mounts the override file in `docker/docker-compose.yml`.
After applying, idle CPU dropped from 325% to ~12% on my machine.
## Test plan
- [ ] `pnpm run docker` brings up the stack cleanly
- [ ] `docker stats clickhouse` shows low idle CPU
- [ ] App functionality unaffected (system log tables are not queried by
the webapp)
## Summary
Queues that use concurrency keys can no longer bypass the per-queue
length cap, and the "Queued | Running" columns in the dashboard now show
the true total across all CK variants instead of 0.
The cap and the dashboard both relied on `ZCARD` of the base queue key,
but CK-keyed runs live under `<base>:ck:<variant>` keys. Any queue that
used concurrency keys read 0 — letting a single CK variant grow
unbounded past the user's configured cap.
## Fix
Two per-base-queue counters are maintained inside the CK Lua scripts:
`<base>:lengthCounter` and `<base>:runningCounter`. Non-CK
enqueue/dequeue paths are untouched.
Counters are lazy-initialized the first time a CK enqueue (or nack)
lands on a queue: the Lua script sums `ZCARD` across the variants
tracked by `ckIndex`, sets the counter, then `INCR`s. Pre-existing CK
backlog on already-populated queues is captured automatically — no batch
migration required.
`INCR`/`DECR` is gated on `ZADD`/`SADD` returning 1 (a new entry vs an
idempotent no-op), so duplicate enqueues or re-dequeues don't inflate
the counter.
The counter is `SET` with a 24-hour TTL on init. `INCR`/`DECR` do not
extend the TTL, so the counter expires daily and the next CK operation
re-seeds it from `ckIndex`. This bounds any drift that accumulates
during the rolling-deploy overlap window — where old (un-Tracked) and
new (Tracked) webapp instances briefly coexist — to ≤24 hours, with no
admin sweep or background reconciler needed.
Read paths pipeline `ZCARD`/`SCARD` on the base key + `GET` on the
counter and sum. A missing counter is treated as 0, so pure non-CK
queues see the same answer as before.
The counter-aware scripts ship alongside the originals with a `Tracked`
suffix for rolling-deploy safety; a follow-up PR will drop the originals
once this has rolled out.
## Test plan
- [ ] `pnpm run test --filter @internal/run-engine` — 116 tests pass,
including a new `ckCounters.test.ts` covering lazy init from
pre-existing backlog, churn, floor-at-zero, the non-CK regression case,
mixed CK + non-CK on the same base queue, idempotent re-enqueue
(ZADD-already-exists), 24h TTL on the counter, and nack re-seeding after
counter expiry.
- [ ] Verified end-to-end against a live local environment:
- Triggered 24 CK enqueues across 4 variants → `lengthCounter=16`,
`runningCounter=8`, dashboard showed Queued=16 / Running=8 for the CK
queue.
- Set the env queue cap to 16, triggered 12 more enqueues → 8 succeeded,
4 rejected with `QueueSizeLimitExceededError`.
- Deleted the counter on a queue with 31 messages already sitting in CK
variants, triggered one more enqueue → counter materialized to 31 from
the `ckIndex` sum, then INCR'd.
## Summary
Consolidates the webapp's authentication and authorization into a small
set of route helpers, replacing the ad-hoc `requireUser` /
`requireUserId` / `authenticatedEnvironmentForAuthentication` calls
scattered across routes. Same security model, but the per-request flow
(authenticate → authorize → load) now lives in one place per route
family.
Introduces a plugin seam (`@trigger.dev/plugins`) that lets the cloud
build install a richer RBAC implementation without touching webapp code.
The OSS fallback keeps the pre-RBAC permissive behaviour intact, so
self-hosted deployments work unchanged.
Adds a comprehensive end-to-end auth test suite that didn't exist before
— 193 `it()` blocks (vitest reports ~199 after `it.each` expansion)
covering API key, PAT and JWT auth across the public API surface, plus
dashboard session auth for admin pages.
## Changes
### Plugin contract — `@trigger.dev/plugins`
`RoleBaseAccessController` interface authoritative for both OSS
(fallback) and cloud (enterprise plugin):
- `authenticateBearer(request, { allowJWT? })` — API-key / public-JWT
auth, returns env + ability
- `authenticateSession(request, { userId, organizationId?, projectId?
})` — dashboard auth, caller resolves `userId` from the session cookie
and passes it in (no `helpers.getSessionUserId` callback — decouples the
plugin host from session-cookie code)
- `authenticatePat(request, { organizationId?, projectId? })` — PAT
auth, returns identity + `lastAccessedAt` so the host can throttle the
per-request update
- `authenticateAuthorize*` variants for the auth-and-check-in-one-call
cases
- `isUsingPlugin(): Promise<boolean>` — capability flag for UI /
branching where plugin-present-ness matters; replaces the
sentinel-string coupling that had `personalAccessToken.server` matching
`"RBAC plugin not installed"` literally
### Dashboard auth (started, partial rollout)
Admin and settings pages migrated to a unified `dashboardLoader` /
`dashboardAction` helper that authenticates the session, runs an
authorization check, and exposes the result to the route. Other
dashboard routes still on the old pattern; remaining migration tracked
in TRI-8730.
Migrated routes:
- `admin.*` (14 admin / back-office / feature-flags / LLM-models /
notifications / orgs / concurrency pages)
- `_app.orgs.$organizationSlug.settings.team`
- `_app.orgs.$organizationSlug.settings.roles`
### API / realtime / engine auth (complete for the migrated families)
71 routes migrated to a unified `apiBuilder` that centralizes Bearer /
PAT / Public-JWT authentication and applies the per-route authorization
check before the handler runs. Includes:
- `api.v1.*` and `api.v2.*` and `api.v3.*` — tasks, runs, batches,
queues, prompts, deployments, query, sessions, waitpoints, packets,
workers, idempotency keys
- `realtime.v1.*` — runs, batches, sessions, streams
- `engine.v1.*` — dev / worker-action protocols
29 routes still on the legacy `authenticateApiRequest*` helpers —
tracked as a post-deploy follow-up in TRI-9228.
Multi-resource auth direction is now explicit at the call site via
`anyResource(...)` (OR) and `everyResource(...)` (AND). Bare arrays no
longer typecheck — fixes a class of bug where a JWT scoped to one
resource could implicitly access others under OR semantics.
PAT auth path consolidated: was three DB queries per request (legacy
`authenticateApiRequestWithPersonalAccessToken` findFirst +
`rbac.authenticatePat` join + `lastAccessedAt` update). Now one query in
the steady state — plugin returns `lastAccessedAt`, host smart-skips the
update via JS-side throttle when fresh.
Side effect: action aliases preserved historic JWT scope semantics where
the new model is stricter (e.g. a `write:tasks` JWT now also satisfies
`trigger` / `batchTrigger` / `update` actions on the same resource —
matched at the auth boundary, not in the route handler).
### Backwards-compat fixes
The strict-match model regressed several real-world JWT shapes. Each
preserved via explicit `anyResource(...)` entries in the route's authz
block:
- **Batch retrieve routes** (`api.v1.batches.$batchId`, `api.v2.*`,
`realtime.v1.batches.*`) accept `read:runs` JWTs again (pre-RBAC
literal-match superScope behaviour)
- **Runs list routes** (`api.v1.runs`, `realtime.v1.runs`) accept
type-level `read:tasks` / `read:tags` on unfiltered queries (matched the
legacy `Object.keys` iteration semantic)
- **PAT/OAT auth shape** normalized through `toAuthenticated` so all
auth methods return the same slim `AuthenticatedEnvironment` (was:
API-key returned the slim shape but PAT/OAT returned raw Prisma
`Decimal` / no `orgMember`)
- **Scope `:` preservation** in resource ids — `read:tags:env:staging`
now correctly identifies the tag id as `env:staging`, not `env`
### Slim `AuthenticatedEnvironment`
Extracted to `@trigger.dev/core/v3/auth/environment` — a structural
shape independent of `@trigger.dev/database`. The plugin contract
returns this; webapp consumers import from there; the cloud plugin
(Drizzle) returns the same shape without Prisma's `Decimal` class
leaking into the public surface. Lets internal-packages (run-engine,
etc.) refer to `AuthenticatedEnvironment` without pulling Prisma in.
### Auth test suite (new — `*.e2e.full.test.ts`)
193 e2e tests run against a real spawned webapp + Postgres (no mocks).
Coverage matrix:
- **API key auth** — read / write / trigger / batchTrigger / deploy
actions across runs, batches, deployments, prompts, queues, query,
sessions, input-streams, waitpoints, tasks, idempotency keys; multi-key
resources (a run carries batch / tag / task identifiers — auth must
accept any matching scope)
- **Personal Access Token auth** — comprehensive matrix: scope match,
scope mismatch, missing scope, expired token, malformed token
- **Public JWT auth** — sub-vs-URL environment resolution, expired JWTs,
signature verification, scope checking, otu (one-time-use) token
semantics, branch-environment signing-key fallback
- **Dashboard session auth** — admin-only pages reject non-admins;
per-action gating
- **Cross-cutting edge cases** — revoked API key grace window, JWT
cross-environment isolation, MissingResource branch behaviour
### Hygiene cleanups
- Deleted dead `app/services/authorization.server.ts` (legacy
`checkAuthorization` + types — no live consumers post-migration) and its
orphaned test
- Dropped the never-populated `scopes` field from
`ApiAuthenticationResultSuccess`
- `scheduleEmail` moved out of `email.server.ts` into its own module —
breaks a `commonWorker → marqs/V1` import chain that was poisoning the
auth test graph
- OSS Roles page shows a deployment-aware empty state ("Roles aren't
available in this self-hosted deployment" vs the plan-upsell copy) via
`rbac.isUsingPlugin()`
- Team action handler: explicit per-intent ability gates
(`manage:billing` for purchase-seats, `manage:members` for set-role +
remove-member with self-leave carve-out)
### Cross-repo coordination
All public-package contract changes paired in `triggerdotdev/cloud#763`
(rbac-packages branch) — the enterprise plugin implements the same
`RoleBaseAccessController` interface against Drizzle.
## Test plan
- [x] `pnpm run typecheck --filter webapp` clean
- [x] `pnpm --filter webapp exec vitest run --config
vitest.e2e.full.config.ts` — 193/193 pass (requires Docker for
testcontainers)
- [x] Spot-check an authed API endpoint with a valid + invalid API key
against a local stack
- [x] Spot-check the migrated admin pages render and gate non-admins
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
## Summary
Adds `.claude/REVIEW.md` — a repo-specific source of truth for what AI /
agent code reviewers should treat as critical in this codebase
(rolling-deploy safety, hot-table indexes, recovery-path queries,
testcontainers usage, etc.). Pairs with a Claude-based PR audit that
flags drift between REVIEW.md and the code as it evolves.
## How the audit works
Mirrors the existing `.github/workflows/claude-md-audit.yml` pattern. On
non-draft, non-fork PRs that touch code, `anthropics/claude-code-action`
reads REVIEW.md, samples the PR diff, and posts a sticky comment with up
to 3 of:
- `[stale]` — rule cites a path / function / table that's been removed
or renamed
- `[contradiction]` — code in the PR violates a current rule
- `[missing]` — PR introduces a new pattern future reviewers should know
about
- `[obsolete]` — rule asserts a constraint the repo has moved past
If nothing's off, posts `✅ REVIEW.md looks current for this PR.`
## Test plan
- [ ] Convert this PR to ready-for-review, confirm the audit runs and
posts a sticky comment
- [ ] Verify the audit doesn't run on fork PRs (gated by
`head.repo.full_name == github.repository`)
- [ ] Verify suggestions are actionable on at least one follow-up PR
Closes
[TRI-9234](https://linear.app/triggerdotdev/issue/TRI-9234/retry-task-process-sigsegv-errors-respecting-user-retry-config)
## What this changes
SIGSEGV crashes (`TASK_PROCESS_SIGSEGV`) will now be **retried when an
attempt fails**, in line with the task's configured retry settings
(`retry.maxAttempts` etc.) — the same path SIGTERM and uncaught
exceptions already use. Previously SIGSEGV was hard-classified as
non-retriable and failed the run on the first segfault, ignoring the
user's retry policy.
Tasks without a retry policy still fail fast on the first SIGSEGV.
Behaviour is unchanged for OOM kills (separate machine-bump retry path)
and SIGKILL_TIMEOUT.
## Deploy
**Only the webapp needs to ship.** The retry decision lives entirely in
the webapp:
- V2 path: `internal-packages/run-engine` (bundled into the webapp)
- V1 path: `apps/webapp/app/v3/services/completeAttempt.server.ts`
No supervisor, CLI, SDK, or customer-task-image changes required.
Customers do not need to redeploy. The `@trigger.dev/core` changeset is
just keeping the public package in sync — the published npm version
isn't what makes the fix work.
## Why retry
SIGSEGV in Node tasks is frequently non-deterministic across processes:
- **Native addon races** (`sharp`, `canvas`, `better-sqlite3`,
`node-rdkafka`, `bcrypt`, …) — libuv thread-pool work stepping on V8
handles. Different heap layout / thread schedule on a fresh process →
retry often succeeds.
- **JIT / GC interaction** — V8 turbofan deopt or GC during a native
callback. Timing-dependent.
- **Near-OOM in native code** — when RSS approaches the cgroup limit,
native allocations fail and poorly-written addons dereference NULL →
SIGSEGV instead of clean OOM-kill.
- **Host / hardware issues** — bit flips, kernel quirks. Retry lands on
a different host.
The genuinely deterministic case (a user-code bug always tripping the
same addon) is real, but a subset — and `maxAttempts` bounds the damage.
## Pre-existing inconsistency this resolves
- `shouldRetryError` returned `false` for `TASK_PROCESS_SIGSEGV` →
`fail_run`.
- `shouldLookupRetrySettings` already listed `TASK_PROCESS_SIGSEGV` as
retry-config-aware — but that branch was unreachable because
`shouldRetryError` short-circuited first in `retrying.ts:86-90`.
- We already retry `TASK_RUN_UNCAUGHT_EXCEPTION` (clearly a user-code
bug) under the user's retry policy; refusing to retry SIGSEGV was the
odd one out.
## Test plan
- [x] `pnpm exec vitest run test/errors.test.ts` in `packages/core` —
26/26 pass (4 new)
- [x] `pnpm run build --filter @trigger.dev/core`
- [ ] CI green on PR
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
## Summary
1 improvement, 1 bug fix.
## Improvements
- Fail attempts on uncaught exceptions instead of hanging to
`MAX_DURATION_EXCEEDED`. A Node `EventEmitter` (e.g. `node-redis`)
emitting `"error"` with no `.on("error", ...)` listener escalates to
`uncaughtException`, which the worker previously reported but did not
act on — runs drifted to maxDuration with empty attempts. They now fail
fast with the original error and status `FAILED`, and respect the task's
normal retry policy. You should still attach `.on("error", ...)`
listeners to long-lived clients to handle errors gracefully.
([#3529](https://github.com/triggerdotdev/trigger.dev/pull/3529))
## Bug fixes
- Fix dev workers spinning at 100% CPU after the parent CLI disconnects.
Orphaned `trigger-dev-run-worker` (and indexer) processes were caught in
an `uncaughtException` feedback loop: a periodic IPC send via
`process.send` would throw `ERR_IPC_CHANNEL_CLOSED` once the parent
closed the channel, which re-entered the same handler that itself called
`process.send`, scheduled via `setImmediate` and amplified by
source-map-support's `prepareStackTrace`. Fixed by (1) silently dropping
packets in `ZodIpcConnection` when the channel is disconnected, (2)
adding a `process.on("disconnect", ...)` handler in dev workers so they
exit cleanly when the CLI closes the IPC channel, and (3) wrapping all
`uncaughtException`-path `process.send` calls in a `safeSend` guard that
checks `process.connected` and swallows synchronous throws.
([#3491](https://github.com/triggerdotdev/trigger.dev/pull/3491))
<details>
<summary>Raw changeset output</summary>
# Releases
## @trigger.dev/build@4.4.6
### Patch Changes
- Updated dependencies:
- `@trigger.dev/core@4.4.6`
## trigger.dev@4.4.6
### Patch Changes
- Fix dev workers spinning at 100% CPU after the parent CLI disconnects.
Orphaned `trigger-dev-run-worker` (and indexer) processes were caught in
an `uncaughtException` feedback loop: a periodic IPC send via
`process.send` would throw `ERR_IPC_CHANNEL_CLOSED` once the parent
closed the channel, which re-entered the same handler that itself called
`process.send`, scheduled via `setImmediate` and amplified by
source-map-support's `prepareStackTrace`. Fixed by (1) silently dropping
packets in `ZodIpcConnection` when the channel is disconnected, (2)
adding a `process.on("disconnect", ...)` handler in dev workers so they
exit cleanly when the CLI closes the IPC channel, and (3) wrapping all
`uncaughtException`-path `process.send` calls in a `safeSend` guard that
checks `process.connected` and swallows synchronous throws.
([#3491](https://github.com/triggerdotdev/trigger.dev/pull/3491))
- Fail attempts on uncaught exceptions instead of hanging to
`MAX_DURATION_EXCEEDED`. A Node `EventEmitter` (e.g. `node-redis`)
emitting `"error"` with no `.on("error", ...)` listener escalates to
`uncaughtException`, which the worker previously reported but did not
act on — runs drifted to maxDuration with empty attempts. They now fail
fast with the original error and status `FAILED`, and respect the task's
normal retry policy. You should still attach `.on("error", ...)`
listeners to long-lived clients to handle errors gracefully.
([#3529](https://github.com/triggerdotdev/trigger.dev/pull/3529))
- Updated dependencies:
- `@trigger.dev/core@4.4.6`
- `@trigger.dev/build@4.4.6`
- `@trigger.dev/schema-to-json@4.4.6`
## @trigger.dev/core@4.4.6
### Patch Changes
- Fix dev workers spinning at 100% CPU after the parent CLI disconnects.
Orphaned `trigger-dev-run-worker` (and indexer) processes were caught in
an `uncaughtException` feedback loop: a periodic IPC send via
`process.send` would throw `ERR_IPC_CHANNEL_CLOSED` once the parent
closed the channel, which re-entered the same handler that itself called
`process.send`, scheduled via `setImmediate` and amplified by
source-map-support's `prepareStackTrace`. Fixed by (1) silently dropping
packets in `ZodIpcConnection` when the channel is disconnected, (2)
adding a `process.on("disconnect", ...)` handler in dev workers so they
exit cleanly when the CLI closes the IPC channel, and (3) wrapping all
`uncaughtException`-path `process.send` calls in a `safeSend` guard that
checks `process.connected` and swallows synchronous throws.
([#3491](https://github.com/triggerdotdev/trigger.dev/pull/3491))
- Fail attempts on uncaught exceptions instead of hanging to
`MAX_DURATION_EXCEEDED`. A Node `EventEmitter` (e.g. `node-redis`)
emitting `"error"` with no `.on("error", ...)` listener escalates to
`uncaughtException`, which the worker previously reported but did not
act on — runs drifted to maxDuration with empty attempts. They now fail
fast with the original error and status `FAILED`, and respect the task's
normal retry policy. You should still attach `.on("error", ...)`
listeners to long-lived clients to handle errors gracefully.
([#3529](https://github.com/triggerdotdev/trigger.dev/pull/3529))
## @trigger.dev/python@4.4.6
### Patch Changes
- Updated dependencies:
- `@trigger.dev/core@4.4.6`
- `@trigger.dev/build@4.4.6`
- `@trigger.dev/sdk@4.4.6`
## @trigger.dev/react-hooks@4.4.6
### Patch Changes
- Updated dependencies:
- `@trigger.dev/core@4.4.6`
## @trigger.dev/redis-worker@4.4.6
### Patch Changes
- Updated dependencies:
- `@trigger.dev/core@4.4.6`
## @trigger.dev/rsc@4.4.6
### Patch Changes
- Updated dependencies:
- `@trigger.dev/core@4.4.6`
## @trigger.dev/schema-to-json@4.4.6
### Patch Changes
- Updated dependencies:
- `@trigger.dev/core@4.4.6`
## @trigger.dev/sdk@4.4.6
### Patch Changes
- Updated dependencies:
- `@trigger.dev/core@4.4.6`
</details>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
### Style updates to the notifications
- Tightened up the typography
- Brighter background to make it stand out a bit more
- A bit more padding to make it more readable
- Show the close button on hover instead
- Turned the notification into a separate component as it's shared on
the admin page modal
- Minor tweaks to the behavior of toggling the notification beween
open/closed side menu states
### Before
<img width="224" height="313" alt="before"
src="https://github.com/user-attachments/assets/c9a9377c-4a3b-4477-921a-3c86385d3f0b"
/>
### After (with image)
<img width="239" height="284" alt="CleanShot 2026-05-11 at 17 22 01"
src="https://github.com/user-attachments/assets/311b4dbc-4853-4e6c-9f83-8173b38bd466"
/>
### After (no image)
<img width="239" height="189" alt="after"
src="https://github.com/user-attachments/assets/884e062b-3608-4cb3-a462-d50597257753"
/>
---------
Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com>
## Summary
Make `taskIdentifier` optional on the run-queue message schema. No
behavior change in this PR; readers continue to accept payloads that
include the field. A separate change will stop writing it on the wire to
shrink the per-run payload that lives in Redis while runs wait to be
dequeued.
## Design
The field is written into every payload at enqueue time but no consumer
reads it back on the dequeue path. Both the run-engine and supervisor
derive `taskIdentifier` from the loaded `TaskRun` row instead. Relaxing
the schema first means readers tolerate payloads that omit it, so the
writer-side change can ship without producing schema-parse errors during
a rolling deploy.
`projectId` is left required: `WorkerQueueResolver.#getOverride` reads
it for project-scoped runtime worker-queue overrides.
## Test plan
- [x] `pnpm run typecheck --filter @internal/run-engine`
- [x] `pnpm run typecheck --filter webapp`
- [x] `pnpm run test ./src/run-queue/tests/enqueueMessage.test.ts
./src/run-queue/tests/workerQueueResolver.test.ts --run` (28/28 passing)
## Summary
TTL expiration on queued runs was being scheduled twice: once via a
per-run `expireRun` worker job (the original implementation) and once
via the batch TTL system (added more recently). Both paths attempt to
flip the same run to `EXPIRED`. The per-run job almost always won the
race, leaving the batch consumer to observe runs already expired by the
older path.
This collapses TTL expiration onto the batch path so every queued TTLed
run goes through a single Redis-backed sorted set + batch consumer
instead of also getting its own scheduled redis-worker job.
## Design
`engine.trigger` and `delayedRunSystem.enqueueDelayedRun` no longer call
`ttlSystem.scheduleExpireRun`. The remaining `enqueueSystem.enqueueRun({
includeTtl: true })` already adds the run to the TTL sorted set;
`TtlSystem.expireRunsBatch` flips it to `EXPIRED` when the TTL fires.
Delayed runs get the same coverage by passing `includeTtl: true` on
their post-delay enqueue, so the TTL is armed from the moment the run
enters the queue (matching how the old job behaved —
`parseNaturalLanguageDuration` is evaluated at enqueue time).
The new path explicitly does not re-expire runs once they have been
allocated a concurrency slot. That is intentional: TTL is for runs that
are queued and have never started. Once a run has a slot it is on its
way to executing.
## Test plan
- [x] `pnpm run test --filter @internal/run-engine
./src/engine/tests/ttl.test.ts` — 15 tests, including a new "Re-enqueued
runs are not expired by TTL once they have started" that locks in the
queued-and-never-started contract.
- [x] `pnpm run test --filter @internal/run-engine
./src/engine/tests/delays.test.ts` — 5 tests, including "Delayed run
with a ttl" which now also asserts the TTL is armed from queue-enter
time, not `createdAt`.
- [x] `pnpm run test --filter @internal/run-engine
./src/engine/tests/lazyWaitpoint.test.ts` — 12 tests.
- [x] `pnpm run typecheck --filter @internal/run-engine`.
## Summary
- Adds admin-only editors on the back-office org page for
`Organization.maximumProjectCount` and
`Organization.batchRateLimitConfig`, alongside the existing API rate
limit editor.
- Splits the back-office org page into per-section components
(`ApiRateLimitSection`, `BatchRateLimitSection`, `MaxProjectsSection`)
so each tool is self-contained — adding new sections later doesn't bloat
the route.
- Generalizes the rate-limit form into a reusable `RateLimitSection`
component + `RateLimitDomain` server config so API and batch share the
same UI, validation, and action handler. Each domain only owns its env
defaults, DB column, and logger key.
- "Saved." banner and validation errors are scoped to the section that
submitted, not the page.
Heads-up: the API rate-limit log key was renamed
`admin.backOffice.rateLimit` → `admin.backOffice.apiRateLimit` for
symmetry with the new `admin.backOffice.batchRateLimit`.
## Test plan
- [ ] As an admin, visit `/admin/back-office/orgs/:orgId` and confirm
all three sections render with the org's current values (or system
defaults).
- [ ] Edit and save each section; confirm only that section shows the
"Saved." banner.
- [ ] Submit invalid input (e.g. `0` tokens, malformed interval);
confirm errors render in the offending form only and the other sections
stay closed.
- [ ] Confirm a non-admin user is redirected away from the route.
- [ ] After saving a rate-limit override, hit the org with traffic and
confirm the new limit is enforced (API rate limit + batch rate limit
code paths read the column at request time).
## Summary
When ElastiCache demotes a primary to replica — during a Multi-AZ
failover or a vertical node-type change — the demoting primary issues an
`UNBLOCKED` reply to any in-flight blocking commands (`BLPOP`, `BRPOP`,
`BLMOVE`, `XREADGROUP ... BLOCK`, etc.) to clear them before the role
flips. ioredis surfaces these as `ReplyError` to caller code.
The shared `defaultReconnectOnError` added in #3548 only matches
`READONLY` and `LOADING`. This extends it to `UNBLOCKED` so the
disconnect-reconnect-retry cycle handles BLPOP-shaped errors the same
way the existing two cases handle non-blocking-command errors.
## Fix
```ts
export function defaultReconnectOnError(err: Error): boolean | 1 | 2 {
const msg = err.message ?? "";
if (
msg.startsWith("READONLY") ||
msg.startsWith("LOADING") ||
msg.startsWith("UNBLOCKED")
) {
return 2;
}
return false;
}
```
Returning `2` tells ioredis to disconnect, reconnect, and re-issue the
command. For a BLPOP that means a fresh BLPOP against the new primary
instead of the `UNBLOCKED` error escaping to the caller.
## Test plan
- [ ] CI green
- [ ] Trigger a Multi-AZ failover or a vertical scale event on an
ElastiCache replication group whose clients are running blocking
commands and confirm no `UNBLOCKED` errors surface to caller code during
the cutover.
## Summary
During an ElastiCache role swap (failover) or node-type change (vertical
scale), the ioredis TCP/TLS connection stays open but the server starts
answering with `READONLY` (the client is talking to a node that became a
replica) or `LOADING` (node still loading data from disk). Without an
explicit hook, those errors surface to caller code as `ReplyError`
instances — every write op on the affected connection fails until the
cluster fully cuts over.
This PR adds `reconnectOnError` to every prod ioredis client so the
disconnect + reconnect + retry cycle absorbs these errors and caller
code never sees them.
## Fix
```ts
export function defaultReconnectOnError(err: Error): boolean | 1 | 2 {
const msg = err.message ?? "";
if (msg.startsWith("READONLY") || msg.startsWith("LOADING")) return 2;
return false;
}
```
Returning `2` tells ioredis to disconnect, reconnect, and re-issue the
failed command. After reconnect, DNS / SG state routes the new socket to
a writable node.
The helper lives in `@internal/redis` and is wired into both the shared
`createRedisClient` (which covers RunQueue, schedule-engine,
redis-worker, and every other internal-package consumer) and the direct
`new Redis(...)` call sites in the webapp.
V1-only marqs files are intentionally not migrated.
## Test plan
- [x] `pnpm run typecheck --filter webapp`
- [x] `pnpm run typecheck --filter @internal/run-engine`
- [x] Verified end-to-end against a live ElastiCache vertical-scale
event — caller-surfaced errors went from tens of thousands during the
cutover window down to a handful per ioredis client
- [ ] Confirm steady-state behavior unchanged after deploy
## Summary
- Run-view inspector panel was glitching out on Firefox: visual flicker
on close, locking up at min size, and intermittent `panelHasSpace`
invariant errors. Root cause is the underlying `react-window-splitter`
library's collapse animation, which uses `@react-spring/rafz` and
interacts poorly with Firefox.
- Disabled the library's collapse animation on Firefox only, app-wide
(every consumer of `RESIZABLE_PANEL_ANIMATION`). Chromium and Safari
behaviour is unchanged.
## Changes
- **Firefox animation skip** in `RESIZABLE_PANEL_ANIMATION` —
UA-detected at module load, resolves to `undefined` for Firefox so the
library's animation actor completes in one frame instead of running its
rAF loop.
- **Inspector min raised 50px → 250px** so dragging can't shrink the
panel into a near-useless width.
- **`autosaveId` bumped `v2` → `v3`** to invalidate stale persisted
snapshots (the library has a `// TODO` branch that ignores prop changes
for already-registered panels, so existing users would otherwise still
see the old 50px min).
- **`react-window-splitter` pinned** to exact `0.4.1` to protect the
patch from drifting if line offsets change in a patch release.
- **Two hunks added to the existing `@window-splitter/state` patch:**
- Removed the library's auto-collapse-on-drag block entirely. Every
collapsible panel in the app is parent-controlled, and that block was
triggering state-machine deadlocks when handlers were no-ops.
Drag-to-collapse is now disabled across the app; collapse is only
triggered explicitly (close button, ESC, URL change, etc.).
- In `getDeltaForEvent`, fall back to the panel's `default` before its
`min` when expanding — so the first ever click on a span opens the
inspector at 500px, not 250px.
## Local testing confirmed
- [x] Firefox: open a run, click various spans → panel opens instantly
at 500px, drags freely between 250px and max, closes instantly to 0. No
console errors.
- [x] Chrome/Chromium: same flow, but with smooth open/close animation
as before.
- [x] Safari: same as Chrome.
- [x] Reload mid-session → panel restores cleanly to the dragged size.
- [x] Other resizable panels in the app (logs, deployments, schedules,
batches, bulk-actions, runs index) still animate on Chromium/Safari.
## Notes
- Linear: TRI-8584
- Branch contains intermediate commits exploring an unsuccessful
snapshot-validator approach; they're reverted by the final commit.
Cumulative diff is 6 files. Squash on merge if you'd prefer a clean
history.
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
## Summary
Adds a Redis pub/sub reload path to the webapp's in-memory LLM pricing
registry. When enabled on a process, the registry reloads from the
database whenever a publish lands on the configured channel — instead of
waiting for the existing 5-minute interval. Lets pricing/model changes
propagate to cost enrichment within seconds.
Subscription is **off by default** and opt-in per process. Only
OTel-ingesting services need real-time freshness; dashboard and worker
services run fine on the periodic interval and shouldn't pile onto each
publish with a full-table reload.
## Design
When `LLM_PRICING_RELOAD_PUBSUB_ENABLED=true`, subscribes via
`createRedisClient` against `COMMON_WORKER_REDIS_*` and listens on
`LLM_PRICING_RELOAD_CHANNEL` (default `llm-registry:reload`). The
5-minute periodic reload stays as a backstop, and a SIGTERM/SIGINT
handler closes the subscription cleanly.
The publisher side lives outside this PR — any process running in the
same Redis namespace can trigger a reload by `PUBLISH
llm-registry:reload <anything>`. Includes a `.server-changes/` note for
the changelog.
### Debounced reload
Bursts of publishes are coalesced. The first publish schedules a reload
at T+`LLM_PRICING_RELOAD_DEBOUNCE_MS` (default 1s); subsequent publishes
during that window are no-ops because the trailing reload picks up
everything when it queries the DB. Bounds reload rate to at most 1 per
debounce window regardless of publisher chattiness, so a runaway
upstream publisher can't fan out into a flood of full-table-scan
reloads.
## Test plan
- [ ] With `LLM_PRICING_RELOAD_PUBSUB_ENABLED=false` (default):
`redis-cli PUBSUB NUMSUB llm-registry:reload` returns `0` while the
webapp is up
- [ ] With it set to `true`: returns `>= 1`
- [ ] `redis-cli PUBLISH llm-registry:reload test` returns `1` (one
subscriber received) on a subscribed process
- [ ] Mutate an `LlmModel` row externally, publish on the channel,
observe the registry's match() picks up the change without waiting for
the 5-min tick
- [ ] Publish 100x in rapid succession; confirm only one reload fires
within the debounce window
When a webapp API route's catch-all 500 branch handles a non-typed
exception, it returns the raw `error.message` to the caller. If the
exception originates from an internal subsystem (the ORM client, an
infra dependency, etc.) the server-side error string is surfaced
verbatim in the response body — exposing implementation details the API
surface shouldn't carry.
The leak shows up in three shapes across the routes:
- `return json({ error: error.message }, { status: 500 })`
- `return json({ error: error instanceof Error ? error.message :
"Internal Server Error" }, { status: 500 })`
- ``return json({ error: `Internal server error: ${error.message}` }, {
status: 500 })``
(plus a couple of analogous neverthrow-Result variants on admin routes.)
## Fix
Across 19 webapp routes, replace each leaking branch with a generic body
(`"Something went wrong"` / `"Internal Server Error"` to match the
file's existing fallback) and add `logger.error(...)` so full visibility
is preserved server-side. Catch blocks that branch on typed user-input
errors (`ServiceValidationError`, `EngineServiceValidationError`,
`OutOfEntitlementError`, `PrismaClientKnownRequestError`) are left
intact — those messages are constructed deliberately and intended to be
customer-facing.
## Test plan
- [x] `pnpm run typecheck --filter webapp`
- [x] Per-route manual probe: inject a synthetic `Error` at the top of
the catch'd `try` block (or fake the wrapped call's rejection / Result
error), curl the route with the dev API key, confirm the response body
changed from the synthetic message verbatim → generic body. 21/21 leak
sites verified end-to-end.
- [x] 4xx-typed-error paths spot-checked: throwing
`ServiceValidationError` from inside the catch'd try still surfaces its
message at 422 as intended.
## Summary
Stamps the active OpenTelemetry `trace_id` and `span_id` onto every
Sentry event captured from the webapp, so engineers can copy a
`trace_id` from a Sentry issue and search for the corresponding trace in
any OTel-aware backend. Also adds an `otel_sampled` tag to indicate
whether the trace was head-sampled — a cheap signal for whether the link
will resolve to span data or hit a missing trace.
## Why
Sentry and OTel were OTel-disconnected: `apps/webapp/sentry.server.ts`
initialised Sentry with `skipOpenTelemetrySetup: true`, and no
error-capture site (`logger.server.ts`, the Remix-wrapped `handleError`,
the root `ErrorBoundary`) attached OTel context to the event. With many
spans/sec across services, getting from a Sentry issue to its trace was
guesswork.
## Approach
Single global Sentry event processor, registered immediately after
`Sentry.init`. On each event it reads
`trace.getActiveSpan()?.spanContext()` via `@opentelemetry/api`, then
writes:
- `event.contexts.trace.trace_id` and `event.contexts.trace.span_id`
(Sentry's native trace context fields)
- `event.tags.otel_sampled` = `"true"` | `"false"` (derived from
`traceFlags`)
If no active span (module-load errors, scheduled timers without a
context, primary cluster process), the processor returns the event
unmodified — Sentry's default propagation context fills in.
Implementation is co-located in `apps/webapp/sentry.server.ts` (no
separate helper module — `sentry.server.ts` is built standalone by
esbuild and a separate import would have required a new bundling step).
Helper functions are exported so the unit tests can reach them without
re-running `Sentry.init`.
## Non-goals (deliberate)
- No sample rate change. ~95% of Sentry events will carry a `trace_id`
that returns no spans in the tracing backend (head-sampled out). The
`otel_sampled` tag makes that obvious at a glance. Raising find-rate is
a separate conversation with cost trade-offs.
- No user/org tags or `Sentry.setUser` (would need auth-helper +
per-request scope wiring across multiple worker entrypoints — separate
ticket).
- Webapp image only. No changes to supervisor or CLI workers.
## Test plan
- [x] Unit tests in `apps/webapp/test/sentryTraceContext.server.test.ts`
— 9 tests covering: helper returns \`undefined\` with no active span;
returns \`traceId\`/\`spanId\`/\`sampled=true\` for a recording span;
returns \`sampled=false\` for a non-recording span; processor leaves the
event unchanged with no active span; processor stamps
\`trace_id\`/\`span_id\` onto \`contexts.trace\`; preserves existing
\`contexts.trace\` fields; tags \`otel_sampled\` correctly for both
sampled and non-sampled cases; never throws if \`@opentelemetry/api\`
access throws.
- [x] \`pnpm run typecheck --filter webapp\` passes.
- [x] Manually verified end-to-end against a sandboxed Sentry project:
confirmed both sampled and non-sampled traces correctly populate
\`contexts.trace.trace_id\` matching the OTel ids logged from the
loader, and the \`otel_sampled\` tag appears with the expected value.
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
## Summary
Both Claude Code workflows (`claude.yml` and `claude-md-audit.yml`)
authenticated via `CLAUDE_CODE_OAUTH_TOKEN`, which broke when the org
disabled Claude subscription access for Claude Code:
> Your organization has disabled Claude subscription access for Claude
Code · Use an Anthropic API key instead, or ask your admin to enable
access
This switches both workflows to `anthropic_api_key: ${{
secrets.ANTHROPIC_API_KEY }}` (secret already added to the repo).
## Test plan
- [ ] Confirm `📝 CLAUDE.md Audit` runs to completion on this PR
- [ ] Confirm `@claude` mention in a PR comment still triggers the
`Claude Code` workflow successfully
When a Node EventEmitter (e.g. node-redis) emits an "error" event with
no
listener attached, Node escalates it to process.on("uncaughtException")
in
the task worker. The worker reported the error via the
UNCAUGHT_EXCEPTION
IPC event but did not exit, and the supervisor-side handler in
taskRunProcess only logged the message at debug level — leaving the
run()
promise orphaned until maxDuration fired and producing empty attempts
(durationMs=0, costInCents=0).
The supervisor now rejects the in-flight attempt with an
UncaughtExceptionError and gracefully terminates the worker (preserving
the OTEL flush window) on UNCAUGHT_EXCEPTION. The attempt fails fast
with
TASK_EXECUTION_FAILED, surfacing the original error name, message, and
stack trace, and falls under the normal retry policy. This mirrors the
existing indexing-side behavior in indexWorkerManifest. Apply the same
handling to unhandled promise rejections, which Node already routes
through uncaughtException by default.
- Tags webapp images by full commit SHA on `main` pushes
(`ghcr.io/triggerdotdev/trigger.dev:<sha>`) so any commit can be
resolved to a digest easily.
- Adds OCI labels (`source`, `revision`, `version`, `created`) so
`docker inspect`, vulnerability scanners, and
registry browsers see source/commit/version directly.
- Signs each pushed digest with SLSA build provenance via
`actions/attest-build-provenance@v4.1.0` (pinned by SHA), enabling `gh
attestation verify oci://...` against the source commit and workflow.
Large deploys (projects with many tasks or source files) blocked the
webapp event loop for several seconds inside Prisma's client-side
serializer on `BackgroundWorker.create`, tail-latencying every other
in-flight request on the same Node process. The `metadata` JSON column
was being written with the full deploy manifest — every task's config,
every queue and prompt, and the full source of every file — all of which
already live on dedicated columns or in dedicated tables.
Fix: project the manifest to `{ packageVersion, contentHash, tasks: [{
id, filePath, schedule }] }` on insert. The only post-write read site is
`changeCurrentDeployment`, which feeds `tasks[].schedule` into
`syncDeclarativeSchedules` at deploy promotion. The retained top-level
keys and per-task `filePath` are kept solely so
`BackgroundWorkerMetadata.safeParse` still succeeds on read.
## Test plan
- [ ] Deploy a project with declarative schedules; verify schedules are
created on first deploy
- [ ] Modify / remove schedules across subsequent deploys; verify sync
- [ ] Roll back to a previous deploy; verify `changeCurrentDeployment`
re-syncs schedules
- [ ] Inspect `BackgroundWorker.metadata` on a fresh deploy — should be
a small object, not the full manifest
`dac9c83bd` added `ignoreErrors: /^ServiceValidationError(?::|$)/` in
`apps/webapp/sentry.server.ts` to drop SVEs before they reach Sentry.
The
filter only matches when the captured event's *type* is
`ServiceValidationError`, but nine call sites in the webapp catch SVE
(and
analogous user-input error types — `OutOfEntitlementError`,
`CreateDeclarativeScheduleError`, `QueryError`) and call
`logger.error("wrapper message", { error: e })` *before* the type check.
The captured event is then titled with the wrapper message, with the
inner
error buried in `extra.error` — invisible to the SDK filter. Result: a
steady stream of expected user-input failures escalating as
`error`-level
events when they should be `warn`.
Each catch block now type-discriminates first, logs expected types at
`warn`,
and keeps unknown-error fall-throughs at `error`. For service sites that
wrap into SVE (`createBackgroundWorker`,
`createDeploymentBackgroundWorkerV4`),
the inner error is logged at `error` before wrapping — mirrors the
`waitpointCompletionPacket.server.ts` pattern from `dac9c83bd`.
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Fixes#3520. The bundled bitnami clickhouse subchart was pinned at
`9.3.7` (clickhouse `25.6.1-debian-12-r0`), which hits a memory-tracker
accounting bug under sustained ingest - the global counter overflows to
~7 EiB and every query gets rejected by OvercommitTracker until the pod
is restarted. Self-hosters running 4.0.5 through 4.4.5 are exposed
regardless of chart version since the subchart pin hadn't moved.
Bumping to `9.4.4` (clickhouse `25.7.5-debian-12-r0`) pulls in the
25.7.x memory-tracker fixes. This is also the latest publicly packaged
release at `oci://registry-1.docker.io/bitnamicharts` - that registry
has been frozen since 2025-08-28 (Bitnami catalog changes), but the
chart source remains under Apache 2 on `bitnami/charts`. The image
continues to resolve via `bitnamilegacy/clickhouse` per the existing
`values.yaml` override, since `bitnami/clickhouse` itself moved to
paid-only.
Verified locally: `helm dependency update` + `helm lint` + `helm
template` + kubeconform across all 57 rendered manifests. Rendered
statefulset image is
`docker.io/bitnamilegacy/clickhouse:25.7.5-debian-12-r0`.
## Summary
Move from a single shared S2 basin to **per-org basins** with retention
tied to the org's billing plan. Stops S2 from deleting streams out from
under live chat sessions when basin retention fires before the chat
ends, and unlocks per-org cost attribution.
OSS / s2-lite installs are unaffected: provisioning is gated by
`REALTIME_STREAMS_PER_ORG_BASINS_ENABLED` (default `false`), and the
read precedence falls back to the global basin env var when an entity
has no stamped basin.
```
basin = run.streamBasinName ?? session.streamBasinName ?? env.REALTIME_STREAMS_S2_BASIN
```
## Design
Three nullable `streamBasinName` columns (`Organization`, `TaskRun`,
`Session`) plus a provisioner that idempotently creates the basin and
reconfigures retention on plan changes. The trigger and session-create
paths stamp the org's basin onto new rows; the realtime read path picks
the basin from the entity context.
Admin routes back-fill existing orgs and force-reconfigure a single org.
## Test plan
- [x] `pnpm run typecheck --filter webapp --filter @internal/run-engine`
- [x] Backfill admin route end-to-end (provision + DB stamp + S2 basin
config).
- [x] Reconfigure on plan change (all retention tiers).
- [x] chat.agent multi-turn drives streams into the per-org basin.
- [x] Legacy fallback when entity has no stamped basin.
- [x] Provisioner is a no-op when the flag is off.
Adds zizmor alongside the actionlint job from #3503. Both now run as
parallel jobs in a single `.github/workflows/workflow-checks.yml`,
triggered on `.github/workflows/**` and `.github/actions/**` changes.
Zizmor is configured with `unpinned-uses: hash-pin` policy via
`.github/zizmor.yml`, so any future unpinned action will fail CI.
Findings upload SARIF to the Security tab alongside CodeQL.
Bulk of the diff is cleanup of the findings zizmor surfaced on first
run. `zizmor --fix=all` handled most of them mechanically; the rest were
judgment calls.
<img width="2284" height="2028" alt="CleanShot 2026-05-01 at 18 53
50@2x"
src="https://github.com/user-attachments/assets/4f58cbb1-0168-40fb-a523-017f2ba625a1"
/>
## Performance
- **Per-request DB hit**: `getUserId` runs `getEffectiveSessionDuration`
(User lookup + Org `aggregate`) on *every* authenticated request,
including each fetcher poll. Consider caching the effective duration in
the session cookie with a short TTL (e.g. 60s) and revalidating in the
background.
- **Double session commit in `root.tsx`**: `getUser` already runs the
expiry check; then `commitAuthenticatedSessionLazy` commits the cookie
again. Fine, but doubles `Set-Cookie` headers on every page load — worth
a quick perf check.
## Correctness / Edge cases
- **Lazy backfill assumes a root.tsx hit first**: users whose first
post-deploy request is a fetcher/API route (`/resources/*`) skip the
backfill until they navigate to a page. Not a security hole, but
`getUserId` could backfill itself for completeness.
- **No upper bound on `Organization.maxSessionDuration`**: admin API
accepts `1` second, which would instant-logout every member on next
request. Add a `min(60)` (or `min(300)` to match the lowest user option)
to the Zod schema.
- **No clock-skew tolerance**: `isSessionExpired` is exact-millisecond.
Multi-instance deploys with skewed clocks could log users out a few
seconds early/late. Probably fine for the 5-min minimum, but worth
noting.
## Security
- **Auto-logout audit log lacks IP/orgId**: HIPAA forensics typically
wants source IP and which org context. Currently logs only `userId` +
path. IP isn't PII for audit purposes; orgIds help correlate. Add both.
- **Cookie `Max-Age` is 1 year regardless of user's setting**:
intentional (server-side `issuedAt` is the source of truth), but
reviewers will ask. Add a one-line comment on the cookie config
explaining why.
## API surface
- **`maxSessionDuration` is admin-PAT only**: no in-app UI for org
owners to set/change their own cap. If this is "Trigger staff sets it
during HIPAA onboarding", say so in the PR description; otherwise add an
org-settings UI.
- **Auto-submit dropdown has no confirmation**: misclicking "5 minutes"
immediately shortens the user's session window with no undo. Consider a
save button or 3-sec undo toast.
## Schema / migration
- **`User.sessionDuration NOT NULL DEFAULT 31556952`**: instant on PG
11+ (metadata-only), but call out in the PR description so reviewers
don't worry about a table rewrite on the User table.
- **No DB-level constraint matching `SESSION_DURATION_OPTIONS`**: if the
option list changes, existing users keep orphaned values. The dropdown's
tag-along behaviour hides this — fine for now, but if you ever drop an
option you'll need a backfill.
## UX
- **Session expiry only fires on next request**: an idle authenticated
tab keeps showing UI past the cap (until SSE/polling catches it, ~60s).
Add a client-side timer based on the user's effective duration that
triggers a fetcher to `/account` or `/logout` at expiry.
- **No "you were signed out" message on logout**: users hitting their
cap are bounced to `/` with no explanation. Was intentionally reverted
in this PR — call that out so reviewers don't request it.
## Tests
- Unit coverage on `sessionDuration.server.ts` is solid (215 lines).
Missing: integration test for `getUserId` → expired session → redirect
to `/logout`, and one for the loader's clamping fix (the most recent
bug). Add at least the second one to lock in the regression.
---------
Co-authored-by: Matt Aitken <matt@mattaitken.com>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Reported by external contributor. The supervisor template hardcoded a
short DNS name for `OTEL_EXPORTER_OTLP_ENDPOINT`, which the supervisor
then propagates verbatim into runner pods
(`apps/supervisor/src/workloadManager/kubernetes.ts:196`). When runners
are spawned in a different namespace via
`supervisor.config.kubernetes.namespace`, the short name doesn't resolve
and span/log export silently fails - runs complete fine but the
dashboard shows nothing.
Same FQDN pattern the chart already uses for
`TRIGGER_WORKLOAD_API_DOMAIN` (line 203). Verified with `helm template
trigger . --namespace my-ns` - renders
`http://trigger-webapp.my-ns.svc.cluster.local:3030/otel`.
Cheers Niels
## Summary
Delete 34 `.server-changes/*.md` files that should have been cleaned up
automatically when v4.4.5 (#3406) was merged but were stranded by a
workflow race.
## Why these are stale
The `update-lockfile` job in `.github/workflows/changesets-pr.yml` is
what cleans up consumed `.server-changes/*.md` files on the release
branch. When v4.4.5 was merged on 2026-05-01, the post-merge workflow
run on `main` failed at `pnpm install --frozen-lockfile` (stale lockfile
in the merge commit), and `cancel-in-progress: true` cancelled the
in-flight run from the previous push — so `update-lockfile` never
reached the cleanup step.
Result: the 34 files described changes that v4.4.5 already shipped, and
they were re-appearing in the v4.4.6 release PR (#3501) under "Server
changes" plus showing up as deletions in its diff.
## What this PR keeps
- `fix-rollback-schedule-sync.md` — genuinely new for v4.4.6 (#3468),
the only server change introduced after v4.4.5
- `README.md`, `.gitkeep` — directory infrastructure
- `dev-cli-disconnect-md` — leaving alone (typo'd filename from March,
no `.md` extension, not picked up by the cleanup glob anyway)
## After merge
The next run of `changesets-pr.yml` will refresh #3501 with a "Server
changes" section that only lists the v4.4.6 entry, and the only
`.server-changes/` deletion in its diff will be
`fix-rollback-schedule-sync.md`.
## Related
- #3505 is the proper underlying fix — collapses the three-job graph
into a single atomic commit by `changesets/action` so this race can't
strand the cleanup again. This PR is just the one-time catch-up for the
files that already got stranded.
Follow-up to the v4.4.5 release incident where the release PR (#3406)
was merged with a stale lockfile and stale Chart.yaml, breaking npm +
helm releases. The two automation jobs (`update-lockfile`,
`bump-chart-version`) got cancelled mid-flight by `cancel-in-progress`
when the merge fired the workflow again on `main`.
This restructures `changeset:version` so all the post-version-bump
fixups happen in the same script and end up in a single atomic commit on
`changeset-release/main`, via `changesets/action`'s normal commit step.
Pattern borrowed from Cloudflare workers-sdk, Astro, shadcn/ui.
## Before
```
push: main
└── release-pr (changeset version → bumps package.jsons, opens PR)
└── update-lockfile (separate job, separate commit)
└── bump-chart-version (separate job, separate commit)
```
Three jobs, three commits to the release branch.
## After
```
push: main
└── release-pr
└── changesets/action runs:
changeset version
pnpm install --lockfile-only
node scripts/bump-helm-chart.mjs
node scripts/cleanup-server-changes.mjs
...all staged and committed as ONE commit by the action
```
One job, one commit.
Adds an `actionlint` job that runs on changes to `.github/workflows/**`
and `.github/actions/**`. Catches workflow bugs at PR time — expression
typos, deprecated runner labels, broken matrices, and shellcheck issues
in `run:` blocks.
Run from the official `docker://rhysd/actionlint` image, digest-pinned
alongside everything else.
Existing workflows had 6 shellcheck findings, all fixed.
Updates the compute private beta page with the May 1 release entry, plus
a deploy-time warning when `us-east-1-next` is the project default.
The new What's new entry, verbatim:
### May 1, 2026
- **Cold starts are faster across all machine sizes.** Every preset
starts faster, including `micro` and `small-1x` - there's no longer a
cold-start penalty for picking a smaller machine.
- **First runs after a deploy are faster on every preset.** Boot
snapshot creation is significantly quicker across the board, so the cold
path is consistently snappier.
- **`large-1x` and `large-2x` no longer hard-fail.** They're still not
recommended - cold-start performance trails the smaller presets and
we're ironing out reliability issues.
Follow-up to #3472 and #3479.
Orphaned `trigger-dev-run-worker` processes were pinning CPU at 100%
after the dev CLI exited — stuck in an uncaughtException feedback loop
where a closed IPC channel kept throwing `ERR_IPC_CHANNEL_CLOSED` back
into a handler that itself called `process.send`.
Fix:
- `ZodIpcConnection` no-ops sends when the channel is disconnected.
- Dev workers exit on `process.disconnect` instead of being re-parented
to init.
- All worker `uncaughtException` handlers route through a `safeSend`
guard so the handler can never re-enter itself.
Verified end-to-end: `kill -9` of the dev CLI now cleans up all child
workers within ~2s.
Last action still firing the Node 20 deprecation warning after #3494.
`buildjet/setup-node@v4.0.4` (the latest tag) declares `runs: using:
'node20'` and the repo hasn't shipped a node24 update.
Workflows here run on `ubuntu-latest` (not buildjet runners), so the
buildjet fork wasn't giving us anything we don't get from
`actions/setup-node` directly. Swapping to `actions/setup-node@v6.4.0`
(node24 runtime) silences the warning.
## Summary
Each successful PAT (`PersonalAccessToken`) or OAT
(`OrganizationAccessToken`) authentication issues a `prisma.X.update({
lastAccessedAt: new Date() })` to bump the timestamp. For tokens used at
high frequency (CLI clients, integrations) this generates a per-request
DB write that is mostly redundant — the `lastAccessedAt` field is only
surfaced on the settings page so users can decide which tokens to
revoke, and "within the last 5 minutes" is plenty of granularity for
that.
## Design
Replace each unconditional `update` with a conditional `updateMany`
whose `WHERE` requires the existing `lastAccessedAt` to be `NULL` or
strictly older than 5 minutes:
```ts
await prisma.personalAccessToken.updateMany({
where: {
id: personalAccessToken.id,
OR: [
{ lastAccessedAt: null },
{ lastAccessedAt: { lt: new Date(Date.now() - PAT_LAST_ACCESSED_THROTTLE_MS) } },
],
},
data: { lastAccessedAt: new Date() },
});
```
The conditional runs inside the SQL `UPDATE`, so concurrent auths can't
race into a double-write.
No schema change. No migration. No new infrastructure. Throttle is a
hardcoded constant (`5 * 60 * 1000`) — easy to revisit.
## Test plan
- [x] `pnpm run typecheck --filter webapp`
- [x] `pnpm vitest run ./test/services/personalAccessToken.test.ts
./test/services/organizationAccessToken.test.ts` — 6/6 pass, verifying
the throttle `WHERE` clause is constructed correctly and the `update` is
skipped on token-not-found / wrong-prefix paths
Follow-up to v4.4.5 release. The `bump-chart-version` job on the release
PR was cancelled before it could run, so Chart.yaml was merged still
pointing at 4.4.4. The helm release job ([failed
run](https://github.com/triggerdotdev/trigger.dev/actions/runs/25218553990/job/73947054128))
caught it via its version-match guard.
Once this merges I'll re-run the helm release workflow manually.
## ✅ Checklist
- [x] I have followed every step in the [contributing
guide](https://github.com/triggerdotdev/trigger.dev/blob/main/CONTRIBUTING.md)
- [x] The PR title follows the convention.
- [x] I ran and tested the code works
---
## Testing
- Reviewed the code flow for deployment rollback
(`ChangeCurrentDeploymentService`) and confirmed it was missing schedule
sync
- Verified all 4 callers of `ChangeCurrentDeploymentService` (UI
rollback, UI promote, API promote, finalize deployment) are now covered
- Ran `pnpm run typecheck --filter webapp` — passes cleanly
---
## Changelog
When rolling back (or manually promoting) a deployment, declarative
schedules were not being synced to match the target deployment's worker
metadata. Schedules remained as configured by the most recent deployment
rather than reflecting the target version's schedule configuration.
This fix adds a call to `syncDeclarativeSchedules` in
`ChangeCurrentDeploymentService` after the deployment promotion is
updated. It parses the target deployment's stored
`BackgroundWorkerMetadata` to restore the correct schedule state. This
covers both rollback and promote paths (UI and API). Errors are handled
gracefully so they don't block the deployment change itself.
---
## Screenshots
N/A — backend-only change.
💯
Link to Devin session:
https://app.devin.ai/sessions/0debf012b58c4132be778f8ea88cd2b6
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: nick <55853254+nicktrn@users.noreply.github.com>
## Summary
The v4.4.5 release PR (#3406) was merged before the automated
lockfile-update job in
[\`changesets-pr.yml\`](.github/workflows/changesets-pr.yml) could push
its commit. As a result main now has \`package.json\` bumped to
\`4.4.5\` but \`pnpm-lock.yaml\` still pinned to \`4.4.4\`.
This blocks every subsequent \`pnpm install --frozen-lockfile\` run,
including:
- \`release.yml\` for v4.4.5 publish ([run
#25217579660](https://github.com/triggerdotdev/trigger.dev/actions/runs/25217579660))
— never published packages to npm
- \`changesets-pr.yml\` on the next push to main ([run
#25217579645](https://github.com/triggerdotdev/trigger.dev/actions/runs/25217579645))
## Root cause (from CI logs)
\`\`\`
ERR_PNPM_OUTDATED_LOCKFILE Cannot install with "frozen-lockfile" because
pnpm-lock.yaml is not up to date with <ROOT>/packages/build/package.json
- @trigger.dev/core (lockfile: workspace:4.4.4, manifest:
workspace:4.4.5)
\`\`\`
Regenerated via \`pnpm install --lockfile-only\` against current main.
The diff is exactly what the canceled \`update-lockfile\` job would have
produced:
- 12 \`workspace:4.4.4\` → \`workspace:4.4.5\` specifier bumps
- pnpm metadata refresh (deprecation annotations on transitive deps, one
optional \`bufferutil\` peer resolution on \`react-email\`)
No new direct dependencies, no version drops.
## Follow-ups (separate PRs)
1. **Re-run release.yml** via \`workflow_dispatch\` (\`type: release\`,
\`ref\` = merge commit on main once this lands) to actually publish
4.4.5 to npm.
2. **Workflow fix** to prevent recurrence: fold the lockfile update into
\`changeset:version\` so the \`release-pr\` job creates a single commit
with version bumps + lockfile in sync. Removes the race window where the
release PR is mergeable before \`update-lockfile\` runs.
Dependabot's first auto-bump PR (#3495) was auto-closed because
`dependabot[bot]` isn't in the vouch list and isn't exempt from the
require-draft check.
Two changes:
- Add `dependabot[bot]` to `.github/VOUCHED.td` so the vouch check
passes.
- Add `dependabot[bot]` to the require-draft exception in
`vouch-check-pr.yml` (alongside `devin-ai-integration[bot]`) so its PRs
aren't closed for being non-draft.
Without both, dependabot bumps will keep getting closed and we lose the
weekly action update flow that #3494 set up.
## Summary
8 new features, 18 improvements, 11 bug fixes.
## Breaking changes
- Add server-side deprecation gate for deploys from v3 CLI versions
(gated by `DEPRECATE_V3_CLI_DEPLOYS_ENABLED`). v4 CLI deploys are
unaffected.
([#3415](https://github.com/triggerdotdev/trigger.dev/pull/3415))
## Improvements
- Add `--no-browser` flag to `init` and `login` to skip auto-opening the
browser during authentication. Also error loudly when `init` is run
without `--yes` under non-TTY stdin (previously default-and-exited
silently, leaving the project half-initialized). Both commands now show
an `Examples` section in `--help`.
([#3483](https://github.com/triggerdotdev/trigger.dev/pull/3483))
- Add `isReplay` boolean to the run context (`ctx.run.isReplay`),
derived from the existing `replayedFromTaskRunFriendlyId` database
field. Defaults to `false` for backwards compatibility.
([#3454](https://github.com/triggerdotdev/trigger.dev/pull/3454))
- Redact the `resolveWaitpoint` runtime log so it only emits `id` and
`type` instead of the full completed waitpoint. Previously the log
printed the entire waitpoint (including `output`) to stdout in
production runs, which could leak sensitive payloads. The value returned
by `wait.forToken()` is unchanged.
([#3490](https://github.com/triggerdotdev/trigger.dev/pull/3490))
- Add `SessionId` friendly ID generator and schemas for the new durable
Session primitive. Exported from `@trigger.dev/core/v3/isomorphic`
alongside `RunId`, `BatchId`, etc. Ships the
`CreateSessionStreamWaitpoint` request/response schemas alongside the
main Session CRUD.
([#3417](https://github.com/triggerdotdev/trigger.dev/pull/3417))
- Truncate large error stacks and messages to prevent OOM crashes. Stack
traces are capped at 50 frames (keeping top 5 + bottom 45 with an
omission notice), individual stack lines at 1024 chars, and error
messages at 1000 chars. Applied in parseError, sanitizeError, and OTel
span recording.
([#3405](https://github.com/triggerdotdev/trigger.dev/pull/3405))
## Server changes
These changes affect the self-hosted Docker image and Trigger.dev Cloud:
- Add a "Back office" tab to `/admin` and a per-organization detail page
at `/admin/back-office/orgs/:orgId`. The first action available on that
page is editing the org's API rate limit: admins can save a
`tokenBucket` override (refill rate, interval, max tokens) and see a
plain-English preview of the resulting sustained rate and burst
allowance. Writes are audit-logged via the server logger.
([#3434](https://github.com/triggerdotdev/trigger.dev/pull/3434))
- Optional `DEPLOY_REGISTRY_ECR_DEFAULT_REPOSITORY_POLICY` env var to
apply a default repository policy when the webapp creates new ECR repos
([#3467](https://github.com/triggerdotdev/trigger.dev/pull/3467))
- Ship the Errors page to all users, with a polish + bug-fix pass:
pinned "No channel" item in the Slack alert channel picker,
viewer-timezone alert timestamps via Slack's `<!date^>` token, Activity
sparkline peak tooltip, centered loading spinner and bug-icon empty
state on the error detail page, ellipsis on the Configure alerts
trigger.
([#3477](https://github.com/triggerdotdev/trigger.dev/pull/3477))
- Configure the set of machine presets to build boot snapshots for at
deploy time via `COMPUTE_TEMPLATE_MACHINE_PRESETS` (CSV of preset names,
default `small-1x`). Use `COMPUTE_TEMPLATE_MACHINE_PRESETS_REQUIRED`
(CSV, default = full PRESETS list) to scope which preset failures fail a
required-mode deploy. Optional preset failures are logged and don't
block the deploy.
([#3492](https://github.com/triggerdotdev/trigger.dev/pull/3492))
- Regenerating a RuntimeEnvironment API key no longer invalidates the
previous key immediately. The old key is recorded in a new
`RevokedApiKey` table with a 24 hour grace window, and
`findEnvironmentByApiKey` falls back to it when the submitted key
doesn't match any live environment. The grace window can be ended early
(or extended) by updating `expiresAt` on the row.
([#3420](https://github.com/triggerdotdev/trigger.dev/pull/3420))
- Add the `Session` primitive — a durable, task-bound, bidirectional I/O
channel that outlives a single run and acts as the run manager for
`chat.agent`. Ships the Postgres `Session` + `SessionRun` tables,
ClickHouse `sessions_v1` + replication service, the `sessions` JWT
scope, and the public CRUD + realtime routes (`/api/v1/sessions`,
`/realtime/v1/sessions/:session/:io`) including `end-and-continue` for
server-orchestrated run handoffs and session-stream waitpoints.
([#3417](https://github.com/triggerdotdev/trigger.dev/pull/3417))
- Add `KUBERNETES_POD_DNS_NDOTS_OVERRIDE_ENABLED` flag (off by default)
that overrides the cluster default and sets `dnsConfig.options.ndots` on
runner pods (defaulting to 2, configurable via
`KUBERNETES_POD_DNS_NDOTS`). Kubernetes defaults pods to `ndots: 5`, so
any name with fewer than 5 dots — including typical external domains
like `api.example.com` — is first walked through every entry in the
cluster search list (`<ns>.svc.cluster.local`, `svc.cluster.local`,
`cluster.local`) before being tried as-is, turning one resolution into
4+ CoreDNS queries (×2 with A+AAAA). Using a lower `ndots` value reduces
DNS query amplification in the `cluster.local` zone.
Note: before enabling, make sure no code path relies on search-list
expansion for names with dots ≥ the configured value — those names will
hit their as-is form first and could resolve externally before falling
back to the cluster search path.
([#3441](https://github.com/triggerdotdev/trigger.dev/pull/3441))
- Vercel integration option to disable auto promotions
([#3376](https://github.com/triggerdotdev/trigger.dev/pull/3376))
- Make it clear in the admin that feature flags are global and should
rarely be changed.
([#3408](https://github.com/triggerdotdev/trigger.dev/pull/3408))
- Admin worker groups API: add GET loader and expose more fields on
POST. ([#3390](https://github.com/triggerdotdev/trigger.dev/pull/3390))
- Add 60s fresh / 60s stale SWR cache to `getEntitlement` in
`platform.v3.server.ts`. Eliminates a synchronous billing-service HTTP
round trip on every trigger. Reuses the existing `platformCache` (LRU
memory + Redis) pattern already used for `limits` and `usage`. Cache key
is `${orgId}`. Errors return a permissive `{ hasAccess: true }` fallback
(existing behavior) and are also cached to prevent thundering-herd on
billing outages.
([#3388](https://github.com/triggerdotdev/trigger.dev/pull/3388))
- Show a `MicroVM` badge next to the region name on the regions page.
([#3407](https://github.com/triggerdotdev/trigger.dev/pull/3407))
- Increase default maximum project count per organization from 10 to 25
([#3409](https://github.com/triggerdotdev/trigger.dev/pull/3409))
- Merge execution snapshot creation into the dequeue taskRun.update
transaction, reducing 2 DB commits to 1 per dequeue operation
([#3395](https://github.com/triggerdotdev/trigger.dev/pull/3395))
- Add per-worker Node.js heap metrics to the OTel meter —
`nodejs.memory.heap.used`, `nodejs.memory.heap.total`,
`nodejs.memory.heap.limit`, `nodejs.memory.external`,
`nodejs.memory.array_buffers`, `nodejs.memory.rss`. Host-metrics only
publishes RSS, which overstates V8 heap by the external + native
footprint; these give direct heap visibility per cluster worker so
`NODE_MAX_OLD_SPACE_SIZE` can be sized against observed heap peaks
rather than RSS.
([#3437](https://github.com/triggerdotdev/trigger.dev/pull/3437))
- Tag Prisma spans with `db.datasource: "writer" | "replica"` so
monitors and trace queries can distinguish the writer pool from the
replica pool. Applies to all `prisma:engine:*` spans (including
`prisma:engine:connection` used by the connection-pool monitors) and the
outer `prisma:client:operation` span.
([#3422](https://github.com/triggerdotdev/trigger.dev/pull/3422))
- Clarify the cross-region intent in the Terraform and AI-prompt helpers
on the Add Private Connection page. Both already default
`supported_regions` to `["us-east-1", "eu-central-1"]`; added an inline
comment / parenthetical so the user understands why both regions are
listed (Trigger.dev runs in both, so the service must be consumable from
either).
([#3465](https://github.com/triggerdotdev/trigger.dev/pull/3465))
- Add `RUN_ENGINE_READ_REPLICA_SNAPSHOTS_SINCE_ENABLED` flag (default
off) to route the Prisma reads inside `RunEngine.getSnapshotsSince`
through the read-only replica client. Offloads the snapshot polling
queries (fired by every running task runner) from the primary. When
disabled, behavior is unchanged.
([#3423](https://github.com/triggerdotdev/trigger.dev/pull/3423))
- Stop creating TaskRunTag records and _TaskRunToTaskRunTag join table
entries during task triggering. The denormalized runTags string array on
TaskRun already stores tag names, making the M2M relation redundant
write overhead.
([#3369](https://github.com/triggerdotdev/trigger.dev/pull/3369))
- Stop writing per-tick state (`lastScheduledTimestamp`,
`nextScheduledTimestamp`, `lastRunTriggeredAt`) on `TaskSchedule` and
`TaskScheduleInstance`. The schedule engine now carries the previous
fire time forward via the worker queue payload, eliminating ~270K
dead-tuple-driven autovacuums per year on these hot tables and the
associated `IO:XactSync` mini-spikes on the writer. Customer-facing
`payload.lastTimestamp` semantics are unchanged.
([#3476](https://github.com/triggerdotdev/trigger.dev/pull/3476))
- Replace the expensive DISTINCT query for task filter dropdowns with a
dedicated TaskIdentifier registry table backed by Redis. Environments
migrate automatically on their next deploy, with a transparent fallback
to the legacy query for unmigrated environments. Also fixes duplicate
dropdown entries when a task changes trigger source, and adds
active/archived grouping for removed tasks. Moves BackgroundWorkerTask
reads in the trigger hot path to the read replica.
([#3368](https://github.com/triggerdotdev/trigger.dev/pull/3368))
- Public Access Tokens (PATs) minted before an API key rotation now keep
working during the 24h grace window. `validatePublicJwtKey` falls back
to any non-expired `RevokedApiKey` rows for the signing environment when
the primary signature check against the env's current `apiKey` fails.
The fallback query only runs on the failure path, so the hot success
path is unchanged.
([#3464](https://github.com/triggerdotdev/trigger.dev/pull/3464))
- Batch items that hit the environment queue size limit now fast-fail
without
retries and without creating pre-failed TaskRuns.
([#3352](https://github.com/triggerdotdev/trigger.dev/pull/3352))
- Show the cancel button in the runs list for runs in `DEQUEUED` status.
`DEQUEUED` was missing from `NON_FINAL_RUN_STATUSES` so the list hid the
button even though the single run page allowed it.
([#3421](https://github.com/triggerdotdev/trigger.dev/pull/3421))
- Reduce 5xx feedback loops on hot debounce keys by quantizing
`delayUntil`,
adding an unlocked fast-path skip, and gracefully handling redlock
contention in `handleDebounce` so the SDK no longer retries into a herd.
([#3453](https://github.com/triggerdotdev/trigger.dev/pull/3453))
- Fix RSS memory leak in the realtime proxy routes. `/realtime/v1/runs`,
`/realtime/v1/runs/:id`, and `/realtime/v1/batches/:id` called `fetch()`
into Electric with no abort signal, so when a client disconnected mid
long-poll, undici kept the upstream socket open and buffered response
chunks that would never be consumed — retained only in RSS, invisible to
V8 heap tooling. Thread `getRequestAbortSignal()` through
`RealtimeClient.streamRun/streamRuns/streamBatch` to `longPollingFetch`
and cancel the upstream body in the error path. Isolated reproducer
showed ~44 KB retained per leaked request; signal propagation releases
it cleanly.
([#3442](https://github.com/triggerdotdev/trigger.dev/pull/3442))
- Fix memory leak where every aborted SSE connection pinned the full
request/response graph on Node 20, caused by `AbortSignal.any()` in
`sse.ts` retaining its source signals indefinitely (see
nodejs/node#54614, nodejs/node#55351). Also clear the
`setTimeout(abort)` timer in `entry.server.tsx` so successful HTML
renders don't pin the React tree for 30s per request.
([#3430](https://github.com/triggerdotdev/trigger.dev/pull/3430))
- Preserve filters on the queues page when submitting modal actions.
([#3471](https://github.com/triggerdotdev/trigger.dev/pull/3471))
- Fix Redis connection leak in realtime streams and broken abort signal
propagation.
**Redis connections**: Non-blocking methods (ingestData, appendPart,
getLastChunkIndex) now share a single Redis connection instead of
creating one per request. streamResponse still uses dedicated
connections (required for XREAD BLOCK) but now tears them down
immediately via disconnect() instead of graceful quit(), with a 15s
inactivity fallback.
**Abort signal**: request.signal is broken in Remix/Express due to a
Node.js undici GC bug (nodejs/node#55428) that severs the signal chain
when Remix clones the Request internally. Added getRequestAbortSignal()
wired to Express res.on("close") via httpAsyncStorage, which fires
reliably on client disconnect. All SSE/streaming routes updated to use
it. ([#3399](https://github.com/triggerdotdev/trigger.dev/pull/3399))
- Prevent dashboard crash (React error #31) when span accessory item
text is not a string. Filters out malformed accessory items in
SpanCodePathAccessory instead of passing objects to React as children.
([#3400](https://github.com/triggerdotdev/trigger.dev/pull/3400))
- Upgrade Remix packages from 2.1.0 to 2.17.4 to address security
vulnerabilities in React Router
([#3372](https://github.com/triggerdotdev/trigger.dev/pull/3372))
- Fix Vercel integration settings page (remove redundant section
toggles) and improve the Vercel onboarding flow so the modal closes
after connecting a GitHub repo and the marketplace `next` URL is
preserved across the GitHub app install redirect.
([#3424](https://github.com/triggerdotdev/trigger.dev/pull/3424))
<details>
<summary>Raw changeset output</summary>
# Releases
## @trigger.dev/build@4.4.5
### Patch Changes
- Updated dependencies:
- `@trigger.dev/core@4.4.5`
## trigger.dev@4.4.5
### Patch Changes
- Add `--no-browser` flag to `init` and `login` to skip auto-opening the
browser during authentication. Also error loudly when `init` is run
without `--yes` under non-TTY stdin (previously default-and-exited
silently, leaving the project half-initialized). Both commands now show
an `Examples` section in `--help`.
([#3483](https://github.com/triggerdotdev/trigger.dev/pull/3483))
- Updated dependencies:
- `@trigger.dev/core@4.4.5`
- `@trigger.dev/build@4.4.5`
- `@trigger.dev/schema-to-json@4.4.5`
## @trigger.dev/core@4.4.5
### Patch Changes
- Add `isReplay` boolean to the run context (`ctx.run.isReplay`),
derived from the existing `replayedFromTaskRunFriendlyId` database
field. Defaults to `false` for backwards compatibility.
([#3454](https://github.com/triggerdotdev/trigger.dev/pull/3454))
- Redact the `resolveWaitpoint` runtime log so it only emits `id` and
`type` instead of the full completed waitpoint. Previously the log
printed the entire waitpoint (including `output`) to stdout in
production runs, which could leak sensitive payloads. The value returned
by `wait.forToken()` is unchanged.
([#3490](https://github.com/triggerdotdev/trigger.dev/pull/3490))
- Add `SessionId` friendly ID generator and schemas for the new durable
Session primitive. Exported from `@trigger.dev/core/v3/isomorphic`
alongside `RunId`, `BatchId`, etc. Ships the
`CreateSessionStreamWaitpoint` request/response schemas alongside the
main Session CRUD.
([#3417](https://github.com/triggerdotdev/trigger.dev/pull/3417))
- Truncate large error stacks and messages to prevent OOM crashes. Stack
traces are capped at 50 frames (keeping top 5 + bottom 45 with an
omission notice), individual stack lines at 1024 chars, and error
messages at 1000 chars. Applied in parseError, sanitizeError, and OTel
span recording.
([#3405](https://github.com/triggerdotdev/trigger.dev/pull/3405))
## @trigger.dev/python@4.4.5
### Patch Changes
- Updated dependencies:
- `@trigger.dev/core@4.4.5`
- `@trigger.dev/build@4.4.5`
- `@trigger.dev/sdk@4.4.5`
## @trigger.dev/react-hooks@4.4.5
### Patch Changes
- Updated dependencies:
- `@trigger.dev/core@4.4.5`
## @trigger.dev/redis-worker@4.4.5
### Patch Changes
- Updated dependencies:
- `@trigger.dev/core@4.4.5`
## @trigger.dev/rsc@4.4.5
### Patch Changes
- Updated dependencies:
- `@trigger.dev/core@4.4.5`
## @trigger.dev/schema-to-json@4.4.5
### Patch Changes
- Updated dependencies:
- `@trigger.dev/core@4.4.5`
## @trigger.dev/sdk@4.4.5
### Patch Changes
- Updated dependencies:
- `@trigger.dev/core@4.4.5`
</details>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
The webapp's compute template creation hardcoded a single machine preset
(`small-1x`) at deploy time, regardless of which presets a project
actually uses. Tasks running on any other preset paid full cold-snapshot
creation cost on first run.
Two new env vars:
- `COMPUTE_TEMPLATE_MACHINE_PRESETS` - CSV of preset names to build boot
snapshots for during deploy. Defaults to `small-1x` so existing deploys
don't change behavior.
- `COMPUTE_TEMPLATE_MACHINE_PRESETS_REQUIRED` - CSV of presets whose
failure fails a required-mode deploy. Defaults to the full `PRESETS`
list. Optional preset failures are logged but don't block the deploy.
The compute client now sends the multi-config request shape; the service
evaluates per-preset outcomes against the required set and surfaces a
combined failure message when a required preset fails.
Both env vars are validated at boot via the env schema - unknown preset
names or `_REQUIRED` entries that aren't a subset of `_PRESETS` fail
loudly at startup rather than silently per-deploy.
Most actions in this repo were several major versions behind, which is
why every CI run has been emitting Node 20 deprecation warnings.
Pinning every action to a commit SHA (with the version as a trailing
comment) means each CI run uses the exact code that was reviewed when
the bump landed, instead of whatever a maintainer last pointed the major
tag at. Dependabot is configured to group all action bumps into one
weekly PR with a 7-day cooldown.
Worth flagging:
- The Claude Code action ships ~daily but the model is set separately
via `--model` in `claude_args`, so SHA-pinning the action gives
reproducibility without locking the model.
- The kubeconform container is digest-pinned
(`docker://image:tag@sha256:...`). Dependabot's github-actions ecosystem
doesn't track `docker://` references ([explicit TODO in
dependabot-core](https://github.com/dependabot/dependabot-core/blob/main/github_actions/lib/dependabot/github_actions/file_parser.rb)),
so it needs manual bumps either way - but the digest pin protects
against tag repointing for free.
Redact the `resolveWaitpoint` runtime log so it only emits `id` and
`type` instead of the full completed waitpoint. Previously the log
printed the entire waitpoint (including `output`) to stdout in
production runs, which could leak sensitive payloads. The value returned
by `wait.forToken()` is unchanged.
The Activity peak count tooltip in the errors list rendered a `<button>`
(from `SimpleTooltip`'s default `TooltipTrigger`) inside the row's `<a>`
link (`TableCell to={errorPath}`). Interactive content nested inside
other interactive content is invalid HTML and triggers accessibility
warnings. Adding `asChild` to `SimpleTooltip` makes the existing
`<span>` the trigger directly, removing the nested `<button>`.
Closes the most common friction point hit while setting up a fresh
project from an agent harness: the CLI auto-opens the user's default
browser during auth and there is no supported way to skip it (the
existing `isLinuxServer()` path only triggers when `xdg-open` is missing
entirely).
`--no-browser` on `login` and `init` prints the URL and waits to be
visited from any browser. The flag threads through the embedded
`login()` call inside `init`.
While here:
- `init` now errors loudly when stdin is non-TTY without `--yes` instead
of default-and-exiting silently at the first prompt (which left the
project half-initialized: deps installed, no config or example file).
- Both commands gain an `Examples` block in `--help` rendered between
the description and the arguments/options list, so `--help | head`
surfaces the common invocations.
Other commands also call `login()` embedded and would benefit from
`--no-browser` too, but kept this PR scoped to the cases the friction
log called out.
## Summary
Each scheduled-task tick previously issued **3 Prisma `UPDATE`s**
against
`TaskSchedule.lastRunTriggeredAt`,
`TaskScheduleInstance.lastScheduledTimestamp`,
and `TaskScheduleInstance.nextScheduledTimestamp`. All three were pure
denormalization — every value can be derived without persisting.
After this PR `TaskSchedule` and `TaskScheduleInstance` become **near
read-only**:
writes happen only on schedule create / update / delete (rare admin
actions),
so the per-tick autovacuum churn on these hot tables disappears.
## Design
The previous fire time travels forward through the **schedule worker
payload**,
not through the database. Concretely:
- The `schedule.triggerScheduledTask` worker payload gains an optional
`lastScheduleTime: z.coerce.date().optional()` field.
- When the engine fires a schedule, it re-enqueues the next tick with
`lastScheduleTime = scheduleTimestamp` (the just-fired time).
- When the next tick dequeues, `payload.lastTimestamp` is sourced from
`params.lastScheduleTime` directly. No DB round-trip, no cron-derivation
drift across DST boundaries, no caveats around recently-edited cron
expressions.
`payload.lastTimestamp` keeps its `Date | undefined` SDK shape.
First-ever
fires still report `undefined`, so customer `if
(!payload.lastTimestamp)`
first-run patterns keep working.
For Redis jobs that were enqueued **before** this change (which lack
`lastScheduleTime` in their payload), the engine falls back to
`instance.lastScheduledTimestamp` once. Once those drain, the column is
never read again. Revert is code-only; the columns stay in place and can
be dropped in a follow-up once the rollout is stable.
## Files
- `internal-packages/schedule-engine/*` — engine refactor,
`workerCatalog`
schema field, `TriggerScheduleParams` extension, tests updated to assert
on the worker-payload flow rather than DB readbacks.
- `internal-packages/database/prisma/schema.prisma` — `/// @deprecated`
triple-slash docstrings on the three columns. No migration.
- `apps/webapp/app/presenters/v3/ScheduleListPresenter.server.ts` —
drops
the `lastRunTriggeredAt` Prisma select; "Last run" cell is approximated
from the cron expression's previous slot, gated on `schedule.createdAt`
so brand-new schedules show "–". UI is best-effort; the runs page is the
source of truth.
- `apps/webapp/app/v3/utils/calculateNextSchedule.server.ts` — adds a
`previousScheduledTimestamp` helper for the UI cell above. Public API
responses (`api.v1.schedules.*`) already compute `nextRun` from cron and
don't expose `lastTimestamp` — no public API change.
- `references/scheduled-tasks/` — new reference project with declarative
schedules at multiple cadences and three throw-on-fail validators
(`first-fire-detector`, `interval-validator`, `upcoming-validator`) for
E2E-verifying the worker-payload flow.
Refs TRI-8891
## Test plan
- [x] `pnpm run typecheck --filter @internal/schedule-engine --filter
webapp`
- [x] `pnpm run build --filter @trigger.dev/core`
- [x] `pnpm run test --filter @internal/schedule-engine` — integration
test
asserts first-fire `lastTimestamp === undefined`, second fire carries
the previous fire's timestamp exactly.
- [x] E2E against local webapp via `references/scheduled-tasks`:
- Fresh schedules attached → all three deprecated columns stay `NULL`
after
multiple fires.
- Redis payload at second fire contains
`"lastScheduleTime":"<previous fire timestamp>"`.
- `TaskRun.payload` and the every-minute task's returned output both
confirm
`lastTimestamp = null` on first fire and `lastTimestamp = <prev fire>`
on
second fire, exactly 60s apart.
- All three throw-on-FAIL validators completed successfully on every
non-first fire.
- [x] Schedules REST API end-to-end (`POST` / `GET` / `PUT` / `activate`
/
`deactivate` / `DELETE`) — `nextRun` recomputed live from cron + tz on
every response, no reads of deprecated columns.
## What this does
Polish + bug-fix pass on the Errors page so it can ship to everyone.
Touches the Slack alert config UX, errors list, error detail page, and
unhides the SideMenu entry for non-admins.
## Decisions
**"No channel" item over standalone Remove button**
Chose pinning a `<XMarkIcon /> No channel` `SelectItem` above the
channel list. Rejected the standalone "Remove channel" link in a
`<Hint>` — color/hover behaviour clashed with the sibling `<TextLink>`,
and "channel selection" is the right context for clearing. Server action
already deletes the channel when `slackChannel=""` is submitted.
**Slack `<!date^>` token over per-user TZ field for alerts**
Chose Slack's native `<!date^TS^…>` token so each viewer sees timestamps
in their own timezone (UTC fallback). Rejected per-user/per-org TZ
schema work — works for multi-region channels for free. Email/dashboard
TZ source-of-truth filed as TRI-8885 / TRI-8886.
**Make errors GA**
## Summary
`POST /api/v1/sessions` was secret-key-only because the customer browser
flow runs through `chat.createStartSessionAction` (server-side, holds
the secret key). But the `cli-v3` MCP `start_agent_chat` tool is itself
a server-side surface — developer's CLI/IDE acting as their own server —
and only holds a JWT minted from the user's PAT. Without JWT support on
this route the entire MCP agent toolkit (`start_agent_chat`,
`send_agent_message`, `close_agent_chat`) is blocked at session
creation.
Add `allowJWT: true` plus an `authorization` block requiring the
`write:sessions` (or `admin`) super-scope.
## Why a wildcard `sessions` resource
Resource scoping by `taskIdentifier` isn't possible at auth-resolve time
— action routes don't pass `body` to the `resource` callback, and the
task name only lives in the body. So the resource is `sessions: "*"` and
the super-scope does the actual gating. The JWT-issuer (cli-v3 MCP,
customer servers wrapping their own auth helpers, etc.) decides which
scopes to mint, which is where per-task narrowing lives.
## Test plan
- [x] Verified end-to-end against local:
`mcp__trigger__start_agent_chat` → `send_agent_message("pong")` →
`send_agent_message("echo")` → `close_agent_chat` all succeed. Two
assistant turns reuse the same runId (continuation in the idle window).
- [ ] Browser-mediated `chat.createStartSessionAction` flow continues to
work unchanged (still uses secret-key path under the hood).
- [ ] Loader (GET) and other session routes — unchanged, no scope drift.
## Notes
This unblocks T17 in the [ai-chat e2e smoke
catalog](https://github.com/triggerdotdev/trigger.dev/blob/feature/tri-7532-ai-sdk-chat-transport-and-chat-task-system/.claude/skills/ai-chat-e2e/SMOKE-TESTS.md)
(which lives in the feature branch's skill catalog, not this repo).
Pairs with the cli-v3 MCP fix on the feature branch (`feat: AI SDK
custom useChat transport & chat.task harness`, PR #3173) — that PR's
`agentChat.ts` change makes the call shape correct (`taskIdentifier` +
`triggerConfig`); this PR opens the door for the JWT to actually pass.
## Summary
Self-hosters that operate the webapp's ECR account separately from the
account running the EKS workers (e.g., a shared platform account that
hosts the registry plus per-team accounts that host clusters) currently
hit a 403 Forbidden the first time **any** project is deployed:
```
Failed to pull image "<acct-A>.dkr.ecr.<region>.amazonaws.com/<namespace>/proj_…:…":
unexpected status from HEAD request to .../v2/.../manifests/sha256:…: 403 Forbidden
```
`ensureEcrRepositoryExists` in
`apps/webapp/app/v3/getDeploymentImageRef.server.ts` calls
`CreateRepository` and `PutLifecyclePolicy`, but never
`SetRepositoryPolicy` — so the new repo inherits the AWS default (only
the registry-owner account can read/pull). Workers in the cluster
account get 403 every single deploy. The only workarounds today are
running a one-off post-create script or pre-creating every repo by hand.
## Proposed change
Add an optional env var:
```
DEPLOY_REGISTRY_ECR_DEFAULT_REPOSITORY_POLICY (V4 mirror: V4_DEPLOY_REGISTRY_ECR_DEFAULT_REPOSITORY_POLICY)
```
Raw IAM policy JSON. When set, the webapp calls `SetRepositoryPolicy`
immediately after `CreateRepository` so every new repo carries that
policy from creation. Operators control the principal/actions; we don't
bake in any opinions about cross-account boundaries.
Example value (for the typical self-host case — grant pull to the
cluster account):
```json
{
"Version": "2012-10-17",
"Statement": [{
"Sid": "AllowClusterAccountPull",
"Effect": "Allow",
"Principal": {"AWS": "arn:aws:iam::<cluster-account-id>:root"},
"Action": [
"ecr:GetDownloadUrlForLayer",
"ecr:BatchGetImage",
"ecr:BatchCheckLayerAvailability"
]
}]
}
```
## Why env var (not a chart-level field)
- Mirrors the shape of the sibling vars (`DEPLOY_REGISTRY_ECR_TAGS`,
`DEPLOY_REGISTRY_ECR_ASSUME_ROLE_ARN`, etc.) which are already
operator-supplied via `webapp.extraEnvVars` in self-host setups.
- Cloud is unaffected — the env var is optional, unset by default;
existing behavior unchanged.
- Existing repos are unaffected — only newly-created repos get the
policy.
- `RepositoryCreationTemplate` from the AWS provider isn't an
alternative here: it only applies to repos created via
pull-through-cache or replication, not to `ecr:CreateRepository` API
calls.
## Implementation
- `apps/webapp/app/env.server.ts` — declare
`DEPLOY_REGISTRY_ECR_DEFAULT_REPOSITORY_POLICY` and the V4 fallback.
- `apps/webapp/app/v3/registryConfig.server.ts` — propagate
`ecrDefaultRepositoryPolicy` to `RegistryConfig`.
- `apps/webapp/app/v3/getDeploymentImageRef.server.ts` —
`createEcrRepository` accepts the policy; if set, calls
`SetRepositoryPolicy` after `PutLifecyclePolicy`.
- `docs/self-hosting/env/webapp.mdx` — documentation row added under
**Deploy & Registry**.
## Verification
Verified end-to-end against a self-hosted Trigger.dev on EKS where the
ECR account is separate from the cluster account:
- **Without the env var** (current `main`): the new project's first run
pod stays in `ImagePullBackOff` with `403 Forbidden`.
- **With the env var set** to a JSON granting
`ecr:BatchGetImage`/`GetDownloadUrlForLayer`/`BatchCheckLayerAvailability`
to the cluster account: a fresh `trigger.dev deploy --env prod` followed
by a `hello-world` run completes in ~5s end-to-end on the first try.
Manually also confirmed that existing repos are untouched (the call only
fires inside `createEcrRepository`, which only runs when
`DescribeRepositories` returned `RepositoryNotFoundException`).
## Out of scope
- Chart values surface for this — operators already pass the existing
ECR vars via `webapp.extraEnvVars`, so this follows the same pattern.
Happy to add a first-class chart field in a follow-up if that's the
preferred direction.
- IAM-policy validation in the webapp — we forward the JSON verbatim to
AWS and surface AWS's error messages on misuse, matching how
`DEPLOY_REGISTRY_ECR_TAGS` is handled today.
This is a draft pending CI / CodeRabbit pass — happy to iterate on
direction (e.g., split into per-action env vars, or extend the chart
values schema) if any of the above choices feels off.
---------
Co-authored-by: nicktrn <55853254+nicktrn@users.noreply.github.com>
Queues page action handler was rebuilding the redirect URL with only
`?page=`, so any pause/resume/override modal confirmation wiped the
user's search query. With hundreds of queues filtered down to a handful,
every confirmation dropped you back to the unfiltered list - and
pagination still pointed at the previous numeric page, so you'd land on
a different slice than you came from.
Swap the manual rebuild for `url.search` so the full querystring
(including any future filter params) flows through. Drops the now-unused
`SearchParamsSchema.parse` call inside `action`; the loader still
validates on the way back.
## Summary
Follow-up to #3420. PATs (public access tokens) minted before an API key
rotation 401'd immediately on the realtime stream endpoints, even though
the rotation flow advertises a 24h overlap. This fixes the gap.
## Root cause
PATs are JWTs signed with the env's `apiKey` at mint time. When that
secret is rotated, `validatePublicJwtKey`
(`apps/webapp/app/services/realtime/jwtAuth.server.ts`) only verifies
the signature against `environment.parentEnvironment?.apiKey ??
environment.apiKey` — i.e. the env's *current* canonical key. Any PAT in
the wild signed with the previous key fails signature verification →
401, even within the grace window.
#3420 wired up the grace-window fallback in two places —
`findEnvironmentByApiKey` (raw secret-key auth) and `api.v1.auth.jwt.ts`
(signs new JWTs with the canonical key when minting from an old one) —
but the *verify* path for already-issued PATs was never updated.
In a typical app, `POST /api/v1/tasks/.../trigger` (Bearer secret) keeps
working through rotation because that path has the fallback, but `GET
/realtime/v1/streams/run_*/...` and `POST
/realtime/v1/streams/run_*/input/...` 401 for runs that were already in
flight when the rotation happened.
## Fix
After the primary `validateJWT` against the env's current `apiKey`, fall
back to non-expired `RevokedApiKey` rows for the signing env (parent env
when the request is against a child) — but **only on the failure path**,
so the hot success path is unchanged. Uses `$replica` to match the rest
of the auth path.
Symmetrical to the `findEnvironmentByApiKey` two-step from #3420.
## Changes
- `apps/webapp/app/services/realtime/jwtAuth.server.ts` —
`validateAgainstRevokedApiKeys` helper invoked only on `!result.ok`
- `apps/webapp/app/models/runtimeEnvironment.server.ts` —
`findEnvironmentById` also selects `parentEnvironment.id` so we can
scope the revoked-keys lookup to the correct env
## Test plan
E2E verified locally via curl against `GET /realtime/v1/runs/{runId}`
(PAT-authenticated):
- [x] Pre-rotation, PAT signed with K1 → **200** with run body
- [x] Simulate rotation (insert `RevokedApiKey` row + flip env `apiKey`
to K2 in a single transaction, mirroring `regenerateApiKey`)
- [x] Same PAT (K1) within grace window → **200** with run body —
fallback hits
- [x] Fresh PAT signed with K2 → **200** — current key still works
- [x] Set `RevokedApiKey.expiresAt` to past → **401** — fallback finds
no live row
- [x] Bogus signature (no rotation) → **401**
- [x] Cleanup verified: env `apiKey` restored, `RevokedApiKey` row
deleted
- [x] `pnpm run typecheck --filter webapp` passes
## Summary
Several boundary catches and customer-input validation paths were
logging at `error` level for failures the system already handles
gracefully — disconnect on auth failure, return undefined, skip retries,
etc. This batch routes them to `warn` (which stays in stdout) or counts
them as OTel metrics, so visibility is preserved without surfacing them
as alerts.
## Changes
**New helper / pattern:**
- `apiBuilder.server.ts` — `logBoundaryError(message, error, url)`
inspects the inner error type at loader/action boundary catches;
downgrades to `warn` for `AbortError`, `ServiceValidationError`, and
`EngineServiceValidationError`.
- `platform.v3.server.ts` — `platform_client.failures_total` OTel
counter with `{function, kind}` labels; helper
`recordPlatformFailure(fn, kind)` replaces the previous error-level
logging across all `BillingClient` wrappers.
**Log-level downgrades:**
- `handleSocketIo.server.ts` — `Worker authentication failed` → warn
(system disconnects on failure; refs TRI-8863)
- `waitpointSystem.ts` — when `runStatus === "CANCELED"` in the
suspended-without-checkpoint branch, skip the throw and warn instead
(benign cancel-vs-resume race, nothing to resume)
- `runAttemptSystem.ts` — `flushedMetadata` parse/validate failures →
warn (customer-side data shape, system returns gracefully)
- `batch-queue/index.ts` — final-attempt failures with
`result.skipRetries` → warn (callbacks already opted out of retry, e.g.
queue size limit hit)
- `queryPerformanceMonitor.server.ts` — slow queries → warn
(observability signal, not an application error)
- `timeoutDeployment.server.ts` — deployment-state mismatch in the
timeout job → warn (timeout-vs-completion race)
**Inner error preservation:**
- `waitpointCompletionPacket.server.ts` — `logger.error(uploadError)`
before throwing the `ServiceValidationError` wrapper, so the underlying
upload error stays visible.
## Why
The pattern across all of these is the same: a boundary log treated any
thrown/returned error as `error` regardless of cause, even when the
cause was an expected, system-handled condition (client disconnect,
customer quota, race condition, schema validation of customer data).
That made the logs noisy and made it harder to spot real bugs.
Where the underlying signal is still useful operationally (slow queries,
billing call failures), we route it to OTel metrics with low-cardinality
labels so dashboards and alerts can be tuned independently of error
logs.
## Test plan
- [ ] `pnpm run typecheck --filter webapp`
- [ ] `pnpm run build --filter @internal/run-engine`
- [ ] Trigger a run on hello-world and verify task lifecycle is
unaffected
- [ ] Cancel a suspended run and verify the cancel-while-suspended
branch in `waitpointSystem.ts` returns `{status: "skipped"}` instead of
throwing
- [ ] Confirm `platform_client.failures_total` counter shows up in
metrics with `{function, kind}` labels when the billing client errors
Today the helm prerelease workflow only fires on PRs that touch
`hosting/k8s/helm/**`. Two consequences we ran into:
1. The `changeset-release/main` PR's prerelease comment goes stale once
the release branch gets force-pushed without a helm-touching commit (the
bot's `Chart.yaml` bump alone doesn't seem to refire the trigger
reliably).
2. The release PR's chart references an `appVersion` (e.g. `v4.4.5`)
whose Docker images don't exist until *after* merge + tag. So that
prerelease chart can't actually be installed end-to-end.
Renames the workflow to `helm-prerelease.yml` and adds two new triggers:
- **`push: main`** with `paths: hosting/k8s/helm/**` -> rolling
prereleases versioned `<base>-main.<sha>`. `appVersion` stays at
whatever `Chart.yaml` has (i.e. last released), so installs pull real
images. Tests that chart structure is deployable, even if the app code
is one release behind.
- **`workflow_dispatch`** with optional `app_version` input -> manually
trigger a prerelease and optionally override `appVersion` (e.g. pin to
`main` or a specific tag). Useful for testing chart + app-version
combinations on demand.
PR behavior unchanged: same `<base>-pr<N>.<sha>` versioning, same
posted/updated comment.
Why not also bypass paths for `changeset-release/main`? The release PR's
chart references not-yet-built `v4.4.5` images, so those prereleases
aren't actually installable. The rolling main prerelease covers the
testable case better.
Why not SHA-pin `appVersion` to a built image like `main-<sha>`? Bigger
change - the docker publish workflows currently only push `:main` (no
SHA-suffixed tag). Worth doing later if we want first-class "install one
chart, get exactly that commit's app code" testing, but out of scope
here.
Diff is mostly a rename. Substantive changes:
- new `push` and `workflow_dispatch` triggers
- `prerelease` job `if:` extended for the new event types
- version logic branches per event
- new "Override appVersion" step (workflow_dispatch only)
- new "Write run summary" step so non-PR runs surface the install
instructions
- PR comment steps gated on `github.event_name == 'pull_request'`
- concurrency group falls back to `github.ref` for non-PR runs
> ⚠️ **Not released yet.** This PR is the server-side foundation only.
The SDK changes that customers will actually use (`chat.agent`
migration, `chat.createStartSessionAction`, `useTriggerChatTransport`
updates) live on a separate branch and ship together in an upcoming
`@trigger.dev/sdk` prerelease. Until that prerelease is published, this
surface is reachable only via direct HTTP.
## What this gives Trigger.dev users
A new first-class primitive, **Session**, for durable, task-bound,
bidirectional I/O that outlives any single run. Sessions are the run
manager for `chat.agent` going forward, and they unblock anything else
that needs "one identifier, many runs over time" with a stable channel
pair the client can write to and subscribe to.
### Use cases unblocked
- **Chat agents that persist across many runs.** One session per chat
(keyed on your own `chatId` via `externalId`), turns 1..N attach to the
same Session, the UI subscribes once and keeps receiving output as new
runs take over.
- **Approval loops and long-running tasks with user feedback.** The task
waits on `.in`, the client writes to `.in`, the server enforces
no-writes-after-close.
- **Workflow progress streams that live past the run.** Subscribe to
`.out` after the task finishes to replay history.
- **Resume-next-day flows.** A session is a durable row, not a transient
stream. Send a message a day later and the server triggers a fresh run
on the same session.
### How it works (Session-as-run-manager)
A Session row is task-bound (`taskIdentifier` + `triggerConfig` are
required) and owns its current run via `currentRunId` +
`currentRunVersion` for optimistic claim. Three trigger paths:
1. **Session create** — `POST /api/v1/sessions` creates the row and
triggers the first run synchronously.
2. **Append-time probe** — `POST
/realtime/v1/sessions/:session/in/append` checks if the current run is
alive; if it has terminated (idle exit, crash, etc.), the server
triggers a new run before processing the append.
3. **End-and-continue handoff** — `POST
/api/v1/sessions/:session/end-and-continue`, called by the running
agent, triggers a fresh run and atomically swaps `currentRunId`. Used by
`chat.requestUpgrade()` for version handoffs.
Every triggered run is recorded in the `SessionRun` audit table with a
reason (`initial`, `continuation`, `upgrade`, `manual`).
## Public API surface
### Control plane
- `POST /api/v1/sessions` — create. Idempotent on `(env, externalId)`.
Triggers the first run, returns the session and a session-scoped public
access token. Returns 409 if the upserted row is already closed.
- `GET /api/v1/sessions/:session` — retrieve by friendlyId
(`session_abc...`) or by your own externalId (server disambiguates by
prefix).
- `GET /api/v1/sessions` — list with filters (`type`, `tag`,
`taskIdentifier`, `externalId`, derived `status` ACTIVE/CLOSED/EXPIRED,
created-at range) and cursor pagination. Backed by ClickHouse.
- `PATCH /api/v1/sessions/:session` — update tags / metadata /
externalId.
- `POST /api/v1/sessions/:session/close` — terminate. Idempotent,
hard-blocks new server-brokered writes.
- `POST /api/v1/sessions/:session/end-and-continue` — agent-only handoff
to a fresh run.
### Realtime
- `PUT /realtime/v1/sessions/:session/:io` — initialize a channel.
Returns S2 credentials in headers so high-throughput clients can write
direct to S2.
- `GET /realtime/v1/sessions/:session/:io` — SSE subscribe. Supports
Last-Event-ID resume and an opt-in `X-Peek-Settled: 1` header that
fast-closes the stream when the upstream is already settled
(`trigger:turn-complete`), eliminating long-poll wait on
reconnect-on-reload paths.
- `POST /realtime/v1/sessions/:session/:io/append` — server-side
appends.
- `POST /api/v1/runs/:runFriendlyId/session-streams/wait` — runs wait on
a session stream as a waitpoint, with a race-check to avoid suspending
if data already landed.
### Auth scopes
`sessions` is a new resource type. `read:sessions:{id}`,
`write:sessions:{id}`, `admin:sessions:{id}` flow through the existing
JWT validator. Session-scoped public access tokens minted by the server
replace browser-held trigger-task tokens for chat-style flows — the
browser never sees a run identifier or a run-scoped token in steady
state.
## What's coming after this PR
- **SDK + chat.agent migration**: separate branch, separate PR, ships in
the next `@trigger.dev/sdk` prerelease alongside this server deploy.
Customers using the prerelease `chat.agent` will follow the [upgrade
guide](https://github.com/triggerdotdev/trigger.dev/blob/docs/tri-7532-ai-sdk-chat-transport-and-chat-task-system/docs/ai-chat/upgrade-guide.mdx).
- **Dashboard surfaces**: dedicated agent list, agent playground, agent
view on the run dashboard. Tracking separately.
## Implementation notes
- **Postgres `Session` table**: scalar scoping columns (`projectId`,
`runtimeEnvironmentId`, `environmentType`, `organizationId`) without
FKs, matching the January TaskRun FK-removal decision. Point-lookup
indexes only — list queries go to ClickHouse. Terminal markers
(`closedAt`, `expiresAt`) are write-once.
- **ClickHouse `sessions_v1`**: ReplacingMergeTree, partitioned by
month, ordered by `(org_id, project_id, environment_id, created_at,
session_id)`. Tags indexed via `tokenbf_v1` skip index.
- **`SessionsReplicationService`**: mirrors `RunsReplicationService`
exactly — leader-locked logical replication consumer,
`ConcurrentFlushScheduler`, retry with exponential backoff + jitter,
identical metric shape. Dedicated slot + publication so the two consume
independently.
- **S2 keys**: `sessions/{addressingKey}/{out|in}`. The existing
`runs/{runId}/{streamId}` key format for run-scoped streams is
untouched.
- **Optimistic claim**: `ensureRunForSession` triggers a run upfront
(cheap to cancel if it loses the race), then attempts an `updateMany`
keyed on `currentRunVersion`. Loser cancels its triggered run and reuses
the winner's. No DB lock held across the trigger.
### What did NOT change
Run-scoped `streams.pipe` / `streams.input` and the existing
`/realtime/v1/streams/{runId}/...` routes are unchanged. Sessions are
net-new — not a reshaping of the current streams API.
## Deploy notes
- Set `SESSION_REPLICATION_CLICKHOUSE_URL` and
`SESSION_REPLICATION_ENABLED=1` to enable the replication consumer.
- The `Session` table needs `REPLICA IDENTITY FULL` set on the prod
source DB before the publication is created (same one-time DDL we did
for `TaskRun`). Required for delete events to carry full column values.
- Cross-form authorization on the `GET /api/v1/sessions/:session` loader
(a JWT minted for either form authorizes both URL forms). Action routes
are URL-form-specific, matching how the SDK mints PATs.
## Verification
- Webapp typecheck clean (10/10).
- `apps/webapp/test/sessionsReplicationService.test.ts` — round-trip
tests for insert/update/delete through Postgres logical replication into
ClickHouse via testcontainers.
- Live end-to-end against local dev: create + retrieve (both forms) +
update + close, `.out.initialize` + `.out.append` x2 + `.in.send` +
`.out.subscribe` over SSE, list with all filter combinations +
pagination, `end-and-continue` swap, `X-Peek-Settled` fast-close
(verified in browser via reconnect-on-reload and via curl). Replicated
row lands in ClickHouse within ~1s.
- Multi-round Devin + CodeRabbit review feedback addressed
(read-after-write paths use `prisma` writer, info-leak on auth-routes
masked as 403, peek-settled discriminator parsing fix, etc.).
## Test plan
- [ ] `pnpm run typecheck --filter webapp`
- [ ] `pnpm run test --filter webapp
./test/sessionsReplicationService.test.ts --run`
- [ ] Start the webapp with `SESSION_REPLICATION_CLICKHOUSE_URL` and
`SESSION_REPLICATION_ENABLED=1`. Confirm the slot and publication
auto-create on boot.
- [ ] `POST /api/v1/sessions` and verify the row replicates to
`trigger_dev.sessions_v1` within a couple of seconds.
- [ ] `POST /api/v1/sessions/:id/close`, then confirm `POST
/realtime/v1/sessions/:id/out/append` returns 400.
- [ ] Reuse a closed session's `externalId` on `POST /api/v1/sessions`
and confirm 409.
- [ ] `GET /realtime/v1/sessions/:id/out` with `X-Peek-Settled: 1` after
a turn completes and confirm `X-Session-Settled: true` response header +
immediate close.
## Changes
Three changes in
`internal-packages/run-engine/src/engine/systems/debounceSystem.ts`, in
order of impact:
1. **Fast-path skip before the lock.** In `handleExistingRun`, do an
unlocked read of `delayUntil` (and `createdAt` for the max-duration
check) from the run row before entering `runLock.lock("handleDebounce",
...)`. If `newDelayUntil <= currentDelayUntil` and the run is still
within its max-duration window, return the existing run immediately
without taking the lock. Safe because debounce is monotonic-forward only
— a stale read either matches reality or undershoots, both of which
decay correctly (re-checked properly inside the lock by whichever caller
is actually pushing forward). Trailing-mode triggers carrying
`updateData` still take the lock so the data update is applied.
2. **Quantize `newDelayUntil`.** Round the computed `newDelayUntil` to
1-second buckets (configurable via `quantizeNewDelayUntilMs`, set to 0
to disable). Without quantization, every call has a slightly larger
`newDelayUntil` than the last and they all pass the fast-path check.
With it, concurrent callers on the same key share a target time and ~95%
short-circuit. User-visible effect: a debounced run might fire up to 1s
earlier than the strict spec — non-issue for typical debounce use cases
(chat summarization, batched notifications, etc.).
3. **Graceful lock-contention fallback.** Wrap the `runLock.lock(...)`
call so `LockAcquisitionTimeoutError` and Redlock `ExecutionError` /
`ResourceLockedError` return the existing run id with success instead of
propagating a 5xx. Debounce is best-effort: if we can't take the lock,
the herd is already updating it for us; fall in line. This kills the 5xx
→ SDK-retry feedback loop. With (1)+(2) this rarely fires; without them
it's the difference between 5xx and 200.
Defaults preserve current behaviour aside from quantization (1s) and
fast-path (on). Both are configurable via `RunEngineOptions.debounce`.
## ✅ Checklist
- [x] I have followed every step in the [contributing
guide](https://github.com/triggerdotdev/trigger.dev/blob/main/CONTRIBUTING.md)
- [x] The PR title follows the convention.
- [x] I ran and tested the code works
---
## Changelog
Reduce 5xx feedback loops on hot debounce keys by quantizing
`delayUntil`, adding an unlocked fast-path skip before the redlock, and
gracefully handling redlock contention in `handleDebounce` so the SDK no
longer retries into a herd.
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
## Summary
Adds `isReplay` boolean to the run context (`ctx.run.isReplay`),
following the same pattern as the existing `isTest`. The value is
derived from the existing `replayedFromTaskRunFriendlyId` database
field, so no schema migration is needed.
## ✅ Checklist
- [x] I have followed every step in the [contributing
guide](https://github.com/triggerdotdev/trigger.dev/blob/main/CONTRIBUTING.md)
- [x] The PR title follows the convention.
- [x] I ran and tested the code works
---
## Testing
- Verified `@trigger.dev/core` builds successfully
- Verified `webapp` typechecks successfully
- All new fields use `default(false)` for backwards compatibility
---
## Changelog
- Added `isReplay` to `TaskRun` and `V3TaskRun` schemas in `common.ts`
- Added `RUN_IS_REPLAY` semantic attribute and wired it in `taskContext`
- Propagated `isReplay` through the dequeue system, run attempt system,
and all execution context construction paths (V1 + V2)
- Added `isReplay` to `DequeuedMessage` and
`TaskRunExecutionLazyAttemptPayload` schemas
- Added patch changeset for `@trigger.dev/core`
- Updated docs: added `isReplay` to context reference, added "Detecting
replays" section to replaying page
---
💯
Link to Devin session:
https://app.devin.ai/sessions/1d6f1b3cc39a4623b72d05bf00f2d70c
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: nick <55853254+nicktrn@users.noreply.github.com>
Fork PRs can't access org secrets or push to GHCR, so these two
`pull_request` jobs hard-fail with no path to passing:
- `claude-md-audit` - needs `CLAUDE_CODE_OAUTH_TOKEN`
- `helm-pr-prerelease` `prerelease` job - needs `packages: write` to
push the chart
Hit this on #3449. Approving the run didn't help; the jobs ran and
failed at the privileged step. The chart-validation `lint-and-test` job
is fork-safe and stays untouched - that remains the merge gate for Helm
changes.
Gate both jobs on same-repo head:
```yaml
if: github.event.pull_request.head.repo.full_name == github.repository
```
Other PR workflows already handle forks fine: `pr_checks`
(typecheck/units/e2e/sdk-compat) falls back to anonymous DockerHub pulls
when secrets are missing.
## Summary
When the official Helm chart is deployed with an external ClickHouse and
`clickhouse.external.existingSecret` set — the documented path for not
committing secrets to `values.yaml` — the webapp pod crash-loops on
startup:
```
goose run: parse "http://default:${CLICKHOUSE_PASSWORD}@<host>:8123?secure=false": net/url: invalid userinfo
```
Context in vouch request #3443. Re-opening in draft status per bot
policy (previous attempt was #3445, closed by automation because it
wasn't draft; no changes to the patch).
## Root cause
Two pieces interact:
1. `hosting/k8s/helm/templates/_helpers.tpl` renders `CLICKHOUSE_URL`
(and `RUN_REPLICATION_CLICKHOUSE_URL`) with a shell-style literal
`${CLICKHOUSE_PASSWORD}` expecting bash expansion at container start.
2. `docker/scripts/entrypoint.sh` does `export
GOOSE_DBSTRING="$CLICKHOUSE_URL"` — single-pass POSIX sh substitution,
so the inner `${...}` survives as literal text and goose rejects it.
Reproduces against the latest published chart
(`oci://ghcr.io/triggerdotdev/charts/trigger:4.0.5`) and `main`.
## Fix
Switch the two helpers (external + `existingSecret` branch) from
shell-style `${CLICKHOUSE_PASSWORD}` to Kubernetes'
`$(CLICKHOUSE_PASSWORD)`. Kubelet substitutes `$(VAR)` at pod-creation
time from earlier env entries, and the chart already declares
`CLICKHOUSE_PASSWORD` from the Secret immediately before
`CLICKHOUSE_URL`, so the URL reaches the entrypoint with the real
password already inlined. No entrypoint change, no image change. The
plain-password branch (no `existingSecret`) is unchanged.
Operator caveat added as template comments: `CLICKHOUSE_PASSWORD` must
be URL-userinfo-safe since kubelet substitutes verbatim without
percent-encoding. Hex-encoded passwords (e.g. `openssl rand -hex 32`)
are safe by construction.
## Verification
- `helm template` against `external.existingSecret` now renders `value:
"http://default:$(CLICKHOUSE_PASSWORD)@<host>:8123?secure=false"` (was
`${CLICKHOUSE_PASSWORD}`).
- `helm template` against the plain-password branch is byte-identical to
before.
- Deployed end-to-end on a staging EKS cluster (Meistrari platform):
webapp container reaches `goose: successfully migrated database to
version: 6`, Node.js ClickHouse client connects at runtime.
## Alternatives considered
- **Change `entrypoint.sh`** to `eval` / `envsubst` the URL — larger
surface, touches every deployment mode (Docker Compose + k8s) and every
container image.
- **Mirror the Postgres pattern** (chart reads the full URL via
`valueFrom.secretKeyRef`, as in `trigger-v4.postgres.useSecretUrl`) —
cleaner long-term but requires a new `values.yaml` field and a migration
path for existing users. Happy to follow up with that as a separate PR
if the minimal fix here isn't the preferred direction.
## Changeset
None added — the Helm chart isn't versioned through `@changesets/cli`
(docs/chart-only PRs historically merge without a changeset, e.g.
#2671). Happy to add one if the policy changed.
Closes#3443.
## Summary
Two fixes that together get a fresh-machine setup working from
`CONTRIBUTING.md` end-to-end with no manual workarounds:
### `CONTRIBUTING.md`
- Fix wrong path in the migration walkthrough: `cd packages/database` →
`cd internal-packages/database`. The current path doesn't exist; this
breaks step 2 for every contributor adding a migration.
- Renumber duplicate `4.` steps in **Adding migrations** and the skipped
`5.` in the hello-world **Running** section.
- Combine three sequential `pnpm run build --filter ...` calls into one
(Turbo parallelizes filters): `pnpm run build --filter webapp --filter
trigger.dev --filter @trigger.dev/sdk`.
- Add a `pnpm run db:seed` step after migrate. The seed creates the
local user, `References` org, and reference projects (including
`hello-world` with the stable `proj_rrkpdguyagvsoktglnod`). Removes the
manual instruction to edit the `externalRef` column in Postgres.
- Mention ClickHouse and the ClickHouse migrator alongside
Postgres/Redis in the Docker step (they're already part of `pnpm run
docker`, just invisible in the docs).
- Remove the V1-era **Add sample jobs** section.
`references/job-catalog` no longer exists; the hello-world flow above
replaces it.
### `turbo.json`
Scope `db:seed` to `webapp#db:seed → webapp#build`. The previous
root-level entry queued `build` for every workspace package — including
`references-*`, `docs`, `kubernetes-provider`, `coordinator`, etc. Only
`webapp` actually has a `db:seed` script, so the rest of those builds
were dead weight. Worse: a single broken reference (today,
`references-realtime-hooks-test` failing under Turbopack with
`node:fs/promises`) kills the whole seed pipeline.
After the change, `turbo run db:seed --dry-run` plan drops from 27 tasks
to 20 — only `webapp` and its real transitive workspace deps. Reference
projects no longer block seeding.
## Test plan
- [x] Fresh-machine setup followed end-to-end on a wiped Postgres +
ClickHouse: migrate → seed → build → webapp → CLI login → `trigger dev`
→ triggered `hello-world`, run completed with `{"message":"Hello,
world!"}`.
- [x] `turbo run db:seed --dry-run=json` confirms 20 tasks, all webapp
deps, no reference packages.
- [ ] CI green on the renamed turbo task name.
## Summary
Fixes an RSS-only memory leak in the three realtime proxy routes
(`/realtime/v1/runs`, `/realtime/v1/runs/:id`,
`/realtime/v1/batches/:id`). Client disconnects during an in-flight
long-poll would leave the upstream fetch to Electric running with no way
to abort it, so undici kept the socket open and buffered response chunks
that would never be consumed.
## Root cause
All three routes flow through
`RealtimeClient.streamRun/streamRuns/streamBatch` → `#streamRunsWhere` →
`#performElectricRequest` → `longPollingFetch(url, { signal })`. The
chain was already signal-aware, but `#streamRunsWhere` hardcoded
`signal=undefined` when calling `#performElectricRequest`, so no signal
ever reached `longPollingFetch`.
When a downstream client aborts a long-poll mid-flight:
1. Express tears down the downstream response socket.
2. The `longPollingFetch` promise has already resolved (it returns as
soon as upstream headers arrive) and handed back `new
Response(upstream.body, {...})`.
3. `undici` keeps the upstream socket open and continues buffering
chunks into the `ReadableStream` that nothing will ever read from.
4. The upstream connection is eventually closed by Electric's own poll
timeout (~20s). During that window the per-request buffers stay in
native memory.
These buffers live below V8's accounting — no `heapUsed` or `external`
growth, no sign in heap snapshots, only RSS. An isolated standalone
reproducer (`fetch` against a slow-streaming upstream, discard the
`Response` before consuming its body) measures **~44 KB retained per
leaked request** after GC. That's consistent with the undici socket +
receive buffer + HTTP parser state for a long-lived chunked response.
The pattern is the shape documented in
[nodejs/undici#1108](https://github.com/nodejs/undici/issues/1108) and
[#2143](https://github.com/nodejs/undici/issues/2143).
## What changed
- **`realtimeClient.server.ts`** — add optional `signal` parameter to
`streamRun`, `streamRuns`, `streamBatch`, and the shared
`#streamRunsWhere`; thread it through to `#performElectricRequest`
instead of hardcoding `undefined`.
- **`realtime.v1.runs.$runId.ts`, `realtime.v1.runs.ts`,
`realtime.v1.batches.$batchId.ts`** — pass `getRequestAbortSignal()`
(from `httpAsyncStorage.server.ts`) at the call site. This is the signal
wired to `res.on('close')` and fires reliably on downstream disconnect.
- **`longPollingFetch.ts`** — belt-and-suspenders: cancel the upstream
body explicitly in the error path, and treat `AbortError` as a clean
`499` instead of a `500`. This both releases undici's buffers
deterministically on error and avoids spurious 500s in request logs when
a client legitimately walks away.
## Verification
Standalone reproducer: slow upstream server streams 32 KB chunks every
100 ms for 5 seconds per request. The proxy does `fetch(url)` with
varying signal/cancel strategies, creates `new Response(upstream.body,
...)`, and discards it without consuming the body (simulating the leak
path).
Results from 1 000 parallel fetches per variant, measured post-GC:
| variant | Δ heap | Δ external | Δ RSS |
| --- | --- | --- | --- |
| A. no signal, body never consumed (the bug) | +0.3 MB | 0 MB | **+59.4
MB** |
| B. signal propagated, aborted after headers (this fix) | −0.1 MB | 0
MB | +15.4 MB |
| C. no signal, explicit `res.body.cancel()` | 0 MB | 0 MB | −25.4 MB |
10-round sustained test of variant B to distinguish accumulating
retention from one-time allocator overhead:
```
round 1/10 Δ=+3.2 MB round 6/10 Δ=-12.5 MB
round 2/10 Δ=-7.6 MB round 7/10 Δ=-11.9 MB
round 3/10 Δ=-11.7 MB round 8/10 Δ=-2.6 MB
round 4/10 Δ=+3.2 MB round 9/10 Δ=-8.0 MB
round 5/10 Δ=-1.2 MB round 10/10 Δ=-12.6 MB
```
RSS oscillates in a 49-65 MB band with no upward trend — signal
propagation fully releases the buffers.
## Risk
- Behavior change only on aborted long-polls: the upstream fetch now
cancels promptly instead of running to its natural timeout. This saves
both memory and outbound traffic to Electric.
- `AbortError` now surfaces as `499` rather than `500`. Any dashboard or
alert that counts 500s in request logs will see slightly fewer of them;
this is the intended behavior.
- Signal-aware parameter is optional on
`RealtimeClient.streamRun/streamRuns/streamBatch`, so callers that don't
opt in get the previous behavior.
## Test plan
- [ ] Existing realtime integration tests pass
- [ ] Dashboard realtime views (runs list, batch details) continue
working normally across tab open/close cycles
- [ ] Under a burst of aborted long-polls, server RSS returns to
baseline rather than climbing
Adds @GautamBytes to the vouch list so they can contribute to the
repository.
Closes#3307
Co-authored-by: claude[bot] <41898282+claude[bot]@users.noreply.github.com>
Co-authored-by: Eric Allam <ericallam@users.noreply.github.com>
Adds a minimal end-to-end test harness that spawns the compiled webapp
as a child
process against a throwaway Postgres container, plus a baseline of 8
auth-behaviour
tests. These tests will be used as a regression check before and after
the upcoming
apiBuilder RBAC migration to confirm auth behaviour is unchanged.
## What's included
**`internal-packages/testcontainers/src/webapp.ts`** (new)
Spawns `build/server.js` with a dynamically allocated port, polls
`/healthcheck`,
and exposes `WebappInstance` and `startTestServer()` (postgres container
+ webapp +
PrismaClient in one call). Key details:
- Uses `process.execPath` so the correct Node binary is found in forked
test processes
- Sets `NODE_PATH` to `node_modules/.pnpm/node_modules` so pnpm-hoisted
transitive
deps (e.g. `eventsource-parser`) resolve correctly inside the subprocess
- Overrides both `PORT` and `REMIX_APP_PORT` so Vite's automatic `.env`
loading
doesn't override the dynamically allocated port
**`internal-packages/testcontainers/package.json`**
Adds `./webapp` sub-path export so tests can `import from
"@internal/testcontainers/webapp"`.
**`internal-packages/testcontainers/src/index.ts`**
Exports `createPostgresContainer` (used internally by `webapp.ts`).
**`apps/webapp/test/helpers/seedTestEnvironment.ts`** (new)
Creates a minimal org → project → environment row set with random
suffixes.
**`apps/webapp/test/api-auth.e2e.test.ts`** (new)
8 tests across two suites:
- API-key bearer: valid key (auth passes, 404), missing header (401),
invalid key (401), error body shape
- JWT bearer: valid JWT on JWT-enabled route (passes), valid JWT on
non-JWT route (401), empty-scope JWT (403), wrong signing key (401)
## How to run
```bash
# Build required first (one-time)
pnpm run build --filter webapp
cd apps/webapp && pnpm exec vitest run test/api-auth.e2e.test.ts
```
## Test plan
- [x] All 8 tests pass against the current webapp build
- [x] Webapp healthcheck returns 200 on startup
- [ ] CI passes
---------
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Adds `KUBERNETES_POD_DNS_NDOTS_OVERRIDE_ENABLED` flag (off by default)
that overrides the cluster default and sets `dnsConfig.options.ndots` on
runner pods (defaulting to 2, configurable via
`KUBERNETES_POD_DNS_NDOTS`).
Kubernetes defaults pods to `ndots: 5`, so any name with fewer than 5
dots, including typical external domains like `api.example.com`, is
first walked through every entry in the cluster search list
(`<ns>.svc.cluster.local`, `svc.cluster.local`, `cluster.local`) before
being tried as-is, turning one resolution into 4+ CoreDNS queries (×2
with A+AAAA).
Using a lower `ndots` value reduces DNS query amplification in the
`cluster.local` zone.
## Summary
- The `processMasterQueueForEnvironment` call in the priority test was
racing against background `processQueueForWorkerQueue` jobs scheduled
50ms after each trigger
- With a 50ms debounce (`processWorkerQueueDebounceMs: 50`) and runs
triggered sequentially, the RunQueue Worker could process those jobs
mid-sequence, pushing partial batches to the worker queue in the wrong
overall priority order
- `masterQueueConsumersDisabled: true` only blocks the shard-level
polling loops — it does not prevent the RunQueue's own Worker from
processing these debounced jobs
- Fix: add `worker.disabled: true` to the test 1 engine config, which
propagates to `workerOptions.disabled` in the RunQueue constructor and
prevents the Worker from starting
## Test plan
- [x] Both priority tests pass: `pnpm run test
./src/engine/tests/priority.test.ts --run`
- [x] Test 1 log confirms no `✅ Starting run engine worker` or worker
loop messages — workers fully disabled
- [x] Test 2 unaffected (uses master queue consumers for automatic
promotion, no `disabled` flag added)
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Wires up automatic Helm chart releases to ride along with the existing
changeset-driven package release flow.
Today `Chart.yaml` is bumped by hand and `release-helm.yml` fires only
when a human pushes a `helm-v*` tag. With this, the changeset release PR
also carries a `Chart.yaml` bump so main always matches the published
version, and `release.yml` invokes `release-helm.yml` via
`workflow_call` after Docker images are published.
`helm-v${VERSION}` tag is pushed as a marker (same GITHUB_TOKEN trick as
`v.docker.*`). Manual `helm-v*` tag flow still works. Chart.yaml
consistency check in `release-helm.yml` is the safety net if the bump
job ever drifts.
First rollout: the open `changeset-release/main` PR has stale
Chart.yaml. Bump it manually on that branch before merging, otherwise
the first automated helm release fails at the consistency check.
## Summary
Adds direct V8 heap and process-memory gauges to the webapp's
OpenTelemetry meter. The webapp already exports per-cluster-worker
Node.js runtime metrics (event-loop lag / utilization, active handles,
active requests, libuv threadpool size) via a custom meter under the
`trigger.dev` scope. Heap and memory were missing; this PR adds them
alongside, in the same observable-batch pattern.
## New gauges
| Metric | Source | Unit |
| --- | --- | --- |
| `nodejs.memory.heap.used` | `process.memoryUsage().heapUsed` | bytes |
| `nodejs.memory.heap.total` | `process.memoryUsage().heapTotal` | bytes
|
| `nodejs.memory.heap.limit` | `v8.getHeapStatistics().heap_size_limit`
| bytes |
| `nodejs.memory.external` | `process.memoryUsage().external` | bytes |
| `nodejs.memory.array_buffers` | `process.memoryUsage().arrayBuffers` |
bytes |
| `nodejs.memory.rss` | `process.memoryUsage().rss` | bytes |
Gated by the existing `INTERNAL_OTEL_NODEJS_METRICS_ENABLED` flag, same
as the adjacent event-loop / handle gauges. Zero overhead when disabled.
## Why
`@opentelemetry/host-metrics` publishes `process.memory.usage`, which is
RSS only. RSS is the sum of V8 heap, external memory (Buffers, etc.),
native code, and thread stacks. Without a direct heap metric it is not
possible to size the V8 heap cap (`--max-old-space-size`) from metrics
alone, because RSS overstates heap by the external + native footprint. A
worker can have a 4 GB RSS with a 2.5 GB heap and 1.5 GB of buffers; the
former constrains `--max-old-space-size`, the latter does not.
`nodejs.memory.heap.limit` also surfaces the configured
`--max-old-space-size` (read from
`v8.getHeapStatistics().heap_size_limit`), so operators can see the
current limit in the same dashboard as actual usage rather than
cross-referencing container environment variables.
## Risk
Minimal. Observable gauges are sampled at the configured metric-export
interval. `v8.getHeapStatistics()` and `process.memoryUsage()` are each
microsecond-level calls, and six gauges are added to the same batch
callback that already reads ~20 other Node.js runtime values per sample.
Same registration pattern as the existing event-loop metrics in the
file.
## Test plan
- [ ] Deploy and confirm the six new gauges appear at the configured
exporter
- [ ] In cluster mode, confirm per-worker granularity (one series per
cluster worker, tagged by `process.executable.name` /
`service.instance.id`)
- [ ] Confirm `nodejs.memory.heap.limit` reports the configured
`--max-old-space-size` value in bytes
## Summary
- New **Back office** tab at `/admin`, per-org detail page at
`/admin/back-office/orgs/:orgId` designed to host future per-org admin
actions (project count, delete account, YC deals).
- First action: edit an organization's API rate limit — tokenBucket
override (refill rate, interval, max tokens), with a live plain-English
preview (e.g. *"1,500 requests per minute · 750 request burst
allowance"*). Writes are audit-logged via the server logger.
- Cleanup: removed unused `v2?` / `v3?` columns from the admin orgs list
(display only — Prisma select untouched).
## Test plan
- [ ] Back office tab visible in admin nav and highlighted when on a
sub-route
- [ ] `/admin/orgs` shows a Back office "Open" link per row; no v2/v3
columns
- [ ] Empty state at `/admin/back-office` links back to `/admin/orgs`
- [ ] Detail page renders the effective rate limit in view mode; Edit
reveals the form
- [ ] Save writes `Organization.apiRateLimiterConfig`, returns to view
mode, shows "Rate limit saved." banner
- [ ] Invalid values surface inline field errors and keep edit mode
- [ ] Non-admins hitting any new route are redirected to `/`
- [ ] Server logs show `admin.backOffice.rateLimit` info line per
mutation
## Summary
Fixes a server-side memory leak in the webapp's SSE helper. Every
aborted SSE connection (client tab close, navigation, timeout) was
pinning its full request/response graph indefinitely on Node 20, so any
long-running webapp process accumulated retained memory proportional to
streaming-request churn.
## Root cause
`apps/webapp/app/utils/sse.ts` combined four abort signals via
`AbortSignal.any([requestAbortSignal, timeoutSignal,
internalController.signal])`. The composite signal tracks its source
signals in an internal `Set<WeakRef>` registered against a
`FinalizationRegistry`; under sustained traffic those entries accumulate
faster than they're cleaned up, pinning every source signal (and its
listeners, and anything those listeners close over) until the parent
signal itself is GC'd or aborts.
This is a long-standing Node issue with multiple open reports:
- [nodejs/node#54614](https://github.com/nodejs/node/issues/54614) —
original report, still open. A [follow-up from
ChainSafe](https://github.com/nodejs/node/issues/54614#issuecomment-4055656572)
describes the exact same shape in a Lodestar production workload (req +
timeout signals composed per request accumulating in long-running
worker) and the same mitigation: drop `AbortSignal.any`, compose
manually.
- [nodejs/node#55351](https://github.com/nodejs/node/issues/55351) —
mechanism confirmed by Node member @jasnell: *"the set of dependent
signals known to the AbortSignal are kept in an internal Set using
WeakRefs. The AbortSignals are being properly gc'd but the Set is never
cleaned out of the WeakRefs making those leak."* Partially fixed by [PR
#55354](https://github.com/nodejs/node/pull/55354), shipped in Node
22.12.0 — but only covers the tight-loop case, not long-lived parent
signals.
- [nodejs/node#57584](https://github.com/nodejs/node/issues/57584) —
circular-dependency variant, still open.
- [nodejs/node#62363](https://github.com/nodejs/node/issues/62363) —
regression in Node 24/25 from an unrelated V8 change ("Don't pretenure
WeakCells"). Different root cause, same symptom.
A separate issue in `apps/webapp/app/entry.server.tsx` —
`setTimeout(abort, ABORT_DELAY)` with no `clearTimeout` on success paths
— kept the React render tree + `remixContext` alive for 30s per
successful HTML request. Same pattern fixed upstream in React Router
templates
([react-router#14200](https://github.com/remix-run/react-router/pull/14200)),
never backported to Remix v2.
## What changed
- **`apps/webapp/app/utils/sse.ts`** — single-signal abort chain.
`AbortSignal.any` removed; `AbortSignal.timeout` replaced by a plain
`setTimeout` cleared when the controller aborts; named sentinel
constants used as stackless abort reasons; request-abort handler
explicitly removed on cleanup.
- **`apps/webapp/app/entry.server.tsx`** — clears the `setTimeout(abort,
ABORT_DELAY)` timer in `onShellReady` / `onAllReady` / `onShellError`.
- **`apps/webapp/app/v3/tracer.server.ts` + `env.server.ts`** — gates
OpenTelemetry `HttpInstrumentation` and `ExpressInstrumentation` behind
`DISABLE_HTTP_INSTRUMENTATION=true` as an escape hatch for future
OTel-listener retention patterns. Defaults to enabled.
- **`apps/webapp/app/presenters/v3/RunStreamPresenter.server.ts`** —
uses the shared `ABORT_REASON_SEND_ERROR` sentinel.
## Verification
### Full-app reproduction (memlab)
Isolated local harness, 500 abrupt SSE disconnects against a
dev-presence route, GC between passes, heap snapshot diff with
[memlab](https://facebook.github.io/memlab/):
| Run | Heap delta after 500 conns + GC | memlab retained leaks |
| --- | --- | --- |
| Before | +16.0 MB (linear with request count) | 158 clusters; 250
`ServerResponse`, 1000 `AbortController`, 250 `SpanImpl` retained |
| After | **+3.3 MB (noise)** | **0 app-code leaks** |
### Standalone mechanism isolation
To confirm *which* axis of the change is load-bearing, a separate
standalone Node script (`/tmp/abort-leak-test.mjs`) ran 2000 requests ×
200 KB payload per variant:
| Variant | Heap delta after GC |
| --- | --- |
| baseline (no signal machinery) | 0 MB |
| V1: `AbortSignal.any` + string abort reason | **+9.1 MB** |
| V2: `AbortSignal.any` only (no reason) | **+10.8 MB** |
| V3: string reason only (no `AbortSignal.any`) | 0 MB |
| V4: neither (the fix) | 0 MB |
| V5: `AbortSignal.any` with no listener on the composite | **+10.2 MB**
|
This proves `AbortSignal.any` is the sole mechanism. The reason type
(`.abort()` vs `.abort("string")`) is irrelevant for retention — V3 is
clean, V5 leaks even without a listener on the composite.
## Risk
- `sse.ts` is used by the dev-presence routes. Behaviour is equivalent —
timeouts and client disconnects still abort the stream. `signal.reason`
is now a named string sentinel (`"timeout"`, `"request_aborted"`, etc.)
instead of the previous string arg or default `AbortError`. No in-tree
reader of `signal.reason` exists.
- `entry.server.tsx` change is a standard cleanup of an abort timer,
matches upstream React Router guidance.
- `tracer.server.ts` change is env-gated and defaults to current
behaviour.
- Three other webapp `AbortSignal.timeout()` callsites (alert delivery,
remote-build status) are fire-and-forget passed directly to `fetch` —
not composed with anything long-lived, no retention risk, untouched.
## Test plan
- [ ] Existing SSE integration tests pass
- [ ] Dev-presence SSE behaves normally across tab open/close cycles
- [ ] No heap growth under sustained aborted-connection traffic (heap
snapshot diff)
## Follow-up
The same `AbortSignal.any([userSignal, internalSignal])` pattern exists
in several SDK/core callsites that ship to customers
(`packages/core/src/v3/realtimeStreams/manager.ts`,
`packages/trigger-sdk/src/v3/{ai,chat,chat-client,sessions}.ts`,
`packages/core/src/v3/workers/warmStartClient.ts`). Whether those leak
in practice depends on the user passing a long-lived signal. Tracked
separately.
Mirrors the existing `supervisor.serviceAccount` pattern onto webapp so
operators can annotate the SA (IRSA `eks.amazonaws.com/role-arn`,
Workload Identity, etc.) or bring their own SA. Without this,
`webapp.serviceAccount.annotations` isn't exposed and operators have to
patch the SA out-of-band.
```yaml
webapp:
serviceAccount:
create: true
name: ""
annotations:
eks.amazonaws.com/role-arn: arn:aws:iam::123456789012:role/trigger-webapp
```
Three pieces, same as supervisor:
- `webapp.serviceAccount.create` toggle on the SA block
- `webapp.serviceAccount.annotations` + `name` values
- `trigger-v4.webappServiceAccountName` helper, used by the SA, the
token-syncer RoleBinding subject, and the Deployment's
`serviceAccountName`
Role + RoleBinding are left unguarded (matching supervisor's shape where
`rbac.create` is a separate toggle from `serviceAccount.create`) -
BYO-SA users take on the responsibility of ensuring the SA they supply
has the permissions the RoleBinding grants.
Verified with `helm template` against default values, an IRSA annotation
override, and `create: false` with a custom name.
## Problem
When `batchTrigger()` is called with large payloads, each item's payload
is uploaded to R2 server-side during the streaming loop before being
enqueued. This makes the loop slow — around 3 seconds per item. Workers
pick up and execute each item as it's enqueued, running concurrently
with the ongoing stream.
For the last item in the batch, a race exists between the streaming loop
finishing and the batch completion cleanup:
1. The loop enqueues the last item and returns from `enqueueBatchItem()`
2. A waiting worker picks up the item almost instantly and executes it
3. `recordSuccess()` fires, `processedCount` hits the expected total,
`finalizeBatch()` runs
4. `cleanup()` deletes all Redis keys for the batch, including
`enqueuedItemsKey`
5. The streaming loop exits and calls `getBatchEnqueuedCount()` — reads
the now-deleted key — returns 0
The count check finds `enqueuedCount (0) !== batch.runCount`, falls
through to a Postgres fallback, but the fallback only checked `sealed`.
The BatchQueue completion path sets `status = COMPLETED` in Postgres
without setting `sealed = true` (that's the streaming endpoint's job),
so the fallback misses it too.
This causes the endpoint to return `sealed: false`. The SDK treats this
as retryable and retries up to 5 times with exponential backoff. Each
retry calls `enqueueBatchItem()`, which reads the batch meta key from
Redis — also deleted by `cleanup()` — and throws "Batch not found or not
initialized" (500). The final retry gets a 422 because the batch is
already COMPLETED, which the SDK does not retry, causing an `ApiError`
to be thrown from `await batchTrigger()` in the parent run — even though
all child runs completed successfully.
## Fix
In the Postgres fallback inside `StreamBatchItemsService`, also check
`status === "COMPLETED"` alongside `sealed`. This covers the
fast-completion path where the BatchQueue finishes all runs before the
streaming endpoint gets to seal the batch normally.
Also switches `findUnique` to `findFirst` per webapp convention.
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
## Summary
Adds `RUN_ENGINE_READ_REPLICA_SNAPSHOTS_SINCE_ENABLED` (default `"0"`).
When enabled, the Prisma reads inside `RunEngine.getSnapshotsSince` run
against the read-only replica client instead of the primary. Offloads
the snapshot-polling queries fired by every running task runner off the
writer.
## Why
`getSnapshotsSince` is called from the managed runner's
fetch-and-process loop (once per poll interval, plus on every
snapshot-change notification). It runs four sequential reads per call —
one `findFirst` by snapshot id, one `findMany` on snapshots with
`createdAt > X`, one raw SQL against `_completedWaitpoints`, and chunked
`findMany` on `waitpoint`. Per concurrent run, every few seconds. It's
read-only, tolerates a small amount of staleness, and is an obvious
candidate for the replica.
## Replica-lag considerations
- **Step 1 "since snapshot not found"**: if the runner just received a
snapshot id from the primary and asks the replica before it replicates,
the function throws and the caller treats the response as an error
(runner falls back to a metadata refresh). Self-correcting, not silent.
- **Step 2 missing newly-created snapshots**: the next poll's `createdAt
> sinceSnapshot.createdAt` filter still picks them up once the replica
catches up.
- **Waitpoint junction race**: the riskiest path — if a latest snapshot
is replicated but its `_completedWaitpoints` join rows aren't yet, the
runner could advance past that snapshot with `completedWaitpoints: []`.
WAL/storage-level replication replays commits in order, so in practice
both should appear atomically on the reader, but the race window is why
the flag ships disabled.
Aurora reader shrinks all three windows to single-digit ms in typical
conditions, and its storage-level replication gives atomic visibility of
committed transactions on the reader.
## Test plan
- [ ] Flip the flag on in a non-prod environment, confirm snapshot
polling behaves normally and `getSnapshotsSince` errors in Sentry stay
flat.
- [ ] Verify writer query volume drops and reader query volume rises on
the snapshot-polling queries.
- [ ] Keep an eye on `AuroraReplicaLag` (or equivalent) during rollout.
## Summary
Stamp every Prisma span with `db.datasource: "writer" | "replica"` so
traces can distinguish which client the query went through.
Both `PrismaClient` instances share the same global
`@prisma/instrumentation`, so their spans come out with identical names
and attributes today. This makes them trivially filterable.
## How
Two pieces in `apps/webapp/app/`:
1. **`v3/tracer.server.ts`** — a `DatasourceAttributeSpanProcessor`
reads an OTel context key in `onStart` and calls
`span.setAttribute("db.datasource", value)`. Registered as the first
span processor.
2. **`db.server.ts`** — `tagDatasource(datasource, client)` wraps each
`PrismaClient` with `$extends({ query: { $allOperations } })`. The
middleware sets the context key around the query and directly tags the
active span (to catch `prisma:client:operation`, which Prisma creates
before the middleware fires).
### Context-propagation gotcha
`PrismaPromise` is lazy — `query(args)` returns a thenable that only
starts when someone `.then()`s it. The naive `context.with(ctx, () =>
query(args))` restores ALS synchronously, so when Prisma's internal code
awaits the thenable later, the engine spans fire with the original ALS.
Wrapping as `async () => await query(args)` forces the `.then()` inside
the `context.with` callback, so ALS stays on our context for the engine
spans.
### Coverage
- **Tagged**: all `prisma:engine:*` (`connection`, `db_query`,
`serialize`, `query`, etc.), `prisma:client:operation`,
`prisma:client:serialize`, `prisma:client:connect`
- **Not tagged**: `prisma:client:load_engine` — one-time startup, fires
before any query
Concurrent `Promise.all([writer.x, replica.y])` correctly tags each pool
separately (ALS isolates per-Promise chain).
### Performance
One `context.with` (~200ns) and one `setAttribute` per span (effectively
free per OTel JS benchmarks) per Prisma op. Negligible against a query
path measured in milliseconds.
## Test plan
- [ ] Verify `db.datasource` appears on `prisma:engine:connection` spans
after the webapp is restarted
- [ ] Spot-check a handful of real traces carry the attribute
The cancel button was missing from the runs list for runs in `DEQUEUED`
status. The runs list gates the button on `run.isCancellable`, which
goes through `isCancellableRunStatus` -> `CANCELLABLE_RUN_STATUSES` =
`NON_FINAL_RUN_STATUSES`. `DEQUEUED` was never added to that list when
it was introduced in the run engine.
The single run page uses a separate check (`!run.isFinished`, i.e. the
inverse of `FINAL_RUN_STATUSES`), so cancellation already worked there -
only the list was affected.
Adding `DEQUEUED` to `NON_FINAL_RUN_STATUSES` also flips
`isCrashableRunStatus` and `isFailableRunStatus`, but:
- The crash path is the right behaviour - a `DEQUEUED` run (worker has
claimed but not yet executing) can legitimately crash before
`EXECUTING`, same as `PENDING`/`DELAYED` already do.
- The fail path (`failedTaskRun.server.ts`) is only reached from V1 code
paths (marqs consumers, v1 heartbeat handler). `DEQUEUED` is a
V2-engine-only status, so V1 consumers never see it.
When cancelling a `DEQUEUED` run the execution snapshot goes to
`PENDING_CANCEL` (worker must ack) but `TaskRun.status` flips to
`CANCELED` immediately - the UI reflects cancellation without waiting
for the worker. Added an integration test in
`run-engine/src/engine/tests/cancelling.test.ts` covering the full
trigger -> dequeue -> cancel -> worker-ack flow.
## Stall safety
The stall recovery path (PENDING_EXECUTING heartbeat miss ->
nack-and-requeue -> back to QUEUED) lives entirely inside
`@internal/run-engine` and never touches the webapp's `taskStatus.ts`
helpers - the engine has zero imports from `~/v3/taskStatus` and doesn't
know `CrashTaskRunService` / `FailedTaskRunService` exist. A stalled
DEQUEUED run still goes back to the queue for retry; this change cannot
cause stalls to crash or fail.
The only realistic impact is the intended UI fix - the theoretical V1
crash/fail branches for DEQUEUED are unreachable in practice because V1
runs never have DEQUEUED status.
## Summary
Regenerating a RuntimeEnvironment API key no longer immediately
invalidates the previous one. Rotation is now overlap-based: the old key
keeps working for 24 hours so customers can roll it out in their env
vars without downtime, then stops working.
## Design
- **New `RevokedApiKey` table** (one row per revocation). Holds the
archived `apiKey`, a FK to the env, an `expiresAt`, and a `createdAt`.
Indexed on `apiKey` (high-cardinality equality — single-row hits) and on
`runtimeEnvironmentId`.
- **`regenerateApiKey` wraps both writes in a single `$transaction`:**
insert a `RevokedApiKey` with `expiresAt = now + 24h`, update the env
with the new `apiKey`/`pkApiKey`.
- **`findEnvironmentByApiKey` does a two-step lookup:** primary
unique-index hit on `RuntimeEnvironment.apiKey` first; on miss,
`RevokedApiKey.findFirst({ apiKey, expiresAt: { gt: now } })` with an
`include: { runtimeEnvironment }`. Two-step (not `OR`-join) keeps the
hot path identical to today and puts the fallback cost only on invalid
keys. Both lookups use `$replica`.
- **Admin endpoint** `POST /admin/api/v1/revoked-api-keys/:id` accepts
`{ expiresAt }` and updates the row. Setting to `now` ends the grace
window immediately; setting to the future extends it.
- **Modal copy** on the regenerate dialog updated — previously warned of
downtime, now explains the 24h overlap.
## Why a separate table instead of columns on `RuntimeEnvironment`
- Keeps the hot auth path's primary lookup unchanged — no
OR/nullable-apiKey semantics to reason about.
- Naturally supports multiple in-flight grace windows (regenerate twice
in a day → two old keys valid until their independent expiries).
- FK + cascade cleans up correctly when an env is deleted; nothing to
backfill.
## Test plan
Verified locally against hello-world with dev and prod env keys:
- [x] baseline — current key authenticates (`GET /api/v1/runs`) → `200`
- [x] regenerate via UI — DB shows old key in `RevokedApiKey` with
`expiresAt ≈ now+24h`, env has new key
- [x] grace window — both old and new keys → `200`; bogus key → `401`
- [x] admin endpoint: `expiresAt = now` → old key `401`
- [x] admin endpoint: `expiresAt = +1h` (after early-expire) → old key
`200` again
- [x] admin endpoint: `expiresAt = past` → old key `401`
- [x] admin 400 (invalid body), 404 (unknown id), 401 (missing/non-admin
PAT)
- [x] same flow exercised end-to-end on a PROD-typed env — behavior
identical
- [x] `pnpm run typecheck --filter webapp` passes
## ✅ Checklist
- [x] I have followed every step in the [contributing
guide](https://github.com/triggerdotdev/trigger.dev/blob/main/CONTRIBUTING.md)
- [x] The PR title follows the convention.
- [x] I ran and tested the code works
---
## Summary
Adds a server-side gate that detects deploy attempts from v3 CLI
versions (i.e. `trigger.dev@3.x`) at the `POST /api/v1/deployments`
entry point and, when enabled, rejects them with a clear upgrade
message. v4 CLI deploys are completely unaffected.
The last 3.x CLI release was `3.3.7`, which we can't update. This
approach short-circuits the deploy before any DB writes, image-ref
generation, S2 stream creation, or queue enqueue — no side effects in
either mode.
## How v3 vs v4 are distinguished
I pulled the published CLI tarballs for `trigger.dev@3.3.7`, `4.0.0`,
`4.0.1`, `4.0.5`, `4.1.0`, `4.2.0`, and the current `4.4.4` in the repo.
The cleanest, most reliable signal is the request body to `POST
/api/v1/deployments`:
| Field on initialize | v3.3.7 CLI | v4.x CLI |
|---|---|---|
| `type` | **never sent** | always sent — `"MANAGED"` (run_engine_v2) or
`"V1"` |
| `isNativeBuild` / `gitMeta` / `triggeredVia` / `runtime` | not sent |
sent |
| `registryHost` / `namespace` | sent (v3-only; stripped by current Zod
schema) | not sent |
Every v4 call site I inspected sets `type: features.run_engine_v2 ?
"MANAGED" : "V1"` unconditionally. `payload.type` is `undefined` if and
only if the client is a 3.x CLI.
## Behavior
- Detection always runs and emits `logger.warn("Detected deploy from
deprecated v3 CLI", { environmentId, projectId, organizationId, enforced
})`, which lets us watch how many v3 deploys are still happening before
enforcement is flipped.
- Enforcement is gated behind `DEPRECATE_V3_CLI_DEPLOYS_ENABLED`
(default `"0"`, off). When `"1"`, the server returns `400` with:
> The trigger.dev CLI v3 is no longer supported for deployments. Please
upgrade your project to v4: https://trigger.dev/docs/migrating-from-v3
The v3 CLI surfaces this verbatim as `Failed to start deployment:
<message>` because `zodfetch` throws `ApiError` for non-retryable 4xx
(400/422) and `deploy.js` in 3.3.7 prints `error.message`.
## Out of scope (intentionally)
- `api.v1.deployments.$deploymentId.finalize.ts` /
`FinalizeDeploymentService` /
`createDeploymentBackgroundWorkerV3.server.ts` are V1-engine paths, not
the v3 CLI gate. Leaving them alone per review.
- Container-side `createDeploymentBackgroundWorker` call in
`managed-index-controller.ts` is still used by v4's in-image indexer.
Not touched.
- v3 `trigger dev` flow (different code path) — separate deprecation
if/when needed.
## Testing
- Ran `pnpm run typecheck --filter webapp` locally — passes.
- Verified v4 tarballs (4.0.0, 4.0.1, 4.0.5, 4.1.0, 4.2.0, 4.4.4) all
include `type:` in the `initializeDeployment` call site, so none will be
accidentally blocked.
- Verified v3.3.7 tarball's `initializeDeployment` payload has no `type`
field.
Rollout plan after merge:
1. Deploy with `DEPRECATE_V3_CLI_DEPLOYS_ENABLED` unset → watch
`Detected deploy from deprecated v3 CLI` log volume.
2. When comfortable, set `DEPRECATE_V3_CLI_DEPLOYS_ENABLED=1` to
enforce.
---
## Changelog
Detect v3 CLI deploys on `/api/v1/deployments` and, when
`DEPRECATE_V3_CLI_DEPLOYS_ENABLED=1`, reject them with an upgrade
message pointing at https://trigger.dev/docs/migrating-from-v3. v4 CLI
deploys are unaffected.
Link to Devin session:
https://app.devin.ai/sessions/b242c11bd86e4099aeec8b59bab62143
Requested by: @ericallam
Example cURL call using an admin user PAT (replace with a real one):
```sh
curl -X PUT https://cloud.trigger.dev/admin/api/v1/environments/<environmentId>/burst-factor \
-H "Authorization: Bearer tr_pat_1234" \
-H "Content-Type: application/json" \
-d '{"burstFactor": 1.5}'
```
Adds a `MicroVM` badge next to the region name on the regions page. Uses
the existing `small` badge variant for visual consistency with the
`Default` badge already on this page.
## Summary
Large error stacks and messages can OOM the worker process when
serialized into OTel spans or `TaskRunError` objects. This was reported
when throwing an error with a massive `.stack` property from a chat
agent hook.
This adds frame-based stack truncation (similar to Sentry's approach)
plus message length limits, applied consistently across all error
serialization paths.
### What changed
**`packages/core/src/v3/errors.ts`**
- `truncateStack()` — parses `error.stack` into message lines + frame
lines, caps at 50 frames (keep top 5 closest to throw + bottom 45 entry
points, with "... N frames omitted ..." in between). Individual lines
capped at 1024 chars.
- `truncateMessage()` — caps error messages at 1000 chars
- Applied in `parseError()` and `sanitizeError()`
**`packages/core/src/v3/otel/utils.ts`**
- `sanitizeSpanError()` now uses `truncateStack` and `truncateMessage`
from `errors.ts` instead of duplicating truncation logic
- Non-Error values (strings, JSON) capped at 5000 chars
**`packages/core/src/v3/tracer.ts`**
- `startActiveSpan` catch block now delegates to `recordSpanException()`
instead of calling `span.recordException()` directly
### Limits
| What | Limit | Rationale |
|------|-------|-----------|
| Stack frames | 50 | Matches Sentry's `STACKTRACE_FRAME_LIMIT` |
| Top frames kept | 5 | Closest to throw site |
| Bottom frames kept | 45 | Entry points / framework frames |
| Per-line length | 1024 | Matches Sentry, prevents regex DoS |
| Message length | 1000 | Bounded but generous |
| Generic string (non-Error) | 5000 | Fallback for JSON/string errors in
spans |
## Test plan
- [x] 17 unit tests in `packages/core/test/errors.test.ts`
- [x] E2E: threw a 300-frame / 5000-char-message error in the ai-chat
reference app, verified truncated stack and message in span via
`get_span_details`
- [x] Verified the run survived the error (no OOM, continued waiting for
next message)
Two changes to cut error volume from logs that represent handled
conditions, not real errors (combined ~1600/hr in prod):
1. api.v1.waitpoints.tokens.$waitpointFriendlyId.complete.ts
The route throws `json(..., { status: 404 })` when a waitpoint
isn't found, but the generic catch block caught that Response,
logged it as an error (with an empty {} body because Error fields
are non-enumerable), and rethrew as a 500 — so clients saw a 500
instead of the intended 404, and every stale-waitpoint request
produced a Sentry event.
Fix: re-throw Response objects unchanged so the correct status
propagates and we don't log user 404s as errors. Also serialize
remaining Error instances explicitly (name/message/stack) so the
logs are actionable when we do hit a real error.
2. v3/marqs/sharedQueueConsumer.server.ts:603
"Task run has invalid status for execution. Going to ack" — the
message itself says we're handling it gracefully. Benign race
between dequeue and completion/cancellation. Demote to warn.
## Summary
Nests the `TaskRunExecutionSnapshot` creation inside the
`taskRun.update()` Prisma call in the dequeue flow, reducing **2 DB
commits → 1** per dequeue operation. This is the highest-volume of the
five unmerged flows identified in TRI-8450 (~9,200 commits/sec on the
engine service).
**Pattern**: Follows the same nested-write approach already used in the
completion path (`runAttemptSystem.ts:735`) and trigger path
(`engine/index.ts:674`).
**Changes**:
- `dequeueSystem.ts`: Moved snapshot creation into `executionSnapshots:
{ create: {...} }` within the existing `taskRun.update()`. Pre-generates
the snapshot ID via `generateInternalId()` (plain cuid, matching what
Prisma's `@default(cuid())` produces) so the event emission, heartbeat
enqueue, and return value can all be constructed from data already in
scope — **no extra DB read needed** after the merged write.
`SnapshotId.toFriendlyId()` is used only for the return value's
`friendlyId` field, matching the original `createExecutionSnapshot`
behavior.
- `executionSnapshotSystem.ts`: Added public
`enqueueHeartbeatIfNeeded()` method that exposes the heartbeat
scheduling logic (previously only available internally via
`createExecutionSnapshot`). This is needed because `PENDING_EXECUTING`
requires a heartbeat, unlike the `FINISHED` status in the completion
reference pattern. This method is reusable by future merge targets
(retry-immediate, checkpoint, cancel, requeue).
**Net DB change per dequeue**: eliminates 1 write transaction (the
separate `TaskRunExecutionSnapshot.create`). No extra reads added — the
snapshot ID is pre-generated and the `executionSnapshotCreated` event
payload is constructed inline from values already available in the
closure.
## Review & Testing Checklist for Human
- [ ] **Verify manually-constructed event payload matches DB state**:
The `executionSnapshotCreated` event is now built inline (not read back
from DB). Confirm the field values (`runStatus: "PENDING"`,
`attemptNumber`, `checkpointId`, `workerId`, `runnerId`,
`completedWaitpointIds`) match what Prisma actually writes. A mismatch
here would be silent — event consumers would get stale/wrong data.
- [ ] **Verify `attemptNumber` source is equivalent**: Old code used
`lockedTaskRun.attemptNumber` (post-update result). New code uses
`result.run.attemptNumber` (pre-update). The `taskRun.update()` data
payload does NOT include `attemptNumber`, so they should be identical —
but confirm this assumption holds for all dequeue scenarios (e.g.
retried runs).
- [ ] **Verify `isValid` defaults to `true` in schema**: The old
`createExecutionSnapshot` explicitly set `isValid: error ? false :
true`. The nested create omits `isValid` (no error in the dequeue happy
path). Confirm the Prisma schema default for
`TaskRunExecutionSnapshot.isValid` is `true`.
- [ ] **Verify `runStatus: "PENDING"` hardcoding matches the mapping**:
The old code passed `lockedTaskRun.status` ("DEQUEUED") to
`createExecutionSnapshot`, which mapped it to "PENDING" via `run.status
=== "DEQUEUED" ? "PENDING" : run.status`. The new code hardcodes
`"PENDING"` directly. This is correct but brittle if `status` ever
changes from "DEQUEUED" to something else upstream.
- [ ] **Spot-check `completedWaitpoints` connect + order logic**: The
nested create replicates the connect/order logic from
`createExecutionSnapshot` (lines 387-393). Verify the
`snapshot.completedWaitpoints` type provides `id` and `index` fields
compatible with this usage.
- [ ] **Verify `checkpoint` in return value**: The return now uses
`snapshot.checkpoint` (from the *previous* snapshot) instead of reading
the newly-created snapshot's checkpoint relation. Since `checkpointId`
is passed through unchanged, they should be identical — but worth a
sanity check.
**Recommended test plan**: deploy to staging, run the
`sample_pg_activity.py` sampler for a 5-minute window, and verify the
COMMIT count drop on the engine service + proportional `IO:XactSync`
reduction.
### Notes
- This only covers the **dequeue** flow (flow #1 from TRI-8450). The
remaining four flows (retry-immediate, checkpoint, requeue, cancel) are
separate follow-ups.
- The new `enqueueHeartbeatIfNeeded` method is deliberately designed for
reuse by those follow-up PRs.
- CI note: the `priority.test.ts` failure in shard 7 is a flaky ordering
assertion unrelated to this change (it compares `friendlyId` values in
dequeue order). The `audit` check is also pre-existing/unrelated.
Link to Devin session:
https://app.devin.ai/sessions/034fe0e7224f49278a2de260203e1377
Requested by: @ericallam
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: Eric Allam <eallam@icloud.com>
Wraps getEntitlement in platform.v3.server.ts with the existing
platformCache (LRU memory + Redis) under a new `entitlement` namespace.
Eliminates a synchronous billing-service HTTP round trip on every
trigger.
Cache config: 60s fresh / 60s stale SWR. Cache key is the
organization id. Errors are caught inside the loader and return the
existing permissive { hasAccess: true } fallback, which is also
cached to prevent thundering-herd on billing outages.
Trade-off: plan upgrade/downgrade is now visible after up to ~120s
worst-case (60s fresh + 60s stale revalidation). Acceptable since
the existing limits and usage namespaces use 5min/10min, and the
defensive hasAccess: true fallback already exists.
Replace the expensive DISTINCT query for task filter dropdowns with a
dedicated TaskIdentifier registry table backed by Redis. Environments
migrate automatically on their next deploy, with a transparent fallback
to the legacy query for unmigrated environments. Also fixes duplicate
dropdown entries when a task changes trigger source, and adds
active/archived grouping for removed tasks. Moves BackgroundWorkerTask
reads in the trigger hot path to the read replica.
Pool Redis connections for non-blocking ops (ingestData, appendPart,
getLastChunkIndex)
using a shared singleton instead of new Redis() per request. Use
redis.disconnect()
for immediate teardown in streamResponse cleanup. Add 15s inactivity
timeout fallback.
Fix broken request.signal in Remix/Express by wiring Express
res.on('close') to an
AbortController via httpAsyncStorage. All SSE/streaming routes now use
getRequestAbortSignal() which fires reliably on client disconnect,
bypassing the
Node.js undici GC bug (nodejs/node#55428) that severs the signal chain.
Extends the admin worker groups endpoint with a GET loader and more
fields on POST (type, hidden, workloadType, cloudProvider, location,
staticIPs, enableFastPath), and pulls the PAT + admin check that was
inlined or locally duplicated across every admin.api route into a shared
helper in personalAccessToken.server.ts. The generic
authenticateAdminRequest returns a discriminated result;
requireAdminApiRequest is the thin Remix loader/action wrapper that
throws. The neverthrow-style route (platform-notifications.ts) now
composes the generic helper instead of duplicating the check. Verified
locally against GET (listing) and POST (new fields, invalid enum,
minimal backwards-compat).
## Summary
Upgrades all `@remix-run/*` packages in `apps/webapp` from **2.1.0 →
2.17.4** to address security vulnerabilities. Recreation of #2951 on a
fresh checkout of `main`.
**Updated packages (`apps/webapp/package.json`):**
- `@remix-run/express`, `@remix-run/node`, `@remix-run/react`,
`@remix-run/serve`, `@remix-run/server-runtime`: 2.1.0 → 2.17.4
- `@remix-run/router`: ^1.15.3 → ^1.23.2
- `@remix-run/dev`, `@remix-run/eslint-config`, `@remix-run/testing`:
2.1.0 → 2.17.4
**Root `package.json` overrides:**
- `@remix-run/dev@2.17.4>tar-fs`: 2.1.3 → 2.1.4
- `testcontainers@10.28.0>tar-fs`: 3.0.9 → 3.1.1
**Documentation:** Updated Remix version references in `CLAUDE.md`,
`apps/webapp/CLAUDE.md`, and `.cursor/rules/webapp.mdc`.
**Server changes:** Added `.server-changes/upgrade-remix-security.md`
for release tracking per `CONTRIBUTING.md`.
No application code changes — only `package.json` files, documentation,
a server-changes entry, and the regenerated `pnpm-lock.yaml`.
### Updates since last revision
Addressed all 3 Devin Review findings:
1. **Missing `.server-changes/` file** — added
`.server-changes/upgrade-remix-security.md` (commit ce22a0bd4)
2. **Sentry Remix patch (`@sentry/remix@9.46.0`)** — verified the patch
at `patches/@sentry__remix@9.46.0.patch` applies cleanly against 2.17.4.
The patch modifies Sentry's own `RemixInstrumentation` wrapper (removing
`request.clone()` and form data attributes), not Remix internals. The
underlying Remix APIs it hooks into (`callRouteAction`,
`callRouteLoader`) are stable across 2.1→2.17.
3. **`remix-typedjson@0.3.1` compatibility** — peer deps declare
`@remix-run/react: ^1.16.0 || ^2.0`, covering 2.17.4. Confirmed working
at runtime across all 22 tested pages that use it (root.tsx, hooks,
route loaders).
### Verification performed during this session
- **Runtime:** Express+Remix integration, magic link login, client-side
routing, MetaFunction rendering
- **Operational:** hello-world task triggered via API, runs list, run
detail, tasks page
- **Comprehensive UI:** 22 pages, 11 filter types, environment/project
switchers, interactive elements
- **Docker:** Production Dockerfile (`docker/webapp/Dockerfile`) builds
successfully
- **Changelog audit:** All 16 minor versions reviewed — every breaking
change is behind opt-in future flags the webapp doesn't enable
## Review & Testing Checklist for Human
- [ ] **Verify auth flows in staging** — `remix-auth`,
`remix-auth-email-link`, and `remix-auth-github` declare peer deps on
`@remix-run/server-runtime@^1.x`, which is now 2.17.4. Login (magic link
+ OAuth) should be tested in a staging environment since local dev
testing may not exercise all auth code paths.
- [ ] **Verify tar-fs override versions** resolve the targeted security
advisories (2.1.4 and 3.1.1)
- [ ] **Review new transitive dependencies** added by the upgrade:
`turbo-stream@2.4.1`, `undici@6.25.0`, `valibot@1.3.1`, `ws@7.5.10`
Recommended test plan: deploy to staging and exercise core webapp flows
— login (email magic link + GitHub OAuth), dashboard navigation, task
triggering/viewing, and API endpoints — to catch runtime regressions not
covered by local testing.
### Notes
- Peer dependency warnings for `remix-auth-*` packages (expecting
`@remix-run/server-runtime@^1.x`) were present in the original PR #2951
as well and appear to be pre-existing
- The lockfile diff is large (~1200 lines) but mechanical — driven by
the Remix version bump cascading through transitive dependencies
- CI failures (`audit`, `units/internal/1-of-8`) are unrelated: `audit`
is a `claude-code-action` bot permissions issue; the internal test
failure is a ClickHouse testcontainers `Failed to connect to Reaper`
flake
Link to Devin session:
https://app.devin.ai/sessions/d9fa9953b9bf40e5a8d12b8f5ba5b86b
Requested by: @ericallam
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: Eric Allam <eallam@icloud.com>
Adds missing list deployments API page, fixes defaultMachine → machine
in config docs, and clarifies browser CORS usage for wait token
completion with corrected warning placement
The TaskRun.runTags string array already stores tag names, making the
TaskRunTag M2M relation redundant write overhead. Remove createTags
calls, connect: tags, and join table writes from both V1 and V2 trigger
paths. Simplify the add-tags API to just push to runTags directly.
## Summary
12 new features, 59 improvements, 17 bug fixes.
## Highlights
- Add support for setting TTL (time-to-live) defaults at the task level
and globally in trigger.config.ts, with per-trigger overrides still
taking precedence
([#3196](https://github.com/triggerdotdev/trigger.dev/pull/3196))
- Large run outputs can use the new API which allows switching object
storage providers.
([#3275](https://github.com/triggerdotdev/trigger.dev/pull/3275))
## Improvements
- Add platform notifications support to the CLI. The `trigger dev` and
`trigger login` commands now fetch and display platform notifications
(info, warn, error, success) from the server. Includes discovery-based
filtering to conditionally show notifications based on project file
patterns, color markup rendering for styled terminal output, and a
non-blocking display flow with a spinner fallback for slow fetches. Use
`--skip-platform-notifications` flag with `trigger dev` to disable the
notification check.
([#3254](https://github.com/triggerdotdev/trigger.dev/pull/3254))
- Add `get_span_details` MCP tool for inspecting individual spans within
a run trace.
([#3255](https://github.com/triggerdotdev/trigger.dev/pull/3255))
- New `get_span_details` tool returns full span attributes, timing,
events, and AI enrichment (model, tokens, cost, speed)
- Span IDs now shown in `get_run_details` trace output for easy
discovery
- New API endpoint `GET /api/v1/runs/:runId/spans/:spanId`
- New `retrieveSpan()` method on the API client
- `get_query_schema` — discover available TRQL tables and columns
- `query` — execute TRQL queries against your data
- `list_dashboards` — list built-in dashboards and their widgets
- `run_dashboard_query` — execute a single dashboard widget query
- `whoami` — show current profile, user, and API URL
- `list_profiles` — list all configured CLI profiles
- `switch_profile` — switch active profile for the MCP session
- `start_dev_server` — start `trigger dev` in the background and stream
output
- `stop_dev_server` — stop the running dev server
- `dev_server_status` — check dev server status and view recent logs
- `GET /api/v1/query/schema` — query table schema discovery
- `GET /api/v1/query/dashboards` — list built-in dashboards
- `--readonly` flag hides write tools (`deploy`, `trigger_task`,
`cancel_run`) so the AI cannot make changes
- `read:query` JWT scope for query endpoint authorization
- `get_run_details` trace output is now paginated with cursor support
- MCP tool annotations (`readOnlyHint`, `destructiveHint`) for all tools
- `get_query_schema` now requires a table name and returns only one
table's schema (was returning all tables)
- `get_current_worker` no longer inlines payload schemas; use new
`get_task_schema` tool instead
- Query results formatted as text tables instead of JSON (~50% fewer
tokens)
- `cancel_run`, `list_deploys`, `list_preview_branches` formatted as
text instead of raw JSON
- Schema and dashboard API responses cached to avoid redundant fetches
- Adapted the CLI API client to propagate the trigger source via http
headers.
([#3241](https://github.com/triggerdotdev/trigger.dev/pull/3241))
- Propagate run tags to span attributes so they can be extracted
server-side for LLM cost attribution metadata.
([#3213](https://github.com/triggerdotdev/trigger.dev/pull/3213))
- New `get_span_details` tool returns full span attributes, timing,
events, and AI enrichment (model, tokens, cost, speed)
- Span IDs now shown in `get_run_details` trace output for easy
discovery
- New API endpoint `GET /api/v1/runs/:runId/spans/:spanId`
- New `retrieveSpan()` method on the API client
- `get_query_schema` — discover available TRQL tables and columns
- `query` — execute TRQL queries against your data
- `list_dashboards` — list built-in dashboards and their widgets
- `run_dashboard_query` — execute a single dashboard widget query
- `whoami` — show current profile, user, and API URL
- `list_profiles` — list all configured CLI profiles
- `switch_profile` — switch active profile for the MCP session
- `start_dev_server` — start `trigger dev` in the background and stream
output
- `stop_dev_server` — stop the running dev server
- `dev_server_status` — check dev server status and view recent logs
- `GET /api/v1/query/schema` — query table schema discovery
- `GET /api/v1/query/dashboards` — list built-in dashboards
- `--readonly` flag hides write tools (`deploy`, `trigger_task`,
`cancel_run`) so the AI cannot make changes
- `read:query` JWT scope for query endpoint authorization
- `get_run_details` trace output is now paginated with cursor support
- MCP tool annotations (`readOnlyHint`, `destructiveHint`) for all tools
- `get_query_schema` now requires a table name and returns only one
table's schema (was returning all tables)
- `get_current_worker` no longer inlines payload schemas; use new
`get_task_schema` tool instead
- Query results formatted as text tables instead of JSON (~50% fewer
tokens)
- `cancel_run`, `list_deploys`, `list_preview_branches` formatted as
text instead of raw JSON
- Schema and dashboard API responses cached to avoid redundant fetches
- Add optional `hasPrivateLink` field to the dequeue message
organization object for private networking support
([#3264](https://github.com/triggerdotdev/trigger.dev/pull/3264))
- Define and manage AI prompts with `prompts.define()`. Create typesafe
prompt templates with variables, resolve them at runtime, and manage
versions and overrides from the dashboard without redeploying.
([#3244](https://github.com/triggerdotdev/trigger.dev/pull/3244))
## Bug fixes
- Fix dev CLI leaking build directories on rebuild, causing disk space
accumulation. Deprecated workers are now pruned (capped at 2 retained)
when no active runs reference them. The watchdog process also cleans up
`.trigger/tmp/` when the dev CLI is killed ungracefully (e.g. SIGKILL
from pnpm).
([#3224](https://github.com/triggerdotdev/trigger.dev/pull/3224))
- Fix `--load` flag being silently ignored on local/self-hosted builds.
([#3114](https://github.com/triggerdotdev/trigger.dev/pull/3114))
- Fixed `search_docs` tool failing due to renamed upstream Mintlify tool
(`SearchTriggerDev` → `search_trigger_dev`)
- Fixed `list_deploys` failing when deployments have null
`runtime`/`runtimeVersion` fields (#3139)
- Fixed `list_preview_branches` crashing due to incorrect response shape
access
- Fixed `metrics` table column documented as `value` instead of
`metric_value` in query docs
- Fixed dev CLI leaking build directories on rebuild — deprecated
workers now clean up their build dirs when their last run completes
- Fixed `search_docs` tool failing due to renamed upstream Mintlify tool
(`SearchTriggerDev` → `search_trigger_dev`)
- Fixed `list_deploys` failing when deployments have null
`runtime`/`runtimeVersion` fields (#3139)
- Fixed `list_preview_branches` crashing due to incorrect response shape
access
- Fixed `metrics` table column documented as `value` instead of
`metric_value` in query docs
- Fixed dev CLI leaking build directories on rebuild — deprecated
workers now clean up their build dirs when their last run completes
## Server changes
These changes affect the self-hosted Docker image and Trigger.dev Cloud:
- Add admin UI for viewing and editing feature flags (org-level
overrides and global defaults).
([#3291](https://github.com/triggerdotdev/trigger.dev/pull/3291))
- AI prompt management dashboard and enhanced span inspectors.
**Prompt management:**
- Prompts list page with version status, model, override indicators, and
24h usage sparklines
- Prompt detail page with template viewer, variable preview, version
history timeline, and override editor
- Create, edit, and remove overrides to change prompt content or model
without redeploying
- Promote any code-deployed version to current
- Generations tab with infinite scroll, live polling, and inline span
inspector
- Per-prompt metrics: total generations, avg tokens, avg cost, latency,
with version-level breakdowns
**AI span inspectors:**
- Custom inspectors for `ai.generateText`, `ai.streamText`,
`ai.generateObject`, `ai.streamObject` parent spans
- `ai.toolCall` inspector showing tool name, call ID, and input
arguments
- `ai.embed` inspector showing model, provider, and input text
- Prompt tab on AI spans linking to prompt version with template and
input variables
- Compact timestamp and duration header on all AI span inspectors
**AI metrics dashboard:**
- Operations, Providers, and Prompts filters on the AI Metrics dashboard
- Cost by prompt widget
- "AI" section in the sidebar with Prompts and AI Metrics links
**Other improvements:**
- Resizable panel sizes now persist across page refreshes
- Fixed `<div>` inside `<p>` DOM nesting warnings in span titles and
chat messages
([#3244](https://github.com/triggerdotdev/trigger.dev/pull/3244))
- Add allowRollbacks query param to the promote deployment API to enable
version downgrades
([#3214](https://github.com/triggerdotdev/trigger.dev/pull/3214))
- Pre-warm compute templates on deploy for orgs with compute access.
Required for projects using a compute region, background-only for
others.
([#3114](https://github.com/triggerdotdev/trigger.dev/pull/3114))
- Add automatic LLM cost calculation for spans with GenAI semantic
conventions. When a span arrives with `gen_ai.response.model` and token
usage data, costs are calculated from an in-memory pricing registry
backed by Postgres and dual-written to both span attributes
(`trigger.llm.*`) and a new `llm_metrics_v1` ClickHouse table that
captures usage, cost, performance (TTFC, tokens/sec), and behavioral
(finish reason, operation type) metrics.
([#3213](https://github.com/triggerdotdev/trigger.dev/pull/3213))
- Add API endpoint `GET /api/v1/runs/:runId/spans/:spanId` that returns
detailed span information including properties, events, AI enrichment
(model, tokens, cost), and triggered child runs.
([#3255](https://github.com/triggerdotdev/trigger.dev/pull/3255))
- Multi-provider object storage with protocol-based routing for
zero-downtime migration
([#3275](https://github.com/triggerdotdev/trigger.dev/pull/3275))
- Add IAM role-based auth support for object stores (no access keys
required).
([#3275](https://github.com/triggerdotdev/trigger.dev/pull/3275))
- Add platform notifications to inform users about new features,
changelogs, and platform events directly in the dashboard.
([#3254](https://github.com/triggerdotdev/trigger.dev/pull/3254))
- Add private networking support via AWS PrivateLink. Includes
BillingClient methods for managing private connections, org settings UI
pages for connection management, and supervisor changes to apply
`privatelink` pod labels for CiliumNetworkPolicy matching.
([#3264](https://github.com/triggerdotdev/trigger.dev/pull/3264))
- Reduce run start latency by skipping the intermediate queue when
concurrency is available. This optimization is rolled out per-region and
enabled automatically for development environments.
([#3299](https://github.com/triggerdotdev/trigger.dev/pull/3299))
- Extended the search filter on the environment variables page to match
on environment type (production, staging, development, preview) and
branch name, not just variable name and value.
([#3302](https://github.com/triggerdotdev/trigger.dev/pull/3302))
- Set `application_name` on Prisma connections from SERVICE_NAME so DB
load can be attributed by service
([#3348](https://github.com/triggerdotdev/trigger.dev/pull/3348))
- Fix transient R2/object store upload failures during batchTrigger()
item streaming.
- Added p-retry (3 attempts, 500ms–2s exponential backoff) around
`uploadPacketToObjectStore` in `BatchPayloadProcessor.process()` so
transient network errors self-heal server-side rather than aborting the
entire batch stream.
- Removed `x-should-retry: false` from the 500 response on the batch
items route so the SDK's existing 5xx retry path can recover if
server-side retries are exhausted. Item deduplication by index makes
full-stream retries safe.
([#3331](https://github.com/triggerdotdev/trigger.dev/pull/3331))
- Concurrency-keyed queues now use a single master queue entry per base
queue instead of one entry per key. Prevents high-CK-count tenants from
consuming the entire parentQueueLimit window and starving other tenants
on the same shard.
([#3219](https://github.com/triggerdotdev/trigger.dev/pull/3219))
- Reduce lock contention when processing large `batchTriggerAndWait`
batches. Previously, each batch item acquired a Redis lock on the parent
run to insert a `TaskRunWaitpoint` row, causing
`LockAcquisitionTimeoutError` with high concurrency (880 errors/24h in
prod). Since `blockRunWithCreatedBatch` already transitions the parent
to `EXECUTING_WITH_WAITPOINTS` before items are processed, the per-item
lock is unnecessary. The new `blockRunWithWaitpointLockless` method
performs only the idempotent CTE insert without acquiring the lock.
([#3232](https://github.com/triggerdotdev/trigger.dev/pull/3232))
- Strip `secure` query parameter from QUERY_CLICKHOUSE_URL before
passing to ClickHouse client. This was already done for the main and
logs ClickHouse clients but was missing for the query client, causing a
startup crash with `Error: Unknown URL parameters: secure`.
([#3204](https://github.com/triggerdotdev/trigger.dev/pull/3204))
- Fix `OrganizationsPresenter.#getEnvironment` matching the wrong
development environment on teams with multiple members. All dev
environments share the slug `"dev"`, so the previous `find` by slug
alone could return another member's environment. Now filters DEVELOPMENT
environments by `orgMember.userId` to ensure the logged-in user's dev
environment is selected.
([#3273](https://github.com/triggerdotdev/trigger.dev/pull/3273))
<details>
<summary>Raw changeset output</summary>
# Releases
## @trigger.dev/build@4.4.4
### Patch Changes
- Updated dependencies:
- `@trigger.dev/core@4.4.4`
## trigger.dev@4.4.4
### Patch Changes
- Add platform notifications support to the CLI. The `trigger dev` and
`trigger login` commands now fetch and display platform notifications
(info, warn, error, success) from the server. Includes discovery-based
filtering to conditionally show notifications based on project file
patterns, color markup rendering for styled terminal output, and a
non-blocking display flow with a spinner fallback for slow fetches. Use
`--skip-platform-notifications` flag with `trigger dev` to disable the
notification check.
([#3254](https://github.com/triggerdotdev/trigger.dev/pull/3254))
- Fix dev CLI leaking build directories on rebuild, causing disk space
accumulation. Deprecated workers are now pruned (capped at 2 retained)
when no active runs reference them. The watchdog process also cleans up
`.trigger/tmp/` when the dev CLI is killed ungracefully (e.g. SIGKILL
from pnpm).
([#3224](https://github.com/triggerdotdev/trigger.dev/pull/3224))
- Fix `--load` flag being silently ignored on local/self-hosted builds.
([#3114](https://github.com/triggerdotdev/trigger.dev/pull/3114))
- Add `get_span_details` MCP tool for inspecting individual spans within
a run trace.
([#3255](https://github.com/triggerdotdev/trigger.dev/pull/3255))
- New `get_span_details` tool returns full span attributes, timing,
events, and AI enrichment (model, tokens, cost, speed)
- Span IDs now shown in `get_run_details` trace output for easy
discovery
- New API endpoint `GET /api/v1/runs/:runId/spans/:spanId`
- New `retrieveSpan()` method on the API client
- MCP server improvements: new tools, bug fixes, and new flags.
([#3224](https://github.com/triggerdotdev/trigger.dev/pull/3224))
**New tools:**
- `get_query_schema` — discover available TRQL tables and columns
- `query` — execute TRQL queries against your data
- `list_dashboards` — list built-in dashboards and their widgets
- `run_dashboard_query` — execute a single dashboard widget query
- `whoami` — show current profile, user, and API URL
- `list_profiles` — list all configured CLI profiles
- `switch_profile` — switch active profile for the MCP session
- `start_dev_server` — start `trigger dev` in the background and stream
output
- `stop_dev_server` — stop the running dev server
- `dev_server_status` — check dev server status and view recent logs
**New API endpoints:**
- `GET /api/v1/query/schema` — query table schema discovery
- `GET /api/v1/query/dashboards` — list built-in dashboards
**New features:**
- `--readonly` flag hides write tools (`deploy`, `trigger_task`,
`cancel_run`) so the AI cannot make changes
- `read:query` JWT scope for query endpoint authorization
- `get_run_details` trace output is now paginated with cursor support
- MCP tool annotations (`readOnlyHint`, `destructiveHint`) for all tools
**Bug fixes:**
- Fixed `search_docs` tool failing due to renamed upstream Mintlify tool
(`SearchTriggerDev` → `search_trigger_dev`)
- Fixed `list_deploys` failing when deployments have null
`runtime`/`runtimeVersion` fields (#3139)
- Fixed `list_preview_branches` crashing due to incorrect response shape
access
- Fixed `metrics` table column documented as `value` instead of
`metric_value` in query docs
- Fixed dev CLI leaking build directories on rebuild — deprecated
workers now clean up their build dirs when their last run completes
**Context optimizations:**
- `get_query_schema` now requires a table name and returns only one
table's schema (was returning all tables)
- `get_current_worker` no longer inlines payload schemas; use new
`get_task_schema` tool instead
- Query results formatted as text tables instead of JSON (~50% fewer
tokens)
- `cancel_run`, `list_deploys`, `list_preview_branches` formatted as
text instead of raw JSON
- Schema and dashboard API responses cached to avoid redundant fetches
- Add support for setting TTL (time-to-live) defaults at the task level
and globally in trigger.config.ts, with per-trigger overrides still
taking precedence
([#3196](https://github.com/triggerdotdev/trigger.dev/pull/3196))
- Adapted the CLI API client to propagate the trigger source via http
headers.
([#3241](https://github.com/triggerdotdev/trigger.dev/pull/3241))
- Updated dependencies:
- `@trigger.dev/core@4.4.4`
- `@trigger.dev/build@4.4.4`
- `@trigger.dev/schema-to-json@4.4.4`
## @trigger.dev/core@4.4.4
### Patch Changes
- Fix `list_deploys` MCP tool failing when deployments have null
`runtime` or `runtimeVersion` fields.
([#3224](https://github.com/triggerdotdev/trigger.dev/pull/3224))
- Propagate run tags to span attributes so they can be extracted
server-side for LLM cost attribution metadata.
([#3213](https://github.com/triggerdotdev/trigger.dev/pull/3213))
- Add `get_span_details` MCP tool for inspecting individual spans within
a run trace.
([#3255](https://github.com/triggerdotdev/trigger.dev/pull/3255))
- New `get_span_details` tool returns full span attributes, timing,
events, and AI enrichment (model, tokens, cost, speed)
- Span IDs now shown in `get_run_details` trace output for easy
discovery
- New API endpoint `GET /api/v1/runs/:runId/spans/:spanId`
- New `retrieveSpan()` method on the API client
- MCP server improvements: new tools, bug fixes, and new flags.
([#3224](https://github.com/triggerdotdev/trigger.dev/pull/3224))
**New tools:**
- `get_query_schema` — discover available TRQL tables and columns
- `query` — execute TRQL queries against your data
- `list_dashboards` — list built-in dashboards and their widgets
- `run_dashboard_query` — execute a single dashboard widget query
- `whoami` — show current profile, user, and API URL
- `list_profiles` — list all configured CLI profiles
- `switch_profile` — switch active profile for the MCP session
- `start_dev_server` — start `trigger dev` in the background and stream
output
- `stop_dev_server` — stop the running dev server
- `dev_server_status` — check dev server status and view recent logs
**New API endpoints:**
- `GET /api/v1/query/schema` — query table schema discovery
- `GET /api/v1/query/dashboards` — list built-in dashboards
**New features:**
- `--readonly` flag hides write tools (`deploy`, `trigger_task`,
`cancel_run`) so the AI cannot make changes
- `read:query` JWT scope for query endpoint authorization
- `get_run_details` trace output is now paginated with cursor support
- MCP tool annotations (`readOnlyHint`, `destructiveHint`) for all tools
**Bug fixes:**
- Fixed `search_docs` tool failing due to renamed upstream Mintlify tool
(`SearchTriggerDev` → `search_trigger_dev`)
- Fixed `list_deploys` failing when deployments have null
`runtime`/`runtimeVersion` fields (#3139)
- Fixed `list_preview_branches` crashing due to incorrect response shape
access
- Fixed `metrics` table column documented as `value` instead of
`metric_value` in query docs
- Fixed dev CLI leaking build directories on rebuild — deprecated
workers now clean up their build dirs when their last run completes
**Context optimizations:**
- `get_query_schema` now requires a table name and returns only one
table's schema (was returning all tables)
- `get_current_worker` no longer inlines payload schemas; use new
`get_task_schema` tool instead
- Query results formatted as text tables instead of JSON (~50% fewer
tokens)
- `cancel_run`, `list_deploys`, `list_preview_branches` formatted as
text instead of raw JSON
- Schema and dashboard API responses cached to avoid redundant fetches
- Large run outputs can use the new API which allows switching object
storage providers.
([#3275](https://github.com/triggerdotdev/trigger.dev/pull/3275))
- Add optional `hasPrivateLink` field to the dequeue message
organization object for private networking support
([#3264](https://github.com/triggerdotdev/trigger.dev/pull/3264))
- Add support for setting TTL (time-to-live) defaults at the task level
and globally in trigger.config.ts, with per-trigger overrides still
taking precedence
([#3196](https://github.com/triggerdotdev/trigger.dev/pull/3196))
- Adapted the CLI API client to propagate the trigger source via http
headers.
([#3241](https://github.com/triggerdotdev/trigger.dev/pull/3241))
## @trigger.dev/python@4.4.4
### Patch Changes
- Updated dependencies:
- `@trigger.dev/sdk@4.4.4`
- `@trigger.dev/core@4.4.4`
- `@trigger.dev/build@4.4.4`
## @trigger.dev/react-hooks@4.4.4
### Patch Changes
- Updated dependencies:
- `@trigger.dev/core@4.4.4`
## @trigger.dev/redis-worker@4.4.4
### Patch Changes
- Adapted the CLI API client to propagate the trigger source via http
headers.
([#3241](https://github.com/triggerdotdev/trigger.dev/pull/3241))
- Updated dependencies:
- `@trigger.dev/core@4.4.4`
## @trigger.dev/rsc@4.4.4
### Patch Changes
- Updated dependencies:
- `@trigger.dev/core@4.4.4`
## @trigger.dev/schema-to-json@4.4.4
### Patch Changes
- Updated dependencies:
- `@trigger.dev/core@4.4.4`
## @trigger.dev/sdk@4.4.4
### Patch Changes
- Define and manage AI prompts with `prompts.define()`. Create typesafe
prompt templates with variables, resolve them at runtime, and manage
versions and overrides from the dashboard without redeploying.
([#3244](https://github.com/triggerdotdev/trigger.dev/pull/3244))
- Add support for setting TTL (time-to-live) defaults at the task level
and globally in trigger.config.ts, with per-trigger overrides still
taking precedence
([#3196](https://github.com/triggerdotdev/trigger.dev/pull/3196))
- Adapted the CLI API client to propagate the trigger source via http
headers.
([#3241](https://github.com/triggerdotdev/trigger.dev/pull/3241))
- Updated dependencies:
- `@trigger.dev/core@4.4.4`
</details>
---------
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Adds region-level gating so MICROVM regions are only visible and usable
by orgs with the `hasComputeAccess` feature flag. Admins and explicit
allowlist behavior unchanged.
- New shared helper (`regionAccess.server.ts`) with
`resolveComputeAccess`, `defaultVisibilityFilter`, and
`isComputeRegionAccessible`
- `RegionsPresenter` filters out MICROVM regions for non-compute orgs
- `SetDefaultRegionService` blocks setting a MICROVM region as default
without compute access
- `WorkerGroupService` blocks triggering runs in MICROVM regions without
compute access
- `computeTemplateCreation` refactored to use shared
`resolveComputeAccess`
- Updated snapshot callback schema
Sets `application_name` on the Prisma writer and replica connection
strings using the existing `SERVICE_NAME` env var, so DB load can be
attributed by service.
A single "fetch failed" from the object store was aborting the entire
batch stream with no retry. Added p-retry (3 attempts, 500ms-2s backoff)
around ploadPacketToObjectStore so transient network errors self-heal
server-side instead of propagating to the SDK.
### Text wrapping fix
- Fixes message text not wrapping on the run inspector if there were no
spaces in the text
- Fixes inspector title truncation
- Adds a copy text button for the Message property
<img width="468" height="740" alt="CleanShot 2026-04-04 at 10 19 02@2x"
src="https://github.com/user-attachments/assets/71e42bf3-d103-44a2-b3b4-937c0b60a4bc"
/>
This is a small improvement mainly with the UI Skills file:
- Animate open and close the Resizable panels
- Uses the built in animation hooks from react-window-splitter
- Includes a global variable for the animation easing and timing for
consistency
https://github.com/user-attachments/assets/50ed0019-ed12-4e08-b95c-7c6d1fe5bac0
## Summary
- Drop all 8 foreign key constraints on TaskRun. The run listing path is
now fully ClickHouse-backed so we no longer need Postgres to enforce
referential integrity on this table. The FK constraints add write
overhead on every insert/update with no remaining benefit. Prisma
queries are unaffected.
- Remove PostgresRunsRepository and its associated feature flag
(runsListRepository), which was the last remaining code path querying
TaskRun directly for list/count operations.
- Drop three indexes that were only useful for the Postgres run list
path and have no remaining query consumers:
- TaskRun_runtimeEnvironmentId_id_idx — was the cursor pagination index
for PostgresRunsRepository; superseded by the (runtimeEnvironmentId,
createdAt DESC) composite index
- TaskRun_scheduleId_idx — redundant with the (scheduleId, createdAt
DESC) composite index; no direct Postgres queries filter by scheduleId
alone
- TaskRun_rootTaskRunId_idx — no queries filter TaskRun by rootTaskRunId
as a WHERE clause anywhere in the codebase
All index drops use CONCURRENTLY IF EXISTS to avoid table locks in
production.
## Test plan
- pnpm run db:migrate:deploy applies all migrations cleanly
- pnpm run typecheck --filter webapp passes
- Run list pages load correctly in the dashboard (ClickHouse path)
- Scheduled task runs still trigger and appear correctly
This allows seamless migration to different object storage.
Existing runs that have offloaded payloads/outputs will continue to use
the default object store (configured using `OBJECT_STORE_*` env vars).
You can add additional stores by setting new env vars:
- `OBJECT_STORE_DEFAULT_PROTOCOL` this determines where new run large
payloads will get stored.
- If you set that you need to set new env vars for that protocol.
Example:
```
OBJECT_STORE_DEFAULT_PROTOCOL=“s3"
OBJECT_STORE_S3_BASE_URL=https://s3.us-east-1.amazonaws.com
OBJECT_STORE_S3_ACCESS_KEY_ID=<val>
OBJECT_STORE_S3_SECRET_ACCESS_KEY=<val>
OBJECT_STORE_S3_REGION=us-east-1
OBJECT_STORE_S3_SERVICE=s3
```
---------
Co-authored-by: nicktrn <55853254+nicktrn@users.noreply.github.com>
Adds a migration reference for users moving from n8n to Trigger.dev.
Includes a concept map, four common patterns covering the
migration-specific gaps, and a full customer onboarding example. The
onboarding workflow highlights the 3-day wait pattern, an area where
n8n's execution model has known reliability issues at production scale
that Trigger.dev handles natively
- Added versions filtering on the Errors list and page
- Added errors stacked bars to the graph on the individual error page
---------
Co-authored-by: James Ritchie <james@trigger.dev>
The @internal/compute package had its main/types pointing to
./src/index.ts with no build step. This works in dev (tsc resolves .ts
at compile time) but fails at runtime in Docker because Node.js can't
load .ts files directly.
Added tsconfig.build.json and build/clean/dev scripts matching the
pattern used by schedule-engine and other internal packages. Exports now
point to dist/.
Temporary workaround that enables filtering by environment in the
envvars page, without changing any UI.
---------
Co-authored-by: Claude <noreply@anthropic.com>
## Summary
Currently, every triggered run follows a two-step path through Redis:
1. **Enqueue** — A Lua script atomically adds the message to a queue
sorted set (ordered by priority-adjusted timestamp)
2. **Dequeue** — A debounced `processQueueForWorkerQueue` job fires
~500ms later, checks concurrency limits, removes the message from the
sorted set, and pushes it to a worker queue (Redis list) where workers
pick it up via `BLPOP`
This means every run pays at least ~500ms of latency between being
triggered and being available for a worker to execute, even when the
queue is empty and concurrency is wide open.
### What changed
The enqueue Lua scripts now atomically decide whether to **skip the
queue sorted set entirely** and push directly to the worker queue. This
happens inside the same Lua script that handles normal enqueue, so the
decision is atomic with respect to concurrency bookkeeping.
A run takes the **fast path** when all of these are true:
- **Fast path is enabled** for this worker queue (gated per
`WorkerInstanceGroup`)
- **No available messages** in the queue (`ZRANGEBYSCORE` finds nothing
with score ≤ now) — this respects priority ordering and allows fast path
even when the queue has future-scored messages (e.g. nacked retries with
delay)
- **Environment concurrency** has capacity
- **Queue concurrency** has capacity (including per-concurrency-key
limits for CK queues)
When the fast path is taken:
- The message is stored and pushed directly to the worker queue
(`RPUSH`)
- Concurrency slots are claimed (`SADD` to the same sets used by the
normal dequeue path)
- The `processQueueForWorkerQueue` job is **not scheduled** (no work to
do)
- TTL sorted set is skipped (the `expireRun` worker job handles TTL
independently)
When any condition fails, the existing slow path runs unchanged.
### Rollout gating
- **Development environments**: Fast path is always enabled
- **Production environments**: Gated by a new `enableFastPath` boolean
on `WorkerInstanceGroup` (defaults to `false`), allowing
region-by-region rollout
### Rolling deploy safety
Each process registers its own Lua scripts via `defineCommand`
(identified by SHA hash). Old and new processes never share scripts. The
Redis data structures are fully compatible in both directions — ack,
nack, and release operations work identically regardless of which path a
message took.
## Test plan
- [x] Fast path taken when queue is empty and concurrency available
- [x] Slow path when `enableFastPath` is false
- [x] Slow path when queue has available messages (respects priority
ordering)
- [x] Fast path when queue only has future-scored messages
- [x] Slow path when env concurrency is full
- [x] Fast-path message can be acknowledged correctly
- [x] Fast-path message can be nacked and re-enqueued to the queue
sorted set
- [x] Run all existing run-queue tests (ack, nack, CK, concurrency
sweeper, dequeue) to verify no regressions
- [x] Typecheck passes for run-engine and webapp
Add TTL (time-to-live) defaults at task-level and config-level, with
precedence: per-trigger > task > config > dev default (10m).
Docs PR: #3200 (merge after packages are released)
- Rebuild llm_pricing_tiers and llm_prices in syncLlmCatalog for
source=default
- Add vitest config, sync regression tests, and pin vitest 3.1.4
- Update pnpm-lock.yaml for the new devDependency
Adds a dialog to the admin orgs page for viewing and editing per-org
feature flag overrides. Flags are introspected from the catalog so the
UI stays in sync with available flags automatically. Also adds a new tab
for global flags.
Refactors featureFlags.server.ts to split catalog definition (shared)
from server-only runtime (flags(), makeSetMultipleFlags). The shared
module exports flag metadata and validation so both the UI and API
routes can use it without pulling in server dependencies.
Adds support for taint tolerations for scheduled runs. Useful for
selectively tolerating taints on dedicated node pools.
The new `KUBERNETES_SCHEDULED_RUN_TOLERATIONS` env variable accepts a
comma-separated list in the format key=value:effect (or key:effect for
the Exists operator).
Drive-by: renames all `KUBERNETES_SCHEDULE_*` affinity env vars to
KUBERNETES_SCHEDULED_RUN_* for clarity — this feature isn't used in
production yet or published in a tagged image; the name change is fine.
For human reviewer:
- Check if Redis connection + code makes sense
- Check CLI methods (it's on a hotpath)
- Check DB Migrations and new tables
## ✅ Checklist
- [x] I have followed every step in the [contributing
guide](https://github.com/triggerdotdev/trigger.dev/blob/main/CONTRIBUTING.md)
- [x] The PR title follows the convention.
- [x] I ran and tested the code works
---
## Testing
Spawning new CLI / Dashboard notifications, check MVP, check if failures
not produce any problems with CLI/Dashboard
---
## Changelog
Added notifications mechanism for Dashboard and CLI
---
## Screenshots
💯
Adds three new top-level columns to the ClickHouse task_runs_v2 table
primarily for analytics:
- `trigger_source` / `root_trigger_source` - extracted from the existing
TaskRun.annotations JSON during WAL
replication
- `is_warm_start` - new nullable boolean on TaskRun in Postgres, set in
the existing taskRun.update() at attempt
start (no additional write). null until the first attempt starts.
Run region is already available via the existing `worker_queue` column
in ClickHouse.
Scheduled runs create predictable hourly spikes that compete with
on-demand runs for node capacity. Runs triggered "on-demand" via the
SDK, API, or dashboard, are more sensitive to cold start latency since
users are typically
waiting on the result. When a burst of scheduled runs lands at the top
of the hour, it can saturate the shared pool resources causing
contention, affecting cold starts across the board.
The idea in this change is to absorb these periodic spikes in a
dedicated pool without affecting the cold starts of on-demand runs.
Scheduled runs are inherently less sensitive to cold starts.
### Changes in this PR
Follows up on run annotations (#3241), which made trigger origin
available on every run in the tree. This PR exposes
annotations at dequeue time to the supervisor. This enables scheduling
decisions based on trigger source.
The affinities are soft preferences at schedule time, so runs fall back
gracefully if the target pool is out out of capacity.
Queue limit ServiceValidationErrors were being logged at error level.
These are
expected validation rejections, not bugs.
- Add logLevel property to ServiceValidationError (webapp + run-engine)
- Set logLevel: warn on all queue limit throws
- Schedule engine: detect queue limit failures and log as warn
- Redis-worker: respect logLevel on thrown errors
Adds an `annotations` JSONB column to task runs that captures where and
how each run was triggered.
This enables filtering and analyzing trigger origins without querying up
the run tree. Also enables making scheduling decisions based on the
trigger source, e.g., use separate affinities for scheduled runs.
Each run records:
- **triggerSource**: who initiated it (sdk, api, dashboard, cli, mcp,
schedule)
- **triggerAction**: what kind of action (trigger, replay, test)
- **rootTriggerSource**: the trigger source of the root ancestor,
propagated through the entire run
tree
- **rootScheduleId**: schedule id, in case the run tree was triggered
from a schedule
Currently the main motivation for annotations it to determine whether a
run is part of a schedule-originated tree without traversing ancestors.
### A couple of design considerations
- **Decoupled source from method**: triggerSource and triggerAction are
separate fields to avoid
combinatorial explosion (every new source × every new action)
- **Server-side first**: all annotation values are primarily determined
on the server, only a minor SDK change needed
- **Forward-compatible**: annotation fields use
`z.enum([...]).or(anyString)` so new values can be
added without breaking validation; we currently don't need an explicit
version field for annotations.
Note: `metadata` would have been a more fitting name for the db column,
as it is consistent with other tables where we store this type of
information. It is already in use to store user metadata though, so we
go with `annotations` instead.
- Full prompt management UI: list, detail, override, and version
management for AI prompts defined with `prompts.define()`
- Rich AI span inspectors for all AI SDK operations with token usage,
messages, and prompt context
- Real-time generation tracking with live polling and filtering
## Prompt management
Define prompts in your code with `prompts.define()`, then manage
versions and overrides from the dashboard without redeploying:
```typescript
import { task, prompts } from "@trigger.dev/sdk";
import { generateText } from "ai";
import { openai } from "@ai-sdk/openai";
import { z } from "zod";
const supportPrompt = prompts.define({
id: "customer-support",
model: "gpt-4o",
variables: z.object({
customerName: z.string(),
plan: z.string(),
issue: z.string(),
}),
content: `You are a support agent for Acme SaaS.
Customer: {{customerName}} ({{plan}} plan)
Issue: {{issue}}
Respond with empathy and precision.`,
});
export const supportTask = task({
id: "handle-support",
run: async (payload) => {
const resolved = await supportPrompt.resolve({
customerName: payload.name,
plan: payload.plan,
issue: payload.issue,
});
const result = await generateText({
model: openai(resolved.model ?? "gpt-4o"),
system: resolved.text,
prompt: payload.issue,
...resolved.toAISDKTelemetry(),
});
return { response: result.text };
},
});
```
The prompts list page shows each prompt with its current version, model,
override status, and a usage sparkline over the last 24 hours.
From the prompt detail page you can:
- **Create overrides** to change the prompt template or model without
redeploying. Overrides take priority over the deployed version when
`prompt.resolve()` is called.
- **Promote** any code-deployed version to be the current version
- **Browse generations** across all versions with infinite scroll and
live polling for new results
- **Filter** by version, model, operation type, and provider
- **View metrics** (total generations, avg tokens, avg cost, latency)
broken down by version
## AI span inspectors
Every AI SDK operation now gets a custom inspector in the run trace
view:
- **`ai.generateText` / `ai.streamText`** — Shows model, token usage,
cost, the full message thread (system prompt, user message, assistant
response), and linked prompt details
- **`ai.generateObject` / `ai.streamObject`** — Same as above plus the
JSON schema and structured output
- **`ai.toolCall`** — Shows tool name, call ID, and input arguments
- **`ai.embed`** — Shows model and the text being embedded
For generation spans linked to a prompt, a "Prompt" tab shows the prompt
metadata, the input variables passed to `resolve()`, and the template
content from the prompt version.
All AI span inspectors include a compact timestamp and duration header.
## Other improvements
- Resizable panel sizes now persist across page refreshes (patched
`@window-splitter/state` to fix snapshot restoration)
- Run page panels also persist their sizes
- Fixed `<div>` inside `<p>` DOM nesting warnings in span titles and
chat messages
- Added Operations and Providers filters to the AI metrics dashboard
## Screenshots
<img width="3680" height="2392" alt="CleanShot 2026-03-21 at 10 14
17@2x"
src="https://github.com/user-attachments/assets/f3e59989-a2fa-4990-a9d0-3cacda431868"
/>
<img width="3680" height="2392" alt="CleanShot 2026-03-21 at 10 15
37@2x"
src="https://github.com/user-attachments/assets/2f2d02df-2d2b-44fb-ac6f-9153f6a6c387"
/>
<img width="3680" height="2392" alt="CleanShot 2026-03-21 at 10 15
54@2x"
src="https://github.com/user-attachments/assets/baa161e0-ef91-4fa4-a55f-986b71cccdf0"
/>
- Automatic LLM cost enrichment for AI SDK spans (streamText,
generateText, generateObject) or any other spans that use semantic
gen_ai attributes with support for 145+ models
- New AI span inspector sidebar showing model, tokens, cost, messages,
tool calls, and response text
- LLM metrics dual-write to ClickHouse `llm_metrics_v1` table for
analytics
- LLM metrics built-in dashboard (unlinked at the moment)
- Provider cost fallback — uses gateway/OpenRouter reported costs from
`providerMetadata` when registry pricing is unavailable
- Prefix-stripping for gateway/OpenRouter model names (e.g.
`mistral/mistral-large-3` matches `mistral-large-3` pricing)
- Admin dashboard for managing LLM model pricing (list, create, edit,
delete, search, test pattern matching)
- Missing models detection page — queries ClickHouse for unpriced models
with sample spans and Claude Code-ready prompts for adding pricing
- AI span seed script (`pnpm run db:seed:ai-spans`) with 51 spans across
12 provider systems for local dev testing
- UI fixes: `completionTokens`/`promptTokens` aliases,
`ai.response.object` display for generateObject, cache read/write token
breakdown
## Screenshots:
<img width="1030" height="104" alt="CleanShot 2026-03-17 at 16 48 54@2x"
src="https://github.com/user-attachments/assets/bc8fccda-e48b-4d0c-bfb1-e620064e5979"
/>
<img width="1094" height="1512" alt="CleanShot 2026-03-17 at 16 49
23@2x"
src="https://github.com/user-attachments/assets/c2424569-d07e-4d67-a436-e8250043a1ee"
/>
<img width="1074" height="1412" alt="CleanShot 2026-03-17 at 16 49
18@2x"
src="https://github.com/user-attachments/assets/22342ac4-4769-45d1-a328-a24fb9a82a50"
/>
<img width="1012" height="2292" alt="CleanShot 2026-03-17 at 16 39
01@2x"
src="https://github.com/user-attachments/assets/59e327d1-6652-4293-8be0-bb8326e5fbc5"
/>
<img width="3680" height="2392" alt="CleanShot 2026-03-15 at 08 29
38@2x"
src="https://github.com/user-attachments/assets/1f77beb8-de67-495b-b890-bcdb8d7f1fe8"
/>
---------
Co-authored-by: James Ritchie <james@trigger.dev>
When processing batchTriggerAndWait items, each batch item was acquiring
a
Redis lock on the parent run to insert a TaskRunWaitpoint row. With high
concurrency (processingConcurrency=50), this caused
LockAcquisitionTimeoutError
(880 errors/24h in prod), orphaned runs, and stuck parent runs.
Since blockRunWithCreatedBatch already transitions the parent to
EXECUTING_WITH_WAITPOINTS before items are processed, the per-item lock
is
unnecessary. The new blockRunWithWaitpointLockless method performs only
the
idempotent CTE insert and timeout scheduling without acquiring the lock.
## Summary
Major expansion of the MCP server (14 → 25 tools), context efficiency
optimizations, new API endpoints, and a fix for the dev CLI leaking
build directories on disk.
### New MCP tools
- **Query & analytics**: `get_query_schema`, `query`, `list_dashboards`,
`run_dashboard_query` — query your data using TRQL directly from AI
assistants
- **Profile management**: `whoami`, `list_profiles`, `switch_profile` —
see and switch CLI profiles per-project (persisted to
`.trigger/mcp.json`)
- **Dev server control**: `start_dev_server`, `stop_dev_server`,
`dev_server_status` — start/stop `trigger dev` and stream build output
- **Task introspection**: `get_task_schema` — get payload schema for a
specific task (split out from `get_current_worker` to reduce context)
### New API endpoints
- `GET /api/v1/query/schema` — discover TRQL tables and columns
(server-driven, multi-table)
- `GET /api/v1/query/dashboards` — list built-in dashboard widgets and
their queries
### New features
- **`--readonly` flag** — hides write tools (`deploy`, `trigger_task`,
`cancel_run`) so agents can't make changes
- **`read:query` JWT scope** — new authorization scope for query
endpoints, with per-table granularity (`read:query:runs`,
`read:query:llm_metrics`, etc.)
- **Paginated trace output** — `get_run_details` now paginates trace
events via cursor, caching the full trace in a temp file so subsequent
pages don't re-fetch
- **MCP tool annotations** — all tools now have
`readOnlyHint`/`destructiveHint` annotations for clients that support
them
- **Project-scoped profile persistence** — `switch_profile` saves to
`.trigger/mcp.json` (gitignored), automatically loaded on next MCP
server start
### Context optimizations
- `get_query_schema` requires a table name — returns one table's schema
instead of all tables (60-80% fewer tokens)
- `get_current_worker` no longer inlines payload schemas — use
`get_task_schema` for specific tasks
- Query results formatted as text tables instead of JSON (~50% fewer
tokens for flat data)
- `cancel_run`, `list_deploys`, `list_preview_branches` formatted as
text instead of raw `JSON.stringify()`
- Schema and dashboard API responses cached (1hr and 5min respectively)
### Bug fixes
- Fixed `search_docs` failing due to renamed upstream Mintlify tool
(`SearchTriggerDev` → `search_trigger_dev`)
- Fixed `list_deploys` failing when deployments have null
`runtime`/`runtimeVersion` fields (fixes#3139)
- Fixed `list_preview_branches` crashing due to incorrect response shape
access
- Fixed `metrics` table column documented as `value` instead of
`metric_value` in query docs
- Fixed `/api/v1/query` not accepting JWT auth (added `allowJWT: true`)
### Dev CLI build directory fix
The dev CLI was leaking `build-*` directories in `.trigger/tmp/` on
every rebuild, accumulating hundreds of MB over time (842MB observed).
Three layers of protection added:
1. **During session**: deprecated workers are pruned (capped at 2
retained) when no active runs reference them, preventing unbounded
accumulation
2. **On SIGKILL/crash**: the watchdog process now cleans up
`.trigger/tmp/` when it detects the parent CLI was killed
3. **On next startup**: existing `clearTmpDirs()` wipes any remaining
orphans
## Test plan
- [ ] `pnpm run mcp:smoke` — 17 automated smoke tests for all read-only
MCP tools
- [ ] `pnpm run mcp:test list` — verify 25 tools registered (21 in
`--readonly` mode)
- [ ] `pnpm run mcp:test --readonly list` — verify write tools hidden
- [ ] Manual: start dev server, trigger task, rebuild multiple times,
verify build dirs stay capped at 4
- [ ] Manual: SIGKILL the dev CLI, verify watchdog cleans up
`.trigger/tmp/`
- [ ] Verify new API endpoints return correct data: `GET
/api/v1/query/schema`, `GET /api/v1/query/dashboards`
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Queues with concurrency keys now appear as a single entry in the master
queue instead of one entry per key. This prevents high-CK-count tenants
from consuming the entire `parentQueueLimit` window and starving other
tenants on the same shard.
A new per-queue **CK index** (sorted set) tracks active concurrency key
sub-queues. The master queue gets one `:ck:*` wildcard entry per base
queue. Dequeuing from that entry round-robins across sub-queues,
maintaining per-CK concurrency tracking and fairness.
All existing operations (enqueue, dequeue, ack, nack, DLQ, TTL expiry)
are CK-index-aware and keep the index consistent. Old-format entries
drain naturally during rollout — no migration step needed, single
deploy.
## Adds 2 self serve features
### 1. self serve preview branches
- Copies the patterns of the self serve concurrency
- Self serve only available on Pro plan (otherwise you are linked to the
billing plans page)
- Global self serve branches limit: 180 (+20 for the Pro plan). It can
be overridden per Org
- You need to archive branches before reducing the number of extra
branches you're paying for
- Branches are removed immediately but remain billed until the end of
the billing cycle like extra concurrency
### 2. self serve team members
- Copies the patterns of the self serve concurrency
- Self serve only available on Pro plan (otherwise you are linked to the
billing plans page)
- Global self serve members is unlimited but can be limited with the
same env var quota and overridden per org if needed
- You need to remove team members before reducing the number of members
you pay for
- Team members are removed immediately but remain billed until the end
of the billing cycle like extra concurrency
Deprecates the syncVercelEnvVars build extension and adds warnings in
both the Vercel integration docs and the extension's own page to prevent
conflicts with the native env var sync
## Summary
2 new features, 2 improvements.
## Improvements
- Add syncSupabaseEnvVars to pull database connection strings and save
them as trigger.dev environment variables
([#3152](https://github.com/triggerdotdev/trigger.dev/pull/3152))
- Auto-cancel in-flight dev runs when the CLI exits, using a detached
watchdog process that survives pnpm SIGKILL
([#3191](https://github.com/triggerdotdev/trigger.dev/pull/3191))
## Server changes
These changes affect the self-hosted Docker image and Trigger.dev Cloud:
- A new Errors page for viewing and tracking errors that cause runs to
fail
- Errors are grouped using error fingerprinting
- View top errors for a time period, filter by task, or search the text
- View occurrences over time
- View all the runs for an error and bulk replay them
([#3172](https://github.com/triggerdotdev/trigger.dev/pull/3172))
- Add sidebar tabs (Options, AI, Schema) to the Test page for schemaTask
payload generation and schema viewing.
([#3188](https://github.com/triggerdotdev/trigger.dev/pull/3188))
<details>
<summary>Raw changeset output</summary>
# Releases
## @trigger.dev/build@4.4.3
### Patch Changes
- Add syncSupabaseEnvVars to pull database connection strings and save
them as trigger.dev environment variables
([#3152](https://github.com/triggerdotdev/trigger.dev/pull/3152))
- Updated dependencies:
- `@trigger.dev/core@4.4.3`
## trigger.dev@4.4.3
### Patch Changes
- Auto-cancel in-flight dev runs when the CLI exits, using a detached
watchdog process that survives pnpm SIGKILL
([#3191](https://github.com/triggerdotdev/trigger.dev/pull/3191))
- Updated dependencies:
- `@trigger.dev/core@4.4.3`
- `@trigger.dev/build@4.4.3`
- `@trigger.dev/schema-to-json@4.4.3`
## @trigger.dev/core@4.4.3
### Patch Changes
- Auto-cancel in-flight dev runs when the CLI exits, using a detached
watchdog process that survives pnpm SIGKILL
([#3191](https://github.com/triggerdotdev/trigger.dev/pull/3191))
## @trigger.dev/python@4.4.3
### Patch Changes
- Updated dependencies:
- `@trigger.dev/core@4.4.3`
- `@trigger.dev/build@4.4.3`
- `@trigger.dev/sdk@4.4.3`
## @trigger.dev/react-hooks@4.4.3
### Patch Changes
- Updated dependencies:
- `@trigger.dev/core@4.4.3`
## @trigger.dev/redis-worker@4.4.3
### Patch Changes
- Updated dependencies:
- `@trigger.dev/core@4.4.3`
## @trigger.dev/rsc@4.4.3
### Patch Changes
- Updated dependencies:
- `@trigger.dev/core@4.4.3`
## @trigger.dev/schema-to-json@4.4.3
### Patch Changes
- Updated dependencies:
- `@trigger.dev/core@4.4.3`
## @trigger.dev/sdk@4.4.3
### Patch Changes
- Updated dependencies:
- `@trigger.dev/core@4.4.3`
</details>
---------
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
When the dev CLI exits (e.g. ctrl+c via pnpm), runs that were
mid-execution
previously stayed stuck in EXECUTING status for up to 5 minutes until
the
heartbeat timeout fired. Now they are cancelled within seconds.
The dev CLI spawns a lightweight detached watchdog process at startup.
The
watchdog monitors the CLI process ID and, when it detects the CLI has
exited,
calls a new POST /engine/v1/dev/disconnect endpoint to cancel all
in-flight
runs immediately (skipping PENDING_CANCEL since the worker is known to
be dead).
Watchdog design:
- Fully detached (detached: true, stdio: ignore, unref()) so it survives
even when pnpm sends SIGKILL to the process tree
- Active run IDs maintained via atomic file write
(.trigger/active-runs.json)
- Single-instance guarantee via PID file (.trigger/watchdog.pid)
- Safety timeout: exits after 24 hours to prevent zombie processes
- On clean shutdown, the watchdog is killed (no disconnect needed)
Disconnect endpoint:
- Rate-limited: 5 calls/min per environment
- Capped at 500 runs per call
- Small counts (<= 25): cancelled inline with pMap concurrency 10
- Large counts: delegated to the bulk action system
- Uses finalizeRun: true to skip PENDING_CANCEL and go straight to
FINISHED
Run engine change:
- cancelRun() now respects finalizeRun when the run is in EXECUTING
status,
skipping the PENDING_CANCEL waiting state and going directly to FINISHED
### Fixes and improvements to the onboarding questions:
**This change is worth double checking @matt-aitken**
- Update to the Button.tsx file: it now takes `isLoading` that shows a
spinner in the middle of the button (replacing the button text and any
icons) and sets it to `disabled`. It does this nicely by keeping the
button width the same so there's no layout shift.
**Other fixes**
- Fixes an issue where if you type a custom option in the "What
technologies do you use" question, it doesn't check the list to see if
it matches. Now it checks the box if you've typed an option from that
list.
- When we randomize the list of onboarding question options, we now
store the position they appeared in the list
<img width="2191" height="1023" alt="CleanShot 2026-03-06 at 13 36 53"
src="https://github.com/user-attachments/assets/4eba0d1a-1528-49a3-be5b-6bde89030193"
/>
<img width="411" height="1069" alt="CleanShot 2026-03-06 at 13 37 28"
src="https://github.com/user-attachments/assets/e5f7bb9c-c894-41cc-9ca6-96b43fcf6005"
/>
Add a tabbed sidebar to the Test page for standard tasks, reusing the
ClientTabs pattern from the Query page.
- Options tab: existing sidebar content (machine, version, queue, etc.)
- AI tab: AI-powered payload generation with streaming, supports JSON
Schema, inferred schema from recent runs, and task source code lookup
via tool calling for tasks without schemas
- Schema tab: displays payload JSON Schema (from schemaTask), inferred
schema (from recent runs via @jsonhero/schema-infer), or empty state
with schemaTask docs and example code
Data layer changes:
- Surface payloadSchema and inferredPayloadSchema from TestTaskPresenter
- Add payloadSchema and fileId to WorkerDeploymentWithWorkerTasks type
- Decompress zlib-deflated source files for AI context
A top-level Errors page that aggregates errors from failed runs with
occurrences metrics.
https://github.com/user-attachments/assets/8f0ef55e-90dd-4faa-9051-59f4665181e4
Errors are “fingerprinted” so similar errors are grouped together (e.g.
has an ID in the error message).
You can view an individual error to view a timeline of when it fired,
the runs, and bulk replay them.
# trigger.dev v4.4.2
## Summary
2 new features, 2 improvements, 8 bug fixes.
## Improvements
- Add input streams for bidirectional communication with running tasks.
Define typed input streams with `streams.input<T>({ id })`, then consume
inside tasks via `.wait()` (suspends the process), `.once()` (waits for
next message), or `.on()` (subscribes to a continuous stream). Send data
from backends with `.send(runId, data)` or from frontends with the new
`useInputStreamSend` React hook.
([#3146](https://github.com/triggerdotdev/trigger.dev/pull/3146))
- Add PAYLOAD_TOO_LARGE error to handle graceful recovery of sending
batch trigger items with payloads that exceed the maximum payload size
([#3137](https://github.com/triggerdotdev/trigger.dev/pull/3137))
## Bug fixes
- Fix slow batch queue processing by removing spurious cooloff on
concurrency blocks and fixing a race condition where retry attempt
counts were not atomically updated during message re-queue.
([#3079](https://github.com/triggerdotdev/trigger.dev/pull/3079))
- fix(sdk): batch triggerAndWait variants now return correct
run.taskIdentifier instead of unknown
([#3080](https://github.com/triggerdotdev/trigger.dev/pull/3080))
## Server changes
These changes affect the self-hosted Docker image and Trigger.dev Cloud:
- Two-level tenant dispatch architecture for batch queue processing.
Replaces the
single master queue with a two-level index: a dispatch index (tenant →
shard)
and per-tenant queue indexes (tenant → queues). This enables O(1) tenant
selection and fair scheduling across tenants regardless of queue count.
Improves batch queue processing performance.
([#3133](https://github.com/triggerdotdev/trigger.dev/pull/3133))
- Add input streams with API routes for sending data to running tasks,
SSE reading, and waitpoint creation. Includes Redis cache for fast
`.send()` to `.wait()` bridging, dashboard span support for input stream
operations, and s2-lite support with configurable S2 endpoint, access
token skipping, and S2-Basin headers for self-hosted deployments. Adds
s2-lite to Docker Compose for local development.
([#3146](https://github.com/triggerdotdev/trigger.dev/pull/3146))
- Speed up batch queue processing by disabling cooloff and increasing
the batch queue processing concurrency limits on the cloud:
- Pro plan: increase to 50 from 10.
- Hobby plan: increase to 10 from 5.
- Free plan: increase to 5 from 1.
([#3079](https://github.com/triggerdotdev/trigger.dev/pull/3079))
- Move batch queue global rate limiter from FairQueue claim phase to
BatchQueue worker queue consumer for accurate per-item rate limiting.
Add worker queue depth cap to prevent unbounded growth that could cause
visibility timeouts.
([#3166](https://github.com/triggerdotdev/trigger.dev/pull/3166))
- Fix a race condition in the waitpoint system where a run could be
blocked by a completed waitpoint but never be resumed because of a
PostgreSQL MVCC issue. This was most likely to occur when creating a
waitpoint via `wait.forToken()` at the same moment as completing the
token with `wait.completeToken()`. Other types of waitpoints (timed,
child runs) were not affected.
([#3075](https://github.com/triggerdotdev/trigger.dev/pull/3075))
- Fix metrics dashboard chart series colors going out of sync and
widgets not reloading stale data when scrolled back into view
([#3126](https://github.com/triggerdotdev/trigger.dev/pull/3126))
- Gracefully handle oversized batch items instead of aborting the
stream.
When an NDJSON batch item exceeds the maximum size, the parser now emits
an error marker instead of throwing, allowing the batch to seal
normally. The oversized item becomes a pre-failed run with
`PAYLOAD_TOO_LARGE` error code, while other items in the batch process
successfully. This prevents `batchTriggerAndWait` from seeing connection
errors and retrying with exponential backoff.
Also fixes the NDJSON parser not consuming the remainder of an oversized
line split across multiple chunks, which caused "Invalid JSON" errors on
subsequent lines.
([#3137](https://github.com/triggerdotdev/trigger.dev/pull/3137))
- Require the user is an admin during an impersonation session.
Previously only the impersonation cookie was checked; now the real
user's admin flag is verified on every request. If admin has been
revoked, the session falls back to the real user's ID.
([#3078](https://github.com/triggerdotdev/trigger.dev/pull/3078))
<details>
<summary>Raw changeset output</summary>
# Releases
## @trigger.dev/build@4.4.2
### Patch Changes
- Updated dependencies:
- `@trigger.dev/core@4.4.2`
## trigger.dev@4.4.2
### Patch Changes
- Updated dependencies:
- `@trigger.dev/build@4.4.2`
- `@trigger.dev/core@4.4.2`
- `@trigger.dev/schema-to-json@4.4.2`
## @trigger.dev/python@4.4.2
### Patch Changes
- Updated dependencies:
- `@trigger.dev/sdk@4.4.2`
- `@trigger.dev/build@4.4.2`
- `@trigger.dev/core@4.4.2`
## @trigger.dev/react-hooks@4.4.2
### Patch Changes
- Add input streams for bidirectional communication with running tasks.
Define typed input streams with `streams.input<T>({ id })`, then consume
inside tasks via `.wait()` (suspends the process), `.once()` (waits for
next message), or `.on()` (subscribes to a continuous stream). Send data
from backends with `.send(runId, data)` or from frontends with the new
`useInputStreamSend` React hook.
([#3146](https://github.com/triggerdotdev/trigger.dev/pull/3146))
Upgrade S2 SDK from 0.17 to 0.22 with support for custom endpoints
(s2-lite) via the new `endpoints` configuration, `AppendRecord.string()`
API, and `maxInflightBytes` session option.
- Updated dependencies:
- `@trigger.dev/core@4.4.2`
## @trigger.dev/redis-worker@4.4.2
### Patch Changes
- Fix slow batch queue processing by removing spurious cooloff on
concurrency blocks and fixing a race condition where retry attempt
counts were not atomically updated during message re-queue.
([#3079](https://github.com/triggerdotdev/trigger.dev/pull/3079))
- Updated dependencies:
- `@trigger.dev/core@4.4.2`
## @trigger.dev/rsc@4.4.2
### Patch Changes
- Updated dependencies:
- `@trigger.dev/core@4.4.2`
## @trigger.dev/schema-to-json@4.4.2
### Patch Changes
- Updated dependencies:
- `@trigger.dev/core@4.4.2`
## @trigger.dev/sdk@4.4.2
### Patch Changes
- Add input streams for bidirectional communication with running tasks.
Define typed input streams with `streams.input<T>({ id })`, then consume
inside tasks via `.wait()` (suspends the process), `.once()` (waits for
next message), or `.on()` (subscribes to a continuous stream). Send data
from backends with `.send(runId, data)` or from frontends with the new
`useInputStreamSend` React hook.
([#3146](https://github.com/triggerdotdev/trigger.dev/pull/3146))
Upgrade S2 SDK from 0.17 to 0.22 with support for custom endpoints
(s2-lite) via the new `endpoints` configuration, `AppendRecord.string()`
API, and `maxInflightBytes` session option.
- fix(sdk): batch triggerAndWait variants now return correct
run.taskIdentifier instead of unknown
([#3080](https://github.com/triggerdotdev/trigger.dev/pull/3080))
- Add PAYLOAD_TOO_LARGE error to handle graceful recovery of sending
batch trigger items with payloads that exceed the maximum payload size
([#3137](https://github.com/triggerdotdev/trigger.dev/pull/3137))
- Updated dependencies:
- `@trigger.dev/core@4.4.2`
## @trigger.dev/core@4.4.2
</details>
---------
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
The global rate limiter was being applied at the FairQueue claim phase,
consuming 1 token per queue-claim-attempt rather than per item
processed.
With many small queues (each batch is its own queue), consumers burned
through tokens on empty or single-item queues, causing aggressive
throttling well below the intended items/sec limit.
Changes:
- Move rate limiter from FairQueue claim phase to BatchQueue worker
queue
consumer loop (before blockingPop), so each token = 1 item processed
- Replace the FairQueue rate limiter with a worker queue depth cap to
prevent unbounded growth that could cause visibility timeouts
- Add BATCH_QUEUE_WORKER_QUEUE_MAX_DEPTH env var (optional, disabled by
default)
Input streams enable sending typed data to executing tasks from external
callers — backends, frontends, or other tasks. This unlocks interactive
use cases like approval UIs, cancel buttons, chat interfaces, and
human-in-the-loop AI workflows where the task needs to receive data
while running.
Three consumption patterns inside a task:
* `.wait()` — Suspend the task until data arrives (process freed, most
efficient)
* `.once()` — Wait for the next message (process stays alive)
* `.on()` — Subscribe to a continuous stream of messages
One send pattern from outside:
* `.send(runId, data)` — Send typed data to a specific run's input
stream
## User-facing API
### Define a typed input stream
```ts
import { streams, task } from "@trigger.dev/sdk";
const approval = streams.input<{ approved: boolean; reviewer: string }>({ id: "approval" });
```
### Consume inside a task
```ts
export const myTask = task({
id: "my-task",
run: async () => {
// Pattern 1: Suspend until data arrives (most efficient — frees the process)
const result = await approval.wait({ timeout: "5m" });
// Pattern 2: Wait for next message (process stays alive)
const data = await approval.once().unwrap();
// Pattern 3: Subscribe to multiple messages
approval.on((data) => { /* handle each message */ });
},
});
```
### Send from outside
```ts
// From a backend (using secret API key)
await approval.send(runId, { approved: true, reviewer: "alice" });
// From a frontend (using public JWT token from trigger response)
const { send } = useInputStreamSend("approval", runId, { accessToken });
send({ approved: true, reviewer: "alice" });
```
---------
Co-authored-by: Claude <noreply@anthropic.com>
- New User onboarding questions added and stored in a new
`onboardingData` col
- Keeps the same Org creation screen and stores the data in the same
format in same DB column
- New Org onboarding questions addded and stored in a new
`onboardingData` col
https://github.com/user-attachments/assets/244e4bae-f74d-4ed4-a545-92c9b927e98b
---------
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Pins vouch actions to `c6d80ead49839655b61b422700b7a3bc9d0804a9`
(v1.4.2) in favor of security practices. We were previously using the
`@main` tag.
Also removes the checkout steps as they're not needed in these
workflows.
There's a bug in react-window-splitter on Firefox. When trying to expand
the inspector panel in the query editor, it checks if the main panel has
space but gets an object instead of a number for the auto-sized
query-main panel. This causes the expand calculation to fail and it
snaps it back to collapsed.
I've removed this behavior for now as it's not an important feature.
Gracefully handle oversized batch items instead of aborting the stream.
When an NDJSON batch item exceeds the maximum size, the parser now emits
an error marker instead of throwing, allowing the batch to seal
normally. The oversized item becomes a pre-failed run with
`PAYLOAD_TOO_LARGE` error code, while other items in the batch process
successfully. This prevents `batchTriggerAndWait` from seeing connection
errors and retrying with exponential backoff.
Also fixes the NDJSON parser not consuming the remainder of an oversized
line split across multiple chunks, which caused "Invalid JSON" errors on
subsequent lines.
Replace flat master queue index with two-level tenant dispatch to fix
noisy neighbor problem. When a tenant has many queues at capacity, the
scheduler now iterates tenants (Level 1) not queues, then fetches
per-tenant queues (Level 2) only for eligible tenants.
Single-deploy migration: new enqueues write to dispatch indexes only,
consumer drains old master queue alongside new dispatch path until
empty.
Clarifies in the Query docs that run metadata is not available on the
Query page and that the output column is JSON, so dot notation (e.g.
output.externalId) should be used for selecting and filtering. Adds an
example that filters by an output field in WHERE
Split the CTE in blockRunWithWaitpoint so the pending waitpoint check
is a separate SQL statement. In READ COMMITTED isolation, each statement
gets its own snapshot, so a separate SELECT sees the latest committed
state from concurrent completeWaitpoint calls.
Previously, the CTE did INSERT + pending check in one statement (one
snapshot). If completeWaitpoint committed between the CTE start and
the SELECT, the SELECT would still see PENDING due to the stale
snapshot. Neither side would enqueue continueRunIfUnblocked, leaving
the run stuck forever.
Closes #
## ✅ Checklist
- [ ] I have followed every step in the [contributing
guide](https://github.com/triggerdotdev/trigger.dev/blob/main/CONTRIBUTING.md)
- [ ] The PR title follows the convention.
- [ ] I ran and tested the code works
---
## Testing
Verified the pricing definition displays the correct tier amount on the
plan selection page.
---
## Changelog
Fixed incorrect pricing tier for additional realtime connections from
$10/month per 100 to $10/month per 1000.
---
## Screenshots
N/A
💯https://claude.ai/code/session_015QrZZJHPWta3QCBnhX2Pff
Co-authored-by: Claude <noreply@anthropic.com>
Fix slow fair queue processing by removing spurious cooloff on
concurrency blocks and fixing a race condition where retry attempt
counts were not atomically updated during message re-queue.
Removed cooloff entirely from the batch queue
- Fix for series color assignment being out of sync with the graph
(ensures added series colors match their graph representation)
- Reload widgets when returning to screen if props changed (prevents
stale widgets after filtering and scrolling)
- Add .server-changes/ convention for tracking server-only changes
- Create scripts/enhance-release-pr.mjs to deduplicate and categorize
changeset PR body
- Create scripts/generate-github-release.mjs to format unified GitHub
release body
- Change release.yml to create one unified GitHub release instead of
per-package releases
- Add update-release job to patch Docker image link after images are
pushed to GHCR
- Update changesets-pr.yml to trigger on .server-changes, enhance PR
body, and clean up consumed files
- Document server changes in CLAUDE.md, CONTRIBUTING.md, CHANGESETS.md,
and RELEASE.md
What changed
- Fixed some functions like dateAdd, toString, ifNotFinite
- Removed all functions that accept lambdas as they're not supported
(yet)
- Added tests for all TRQL functions that use ClickHouse
Adds OpenAPI specs and sidebar pages for four previously undocumented
public endpoints: retrieve
run result, per-task batch trigger, retrieve batch, and retrieve batch
results.
This PR implements a new run TTL system and queue size limits to prevent
unbounded queue growth which should help prevent situations where queues
enter a "death spiral" where the queue will never be able to catch up.
The main/correct way to battle this situation is to enforce a maximum
TTL on all runs (e.g. up to 14 days) where runs that have been queued
for that maximum TTL will get auto-expired, making room for newer runs
to execute. This required creating a new TTL system that can handle
higher workloads and is now deeply integrated into the RunQueue. When
runs are enqueued with a TTL, they are added to their normal queue as
well as to the TTL queue. When runs are dequeued, they are removed from
both their normal queue and the TTL queue. If runs are dequeued by the
TTL system, they are removed from their normal queue. Both these
dequeues happen automatically so there is no race condition.
The TTL expiration system is also made reliable by expiring runs via a
Redis worker, which is enqueued to atomically inside the TTL dequeue lua
script.
### Optional associated waitpoints
Additionally, this PR implements an optimization where runs that aren't
triggered with a dependent parent run will no longer create an
associated waitpoint. Associated waitpoints are then lazily created if a
dependent run wants to wait for the child run post-facto (via debounce
or idempotency), which is a rare situation but is possible. This means
fewer waitpoint creations but also fewer waitpoint completions for runs
with no dependencies.
### Environment Queue Limits
Prevents any single queue growing too large by enforcing queue size
limits at trigger time.
- Queue size checks happen at trigger time - runs are rejected if queue
would exceed limit
- Dashboard UI shows queue limits on both the Queues page and a new
Limits page
- In-memory caching for queue size checks to reduce Redis load
### Batch trigger fixes
Currently when a batch item cannot be created for whatever reason (e.g.
queue limits) the run will never get created, which means a stalled run
if using `batchTriggerAndWait`. We've updated the system to handle this
differently: now when a batch item cannot be triggered and converted
into a run, we will eventually (after retrying 8 times up to 30s) we
will create a "pre-failed" run with the error details, correctly
resolving the batchTriggerAndWait.
Without doing an expensive query we can’t tell if it’s definitely a v3
projects – like getting run counts.
So let’s just assume if the project hasn’t been upgraded to v4 (by
running dev/deploy CLI with v4) AND the project is older than the v4
release then it’s v3.
Small fixes and improvements to the logs page:
- Clicking the Run ID didn't open inspector
- Swapped the "open link in tab" icon with Runs icon
- Prevent tooltip hovering on Level info
<img width="350" height="206" alt="CleanShot 2026-02-20 at 10 00 37@2x"
src="https://github.com/user-attachments/assets/3e82f24a-c0a1-4c01-a8e9-9e06a8af982a"
/>
This PR was opened by the [Changesets
release](https://github.com/changesets/action) GitHub action. When
you're ready to do a release, you can merge this and publish to npm
yourself or [setup this action to publish
automatically](https://github.com/changesets/action#with-publishing). If
you're not ready to do a release yet, that's fine, whenever you add more
changesets to main, this PR will be updated.
# Releases
## @trigger.dev/build@4.4.1
### Patch Changes
- Updated dependencies:
- `@trigger.dev/core@4.4.1`
## trigger.dev@4.4.1
### Patch Changes
- Add OTEL metrics pipeline for task workers. Workers collect process
CPU/memory, Node.js runtime metrics (event loop utilization, event loop
delay, heap usage), and user-defined custom metrics via
`otel.metrics.getMeter()`. Metrics are exported to ClickHouse with
10-second aggregation buckets and 1m/5m rollups, and are queryable
through the dashboard query engine with typed attribute columns,
`prettyFormat()` for human-readable values, and AI query support.
([#3061](https://github.com/triggerdotdev/trigger.dev/pull/3061))
- Updated dependencies:
- `@trigger.dev/build@4.4.1`
- `@trigger.dev/core@4.4.1`
- `@trigger.dev/schema-to-json@4.4.1`
## @trigger.dev/python@4.4.1
### Patch Changes
- Updated dependencies:
- `@trigger.dev/sdk@4.4.1`
- `@trigger.dev/build@4.4.1`
- `@trigger.dev/core@4.4.1`
## @trigger.dev/react-hooks@4.4.1
### Patch Changes
- Updated dependencies:
- `@trigger.dev/core@4.4.1`
## @trigger.dev/redis-worker@4.4.1
### Patch Changes
- Updated dependencies:
- `@trigger.dev/core@4.4.1`
## @trigger.dev/rsc@4.4.1
### Patch Changes
- Updated dependencies:
- `@trigger.dev/core@4.4.1`
## @trigger.dev/schema-to-json@4.4.1
### Patch Changes
- Updated dependencies:
- `@trigger.dev/core@4.4.1`
## @trigger.dev/sdk@4.4.1
### Patch Changes
- Add OTEL metrics pipeline for task workers. Workers collect process
CPU/memory, Node.js runtime metrics (event loop utilization, event loop
delay, heap usage), and user-defined custom metrics via
`otel.metrics.getMeter()`. Metrics are exported to ClickHouse with
10-second aggregation buckets and 1m/5m rollups, and are queryable
through the dashboard query engine with typed attribute columns,
`prettyFormat()` for human-readable values, and AI query support.
([#3061](https://github.com/triggerdotdev/trigger.dev/pull/3061))
- Updated dependencies:
- `@trigger.dev/core@4.4.1`
## @trigger.dev/core@4.4.1
---------
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Adds a direct Vercel Marketplace link, documents configuring build
options via the project config page, and adds a warning and workaround
for projects using a Vercel Root Directory
- Adds an end-to-end OTEL metrics pipeline: task workers collect and
export metrics via OpenTelemetry, the webapp ingests them into
ClickHouse, and they're queryable through the existing dashboard query
engine
- Workers emit process CPU/memory metrics (via
`@opentelemetry/host-metrics`) and Node.js runtime metrics (event loop
utilization, event loop delay, heap usage)
- Users can create custom metrics in their tasks via
`otel.metrics.getMeter()` from `@trigger.dev/sdk`
- Metrics are automatically tagged with run context (run ID, task slug,
machine, worker version) so they can be sliced per-run, per-task, or
per-machine
- The TSQL query engine gains metrics table support with typed attribute
columns, `prettyFormat()` for human-readable values, and per-schema time
bucket thresholds
- Includes reference tasks
(`references/hello-world/src/trigger/metrics.ts`) demonstrating
CPU-intensive, memory-ramp, bursty workload, and custom metrics patterns
## What changed
### Metrics collection (packages/core, packages/cli-v3)
- **Metrics export pipeline** — `TracingSDK` now sets up a
`MeterProvider` with a `PeriodicExportingMetricReader` that chains
through `TaskContextMetricExporter` (adds run context attributes) and
`BufferingMetricExporter` (batches exports to reduce overhead)
- **Host metrics** — Enabled `@opentelemetry/host-metrics` for process
CPU, memory, and system-level metrics
- **Node.js runtime metrics** — New `nodejsRuntimeMetrics.ts` module
using `performance.eventLoopUtilization()`, `monitorEventLoopDelay()`,
and `process.memoryUsage()` to emit 6 observable gauges
- File system and diskio metrics
- **Custom metrics** — Exposed `otel.metrics` from `@trigger.dev/sdk` so
users can create counters, histograms, and gauges in their tasks
- **Machine ID** — Stable per-worker machine identifier for grouping
metrics
- **Dev worker** — Drops `system.*` metrics to reduce noise, keeps
sending metrics between runs in warm workers
### Metrics ingestion (apps/webapp)
- **OTEL endpoint** — `otel.v1.metrics.ts` accepts OTEL metric export
requests (JSON and protobuf), converts to ClickHouse rows
- **ClickHouse schema** — `017_create_metrics_v1.sql` with 10-second
aggregation buckets, JSON attributes column, 60-day TTLs
### Query engine (internal-packages/tsql, apps/webapp)
- **Metrics query schema** — Typed columns for metric attributes
(`task_identifier`, `run_id`, `machine_name`, `worker_version`, etc.)
extracted from the JSON attributes column
- **`prettyFormat()`** — TSQL function that annotates columns with
format hints (`bytes`, `percent`, `durationSeconds`) for frontend
rendering without changing the underlying data
- **Per-schema time buckets** — Different tables can define their own
time bucket thresholds (metrics uses tighter intervals than runs)
- **AI query integration** — The AI query service knows about the
metrics table and can generate metric queries
- **Chart improvements** — Better formatting for byte values,
percentages, and durations in charts and tables
### Reference project
- **`references/hello-world/src/trigger/metrics.ts`** — 6 example tasks:
`cpu-intensive`, `memory-ramp`, `bursty-workload`, `sustained-workload`,
`concurrent-load`, `custom-metrics`
## Test plan
- [ ] Build all packages and webapp
- [ ] Start dev worker with hello-world reference project
- [ ] Run `cpu-intensive`, `memory-ramp`, and `custom-metrics` tasks
- [ ] Verify metrics in ClickHouse: `SELECT DISTINCT metric_name FROM
metrics_v1`
- [ ] Query via dashboard AI: "show me CPU utilization over time"
- [ ] Verify `prettyFormat` renders correctly in chart tooltips and
table cells
- [ ] Confirm dev worker drops `system.*` metrics but keeps `process.*`
and `nodejs.*`
This PR was opened by the [Changesets
release](https://github.com/changesets/action) GitHub action. When
you're ready to do a release, you can merge this and publish to npm
yourself or [setup this action to publish
automatically](https://github.com/changesets/action#with-publishing). If
you're not ready to do a release yet, that's fine, whenever you add more
changesets to main, this PR will be updated.
# Releases
## @trigger.dev/sdk@4.4.0
### Minor Changes
- Added `query.execute()` which lets you query your Trigger.dev data
using TRQL (Trigger Query Language) and returns results as typed JSON
rows or CSV. It supports configurable scope (environment, project, or
organization), time filtering via `period` or `from`/`to` ranges, and a
`format` option for JSON or CSV output.
([#3060](https://github.com/triggerdotdev/trigger.dev/pull/3060))
```typescript
import { query } from "@trigger.dev/sdk";
import type { QueryTable } from "@trigger.dev/sdk";
// Basic untyped query
const result = await query.execute("SELECT run_id, status FROM runs
LIMIT 10");
// Type-safe query using QueryTable to pick specific columns
const typedResult = await query.execute<QueryTable<"runs", "run_id" |
"status" | "triggered_at">>(
"SELECT run_id, status, triggered_at FROM runs LIMIT 10"
);
typedResult.results.forEach((row) => {
console.log(row.run_id, row.status); // Fully typed
});
// Aggregation query with inline types
const stats = await query.execute<{ status: string; count: number }>(
"SELECT status, COUNT(*) as count FROM runs GROUP BY status",
{ scope: "project", period: "30d" }
);
// CSV export
const csv = await query.execute("SELECT run_id, status FROM runs", {
format: "csv",
period: "7d",
});
console.log(csv.results); // Raw CSV string
```
### Patch Changes
- Add `maxDelay` option to debounce feature. This allows setting a
maximum time limit for how long a debounced run can be delayed, ensuring
execution happens within a specified window even with continuous
triggers.
([#2984](https://github.com/triggerdotdev/trigger.dev/pull/2984))
```typescript
await myTask.trigger(payload, {
debounce: {
key: "my-key",
delay: "5s",
maxDelay: "30m", // Execute within 30 minutes regardless of continuous
triggers
},
});
```
- Aligned the SDK's `getRunIdForOptions` logic with the Core package to
handle semantic targets (`root`, `parent`) in root tasks.
([#2874](https://github.com/triggerdotdev/trigger.dev/pull/2874))
- Export `AnyOnStartAttemptHookFunction` type to allow defining
`onStartAttempt` hooks for individual tasks.
([#2966](https://github.com/triggerdotdev/trigger.dev/pull/2966))
- Fixed a minor issue in the deployment command on distinguishing
between local builds for the cloud vs local builds for self-hosting
setups.
([#3070](https://github.com/triggerdotdev/trigger.dev/pull/3070))
- Updated dependencies:
- `@trigger.dev/core@4.4.0`
## @trigger.dev/build@4.4.0
### Patch Changes
- Updated dependencies:
- `@trigger.dev/core@4.4.0`
## trigger.dev@4.4.0
### Patch Changes
- Fix runner getting stuck indefinitely when `execute()` is called on a
dead child process.
([#2978](https://github.com/triggerdotdev/trigger.dev/pull/2978))
- Add optional `timeoutInSeconds` parameter to the
`wait_for_run_to_complete` MCP tool. Defaults to 60 seconds. If the run
doesn't complete within the timeout, the current state of the run is
returned instead of waiting indefinitely.
([#3035](https://github.com/triggerdotdev/trigger.dev/pull/3035))
- Fixed a minor issue in the deployment command on distinguishing
between local builds for the cloud vs local builds for self-hosting
setups.
([#3070](https://github.com/triggerdotdev/trigger.dev/pull/3070))
- Updated dependencies:
- `@trigger.dev/core@4.4.0`
- `@trigger.dev/build@4.4.0`
- `@trigger.dev/schema-to-json@4.4.0`
## @trigger.dev/core@4.4.0
### Patch Changes
- Add `maxDelay` option to debounce feature. This allows setting a
maximum time limit for how long a debounced run can be delayed, ensuring
execution happens within a specified window even with continuous
triggers.
([#2984](https://github.com/triggerdotdev/trigger.dev/pull/2984))
```typescript
await myTask.trigger(payload, {
debounce: {
key: "my-key",
delay: "5s",
maxDelay: "30m", // Execute within 30 minutes regardless of continuous
triggers
},
});
```
- Fixed a minor issue in the deployment command on distinguishing
between local builds for the cloud vs local builds for self-hosting
setups.
([#3070](https://github.com/triggerdotdev/trigger.dev/pull/3070))
- fix: vendor superjson to fix ESM/CJS compatibility
([#2949](https://github.com/triggerdotdev/trigger.dev/pull/2949))
Bundle superjson during build to avoid `ERR_REQUIRE_ESM` errors on
Node.js versions that don't support `require(ESM)` by default (<
22.12.0) and AWS Lambda which intentionally disables it.
- Add Vercel integration support to API schemas: `commitSHA` and
`integrationDeployments` on deployment responses, and `source` field for
environment variable imports.
([#2994](https://github.com/triggerdotdev/trigger.dev/pull/2994))
## @trigger.dev/python@4.4.0
### Patch Changes
- Updated dependencies:
- `@trigger.dev/core@4.4.0`
- `@trigger.dev/sdk@4.4.0`
- `@trigger.dev/build@4.4.0`
## @trigger.dev/react-hooks@4.4.0
### Patch Changes
- Fix `onComplete` callback firing prematurely when the realtime stream
disconnects before the run finishes.
([#2929](https://github.com/triggerdotdev/trigger.dev/pull/2929))
- Updated dependencies:
- `@trigger.dev/core@4.4.0`
## @trigger.dev/redis-worker@4.4.0
### Patch Changes
- Updated dependencies:
- `@trigger.dev/core@4.4.0`
## @trigger.dev/rsc@4.4.0
### Patch Changes
- Updated dependencies:
- `@trigger.dev/core@4.4.0`
## @trigger.dev/schema-to-json@4.4.0
### Patch Changes
- Updated dependencies:
- `@trigger.dev/core@4.4.0`
---------
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Closes #<issue>
## ✅ Checklist
- [ ] I have followed every step in the [contributing
guide](https://github.com/triggerdotdev/trigger.dev/blob/main/CONTRIBUTING.md)
- [ ] The PR title follows the convention.
- [ ] I ran and tested the code works
---
## Testing
N/A - Documentation and OpenAPI schema updates only.
---
## Changelog
Added comprehensive Queue Management API support:
**OpenAPI Endpoints:**
- `GET /api/v1/queues` - List all queues with pagination support
- `GET /api/v1/queues/{queueParam}` - Retrieve a specific queue by ID,
task ID, or custom queue name
- `POST /api/v1/queues/{queueParam}/pause` - Pause or resume a queue
- `POST /api/v1/queues/{queueParam}/concurrency/override` - Override
queue concurrency limits
- `POST /api/v1/queues/{queueParam}/concurrency/reset` - Reset
concurrency limits to base values
**Schema Definitions:**
- `QueueObject` - Complete queue representation with concurrency details
- `ListQueuesResult` - Paginated queue listing response
**Documentation:**
- Updated `queue-concurrency.mdx` with SDK usage examples for queue
management
- Added 5 new management API documentation pages for each endpoint
- Updated `docs.json` navigation structure with new "Queues API" section
All endpoints support flexible queue identification (by ID, task ID, or
custom queue name) and include TypeScript code samples.
---
## Screenshots
N/A
💯https://claude.ai/code/session_01LyrXwxHCbejvi34fykifPP
Co-authored-by: Claude <noreply@anthropic.com>
Adds a region selector to the Test task page and Replay run dialog, so
users can override the region from the dashboard. Disabled with a
placeholder for dev environments.
Closes#3016
This will prevent internal logs to be added to the
task_events_search_table
Closes #<issue>
## ✅ Checklist
- [x] I have followed every step in the [contributing
guide](https://github.com/triggerdotdev/trigger.dev/blob/main/CONTRIBUTING.md)
- [x] The PR title follows the convention.
- [x] I ran and tested the code works
---
## Testing
Ran the migration, deleted the old invalid rows and ran new tasks.
The undesired logs are not added to the table.
---
## Changelog
Updated the MATERIALIZED VIEW to also filter for `trace_id != ''`
---------
Co-authored-by: Matt Aitken <matt@mattaitken.com>
Expand documentation for the Vercel integration with detailed usage,
installation, environment variable sync, atomic deployments, and
environment mapping. Replace the previous "coming soon" placeholder with
complete instructions and UI flow for connecting via the Trigger.dev
dashboard or the Vercel Marketplace. Explain required GitHub
integration,
how env vars sync in both directions, which vars are excluded, and how
to control sync behavior. Describe atomic deployments (default for
production), how they gate Vercel deployments to ensure task/app
consistency, and note related configuration changes. Add tips and notes
to guide setup and troubleshooting.
This provides users with actionable guidance to connect Vercel, map
environments, and keep app and tasks in sync without custom CI scripts.
Fixes an issue introduced in #3024.
The behavior for local builds in older CLI versions relies on
`externalBuildData` to be defined to distinguish from the self-hosting
local build path, even though it doesn't actually use the token.
Summary
- Add API endpoint to run TRQL queries
- Implement SDK function for executing queries
## SDK
Added `query.execute()` which lets you query your Trigger.dev data using
TRQL (Trigger Query Language) and returns results as typed JSON rows or
CSV. It supports configurable scope (environment, project, or
organization), time filtering via `period` or `from`/`to` ranges, and a
`format` option for JSON or CSV output.
```typescript
import { query } from "@trigger.dev/sdk";
import type { QueryTable } from "@trigger.dev/sdk";
// Basic untyped query
const result = await query.execute("SELECT run_id, status FROM runs LIMIT 10");
// Type-safe query using QueryTable to pick specific columns
const typedResult = await query.execute<QueryTable<"runs", "run_id" | "status" | "triggered_at">>(
"SELECT run_id, status, triggered_at FROM runs LIMIT 10"
);
typedResult.results.forEach(row => {
console.log(row.run_id, row.status); // Fully typed
});
// Aggregation query with inline types
const stats = await query.execute<{ status: string; count: number }>(
"SELECT status, COUNT(*) as count FROM runs GROUP BY status",
{ scope: "project", period: "30d" }
);
// CSV export
const csv = await query.execute(
"SELECT run_id, status FROM runs",
{ format: "csv", period: "7d" }
);
console.log(csv.results); // Raw CSV string
```
Documents the skipColumns option on useRealtimeRun and
useRealtimeRunsWithTag for status-only subscriptions (smaller payloads,
e.g. for progress/completion UI). Adds a troubleshooting section for the
“Failed to index deployment” source-map error when using the Bun
runtime, with a pnpm patch workaround and link to the GitHub issue
Extract `applyPeriod` callback from `applySelection` so preset period
buttons ("Created in the last X") apply immediately when clicked,
instead of only updating the selection state and requiring a separate
apply step.
Also validates `maxPeriodDays` on instant-apply so the upgrade prompt
still works correctly for plan-limited periods.
For now we’re going to always add FINAL to TRQL queries for data
correctness.
In the future we will implement an automated optimization where we use
`SELECT argMax(column, _version)` and `WHERE _is_deleted = 0`. But this
is a more complex change and needs more investigation of downsides.
A customer experienced a bug where their subscription downgraded to the
free plan unintentionally. This was due to a concurrency upgrade payment
attempt that failed a card check. We auto retry the payment across 2
weeks of attempts. When the final attempt failed, the whole subscription
downgraded.
Now we check if the payment is successful and if not, return an error
immediately so the subscription isn't modified until a successful
payment is made for an upgrade.
Summary
- Remove LIMIT from built-in dashboard queries
- Make concurrency configurable per project via environment variables
- Fix widget fallback period to Metrics default (1d) instead of 7d
- Handle concurrency at the project level
- Sort series for graphs so largest is displayed at the bottom (legend
shows largest at top)
- Use average aggregation for some built-in charts
- Improve aggregation handling for the legend
- Only render chart points when there is data; render dots on line
charts
- Truncate legend items and show tooltip on hover
- Better preserve chart configuration when the underlying query changes
## ✅ Checklist
- [X] I have followed every step in the [contributing
guide](https://github.com/triggerdotdev/trigger.dev/blob/main/CONTRIBUTING.md)
- [X] The PR title follows the convention.
- [X] I ran and tested the code works
---
## Testing
Tested the migration on test env and locally.
Tested query and merge performance.
Generated tasks and observed the ingested data and searches.
---
## Changelog
* New ClickHouse table & MV (task_events_search_v1): A search-optimized
materialized view that filters out debug events, partial spans, and
empty span events at ingestion time.
* ClickHouse client updates: New getLogsSearchListQueryBuilder and
taskEventsSearch accessor on the ClickHouse class.
* LogsListPresenter: Switches to the new search table, uses
triggered_timestamp for cursor pagination instead of unixTimestamp.
* Spans route: Also switches to the new search query builder.
* Seed spanSpammer task: Adds a 10s trace with events and metadata
operations for testing.
Summary
- Implemented metrics dashboards with a built-in dashboard and custom
dashboards
- Added a "Big number” display type
What changed
- New data format for metric layouts and saving/editing layouts
(editing, saving, cancel revert)
- QueryWidget usable on Query page and Metrics dashboards
- Time filtering, auto-reloading and timeBucket() auto-bin support
- Filters added to metrics; widget popover/improved history and blank
states
- Side menu:
- Metrics/Insights section with icons, colors, padding, collapsible
behavior and reordering of custom dashboards
- Move action logic into service for reuse and API querying; refactor
reordering for reuse
<!-- devin-review-badge-begin -->
---
<a
href="https://app.devin.ai/review/triggerdotdev/trigger.dev/pull/3019"
target="_blank">
<picture>
<source media="(prefers-color-scheme: dark)"
srcset="https://static.devin.ai/assets/gh-open-in-devin-review-dark.svg?v=1">
<img
src="https://static.devin.ai/assets/gh-open-in-devin-review-light.svg?v=1"
alt="Open with Devin">
</picture>
</a>
<!-- devin-review-badge-end -->
---------
Co-authored-by: James Ritchie <james@trigger.dev>
## Summary
- Adds an optional `timeoutInSeconds` parameter (default 60s) to the
`wait_for_run_to_complete` MCP tool
- If the run doesn't complete within the timeout, returns the current
run state instead of blocking indefinitely
- Uses `AbortSignal.timeout()` combined with the existing MCP signal
Fixes#3032
## ✅ Checklist
- [X] I have followed every step in the [contributing
guide](https://github.com/triggerdotdev/trigger.dev/blob/main/CONTRIBUTING.md)
- [X] The PR title follows the convention.
- [X] I ran and tested the code works
---
## Testing
Manually tested each implementation.
---
## Changelog
* Updated Logs Page with the new implementation in time filter component
* In TRQL editor users can now click on empty/blank spaces in the editor
and the cursor will appear
* Added CMD + / for line commenting in TRQL
* Activated proper undo/redo functionality in CodeMirror (TRQL editor)
* Added a check for new logs button, previously once the user got to the
end of the logs he could not check for newer logs
* Added showing MS in logs page Dates
* Removed LOG_INFO internal logs, they are available with Admin Debug
flag
* Added support for correct timezone render on server side.
* Increased CLICKHOUSE_LOGS_LIST_MAX_MEMORY_USAGE to 1GB
* Changed Previous run/ Next run to J/K, consistent with previous/next
page in Runs list
## Summary
- **Fix Docker publish automation**: The `v.docker.*` tags pushed by the
release workflow using `GITHUB_TOKEN` don't trigger the publish workflow
(GitHub Actions limitation to prevent infinite loops). Added a
`workflow_call` to `publish.yml` directly from the release job so Docker
images are built automatically after npm publish. Tags are still pushed
for reference.
- **Fix worker Containerfiles**: The coordinator, docker-provider, and
kubernetes-provider builds have been failing since the superjson
vendoring change in `@trigger.dev/core` (#2949). The Containerfiles now
run `bundle-vendor` before `build:bundle` to generate the vendor files
that esbuild needs.
### Context
- Docker images on GHCR have been stuck at v4.3.0 — v4.3.1, v4.3.2,
v4.3.3 tags existed on GitHub but never triggered publish runs
- The worker builds (publish-worker) have been failing on every push to
main since Jan 30
## Test plan
- [x] Verified kubernetes-provider Containerfile builds locally with the
fix
- [x] Manually dispatched publish workflow for v4.3.1 — all jobs
succeeded
<!-- devin-review-badge-begin -->
---
<a
href="https://app.devin.ai/review/triggerdotdev/trigger.dev/pull/3013"
target="_blank">
<picture>
<source media="(prefers-color-scheme: dark)"
srcset="https://static.devin.ai/assets/gh-open-in-devin-review-dark.svg?v=1">
<img
src="https://static.devin.ai/assets/gh-open-in-devin-review-light.svg?v=1"
alt="Open with Devin">
</picture>
</a>
<!-- devin-review-badge-end -->
This pull request overhauls the "Building with AI" documentation
section. It includes a comprehensive restructuring of the main
building-with-ai page with new setup guides and troubleshooting
sections, reorganizes the navigation hierarchy to elevate
mcp-agent-rules as a top-level page, and updates multiple documentation
pages to clarify the relationships between three AI tools: Skills, Agent
Rules, and MCP Server. Changes also include formatting improvements,
such as replacing italicized text with inline code formatting, and
consistent additions of explanatory Note blocks and CardGroup components
across related pages.
Display the deployment trigger source (CLI, CI/CD, Dashboard, GitHub
Integration) with appropriate icons on the deployment details page. The
triggeredVia field was already in the database but not displayed.
Co-authored-by: Claude <noreply@anthropic.com>
Adds optional pod affinity so pods from the same project prefer
scheduling on the same node. This can help improve image cache hit
rates; subsequent pods benefit from already-pulled image layers,
reducing startup time.
Complements the built-in ImageLocality scheduler plugin by helping
during burst scheduling scenarios. Pod affinity sees scheduled pods
immediately, while ImageLocality only sees images after they're fully
pulled.
Configuration:
- `KUBERNETES_PROJECT_AFFINITY_ENABLED` - Enable/disable (default:
false)
- `KUBERNETES_PROJECT_AFFINITY_WEIGHT` - Scheduler weight 1-100
(default: 50)
- `KUBERNETES_PROJECT_AFFINITY_TOPOLOGY_KEY` - Topology key (default:
kubernetes.io/hostname)
Uses soft (preferred) affinity so pods always schedule even if preferred
node is full.
<!-- devin-review-badge-begin -->
---
<a
href="https://app.devin.ai/review/triggerdotdev/trigger.dev/pull/2995">
<picture>
<source media="(prefers-color-scheme: dark)"
srcset="https://static.devin.ai/assets/gh-open-in-devin-review-dark.svg?v=1">
<img
src="https://static.devin.ai/assets/gh-open-in-devin-review-light.svg?v=1"
alt="Open with Devin">
</picture>
</a>
<!-- devin-review-badge-end -->
## Summary
- When a child process crashes and a retry (`RETRY_IMMEDIATELY`) is
attempted on the same `TaskRunProcess`, `execute()` hangs forever
because the IPC send is silently skipped and the attempt promise can
never resolve
- This caused runner pods to stay up indefinitely with no heartbeats or
polls
- Fix: reject the attempt promise immediately when the child is not
connected, so the controller can proceed to warm start or exit
## Test plan
- [x] Added `taskRunProcess.test.ts` — verifies `execute()` rejects
promptly instead of hanging when the child process is dead
- [x] Deploy and verify no more stuck runner pods accumulate over time
Closes#2798
When a run finished the logs UI could get stuck and so be pending and
never update again. If you did a hard reload it would be correct.
This happened because when we insert a log/span we ping Redis which
causes a reload of the UI. However there was a race condition – the
insert into ClickHouse can take a while so we were refreshing the UI too
early. Then never refreshing it again.
Changes
- Send refresh pings every 5s to keep run page logs live
- Throttle updates so the run UI is never updated more than once per
second
- Stop auto-reloading when a run has been completed for >= 30s
- Add type inference improvements for the throttle function
<!-- devin-review-badge-begin -->
---
<a
href="https://app.devin.ai/review/triggerdotdev/trigger.dev/pull/2971">
<picture>
<source media="(prefers-color-scheme: dark)"
srcset="https://static.devin.ai/assets/gh-open-in-devin-review-dark.svg?v=1">
<img
src="https://static.devin.ai/assets/gh-open-in-devin-review-light.svg?v=1"
alt="Open with Devin">
</picture>
</a>
<!-- devin-review-badge-end -->
## ✅ Checklist
- [ ] I have followed every step in the [contributing
guide](https://github.com/triggerdotdev/trigger.dev/blob/main/CONTRIBUTING.md)
- [ ] The PR title follows the convention.
- [ ] I ran and tested the code works
---
## Description
This PR standardizes the `@types/node` dependency across the entire
monorepo to version `20.14.14`. Previously, different packages were
using different versions (ranging from 12.20.55 to 22.13.9), which could
cause type conflicts and inconsistencies.
### Changes Made
1. **tsconfig.json** - Added `"node"` to the `types` array in
`apps/webapp/tsconfig.json` to ensure Node.js types are properly
recognized
2. **package.json overrides** - Added `@types/node` version override to
`20.14.14` in the root `package.json`
3. **pnpm-lock.yaml** - Updated lock file to reflect the standardized
version across all packages and their dependencies
4. **Fixture package.json** - Updated
`packages/cli-v3/e2e/fixtures/emit-decorator-metadata/package.json` to
use the standardized version
This ensures consistent type definitions across the monorepo and
prevents version mismatches that could lead to type errors or unexpected
behavior.
---
## Testing
- Verified that all package references to `@types/node` now point to
version `20.14.14`
- Confirmed that the lock file properly reflects the override across all
transitive dependencies
- Ensured TypeScript configuration includes Node.js types for proper
type checking
---
## Changelog
- Standardized `@types/node` to version `20.14.14` across all packages
in the monorepo
- Added `"node"` to TypeScript compiler types in webapp configuration
- Updated all package dependencies to use the consistent version through
pnpm overrides
💯https://claude.ai/code/session_018eqp2LvvErkFSN9oK5xBh1
<!-- devin-review-badge-begin -->
---
<a
href="https://app.devin.ai/review/triggerdotdev/trigger.dev/pull/2970">
<picture>
<source media="(prefers-color-scheme: dark)"
srcset="https://static.devin.ai/assets/gh-open-in-devin-review-dark.svg?v=1">
<img
src="https://static.devin.ai/assets/gh-open-in-devin-review-light.svg?v=1"
alt="Open with Devin">
</picture>
</a>
<!-- devin-review-badge-end -->
---------
Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: claude[bot] <41898282+claude[bot]@users.noreply.github.com>
Co-authored-by: Eric Allam <ericallam@users.noreply.github.com>
Bundle superjson and its dependency (copy-anything) during build to
avoid
ERR_REQUIRE_ESM errors on Node.js versions that don't support
require(ESM)
by default (< 22.12.0) and AWS Lambda which intentionally disables it.
- Add scripts/bundle-superjson.mjs to bundle superjson with esbuild
- Update build script to bundle vendor files before tshy compilation
- Move superjson from dependencies to devDependencies
- Update imports to use vendored bundles
Fixes#2937
<!-- devin-review-badge-begin -->
---
<a
href="https://app.devin.ai/review/triggerdotdev/trigger.dev/pull/2949">
<picture>
<source media="(prefers-color-scheme: dark)"
srcset="https://static.devin.ai/assets/gh-open-in-devin-review-dark.svg?v=1">
<img
src="https://static.devin.ai/assets/gh-open-in-devin-review-light.svg?v=1"
alt="Open with Devin">
</picture>
</a>
<!-- devin-review-badge-end -->
---------
Co-authored-by: claude[bot] <41898282+claude[bot]@users.noreply.github.com>
Co-authored-by: Eric Allam <ericallam@users.noreply.github.com>
Summary
- Query: add time limits, performance improvements, and styling updates
Changes
- Add ClickHouse output_text and error_text columns with indexes
- Automatically use _text columns for JSON based on query pattern;
support JSON column data prefixes
- Add idempotency key and scope columns
- Add enforcedWhereClause for tenant and time restrictions, instead of
the old tenant stuff.
- Implement basic time filter limiting and set default time period based
on plan; show message when results are clipped
- UX: resizable code area (including vertical splits), collapsible
sidebar, fix table/chart vertical sizing, max height for chart legend in
fullscreen
- Styling and UI tweaks: improved chart legend styling, more chart
colours, thinner line chart stroke, pricing callout color, improved
layout for callouts
- Features: generate and save AI titles
<!-- devin-review-badge-begin -->
---
<a
href="https://app.devin.ai/review/triggerdotdev/trigger.dev/pull/2953">
<picture>
<source media="(prefers-color-scheme: dark)"
srcset="https://static.devin.ai/assets/gh-open-in-devin-review-dark.svg?v=1">
<img
src="https://static.devin.ai/assets/gh-open-in-devin-review-light.svg?v=1"
alt="Open with Devin">
</picture>
</a>
<!-- devin-review-badge-end -->
Closes #
## ✅ Checklist
- [ ] I have followed every step in the [contributing
guide](https://github.com/triggerdotdev/trigger.dev/blob/main/CONTRIBUTING.md)
- [ ] The PR title follows the convention.
- [ ] I ran and tested the code works
---
## Testing
Tested the MiddleTruncate component in the TasksDropdown by:
1. Verifying that long task names (e.g.,
"namespace:category:subcategory:task-name") are truncated in the middle
2. Confirming the full text appears in a tooltip on hover
3. Testing responsive behavior - truncation adjusts when the container
is resized
4. Verifying that short task names that fit within the container are
displayed in full without truncation
---
## Changelog
Added a new `MiddleTruncate` primitive component that intelligently
truncates text in the middle while preserving the beginning and end
portions. This is particularly useful for long hierarchical identifiers
like task slugs.
**Key features:**
- Truncates text in the middle with an ellipsis (…) when it exceeds
available width
- Shows full text in a tooltip on hover when truncated
- Responsive - recalculates truncation on container resize using
ResizeObserver
- Maintains minimum character visibility (4 chars minimum on each side
for readability)
- Integrated into TasksDropdown to handle long task names
**Changes:**
- Created new `MiddleTruncate.tsx` component with binary search
algorithm for optimal character distribution
- Updated TasksDropdown to use MiddleTruncate for task slug display
- Increased TasksDropdown popover width from 240px to 360px to provide
better space for truncated text
---
## Screenshots
💯https://github.com/user-attachments/assets/a7a2191a-2e36-437e-ab3f-517fe7620b93
<!-- devin-review-badge-begin -->
---
<a
href="https://app.devin.ai/review/triggerdotdev/trigger.dev/pull/2946">
<picture>
<source media="(prefers-color-scheme: dark)"
srcset="https://static.devin.ai/assets/gh-open-in-devin-review-dark.svg?v=1">
<img
src="https://static.devin.ai/assets/gh-open-in-devin-review-light.svg?v=1"
alt="Open with Devin">
</picture>
</a>
<!-- devin-review-badge-end -->
---------
Co-authored-by: Claude <noreply@anthropic.com>
## ✨ Changes
### UI & UX
- Normalized log level display across table and detail view
- Fixed table header scroll behavior and sidebar positioning
- Improved loading state with taller segment and disabled resizing
- Added "no more logs" message with count
- Enhanced keyboard shortcuts
### Filtering & Search
- Streamlined filters: RunId and Task only (removed run filters)
- Side panel closes when filters change
- Fixed logs from previous search remaining in table
- Fixed table scroll position when changing filters
### Backend
- Added performance indexes on message and attributes
(`014_add_task_runs_v2_search_indexes.sql`)
- Added DEBUG level logging by default
- Removed internal logs from display
- Fixed ServiceValidationError forwarding to frontend
- Removed v1 logs API support
## Summary
Fixes#2856 - The `onComplete` callback in `useRealtimeRun` was firing prematurely
## Root Cause
The callback was triggered when the long-poll stream ended, regardless
of whether the run had actually completed. Reverse proxies often close
idle connections, causing the stream to end prematurely. In this case it
was caused by fetch abort due to React strict mode.
## Fix
Changed the condition from checking if `run` exists to checking if
`run?.finishedAt` exists, ensuring `onComplete` only fires when the run
has reached a terminal state.
---------
Co-authored-by: claude[bot] <41898282+claude[bot]@users.noreply.github.com>
Co-authored-by: nicktrn <nicktrn@users.noreply.github.com>
## Summary
Fixes a concurrency leak in the batch queue where visibility timeout
reclaims do not release concurrency slots.
**The bug:** When a message visibility timeout expires (60s),
`reclaimTimedOut` puts the message back in the queue but does NOT
release the concurrency slot. The messageId stays in the concurrency set
(`engine:batch:concurrency:tenant:{envId}`), counting against the tenant
limit even though the message is no longer in-flight.
This causes:
1. Tenant appears at capacity when checking `SCARD >= limit`
2. New messages get released back to queue instead of being processed
3. Messages stuck in infinite loop, master queue grows indefinitely
**The fix:**
- Modified `reclaimTimedOut` to capture message data (including
tenantId) BEFORE releasing from in-flight
- Returns `ReclaimedMessageInfo[]` with messageId, queueId, tenantId,
and metadata
- `#reclaimTimedOutMessages` now iterates over reclaimed messages and
calls `concurrencyManager.release()` for each
## Test plan
- [x] Added test: `should return reclaimed message info with tenantId
for concurrency release`
- [x] Added test: `should return empty array when no messages have timed
out`
- [x] Added test: `should reclaim multiple timed-out messages and return
all their info`
- [x] Updated `raceConditions.test.ts` for new return type
- [x] All tests passing
- [ ] Monitor production after deploy for concurrency leak recurrence
refs TRI-7049
<!-- devin-review-badge-begin -->
---
<a
href="https://app.devin.ai/review/triggerdotdev/trigger.dev/pull/2907">
<picture>
<source media="(prefers-color-scheme: dark)"
srcset="https://static.devin.ai/assets/gh-open-in-devin-review-dark.svg?v=1">
<img
src="https://static.devin.ai/assets/gh-open-in-devin-review-light.svg?v=1"
alt="Open with Devin">
</picture>
</a>
<!-- devin-review-badge-end -->
Adds documentation for creating additional worker groups via the admin
API endpoint, including how to make users admin (new vs existing users),
and clarifies that ADMIN_EMAILS only applies on signup.
Adds documentation notes clarifying that:
- delayed runs execute on the current deployment version
- guidance for Date objects in payloads
- static IP availability
- version locking behavior for delayed runs
This PR was opened by the [Changesets
release](https://github.com/changesets/action) GitHub action. When
you're ready to do a release, you can merge this and publish to npm
yourself or [setup this action to publish
automatically](https://github.com/changesets/action#with-publishing). If
you're not ready to do a release yet, that's fine, whenever you add more
changesets to main, this PR will be updated.
# Releases
## @trigger.dev/build@4.3.3
### Patch Changes
- Updated dependencies:
- `@trigger.dev/core@4.3.3`
## trigger.dev@4.3.3
### Patch Changes
- Updated dependencies:
- `@trigger.dev/core@4.3.3`
- `@trigger.dev/build@4.3.3`
- `@trigger.dev/schema-to-json@4.3.3`
## @trigger.dev/core@4.3.3
### Patch Changes
- Add support for AI SDK v6 (Vercel AI SDK)
([#2919](https://github.com/triggerdotdev/trigger.dev/pull/2919))
- Updated peer dependency to allow `ai@^6.0.0` alongside v4 and v5
- Updated internal code to handle async validation from AI SDK v6's
Schema type
- Expose user-provided idempotency key and scope in task context.
`ctx.run.idempotencyKey` now returns the original key passed to
`idempotencyKeys.create()` instead of the hash, and
`ctx.run.idempotencyKeyScope` shows the scope ("run", "attempt", or
"global").
([#2903](https://github.com/triggerdotdev/trigger.dev/pull/2903))
- Fix batch trigger failing with "ReadableStream is locked" error when
network failures occur mid-stream. Added safe stream cancellation that
gracefully handles locked streams during retry attempts.
([#2917](https://github.com/triggerdotdev/trigger.dev/pull/2917))
- Add a maxDepth to flatten/unflattenAttributes to prevent possible
issues ([#2890](https://github.com/triggerdotdev/trigger.dev/pull/2890))
## @trigger.dev/python@4.3.3
### Patch Changes
- Updated dependencies:
- `@trigger.dev/sdk@4.3.3`
- `@trigger.dev/core@4.3.3`
- `@trigger.dev/build@4.3.3`
## @trigger.dev/react-hooks@4.3.3
### Patch Changes
- Updated dependencies:
- `@trigger.dev/core@4.3.3`
## @trigger.dev/redis-worker@4.3.3
### Patch Changes
- Updated dependencies:
- `@trigger.dev/core@4.3.3`
## @trigger.dev/rsc@4.3.3
### Patch Changes
- Updated dependencies:
- `@trigger.dev/core@4.3.3`
## @trigger.dev/schema-to-json@4.3.3
### Patch Changes
- Updated dependencies:
- `@trigger.dev/core@4.3.3`
## @trigger.dev/sdk@4.3.3
### Patch Changes
- Add support for AI SDK v6 (Vercel AI SDK)
([#2919](https://github.com/triggerdotdev/trigger.dev/pull/2919))
- Updated peer dependency to allow `ai@^6.0.0` alongside v4 and v5
- Updated internal code to handle async validation from AI SDK v6's
Schema type
- Expose user-provided idempotency key and scope in task context.
`ctx.run.idempotencyKey` now returns the original key passed to
`idempotencyKeys.create()` instead of the hash, and
`ctx.run.idempotencyKeyScope` shows the scope ("run", "attempt", or
"global").
([#2903](https://github.com/triggerdotdev/trigger.dev/pull/2903))
- Updated dependencies:
- `@trigger.dev/core@4.3.3`
---------
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
There’s an edge case that means runs can end up in the
currentConcurrency set when they’re not in the correct state for
execution. This means they will be permanently stuck in queued.
Given an environmentId this will fix those runs.
This is a temporary fix while we permanently fix the issue.
---------
Co-authored-by: claude[bot] <41898282+claude[bot]@users.noreply.github.com>
Co-authored-by: Eric Allam <ericallam@users.noreply.github.com>
When we auth API keys we get the environment, project and org. This is a
very hot path so even though these queries are fast they contribute a
significant percentage of total load.
This moves them to use the read replica instead.
Moving usage updates into the run engine to prevent inefficient &
additional incremental updates to the TaskRun table. Read/Modify/Write
pattern is safe inside of the run engine because of the run lock. We can
also now cap the usageDurationMs value from overflowing and causing an
error.
## Why?
This is preventing at least one update per TaskRun and instead updating
these values piggybacking on other updates.
## Aurora PostgreSQL Reader Consistency Notes
### TL;DR
Aurora readers share the same storage as the writer, but maintain
separate in-memory page caches. This means:
- **Storage is always consistent** - writes are synchronously committed
to shared storage
- **Page cache can lag** - typically <100ms, but can cause stale reads
if data is cached
### How It Works
1. Writer commits to shared storage (synchronous 4/6 quorum)
2. Writer sends cache invalidation messages to readers (asynchronous)
3. If reader has data in cache → returns cached (potentially stale)
value
4. If reader has cache miss → fetches from shared storage (always
current)
### Monitoring
```sql
SELECT server_id,
CASE WHEN session_id = 'MASTER_SESSION_ID' THEN 'Writer' ELSE 'Reader' END AS role,
replica_lag_in_msec
FROM aurora_replica_status();
```
Move expensive findMany queries for PENDING_VERSION and
WAITING_FOR_DEPLOY
runs to read replicas to avoid blocking migrations on the primary
database.
Changes:
- Add readOnlyPrisma to SystemResources type
- Pass readOnlyPrisma to systems in RunEngine constructor
- Update pendingVersionSystem to use readOnlyPrisma for findMany
- Update executeTasksWaitingForDeploy to use _replica for findMany
## Summary
- Adds `IF NOT EXISTS` to the migration that adds
`idempotencyKeyOptions` column to prevent errors if the column already
exists
## Migration Checksum Fix
If you've already applied the previous version of this migration, you'll
need to update the checksum in your `_prisma_migrations` table to match
the new migration file.
**Previous checksum:**
`f8876e274e3f7735312275eb24a9c4b40f512ac12a286b2de3add47f66df5b27`
**New checksum:**
`0620a914ddbaf01279576274432e51c41f41502cd4c8de38621625380750e397`
### Fix instructions
Run this SQL command against your database:
```sql
UPDATE "_prisma_migrations"
SET checksum = '0620a914ddbaf01279576274432e51c41f41502cd4c8de38621625380750e397'
WHERE migration_name = '20260116154810_add_idempotency_key_options_to_task_run';
```
This updates the stored checksum to match the modified migration file,
allowing future migrations to proceed without checksum mismatch errors.
## Test plan
- [x] Verified migration applies cleanly on fresh database
- [ ] Verified checksum update works on database with previous migration
applied
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Adds support for skipping Postgres migrations on container startup via
the new `SKIP_POSTGRES_MIGRATIONS` environment variable.
Set `SKIP_POSTGRES_MIGRATIONS=1` to skip migrations, matching the
existing behavior of `SKIP_CLICKHOUSE_MIGRATIONS`.
What changed
- Upgraded recharts to 2.15.2
- Added multiple chart types and components: big number, line, stacked,
bar (including zoomable & reference line), big dataset bar, and usage
graph
- Implemented custom legend with animated values, tooltip showing x-axis
data, and hover/highlight behaviors for stacks and legend
- Added loading, no-data, and invalid chart states plus loading spinners
and improved loading animations/layout
- Storybook integration: initial charts setup, separate chart files,
alphabetized menu, chart state toggles, and story updates
- Interaction & UX improvements: zooming (drag/select), crosshair
pointer, show/select dates while zooming, prevent text selection on
drag, hide mouse wheel zoom, capped legend items, axis/legend styling
tweaks, better spacing, and min-height for charts
- Data & state handling: moved date data to route for unified zooming,
moved chartState to main Chart component, moved hard-coded/mock data out
of components, and set chart data when zooming to start/end dates
- Performance & animation: turned off/reduced chart animations, sped up
animated numbers, removed hover transitions for bars
- New UI primitives and layout: Card component, small card updates, SVG
icons, improved segmented control and popover variants, table
improvements (resizable columns, filtering, sorting, scrolling fixes)
- Various fixes and polish: tooltip style fixes, legend value updates,
hover/leave state resets, bar width fixes for small datasets,
type/import fixes, and numerous small style/typo tweaks
---------
Co-authored-by: James Ritchie <james@trigger.dev>
## Summary
- Add support for Vercel AI SDK v6 as a peer dependency
- Update internal code to handle async validation from AI SDK v6's
Schema type
Closes#2918
Co-authored-by: claude[bot] <41898282+claude[bot]@users.noreply.github.com>
Co-authored-by: Eric Allam <ericallam@users.noreply.github.com>
When fetch crashes mid-stream during batch item upload (e.g., connection
reset, timeout), the request stream may remain locked by fetch's
internal reader. Attempting to cancel a locked stream throws 'Invalid
state: ReadableStream is locked', causing the batch operation to fail.
Added safeStreamCancel() helper that gracefully handles locked streams
by catching and ignoring the locked error. The stream will be cleaned up
by garbage collection when fetch eventually releases the reader.
Fixes customer issue where batchTrigger failed with ReadableStream
locked error during network instability.
## Summary
- Store the original user-provided idempotency key and scope alongside
the hash
- Expose `ctx.run.idempotencyKey` as the user-provided key (not the
hash)
- Add `ctx.run.idempotencyKeyScope` to show the scope ("run", "attempt",
or "global")
<img width="539" height="450" alt="CleanShot 2026-01-19 at 11 40 46"
src="https://github.com/user-attachments/assets/b6f42991-697e-4314-a164-aef77b8fd25c"
/>
## Problem
Idempotency keys were hashed (SHA-256) before storage, making debugging
difficult since users couldn't see the value they originally set or
search for runs by idempotency key.
## Solution
Attach metadata to the `String` object returned by
`idempotencyKeys.create()` using a Symbol, extract it in the SDK before
the API call, and store it in the database alongside the hash.
```typescript
const key = await idempotencyKeys.create("my-key", { scope: "global" });
await childTask.triggerAndWait(payload, { idempotencyKey: key });
// In child task:
ctx.run.idempotencyKey // "my-key" (previously showed the hash)
ctx.run.idempotencyKeyScope // "global"
```
Test plan
- Trigger task with idempotencyKeys.create() using different scopes (run, attempt, global)
- Verify ctx.run.idempotencyKey returns user-provided key
- Verify ctx.run.idempotencyKeyScope returns correct scope
- Verify PostgreSQL stores idempotencyKeyOptions JSON
- Verify ClickHouse receives idempotency_key_user and idempotency_key_scope via replication
---------
Co-authored-by: James Ritchie <james@trigger.dev>
Added documentation for the new Limits page feature that allows users to
view their current limits, quotas, and rate limit usage in real-time
from the dashboard. The page displays rate limit token availability,
quota usage, and plan features for organizations.
## Files changed
- `docs/limits.mdx` - Added introductory paragraph about the new Limits
page in the dashboard
Generated from [feat(webapp): New limits
page](https://github.com/triggerdotdev/trigger.dev/pull/2885) @samejr
Co-authored-by: mintlify[bot] <109931778+mintlify[bot]@users.noreply.github.com>
Co-authored-by: James Ritchie <james@trigger.dev>
Adds support to configure CPU/memory request ratios per machine preset.
Falls back to the global request ratio configs if no specific override
is specified.
Runs across different machine presets have different usage patters, so
this enables use to manage the available capacity better.
Closes #<issue>
## ✅ Checklist
- [ ] I have followed every step in the [contributing
guide](https://github.com/triggerdotdev/trigger.dev/blob/main/CONTRIBUTING.md)
- [ ] The PR title follows the convention.
- [ ] I ran and tested the code works
---
## Testing
_[Describe the steps you took to test this change]_
---
## Changelog
_[Short description of what has changed]_
---
## Screenshots
_[Screenshots]_
💯
---------
Co-authored-by: Matt Aitken <matt@mattaitken.com>
Summary
- Add nullable projectId field to TaskScheduleInstance.
- Create an index for TaskScheduleInstance.environmentId (added only if
it doesn’t exist, handled concurrently).
- Ensure TaskScheduleInstance.projectId is set everywhere in the
codebase.
Backfilling projectIds, once this is live
```sql
UPDATE "TaskScheduleInstance" tsi
SET "projectId" = ts."projectId"
FROM "TaskSchedule" ts
WHERE tsi."taskScheduleId" = ts."id";
```
* Removed EVENT_REPOSITORY_CLICKHOUSE_ROLLOUT_PERCENT
* Added hasLogsPageAccess featureFlag for logs page
* Replaced attributes with attributes_text for logs to reduce memory
usage and improve query performance
* Added support for event_v1 for logs, now depending on the settings the
logs are fetched either from `task_events_v1` or `task_events_v2`
* Show an error in the interface in cast the repository store is
`postgres`
## Summary
- Upgrades Node.js from 20.19.0 to 20.20.0 (and 22.12.0 to 22.22.0 for
supervisor) to address the async_hooks stack overflow DoS vulnerability
- Adds `maxDepth` parameter (default 128) to `flattenAttributes` and
`unflattenAttributes` to prevent stack overflow on maliciously deep
nested structures
## Details
The vulnerability (patched in Node.js 20.20.0, 22.22.0, 24.13.0, 25.3.0)
causes unrecoverable crashes (exit code 7) when stack overflow occurs
during async_hooks callbacks. Since the webapp uses `AsyncLocalStorage`,
it was theoretically vulnerable.
### Changes
**Node.js version updates:**
- `docker/Dockerfile`: 20.11.1 → 20.20.0
- `apps/supervisor/Containerfile`: 22-alpine → 22.22.0-alpine
- `.nvmrc`: 20.19.0 → 20.20.0
- `apps/supervisor/.nvmrc`: 22.12.0 → 22.22.0
- `references/prisma-7/.nvmrc`: 20.19.0 → 20.20.0
- All GitHub workflows: 20.19.0 → 20.20.0
**Defense in depth:**
- Added `maxDepth` parameter to `flattenAttributes()` and
`unflattenAttributes()` in `packages/core` to prevent stack overflow on
deeply nested user input
## Test plan
- [x] All existing `flattenAttributes` tests pass (50 tests)
- [x] New tests for depth limiting added
- [x] Verify Docker builds work with new base images
Fixed documentation examples to use correct 'data' field instead of
'output' for the waitpoint token completion endpoint.
The API schema expects 'data' in the request body, but all code examples
(curl, Python, Ruby, Go) incorrectly showed 'output', causing waitpoints
to complete with empty/undefined output when users followed the docs.
Fixes#2872
Closes #<issue>
## ✅ Checklist
- [x] I have followed every step in the [contributing
guide](https://github.com/triggerdotdev/trigger.dev/blob/main/CONTRIBUTING.md)
- [x] The PR title follows the convention.
- [x] I ran and tested the code works
---
## Testing
_[Describe the steps you took to test this change]_
This is simple doc fix
---
## Changelog
_[Short description of what has changed]_
Documentation example for the specific api route had a wrong field,
fixed that to have correct field
---
## Screenshots
_[Screenshots]_
<img width="780" height="252" alt="Screenshot 2026-01-14 at 10 49 50 PM"
src="https://github.com/user-attachments/assets/50a03f2c-edfb-4bd4-bb72-a0fd79e77216"
/>
The above image shows the correct request format, but docs previously
had incorrect payload.
<img width="723" height="307" alt="Screenshot 2026-01-14 at 11 24 01 PM"
src="https://github.com/user-attachments/assets/b9096225-3f7c-4511-b8c2-e8144c896900"
/>
This is the exact wrong field in docs, that was fixed
https://trigger.dev/docs/wait-for-token#from-another-language💯
Co-authored-by: appdevelopers9a <appdeveloper@s9alabs.com>
It’s useful to know when they were modified for debugging and auditing.
For existing rows createdAt and updatedAt are set to now() during the
migration, to avoid a nullable column.
When users set `secrets.enabled=false` to use an external secret via
`secrets.existingSecret`, the environment variables `SESSION_SECRET`,
`MAGIC_LINK_SECRET`, `ENCRYPTION_KEY`, and `MANAGED_WORKER_SECRET` were
not being populated from the secret.
Fixes#2859
Also adds automatic helm prereleases for PRs
---------
Co-authored-by: claude[bot] <41898282+claude[bot]@users.noreply.github.com>
Co-authored-by: nicktrn <nicktrn@users.noreply.github.com>
Change EVENT_REPOSITORY_DEFAULT_STORE default from "postgres" to
"clickhouse_v2" so new deployments favor the ClickHouse v2 event store
by default. This updates runtime behavior to use the newer store
implementation unless explicitly overridden.
Don’t allow aliased columns to be queried – it was actually safe but
confusing. We call `created_at` -> `triggered_at` but we still allowed
created_at which was confusing.
Now we have nice errors if you try select columns that aren’t
selectable.
Also removed a ClickHouse setting `allow_experimental_object_type` which
worked fine locally but stopped all queries working on ClickHouse Cloud
🤦♂️
The Cancel button was missing an onClick handler to close the modal
dialog. This caused confusing behavior where clicking Cancel would not
dismiss the dialog. Also added type="button" to prevent form submission
since the button is inside a form.
Co-authored-by: Claude <noreply@anthropic.com>
## Summary
Optimizes the runs replication service for better CPU efficiency and
throughput when inserting task runs into ClickHouse.
### Key Changes
- **Switch to compact array format** - Uses
`JSONCompactEachRowWithNames` instead of `JSONEachRow` for ClickHouse
inserts, reducing JSON serialization overhead
- **Type-safe tuple arrays** - Introduces `TaskRunInsertArray` and
`PayloadInsertArray` tuple types with compile-time column order
validation
- **Pre-sorted batch inserts** - Sorts inserts by primary key before
flushing for better ClickHouse insert performance
- **Programmatic index generation** - `TASK_RUN_INDEX` and
`PAYLOAD_INDEX` are generated from column arrays to prevent manual
synchronization errors
### Files Changed
- `runsReplicationService.server.ts` - Core optimization to use compact
array inserts
- `@internal/clickhouse` - Added `insertCompactRaw` method and tuple
types
- `taskRuns.ts` - Column definitions, index constants, and insert
functions
---------
Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
Closes #<issue>
## ✅ Checklist
- [x] I have followed every step in the [contributing
guide](https://github.com/triggerdotdev/trigger.dev/blob/main/CONTRIBUTING.md)
- [x] The PR title follows the convention.
- [x] I ran and tested the code works
---
## Testing
- Verified log detail view displays correctly with message, metadata,
and attributes
- Tested search highlighting functionality in log messages (escapes
special regex characters)
- Confirmed tabs (Details/Run) switch properly with keyboard shortcuts
(d/r)
- Verified run information loads via async fetcher in Run tab
- Tested close button and Escape key for dismissing the panel
- Verified log details display correct information: level badges, kind
badges, timestamps, trace IDs, span IDs
- Confirmed links to parent spans and run pages work correctly
- Tested with various log levels (ERROR, WARN, INFO, DEBUG, TRACE) and
kinds (SPAN, SPAN_EVENT, LOG_*)
- Verified admin-only fields display correctly when user has admin
access
- Tested data loading states and error states (log not found, run not
found)
---
## Changelog
Created new Logs page.
The information shown is gathered from the spans from each run.
The feature supports all run filters with two new filters for level and
logs text search.
---
## Screenshots
<img width="2059" height="1196" alt="Logs page preview"
src="https://github.com/user-attachments/assets/70b667b4-98cc-4728-855a-2766dd5c1aa5"
/>
💯
---------
Co-authored-by: James Ritchie <james@trigger.dev>
Summary
- Improve query experience and safety across ClickHouse and TSQL.
Changes
- Display JSON columns when in non-pretty mode (no longer show [Object
Object]).
- Sanitize ClickHouse errors originating from TSQL.
- Remove tenant details from errors.
- Add AI-assisted error-fixing for queries.
- Improve code quality and readability.
- Provide autocomplete support for enum values.
- Enforce limits on ClickHouse queries (10s query limit).
- Add org-level and global concurrency limits.
- Warn and train AI to avoid SELECT *; when used, only return core
columns and show info.
- If AI suggests no time range, default to past 7 days.
- Format the default query for readability.
- Add an admin-only EXPLAIN button.
- Prevent impersonation queries from being saved to history.
**Background**
Runs with `large-1x` or `large-2x` machine presets are disproportionally
affected by scheduling delays during peak times. This is in part caused
by the fact that the worker pool is shared for all runs, meaning large
runs compete with smaller runs for available capacity. Because large
runs require significantly more CPU and memory, they are harder for the
scheduler to bin-pack onto existing nodes, often requiring a node with a
significant amount of free resources or waiting for a new node to spin
up entirely. This effect is amplified during peak times when nodes are
already densely packed with smaller workloads, leaving insufficient
contiguous resources for large runs. Also, large runs make up a small
percentage of the total runs.
**Changes**
This PR adds Kubernetes node affinity settings to separate large and
standard machine workloads across node pools.
- Controlled via `KUBERNETES_LARGE_MACHINE_POOL_LABEL` env var (disabled
when not set)
- Large machine presets (large-*) get a soft preference to schedule on
the large pool, with fallback to standard nodes
- Non-large machines are excluded from the large pool via required
anti-affinity
- This ensures the large machine pool is reserved for large workloads
while allowing large workloads to spill over to standard nodes if needed
UI/UX improvement to the date/time picker:
- You can now choose a custom duration
- Adds a new DateTimePicker.tsx component, using a new shadcn
Calendar.tsx component
- Clear UI separation between the 2 actions, choosing a duration or
choosing date range
- Adds new quick select options for picking a date range quickly
https://github.com/user-attachments/assets/6b59b49d-2a56-4354-ad72-d8426437e56e
---------
Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: claude[bot] <41898282+claude[bot]@users.noreply.github.com>
Co-authored-by: Eric Allam <ericallam@users.noreply.github.com>
- Add CLAUDE.md providing Claude Code guidance and documenting the
Claude Code skill
- Add trigger-dev-tasks skill to assist writing Trigger.dev tasks
- Add SDK rules version 4.3.0 including batch trigger v2 and debouncing
features
TRQL (pronounced Treacle like the delicious British dark sweet syrup) is
the TRiggerQueryLanguage. It allows users to safely write queries on
their data. The queries are safely turned into ClickHouse queries which
are tenant-safe and not SQL injectable.
https://github.com/user-attachments/assets/bbfca473-b3fc-4150-8fe6-79e8840a2d29
This started out as a translation of HogQL by PostHog from Python to
TypeScript.
Features
- Tenant safe queries.
- Many underlying ClickHouse features including functions and
aggregations.
- Virtual columns, which are exposed to users as real columns but are
actually expressions.
- Transformations of data types and where clauses.
- Simple JSON path querying.
- Limits on execution time.
- Reporting of query statistics.
## Query page
There’s a new Query page (currently behind a feature flag) where you can
write TRQL queries and execute them against your environment, project or
organization.
Features
- Executing TRQL queries
- Syntax highlighting and errors
- Autocomplete
- AI generation/editing of queries
- Help and examples
- Table with auto-inferred data types from the table schema
- Table cell renderers for our special types like Run ids, environments,
machines, tasks, queues, etc.
- Copy/export as CSV/JSON
- Line and bar graphs with grouping and stacking
- History of queries
Replaces `redirectDocument` with `useFetcher` for editing environment
variables. This allows background form submission without full page
reload, which preserves:
- Scroll position in the env vars list
- "Reveal values" toggle state
- Search filter state
Fixes#2845
Generated with [Claude Code](https://claude.ai/code)
Co-authored-by: claude[bot] <41898282+claude[bot]@users.noreply.github.com>
Co-authored-by: Eric Allam <ericallam@users.noreply.github.com>
Updated the billing alerts documentation to reflect the new spike alerts
feature added in PR #2829. The documentation now explains both standard
alerts (75%, 90%, 100%, 200%, 500%) and spike alerts (10x, 20x, 50x,
100x) that help catch runaway usage from bugs or errors.
## Files changed
- `docs/how-to-reduce-your-spend.mdx` - Added section explaining the two
types of billing alerts
Generated from [Chore(webapp): Adds additional billing
alerts](https://github.com/triggerdotdev/trigger.dev/pull/2829) @samejr
Co-authored-by: mintlify[bot] <109931778+mintlify[bot]@users.noreply.github.com>
Add colored console warnings when the event loop is blocked and wire
a feature flag to enable/disable notifications- Introduce
notifyEventLoopBlocked() in eventLoopMonitor.server.ts to
log a colored warning with blocked and async type.
- Call notifyEventLoopBlocked() when an event-loop stall is detected.
- Add EVENT_LOOP_MONITOR_NOTIFY_ENABLED to env schema with a default of
"0" so notifications are off by default.
- Will notify when over the `EVENT_LOOP_MONITOR_THRESHOLD_MS` env var
This makes it easier to spot long event-loop stalls during development
or when notifications are explicitly enabled.
<img width="840" height="132" alt="CleanShot 2026-01-07 at 15 03 24@2x"
src="https://github.com/user-attachments/assets/be20fa6a-be2b-46a1-aa89-d0913ed8b5b3"
/>
This PR fixes some issues with the new BatchQueue by implementing the
full two-phase dequeue process in the FairQueue, and moving the
responsibility of consuming the worker queue to the BatchQueue and
independently enabling it via the `BATCH_QUEUE_WORKER_QUEUE_ENABLED` env
var. We've also introduced the `BATCH_QUEUE_SHARD_COUNT` env var to
control the count of master queue shards in the FairQueue. We can also
control how many queues are considered in each iteration of the master
queue consumer via the `BATCH_QUEUE_MASTER_QUEUE_LIMIT` env var.
This PR will also now skip trying to dequeue from tenants that are at
concurrency capacity, which should lead to fewer issues with low
concurrency tenants blocking higher concurrency tenants from processing.
The Override concurrency limit modal has 2 type="submit" buttons. The
first one in the DOM was firing when the "enter" key is hit which
canceled and reset the limit instead which is a bad UX.
### The fix
This fix adds a hidden button above in the DOM order which mirrors the
Update Override button. Having a double submit button is rare in our
modals so feels safe to add this to the specific modal that needs it.
### Alternative solution
Switching the order of the buttons in the main FormButton component,
then using `flex-row-reverse` to flip them back in CSS works, but it
reverses the tab order. Adding a `tabIndex` to fix that issue didn't
seem to work reliably.
### UI Improvements to the Concurrency page:
- Truncates long branch names and includes a tooltip
- The Tables have a new variant if you don't want the rows to highlight
on hover
- Small fix to pluralize some words in the purchase modal
- Fix to prevent tooltip buttons being `type=submit`
- Updates the /limits docs page to include purchasing more concurrency
- Adds a clear banner when you have a positive balance of unallocated
concurrency
https://github.com/user-attachments/assets/54d927c3-84e3-4d55-8f42-726098f4daf0
- Adds 4 additional alert thresholds to ensure customers are emailed if
they have runaway usage.
- Separated these into a new section called "Spike alerts" with a
tooltip so it's clear what they are.
- Tooltip message is: "Catch runaway usage from bugs or errors. We
recommend keeping these enabled as a safety net."
- A billing service PR now returns all orgs to populate the email list,
rather than oldest 5.
- Adds `defaultChecked` logic to honour existing orgs who have
configured alerts in the DB. New orgs get all alerts checked on by
default.
<img width="1316" height="1560" alt="CleanShot 2026-01-05 at 09 43
56@2x"
src="https://github.com/user-attachments/assets/ce749407-2b7f-4864-9c09-9333c5ac495a"
/>
Fixes#2835
There were still some flags in here we removed, deploying is a lot
simpler now for self-hosters.
Also updates the github actions guide.
---------
Co-authored-by: Claude <noreply@anthropic.com>
**Improvements to the run ID copy button and run navigation buttons for
consistency**
- Adds some x-padding and layout adjustment to the copy ID button.
<img width="664" height="114" alt="CleanShot 2025-12-19 at 16 05 21@2x"
src="https://github.com/user-attachments/assets/ebc8e0de-011b-419c-bdcc-eb4157553d1c"
/>
- New custom navigation icons that work better at tiny sizes
<img width="330" height="196" alt="CleanShot 2025-12-19 at 16 06 38@2x"
src="https://github.com/user-attachments/assets/bfd8d6b8-8a65-4eac-9ce1-d70acf0ad265"
/>
Some other small improvements/fixes:
- Fixes a browser html error where there was a <button> inside a
<button>
- Updates the shortcut description to match the tooltip text for
consistency
- Made the hover states more consistent
- The shortcut bar at the bottom snaps to the list sooner because there
are more items now
Closes #<issue>
## ✅ Checklist
- [ ] I have followed every step in the [contributing
guide](https://github.com/triggerdotdev/trigger.dev/blob/main/CONTRIBUTING.md)
- [ ] The PR title follows the convention.
- [ ] I ran and tested the code works
---
## Testing
_[Describe the steps you took to test this change]_
---
## Changelog
_[Short description of what has changed]_
---
## Screenshots
_[Screenshots]_
💯
This fixes a regression introduced in #2778 - stable sort is required
for deterministic builds, but we can safely preserve order for the user
package.json during package updates
This PR was opened by the [Changesets
release](https://github.com/changesets/action) GitHub action. When
you're ready to do a release, you can merge this and publish to npm
yourself or [setup this action to publish
automatically](https://github.com/changesets/action#with-publishing). If
you're not ready to do a release yet, that's fine, whenever you add more
changesets to main, this PR will be updated.
# Releases
## @trigger.dev/build@4.3.1
### Patch Changes
- Updated dependencies:
- `@trigger.dev/core@4.3.1`
## trigger.dev@4.3.1
### Patch Changes
- Updated dependencies:
- `@trigger.dev/core@4.3.1`
- `@trigger.dev/build@4.3.1`
- `@trigger.dev/schema-to-json@4.3.1`
## @trigger.dev/core@4.3.1
### Patch Changes
- Added support for idempotency reset
([#2777](https://github.com/triggerdotdev/trigger.dev/pull/2777))
## @trigger.dev/python@4.3.1
### Patch Changes
- Updated dependencies:
- `@trigger.dev/sdk@4.3.1`
- `@trigger.dev/core@4.3.1`
- `@trigger.dev/build@4.3.1`
## @trigger.dev/react-hooks@4.3.1
### Patch Changes
- Updated dependencies:
- `@trigger.dev/core@4.3.1`
## @trigger.dev/redis-worker@4.3.1
### Patch Changes
- Updated dependencies:
- `@trigger.dev/core@4.3.1`
## @trigger.dev/rsc@4.3.1
### Patch Changes
- Updated dependencies:
- `@trigger.dev/core@4.3.1`
## @trigger.dev/schema-to-json@4.3.1
### Patch Changes
- Updated dependencies:
- `@trigger.dev/core@4.3.1`
## @trigger.dev/sdk@4.3.1
### Patch Changes
- feat: Support for new batch trigger system
([#2779](https://github.com/triggerdotdev/trigger.dev/pull/2779))
- feat(sdk): Support debouncing runs when triggering with new debounce
options
([#2794](https://github.com/triggerdotdev/trigger.dev/pull/2794))
- Added support for idempotency reset
([#2777](https://github.com/triggerdotdev/trigger.dev/pull/2777))
- Updated dependencies:
- `@trigger.dev/core@4.3.1`
---------
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Add support for resetting idempotency keys both from ui and sdk
## ✅ Checklist
- [x] I have followed every step in the [contributing
guide](https://github.com/triggerdotdev/trigger.dev/blob/main/CONTRIBUTING.md)
- [x] The PR title follows the convention.
- [x] I ran and tested the code works
---
## Testing
- Created a new run with a idempotency idempotencyKey.
- Started a new run with the same task and got redirected to the first
run.
- Deleted the key from the UI on the run details
- Started a new run with the same task and it created a new one
- Did the above steps using the SDK
---
## Changelog
- Add new action route for resetting idempotency keys via UI
- Add reset button in Idempotency section of run detail view
- Added API and SDK for resetting imdepotency
- Updated docs page for this feature
---
## Screenshots
_[Screenshots]_
<img width="438" height="363" alt="Screenshot 2025-12-11 at 11 56 37"
src="https://github.com/user-attachments/assets/30b8ef5e-8aac-4d04-b57a-9bf30d085dcb"
/>
Adds support for **debounced task runs** - when triggering a task with a
debounce key, subsequent triggers with the same key will reschedule the
existing delayed run instead of creating new runs. This continues until
no new triggers occur within the delay window.
## Usage
```typescript
await myTask.trigger({ userId: "123" }, {
debounce: {
key: "user-123-update",
delay: "5s",
mode: "leading", // default
}
});
```
- **key**: Scoped to the task identifier
- **delay**: How long to wait before executing (supports duration
strings like `"5s"`, `"1m"`)
- **mode**: Either `"leading"` or `"trailing"`. Leading debounce will
use the payload and options from the first run created with the debounce
key. Trailing will use payload and options from the last run.
### "trailing" mode overrides
When using `mode: "trailing"` with debounce, the following options are
updated from the **last** trigger:
- **`payload`** - The task input data
- **`metadata`** - Run metadata
- **`tags`** - Run tags (replaces existing tags)
- **`maxAttempts`** - Maximum retry attempts
- **`maxDuration`** - Maximum compute time
- **`machine`** - Machine preset (cpu/memory)
## Behavior
- **First run wins**: The first trigger creates the run, subsequent
triggers push its execution time later
- **Idempotency keys take precedence**: If both are specified,
idempotency is checked first
- **Max duration**: Configurable via `DEBOUNCE_MAX_DURATION_MS` env var
(default: 10 minutes)
Works with `triggerAndWait` - parent runs correctly block on the
debounced run.
New batch trigger system with larger payloads, streaming ingestion,
larger batch sizes, and a fair processing system.
This PR introduces a new `FairQueue` abstraction inspired by our own
`RunQueue` that enables multi-tenant fair queueing with concurrency
limits. The new `BatchQueue` is built on top of the `FairQueue`, and
handles processing Batch triggers in a fair manner with per-environment
concurrency limits defined per-org. Additionally, there is a global
concurrency limit to prevent the BatchQueue system from creating too
many runs too quickly, which can cause downstream issues.
For this new BatchQueue system we have a completely new batch trigger
creation and ingestion system. Previously this was a single endpoint
with a single JSON body that defined details about the batch as well as
all the items in the batch.
We're introducing a two-phase batch trigger ingestion system. In the
first phase, the BatchTaskRun record is created (and possibly rate
limited). The second phase is another endpoint that accepts an NDJSON
body with each line being a single item/run with payload and options.
At ingestion time all items are added to a queue, in order, and then
processed by the BatchQueue system.
## New batch trigger rate limits
This PR implements a new batch trigger specific rate limit, configured
on the `Organization.batchRateLimitConfig` column, and defaults using
these environment variables:
- `BATCH_RATE_LIMIT_REFILL_RATE` defaults to 10
- `BATCH_RATE_LIMIT_REFILL_INTERVAL` the duration interval, defaults to
`"10s"`
- `BATCH_RATE_LIMIT_MAX` defaults to 1200
This rate limiter is scoped to the environment ID and controls how many
runs can be submitted via batch triggers per interval. The SDK handles
the retrying side.
## Batch queue concurrency limits
The new column `Organization.batchQueueConcurrencyConfig` now defines an
org specific `processingConcurrency` value, with a backup of the env var
`BATCH_CONCURRENCY_LIMIT_DEFAULT` which defaults to 10. This controls
how many batch queue items are processed concurrently per environment.
There is also a global rate limit for the batch queue set via the
`BATCH_QUEUE_GLOBAL_RATE_LIMIT` which defaults to being disabled. If
set, the entire batch queue system won't process more than
`BATCH_QUEUE_GLOBAL_RATE_LIMIT` items per second. This allows
controlling the maximum number of runs created per second via batch
triggers.
## Batch trigger settings
- `STREAMING_BATCH_MAX_ITEMS` controls the maximum number of items in a
single batch
- `STREAMING_BATCH_ITEM_MAXIMUM_SIZE` controls the maximum size of each
item in a batch
- `BATCH_CONCURRENCY_DEFAULT_CONCURRENCY` controls the default
environment concurrency
- `BATCH_QUEUE_DRR_QUANTUM` how many credits each environment gets each
round for the DRR scheduler
- `BATCH_QUEUE_MAX_DEFICIT` the maximum deficit for the DRR scheduler
- `BATCH_QUEUE_CONSUMER_COUNT` how many queue consumers to run
- `BATCH_QUEUE_CONSUMER_INTERVAL_MS` how frequently they poll for items
in the queue
### Configuration Recommendations by Use Case
**High-throughput priority (fairness acceptable at 0.98+):**
```env
BATCH_QUEUE_DRR_QUANTUM=25
BATCH_QUEUE_MAX_DEFICIT=100
BATCH_QUEUE_CONSUMER_COUNT=10
BATCH_QUEUE_CONSUMER_INTERVAL_MS=50
BATCH_CONCURRENCY_DEFAULT_CONCURRENCY=25
```
**Strict fairness priority (throughput can be lower):**
```env
BATCH_QUEUE_DRR_QUANTUM=5
BATCH_QUEUE_MAX_DEFICIT=25
BATCH_QUEUE_CONSUMER_COUNT=3
BATCH_QUEUE_CONSUMER_INTERVAL_MS=100
BATCH_CONCURRENCY_DEFAULT_CONCURRENCY=5
```
## Changelog
- Add disableAdjacentRows prop to TaskRunsTable component to control
table state encoding
- Pass rootOnlyDefault prop from loader to TaskRunsTable for proper
state management
- Disable adjacent run navigation in schedule, waitpoint, and other
inspector views
- Preserve adjacent run navigation on main runs list page with rootOnly
filter support
This PR was opened by the [Changesets
release](https://github.com/changesets/action) GitHub action. When
you're ready to do a release, you can merge this and publish to npm
yourself or [setup this action to publish
automatically](https://github.com/changesets/action#with-publishing). If
you're not ready to do a release yet, that's fine, whenever you add more
changesets to main, this PR will be updated.
# Releases
## trigger.dev@4.3.0
### Minor Changes
- feat(cli): deterministic image builds for deployments
([#2778](https://github.com/triggerdotdev/trigger.dev/pull/2778))
- feat(cli): enable zstd compression for deployment images
([#2773](https://github.com/triggerdotdev/trigger.dev/pull/2773))
### Patch Changes
- The new `triggeredVia` field is now populated in deployments via the
CLI. ([#2767](https://github.com/triggerdotdev/trigger.dev/pull/2767))
- fix(dev): stop max listeners exceeded warning messages when running
more than 10 runs concurrently
([#2771](https://github.com/triggerdotdev/trigger.dev/pull/2771))
- Upgrade @modelcontextprotocol/sdk to 1.24.3
([#2768](https://github.com/triggerdotdev/trigger.dev/pull/2768))
- Updated dependencies:
- `@trigger.dev/core@4.3.0`
- `@trigger.dev/build@4.3.0`
- `@trigger.dev/schema-to-json@4.3.0`
## @trigger.dev/core@4.3.0
### Minor Changes
- feat(cli): deterministic image builds for deployments
([#2778](https://github.com/triggerdotdev/trigger.dev/pull/2778))
### Patch Changes
- The new `triggeredVia` field is now populated in deployments via the
CLI. ([#2767](https://github.com/triggerdotdev/trigger.dev/pull/2767))
## @trigger.dev/build@4.3.0
### Patch Changes
- Updated dependencies:
- `@trigger.dev/core@4.3.0`
## @trigger.dev/python@4.3.0
### Patch Changes
- Updated dependencies:
- `@trigger.dev/core@4.3.0`
- `@trigger.dev/build@4.3.0`
- `@trigger.dev/sdk@4.3.0`
## @trigger.dev/react-hooks@4.3.0
### Patch Changes
- Updated dependencies:
- `@trigger.dev/core@4.3.0`
## @trigger.dev/redis-worker@4.3.0
### Patch Changes
- Updated dependencies:
- `@trigger.dev/core@4.3.0`
## @trigger.dev/rsc@4.3.0
### Patch Changes
- Updated dependencies:
- `@trigger.dev/core@4.3.0`
## @trigger.dev/schema-to-json@4.3.0
### Patch Changes
- Updated dependencies:
- `@trigger.dev/core@4.3.0`
## @trigger.dev/sdk@4.3.0
### Patch Changes
- Updated dependencies:
- `@trigger.dev/core@4.3.0`
---------
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
This PR makes our image builds deterministic and reproducible by
ensuring that identical source code always produces the same image
layers and image digest. This means that deployments where nothing has
changed will no longer invalidate the image cache in our worker cluster
nodes, thus avoid making the cold starts for runs worse.
**Context**
New deployments currently increase the cold start times for runs, as
they generate a new image which needs to be pulled in the worker cluster
where runs are executed. It happens also when the source code for the
deployment has not changed due to non-deterministic steps in our build
system. This addresses the latter issue by making builds reproducible.
**Main changes**
- Avoided baking `TRIGGER_DEPLOYMENT_ID` and
`TRIGGER_DEPLOYMENT_VERSION` in the image, we now pass these via the
supervisor instead.
- Used `json-stable-stringify` for consistent key ordering in the files
we generate for the build, e.g., `package.json`, `build.json`,
`index.json`.
- Removed `metafile.json` from the image contents as it is not actually
used in the container. This is only relevant for the `analyze` command.
- Added `SOURCE_DATE_EPOCH=0` and `rewrite-timestamp=true` to Docker
builds to normalize file timestamps.
- Removed some `timings` and `outputHashes` from build outputs and
manifests.
The builds are now reproducible for both native build server and Depot
paths. This should also lead to better image layer cache reuse in
general.
- Add replace prop to LinkButton to use history replacement for adjacent
run navigation
- Preserve span and tab params when navigating between adjacent runs
- Disable animations for completed spans in timeline to improve
performance
- Include spanId in runs list navigation for better context preservation
- Direct link to task test page when filtering by single task with no
runs
- Fix minor styling issue with run friendlyId display padding
## ✅ Checklist
- [x] I have followed every step in the [contributing
guide](https://github.com/triggerdotdev/trigger.dev/blob/main/CONTRIBUTING.md)
- [x] The PR title follows the convention.
- [x] I ran and tested the code works
---
## Testing
Set up the local github application, and tested its connection with
trigger.dev
Checked:
- Change is backwards compatible
- Actions/ux are uniform across the pages (empty Tasks, Deployments,
project settings)
- Connecting GH, Connecting Repo, disconnecting Repo, modifying settings
---
## Changelog
- Create new resource route for GitHub settings management with loader
and actions
- Add GitHubSettingsPresenter to fetch connected repos and installations
- Implement GitHubSettingsPanel component for reusable GitHub
configuration UI
- Refactor project settings page to use shared GitHubSettingsPanel
component
- Integrate GitHub connection flow into empty state onboarding for Tasks
and Deployments
- Add support for GitHub repo connection, disconnection, and branch
tracking settings
- Include redirect URL support for seamless navigation after GitHub
actions
- Remove duplicate GitHub connection code from project settings route
---
## Screenshots
https://github.com/user-attachments/assets/8fc24699-640b-4f9e-afd8-b26edc945218🐐
---------
Co-authored-by: James Ritchie <james@trigger.dev>
This will speed up ice cold starts (*) for two reasons:
- better compression ratio
- faster decompression
This is a minor release because zstd compression will now be enabled by
default for all deployments.
(*) ice cold starts happen when deploy images are not cached on the
worker node yet. These cold start durations are highly dependent on
image size and as it turns out, also the type of compression used.
Fixed the issue where Cmd+Left Arrow was being intercepted by the
TreeView component on task runs screen.
Solution:
Added a check in the getTreeProps keyboard handler to detect when
metaKey (Cmd on macOS) is pressed with Left Arrow. When detected, the
handler returns early without preventing the default browser behavior,
allowing Chrome's native back navigation to work.
Closes #<issue>
## ✅ Checklist
- [ x] I have followed every step in the [contributing
guide](https://github.com/triggerdotdev/trigger.dev/blob/main/CONTRIBUTING.md)
- [ x] The PR title follows the convention.
- [ x] I ran and tested the code works
---
## Testing
Went to the task details page and confirmed that Cmd + Left Arrow will
navigate back
---
## Changelog
Modified TreeView.tsx to check for e.metaKey before handling Left Arrow
key events
When Cmd+Left is pressed, the event is no longer prevented, allowing
browser default behavior
---
Co-authored-by: Mihai Popescu <mihaipopescu@Mihais-MacBook-Pro.local>
Deployments are affected by general API rate limits, this is just a
quick fix by whitelisting the deployment related endpoints. In a follow
up PR we'll add a separate rate limiter for this group of endpoints.
## ✅ Checklist
- [x] I have followed every step in the [contributing
guide](https://github.com/triggerdotdev/trigger.dev/blob/main/CONTRIBUTING.md)
- [x] The PR title follows the convention.
- [x] I ran and tested the code works
---
## Testing
Manual testing of the task run pages
---
## Changelog
- Add previous/next run navigation buttons to run detail page header
- Support [ and ] keyboard shortcuts to jump between adjacent runs
- Preserve runs table state (filters, pagination) when navigating
- Preload adjacent page runs at boundaries for seamless navigation
- Add actions prop to PageTitle component
- Document shortcut in keyboard shortcuts panel
- Store current filter state from runs table as `tableState` search
param when navigating to individual run pages
- Restore filters when navigating back from run detail view to runs list
- Update `v3RunPath` and `v3RunSpanPath` helpers to accept optional
searchParams
- Use `useOptimisticLocation` to capture current search params in
TaskRunsTable
- Parse `tableState` param in run detail route and pass filters to back
button
- This improves UX by remembering filter selections (task, status, date
range, etc.) when users click into a run and then navigate back to the
runs list
- Add new text-below variant that shows "Click to copy" tooltip on hover
and "Copied" on click. Also add controlled open/onOpenChange props to
SimpleTooltip for managing tooltip visibility.
---
## Screenshots
https://github.com/user-attachments/assets/5067bbe0-1bcd-4e75-80a7-f56dabd5ed69
Changes in this PR:
- Arbitrary refs are now allowed when triggering the release workflow
manually (ref must be on the main branch).
- Release summary is now displayed in the GH job output; makes for a
nicer experience when approving the release workflow.
<!-- CURSOR_SUMMARY -->
> [!NOTE]
> Updates `BatchTriggerV3Service` to send `batch.id` (not `friendlyId`)
as `batchId` to `TriggerTaskService.call`.
>
> <sup>Written by [Cursor
Bugbot](https://cursor.com/dashboard?tab=bugbot) for commit
68bf8df4a29bdcac5bfb2806bc417541f69ffd6e. This will update automatically
on new commits. Configure
[here](https://cursor.com/dashboard?tab=bugbot).</sup>
<!-- /CURSOR_SUMMARY -->
<!-- CURSOR_SUMMARY -->
> [!NOTE]
> Centralizes SIGTERM handling in `DevSupervisor` and removes per-run
SIGTERM listeners in `DevRunController` to avoid
MaxListenersExceededWarning under high concurrency.
>
> - **Dev runtime**:
> - **SIGTERM handling**: Add centralized handler in
`packages/cli-v3/src/dev/devSupervisor.ts` to gracefully stop all run
controllers; unregisters on `shutdown()`.
> - **Cleanup**: Remove per-controller `SIGTERM` listener and handler
from `packages/cli-v3/src/entryPoints/dev-run-controller.ts` to reduce
event listeners and warnings.
> - **Changeset**: Add patch note in
`.changeset/fuzzy-ghosts-admire.md`.
>
> <sup>Written by [Cursor
Bugbot](https://cursor.com/dashboard?tab=bugbot) for commit
5ad2f5341829cebf6fd37a3c616a2db5e4ad936a. This will update automatically
on new commits. Configure
[here](https://cursor.com/dashboard?tab=bugbot).</sup>
<!-- /CURSOR_SUMMARY -->
This PR applies a small change to the deployments table to keep track
of:
- where the deployment was triggered from
- build server metadata, if the build server was involved
We recently upgraded the ECR sdk version. Our ECR repo exists check
relies on the type of the error thrown and the new ECR sdk version seems
to have broken that behavior. This PR adds a workaround to the issue.
The manual trigger is currently enabled only for the prerelease job.
This PR adds it for the normal release flow too, as it is useful to
retrigger failed release workflows.
This PR was opened by the [Changesets
release](https://github.com/changesets/action) GitHub action. When
you're ready to do a release, you can merge this and publish to npm
yourself or [setup this action to publish
automatically](https://github.com/changesets/action#with-publishing). If
you're not ready to do a release yet, that's fine, whenever you add more
changesets to main, this PR will be updated.
# Releases
## trigger.dev@4.2.0
### Minor Changes
- feat(cli): upgrade bun deployments to v1.3.3
([#2756](https://github.com/triggerdotdev/trigger.dev/pull/2756))
### Patch Changes
- fix(otel): exported logs and spans will now have matching trace IDs
([#2724](https://github.com/triggerdotdev/trigger.dev/pull/2724))
- The `--force-local-build` flag is now renamed to just `--local-build`
([#2702](https://github.com/triggerdotdev/trigger.dev/pull/2702))
- fix(cli): header will always print the correct profile
([#2728](https://github.com/triggerdotdev/trigger.dev/pull/2728))
- feat: add ability to set custom resource properties through
trigger.config.ts or via the OTEL_RESOURCE_ATTRIBUTES env var
([#2704](https://github.com/triggerdotdev/trigger.dev/pull/2704))
- feat(cli): implements content-addressable store for the dev CLI build
outputs, reducing disk usage
([#2725](https://github.com/triggerdotdev/trigger.dev/pull/2725))
- Added support for native build server builds in the deploy command
(`--native-build-server`)
([#2702](https://github.com/triggerdotdev/trigger.dev/pull/2702))
- Updated dependencies:
- `@trigger.dev/build@4.2.0`
- `@trigger.dev/core@4.2.0`
- `@trigger.dev/schema-to-json@4.2.0`
## @trigger.dev/build@4.2.0
### Patch Changes
- syncVercelEnvVars to skip API and read env vars directly from
env.process for Vercel build environments. New syncNeonEnvVars build
extension for syncing environment variablesfrom Neon database projects
to Trigger.dev. The extension automatically detects branches and builds
appropriate PostgreSQL connection strings for non-production, non-dev
environments (staging, preview).
([#2729](https://github.com/triggerdotdev/trigger.dev/pull/2729))
- Updated dependencies:
- `@trigger.dev/core@4.2.0`
## @trigger.dev/core@4.2.0
### Patch Changes
- fix: prevent ERR_IPC_CHANNEL_CLOSED errors from causing an unhandled
exception on TaskRunProcess
([#2743](https://github.com/triggerdotdev/trigger.dev/pull/2743))
- Added support for native build server builds in the deploy command
(`--native-build-server`)
([#2702](https://github.com/triggerdotdev/trigger.dev/pull/2702))
## @trigger.dev/python@4.2.0
### Patch Changes
- Updated dependencies:
- `@trigger.dev/build@4.2.0`
- `@trigger.dev/sdk@4.2.0`
- `@trigger.dev/core@4.2.0`
## @trigger.dev/react-hooks@4.2.0
### Patch Changes
- fix: prevent infinite useEffect when passing an array of tags to
useRealtimeRunsWithTag
([#2705](https://github.com/triggerdotdev/trigger.dev/pull/2705))
- Updated dependencies:
- `@trigger.dev/core@4.2.0`
## @trigger.dev/redis-worker@4.2.0
### Patch Changes
- Updated dependencies:
- `@trigger.dev/core@4.2.0`
## @trigger.dev/rsc@4.2.0
### Patch Changes
- Updated dependencies:
- `@trigger.dev/core@4.2.0`
## @trigger.dev/schema-to-json@4.2.0
### Patch Changes
- Updated dependencies:
- `@trigger.dev/core@4.2.0`
## @trigger.dev/sdk@4.2.0
### Patch Changes
- fix(sdk): Re-export schemaTask types to prevent the TypeScript error
TS2742: The inferred type of 'task' cannot be named without a reference
to '@trigger.dev/core/v3'. This is likely not portable.
([#2735](https://github.com/triggerdotdev/trigger.dev/pull/2735))
- feat: add ability to set custom resource properties through
trigger.config.ts or via the OTEL_RESOURCE_ATTRIBUTES env var
([#2704](https://github.com/triggerdotdev/trigger.dev/pull/2704))
- Updated dependencies:
- `@trigger.dev/core@4.2.0`
---------
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
**Background**
Currently the changeset PR creation and the publishing is handled by the
same workflow. This is not ideal:
- The build steps are executed on every run of the pipeline, even though
they're only needed for the publish case.
- The PR creation workflow does not need permissions to publish to npm,
only the release path needs them.
- Adding an approval step is painful as we'd need to also approve each
changeset PR creation workflow run.
**Changes in this PR**
- Separated the changeset PR creation into its own workflow and minimum
permission set.
- Added a GH environment with an approval step for the package
publishing workflow (also for prereleases).
- New publish workflow runs will not cancel in-progress runs; helps
avoid partial failures in publishing.
These changes also enable hardening the npm OIDC setup by tying it to a
GH environment that requires approval.
Vercel's NeonDB integration renders database connection environment
variables at runtime, which means Trigger.dev cannot directly sync these
values during the build process. This change adds support for fetching
branch-specific NeonDB connection strings via the Neon API.
feat(build): Add syncNeonEnvVars extension and improve Vercel env var
syncing
Add a new `syncNeonEnvVars` build extension for syncing environment
variables
from Neon database projects to Trigger.dev. The extension automatically
detects
branches and builds appropriate PostgreSQL connection strings for
non-production
environments (staging, dev, preview).
Features of `syncNeonEnvVars`:
- Fetches branch-specific database credentials from Neon API
- Generates all standard Postgres connection strings (DATABASE_URL,
POSTGRES_URL,
POSTGRES_PRISMA_URL, etc.) with both pooled and unpooled variants
- Supports custom database name, role name, and env var prefix options
- Skips automatically in Vercel environments (Neon's Vercel integration
handles this)
- Skips for production environments (designed for preview/staging/dev
branches)
Improvements to `syncVercelEnvVars`:
- When running in a Vercel build environment (detected via VERCEL env
var),
values are now read from process.env instead of the Vercel API response
- This ensures the build uses the actual runtime values Vercel provides
- Removed embedded Neon-specific logic (now handled by separate
extension)
- Simplified and cleaned up the extension code
Documentation updates for both extensions with usage examples and
configuration
options.
Closes#2714
## ✅ Checklist
- [x] I have followed every step in the [contributing
guide](https://github.com/triggerdotdev/trigger.dev/blob/main/CONTRIBUTING.md)
- [x] The PR title follows the convention.
- [x] I ran and tested the code works
---
## Testing
Set up Vercel + Trigger.dev envs, used Vercel's chat-bot-ai template.
When viewing runs filtered to a single task, the "Create a test run" and
"Run a test" buttons now navigate directly to the task-specific test
page instead of the generic test page.
This improves UX by pre-populating the test form with the filtered task,
saving users from having to manually select it again.
## ✅ Checklist
- [x] I have followed every step in the [contributing
guide](https://github.com/triggerdotdev/trigger.dev/blob/main/CONTRIBUTING.md)
- [x] The PR title follows the convention.
- [x] I ran and tested the code works
## Screenshots
<img width="1706" height="1392" alt="image"
src="https://github.com/user-attachments/assets/d8b5d445-73b5-426c-83a4-90ac2a95b955"
/>
Add validation for logical replication publication configuration. Helps
diagnose an issue where runs are no longer replicated to clickhouse
because of a configuration issue with the replication publication.
## Problem
The `LogicalReplicationClient` only checked if a publication existed,
not if it was correctly configured. This caused a silent failure where:
- Replication would start successfully
- Transaction boundaries (begin/commit) were received
- **But no actual data changes were replicated**
This happened when a publication existed but:
1. Had no tables associated with it
2. Was missing required actions (e.g., `delete`)
## Solution
Added `#validatePublicationConfiguration()` method that validates:
- ✅ Publication includes the expected table
- ✅ Publication has all required actions configured
When validation fails, error messages include the exact SQL command to
fix the issue:
**Missing table:**
```
Publication 'task_runs_to_clickhouse_v1_publication' exists but has NO TABLES configured.
Expected table: "public.TaskRun".
Run: ALTER PUBLICATION task_runs_to_clickhouse_v1_publication ADD TABLE "TaskRun";
```
**Missing actions:**
```
Publication 'task_runs_to_clickhouse_v1_publication' is missing required actions.
Expected: [insert, update, delete], Current: [insert, update], Missing: [delete].
Run: ALTER PUBLICATION task_runs_to_clickhouse_v1_publication SET (publish = 'insert, update, delete');
```
This prevents silent data loss and makes debugging configuration issues
much easier.
Adds a manual trigger to the `release.yml` workflow for publishing
prerelease versions. Needs to be in the same workflow file due to a NPM
limitation on how OIDC claims are checked.
Currently there is a validation step on the ref for the prerelease: it
must be merged to the main branch. We can revisit this in the future in
case we find it too limiting.
To avoid doing a migration for the ECR repo cache settings, we lazily do
it on the next deployment for that project. Failures to update the repo
settings are just logged and will not cause the deployment to fail.
Fixes this type of error when exporting a `schemaTask` in a monorepo:
```
error TS2742: The inferred type of 'helloWorldSchema' cannot be named without a reference to '@trigger.dev/core/v3'. This is likely not portable.
```
This PR adds support for CLI deployments using the native build server.
**Background**
The deployment command currently does the following:
- bundles the code
- submits the build context to our external build provider and waits for
the build
- triggers deployment state transitions using the platform API
Upstream build provider outages cause issue with deployments,
potentially blocking deployments entirely. We recently introduced the
`--force-local-build` flag as a fallback to enable deployment without a
dependency on the upstream build provider, though it requires users to
have docker in their systems. This PR continues that work by providing a
remote build path which uses our own build server and does not rely on
the external provider.
**Changes in this PR**
Introduced the new `--native-build-server` flag, which does the
following:
- scans all files relevant for the Trigger deployment and evaluates
ignore rules
- packages it up in an archive and uploads it as a deployment artifact
- queues the deployment and triggers the build
- streams logs from the build server
This no longer relies on external build services. Also deployment state
transitions happen on the server-side, giving us more flexibility to
evolve the flow and schemas of related deployment API endpoints. In
general it gives us better control of the whole build and deployment
process. This path will eventually become the default.
The `--detach` flag is also new, allowing to trigger deployments without
waiting for the result.
The deployment artifacts are uploaded via pre-signed URLs to avoid
unnecessary load on the platform. The new `/artifacts` endpoint
generates the pre-signed URLs; size limits are enforced on s3. This
endpoint is deliberately generic, we could extend it in the future to
upload other artifacts client-side in a similar way, e.g., large payload
packets.
This is a step which we currently need to do manually and it's rather
painful. The lockfile update is necessary due to cross references in our
packages.
Added it as a separate job instead of a step to start from fresh
workspace, as the state that the `changeset` step leaves the workdir is
not explicitly clear to the reader.
Add a new `syncNeonEnvVars` build extension for syncing environment variables
from Neon database projects to Trigger.dev. The extension automatically detects
branches and builds appropriate PostgreSQL connection strings for non-production
environments (staging, dev, preview).
Features of `syncNeonEnvVars`:
- Fetches branch-specific database credentials from Neon API
- Generates all standard Postgres connection strings (DATABASE_URL, POSTGRES_URL,
POSTGRES_PRISMA_URL, etc.) with both pooled and unpooled variants
- Supports custom database name, role name, and env var prefix options
- Skips automatically in Vercel environments (Neon's Vercel integration handles this)
- Skips for production environments (designed for preview/staging/dev branches)
Improvements to `syncVercelEnvVars`:
- When running in a Vercel build environment (detected via VERCEL env var),
values are now read from process.env instead of the Vercel API response
- This ensures the build uses the actual runtime values Vercel provides
- Removed embedded Neon-specific logic (now handled by separate extension)
- Simplified and cleaned up the extension code
Documentation updates for both extensions with usage examples and configuration
options.
* Add the release version to changeset PRs
* Add missing id-token permission, needed for oidc
* Remove a couple of unnecesary steps
* Reference the `changeset-release/main` branch explicitly
Vercel's NeonDB integration renders database connection environment
variables at runtime, which means Trigger.dev cannot directly sync
these values during the build process. This change adds support for
fetching branch-specific NeonDB connection strings via the Neon API.
Changes:
- Discover NEON_PROJECT_ID from incoming Vercel environment variables
- Call NeonDB API to search for branches matching the git branch name
- Filter branches to find exact matches with Vercel environment prefix
(e.g., "preview/branch-name") to avoid false positives from partial
string matches
- Retrieve branch endpoints and select the write endpoint (or first
available)
- Build connection strings (DATABASE_URL, POSTGRES_URL, etc.) using
the branch endpoint host while preserving user/password credentials
Safety measures for non-production environments:
- Filter out all Neon-related env vars (DATABASE_URL, PGHOST, etc.)
before calling the Neon API to prevent accidental use of production
database credentials
- Only add branch-specific database env vars if a matching Neon branch
is found and the API call succeeds
- If neonDbAccessToken is not provided or the API fails, non-production
environments will not receive any database connection env vars
Usage:
Users must provide a NEON_ACCESS_TOKEN (via options or env var) to
enable automatic branch resolution for preview deployments. Production
environments continue to use Vercel's standard env var sync without
modification.
schedule spans can sometimes show as generic spans when using the
task_events_v2 table because of the inserted_at filter. Increasing the
buffer for the start time does the trick and doesn’t cause any perf
Issues (and is in general just more robust)
When using custom OTLP exporters via `telemetry.exporters` and
This occurred when tasks were triggered **without** a parent trace
context (e.g., via API or dashboard). In this scenario: - Spans were
correctly rewritten to use the generated `externalTraceId` - Logs kept
their original internal trace ID due to a bug in the early return logic
### Root Cause
In `ExternalLogRecordExporterWrapper.transformLogRecord()`, the early
return condition incorrectly included `!this.externalTraceContext`:
```typescript
if (!logRecord.spanContext || !this.externalTraceId ||
!this.externalTraceContext) { return logRecord; // Bug: Returns early
when externalTraceContext is undefined }
// This fallback logic was never reached:
const externalTraceId = this.externalTraceContext
? this.externalTraceContext.traceId
: this.externalTraceId;
```
### Fix
1. **Reordered logic in `transformLogRecord()`**: Move the
1. `externalTraceId` calculation before the early return, and check the
1. culated value instead of `this.externalTraceContext`:
```typescript
const externalTraceId = this.externalTraceContext
? this.externalTraceContext.traceId
: this.externalTraceId;
if (!logRecord.spanContext || !externalTraceId) {
return logRecord;
}
```
2. **Clarified `_isExternallySampled` logic**: Updated both
2. `ExternalSpanExporterWrapper` and `ExternalLogRecordExporterWrapper`
2. explicitly handle the case where there's no external trace context
2. a generated `externalTraceId` exists:
```typescript
this._isExternallySampled = externalTraceContext
? isTraceFlagSampled(externalTraceContext.traceFlags)
: !!externalTraceId;
```
### Impact
Logs and spans from the same task run will now have matching trace IDs
when exported to external observability tools, enabling proper trace correlation regardless of whether the task was triggered with or without a parent trace context.
`telemetry.logExporters` in `trigger.config.ts`, logs and spans were
exported with **different trace IDs**, breaking trace correlation in
external observability tools like Datadog.
* stop deleting the first dev version files on the first change, prevents system failures
* prevent dev runs getting stuck in dequeued status by deleting workers
* add changeset
This PR fixes the `x-trigger-branch` header support for targeting specific preview branches when managing environment variables. The header was documented but not actually being extracted or used in the environment variable API routes. Additionally, the query logic in `authenticatedEnvironmentForAuthentication` was fundamentally broken—it searched for environments with both `slug: "preview"` (parent environment property) AND a specific `branchName` (child environment property), which no environment could satisfy simultaneously. The fix extracts the branch name using `branchNameFromRequest()` and correctly queries for child branch environments using `type: "PREVIEW"` and the specific `branchName`. This ensures that environment variable operations (create, update, get, list) properly target individual preview branches instead of affecting all preview environments.
* Don't use the organization max concurrency anymore
* Early draft of the concurrency page
* WIP adding a new stepper input component
* Move stepper to be alphabetical
* When max value is reached, disabled the + button
* Show placeholder if you delete all numbers
* Make all the html input values available to the component
* Adds size variants
* Move stepper into its own component
* Work on showing the extra concurrency
* The purchase form styling and functionality (minus actually purchasing)
* New style for outline input fields
* Concurrency purchasing working
* Purchasing concurrency and quota emails working
* Improvements to the modal
* Show cost breakdown in the modal
* Fix for allocated concurrency including DEV
* Improved types
* Allocating concurrency is working
* Live updates total env concurrency
* Implemented reset
* Fix for concurrency allocation editing across multiple projects
* Tabular numbers
* Added an error from allocating concurrency
* Fixes for allocating concurrency where it didn't calculate correctly
* "Increase limit" link to concurrency page
* Indent environments
* Added Preview limit when updating concurrency for an org
* Show error when changing plan fails
* Added maximumProjectCount column to Org
* Limit project count and display a rich error toast (with title and button now)
* Added title and button to toasts. Use it for new project error
* @trigger.dev/platform 1.0.20
* Allow submitting zero concurrency so you can downgrade back to nothing
* Use the server as the truth for omitted environments
* Updated the pricing panels
---------
Co-authored-by: James Ritchie <james@trigger.dev>
* prisma extension fixes WIP
* More prisma stuff
* more prisma stuff
* remove changelog
* upgrade github workflows to use node 20.19 because installing prisma@7 breaks with lower versions
* Don't use generate for the prisma reference projects
* make sure it works if no mode is passed in
* Update aws sdk ecr client to the latest version
* Exlude the cache tag from the immutability enforcement
* Attach a policy to ECR repos to expire untagged images
* Fix filterType
* Fix for the MCP tool that gets logs for debugging runs
This was broken when we changed the data on the backend that returns
log/span data from runs. We changed the data structured and the internal
API that the MCP client uses was failing to parse with the Zod schema
* add changeset
* Revert "add changeset"
This reverts commit 86eca836d5907fa0d0f8ac595d4d5ebade140514.
---------
Co-authored-by: nicktrn <55853254+nicktrn@users.noreply.github.com>
* Use read-only project-scoped s2 tokens for streaming deployment logs
* Add http2 to remix polyfills
Needed for using s2 client-side.
* Stream build-server logs in the deployment details page
* Disable 12-hour format in the DateTime component
* Enable collapsing the logs panel
* Auto-collapse logs for succesful/timedout/queued deployments
* Make S2 env vars optional
* Show the logs section only for gh-triggered deployments
* Cache s2 access tokens in redis
* Reset streaming state
* Expose 12h format as a param for the Datetime components
* docs: deployments with local builds
Adds a section to the deployment docs page about the option to deploy using a locally built image.
* Update installation link
* Add an API endpoint to query remote build provider status
* Show local build hint for failed deployments when Depot is down
* Show the local build flag in the help output
* Add changeset
* Fix import
* Fix docs link
* chore(runner): move max duration logic into parent process
* chore(rsc): remove type-marker package.json
* add changeset
* chore(core): remove irrelevant test after our changes
* chore(core): clarify we don't care about the timeout promise
* fix(webapp): display correct concurrency override base value
* fix(webapp): persist concurrency overrides on deploy
* fix(webapp): use correct override base value type
* fix(webapp): override input is bounded by env concurrency
* Enable skipping image push during deployment finalization step
* Add endpoint to generate registry credentials for a deployment
* Add a --force-local-build flag to the deployment command to skip remote build
* Do not show the new flag in the help output
* Add changeset
* Remove registry login logs from onLog, not useful
* Rename var
* Update platform package to the latest version
* Limit local dev concurrency using the dev environment concurrency limit
Previously it was limited to max of 25, no matter the environment limit
* Have global dev limit
The Postgres query to get the latest run ids for the test page was very
slow when there were a lot of runs and/or versions.
This now uses the standard runs list we use everywhere else.
* feat(queues): add ability to override concurrency limit via API and dashboard
* Updates the modal layout and tweaks copy
* Improves the dropdown menu item
* Popover supports both Button and LinkButton
* Right align the columns and fix the dropdown menu item styles
* Organize imports,
* Fix spinner icon in dropdown menu
* Remove unused props
* Adds a tooltip to the Concurrency override badge
* Fixes console error with popover menu
* typo
* Fixes incorrect className
* Minimal buttons to view runs
---------
Co-authored-by: Eric Allam <eallam@icloud.com>
* WIP using ClickHouse for the tags filter list
* WIP on tags listing
* Webapp: exclude test files when typechecking
* Tags filtering working with CH
* Remove unused import
* The AI filter should only look at the last past 30d of tags
* Do the text query in ClickHouse
* Deal with encoded characters better
* More encoding fixes
* Fix for wrong items being checked
* Put applied tags back
* Add the env.id to the dependencies array
* chore(billing): improve logs to distinguish between failure modes
* fix(engine): default to paid placement on billing errors
* chore(engine): set plan type according to paying field when missing
- Move S3 credentials from secrets.objectStore to s3.auth
- Update external PostgreSQL config to use databaseUrl/directUrl approach
- Add existingSecret support for PostgreSQL with secretKeys
- Add TLS configuration for external Redis
- Add existingSecret support for Redis, ClickHouse and S3
- Add complete external S3 configuration example
- Improve secure credential management documentation
These changes align the production example file with the current values.yaml
structure, making it easier to configure external services with better
support for secret management.
* Allow shortcuts hook to work if undefined
* Conditionally show shortcut button if only 1 result
* LinkButton can accept conditionally shown shortcuts
* docs: deploying using the github integration
* Add hint in the gh actions docs page
* Remove extra space
* Add a couple of hints to the build config fields
* feat(build-server): add option to specify pre-build command
Adds an option to specify a pre-build command in the build settings. Can
be useful for projects that need a step before the build, e.g., to
generate a prisma client.
Also, remove the install directory in favor of simplicity. Both
pre-build and install commands are run from the root of the repo. Users
that need to run the commands in a different dir can just prepend to the
command, e.g., `cd apps/web && pnpm run primsa:migrate`
* Fix spelling
* Show hint if preview branches are disabled in the project
* Enable preview deployments only if the preview environemtn is enabled
* Fix prisma reference
* fix(otel): prevent unpaired unicode surrogate pairs from causing insert errors
* only check parts of the string that are not going to get truncated
remove unnecessary taks
* fix(charts): use bitnamilegacy repo for MinIO Object Browser
The Bitnami `minio-object-browser` image has been removed from Docker
Hub. This patch updates the Trigger Helm chart to reference the
`bitnamilegacy/minio-object-browser` repository under the
`s3.console.image` section, ensuring that the legacy MinIO browser pod
can still be deployed without image pull errors.
All other MinIO components continue using the `bitnamilegacy/minio`
image for consistency across Bitnami Legacy dependencies.
* chore(helm): bump Helm chart version to 4.0.4
The Depot build init with `depot.build.v1.BuildService.createBuild` fails surprisingly often due to transient errors, causing the whole deployment to fail. This PR adds a simple retry mechanism with backoff using p-retry. This should improve the failure rate.
* chore(docker): use bitnami legacy repo
* chore(helm): use bitnami legacy repo
* Make Helm webapp chart images configurable
Adds configurability for init and token syncer container images through
new values in the Helm chart configuration
* chore(helm): refactor utility image config
* chore(helm): bump chart version to 4.0.3
---------
Co-authored-by: LeoKaynan <leokaynan@hotmail.com>
* feat(webapp): rate limit magic-link login attempts
Adds a simple rate limiter to the login with magic link flow. Similar implementation to the MFA rate limits.
* Fix error message
* Add an env var feature flags for login rate limiting
* Use BoolEnv instead of `0`/`1`
* Parse xff properly
* feat(server): add two admin endpoints for queue and environment concurrency debugging and repairing
feat(run-engine): ability to repair runs in QUEUED, SUSPENDED, and FINISHED execution status
* Handle FINISHED snapshot in the repair
* fix: use higher entropy invite tokens
We currently use CUIDs for invite tokens, which are generated using
a pattern and are not cryptographically secure. This PR switches to
a higher entropy string generated with `nanoid`.
* Dedupe the invite emails in the application
* fix: org scoping in the select plan flow
Adds proper org scoping in the loader and action in the plans page.
* Fix billing alerts scope
* Fix org usage page scope
* Fix token revoking flow scope check
* Throw error for failed PAT revokes instead of silent failure
* fix(webapp): toast message issue after gh app installation
Fixes an issue with displaying toasts messages in the project settings
page. The github callback cookie was interfering with the flash cookie used
for toast messages.
* Do not set a tracking branch in the staging env by default
* Add canceledAt to the deployment db schema
* Expose an api endpoint to cancel deployments
* Show the canceled status description in the dashboard
* Enable canceling deployments from the dashboard
* Show cancelation reason in the deployment details
* Make verifyProjectMembership a function for consistency
* Apply some good 🐰 suggestions
* Add installing status to the deployment db schema
* Replace the deployments /start endpoint with /progress
* Show the installing status in the dashboard
* Add installing status to the api schema and cli
* Add changeset
Depot builds have short-lived tokens and their TTL is not exposed in the SDK. As queued deployments can stay in the queue for an arbitrary amount of time, deferring the remote build creation helps avoid expired Depot token issues.
* Cancel run events which then propogate cancellation status to span ancestors
* WIP
* convert closing cached run spans to new system
* converted expired complete span event to new method
* move v3 over to new methods
* Convert getDetailedTraceSummary to use the new ancestor override stuff
* remove debug logs
* Don't return UNSPECIFIED task events in getRunEvents
* fix the call site for cancelling run event in v3
* Add changeset
* remove methods
This PR adapts the deployment initialization endpoint to handle build server deployments with older CLI versions gracefully.
When we introduced automatic deployments via the build server, we slightly changed the deployment flow
mainly in the initialization and starting step: now deployments are first initialized in the `PENDING` status
and updated to `BUILDING` once the build server dequeues the build job.
Newer versions of the `deploy` command in the CLI will automatically attach to the existing deployment
and continue with the build process. For older versions, we can't change the command's client-side behavior,
so we need to handle this case here in the initialization endpoint. As we control the env variables which
the git meta is extracted from in the build server, we can use those to pass the existing deployment ID
to this endpoint. This doesn't affect the git meta on the deployment as it is set prior to this step using the
/start endpoint. It's a rather hacky solution, but it will do for now as it enables us to avoid degrading the
build server experience for users with older CLI versions. We'll eventually be able to remove this workaround
once we stop supporting 3.x CLI versions.
* Fix `current` badge inconsistency in the deployment details page
* Add custom hook for auto revalidation based on an interval and/or focus change
* Use the autoRevalidate hook for live reloading of the deployments page
* Extract autoReloadPollIntervalMs to an env var
* Replace the sse-based autoreload in bulk actions and queues page with the simpler autoRevalidate hook
* Enable setting the initial status on deployment creation
* Expose endpoint to start deployments
* Extend build timeout on deployment start
* Use separate timeout value for queued deployments
* Add startedAt to the deployment schema
* Show the new startedAt instead of createdAt in the dashboard
* Show github user tag also in the deployment details page
* Show `pending` deployment status as `queued` in the dashboard
* Apply some good 🐰 suggestions
* Add missing return
* Improve styling of onboarding pricing plans
* type only import
* Improve the onboarding plan page so it scrolls on smaller screens
* Adds additional pricing for bolt-ons for the Pro plan
* fix text wrapping issue
* add bg color back in
* Change the color to indigo
* Pro tier pricing information now matches the marketing site
* Update the button styles to secondary
* WIP adding separate links to Parent and Root runs
* TextLink now supports optional shortcuts
* Adds shortcut keys to the root and parent links + the shortcut help panel
* Adds new icons for root and parent
* root friendlyId works
* Updates icons for jump to root and parent
* Copy tweak
* Improve how the Free tier shows no preview branches
* Improve the wording in the tooltip
* Align the x icon better
* Show price for additional preview branches
* Change the shortcut key
* Fixes button alignment
* Adds nested dependencies task hello-world
* Fixes typo “Cancelled”
* Removes taskIdentifier, not needed
* Removes unused taskIdentifier
This PR enables setting project build settings in the settings page:
root directory, install command and trigger config file path.
For most cases there should be no need to set these explicitly.
* Mo-Stashed changes
* fix(core): prettyPrintingPacket will now do a structuredClone on non-circular references instead of outputting [Circular]
This also fixes an issue with replaying of runs that include
non-circular references
* Add external build data and image platform to the get deployment endpoint
* If provided, attach to an existing deployment in the deploy command
* Check status for existing deployments
* Add changeset
* feat(engine): Improve execution stalls troubleshooting, align dev and prod behavior, adding heartbeats.yield utility
* A few improvements via the 🐇 review
* Allow treating EXECUTION stalls as OOM errors, improve the error message, add more information to the docs, improve resource monitor and add it to the docs
* Add changeset
* Add a CLI command to list and view env vars
* Add changeset
* Restrict pemissions on env files created with `env pull`
* Escape env vars when exporting to file
* Switch changeset to patch
* Fix settigns page delete project width issue
* Apply a couple of touch-ups to the project settings page
* Add UI flow to connect gh repos
* Enabling adding another gh account in the ui
* Enable connecting a repo to a project
* Enable updating git settings
* Enable disconnecting gh repos from a project
* Remove prisma migration drifts
* Hide git settings when github app is disabled
* Fix migration order
* Avoid using `location` to avoid SSR issues
* Make branch tracking optional
* Disable save buttons when there are no field changes
* Disable delete project button unless the input matches the project slug
* Show connected repo connectedAt date
* Check that tracking branch exists when updating git settings
* Show tracking branch hint in the deployments page
* Fix positioning issue of the pagination pane in the deployments page
* Use mono font for branch names
* Add link to git settings
* Show tracking branch hint for the preview env too
* Add a confirmation prompt on repo disconnect
* Add link to configure repo access in gh
* Add rel prop to github links
* Automatically open repo connection modal after app installation
* Apply some fixes suggested by mr rabbit
* Fix flash cookie issue
* Extract project settings actions into a service
* Extract project settings loader into a presenter service
* Introduce neverthrow for error handling
* Try out neverthrow for error handling in the project setting flows
* Move env gh branch resolution to the presenter service
* feat(webapp): add support for running web services (api, engine, webapp) in cluster mode for better perf
* cleaned up signal handling and resolved some valid 🐇 issues
* Add schemas for gh app installations
* Implement gh app installation flow
* Make the gh app configs optional
* Add additional org check on gh app installation callback
* Save account handle and repo default branch on install
* Do repo hard deletes in favor of simplicity
* Disable github app by default
* Fix gh env schema union issue
* Use octokit's iterator for paginating repos
* Parse gh app install callback with a discriminated union
* Remove duplicate env vars
* Use bigint for github integer IDs
* Sanitize redirect paths in the gh installation and auth flow
* Regenerate migration after rebase on main to fix ordering
* Handle gh install updates separately from new installs
* feat(supervisor): dynamic queue consumer pool
* add changeset
* fix: correctly handle zero median and even samples
* feat(supervisor): consumer pool metrics
* fix tests
* more tests and fixes
* decrease default scaling cooldowns
* don't treat initial pool size as scale up
* handle scale down when queue length drops to zero
* remove changeset, supervisor changes only
* add damping factor env var
* feat(webapp): upgrade otel packages and add more metrics
* add env var to disable additional detectors
* expose more prisma metrics
* chore(webapp): drop node 16 support
* Added MCP to the intro
* Added human-in-the-loop and new build extensions
* Waitpoint notes
* Moved openai guardrails example to Python
* Added a connection limit note
* Added Supabase + Prisma note to supabase auth
* Updates based on what the rabbit said
* Moved the supavisor section to the prismaExtension docs
* docs: update triggering.mdx fixing wrong argument numeration for options
Update the docs triggering.mdx: text indicating options argument position inside `tasks.trigger` function.
Options object is a third argument instead of stated second argument.
* update triggering.mdx to change options argument position in batchTrigger
* feat(mcp): add wait_for_run_to_complete tool so agents don't spam the get_run_details call after triggering
This also fixes the search docs MCP tool
* Install mcp using the latest tag, not the specific version
* more eager dequeuing, queue cooloff periods, return workerQueueLength when dequeueing
* Cache worker group authentication and remove old self-hosted worker code (only managed is currently supported)
* add additional spans during dequeue
* Add env vars and additional spans
* Initial work on upgrading to 6.14.0
Set the output to node_modules still to make it easier
* Use ./generated Prisma folder, update types to fix issues
* Docker compose restart Clickhouse
* Prisma instrumentation update
* Docker
* Removed database dockerignore file, add generated prisma client to the top-level one
* Delete v3-catalog package.json
* Resolved pnpm lock file
* Log errors for very slow queries
* Create schema and migration for organization access tokens
* Add helpers for creating and authenticating OATs
* Adapt the auth service to also accept OATs
* Accept OATs in the whoami v2 endpoint
* Enable deployments with the CLI using OATs
* Avoid reading env variables directly in the token utils
* Remove duplicate cli token utils
* Validate ENCRYPTION_KEY length when parsing env vars
* Make token utils a server-only module
* Disallow revoking already revoked OATs
* Simplify generics in authenticateRequest
* Use 32 bytes mock encryption key in the test setup
* Update dummy encryption key values in tests and templates
* Add a column in the OATs table to differentiate between user and system generated
* Simplify args for v3ProjectPath
Co-authored-by: Matt Aitken <matt@mattaitken.com>
* Add index on org id and createdAt
* Avoid storing the encrypted oat token and its obfuscated version in the DB at all
It is a safer approach. Also we do not need to ever read the decrypted token value after creation.
* Fix prisma update condition
* Add token type to the OAT table index
* Accept OATs in the mcp auth flow
* Simplify env auth flow around the /projects endpoints
---------
Co-authored-by: Matt Aitken <matt@mattaitken.com>
* Add MCP Server documentation to documentation site
This change introduces comprehensive documentation for the Trigger.dev MCP Server, covering its installation, usage, and available tools. The addition is necessary to provide users with detailed guidance on leveraging the MCP Server for managing Trigger.dev projects, tasks, and deployment processes.
- Updated `docs.json` to include a new section for the MCP Server documentation, improving navigation and accessibility.
- Added `mcp-introduction.mdx` as an introductory guide to the MCP Server, detailing its purpose and installation steps.
- Included `mcp-tools.mdx`, a document outlining each tool available in the MCP Server, complete with descriptions and example usages.
- The new content is aimed at improving user understanding and efficiency in using Trigger.dev's MCP Server capabilities.
* Refactor MCP installation documentation
The changes were necessary to streamline the MCP installation guidance and align our documentation with the latest client support capabilities. Removing the manual installation section and redundant information helps reduce confusion for users.
- Updated installation instructions to lead with auto-configuration for supported clients.
- Cleaned up the options and examples sections to enhance clarity.
- Reformatted client configuration details for better readability.
- Removed outdated links in the "Next Steps" section, focusing on more direct resources such as available MCP tools.
* mcp docs edits
* Add debugMode to get_run_details
* add cancel_run
* Add list deployments tools
* Add list preview branches tool
* Docs: adds agent rules page
* Adds redirect for the CLI path to work
* Adds subagent install image
---------
Co-authored-by: James Ritchie <james@trigger.dev>
* fix(v3): prevent saturated queues from dominating dequeue attempts by adding a cooloff period of successive failed dequeues
* Add env vars and add a marqs shutdown on SIGTERM/INT
* fix: span events are optional, as logs don't have them
* remove CI: true env var to prevent massive tool responses
* fix github resolver relative path
* prevent the dev command from asking for user input when not interactive
* Remove lodash.get because it's deprecated
* span output can be object other than an object, like a string
* prevent large traces from causing get_run_details failures
* Decreased max log lines, made it a tool input property
* Some install mcp tweaks
* Remove variables from dequeue log message
* Continue snapshot throw json
* Waiting for deploy error removed
* Realtime ECONNRESET is expected
* Redis worker logErrors changes, removed ids
* Preview branch without a branch shouldn't log an error, it's a user provided issue
* "Task run is not in a cancellable state" isn't an error, it's expected
* "CreateCheckpointService: Child run already resumed" is expected
* "CreateCheckpointService: Batch already resumed" is expected
* "Failed to insert events, will attempt bisection" changed to info, we have errors for complete failures
* Ignore "PrismaClient error"
* Don't log Redis worker DLQ errors if we're ignoring
* "Failed to parse machine config" is fine, sometimes a config is null or undefined
* "Failed to parse machine config" for v3
* MetadataTooLargeError shouldn't log an error
* Don't log an error when the snapshot shouldn't be created, it's normal for this to happen
* Slack alerts, skip `account_inactive` errors
* v3 finalize run with no locked isn't an error
* Another false error
* Finalize run CRASHED runs were logging errors
* All slack alert errors are warnings except invalid blocks
* Adds new features table to top of v4 upgrade guide
* Adds wait idempotency to wait-until, wait-for, and wait-for-token pages
* Adds new priority docs page and updates the v4 upgrade guide
* Adds new task lifecycle hooks
* Removes the message about requiring tasks to be exported
* Adds new global lifecycle hooks section
* Moves sections from upgrade guide into the table
* Adds hidden task page
* Improves the global lifecycle hooks section
* Updates middleware and locals section
* Adds new useWaitToken page to the react hooks section
* Adds a new ai.tool section
* Moves Docker (legacy) page into self-hosting section
* Removes known issues from v4 upgrade guide
* Replace “toolTask” with “ai.tool” in the Streams page example
* Renames guide to “Migrating from v3” and adds redirect
* Remove references to v4
* Removes changelog from migration guide
* The installation guide now references `@latest update`
* Changes all references from `/sdk/v3` to `/sdk`
* Updates @v4-beta to @latest
* Fixed broken link
* Fixes broken link
* Adds an upgrade to v4 using AI section
* Fixes 2 broken links
* Adds an entry for targetting preview branches
* Updates the run statuses
* Adds boolean helpers section to the runs and realtime pages
* Updates the concurrency page
* Updates the test page to include the new options
* Adds SDK and curl options for the preview branch targeting
* Updates new bulk actions page
* Remove the releasing concurrency section
* Got rid of some more @v4-beta mentions
* Improved rate limit docs
* Improved migrating docs
* Removed commented sections of the docs
* useWaitToken hook
* Fixed the description
* Fix for missing test image
---------
Co-authored-by: Matt Aitken <matt@mattaitken.com>
Co-authored-by: Dan <8297864+D-K-P@users.noreply.github.com>
* fix: external traces now respect parent sampling, and prevent broken traces when there is no external trace context
* Add changeset
* improve trace flag handling and better internal host checking
* the traceFlags are now being properly passed through as a number
* add tier scheduling support to supervisor
* add billing info to dequeued message w/o cache
* add cache with best effort invalidation
* fix invalidate circular dep
* add changeset
* use new plan type on runs as fallback during dequeue
* tidy up
* be more explicit with plan type fallback
* remove additional billing check from hot path
* switch to placement tags
* update changeset
* update platform package
* start using new entitlement response
* ensure skipChecks optimization validates at batch level
* add optional items to add to queue manager limits
* make the bool env helper only accept boolean defaults
* remove redundant private field
* update placement tag helper to prevent unsupported tags
Small change that shows the github user in the deployments page in cases
where it was triggered by the github app integration.
These deployments are not triggered using a PAT, so we cannot associate
them with a Trigger user ID.
This PR adapts the deployment command to also evaluate build server env
variables when creating the git meta. The build server will initially
only support deployments triggered by the github app, but we might add
other git providers in the future.
Sticking to the GH actions naming convention for the env variables set
by the build server in favor of consistency.
* fix: sentry memory leak by disabling includeLocalVariables
* Enhance heap snapshot consistency and labeling
To facilitate more accurate and consistent heap memory snapshots, a new function forceConsistentGC was added before taking a snapshot. This ensures the garbage collector (GC) runs multiple times, stabilizing the heap state for more reliable analysis. This is particularly helpful when debugging memory-related issues.
Updates to the memory-leak-detector script now allow labeling of snapshot runs using the --label flag. This helps in distinguishing different runs for easier tracking and comparison of memory usage across test sessions. Additionally, the --expose-gc flag ensures that the GC can be manually triggered during test runs, leading to more consistent memory states and potentially uncovering hidden memory leaks.
* Refactor forceConsistentGC for improved readability
The function forceConsistentGC was refactored to enhance code readability and consistency. The main improvements include:
- Updated syntax for consistent string quotation and spacing.
- Simplified garbage collection by removing specific major/minor GC calls, as the distinction isn't necessary.
- Implemented minor changes to arrow function formatting for consistency.
These changes neither impact the program logic nor the function behavior but help maintain code quality standards and readability.
* Fix memory leak by removing request.clone() usage
Identified that the memory leak in the project was linked to the usage of request.clone() within the `@sentry/remix` package's callRouteAction handler. Although initially suspected as a Sentry issue, the problem appears to arise from the handling of request.clone() in Remix version 2.1.0. By removing the call to request.clone(), the memory leak has been resolved.
- Introduced garbage collection execution before snapshot to manage memory allocation effectively.
- Improved error handling and timeout mechanisms in the memory leak detector to enhance its resilience during runtime.
- Expanded testing for both GET and POST requests to monitor and validate potential memory leaks better. The POST requests involve sending large payloads to stress-test the system.
- The modification particularly focuses on enhancing robust memory tracking and providing detailed progress reporting during request phases.
* patch @sentry/remix to prevent memory leaks
* Fix pnpm lock
* undo some unrelated changes
* The logger now supports metadata
* Added metadata to ServiceValidationError in some critical places
* Don't ack the heartbeat if there's a mismatch, it might prevent a brand new one
* Don't ack the heartbeat inside stalled. By returning it will be acked IF the deduplication key matches
* Only start an attempt if not finished. Send message to worker if pending executing
* Fix the exit process reason tet
* Fixed cancelling test since bug fix
The old behaviour was wrong for pending executing in the test
* set correct run status on snapshot after dequeue
* set run status back to PENDING when we requeue
* remove retrying after failure status from v4 and fix tests
* fix one last test
* Adds a background image dashboard wrapper
* Dashboard background image
* Adds a background image to the welcome onboarding page
* Background is constructed of 3 images
* Adds the background to the create org page
* Adds a background to the choose plan page
* Change the default button spinner color to white
* Adds the background to the create new project page
* Updates the invite team member page
* Adds background image to received invite page
* Reordered react hooks + frontend sections
* Updated the overview and nav
* Separated out SWR hooks
* Restructured metadata sections
* Improved backend docs
* Fixed broken link
* Fixed broken links
* Updated the structure
* Restructured overview
* Updated examples cards
* Improved overview
* Updated how it works
* Updated auth
* Added type safety to the run object page
* made the subscribe description clearer
* Fixed links in triggering
* Fixed link
* Removed examples footers
* Copy tweak
* Consolidated metadata and subscribe pages
* moved metadata task examples to metadata
* Fixed links
* Fixing links like zelda
* Removed dead import
* Clearer titles
* Cap R for Realtime
* Fixes broken link
---------
Co-authored-by: James Ritchie <james@trigger.dev>
* Map new allowedMasterQueues → allowedWorkerQueues
* ClickHouse worker_queue on task runs
* Added the Region to the run inspector
* Pass a region in when triggering
* Added a changeset
* Added triggering regions docs
* Added region to the ctx
* Fix for backfiller masterQueue/workerQueue
* Add manual monorepo setup guide for Trigger.dev
This commit introduces a new "Manual setup" guide for setting up Trigger.dev in projects, specifically focused on monorepo configurations. The guide outlines two primary approaches for monorepos: creating a dedicated tasks package and integrating tasks directly within apps. This detailed documentation aims to help developers manually configure their projects, bypassing automated steps, and understanding the setup better.
- Provides step-by-step instructions for both 'Tasks as a package' and 'Tasks in apps' approaches.
- Includes example configurations for various package managers, environment setups, and runtime options.
- Enhances user understanding of Trigger.dev's configuration requirements in complex monorepo environments.
The purpose of adding this guide is to enable developers to seamlessly integrate Trigger.dev into their monorepos, whether they choose to abstract tasks into packages or embed them within individual applications. This flexibility supports diverse project structures while maintaining consistency with Trigger.dev's operational prerequisites.
* Correct tasks usage in documentation
The 'tasks as package' section in the documentation had an incorrect example under 'Use tasks in your apps' which needed correction to align with the actual package usage.
- Fixed incorrect import of tasks by updating to the correct import from '@repo/tasks/trigger'.
- Updated the syntax to use 'tasks.trigger' with type parameters, following the new pattern established for triggering tasks with TypeScript.
- Added error handling to catch and log errors during task execution, returning a meaningful error message instead of just failing silently.
This update ensures developers have an accurate reference when implementing tasks in their applications, especially given the breaking changes in TypeScript compatibility due to recent package updates.
* Update package configuration for Zod 3-4 compatibility
The recent Zod package updates from version 3 to 4 introduce breaking changes in TypeScript compatibility that require adjustments in our project configuration files. The primary updates involve adding type annotations and modifying import statements to support the shift without breaking existing functionality.
- Updated `package.json` and initialization files to align with new compatibility requirements.
- Modified server task examples to explicitly use type imports for improved error handling and consistency.
- Adjusted workspace and project settings to maintain compatibility and improve configuration clarity.
- Provided references to current examples for better implementation guidance.
* More manual setup guide steps
* Fix bun docs link
* Initial Regions page
* Fix for bad attribute name
* Switching regions is working. Some style improvements
* Use a dialog for confirmation, not great yet
* Added allowedMasterQueues
* Improves flag icons
* Improves the region switch modal with more info
* Adds “suggest a region” table row
* New icons for the buttons
* Improved the tooltip information
* New “small” badge style
* Make the default badge live in its column
* Better DO icon size
* Remove unused export of regions options
* Show upgrade message for free users to get static IPs
* Admins can view all regions and switch at will
---------
Co-authored-by: James Ritchie <james@trigger.dev>
* Update zod package to version 3.25.76 across all modules
Update the zod library from version 3.23.8 to 3.25.76 in multiple package files to ensure compatibility and take advantage of new features or bug fixes introduced in recent releases. Keeping all modules synchronized with the latest version of zod helps maintain consistency across the project and reduces potential compatibility issues.
- Modified zod version in apps/supervisor, webapp, and various internal packages.
- Updated zod references in pnpm-lock.yaml to reflect the new version.
- Ensure dependencies that rely on zod are using the updated version to avoid mismatches.
* Add changeset
* Add payload schema handling for task indexing
This change introduces support for handling payload schemas during task indexing. By incorporating the `payloadSchema` attribute into various components, we ensure that each task's payload structure is clearly defined and can be validated before processing.
- Updated the TaskManifest and task metadata structures to include an optional `payloadSchema` attribute. This addition allows for more robust validation and handling of task payloads.
- Enhanced several core modules to export and utilize the new `getSchemaToJsonSchema` function, providing easier conversion of schema types to JSON schemas.
- Modified the database schema to store the `payloadSchema` attribute, ensuring that the payload schema information is persisted.
- The change helps in maintaining consistency in data handling and improves the integrity of task data across the application.
* Refactor: Remove getSchemaToJsonSchema in favor of schemaToJsonSchema
The `getSchemaToJsonSchema` function was removed and replaced with `schemaToJsonSchema` across the codebase. This update introduces a new `@trigger.dev/schema-to-json` package to handle conversions of schema validation libraries to JSON Schema format, centralizing the functionality and improving maintainability.
- Removed `getSchemaToJsonSchema` exports and references.
- Added new schema conversion utility `@trigger.dev/schema-to-json`.
- Updated `trigger-sdk` package to utilize `schemaToJsonSchema` for payloads.
- Extensive testing coverage included to ensure conversion accuracy across various schema libraries including Zod, Yup, ArkType, Effect, and TypeBox.
- The update ensures consistent and reliable schema conversions, facilitating future enhancements and supporting additional schema libraries.
* Add support for Zod 4 in schema-to-json
This change enhances the schema-to-json package by adding support for Zod version 4, which introduces the native `toJsonSchema` method. This method facilitates a direct conversion of Zod schemas to JSON Schema format, improving performance and reducing reliance on the `zod-to-json-schema` library.
- Updated README to reflect Zod 4 support with native method and retained support for Zod 3 via existing library.
- Modified package.json to allow installation of both Zod 3 and 4 versions.
- Implemented handling for Zod 4 schemas in `src/index.ts` using their native method.
- Added a test case to verify the proper conversion of Zod 4 schemas to JSON Schema.
- Included a script for updating the package version based on the root package.json.
- Introduced a specific TypeScript config for source files.
* Revise schema-to-json for bundle safety and tests
The package @trigger.dev/schema-to-json has been revised to ensure bundle safety by removing direct dependencies on schema libraries such as Zod, Yup, and Effect. This change minimizes bundle size and enhances tree-shaking by allowing external conversion libraries to be utilized only at runtime if necessary. As a result, the README was updated to reflect this usage pattern.
- Introduced `initializeSchemaConverters` function to load necessary conversion libraries at runtime, keeping the base package slim.
- Adjusted test suite to initialize converters before tests, ensuring accurate testing of schema conversion capabilities.
- Updated `schemaToJsonSchema` function to dynamically check for availability of conversion libraries, improving flexibility without increasing the package size.
- Added configuration files for Vitest to support the new testing framework, reflecting the transition from previous test setups.
These enhancements ensure that only the schema libraries actively used in an application are bundled, optimizing performance and resource usage.
* Refine JSON Schema typing across packages
The changes introduce stricter typing for JSON Schema-related definitions, specifically replacing vague types with more precise ones, such as using `z.record(z.unknown())` instead of `z.any()` and `Record<string, unknown>` in place of `any`. This is part of an effort to better align with common practices and improve type safety in the packages.
- Updated the `payloadSchema` in several files to use `z.record(z.unknown())`, enhancing the type strictness and consistency with JSON Schema Draft 7 recommendations.
- Added `@types/json-schema` as a dependency, utilizing its definitions for improved type clarity and adherence to best practices in TypeScript.
- Modified various comments to explicitly mention JSON Schema Draft 7, ensuring developers are aware of the JSON Schema version being implemented.
- These adjustments are informed by research into how popular libraries and tools handle JSON Schema typing, aiming to integrate best practices for improved maintainability and interoperability.
* Add JSON Schema examples using various libraries
The change introduces extensive examples of using JSON Schemas in the 'references/hello-world' project within the 'trigger.dev' repository. These examples utilize libraries like Zod, Yup, and TypeBox for JSON Schema conversion and validation. The new examples demonstrate different use cases, including automatic conversion with schemaTask, manual schema provision, and schema conversion at build time. We also updated the dependencies in 'package.json' to include the necessary libraries for schema conversion and validation.
- Included examples of processing tasks with JSON Schema using libraries such as Zod, Yup, TypeBox, and ArkType.
- Showcased schema conversion techniques and type-safe JSON Schema creation.
- Updated 'package.json' to ensure all necessary dependencies for schema operations are available.
- Created illustrative scripts that cover task management from user processing to complex schema implementations.
* Refactor SDK to encapsulate schema-to-json package
The previous implementation required users to directly import and initialize functions from the `@trigger.dev/schema-to-json` package, which was not the intended user experience. This change refactors the SDK so that all necessary functions and types from `@trigger.dev/schema-to-json` are encapsulated within the `@trigger.dev/*` packages.
- The examples in `usage.ts` have been updated to clearly mark `@trigger.dev/schema-to-json` as an internal-only package.
- Re-export JSON Schema types and conversions in the SDK to improve developer experience (DX).
- Removed unnecessary direct dependencies on `@trigger.dev/schema-to-json` from user-facing code, ensuring initialization and conversion logic is handled internally.
- Replaced instances where users were required to manually perform schema conversions with automatic handling within the SDK for simplification and better maintainability.
* Add JSONSchema type for payloadSchema in tasks
The change was necessary to improve type safety by using a proper JSONSchema type definition instead of a generic Record<string, unknown>. This enhances the developer experience and ensures that task payloads conform to the JSON Schema Draft 7 specification. The JSONSchema type is now re-exported from the SDK for user convenience, hiding internal complexity and maintaining a seamless developer experience.
- Added JSONSchema type based on Draft 7 specification
- Updated task metadata and options to use JSONSchema type
- Hid internal schema conversion logic from users by re-exporting types from SDK
- Improved bundle safety and dependency management
* Add JSON schema testing and revert package dependencies
This commit introduces a comprehensive set of JSON schema testing within the monorepo, specifically adding a new test project in `references/json-schema-test`. This includes a variety of schema definitions and tasks utilizing multiple validation libraries to ensure robust type-checking and runtime validation.
Additionally, the dependency versions for `@effect/schema` have been adjusted from `^0.76.5` to `^0.75.5` to maintain compatibility across the project components. This ensures consistent behavior and compatibility with existing code bases without introducing breaking changes or unexpected behavior due to version discrepancies.
Key updates include:
- Added new test project with extensive schema validation tests.
- Ensured type safety across various task implementations.
- Reverted dependency versions to ensure compatibility.
- Created multiple schema tasks using libraries like Zod, Yup, and others for thorough testing.
* Refactor JSON Schema test files for clarity
Whitespace and formatting changes were applied across the `json-schema-test` reference project to enhance code readability and cohesion. This included removing unnecessary trailing spaces and ensuring consistent indentation patterns, which improves maintainability and readability by following the project's code style guidelines.
- Renamed JSONSchema type annotations to adhere to TypeScript conventions, ensuring that all schema definitions properly satisfy the JSONSchema interface.
- Restructured some object declarations for improved clarity, especially within complex schema definitions.
- These adjustments are crucial for better future maintainability, reducing potential developer errors when interacting with these test schemas.
* Fixed some stuff
* WIP
* we now convert schema to jsonSchema on the CLI side via the indexing
* Remove the json-schema-test reference project
* Improve schema-to-json peer deps and fix effect schema
* Explain the casting and match the version numbers
* Fixed a bunch more schema stuff
* Don't clean files that might be written to
* Don't use a custom version of vitest in the new package
* fix attw in schema-to-json
* prevent infinite retry loop for unicode errors
* structured logs for prisma events
* preserve all prisma event fields
* either use structured logs or stdout, never both
* split runs repo tests
* decrease test shards to 8
* First draft billing alerts page
* Budget alert form working
* Don't let free plan users change the billing alert amount
* Fix missing key in map in the form
* Disable queues/org from admin API endpoint
* Don't allow resuming if runsEnabled is false
* Refer to "Billing alerts" not "Plans"
Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>
* Form missing dependencies fix
Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>
* Deal with thrown errors, fix for duplicating email fields
* Added a RuntimeEnvironment organizationId index
---------
Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>
* Cleanup context and execution creation, cache stuff, add parent and root task run ids
* more efficient by using friendly IDs instead of doing joins
* metadata.root/parent now reference current run when run has no root/parent
* Adding changeset
* try to make test less flaky
* Clean imports
* Another attempt to fix the flaky test
* Fix usage by still passing durationMs and costInCents to the execution, just not the run.ctx
* Improve dynamic flush scheduler for otel data
The changes introduce a more flexible and adaptive dynamic flush scheduler to address production issues where the system wasn't flushing data fast enough, causing memory growth and crashes. This issue arises from the existing scheduler handling only a single flush at a time, limiting concurrency and failing to cope with the influx of logs.
- Added configuration options for setting minimum and maximum concurrency levels, maximum batch size, and memory pressure threshold. These parameters ensure that flush operations adjust dynamically based on workload and pressure.
- Implemented `pLimit` to facilitate concurrent flush operations, with adjustments made according to batch queue length and memory pressure.
- Metrics reporting improvements were added to monitor the dynamic behavior of the flush scheduler, aiding in identifying performance issues and optimizing the operation accordingly.
* Implement load shedding for TaskEvent records
This change introduces load shedding mechanisms to manage TaskEvent
records, particularly those of kind LOG, when the system experiences
high volumes and is unable to flush to the database in a timely
manner. The addition aims to prevent overwhelming the system and
ensure critical tasks are prioritized.
- Added configuration options for `loadSheddingThreshold` and
`loadSheddingEnabled` in multiple modules to activate load shedding.
- Introduced `isDroppableEvent` function to allow specific events to
be dropped when load shedding is enabled.
- Ensured metrics are updated to reflect dropped events and load
shedding status, providing visibility into system performance
during high load conditions.
- Updated loggers to inform about load shedding state changes,
ensuring timely awareness of load management activities.
* Fix undefined 'queuePressure' variable in DynamicFlushScheduler
The 'queuePressure' variable was being used without being defined
in the DynamicFlushScheduler class, causing potential runtime
errors. This commit adds the missing definition and ensures that
the variable is correctly calculated based on the 'totalQueuedItems'
and 'memoryPressureThreshold'.
- Addressed code inconsistencies and improved formatting.
- Defined 'queuePressure' in the 'adjustConcurrency' method
to prevent potential undefined errors.
- Enhanced readability by maintaining consistent spacing and
format across the file, contributing to the stability and
maintainability of the code.
- Adjusted batch size logic based on the newly defined 'queuePressure'
variable.
* Refactor concurrency adjustment logic in scheduler
The concurrency adjustment logic in the dynamic flush scheduler has been refactored to improve clarity and maintainability. This change moves the calculation of pressure metrics outside of the conditional blocks to ensure they are always determined prior to decision-making.
- The queue pressure and time since last flush calculations were moved up in the code to be independent of the 'backOff' condition.
- This refactor sets up the groundwork for more reliable concurrency scaling and better performance monitoring capabilities. The overall logic of adjusting concurrency based on system pressure metrics remains unchanged.
This adjustment addresses ongoing issues with the scheduler that were not resolved by previous changes.
* Some tweaks
* Sentry WIP
* Configure sentry for uploading and releasing during the publish webapp step
* Delete source maps after uploading
* Forward logger.error calls to sentry through Logger.onError
* Couple tweaks to the dockerfile
* Premerge run batch before sending it to clickhouse
* Pre-order batch items before sending to clickhouse in favor of performance
* existing
* Emit event on batch flushes
* When merging batches, keep the last occurrence items with the same version
* Add a couple of tests
* Initial burst changes to queue page
* Added tooltip and changed wording around
* View runs from Queues page
* Fix for ugly Version filter "Current" badge
The normal database needs to be mocked because an import somewhere is
using it which causes issues with the test container dbs and hanging at
the end.
Strategy copied from the trigger test file.
* Set the default replication concurrency to 2 for self-hosters
100 was a bit crazy
* Made the run repository an interface, deferring just to CH for now
* Added run repository feature flag, allowing passing a default when getting a flag
* Switch run repository using a feature flag
* Added spans
* Pass the default repository in, so we can try Postgres in the tests
* Fallback to Postgres if ClickHouse errors
* Update feature flags API endpoint
* Queue in run table and filtering
* Debounce the filter changes
* Remove console log
* Added machine filtering
* Added version filtering
* Filter by version in the db
* Removed duplicate classes
* Version filtering hasFilters consistency
* Added queues and machines to the bulk action summary
* runs.list filtering for queue and machine
* Fix for machine errors
* Input field now has accessory instead of shortcut
* First experiments with the UI
* Got the fake filtering working
* AI filtering is working pretty well ✨
* Started working on tool calling
* Tool calling is working
* Styling progress
* Working on the error
* Errors work, improved the styling
* Nice glow effect
* Tweak the darkness of the text field
* Re-ordered the UI, set AI settings to use system prompt and telemetry
* Refactored to make it testable
* Added basic evals
* Better time inputs and evals
* Removed some code comments
* Remove unused useSearchParam change
* Tidy imports
* If no OpenAI API key send json back
* Fix for merge conflict with duplicate query filters
* Another conflict resolved
* Another merge conflict resolved
* Pass the model in, allow changing it
* Queue in run table and filtering
* Debounce the filter changes
* Remove console log
* Added machine filtering
* Added version filtering
* Filter by version in the db
* Removed duplicate classes
* Version filtering hasFilters consistency
* Added queues and machines to the bulk action summary
* runs.list filtering for queue and machine
* Fix for machine errors
* Added experimental_devProcessCwdInBuildDir config option
Added experimental_devProcessCwdInBuildDir config option to opt-in to new process.cwd behavior when executing tasks in the dev CLI. Currently process.cwd maps to the "root" of your trigger.dev project (the directory that contains your trigger.config.ts file). Setting experimental_devProcessCwdInBuildDir to true changes process.cwd to instead be the temporary build directory inside of the .trigger directory.
This makes it so the files added via the additionalFiles extension can be read using the same path in dev and deployed tasks.
* Remove claude code reference because it breaks windows tests 😡
* WIP
* Make release concurrency system extremely simple, everything just releases all the time
* update the deadlock detection to use the new lockedQueueReleaseConcurrencyOnWaitpoint column
* WIP new release concurrency system
* Remove releaseConcurrency and releaseConcurrencyOnWaitpoint
Also removed deadlock detection, and added environment burst concurrency
* Added new DEQUEUED status
Cleaned up the API run statuses, including now detecting new clients and not breaking older clients by adding an API version header to all requests
* Introduce the new "current dequeued concurrency set"
* Remove QUEUED_EXECUTING because we no longer "eagerly" release before checkpointing
* Remove waitpoint test for QUEUED_EXECUTING
* Add isWaiting
* Add changeset
* Use createdAt for ordering realtime runs instead of number
* Clarify the envCurrentDequeuedKey usage
* mock the db.server file to fix the tests
* Updated changset "EXECUTED" -> "EXECUTING"
---------
Co-authored-by: Matt Aitken <matt@mattaitken.com>
* Access machinePreset from the run list presenter
* New icons for machine presets
* New icon + name combo label for the machine preset
* Adds new “Machine” column to the runs list
* Make a separate component for the machine tooltip info
* add machinePreset to the span presenter
* Show the Machine in the Details tab in the Run inspector
* Show an admin only separator
* Fix docs icon in the button
* Small padding tweak
* Move the Machine nearer the costs
* Don't cast the machine preset
* Remove typecast, better to have a bad label if we add a new machine and don't update thos
---------
Co-authored-by: Matt Aitken <matt@mattaitken.com>
* Use sexy scrollbars for templates and recent runs popover content
@Nick this is for you!!1
* Fix TabButton component disabled state
* Disable payload tab when replaying runs with large payloads
* Fix for filtering by run id
Simplified to have a single filter for runId which accepts an array. This unifies the filtering by multiple runs (used by the waitpoint page) and the individual run filtering. It fixes a bug with bulk actions
* Truncate long bulk action titles
* Fix regression that broke superjson behavior for the replay flow
* Ignore undefined type metadata when overriding superjson payloads
Addresses https://github.com/triggerdotdev/trigger.dev/issues/1968
* Add a few unit tests for replaceSuperJsonPayload
* Add ID of the replayedFrom run to the TaskRun schema
* Propagate the replayedFrom run ID in the replay flow
* Link the replayed run in the run details pane
* useSearchParams has
* useSearchParams has
* useSearchParams has
* Consistent way to get the run filters
* Consistent way to get the run filters
* Consistent way to get the run filters
* Initial work on the new bulk actions
* Initial work on the new bulk actions
* Initial work on the new bulk actions
* WIP actions and filtering
* WIP actions and filtering
* WIP actions and filtering
* Empty filter arrays are set to undefined
* Empty filter arrays are set to undefined
* Empty filter arrays are set to undefined
* WIP prisma schema
Removed extra runtimeEnvironmentId
* WIP prisma schema
Removed extra runtimeEnvironmentId
* WIP prisma schema
Removed extra runtimeEnvironmentId
* Migrations
* Migrations
* Migrations
* BulkActionGroup changed some columns around
* BulkActionGroup changed some columns around
* BulkActionGroup changed some columns around
* New badge variant, removed unused ones
* New badge variant, removed unused ones
* New badge variant, removed unused ones
* Bulk action button
* Bulk action button
* Bulk action button
* Make the next runs page the default now
* Make the next runs page the default now
* Make the next runs page the default now
* Improved the RadioButton style
* Improved the RadioButton style
* Improved the RadioButton style
* Remove the old bulk action bar
* Remove the old bulk action bar
* Remove the old bulk action bar
* More UI progress
* More UI progress
* More UI progress
* Lots of UI changes to the Runs page
* Lots of UI changes to the Runs page
* Lots of UI changes to the Runs page
* Fixed period filter resetting everything
* Fixed period filter resetting everything
* Fixed period filter resetting everything
* Improved the Switch secondary style
* Improved the Switch secondary style
* Improved the Switch secondary style
* Buggy filter fixes
* Buggy filter fixes
* Buggy filter fixes
* Improved the filter display and fixed a bug with search param from object
* Improved the filter display and fixed a bug with search param from object
* Improved the filter display and fixed a bug with search param from object
* Clear button is minimal
* Clear button is minimal
* Clear button is minimal
* Using a presenter now
* Using a presenter now
* Using a presenter now
* Bulk actions are created, but not actually processed (yet)
* Bulk actions are created, but not actually processed (yet)
* Bulk actions are created, but not actually processed (yet)
* Bulk replay/cancel is working
* Bulk replay/cancel is working
* Bulk replay/cancel is working
* Multiple fixes, added bulk column to PG
* Multiple fixes, added bulk column to PG
* Multiple fixes, added bulk column to PG
* Bulk action run filtering working using CH
* Bulk action run filtering working using CH
* Bulk action run filtering working using CH
* Replay setting the bulk id on the runs
* Replay setting the bulk id on the runs
* Replay setting the bulk id on the runs
* Properly cap the time when doing a bulk action
* Properly cap the time when doing a bulk action
* Properly cap the time when doing a bulk action
* If the bulk action isn't recent, add it to the dropdown anyway
* If the bulk action isn't recent, add it to the dropdown anyway
* If the bulk action isn't recent, add it to the dropdown anyway
* Blank version of the bulk actions page
* Blank version of the bulk actions page
* Blank version of the bulk actions page
* Individually selected runs working
* Individually selected runs working
* Individually selected runs working
* Use selected mode if runs are checked
* Use selected mode if runs are checked
* Use selected mode if runs are checked
* Added the modal
* Added the modal
* Added the modal
* Marked the old bulk actions stuff as deprecated
* Marked the old bulk actions stuff as deprecated
* Marked the old bulk actions stuff as deprecated
* Renamed bulk action file
* Renamed bulk action file
* Renamed bulk action file
* Bulk run filter with the name and a default
* Bulk run filter with the name and a default
* Bulk run filter with the name and a default
* WIP on bulk actions page
* WIP on bulk actions page
* WIP on bulk actions page
* Updated panel, added new truncated id component
* Updated panel, added new truncated id component
* Updated panel, added new truncated id component
* Style improvements to the radio buttons
* Style improvements to the radio buttons
* Style improvements to the radio buttons
* Added an option action completion email
* Added an option action completion email
* Added an option action completion email
* Adds a blank state for the bulk actions page
* Adds a blank state for the bulk actions page
* Adds a blank state for the bulk actions page
* Nicer completed email
* Nicer completed email
* Nicer completed email
* Don't open the bulk action panel if there are no runs
* Don't open the bulk action panel if there are no runs
* Don't open the bulk action panel if there are no runs
* Runs blank state and bulk action accordion
* Runs blank state and bulk action accordion
* Runs blank state and bulk action accordion
* Updates secondary/small switch style
* Updates secondary/small switch style
* Updates secondary/small switch style
* Pagination buttons no longer split in twain (WIP)
* Pagination buttons no longer split in twain (WIP)
* Pagination buttons no longer split in twain (WIP)
* Aborting working
* Aborting working
* Aborting working
* Bulk action live reloading
* Bulk action live reloading
* Bulk action live reloading
* ListPagination works correctly in all states
* ListPagination works correctly in all states
* ListPagination works correctly in all states
* Run page, show friendlyId instead of number
* Run page, show friendlyId instead of number
* Run page, show friendlyId instead of number
* Bulk action help open by default if you have none
* Bulk action help open by default if you have none
* Bulk action help open by default if you have none
* Extra status filtering step because of replication delay
* Extra status filtering step because of replication delay
* Extra status filtering step because of replication delay
* Wider bulk action onboarding
* Wider bulk action onboarding
* Wider bulk action onboarding
* More sensible widths on the bulk action side panel
* More sensible widths on the bulk action side panel
* More sensible widths on the bulk action side panel
* Border color tweak to the RadioButton
* Border color tweak to the RadioButton
* Border color tweak to the RadioButton
* Improved the accordion component hover states
* Improved the accordion component hover states
* Improved the accordion component hover states
* Updates the bulk action blank state images to the latest UI
* Updates the bulk action blank state images to the latest UI
* Updates the bulk action blank state images to the latest UI
* Added R and C shortcuts back in
* Added R and C shortcuts back in
* Added R and C shortcuts back in
* Fix for selecting a single run
* Fix for selecting a single run
* Fix for selecting a single run
* Improved exit icon, added shortcut to modal
* Improved exit icon, added shortcut to modal
* Improved exit icon, added shortcut to modal
* Tidy imports
* Tidy imports
* Tidy imports
* Tidy imports
* Tidy imports
* Tidy imports
* Tidy imports
* Tidy imports
* Tidy imports
* Tidy imports
* Fix for grid layout when 1 page of bulk actions visible
* Fix for grid layout when 1 page of bulk actions visible
* Fix for grid layout when 1 page of bulk actions visible
* Removed the ... on the abort button
* Removed the ... on the abort button
* Removed the ... on the abort button
* Removed the ... on the abort button
* Animate the progress bar
* Set TZ="UTC" in the env example
* Filter summary in the bulk inspector
* Improves the pagination styling
* Improves the pagination styling
* Delete old bulk action routes
* Removed old Postgres RunListPresenter
* Retry any replication error where the message contains "timeout"
* Increase wait to make test less flaky
* The test was using run id instead of friendly id
* Safer array access
* Remove error log if there's a bad status
* Nicer frontend type safety with the bulk action and mode
* Switched a log to a debug log
* Retry replication unless the error is a known non-retry error
Flip the strategy to retry by default
* Make ClickHouse required
* Backfill run replication admin API endpoint
* Set a CLICKHOUSE_URL for unit tests
---------
Co-authored-by: James Ritchie <james@trigger.dev>
* Rearrange the layout of the replay modal
* Add run options to the replay modal
* Handle payload and metadata correctly in the shared json editor
* Apply run options in replays
* Fix clear button issue in json editor
Memoization of the clear function caused some problems. Removing it should not cause performance issues.
* Update replay modal hint
* Move machine and version fields to the top for visibility
* Use the same field ordering in the test page
* Reload queues and versions on env override
* Adapt json editor to fill full height
* Clean up a few excessive ternaries
* Avoid ui jump on env selection
* Switch to sexy scrollbars for scheduled tasks in the test page
* Improve TaskRunProcess health detection so we don't try and reuse an unhealthy process
This was happening after the process was killed internally, like by an OOM error
* Add changeset
* don't try interactive login in ci, link to docs
* add ci note to cli deploy docs
* add ci note to github actions
* add changeset
* make cron test less strict
* update node-22 image
* update bun image
* disable io_uring
* fix fallback bun path
* prevent duplicate warnings
* add runtime and version to deployments
* runtime icons
* fallback to nodejs
* prevent empty table cell menu
* log if local build on deploy
* pass io_uring env var to child
* denormalize runtime and version, display on run details
* add changesets
* disable pr checks for changeset commits..
* add runtime data to deployed bg workers
* Introduce request idempotency to prevent duplicate triggers
* Implement request idempotency on trigger
* Use x-trigger-request-idempotency-key header instead
Plus cleanup shared logic in the routes
* Add changeset
* Oops, lets not hardcode a 408
* A couple of improvements
* Add new prisma model for task run templates
* Create run templates in a new service
* Add modal to create run templates in the test page
* Show templates list and apply values when selected
* Hide template creation time in the dropdown list, only show date
* Enable deleting run templates
* Show success toast on template creation
* Validate template label length
* Improve the template creation success indicator
* Use formAction consistently to differentiate submissions
* Type formAction for better editor support
* Prettify run template payload and metadata
* Add triggerSource, concurrencyKey and ttl to run templates
* Switch to sexy scrollbars in the options pane
* Add missing field descriptions
* Add a run options link to the docs
* Make the copy for the field descriptions more concise
* Fix tooltip issue in the Select component
* Implement a new primitive UI component for picking durations
* Implement a new component to input run tags
* Expose all run options in the test run page
* Add subtle animations when adding/removing run tags in the test page
* Add a new resource endpoint for fetching queues
* Fetch usable queues for the selected task
* Fix width display issue in the select component
* Enable locking a run to a version from the test page
* Disable entering max attemps <0
* Validate tags
* Add recent runs popover
* Only show latest version for development environments
* Update run options when selecting a recent run
* Rearrange the test page layout
* Add subtle animation to the duration picker segments on focus
* Improve queue selection dropdown styling
* Fix disabled state issue for the SelectTrigger component
* Disable version selection field for dev envs
* Add usage hints next to the run option fields
* Add machine preset to the run options list
* Allow arbitrary queue inputs for v1 engine runs
* Show truncated run ID instead of run numbers for recent runs
Run numbers will soon get deprecated due to contention issues
* Fix duplicate queue issue
* Extract common elements across the standard and scheduled test task forms
* Apply values from recent runs to scheduled tasks too
* Add additional run options for scheduled tasks
* Use a slightly smaller font size for run option labels
* Disallow commas in the run tag input field
* Switch to a custom icon for recent runs button
* Flatten the load function test task result object
* Avoid redefining machine presets, use zod schema instead
* Fix ClockRotateLeftIcon jsx issues
* Remove recent runs button tooltip as it causes nesting errors
* Adjust the page layout to make it clear which task is currently selected
* Inline the tab group with the copy/clear buttons
* Adds a new route for logging in with mfa
* New path for security page
* Adds “Security” link to account side menu
* Update the Switch component to allow label positions left and right
* Optionally hide the Close button in the Dialog title bar
* Installs `qrcode` react package for generating QR codes.
* CopyButton component now takes children
* New Security route for setting up MFA
* Adds new OTP package for the chadcn InputOTP component
* Adds new InputOTP chadcn component
* Adds InputOTP chadcn component to the MFA login screen
* InputOTP component supports variant styles
* Improvements to form handling
* Show a confirmation modal before you can disable MFA
* Revert redirect back to the dashboard for now
* Implement MFA enabling and disabling
* Refactor and cleanup mfa management code
* More cleanup
* Handle errors in the management action
* Implement mfa login flow
* recovery code input should be password
* Implement rate limiting on the mfa validation endpoint
* Better error ux
* Implement mfa emails and apply James' updates
* Use latest @better-auth/utils
* Improvements via CodeRabbit review
---------
Co-authored-by: James Ritchie <james@trigger.dev>
* create repo if doesn't exist
* fresh auth token for each deploy
* optional assume role
* log when machine overrides enabled
* make test repo namespace configurable
* assume role fix and env var changes
* improve ecr check
* improve tag parsing
* tag parsing tests
* track if repo created and fix test
* missing tryCatch for sts call
* processKeepAlive disabled performance fix
When processKeepAlive was disabled, the eagerly created TaskRunProcess being discarded and a whole new TaskRunProcess created at execution time.
* Don't use a TaskRunProcess if it's being killed, fixes retryImmediately
* Add retry logic for insert operations
Add a generic retry mechanism for task run and payload inserts to handle
transient connection errors. The new #insertWithRetry method retries up to
three times with exponential backoff and jitter on retryable connection
errors such as connection resets or timeouts. Errors are logged and
recorded in tracing spans to improve observability and robustness of the
replication service.
* Replication settings are configurable
* Log out the runIds for failed batches
* Detecting bad JSON in run replication and ignoring it
* Reproduced split unicode error
* Move output file
* Massively improved the performance
* Minor performance improvements
* Unskip tests
* Remove unused test in CH package
* Fix for the ClickHouse UI explorer
* RunReplication keepAlive defaults to false
* Add concurrency_key and bulk_action_group_ids to ClickHouse task runs
* ClickHouse package doesn't need to be built anymore for the webapp
* Set the concurrency_key from the run replication service
* v4: current concurrency sweeper
* Fix webapp tests
* Ensure only a single instance performs concurrency sweeping by using redis-worker cron jobs
* Improved the mark phase
* Ensure cron jobs get rescheduled even if the handler throws an error
* Better property names
* fix both internal and external registry host
* add next steps to self-hosting overview
* separate into webapp and registry ingress
* fix external registry config check
* fix localhost registry warning
* bump chart version
* Fix incorrect logs on new schedule engine triggered taskss
Also added the ability to recover schedules in the schedule engine via an Admin API endpoint in the new schedule engine
* Fixed schedule recovery failing test
* shallow clone
* fix image tag locking example
* electric should use db url env var
* add htpasswd note
* use local driver with log rotation by default
* make worker urls more easily configurable
* configure dev otel endpoint via .env
* increase min recommended worker specs
* move worker url section
* Runs filter by org id and add created at to ordering
* CopyableText can accept an alternative value for copying
* The runs table now shows the ID instead of run number
* Paginating back/forwards fix
* The task stats need org id and project id too
* Fix realtime re-subscribing stale data issue
Fixes an issue with realtime when re-subscribing to a run, that would temporarily display stale data and the changes. Now when re-subscribing to a run only the latest changes will be vended
* removed logs
* Install the kapa sdk
* WIP using the SDK for the Kapa Ask AI widget
* Removes old kapa from root
* Now rendering everything inside the dialog component
* Fixes min-height of dialog content
* Remove kapa from root
* prevents kapa using reCaptcha
* Adds more functionailty with temporary UI placement for now
* Reset conversation button
* Adds a new sparkle list icon
* Adds some example questions as a blank state
* Animate in the example questions
* use “marked” package to render markdown
* Improve some animations
* Submit a question from the URL param
* adds custom scroll bar styling
* fixes modal to correct height after re-opening it
* Add button to stop generating answer mid-stream
* Adds buttons states to show submitting, generating, submittable
* Adds a helpfull sentence in the blank state
* Show a message if the chat returns an error
* Adds reset chat and feedback buttons to the bottom of an answer
* Makes sure you can give feedback in the different states of chat
* Adds a suble background to the dialog
* Fix a button inside button error
* Improve the shortcut esc key on dialog and sheet component
* Fix classname error
* organize imports
* Use our custom focus-visible
* Move the Tooltip for the button into the AskAI component
* Improved error message
* Organize imports
* Animated the modal gradient
* Small layout improvements
* Adds most asked questions from Kapa
* border glow tweak
* AskAI component is now a hook that can take a question
* remove kapa script
* Add a delay before the modal opens when usign the URL params
* Remove old component
* Update to the latest Kapa version
* Rephrased error message
* Use correct types for conversation
* Fixed types for addFeedback
* Adds DOMPurify package
* removed unused const
* Removed unnecessary platform specification
* Reset the timeout when the ai panel pops up
Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>
* Fix for coderabbit bad commit
* Clean up imports
---------
Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>
### PR: Optimize **TaskRun** indexes for hot-path queries
**What changed**
| Object | Type | Purpose |
| ------------------------------- | --------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------- |
| `taskrun_runtime_id_desc_idx` | **BTREE** `(runtimeEnvironmentId, id DESC) INCLUDE (createdAt)` | Eliminates explicit sort for the “latest task runs” query (`ORDER BY id DESC`) while remaining index-only. |
| `taskrun_runtime_createdat_idx` | **BTREE** `(runtimeEnvironmentId, createdAt DESC) INCLUDE (id)` | Accelerates the filter-only path that scans by `createdAt >= …` without any ordering requirement. |
| `taskrun_createdat_brin` | **BRIN** on `createdAt` (`pages_per_range = 128`) | Lets the planner skip whole blocks older than the time window for both queries at < 100 MB cost. |
| *(cleanup)* | **DROP** `TaskRun_runtimeEnvironmentId_createdAt_id_idx` | Retires the 3-column index once the new ones are built. |
**Key details**
* All indexes created **CONCURRENTLY** to avoid write blocking.
* `fillfactor = 90` on b-trees for balanced space vs. future growth.
* Net disk usage drops **≈ 15–20 GB** while each query now gets a purpose-built access path.
**Why**
* Remove planner Sort nodes for the top-N “latest runs” view.
* Speed up environment-filtered range scans.
* Shrink index bloat and improve cache efficiency.
* Add createdAt filter to realtime subscribing with tags
* Filter realtime colums and expose ability to skip some columns
* Add sharding support for electric
* Use unkey cache for the created at filter caching
* Remove 2 unused indexes on TaskRun
* Run list now filters by a single runtime environment
* Remove project ID indexes
* Use clickhouse in task list aggregation queries instead of pg (keep pg for self-hosters)
* WIP clickhouse powered runs list
stuff
* Improve the query to get the latest tasks for the task list presenter
* Update the usage task list to use clickhouse
* Implement next runs list powered by clickhouse
* Add new index for TaskRun for the runs list, by environment ID
* Add runTags gin index
* Handle possibly malicious inputs
* Ignore claude settings
* Better handling not finding an environment on the schedule page
* Use ms since epoch in test, not seconds
* Remove unused function
* Fix test
* Use an env var for the realtime maximum createdAt filter duration (defaults to 1 day)
* Fixed the query builder to correct the group by / order by order
* Make sure runs.list still works
* Create small-birds-arrive.md
* remove unused env vars
* actually use ALERT_REPLY_TO_EMAIL for alerts (non-breaking)
* increase fallback branch limit to 100M
* more unused env vars and example
* hide usage page when self-hosted
* fix for init flow
* set default or concurrency to 300 to match 100 on env
* fail fast when registry env vars are empty strings
* WIP
* Run queue now works with the worker queue / master queue split
* Acking should also cause the master queue to be processed
* Convert run engine tests and run engine to use runQueue changes
* Include the util files in the test tsconfig
* coordinator target should be es2020 as well
* providers target 2020
* Fix the triggerTask tests in the webapp
* v4 now working with the new worker queues, and added the legacy master queue migration stuff
* report worker queue lengths via opentelemetry metrics
* Adding lock metrics
* Release concurrency bucket metrics
* • Updated RunQueue.removeEnvironmentQueuesFromMasterQueue() method signature to take runtimeEnvironmentId instead of masterQueue parameter
• Added automatic master queue shard calculation using this.keys.masterQueueKeyForEnvironment(runtimeEnvironmentId, this.shardCount)
• Updated RunEngine wrapper method to use new runtimeEnvironmentId parameter
• Updated DeleteProjectService to call the method once per environment instead of once per master queue
• Simplified API by encapsulating master queue sharding logic within RunQueue class
* metrics now working, configure the run queue settings, additional metrics for run engine and redis-worker
* Fix CodeRabbit suggestions
* return undefined from dequeueFromWorkerQueue, not null
* Remove message from worker queue in certain circumstances when acking
* Update log
* Ensure master queue consumers cannot stop from a processing error, and make the consumer interval configurable via an env var
* Change how the run queue master queue consumers are disabled internally
* Fixed tests
* process the queue on nack
* Fix more tests
* Fix priority tests
* Fixed dequeueing test
* remove registry proxy
* remove --self-hosted flag
* automatically set network build flag
* update syncEnvVars debug log
* improve switch command
* always display deploy errors if they exist
* fix stuck deploy command after finalize error
* webapp-driven deploys, multi-platform support, lots of fixes
* add worker deployment migration
* rename image platform env var
* only try to sync parent env vars for preview deployments
* add KEEP_TMP_DIRS
* supervisor: docker api version lock, auth, multi-platform
* set image ref on create, validate digest
* use metadata for digest, fix local multi-platform builds
* print git meta branch before commit
* improve push and load flag handling
* make runs after local builds compatible with load and push
* small improvement for platform overrides
* add image platform to dequeued message
* remove deprecated init request body fields
* fix fail deployment id param
* remove build debug logs
* pass report merge with no tests
* structured run debug logs
* add required env var for tests
* should not be an error log
* add changeset
* Draft preview branches docs
* More wip on docs
* Explain what the GH action does
* Added manual deploy/archive instructions
* Added dashboard section to the docs
* More wip on the preview branch docs
* Added preview/branch to the CLI deploy docs
* Add preview branch note to triggering docs
* Added preview branch notes to the API keys docs
* Added preview stuff to the Context docs
* CLI preview archive command
* Add a better explanation and details about env vars
* Fix for weird reference to the dashboard
* Prefer branch variables, but load preview envs too
* WIP with vercel preview env var syncing
* Return success if either child or parent env vars are synced
* Removed log of env vars
* Remove another log
* disable run debug logs by default
* lightweight webapp health check
* disable debug logs for dev runs
* disable run debug logs for supervisor client
* add changeset
* cursor should ignore .env files
* fix for duplicate builds when starting dev
* output metafile in dev
* attach metafile to background worker
* attach import timings to worker manifest
* warn during dev if imports take more than 1s
* add analyze command
* update disable warnings flag message
* add changeset
* Initial preview migrations
* Modified the staging endpoint to create preview environments
* Added isBranchableEnvironment to RuntimeEnvironment
* Staging = yellow Preview = orange
* Changed the env sort order
* Set isBranchableEnvironment correctly. Create preview for new projects
* Very basic branch menu
* Creating branches from the dashboard
* Fix for string icons on project delete page
* Don’t show branch API keys
* WIP on the manage branches page
* RuntimeEnvironment added projectId index
* Only create the parentEnvironmentId column if it doesn’t exist already
* Improved the limit wording
* Add search to the branch list
* contains in both places
* Many style improvements
* Branch dropdown and v4 badge
* Arching/unarchive branches working in the dashboard
* Tidied imports
* Change preview slug from `prev` to `preview`
* Use correct color for side menu preview branch icon
* Upsert the branch and use the shortcode as a unique constraint
* Upserting working with nice messages in the dashboard
* Better errors when upserting branches
* Button shortcut, don’t allow event to propagate
* Better duplicate error message
* Filter out archived branches from the env selector
* Archiving/creating tweaked some more
* Add an archived banner to the app, fixes for archived branches and upsells
* Fixed pagination
* Disable editing schedules, pausing queues, testing tasks
* Don’t allow replaying if the env is archived
* When deploying detect the correct environment
* Get the projectClient when there’s a branch
* createGitMeta function, most code from the vercel CLI repo
* Deploy, getting the correct environment client
* Added git column to WorkerDeployment
* Add GitMeta to core schemas
* Create branch when deploying
* WIP on branch support in the API
* Delete old createTaskRunAttempt fn
* apiAuth remove export from internal functions
* Rename env var to “TRIGGER_PREVIEW_BRANCH”
* Add TRIGGER_PREVIEW_BRANCH to resolved env vars for runs
* First preview deploy and run working
* Set the preview branch in the main SDK
* Added git links to the preview branches table
* Better errors when replaying/testing archived branches
* Don’t dequeue archived environments
* Env var resolution with parent environment
* Hello world default machine small-2x to save my memory
* Fix for more env var functions
* Only return non-archived envs
* Switch to controlled state for the checkboxes
* Uncheck everything when PREVIEW is checked
* WIP on branch UI
* Show the preview branch label on the env vars list
* Fix for overriding env vars
* Adding preview branch env vars working
* Progress on new env vars
* Only allow selecting a single branch
* Layout fix when there are errors
* Set the defaultValue so there are some fields
* Conform fix for team invite page
* Archived environments don’t run scheduled tasks
* Added Git data to deployments
* Added git data to the deployment inspector
* Don’t allow upserting schedules when archived
* Deduplicate and blacklist some env vars
* Fix for wrong conform function being used
* Show a better error if all vars were blacklisted
* Added environment variable search (by key and value)
* Improved preview branch icon
* Replay now supports branches
* Schedule page render branches properly
* Show the env icon in bottom-left of the test page
* When editing older schedules (that have multi-env) show preview branches correctly
* Fix for incorrect disallowed branch name character
* Extract and improve the directory verification code
* WIP for CLI preview archive command
* Improved the preview branch action buttons
* Redirect to the project if we don’t find a matching env
* Archiving branch via the CLI working
* Fix for archiving branches
* Public access token test task
* JWTs working are with preview branches
* Add branch and git data to the Run ctx
* Updated GitMeta functions to work in CI
* Added pullRequestState
* Archive when deploying if the PR is closed/merged
* Fix for the changesets guide
* Fix for CLI dev bug introduced
* CLI promote now supports preview branches
* Add PR title. Reordered them and added tooltips
* syncEnvVars working with branches
* Added preview branch support to syncVercelEnvVars()
* Detect the branch from Vercel env var (set during build)
* Allow passing a branch in
* Use process.env.VERCEL_TOKEN as well… this used in Vercel CI
* Temp delete
* Improved regenerate api key modal
* Added Accordion component (with styles)
* Redesigned the API keys page
* Revert "Temp delete"
This reverts commit 177b92cd935a6161456bde65d01294e23ecfd47f.
* Changeset
* Fixed docs link
* The new branch panel closes when a branch is created
* Update apps/webapp/app/services/upsertBranch.server.ts
Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>
* Removed findUniques from WorkerGroupTokenService
* Made the parentEnvironmentId migrations safe
* Latest lockfile
* Update packages/cli-v3/src/commands/workers/build.ts
Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>
* Move isValidGitBranchName to a separate file
* Move the sanitize fn too
* removeBlacklistedVariables moved to a separate file
* Moved deduplicateVariableArray to a separate file…
* Fix broken sanitizeBranchName import
* Another import fix…
* Improved blacklisted error message
* SImplified migration to use `ADD COLUMN IF NOT EXISTS "parentEnvironmentId" TEXT`
---------
Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>
* v4: Can now set project ref using the TRIGGER_PROJECT_REF env var
* Add changeset
* Improve the name of the resolvEnvVars function and move setting NODE_ENV into the dev supervisor
* Removed moved function
* Simulate "next steps" when calculating the next timestamp to prevent millions of iterations on dev schedues
* Add additional tests
* Fix typescript issue
* shard unit tests
* temp enable for all pushes
* fix test workflow
* update to latest vitest and only add to root package.json
* additionally use default reporter
* gather reports before uploading
* split up slow replication tests
* split up unit tests workflow
* move workflows to parent dir
* use new paths in parent workflow
* prevent artifact clashes
* we always need to create the reports dir
* speed up merge reports
* gather reports even when tests fail
* fix artifact patterns
* increase shards
* disable push trigger again
* improve dequeue snapshot test reliability
* Refill release concurrency tokens when a run is cancelled
* Improved release concurrency accounting system + a sweeper to auto-refill tokens for snapshots that are no longer the latest snapshot on a run (e.g. the run has moved to a new snapshot state)
* Fix order of arguments to the releasings sweeper
* Add a heartbeat for SUSPENDED snapshots, where when stalled will attempt to continue the run if unblocked
* v4: fix race condition when continuing run when blocked at the same time
* Attempt to fix flaky e2e test
* Make waitpoint race condition test less flaky
* Upgrade posthog-node to clear axios vulns
* Upgrade @slack/web-api to use a secure version of axios
* Update parse-duration to fix security vulns
* Mitigate against the ws DoS vuln
* Upgrade body-parser to 1.20.3 in the webapp
* Upgrade react-use to 17.5.1 to remove the fast-loops transitive dep
* Remove unused babel dev deps and config file that's no longer used
* Upgrade prismjs to 1.30.0 and bundle parse-duration now
* upgrade express to 4.20.0 to fix issue with XSS when redirecting
* Upgrade @conform/zod to 0.9.2
* runs replication leader lock expiration fix
* Allow configuring the container image --max-old-space-size using NODE_MAX_OLD_SPACE_SIZE
* Ability to configure the clickhouse keep alive settings
* Add some logging because we might not be able to do telemetry
* Add external log exporters and fix missing external trace exporters in deployed tasks
* Generate the external traceID correctly and exporter 3rd party logs with the external traceID as well
* WIP clickhouse package with test containers setup
* More clickhouse client setup now with otel and real tests, and the v1 of raw run events
* Add some additional columns to raw_run_events_v1
* WIP runs dashboard service
* Create a new run engine event bus event for the runs dashboard to hook into
* Track run events in the run engine
* make sure engine v1 runs get synced to CH
* Update the attemptNumber of v3 task runs
* Restructure the run events to be more sparse
* emit more stuff
* Setup replication package
* scaffold the replication package
* replication wip
* resolve conflicts
* more replication stuff
* Add ability to drop the replication slot completely on teardown
* Use the new single replacingmergetree task events table for replication
* get it working
* insert payloads into their own table only on insert and then join
* prepare for using clickhouse cloud and now running ch migrations during boot in the entrypoint.sh
* Handover WIP and tests
* Testing the replication service
* Remove the runs dashboard stuff that we aren't using anymore
* Added a test for large payloads
* hacky typecheck fix
* Fix new internal package typecheck issues and start adding telemetry to the replication service
* tracing over spans, some other improvements
* Improvements to the runs replication service, now ready for testing
* Some fixes and cleanups
* Don't need this code anymore
* move transaction types into the runs replication service
* only send spans where there are transaction events
* A couple of suggested tweaks
* WIP adding a side menu panel to display incident statuses
* Fixes re-rendering bug and copy tweak
* cache the betterstack response using unkey
* Style the button to fit the panel colors
* improve route to add worker group, handles existing groups gracefully
* add option to remove default worker group from project
* separate project id field
* update supervisor readme with route changes
* Initial commit with a plan for what we’re going to do
* Some initial types and improved plan
* Add Waitpoint resolver
* Add resolver + status index
* Remove type + status index
* Only drop if exists
* Remove type index
* Update waitpoint list presenter to use resolver
* Added resolver to the engine
* Made the existing waitpoint list presenter more flexible
* Initial implentation ofr wait.forHttpCallback()
* Added the callback endpoint (no API rate limit)
* schema version
* Added jsdocs, removed schema version because of errors
* Show callback URL if it’s set
* Dashboard pages and panels
* Remove todos
* Added temporary icon
* Added a blank state
* Some tweaks and added a Replicate example
* Implement unwrap() for httpCallback
* Added unwrap to wait.forToken() as well
* Improved jsdocs
* Added docs
* Added unwrap to the token docs
* Show a dash if there are no tags
Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>
* Make the timeout error safer
* Fixed migrations… should use id desc not createdAt desc
* Fixed page title
* Fixed migration so it only adds them if they don’t exist. This allows us to manuall run in cloud first
* Respect the max content length by getting the length of the body
* Added more docs details about the callback format
* Remove code comment
* Improved the error
* Added a hash to the HTTP callback URLs
* Add the apiKey to the API input type to fix TS error
* Return the error responses. They were being caught and not preserved
* The content-length header is required. Deal with an empty body
* Removed unused types
* Added some new span icons
* Reworked http callback to be a create call then just use wait.forToken()
* Added a changeset
* Updated the docs
* Updated the wait overview docs
* Simplify to just a call
* WIP stripping right back to waitpoints just having a URL associated with them…
* More deletions
* Remove missing icon
* Updated the changeset
* Add URL to the token return types
* Remove wait for http callback page
* Updated docs
* More tidying
* Type and import fix
* Remove unused import
* Some type fixes for the retrieve
---------
Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>
* remove dead code
* rename managed to shared runtime manager
* rename to resolve waitpoint for clarity
* add resolver id helper
* store and correctly resolve waipoints that come in early
* fix ipc message type change
* branded type for resolver ids
* add fixme comments
* remove more unused ipc schemas
* fix entitlement validation when client doesn't exist
* restore hello world reference workspace imports
* runtime manager debug logs
* prefix engine run logs
* managed run logger accepts nested props
* runtime suspendable state and improved logs
* require suspendable state for checkpoints, fix snapshot processing queue
* add terminal link as cli module so we can more easily patch it
* apply cursor patch
* add license info
* remove terminal-link package and add deprecation notice
* remove old patch
* remove terminal-link from sdk
* rename snapshot module
* add cli test tsconfig
* add run logger base type
* add snapshot manager tests
* fix cli builds
* improve QUEUED_EXECUTING test
* changeset
* make testcontainers wait until container has stopped
* require unit tests for publishing again
* avoid mutation during iteration when resolving pending waitpoints
* improve debug logs and make them less noisy
* always update poller snapshot id for accurate logs
* detach task run process handlers
* check for env overrides in a few more places and add verbose logs
* log when poller is still executing when we stop it
* add supervisor to publish workflow
* always print full deploy logs in CI
* Revert "avoid mutation during iteration when resolving pending waitpoints"
This reverts commit 87b0ce1e5b.
* disable pre
* print prerelease script errors
* Revert "disable pre"
This reverts commit 9403409637.
* misc fixes
* better debug logs
* add snapshots since methods and route
* prep for snapshots since
* improve deprecated execution detection
* update supervisor and schema
* properly log http server errors
* detect restore after failed snapshot fetch
* run and snapshot id can be overridden
* fix restore detection
* fix deprecation checks, move into snapshot manager
* less logs
* rename snapshot manager stop
* restore detection was moved into snapshot manager
* fix notifier logs
* make runtime manager status a debug log
* no need to attach runtime status twice
* findUnique -> findFirst
* sort snapshots by created at everywhere
* update testcontainers packages
* increase cleanup timeout and add better logs
* small tweaks
* decrease docker network size so we can have more of them
* add a test flow to check this all works
* await all engine.quit calls
* add docker diagnostics
* reduce number of test runs
* improve network attachment output
* add setup logs
* log redis setup errors
* add cleanup log type
* stop redis container if setup fails
* disable ipv6
* tidy things up a bit
* fix unit tests workflow
* decrease container cleanup timeout
* fix types
* fix webapp typecheck
* make testcontainers wait until container has stopped
* require unit tests for publishing again
* add failing test case
* make it pass
* add retry threshold ms env var and use it
* Combines the Running and Concurrency limit cols into 1
* Display a badge when a queue is at the concurrency limit
* Colors the Running/Limit column text amber if the concurrency limit is hit
* Turns the “Running” big number amber and shows “At concurrency limit” text
* BigNumber now handles big values using formatNumber and formatNumberCompact
Also includes some responsive improvements to make sure things wrap when it gets tight
* Adds a new col for showing how the queue is limited
* Reinstates a threshold for making very big numbers compact
* Added border to make the search bar not float
* Adds a tooltip to display the UTC, timezone and local time for DateTime
* Made it safer for SSR
* Makes the copy function into a hook and separate button + adds copy dateTime button to the date tooltip
* Adds an extra-small size copy button
* Show the offset (UTC +1) next to your local time
* Optionally don’t show the date tooltip. Defaults to true (always show)
* Tidy imports
* Refactor the tooltip content to remove duplicated markup
* Coderabbit suggestion to include support for minutes for some timezones like Nepal
* add terminal link as cli module so we can more easily patch it
* apply cursor patch
* add license info
* remove terminal-link package and add deprecation notice
* remove old patch
* remove terminal-link from sdk
* changeset
* Made a better AI icon and improved how it looks on the create new schedules inspector
* WIP adding kapa ai to the app
* WIP adding a new Ask AI button to the side menu
* WIP using the react example from the docs
* Align the AI button in the bottom bar
* Kapa widget now works
* Trigger the Kapa modal from the custom button
* Fix imports
* Adds Ask AI shortcut to Shortcuts panel
* Adds a new enter shortcut key
* Adds a prop so you can optionally hide the shortcut key
* Latest
* Moved the Kapa/Help stuff into a component, out of root
* WIP using onModalClose
* Creates a wrapper provider to block shortcuts while kapa modal is open (has bugs)
* Fixes button alignment
* Hide the shortcut key at the button layer
* Fix for enable/disable shortcut keys globally
* Kapa is working
* You can bring up the shortcut keys without opening the help panel
* TODO remove listeners
* remove imports and fix invalid tailwind class
* style kapa widget as best i can
* Remove Kapa event listeners
* Allow passing in a query
* Open the AI widget if there’s a URL param
* Much cleaner implementation for Kapa
* Trying to auto-open the Kapa widget when the page loads
* Delay opening the widget because it was causing issues
* Improved Kapa widget colors
* Added an AI help link to the CLI
* We don’t need this anymore
* Exit with 1, indicating an error. This is important for CI
* Removed old auto-opening code
* Added some code comments to explain some of the annoying stuff
---------
Co-authored-by: James Ritchie <james@trigger.dev>
* realtime: fix streams missing chunks when streams are longer than 5 minutes and receive 408 Request Timeout errors. Also now support multiple client streams being sent to a single stream key
* Safely release the reader lock
* Fix for Schedules list page slow loading
Getting BackgroundWorkerTask was very slow (Prisma was getting every single one…)
* Same fix for the upserting of schedules in the dashboard
* Fix priority tests by dequeuing in sequence
* Expect this test to fail until we support parallel dequeuing of an env with streaming
* Fix one and expect fail from the other
* Fix for batch trigger test dequeuing
* Fixed pending version test
* Locked task runs will now require queues and tasks to be in the locked version
* Client errors caught in a run function now will skip retrying
* Extracted out the trigger queues logic
* extract validation, idempotency keys, payloads to concerns
* Extracted out a bunch of more stuff and getting trigger tests to work
* Add queue and locked version tests
* Deadlock detection WIP
* more deadlock detection
* Only detect deadlocks when the parent run is waiting on the child run
* Improve the error experience around deadlocks
* A couple tweaks to make CodeRabbit happy and fixing the tests in CI
* Fixed failing test
* Changeset
* wip
* Make sure to scope queries to the runtime env
* Added error throwing tests to runlock
* Added some more run lock tests
* Added customConditions so the test can use the Logger
* Fix for JSON error
* Added a span to the run lock with a name
* Increase v4 visibility timeouts
* Added some logging for #continueRunIfUnblocked: run has no checkpoint
* If a run is finished or pending cancel, don’t try and continue it
* Remove setting the invisibility timeout because it’s already done in the dequeue Lua script
* Remove orphaned queue items when dequeuing
* Added an ack to the visibility timeout test
* refactor: docker compose migration
* fix compose download link
* set static name for electric container
---------
Co-authored-by: nicktrn <55853254+nicktrn@users.noreply.github.com>
* Add a custom react hook to enable fuzzy searching in item lists
* Use fuzzy filtering in the tasks view list
* Use fuzzy filtering in the test tasks list
* Remove the old tasks filtering react hook
* Allow empty payloads in test runs
* Expose a basic linter configs in the code editor component
* Touch up the linting error elements in the code editor component
* Added some Redis worker debounce tests (one failing that reproduces a bug)
* Added some tests for acking
* Added a deduplicationKey to prevent acking when items are queued
* The worker passes the deduplicationKey back in for acking
* Improved logs and removed events from test
* Fixes runLimiter check on #dequeueRuns
This was causing issues in the dev environment with logs being spammed with `[DevSupervisor] dequeueRuns. Run limit reached, trying again later` and making the worker unresponsive and lose connection.
* Create polite-impalas-care.md
---------
Co-authored-by: Eric Allam <eallam@icloud.com>
* multiple queue consumer in the same supervisor instance
* disable pre
* Revert "disable pre"
This reverts commit 4b154398982758d09667531e3a64761da83664eb.
* Completed batch waitpoints when we completed the BatchTaskRun
* Try complete the batch faster now it’s being used operationally
* Fix for tests that were using the old engine.unblockRunForCreatedBatch() function
* WIP on secret env vars
* Editing individual env var values is working
* Sort the env vars by the key
* Deleting values
* Allowing setting secret env vars
* Added medium switch style
* Many style changes to the env var form
* “Copy text” -> “Copy”
* Draw a divider between hidden buttons
* Env var tweaks
* Don’t show Dev:you anymore
* Grouping the same env var keys together
* Styles improved
* Improved styling of edit panel
* Fix bun detection, dev flushing, and init command (#1914)
* update nypm to support text-based bun lockfiles
* add nypm changeset
* handle dev flushing failures gracefully
* fix path normalization for init.ts
* add changesets
* chore: remove pre.json after exiting pre mode
* init command to install v4-beta packages
* Revert "chore: remove pre.json after exiting pre mode"
This reverts commit f5694fde9314114c74a220c2213d19667bca1a6c.
* make init default to cli version for all packages
* Release 4.0.0-v4-beta.1 (#1916)
* chore: Update version for release (v4-beta)
* Release 4.0.0-v4-beta.1
---------
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: nicktrn <55853254+nicktrn@users.noreply.github.com>
* Both run engines will only lock to versions they can handle (#1922)
* run engine v1 will only lock to v1 deployments
* run engine v2 will only lock to managed v2 deployments
* test: create background worker and deployment with correct engine version
* Add links to and from deployments (#1921)
* link from deployments tasks to filtered runs view
* jump to deployment
* don't add version links for dev (yet)
* Fix current worker deployment getter (#1924)
* only return last v1 deployment in the shared queue consumer
* be explicit about only returning managed deployments
* Add a docs page for the human-in-the-loop example project (#1919)
* Add a docs page for the human-in-the-loop example project
* Order guides, example projects and example tasks alphabetically in the docs list
* Managed run controller revamp (#1927)
* update nypm to support text-based bun lockfiles
* fix retry spans
* only download debug logs if admin
* add nypm changeset
* pull out env override logic
* use runner env gather helper
* handle dev flushing failures gracefully
* fix path normalization for init.ts
* add logger
* add execution heartbeat service
* add snapshot poller service
* fix poller
* add changesets
* create socket in constructor
* enable strictPropertyInitialization
* deprecate dequeue from version
* start is not async
* dependency injection in prep for tests
* add warm start count to all controller logs
* add restore count
* pull out run execution logic
* temp disable pre
* add a controller log when starting an execution
* refactor execution and squash some bugs
* cleanup completed docker containers by default
* execution fixes and logging improvements
* don't throw afet abort cleanup
* poller should use private interval
* rename heartbeat service file
* rename HeartbeatService to IntervalService
* restore old heartbeat service but deprecate it
* use the new interval service everywhere
* Revert "temp disable pre"
This reverts commit e03f4179de6a731c17253b68a6e00bcb7ac1736b.
* add changeset
* replace all run engine find uniques with find first
* Release 4.0.0-v4-beta.2 (#1928)
* chore: Update version for release (v4-beta)
* Release 4.0.0-v4-beta.2
---------
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: nicktrn <55853254+nicktrn@users.noreply.github.com>
* Remove batch ID carryover for non-batch waits (#1930)
* add failing test case
* do not carry over previous batch id when blocking with waitpoint
* delete irrelevant test
* Delete project (#1913)
* Delete project
- Don’t schedule tasks if the project is deleted
- Delete queues from the master queues
- Add the old delete project UI back in
* Mark the project as deleted last
* Fix for overriding local variable
* Added a todo for deleting env queues
* Remove todo
* Improve usage flushing (#1931)
* add flush to global usage api
* enable controller debug logs
* initialize usage manager after env overrides
* add previous run id to more debug logs
* add changeset
* For secret env vars, don’t return the value
* Added a new env var repository function for getting secrets with redactions
* Test task for env vars
* Delete heartbeat file, merge mess up
---------
Co-authored-by: nicktrn <55853254+nicktrn@users.noreply.github.com>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: Saadi Myftija <saadi.myftija@gmail.com>
* drive run heartbeats from child process
* track last heartbeat
* add changeset
* use v4-specific heartbeat interval env var and decrease to 20s
* delete old heartbeat service
* remove heartbeat handler during execution cleanup
* add flush to global usage api
* enable controller debug logs
* initialize usage manager after env overrides
* add previous run id to more debug logs
* add changeset
* Delete project
- Don’t schedule tasks if the project is deleted
- Delete queues from the master queues
- Add the old delete project UI back in
* Mark the project as deleted last
* Fix for overriding local variable
* Added a todo for deleting env queues
* Remove todo
* update nypm to support text-based bun lockfiles
* fix retry spans
* only download debug logs if admin
* add nypm changeset
* pull out env override logic
* use runner env gather helper
* handle dev flushing failures gracefully
* fix path normalization for init.ts
* add logger
* add execution heartbeat service
* add snapshot poller service
* fix poller
* add changesets
* create socket in constructor
* enable strictPropertyInitialization
* deprecate dequeue from version
* start is not async
* dependency injection in prep for tests
* add warm start count to all controller logs
* add restore count
* pull out run execution logic
* temp disable pre
* add a controller log when starting an execution
* refactor execution and squash some bugs
* cleanup completed docker containers by default
* execution fixes and logging improvements
* don't throw afet abort cleanup
* poller should use private interval
* rename heartbeat service file
* rename HeartbeatService to IntervalService
* restore old heartbeat service but deprecate it
* use the new interval service everywhere
* Revert "temp disable pre"
This reverts commit e03f4179de6a731c17253b68a6e00bcb7ac1736b.
* add changeset
* replace all run engine find uniques with find first
* run engine v1 will only lock to v1 deployments
* run engine v2 will only lock to managed v2 deployments
* test: create background worker and deployment with correct engine version
* update nypm to support text-based bun lockfiles
* add nypm changeset
* handle dev flushing failures gracefully
* fix path normalization for init.ts
* add changesets
* chore: remove pre.json after exiting pre mode
* init command to install v4-beta packages
* Revert "chore: remove pre.json after exiting pre mode"
This reverts commit f5694fde9314114c74a220c2213d19667bca1a6c.
* make init default to cli version for all packages
* WIP on hiding the queued time
* Fix for position/sizes of spans
* Fix for duration squashing
* Tweaked some styles on the timeline
* Fix for retry spans appearing in the wrong place
* Added “Q” shortcut key
* Fix for in progress span durations
* WIP upgrade to v4 docs
* WIP
* Upgrade to v4 docs
* Add wait for token docs and improve upgrade guide
* Added known issues to upgrade guide, and moved the migration section higher up
* Update to say "There are no known issues at the moment.”
* Docs upgrade guide package version. Need to use exact packages for the CLI to work
---------
Co-authored-by: Matt Aitken <matt@mattaitken.com>
* attach all run controller logs to the run
* make controller-level pre-suspend wait duration configurable
* snapshot status should remain EXECUTING for short retry delays
* add suspended process error
* ensure clean slate before waiting for next run
* treat immediate retries as warm starts
* fix for finished runs waiting forever
* Use completedAt instead of updatedAt
* Disable verbose SSE logs
* Engine rate limiter: whitelist engine worker-actions
* Show the compute column in the runs table
* The compute column should be invocation + compute cost
* don't hardcode max old space size in deployment images
* flags: treat underscores as hyphens
* append attempt number to runner name if >1
* improve retry spans for oom errors
* Make sure releaseConcurrencyOnWaitpoint is set on taskQueue
* Allow task queue to have releaseConcurrencyOnWaitpoint (required to fix custom queues)
* Test for releaseConcurrencyOnWaitpoint
* Added a warning if you use the same queue twice with different settings
* Improved the error some more
* set correct worker name on create
* dev: kill run process on failed cancel attempt
* add missing prod timeline events
* add more debug logs after api call failures
* import events are always visible
* set dequeued at on new runners
* skip fork event test
* fix timeline test for import event
* Better trace icon
* Better Waitpoint token icon
* Fix for bad jsx
* Warm and cold start icons
* Tooltips now use a <Portal> so they appear on top
* Warm start components
* Added warm start markers to the Run page and inspector
* Fix for getting the correct value from the metadata
* Better trace icon, with fallback to a passed in one
* Removed unused isWarmStart function
* make worker nodetype configurable
* make max dequeue count configurable from supervisor
* deterministic runner ids
* update runner id on restore
* disable resource monitor
* explicit run controller type imports
* enhance debug logs
* run engine: add ability to selectively disable release concurrency queue consumers via env var
Also added some additional logging
* actually pass the disableConsumers option through to the release concurrency system
* Fixed CLI e2e failing tests
* v4: eagerly fork child process before warm start
Also fixes an issue where the attempt span events weren't coming through in the partial spans
* Making flushing a safer operation
* Add changeset
* Improve attempt metrics when in a warm start, and pass the STYLE_VARIANT for a warm or cold start on attempt spans
* Improved the react email docs and added a video
* Copy
* Added a link to the cursor rules
* Added satori example
* Copy updates
* Added react pdf link
* remove dummy metrics from heartbeat
* fix heartbeat timeouts..
* reset clock when resolving waitpoints
* optionally set metadata url when scheduling run
* don't use global vitest as it's outdated
* don't run test files in parallel
* handle special graceful shutdown code
* exit codes for success and failure can now be set and overridden
* ensure immediate cleanup in tests
* update lockfile after conflict
* AI SDK sql generator and approval example WIP
* Create some nice spans and extract the tools out so we can get the type of the streams
* Enable AI SDK telemetry
* Adding ai.tool and removing toolTask, 3rd party telemetry spans now wil create partials, better ai SDK telemetry icons
* Created a separate d3-chat example, split out from the openai-agents example
* Fixed the tool options being passed to metadata in `ai.tool`
* Add a link to the run
* Slightly improved design
* Add a crawler task using crawl4ai
* Use a tool to get the userId
* Couple of tweaks
* Adding markdown rendering to assistant messages and added an e2b based chart rendering task
* Backup to anthropic
* Add changeset
* Delete old wait-for pages & create new wait-for-token page
* Link to new wait-for-token page and update description
* Adds an upgrade to v4 coming soon docs page
* Removes unncessary property from Note component
* Adds DevPresenceBanner to the Run page
* Adds DevPresenceBanner to the Run list page
* Makes the DevConnection component reusable and moves components to the DevPresence.tsx file
* SideMenu: only show dev presence when project engine === V2
* WIP on disconnected banner on v3 and v4
* v3 dev connection working. Made it slightly red
* Only show the disconnected banner on v3 if the run is not completed
---------
Co-authored-by: Matt Aitken <matt@mattaitken.com>
* logs for optional services
* print env vars on startup in debug mode
* routes need to explicitly ask to keep connection alive
* log indicators for now
* make workload api listen host configurable
* expose supervisor metrics and make more configurable
* configurable pull secrets, no defaults
* remove restore route
* run controller to handle queued executing
* fix v3 deploys in v4 project
* update admin worker route
* only start pod cleaner et al in k8s mode
* set new worker group as default if none yet
* make image ref optional
* checkpoint image ref is optional for output as well
* export feature flag const
* one last image ref type fix
* make runner intervals configurable
* ability to set arbitrary env vars on new runners
* set default runtime back to node 21
* move all runner env vars to the same section
* cli: add dev lock file to prevent 2 dev processes running at the same time in the same dir
* Make sure the .trigger dir exists before creating the dev.lock file
* logs for optional services
* print env vars on startup in debug mode
* routes need to explicitly ask to keep connection alive
* log indicators for now
* make workload api listen host configurable
* expose supervisor metrics and make more configurable
* configurable pull secrets, no defaults
* remove restore route
* run controller to handle queued executing
* fix v3 deploys in v4 project
* update admin worker route
* only start pod cleaner et al in k8s mode
* set new worker group as default if none yet
* Ensure the task list uses the dev environment of the logged in user
* Ensure the test task action uses the correct dev environment
* Fixed issue with missing case statement scope, wrapping in block
* Updates the large login buttons to the new purple style
* Updates the dev checking connection icon to include 3 dots on the screen
* Fixes svg property names
* Removed old condition that handled old v2 tables
* Adds support for a single button to be shown on hover in a table cell
* Makes the “hiddenButtons” support a single button instead of adding a new property
* Format the docs link for Schedules like the other for consistency
* D3 demo WIP
* Agent working and completing token successfully
* Sending OpenAI Agent SDK spans through to the platform now works
* A couple of perf tweaks for enriching events
* New time period filter (permanently displayed)
* Batches, and fix for blank state
* Waitpoint token filtering
* Tags query: remove things we’re not using
* Run tag and waitpoint tags use startsWith for faster search
* Fix for run page on wrong env. Added schedule last triggered column
* Removed the Redis pubsub, just use the presence key
* Improve the dev presence responsiveness
* The CLI presence connection recovers when the webapp is restarted
* Dev schedules are now working for v4
* Refactored to make the dev presence stuff
* Got rid of stupid extra /dev, added a connecting state with icon
* Remove unused Redis client
* Move to indigo primary button to better fit in with the app UI style
* Updates Pause queue button to secondary
* Updates some blank state CTA buttons to primary
* WIP
* Completing waitpoints using public access tokens
* fixed example
* adding agent-loops reference, adding useWaitToken react hook, adding a way to throw an error to output from the run, adding an icon option to logger.trace
* Fixed metadata system
* Check if the waitpoint is COMPLETED already in the complete endpoint and return true
* New format for the functions instead of “onWait() task”
* Make the icon sizes match the hero icons
* Updating icons for the new task hooks
* Adds new icons to the RunIcon.tsx component
* Improves the size of the menu icons
* Fixes the missaligned environment dropdown and dev connected button
* Changes the button colours so they are all unique
* More icon updates
* Adds new variables for main page icons
* Improves icons sizes
* Using new color variables in the side menu and blank states
* Adds preview environment color variable
* Align the env icon in the menu
* Updates the env icons to be the correct size and padding to match hero icons
* Label uses new colors, removes unused cases and new env icons
* Lower case env label
* Removes unused code
* Use full env title
* Organize imports
* More variants of the runs icon to work at smaller sizes
* small padding adjustment
* reformat init function span
* Fix last init formatting
* New colour for settings
* Organize switch statement for icons nicely
* Use new color variable
* Update icon svgs
* Renamed lifecycle hook icon to Function icon
* Renamed function icon file name
* Added waitpoints/tokens to the sidebar
* Added indexes to the Waitpoint time for filtering
* Begun work on `WaitpointTokenListPresenter`, the pag is a copy of the Queues page for now
* MVP of waitpoint token page
* Added status
* Expiry of timeout/ttl
* Improvements to the waitpoint table
* Improved columns and icon
* Changes from the RunTag copy on hover branch
* Fix for nested button error
* Added waitpoint tags to the DB/table
* Applied Eric’s task run tag fix (it’s live on prod in the legacy run engine branch)
* Added tags to waitpoints
* Removed todos that have been done
* Added token support for releaseConcurrency. Also added a ton of JSDocs
* Added releaseConcurrency to the API token endpoint…
* WIP on waitpoint page filters
* Fix for tags filtering
* Waitpoint filters working
* Fix for badly named function
* WaitpointPresenter used from SpanPresenter
* Waitpoint detail panel WIP
* Fix for server client hydration issue with CodeBlock
* Selected waitpoint panel
* Added a blank state
* Added waitpoint docs link
* Fix for animated number going past the target
* Fix for the queue list pagination and upgrade status
* Engine version error for waitpoint token list
* RunTag component doesn’t get squished and hover behaviour is nicer
* Associating runs with waitpoints
* Added triggered icon
* Link directly to the waitpoint
* Fix for TS error on waitpoint retrieve
* Added CopyableText component, used for waitpoint id in the table
* Removed the confetti 🎊
* Deleted some old images
* Moved some schemas/types to core. Use `id` instead of `friendlyId`
* Added wait.listTokens() function. Made some changes to the types to make it nicer
* WIP wait.retrieveToken()
* wait.retrieveToken working
* Added data to retrieve token
* Separate ApiWaitpointPresenter completely
* Added completed time to the waitpoint detail panel
* Fix for the Avatar component having SSR issues. Specify the size in rems and removed the useLayoutEffect
* Fix for applied idempotency key filter dropdown showing the id field
* Use parentheses to make sure the token list query respects idempotency key correctly
* Use the proper logger, and have a decent message with info to track the bug down
* Pass the org title into the Avatar
* Better error when failing to creating a manual waitpoint after X attempts
* Improve the new run engine Trigger/Batch trigger service class names
* centralize queue timestamp logic in EnqueueSystem, adding queueTimestamp support and propagation
* Fixing the create tags in the new run engine trigger task service
* patch k8s client to allow field selector on informers
* add pod cleaner and tests
* add failed pod handler and tests
* remove supervisor test script for now
* Adds “plan” to the plan name if it’s a paid plan so they all format correctly
* Fixes menu item having incorrect height
* Adds gap between switch org menu items
* Moves main action button to right side in blank state
* Simpler way to add “plan” to the plan name
* Styled the plan badge a bit nicer
* Project page now matches the other settings page layouts
* Fixes colour of queue icon
* Fixes table blank state not spanning enough cols
* Organize imports
* Unified the blank state panels and improved some copy in them
* Improved the copy in the no deployed tasks blank state
* Make the upgrade icon solid and indigo
* Transition the hover state on the side menu section
* Replaces many props with a single ‘accessory’ React node on the Info Panel component
* Organize imports
* Organize imports
* Removes animation on the dropdown menu icon
* Improves Settings and Usage button states
* Side menu sections animate with easeInOut
* Incorporates docs button into the text and adds title to the clipboard field
* Icons look better as the small variant
* Improved the typography layout of the org name and small links
* Made the divide line show when the side menu scrolls
* Made the impersonating icon amber so it stands out more
* Adds an optional amber border to the org dropdown
* Delete the proxy app (was v2)
* Delete RunPresenterElectric
* Select the best proj/org/env
* Storing current proj/env in DB. Initial selection logic working with tasks page
* 2sm needed to be in the Tailwind merge list
* Move the task stream route (although we don’t actually use the env for now)
* Alerts moved from /v3
* API keys page moved from /v3
* Concurrency page moved from /v3
* WIP on side menu sections
* Improved the accordion animation
* Moved schedules from /v3
* More pages moved
* Move pages working
* Run page working
* Schedules working
* Moved deployments
* Alert pages moved
* Delete electric hooks, not used
* Started setting up blank states
* Test page working
* Removed “Select task” from the test page
* Some work on deployment page
* Style tweaks
* Redirect from project root to approriate env
* Improved env selector styling
* Fix for jsx errors
* Better min width on env selector
* Improved the env switching logic
* Added deployments to env routing
* Redirect deployments to the correct env
* Redirect run from proj to env
* JSX icon fix
* Only allow single env schedules from now on
* Remove env var count from the API keys page
* Move improvements and redirects
* Project settings moved
* Fix for scroll area on test page
* Tweaked the test design
* Made recent payloads column narrower
* Improved the test layout some more
* Added org icon, new project selector menu
* WIP on org switching menu
* Org switching is working
* New menu working well, removed old side menu items
* Buttons can now have a component name or an actual component for their icons
* Removed the Projects page, instead redirect appropriately
* Fix for broken blank states
* Minor run table improvements
* Removed unused switcher log and logic
* Concurrency page fix for invalid html, improved layout
* Minor improvements
* Moved the side menu to the project level
* Improved account styling
* Moved org settings pages (with redirects)
* Add current plan to billing side menu link
* Upgrade to get staging from env dropdown
* New env badge on concurrency limits page
* Show Run Engine version in span presenter
* New promote icon
* Concurrency limits page is the sum of engine v1 + v2 queues
* Fix for missing batch import
* Added currentConcurrencyOfEnvQueue function
* Basic avatar setting working
* Avatar setting is working
* You can change the color of your icon
* Avatar improvements
* Bugfix for mising prop
* Removed some old env badges
* Fixed replaying
* Removed EnvironmentLabel
* Old env badge deleted, changed everywhere to the new one
* Fix for Slack integration paths
* Fix for waitpoint completion form moving
* Bulk replay/cancel env fix
* Fix for alert webhook path
* Redirect projects/v3/* to project/*
* Fixes for CLI redirect routes
* Remove welcome email (unused)
* Change how we count schedules towards your limits
* Use new schedules limits when checking a schedule
* Added projectId back in to task queries (indexes)
* WIP dev presence
* CLI modal
* Moved things around and use Context
* Fix for p inside p
* Dev connected status on run page
* Correct dev env (not a teammates)
* Show disconnected message at the end
* Minor tweak on project dropdown icon padding
* Fix for inconsistent date format for presence
* Added a message when pushing to the billing page
* Center the team page
* Project settings page centered
* Improvements to the dev presence
* handle warm start service failure on supervisor side
* export zodfetch wrapper
* add warm start client
* rename to keepalive
* add missing warm start header
* make heartbeat and snapshot poll interval configurable
* create warm start client in constructor
* add warm start run debug log
* re-enable checkpoints and improve error messages
* reduce run pod container name cardinality
* move runner id generator into core
* fix runner id import
* log when no checkpoint client and we try to restore
* move run controller constructor to the top
* fix import
* support env overrides after restore
* fix and restructure dockerignore
* switch to using pnpm deploy
* pass webapp node image as build arg
* ensure pnpm is downloaded at build, not runtime
This just adds the main components for the MCP server. It hooks on the
existing `dev` command and can be enabled by passing the `--mcp` flag.
Currently only the `trigger-task` tool is exposed, which enables users
trigger tasks via MCP and see the resulting run.
Still WIP :)
* Delete v2 Stripe routes
* Delete v2 billing/usage pages
* Delete v2 integration pages
* Delete v2 project pages
* Deleted a load of components and services
* Deleted a load more components, presenters and services
* Deleted another 100 files or so…
* Removed old v2 paths
* Removed named icons from form titles
* Removed more string icons
* Delete NamedIcon
* Fixed some type errors
* Delete endpointApi
* Removed v2 from core/sdk
* Post merge fixes
* added explicit return types
* using the new sdk export without v3
* Delete old v2 file
* Added explicit return types because TS was complaining…
* Don’t export RuntimeEnvironmentType from two core files. Was causing TS issue
* Fix for removal of NamedIcon in new route
* Removed strange eslintrc rule
* Use the new redis client
---------
Co-authored-by: James Ritchie <james@trigger.dev>
* re2: fix @trigger.dev/core exports
* re2: WIP env based queue selection algo
* more wip
* WIP
* Get run engine tests to pass
* Adding tests for the fair dequeueing strat in the run engine
* Configure the new queue selection strategy in the webapp and get it all building and typechecks passing
* webapp now uses built packages, building redis-worker, run-engine, database, using better tsconfig setups for tests, moving isomorphic code into core/v3/isomorphic
* Fixed webapp typechecks
* dev now depends on build, fixed supervisor typecheck
* Fixed run engine tests
* Fixed e2e tests
* dockerignore node_modules in subdirectories
* image tag action should handle re2 tags
* add supervisor containerfile
* add publish worker re2 workflow
* fix copypasta
* require branch check
* add more granular service control to supervisor session
* fix supervisor api domain for split setups
* remove default workload api domain
* option to disable workload api
* fix bool env var coercion
* Added describe to tests that were missing it
* Added a function to get the maxOldSpaceSize
* Make it easy to take `NODE_OPTIONS` and set the old space flag
* Added a zed task to rebuild the packages
* Moved isOOMRunError and added SIBABRT condition
* Deduplication flags function with tests
* Export flags file
* On TaskRunProcess, set max old space and deduplicate the flags with priority order
* Move retrying logic to a separate function, it was getting very messy
* Created new test file for attempt failures
* Allow setting retry settings for tests
* Some retrying tests, including OOM
* More failure condition tests
* Fix for OOM retrying
* Complete the attempt span if it was an OOM error
* Remove old broken import
* Fixed order of exports
* Fix for docker-provider checkpoints import
* Adds “Explore by build extension” to the intro page
* Adds instructions for clearing the build cache
* Add a tip for running trigger dev + next dev concurrently
* left align table headers
* Adds trouble shooting for pino
* Adds corepack bug and workaround
* Removes reference to undici as it messes with spans
* Left align table headers
* Adds example for exporting logs to Axiom
* In dev, the worker group is optional when triggering tasks (the master queue is defined by the environment). Also deprecated the TaskEvent.isDebug column and using TaskEventKind.LOG instead for debug events
* Fixed a couple of type issues
* More type fixes
* bump worker version
* Suggested glossary for the RunEngine, TBC
* Removed BatchTaskRun changes from this branch, they were done in main
* Set the BatchTaskRun status to completed when all runs are completed
* When dequeuing respect passed in maxResources
* Ported over the new run props: idempotencyKeyExpiresAt, versions, oneTimeUseToken, maxDurationInSeconds
* Didn’t hit save… the new props when triggering tasks passed through
* Idempotency expiration + waitpoint edge case
* WIP on creating checkpoint, parking for now
* fix worker routes
* upgrade webapp node types to support generic event emitter
* separate event bus handler singleton and run failure alerts
* duration waits
* fix execution snapshot debug spans
* task waits
* fix event bus types
* temporary fix for react hook run handle type
* disable run notifications for now
* convert any typecasts to expect errors to more easily fix later
* fix webapp types after node types upgrade
* updateEnvConcurrencyLimits across marqs and the runqueue
* Pass proper values into the run engine
* RunQueue settings and removed unused rebalancing workers
* Remove rebalancing prop
* Tidied more things up
* Update/remove queue limits for MARQS and RunQueue
* taskQueue/concurrencyLimit changes ported back into the RunEngine
* Reworked completing waitpoints to improve performance and reduce race conditions
* Improved test robustness
* Down to a single run lock only when a run is totally unblocked and ready to continue
* warm starts, worker notifications, wait fixes
* Fix for Run Engine poll interval env var
* Expect the waitpoint to be completed quickly
* If a run is locked then it’s too late to expire it
* Added VALKEY_ env vars and plugged them into the run engine
* Extracted and updated the guard queue function so it can be used when batching
* Added logging and universal concurrency changes to trigger task v1
* Added notes back in
* Bump @trigger.dev/worker to 3.3.7
* reportInvocationUsage for the runAttemptStarted event
* improve execution snapshot span debug span start times
* Unfriendly IDs
* update lockfile
* Created a shared determineEngineVersion function
* disable unfinished commands
* save new cli config to different location, misc fixes
* add basic engine version check via current deploy
* new run engine will default to node 22 runtime
* block some actions for projects on previous run engine
* fix worker group tests
* fix triggerAndWait test
* one typescript version to rule them all
* redlock type patch
* fix type issues caused by ts-reset
* improve cleanup scripts
* add missing socket.io dep
* fix run notification handler type
* fix worker group test again
* generate prisma client for e2e tests
* remove worker group tests for now
* prevent image pull rate limits during unit tests
* increase timeout for queue concurrency limit test
* generate prisma client for preview release
* same node types everywhere
* Updated engine readme, removed legacy system notes
* use default machine preset from platform package
* worker instances plural in schema
* disable pnpm update notifications
* return worker group details from connect call
* add workers admin route
* fix heartbeat route return type
* move deployment labels to core apps
* refactor run controller env schema
* Add firstAttemptStartedAt to TaskRun
* RunEngine 2.0 batch trigger support (#1581)
* Make it clear when BatchTriggerV2Service is used
* Copy of BatchTriggerV2Service
* WIP batch triggering
* Allow blocking a run with multiple waitpoints at once. Made it atomic
* Removed unused param
* New batch service
* Pass through the parentRunId and resumeParentOnCompletion
* Use the new batch service, and correct trigger task version
* Force V1 engine if using BatchTriggerV2Service, we’ve already done the check at this point
* Removed the $transaction and early exit if nothing changed
* Adedd a simple batch task to the hello world reference catalog
* Fix for batch waits not working
* Added parentRunId in a couple more places
* Removed waitForBatch log
* Added another parentRunId
* Expanded the example to include all the different triggers
* More changes to blocking to support continuing after idempotent completed runs
* Fix for the wrong type when blocking a run
* remove @map
* optimise worker auth query
* add engine version header to core api client requests
* remove unique constraint for default group id
* consolidate migrations
* the first managed worker becomes the global default
* Debug events off by default, added an admin toggle to show them
* worker group name can't be an empty string
* add exec helper to core
* move machine resources to core
* add pre-dequeue callback to determine max resources
* optionally skip dequeue
* bump worker package
* move worker to core
* fix ReadableStream type error
* fix another type issue
* update a few more tsconfigs
* add metadata changes introduced in #1563
* Run Engine 2.0 trigger idempotency (#1613)
* Return isCached from the trigger API endpoint
* Fix for the wrong type when blocking a run
* Render the idempotent run in the inspector
* Event repository for idempotency
* Debug events off by default, added an admin toggle to show them
* triggerAndWait idempotency span
* Some improvements to the reference idempotency task
* Removed the cached tracing from the SDK
* Server-side creating cached span
* Improved idempotency test task
* Create cached task spans in a better way
* Idempotency span support inc batch trigger
* Simplified how the spans are done, using more of the existing code
* Improved the idempotency test task
* Added Waitpoint Batch type, add to TaskRunWaitpoint with order
* Pass batch ids through to the run engine when triggering
* Added batchIndex
* Better batch support in the run engine
* Added settings to batch trigger service, before major overhaul
* Allow the longer run/batch ids in the filters
* Changed how batching works, includes breaking changes in CLI
* Removed batch idempotency because it gets put on the runs instead
* Added `runs` to the batch.retrieve call/API
* Set firstAttemptStartedAt when creating the first attempt
* Do nothing when receiving a BATCH waitpoint
* Some fixes in the new batch trigger service… mostly just passing missing optional params through
* Tweaked the idempotency test task for more situations
* Only block with a batch if it’s a batchTriggerAndWait… 🤦♂️
* Added another case to the idempotency test task: multiple of the same idempotencyKey in a single batch
* Support for the same run multiple times in the same batch
* Small tweaks
* Make sure to complete batches, even if they’re not andWait ones
* Export RunDuplicateIdempotencyKeyError from the run engine
* Latest lockfile
* Trigger with a machine (old run engine)
* RE2, allow setting machine when triggering
* Fix for new glob patterns
* add max run count to dequeue from version route
* add worker instance name env var and header
* queue consumer pre skip callback
* poll for more runs after final execution errors
* fix dequeue search param schema
* add shortcut to debug switch
* expose run engine timeouts as env vars
* make warm start durations configurable
* add optional status to json reply helper
* fix preSkip hook, add debug logs
* BLOCKED_BY_WAITPOINTS -> SUSPENDED
* exit controller when run suspended
* check if already replied before http reply
* run controller will wait for next run after the current one is suspended
* cancel run button shortcut
* minimal event repository environment type
* fix update metadata call
* run suspension and misc fixes wip
* change debug shortcut to shift + D
* Started work on the Dev supervisor
* Formatting
* Fix for bad imports
* Before rebuilding SSE
* Presence updating from the CLI working via SSE
* add worker notification debug logs
* send run:stop when exiting run phase
* skip current snapshot poll on worker notification
* add more logs and route to submit run debug logs
* add worker and runner ids to snapshots
* improve run notification debug logs
* add workload debug log route
* misc run controller fixes and refactor
* prevent parallel execution of critical functions
* update bun to 1.2.1
* WIP with dev dequeuing
* Method to convert friendlyIds to non-friendly, do nothing with actual ids
* Set the engine on BackgroundWorker, lazily upgrade projects to engine V2
* Runs with ttls were getting immediately expired… oops.
* Pass the Waiting for deploy reason through, so we have it on the execution snapshots
* Fixed the logic for getting the right background worker for a run
* Use the correct ID when dequeuing…
* determineEngineVersion is now fully functional
* Rate limiter ignores the dev endpoints
* Retrieving a batch gives you the runIds
* Set a unique version for the RE2 BatchTaskRun
* add provisional changeset
* The start of dev run execution is working
* First dev run working
* Moved the dev run controller closer to what Nick did with the managed one
* export exec output type
* Heartbeat fix: don’t heartbeat if _isHeartbeating == false
* Dev runs get notifications, some dev bug fixes
* Improved logging or dequeuing
* We need to dequeue runs from the latest version too, for triggerAndWait
* Ported Eric’s validateWorkerManifest with nicer errors
* When flattening an idempotency key if part is undefined, return undefined
* Dev logging fixes
* Remove sigterm listener
* Deprecating workers. Don’t specify a BackgroundWorker when dequeuing an environment
* Deleted some old files. Renamed “managed” to “deploy”
* When a build finishes, always copy the build dir (otherwise the first one gets trampled on by the 2nd)
* Dev master queues should work differently
* Deleting old workers
* Added debounce function to core
* Improvement to canceling
* WIP on debounce canceling on socket disconnection
* Added environment data to execution snapshots
* Dev runs that have stalled get “Canceled” with a reason explaining why
* Show CLI messaged when a connection to the platform is lost/restored
* Fix TriggerTask after merge
* Add trigger task v2 max attempts, replace some findUniques
* Port the new queue logic to the run engine
* More fixes post-merge
* We weren’t setting a `retryConfig` up for the tests… it’s now required
* Start the Redis worker inside the Run Engine… 🤦♂️
* Trying to make the testcontainers more reliable
* Added keyPrefix: "engine:”
* Badly placed bracket in trigger task
* Better Redis namespacing
* Fix for expired run not getting removed from the queue
* Don’t create a redis client in the testcontainers, return the redisOptions instead
* Cleanup redis client in the run lock tests
* Fix for the RunQueue not supporting keyPrefix
* Updated more of the RunQueue scripts rebalancing
* Trying to make Redis more robust in the tests…
* Improved test resiliciency more
* Fix for delays (checkpoint check)
* Increase the timeout slightly to fix ttl test
* Added priority support when triggering
* More wip trying to make test containers more reliable
* batchTriggerAndWait test is still failing… some wip to try fix it
* Fixed redis tests now we’re not providing a client
* Separate Redis clients for the run engine worker/queue/runlock
* Made the wait for duration test more resilient
* Added idempotencyKeyExpiresAt to Waitpoints
* Waitpoint timeouts and idempotency expiry
* Use finishWaitpoint, removed extra worker job
* Added waitpoint idempotency tests
* Creating resume tokens is working
* Some improvements to the resume tokens
* Moved resumeTokens to just be wait functions 🥳
* Delete old RuntimeManagers
* Wait for token is working
* Better test for the wait tokens
* Improved the test task some more
* Hide the accessories in the span inspector
* WIP on waitpoint inspector
* WIP on complete waitpoint form
* Span overview panel can be changed based on the entity type
* Improved the waitpoint display
* WIP on completing waitpoint form
* Use the existing CodeBlock for the tip
* Style improvements
* Complete waitpoint
* All waitpoint sidebar variants
* Waits now use a pause icon
* Durations waits use the API to create/block with a waitpoint, not the runtime
* Fix for engine.blockRunWithWaitpoint required org id
* Removed old wait code from the run controllers/task run process
* Form action for skipping a datetime waitpoint
* Move testDockerCheckpoint to a separate core package export (it can’t be bundled on the client)
* Fix for glitchy hourglass animation
* Completed waitpoints display better
* Increase Redis maxRetriesPerRequest to 20 (default)
* Completing and skipping waitpoints is working
* Remove the database prisma dev command, since we need to use create only now. Updated docs
* Added skip timeout, reworked the UI
* Tweaked spacing
* Added payload limit to waitpoint token completion from dashboard
* Test idempotency works on wait.for and wait.until
* Moved the worker-actions to /engine/ from /api/
* Moved dev engine endpoints to /engine/ from /api/
* Separate /engine/ rate limiter
* Added parallel wait prevention, it’s working for duration waits but not well for triggerAndWait yet
* WIP post-merge conflicts
* Set taskEventStore column in the new engine
* Remove duplicate keys
* Post-merge fixes
* Fix for span merge layout
* Use executedAt instead of firstAttemptStartedAt
---------
Co-authored-by: Matt Aitken <matt@mattaitken.com>
* Removed unnecessary classes from Dialog
* Display large modal with code and copy button
* Fixes button position if chrome is shown
* Removes the duplication of the Highlight component
* Record cold start and execution metrics on attempt executions. Add cold start metrics as span events on attempt spans and display them in the run dashboard
* Add deployed tasks run timeline metrics
* Add Dequeued event to run timeline and cleanup the run timeline code
* Adds variants to storybook
* WIP adding new span styles
* Added offset progress bar animation
* More storybook states
* Adds support for the full vertical span to show the same state
* Adds error state to timelineLine
* Added additional state
* Added more line styling
* Added progress state to dequeued
* Added another state to storybook
* Fixed classname error
* Updated styles for the span timeline points
* Fixes alignment of timeline follow cursor indicator
* Adds help text tooltip to timeline span type titles
* Fixes type error
* Tweaked wording of tooltips
* Fixed type error (check this)
* Moved isAdmin to a higher level
* removed unused admin props
* Removed unused Admin filter
* Fixed border styling
* made the opacity of the timeline states 30% less
* Undo type cast
* Added a diminished style that’s used for spans (grey progress bar)
* Adds new storybook state
* Fixed timeline state
* Removed state if span isn’t the first
* Changed the timestamp span icon
---------
Co-authored-by: James Ritchie <james@trigger.dev>
* Improve the MarQS priority system by moving future messages into the LRE worker and using a priority timestamp offset to define priority in messages
* Add a test to make sure priority offsets don't unfairly favor environments
* requeuing should clear concurrency sets
* Heartbeats should only reschedule existing heartbeat jobs
* Fix type error
* Moves API reference to a new dropdown section
* Move toubleshooting higher up
* Separates the reference Overview page into new pages
* Removed Projects api and redirects old page to overview
* typo
* Format the machine override example code
* Adds note to add `fluent-ffmpeg` to external
* Updates onFailure with more clarification
* Adds debugging in vs code to Troubleshooting
* Removed link
* New MarQS method requeueMessage, an atomic version of replace message
* Remove redundant call to remove the message from the env queue in requeueMessage
* run engine v1: orgs are no longer considered for concurrency
* Add reserve concurrency concept to allow waiting to resume parent tasks to release concurrency at the env level for child tasks to use (or else there is a deadlock). WIP recursive tasks
* child tasks inherit the queue timestamp from their parent tasks to prioritize completing child tasks based on when their parent started
* handle reserve concurrency with recursive deadlocks
* Finish docs update for concurrency
* Some fixes from badge conflict resolution
* WIP priority queues
* Implement MarQS priority queues
* Fix the migrations
* Adds more Prisma documentation
* Adds more info and a diagram to improve idempotencyKeyTTLs
* Adds more clarity to the run with TTL docs section
* Limits page copy improvements
* Adds schema properties to the alert webhooks
* typo
* Adds feature list to the sveltekit plugin page
* Removed Migrations section in side menu
* Tidy up the table links
* Creates a community packages section
* Small tweak
* Adds info and links to community packages
* Updated mintlify broken links version in workflow file
* fixed broken link
* add timeout support to sendWithAck
* coordinator will retry completion submission
* actually retry
* increase default retries
* something went wrong there, add this back in
* add changeset
* don't crash run on lazy attempt errors and count on retry
* remove abort controller after checkpoint in all cases
* duration checkpoints
* manual checkpoints
* add busybox to coordinator image
* add async toggle to process in background
* remove early returns
* add changeset for manual checkpoint schema
* Uses image cards for the frameworks
* Removes old snippets
* New AI agents side menu section
* WIP adding new ai agent pages
* Better overview page
* More copy added to the agent example pages
* Copy improvements
* Removes “Creating a project” page and side menu section
* Fixes broken links
* Updates to the latest Mintlify version, fixes issues, changes theme
* Adds descriptions to the main dropdown menu items
* Reformatted Introduction docs ‘landing page’
* Retry heartbeat timeouts by putting back in the queue (#1689)
* If there’s a heartbeat error and no attempts we put it back in the queue to try again
* When nacking, return whether it was put back in the queue or not
* Try and nack, if it fails then fail the run
* Consolidated switch statement
* Fail executing/retrying runs
* OOM retrying on larger machines (#1691)
* OOM retrying on larger machines
* Create forty-windows-shop.md
* Update forty-windows-shop.md
* Only retry again if the machine is different from the original
* Kubernetes OOMs appear as non-zero sigkills, adding support for treating these as OOMs
* Complete the original attempt span if retrying due to an OOM
* Revert "Complete the original attempt span if retrying due to an OOM"
This reverts commit 5f652c6212.
* chore: Update version for release (#1666)
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
* Release 3.3.14
* Set machine when triggering docs
* Batch queue runs that are waiting for deploy (#1693)
* Detect ffmpeg OOM errors, added manual OutOfMemoryError (#1694)
* Detect ffmpeg OOM errors, added manual OutOfMemoryError
* Create eighty-spies-knock.md
* Improved the machines docs, including the new OutOfMemoryError
* chore: Update version for release (#1695)
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
* Release 3.3.15
* Create new partitioned TaskEvent table, and switch to it gradually as new runs are created (#1696)
* Create new partitioned TaskEvent table, and switch to it gradually as new runs are created
* Add env var for partition window in seconds
* Make startCreatedAt required in task event store
* Don't create an attempt if the run is final, batchTriggerAndWait bad continue fix (#1698)
* WIP fix for ResumeAttemptService selecting the wrong attempt (which has no error or output)
* Don’t create an attempt if the run is already in a final status
* Don’t get all the columns for the query. Improved the logging.
* Added a log to the batch example
* Filter out the undefined values
* Fix missing logs on child runs by using the root task run createdAt if it exists (#1697)
* Provider changes to support image cache (#1700)
* add env var for additional pull secrets
* make static images configurable
* optional image prefixes
* optional labels with sample rates
* add missing core paths
* remove excessive logs
* Fix run container exits after OOM retries (#1701)
* remove unused imports
* tell run to exit before force requeue
* handle exit for case where we already retried after oom
* improve retry span and add machine props
* don't try to exit run in dev
* Upgrade local dev to use electric beta.15 (#1699)
* Text fixes
* Removed pnpm files
---------
Co-authored-by: Matt Aitken <matt@mattaitken.com>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Co-authored-by: Eric Allam <eric@trigger.dev>
Co-authored-by: nicktrn <55853254+nicktrn@users.noreply.github.com>
* WIP with webhook SDK function and types
* JSDocs added to the schema
* Webhooks are working
* Expanded the alert docs
* Remove duplicate export of waitUntil.js
* Use uncrypto
* Don’t rate limit webhooks
* Create slow-olives-fix.md
* remove unused imports
* tell run to exit before force requeue
* handle exit for case where we already retried after oom
* improve retry span and add machine props
* don't try to exit run in dev
* WIP fix for ResumeAttemptService selecting the wrong attempt (which has no error or output)
* Don’t create an attempt if the run is already in a final status
* Don’t get all the columns for the query. Improved the logging.
* Added a log to the batch example
* Filter out the undefined values
* Create new partitioned TaskEvent table, and switch to it gradually as new runs are created
* Add env var for partition window in seconds
* Make startCreatedAt required in task event store
* OOM retrying on larger machines
* Create forty-windows-shop.md
* Update forty-windows-shop.md
* Only retry again if the machine is different from the original
* If there’s a heartbeat error and no attempts we put it back in the queue to try again
* When nacking, return whether it was put back in the queue or not
* Try and nack, if it fails then fail the run
* Consolidated switch statement
* Fail executing/retrying runs
* missing construction option in `AudioWaveformExtension`
* Create lemon-fireants-repair.md
---------
Co-authored-by: Matt Aitken <matt@mattaitken.com>
* Added isPrismaRetriableError()
* Retry completeBatchTaskRunItem if they fail because of a retriable Prisma error
* Retry using Redis worker
* Handle more retriable errors. Add special condition in for race condition error
* Added Postgres connection_timeout with default 20s
* Added a simple batchTriggerAndWait example
* Move the task run heartbeats to RedisWorker
* Move alerts to redis worker, improving redis worker
* Fix typecheck errors
* Use single threaded tests for redis worker
* Enable/disable the redis workers independently
* Remove preview release from PR checks
- remove calls to findUnique, especially when including/selecting relations
- add some missing indexes
- add spans to $transaction calls to help track down long running txs
* Automatically retry TriggerTaskService when hitting a unique constraint error on idempotency key
* improve batch completion system for run engine v1
* Rename batch stuff to v3 so it's not confusing
* Handle unique constraint error on BatchTaskRunItem creation and allow different limits for batchTrigger and batchTriggerAndWait
* Documentation edits made through Mintlify web editor
* Moves files to correct path and improves copy
* Corrects video props to camel case and removes deprecated prop
* Link to the new bulk actions page from the replaying page
---------
Co-authored-by: mintlify[bot] <109931778+mintlify[bot]@users.noreply.github.com>
Co-authored-by: James Ritchie <james@trigger.dev>
* Remove unnecessary disabled org check
* Add support for separate redis servers for caching, rate limiter, marqs, and the pub/sub stuff (they all default to the previously used env vars for redis)
* Remove the socket count from shared queue consumer cause it can take up to 5s to complete
* Add support for re-using queue snapshots across dequeues
* Only consider the top N orgs when dequeuing, to help mitigate large spikes in queues (like around the hour and half hour marks)
* Various fixes for run engine v1
- Make sure there are connected providers before sending a scheduled attempt message, nack and retry if there are not
- Fail runs that fail task heartbeats when pending and locked
- More and better logging around shared queue consumer
- Fix bug when failing a task run with no attempt
* Prevent findUnique from bringing down our database
* engine v1 fairness improvements (env based queue selection)
* Add ability to bias env selection by concurrency limit and capacity
* Add some randomization to the list of queues within each evn
* Configure biases using env vars
* Typo
* revert v3 catalog project ref change
* Fix v2 marqs
* reduces the thickness of the focus-visible
* More subtle Input style
* Updates focus-visible styles for the SimpleSelect
* Tables tab through rows and action cells only
* Refined focus visible style
* Adds the custom focus visible to the TextLink
* Moves the onClick handling to the table row rather than the cell
* Makes table row heights consistent
* Adds gap between task search bar and button
* Prevents long tag values from wrapping
* Removes unnecessary rows from table header
* Added gap between search and filters
* Fixes the schedules table pagination staying fixed to the bottom of the page
* Removed more unnecessary header table rows to prevent them being selectable
* Removed isSelected styles (not working)
* Added <tr> back to the main Table compontent
* Table row handles modifier keys
* Adds to={path} to the TableRow only
* Revert "Adds to={path} to the TableRow only"
This reverts commit 8a814d434ff85b8ab9c2305d336074f659d86094.
* Revert "Table row handles modifier keys"
This reverts commit b2394741fa2ed6b908e00673b1a395ae89ca7146.
* Table reverted to use linked cells rather than rows
* Set the tab index of a cell and style the table row when tabbed
* Tabbed row style applied to the sticky cells
* Adds isTabbableCell to each table
* Improves the spcificity of the row highlighting
* Reduces the height of the task rows to match the other tables
* Adds tab styles to fill in row dividers top and bottom
* Removed old row onClick and to props
* Removed duplicate table header row
* Creates table style variants so tables look good in the inspector panels
* Full width tables on the schedules inspector panel
* Reduce padding on Alerts page table rows
* Updates the Deploy page inspector table to the new style
* Removes the duplicate Table Row from the table headers in v2
* Adds TableRow to the EventsTable
* Adds TableRow to more tables
* Adds TableRow to more tables
* Adds TableRow to more tables
* Adds TableRow to more tables
* Fix for showing correct cursor for linked table rows
* Removed variants from some of the table components
* Efficiency improvements to SharedQueueConsumer and MarQS for run engine v1
* Reliabily resolve task queue for a run and ack runs where we can't find the queue
* engine v1: stop nacking after reaching the max nack count, ack to remove the message
* Handle division by 0 possibility in distributeQueues
* Add --experimental-global-webcrypto node option fix "crypto is not defined error" on Node.js 18 in dev
* Only add flag when on node 18
* Missed the function
* Show a good error if TRIGGER_ACCESS_TOKEN isn’t a PAT
* Require maxDuration in trigger.config from now on
* Changeset
* Update the trigger.config templates copy, and set the default to 1hr
* Set the maxDuration in the trigger.config files inside the repo
* Fixed the e2e tests by setting the package to 0.0.0-prerelease-20250116195421
* WIP preventing concurrent waits, throw an error
* Added ConcurrentWaitError (not retryable)
* Move preventMultipleWaits out of the RuntimeAPI
* Added preventMultipleWaits to the devRuntimeManager
* Added throwable InternalError. Plus new TASK_DID_CONCURRENT_WAIT code
* Docs link for troubleshooting concurrent waits
* Docs for troubleshooting concurrent waits
* preventMultipleWaits function
* Added TASK_DID_CONCURRENT_WAIT code
* Deal with InternalErrors that skipRetrying
* Added preventMultipleWaits to prod
* workerDeployment change findUnique to findFirst
* Changed the worker deployment queries to only select what they need, some of the columns are huge
* Only get the bare minimum data for tasks too
* WIP langsmith & AI SDK integration
* Add changeset
* Add exporter support to deployed tasks
* Better support for external exporters and group exporters and instrumentations under the telemetry config property
* Missing changes
* Added shortcuts (with tooltip) to the pagination
* WIP fixing the disabled hover state
* Styled the tooltip
* Added a shortcuts panel to the help menu
* Adds new shortcut to list
* Changes “meta” for “mod”
* Adds more shortcuts to the list
* Adding shortcut to open the shortcuts panel
* tweak gap between shortcut letters
* button component now has icon spacing adjustment (for lucide icons)
* Fixed some ilegal markup
* Pagination uses disabled prop rather than a disabled wrapper
* Improved the Switch styles
* Makes the shortcut modifier optional
* Added new icon based shortcut keys for mac and win
* Updated PC modifier shortcuts
* Adds a new windows key icon
* Allows variants and react nodes to be used as the modifier key
* Adds more shortcuts to the storybook
* Adds missing focus-visible styles to the pagination
* Removed test modifier keys
* number style is tabular
* Update apps/webapp/app/components/primitives/ShortcutKey.tsx
Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>
* Tooltip now just 1 prop on the button component
---------
Co-authored-by: Matt Aitken <matt@mattaitken.com>
Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>
* Increase the number of active streams from 2 to 5
* WIP parent metadata updates
* Fix noop metadata manager
* Implement run metadata updates from ancestor tasks
* Add changeset
* Add ability to stream into parent and root task runs
* Don't duplicate exporting run metadata types
* Add ability to fetch streams through metadata
* Couple of fixes from CodeRabbit
* Fix metadata tests
* Make sure streams are subscribed in the "background"
* Move the stream subscription stuff to the API client, expose it through `runs.fetchStream`
* Fixed run stream tests
* Fix realtime safari bug because of missing ReadableStream async iterable support
Limit to only safari
* Fix missing TIMED_OUT run status
* When coercing realtime date strings, make sure they are set to UTC
* Added new side menu section and updated intro
* Updated the fal ai realtime project
* Copy update
* Added example project
* Swapped walkthrough and repo
* Restore original fal task page
* Added links back to the intro
* Updated login page logos
* Adds an isSelected state to the Table
* Toast style now matches the design
* Adds a space between the upgrade panel and the list of users
* Fix streaming splits in realtime streams v2
* Add changeset
* Skip all flaky tests 😡
* Improve the way we stream from tasks to the server
* Improve the v1 realtime streams (Redis)
* Turn on the relay realtime stream service
* Improved the relay realtime cleanup
* Fixed consuming realtime runs w/streams after the run is already finished
* Remove some logs
* Update changeset
* Fixed runStream tests
* Upgrading from Hobby to Pro displays a dialog to confirm
* Improved the logic for upgrading to pro
* Added the spinner component to storybook
* Improved the dark style spinner
* Realtime streams now powered by electric, and fix the streaming update duplicate issues by converting the electric Shape materialized view into a ReadableStream of changes
* Ensure realtime subscription stops when runs are finished, and add an onComplete handle to use realtime hooks
* Fix tests
Add Agent Skills for `chat.agent`. Drop a folder with a `SKILL.md` and any helper scripts/references next to your task code, register it with `skills.define({ id, path })`, and the CLI bundles it into the deploy image automatically — no `trigger.config.ts` changes. The agent gets a one-line summary in its system prompt and discovers full instructions on demand via `loadSkill`, with `bash` and `readFile` tools scoped per-skill (path-traversal guards, output caps, abort-signal propagation).
Built on the [AI SDK cookbook pattern](https://ai-sdk.dev/cookbook/guides/agent-skills) — portable across providers. SDK + CLI only for now; dashboard-editable `SKILL.md` text is on the roadmap.
**AI Prompts** — define prompt templates as code alongside your tasks, version them on deploy, and override the text or model from the dashboard without redeploying. Prompts integrate with the Vercel AI SDK via `toAISDKTelemetry()` (links every generation span back to the prompt) and with `chat.agent` via `chat.prompt.set()` + `chat.toStreamTextOptions()`.
```ts
import{prompts}from"@trigger.dev/sdk";
import{generateText}from"ai";
import{openai}from"@ai-sdk/openai";
import{z}from"zod";
exportconstsupportPrompt=prompts.define({
id:"customer-support",
model:"gpt-4o",
config:{temperature: 0.7},
variables: z.object({
customerName: z.string(),
plan: z.string(),
issue: z.string(),
}),
content:`You are a support agent for Acme.
Customer: {{customerName}} ({{plan}} plan)
Issue: {{issue}}`,
});
constresolved=awaitsupportPrompt.resolve({
customerName:"Alice",
plan:"Pro",
issue:"Can't access billing",
});
constresult=awaitgenerateText({
model: openai(resolved.model??"gpt-4o"),
system: resolved.text,
prompt:"Can't access billing",
...resolved.toAISDKTelemetry(),
});
```
**What you get:**
- **Code-defined, deploy-versioned templates** — define with `prompts.define({ id, model, config, variables, content })`. Every deploy creates a new version visible in the dashboard. Mustache-style placeholders (`{{var}}`, `{{#cond}}...{{/cond}}`) with Zod / ArkType / Valibot-typed variables.
- **Dashboard overrides** — change a prompt's text or model from the dashboard without redeploying. Overrides take priority over the deployed "current" version and are environment-scoped (dev / staging / production independent).
- **Resolve API** — `prompt.resolve(vars, { version?, label? })` returns the compiled `text`, resolved `model`, `version`, and labels. Standalone `prompts.resolve<typeof handle>(slug, vars)` for cross-file resolution with full type inference on slug and variable shape.
- **AI SDK integration** — spread `resolved.toAISDKTelemetry({ ...extra })` into any `generateText` / `streamText` call and every generation span links to the prompt in the dashboard alongside its input variables, model, tokens, and cost.
- **`chat.agent` integration** — `chat.prompt.set(resolved)` stores the resolved prompt run-scoped; `chat.toStreamTextOptions({ registry })` pulls `system`, `model` (resolved via the AI SDK provider registry), `temperature` / `maxTokens` / etc., and telemetry into a single spread for `streamText`.
- **Dashboard** — prompts list with per-prompt usage sparklines; per-prompt detail with Template / Details / Versions / Generations / Metrics tabs. AI generation spans get a custom inspector showing the linked prompt's metadata, input variables, and template content alongside model, tokens, cost, and the message thread.
See [/docs/ai/prompts](https://trigger.dev/docs/ai/prompts) for the full reference — template syntax, version resolution order, override workflow, and type utilities (`PromptHandle`, `PromptIdentifier`, `PromptVariables`).
Adds AI SDK 7 support. The `ai` peer range now includes v7, and the `chat.agent` / chat surfaces work against v7's ESM-only build. On v7, install `@ai-sdk/otel` alongside `ai` and the SDK registers it for you so `experimental_telemetry` spans keep flowing into your run traces (v7 stopped emitting them from `ai` core). v5 and v6 keep working unchanged.
Add `ai.toolExecute(task)` so you can wire a Trigger subtask in as the `execute` handler of an AI SDK `tool()` while defining `description` and `inputSchema` yourself — useful when you want full control over the tool surface and just need Trigger's subtask machinery for the body.
```ts
constmyTool=tool({
description:"...",
inputSchema: z.object({...}),
execute: ai.toolExecute(mySubtask),
});
```
`ai.tool(task)` (`toolFromTask`) keeps doing the all-in-one wrap and now aligns its return type with AI SDK's `ToolSet`. Minimum `ai` peer raised to `^6.0.116` to avoid cross-version `ToolSet` mismatches in monorepos.
Fix `chat.agent` skills silently missing in `trigger dev` for projects whose task files read `process.env` at module top level (e.g. a third-party SDK client initialized at import). Skill folders now bundle into `.trigger/skills/` reliably regardless of which env vars are set when the CLI launches.
Reject overlong `idempotencyKey` values at the API boundary so they no longer trip an internal size limit on the underlying unique index and surface as a generic 500. Inputs are capped at 2048 characters — well above what `idempotencyKeys.create()` produces (a 64-character hash) and above any realistic raw key. Applies to `tasks.trigger`, `tasks.batchTrigger`, `batch.create` (Phase 1 streaming batches), `wait.createToken`, `wait.forDuration`, and the input/session stream waitpoint endpoints. Over-limit requests now return a structured 400 instead.
Reliability fixes for `chat.agent`. A user message sent while the agent is streaming is no longer delivered twice (which could run a duplicate turn), input appends now carry an idempotency key so a retried send can't duplicate a message, stopping a generation clears the streaming state so a page reload doesn't replay the stopped turn, and runs can now carry the full set of dashboard tags instead of being silently truncated. `onTurnComplete` now fires on errored turns (with the thrown error attached) and the failed turn's user message is persisted so it isn't lost on the next run. Custom agents and manual `chat.writeTurnComplete` callers now trim the output stream, sending a custom action no longer leaves a second stream reader running, and a long-lived `watch` subscription no longer grows its dedupe set without bound.
Adds `onBoot` to `chat.agent` — a lifecycle hook that fires once per worker process picking up the chat. Runs for the initial run, preloaded runs, AND reactive continuation runs (post-cancel, crash, `endRun`, `requestUpgrade`, OOM retry), before any other hook. Use it to initialize `chat.local`, open per-process resources, or re-hydrate state from your DB on continuation — anywhere the SAME run picking up after suspend/resume isn't enough.
Use `onBoot` (not `onChatStart`) for state setup that must run every time a worker picks up the chat — `onChatStart` fires once per chat and won't run on continuation, leaving `chat.local` uninitialized when `run()` tries to use it.
Add a `tools` option to `chat.agent`. Declaring your tools here threads them into the SDK's internal `convertToModelMessages`, so each tool's `toModelOutput` is re-applied when prior-turn history is re-converted.
**AI Agents** — run AI SDK chat completions as durable Trigger.dev agents instead of fragile API routes. Define an agent in one function, point `useChat` at it from React, and the conversation survives page refreshes, network blips, and process restarts.
- **AI SDK `useChat` integration** — a custom [`ChatTransport`](https://sdk.vercel.ai/docs/ai-sdk-ui/transport) (`useTriggerChatTransport`) plugs straight into Vercel AI SDK's `useChat` hook. Text streaming, tool calls, reasoning, and `data-*` parts all work natively over Trigger.dev's realtime streams. No custom API routes needed.
- **First-turn fast path (`chat.headStart`)** — opt-in handler that runs the first turn's `streamText` step in your warm server process while the agent run boots in parallel, cutting cold-start TTFC by roughly half (measured 2801ms → 1218ms on `claude-sonnet-4-6`). The agent owns step 2+ (tool execution, persistence, hooks) so heavy deps stay where they belong. Web Fetch handler works natively in Next.js, Hono, SvelteKit, Remix, Workers, etc.; bridge to Express/Fastify/Koa via `chat.toNodeListener`. New `@trigger.dev/sdk/chat-server` subpath.
- **Multi-turn durability via Sessions** — every chat is backed by a durable Session that outlives any individual run. Conversations resume across page refreshes, idle timeout, crashes, and deploys; `resume: true` reconnects via `lastEventId` so clients only see new chunks. `sessions.list` enumerates chats for inbox-style UIs.
- **Auto-accumulated history, delta-only wire** — the backend accumulates the full conversation across turns; clients only ship the new message each turn. Long chats never hit the 512 KiB body cap. Register `hydrateMessages` to be the source of truth yourself.
- **Lifecycle hooks** — `onPreload`, `onChatStart`, `onValidateMessages`, `hydrateMessages`, `onTurnStart`, `onBeforeTurnComplete`, `onTurnComplete`, `onChatSuspend`, `onChatResume` — for persistence, validation, and post-turn work.
- **Stop generation** — client-driven `transport.stopGeneration(chatId)` aborts mid-stream; the run stays alive for the next message, partial response is captured, and aborted parts (stuck `partial-call` tools, in-progress reasoning) are auto-cleaned.
- **Tool approvals (HITL)** — tools with `needsApproval: true` pause until the user approves or denies via `addToolApprovalResponse`. The runtime reconciles the updated assistant message by ID and continues `streamText`.
- **Steering and background injection** — `pendingMessages` injects user messages between tool-call steps so users can steer the agent mid-execution; `chat.inject()` + `chat.defer()` adds context from background work (self-review, RAG, safety checks) between turns.
- **Actions** — non-turn frontend commands (undo, rollback, regenerate, edit) sent via `transport.sendAction`. Fire `hydrateMessages` + `onAction` only — no turn hooks, no `run()`. `onAction` can return a `StreamTextResult` for a model response, or `void` for side-effect-only.
- **Typed state primitives** — `chat.local<T>` for per-run state accessible from hooks, `run()`, tools, and subtasks (auto-serialized through `ai.toolExecute`); `chat.store` for typed shared data between agent and client; `chat.history` for reading and mutating the message chain; `clientDataSchema` for typed `clientData` in every hook.
- **`chat.toStreamTextOptions()`** — one spread into `streamText` wires up versioned system [Prompts](https://trigger.dev/docs/ai/prompts), model resolution, telemetry metadata, compaction, steering, and background injection.
- **Multi-tab coordination** — `multiTab: true` + `useMultiTabChat` prevents duplicate sends and syncs state across browser tabs via `BroadcastChannel`. Non-active tabs go read-only with live updates.
- **Network resilience** — built-in indefinite retry with bounded backoff, reconnect on `online` / tab refocus / bfcache restore, `Last-Event-ID` mid-stream resume. No app code needed.
See [/docs/ai-chat](https://trigger.dev/docs/ai-chat/overview) for the full surface — quick start, three backend approaches (`chat.agent`, `chat.createSession`, raw task), persistence and code-sandbox patterns, type-level guides, and API reference.
Continuation chat boots no longer stall for around 10 seconds before the first turn. The `session.in` resume cursor is now found with a non-blocking records read instead of draining an SSE long-poll (which always waited out its full 5 second inactivity window, twice per boot), the boot reads run concurrently, and chat snapshots carry the cursor so subsequent boots skip the scan entirely.
Fix `chat.headStart` when `hydrateMessages` is registered. The warm route's step-1 partial now reaches the agent's accumulator on the hydrate path, so `onTurnComplete` carries the full first turn (the head-start user message included), tool-call handovers resume from step 2 instead of re-running step 1, and the assistant `messageId` stays stable across the handover.
Preserve reasoning parts across the `chat.headStart` handover. Extended-thinking models' step-1 reasoning now lands in the durable session history (and `onTurnComplete`) under the same assistant `messageId`, with provider metadata intact so Anthropic thinking signatures survive replays.
Add read primitives to `chat.history` for HITL flows: `getPendingToolCalls()`, `getResolvedToolCalls()`, `extractNewToolResults(message)`, `getChain()`, and `findMessage(messageId)`. These lift the accumulator-walking logic that customers building human-in-the-loop tools were re-implementing into the SDK.
Use `getPendingToolCalls()` to gate fresh user turns while a tool call is awaiting an answer. Use `extractNewToolResults(message)` to dedup tool results when persisting to your own store — the helper returns only the parts whose `toolCallId` is not already resolved on the chain.
```ts
constpending=chat.history.getPendingToolCalls();
if(pending.length>0){
// an addToolOutput is expected before a new user message
Stamp `gen_ai.conversation.id` (the chat id) on every span and metric emitted from inside a `chat.task` or `chat.agent` run. Lets you filter dashboard spans, runs, and metrics by the chat conversation that produced them — independent of the run boundary, so multi-run chats correlate cleanly. No code changes required on the user side.
Fix `chat.agent` HITL continuations on reasoning-heavy turns. Two changes that work together:
- The per-turn merge now overlays the wire copy's tool-part state advancement onto the agent's existing chain — `state` + the matching resolution field (`output` / `errorText` / `approval`) come from the wire, everything else (text, reasoning, tool `input`, provider metadata) stays whatever the snapshot or `hydrateMessages` returned. Previously a full-message replace overwrote those fields with whatever the client shipped, so a slimmed wire copy landed a tool call with no `arguments` on the next LLM call. Covers `output-available` / `output-error` (HITL `addToolOutput`) and `approval-responded` / `output-denied` (approval flow).
-`TriggerChatTransport.sendMessages` and `AgentChat.sendRaw` now slim assistant messages that carry advanced tool parts. The wire payload is just `{ id, role, parts: [<state + resolution field>] }` for `submit-message` continuations; everything else passes through. Reasoning blobs and full tool inputs no longer ride the wire on every `addToolOutput` / `addToolApproveResponse`, so continuation payloads stay well under the `.in/append` cap on long agent loops.
Note: `onValidateMessages` receives the slim wire on HITL turns. If you call `validateUIMessages` from `ai` against the full `messages` array it will reject the slim assistant; filter to user messages (or skip on HITL turns) — see the updated docstring on `onValidateMessages` for the recommended pattern.
For `hydrateMessages` hooks that persist the chain, this release also adds a small helper to the `@trigger.dev/sdk/ai` surface:
It pushes fresh user messages by id, no-ops on HITL continuations (the incoming shares an id with the existing assistant — the runtime overlays the new tool-state advance), and skips on non-`submit-message` triggers. Returns `true` if it mutated `stored` so the caller knows whether to persist.
Net effect: `chat.addToolOutput(...)` / `chat.addToolApproveResponse(...)` on multi-step reasoning agents (OpenAI Responses with `store: false`, Anthropic extended thinking, etc.) no longer blows the cap and no longer corrupts the LLM input.
`ChatStartSessionParams` gains a typed `clientData` field — folded into the first run's `payload.metadata` so `onPreload` / `onChatStart` see the same shape per-turn `metadata` carries via the transport. The opaque session-level `metadata` field is unchanged.
`trigger init` now sets up your AI coding assistant as part of project setup: pick the MCP server, the agent skills, or both, then scaffold with the CLI or hand off to your assistant. Adds a new `getting-started` agent skill that teaches assistants how to bootstrap Trigger.dev (install the SDK, write `trigger.config.ts`, create a first task, run `trigger dev`), so the AI-driven setup path works end to end. It ships in the CLI alongside the existing skills, version-matched to your SDK.
Coerce numeric `concurrencyKey` values to string at the API boundary across `tasks.trigger`, `tasks.batchTrigger`, and the Phase-2 streaming batch endpoint.
Record client-side dequeue API latency in the supervisor consumer pool as a Prometheus histogram (`queue_consumer_pool_dequeue_duration_seconds`, labelled by `outcome`: success/empty/error).
`dev` and `deploy` now fail with a clear error when two tasks are defined with the same id, including across different task types (e.g. a scheduled task and a regular task sharing an id). Previously the second definition silently overwrote the first, so one of the tasks would vanish with no warning. Task ids are detected as duplicates during indexing (naming each offending id and the files it was found in), and the same rule is enforced server-side when the background worker is registered.
Fix `@trigger.dev/core` build: cast the underlying log record exporter when calling `forceFlush` so it typechecks against the updated OpenTelemetry `LogRecordExporter` type (which no longer declares `forceFlush`).
`envvars.upload` now accepts an optional `isSecret` flag, letting you create the imported variables as secret (redacted) environment variables. When omitted, variables default to non-secret.
Offload large trigger payloads to object storage before sending the trigger API request. The SDK uploads packets at or above the existing 128KB limit and sends an `application/store` pointer instead of embedding large JSON in the request body. `TriggerTaskRequestBody` now validates that `application/store` payloads are non-empty storage paths.
Payload uploads use the same resolved `ApiClient` as the trigger call (including `requestOptions.clientConfig`), not only the global `apiClientManager.client` — so custom `baseURL`, access token, and preview branch apply to both presign and trigger.
Fix `LocalsKey<T>` type incompatibility across dual-package builds. The phantom value-type brand no longer uses a module-level `unique symbol`, so a single TypeScript compilation that resolves the type from both the ESM and CJS outputs (which can happen under certain pnpm hoisting layouts) no longer sees two structurally-incompatible variants of the same type.
The CLI MCP server's agent-chat tools (`start_agent_chat`, `send_agent_message`, `close_agent_chat`) now run on the new Sessions primitive, so AI assistants driving a `chat.agent` get the same idempotent-by-`chatId`, durable-across-runs behavior the browser transport gets. Required PAT scopes go from `write:inputStreams` to `read:sessions` + `write:sessions`.
The MCP server no longer tells the AI agent to wait for a run to complete after every `trigger_task` call. Waiting is now opt-in: the agent only waits when you ask it to (for example "trigger and then wait for it to finish"). This avoids burning tokens polling runs you didn't need to block on and keeps responses clearer.
Unit-test `chat.agent` definitions offline with `mockChatAgent` from `@trigger.dev/sdk/ai/test`. Drives a real agent's turn loop in-process — no network, no task runtime — so you can send messages, actions, and stop signals via driver methods, inspect captured output chunks, and verify hooks fire. Pairs with `MockLanguageModelV3` from `ai/test` for model mocking. `setupLocals` lets you pre-seed `locals` (DB clients, service stubs) before `run()` starts.
The broader `runInMockTaskContext` harness it's built on lives at `@trigger.dev/core/v3/test` — useful for unit-testing any task code, not just chat.
Pipeline the per-entry `HGETALL` fetches in `MollifierBuffer.listEntriesForEnv`. The previous serial implementation issued one Redis round-trip per runId returned by `LRANGE`, which dominated stale-sweep wall-time at any meaningful backlog (at the sweep's default maxCount=1000, this is ~1000 RTTs per env per pass). Behaviour is unchanged — entries are still skipped when the entry hash has been torn down by a concurrent drainer ack/fail between the LRANGE and the HGETALL.
Make mollifier buffer and drainer internals configurable. `MollifierBuffer` now accepts `ackGraceTtlSeconds`, `maxRetriesPerRequest`, `reconnectStepMs`, and `reconnectMaxMs` options, and `MollifierDrainer` accepts `maxBackoffMs` and `backoffFloorMs`. All default to their previous hardcoded values, so existing behaviour is unchanged.
`MollifierDrainer` accepts a `drainBatchSize` option (default 1) that controls how many entries are popped per env per tick — in-flight handlers remain capped by the global `concurrency`. `MollifierBuffer` also gains `getDrainingCount()` / `listStaleDraining()`, backed by a new `mollifier:draining` ZSET maintained atomically with pop/ack/fail/requeue (observability-only).
Add MollifierBuffer and MollifierDrainer primitives for trigger burst smoothing.
MollifierBuffer (`accept`, `pop`, `ack`, `requeue`, `fail`, `evaluateTrip`) is a per-env FIFO over Redis with atomic Lua transitions for status tracking. `evaluateTrip` is a sliding-window trip evaluator the webapp gate uses to detect per-env trigger bursts.
MollifierDrainer pops entries through a polling loop with a user-supplied handler. The loop survives transient Redis errors via capped exponential backoff (up to 5s), and per-env pop failures don't poison the rest of the batch — one env's blip is logged and counted as failed for that tick. Rotation is two-level: orgs at the top, envs within each org. The buffer maintains `mollifier:orgs` and `mollifier:org-envs:${orgId}` atomically with per-env queues, so the drainer walks orgs → envs directly without an in-memory cache. The `maxOrgsPerTick` option (default 500) caps how many orgs are scheduled per tick; for each picked org, one env is popped (rotating round-robin within the org). An org with N envs gets the same per-tick scheduling slot as an org with 1 env, so tenant-level drainage throughput is determined by org count rather than env count.
Mollifier `mutateSnapshot` now enforces a tag cap: an `append_tags` patch carrying `maxTags` returns `"limit_exceeded"` (writing nothing) when the deduped tag count would exceed the limit, so a buffered run can't accumulate more tags via the tags API than the trigger validator allows at creation.
Update the bundled OpenTelemetry packages to their latest releases (`@opentelemetry/sdk-node` 0.218.0, `@opentelemetry/core` 2.7.1, `@opentelemetry/host-metrics` 0.38.3).
Add `GetProjectEnvironmentsResponseBody` and `ProjectEnvironment` schemas for the new `GET /api/v1/projects/{projectRef}/environments` endpoint, which lists the parent environments (dev, staging, preview, prod) a personal access token can access for a project. Dev is scoped to the token owner and branch (preview child) environments are excluded.
Fix `COULD_NOT_FIND_EXECUTOR` when a task's definition is loaded via `await import(...)` from inside another task's `run()`. The runtime workers now register such tasks with a sentinel file context, and the catalog logs a one-time warning per task id.
Retry `TASK_MIDDLEWARE_ERROR` under the task's retry policy instead of failing the run on the first attempt. The error was already classified as retryable by `shouldRetryError`, but `shouldLookupRetrySettings` did not include it, so the retry flow fell through to `fail_run`. Fixes #3231.
Retry `TASK_PROCESS_SIGSEGV` task crashes under the user's retry policy instead of failing the run on the first segfault. SIGSEGV in Node tasks is frequently non-deterministic (native addon races, JIT/GC interaction, near-OOM in native code, host issues), so retrying on a fresh process often succeeds. The retry is gated by the task's existing `retry` config + `maxAttempts` — same path `TASK_PROCESS_SIGTERM` and uncaught exceptions already use — so tasks without a retry policy still fail fast.
Add `region` to the runs list / retrieve API: filter runs by region (`runs.list({ region: "..." })` / `filter[region]=<masterQueue>`) and read each run's executing region from the new `region` field on the response.
Bump `@s2-dev/streamstore` to `0.22.10` to fix a `TASK_RUN_UNCAUGHT_EXCEPTION` ("Invalid state: Unable to enqueue") when a `chat.agent` turn is aborted mid-stream.
**Sessions** — a durable, run-aware stream channel keyed on a stable `externalId`. A Session is the unit of state that owns a multi-run conversation: messages flow through `.in`, responses through `.out`, both survive run boundaries. Sessions back the new `chat.agent` runtime, and you can build on them directly for any pattern that needs durable bi-directional streaming across runs.
```ts
import{sessions,tasks}from"@trigger.dev/sdk";
// Trigger a task and subscribe to its session output in one call
See [/docs/ai-chat/overview](https://trigger.dev/docs/ai-chat/overview) for the full surface — Sessions powers the durable, resumable chat runtime described there.
Add `TriggerClient` for running multiple SDK clients side-by-side, each with its own auth, preview branch, and baseURL. Useful when a single process needs to trigger tasks or read runs across multiple projects, environments, or preview branches without mutating shared global state.
`trigger skills` installs Trigger.dev agent skills into your coding agent so it knows how to write tasks, schedules, realtime, and chat.agent code. The skills ship with the CLI and are copied into each tool's native skills directory (Claude Code, Cursor, GitHub Copilot, and Codex / AGENTS.md), and `trigger dev` offers to install them on first run.
```bash
trigger skills --target claude-code
```
Replaces the previous `install-rules` command, which stays as an alias.
Fix `TypeError` in `unflattenAttributes` when the input attribute map contains conflicting dotted key paths (e.g. both `a.b` set to a scalar and `a.b.c` set to a value). The path-walk loop now applies last-write-wins when a prior key wrote a primitive, null, or array at an intermediate slot, matching the existing precedent in `AttributeFlattener.addAttribute`. Callers no longer crash when handed malformed external attribute inputs.
Fix external trace context leaking across runs on warm-started workers with `processKeepAlive` enabled. Every subsequent run's attempt span was being exported with the first run's `traceId` and `parentSpanId`, breaking causal-chain navigation in external APM tools. Runs without an external trace context are unaffected.
Repo-specific signal for anyone (human or agent) reviewing a PR in this codebase. Calibrates what counts as critical, what to always check, and what to skip.
## What makes a 🔴 Important finding here
Reserve 🔴 for things that would page someone or block a rollback. In this codebase, that means:
- **Rolling-deploy breakage.** Old and new versions of the webapp/supervisor run side-by-side during deploys. A change is broken if:
- A Lua script's behavior changes for a given key set without versioning (rename the script with a behavior-descriptive suffix like `Tracked` rather than `V2` — both versions must coexist safely).
- A Redis data shape used by both versions changes in place. New shapes need a new key namespace.
- A migration is not backward-compatible with the prior image.
- **Schema / migration safety.** Prisma migrations must be backward-compatible with the prior deploy. Adding NOT NULL without a default, dropping a column an old image still reads, renaming a column — all 🔴.
- **ClickHouse migration ordering + idempotency.** Goose runs in strict mode in the deploy pipeline and refuses to apply a missing version below the current version — slotting a new file in below the latest already-applied version blocks the deploy. New ClickHouse migration files MUST use the next available number (`max(files in internal-packages/clickhouse/schema/) + 1`); if main has added migrations while you've been on a branch, renumber yours. DDL must also be idempotent (`ADD COLUMN IF NOT EXISTS`, `DROP COLUMN IF EXISTS`, `CREATE TABLE IF NOT EXISTS`, `ADD INDEX IF NOT EXISTS`) so a partial / `--allow-missing` apply elsewhere doesn't fail on retry. Either fault is 🔴 — both break test/prod deploys. Rules live in `internal-packages/clickhouse/CLAUDE.md`.
- **Queue / concurrency correctness.** RunQueue, MarQS (V1, legacy), redis-worker — any change to enqueue / dequeue / locking semantics. Re-derive the invariant on paper before flagging or accepting.
- **Missing index on a hot table.** New Prisma queries against `TaskRun`, `TaskRunExecutionSnapshot`, `JobRun`, `Project`, etc. must use an existing index. Check `internal-packages/database/prisma/schema.prisma` for the relevant `@@index` lines — don't guess and don't propose `EXPLAIN`.
- **Recovery-path queries.** Any `TaskRun.findFirst` / `findMany` added to a schedule, run-recovery, or restart loop. Recovery fan-outs (Redis crash, restart storms) turn "rare indexed query" into a DB incident. 🔴 even if indexed.
- **Aggregations on hot tables.** No `COUNT` / `GROUP BY` on `TaskRun` or other multi-million-row tables. Use Redis or ClickHouse for counts.
- **Prod Redis blast-radius.** New code paths that `SCAN` with broad patterns (`*foo*`) on prod-shaped Redis, or `EVAL` Lua with `SCAN` loops inside. Both are 🔴.
- **`@trigger.dev/core` direct import** from anywhere outside the SDK package. Always import from `@trigger.dev/sdk`. Core direct imports are 🔴 — they break the public API contract.
- **Heavy execute-deps imported into request-handler bundles.** Specifically `chat.handover` and similar split-bundle entry points must not transitively import the agent task's execute path. Watch for new imports added at module top-level of route files.
- **V1 engine code modified in a "V2 only" PR.** The `apps/webapp/app/v3/` directory contains both. If the PR description says V2-only but it touches `triggerTaskV1`, `cancelTaskRunV1`, `MarQS`, etc. — 🔴.
## Always check
- **Tests use testcontainers, not mocks.** Vitest with `redisTest` / `postgresTest` / `containerTest` from `@internal/testcontainers`. Any new `vi.mock(...)` on Redis, Postgres, BullMQ, or other infra is wrong here — 🔴 if added in production-path tests, 🟡 if isolated unit test.
- **Public-package changes have a changeset.** `pnpm run changeset:add` produces `.changeset/*.md`. Required for any edit under `packages/*`. Missing → 🟡; missing on a breaking change → 🔴.
- **Server-only changes have `.server-changes/*.md`.** Required for `apps/webapp/`, `apps/supervisor/` edits with no public-package change. Body should be 1-2 sentences (it has to fit as one bullet in a future changelog). Missing → 🟡.
- **Lua script naming.** Coexisting scripts use behavior-descriptive suffixes (`Tracked`), never `V2`. Old name must keep working until the next deploy clears it.
- **RunQueue payload shape.** V2 run-queue payload's `projectId` is consumed by `workerQueueResolver` for override matching. If a PR drops it from the payload, 🔴.
- **`safeSend` scope.** Defensive IPC wrappers belong on loop / interval / handler contexts, not one-shot terminal sends. If the PR adds `safeSend` to a single terminal call for consistency, 🟡 with a "remove this" suggestion.
- **Zod version.** Pinned to `3.25.76` monorepo-wide. New package adding zod with a different version or range — 🔴.
## Skip (do NOT flag)
- Anything Prettier / ESLint catches. CI runs both.
- TypeScript style preferences (`type` vs `interface`) — already covered by repo standards.
- Test coverage exhortations as a generic suggestion. Only flag missing tests when a specific code path is genuinely untested and the path has prior incidents.
-`agentcrumbs` markers (`// @crumbs`, `// #region @crumbs`) and `agentcrumbs` imports — these are temporary debug instrumentation stripped before merge.
-`// removed comments for removed code`, renamed `_unused` vars, re-exported types as "backwards compatibility shims" — also covered by repo standards.
- Suggestions to "add error handling" without naming a specific scenario that breaks.
The `apps/webapp/app/v3/` directory name is misleading — most code there is V2. Only specific files are V1-only legacy: `MarQS` queue, `triggerTaskV1`, `cancelTaskRunV1`, and a handful of others (see `apps/webapp/CLAUDE.md` for the exact list). Don't flag "you should refactor this to use V2" on those — they're frozen.
## Confidence calibration for this repo
The most common false-positive pattern: speculating about race conditions in code paths the agent doesn't have runtime visibility into. If the only evidence is "this *could* race", drop it. If you can point to a specific interleaving with file:line for each step, surface it.
- When adding indexes to **existing tables**, use `CREATE INDEX CONCURRENTLY IF NOT EXISTS` to avoid table locks. These must be in their own separate migration file (one index per file).
- Indexes on **newly created tables** (same migration as `CREATE TABLE`) do not need CONCURRENTLY.
- When indexing a **new column on an existing table**, split into two migrations: first `ADD COLUMN IF NOT EXISTS`, then `CREATE INDEX CONCURRENTLY IF NOT EXISTS` in a separate file.
- After generating a migration with Prisma, remove extraneous lines for: `_BackgroundWorkerToBackgroundWorkerFile`, `_BackgroundWorkerToTaskQueue`, `_TaskRunToTaskRunTag`, `_WaitpointRunConnections`, `_completedWaitpoints`, `SecretStore_key_idx`, and unrelated TaskRun indexes.
- Never drop columns or tables without explicit approval.
- New code should target `RunEngineVersion.V2` only.
The `v3/` directory name is misleading - most code here is actively used by the current V2 engine. Only the specific files below are legacy V1-only code.
When adding a new dependency to any package.json in the monorepo:
1.**Look up the latest version** on npm before adding:
```bash
pnpm view <package-name> version
```
If unsure which version to use (e.g. major version compatibility), confirm with the user.
2. **Edit the package.json directly** — do NOT use `pnpm add` as it can cause issues in the monorepo. Add the dependency with the correct version range (typically `^x.y.z`).
3. **Run `pnpm i` from the repo root** after editing to install and update the lockfile:
```bash
pnpm i
```
Always run from the repo root, not from the package directory.
description: Use when adding, modifying, or debugging OTel span timeline events in the trace view. Covers event structure, ClickHouse storage constraints, rendering in SpanTimeline component, admin visibility, and the step-by-step process for adding new events.
The trace view's right panel shows a timeline of events for the selected span. These are OTel span events rendered by `app/utils/timelineSpanEvents.ts` and the `SpanTimeline` component.
## How They Work
1.**Span events** in OTel are attached to a parent span. In ClickHouse, they're stored as separate rows with `kind: "SPAN_EVENT"` sharing the parent span's `span_id`. The `#mergeRecordsIntoSpanDetail` method reassembles them into the span's `events` array at query time.
2. The timeline only renders events whose `name` starts with `trigger.dev/` - all others are silently filtered out.
3. The **display name** comes from `properties.event` (not the span event name), mapped through `getFriendlyNameForEvent()`.
4. Events are shown on the **span they belong to** - events on one span don't appear in another span's timeline.
## ClickHouse Storage Constraint
When events are written to ClickHouse, `spanEventsToTaskEventV1Input()` filters out events whose `start_time` is not greater than the parent span's `startTime`. Events at or before the span start are silently dropped. This means span events must have timestamps strictly after the span's own `startTimeUnixNano`.
## Timeline Rendering (SpanTimeline component)
The `SpanTimeline` component in `app/components/run/RunTimeline.tsx` renders:
1.**Events** (thin 1px line with hollow dots) - all events from `createTimelineSpanEventsFromSpanEvents()`
2.**"Started"** marker (thick cap) - at the span's `startTime`
3.**Duration bar** (thick 7px line) - from "Started" to "Finished"
4.**"Finished"** marker (thick cap) - at `startTime + duration`
The thin line before "Started" only appears when there are events with timestamps between the span start and the first child span. For the Attempt span this works well (Dequeued -> Pod scheduled -> Launched -> etc. all happen before execution starts). Events all get `lineVariant: "light"` (thin) while the execution bar gets `variant: "normal"` (thick).
## Trace View Sort Order
Sibling spans (same parent) are sorted by `start_time ASC` from the ClickHouse query. The `createTreeFromFlatItems` function preserves this order. Event timestamps don't affect sort order - only the span's own `start_time`.
## Event Structure
```typescript
// OTel span event format
{
name:"trigger.dev/run",// Must start with "trigger.dev/" to render
timeUnixNano:"1711200000000000000",
attributes:[
{key:"event",value:{stringValue:"dequeue"}},// The actual event type
{key:"duration",value:{intValue: 150}},// Optional: duration in ms
]
}
```
## Admin-Only Events
`getAdminOnlyForEvent()` controls visibility. Events default to **admin-only** (`true`).
| Event | Admin-only | Friendly name |
|-------|-----------|---------------|
| `dequeue` | No | Dequeued |
| `fork` | No | Launched |
| `import` | No (if no fork event) | Importing task file |
description: Use this skill when writing, designing, or optimizing Trigger.dev background tasks and workflows. This includes creating reliable async tasks, implementing AI workflows, setting up scheduled jobs, structuring complex task hierarchies with subtasks, configuring build extensions for tools like ffmpeg or Puppeteer/Playwright, and handling task schemas with Zod validation.
You are an expert Trigger.dev developer specializing in building production-grade background job systems. Tasks deployed to Trigger.dev run in Node.js 21+ and use the `@trigger.dev/sdk` package.
## Critical Rules
1.**Always use `@trigger.dev/sdk`** - Never use `@trigger.dev/sdk/v3` or deprecated `client.defineJob` pattern
2.**Never use `node-fetch`** - Use the built-in `fetch` function
3.**Export all tasks** - Every task must be exported, including subtasks
4.**Never wrap wait/trigger calls in Promise.all** - `triggerAndWait`, `batchTriggerAndWait`, and `wait.*` calls cannot be wrapped in `Promise.all` or `Promise.allSettled`
mode:"trailing",// "leading" (default) or "trailing"
},
});
```
## Debouncing
Consolidate multiple triggers into a single execution:
```ts
// Rapid triggers with same key = single execution
awaitmyTask.trigger({userId:"123"},{
debounce:{
key:"user-123-update",
delay:"5s",
},
});
// Trailing mode: use payload from LAST trigger
awaitmyTask.trigger({data:"latest"},{
debounce:{
key:"my-key",
delay:"10s",
mode:"trailing",
},
});
```
Use cases: user activity updates, webhook deduplication, search indexing, notification batching.
## Batch Triggering
Up to 1,000 items per batch, 3MB per payload:
```ts
constresults=awaitmyTask.batchTriggerAndWait([
{payload:{userId:"1"}},
{payload:{userId:"2"}},
]);
for(constresultofresults){
if(result.ok)console.log(result.output);
}
```
## Machine Presets
| Preset | vCPU | Memory |
|-------------|------|--------|
| micro | 0.25 | 0.25GB |
| small-1x | 0.5 | 0.5GB |
| small-2x | 1 | 1GB |
| medium-1x | 1 | 2GB |
| medium-2x | 2 | 4GB |
| large-1x | 4 | 8GB |
| large-2x | 8 | 16GB |
## Design Principles
1.**Break complex workflows into subtasks** that can be independently retried and made idempotent
2.**Don't over-complicate** - Sometimes `Promise.allSettled` inside a single task is better than many subtasks (each task has dedicated process and is charged by millisecond)
3.**Always configure retries** - Set appropriate `maxAttempts` based on the operation
4.**Use idempotency keys** - Especially for payment/critical operations
5.**Group related subtasks** - Keep subtasks only used by one parent in the same file, don't export them
6.**Use logger** - Log at key execution points with `logger.info()`, `logger.error()`, etc.
## Reference Documentation
For detailed documentation on specific topics, read these files:
Almost all commands in the monorepo should be executed when `pnpm run ...` from the root of the monorepo. For example, running tests for the `@internal/run-engine` internal package:
```
pnpm run dev --filter webapp
```
But often, when running tests, it's better to `cd` into the directory and then run tests:
```
cd apps/webapp
pnpm run test --run
```
This way you can run for a single file easily:
```
cd internal-packages/run-engine
pnpm run test ./src/engine/tests/ttl.test.ts --run
description: Guidelines for creating OpenTelemetry metrics to avoid cardinality issues
globs:
- "**/*.ts"
---
# OpenTelemetry Metrics Guidelines
When creating or editing OTEL metrics (counters, histograms, gauges), always ensure metric attributes have **low cardinality**.
## What is Cardinality?
Cardinality refers to the number of unique values an attribute can have. Each unique combination of attribute values creates a new time series, which consumes memory and storage in your metrics backend.
description: Making updates to the main trigger.dev remix webapp
globs: apps/webapp/**/*.tsx,apps/webapp/**/*.ts
alwaysApply: false
---
The main trigger.dev webapp, which powers it's API and dashboard and makes up the docker image that is produced as an OSS image, is a Remix 2.17.4 app that uses an express server, written in TypeScript. The following subsystems are either included in the webapp or are used by the webapp in another part of the monorepo:
- `@trigger.dev/database` exports a Prisma 6.14.0 client that is used extensively in the webapp to access a PostgreSQL instance. The schema file is [schema.prisma](mdc:internal-packages/database/prisma/schema.prisma)
- `@trigger.dev/core` is a published package and is used to share code between the `@trigger.dev/sdk` and the webapp. It includes functionality but also a load of Zod schemas for data validation. When importing from `@trigger.dev/core` in the webapp, we never import the root `@trigger.dev/core` path, instead we favor one of the subpath exports that you can find in [package.json](mdc:packages/core/package.json)
- `@internal/run-engine` has all the code needed to trigger a run and take it through it's lifecycle to completion.
- `@trigger.dev/redis-worker` is a custom redis based background job/worker system that's used in the webapp and also used inside the run engine.
## Environment variables and testing
In the webapp, all environment variables are accessed through the `env` export of [env.server.ts](mdc:apps/webapp/app/env.server.ts), instead of directly accessing `process.env`.
Ideally, the `env.server.ts` file would never be imported into a test file, either directly or indirectly. Tests should only imported classes and functions from a file matching `app/**/*.ts` of the webapp, and that file should not use environment variables, everything should be passed through as options instead. This "service/configuration" separation is important, and can be seen in a few places in the code for examples:
- [realtimeClient.server.ts](mdc:apps/webapp/app/services/realtimeClient.server.ts) is the testable service, and [realtimeClientGlobal.server.ts](mdc:apps/webapp/app/services/realtimeClientGlobal.server.ts) is the configuration
Also for writing tests in the webapp, checkout our [tests.md](mdc:ai/references/tests.md) guide
## Legacy run engine vs Run Engine 2.0
We originally the Trigger.dev "Run Engine" not as a single system, but just spread out all over the codebase, with no real separate or encapsulation. And we didn't even call it a "Run Engine". With Run Engine 2.0, we've completely rewritten big parts of the way the system works, and moved it over to an internal package called `@internal/run-engine`. So we've retroactively named the previous run engine "Legacy run engine". We're focused almost exclusively now on moving to Run Engine 2.0 and will be deprecating and removing the legacy run engine code eventually.
## Where to look for code
- The trigger API endpoint is [api.v1.tasks.$taskId.trigger.ts](mdc:apps/webapp/app/routes/api.v1.tasks.$taskId.trigger.ts)
- The batch trigger API endpoint is [api.v1.tasks.batch.ts](mdc:apps/webapp/app/routes/api.v1.tasks.batch.ts)
- Setup code for the prisma client is in [db.server.ts](mdc:apps/webapp/app/db.server.ts)
- The run engine is configured in [runEngine.server.ts](mdc:apps/webapp/app/v3/runEngine.server.ts)
- All the "services" that are found in app/v3/services/\*_/_.server.ts
- The code for the TaskEvent data, which is the otel data sent from tasks to our servers, is in both the [eventRepository.server.ts](mdc:apps/webapp/app/v3/eventRepository.server.ts) and also the [otlpExporter.server.ts](mdc:apps/webapp/app/v3/otlpExporter.server.ts). The otel endpoints which are hit from production and development otel exporters is [otel.v1.logs.ts](mdc:apps/webapp/app/routes/otel.v1.logs.ts) and [otel.v1.traces.ts](mdc:apps/webapp/app/routes/otel.v1.traces.ts)
- We use "presenters" to move more complex loader code into a class, and you can find those are app/v3/presenters/\*_/_.server.ts
- All the "services" that are found in app/v3/services/\*_/_.server.ts
- The code for the TaskEvent data, which is the otel data sent from tasks to our servers, is in both the [eventRepository.server.ts](mdc:apps/webapp/app/v3/eventRepository.server.ts) and also the [otlpExporter.server.ts](mdc:apps/webapp/app/v3/otlpExporter.server.ts). The otel endpoints which are hit from production and development otel exporters is [otel.v1.logs.ts](mdc:apps/webapp/app/routes/otel.v1.logs.ts) and [otel.v1.traces.ts](mdc:apps/webapp/app/routes/otel.v1.traces.ts)
- We use "presenters" to move more complex loader code into a class, and you can find those are app/v3/presenters/\*_/_.server.ts
# This is used for logging in via GitHub. You can leave these commented out if you don't want to use GitHub for authentication.
# AUTH_GITHUB_CLIENT_ID=
# AUTH_GITHUB_CLIENT_SECRET=
# Resend is an email service used for signing in to Trigger.dev via a Magic Link.
# Emails will print to the console if you leave these commented out
# Configure an email transport to allow users to sign in to Trigger.dev via a Magic Link.
# If none are configured, emails will print to the console instead.
# Uncomment one of the following blocks to allow delivery of
# Resend
### Visit https://resend.com, create an account and get your API key. Then insert it below along with your From and Reply To email addresses. Visit https://resend.com/docs for more information.
# RESEND_API_KEY=<api_key>
# EMAIL_TRANSPORT=resend
# FROM_EMAIL=
# REPLY_TO_EMAIL=
# RESEND_API_KEY=
# Generic SMTP
### Enter the configuration provided by your mail provider. Visit https://nodemailer.com/smtp/ for more information
### SMTP_SECURE = false will use STARTTLS when connecting to a server that supports it (usually port 587)
# EMAIL_TRANSPORT=smtp
# FROM_EMAIL=
# REPLY_TO_EMAIL=
# SMTP_HOST=
# SMTP_PORT=587
# SMTP_SECURE=false
# SMTP_USER=
# SMTP_PASSWORD=
# AWS Simple Email Service
### Authentication is configured using the default Node.JS credentials provider chain (https://docs.aws.amazon.com/AWSJavaScriptSDK/v3/latest/Package/-aws-sdk-credential-providers/#fromnodeproviderchain)
# EMAIL_TRANSPORT=aws-ses
# FROM_EMAIL=
# REPLY_TO_EMAIL=
# Remove the following line to enable logging telemetry traces to the console
LOG_TELEMETRY="false"
# CLOUD VARIABLES
POSTHOG_PROJECT_KEY=
PLAIN_API_KEY=
CLOUD_AIRTABLE_CLIENT_ID=
CLOUD_AIRTABLE_CLIENT_SECRET=
CLOUD_GITHUB_CLIENT_ID=
CLOUD_GITHUB_CLIENT_SECRET=
CLOUD_LINEAR_CLIENT_ID=
CLOUD_LINEAR_CLIENT_SECRET=
CLOUD_SLACK_APP_HOST=
CLOUD_SLACK_CLIENT_ID=
CLOUD_SLACK_CLIENT_SECRET=
# v3 variables
PROVIDER_SECRET=provider-secret # generate the actual secret with `openssl rand -hex 32`
COORDINATOR_SECRET=coordinator-secret # generate the actual secret with `openssl rand -hex 32`
# Uncomment the following line to enable the registry proxy
# ENABLE_REGISTRY_PROXY=true
# DEPOT_ORG_ID=<Depot org id>
# DEPOT_TOKEN=<Depot org token>
# DEPOT_PROJECT_ID=<Depot project id>
# DEPLOY_REGISTRY_HOST=${APP_ORIGIN} # This is the host that the deploy CLI will use to push images to the registry
# CONTAINER_REGISTRY_ORIGIN=<Container registry origin e.g. https://registry.digitalocean.com>
# CONTAINER_REGISTRY_USERNAME=<Container registry username e.g. Digital ocean email address>
# CONTAINER_REGISTRY_PASSWORD=<Container registry password e.g. Digital ocean PAT>
uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2 # zizmor:ignore[artipacked] changesets/action pushes the release branch; no artifact upload here so no leak path
You are auditing this PR for drift against `.claude/REVIEW.md`.
## Context
`.claude/REVIEW.md` is the repo's source of truth for what AI / agent code reviewers should treat as critical findings (rolling-deploy safety, hot-table indexes, recovery-path queries, testcontainers usage, Lua versioning, etc.). It is consumed by review agents to calibrate severity. If REVIEW.md goes stale, every future agent review degrades.
## Strategy — read this first
You have a hard turn budget. Spend it on signal, not coverage. The audit is allowed to miss things; it is NOT allowed to time out.
1. Read `.claude/REVIEW.md` once, in full.
2. Run `git diff origin/main...HEAD --name-only` to get the list of changed files. Do NOT read the diff content yet.
3. Scan the file-list for relevance to REVIEW.md scope. Relevance signals: changes to Prisma schema, Redis / queue / Lua code, hot tables, recovery / restart loops, new packages, deletions of paths REVIEW.md cites. Skim everything else.
4. Open at most **5 files** total — only the ones most likely to surface a real signal. If nothing in the file-list looks relevant to any REVIEW.md rule, do NOT read any files; go straight to the verdict.
5. Form a verdict and stop. Do not exhaust the turn budget exploring.
Large PRs (>50 files changed) are a strong signal to be MORE selective, not more thorough. Pick 3-5 files at most.
## What to look for
- **Stale references** — does any REVIEW.md rule cite a file, directory, function, table, Prisma model, or package name that has been removed or renamed in this PR (or is already gone from `main`)?
- **Contradictions** — does code in this PR clearly violate a current REVIEW.md rule? (Don't re-review the PR. Only flag if REVIEW.md and the PR plainly disagree.)
- **Missing rules** — does this PR introduce a new pattern future reviewers should know about? Examples: a new hot table, a new Lua-script versioning convention, a new safety wrapper, a new "must always check" invariant.
- **Obsolete rules** — has the repo moved past a constraint REVIEW.md still asserts? (e.g. a deprecated path is gone, a pattern is now linted, V1 code is deleted.)
## Response format
If nothing needs changing:
✅ REVIEW.md looks current for this PR.
Otherwise:
📝 **REVIEW.md updates suggested:**
- **[stale]** `<rule excerpt>` — <what's stale and why>
- **[contradiction]** `<rule excerpt>` — <what in this PR disagrees>
- **[missing]** under `## <section>` — <one-sentence draft rule>
- **[obsolete]** `<rule excerpt>` — <why this rule no longer applies>
## Rules
- Maximum 3 suggestions per audit. Pick the highest-signal ones.
- Only flag things that would actually mislead a future reviewer. Style and wording do not count.
- Do NOT review the PR itself. Do NOT propose rules outside REVIEW.md's existing sections.
- Do NOT propose rules for one-off PR specifics that don't generalize to future PRs.
- If REVIEW.md does not exist in the repo, respond with `(skip)` and stop.
- When in doubt between "one more file read" and "finish now" — finish now.
Some files were not shown because too many files have changed in this diff
Show More
Reference in New Issue
Block a user
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
Eric Allam
