## Summary
When the realtime runs feed (the backend behind the `realtimeBackend`
feature flag) hydrates a change from a Postgres read replica, the read
can race the replica's apply of the very write that triggered it. The
delivered row then carries the previous change's content, and an
isolated final change (for example a last `metadata.set` before a run
goes quiet) is not corrected until the roughly 20 second backstop poll.
Measured against a replica with deliberate apply delay, every delivery
trailed exactly one change behind and a final change stranded for the
full backstop interval.
## Fix
Publishers stamp each change record with the committed row's
`updatedAt`, taken from writes they already perform, so the stamp costs
no extra queries. The router delays its wake hydrate until the replica's
measured lag has passed, anchored to that timestamp: a record that has
already spent longer than the lag in transit is hydrated immediately, so
only the racing leading edge ever waits. After hydrating, a tripwire
compares each row against its record's watermark. Still-stale rows are
withheld and retried briefly, and each detection feeds the lag estimate.
If retries run out, the rows are delivered anyway (liveness over
freshness) and follow-up re-hydrates emit the fresh version through the
normal working-set diff once the replica catches up, with the backstop
as the terminal net.
Replica lag is sampled reader-side only, and only while feeds are
active. Aurora reports live lag via `aurora_replica_status()`; vanilla
Postgres can only report "caught up or not" (mid-apply lag is not
honestly measurable from a replica), so tripwire observations floor the
estimate there. Deployments without a replica resolve to zero lag and
skip the gate entirely. Tunables live under
`REALTIME_BACKEND_NATIVE_REPLICA_LAG_*`, and
`realtime_native.stale_hydrates` plus
`realtime_native.replica_lag_estimate_ms` make replica health
observable.
Two adjacent fixes: a metadata update that writes nothing no longer
publishes a change record, and buffered parent and root metadata
operations now publish when the flusher writes them, so those changes
wake live feeds instead of waiting for the backstop.
For local testing, `docker-compose` gains an opt-in `database-replica`
service (compose profile `replica`) with a configurable
`recovery_min_apply_delay`, which reproduces replica-lag behavior
deterministically. With the gate disabled this rig reproduces the
one-change-behind delivery exactly; with it enabled, deliveries arrive
with current content at roughly the true replica lag, across write rates
faster and slower than the lag itself.
## Summary
The environment variables page loaded every variable value in the
project, unfiltered by environment. Archiving a preview branch does not
delete its environment variable value rows, so projects that churn
preview branches accumulate values forever, and every page view loaded
all of them. On large projects this made the page loader take many
seconds and stalled the server while deserializing the oversized result.
## Fix
The presenter now loads the displayed environments first and filters the
`values` relation to those environment IDs. That matches the display
semantics exactly (per-user dev environments and active branch
environments included), and the lookup is covered by the existing unique
index on `(variableId, environmentId)`. Values in archived branch
environments are no longer fetched at all.
Covered by a new testcontainers test asserting that values from active
environments (including branch environments) are returned while archived
branch environments are excluded.
## Summary
`test/runsRepositoryCursor.test.ts` pinned its fixture runs to
`createdAt = 2026-06-04T16:55:07Z`. `listRuns` applies the default 7 day
window when no time filter is given, so the fixtures aged out of the
window at 16:55 UTC on 2026-06-11 and all five tests started failing for
every branch, regardless of what the branch changed. The tests were
green on their own CI two days earlier because the fixtures were only
five days old at the time.
This switches the fixture base to a relative timestamp (one hour ago),
so the fixtures stay inside the default window permanently. Verified the
suite goes 5/5 green with this change on the same environment where the
pinned dates fail 5/5.
## Summary
The run trace page loader serialized every span's raw OTel events (with
full properties) into the response, even though the tree UI only renders
the derived `timelineEvents` and the span detail panel refetches what it
needs. On event-heavy traces that inflated both the loader payload and
the server-side heap copies built per request. This PR keeps raw span
events server-side and pairs that with a few related trace-view
improvements:
- A new optional `TRACE_VIEW_EMERGENCY_SPAN_CAP` env var (unset by
default) clamps the trace summary and detailed trace summary span limits
on both event store paths, including the public run trace endpoint, so
operators can bound trace query sizes in one place without retuning the
per-store limits.
- The TreeView virtualizer resolved every rendered row with a linear
scan over the whole tree (and `getNodeProps` did the same via
`findIndex`); rows now resolve through memoized id lookup maps, which
matters once traces reach tens of thousands of spans.
- The run stream SSE lookup now applies the same organization membership
scoping as the rest of the run page presenters, for consistency.
Behavior is unchanged by default: the trace tree renders from the same
`timelineEvents` it always has, and the new cap only takes effect when
set.
## Summary
HIPAA BAA is offered as a paid add-on on every paid plan. Each paid tier
on the in-app pricing card now has a "HIPAA BAA add-on" row with a
"Request a BAA" link that opens the existing contact dialog pre-filled
with a new `hipaa` inquiry type, prompting the user for their company
name and a brief description of the PHI workload.
The contact form's `feedbackTypes` are restructured to match the
marketing /contact form: every inquiry type carries a Plain label ID and
a "Contact form: ..." thread title, so threads land in Plain identically
whether they come from the dashboard or the marketing site. The
included-compute line on each tier also picks up the credits wording
from the marketing pricing page, and the Enterprise tier lifts its title
above the features row.
The compute suspend flow delays snapshots by `snapshotDelayMs` (~30s) so
short-lived waitpoints skip the snapshot entirely, with the intent that
a run continuing before the delay expires cancels the pending snapshot.
But the only `cancel()` call site was the `/continue` action, which
runners only invoke when restoring from an already-taken snapshot — so
pending snapshots were never cancelled (zero `snapshot.canceled` events
ever emitted in prod). When a run resumed and completed inside the
window, the stale snapshot fired ~30s later anyway, pausing the VM 6–13s
mid warm-start long-poll; the frozen guest couldn't fire its abort timer
or send a FIN, causing stalls and run-engine driven retries.
### Change
- Cancel the pending snapshot on `attempt.complete` — after the platform
accepts the completion, before the HTTP reply (so it can't reorder with
the runner's next `/suspend`).
- Cancel on `runDisconnected` (crash, exit, or run replaced on the
socket).
- Both cancels are guarded by a runnerId match (new
`TimerWheel.peek()`): a stale duplicate runner for a reassigned run must
not cancel the fresh runner's pending snapshot. A missing runnerId falls
through to an unconditional cancel (the pre-existing `/continue`
behavior is unchanged).
Waitpoint suspensions keep the runner socket connected and the attempt
incomplete, so neither hook touches a snapshot that is still wanted.
Known limitation (fail-safe direction): `socket.data.runnerId` is frozen
at the websocket handshake, so after a same-supervisor restore the
disconnect-path guard refuses the cancel. The `attempt.complete` path
uses the runner's current header id and is unaffected.
`ComputeWorkloadManager.create` swallows gateway errors currently, so a
cold start that fails placement (e.g. a netns slot with a busy tap, a
full node disk) silently abandons the dequeued run until the run
engine's `PENDING_EXECUTING` heartbeat timeout redrives it via stall
detection.
### Changes
- Retry `instances.create` with short backoff (default 3 attempts, 250ms
backoff), recording `createAttempts` in the wide event.
- **Only statuses where the create definitely did not commit are
retried**: 500 (agent/fcrun create failed) and 503 (no placement).
502/504 are excluded — the gateway emits those when it fails to reach
the node or read its response, which can happen *after* the agent
committed the create; the gateway only records the instance name on a
clean 201, so a same-name retry would miss the collision check and could
double-create the VM on another node. Network-level fetch failures are
retried (if the gateway processed the create, its name index is
populated and the retry 409s harmlessly). Timeouts are not retried.
- **Retry attempts after a 5xx use a deterministic `-rN` name suffix**:
a failed create can leave its name registered until async cleanup runs.
Attempt 1 keeps the unsuffixed name.
Fixes an intermittent `Test timed out in 30000ms` in the
`streamBatchItems` suite. Not a logic hang — the 30s budget covers
container setup, and each case boots its own per-test Redis container +
a full `RunEngine`, so under CI Docker contention a cold boot can cross
30s (which is why the failure moved between tests).
- New `containerTestWithIsolatedRedisNoClickhouse` fixture (Postgres
clone + per-test Redis, no ClickHouse) — this suite never uses
ClickHouse, but the old fixture's auto `resetClickhouse` forced a
ClickHouse boot + migration onto the cold-start test.
- Raised `testTimeout` 30s → 120s, matching the run-engine convention
for this footprint.
## Summary
Reliability and authorization fixes for realtime chat sessions:
- Session-stream waitpoint delivery is scoped to the environment, so two
environments using the same session `externalId` can no longer complete
each other's waitpoints.
- The session snapshot-url routes now enforce per-session authorization,
and appending to a session's `out` channel requires secret-key auth, so
a session-scoped token can't read another session's snapshot or forge
assistant output.
- Appends that carry an `X-Part-Id` header are deduplicated on retry, so
a retried send can't duplicate a message.
- Session creation rejects expired sessions (instead of triggering a run
that can never receive input), `externalId` is immutable after creation,
and the sessions list endpoint returns friendly `run_*` ids to match the
single-session routes.
## Rollout
The waitpoint cache key gains an environment prefix. To keep waitpoints
registered by the previous deploy working across the boundary, the drain
reads both the new and the previous key for this release; the legacy
read can be removed a release later once no pre-deploy waitpoints
remain.
## Summary
Adds a second backend for the realtime runs feed (`useRealtimeRun`,
`subscribeToRunsWithTag`, `subscribeToBatch`), built to stay healthy
when a single busy environment has many subscribers watching many runs
at once. It is gated behind a feature flag with the existing backend as
the default, so nothing changes for users until it is enabled per
environment.
## Design
A run change is published once, as a small self-describing record, to a
single per-environment channel. Every feed is then a predicate over that
one stream rather than owning a channel:
- A per-instance router indexes the currently-held feeds by run, tag,
and batch. When a run changes it hydrates the affected rows once and
serializes them once, then fans the result to every matching feed. One
hot shared tag watched by many subscribers costs a single database query
and serialize, not one per subscriber.
- Feeds that don't match a change are never woken, wake delivery per
environment is coalesced on a leading edge (250ms default) so a burst of
changes costs one wake, and cold reads coalesce onto a single
short-TTL-cached resolve.
- An admission gate bounds how many cold ClickHouse resolves run
concurrently, so a mass reconnect across many distinct filters queues
instead of stampeding the database.
- Changes that land while a client is between long-polls are delivered
on its next poll instead of waiting for the periodic backstop: each
environment buffers its recent change records, subscriptions linger
briefly after the last feed closes, and a newly-armed poll replays
exactly the connection's gap.
- The per-connection replay cursors behind that are shared across
instances via Redis (a single timestamp each), so a poll landing on a
different instance behind the load balancer still reads the connection's
true gap instead of falling back to a cold resolve. Cursor reads have a
bounded deadline and degrade to the cold-read path on any Redis trouble.
- Tag subscriptions with multiple tags match runs carrying all of the
tags, mirroring the existing backend's filter semantics, and live
long-polls hold for about 20 seconds to match its cadence.
- The per-environment channel supports Redis Cluster sharded pub/sub, so
the wake path scales horizontally across shards by environment.
- The backend reports its health through OpenTelemetry metrics (delivery
lag, poll resolution paths, backstop outcomes, replay and cursor-store
activity), with a provisioned Grafana dashboard for local development.
Everything is behind the feature flag and tunable via env vars; the
existing backend remains the default.
## Summary
When `RUN_ENGINE_READ_REPLICA_SNAPSHOTS_SINCE_ENABLED` is on,
`RunEngine.getSnapshotsSince` reads from the read replica. During write
spikes the replica can briefly lag, so the snapshot id a runner just
learned from the writer isn't visible there yet: the lookup threw, the
worker route returned a 500, and the runner waited for its next poll —
turning sub-second snapshot notifications into poll-interval latency
exactly when things are busiest. This PR makes the flag safe to enable:
a replica miss of the since snapshot gets one jittered retry on the
replica (most lag windows are shorter than the ~50–200ms wait, so the
writer is never touched), then falls back to the primary, observed via a
new `run_engine.snapshots_since.replica_miss` counter with an `outcome`
attribute (`replica_retry` vs `primary`). Only genuine misses — absent
on the primary too — remain errors.
## Design
- `getExecutionSnapshotsSince` now throws a typed
`ExecutionSnapshotNotFoundError` so the engine can distinguish the
expected lag miss from real failures. The message string is unchanged
and the error never leaves the engine.
- The recovery path only engages when the flag is on, a distinct replica
client is configured, and no transaction client was passed. With the
flag off, the path is behaviorally identical to before.
- Retry delay bounds are configurable
(`RUN_ENGINE_SNAPSHOTS_SINCE_REPLICA_RETRY_MIN_MS`/`MAX_MS`, default
50/200; `MAX_MS=0` skips the replica retry and goes straight to the
primary).
- The warn log fires only when the primary serves the read (the writer
spill is the operationally interesting event); replica-retry recoveries
are counted but quiet. A permanently-missing snapshot id stays an
error-level failure with a `failedDuring` field, so lag metrics aren't
polluted by bogus ids.
- Stale-tail lag (replica has the since snapshot but not newer rows)
deliberately still returns the replica's view; the next poll catches up.
- The since-snapshot anchor lookup is now scoped to the polled run
(`where: { id, runId }`), so a snapshot id from a different run raises
not-found instead of silently anchoring a too-wide window of the run's
snapshots.
## Test plan
All vitest + testcontainers, no mocks. A new `schemaOnlyPrisma` fixture
(migrated-but-empty clone database) simulates a replica that hasn't
caught up, and a real in-memory OTel meter pins the counter semantics
per outcome.
- [x] Replica catches up during the jittered retry window → served by
the replica, `outcome=replica_retry` = 1, primary never consulted
- [x] Replica permanently missing the since snapshot → served by the
primary, `outcome=primary` = 1
- [x] Snapshot missing on both replica and primary → null, counter = 0
- [x] Replica has the since snapshot but lags by one → the replica's
view is served, no fallback (verified discriminating power: the test
fails if reads secretly hit the primary)
- [x] Flag off with a replica configured → primary serves the read
- [x] Transaction client provided → bypasses the replica entirely
- [x] Since snapshot belonging to a different run → null
- [x] Existing getSnapshotsSince + waitpoints suites green; run-engine,
testcontainers, and webapp typechecks pass
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
## Summary
Adds `GET /api/v1/projects/{projectRef}/environments` (personal access
token auth), which lists the base environments a user can access for a
project — their own dev environment plus the project's staging, preview,
and production environments.
## Details
- Built on the PAT route builder, so it inherits org-membership auth and
the per-resource ability check.
- `dev` is scoped to the token owner; archived environments are
excluded.
- Returns the branchable **parent** preview environment — preview branch
children are not included. A consumer targets the parent; branch-level
overrides are handled separately.
- Sorted to match the dashboard's environment switcher (dev → staging →
preview → prod), and never returns API keys.
Example response:
```json
[
{ "id": "...", "slug": "dev", "type": "DEVELOPMENT", "isBranchableEnvironment": false, "branchName": null, "paused": false },
{ "id": "...", "slug": "stg", "type": "STAGING", "isBranchableEnvironment": false, "branchName": null, "paused": false },
{ "id": "...", "slug": "preview", "type": "PREVIEW", "isBranchableEnvironment": true, "branchName": null, "paused": false },
{ "id": "...", "slug": "prod", "type": "PRODUCTION", "isBranchableEnvironment": false, "branchName": null, "paused": false }
]
```
## Summary
The dashboard's Agent view rendered `source-url` and `file` message
parts by putting their `url` straight into an `href`/`src`. Those URLs
come from streamed agent and tool data, so a tool that emitted something
like `javascript:alert(1)` produced a clickable XSS payload in the
dashboard.
## Fix
A `toSafeUrl` helper now gates every URL before it reaches an
`href`/`src`: it allows only `http:`/`https:`/`blob:` (and
`data:image/...` for inline images) and returns `null` for anything
else. Unsafe values render as plain text instead of a link or image, so
a hostile or malformed URL degrades gracefully rather than becoming
clickable. Safe URLs render exactly as before. Covered by a unit test
over the allow/deny list.
## Summary
During `trigger()` worker-queue resolution, `getWorkerQueue` wrapped any
error from `getDefaultWorkerGroupForProject` into a client-facing
`ServiceValidationError` (HTTP 422) carrying `error.message`. That
method runs `project.findFirst` on the **writer**; when the writer is
unreachable Prisma throws a connection error (P1001) whose message
includes the database host, and that raw message was returned to the API
client and surfaced in the run view via the SDK's `TriggerApiError`.
It also mis-classifies a transient outage: a 422 is not retried by the
SDK, so triggers failed permanently instead of riding out a brief writer
blip.
## Design
This is the only place on the trigger path that folds a *caught* error's
message into a client-facing error — every other DB failure on the path
propagates to the route's generic 500 handler (scrubbed, and retried by
the SDK). So the fix is local:
- Add `isInfrastructureError()` — true for Prisma connection-level
failures (the DB-unreachable family: P1001/P1002/P1008/P1017, plus the
init/panic/unknown client error classes), false for query/validation
errors (e.g. P2002).
- At the wrap site, rethrow infrastructure errors so they reach the
generic 500 handler (no raw message, and retryable). Genuine domain
failures (e.g. "Project not found.") still become a 422.
Only P1001 ("can't reach database server") has been observed in
practice; the rest of the connection family is included as same-class
forward-proofing.
## Test plan
- [x] Unit: `isInfrastructureError` classifies a P1001 (incl. the Prisma
6.x `PrismaClientKnownRequestError` shape) and init errors as
infrastructure; P2002 and a plain `Error` as not
- [x] `getWorkerQueue` rethrows a P1001 unchanged instead of wrapping it
in a `ServiceValidationError`; still wraps a domain failure as a
`ServiceValidationError` — RED on current code, GREEN after
- [ ] (optional) toxiproxy e2e: trigger with the writer cut → HTTP 500
generic body, no DB host in the response
---------
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
## Problem
Two backward-pagination bugs in `ClickHouseRunsRepository.listRunIds`,
both pre-existing (they predate the composite-cursor work in #3852 and
were spotted during/after it):
**1. Wrong slice (straddled pages).** `listRunRows` fetches `page.size +
1` rows to detect `hasMore`. That extra row is the one *farthest from
the cursor* in both directions (forward orders DESC; backward orders
ASC), so it's always the *trailing* element. Forward correctly used
`rows.slice(0, size)`, but backward+`hasMore` used `rows.slice(1, size +
1)` — dropping the row *closest* to the cursor and keeping the has-more
sentinel. The page straddled two logical pages (one run from the correct
previous page + one from the page before it), so paging "newer" across a
boundary **repeated and skipped** runs.
**2. Stranded forward cursor on a partial backward page.** In the
backward `!hasMore` branch, `nextCursor` was `reversedRows.at(page.size
- 1)`. On a partial page (fewer than `page.size` rows — reachable via
`runs.list` by passing a forward page's cursor as `page[before]`), that
index overshoots → `undefined` → `nextCursor` becomes `null`, leaving no
way to page forward again.
## Fix
- **Slice:** both directions now slice `rows.slice(0, size)` (the
sentinel is the trailing element either way).
- **Partial-page cursor:** the backward `!hasMore` branch takes the
oldest row on the page, `rows.at(0)`, for `nextCursor` — equivalent to
the old expression for full pages, correct for partial ones.
Forward pagination, the cursor *values* for full pages, and the `hasMore
=== true` paths were already correct and are unchanged.
## Tests
`runsRepositoryCursor.test.ts` gains two cases (both fail on `main`,
pass here):
- **multi-page backward walk:** forward across all pages, then backward
from the last page — each backward page must *exactly* reproduce the
corresponding forward page (no straddling: `main` returns `{b,c}`
instead of `{c,d}`), and the full traversal covers every run once.
- **partial backward page:** backward onto a partial first page must
still expose a working forward cursor (and paging forward from it
reaches the rest) — `main` returns a `null` nextCursor.
The three existing cursor tests (forward completeness, backward
round-trip, legacy cursor) still pass.
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
## What
`dev` and `deploy` now fail with a clear error when two tasks are
defined with the same id — including across task types (e.g. a scheduled
task and a regular task sharing an id).
## Why
Tasks are registered into the resource catalog keyed by id, so a second
definition with the same id silently overwrote the first. One of the
tasks would just vanish from the worker with no warning — easy to miss,
hard to debug. (Any earlier duplicate-id check ran against the
post-registration task list, which is already de-duplicated, so it never
actually fired.)
## How
- **Detect at registration** (`@trigger.dev/core`):
`StandardResourceCatalog` records a collision when a task id is
registered more than once, capturing the files involved — the only point
where both definitions are visible before the id-keyed map collapses
them. Exposed via `listTaskIdCollisions()`.
- **Fail indexing** (`trigger.dev` CLI): both index workers report
collisions via a new `TASKS_FAILED_TO_INDEX` message;
`indexWorkerManifest` rejects with a new `DuplicateTaskIdsError`. `dev`
renders a dedicated error (offending ids + files + docs link); `deploy`
fails with the same message. Runtime worker boot is unaffected — it
never reads the collisions.
- **Server-side backstop** (webapp): background-worker registration also
rejects duplicate ids with a clear `ServiceValidationError`, so
duplicates are caught even from an older CLI.
## Testing
- Unit tests for collision collection in the catalog and for the
error-message formatting (standard, same-file, and 3+-definition cases).
- Verified end to end against a local webapp: a project with a regular
task and a scheduled task sharing an id now fails `dev` with the
dedicated error; a project with distinct ids still starts normally.
## Changeset
Patch for `@trigger.dev/core` and `trigger.dev`.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
## Summary
- `internal-packages/rbac/src/index.ts` — in `LazyController.load()`'s
catch block, throw an Error when `process.env.REQUIRE_PLUGINS === "1"`
instead of silently falling back. The throw is captured into the lazy
controller's init promise, so it surfaces on the first method call.
- `apps/webapp/app/routes/healthcheck.tsx` — `await
rbac.isUsingPlugin()` after the DB ping. With `REQUIRE_PLUGINS=1` and a
failed plugin load, the throw surfaces here and the healthcheck returns
500 → readiness probe fails → rollout is rolled back. Noop for
self-hosters.
- `.server-changes/require-plugins-fail-fast.md` — server-changes entry.
- `internal-packages/rbac/src/require-plugins.test.ts` — 4 unit tests
covering loader branching: unset → fallback, `=1` → throw,
`forceFallback: true` wins, only exactly `"1"` enforces.
- `internal-packages/testcontainers/src/webapp.ts` — adds
`requirePlugins?: boolean` to `StartWebappOptions`. Implies
`forceRbacFallback: false`.
- `apps/webapp/test/healthcheck-require-plugins.e2e.test.ts` — e2e
closes the loop: spawns a real webapp, hits `/healthcheck` via HTTP,
asserts 500 with `REQUIRE_PLUGINS=1` and 200 without.
## Motivation
Today the RBAC plugin loader catches any plugin-load failure (missing
module, broken transitive dep, init throw) and silently returns the
default fallback implementation. This is the correct behaviour for
self-hosters who don't ship the plugin — but it's dangerous in
deployments where the plugin is expected to load: an
accidentally-missing or broken plugin would silently disable
enforcement.
`REQUIRE_PLUGINS=1` makes the loader fail loudly in those deployments.
The variable name is intentionally plural and generic — future plugin
contracts (audit logs, SSO) can read the same flag without renaming.
Closes
[TRI-9852](https://linear.app/triggerdotdev/issue/TRI-9852/require-plugins1-fail-fast-for-required-plugin-loads).
## Test plan
- [x] `pnpm run test --filter @trigger.dev/rbac` — 38/38 tests pass,
including the 4 new loader tests
- [x] `pnpm run typecheck --filter webapp` — passes
- [x] `pnpm run typecheck --filter @trigger.dev/rbac --filter
@internal/testcontainers` — passes
- [x] e2e test added (`healthcheck-require-plugins.e2e.test.ts`) — CI
runs it via `e2e-webapp.yml`. Couldn't run locally (no Docker daemon
up); CI has Docker provisioned.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
## Summary
Runs routed onto a dedicated scheduled worker queue were showing under a
phantom region in the dashboard, run details, and the API, and slipped
through region filters. They now resolve to their real region
everywhere.
## Fix
A worker queue can carry a `:scheduled` suffix that routes
scheduled-lineage runs onto their own list. That suffix is an internal
routing detail, but it was leaking anywhere the worker queue is read as
a region. A `baseWorkerQueue` helper strips any `:<class>` suffix back
to the base region (region names never contain a colon, so it's
everything before the first colon) and is applied at every region read
site: the runs list, run detail, the public API, and replay's region
override. The runs-replication writer also stores the base region in
ClickHouse so the region filter matches.
## Summary
The `mollifier.decisions` metric only carried an `outcome` label, so for
an org that has the mollifier enabled there was no way to see how often
its triggers pass through the gate instead of being diverted — making it
hard to tell why the trip isn't firing for an opted-in org.
This adds two bounded labels: `enrolled` (`"true"`/`"false"`, the
per-org flag) and `org` (the org id, attached **only** when `enrolled`
is true). For an enrolled org you can now compare directly:
`mollifier.decisions{outcome="pass_through", enrolled="true",
org="<id>"}` vs `{outcome="mollify", ...}`.
## Design
`recordDecision` now takes an options object (`{ reason?, enrolled,
orgId? }`). The `org` label is restricted to the enrolled cohort to keep
cardinality bounded — the guard lives in a pure `decisionLabels` helper,
so a non-enrolled org id can never be attached even if one is passed.
The enrolled set is small and capped operationally.
The per-org flag is resolved once at the top of `evaluateGate`
(in-memory, no DB round-trip on the trigger hot path) so every decision
— including the debounce / one-time-use-token / triggerAndWait bypasses
— is labelled consistently.
## Test plan
- [x] `mollifierGate.test.ts` cascade asserts `enrolled`/`org` on every
gate branch
- [x] `mollifierDecisionLabels.test.ts` (new) proves `org` is dropped
for non-enrolled even when an id is passed (cardinality guard)
- [x] `vitest run mollifierGate mollifierDecisionLabels` — 34/34 pass
- [x] `pnpm run typecheck --filter webapp` clean
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Add an optional network_labels field to the internal compute client's
create and restore request schemas and forward per-VM endpoint labels on
both paths, so a restored VM keeps the same labels as a freshly-booted
one. Mirrors the label the Kubernetes workload manager already sets on
the run pod.
---------
Co-authored-by: nicktrn <55853254+nicktrn@users.noreply.github.com>
## Problem
`ClickHouseRunsRepository.listRunIds` / `listRuns` order results by the
composite key `(created_at, run_id)`, but the cursor predicate cut on
`run_id` **alone**:
```ts
.where("run_id < {runId: String}", { runId: cursor })
.orderBy("created_at DESC, run_id DESC")
```
This is only sound when `run_id` lexicographic order matches
`created_at` order. `run_id`s are cuids — only coarsely time-sortable —
so when a burst of runs is created within a sub-second window, the two
orders can diverge. When they do, the next-page predicate (`run_id <
cursor`, where `cursor` is the *last* page element = the smallest
`created_at`, not necessarily the smallest `run_id`):
- **re-includes** rows already returned on a previous page (duplicates),
and
- **skips** rows it should have returned (silent data loss).
For bulk **replay** this caused runs to be replayed more than once
(replay has no idempotency guard). For the dashboard and the `runs.list`
API it could silently repeat or skip runs at page boundaries.
## Fix
Make the cursor predicate match the composite ordering:
- Cursors now encode the full `(created_at, run_id)` key as an **opaque
URL-safe base64 token**
(`base64url({"c":<createdAtMs>,"r":"<runId>"})`), and the query cuts on
the matching tuple — `(created_at, run_id) < (…)` forward / `> (…)`
backward.
- The `ORDER BY` is unchanged, so the query stays aligned with the
table's primary key — no performance regression (the tuple range
predicate is actually more index-friendly than `run_id <` alone).
- Cursors are **server-issued opaque tokens** (the SDK only echoes
`pagination.next` / `pagination.previous` back), so this needs **no
client/SDK update**. Legacy cursors were the bare internal `run_id`;
they're detected by decode failure (a cuid isn't a valid base64-wrapped
JSON payload) and fall back to the old `run_id`-only predicate, so
in-flight cursors keep working and drain naturally. New cursors also no
longer expose a bare internal run id.
- `listRunIds` is now the single cursor-aware list primitive: it returns
`{ runIds, pagination: { nextCursor, previousCursor } }`, and `listRuns`
builds on it (one place constructs cursors). Bulk actions consume the
same method and advance by `pagination.nextCursor`, finishing when it's
`null`.
- `getTaskRunsQueryBuilder` now also selects
`toUnixTimestamp64Milli(created_at) AS created_at_ms`, using a dedicated
`TaskRunListQueryResult` schema. The shared `TaskRunV2QueryResult` stays
`run_id`-only so the run-engine pending-version lookup
(`getPendingVersionIdsQueryBuilder`, which selects only `run_id`)
doesn't fail validation on a column it doesn't query.
## Tests
New `runsRepositoryCursor.test.ts` (testcontainer-backed, real
Postgres→ClickHouse replication):
- **forward** pagination returns every run exactly once when `run_id`
order is the reverse of `created_at` order (reproduces the
duplicate/skip bug — fails on `main`; this
walk-until-`nextCursor`-null-and-assert-complete is exactly the bulk
action's iteration),
- **backward** pagination round-trips to the previous page across a
boundary,
- **legacy** bare-`run_id` cursor still uses the old predicate
(backwards compatibility).
The existing `runsRepository` suites (part1–4) still pass; `part4`'s
`count new runs with listRunIds` test was updated for the new `{ runIds,
pagination }` return shape, and the `clickhouse` `taskRuns`
query-builder snapshots were regenerated for the added `created_at_ms`
column.
## Notes
- Separate, pre-existing issue (out of scope, not introduced here):
`listRuns`' backward display-slicing (`rows.slice(1, size+1)` when
`hasMore`) has an off-by-one that can return a straddled page. Tracked
separately.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Speeds up and de-flakes the unit-test suite: testcontainers booted once
per vitest worker (per-test isolation kept only where a test runs
background redis work that outlives it), a duration-weighted shard
sequencer so each shard does roughly equal work, the slowest suites
split, two genuine flakes fixed (`streamBatchItems` shared-redis leak;
run-engine waits that relied on fixed sleeps), and transient DockerHub
pulls retried.
**Timings (CI, per-shard wall):** worst unit-test shard ~771s → ~294s;
packages/webapp shards ~250-270s, most internal ~190-240s. All 25 shards
green.
A shard breaks down as ~70s fixed setup (install / image-pull /
generate) + ~70s cold `^build` + the actual container tests. So the
remaining cost is mostly the tests themselves plus that fixed setup.
**Next (separate, timings):**
- **typecheck (~6m24s)** — the slowest check overall; bound by
full-graph `tsc`, not the TS version (a TS6 branch is still ~6m17s). The
real lever is **tsgo** (the Go compiler).
- Possible later: turbo CI caching could trim the ~70s cold build on
*warm* runs, but it's conditional (cold runs rebuild anyway) and doesn't
touch setup or test time — secondary.
`cli-v3` e2e and `sdk-compat` are path-gated (don't run on test-infra
changes) and already comfortably fast.
## Summary
Adds a trace export to the run page. From the new **Export trace** menu
you can
copy a run's full trace to the clipboard as Markdown (for pasting into
an AI
assistant) or download it as a flat Log, a Markdown table, or JSON
Lines.
Internal engine-debug events are filtered out by default, and errors are
surfaced inline with their message.
## Design
The export streams events from the store to the gzipped response one at
a time
and never materialises the span tree, so a trace of any size exports
with
bounded memory and without stalling the server. Output is flat and
chronological: each line carries its own `spanId ← parentSpanId`, so the
hierarchy is reconstructable without nesting. Formats share a single
streaming
pipeline and are pluggable via `?format=log|jsonl|markdown`, so adding a
format
is an isolated change.
## Screenshots
**Export menu**
<img width="370" height="252" alt="trace-export-menu"
src="https://github.com/user-attachments/assets/3d10304a-8c49-4606-b15d-2859b137419f"
/>
**In context**
<img width="2400" height="1802" alt="trace-export-run-page"
src="https://github.com/user-attachments/assets/46c80b30-303b-47c6-9ace-a2fb06f6cb61"
/>
## Summary
Opening or closing the Bulk action inspector should not affect the Runs
list, but it was still triggering refresh-related UI behavior. This PR
fixes that and smooths out a few related inspector interactions.
## Changelog
Stop reloading the runs list (and flashing its loading state) when
opening or closing the
Bulk action inspector. Filtering, pagination, and explicit refresh are
unaffected.
The supervisor can now pause dequeuing - and freeze consumer-pool
scale-up - when a backpressure signal says the cluster can't place more
work, then ramp dequeuing back up gradually once it clears. The signal
is a verdict published to a Redis key by a cluster-side component; the
supervisor reads it on a short refresh and gates `preDequeue` on it.
Off by default (`TRIGGER_DEQUEUE_BACKPRESSURE_ENABLED`). Everything
fails open: a missing, stale, or unreadable verdict never pins the
brake, and the hot-path read is a synchronous cached lookup with no I/O.
The scale-up freeze leaves scale-down untouched, and on release the
resume is ramped so a deep queue isn't hammered all at once.
Dry-run is on by default (`TRIGGER_DEQUEUE_BACKPRESSURE_DRY_RUN`): even
once enabled it only logs what it would have done, and surfaces the
computed state through metrics, until explicitly set to act. Prometheus:
`supervisor_backpressure_engaged`, `_dry_run`,
`_skipped_dequeues_total`.
Refs TRI-5354
## Summary
Scheduled runs and their descendants can now be routed to a dedicated
per-region worker queue, processed by a separate worker fleet, so a
burst of scheduled crons no longer competes with standard and agent runs
for the same queue and inflates their startup latency. It is off by
default and enabled per organization via a feature flag (with a global
default), so nothing changes until it is turned on.
## Design
At trigger time, any run whose lineage originates from a schedule
(`rootTriggerSource === "schedule"`, which already propagates from a
scheduled run down to all of its children) gets its worker queue
suffixed with `:scheduled`. The worker queue name is an opaque string
persisted on the run and used verbatim by enqueue and dequeue, so this
needs no Lua, message-envelope, or concurrency changes. Concurrency
stays keyed by environment and queue, not by worker queue.
On the consumer side, the dequeue endpoint gains an optional
`queueClass` selector. A supervisor sends `queueClass: "scheduled"` and
the server derives the actual queue from the worker's own group, so a
token can only ever reach its own region's queues. A fleet picks its
class with the `TRIGGER_WORKER_QUEUE_CLASS` env var (`default` or
`scheduled`), so a dedicated scheduled fleet can run alongside the
standard one.
Verified end to end against a local managed-worker setup: scheduled runs
route to the dedicated queue, are drained only by the scheduled fleet,
and standard runs are left untouched.
## Summary
This PR adds packet path validation before key construction and
presigning. Invalid paths are rejected before reaching either
object-store client implementation, ensuring consistent behavior
regardless of the underlying storage configuration.
## Summary
Three fixes to the AI agent surface in the dashboard, all surfaced while
verifying AI SDK 7 support (the SDK side is #3833):
- **AI SDK 7 telemetry rendering.** The generation-span inspector and
run metrics now read both the v6 (`ai.*`) and v7 (`gen_ai.*`) telemetry
attribute shapes. On v7 the Messages, Provider, and Tools views showed
empty/unknown before; now they render correctly.
- **HITL approvals in the conversation view.** The agent conversation
view renders human-in-the-loop tool approvals and denials (awaiting
approval, approved, denied with reason) instead of leaving the tool part
blank. This gap predated v7.
- **Chat snapshot store mismatch.** Chat session snapshots now resolve
through a single storage-key helper shared by the SDK write and the
dashboard read. Previously the write applied the default object-store
protocol to the key while the read fell back to a different store, so
the dashboard 404'd on the snapshot and showed only a partial
conversation.
## Summary
On a busy webapp the trace/log/metric ingestion path emits several
`info` logs per insert batch, which makes up the bulk of the service's
log output. This moves that per-batch chatter to `debug` and adds an
opt-in to drop successful HTTP access logs, cutting log volume with no
loss of error signal.
## Details
The per-batch ClickHouse insert logs, the flush scheduler's concurrency
adjustments, and the event-loop utilization sample (already exported as
a metric, so the log line was redundant) now log at `debug`. Error and
warning logs are untouched.
New `HTTP_ACCESS_LOG_DISABLED=1` env var: when set, the HTTP access
logger skips successful (2xx) requests while still logging non-2xx
responses. Defaults off, so existing deployments are unchanged.
## Summary
This PR improves performance across the Environment Variables page.
## Changes
### Targeted value loading
- load only the non-secret (environmentId, key) pairs required by the
page. Secret values continue to be redacted in the UI.
### SSR windowing + virtualization
- SSR-render only the first 50 rows
- hydrate those rows
- virtualize the remaining dataset client-side
- search is now URL-driven during SSR, ensuring deep links such as
`?search=DATABASE_URL`
### Lightweight 'Create' flow
- 'Create' page no longer loads the full Environment Variables dataset.
## Results
Large projects no longer render thousands of rows during SSR.
Example (~11k rendered rows):
Metric | Before | After
-- | -- | --
Document size | ~150 MB | ~5 MB
SSR rows | ~11k | 50
Browser DOM rows | Thousands | ~26–38
## Summary
The mollifier had ~21 behavioural constants baked in as hardcoded values
— the buffer's ack-grace TTL and Redis retry/reconnect tuning, the
drainer's poll interval and backoff envelope, the pre-gate idempotency
claim TTL/wait/poll, the buffered-run mutate-with-fallback wait loop,
the metadata CAS retry budget and backoff, the stale-sweep scan bounds,
and the draining-gauge interval. None could be adjusted without a code
change, which makes tuning the system under production load impossible.
This exposes all of them as `TRIGGER_MOLLIFIER_*` environment variables,
each defaulting to its previous hardcoded value. Behaviour is identical
unless an operator sets a var, so it's a safe no-op deploy.
## Design
The package-level classes (`MollifierBuffer`, `MollifierDrainer` in
`@trigger.dev/redis-worker`) gain optional constructor options
defaulting to the old constants — backward compatible, hence a patch
changeset. The webapp factories and worker bootstraps read the env and
pass them through. The route- and concern-level pure helpers
(mutate-with-fallback, metadata mutation, idempotency claim, stale-sweep
state) keep their existing `?? DEFAULT` option fallbacks and are fed env
values at their call sites, so they stay unit-testable without importing
`env.server`.
## Test plan
- [x] `@trigger.dev/redis-worker` builds
- [x] webapp typecheck passes
- [x] mollifier buffer + drainer testcontainer suites pass (modulo a
couple of pre-existing flaky timing tests)
- [x] Reviewer: confirm the `TRIGGER_MOLLIFIER_*` env var names match
ops conventions
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>
## Summary
Adds a nullable `pricing_unit` column to the LLM model registry's
`llm_models` table, recording how each model is billed ("tokens",
"characters", "images", "minutes", "requests", "free", "not_findable").
It lets pricing-coverage reporting exclude models that aren't priced
per-token (image/video/audio models currently drag the "% priced" number
down even though they can never carry a per-token price), and lays the
groundwork for non-token pricing.
The default model catalog is entirely per-token, so `seed` and
`syncLlmCatalog` set `pricing_unit="tokens"` on those rows. The admin
LLM model form (create + edit) and the admin API get a pricing-unit
selector so admin-curated models can set it; existing rows can stay
unset.
Auto-discovered models get their unit from the model-registry pipeline,
which lands separately.
---------
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
The reference/example projects (`references/`) now live in their own
repo, https://github.com/triggerdotdev/references, so their heavy,
frequently-changing dependencies are no longer part of this repo's
lockfile and tooling. This removes them here and repoints everything
that referenced them.
- Deletes `references/`; updates the pnpm workspace + lockfile.
- Clears the references-only CI rules and `.vscode` configs.
- Repoints the docs (contributor/agent + one public page) to the new
repo.
- `seed.mts` keeps the local-dev projects (hello-world, d3-chat,
realtime-streams).
Brings the OpenTelemetry packages up to the latest coherent release
across the webapp and the published packages (`@trigger.dev/core`, the
CLI, `@trigger.dev/sdk`) plus
`internal-packages/{tracing,testcontainers}`:
- `@opentelemetry/sdk-node` 0.218.0
- `@opentelemetry/core` 2.7.1
- `@opentelemetry/host-metrics` 0.38.3
We were already on the otel 2.x line, so this is a same-major minor move
- the versions are pinned to `@opentelemetry/sdk-node@0.218.0`'s own
declared dependency set so the experimental (0.2xx) and stable (2.x)
packages stay coherent (mixing them is the usual otel breakage).
**One code change:** otel 0.215 made `forceFlush()` a required method on
`LogRecordExporter`, so `ExternalLogRecordExporterWrapper` (core's
tracing SDK) gains a `forceFlush()` that delegates to the underlying
exporter.
**Notable upgrades along the way:** OTLP exporters can take a custom
HTTP agent (connection pooling/keepAlive on the export path), HTTP
request headers are captured at span creation, and core hot-path perf
improvements in 2.6.1/2.7. `host-metrics` 0.37→0.38 is a clean upgrade.
Patch changeset added for the three published packages. References
projects are intentionally untouched.
Verified: `@trigger.dev/core` / CLI / `@trigger.dev/sdk` build, webapp +
`@internal/tracing` typecheck - all green.
Adds wide-event observability for the supervisor: one flat-keyed JSON
line per dequeue iteration, workload-server route, and run socket
lifecycle event. Events carry `trace_id` sourced from the inbound W3C
traceparent plus `meta.run_id` and related identifiers, so they join
across services by run.
The outbound warm-start POST also forwards the inbound traceparent so
the upstream receiver continues the same trace instead of minting a new
one.
Off by default behind `TRIGGER_WIDE_EVENTS_ENABLED`. With the flag off,
no events are emitted, no ALS state is allocated, and the outbound
warm-start request is unchanged — every call site was audited to confirm
the off path is byte-identical to current behavior.
Dequeue-path phase timings recorded under `phase.<name>.duration_ms`:
`restore`, `warm_start`, `workload_create`. A `path_taken` extra
distinguishes `restore` / `warm_start` / `cold_create` /
`skipped_no_image`.
Refs TRI-9480.
## Summary
The Runs list now updates live without requiring a page refresh. Status
changes and other run fields are updated in place while runs are
executing.
When new runs matching the current filters are created, a "New runs
created" refresh button appears above the list.
Root runs now show a live child-run status breakdown directly in the
status tooltip.
### List live update
- Visible runs update in place while they are still running.
- A "New runs created" refresh button appears when new matching runs are
detected.
- Polling stops when all visible runs have finished and a refresh button
is already shown.
- Polling pauses when the browser tab is not visible.
- Runs list status updates and new-run detection share a single
runs/live polling path.
### Child-status tooltip
- Root run tooltips now display a breakdown of child run statuses.
- Child statuses are loaded when the tooltip opens (after a 400ms hover
delay).
- The tooltip stays up to date while child runs are still changing
state.
- Handles cases where child runs continue running after their parent run
has completed, or have not yet been created.
### Supporting changes
- Added hidden-tab awareness to polling.
- Added safeguards around polling inputs (`runIds` deduping and limits).
## Test plan
- [x] pnpm run typecheck --filter webapp passes
- [x] cd apps/webapp && pnpm run test
./test/presenters/mapRunToLiveFields.test.ts --run passes
- [x] cd apps/webapp && pnpm run test
./test/runsRepository.part2.test.ts --run -t "hasNewRuns" passes
### Manual smoke:
- [x] Active runs update without a page refresh.
- [x] A new matching run shows the refresh banner and the banner actions
work as expected.
- [x] Root run tooltips show live child-status updates and stop polling
once child runs settle.
---------
Co-authored-by: Ekaterina Bulatova <kathiekiwi@Ekaterinas-MacBook-Pro.local>
## Problem
On environments where runs carry a Postgres-backed `taskEventStore`
value (`taskEvent` or `taskEventPartitioned`), OTLP ingest endpoints
(`POST /otel/v1/traces` and `/otel/v1/logs`) were returning HTTP 500.
**Root cause:** The org-scoped ClickHouse factory introduced in a recent
PR routes all OTLP spans through `getEventRepositoryForOrganizationSync`
→ `buildEventRepository`. That function only handles `"clickhouse"` and
`"clickhouse_v2"` store values and throws `Unknown ClickHouse event
repository store: <value>` for anything else. The throw occurred inside
the grouping loop of `#exportEvents`, unwinding the entire method and
returning 500 for the whole batch.
The OpenTelemetry collector's `otlphttp` exporter treats HTTP 500 as
non-retryable and drops the batch — causing real span loss.
**Fix:** Guard the `getEventRepositoryForOrganizationSync` call in
`#exportEvents` so it is only invoked for `clickhouse` / `clickhouse_v2`
store values. All other values are routed directly to the Postgres
`eventRepository`, matching the guard pattern already present in
`resolveEventRepositoryForStore` and `getEventRepositoryForStore` in
`eventRepository/index.server.ts`.
The ClickHouse factory call is also wrapped in a try/catch that falls
back to Postgres so any unexpected store value in a future OTLP batch
degrades gracefully instead of failing the whole request.
## Changes
- `apps/webapp/app/v3/otlpExporter.server.ts` — add Postgres routing
guard and try/catch fallback in `#exportEvents`
## Testing
The `eventRepository/index.server.ts` module already has the same guard
pattern thoroughly covered. The fix brings `#exportEvents` into
alignment with that existing, tested pattern. Manual verification:
confirm OTLP batches containing Postgres-store spans return 200 and
route to the correct repository.
## Summary
The environment variables import API now accepts an optional `isSecret`
flag, so imported variables can be created as secret (redacted)
environment variables instead of plaintext. When the flag is omitted,
variables default to non-secret, preserving existing behavior for CLI
deploys and dashboard imports.
This is useful for tools that push secrets into Trigger.dev (for
example, syncing from a secrets manager) and want them stored as secrets
rather than plain environment variables.
It's available through `envvars.import` in the SDK and the `POST
/api/v1/projects/{projectRef}/envvars/{slug}/import` endpoint, and is
honored for both regular and preview-branch environments.
```ts
await envvars.import("proj_1234", "prod", {
variables: { STRIPE_SECRET_KEY: "sk_live_..." },
isSecret: true,
});
```
## Summary
Adds `drainBatchSize` to `MollifierDrainer` (default `1` — preserves
existing behaviour) and wires `TRIGGER_MOLLIFIER_DRAIN_BATCH_SIZE`
through the webapp (default `50`). Each tick the drainer now pops up to
`drainBatchSize` from each chosen env, then dispatches every popped
entry through the shared `concurrency`-bounded `pLimit`. Per-org/per-env
fairness is unchanged — only the in-env pop count grows.
Pre-existing behaviour was one pop per env per tick. For a single-env
burst that single-flighted the drain at the per-tick floor of `pop +
engine.trigger ≈ 50–60 ms`. With buffer entries piling up under a
real-world tenant burst that's tens of minutes of tail latency to fully
materialise — even though PG itself could comfortably sustain the
writes.
## Why this matters — heavy-tail illustration
Scenario: 100 customers in one window — 94 fire 20 triggers each, 5 fire
100, 1 fires 1000. Gate at `THRESHOLD=10/s`, `HOLD_MS=500`. First 10 of
each burst hit PG directly; the rest buffer.
| Customers | Triggers each | PG direct | Buffered each | Total buffered
|
|---|---|---|---|---|
| 94 small | 20 | 10 | 10 | 940 |
| 5 medium | 100 | 10 | 90 | 450 |
| 1 heavy | 1000 | 10 | 990 | 990 |
**With `DRAIN_BATCH_SIZE=50`, `DRAIN_CONCURRENCY=50`, ~50 ms
`engine.trigger`:**
| Tick | Pops | Dispatch waves | Wall-clock |
|---|---|---|---|
| 1 | 94×10 + 5×50 + 1×50 = 1 240 | 25 × 50 ms | ~1 300 ms (94 smalls
done) |
| 2 | 5×40 + 1×50 = 250 | 5 × 50 ms | ~300 ms (5 mediums done) |
| 3–20 | heavy alone, 50/tick | 1 × 50 ms | ~100 ms each |
| Customer class | Buffered fully drained |
|---|---|
| 94 small | **~1.3 s** |
| 5 medium | **~1.6 s** |
| 1 heavy | **~3.4 s** |
**Without batching (one pop per env per tick — current behaviour):**
| Customer class | Buffered fully drained |
|---|---|
| 94 small | ~500 ms |
| 5 medium | ~4.5 s |
| 1 heavy | **~49 s** |
So the heavy single-tenant tail drops from ~49 s to ~3.4 s (~14× faster)
without changing PG load characteristics. Smalls go up slightly in this
scenario (500 ms → 1.3 s) because all 100 envs share one tick's dispatch
queue — that's the trade we accept for the heavy tail; the worst-case
small wait is still inside one tick. PG load is identical either way (50
concurrent inserts at a time, capped by `DRAIN_CONCURRENCY`).
## What changed
**`packages/redis-worker`**
- New `drainBatchSize` option (default 1 — full backward compat).
- `runOnce()` refactored to pop per-env batches in parallel, then
dispatch all popped entries through the existing global `pLimit`.
Mid-batch pop failure aborts only that env's batch and counts as one
failure (same semantic as the old per-env path).
- Removed the now-unused `processOneFromEnv` helper.
**`apps/webapp`**
- `TRIGGER_MOLLIFIER_DRAIN_BATCH_SIZE` env var (default 50, matching
`DRAIN_CONCURRENCY`).
- Wired into `mollifierDrainer.server.ts`.
**Test cloud config** (separate cloud PR):
`TRIGGER_MOLLIFIER_DRAIN_BATCH_SIZE="50"` on the worker service.
Production rollout deferred until we've watched it on test cloud.
## Test plan
- [x] All 25 stub-based drainer tests pass (18 pre-existing + 7 new). 7
new tests under `MollifierDrainer.drainBatchSize`:
- pops up to `drainBatchSize` across ticks
- global `concurrency` cap still holds when batch > concurrency
- mid-batch pop failure isolation
- multi-env batch fan-out in one tick
- **hierarchical org fairness preserved at `drainBatchSize > 1`**
(load-bearing — guards against future regressions to
per-env-instead-of-per-org rotation)
- mixed success/failure accounting in a batched tick
- bounded pops on empty queue (no Lua spam past `drainBatchSize`)
- [x] All pre-existing tests still pass unchanged at default
`drainBatchSize=1` → backward-compat locked.
- [x] `pnpm run build --filter @trigger.dev/redis-worker` clean.
- [x] `pnpm run typecheck --filter webapp` clean.
- [x] `redisTest` block (real Redis via testcontainers) — couldn't run
locally on this branch due to testcontainers runtime discovery; will
validate in CI.
- [ ] Test-cloud smoke after cloud PR lands: fire `burst 50` against a
flagged env and confirm the 50th entry's drain time drops from ~2.5 s to
<200 ms.
## Notes
- Per-tick memory bound: `maxOrgsPerTick × drainBatchSize` entries can
sit in the JS pLimit queue between pop and dispatch. At defaults that's
`500 × 50 = 25 000` × ~5 KB snapshot ≈ ~125 MB worst case per worker —
well within headroom.
- The pre-batch model's strict per-env throughput cap of `1/tick` is
documented as the fairness baseline elsewhere. Org-level fairness is
what callers actually rely on; this change does not weaken that.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Follow-up to #3796, which bumped the slack-client axios paths but left
posthog-node's transitive `axios@1.15.1` in place.
`posthog-node` 4.17.1 → 5.35.6. v5 drops the axios dependency entirely
(it's now fetch-based via `@posthog/core`), so posthog's old axios path
disappears. With #3796 already on main (webapp + d3 references on
`@slack/web-api@7.16.0`), nothing else pins the old line, so the
now-dead `axios@>=1.0.0 <1.15.0` override is removed and axios resolves
to a single patched `1.16.1` repo-wide. This closes the remaining axios
advisories.
Compat: the webapp's usage in `telemetry.server.ts` (`new PostHog(key, {
host })`, `.identify`, `.groupIdentify`, `.capture`) is all object-form
API that v5 preserves; `pnpm run typecheck --filter webapp` passes.
Node: posthog-node v5 requires Node `^20.20.0 || >=22.22.0`. We run
20.20.0 in dev (`.nvmrc`), CI, and the published Docker image
(`node:20.20-bullseye-slim`), so we're compliant.
Bumps `@slack/web-api` 7.9.1 → 7.16.0 in the webapp and the two
`references` examples (d3-chat, d3-openai-agents). 7.16.0 depends on
`axios@^1.16.0`, so every slack-client axios path resolves to 1.16.1
instead of 1.15.1.
This clears the slack and references axios paths. `posthog-node`'s
transitive axios still resolves the older line - that's handled in a
follow-up that upgrades posthog-node to v5 (which drops the axios
dependency entirely and lets us retire the now-stale axios override).
The dependabot axios advisories fully close once both land.
## Summary
A `chat.agent` turn that gets aborted mid-stream (stop generation, idle
suspend, cancellation) could surface a `TASK_RUN_UNCAUGHT_EXCEPTION` —
`TypeError: Invalid state: Unable to enqueue` — in the dashboard. The
run kept working, but the exceptions were loud and confusing.
## Root cause
The realtime stream writer batches chunks through
`@s2-dev/streamstore`'s `BatchTransform`, which holds them for a short
linger window before flushing. When the turn's abort signal fires while
a record is still buffered, the stream's writable is aborted and the
transform's readable controller is closed — but the pending linger
`setTimeout` still fires and calls `controller.enqueue()` on the dead
controller, throwing from a timer callback where nothing can catch it.
Fixed upstream in `@s2-dev/streamstore@0.22.10`, which wraps the linger
flush in a try/catch that discards the closed-controller error. This
bumps the dependency across core, the CLI, and the webapp, and adds a
regression test against the real `BatchTransform`.
Verified end-to-end: a mid-stream stop that previously failed the run
with `TASK_RUN_UNCAUGHT_EXCEPTION` now leaves the run healthy.
## Summary
Dashboard surfaces handle buffered runs by falling back to the mollifier
snapshot:
- Run detail, span detail, streams view (`_app.../runs.\$runParam`,
`resources.../spans.\$spanParam`, `resources.../streams.\$streamKey`).
- Redirect routes (`@.runs.\$runParam`, `runs.\$runParam`,
`projects.v3.\$projectRef.runs.\$runParam`).
- Action routes — cancel / replay / idempotency-reset / debug — under
`resources.taskruns/...` and `resources.../idempotencyKey.reset`.
- Logs download.
- Realtime subscription route + per-run resource
(`realtime.v1.runs.\$runId`, `resources.../realtime.v1.*`).
- `CancelRunDialog` gains an `onCancelSubmitted` callback so submit
isn't raced by the Radix `DialogClose` wrapper.
Stacked on the mutations PR.
## Test plan
- [x] \`pnpm run typecheck --filter webapp\` passes
- [x] \`pnpm run test --filter webapp
test/mollifierRealtimeRunResource.test.ts\` passes
- [x] \`pnpm run test --filter webapp
test/mollifierRealtimeRunResourceBuffer.test.ts\` passes
- [x] \`pnpm run test --filter webapp
test/mollifierRealtimeSubscription.test.ts\` passes
- [x] Manual smoke: trigger a buffered run, open it in the dashboard,
replay/cancel from the UI
---
## Ship-gate follow-up fixes
- **Auto-redirect to root span on direct nav** — loader sets `?span=`
from root span (PG) or buffered snapshot spanId before 302'ing, so
bookmark/share-link/direct-nav doesn't leave the panel collapsed.
- **RunPresenter switches from `findFirstOrThrow` to `findFirst` + typed
`RunNotInPgError`** — kills the per-poll `PrismaClient error` log spam
for buffered runs without changing the route-loader's fallback flow.
- **Span detail panel renders for buffered runs** — `SpanPresenter.call`
now falls back to `findRunByIdWithMollifierFallback` +
`buildSyntheticSpanRun` instead of returning undefined and triggering
the "Event not found" toast loop.
- **Logs download for buffered runs returns a gzipped placeholder line**
— replaces the 404 with a content-encoded line explaining the run is
queued. Same org-membership gate as the PG path.
- **Admin Debug-Run button hidden for buffered runs + SpanRun circular
type alias broken** (squashed) — buttons gate on a new `isBuffered` flag
on the synthetic SpanRun. Required grounding SpanRun in
`SpanPresenter.getRun` to break a circular type alias TS no longer
tolerates once `isBuffered` is a literal field on the shape.
- **Replay action requires user auth + org-membership** (🚩 Devin
finding) — `action` was unauthenticated and the PG `findFirst` had no
org filter, so any caller with a valid `runParam` could replay any run.
Buffered fallback inherited the same gap. Fixed to mirror the cancel
route.
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
## Summary
Cancel, replay, reschedule, metadata, tags, and idempotency-key-reset
now succeed against a run that's still in the mollifier buffer.
Mutations are applied to the buffered snapshot via Lua CAS; the drainer
carries the mutation forward when it replays.
Primitives added:
- `mutateWithFallback` — PG-first / buffer-fallback resolver with
bounded-wait safety net for entries that transition mid-mutation.
- `applyMetadataMutation` — buffered metadata PUT mirroring the PG-side
retry loop with CAS atomicity.
- `resolveRunForMutation` — discriminated-union resolver used by route
`findResource` so the route builder's pre-action 404 check sees buffered
runs.
Routes wired (whole files, no GET/POST splits):
- `api.v2.runs.\$runParam.cancel.ts`
- `api.v1.runs.\$runParam.replay.ts`
- `api.v1.runs.\$runParam.reschedule.ts`
- `api.v1.runs.\$runId.metadata.ts`
- `api.v1.runs.\$runId.tags.ts`
- `resetIdempotencyKey.server.ts`
Stacked on the reads PR.
## Test plan
- [x] \`pnpm run typecheck --filter webapp\` passes
- [x] \`pnpm run test --filter webapp
test/mollifierMutateWithFallback.test.ts\` passes
- [x] \`pnpm run test --filter webapp
test/mollifierApplyMetadataMutation.test.ts\` passes
- [x] \`pnpm run test --filter webapp
test/mollifierResolveRunForMutation.test.ts\` passes
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
## Summary
Synthesise QUEUED/FAILED responses from the mollifier buffer when a
TaskRun row hasn't landed in Postgres yet. Wires the synthesis into:
- `ApiRetrieveRunPresenter`
- v1 trace GET route
- v1 spans GET route
- attempts route gains a GET loader (fixes pre-existing Remix "no
loader" 400)
The `readFallback` infra itself lives on the trigger PR (consumed by
`IdempotencyKeyConcern`); this PR adds the route-level
synthetic-rendering primitives.
Stacked on the replay PR.
## Test plan
- [x] \`pnpm run typecheck --filter webapp\` passes
- [x] \`pnpm run test --filter webapp
test/mollifierSyntheticRedirectInfo.test.ts\` passes
- [x] \`pnpm run test --filter webapp
test/mollifierSyntheticSpanRun.test.ts\` passes
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
## Summary
The replay side of the mollifier:
- `DrainerHandler`: reads buffered snapshots and replays them through
`engine.trigger` to materialise PG rows.
- `RunEngine.createCancelledRun`: new public method the handler uses to
write CANCELED rows directly from snapshots (bypass queue + waitpoint,
emit `runCancelled`). Tolerates the cjson empty-table tags edge case
found during validation.
- Drainer fairness: org → env rotation so a heavy env doesn't starve
light ones in the same org.
- Stale-entry sweep + telemetry + alertable gauge so a stuck/offline
drainer surfaces in alerts.
Both the drainer and sweep default-off; nothing fires unless flagged on
(`TRIGGER_MOLLIFIER_DRAINER_ENABLED`,
`TRIGGER_MOLLIFIER_STALE_SWEEP_ENABLED`).
Stacked on the trigger-time decisions PR.
## Test plan
- [x] \`pnpm run typecheck --filter webapp\` passes
- [x] \`pnpm run test --filter webapp
test/mollifierDrainerHandler.test.ts\` passes
- [x] \`pnpm run test --filter webapp test/mollifierStaleSweep.test.ts\`
passes
- [x] \`pnpm run test --filter @internal/run-engine
src/engine/tests/createCancelledRun.test.ts\` passes
- [x] \`pnpm run test --filter @trigger.dev/redis-worker
packages/redis-worker/src/mollifier/drainer.test.ts\` passes
---
## Ship-gate follow-up fix
**Drainer writes SYSTEM_FAILURE on max-attempts exhaustion.** Adds an
`onTerminalFailure` callback on `MollifierDrainerOptions` so the
customer's run lands a SYSTEM_FAILURE PG row even when the drainer
exhausts `MAX_ATTEMPTS` on a retryable PG error (previously
`buffer.fail()` was called with no row written → silent data loss). The
callback runs before `buffer.fail()` on every terminal path
(non-retryable AND max-attempts-exhausted), and re-throwing a retryable
error from the callback causes the drainer to requeue rather than fail.
Bumps `@trigger.dev/redis-worker` to a **minor** changeset (additive
option + new exported types). Includes 5 unit tests covering both
terminal causes plus the requeue-on-retryable-callback-failure path and
no-callback back-compat.
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Eric Allam