164 Commits

Author	SHA1	Message	Date
Eric Allam	79393deacf	feat(webapp): gate worker dequeues by worker queue via env var ... Add RUN_ENGINE_DEQUEUE_DISABLED_WORKER_QUEUES: a comma-separated list of worker queues (or base regions, which also cover the :scheduled split) for which the engine API refuses worker dequeue requests and returns no work, so those runs stay queued instead of being handed to workers that cannot run them. Unset means no gating. Blocked dequeues increment the run_engine.dequeue.blocked otel counter, tagged by worker_queue and region.	2026-06-23 10:17:29 +01:00
Eric Allam	954ee5c572	fix(webapp): deliver realtime changes with current content when the read replica lags (#3910) ... ## Summary When the realtime runs feed (the backend behind the `realtimeBackend` feature flag) hydrates a change from a Postgres read replica, the read can race the replica's apply of the very write that triggered it. The delivered row then carries the previous change's content, and an isolated final change (for example a last `metadata.set` before a run goes quiet) is not corrected until the roughly 20 second backstop poll. Measured against a replica with deliberate apply delay, every delivery trailed exactly one change behind and a final change stranded for the full backstop interval. ## Fix Publishers stamp each change record with the committed row's `updatedAt`, taken from writes they already perform, so the stamp costs no extra queries. The router delays its wake hydrate until the replica's measured lag has passed, anchored to that timestamp: a record that has already spent longer than the lag in transit is hydrated immediately, so only the racing leading edge ever waits. After hydrating, a tripwire compares each row against its record's watermark. Still-stale rows are withheld and retried briefly, and each detection feeds the lag estimate. If retries run out, the rows are delivered anyway (liveness over freshness) and follow-up re-hydrates emit the fresh version through the normal working-set diff once the replica catches up, with the backstop as the terminal net. Replica lag is sampled reader-side only, and only while feeds are active. Aurora reports live lag via `aurora_replica_status()`; vanilla Postgres can only report "caught up or not" (mid-apply lag is not honestly measurable from a replica), so tripwire observations floor the estimate there. Deployments without a replica resolve to zero lag and skip the gate entirely. Tunables live under `REALTIME_BACKEND_NATIVE_REPLICA_LAG_*`, and `realtime_native.stale_hydrates` plus `realtime_native.replica_lag_estimate_ms` make replica health observable. Two adjacent fixes: a metadata update that writes nothing no longer publishes a change record, and buffered parent and root metadata operations now publish when the flusher writes them, so those changes wake live feeds instead of waiting for the backstop. For local testing, `docker-compose` gains an opt-in `database-replica` service (compose profile `replica`) with a configurable `recovery_min_apply_delay`, which reproduces replica-lag behavior deterministically. With the gate disabled this rig reproduces the one-change-behind delivery exactly; with it enabled, deliveries arrive with current content at roughly the true replica lag, across write rates faster and slower than the lag itself.	2026-06-12 07:34:50 +01:00
Eric Allam	8dc77c0ccd	fix(webapp): only load env var values for displayed environments (#3903) ... ## Summary The environment variables page loaded every variable value in the project, unfiltered by environment. Archiving a preview branch does not delete its environment variable value rows, so projects that churn preview branches accumulate values forever, and every page view loaded all of them. On large projects this made the page loader take many seconds and stalled the server while deserializing the oversized result. ## Fix The presenter now loads the displayed environments first and filters the `values` relation to those environment IDs. That matches the display semantics exactly (per-user dev environments and active branch environments included), and the lookup is covered by the existing unique index on `(variableId, environmentId)`. Values in archived branch environments are no longer fetched at all. Covered by a new testcontainers test asserting that values from active environments (including branch environments) are returned while archived branch environments are excluded.	2026-06-11 19:30:59 +01:00
Eric Allam	187c0476c3	perf(webapp): shrink run trace loader payload and add trace span cap controls (#3906) ... ## Summary The run trace page loader serialized every span's raw OTel events (with full properties) into the response, even though the tree UI only renders the derived `timelineEvents` and the span detail panel refetches what it needs. On event-heavy traces that inflated both the loader payload and the server-side heap copies built per request. This PR keeps raw span events server-side and pairs that with a few related trace-view improvements: - A new optional `TRACE_VIEW_EMERGENCY_SPAN_CAP` env var (unset by default) clamps the trace summary and detailed trace summary span limits on both event store paths, including the public run trace endpoint, so operators can bound trace query sizes in one place without retuning the per-store limits. - The TreeView virtualizer resolved every rendered row with a linear scan over the whole tree (and `getNodeProps` did the same via `findIndex`); rows now resolve through memoized id lookup maps, which matters once traces reach tens of thousands of spans. - The run stream SSE lookup now applies the same organization membership scoping as the rest of the run page presenters, for consistency. Behavior is unchanged by default: the trace tree renders from the same `timelineEvents` it always has, and the new cap only takes effect when set.	2026-06-11 18:49:05 +01:00
DKP	93b4715967	feat(webapp): hipaa baa add-on on paid pricing tiers (#3904) ... ## Summary HIPAA BAA is offered as a paid add-on on every paid plan. Each paid tier on the in-app pricing card now has a "HIPAA BAA add-on" row with a "Request a BAA" link that opens the existing contact dialog pre-filled with a new `hipaa` inquiry type, prompting the user for their company name and a brief description of the PHI workload. The contact form's `feedbackTypes` are restructured to match the marketing /contact form: every inquiry type carries a Plain label ID and a "Contact form: ..." thread title, so threads land in Plain identically whether they come from the dashboard or the marketing site. The included-compute line on each tier also picks up the credits wording from the marketing pricing page, and the Enterprise tier lifts its title above the features row.	2026-06-11 17:32:06 +01:00
Saadi Myftija	d0b2d79b3b	fix(supervisor): cancel pending delayed snapshots when the run completes or disconnects (#3894) ... The compute suspend flow delays snapshots by `snapshotDelayMs` (~30s) so short-lived waitpoints skip the snapshot entirely, with the intent that a run continuing before the delay expires cancels the pending snapshot. But the only `cancel()` call site was the `/continue` action, which runners only invoke when restoring from an already-taken snapshot — so pending snapshots were never cancelled (zero `snapshot.canceled` events ever emitted in prod). When a run resumed and completed inside the window, the stale snapshot fired ~30s later anyway, pausing the VM 6–13s mid warm-start long-poll; the frozen guest couldn't fire its abort timer or send a FIN, causing stalls and run-engine driven retries. ### Change - Cancel the pending snapshot on `attempt.complete` — after the platform accepts the completion, before the HTTP reply (so it can't reorder with the runner's next `/suspend`). - Cancel on `runDisconnected` (crash, exit, or run replaced on the socket). - Both cancels are guarded by a runnerId match (new `TimerWheel.peek()`): a stale duplicate runner for a reassigned run must not cancel the fresh runner's pending snapshot. A missing runnerId falls through to an unconditional cancel (the pre-existing `/continue` behavior is unchanged). Waitpoint suspensions keep the runner socket connected and the attempt incomplete, so neither hook touches a snapshot that is still wanted. Known limitation (fail-safe direction): `socket.data.runnerId` is frozen at the websocket handshake, so after a same-supervisor restore the disconnect-path guard refuses the cancel. The `attempt.complete` path uses the runner's current header id and is unaffected.	2026-06-11 18:29:54 +02:00
Saadi Myftija	2397ca2999	fix(supervisor): retry transient instance create failures in compute workload manager (#3902) ... `ComputeWorkloadManager.create` swallows gateway errors currently, so a cold start that fails placement (e.g. a netns slot with a busy tap, a full node disk) silently abandons the dequeued run until the run engine's `PENDING_EXECUTING` heartbeat timeout redrives it via stall detection. ### Changes - Retry `instances.create` with short backoff (default 3 attempts, 250ms backoff), recording `createAttempts` in the wide event. - **Only statuses where the create definitely did not commit are retried**: 500 (agent/fcrun create failed) and 503 (no placement). 502/504 are excluded — the gateway emits those when it fails to reach the node or read its response, which can happen *after* the agent committed the create; the gateway only records the instance name on a clean 201, so a same-name retry would miss the collision check and could double-create the VM on another node. Network-level fetch failures are retried (if the gateway processed the create, its name index is populated and the retry 409s harmlessly). Timeouts are not retried. - **Retry attempts after a 5xx use a deterministic `-rN` name suffix**: a failed create can leave its name registered until async cleanup runs. Attempt 1 keeps the unsuffixed name.	2026-06-11 18:29:40 +02:00
Oskar Otwinowski	1c7e64acde	feat(supervisor): stamp org identity label on compute microVMs (#3899)	2026-06-11 11:49:56 +01:00
Eric Allam	b82d100483	fix(webapp): harden the realtime session routes (#3890) ... ## Summary Reliability and authorization fixes for realtime chat sessions: - Session-stream waitpoint delivery is scoped to the environment, so two environments using the same session `externalId` can no longer complete each other's waitpoints. - The session snapshot-url routes now enforce per-session authorization, and appending to a session's `out` channel requires secret-key auth, so a session-scoped token can't read another session's snapshot or forge assistant output. - Appends that carry an `X-Part-Id` header are deduplicated on retry, so a retried send can't duplicate a message. - Session creation rejects expired sessions (instead of triggering a run that can never receive input), `externalId` is immutable after creation, and the sessions list endpoint returns friendly `run_*` ids to match the single-session routes. ## Rollout The waitpoint cache key gains an environment prefix. To keep waitpoints registered by the previous deploy working across the boundary, the drain reads both the new and the previous key for this release; the legacy read can be removed a release later once no pre-deploy waitpoints remain.	2026-06-11 10:35:36 +01:00
Eric Allam	f9d57d3bd5	feat(webapp): add a new backend for the realtime runs feed (#3864) ... ## Summary Adds a second backend for the realtime runs feed (`useRealtimeRun`, `subscribeToRunsWithTag`, `subscribeToBatch`), built to stay healthy when a single busy environment has many subscribers watching many runs at once. It is gated behind a feature flag with the existing backend as the default, so nothing changes for users until it is enabled per environment. ## Design A run change is published once, as a small self-describing record, to a single per-environment channel. Every feed is then a predicate over that one stream rather than owning a channel: - A per-instance router indexes the currently-held feeds by run, tag, and batch. When a run changes it hydrates the affected rows once and serializes them once, then fans the result to every matching feed. One hot shared tag watched by many subscribers costs a single database query and serialize, not one per subscriber. - Feeds that don't match a change are never woken, wake delivery per environment is coalesced on a leading edge (250ms default) so a burst of changes costs one wake, and cold reads coalesce onto a single short-TTL-cached resolve. - An admission gate bounds how many cold ClickHouse resolves run concurrently, so a mass reconnect across many distinct filters queues instead of stampeding the database. - Changes that land while a client is between long-polls are delivered on its next poll instead of waiting for the periodic backstop: each environment buffers its recent change records, subscriptions linger briefly after the last feed closes, and a newly-armed poll replays exactly the connection's gap. - The per-connection replay cursors behind that are shared across instances via Redis (a single timestamp each), so a poll landing on a different instance behind the load balancer still reads the connection's true gap instead of falling back to a cold resolve. Cursor reads have a bounded deadline and degrade to the cold-read path on any Redis trouble. - Tag subscriptions with multiple tags match runs carrying all of the tags, mirroring the existing backend's filter semantics, and live long-polls hold for about 20 seconds to match its cadence. - The per-environment channel supports Redis Cluster sharded pub/sub, so the wake path scales horizontally across shards by environment. - The backend reports its health through OpenTelemetry metrics (delivery lag, poll resolution paths, backstop outcomes, replay and cursor-store activity), with a provisioned Grafana dashboard for local development. Everything is behind the feature flag and tunable via env vars; the existing backend remains the default.	2026-06-11 07:56:10 +01:00
Daniel Sutton	6afc9bfa4c	fix(run-engine): retry getSnapshotsSince on the replica then primary when the read replica lags (#3889) ... ## Summary When `RUN_ENGINE_READ_REPLICA_SNAPSHOTS_SINCE_ENABLED` is on, `RunEngine.getSnapshotsSince` reads from the read replica. During write spikes the replica can briefly lag, so the snapshot id a runner just learned from the writer isn't visible there yet: the lookup threw, the worker route returned a 500, and the runner waited for its next poll — turning sub-second snapshot notifications into poll-interval latency exactly when things are busiest. This PR makes the flag safe to enable: a replica miss of the since snapshot gets one jittered retry on the replica (most lag windows are shorter than the ~50–200ms wait, so the writer is never touched), then falls back to the primary, observed via a new `run_engine.snapshots_since.replica_miss` counter with an `outcome` attribute (`replica_retry` vs `primary`). Only genuine misses — absent on the primary too — remain errors. ## Design - `getExecutionSnapshotsSince` now throws a typed `ExecutionSnapshotNotFoundError` so the engine can distinguish the expected lag miss from real failures. The message string is unchanged and the error never leaves the engine. - The recovery path only engages when the flag is on, a distinct replica client is configured, and no transaction client was passed. With the flag off, the path is behaviorally identical to before. - Retry delay bounds are configurable (`RUN_ENGINE_SNAPSHOTS_SINCE_REPLICA_RETRY_MIN_MS`/`MAX_MS`, default 50/200; `MAX_MS=0` skips the replica retry and goes straight to the primary). - The warn log fires only when the primary serves the read (the writer spill is the operationally interesting event); replica-retry recoveries are counted but quiet. A permanently-missing snapshot id stays an error-level failure with a `failedDuring` field, so lag metrics aren't polluted by bogus ids. - Stale-tail lag (replica has the since snapshot but not newer rows) deliberately still returns the replica's view; the next poll catches up. - The since-snapshot anchor lookup is now scoped to the polled run (`where: { id, runId }`), so a snapshot id from a different run raises not-found instead of silently anchoring a too-wide window of the run's snapshots. ## Test plan All vitest + testcontainers, no mocks. A new `schemaOnlyPrisma` fixture (migrated-but-empty clone database) simulates a replica that hasn't caught up, and a real in-memory OTel meter pins the counter semantics per outcome. - [x] Replica catches up during the jittered retry window → served by the replica, `outcome=replica_retry` = 1, primary never consulted - [x] Replica permanently missing the since snapshot → served by the primary, `outcome=primary` = 1 - [x] Snapshot missing on both replica and primary → null, counter = 0 - [x] Replica has the since snapshot but lags by one → the replica's view is served, no fallback (verified discriminating power: the test fails if reads secretly hit the primary) - [x] Flag off with a replica configured → primary serves the read - [x] Transaction client provided → bypasses the replica entirely - [x] Since snapshot belonging to a different run → null - [x] Existing getSnapshotsSince + waitpoints suites green; run-engine, testcontainers, and webapp typechecks pass 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com>	2026-06-10 16:54:03 +00:00
Eric Allam	b28c6d0b90	fix(webapp): sanitize streamed agent URLs before rendering in the agent view (#3882) ... ## Summary The dashboard's Agent view rendered `source-url` and `file` message parts by putting their `url` straight into an `href`/`src`. Those URLs come from streamed agent and tool data, so a tool that emitted something like `javascript:alert(1)` produced a clickable XSS payload in the dashboard. ## Fix A `toSafeUrl` helper now gates every URL before it reaches an `href`/`src`: it allows only `http:`/`https:`/`blob:` (and `data:image/...` for inline images) and returns `null` for anything else. Unsafe values render as plain text instead of a link or image, so a hostile or malformed URL degrades gracefully rather than becoming clickable. Safe URLs render exactly as before. Covered by a unit test over the allow/deny list.	2026-06-10 09:52:24 +01:00
Daniel Sutton	bc01f6ea3a	fix(webapp): stop writer DB connectivity errors leaking to trigger() API clients (#3874) ... ## Summary During `trigger()` worker-queue resolution, `getWorkerQueue` wrapped any error from `getDefaultWorkerGroupForProject` into a client-facing `ServiceValidationError` (HTTP 422) carrying `error.message`. That method runs `project.findFirst` on the **writer**; when the writer is unreachable Prisma throws a connection error (P1001) whose message includes the database host, and that raw message was returned to the API client and surfaced in the run view via the SDK's `TriggerApiError`. It also mis-classifies a transient outage: a 422 is not retried by the SDK, so triggers failed permanently instead of riding out a brief writer blip. ## Design This is the only place on the trigger path that folds a *caught* error's message into a client-facing error — every other DB failure on the path propagates to the route's generic 500 handler (scrubbed, and retried by the SDK). So the fix is local: - Add `isInfrastructureError()` — true for Prisma connection-level failures (the DB-unreachable family: P1001/P1002/P1008/P1017, plus the init/panic/unknown client error classes), false for query/validation errors (e.g. P2002). - At the wrap site, rethrow infrastructure errors so they reach the generic 500 handler (no raw message, and retryable). Genuine domain failures (e.g. "Project not found.") still become a 422. Only P1001 ("can't reach database server") has been observed in practice; the rest of the connection family is included as same-class forward-proofing. ## Test plan - [x] Unit: `isInfrastructureError` classifies a P1001 (incl. the Prisma 6.x `PrismaClientKnownRequestError` shape) and init errors as infrastructure; P2002 and a plain `Error` as not - [x] `getWorkerQueue` rethrows a P1001 unchanged instead of wrapping it in a `ServiceValidationError`; still wraps a domain failure as a `ServiceValidationError` — RED on current code, GREEN after - [ ] (optional) toxiproxy e2e: trigger with the writer cut → HTTP 500 generic body, no DB host in the response --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-10 08:53:57 +01:00
Eric Allam	3bc88c453e	perf(webapp): memoize react-router per-request route matching via pnpm patch (#3877) ... ## Summary Under high request load the webapp spends most of its CPU inside react-router's `matchRoutes`, not in application code. `@remix-run/router@1.23.2` (the React Router v6 / Remix 2 core) re-flattens, re-ranks, and recompiles the entire route table on every request, and with the webapp's ~436 routes that cost dominates once request rates climb. There is no `NODE_ENV` gate, so production pays it too. This adds a pnpm patch that memoizes the parts that depend only on the static route manifest: it caches the flattened/ranked branches per route tree, hoists the loop-invariant `decodePath` out of the match loop, and caches compiled path regexes. ## Benchmark CPU profile over the same load (100 concurrent tag feeds, ~425 req/s), `NODE_ENV=production`, before vs after the patch: | Metric | Before | After | | --- | --- | --- | | Active CPU (self-time over the window) | 28.3s | 18.5s (-34%) | | Route-matching self-time | 19.2s | 7.5s (-61%) | | Event-loop lag p99 | 322ms | 113ms (-65%) | | Idle headroom | 26% | 52% | Application/realtime code was ~0% of CPU in both profiles; the bottleneck was entirely generic per-request route matching. ## Why a patch instead of an upgrade The inefficiency is acknowledged upstream ([remix-run/react-router#8653](https://github.com/remix-run/react-router/issues/8653)). A contributor PR doing exactly this ([remix-run/react-router#14866](https://github.com/remix-run/react-router/pull/14866)) was closed in favor of a narrower fix ([remix-run/react-router#14967](https://github.com/remix-run/react-router/pull/14967), branch caching only, shipped in React Router v7), with the maintainer suggesting patch-package as the interim until the Remix 3 route-pattern rewrite (see [remix-run/remix#4786](https://github.com/remix-run/remix/discussions/4786)). We are on the v6-era core and cannot pick up even the partial fix without a framework migration, so this patch is the sanctioned stopgap, and it also includes the compiled-regex cache the merged PR left out. [`patches/README.md`](https://github.com/triggerdotdev/trigger.dev/blob/perf/react-router-route-matching/patches/README.md) documents the full rationale, the safety argument (deterministic, internal-only, bounded caches), and when to remove the patch.	2026-06-09 22:37:26 +01:00
Matt Aitken	1b0f2c71dd	fix(webapp): correct backward pagination slice in listRunIds (#3867) ... ## Problem Two backward-pagination bugs in `ClickHouseRunsRepository.listRunIds`, both pre-existing (they predate the composite-cursor work in #3852 and were spotted during/after it): **1. Wrong slice (straddled pages).** `listRunRows` fetches `page.size + 1` rows to detect `hasMore`. That extra row is the one *farthest from the cursor* in both directions (forward orders DESC; backward orders ASC), so it's always the *trailing* element. Forward correctly used `rows.slice(0, size)`, but backward+`hasMore` used `rows.slice(1, size + 1)` — dropping the row *closest* to the cursor and keeping the has-more sentinel. The page straddled two logical pages (one run from the correct previous page + one from the page before it), so paging "newer" across a boundary **repeated and skipped** runs. **2. Stranded forward cursor on a partial backward page.** In the backward `!hasMore` branch, `nextCursor` was `reversedRows.at(page.size - 1)`. On a partial page (fewer than `page.size` rows — reachable via `runs.list` by passing a forward page's cursor as `page[before]`), that index overshoots → `undefined` → `nextCursor` becomes `null`, leaving no way to page forward again. ## Fix - **Slice:** both directions now slice `rows.slice(0, size)` (the sentinel is the trailing element either way). - **Partial-page cursor:** the backward `!hasMore` branch takes the oldest row on the page, `rows.at(0)`, for `nextCursor` — equivalent to the old expression for full pages, correct for partial ones. Forward pagination, the cursor *values* for full pages, and the `hasMore === true` paths were already correct and are unchanged. ## Tests `runsRepositoryCursor.test.ts` gains two cases (both fail on `main`, pass here): - **multi-page backward walk:** forward across all pages, then backward from the last page — each backward page must *exactly* reproduce the corresponding forward page (no straddling: `main` returns `{b,c}` instead of `{c,d}`), and the full traversal covers every run once. - **partial backward page:** backward onto a partial first page must still expose a working forward cursor (and paging forward from it reaches the rest) — `main` returns a `null` nextCursor. The three existing cursor tests (forward completeness, backward round-trip, legacy cursor) still pass. --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-06-09 12:11:11 +01:00
Matt Aitken	6bcd369ea1	feat(webapp,rbac): REQUIRE_PLUGINS=1 fail-fast for required plugin loads [TRI-9852] (#3734) ... ## Summary - `internal-packages/rbac/src/index.ts` — in `LazyController.load()`'s catch block, throw an Error when `process.env.REQUIRE_PLUGINS === "1"` instead of silently falling back. The throw is captured into the lazy controller's init promise, so it surfaces on the first method call. - `apps/webapp/app/routes/healthcheck.tsx` — `await rbac.isUsingPlugin()` after the DB ping. With `REQUIRE_PLUGINS=1` and a failed plugin load, the throw surfaces here and the healthcheck returns 500 → readiness probe fails → rollout is rolled back. Noop for self-hosters. - `.server-changes/require-plugins-fail-fast.md` — server-changes entry. - `internal-packages/rbac/src/require-plugins.test.ts` — 4 unit tests covering loader branching: unset → fallback, `=1` → throw, `forceFallback: true` wins, only exactly `"1"` enforces. - `internal-packages/testcontainers/src/webapp.ts` — adds `requirePlugins?: boolean` to `StartWebappOptions`. Implies `forceRbacFallback: false`. - `apps/webapp/test/healthcheck-require-plugins.e2e.test.ts` — e2e closes the loop: spawns a real webapp, hits `/healthcheck` via HTTP, asserts 500 with `REQUIRE_PLUGINS=1` and 200 without. ## Motivation Today the RBAC plugin loader catches any plugin-load failure (missing module, broken transitive dep, init throw) and silently returns the default fallback implementation. This is the correct behaviour for self-hosters who don't ship the plugin — but it's dangerous in deployments where the plugin is expected to load: an accidentally-missing or broken plugin would silently disable enforcement. `REQUIRE_PLUGINS=1` makes the loader fail loudly in those deployments. The variable name is intentionally plural and generic — future plugin contracts (audit logs, SSO) can read the same flag without renaming. Closes [TRI-9852](https://linear.app/triggerdotdev/issue/TRI-9852/require-plugins1-fail-fast-for-required-plugin-loads). ## Test plan - [x] `pnpm run test --filter @trigger.dev/rbac` — 38/38 tests pass, including the 4 new loader tests - [x] `pnpm run typecheck --filter webapp` — passes - [x] `pnpm run typecheck --filter @trigger.dev/rbac --filter @internal/testcontainers` — passes - [x] e2e test added (`healthcheck-require-plugins.e2e.test.ts`) — CI runs it via `e2e-webapp.yml`. Couldn't run locally (no Docker daemon up); CI has Docker provisioned. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>	2026-06-09 12:08:20 +01:00
Eric Allam	0ee1461d86	fix(webapp): show scheduled runs under their correct region (#3873) ... ## Summary Runs routed onto a dedicated scheduled worker queue were showing under a phantom region in the dashboard, run details, and the API, and slipped through region filters. They now resolve to their real region everywhere. ## Fix A worker queue can carry a `:scheduled` suffix that routes scheduled-lineage runs onto their own list. That suffix is an internal routing detail, but it was leaking anywhere the worker queue is read as a region. A `baseWorkerQueue` helper strips any `:<class>` suffix back to the base region (region names never contain a colon, so it's everything before the first colon) and is applied at every region read site: the runs list, run detail, the public API, and replay's region override. The runs-replication writer also stores the base region in ClickHouse so the region filter matches.	2026-06-09 11:46:44 +01:00
Daniel Sutton	d7028e278e	feat(webapp): label mollifier decisions by enrolled org (#3869) ... ## Summary The `mollifier.decisions` metric only carried an `outcome` label, so for an org that has the mollifier enabled there was no way to see how often its triggers pass through the gate instead of being diverted — making it hard to tell why the trip isn't firing for an opted-in org. This adds two bounded labels: `enrolled` (`"true"`/`"false"`, the per-org flag) and `org` (the org id, attached **only** when `enrolled` is true). For an enrolled org you can now compare directly: `mollifier.decisions{outcome="pass_through", enrolled="true", org="<id>"}` vs `{outcome="mollify", ...}`. ## Design `recordDecision` now takes an options object (`{ reason?, enrolled, orgId? }`). The `org` label is restricted to the enrolled cohort to keep cardinality bounded — the guard lives in a pure `decisionLabels` helper, so a non-enrolled org id can never be attached even if one is passed. The enrolled set is small and capped operationally. The per-org flag is resolved once at the top of `evaluateGate` (in-memory, no DB round-trip on the trigger hot path) so every decision — including the debounce / one-time-use-token / triggerAndWait bypasses — is labelled consistently. ## Test plan - [x] `mollifierGate.test.ts` cascade asserts `enrolled`/`org` on every gate branch - [x] `mollifierDecisionLabels.test.ts` (new) proves `org` is dropped for non-enrolled even when an id is passed (cardinality guard) - [x] `vitest run mollifierGate mollifierDecisionLabels` — 34/34 pass - [x] `pnpm run typecheck --filter webapp` clean Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-08 17:02:30 +00:00
Oskar Otwinowski	93532cdb99	feat(supervisor): forward per-run labels to the compute provider (#3821) ... Add an optional network_labels field to the internal compute client's create and restore request schemas and forward per-VM endpoint labels on both paths, so a restored VM keeps the same labels as a freshly-booted one. Mirrors the label the Kubernetes workload manager already sets on the run pod. --------- Co-authored-by: nicktrn <55853254+nicktrn@users.noreply.github.com>	2026-06-08 18:45:01 +02:00
Matt Aitken	ef04cc39ef	fix(webapp): use composite keyset cursor for run pagination (#3852) ... ## Problem `ClickHouseRunsRepository.listRunIds` / `listRuns` order results by the composite key `(created_at, run_id)`, but the cursor predicate cut on `run_id` **alone**: ```ts .where("run_id < {runId: String}", { runId: cursor }) .orderBy("created_at DESC, run_id DESC") ``` This is only sound when `run_id` lexicographic order matches `created_at` order. `run_id`s are cuids — only coarsely time-sortable — so when a burst of runs is created within a sub-second window, the two orders can diverge. When they do, the next-page predicate (`run_id < cursor`, where `cursor` is the *last* page element = the smallest `created_at`, not necessarily the smallest `run_id`): - **re-includes** rows already returned on a previous page (duplicates), and - **skips** rows it should have returned (silent data loss). For bulk **replay** this caused runs to be replayed more than once (replay has no idempotency guard). For the dashboard and the `runs.list` API it could silently repeat or skip runs at page boundaries. ## Fix Make the cursor predicate match the composite ordering: - Cursors now encode the full `(created_at, run_id)` key as an **opaque URL-safe base64 token** (`base64url({"c":<createdAtMs>,"r":"<runId>"})`), and the query cuts on the matching tuple — `(created_at, run_id) < (…)` forward / `> (…)` backward. - The `ORDER BY` is unchanged, so the query stays aligned with the table's primary key — no performance regression (the tuple range predicate is actually more index-friendly than `run_id <` alone). - Cursors are **server-issued opaque tokens** (the SDK only echoes `pagination.next` / `pagination.previous` back), so this needs **no client/SDK update**. Legacy cursors were the bare internal `run_id`; they're detected by decode failure (a cuid isn't a valid base64-wrapped JSON payload) and fall back to the old `run_id`-only predicate, so in-flight cursors keep working and drain naturally. New cursors also no longer expose a bare internal run id. - `listRunIds` is now the single cursor-aware list primitive: it returns `{ runIds, pagination: { nextCursor, previousCursor } }`, and `listRuns` builds on it (one place constructs cursors). Bulk actions consume the same method and advance by `pagination.nextCursor`, finishing when it's `null`. - `getTaskRunsQueryBuilder` now also selects `toUnixTimestamp64Milli(created_at) AS created_at_ms`, using a dedicated `TaskRunListQueryResult` schema. The shared `TaskRunV2QueryResult` stays `run_id`-only so the run-engine pending-version lookup (`getPendingVersionIdsQueryBuilder`, which selects only `run_id`) doesn't fail validation on a column it doesn't query. ## Tests New `runsRepositoryCursor.test.ts` (testcontainer-backed, real Postgres→ClickHouse replication): - **forward** pagination returns every run exactly once when `run_id` order is the reverse of `created_at` order (reproduces the duplicate/skip bug — fails on `main`; this walk-until-`nextCursor`-null-and-assert-complete is exactly the bulk action's iteration), - **backward** pagination round-trips to the previous page across a boundary, - **legacy** bare-`run_id` cursor still uses the old predicate (backwards compatibility). The existing `runsRepository` suites (part1–4) still pass; `part4`'s `count new runs with listRunIds` test was updated for the new `{ runIds, pagination }` return shape, and the `clickhouse` `taskRuns` query-builder snapshots were regenerated for the added `created_at_ms` column. ## Notes - Separate, pre-existing issue (out of scope, not introduced here): `listRuns`' backward display-slicing (`rows.slice(1, size+1)` when `hasMore`) has an off-by-one that can return a straddled page. Tracked separately. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-06-08 12:57:47 +01:00
Eric Allam	97036fb741	feat(webapp,clickhouse): export run traces as log, markdown, or jsonl (#3851) ... ## Summary Adds a trace export to the run page. From the new **Export trace** menu you can copy a run's full trace to the clipboard as Markdown (for pasting into an AI assistant) or download it as a flat Log, a Markdown table, or JSON Lines. Internal engine-debug events are filtered out by default, and errors are surfaced inline with their message. ## Design The export streams events from the store to the gzipped response one at a time and never materialises the span tree, so a trace of any size exports with bounded memory and without stalling the server. Output is flat and chronological: each line carries its own `spanId ← parentSpanId`, so the hierarchy is reconstructable without nesting. Formats share a single streaming pipeline and are pluggable via `?format=log|jsonl|markdown`, so adding a format is an isolated change. ## Screenshots **Export menu** <img width="370" height="252" alt="trace-export-menu" src="https://github.com/user-attachments/assets/3d10304a-8c49-4606-b15d-2859b137419f" /> **In context** <img width="2400" height="1802" alt="trace-export-run-page" src="https://github.com/user-attachments/assets/46c80b30-303b-47c6-9ace-a2fb06f6cb61" />	2026-06-06 21:24:37 +01:00
Katia Bulatova	4711adef84	fix(webapp): don't reload runs list when toggling bulk action inspector (#3841) ... ## Summary Opening or closing the Bulk action inspector should not affect the Runs list, but it was still triggering refresh-related UI behavior. This PR fixes that and smooths out a few related inspector interactions. ## Changelog Stop reloading the runs list (and flashing its loading state) when opening or closing the Bulk action inspector. Filtering, pagination, and explicit refresh are unaffected.	2026-06-05 15:52:35 +02:00
github-actions[bot]	a730faadfe	chore: release v4.5.0-rc.5 (#3808) ... ## Summary 1 new feature, 8 improvements, 1 bug fix. ## Highlights - Add optional `shouldPauseScaling` to the supervisor consumer pool scaling options to freeze scale-up while it returns true (scale-down stays allowed). ([#3836](https://github.com/triggerdotdev/trigger.dev/pull/3836)) ## Improvements - The MCP server no longer tells the AI agent to wait for a run to complete after every `trigger_task` call. Waiting is now opt-in: the agent only waits when you ask it to (for example "trigger and then wait for it to finish"). This avoids burning tokens polling runs you didn't need to block on and keeps responses clearer. ([#3838](https://github.com/triggerdotdev/trigger.dev/pull/3838)) - Update the bundled OpenTelemetry packages to their latest releases (`@opentelemetry/sdk-node` 0.218.0, `@opentelemetry/core` 2.7.1, `@opentelemetry/host-metrics` 0.38.3). ([#3810](https://github.com/triggerdotdev/trigger.dev/pull/3810)) - `envvars.upload` now accepts an optional `isSecret` flag, letting you create the imported variables as secret (redacted) environment variables. When omitted, variables default to non-secret. ([#3809](https://github.com/triggerdotdev/trigger.dev/pull/3809)) - Offload large trigger payloads to object storage before sending the trigger API request. The SDK uploads packets at or above the existing 128KB limit and sends an `application/store` pointer instead of embedding large JSON in the request body. `TriggerTaskRequestBody` now validates that `application/store` payloads are non-empty storage paths. ([#3785](https://github.com/triggerdotdev/trigger.dev/pull/3785)) - Make mollifier buffer and drainer internals configurable. `MollifierBuffer` now accepts `ackGraceTtlSeconds`, `maxRetriesPerRequest`, `reconnectStepMs`, and `reconnectMaxMs` options, and `MollifierDrainer` accepts `maxBackoffMs` and `backoffFloorMs`. All default to their previous hardcoded values, so existing behaviour is unchanged. ([#3822](https://github.com/triggerdotdev/trigger.dev/pull/3822)) - `MollifierDrainer` accepts a `drainBatchSize` option (default 1) that controls how many entries are popped per env per tick — in-flight handlers remain capped by the global `concurrency`. `MollifierBuffer` also gains `getDrainingCount()` / `listStaleDraining()`, backed by a new `mollifier:draining` ZSET maintained atomically with pop/ack/fail/requeue (observability-only). ([#3797](https://github.com/triggerdotdev/trigger.dev/pull/3797)) - Adds AI SDK 7 support. The `ai` peer range now includes v7, and the `chat.agent` / chat surfaces work against v7's ESM-only build. On v7, install `@ai-sdk/otel` alongside `ai` and the SDK registers it for you so `experimental_telemetry` spans keep flowing into your run traces (v7 stopped emitting them from `ai` core). v5 and v6 keep working unchanged. ([#3833](https://github.com/triggerdotdev/trigger.dev/pull/3833)) - `useTriggerChatTransport` now recovers when restored session state points at a session that no longer exists in the current environment ([#3816](https://github.com/triggerdotdev/trigger.dev/pull/3816)) ## Bug fixes - Fix `@trigger.dev/core` build: cast the underlying log record exporter when calling `forceFlush` so it typechecks against the updated OpenTelemetry `LogRecordExporter` type (which no longer declares `forceFlush`). ([#3829](https://github.com/triggerdotdev/trigger.dev/pull/3829)) <details> <summary>Raw changeset output</summary> ⚠️⚠️⚠️⚠️⚠️⚠️ `main` is currently in **pre mode** so this branch has prereleases rather than normal releases. If you want to exit prereleases, run `changeset pre exit` on `main`. ⚠️⚠️⚠️⚠️⚠️⚠️ # Releases ## @trigger.dev/build@4.5.0-rc.5 ### Patch Changes - Updated dependencies: - `@trigger.dev/core@4.5.0-rc.5` ## trigger.dev@4.5.0-rc.5 ### Patch Changes - The MCP server no longer tells the AI agent to wait for a run to complete after every `trigger_task` call. Waiting is now opt-in: the agent only waits when you ask it to (for example "trigger and then wait for it to finish"). This avoids burning tokens polling runs you didn't need to block on and keeps responses clearer. ([#3838](https://github.com/triggerdotdev/trigger.dev/pull/3838)) - Update the bundled OpenTelemetry packages to their latest releases (`@opentelemetry/sdk-node` 0.218.0, `@opentelemetry/core` 2.7.1, `@opentelemetry/host-metrics` 0.38.3). ([#3810](https://github.com/triggerdotdev/trigger.dev/pull/3810)) - Updated dependencies: - `@trigger.dev/core@4.5.0-rc.5` - `@trigger.dev/build@4.5.0-rc.5` - `@trigger.dev/schema-to-json@4.5.0-rc.5` ## @trigger.dev/core@4.5.0-rc.5 ### Patch Changes - Add optional `shouldPauseScaling` to the supervisor consumer pool scaling options to freeze scale-up while it returns true (scale-down stays allowed). ([#3836](https://github.com/triggerdotdev/trigger.dev/pull/3836)) - Fix `@trigger.dev/core` build: cast the underlying log record exporter when calling `forceFlush` so it typechecks against the updated OpenTelemetry `LogRecordExporter` type (which no longer declares `forceFlush`). ([#3829](https://github.com/triggerdotdev/trigger.dev/pull/3829)) - `envvars.upload` now accepts an optional `isSecret` flag, letting you create the imported variables as secret (redacted) environment variables. When omitted, variables default to non-secret. ([#3809](https://github.com/triggerdotdev/trigger.dev/pull/3809)) ```ts await envvars.upload("proj_1234", "prod", { variables: { STRIPE_SECRET_KEY: "sk_live_..." }, isSecret: true, }); ``` - Offload large trigger payloads to object storage before sending the trigger API request. The SDK uploads packets at or above the existing 128KB limit and sends an `application/store` pointer instead of embedding large JSON in the request body. `TriggerTaskRequestBody` now validates that `application/store` payloads are non-empty storage paths. ([#3785](https://github.com/triggerdotdev/trigger.dev/pull/3785)) Payload uploads use the same resolved `ApiClient` as the trigger call (including `requestOptions.clientConfig`), not only the global `apiClientManager.client` — so custom `baseURL`, access token, and preview branch apply to both presign and trigger. - Update the bundled OpenTelemetry packages to their latest releases (`@opentelemetry/sdk-node` 0.218.0, `@opentelemetry/core` 2.7.1, `@opentelemetry/host-metrics` 0.38.3). ([#3810](https://github.com/triggerdotdev/trigger.dev/pull/3810)) ## @trigger.dev/plugins@4.5.0-rc.5 ### Patch Changes - Updated dependencies: - `@trigger.dev/core@4.5.0-rc.5` ## @trigger.dev/python@4.5.0-rc.5 ### Patch Changes - Updated dependencies: - `@trigger.dev/sdk@4.5.0-rc.5` - `@trigger.dev/core@4.5.0-rc.5` - `@trigger.dev/build@4.5.0-rc.5` ## @trigger.dev/react-hooks@4.5.0-rc.5 ### Patch Changes - Updated dependencies: - `@trigger.dev/core@4.5.0-rc.5` ## @trigger.dev/redis-worker@4.5.0-rc.5 ### Patch Changes - Make mollifier buffer and drainer internals configurable. `MollifierBuffer` now accepts `ackGraceTtlSeconds`, `maxRetriesPerRequest`, `reconnectStepMs`, and `reconnectMaxMs` options, and `MollifierDrainer` accepts `maxBackoffMs` and `backoffFloorMs`. All default to their previous hardcoded values, so existing behaviour is unchanged. ([#3822](https://github.com/triggerdotdev/trigger.dev/pull/3822)) - `MollifierDrainer` accepts a `drainBatchSize` option (default 1) that controls how many entries are popped per env per tick — in-flight handlers remain capped by the global `concurrency`. `MollifierBuffer` also gains `getDrainingCount()` / `listStaleDraining()`, backed by a new `mollifier:draining` ZSET maintained atomically with pop/ack/fail/requeue (observability-only). ([#3797](https://github.com/triggerdotdev/trigger.dev/pull/3797)) - Updated dependencies: - `@trigger.dev/core@4.5.0-rc.5` ## @trigger.dev/rsc@4.5.0-rc.5 ### Patch Changes - Updated dependencies: - `@trigger.dev/core@4.5.0-rc.5` ## @trigger.dev/schema-to-json@4.5.0-rc.5 ### Patch Changes - Updated dependencies: - `@trigger.dev/core@4.5.0-rc.5` ## @trigger.dev/sdk@4.5.0-rc.5 ### Patch Changes - Adds AI SDK 7 support. The `ai` peer range now includes v7, and the `chat.agent` / chat surfaces work against v7's ESM-only build. On v7, install `@ai-sdk/otel` alongside `ai` and the SDK registers it for you so `experimental_telemetry` spans keep flowing into your run traces (v7 stopped emitting them from `ai` core). v5 and v6 keep working unchanged. ([#3833](https://github.com/triggerdotdev/trigger.dev/pull/3833)) - `useTriggerChatTransport` now recovers when restored session state points at a session that no longer exists in the current environment ([#3816](https://github.com/triggerdotdev/trigger.dev/pull/3816)) - Offload large trigger payloads to object storage before sending the trigger API request. The SDK uploads packets at or above the existing 128KB limit and sends an `application/store` pointer instead of embedding large JSON in the request body. `TriggerTaskRequestBody` now validates that `application/store` payloads are non-empty storage paths. ([#3785](https://github.com/triggerdotdev/trigger.dev/pull/3785)) Payload uploads use the same resolved `ApiClient` as the trigger call (including `requestOptions.clientConfig`), not only the global `apiClientManager.client` — so custom `baseURL`, access token, and preview branch apply to both presign and trigger. - Update the bundled OpenTelemetry packages to their latest releases (`@opentelemetry/sdk-node` 0.218.0, `@opentelemetry/core` 2.7.1, `@opentelemetry/host-metrics` 0.38.3). ([#3810](https://github.com/triggerdotdev/trigger.dev/pull/3810)) - Updated dependencies: - `@trigger.dev/core@4.5.0-rc.5` </details> Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>	2026-06-05 14:09:26 +01:00
Eric Allam	aa9f1112ea	fix(database): include the Prisma CLI in production builds (#3843) ... ## Summary The Prisma CLI was missing from production builds of the webapp image, so anything that shells out to `prisma` at startup failed. The container entrypoint and the standalone migration step both run `prisma migrate deploy` / `prisma migrate status`, and those broke with `Command "prisma" not found`. ## Fix `prisma` was a `devDependency` of `@trigger.dev/database`. It had only been landing in the pruned `--prod` install as a side effect of pnpm auto-installing it as a peer of `@prisma/client`. A recent dependency change shifted peer resolution so prisma stopped being materialized into the production tree, and the CLI disappeared from the image. Moving `prisma` into `dependencies` of `@trigger.dev/database` makes the CLI an explicit part of production installs. It lands in the webapp image only: the separately deployed supervisor, coordinator, and provider images don't reach the database package in their production trees (`core` only `devDepends` on it, so it isn't transitive), so they're unaffected. Verified against a locally built production image: `pnpm --filter @trigger.dev/database exec prisma --version` now resolves the CLI and the schema engine instead of failing.	2026-06-05 13:08:49 +00:00
nicktrn	35c56f1d09	feat(supervisor): add opt-in dequeue backpressure (#3836) ... The supervisor can now pause dequeuing - and freeze consumer-pool scale-up - when a backpressure signal says the cluster can't place more work, then ramp dequeuing back up gradually once it clears. The signal is a verdict published to a Redis key by a cluster-side component; the supervisor reads it on a short refresh and gates `preDequeue` on it. Off by default (`TRIGGER_DEQUEUE_BACKPRESSURE_ENABLED`). Everything fails open: a missing, stale, or unreadable verdict never pins the brake, and the hot-path read is a synchronous cached lookup with no I/O. The scale-up freeze leaves scale-down untouched, and on release the resume is ramped so a deep queue isn't hammered all at once. Dry-run is on by default (`TRIGGER_DEQUEUE_BACKPRESSURE_DRY_RUN`): even once enabled it only logs what it would have done, and surfaces the computed state through metrics, until explicitly set to act. Prometheus: `supervisor_backpressure_engaged`, `_dry_run`, `_skipped_dequeues_total`. Refs TRI-5354	2026-06-05 13:58:19 +01:00
Eric Allam	85886b96da	feat(webapp,supervisor): isolate scheduled runs on a dedicated worker queue (#3839) ... ## Summary Scheduled runs and their descendants can now be routed to a dedicated per-region worker queue, processed by a separate worker fleet, so a burst of scheduled crons no longer competes with standard and agent runs for the same queue and inflates their startup latency. It is off by default and enabled per organization via a feature flag (with a global default), so nothing changes until it is turned on. ## Design At trigger time, any run whose lineage originates from a schedule (`rootTriggerSource === "schedule"`, which already propagates from a scheduled run down to all of its children) gets its worker queue suffixed with `:scheduled`. The worker queue name is an opaque string persisted on the run and used verbatim by enqueue and dequeue, so this needs no Lua, message-envelope, or concurrency changes. Concurrency stays keyed by environment and queue, not by worker queue. On the consumer side, the dequeue endpoint gains an optional `queueClass` selector. A supervisor sends `queueClass: "scheduled"` and the server derives the actual queue from the worker's own group, so a token can only ever reach its own region's queues. A fleet picks its class with the `TRIGGER_WORKER_QUEUE_CLASS` env var (`default` or `scheduled`), so a dedicated scheduled fleet can run alongside the standard one. Verified end to end against a local managed-worker setup: scheduled runs route to the dedicated queue, are drained only by the scheduled fleet, and standard runs are left untouched.	2026-06-05 09:41:57 +01:00
Katia Bulatova	db4074df54	fix(webapp): validate packet storage paths (#3830) ... ## Summary This PR adds packet path validation before key construction and presigning. Invalid paths are rejected before reaching either object-store client implementation, ensuring consistent behavior regardless of the underlying storage configuration.	2026-06-05 10:28:08 +02:00
Eric Allam	8d5cf313bb	fix(webapp): fix AI agent dashboard rendering and snapshot loads (#3834) ... ## Summary Three fixes to the AI agent surface in the dashboard, all surfaced while verifying AI SDK 7 support (the SDK side is #3833): - **AI SDK 7 telemetry rendering.** The generation-span inspector and run metrics now read both the v6 (`ai.*`) and v7 (`gen_ai.*`) telemetry attribute shapes. On v7 the Messages, Provider, and Tools views showed empty/unknown before; now they render correctly. - **HITL approvals in the conversation view.** The agent conversation view renders human-in-the-loop tool approvals and denials (awaiting approval, approved, denied with reason) instead of leaving the tool part blank. This gap predated v7. - **Chat snapshot store mismatch.** Chat session snapshots now resolve through a single storage-key helper shared by the SDK write and the dashboard read. Previously the write applied the default object-store protocol to the key while the read fell back to a different store, so the dashboard 404'd on the snapshot and showed only a partial conversation.	2026-06-05 08:51:29 +01:00
Eric Allam	64151d6ac8	chore(webapp): reduce telemetry ingestion log volume (#3832) ... ## Summary On a busy webapp the trace/log/metric ingestion path emits several `info` logs per insert batch, which makes up the bulk of the service's log output. This moves that per-batch chatter to `debug` and adds an opt-in to drop successful HTTP access logs, cutting log volume with no loss of error signal. ## Details The per-batch ClickHouse insert logs, the flush scheduler's concurrency adjustments, and the event-loop utilization sample (already exported as a metric, so the log line was redundant) now log at `debug`. Error and warning logs are untouched. New `HTTP_ACCESS_LOG_DISABLED=1` env var: when set, the HTTP access logger skips successful (2xx) requests while still logging non-2xx responses. Defaults off, so existing deployments are unchanged.	2026-06-04 15:36:24 +01:00
Katia Bulatova	cae3dcb7dd	Env vars page performance fix (#3829) ... ## Summary This PR improves performance across the Environment Variables page. ## Changes ### Targeted value loading - load only the non-secret (environmentId, key) pairs required by the page. Secret values continue to be redacted in the UI. ### SSR windowing + virtualization - SSR-render only the first 50 rows - hydrate those rows - virtualize the remaining dataset client-side - search is now URL-driven during SSR, ensuring deep links such as `?search=DATABASE_URL` ### Lightweight 'Create' flow - 'Create' page no longer loads the full Environment Variables dataset. ## Results Large projects no longer render thousands of rows during SSR. Example (~11k rendered rows): Metric | Before | After -- | -- | -- Document size | ~150 MB | ~5 MB SSR rows | ~11k | 50 Browser DOM rows | Thousands | ~26–38	2026-06-04 16:28:55 +02:00
Eric Allam	359e2503c9	feat(database,webapp): add LlmModel pricing_unit column and admin selector (#3820) ... ## Summary Adds a nullable `pricing_unit` column to the LLM model registry's `llm_models` table, recording how each model is billed ("tokens", "characters", "images", "minutes", "requests", "free", "not_findable"). It lets pricing-coverage reporting exclude models that aren't priced per-token (image/video/audio models currently drag the "% priced" number down even though they can never carry a per-token price), and lays the groundwork for non-token pricing. The default model catalog is entirely per-token, so `seed` and `syncLlmCatalog` set `pricing_unit="tokens"` on those rows. The admin LLM model form (create + edit) and the admin API get a pricing-unit selector so admin-curated models can set it; existing rows can stay unset. Auto-discovered models get their unit from the model-registry pipeline, which lands separately. --------- Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-06-03 15:08:36 +00:00
nicktrn	d541caeb5e	feat(supervisor): wide events + warm-start trace propagation (#3669) ... Adds wide-event observability for the supervisor: one flat-keyed JSON line per dequeue iteration, workload-server route, and run socket lifecycle event. Events carry `trace_id` sourced from the inbound W3C traceparent plus `meta.run_id` and related identifiers, so they join across services by run. The outbound warm-start POST also forwards the inbound traceparent so the upstream receiver continues the same trace instead of minting a new one. Off by default behind `TRIGGER_WIDE_EVENTS_ENABLED`. With the flag off, no events are emitted, no ALS state is allocated, and the outbound warm-start request is unchanged — every call site was audited to confirm the off path is byte-identical to current behavior. Dequeue-path phase timings recorded under `phase.<name>.duration_ms`: `restore`, `warm_start`, `workload_create`. A `path_taken` extra distinguishes `restore` / `warm_start` / `cold_create` / `skipped_no_image`. Refs TRI-9480.	2026-06-02 21:11:01 +01:00
Katia Bulatova	4f8cf4cc63	feat(webapp): runs live updating ... ## Summary The Runs list now updates live without requiring a page refresh. Status changes and other run fields are updated in place while runs are executing. When new runs matching the current filters are created, a "New runs created" refresh button appears above the list. Root runs now show a live child-run status breakdown directly in the status tooltip. ### List live update - Visible runs update in place while they are still running. - A "New runs created" refresh button appears when new matching runs are detected. - Polling stops when all visible runs have finished and a refresh button is already shown. - Polling pauses when the browser tab is not visible. - Runs list status updates and new-run detection share a single runs/live polling path. ### Child-status tooltip - Root run tooltips now display a breakdown of child run statuses. - Child statuses are loaded when the tooltip opens (after a 400ms hover delay). - The tooltip stays up to date while child runs are still changing state. - Handles cases where child runs continue running after their parent run has completed, or have not yet been created. ### Supporting changes - Added hidden-tab awareness to polling. - Added safeguards around polling inputs (`runIds` deduping and limits). ## Test plan - [x] pnpm run typecheck --filter webapp passes - [x] cd apps/webapp && pnpm run test ./test/presenters/mapRunToLiveFields.test.ts --run passes - [x] cd apps/webapp && pnpm run test ./test/runsRepository.part2.test.ts --run -t "hasNewRuns" passes ### Manual smoke: - [x] Active runs update without a page refresh. - [x] A new matching run shows the refresh banner and the banner actions work as expected. - [x] Root run tooltips show live child-status updates and stop polling once child runs settle. --------- Co-authored-by: Ekaterina Bulatova <kathiekiwi@Ekaterinas-MacBook-Pro.local>	2026-06-02 19:26:46 +02:00
Matt Aitken	6961004a73	fix(webapp): restore Postgres fallback for non-ClickHouse OTLP spans (#3803) ... ## Problem On environments where runs carry a Postgres-backed `taskEventStore` value (`taskEvent` or `taskEventPartitioned`), OTLP ingest endpoints (`POST /otel/v1/traces` and `/otel/v1/logs`) were returning HTTP 500. **Root cause:** The org-scoped ClickHouse factory introduced in a recent PR routes all OTLP spans through `getEventRepositoryForOrganizationSync` → `buildEventRepository`. That function only handles `"clickhouse"` and `"clickhouse_v2"` store values and throws `Unknown ClickHouse event repository store: <value>` for anything else. The throw occurred inside the grouping loop of `#exportEvents`, unwinding the entire method and returning 500 for the whole batch. The OpenTelemetry collector's `otlphttp` exporter treats HTTP 500 as non-retryable and drops the batch — causing real span loss. **Fix:** Guard the `getEventRepositoryForOrganizationSync` call in `#exportEvents` so it is only invoked for `clickhouse` / `clickhouse_v2` store values. All other values are routed directly to the Postgres `eventRepository`, matching the guard pattern already present in `resolveEventRepositoryForStore` and `getEventRepositoryForStore` in `eventRepository/index.server.ts`. The ClickHouse factory call is also wrapped in a try/catch that falls back to Postgres so any unexpected store value in a future OTLP batch degrades gracefully instead of failing the whole request. ## Changes - `apps/webapp/app/v3/otlpExporter.server.ts` — add Postgres routing guard and try/catch fallback in `#exportEvents` ## Testing The `eventRepository/index.server.ts` module already has the same guard pattern thoroughly covered. The fix brings `#exportEvents` into alignment with that existing, tested pattern. Manual verification: confirm OTLP batches containing Postgres-store spans return 200 and route to the correct repository.	2026-06-02 16:19:51 +01:00
Daniel Sutton	139eede41c	feat(redis-worker): batched pop in MollifierDrainer for fast single-env drains (#3797) ... ## Summary Adds `drainBatchSize` to `MollifierDrainer` (default `1` — preserves existing behaviour) and wires `TRIGGER_MOLLIFIER_DRAIN_BATCH_SIZE` through the webapp (default `50`). Each tick the drainer now pops up to `drainBatchSize` from each chosen env, then dispatches every popped entry through the shared `concurrency`-bounded `pLimit`. Per-org/per-env fairness is unchanged — only the in-env pop count grows. Pre-existing behaviour was one pop per env per tick. For a single-env burst that single-flighted the drain at the per-tick floor of `pop + engine.trigger ≈ 50–60 ms`. With buffer entries piling up under a real-world tenant burst that's tens of minutes of tail latency to fully materialise — even though PG itself could comfortably sustain the writes. ## Why this matters — heavy-tail illustration Scenario: 100 customers in one window — 94 fire 20 triggers each, 5 fire 100, 1 fires 1000. Gate at `THRESHOLD=10/s`, `HOLD_MS=500`. First 10 of each burst hit PG directly; the rest buffer. | Customers | Triggers each | PG direct | Buffered each | Total buffered | |---|---|---|---|---| | 94 small | 20 | 10 | 10 | 940 | | 5 medium | 100 | 10 | 90 | 450 | | 1 heavy | 1000 | 10 | 990 | 990 | **With `DRAIN_BATCH_SIZE=50`, `DRAIN_CONCURRENCY=50`, ~50 ms `engine.trigger`:** | Tick | Pops | Dispatch waves | Wall-clock | |---|---|---|---| | 1 | 94×10 + 5×50 + 1×50 = 1 240 | 25 × 50 ms | ~1 300 ms (94 smalls done) | | 2 | 5×40 + 1×50 = 250 | 5 × 50 ms | ~300 ms (5 mediums done) | | 3–20 | heavy alone, 50/tick | 1 × 50 ms | ~100 ms each | | Customer class | Buffered fully drained | |---|---| | 94 small | **~1.3 s** | | 5 medium | **~1.6 s** | | 1 heavy | **~3.4 s** | **Without batching (one pop per env per tick — current behaviour):** | Customer class | Buffered fully drained | |---|---| | 94 small | ~500 ms | | 5 medium | ~4.5 s | | 1 heavy | **~49 s** | So the heavy single-tenant tail drops from ~49 s to ~3.4 s (~14× faster) without changing PG load characteristics. Smalls go up slightly in this scenario (500 ms → 1.3 s) because all 100 envs share one tick's dispatch queue — that's the trade we accept for the heavy tail; the worst-case small wait is still inside one tick. PG load is identical either way (50 concurrent inserts at a time, capped by `DRAIN_CONCURRENCY`). ## What changed **`packages/redis-worker`** - New `drainBatchSize` option (default 1 — full backward compat). - `runOnce()` refactored to pop per-env batches in parallel, then dispatch all popped entries through the existing global `pLimit`. Mid-batch pop failure aborts only that env's batch and counts as one failure (same semantic as the old per-env path). - Removed the now-unused `processOneFromEnv` helper. **`apps/webapp`** - `TRIGGER_MOLLIFIER_DRAIN_BATCH_SIZE` env var (default 50, matching `DRAIN_CONCURRENCY`). - Wired into `mollifierDrainer.server.ts`. **Test cloud config** (separate cloud PR): `TRIGGER_MOLLIFIER_DRAIN_BATCH_SIZE="50"` on the worker service. Production rollout deferred until we've watched it on test cloud. ## Test plan - [x] All 25 stub-based drainer tests pass (18 pre-existing + 7 new). 7 new tests under `MollifierDrainer.drainBatchSize`: - pops up to `drainBatchSize` across ticks - global `concurrency` cap still holds when batch > concurrency - mid-batch pop failure isolation - multi-env batch fan-out in one tick - **hierarchical org fairness preserved at `drainBatchSize > 1`** (load-bearing — guards against future regressions to per-env-instead-of-per-org rotation) - mixed success/failure accounting in a batched tick - bounded pops on empty queue (no Lua spam past `drainBatchSize`) - [x] All pre-existing tests still pass unchanged at default `drainBatchSize=1` → backward-compat locked. - [x] `pnpm run build --filter @trigger.dev/redis-worker` clean. - [x] `pnpm run typecheck --filter webapp` clean. - [x] `redisTest` block (real Redis via testcontainers) — couldn't run locally on this branch due to testcontainers runtime discovery; will validate in CI. - [ ] Test-cloud smoke after cloud PR lands: fire `burst 50` against a flagged env and confirm the 50th entry's drain time drops from ~2.5 s to <200 ms. ## Notes - Per-tick memory bound: `maxOrgsPerTick × drainBatchSize` entries can sit in the JS pLimit queue between pop and dispatch. At defaults that's `500 × 50 = 25 000` × ~5 KB snapshot ≈ ~125 MB worst case per worker — well within headroom. - The pre-batch model's strict per-env throughput cap of `1/tick` is documented as the fairness baseline elsewhere. Org-level fairness is what callers actually rely on; this change does not weaken that. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>	2026-06-02 14:21:37 +01:00
DKP	fa5fc6a9c7	chore(webapp): remove Vercel install Loops event (#3805)	2026-06-02 12:49:08 +02:00
github-actions[bot]	e0681d2394	chore: release v4.5.0-rc.4 (#3788) ... ## Summary 1 new feature, 5 improvements. ## Highlights - Mollifier `mutateSnapshot` now enforces a tag cap: an `append_tags` patch carrying `maxTags` returns `"limit_exceeded"` (writing nothing) when the deduped tag count would exceed the limit, so a buffered run can't accumulate more tags via the tags API than the trigger validator allows at creation. ([#3756](https://github.com/triggerdotdev/trigger.dev/pull/3756)) ## Improvements - Mollifier buffer extensions: idempotency dedup, an atomic `mutateSnapshot` API, metadata CAS, claim primitives, and a `MollifierSnapshot` type. The buffer's Redis client now reconnects with jittered backoff so a fleet of clients doesn't stampede Redis in lockstep after a blip. ([#3752](https://github.com/triggerdotdev/trigger.dev/pull/3752)) - Add `onTerminalFailure` callback to `MollifierDrainerOptions` so the customer's run lands a SYSTEM_FAILURE PG row even when the drainer exhausts `maxAttempts` on a retryable PG error. Previously, retryable-error exhaustion called `buffer.fail()` directly, which atomically marks FAILED + DELs the entry hash with no PG write — silent data loss when PG was unreachable across the full retry budget. The callback fires before `buffer.fail()` on any terminal path (`cause: "non-retryable"` or `"max-attempts-exhausted"`); throwing a retryable error from the callback causes the drainer to requeue rather than fail. ([#3754](https://github.com/triggerdotdev/trigger.dev/pull/3754)) - Bump `@s2-dev/streamstore` to `0.22.10` to fix a `TASK_RUN_UNCAUGHT_EXCEPTION` ("Invalid state: Unable to enqueue") when a `chat.agent` turn is aborted mid-stream. ([#3792](https://github.com/triggerdotdev/trigger.dev/pull/3792)) - Coerce numeric `concurrencyKey` values to string at the API boundary across `tasks.trigger`, `tasks.batchTrigger`, and the Phase-2 streaming batch endpoint. ([#3789](https://github.com/triggerdotdev/trigger.dev/pull/3789)) - Add a `tools` option to `chat.agent`. Declaring your tools here threads them into the SDK's internal `convertToModelMessages`, so each tool's `toModelOutput` is re-applied when prior-turn history is re-converted. ([#3790](https://github.com/triggerdotdev/trigger.dev/pull/3790)) <details> <summary>Raw changeset output</summary> ⚠️⚠️⚠️⚠️⚠️⚠️ `main` is currently in **pre mode** so this branch has prereleases rather than normal releases. If you want to exit prereleases, run `changeset pre exit` on `main`. ⚠️⚠️⚠️⚠️⚠️⚠️ # Releases ## @trigger.dev/redis-worker@4.5.0-rc.4 ### Minor Changes - Mollifier buffer extensions: idempotency dedup, an atomic `mutateSnapshot` API, metadata CAS, claim primitives, and a `MollifierSnapshot` type. The buffer's Redis client now reconnects with jittered backoff so a fleet of clients doesn't stampede Redis in lockstep after a blip. ([#3752](https://github.com/triggerdotdev/trigger.dev/pull/3752)) - Add `onTerminalFailure` callback to `MollifierDrainerOptions` so the customer's run lands a SYSTEM_FAILURE PG row even when the drainer exhausts `maxAttempts` on a retryable PG error. Previously, retryable-error exhaustion called `buffer.fail()` directly, which atomically marks FAILED + DELs the entry hash with no PG write — silent data loss when PG was unreachable across the full retry budget. The callback fires before `buffer.fail()` on any terminal path (`cause: "non-retryable"` or `"max-attempts-exhausted"`); throwing a retryable error from the callback causes the drainer to requeue rather than fail. ([#3754](https://github.com/triggerdotdev/trigger.dev/pull/3754)) ### Patch Changes - Pipeline the per-entry `HGETALL` fetches in `MollifierBuffer.listEntriesForEnv`. The previous serial implementation issued one Redis round-trip per runId returned by `LRANGE`, which dominated stale-sweep wall-time at any meaningful backlog (at the sweep's default maxCount=1000, this is ~1000 RTTs per env per pass). Behaviour is unchanged — entries are still skipped when the entry hash has been torn down by a concurrent drainer ack/fail between the LRANGE and the HGETALL. ([#3752](https://github.com/triggerdotdev/trigger.dev/pull/3752)) - Mollifier `mutateSnapshot` now enforces a tag cap: an `append_tags` patch carrying `maxTags` returns `"limit_exceeded"` (writing nothing) when the deduped tag count would exceed the limit, so a buffered run can't accumulate more tags via the tags API than the trigger validator allows at creation. ([#3756](https://github.com/triggerdotdev/trigger.dev/pull/3756)) - Updated dependencies: - `@trigger.dev/core@4.5.0-rc.4` ## @trigger.dev/build@4.5.0-rc.4 ### Patch Changes - Updated dependencies: - `@trigger.dev/core@4.5.0-rc.4` ## trigger.dev@4.5.0-rc.4 ### Patch Changes - Bump `@s2-dev/streamstore` to `0.22.10` to fix a `TASK_RUN_UNCAUGHT_EXCEPTION` ("Invalid state: Unable to enqueue") when a `chat.agent` turn is aborted mid-stream. ([#3792](https://github.com/triggerdotdev/trigger.dev/pull/3792)) - Updated dependencies: - `@trigger.dev/core@4.5.0-rc.4` - `@trigger.dev/build@4.5.0-rc.4` - `@trigger.dev/schema-to-json@4.5.0-rc.4` ## @trigger.dev/core@4.5.0-rc.4 ### Patch Changes - Coerce numeric `concurrencyKey` values to string at the API boundary across `tasks.trigger`, `tasks.batchTrigger`, and the Phase-2 streaming batch endpoint. ([#3789](https://github.com/triggerdotdev/trigger.dev/pull/3789)) - Bump `@s2-dev/streamstore` to `0.22.10` to fix a `TASK_RUN_UNCAUGHT_EXCEPTION` ("Invalid state: Unable to enqueue") when a `chat.agent` turn is aborted mid-stream. ([#3792](https://github.com/triggerdotdev/trigger.dev/pull/3792)) ## @trigger.dev/plugins@4.5.0-rc.4 ### Patch Changes - Updated dependencies: - `@trigger.dev/core@4.5.0-rc.4` ## @trigger.dev/python@4.5.0-rc.4 ### Patch Changes - Updated dependencies: - `@trigger.dev/sdk@4.5.0-rc.4` - `@trigger.dev/core@4.5.0-rc.4` - `@trigger.dev/build@4.5.0-rc.4` ## @trigger.dev/react-hooks@4.5.0-rc.4 ### Patch Changes - Updated dependencies: - `@trigger.dev/core@4.5.0-rc.4` ## @trigger.dev/rsc@4.5.0-rc.4 ### Patch Changes - Updated dependencies: - `@trigger.dev/core@4.5.0-rc.4` ## @trigger.dev/schema-to-json@4.5.0-rc.4 ### Patch Changes - Updated dependencies: - `@trigger.dev/core@4.5.0-rc.4` ## @trigger.dev/sdk@4.5.0-rc.4 ### Patch Changes - Add a `tools` option to `chat.agent`. Declaring your tools here threads them into the SDK's internal `convertToModelMessages`, so each tool's `toModelOutput` is re-applied when prior-turn history is re-converted. ([#3790](https://github.com/triggerdotdev/trigger.dev/pull/3790)) ```ts chat.agent({ tools: { readFile, search }, run: async ({ messages, tools, signal }) => streamText({ model, messages, tools, abortSignal: signal }), }); ``` Also exports `InferChatUIMessageFromTools<typeof tools>` to derive the chat `UIMessage` type (typed tool parts) directly from a tool set. - Updated dependencies: - `@trigger.dev/core@4.5.0-rc.4` </details> Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>	2026-06-02 09:35:14 +01:00
nicktrn	e35f5742d4	fix(webapp): upgrade posthog-node to v5, drop axios + stale override (#3801) ... Follow-up to #3796, which bumped the slack-client axios paths but left posthog-node's transitive `axios@1.15.1` in place. `posthog-node` 4.17.1 → 5.35.6. v5 drops the axios dependency entirely (it's now fetch-based via `@posthog/core`), so posthog's old axios path disappears. With #3796 already on main (webapp + d3 references on `@slack/web-api@7.16.0`), nothing else pins the old line, so the now-dead `axios@>=1.0.0 <1.15.0` override is removed and axios resolves to a single patched `1.16.1` repo-wide. This closes the remaining axios advisories. Compat: the webapp's usage in `telemetry.server.ts` (`new PostHog(key, { host })`, `.identify`, `.groupIdentify`, `.capture`) is all object-form API that v5 preserves; `pnpm run typecheck --filter webapp` passes. Node: posthog-node v5 requires Node `^20.20.0 || >=22.22.0`. We run 20.20.0 in dev (`.nvmrc`), CI, and the published Docker image (`node:20.20-bullseye-slim`), so we're compliant.	2026-06-02 10:33:44 +02:00
nicktrn	952139d492	fix(webapp): bump @slack/web-api to 7.16.0 for patched axios (#3796) ... Bumps `@slack/web-api` 7.9.1 → 7.16.0 in the webapp and the two `references` examples (d3-chat, d3-openai-agents). 7.16.0 depends on `axios@^1.16.0`, so every slack-client axios path resolves to 1.16.1 instead of 1.15.1. This clears the slack and references axios paths. `posthog-node`'s transitive axios still resolves the older line - that's handled in a follow-up that upgrades posthog-node to v5 (which drops the axios dependency entirely and lets us retire the now-stale axios override). The dependabot axios advisories fully close once both land.	2026-06-02 08:18:28 +01:00
Daniel Sutton	e21b68cc5f	feat(webapp): dashboard parity for mollifier-buffered runs (#3757) ... ## Summary Dashboard surfaces handle buffered runs by falling back to the mollifier snapshot: - Run detail, span detail, streams view (`_app.../runs.\$runParam`, `resources.../spans.\$spanParam`, `resources.../streams.\$streamKey`). - Redirect routes (`@.runs.\$runParam`, `runs.\$runParam`, `projects.v3.\$projectRef.runs.\$runParam`). - Action routes — cancel / replay / idempotency-reset / debug — under `resources.taskruns/...` and `resources.../idempotencyKey.reset`. - Logs download. - Realtime subscription route + per-run resource (`realtime.v1.runs.\$runId`, `resources.../realtime.v1.*`). - `CancelRunDialog` gains an `onCancelSubmitted` callback so submit isn't raced by the Radix `DialogClose` wrapper. Stacked on the mutations PR. ## Test plan - [x] \`pnpm run typecheck --filter webapp\` passes - [x] \`pnpm run test --filter webapp test/mollifierRealtimeRunResource.test.ts\` passes - [x] \`pnpm run test --filter webapp test/mollifierRealtimeRunResourceBuffer.test.ts\` passes - [x] \`pnpm run test --filter webapp test/mollifierRealtimeSubscription.test.ts\` passes - [x] Manual smoke: trigger a buffered run, open it in the dashboard, replay/cancel from the UI --- ## Ship-gate follow-up fixes - **Auto-redirect to root span on direct nav** — loader sets `?span=` from root span (PG) or buffered snapshot spanId before 302'ing, so bookmark/share-link/direct-nav doesn't leave the panel collapsed. - **RunPresenter switches from `findFirstOrThrow` to `findFirst` + typed `RunNotInPgError`** — kills the per-poll `PrismaClient error` log spam for buffered runs without changing the route-loader's fallback flow. - **Span detail panel renders for buffered runs** — `SpanPresenter.call` now falls back to `findRunByIdWithMollifierFallback` + `buildSyntheticSpanRun` instead of returning undefined and triggering the "Event not found" toast loop. - **Logs download for buffered runs returns a gzipped placeholder line** — replaces the 404 with a content-encoded line explaining the run is queued. Same org-membership gate as the PG path. - **Admin Debug-Run button hidden for buffered runs + SpanRun circular type alias broken** (squashed) — buttons gate on a new `isBuffered` flag on the synthetic SpanRun. Required grounding SpanRun in `SpanPresenter.getRun` to break a circular type alias TS no longer tolerates once `isBuffered` is a literal field on the shape. - **Replay action requires user auth + org-membership** (🚩 Devin finding) — `action` was unauthenticated and the PG `findFirst` had no org filter, so any caller with a valid `runParam` could replay any run. Buffered fallback inherited the same gap. Fixed to mirror the cancel route. --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>	2026-06-01 16:50:31 +01:00
Daniel Sutton	e1950778e2	feat(webapp): mollifier API mutations on buffered runs (#3756) ... ## Summary Cancel, replay, reschedule, metadata, tags, and idempotency-key-reset now succeed against a run that's still in the mollifier buffer. Mutations are applied to the buffered snapshot via Lua CAS; the drainer carries the mutation forward when it replays. Primitives added: - `mutateWithFallback` — PG-first / buffer-fallback resolver with bounded-wait safety net for entries that transition mid-mutation. - `applyMetadataMutation` — buffered metadata PUT mirroring the PG-side retry loop with CAS atomicity. - `resolveRunForMutation` — discriminated-union resolver used by route `findResource` so the route builder's pre-action 404 check sees buffered runs. Routes wired (whole files, no GET/POST splits): - `api.v2.runs.\$runParam.cancel.ts` - `api.v1.runs.\$runParam.replay.ts` - `api.v1.runs.\$runParam.reschedule.ts` - `api.v1.runs.\$runId.metadata.ts` - `api.v1.runs.\$runId.tags.ts` - `resetIdempotencyKey.server.ts` Stacked on the reads PR. ## Test plan - [x] \`pnpm run typecheck --filter webapp\` passes - [x] \`pnpm run test --filter webapp test/mollifierMutateWithFallback.test.ts\` passes - [x] \`pnpm run test --filter webapp test/mollifierApplyMetadataMutation.test.ts\` passes - [x] \`pnpm run test --filter webapp test/mollifierResolveRunForMutation.test.ts\` passes --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>	2026-06-01 16:24:10 +01:00
Daniel Sutton	a1dc3c5ae1	feat(webapp): mollifier API GET read-fallback — synthetic primitives + route wiring (#3755) ... ## Summary Synthesise QUEUED/FAILED responses from the mollifier buffer when a TaskRun row hasn't landed in Postgres yet. Wires the synthesis into: - `ApiRetrieveRunPresenter` - v1 trace GET route - v1 spans GET route - attempts route gains a GET loader (fixes pre-existing Remix "no loader" 400) The `readFallback` infra itself lives on the trigger PR (consumed by `IdempotencyKeyConcern`); this PR adds the route-level synthetic-rendering primitives. Stacked on the replay PR. ## Test plan - [x] \`pnpm run typecheck --filter webapp\` passes - [x] \`pnpm run test --filter webapp test/mollifierSyntheticRedirectInfo.test.ts\` passes - [x] \`pnpm run test --filter webapp test/mollifierSyntheticSpanRun.test.ts\` passes --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>	2026-06-01 14:53:58 +01:00
Daniel Sutton	4745754a7a	feat(webapp,run-engine): mollifier drainer replay + stale sweep + cancelled-run engine API (#3754) ... ## Summary The replay side of the mollifier: - `DrainerHandler`: reads buffered snapshots and replays them through `engine.trigger` to materialise PG rows. - `RunEngine.createCancelledRun`: new public method the handler uses to write CANCELED rows directly from snapshots (bypass queue + waitpoint, emit `runCancelled`). Tolerates the cjson empty-table tags edge case found during validation. - Drainer fairness: org → env rotation so a heavy env doesn't starve light ones in the same org. - Stale-entry sweep + telemetry + alertable gauge so a stuck/offline drainer surfaces in alerts. Both the drainer and sweep default-off; nothing fires unless flagged on (`TRIGGER_MOLLIFIER_DRAINER_ENABLED`, `TRIGGER_MOLLIFIER_STALE_SWEEP_ENABLED`). Stacked on the trigger-time decisions PR. ## Test plan - [x] \`pnpm run typecheck --filter webapp\` passes - [x] \`pnpm run test --filter webapp test/mollifierDrainerHandler.test.ts\` passes - [x] \`pnpm run test --filter webapp test/mollifierStaleSweep.test.ts\` passes - [x] \`pnpm run test --filter @internal/run-engine src/engine/tests/createCancelledRun.test.ts\` passes - [x] \`pnpm run test --filter @trigger.dev/redis-worker packages/redis-worker/src/mollifier/drainer.test.ts\` passes --- ## Ship-gate follow-up fix **Drainer writes SYSTEM_FAILURE on max-attempts exhaustion.** Adds an `onTerminalFailure` callback on `MollifierDrainerOptions` so the customer's run lands a SYSTEM_FAILURE PG row even when the drainer exhausts `MAX_ATTEMPTS` on a retryable PG error (previously `buffer.fail()` was called with no row written → silent data loss). The callback runs before `buffer.fail()` on every terminal path (non-retryable AND max-attempts-exhausted), and re-throwing a retryable error from the callback causes the drainer to requeue rather than fail. Bumps `@trigger.dev/redis-worker` to a **minor** changeset (additive option + new exported types). Includes 5 unit tests covering both terminal causes plus the requeue-on-retryable-callback-failure path and no-callback back-compat. --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>	2026-06-01 13:20:05 +01:00
Daniel Sutton	577f35eebe	feat(webapp): mollifier trigger-time decisions — mollify, claim, read fallback (#3753) ... ## Summary The trigger hot path's mollifier integration: - `mollifyTrigger`: when the gate trips, write the engine.trigger snapshot to the buffer and return a synthesised QUEUED response. Postgres write is deferred to drainer-replay (next PR in the stack). - Pre-gate idempotency-key claim: same-key triggers serialise through Redis so a burst lands in PG / buffer exactly once. - Read-fallback extensions: `findRunByIdWithMollifierFallback` for the trigger-time idempotency lookup that must see buffered runs. - Gate bypasses: `debounce`, `oneTimeUseToken`, `parentTaskRunId`/`triggerAndWait` skip the mollify path entirely. - `triggerTask` + `IdempotencyKeyConcern` wired to the above. All behaviour gated by the master `TRIGGER_MOLLIFIER_ENABLED` switch; off-state hot path is unchanged (the gate is not even consulted). Stacked on the buffer extensions PR. ## Test plan - [x] \`pnpm run typecheck --filter webapp\` passes - [x] \`pnpm run test --filter webapp test/mollifierMollify.test.ts\` passes - [x] \`pnpm run test --filter webapp test/mollifierIdempotencyClaim.test.ts\` passes - [x] \`pnpm run test --filter webapp test/mollifierReadFallback.test.ts\` passes - [x] \`pnpm run test --filter webapp test/mollifierGate.test.ts\` passes - [x] \`pnpm run test --filter webapp test/engine/triggerTask.test.ts\` passes --- ## Ship-gate follow-up fixes - **Batch items bypass the mollifier gate** — fixes `BatchTaskRunItem_taskRunId_fkey` FK violation on batch triggers when the gate trips. End-state is a drainer-side `BatchTaskRunItem` create-on-materialise; batch traffic passes through the gate until that lands. - **IdempotencyKeyConcern honours buffered-run TTL on expiry** — buffered path now clears expired idempotency claims (read-side) and resets the buffer's `mollifier:idempotency:*` SETNX binding (write-side) so a re-trigger past the customer's TTL lands as a fresh run instead of echoing the stale buffered runId. --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>	2026-06-01 12:00:24 +01:00
github-actions[bot]	8f066ac389	chore: release v4.5.0-rc.3 (#3763) ... ## Summary 1 improvement, 2 bug fixes. ## Improvements - Retry `TASK_MIDDLEWARE_ERROR` under the task's retry policy instead of failing the run on the first attempt. The error was already classified as retryable by `shouldRetryError`, but `shouldLookupRetrySettings` did not include it, so the retry flow fell through to `fail_run`. Fixes #3231. ([#3676](https://github.com/triggerdotdev/trigger.dev/pull/3676)) ## Bug fixes - Fix `TypeError` in `unflattenAttributes` when the input attribute map contains conflicting dotted key paths (e.g. both `a.b` set to a scalar and `a.b.c` set to a value). The path-walk loop now applies last-write-wins when a prior key wrote a primitive, null, or array at an intermediate slot, matching the existing precedent in `AttributeFlattener.addAttribute`. Callers no longer crash when handed malformed external attribute inputs. ([#3762](https://github.com/triggerdotdev/trigger.dev/pull/3762)) - Fix external trace context leaking across runs on warm-started workers with `processKeepAlive` enabled. Every subsequent run's attempt span was being exported with the first run's `traceId` and `parentSpanId`, breaking causal-chain navigation in external APM tools. Runs without an external trace context are unaffected. ([#3768](https://github.com/triggerdotdev/trigger.dev/pull/3768)) <details> <summary>Raw changeset output</summary> ⚠️⚠️⚠️⚠️⚠️⚠️ `main` is currently in **pre mode** so this branch has prereleases rather than normal releases. If you want to exit prereleases, run `changeset pre exit` on `main`. ⚠️⚠️⚠️⚠️⚠️⚠️ # Releases ## @trigger.dev/build@4.5.0-rc.3 ### Patch Changes - Updated dependencies: - `@trigger.dev/core@4.5.0-rc.3` ## trigger.dev@4.5.0-rc.3 ### Patch Changes - Updated dependencies: - `@trigger.dev/core@4.5.0-rc.3` - `@trigger.dev/build@4.5.0-rc.3` - `@trigger.dev/schema-to-json@4.5.0-rc.3` ## @trigger.dev/core@4.5.0-rc.3 ### Patch Changes - Retry `TASK_MIDDLEWARE_ERROR` under the task's retry policy instead of failing the run on the first attempt. The error was already classified as retryable by `shouldRetryError`, but `shouldLookupRetrySettings` did not include it, so the retry flow fell through to `fail_run`. Fixes #3231. ([#3676](https://github.com/triggerdotdev/trigger.dev/pull/3676)) - Fix `TypeError` in `unflattenAttributes` when the input attribute map contains conflicting dotted key paths (e.g. both `a.b` set to a scalar and `a.b.c` set to a value). The path-walk loop now applies last-write-wins when a prior key wrote a primitive, null, or array at an intermediate slot, matching the existing precedent in `AttributeFlattener.addAttribute`. Callers no longer crash when handed malformed external attribute inputs. ([#3762](https://github.com/triggerdotdev/trigger.dev/pull/3762)) - Fix external trace context leaking across runs on warm-started workers with `processKeepAlive` enabled. Every subsequent run's attempt span was being exported with the first run's `traceId` and `parentSpanId`, breaking causal-chain navigation in external APM tools. Runs without an external trace context are unaffected. ([#3768](https://github.com/triggerdotdev/trigger.dev/pull/3768)) ## @trigger.dev/plugins@4.5.0-rc.3 ### Patch Changes - Updated dependencies: - `@trigger.dev/core@4.5.0-rc.3` ## @trigger.dev/python@4.5.0-rc.3 ### Patch Changes - Updated dependencies: - `@trigger.dev/core@4.5.0-rc.3` - `@trigger.dev/build@4.5.0-rc.3` - `@trigger.dev/sdk@4.5.0-rc.3` ## @trigger.dev/react-hooks@4.5.0-rc.3 ### Patch Changes - Updated dependencies: - `@trigger.dev/core@4.5.0-rc.3` ## @trigger.dev/redis-worker@4.5.0-rc.3 ### Patch Changes - Updated dependencies: - `@trigger.dev/core@4.5.0-rc.3` ## @trigger.dev/rsc@4.5.0-rc.3 ### Patch Changes - Updated dependencies: - `@trigger.dev/core@4.5.0-rc.3` ## @trigger.dev/schema-to-json@4.5.0-rc.3 ### Patch Changes - Updated dependencies: - `@trigger.dev/core@4.5.0-rc.3` ## @trigger.dev/sdk@4.5.0-rc.3 ### Patch Changes - Updated dependencies: - `@trigger.dev/core@4.5.0-rc.3` </details> Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>	2026-06-01 10:34:45 +01:00
Eric Allam	9211032733	chore(database): drop unused TaskRun status composite index (#3743) ... ## Summary Drops the `TaskRun_status_runtimeEnvironmentId_createdAt_id_idx` index from the `TaskRun` table. After #3742 gated the legacy `WAITING_FOR_DEPLOY` drain to V1-engine workers only, this index sees zero scans on both writer and reader replicas. Removing it cuts index maintenance on every `TaskRun` INSERT/UPDATE. ## Why The index existed to support `WHERE status = X AND runtimeEnvironmentId = Y` queries from `ExecuteTasksWaitingForDeployService`, which is V1-only and no longer triggered on V2 deployments. A code grep across `apps/webapp` and `internal-packages/run-engine` confirmed no V2 production query uses this access pattern — every other `status:` filter on `TaskRun` is paired with `id`/`friendlyId`/`parentSpanId` and uses a different index. Dropping it also unlocks HOT updates on the dequeue path. The dequeue `UPDATE` modifies `status` (`QUEUED` -> `DEQUEUED`), and `status` is the leading column of this index — its presence blocked HOT eligibility for every `TaskRun` UPDATE. With the index gone, dequeue UPDATEs can become HOT, reducing WAL bytes and removing the B-tree page contention on this index's right-edge leaves. Uses `DROP INDEX CONCURRENTLY` to avoid blocking writes during the drop. ## Sequencing Should only ship once #3742 has soaked long enough to confirm the index is genuinely cold (24h+ of zero scans on `pg_stat_user_indexes`).	2026-06-01 09:37:37 +01:00
Saadi Myftija	9cb6fd17c3	fix(webapp): idempotent DeploymentBackgroundWorker creation (#3772) ... Make `POST /api/v1/deployments/:deploymentId/background-workers` idempotent so client-side retries no longer collide on the `BackgroundWorker` `(project, env, version)` unique index. Helps make deployments more resilient against the class of indexing failures that surfaces in the dashboard as "Indexing timed out", e.g. during transient database issues.	2026-05-29 10:03:55 +02:00
Matt Aitken	816986d44e	fix(webapp): treat Phase 2 batch-stream retries as idempotent (TRI-9944) (#3766) ... Returns sealed:true when a fast-completing batchTrigger races the stream finalisation, instead of throwing 422/BatchTriggerError.	2026-05-28 09:28:18 +00:00
Matt Aitken	90bbbd1c44	fix(webapp): recover from ClickHouse JSON parse failures on out-of-range integers (#3759) ... ## Summary Second class of poisoned-row failure in the runs replication path. PR #3708 plugged lone UTF-16 surrogates; this one handles bare JSON integer literals outside ClickHouse's `Int64`..`UInt64` range. Recovery stays purely reactive — the existing `sanitizeRows` walker just gains an extra branch, so the hot replication path pays nothing on healthy rows. Fixes the still-firing customer-facing symptom from [TRI-9755](https://linear.app/triggerdotdev/issue/TRI-9755): `scan-social-profiles` runs continued to be stranded in `EXECUTING` on the Tasks page after #3708 deployed. CloudWatch showed `Dropped batch — ClickHouse JSON parse error but sanitizer found nothing to fix` firing **8/8 times** since the previous deploy (zero successful sanitizations). Root cause: upstream JS Number precision loss on a 21-digit Google Plus ID (`117039831458782873093` → `117039831458782870000`) — the precision-lossy value still serialises as a bare integer that exceeds `UInt64.MAX`, which ClickHouse rejects with `INCORRECT_DATA`. ## How the bug ships The customer task emits an output containing a Poshmark profile's `spec_format`: ```json {"key":"gp_id","proper_key":"Gp Id","value":117039831458782870000,"type":"int"} ``` That value is `1.17e20` — comfortably above `UInt64.MAX` (`1.84e19`) but comfortably below `1e21`. `Number.prototype.toString` only switches to exponential form at `|value| >= 1e21`, so `JSON.stringify` emits the bare token `117039831458782870000` and the ClickHouse `JSON(max_dynamic_paths)` column fails with: ``` Code: 117. DB::Exception: Cannot parse JSON object here: {…}: (while reading the value of key output): (at row 1) : While executing ParallelParsingBlockInputFormat. (INCORRECT_DATA) (version 25.12.x) ``` Same error verbatim as prod. The same number quoted (`"117039831458782870000"`) inserts fine — ClickHouse's dynamic JSON column accepts a `String` subtype on the same path. ## What changed `apps/webapp/app/v3/eventRepository/sanitizeRowsOnParseError.server.ts`: - New private `isUnsafeJsonInteger(value)` helper — true iff `value` is a finite integer-valued JS Number where `|value| < 1e21` (so `JSON.stringify` emits integer form, not exponent) **and** `value` falls outside `[Int64.MIN, UInt64.MAX]`. - `sanitizeUnknownInPlace` gains a number-branch: when the predicate holds, replace the Number with `String(value)`. The downstream JSON column dynamic-types the path as String for that row — fine, since the value was already precision-lossy upstream (no JS Number above 2^53 is numerically meaningful anyway). - Float-valued numbers, large floats (>= 1e21), NaN and Infinity are left alone — `JSON.stringify` emits them with exponents or as `null`, both of which ClickHouse accepts. `apps/webapp/test/sanitizeRowsOnParseError.test.ts`: four new unit tests + an extension to `sanitizeRows` covering surrogate + integer fixes counted together across rows. The unit suite now covers: - Positive value above `UInt64.MAX` (`117039831458782870000` — the actual prod value) - Negative value below `Int64.MIN` - Boundary values pass through (`42`, `Number.MAX_SAFE_INTEGER`, `2^63`) - Non-integer numbers untouched (floats, `1e25`, NaN, Infinity) - The actual `scan-social-profiles` nested shape — finds the offending `gp_id` deep inside `output.data.profiles[].spec_format[].platform_variables[].value` `.server-changes/runs-replication-bigint-recovery.md` — release notes entry. ## Why reactive, not pre-flight `#prepareJson` runs millions of times per day on the replication hot path. Walking every JSON tree to look for oversized integers would add bounded-but-real CPU on every healthy row. `sanitizeRows` only fires after a ClickHouse parse-error rejection, which is a few times a day platform-wide. Extending it costs effectively zero on healthy traffic and gains us recovery on the rare poisoned row. ## Verification - Reproduced 1:1 in a throwaway Docker `clickhouse/clickhouse-server:25.12.11.4` (closest available to the prod `25.12.1.1579` build). Pre-sanitize JSON fails with the exact prod error; post-sanitize JSON inserts cleanly and the row is readable with `gp_id` stored as a String subtype. - `pnpm --filter webapp exec vitest run test/sanitizeRowsOnParseError.test.ts` — 22/22 passing (18 existing + 4 new). - `pnpm run typecheck --filter webapp` — clean. ## Test plan - [x] `pnpm run typecheck --filter webapp` - [x] Unit tests pass against new + existing cases - [x] End-to-end Docker ClickHouse repro confirms recovery - [ ] Post-deploy: confirm `Sanitizing batch after ClickHouse JSON parse error` warns fire instead of `Dropped batch …` errors when `scan-social-profiles` outputs trip CH again - [ ] Post-deploy: confirm `permanentlyDroppedBatches` counter stops climbing in `/stp/trigger-app-prod/ecs/replication/service-container/process-logs` ## What this does NOT do - Doesn't backfill the ~120k+ existing stranded `EXECUTING` rows in production. Same as #3708 — that needs a reconciliation/backfill sweep (separate ticket — TRI-9755 fix #3). - Doesn't address the upstream root cause (the customer task emitting a JS-Number-precision-lossy big int). That's a customer-task concern; our replication path needs to be robust to whatever shape arrives. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>	2026-05-27 09:00:53 -07:00
Eric Allam	37eeaa3690	fix(webapp): skip V1 WAITING_FOR_DEPLOY drain on V2 promotes (#3742) ... ## Summary Stops the legacy V1 `WAITING_FOR_DEPLOY` drain from running on every V2 deployment promotion. The drain queries `TaskRun` by `status='WAITING_FOR_DEPLOY'`, which only V1-engine runs ever have — V2 runs use `PENDING_VERSION` and are handled out of band. Calling the drain on V2 promotes produced empty queries against the status index and unnecessary reader-DB load. ## Fix Two layers: 1. Gate the enqueue at the call site in `ChangeCurrentDeploymentService` so it only fires when the deployment's worker is on engine V1. 2. Add a `LEGACY_RUN_ENGINE_WAITING_FOR_DEPLOY_DISABLED` env var (default `0`). When set to `1`, the service returns immediately from `call()` — neuters any jobs already sitting in the worker queue from before the deploy lands. V1 customers see no change; V2 promotes no longer trigger the drain.	2026-05-25 07:10:13 +01:00