helm-v4.4.4
52 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Eric Allam
|
486f49791d |
fix(webapp): eliminate SSE abort-signal memory leak (#3430)
## Summary Fixes a server-side memory leak in the webapp's SSE helper. Every aborted SSE connection (client tab close, navigation, timeout) was pinning its full request/response graph indefinitely on Node 20, so any long-running webapp process accumulated retained memory proportional to streaming-request churn. ## Root cause `apps/webapp/app/utils/sse.ts` combined four abort signals via `AbortSignal.any([requestAbortSignal, timeoutSignal, internalController.signal])`. The composite signal tracks its source signals in an internal `Set<WeakRef>` registered against a `FinalizationRegistry`; under sustained traffic those entries accumulate faster than they're cleaned up, pinning every source signal (and its listeners, and anything those listeners close over) until the parent signal itself is GC'd or aborts. This is a long-standing Node issue with multiple open reports: - [nodejs/node#54614](https://github.com/nodejs/node/issues/54614) — original report, still open. A [follow-up from ChainSafe](https://github.com/nodejs/node/issues/54614#issuecomment-4055656572) describes the exact same shape in a Lodestar production workload (req + timeout signals composed per request accumulating in long-running worker) and the same mitigation: drop `AbortSignal.any`, compose manually. - [nodejs/node#55351](https://github.com/nodejs/node/issues/55351) — mechanism confirmed by Node member @jasnell: *"the set of dependent signals known to the AbortSignal are kept in an internal Set using WeakRefs. The AbortSignals are being properly gc'd but the Set is never cleaned out of the WeakRefs making those leak."* Partially fixed by [PR #55354](https://github.com/nodejs/node/pull/55354), shipped in Node 22.12.0 — but only covers the tight-loop case, not long-lived parent signals. - [nodejs/node#57584](https://github.com/nodejs/node/issues/57584) — circular-dependency variant, still open. - [nodejs/node#62363](https://github.com/nodejs/node/issues/62363) — regression in Node 24/25 from an unrelated V8 change ("Don't pretenure WeakCells"). Different root cause, same symptom. A separate issue in `apps/webapp/app/entry.server.tsx` — `setTimeout(abort, ABORT_DELAY)` with no `clearTimeout` on success paths — kept the React render tree + `remixContext` alive for 30s per successful HTML request. Same pattern fixed upstream in React Router templates ([react-router#14200](https://github.com/remix-run/react-router/pull/14200)), never backported to Remix v2. ## What changed - **`apps/webapp/app/utils/sse.ts`** — single-signal abort chain. `AbortSignal.any` removed; `AbortSignal.timeout` replaced by a plain `setTimeout` cleared when the controller aborts; named sentinel constants used as stackless abort reasons; request-abort handler explicitly removed on cleanup. - **`apps/webapp/app/entry.server.tsx`** — clears the `setTimeout(abort, ABORT_DELAY)` timer in `onShellReady` / `onAllReady` / `onShellError`. - **`apps/webapp/app/v3/tracer.server.ts` + `env.server.ts`** — gates OpenTelemetry `HttpInstrumentation` and `ExpressInstrumentation` behind `DISABLE_HTTP_INSTRUMENTATION=true` as an escape hatch for future OTel-listener retention patterns. Defaults to enabled. - **`apps/webapp/app/presenters/v3/RunStreamPresenter.server.ts`** — uses the shared `ABORT_REASON_SEND_ERROR` sentinel. ## Verification ### Full-app reproduction (memlab) Isolated local harness, 500 abrupt SSE disconnects against a dev-presence route, GC between passes, heap snapshot diff with [memlab](https://facebook.github.io/memlab/): | Run | Heap delta after 500 conns + GC | memlab retained leaks | | --- | --- | --- | | Before | +16.0 MB (linear with request count) | 158 clusters; 250 `ServerResponse`, 1000 `AbortController`, 250 `SpanImpl` retained | | After | **+3.3 MB (noise)** | **0 app-code leaks** | ### Standalone mechanism isolation To confirm *which* axis of the change is load-bearing, a separate standalone Node script (`/tmp/abort-leak-test.mjs`) ran 2000 requests × 200 KB payload per variant: | Variant | Heap delta after GC | | --- | --- | | baseline (no signal machinery) | 0 MB | | V1: `AbortSignal.any` + string abort reason | **+9.1 MB** | | V2: `AbortSignal.any` only (no reason) | **+10.8 MB** | | V3: string reason only (no `AbortSignal.any`) | 0 MB | | V4: neither (the fix) | 0 MB | | V5: `AbortSignal.any` with no listener on the composite | **+10.2 MB** | This proves `AbortSignal.any` is the sole mechanism. The reason type (`.abort()` vs `.abort("string")`) is irrelevant for retention — V3 is clean, V5 leaks even without a listener on the composite. ## Risk - `sse.ts` is used by the dev-presence routes. Behaviour is equivalent — timeouts and client disconnects still abort the stream. `signal.reason` is now a named string sentinel (`"timeout"`, `"request_aborted"`, etc.) instead of the previous string arg or default `AbortError`. No in-tree reader of `signal.reason` exists. - `entry.server.tsx` change is a standard cleanup of an abort timer, matches upstream React Router guidance. - `tracer.server.ts` change is env-gated and defaults to current behaviour. - Three other webapp `AbortSignal.timeout()` callsites (alert delivery, remote-build status) are fire-and-forget passed directly to `fetch` — not composed with anything long-lived, no retention risk, untouched. ## Test plan - [ ] Existing SSE integration tests pass - [ ] Dev-presence SSE behaves normally across tab open/close cycles - [ ] No heap growth under sustained aborted-connection traffic (heap snapshot diff) ## Follow-up The same `AbortSignal.any([userSignal, internalSignal])` pattern exists in several SDK/core callsites that ship to customers (`packages/core/src/v3/realtimeStreams/manager.ts`, `packages/trigger-sdk/src/v3/{ai,chat,chat-client,sessions}.ts`, `packages/core/src/v3/workers/warmStartClient.ts`). Whether those leak in practice depends on the user passing a long-lived signal. Tracked separately. |
||
|
Oskar Otwinowski
|
8eb596f3fe | fix(vercel): Fix vercel settings page (#3424) | ||
|
Eric Allam
|
2d3b2e82e6 |
feat(run-engine): flag to route getSnapshotsSince through read replica (#3423)
## Summary Adds `RUN_ENGINE_READ_REPLICA_SNAPSHOTS_SINCE_ENABLED` (default `"0"`). When enabled, the Prisma reads inside `RunEngine.getSnapshotsSince` run against the read-only replica client instead of the primary. Offloads the snapshot-polling queries fired by every running task runner off the writer. ## Why `getSnapshotsSince` is called from the managed runner's fetch-and-process loop (once per poll interval, plus on every snapshot-change notification). It runs four sequential reads per call — one `findFirst` by snapshot id, one `findMany` on snapshots with `createdAt > X`, one raw SQL against `_completedWaitpoints`, and chunked `findMany` on `waitpoint`. Per concurrent run, every few seconds. It's read-only, tolerates a small amount of staleness, and is an obvious candidate for the replica. ## Replica-lag considerations - **Step 1 "since snapshot not found"**: if the runner just received a snapshot id from the primary and asks the replica before it replicates, the function throws and the caller treats the response as an error (runner falls back to a metadata refresh). Self-correcting, not silent. - **Step 2 missing newly-created snapshots**: the next poll's `createdAt > sinceSnapshot.createdAt` filter still picks them up once the replica catches up. - **Waitpoint junction race**: the riskiest path — if a latest snapshot is replicated but its `_completedWaitpoints` join rows aren't yet, the runner could advance past that snapshot with `completedWaitpoints: []`. WAL/storage-level replication replays commits in order, so in practice both should appear atomically on the reader, but the race window is why the flag ships disabled. Aurora reader shrinks all three windows to single-digit ms in typical conditions, and its storage-level replication gives atomic visibility of committed transactions on the reader. ## Test plan - [ ] Flip the flag on in a non-prod environment, confirm snapshot polling behaves normally and `getSnapshotsSince` errors in Sentry stay flat. - [ ] Verify writer query volume drops and reader query volume rises on the snapshot-polling queries. - [ ] Keep an eye on `AuroraReplicaLag` (or equivalent) during rollout. |
||
|
Eric Allam
|
7c95ee498e |
feat(webapp): tag Prisma spans with db.datasource attribute (#3422)
## Summary
Stamp every Prisma span with `db.datasource: "writer" | "replica"` so
traces can distinguish which client the query went through.
Both `PrismaClient` instances share the same global
`@prisma/instrumentation`, so their spans come out with identical names
and attributes today. This makes them trivially filterable.
## How
Two pieces in `apps/webapp/app/`:
1. **`v3/tracer.server.ts`** — a `DatasourceAttributeSpanProcessor`
reads an OTel context key in `onStart` and calls
`span.setAttribute("db.datasource", value)`. Registered as the first
span processor.
2. **`db.server.ts`** — `tagDatasource(datasource, client)` wraps each
`PrismaClient` with `$extends({ query: { $allOperations } })`. The
middleware sets the context key around the query and directly tags the
active span (to catch `prisma:client:operation`, which Prisma creates
before the middleware fires).
### Context-propagation gotcha
`PrismaPromise` is lazy — `query(args)` returns a thenable that only
starts when someone `.then()`s it. The naive `context.with(ctx, () =>
query(args))` restores ALS synchronously, so when Prisma's internal code
awaits the thenable later, the engine spans fire with the original ALS.
Wrapping as `async () => await query(args)` forces the `.then()` inside
the `context.with` callback, so ALS stays on our context for the engine
spans.
### Coverage
- **Tagged**: all `prisma:engine:*` (`connection`, `db_query`,
`serialize`, `query`, etc.), `prisma:client:operation`,
`prisma:client:serialize`, `prisma:client:connect`
- **Not tagged**: `prisma:client:load_engine` — one-time startup, fires
before any query
Concurrent `Promise.all([writer.x, replica.y])` correctly tags each pool
separately (ALS isolates per-Promise chain).
### Performance
One `context.with` (~200ns) and one `setAttribute` per span (effectively
free per OTel JS benchmarks) per Prisma op. Negligible against a query
path measured in milliseconds.
## Test plan
- [ ] Verify `db.datasource` appears on `prisma:engine:connection` spans
after the webapp is restarted
- [ ] Spot-check a handful of real traces carry the attribute
|
||
|
nicktrn
|
b570586899 |
fix(webapp): allow cancelling runs in DEQUEUED status from the runs list (#3421)
The cancel button was missing from the runs list for runs in `DEQUEUED` status. The runs list gates the button on `run.isCancellable`, which goes through `isCancellableRunStatus` -> `CANCELLABLE_RUN_STATUSES` = `NON_FINAL_RUN_STATUSES`. `DEQUEUED` was never added to that list when it was introduced in the run engine. The single run page uses a separate check (`!run.isFinished`, i.e. the inverse of `FINAL_RUN_STATUSES`), so cancellation already worked there - only the list was affected. Adding `DEQUEUED` to `NON_FINAL_RUN_STATUSES` also flips `isCrashableRunStatus` and `isFailableRunStatus`, but: - The crash path is the right behaviour - a `DEQUEUED` run (worker has claimed but not yet executing) can legitimately crash before `EXECUTING`, same as `PENDING`/`DELAYED` already do. - The fail path (`failedTaskRun.server.ts`) is only reached from V1 code paths (marqs consumers, v1 heartbeat handler). `DEQUEUED` is a V2-engine-only status, so V1 consumers never see it. When cancelling a `DEQUEUED` run the execution snapshot goes to `PENDING_CANCEL` (worker must ack) but `TaskRun.status` flips to `CANCELED` immediately - the UI reflects cancellation without waiting for the worker. Added an integration test in `run-engine/src/engine/tests/cancelling.test.ts` covering the full trigger -> dequeue -> cancel -> worker-ack flow. ## Stall safety The stall recovery path (PENDING_EXECUTING heartbeat miss -> nack-and-requeue -> back to QUEUED) lives entirely inside `@internal/run-engine` and never touches the webapp's `taskStatus.ts` helpers - the engine has zero imports from `~/v3/taskStatus` and doesn't know `CrashTaskRunService` / `FailedTaskRunService` exist. A stalled DEQUEUED run still goes back to the queue for retry; this change cannot cause stalls to crash or fail. The only realistic impact is the intended UI fix - the theoretical V1 crash/fail branches for DEQUEUED are unreachable in practice because V1 runs never have DEQUEUED status. |
||
|
Eric Allam
|
03e4d5fe31 |
feat(webapp,database): API key rotation grace period (#3420)
## Summary
Regenerating a RuntimeEnvironment API key no longer immediately
invalidates the previous one. Rotation is now overlap-based: the old key
keeps working for 24 hours so customers can roll it out in their env
vars without downtime, then stops working.
## Design
- **New `RevokedApiKey` table** (one row per revocation). Holds the
archived `apiKey`, a FK to the env, an `expiresAt`, and a `createdAt`.
Indexed on `apiKey` (high-cardinality equality — single-row hits) and on
`runtimeEnvironmentId`.
- **`regenerateApiKey` wraps both writes in a single `$transaction`:**
insert a `RevokedApiKey` with `expiresAt = now + 24h`, update the env
with the new `apiKey`/`pkApiKey`.
- **`findEnvironmentByApiKey` does a two-step lookup:** primary
unique-index hit on `RuntimeEnvironment.apiKey` first; on miss,
`RevokedApiKey.findFirst({ apiKey, expiresAt: { gt: now } })` with an
`include: { runtimeEnvironment }`. Two-step (not `OR`-join) keeps the
hot path identical to today and puts the fallback cost only on invalid
keys. Both lookups use `$replica`.
- **Admin endpoint** `POST /admin/api/v1/revoked-api-keys/:id` accepts
`{ expiresAt }` and updates the row. Setting to `now` ends the grace
window immediately; setting to the future extends it.
- **Modal copy** on the regenerate dialog updated — previously warned of
downtime, now explains the 24h overlap.
## Why a separate table instead of columns on `RuntimeEnvironment`
- Keeps the hot auth path's primary lookup unchanged — no
OR/nullable-apiKey semantics to reason about.
- Naturally supports multiple in-flight grace windows (regenerate twice
in a day → two old keys valid until their independent expiries).
- FK + cascade cleans up correctly when an env is deleted; nothing to
backfill.
## Test plan
Verified locally against hello-world with dev and prod env keys:
- [x] baseline — current key authenticates (`GET /api/v1/runs`) → `200`
- [x] regenerate via UI — DB shows old key in `RevokedApiKey` with
`expiresAt ≈ now+24h`, env has new key
- [x] grace window — both old and new keys → `200`; bogus key → `401`
- [x] admin endpoint: `expiresAt = now` → old key `401`
- [x] admin endpoint: `expiresAt = +1h` (after early-expire) → old key
`200` again
- [x] admin endpoint: `expiresAt = past` → old key `401`
- [x] admin 400 (invalid body), 404 (unknown id), 401 (missing/non-admin
PAT)
- [x] same flow exercised end-to-end on a PROD-typed env — behavior
identical
- [x] `pnpm run typecheck --filter webapp` passes
|
||
|
Eric Allam
|
881288c615 |
feat(webapp): deprecate v3 CLI deploys server-side (#3415)
## ✅ Checklist - [x] I have followed every step in the [contributing guide](https://github.com/triggerdotdev/trigger.dev/blob/main/CONTRIBUTING.md) - [x] The PR title follows the convention. - [x] I ran and tested the code works --- ## Summary Adds a server-side gate that detects deploy attempts from v3 CLI versions (i.e. `trigger.dev@3.x`) at the `POST /api/v1/deployments` entry point and, when enabled, rejects them with a clear upgrade message. v4 CLI deploys are completely unaffected. The last 3.x CLI release was `3.3.7`, which we can't update. This approach short-circuits the deploy before any DB writes, image-ref generation, S2 stream creation, or queue enqueue — no side effects in either mode. ## How v3 vs v4 are distinguished I pulled the published CLI tarballs for `trigger.dev@3.3.7`, `4.0.0`, `4.0.1`, `4.0.5`, `4.1.0`, `4.2.0`, and the current `4.4.4` in the repo. The cleanest, most reliable signal is the request body to `POST /api/v1/deployments`: | Field on initialize | v3.3.7 CLI | v4.x CLI | |---|---|---| | `type` | **never sent** | always sent — `"MANAGED"` (run_engine_v2) or `"V1"` | | `isNativeBuild` / `gitMeta` / `triggeredVia` / `runtime` | not sent | sent | | `registryHost` / `namespace` | sent (v3-only; stripped by current Zod schema) | not sent | Every v4 call site I inspected sets `type: features.run_engine_v2 ? "MANAGED" : "V1"` unconditionally. `payload.type` is `undefined` if and only if the client is a 3.x CLI. ## Behavior - Detection always runs and emits `logger.warn("Detected deploy from deprecated v3 CLI", { environmentId, projectId, organizationId, enforced })`, which lets us watch how many v3 deploys are still happening before enforcement is flipped. - Enforcement is gated behind `DEPRECATE_V3_CLI_DEPLOYS_ENABLED` (default `"0"`, off). When `"1"`, the server returns `400` with: > The trigger.dev CLI v3 is no longer supported for deployments. Please upgrade your project to v4: https://trigger.dev/docs/migrating-from-v3 The v3 CLI surfaces this verbatim as `Failed to start deployment: <message>` because `zodfetch` throws `ApiError` for non-retryable 4xx (400/422) and `deploy.js` in 3.3.7 prints `error.message`. ## Out of scope (intentionally) - `api.v1.deployments.$deploymentId.finalize.ts` / `FinalizeDeploymentService` / `createDeploymentBackgroundWorkerV3.server.ts` are V1-engine paths, not the v3 CLI gate. Leaving them alone per review. - Container-side `createDeploymentBackgroundWorker` call in `managed-index-controller.ts` is still used by v4's in-image indexer. Not touched. - v3 `trigger dev` flow (different code path) — separate deprecation if/when needed. ## Testing - Ran `pnpm run typecheck --filter webapp` locally — passes. - Verified v4 tarballs (4.0.0, 4.0.1, 4.0.5, 4.1.0, 4.2.0, 4.4.4) all include `type:` in the `initializeDeployment` call site, so none will be accidentally blocked. - Verified v3.3.7 tarball's `initializeDeployment` payload has no `type` field. Rollout plan after merge: 1. Deploy with `DEPRECATE_V3_CLI_DEPLOYS_ENABLED` unset → watch `Detected deploy from deprecated v3 CLI` log volume. 2. When comfortable, set `DEPRECATE_V3_CLI_DEPLOYS_ENABLED=1` to enforce. --- ## Changelog Detect v3 CLI deploys on `/api/v1/deployments` and, when `DEPRECATE_V3_CLI_DEPLOYS_ENABLED=1`, reject them with an upgrade message pointing at https://trigger.dev/docs/migrating-from-v3. v4 CLI deploys are unaffected. Link to Devin session: https://app.devin.ai/sessions/b242c11bd86e4099aeec8b59bab62143 Requested by: @ericallam |
||
|
Iss
|
7d7ebdde52 | feat: Increase default project limit per org from 10 to 25 (#3409) | ||
|
nicktrn
|
9a988ab885 |
chore(webapp): clarify admin feature flags are global (#3408)
global flags are global. |
||
|
nicktrn
|
581db83f64 |
feat(webapp): highlight microVM regions on the regions page (#3407)
Adds a `MicroVM` badge next to the region name on the regions page. Uses the existing `small` badge variant for visual consistency with the `Default` badge already on this page. |
||
|
devin-ai-integration[bot]
|
ff290dfe2f |
perf(run-engine): merge dequeue snapshot creation into taskRun.update transaction [TRI-8450] (#3395)
## Summary
Nests the `TaskRunExecutionSnapshot` creation inside the
`taskRun.update()` Prisma call in the dequeue flow, reducing **2 DB
commits → 1** per dequeue operation. This is the highest-volume of the
five unmerged flows identified in TRI-8450 (~9,200 commits/sec on the
engine service).
**Pattern**: Follows the same nested-write approach already used in the
completion path (`runAttemptSystem.ts:735`) and trigger path
(`engine/index.ts:674`).
**Changes**:
- `dequeueSystem.ts`: Moved snapshot creation into `executionSnapshots:
{ create: {...} }` within the existing `taskRun.update()`. Pre-generates
the snapshot ID via `generateInternalId()` (plain cuid, matching what
Prisma's `@default(cuid())` produces) so the event emission, heartbeat
enqueue, and return value can all be constructed from data already in
scope — **no extra DB read needed** after the merged write.
`SnapshotId.toFriendlyId()` is used only for the return value's
`friendlyId` field, matching the original `createExecutionSnapshot`
behavior.
- `executionSnapshotSystem.ts`: Added public
`enqueueHeartbeatIfNeeded()` method that exposes the heartbeat
scheduling logic (previously only available internally via
`createExecutionSnapshot`). This is needed because `PENDING_EXECUTING`
requires a heartbeat, unlike the `FINISHED` status in the completion
reference pattern. This method is reusable by future merge targets
(retry-immediate, checkpoint, cancel, requeue).
**Net DB change per dequeue**: eliminates 1 write transaction (the
separate `TaskRunExecutionSnapshot.create`). No extra reads added — the
snapshot ID is pre-generated and the `executionSnapshotCreated` event
payload is constructed inline from values already available in the
closure.
## Review & Testing Checklist for Human
- [ ] **Verify manually-constructed event payload matches DB state**:
The `executionSnapshotCreated` event is now built inline (not read back
from DB). Confirm the field values (`runStatus: "PENDING"`,
`attemptNumber`, `checkpointId`, `workerId`, `runnerId`,
`completedWaitpointIds`) match what Prisma actually writes. A mismatch
here would be silent — event consumers would get stale/wrong data.
- [ ] **Verify `attemptNumber` source is equivalent**: Old code used
`lockedTaskRun.attemptNumber` (post-update result). New code uses
`result.run.attemptNumber` (pre-update). The `taskRun.update()` data
payload does NOT include `attemptNumber`, so they should be identical —
but confirm this assumption holds for all dequeue scenarios (e.g.
retried runs).
- [ ] **Verify `isValid` defaults to `true` in schema**: The old
`createExecutionSnapshot` explicitly set `isValid: error ? false :
true`. The nested create omits `isValid` (no error in the dequeue happy
path). Confirm the Prisma schema default for
`TaskRunExecutionSnapshot.isValid` is `true`.
- [ ] **Verify `runStatus: "PENDING"` hardcoding matches the mapping**:
The old code passed `lockedTaskRun.status` ("DEQUEUED") to
`createExecutionSnapshot`, which mapped it to "PENDING" via `run.status
=== "DEQUEUED" ? "PENDING" : run.status`. The new code hardcodes
`"PENDING"` directly. This is correct but brittle if `status` ever
changes from "DEQUEUED" to something else upstream.
- [ ] **Spot-check `completedWaitpoints` connect + order logic**: The
nested create replicates the connect/order logic from
`createExecutionSnapshot` (lines 387-393). Verify the
`snapshot.completedWaitpoints` type provides `id` and `index` fields
compatible with this usage.
- [ ] **Verify `checkpoint` in return value**: The return now uses
`snapshot.checkpoint` (from the *previous* snapshot) instead of reading
the newly-created snapshot's checkpoint relation. Since `checkpointId`
is passed through unchanged, they should be identical — but worth a
sanity check.
**Recommended test plan**: deploy to staging, run the
`sample_pg_activity.py` sampler for a 5-minute window, and verify the
COMMIT count drop on the engine service + proportional `IO:XactSync`
reduction.
### Notes
- This only covers the **dequeue** flow (flow #1 from TRI-8450). The
remaining four flows (retry-immediate, checkpoint, requeue, cancel) are
separate follow-ups.
- The new `enqueueHeartbeatIfNeeded` method is deliberately designed for
reuse by those follow-up PRs.
- CI note: the `priority.test.ts` failure in shard 7 is a flaky ordering
assertion unrelated to this change (it compares `friendlyId` values in
dequeue order). The `audit` check is also pre-existing/unrelated.
Link to Devin session:
https://app.devin.ai/sessions/034fe0e7224f49278a2de260203e1377
Requested by: @ericallam
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: Eric Allam <eallam@icloud.com>
|
||
|
Eric Allam
|
67d2025f33 |
feat(webapp): add 60s/60s SWR cache to getEntitlement (#3388)
Wraps getEntitlement in platform.v3.server.ts with the existing
platformCache (LRU memory + Redis) under a new `entitlement` namespace.
Eliminates a synchronous billing-service HTTP round trip on every
trigger.
Cache config: 60s fresh / 60s stale SWR. Cache key is the
organization id. Errors are caught inside the loader and return the
existing permissive { hasAccess: true } fallback, which is also
cached to prevent thundering-herd on billing outages.
Trade-off: plan upgrade/downgrade is now visible after up to ~120s
worst-case (60s fresh + 60s stale revalidation). Acceptable since
the existing limits and usage namespaces use 5min/10min, and the
defensive hasAccess: true fallback already exists.
|
||
|
Eric Allam
|
79b6053e13 |
feat(server): add TaskIdentifier registry to replace expensive distinct query (#3368)
Replace the expensive DISTINCT query for task filter dropdowns with a dedicated TaskIdentifier registry table backed by Redis. Environments migrate automatically on their next deploy, with a transparent fallback to the legacy query for unmigrated environments. Also fixes duplicate dropdown entries when a task changes trigger source, and adds active/archived grouping for removed tasks. Moves BackgroundWorkerTask reads in the trigger hot path to the read replica. |
||
|
Eric Allam
|
94abe97132 | fix(webapp): prevent dashboard crash when span accessory text is not a string (#3400) | ||
|
Eric Allam
|
02d2334c8a |
fix(webapp): fix Redis connection leak in realtime streams and broken abort signal propagation (#3399)
Pool Redis connections for non-blocking ops (ingestData, appendPart,
getLastChunkIndex)
using a shared singleton instead of new Redis() per request. Use
redis.disconnect()
for immediate teardown in streamResponse cleanup. Add 15s inactivity
timeout fallback.
Fix broken request.signal in Remix/Express by wiring Express
res.on('close') to an
AbortController via httpAsyncStorage. All SSE/streaming routes now use
getRequestAbortSignal() which fires reliably on client disconnect,
bypassing the
Node.js undici GC bug (nodejs/node#55428) that severs the signal chain.
|
||
|
nicktrn
|
93f2ca6bf4 |
feat(webapp): extend admin workers endpoint and unify admin api auth (#3390)
Extends the admin worker groups endpoint with a GET loader and more fields on POST (type, hidden, workloadType, cloudProvider, location, staticIPs, enableFastPath), and pulls the PAT + admin check that was inlined or locally duplicated across every admin.api route into a shared helper in personalAccessToken.server.ts. The generic authenticateAdminRequest returns a discriminated result; requireAdminApiRequest is the thin Remix loader/action wrapper that throws. The neverthrow-style route (platform-notifications.ts) now composes the generic helper instead of duplicating the check. Verified locally against GET (listing) and POST (new fields, invalid enum, minimal backwards-compat). |
||
|
devin-ai-integration[bot]
|
7d82041809 |
fix(security): upgrade Remix packages 2.1.0 → 2.17.4 (#3372)
## Summary
Upgrades all `@remix-run/*` packages in `apps/webapp` from **2.1.0 →
2.17.4** to address security vulnerabilities. Recreation of #2951 on a
fresh checkout of `main`.
**Updated packages (`apps/webapp/package.json`):**
- `@remix-run/express`, `@remix-run/node`, `@remix-run/react`,
`@remix-run/serve`, `@remix-run/server-runtime`: 2.1.0 → 2.17.4
- `@remix-run/router`: ^1.15.3 → ^1.23.2
- `@remix-run/dev`, `@remix-run/eslint-config`, `@remix-run/testing`:
2.1.0 → 2.17.4
**Root `package.json` overrides:**
- `@remix-run/dev@2.17.4>tar-fs`: 2.1.3 → 2.1.4
- `testcontainers@10.28.0>tar-fs`: 3.0.9 → 3.1.1
**Documentation:** Updated Remix version references in `CLAUDE.md`,
`apps/webapp/CLAUDE.md`, and `.cursor/rules/webapp.mdc`.
**Server changes:** Added `.server-changes/upgrade-remix-security.md`
for release tracking per `CONTRIBUTING.md`.
No application code changes — only `package.json` files, documentation,
a server-changes entry, and the regenerated `pnpm-lock.yaml`.
### Updates since last revision
Addressed all 3 Devin Review findings:
1. **Missing `.server-changes/` file** — added
`.server-changes/upgrade-remix-security.md` (commit
|
||
|
Oskar Otwinowski
|
097fab0d8c |
feat(webapp): Vercel integration - disable auto promotions (#3376)
<img width="1044" height="709" alt="image" src="https://github.com/user-attachments/assets/7f2cf25c-3b74-46a9-8794-41be077e04bf" /> |
||
|
Eric Allam
|
ed0c3e4f38 |
fix: stop creating TaskRunTag records and join table entries during triggering (#3369)
The TaskRun.runTags string array already stores tag names, making the TaskRunTag M2M relation redundant write overhead. Remove createTags calls, connect: tags, and join table writes from both V1 and V2 trigger paths. Simplify the add-tags API to just push to runTags directly. |
||
|
Eric Allam
|
417ab876e3 | fix(batch-queue): Batch items that hit the environment queue size limit now fast-fail (#3352) | ||
|
github-actions[bot]
|
5ea36e08f2 |
chore: release v4.4.4 (#3228)
🚀 Publish Trigger.dev Docker / typecheck (push) Failing after 0s
🚀 Publish Trigger.dev Docker / publish-webapp (push) Has been skipped
🚀 Publish Trigger.dev Docker / publish-worker (push) Has been skipped
🚀 Publish Trigger.dev Docker / publish-worker-v4 (push) Has been skipped
🚀 Publish Trigger.dev Docker / units (push) Failing after 0s
## Summary 12 new features, 59 improvements, 17 bug fixes. ## Highlights - Add support for setting TTL (time-to-live) defaults at the task level and globally in trigger.config.ts, with per-trigger overrides still taking precedence ([#3196](https://github.com/triggerdotdev/trigger.dev/pull/3196)) - Large run outputs can use the new API which allows switching object storage providers. ([#3275](https://github.com/triggerdotdev/trigger.dev/pull/3275)) ## Improvements - Add platform notifications support to the CLI. The `trigger dev` and `trigger login` commands now fetch and display platform notifications (info, warn, error, success) from the server. Includes discovery-based filtering to conditionally show notifications based on project file patterns, color markup rendering for styled terminal output, and a non-blocking display flow with a spinner fallback for slow fetches. Use `--skip-platform-notifications` flag with `trigger dev` to disable the notification check. ([#3254](https://github.com/triggerdotdev/trigger.dev/pull/3254)) - Add `get_span_details` MCP tool for inspecting individual spans within a run trace. ([#3255](https://github.com/triggerdotdev/trigger.dev/pull/3255)) - New `get_span_details` tool returns full span attributes, timing, events, and AI enrichment (model, tokens, cost, speed) - Span IDs now shown in `get_run_details` trace output for easy discovery - New API endpoint `GET /api/v1/runs/:runId/spans/:spanId` - New `retrieveSpan()` method on the API client - `get_query_schema` — discover available TRQL tables and columns - `query` — execute TRQL queries against your data - `list_dashboards` — list built-in dashboards and their widgets - `run_dashboard_query` — execute a single dashboard widget query - `whoami` — show current profile, user, and API URL - `list_profiles` — list all configured CLI profiles - `switch_profile` — switch active profile for the MCP session - `start_dev_server` — start `trigger dev` in the background and stream output - `stop_dev_server` — stop the running dev server - `dev_server_status` — check dev server status and view recent logs - `GET /api/v1/query/schema` — query table schema discovery - `GET /api/v1/query/dashboards` — list built-in dashboards - `--readonly` flag hides write tools (`deploy`, `trigger_task`, `cancel_run`) so the AI cannot make changes - `read:query` JWT scope for query endpoint authorization - `get_run_details` trace output is now paginated with cursor support - MCP tool annotations (`readOnlyHint`, `destructiveHint`) for all tools - `get_query_schema` now requires a table name and returns only one table's schema (was returning all tables) - `get_current_worker` no longer inlines payload schemas; use new `get_task_schema` tool instead - Query results formatted as text tables instead of JSON (~50% fewer tokens) - `cancel_run`, `list_deploys`, `list_preview_branches` formatted as text instead of raw JSON - Schema and dashboard API responses cached to avoid redundant fetches - Adapted the CLI API client to propagate the trigger source via http headers. ([#3241](https://github.com/triggerdotdev/trigger.dev/pull/3241)) - Propagate run tags to span attributes so they can be extracted server-side for LLM cost attribution metadata. ([#3213](https://github.com/triggerdotdev/trigger.dev/pull/3213)) - New `get_span_details` tool returns full span attributes, timing, events, and AI enrichment (model, tokens, cost, speed) - Span IDs now shown in `get_run_details` trace output for easy discovery - New API endpoint `GET /api/v1/runs/:runId/spans/:spanId` - New `retrieveSpan()` method on the API client - `get_query_schema` — discover available TRQL tables and columns - `query` — execute TRQL queries against your data - `list_dashboards` — list built-in dashboards and their widgets - `run_dashboard_query` — execute a single dashboard widget query - `whoami` — show current profile, user, and API URL - `list_profiles` — list all configured CLI profiles - `switch_profile` — switch active profile for the MCP session - `start_dev_server` — start `trigger dev` in the background and stream output - `stop_dev_server` — stop the running dev server - `dev_server_status` — check dev server status and view recent logs - `GET /api/v1/query/schema` — query table schema discovery - `GET /api/v1/query/dashboards` — list built-in dashboards - `--readonly` flag hides write tools (`deploy`, `trigger_task`, `cancel_run`) so the AI cannot make changes - `read:query` JWT scope for query endpoint authorization - `get_run_details` trace output is now paginated with cursor support - MCP tool annotations (`readOnlyHint`, `destructiveHint`) for all tools - `get_query_schema` now requires a table name and returns only one table's schema (was returning all tables) - `get_current_worker` no longer inlines payload schemas; use new `get_task_schema` tool instead - Query results formatted as text tables instead of JSON (~50% fewer tokens) - `cancel_run`, `list_deploys`, `list_preview_branches` formatted as text instead of raw JSON - Schema and dashboard API responses cached to avoid redundant fetches - Add optional `hasPrivateLink` field to the dequeue message organization object for private networking support ([#3264](https://github.com/triggerdotdev/trigger.dev/pull/3264)) - Define and manage AI prompts with `prompts.define()`. Create typesafe prompt templates with variables, resolve them at runtime, and manage versions and overrides from the dashboard without redeploying. ([#3244](https://github.com/triggerdotdev/trigger.dev/pull/3244)) ## Bug fixes - Fix dev CLI leaking build directories on rebuild, causing disk space accumulation. Deprecated workers are now pruned (capped at 2 retained) when no active runs reference them. The watchdog process also cleans up `.trigger/tmp/` when the dev CLI is killed ungracefully (e.g. SIGKILL from pnpm). ([#3224](https://github.com/triggerdotdev/trigger.dev/pull/3224)) - Fix `--load` flag being silently ignored on local/self-hosted builds. ([#3114](https://github.com/triggerdotdev/trigger.dev/pull/3114)) - Fixed `search_docs` tool failing due to renamed upstream Mintlify tool (`SearchTriggerDev` → `search_trigger_dev`) - Fixed `list_deploys` failing when deployments have null `runtime`/`runtimeVersion` fields (#3139) - Fixed `list_preview_branches` crashing due to incorrect response shape access - Fixed `metrics` table column documented as `value` instead of `metric_value` in query docs - Fixed dev CLI leaking build directories on rebuild — deprecated workers now clean up their build dirs when their last run completes - Fixed `search_docs` tool failing due to renamed upstream Mintlify tool (`SearchTriggerDev` → `search_trigger_dev`) - Fixed `list_deploys` failing when deployments have null `runtime`/`runtimeVersion` fields (#3139) - Fixed `list_preview_branches` crashing due to incorrect response shape access - Fixed `metrics` table column documented as `value` instead of `metric_value` in query docs - Fixed dev CLI leaking build directories on rebuild — deprecated workers now clean up their build dirs when their last run completes ## Server changes These changes affect the self-hosted Docker image and Trigger.dev Cloud: - Add admin UI for viewing and editing feature flags (org-level overrides and global defaults). ([#3291](https://github.com/triggerdotdev/trigger.dev/pull/3291)) - AI prompt management dashboard and enhanced span inspectors. **Prompt management:** - Prompts list page with version status, model, override indicators, and 24h usage sparklines - Prompt detail page with template viewer, variable preview, version history timeline, and override editor - Create, edit, and remove overrides to change prompt content or model without redeploying - Promote any code-deployed version to current - Generations tab with infinite scroll, live polling, and inline span inspector - Per-prompt metrics: total generations, avg tokens, avg cost, latency, with version-level breakdowns **AI span inspectors:** - Custom inspectors for `ai.generateText`, `ai.streamText`, `ai.generateObject`, `ai.streamObject` parent spans - `ai.toolCall` inspector showing tool name, call ID, and input arguments - `ai.embed` inspector showing model, provider, and input text - Prompt tab on AI spans linking to prompt version with template and input variables - Compact timestamp and duration header on all AI span inspectors **AI metrics dashboard:** - Operations, Providers, and Prompts filters on the AI Metrics dashboard - Cost by prompt widget - "AI" section in the sidebar with Prompts and AI Metrics links **Other improvements:** - Resizable panel sizes now persist across page refreshes - Fixed `<div>` inside `<p>` DOM nesting warnings in span titles and chat messages ([#3244](https://github.com/triggerdotdev/trigger.dev/pull/3244)) - Add allowRollbacks query param to the promote deployment API to enable version downgrades ([#3214](https://github.com/triggerdotdev/trigger.dev/pull/3214)) - Pre-warm compute templates on deploy for orgs with compute access. Required for projects using a compute region, background-only for others. ([#3114](https://github.com/triggerdotdev/trigger.dev/pull/3114)) - Add automatic LLM cost calculation for spans with GenAI semantic conventions. When a span arrives with `gen_ai.response.model` and token usage data, costs are calculated from an in-memory pricing registry backed by Postgres and dual-written to both span attributes (`trigger.llm.*`) and a new `llm_metrics_v1` ClickHouse table that captures usage, cost, performance (TTFC, tokens/sec), and behavioral (finish reason, operation type) metrics. ([#3213](https://github.com/triggerdotdev/trigger.dev/pull/3213)) - Add API endpoint `GET /api/v1/runs/:runId/spans/:spanId` that returns detailed span information including properties, events, AI enrichment (model, tokens, cost), and triggered child runs. ([#3255](https://github.com/triggerdotdev/trigger.dev/pull/3255)) - Multi-provider object storage with protocol-based routing for zero-downtime migration ([#3275](https://github.com/triggerdotdev/trigger.dev/pull/3275)) - Add IAM role-based auth support for object stores (no access keys required). ([#3275](https://github.com/triggerdotdev/trigger.dev/pull/3275)) - Add platform notifications to inform users about new features, changelogs, and platform events directly in the dashboard. ([#3254](https://github.com/triggerdotdev/trigger.dev/pull/3254)) - Add private networking support via AWS PrivateLink. Includes BillingClient methods for managing private connections, org settings UI pages for connection management, and supervisor changes to apply `privatelink` pod labels for CiliumNetworkPolicy matching. ([#3264](https://github.com/triggerdotdev/trigger.dev/pull/3264)) - Reduce run start latency by skipping the intermediate queue when concurrency is available. This optimization is rolled out per-region and enabled automatically for development environments. ([#3299](https://github.com/triggerdotdev/trigger.dev/pull/3299)) - Extended the search filter on the environment variables page to match on environment type (production, staging, development, preview) and branch name, not just variable name and value. ([#3302](https://github.com/triggerdotdev/trigger.dev/pull/3302)) - Set `application_name` on Prisma connections from SERVICE_NAME so DB load can be attributed by service ([#3348](https://github.com/triggerdotdev/trigger.dev/pull/3348)) - Fix transient R2/object store upload failures during batchTrigger() item streaming. - Added p-retry (3 attempts, 500ms–2s exponential backoff) around `uploadPacketToObjectStore` in `BatchPayloadProcessor.process()` so transient network errors self-heal server-side rather than aborting the entire batch stream. - Removed `x-should-retry: false` from the 500 response on the batch items route so the SDK's existing 5xx retry path can recover if server-side retries are exhausted. Item deduplication by index makes full-stream retries safe. ([#3331](https://github.com/triggerdotdev/trigger.dev/pull/3331)) - Concurrency-keyed queues now use a single master queue entry per base queue instead of one entry per key. Prevents high-CK-count tenants from consuming the entire parentQueueLimit window and starving other tenants on the same shard. ([#3219](https://github.com/triggerdotdev/trigger.dev/pull/3219)) - Reduce lock contention when processing large `batchTriggerAndWait` batches. Previously, each batch item acquired a Redis lock on the parent run to insert a `TaskRunWaitpoint` row, causing `LockAcquisitionTimeoutError` with high concurrency (880 errors/24h in prod). Since `blockRunWithCreatedBatch` already transitions the parent to `EXECUTING_WITH_WAITPOINTS` before items are processed, the per-item lock is unnecessary. The new `blockRunWithWaitpointLockless` method performs only the idempotent CTE insert without acquiring the lock. ([#3232](https://github.com/triggerdotdev/trigger.dev/pull/3232)) - Strip `secure` query parameter from QUERY_CLICKHOUSE_URL before passing to ClickHouse client. This was already done for the main and logs ClickHouse clients but was missing for the query client, causing a startup crash with `Error: Unknown URL parameters: secure`. ([#3204](https://github.com/triggerdotdev/trigger.dev/pull/3204)) - Fix `OrganizationsPresenter.#getEnvironment` matching the wrong development environment on teams with multiple members. All dev environments share the slug `"dev"`, so the previous `find` by slug alone could return another member's environment. Now filters DEVELOPMENT environments by `orgMember.userId` to ensure the logged-in user's dev environment is selected. ([#3273](https://github.com/triggerdotdev/trigger.dev/pull/3273)) <details> <summary>Raw changeset output</summary> # Releases ## @trigger.dev/build@4.4.4 ### Patch Changes - Updated dependencies: - `@trigger.dev/core@4.4.4` ## trigger.dev@4.4.4 ### Patch Changes - Add platform notifications support to the CLI. The `trigger dev` and `trigger login` commands now fetch and display platform notifications (info, warn, error, success) from the server. Includes discovery-based filtering to conditionally show notifications based on project file patterns, color markup rendering for styled terminal output, and a non-blocking display flow with a spinner fallback for slow fetches. Use `--skip-platform-notifications` flag with `trigger dev` to disable the notification check. ([#3254](https://github.com/triggerdotdev/trigger.dev/pull/3254)) - Fix dev CLI leaking build directories on rebuild, causing disk space accumulation. Deprecated workers are now pruned (capped at 2 retained) when no active runs reference them. The watchdog process also cleans up `.trigger/tmp/` when the dev CLI is killed ungracefully (e.g. SIGKILL from pnpm). ([#3224](https://github.com/triggerdotdev/trigger.dev/pull/3224)) - Fix `--load` flag being silently ignored on local/self-hosted builds. ([#3114](https://github.com/triggerdotdev/trigger.dev/pull/3114)) - Add `get_span_details` MCP tool for inspecting individual spans within a run trace. ([#3255](https://github.com/triggerdotdev/trigger.dev/pull/3255)) - New `get_span_details` tool returns full span attributes, timing, events, and AI enrichment (model, tokens, cost, speed) - Span IDs now shown in `get_run_details` trace output for easy discovery - New API endpoint `GET /api/v1/runs/:runId/spans/:spanId` - New `retrieveSpan()` method on the API client - MCP server improvements: new tools, bug fixes, and new flags. ([#3224](https://github.com/triggerdotdev/trigger.dev/pull/3224)) **New tools:** - `get_query_schema` — discover available TRQL tables and columns - `query` — execute TRQL queries against your data - `list_dashboards` — list built-in dashboards and their widgets - `run_dashboard_query` — execute a single dashboard widget query - `whoami` — show current profile, user, and API URL - `list_profiles` — list all configured CLI profiles - `switch_profile` — switch active profile for the MCP session - `start_dev_server` — start `trigger dev` in the background and stream output - `stop_dev_server` — stop the running dev server - `dev_server_status` — check dev server status and view recent logs **New API endpoints:** - `GET /api/v1/query/schema` — query table schema discovery - `GET /api/v1/query/dashboards` — list built-in dashboards **New features:** - `--readonly` flag hides write tools (`deploy`, `trigger_task`, `cancel_run`) so the AI cannot make changes - `read:query` JWT scope for query endpoint authorization - `get_run_details` trace output is now paginated with cursor support - MCP tool annotations (`readOnlyHint`, `destructiveHint`) for all tools **Bug fixes:** - Fixed `search_docs` tool failing due to renamed upstream Mintlify tool (`SearchTriggerDev` → `search_trigger_dev`) - Fixed `list_deploys` failing when deployments have null `runtime`/`runtimeVersion` fields (#3139) - Fixed `list_preview_branches` crashing due to incorrect response shape access - Fixed `metrics` table column documented as `value` instead of `metric_value` in query docs - Fixed dev CLI leaking build directories on rebuild — deprecated workers now clean up their build dirs when their last run completes **Context optimizations:** - `get_query_schema` now requires a table name and returns only one table's schema (was returning all tables) - `get_current_worker` no longer inlines payload schemas; use new `get_task_schema` tool instead - Query results formatted as text tables instead of JSON (~50% fewer tokens) - `cancel_run`, `list_deploys`, `list_preview_branches` formatted as text instead of raw JSON - Schema and dashboard API responses cached to avoid redundant fetches - Add support for setting TTL (time-to-live) defaults at the task level and globally in trigger.config.ts, with per-trigger overrides still taking precedence ([#3196](https://github.com/triggerdotdev/trigger.dev/pull/3196)) - Adapted the CLI API client to propagate the trigger source via http headers. ([#3241](https://github.com/triggerdotdev/trigger.dev/pull/3241)) - Updated dependencies: - `@trigger.dev/core@4.4.4` - `@trigger.dev/build@4.4.4` - `@trigger.dev/schema-to-json@4.4.4` ## @trigger.dev/core@4.4.4 ### Patch Changes - Fix `list_deploys` MCP tool failing when deployments have null `runtime` or `runtimeVersion` fields. ([#3224](https://github.com/triggerdotdev/trigger.dev/pull/3224)) - Propagate run tags to span attributes so they can be extracted server-side for LLM cost attribution metadata. ([#3213](https://github.com/triggerdotdev/trigger.dev/pull/3213)) - Add `get_span_details` MCP tool for inspecting individual spans within a run trace. ([#3255](https://github.com/triggerdotdev/trigger.dev/pull/3255)) - New `get_span_details` tool returns full span attributes, timing, events, and AI enrichment (model, tokens, cost, speed) - Span IDs now shown in `get_run_details` trace output for easy discovery - New API endpoint `GET /api/v1/runs/:runId/spans/:spanId` - New `retrieveSpan()` method on the API client - MCP server improvements: new tools, bug fixes, and new flags. ([#3224](https://github.com/triggerdotdev/trigger.dev/pull/3224)) **New tools:** - `get_query_schema` — discover available TRQL tables and columns - `query` — execute TRQL queries against your data - `list_dashboards` — list built-in dashboards and their widgets - `run_dashboard_query` — execute a single dashboard widget query - `whoami` — show current profile, user, and API URL - `list_profiles` — list all configured CLI profiles - `switch_profile` — switch active profile for the MCP session - `start_dev_server` — start `trigger dev` in the background and stream output - `stop_dev_server` — stop the running dev server - `dev_server_status` — check dev server status and view recent logs **New API endpoints:** - `GET /api/v1/query/schema` — query table schema discovery - `GET /api/v1/query/dashboards` — list built-in dashboards **New features:** - `--readonly` flag hides write tools (`deploy`, `trigger_task`, `cancel_run`) so the AI cannot make changes - `read:query` JWT scope for query endpoint authorization - `get_run_details` trace output is now paginated with cursor support - MCP tool annotations (`readOnlyHint`, `destructiveHint`) for all tools **Bug fixes:** - Fixed `search_docs` tool failing due to renamed upstream Mintlify tool (`SearchTriggerDev` → `search_trigger_dev`) - Fixed `list_deploys` failing when deployments have null `runtime`/`runtimeVersion` fields (#3139) - Fixed `list_preview_branches` crashing due to incorrect response shape access - Fixed `metrics` table column documented as `value` instead of `metric_value` in query docs - Fixed dev CLI leaking build directories on rebuild — deprecated workers now clean up their build dirs when their last run completes **Context optimizations:** - `get_query_schema` now requires a table name and returns only one table's schema (was returning all tables) - `get_current_worker` no longer inlines payload schemas; use new `get_task_schema` tool instead - Query results formatted as text tables instead of JSON (~50% fewer tokens) - `cancel_run`, `list_deploys`, `list_preview_branches` formatted as text instead of raw JSON - Schema and dashboard API responses cached to avoid redundant fetches - Large run outputs can use the new API which allows switching object storage providers. ([#3275](https://github.com/triggerdotdev/trigger.dev/pull/3275)) - Add optional `hasPrivateLink` field to the dequeue message organization object for private networking support ([#3264](https://github.com/triggerdotdev/trigger.dev/pull/3264)) - Add support for setting TTL (time-to-live) defaults at the task level and globally in trigger.config.ts, with per-trigger overrides still taking precedence ([#3196](https://github.com/triggerdotdev/trigger.dev/pull/3196)) - Adapted the CLI API client to propagate the trigger source via http headers. ([#3241](https://github.com/triggerdotdev/trigger.dev/pull/3241)) ## @trigger.dev/python@4.4.4 ### Patch Changes - Updated dependencies: - `@trigger.dev/sdk@4.4.4` - `@trigger.dev/core@4.4.4` - `@trigger.dev/build@4.4.4` ## @trigger.dev/react-hooks@4.4.4 ### Patch Changes - Updated dependencies: - `@trigger.dev/core@4.4.4` ## @trigger.dev/redis-worker@4.4.4 ### Patch Changes - Adapted the CLI API client to propagate the trigger source via http headers. ([#3241](https://github.com/triggerdotdev/trigger.dev/pull/3241)) - Updated dependencies: - `@trigger.dev/core@4.4.4` ## @trigger.dev/rsc@4.4.4 ### Patch Changes - Updated dependencies: - `@trigger.dev/core@4.4.4` ## @trigger.dev/schema-to-json@4.4.4 ### Patch Changes - Updated dependencies: - `@trigger.dev/core@4.4.4` ## @trigger.dev/sdk@4.4.4 ### Patch Changes - Define and manage AI prompts with `prompts.define()`. Create typesafe prompt templates with variables, resolve them at runtime, and manage versions and overrides from the dashboard without redeploying. ([#3244](https://github.com/triggerdotdev/trigger.dev/pull/3244)) - Add support for setting TTL (time-to-live) defaults at the task level and globally in trigger.config.ts, with per-trigger overrides still taking precedence ([#3196](https://github.com/triggerdotdev/trigger.dev/pull/3196)) - Adapted the CLI API client to propagate the trigger source via http headers. ([#3241](https://github.com/triggerdotdev/trigger.dev/pull/3241)) - Updated dependencies: - `@trigger.dev/core@4.4.4` </details> --------- Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> |
||
|
nicktrn
|
bd41bb2cbd |
feat(webapp): set application_name on prisma connections (#3348)
Sets `application_name` on the Prisma writer and replica connection strings using the existing `SERVICE_NAME` env var, so DB load can be attributed by service. |
||
|
Matt Aitken
|
def21b26b6 |
fix(batch): retry R2 upload on transient failure in BatchPayloadProcessor (#3331)
⚒️ Publish Worker (v4) / build (supervisor) (push) Has been cancelled
A single "fetch failed" from the object store was aborting the entire batch stream with no retry. Added p-retry (3 attempts, 500ms-2s backoff) around ploadPacketToObjectStore so transient network errors self-heal server-side instead of propagating to the SDK. |
||
|
Matt Aitken
|
68e88d0d71 |
Object Storage seamless migration (#3275)
This allows seamless migration to different object storage. Existing runs that have offloaded payloads/outputs will continue to use the default object store (configured using `OBJECT_STORE_*` env vars). You can add additional stores by setting new env vars: - `OBJECT_STORE_DEFAULT_PROTOCOL` this determines where new run large payloads will get stored. - If you set that you need to set new env vars for that protocol. Example: ``` OBJECT_STORE_DEFAULT_PROTOCOL=“s3" OBJECT_STORE_S3_BASE_URL=https://s3.us-east-1.amazonaws.com OBJECT_STORE_S3_ACCESS_KEY_ID=<val> OBJECT_STORE_S3_SECRET_ACCESS_KEY=<val> OBJECT_STORE_S3_REGION=us-east-1 OBJECT_STORE_S3_SERVICE=s3 ``` --------- Co-authored-by: nicktrn <55853254+nicktrn@users.noreply.github.com> |
||
|
Saadi Myftija
|
1307d97f7a |
fix(webapp): match environment when searching env variables (#3302)
Temporary workaround that enables filtering by environment in the envvars page, without changing any UI. --------- Co-authored-by: Claude <noreply@anthropic.com> |
||
|
Eric Allam
|
a3407287c9 |
feat(engine): enqueue fast path; skip the queue under certain conditions (#3299)
## Summary Currently, every triggered run follows a two-step path through Redis: 1. **Enqueue** — A Lua script atomically adds the message to a queue sorted set (ordered by priority-adjusted timestamp) 2. **Dequeue** — A debounced `processQueueForWorkerQueue` job fires ~500ms later, checks concurrency limits, removes the message from the sorted set, and pushes it to a worker queue (Redis list) where workers pick it up via `BLPOP` This means every run pays at least ~500ms of latency between being triggered and being available for a worker to execute, even when the queue is empty and concurrency is wide open. ### What changed The enqueue Lua scripts now atomically decide whether to **skip the queue sorted set entirely** and push directly to the worker queue. This happens inside the same Lua script that handles normal enqueue, so the decision is atomic with respect to concurrency bookkeeping. A run takes the **fast path** when all of these are true: - **Fast path is enabled** for this worker queue (gated per `WorkerInstanceGroup`) - **No available messages** in the queue (`ZRANGEBYSCORE` finds nothing with score ≤ now) — this respects priority ordering and allows fast path even when the queue has future-scored messages (e.g. nacked retries with delay) - **Environment concurrency** has capacity - **Queue concurrency** has capacity (including per-concurrency-key limits for CK queues) When the fast path is taken: - The message is stored and pushed directly to the worker queue (`RPUSH`) - Concurrency slots are claimed (`SADD` to the same sets used by the normal dequeue path) - The `processQueueForWorkerQueue` job is **not scheduled** (no work to do) - TTL sorted set is skipped (the `expireRun` worker job handles TTL independently) When any condition fails, the existing slow path runs unchanged. ### Rollout gating - **Development environments**: Fast path is always enabled - **Production environments**: Gated by a new `enableFastPath` boolean on `WorkerInstanceGroup` (defaults to `false`), allowing region-by-region rollout ### Rolling deploy safety Each process registers its own Lua scripts via `defineCommand` (identified by SHA hash). Old and new processes never share scripts. The Redis data structures are fully compatible in both directions — ack, nack, and release operations work identically regardless of which path a message took. ## Test plan - [x] Fast path taken when queue is empty and concurrency available - [x] Slow path when `enableFastPath` is false - [x] Slow path when queue has available messages (respects priority ordering) - [x] Fast path when queue only has future-scored messages - [x] Slow path when env concurrency is full - [x] Fast-path message can be acknowledged correctly - [x] Fast-path message can be nacked and re-enqueued to the queue sorted set - [x] Run all existing run-queue tests (ack, nack, CK, concurrency sweeper, dequeue) to verify no regressions - [x] Typecheck passes for run-engine and webapp |
||
|
nicktrn
|
5447cf4843 |
feat(webapp): admin UI for feature flag overrides (#3291)
Adds a dialog to the admin orgs page for viewing and editing per-org feature flag overrides. Flags are introspected from the catalog so the UI stays in sync with available flags automatically. Also adds a new tab for global flags. Refactors featureFlags.server.ts to split catalog definition (shared) from server-only runtime (flags(), makeSetMultipleFlags). The shared module exports flag metadata and validation so both the UI and API routes can use it without pulling in server dependencies. |
||
|
nicktrn
|
9cb3dcb07c |
feat(supervisor): compute workload manager (#3114)
Adds the `ComputeWorkloadManager` for routing task execution through the compute gateway, including full checkpoint/restore support, OTel trace integration, and template pre-warming. ## Changes **Compute workload manager** (`apps/supervisor/src/workloadManager/compute.ts`) - Routes instance create, snapshot, delete, and restore through the compute gateway API - Wide event logging on create with full timing and context - Configurable gateway timeout, auth token, image digest stripping **Compute snapshot service** (`apps/supervisor/src/services/computeSnapshotService.ts`) - Timer wheel for delayed snapshot dispatch (avoids wasted work on short-lived waitpoints) - Configurable dispatch concurrency limit (`COMPUTE_SNAPSHOT_DISPATCH_LIMIT`) - Snapshot-complete callback handler with suspend completion reporting - Trace context management and OTel span emission for snapshot operations **OTel trace service** (`apps/supervisor/src/services/otlpTraceService.ts`) - Fire-and-forget OTLP span emission for compute operations (provision, restore, snapshot) - BigInt nanosecond conversion preserving sub-ms precision for span ordering **Template creation** (`apps/webapp/app/v3/services/computeTemplateCreation.server.ts`) - Three-mode rollout: required (MICROVM projects), shadow (feature flag / percentage), skip - Integrated into deploy finalize flow **Shared compute package** (`internal-packages/compute/`) - Gateway client with namespace-based API (instances, templates, snapshots) - Zod schemas for all gateway request/response types **Database** - `COMPUTE` variant added to `TaskRunCheckpointType` enum - `WorkloadType` enum and column on `WorkerInstanceGroup` - `hasComputeAccess` feature flag **Env / config** - Compute gateway URL, auth token, timeout - Snapshot enable flag, delay, dispatch limit - Dedicated OTLP endpoint for compute spans (`COMPUTE_TRACE_OTLP_ENDPOINT`) |
||
|
Oskar Otwinowski
|
efe24f9c2a |
feat(private-link): Add private links UI (#3264)
⚒️ Publish Worker (v4) / build (supervisor) (push) Has been cancelled
|
||
|
Oskar Otwinowski
|
8244ac6f84 |
feat(Notifications): Platform notifications for CLI and Dashboard (#3254)
For human reviewer: - Check if Redis connection + code makes sense - Check CLI methods (it's on a hotpath) - Check DB Migrations and new tables ## ✅ Checklist - [x] I have followed every step in the [contributing guide](https://github.com/triggerdotdev/trigger.dev/blob/main/CONTRIBUTING.md) - [x] The PR title follows the convention. - [x] I ran and tested the code works --- ## Testing Spawning new CLI / Dashboard notifications, check MVP, check if failures not produce any problems with CLI/Dashboard --- ## Changelog Added notifications mechanism for Dashboard and CLI --- ## Screenshots 💯 |
||
|
Eric Allam
|
efcafdf388 |
fix(webapp): filter dev environments by userId in OrganizationsPresenter (#3273)
fix: filter dev environments by userId in OrganizationsPresenter |
||
|
Eric Allam
|
c00dae006b |
feat(mcp): add get_span_details tool (#3255)
Returns the fully detailed span with attributes and AI enrichment data |
||
|
Eric Allam
|
54d95ee4b9 |
feat: AI prompt management dashboard and enhanced span inspectors (#3244)
- Full prompt management UI: list, detail, override, and version
management for AI prompts defined with `prompts.define()`
- Rich AI span inspectors for all AI SDK operations with token usage,
messages, and prompt context
- Real-time generation tracking with live polling and filtering
## Prompt management
Define prompts in your code with `prompts.define()`, then manage
versions and overrides from the dashboard without redeploying:
```typescript
import { task, prompts } from "@trigger.dev/sdk";
import { generateText } from "ai";
import { openai } from "@ai-sdk/openai";
import { z } from "zod";
const supportPrompt = prompts.define({
id: "customer-support",
model: "gpt-4o",
variables: z.object({
customerName: z.string(),
plan: z.string(),
issue: z.string(),
}),
content: `You are a support agent for Acme SaaS.
Customer: {{customerName}} ({{plan}} plan)
Issue: {{issue}}
Respond with empathy and precision.`,
});
export const supportTask = task({
id: "handle-support",
run: async (payload) => {
const resolved = await supportPrompt.resolve({
customerName: payload.name,
plan: payload.plan,
issue: payload.issue,
});
const result = await generateText({
model: openai(resolved.model ?? "gpt-4o"),
system: resolved.text,
prompt: payload.issue,
...resolved.toAISDKTelemetry(),
});
return { response: result.text };
},
});
```
The prompts list page shows each prompt with its current version, model,
override status, and a usage sparkline over the last 24 hours.
From the prompt detail page you can:
- **Create overrides** to change the prompt template or model without
redeploying. Overrides take priority over the deployed version when
`prompt.resolve()` is called.
- **Promote** any code-deployed version to be the current version
- **Browse generations** across all versions with infinite scroll and
live polling for new results
- **Filter** by version, model, operation type, and provider
- **View metrics** (total generations, avg tokens, avg cost, latency)
broken down by version
## AI span inspectors
Every AI SDK operation now gets a custom inspector in the run trace
view:
- **`ai.generateText` / `ai.streamText`** — Shows model, token usage,
cost, the full message thread (system prompt, user message, assistant
response), and linked prompt details
- **`ai.generateObject` / `ai.streamObject`** — Same as above plus the
JSON schema and structured output
- **`ai.toolCall`** — Shows tool name, call ID, and input arguments
- **`ai.embed`** — Shows model and the text being embedded
For generation spans linked to a prompt, a "Prompt" tab shows the prompt
metadata, the input variables passed to `resolve()`, and the template
content from the prompt version.
All AI span inspectors include a compact timestamp and duration header.
## Other improvements
- Resizable panel sizes now persist across page refreshes (patched
`@window-splitter/state` to fix snapshot restoration)
- Run page panels also persist their sizes
- Fixed `<div>` inside `<p>` DOM nesting warnings in span titles and
chat messages
- Added Operations and Providers filters to the AI metrics dashboard
## Screenshots
<img width="3680" height="2392" alt="CleanShot 2026-03-21 at 10 14
17@2x"
src="https://github.com/user-attachments/assets/f3e59989-a2fa-4990-a9d0-3cacda431868"
/>
<img width="3680" height="2392" alt="CleanShot 2026-03-21 at 10 15
37@2x"
src="https://github.com/user-attachments/assets/2f2d02df-2d2b-44fb-ac6f-9153f6a6c387"
/>
<img width="3680" height="2392" alt="CleanShot 2026-03-21 at 10 15
54@2x"
src="https://github.com/user-attachments/assets/baa161e0-ef91-4fa4-a55f-986b71cccdf0"
/>
|
||
|
Eric Allam
|
1cfc296c6b |
feat(ai): LLM metrics tracking and AI span inspector (#3213)
- Automatic LLM cost enrichment for AI SDK spans (streamText, generateText, generateObject) or any other spans that use semantic gen_ai attributes with support for 145+ models - New AI span inspector sidebar showing model, tokens, cost, messages, tool calls, and response text - LLM metrics dual-write to ClickHouse `llm_metrics_v1` table for analytics - LLM metrics built-in dashboard (unlinked at the moment) - Provider cost fallback — uses gateway/OpenRouter reported costs from `providerMetadata` when registry pricing is unavailable - Prefix-stripping for gateway/OpenRouter model names (e.g. `mistral/mistral-large-3` matches `mistral-large-3` pricing) - Admin dashboard for managing LLM model pricing (list, create, edit, delete, search, test pattern matching) - Missing models detection page — queries ClickHouse for unpriced models with sample spans and Claude Code-ready prompts for adding pricing - AI span seed script (`pnpm run db:seed:ai-spans`) with 51 spans across 12 provider systems for local dev testing - UI fixes: `completionTokens`/`promptTokens` aliases, `ai.response.object` display for generateObject, cache read/write token breakdown ## Screenshots: <img width="1030" height="104" alt="CleanShot 2026-03-17 at 16 48 54@2x" src="https://github.com/user-attachments/assets/bc8fccda-e48b-4d0c-bfb1-e620064e5979" /> <img width="1094" height="1512" alt="CleanShot 2026-03-17 at 16 49 23@2x" src="https://github.com/user-attachments/assets/c2424569-d07e-4d67-a436-e8250043a1ee" /> <img width="1074" height="1412" alt="CleanShot 2026-03-17 at 16 49 18@2x" src="https://github.com/user-attachments/assets/22342ac4-4769-45d1-a328-a24fb9a82a50" /> <img width="1012" height="2292" alt="CleanShot 2026-03-17 at 16 39 01@2x" src="https://github.com/user-attachments/assets/59e327d1-6652-4293-8be0-bb8326e5fbc5" /> <img width="3680" height="2392" alt="CleanShot 2026-03-15 at 08 29 38@2x" src="https://github.com/user-attachments/assets/1f77beb8-de67-495b-b890-bcdb8d7f1fe8" /> --------- Co-authored-by: James Ritchie <james@trigger.dev> |
||
|
Eric Allam
|
411803e49e |
fix(engine): lockless waitpoint insert for batch items to eliminate lock contention (#3232)
When processing batchTriggerAndWait items, each batch item was acquiring a Redis lock on the parent run to insert a TaskRunWaitpoint row. With high concurrency (processingConcurrency=50), this caused LockAcquisitionTimeoutError (880 errors/24h in prod), orphaned runs, and stuck parent runs. Since blockRunWithCreatedBatch already transitions the parent to EXECUTING_WITH_WAITPOINTS before items are processed, the per-item lock is unnecessary. The new blockRunWithWaitpointLockless method performs only the idempotent CTE insert and timeout scheduling without acquiring the lock. |
||
|
Jack Cheng
|
f98e274b06 | feat(webapp): allow version downgrades via promote API (#3214) | ||
|
Eric Allam
|
7672e8d998 |
fix(run-queue): prevent concurrency keys from bloating master queue shards (#3219)
Queues with concurrency keys now appear as a single entry in the master queue instead of one entry per key. This prevents high-CK-count tenants from consuming the entire `parentQueueLimit` window and starving other tenants on the same shard. A new per-queue **CK index** (sorted set) tracks active concurrency key sub-queues. The master queue gets one `:ck:*` wildcard entry per base queue. Dequeuing from that entry round-robins across sub-queues, maintaining per-CK concurrency tracking and fairness. All existing operations (enqueue, dequeue, ack, nack, DLQ, TTL expiry) are CK-index-aware and keep the index consistent. Old-format entries drain naturally during rollout — no migration step needed, single deploy. |
||
|
Dinko Osrecki
|
b8766e7a27 | fix(webapp): strip secure param from query ClickHouse URL (#3204) | ||
|
github-actions[bot]
|
c0b6309c72 |
chore: release v4.4.3 (#3182)
🚀 Publish Trigger.dev Docker / units (push) Failing after 4s
🚀 Publish Trigger.dev Docker / typecheck (push) Failing after 20s
🚀 Publish Trigger.dev Docker / publish-webapp (push) Has been skipped
🚀 Publish Trigger.dev Docker / publish-worker (push) Has been skipped
🚀 Publish Trigger.dev Docker / publish-worker-v4 (push) Has been skipped
## Summary 2 new features, 2 improvements. ## Improvements - Add syncSupabaseEnvVars to pull database connection strings and save them as trigger.dev environment variables ([#3152](https://github.com/triggerdotdev/trigger.dev/pull/3152)) - Auto-cancel in-flight dev runs when the CLI exits, using a detached watchdog process that survives pnpm SIGKILL ([#3191](https://github.com/triggerdotdev/trigger.dev/pull/3191)) ## Server changes These changes affect the self-hosted Docker image and Trigger.dev Cloud: - A new Errors page for viewing and tracking errors that cause runs to fail - Errors are grouped using error fingerprinting - View top errors for a time period, filter by task, or search the text - View occurrences over time - View all the runs for an error and bulk replay them ([#3172](https://github.com/triggerdotdev/trigger.dev/pull/3172)) - Add sidebar tabs (Options, AI, Schema) to the Test page for schemaTask payload generation and schema viewing. ([#3188](https://github.com/triggerdotdev/trigger.dev/pull/3188)) <details> <summary>Raw changeset output</summary> # Releases ## @trigger.dev/build@4.4.3 ### Patch Changes - Add syncSupabaseEnvVars to pull database connection strings and save them as trigger.dev environment variables ([#3152](https://github.com/triggerdotdev/trigger.dev/pull/3152)) - Updated dependencies: - `@trigger.dev/core@4.4.3` ## trigger.dev@4.4.3 ### Patch Changes - Auto-cancel in-flight dev runs when the CLI exits, using a detached watchdog process that survives pnpm SIGKILL ([#3191](https://github.com/triggerdotdev/trigger.dev/pull/3191)) - Updated dependencies: - `@trigger.dev/core@4.4.3` - `@trigger.dev/build@4.4.3` - `@trigger.dev/schema-to-json@4.4.3` ## @trigger.dev/core@4.4.3 ### Patch Changes - Auto-cancel in-flight dev runs when the CLI exits, using a detached watchdog process that survives pnpm SIGKILL ([#3191](https://github.com/triggerdotdev/trigger.dev/pull/3191)) ## @trigger.dev/python@4.4.3 ### Patch Changes - Updated dependencies: - `@trigger.dev/core@4.4.3` - `@trigger.dev/build@4.4.3` - `@trigger.dev/sdk@4.4.3` ## @trigger.dev/react-hooks@4.4.3 ### Patch Changes - Updated dependencies: - `@trigger.dev/core@4.4.3` ## @trigger.dev/redis-worker@4.4.3 ### Patch Changes - Updated dependencies: - `@trigger.dev/core@4.4.3` ## @trigger.dev/rsc@4.4.3 ### Patch Changes - Updated dependencies: - `@trigger.dev/core@4.4.3` ## @trigger.dev/schema-to-json@4.4.3 ### Patch Changes - Updated dependencies: - `@trigger.dev/core@4.4.3` ## @trigger.dev/sdk@4.4.3 ### Patch Changes - Updated dependencies: - `@trigger.dev/core@4.4.3` </details> --------- Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> |
||
|
Eric Allam
|
436f20efc6 |
feat(cli): auto-cancel dev runs on CLI exit via detached watchdog (#3191)
When the dev CLI exits (e.g. ctrl+c via pnpm), runs that were mid-execution previously stayed stuck in EXECUTING status for up to 5 minutes until the heartbeat timeout fired. Now they are cancelled within seconds. The dev CLI spawns a lightweight detached watchdog process at startup. The watchdog monitors the CLI process ID and, when it detects the CLI has exited, calls a new POST /engine/v1/dev/disconnect endpoint to cancel all in-flight runs immediately (skipping PENDING_CANCEL since the worker is known to be dead). Watchdog design: - Fully detached (detached: true, stdio: ignore, unref()) so it survives even when pnpm sends SIGKILL to the process tree - Active run IDs maintained via atomic file write (.trigger/active-runs.json) - Single-instance guarantee via PID file (.trigger/watchdog.pid) - Safety timeout: exits after 24 hours to prevent zombie processes - On clean shutdown, the watchdog is killed (no disconnect needed) Disconnect endpoint: - Rate-limited: 5 calls/min per environment - Capped at 500 runs per call - Small counts (<= 25): cancelled inline with pMap concurrency 10 - Large counts: delegated to the bulk action system - Uses finalizeRun: true to skip PENDING_CANCEL and go straight to FINISHED Run engine change: - cancelRun() now respects finalizeRun when the run is in EXECUTING status, skipping the PENDING_CANCEL waiting state and going directly to FINISHED |
||
|
Eric Allam
|
e64b101138 |
feat(webapp): Add test payload AI generation to the test page based on payload schemas (#3188)
<img width="2191" height="1023" alt="CleanShot 2026-03-06 at 13 36 53" src="https://github.com/user-attachments/assets/4eba0d1a-1528-49a3-be5b-6bde89030193" /> <img width="411" height="1069" alt="CleanShot 2026-03-06 at 13 37 28" src="https://github.com/user-attachments/assets/e5f7bb9c-c894-41cc-9ca6-96b43fcf6005" /> Add a tabbed sidebar to the Test page for standard tasks, reusing the ClientTabs pattern from the Query page. - Options tab: existing sidebar content (machine, version, queue, etc.) - AI tab: AI-powered payload generation with streaming, supports JSON Schema, inferred schema from recent runs, and task source code lookup via tool calling for tasks without schemas - Schema tab: displays payload JSON Schema (from schemaTask), inferred schema (from recent runs via @jsonhero/schema-infer), or empty state with schemaTask docs and example code Data layer changes: - Surface payloadSchema and inferredPayloadSchema from TestTaskPresenter - Add payloadSchema and fileId to WorkerDeploymentWithWorkerTasks type - Decompress zlib-deflated source files for AI context |
||
|
Matt Aitken
|
5f359be286 |
feature: Errors page (#3172)
A top-level Errors page that aggregates errors from failed runs with occurrences metrics. https://github.com/user-attachments/assets/8f0ef55e-90dd-4faa-9051-59f4665181e4 Errors are “fingerprinted” so similar errors are grouped together (e.g. has an ID in the error message). You can view an individual error to view a timeline of when it fired, the runs, and bulk replay them. |
||
|
github-actions[bot]
|
e36b75785e |
chore: release v4.4.2 (#3127)
🚀 Publish Trigger.dev Docker / units (push) Failing after 4s
🚀 Publish Trigger.dev Docker / typecheck (push) Failing after 20s
🚀 Publish Trigger.dev Docker / publish-webapp (push) Has been skipped
🚀 Publish Trigger.dev Docker / publish-worker (push) Has been skipped
🚀 Publish Trigger.dev Docker / publish-worker-v4 (push) Has been skipped
# trigger.dev v4.4.2
## Summary
2 new features, 2 improvements, 8 bug fixes.
## Improvements
- Add input streams for bidirectional communication with running tasks.
Define typed input streams with `streams.input<T>({ id })`, then consume
inside tasks via `.wait()` (suspends the process), `.once()` (waits for
next message), or `.on()` (subscribes to a continuous stream). Send data
from backends with `.send(runId, data)` or from frontends with the new
`useInputStreamSend` React hook.
([#3146](https://github.com/triggerdotdev/trigger.dev/pull/3146))
- Add PAYLOAD_TOO_LARGE error to handle graceful recovery of sending
batch trigger items with payloads that exceed the maximum payload size
([#3137](https://github.com/triggerdotdev/trigger.dev/pull/3137))
## Bug fixes
- Fix slow batch queue processing by removing spurious cooloff on
concurrency blocks and fixing a race condition where retry attempt
counts were not atomically updated during message re-queue.
([#3079](https://github.com/triggerdotdev/trigger.dev/pull/3079))
- fix(sdk): batch triggerAndWait variants now return correct
run.taskIdentifier instead of unknown
([#3080](https://github.com/triggerdotdev/trigger.dev/pull/3080))
## Server changes
These changes affect the self-hosted Docker image and Trigger.dev Cloud:
- Two-level tenant dispatch architecture for batch queue processing.
Replaces the
single master queue with a two-level index: a dispatch index (tenant →
shard)
and per-tenant queue indexes (tenant → queues). This enables O(1) tenant
selection and fair scheduling across tenants regardless of queue count.
Improves batch queue processing performance.
([#3133](https://github.com/triggerdotdev/trigger.dev/pull/3133))
- Add input streams with API routes for sending data to running tasks,
SSE reading, and waitpoint creation. Includes Redis cache for fast
`.send()` to `.wait()` bridging, dashboard span support for input stream
operations, and s2-lite support with configurable S2 endpoint, access
token skipping, and S2-Basin headers for self-hosted deployments. Adds
s2-lite to Docker Compose for local development.
([#3146](https://github.com/triggerdotdev/trigger.dev/pull/3146))
- Speed up batch queue processing by disabling cooloff and increasing
the batch queue processing concurrency limits on the cloud:
- Pro plan: increase to 50 from 10.
- Hobby plan: increase to 10 from 5.
- Free plan: increase to 5 from 1.
([#3079](https://github.com/triggerdotdev/trigger.dev/pull/3079))
- Move batch queue global rate limiter from FairQueue claim phase to
BatchQueue worker queue consumer for accurate per-item rate limiting.
Add worker queue depth cap to prevent unbounded growth that could cause
visibility timeouts.
([#3166](https://github.com/triggerdotdev/trigger.dev/pull/3166))
- Fix a race condition in the waitpoint system where a run could be
blocked by a completed waitpoint but never be resumed because of a
PostgreSQL MVCC issue. This was most likely to occur when creating a
waitpoint via `wait.forToken()` at the same moment as completing the
token with `wait.completeToken()`. Other types of waitpoints (timed,
child runs) were not affected.
([#3075](https://github.com/triggerdotdev/trigger.dev/pull/3075))
- Fix metrics dashboard chart series colors going out of sync and
widgets not reloading stale data when scrolled back into view
([#3126](https://github.com/triggerdotdev/trigger.dev/pull/3126))
- Gracefully handle oversized batch items instead of aborting the
stream.
When an NDJSON batch item exceeds the maximum size, the parser now emits
an error marker instead of throwing, allowing the batch to seal
normally. The oversized item becomes a pre-failed run with
`PAYLOAD_TOO_LARGE` error code, while other items in the batch process
successfully. This prevents `batchTriggerAndWait` from seeing connection
errors and retrying with exponential backoff.
Also fixes the NDJSON parser not consuming the remainder of an oversized
line split across multiple chunks, which caused "Invalid JSON" errors on
subsequent lines.
([#3137](https://github.com/triggerdotdev/trigger.dev/pull/3137))
- Require the user is an admin during an impersonation session.
Previously only the impersonation cookie was checked; now the real
user's admin flag is verified on every request. If admin has been
revoked, the session falls back to the real user's ID.
([#3078](https://github.com/triggerdotdev/trigger.dev/pull/3078))
<details>
<summary>Raw changeset output</summary>
# Releases
## @trigger.dev/build@4.4.2
### Patch Changes
- Updated dependencies:
- `@trigger.dev/core@4.4.2`
## trigger.dev@4.4.2
### Patch Changes
- Updated dependencies:
- `@trigger.dev/build@4.4.2`
- `@trigger.dev/core@4.4.2`
- `@trigger.dev/schema-to-json@4.4.2`
## @trigger.dev/python@4.4.2
### Patch Changes
- Updated dependencies:
- `@trigger.dev/sdk@4.4.2`
- `@trigger.dev/build@4.4.2`
- `@trigger.dev/core@4.4.2`
## @trigger.dev/react-hooks@4.4.2
### Patch Changes
- Add input streams for bidirectional communication with running tasks.
Define typed input streams with `streams.input<T>({ id })`, then consume
inside tasks via `.wait()` (suspends the process), `.once()` (waits for
next message), or `.on()` (subscribes to a continuous stream). Send data
from backends with `.send(runId, data)` or from frontends with the new
`useInputStreamSend` React hook.
([#3146](https://github.com/triggerdotdev/trigger.dev/pull/3146))
Upgrade S2 SDK from 0.17 to 0.22 with support for custom endpoints
(s2-lite) via the new `endpoints` configuration, `AppendRecord.string()`
API, and `maxInflightBytes` session option.
- Updated dependencies:
- `@trigger.dev/core@4.4.2`
## @trigger.dev/redis-worker@4.4.2
### Patch Changes
- Fix slow batch queue processing by removing spurious cooloff on
concurrency blocks and fixing a race condition where retry attempt
counts were not atomically updated during message re-queue.
([#3079](https://github.com/triggerdotdev/trigger.dev/pull/3079))
- Updated dependencies:
- `@trigger.dev/core@4.4.2`
## @trigger.dev/rsc@4.4.2
### Patch Changes
- Updated dependencies:
- `@trigger.dev/core@4.4.2`
## @trigger.dev/schema-to-json@4.4.2
### Patch Changes
- Updated dependencies:
- `@trigger.dev/core@4.4.2`
## @trigger.dev/sdk@4.4.2
### Patch Changes
- Add input streams for bidirectional communication with running tasks.
Define typed input streams with `streams.input<T>({ id })`, then consume
inside tasks via `.wait()` (suspends the process), `.once()` (waits for
next message), or `.on()` (subscribes to a continuous stream). Send data
from backends with `.send(runId, data)` or from frontends with the new
`useInputStreamSend` React hook.
([#3146](https://github.com/triggerdotdev/trigger.dev/pull/3146))
Upgrade S2 SDK from 0.17 to 0.22 with support for custom endpoints
(s2-lite) via the new `endpoints` configuration, `AppendRecord.string()`
API, and `maxInflightBytes` session option.
- fix(sdk): batch triggerAndWait variants now return correct
run.taskIdentifier instead of unknown
([#3080](https://github.com/triggerdotdev/trigger.dev/pull/3080))
- Add PAYLOAD_TOO_LARGE error to handle graceful recovery of sending
batch trigger items with payloads that exceed the maximum payload size
([#3137](https://github.com/triggerdotdev/trigger.dev/pull/3137))
- Updated dependencies:
- `@trigger.dev/core@4.4.2`
## @trigger.dev/core@4.4.2
</details>
---------
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
|
||
|
Eric Allam
|
dee6f1d09e |
fix(batch): move batch queue global rate limiter to worker consumer level (#3166)
The global rate limiter was being applied at the FairQueue claim phase,
consuming 1 token per queue-claim-attempt rather than per item
processed.
With many small queues (each batch is its own queue), consumers burned
through tokens on empty or single-item queues, causing aggressive
throttling well below the intended items/sec limit.
Changes:
- Move rate limiter from FairQueue claim phase to BatchQueue worker
queue
consumer loop (before blockingPop), so each token = 1 item processed
- Replace the FairQueue rate limiter with a worker queue depth cap to
prevent unbounded growth that could cause visibility timeouts
- Add BATCH_QUEUE_WORKER_QUEUE_MAX_DEPTH env var (optional, disabled by
default)
|
||
|
Eric Allam
|
540e1c86a4 |
feat: Input Streams - Bidirectional task communication (#3146)
Input streams enable sending typed data to executing tasks from external
callers — backends, frontends, or other tasks. This unlocks interactive
use cases like approval UIs, cancel buttons, chat interfaces, and
human-in-the-loop AI workflows where the task needs to receive data
while running.
Three consumption patterns inside a task:
* `.wait()` — Suspend the task until data arrives (process freed, most
efficient)
* `.once()` — Wait for the next message (process stays alive)
* `.on()` — Subscribe to a continuous stream of messages
One send pattern from outside:
* `.send(runId, data)` — Send typed data to a specific run's input
stream
## User-facing API
### Define a typed input stream
```ts
import { streams, task } from "@trigger.dev/sdk";
const approval = streams.input<{ approved: boolean; reviewer: string }>({ id: "approval" });
```
### Consume inside a task
```ts
export const myTask = task({
id: "my-task",
run: async () => {
// Pattern 1: Suspend until data arrives (most efficient — frees the process)
const result = await approval.wait({ timeout: "5m" });
// Pattern 2: Wait for next message (process stays alive)
const data = await approval.once().unwrap();
// Pattern 3: Subscribe to multiple messages
approval.on((data) => { /* handle each message */ });
},
});
```
### Send from outside
```ts
// From a backend (using secret API key)
await approval.send(runId, { approved: true, reviewer: "alice" });
// From a frontend (using public JWT token from trigger response)
const { send } = useInputStreamSend("approval", runId, { accessToken });
send({ approved: true, reviewer: "alice" });
```
---------
Co-authored-by: Claude <noreply@anthropic.com>
|
||
|
Eric Allam
|
8003923598 |
feat(server): Gracefully handle oversized batch items instead of aborting the stream (#3137)
Gracefully handle oversized batch items instead of aborting the stream. When an NDJSON batch item exceeds the maximum size, the parser now emits an error marker instead of throwing, allowing the batch to seal normally. The oversized item becomes a pre-failed run with `PAYLOAD_TOO_LARGE` error code, while other items in the batch process successfully. This prevents `batchTriggerAndWait` from seeing connection errors and retrying with exponential backoff. Also fixes the NDJSON parser not consuming the remainder of an oversized line split across multiple chunks, which caused "Invalid JSON" errors on subsequent lines. |
||
|
Eric Allam
|
b1e78a6590 |
feat(batch-queue): two-level tenant dispatch for fair queue (#3133)
Replace flat master queue index with two-level tenant dispatch to fix noisy neighbor problem. When a tenant has many queues at capacity, the scheduler now iterates tenants (Level 1) not queues, then fetches per-tenant queues (Level 2) only for eligible tenants. Single-deploy migration: new enqueues write to dispatch indexes only, consumer drains old master queue alongside new dispatch path until empty. |
||
|
Eric Allam
|
39dd91b098 |
fix(engine) prevent MVCC race in blockRunWithWaitpoint pending check (#3075)
Split the CTE in blockRunWithWaitpoint so the pending waitpoint check is a separate SQL statement. In READ COMMITTED isolation, each statement gets its own snapshot, so a separate SELECT sees the latest committed state from concurrent completeWaitpoint calls. Previously, the CTE did INSERT + pending check in one statement (one snapshot). If completeWaitpoint committed between the CTE start and the SELECT, the SELECT would still see PENDING due to the stale snapshot. Neither side would enqueue continueRunIfUnblocked, leaving the run stuck forever. |
||
|
Eric Allam
|
bed3789c31 |
fix(batch-queue): speed up batch queue processing by disabling cooloff and fixing retry race (#3079)
Fix slow fair queue processing by removing spurious cooloff on concurrency blocks and fixing a race condition where retry attempt counts were not atomically updated during message re-queue. Removed cooloff entirely from the batch queue |
||
|
Eric Allam
|
a482153365 | feat(webapp): require the user is an admin during an impersonation session (#3078) |