## Summary
Fixes a server-side memory leak in the webapp's SSE helper. Every
aborted SSE connection (client tab close, navigation, timeout) was
pinning its full request/response graph indefinitely on Node 20, so any
long-running webapp process accumulated retained memory proportional to
streaming-request churn.
## Root cause
`apps/webapp/app/utils/sse.ts` combined four abort signals via
`AbortSignal.any([requestAbortSignal, timeoutSignal,
internalController.signal])`. The composite signal tracks its source
signals in an internal `Set<WeakRef>` registered against a
`FinalizationRegistry`; under sustained traffic those entries accumulate
faster than they're cleaned up, pinning every source signal (and its
listeners, and anything those listeners close over) until the parent
signal itself is GC'd or aborts.
This is a long-standing Node issue with multiple open reports:
- [nodejs/node#54614](https://github.com/nodejs/node/issues/54614) —
original report, still open. A [follow-up from
ChainSafe](https://github.com/nodejs/node/issues/54614#issuecomment-4055656572)
describes the exact same shape in a Lodestar production workload (req +
timeout signals composed per request accumulating in long-running
worker) and the same mitigation: drop `AbortSignal.any`, compose
manually.
- [nodejs/node#55351](https://github.com/nodejs/node/issues/55351) —
mechanism confirmed by Node member @jasnell: *"the set of dependent
signals known to the AbortSignal are kept in an internal Set using
WeakRefs. The AbortSignals are being properly gc'd but the Set is never
cleaned out of the WeakRefs making those leak."* Partially fixed by [PR
#55354](https://github.com/nodejs/node/pull/55354), shipped in Node
22.12.0 — but only covers the tight-loop case, not long-lived parent
signals.
- [nodejs/node#57584](https://github.com/nodejs/node/issues/57584) —
circular-dependency variant, still open.
- [nodejs/node#62363](https://github.com/nodejs/node/issues/62363) —
regression in Node 24/25 from an unrelated V8 change ("Don't pretenure
WeakCells"). Different root cause, same symptom.
A separate issue in `apps/webapp/app/entry.server.tsx` —
`setTimeout(abort, ABORT_DELAY)` with no `clearTimeout` on success paths
— kept the React render tree + `remixContext` alive for 30s per
successful HTML request. Same pattern fixed upstream in React Router
templates
([react-router#14200](https://github.com/remix-run/react-router/pull/14200)),
never backported to Remix v2.
## What changed
- **`apps/webapp/app/utils/sse.ts`** — single-signal abort chain.
`AbortSignal.any` removed; `AbortSignal.timeout` replaced by a plain
`setTimeout` cleared when the controller aborts; named sentinel
constants used as stackless abort reasons; request-abort handler
explicitly removed on cleanup.
- **`apps/webapp/app/entry.server.tsx`** — clears the `setTimeout(abort,
ABORT_DELAY)` timer in `onShellReady` / `onAllReady` / `onShellError`.
- **`apps/webapp/app/v3/tracer.server.ts` + `env.server.ts`** — gates
OpenTelemetry `HttpInstrumentation` and `ExpressInstrumentation` behind
`DISABLE_HTTP_INSTRUMENTATION=true` as an escape hatch for future
OTel-listener retention patterns. Defaults to enabled.
- **`apps/webapp/app/presenters/v3/RunStreamPresenter.server.ts`** —
uses the shared `ABORT_REASON_SEND_ERROR` sentinel.
## Verification
### Full-app reproduction (memlab)
Isolated local harness, 500 abrupt SSE disconnects against a
dev-presence route, GC between passes, heap snapshot diff with
[memlab](https://facebook.github.io/memlab/):
| Run | Heap delta after 500 conns + GC | memlab retained leaks |
| --- | --- | --- |
| Before | +16.0 MB (linear with request count) | 158 clusters; 250
`ServerResponse`, 1000 `AbortController`, 250 `SpanImpl` retained |
| After | **+3.3 MB (noise)** | **0 app-code leaks** |
### Standalone mechanism isolation
To confirm *which* axis of the change is load-bearing, a separate
standalone Node script (`/tmp/abort-leak-test.mjs`) ran 2000 requests ×
200 KB payload per variant:
| Variant | Heap delta after GC |
| --- | --- |
| baseline (no signal machinery) | 0 MB |
| V1: `AbortSignal.any` + string abort reason | **+9.1 MB** |
| V2: `AbortSignal.any` only (no reason) | **+10.8 MB** |
| V3: string reason only (no `AbortSignal.any`) | 0 MB |
| V4: neither (the fix) | 0 MB |
| V5: `AbortSignal.any` with no listener on the composite | **+10.2 MB**
|
This proves `AbortSignal.any` is the sole mechanism. The reason type
(`.abort()` vs `.abort("string")`) is irrelevant for retention — V3 is
clean, V5 leaks even without a listener on the composite.
## Risk
- `sse.ts` is used by the dev-presence routes. Behaviour is equivalent —
timeouts and client disconnects still abort the stream. `signal.reason`
is now a named string sentinel (`"timeout"`, `"request_aborted"`, etc.)
instead of the previous string arg or default `AbortError`. No in-tree
reader of `signal.reason` exists.
- `entry.server.tsx` change is a standard cleanup of an abort timer,
matches upstream React Router guidance.
- `tracer.server.ts` change is env-gated and defaults to current
behaviour.
- Three other webapp `AbortSignal.timeout()` callsites (alert delivery,
remote-build status) are fire-and-forget passed directly to `fetch` —
not composed with anything long-lived, no retention risk, untouched.
## Test plan
- [ ] Existing SSE integration tests pass
- [ ] Dev-presence SSE behaves normally across tab open/close cycles
- [ ] No heap growth under sustained aborted-connection traffic (heap
snapshot diff)
## Follow-up
The same `AbortSignal.any([userSignal, internalSignal])` pattern exists
in several SDK/core callsites that ship to customers
(`packages/core/src/v3/realtimeStreams/manager.ts`,
`packages/trigger-sdk/src/v3/{ai,chat,chat-client,sessions}.ts`,
`packages/core/src/v3/workers/warmStartClient.ts`). Whether those leak
in practice depends on the user passing a long-lived signal. Tracked
separately.
## Problem
When `batchTrigger()` is called with large payloads, each item's payload
is uploaded to R2 server-side during the streaming loop before being
enqueued. This makes the loop slow — around 3 seconds per item. Workers
pick up and execute each item as it's enqueued, running concurrently
with the ongoing stream.
For the last item in the batch, a race exists between the streaming loop
finishing and the batch completion cleanup:
1. The loop enqueues the last item and returns from `enqueueBatchItem()`
2. A waiting worker picks up the item almost instantly and executes it
3. `recordSuccess()` fires, `processedCount` hits the expected total,
`finalizeBatch()` runs
4. `cleanup()` deletes all Redis keys for the batch, including
`enqueuedItemsKey`
5. The streaming loop exits and calls `getBatchEnqueuedCount()` — reads
the now-deleted key — returns 0
The count check finds `enqueuedCount (0) !== batch.runCount`, falls
through to a Postgres fallback, but the fallback only checked `sealed`.
The BatchQueue completion path sets `status = COMPLETED` in Postgres
without setting `sealed = true` (that's the streaming endpoint's job),
so the fallback misses it too.
This causes the endpoint to return `sealed: false`. The SDK treats this
as retryable and retries up to 5 times with exponential backoff. Each
retry calls `enqueueBatchItem()`, which reads the batch meta key from
Redis — also deleted by `cleanup()` — and throws "Batch not found or not
initialized" (500). The final retry gets a 422 because the batch is
already COMPLETED, which the SDK does not retry, causing an `ApiError`
to be thrown from `await batchTrigger()` in the parent run — even though
all child runs completed successfully.
## Fix
In the Postgres fallback inside `StreamBatchItemsService`, also check
`status === "COMPLETED"` alongside `sealed`. This covers the
fast-completion path where the BatchQueue finishes all runs before the
streaming endpoint gets to seal the batch normally.
Also switches `findUnique` to `findFirst` per webapp convention.
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
## Summary
Adds `RUN_ENGINE_READ_REPLICA_SNAPSHOTS_SINCE_ENABLED` (default `"0"`).
When enabled, the Prisma reads inside `RunEngine.getSnapshotsSince` run
against the read-only replica client instead of the primary. Offloads
the snapshot-polling queries fired by every running task runner off the
writer.
## Why
`getSnapshotsSince` is called from the managed runner's
fetch-and-process loop (once per poll interval, plus on every
snapshot-change notification). It runs four sequential reads per call —
one `findFirst` by snapshot id, one `findMany` on snapshots with
`createdAt > X`, one raw SQL against `_completedWaitpoints`, and chunked
`findMany` on `waitpoint`. Per concurrent run, every few seconds. It's
read-only, tolerates a small amount of staleness, and is an obvious
candidate for the replica.
## Replica-lag considerations
- **Step 1 "since snapshot not found"**: if the runner just received a
snapshot id from the primary and asks the replica before it replicates,
the function throws and the caller treats the response as an error
(runner falls back to a metadata refresh). Self-correcting, not silent.
- **Step 2 missing newly-created snapshots**: the next poll's `createdAt
> sinceSnapshot.createdAt` filter still picks them up once the replica
catches up.
- **Waitpoint junction race**: the riskiest path — if a latest snapshot
is replicated but its `_completedWaitpoints` join rows aren't yet, the
runner could advance past that snapshot with `completedWaitpoints: []`.
WAL/storage-level replication replays commits in order, so in practice
both should appear atomically on the reader, but the race window is why
the flag ships disabled.
Aurora reader shrinks all three windows to single-digit ms in typical
conditions, and its storage-level replication gives atomic visibility of
committed transactions on the reader.
## Test plan
- [ ] Flip the flag on in a non-prod environment, confirm snapshot
polling behaves normally and `getSnapshotsSince` errors in Sentry stay
flat.
- [ ] Verify writer query volume drops and reader query volume rises on
the snapshot-polling queries.
- [ ] Keep an eye on `AuroraReplicaLag` (or equivalent) during rollout.
## Summary
Stamp every Prisma span with `db.datasource: "writer" | "replica"` so
traces can distinguish which client the query went through.
Both `PrismaClient` instances share the same global
`@prisma/instrumentation`, so their spans come out with identical names
and attributes today. This makes them trivially filterable.
## How
Two pieces in `apps/webapp/app/`:
1. **`v3/tracer.server.ts`** — a `DatasourceAttributeSpanProcessor`
reads an OTel context key in `onStart` and calls
`span.setAttribute("db.datasource", value)`. Registered as the first
span processor.
2. **`db.server.ts`** — `tagDatasource(datasource, client)` wraps each
`PrismaClient` with `$extends({ query: { $allOperations } })`. The
middleware sets the context key around the query and directly tags the
active span (to catch `prisma:client:operation`, which Prisma creates
before the middleware fires).
### Context-propagation gotcha
`PrismaPromise` is lazy — `query(args)` returns a thenable that only
starts when someone `.then()`s it. The naive `context.with(ctx, () =>
query(args))` restores ALS synchronously, so when Prisma's internal code
awaits the thenable later, the engine spans fire with the original ALS.
Wrapping as `async () => await query(args)` forces the `.then()` inside
the `context.with` callback, so ALS stays on our context for the engine
spans.
### Coverage
- **Tagged**: all `prisma:engine:*` (`connection`, `db_query`,
`serialize`, `query`, etc.), `prisma:client:operation`,
`prisma:client:serialize`, `prisma:client:connect`
- **Not tagged**: `prisma:client:load_engine` — one-time startup, fires
before any query
Concurrent `Promise.all([writer.x, replica.y])` correctly tags each pool
separately (ALS isolates per-Promise chain).
### Performance
One `context.with` (~200ns) and one `setAttribute` per span (effectively
free per OTel JS benchmarks) per Prisma op. Negligible against a query
path measured in milliseconds.
## Test plan
- [ ] Verify `db.datasource` appears on `prisma:engine:connection` spans
after the webapp is restarted
- [ ] Spot-check a handful of real traces carry the attribute
The cancel button was missing from the runs list for runs in `DEQUEUED`
status. The runs list gates the button on `run.isCancellable`, which
goes through `isCancellableRunStatus` -> `CANCELLABLE_RUN_STATUSES` =
`NON_FINAL_RUN_STATUSES`. `DEQUEUED` was never added to that list when
it was introduced in the run engine.
The single run page uses a separate check (`!run.isFinished`, i.e. the
inverse of `FINAL_RUN_STATUSES`), so cancellation already worked there -
only the list was affected.
Adding `DEQUEUED` to `NON_FINAL_RUN_STATUSES` also flips
`isCrashableRunStatus` and `isFailableRunStatus`, but:
- The crash path is the right behaviour - a `DEQUEUED` run (worker has
claimed but not yet executing) can legitimately crash before
`EXECUTING`, same as `PENDING`/`DELAYED` already do.
- The fail path (`failedTaskRun.server.ts`) is only reached from V1 code
paths (marqs consumers, v1 heartbeat handler). `DEQUEUED` is a
V2-engine-only status, so V1 consumers never see it.
When cancelling a `DEQUEUED` run the execution snapshot goes to
`PENDING_CANCEL` (worker must ack) but `TaskRun.status` flips to
`CANCELED` immediately - the UI reflects cancellation without waiting
for the worker. Added an integration test in
`run-engine/src/engine/tests/cancelling.test.ts` covering the full
trigger -> dequeue -> cancel -> worker-ack flow.
## Stall safety
The stall recovery path (PENDING_EXECUTING heartbeat miss ->
nack-and-requeue -> back to QUEUED) lives entirely inside
`@internal/run-engine` and never touches the webapp's `taskStatus.ts`
helpers - the engine has zero imports from `~/v3/taskStatus` and doesn't
know `CrashTaskRunService` / `FailedTaskRunService` exist. A stalled
DEQUEUED run still goes back to the queue for retry; this change cannot
cause stalls to crash or fail.
The only realistic impact is the intended UI fix - the theoretical V1
crash/fail branches for DEQUEUED are unreachable in practice because V1
runs never have DEQUEUED status.
## Summary
Regenerating a RuntimeEnvironment API key no longer immediately
invalidates the previous one. Rotation is now overlap-based: the old key
keeps working for 24 hours so customers can roll it out in their env
vars without downtime, then stops working.
## Design
- **New `RevokedApiKey` table** (one row per revocation). Holds the
archived `apiKey`, a FK to the env, an `expiresAt`, and a `createdAt`.
Indexed on `apiKey` (high-cardinality equality — single-row hits) and on
`runtimeEnvironmentId`.
- **`regenerateApiKey` wraps both writes in a single `$transaction`:**
insert a `RevokedApiKey` with `expiresAt = now + 24h`, update the env
with the new `apiKey`/`pkApiKey`.
- **`findEnvironmentByApiKey` does a two-step lookup:** primary
unique-index hit on `RuntimeEnvironment.apiKey` first; on miss,
`RevokedApiKey.findFirst({ apiKey, expiresAt: { gt: now } })` with an
`include: { runtimeEnvironment }`. Two-step (not `OR`-join) keeps the
hot path identical to today and puts the fallback cost only on invalid
keys. Both lookups use `$replica`.
- **Admin endpoint** `POST /admin/api/v1/revoked-api-keys/:id` accepts
`{ expiresAt }` and updates the row. Setting to `now` ends the grace
window immediately; setting to the future extends it.
- **Modal copy** on the regenerate dialog updated — previously warned of
downtime, now explains the 24h overlap.
## Why a separate table instead of columns on `RuntimeEnvironment`
- Keeps the hot auth path's primary lookup unchanged — no
OR/nullable-apiKey semantics to reason about.
- Naturally supports multiple in-flight grace windows (regenerate twice
in a day → two old keys valid until their independent expiries).
- FK + cascade cleans up correctly when an env is deleted; nothing to
backfill.
## Test plan
Verified locally against hello-world with dev and prod env keys:
- [x] baseline — current key authenticates (`GET /api/v1/runs`) → `200`
- [x] regenerate via UI — DB shows old key in `RevokedApiKey` with
`expiresAt ≈ now+24h`, env has new key
- [x] grace window — both old and new keys → `200`; bogus key → `401`
- [x] admin endpoint: `expiresAt = now` → old key `401`
- [x] admin endpoint: `expiresAt = +1h` (after early-expire) → old key
`200` again
- [x] admin endpoint: `expiresAt = past` → old key `401`
- [x] admin 400 (invalid body), 404 (unknown id), 401 (missing/non-admin
PAT)
- [x] same flow exercised end-to-end on a PROD-typed env — behavior
identical
- [x] `pnpm run typecheck --filter webapp` passes
## ✅ Checklist
- [x] I have followed every step in the [contributing
guide](https://github.com/triggerdotdev/trigger.dev/blob/main/CONTRIBUTING.md)
- [x] The PR title follows the convention.
- [x] I ran and tested the code works
---
## Summary
Adds a server-side gate that detects deploy attempts from v3 CLI
versions (i.e. `trigger.dev@3.x`) at the `POST /api/v1/deployments`
entry point and, when enabled, rejects them with a clear upgrade
message. v4 CLI deploys are completely unaffected.
The last 3.x CLI release was `3.3.7`, which we can't update. This
approach short-circuits the deploy before any DB writes, image-ref
generation, S2 stream creation, or queue enqueue — no side effects in
either mode.
## How v3 vs v4 are distinguished
I pulled the published CLI tarballs for `trigger.dev@3.3.7`, `4.0.0`,
`4.0.1`, `4.0.5`, `4.1.0`, `4.2.0`, and the current `4.4.4` in the repo.
The cleanest, most reliable signal is the request body to `POST
/api/v1/deployments`:
| Field on initialize | v3.3.7 CLI | v4.x CLI |
|---|---|---|
| `type` | **never sent** | always sent — `"MANAGED"` (run_engine_v2) or
`"V1"` |
| `isNativeBuild` / `gitMeta` / `triggeredVia` / `runtime` | not sent |
sent |
| `registryHost` / `namespace` | sent (v3-only; stripped by current Zod
schema) | not sent |
Every v4 call site I inspected sets `type: features.run_engine_v2 ?
"MANAGED" : "V1"` unconditionally. `payload.type` is `undefined` if and
only if the client is a 3.x CLI.
## Behavior
- Detection always runs and emits `logger.warn("Detected deploy from
deprecated v3 CLI", { environmentId, projectId, organizationId, enforced
})`, which lets us watch how many v3 deploys are still happening before
enforcement is flipped.
- Enforcement is gated behind `DEPRECATE_V3_CLI_DEPLOYS_ENABLED`
(default `"0"`, off). When `"1"`, the server returns `400` with:
> The trigger.dev CLI v3 is no longer supported for deployments. Please
upgrade your project to v4: https://trigger.dev/docs/migrating-from-v3
The v3 CLI surfaces this verbatim as `Failed to start deployment:
<message>` because `zodfetch` throws `ApiError` for non-retryable 4xx
(400/422) and `deploy.js` in 3.3.7 prints `error.message`.
## Out of scope (intentionally)
- `api.v1.deployments.$deploymentId.finalize.ts` /
`FinalizeDeploymentService` /
`createDeploymentBackgroundWorkerV3.server.ts` are V1-engine paths, not
the v3 CLI gate. Leaving them alone per review.
- Container-side `createDeploymentBackgroundWorker` call in
`managed-index-controller.ts` is still used by v4's in-image indexer.
Not touched.
- v3 `trigger dev` flow (different code path) — separate deprecation
if/when needed.
## Testing
- Ran `pnpm run typecheck --filter webapp` locally — passes.
- Verified v4 tarballs (4.0.0, 4.0.1, 4.0.5, 4.1.0, 4.2.0, 4.4.4) all
include `type:` in the `initializeDeployment` call site, so none will be
accidentally blocked.
- Verified v3.3.7 tarball's `initializeDeployment` payload has no `type`
field.
Rollout plan after merge:
1. Deploy with `DEPRECATE_V3_CLI_DEPLOYS_ENABLED` unset → watch
`Detected deploy from deprecated v3 CLI` log volume.
2. When comfortable, set `DEPRECATE_V3_CLI_DEPLOYS_ENABLED=1` to
enforce.
---
## Changelog
Detect v3 CLI deploys on `/api/v1/deployments` and, when
`DEPRECATE_V3_CLI_DEPLOYS_ENABLED=1`, reject them with an upgrade
message pointing at https://trigger.dev/docs/migrating-from-v3. v4 CLI
deploys are unaffected.
Link to Devin session:
https://app.devin.ai/sessions/b242c11bd86e4099aeec8b59bab62143
Requested by: @ericallam
Example cURL call using an admin user PAT (replace with a real one):
```sh
curl -X PUT https://cloud.trigger.dev/admin/api/v1/environments/<environmentId>/burst-factor \
-H "Authorization: Bearer tr_pat_1234" \
-H "Content-Type: application/json" \
-d '{"burstFactor": 1.5}'
```
Adds a `MicroVM` badge next to the region name on the regions page. Uses
the existing `small` badge variant for visual consistency with the
`Default` badge already on this page.
Two changes to cut error volume from logs that represent handled
conditions, not real errors (combined ~1600/hr in prod):
1. api.v1.waitpoints.tokens.$waitpointFriendlyId.complete.ts
The route throws `json(..., { status: 404 })` when a waitpoint
isn't found, but the generic catch block caught that Response,
logged it as an error (with an empty {} body because Error fields
are non-enumerable), and rethrew as a 500 — so clients saw a 500
instead of the intended 404, and every stale-waitpoint request
produced a Sentry event.
Fix: re-throw Response objects unchanged so the correct status
propagates and we don't log user 404s as errors. Also serialize
remaining Error instances explicitly (name/message/stack) so the
logs are actionable when we do hit a real error.
2. v3/marqs/sharedQueueConsumer.server.ts:603
"Task run has invalid status for execution. Going to ack" — the
message itself says we're handling it gracefully. Benign race
between dequeue and completion/cancellation. Demote to warn.
Wraps getEntitlement in platform.v3.server.ts with the existing
platformCache (LRU memory + Redis) under a new `entitlement` namespace.
Eliminates a synchronous billing-service HTTP round trip on every
trigger.
Cache config: 60s fresh / 60s stale SWR. Cache key is the
organization id. Errors are caught inside the loader and return the
existing permissive { hasAccess: true } fallback, which is also
cached to prevent thundering-herd on billing outages.
Trade-off: plan upgrade/downgrade is now visible after up to ~120s
worst-case (60s fresh + 60s stale revalidation). Acceptable since
the existing limits and usage namespaces use 5min/10min, and the
defensive hasAccess: true fallback already exists.
Replace the expensive DISTINCT query for task filter dropdowns with a
dedicated TaskIdentifier registry table backed by Redis. Environments
migrate automatically on their next deploy, with a transparent fallback
to the legacy query for unmigrated environments. Also fixes duplicate
dropdown entries when a task changes trigger source, and adds
active/archived grouping for removed tasks. Moves BackgroundWorkerTask
reads in the trigger hot path to the read replica.
Pool Redis connections for non-blocking ops (ingestData, appendPart,
getLastChunkIndex)
using a shared singleton instead of new Redis() per request. Use
redis.disconnect()
for immediate teardown in streamResponse cleanup. Add 15s inactivity
timeout fallback.
Fix broken request.signal in Remix/Express by wiring Express
res.on('close') to an
AbortController via httpAsyncStorage. All SSE/streaming routes now use
getRequestAbortSignal() which fires reliably on client disconnect,
bypassing the
Node.js undici GC bug (nodejs/node#55428) that severs the signal chain.
Extends the admin worker groups endpoint with a GET loader and more
fields on POST (type, hidden, workloadType, cloudProvider, location,
staticIPs, enableFastPath), and pulls the PAT + admin check that was
inlined or locally duplicated across every admin.api route into a shared
helper in personalAccessToken.server.ts. The generic
authenticateAdminRequest returns a discriminated result;
requireAdminApiRequest is the thin Remix loader/action wrapper that
throws. The neverthrow-style route (platform-notifications.ts) now
composes the generic helper instead of duplicating the check. Verified
locally against GET (listing) and POST (new fields, invalid enum,
minimal backwards-compat).
## Summary
Upgrades all `@remix-run/*` packages in `apps/webapp` from **2.1.0 →
2.17.4** to address security vulnerabilities. Recreation of #2951 on a
fresh checkout of `main`.
**Updated packages (`apps/webapp/package.json`):**
- `@remix-run/express`, `@remix-run/node`, `@remix-run/react`,
`@remix-run/serve`, `@remix-run/server-runtime`: 2.1.0 → 2.17.4
- `@remix-run/router`: ^1.15.3 → ^1.23.2
- `@remix-run/dev`, `@remix-run/eslint-config`, `@remix-run/testing`:
2.1.0 → 2.17.4
**Root `package.json` overrides:**
- `@remix-run/dev@2.17.4>tar-fs`: 2.1.3 → 2.1.4
- `testcontainers@10.28.0>tar-fs`: 3.0.9 → 3.1.1
**Documentation:** Updated Remix version references in `CLAUDE.md`,
`apps/webapp/CLAUDE.md`, and `.cursor/rules/webapp.mdc`.
**Server changes:** Added `.server-changes/upgrade-remix-security.md`
for release tracking per `CONTRIBUTING.md`.
No application code changes — only `package.json` files, documentation,
a server-changes entry, and the regenerated `pnpm-lock.yaml`.
### Updates since last revision
Addressed all 3 Devin Review findings:
1. **Missing `.server-changes/` file** — added
`.server-changes/upgrade-remix-security.md` (commit ce22a0bd4)
2. **Sentry Remix patch (`@sentry/remix@9.46.0`)** — verified the patch
at `patches/@sentry__remix@9.46.0.patch` applies cleanly against 2.17.4.
The patch modifies Sentry's own `RemixInstrumentation` wrapper (removing
`request.clone()` and form data attributes), not Remix internals. The
underlying Remix APIs it hooks into (`callRouteAction`,
`callRouteLoader`) are stable across 2.1→2.17.
3. **`remix-typedjson@0.3.1` compatibility** — peer deps declare
`@remix-run/react: ^1.16.0 || ^2.0`, covering 2.17.4. Confirmed working
at runtime across all 22 tested pages that use it (root.tsx, hooks,
route loaders).
### Verification performed during this session
- **Runtime:** Express+Remix integration, magic link login, client-side
routing, MetaFunction rendering
- **Operational:** hello-world task triggered via API, runs list, run
detail, tasks page
- **Comprehensive UI:** 22 pages, 11 filter types, environment/project
switchers, interactive elements
- **Docker:** Production Dockerfile (`docker/webapp/Dockerfile`) builds
successfully
- **Changelog audit:** All 16 minor versions reviewed — every breaking
change is behind opt-in future flags the webapp doesn't enable
## Review & Testing Checklist for Human
- [ ] **Verify auth flows in staging** — `remix-auth`,
`remix-auth-email-link`, and `remix-auth-github` declare peer deps on
`@remix-run/server-runtime@^1.x`, which is now 2.17.4. Login (magic link
+ OAuth) should be tested in a staging environment since local dev
testing may not exercise all auth code paths.
- [ ] **Verify tar-fs override versions** resolve the targeted security
advisories (2.1.4 and 3.1.1)
- [ ] **Review new transitive dependencies** added by the upgrade:
`turbo-stream@2.4.1`, `undici@6.25.0`, `valibot@1.3.1`, `ws@7.5.10`
Recommended test plan: deploy to staging and exercise core webapp flows
— login (email magic link + GitHub OAuth), dashboard navigation, task
triggering/viewing, and API endpoints — to catch runtime regressions not
covered by local testing.
### Notes
- Peer dependency warnings for `remix-auth-*` packages (expecting
`@remix-run/server-runtime@^1.x`) were present in the original PR #2951
as well and appear to be pre-existing
- The lockfile diff is large (~1200 lines) but mechanical — driven by
the Remix version bump cascading through transitive dependencies
- CI failures (`audit`, `units/internal/1-of-8`) are unrelated: `audit`
is a `claude-code-action` bot permissions issue; the internal test
failure is a ClickHouse testcontainers `Failed to connect to Reaper`
flake
Link to Devin session:
https://app.devin.ai/sessions/d9fa9953b9bf40e5a8d12b8f5ba5b86b
Requested by: @ericallam
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: Eric Allam <eallam@icloud.com>
The TaskRun.runTags string array already stores tag names, making the
TaskRunTag M2M relation redundant write overhead. Remove createTags
calls, connect: tags, and join table writes from both V1 and V2 trigger
paths. Simplify the add-tags API to just push to runTags directly.
Adds region-level gating so MICROVM regions are only visible and usable
by orgs with the `hasComputeAccess` feature flag. Admins and explicit
allowlist behavior unchanged.
- New shared helper (`regionAccess.server.ts`) with
`resolveComputeAccess`, `defaultVisibilityFilter`, and
`isComputeRegionAccessible`
- `RegionsPresenter` filters out MICROVM regions for non-compute orgs
- `SetDefaultRegionService` blocks setting a MICROVM region as default
without compute access
- `WorkerGroupService` blocks triggering runs in MICROVM regions without
compute access
- `computeTemplateCreation` refactored to use shared
`resolveComputeAccess`
- Updated snapshot callback schema
Sets `application_name` on the Prisma writer and replica connection
strings using the existing `SERVICE_NAME` env var, so DB load can be
attributed by service.
A single "fetch failed" from the object store was aborting the entire
batch stream with no retry. Added p-retry (3 attempts, 500ms-2s backoff)
around ploadPacketToObjectStore so transient network errors self-heal
server-side instead of propagating to the SDK.
### Text wrapping fix
- Fixes message text not wrapping on the run inspector if there were no
spaces in the text
- Fixes inspector title truncation
- Adds a copy text button for the Message property
<img width="468" height="740" alt="CleanShot 2026-04-04 at 10 19 02@2x"
src="https://github.com/user-attachments/assets/71e42bf3-d103-44a2-b3b4-937c0b60a4bc"
/>
This is a small improvement mainly with the UI Skills file:
- Animate open and close the Resizable panels
- Uses the built in animation hooks from react-window-splitter
- Includes a global variable for the animation easing and timing for
consistency
https://github.com/user-attachments/assets/50ed0019-ed12-4e08-b95c-7c6d1fe5bac0
## Summary
- Drop all 8 foreign key constraints on TaskRun. The run listing path is
now fully ClickHouse-backed so we no longer need Postgres to enforce
referential integrity on this table. The FK constraints add write
overhead on every insert/update with no remaining benefit. Prisma
queries are unaffected.
- Remove PostgresRunsRepository and its associated feature flag
(runsListRepository), which was the last remaining code path querying
TaskRun directly for list/count operations.
- Drop three indexes that were only useful for the Postgres run list
path and have no remaining query consumers:
- TaskRun_runtimeEnvironmentId_id_idx — was the cursor pagination index
for PostgresRunsRepository; superseded by the (runtimeEnvironmentId,
createdAt DESC) composite index
- TaskRun_scheduleId_idx — redundant with the (scheduleId, createdAt
DESC) composite index; no direct Postgres queries filter by scheduleId
alone
- TaskRun_rootTaskRunId_idx — no queries filter TaskRun by rootTaskRunId
as a WHERE clause anywhere in the codebase
All index drops use CONCURRENTLY IF EXISTS to avoid table locks in
production.
## Test plan
- pnpm run db:migrate:deploy applies all migrations cleanly
- pnpm run typecheck --filter webapp passes
- Run list pages load correctly in the dashboard (ClickHouse path)
- Scheduled task runs still trigger and appear correctly
This allows seamless migration to different object storage.
Existing runs that have offloaded payloads/outputs will continue to use
the default object store (configured using `OBJECT_STORE_*` env vars).
You can add additional stores by setting new env vars:
- `OBJECT_STORE_DEFAULT_PROTOCOL` this determines where new run large
payloads will get stored.
- If you set that you need to set new env vars for that protocol.
Example:
```
OBJECT_STORE_DEFAULT_PROTOCOL=“s3"
OBJECT_STORE_S3_BASE_URL=https://s3.us-east-1.amazonaws.com
OBJECT_STORE_S3_ACCESS_KEY_ID=<val>
OBJECT_STORE_S3_SECRET_ACCESS_KEY=<val>
OBJECT_STORE_S3_REGION=us-east-1
OBJECT_STORE_S3_SERVICE=s3
```
---------
Co-authored-by: nicktrn <55853254+nicktrn@users.noreply.github.com>
- Added versions filtering on the Errors list and page
- Added errors stacked bars to the graph on the individual error page
---------
Co-authored-by: James Ritchie <james@trigger.dev>
Temporary workaround that enables filtering by environment in the
envvars page, without changing any UI.
---------
Co-authored-by: Claude <noreply@anthropic.com>
## Summary
Currently, every triggered run follows a two-step path through Redis:
1. **Enqueue** — A Lua script atomically adds the message to a queue
sorted set (ordered by priority-adjusted timestamp)
2. **Dequeue** — A debounced `processQueueForWorkerQueue` job fires
~500ms later, checks concurrency limits, removes the message from the
sorted set, and pushes it to a worker queue (Redis list) where workers
pick it up via `BLPOP`
This means every run pays at least ~500ms of latency between being
triggered and being available for a worker to execute, even when the
queue is empty and concurrency is wide open.
### What changed
The enqueue Lua scripts now atomically decide whether to **skip the
queue sorted set entirely** and push directly to the worker queue. This
happens inside the same Lua script that handles normal enqueue, so the
decision is atomic with respect to concurrency bookkeeping.
A run takes the **fast path** when all of these are true:
- **Fast path is enabled** for this worker queue (gated per
`WorkerInstanceGroup`)
- **No available messages** in the queue (`ZRANGEBYSCORE` finds nothing
with score ≤ now) — this respects priority ordering and allows fast path
even when the queue has future-scored messages (e.g. nacked retries with
delay)
- **Environment concurrency** has capacity
- **Queue concurrency** has capacity (including per-concurrency-key
limits for CK queues)
When the fast path is taken:
- The message is stored and pushed directly to the worker queue
(`RPUSH`)
- Concurrency slots are claimed (`SADD` to the same sets used by the
normal dequeue path)
- The `processQueueForWorkerQueue` job is **not scheduled** (no work to
do)
- TTL sorted set is skipped (the `expireRun` worker job handles TTL
independently)
When any condition fails, the existing slow path runs unchanged.
### Rollout gating
- **Development environments**: Fast path is always enabled
- **Production environments**: Gated by a new `enableFastPath` boolean
on `WorkerInstanceGroup` (defaults to `false`), allowing
region-by-region rollout
### Rolling deploy safety
Each process registers its own Lua scripts via `defineCommand`
(identified by SHA hash). Old and new processes never share scripts. The
Redis data structures are fully compatible in both directions — ack,
nack, and release operations work identically regardless of which path a
message took.
## Test plan
- [x] Fast path taken when queue is empty and concurrency available
- [x] Slow path when `enableFastPath` is false
- [x] Slow path when queue has available messages (respects priority
ordering)
- [x] Fast path when queue only has future-scored messages
- [x] Slow path when env concurrency is full
- [x] Fast-path message can be acknowledged correctly
- [x] Fast-path message can be nacked and re-enqueued to the queue
sorted set
- [x] Run all existing run-queue tests (ack, nack, CK, concurrency
sweeper, dequeue) to verify no regressions
- [x] Typecheck passes for run-engine and webapp
Add TTL (time-to-live) defaults at task-level and config-level, with
precedence: per-trigger > task > config > dev default (10m).
Docs PR: #3200 (merge after packages are released)
Adds a dialog to the admin orgs page for viewing and editing per-org
feature flag overrides. Flags are introspected from the catalog so the
UI stays in sync with available flags automatically. Also adds a new tab
for global flags.
Refactors featureFlags.server.ts to split catalog definition (shared)
from server-only runtime (flags(), makeSetMultipleFlags). The shared
module exports flag metadata and validation so both the UI and API
routes can use it without pulling in server dependencies.
Adds support for taint tolerations for scheduled runs. Useful for
selectively tolerating taints on dedicated node pools.
The new `KUBERNETES_SCHEDULED_RUN_TOLERATIONS` env variable accepts a
comma-separated list in the format key=value:effect (or key:effect for
the Exists operator).
Drive-by: renames all `KUBERNETES_SCHEDULE_*` affinity env vars to
KUBERNETES_SCHEDULED_RUN_* for clarity — this feature isn't used in
production yet or published in a tagged image; the name change is fine.
For human reviewer:
- Check if Redis connection + code makes sense
- Check CLI methods (it's on a hotpath)
- Check DB Migrations and new tables
## ✅ Checklist
- [x] I have followed every step in the [contributing
guide](https://github.com/triggerdotdev/trigger.dev/blob/main/CONTRIBUTING.md)
- [x] The PR title follows the convention.
- [x] I ran and tested the code works
---
## Testing
Spawning new CLI / Dashboard notifications, check MVP, check if failures
not produce any problems with CLI/Dashboard
---
## Changelog
Added notifications mechanism for Dashboard and CLI
---
## Screenshots
💯
Adds three new top-level columns to the ClickHouse task_runs_v2 table
primarily for analytics:
- `trigger_source` / `root_trigger_source` - extracted from the existing
TaskRun.annotations JSON during WAL
replication
- `is_warm_start` - new nullable boolean on TaskRun in Postgres, set in
the existing taskRun.update() at attempt
start (no additional write). null until the first attempt starts.
Run region is already available via the existing `worker_queue` column
in ClickHouse.
Eric Allam