Sort findMany() by `key` for a stable winner when multiple rows assign the
same `${orgId}:${kind}`, and log an error identifying the winning and
ignored rows instead of overwriting silently. Does not fail the load —
failing the registry would break every customer, not just the misconfigured
orgs.
Co-Authored-By: Matt Aitken <matt@mattaitken.com>
The factory should not import eventRepository.server.ts — doing so pulls
the tracePubSub singleton into any module graph that imports the factory,
which eagerly connects to Redis at module load time (see singleton.ts).
The fallback now lives in index.server.ts via getEventRepositoryForStore,
which is called from RunPresenter, SpanPresenter, and recordRunEvent.
This restores runsReplicationBenchmark.test.ts's module isolation while
still handling non-ClickHouse ("taskEvent"/"postgres") stores.
Co-Authored-By: Matt Aitken <matt@mattaitken.com>
The only way to get a ClickHouse client now is through the factory.
Refactored all existing code to use that and pass in an org.
The runReplication and otlpExporter are the hot paths here which need special attention in reviews.
Adds a minimal end-to-end test harness that spawns the compiled webapp
as a child
process against a throwaway Postgres container, plus a baseline of 8
auth-behaviour
tests. These tests will be used as a regression check before and after
the upcoming
apiBuilder RBAC migration to confirm auth behaviour is unchanged.
## What's included
**`internal-packages/testcontainers/src/webapp.ts`** (new)
Spawns `build/server.js` with a dynamically allocated port, polls
`/healthcheck`,
and exposes `WebappInstance` and `startTestServer()` (postgres container
+ webapp +
PrismaClient in one call). Key details:
- Uses `process.execPath` so the correct Node binary is found in forked
test processes
- Sets `NODE_PATH` to `node_modules/.pnpm/node_modules` so pnpm-hoisted
transitive
deps (e.g. `eventsource-parser`) resolve correctly inside the subprocess
- Overrides both `PORT` and `REMIX_APP_PORT` so Vite's automatic `.env`
loading
doesn't override the dynamically allocated port
**`internal-packages/testcontainers/package.json`**
Adds `./webapp` sub-path export so tests can `import from
"@internal/testcontainers/webapp"`.
**`internal-packages/testcontainers/src/index.ts`**
Exports `createPostgresContainer` (used internally by `webapp.ts`).
**`apps/webapp/test/helpers/seedTestEnvironment.ts`** (new)
Creates a minimal org → project → environment row set with random
suffixes.
**`apps/webapp/test/api-auth.e2e.test.ts`** (new)
8 tests across two suites:
- API-key bearer: valid key (auth passes, 404), missing header (401),
invalid key (401), error body shape
- JWT bearer: valid JWT on JWT-enabled route (passes), valid JWT on
non-JWT route (401), empty-scope JWT (403), wrong signing key (401)
## How to run
```bash
# Build required first (one-time)
pnpm run build --filter webapp
cd apps/webapp && pnpm exec vitest run test/api-auth.e2e.test.ts
```
## Test plan
- [x] All 8 tests pass against the current webapp build
- [x] Webapp healthcheck returns 200 on startup
- [ ] CI passes
---------
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Adds `KUBERNETES_POD_DNS_NDOTS_OVERRIDE_ENABLED` flag (off by default)
that overrides the cluster default and sets `dnsConfig.options.ndots` on
runner pods (defaulting to 2, configurable via
`KUBERNETES_POD_DNS_NDOTS`).
Kubernetes defaults pods to `ndots: 5`, so any name with fewer than 5
dots, including typical external domains like `api.example.com`, is
first walked through every entry in the cluster search list
(`<ns>.svc.cluster.local`, `svc.cluster.local`, `cluster.local`) before
being tried as-is, turning one resolution into 4+ CoreDNS queries (×2
with A+AAAA).
Using a lower `ndots` value reduces DNS query amplification in the
`cluster.local` zone.
## Summary
- The `processMasterQueueForEnvironment` call in the priority test was
racing against background `processQueueForWorkerQueue` jobs scheduled
50ms after each trigger
- With a 50ms debounce (`processWorkerQueueDebounceMs: 50`) and runs
triggered sequentially, the RunQueue Worker could process those jobs
mid-sequence, pushing partial batches to the worker queue in the wrong
overall priority order
- `masterQueueConsumersDisabled: true` only blocks the shard-level
polling loops — it does not prevent the RunQueue's own Worker from
processing these debounced jobs
- Fix: add `worker.disabled: true` to the test 1 engine config, which
propagates to `workerOptions.disabled` in the RunQueue constructor and
prevents the Worker from starting
## Test plan
- [x] Both priority tests pass: `pnpm run test
./src/engine/tests/priority.test.ts --run`
- [x] Test 1 log confirms no `✅ Starting run engine worker` or worker
loop messages — workers fully disabled
- [x] Test 2 unaffected (uses master queue consumers for automatic
promotion, no `disabled` flag added)
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Wires up automatic Helm chart releases to ride along with the existing
changeset-driven package release flow.
Today `Chart.yaml` is bumped by hand and `release-helm.yml` fires only
when a human pushes a `helm-v*` tag. With this, the changeset release PR
also carries a `Chart.yaml` bump so main always matches the published
version, and `release.yml` invokes `release-helm.yml` via
`workflow_call` after Docker images are published.
`helm-v${VERSION}` tag is pushed as a marker (same GITHUB_TOKEN trick as
`v.docker.*`). Manual `helm-v*` tag flow still works. Chart.yaml
consistency check in `release-helm.yml` is the safety net if the bump
job ever drifts.
First rollout: the open `changeset-release/main` PR has stale
Chart.yaml. Bump it manually on that branch before merging, otherwise
the first automated helm release fails at the consistency check.
## Summary
Adds direct V8 heap and process-memory gauges to the webapp's
OpenTelemetry meter. The webapp already exports per-cluster-worker
Node.js runtime metrics (event-loop lag / utilization, active handles,
active requests, libuv threadpool size) via a custom meter under the
`trigger.dev` scope. Heap and memory were missing; this PR adds them
alongside, in the same observable-batch pattern.
## New gauges
| Metric | Source | Unit |
| --- | --- | --- |
| `nodejs.memory.heap.used` | `process.memoryUsage().heapUsed` | bytes |
| `nodejs.memory.heap.total` | `process.memoryUsage().heapTotal` | bytes
|
| `nodejs.memory.heap.limit` | `v8.getHeapStatistics().heap_size_limit`
| bytes |
| `nodejs.memory.external` | `process.memoryUsage().external` | bytes |
| `nodejs.memory.array_buffers` | `process.memoryUsage().arrayBuffers` |
bytes |
| `nodejs.memory.rss` | `process.memoryUsage().rss` | bytes |
Gated by the existing `INTERNAL_OTEL_NODEJS_METRICS_ENABLED` flag, same
as the adjacent event-loop / handle gauges. Zero overhead when disabled.
## Why
`@opentelemetry/host-metrics` publishes `process.memory.usage`, which is
RSS only. RSS is the sum of V8 heap, external memory (Buffers, etc.),
native code, and thread stacks. Without a direct heap metric it is not
possible to size the V8 heap cap (`--max-old-space-size`) from metrics
alone, because RSS overstates heap by the external + native footprint. A
worker can have a 4 GB RSS with a 2.5 GB heap and 1.5 GB of buffers; the
former constrains `--max-old-space-size`, the latter does not.
`nodejs.memory.heap.limit` also surfaces the configured
`--max-old-space-size` (read from
`v8.getHeapStatistics().heap_size_limit`), so operators can see the
current limit in the same dashboard as actual usage rather than
cross-referencing container environment variables.
## Risk
Minimal. Observable gauges are sampled at the configured metric-export
interval. `v8.getHeapStatistics()` and `process.memoryUsage()` are each
microsecond-level calls, and six gauges are added to the same batch
callback that already reads ~20 other Node.js runtime values per sample.
Same registration pattern as the existing event-loop metrics in the
file.
## Test plan
- [ ] Deploy and confirm the six new gauges appear at the configured
exporter
- [ ] In cluster mode, confirm per-worker granularity (one series per
cluster worker, tagged by `process.executable.name` /
`service.instance.id`)
- [ ] Confirm `nodejs.memory.heap.limit` reports the configured
`--max-old-space-size` value in bytes
## Summary
- New **Back office** tab at `/admin`, per-org detail page at
`/admin/back-office/orgs/:orgId` designed to host future per-org admin
actions (project count, delete account, YC deals).
- First action: edit an organization's API rate limit — tokenBucket
override (refill rate, interval, max tokens), with a live plain-English
preview (e.g. *"1,500 requests per minute · 750 request burst
allowance"*). Writes are audit-logged via the server logger.
- Cleanup: removed unused `v2?` / `v3?` columns from the admin orgs list
(display only — Prisma select untouched).
## Test plan
- [ ] Back office tab visible in admin nav and highlighted when on a
sub-route
- [ ] `/admin/orgs` shows a Back office "Open" link per row; no v2/v3
columns
- [ ] Empty state at `/admin/back-office` links back to `/admin/orgs`
- [ ] Detail page renders the effective rate limit in view mode; Edit
reveals the form
- [ ] Save writes `Organization.apiRateLimiterConfig`, returns to view
mode, shows "Rate limit saved." banner
- [ ] Invalid values surface inline field errors and keep edit mode
- [ ] Non-admins hitting any new route are redirected to `/`
- [ ] Server logs show `admin.backOffice.rateLimit` info line per
mutation
## Summary
Fixes a server-side memory leak in the webapp's SSE helper. Every
aborted SSE connection (client tab close, navigation, timeout) was
pinning its full request/response graph indefinitely on Node 20, so any
long-running webapp process accumulated retained memory proportional to
streaming-request churn.
## Root cause
`apps/webapp/app/utils/sse.ts` combined four abort signals via
`AbortSignal.any([requestAbortSignal, timeoutSignal,
internalController.signal])`. The composite signal tracks its source
signals in an internal `Set<WeakRef>` registered against a
`FinalizationRegistry`; under sustained traffic those entries accumulate
faster than they're cleaned up, pinning every source signal (and its
listeners, and anything those listeners close over) until the parent
signal itself is GC'd or aborts.
This is a long-standing Node issue with multiple open reports:
- [nodejs/node#54614](https://github.com/nodejs/node/issues/54614) —
original report, still open. A [follow-up from
ChainSafe](https://github.com/nodejs/node/issues/54614#issuecomment-4055656572)
describes the exact same shape in a Lodestar production workload (req +
timeout signals composed per request accumulating in long-running
worker) and the same mitigation: drop `AbortSignal.any`, compose
manually.
- [nodejs/node#55351](https://github.com/nodejs/node/issues/55351) —
mechanism confirmed by Node member @jasnell: *"the set of dependent
signals known to the AbortSignal are kept in an internal Set using
WeakRefs. The AbortSignals are being properly gc'd but the Set is never
cleaned out of the WeakRefs making those leak."* Partially fixed by [PR
#55354](https://github.com/nodejs/node/pull/55354), shipped in Node
22.12.0 — but only covers the tight-loop case, not long-lived parent
signals.
- [nodejs/node#57584](https://github.com/nodejs/node/issues/57584) —
circular-dependency variant, still open.
- [nodejs/node#62363](https://github.com/nodejs/node/issues/62363) —
regression in Node 24/25 from an unrelated V8 change ("Don't pretenure
WeakCells"). Different root cause, same symptom.
A separate issue in `apps/webapp/app/entry.server.tsx` —
`setTimeout(abort, ABORT_DELAY)` with no `clearTimeout` on success paths
— kept the React render tree + `remixContext` alive for 30s per
successful HTML request. Same pattern fixed upstream in React Router
templates
([react-router#14200](https://github.com/remix-run/react-router/pull/14200)),
never backported to Remix v2.
## What changed
- **`apps/webapp/app/utils/sse.ts`** — single-signal abort chain.
`AbortSignal.any` removed; `AbortSignal.timeout` replaced by a plain
`setTimeout` cleared when the controller aborts; named sentinel
constants used as stackless abort reasons; request-abort handler
explicitly removed on cleanup.
- **`apps/webapp/app/entry.server.tsx`** — clears the `setTimeout(abort,
ABORT_DELAY)` timer in `onShellReady` / `onAllReady` / `onShellError`.
- **`apps/webapp/app/v3/tracer.server.ts` + `env.server.ts`** — gates
OpenTelemetry `HttpInstrumentation` and `ExpressInstrumentation` behind
`DISABLE_HTTP_INSTRUMENTATION=true` as an escape hatch for future
OTel-listener retention patterns. Defaults to enabled.
- **`apps/webapp/app/presenters/v3/RunStreamPresenter.server.ts`** —
uses the shared `ABORT_REASON_SEND_ERROR` sentinel.
## Verification
### Full-app reproduction (memlab)
Isolated local harness, 500 abrupt SSE disconnects against a
dev-presence route, GC between passes, heap snapshot diff with
[memlab](https://facebook.github.io/memlab/):
| Run | Heap delta after 500 conns + GC | memlab retained leaks |
| --- | --- | --- |
| Before | +16.0 MB (linear with request count) | 158 clusters; 250
`ServerResponse`, 1000 `AbortController`, 250 `SpanImpl` retained |
| After | **+3.3 MB (noise)** | **0 app-code leaks** |
### Standalone mechanism isolation
To confirm *which* axis of the change is load-bearing, a separate
standalone Node script (`/tmp/abort-leak-test.mjs`) ran 2000 requests ×
200 KB payload per variant:
| Variant | Heap delta after GC |
| --- | --- |
| baseline (no signal machinery) | 0 MB |
| V1: `AbortSignal.any` + string abort reason | **+9.1 MB** |
| V2: `AbortSignal.any` only (no reason) | **+10.8 MB** |
| V3: string reason only (no `AbortSignal.any`) | 0 MB |
| V4: neither (the fix) | 0 MB |
| V5: `AbortSignal.any` with no listener on the composite | **+10.2 MB**
|
This proves `AbortSignal.any` is the sole mechanism. The reason type
(`.abort()` vs `.abort("string")`) is irrelevant for retention — V3 is
clean, V5 leaks even without a listener on the composite.
## Risk
- `sse.ts` is used by the dev-presence routes. Behaviour is equivalent —
timeouts and client disconnects still abort the stream. `signal.reason`
is now a named string sentinel (`"timeout"`, `"request_aborted"`, etc.)
instead of the previous string arg or default `AbortError`. No in-tree
reader of `signal.reason` exists.
- `entry.server.tsx` change is a standard cleanup of an abort timer,
matches upstream React Router guidance.
- `tracer.server.ts` change is env-gated and defaults to current
behaviour.
- Three other webapp `AbortSignal.timeout()` callsites (alert delivery,
remote-build status) are fire-and-forget passed directly to `fetch` —
not composed with anything long-lived, no retention risk, untouched.
## Test plan
- [ ] Existing SSE integration tests pass
- [ ] Dev-presence SSE behaves normally across tab open/close cycles
- [ ] No heap growth under sustained aborted-connection traffic (heap
snapshot diff)
## Follow-up
The same `AbortSignal.any([userSignal, internalSignal])` pattern exists
in several SDK/core callsites that ship to customers
(`packages/core/src/v3/realtimeStreams/manager.ts`,
`packages/trigger-sdk/src/v3/{ai,chat,chat-client,sessions}.ts`,
`packages/core/src/v3/workers/warmStartClient.ts`). Whether those leak
in practice depends on the user passing a long-lived signal. Tracked
separately.
Mirrors the existing `supervisor.serviceAccount` pattern onto webapp so
operators can annotate the SA (IRSA `eks.amazonaws.com/role-arn`,
Workload Identity, etc.) or bring their own SA. Without this,
`webapp.serviceAccount.annotations` isn't exposed and operators have to
patch the SA out-of-band.
```yaml
webapp:
serviceAccount:
create: true
name: ""
annotations:
eks.amazonaws.com/role-arn: arn:aws:iam::123456789012:role/trigger-webapp
```
Three pieces, same as supervisor:
- `webapp.serviceAccount.create` toggle on the SA block
- `webapp.serviceAccount.annotations` + `name` values
- `trigger-v4.webappServiceAccountName` helper, used by the SA, the
token-syncer RoleBinding subject, and the Deployment's
`serviceAccountName`
Role + RoleBinding are left unguarded (matching supervisor's shape where
`rbac.create` is a separate toggle from `serviceAccount.create`) -
BYO-SA users take on the responsibility of ensuring the SA they supply
has the permissions the RoleBinding grants.
Verified with `helm template` against default values, an IRSA annotation
override, and `create: false` with a custom name.
## Problem
When `batchTrigger()` is called with large payloads, each item's payload
is uploaded to R2 server-side during the streaming loop before being
enqueued. This makes the loop slow — around 3 seconds per item. Workers
pick up and execute each item as it's enqueued, running concurrently
with the ongoing stream.
For the last item in the batch, a race exists between the streaming loop
finishing and the batch completion cleanup:
1. The loop enqueues the last item and returns from `enqueueBatchItem()`
2. A waiting worker picks up the item almost instantly and executes it
3. `recordSuccess()` fires, `processedCount` hits the expected total,
`finalizeBatch()` runs
4. `cleanup()` deletes all Redis keys for the batch, including
`enqueuedItemsKey`
5. The streaming loop exits and calls `getBatchEnqueuedCount()` — reads
the now-deleted key — returns 0
The count check finds `enqueuedCount (0) !== batch.runCount`, falls
through to a Postgres fallback, but the fallback only checked `sealed`.
The BatchQueue completion path sets `status = COMPLETED` in Postgres
without setting `sealed = true` (that's the streaming endpoint's job),
so the fallback misses it too.
This causes the endpoint to return `sealed: false`. The SDK treats this
as retryable and retries up to 5 times with exponential backoff. Each
retry calls `enqueueBatchItem()`, which reads the batch meta key from
Redis — also deleted by `cleanup()` — and throws "Batch not found or not
initialized" (500). The final retry gets a 422 because the batch is
already COMPLETED, which the SDK does not retry, causing an `ApiError`
to be thrown from `await batchTrigger()` in the parent run — even though
all child runs completed successfully.
## Fix
In the Postgres fallback inside `StreamBatchItemsService`, also check
`status === "COMPLETED"` alongside `sealed`. This covers the
fast-completion path where the BatchQueue finishes all runs before the
streaming endpoint gets to seal the batch normally.
Also switches `findUnique` to `findFirst` per webapp convention.
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
## Summary
Adds `RUN_ENGINE_READ_REPLICA_SNAPSHOTS_SINCE_ENABLED` (default `"0"`).
When enabled, the Prisma reads inside `RunEngine.getSnapshotsSince` run
against the read-only replica client instead of the primary. Offloads
the snapshot-polling queries fired by every running task runner off the
writer.
## Why
`getSnapshotsSince` is called from the managed runner's
fetch-and-process loop (once per poll interval, plus on every
snapshot-change notification). It runs four sequential reads per call —
one `findFirst` by snapshot id, one `findMany` on snapshots with
`createdAt > X`, one raw SQL against `_completedWaitpoints`, and chunked
`findMany` on `waitpoint`. Per concurrent run, every few seconds. It's
read-only, tolerates a small amount of staleness, and is an obvious
candidate for the replica.
## Replica-lag considerations
- **Step 1 "since snapshot not found"**: if the runner just received a
snapshot id from the primary and asks the replica before it replicates,
the function throws and the caller treats the response as an error
(runner falls back to a metadata refresh). Self-correcting, not silent.
- **Step 2 missing newly-created snapshots**: the next poll's `createdAt
> sinceSnapshot.createdAt` filter still picks them up once the replica
catches up.
- **Waitpoint junction race**: the riskiest path — if a latest snapshot
is replicated but its `_completedWaitpoints` join rows aren't yet, the
runner could advance past that snapshot with `completedWaitpoints: []`.
WAL/storage-level replication replays commits in order, so in practice
both should appear atomically on the reader, but the race window is why
the flag ships disabled.
Aurora reader shrinks all three windows to single-digit ms in typical
conditions, and its storage-level replication gives atomic visibility of
committed transactions on the reader.
## Test plan
- [ ] Flip the flag on in a non-prod environment, confirm snapshot
polling behaves normally and `getSnapshotsSince` errors in Sentry stay
flat.
- [ ] Verify writer query volume drops and reader query volume rises on
the snapshot-polling queries.
- [ ] Keep an eye on `AuroraReplicaLag` (or equivalent) during rollout.
## Summary
Stamp every Prisma span with `db.datasource: "writer" | "replica"` so
traces can distinguish which client the query went through.
Both `PrismaClient` instances share the same global
`@prisma/instrumentation`, so their spans come out with identical names
and attributes today. This makes them trivially filterable.
## How
Two pieces in `apps/webapp/app/`:
1. **`v3/tracer.server.ts`** — a `DatasourceAttributeSpanProcessor`
reads an OTel context key in `onStart` and calls
`span.setAttribute("db.datasource", value)`. Registered as the first
span processor.
2. **`db.server.ts`** — `tagDatasource(datasource, client)` wraps each
`PrismaClient` with `$extends({ query: { $allOperations } })`. The
middleware sets the context key around the query and directly tags the
active span (to catch `prisma:client:operation`, which Prisma creates
before the middleware fires).
### Context-propagation gotcha
`PrismaPromise` is lazy — `query(args)` returns a thenable that only
starts when someone `.then()`s it. The naive `context.with(ctx, () =>
query(args))` restores ALS synchronously, so when Prisma's internal code
awaits the thenable later, the engine spans fire with the original ALS.
Wrapping as `async () => await query(args)` forces the `.then()` inside
the `context.with` callback, so ALS stays on our context for the engine
spans.
### Coverage
- **Tagged**: all `prisma:engine:*` (`connection`, `db_query`,
`serialize`, `query`, etc.), `prisma:client:operation`,
`prisma:client:serialize`, `prisma:client:connect`
- **Not tagged**: `prisma:client:load_engine` — one-time startup, fires
before any query
Concurrent `Promise.all([writer.x, replica.y])` correctly tags each pool
separately (ALS isolates per-Promise chain).
### Performance
One `context.with` (~200ns) and one `setAttribute` per span (effectively
free per OTel JS benchmarks) per Prisma op. Negligible against a query
path measured in milliseconds.
## Test plan
- [ ] Verify `db.datasource` appears on `prisma:engine:connection` spans
after the webapp is restarted
- [ ] Spot-check a handful of real traces carry the attribute
The cancel button was missing from the runs list for runs in `DEQUEUED`
status. The runs list gates the button on `run.isCancellable`, which
goes through `isCancellableRunStatus` -> `CANCELLABLE_RUN_STATUSES` =
`NON_FINAL_RUN_STATUSES`. `DEQUEUED` was never added to that list when
it was introduced in the run engine.
The single run page uses a separate check (`!run.isFinished`, i.e. the
inverse of `FINAL_RUN_STATUSES`), so cancellation already worked there -
only the list was affected.
Adding `DEQUEUED` to `NON_FINAL_RUN_STATUSES` also flips
`isCrashableRunStatus` and `isFailableRunStatus`, but:
- The crash path is the right behaviour - a `DEQUEUED` run (worker has
claimed but not yet executing) can legitimately crash before
`EXECUTING`, same as `PENDING`/`DELAYED` already do.
- The fail path (`failedTaskRun.server.ts`) is only reached from V1 code
paths (marqs consumers, v1 heartbeat handler). `DEQUEUED` is a
V2-engine-only status, so V1 consumers never see it.
When cancelling a `DEQUEUED` run the execution snapshot goes to
`PENDING_CANCEL` (worker must ack) but `TaskRun.status` flips to
`CANCELED` immediately - the UI reflects cancellation without waiting
for the worker. Added an integration test in
`run-engine/src/engine/tests/cancelling.test.ts` covering the full
trigger -> dequeue -> cancel -> worker-ack flow.
## Stall safety
The stall recovery path (PENDING_EXECUTING heartbeat miss ->
nack-and-requeue -> back to QUEUED) lives entirely inside
`@internal/run-engine` and never touches the webapp's `taskStatus.ts`
helpers - the engine has zero imports from `~/v3/taskStatus` and doesn't
know `CrashTaskRunService` / `FailedTaskRunService` exist. A stalled
DEQUEUED run still goes back to the queue for retry; this change cannot
cause stalls to crash or fail.
The only realistic impact is the intended UI fix - the theoretical V1
crash/fail branches for DEQUEUED are unreachable in practice because V1
runs never have DEQUEUED status.
## Summary
Regenerating a RuntimeEnvironment API key no longer immediately
invalidates the previous one. Rotation is now overlap-based: the old key
keeps working for 24 hours so customers can roll it out in their env
vars without downtime, then stops working.
## Design
- **New `RevokedApiKey` table** (one row per revocation). Holds the
archived `apiKey`, a FK to the env, an `expiresAt`, and a `createdAt`.
Indexed on `apiKey` (high-cardinality equality — single-row hits) and on
`runtimeEnvironmentId`.
- **`regenerateApiKey` wraps both writes in a single `$transaction`:**
insert a `RevokedApiKey` with `expiresAt = now + 24h`, update the env
with the new `apiKey`/`pkApiKey`.
- **`findEnvironmentByApiKey` does a two-step lookup:** primary
unique-index hit on `RuntimeEnvironment.apiKey` first; on miss,
`RevokedApiKey.findFirst({ apiKey, expiresAt: { gt: now } })` with an
`include: { runtimeEnvironment }`. Two-step (not `OR`-join) keeps the
hot path identical to today and puts the fallback cost only on invalid
keys. Both lookups use `$replica`.
- **Admin endpoint** `POST /admin/api/v1/revoked-api-keys/:id` accepts
`{ expiresAt }` and updates the row. Setting to `now` ends the grace
window immediately; setting to the future extends it.
- **Modal copy** on the regenerate dialog updated — previously warned of
downtime, now explains the 24h overlap.
## Why a separate table instead of columns on `RuntimeEnvironment`
- Keeps the hot auth path's primary lookup unchanged — no
OR/nullable-apiKey semantics to reason about.
- Naturally supports multiple in-flight grace windows (regenerate twice
in a day → two old keys valid until their independent expiries).
- FK + cascade cleans up correctly when an env is deleted; nothing to
backfill.
## Test plan
Verified locally against hello-world with dev and prod env keys:
- [x] baseline — current key authenticates (`GET /api/v1/runs`) → `200`
- [x] regenerate via UI — DB shows old key in `RevokedApiKey` with
`expiresAt ≈ now+24h`, env has new key
- [x] grace window — both old and new keys → `200`; bogus key → `401`
- [x] admin endpoint: `expiresAt = now` → old key `401`
- [x] admin endpoint: `expiresAt = +1h` (after early-expire) → old key
`200` again
- [x] admin endpoint: `expiresAt = past` → old key `401`
- [x] admin 400 (invalid body), 404 (unknown id), 401 (missing/non-admin
PAT)
- [x] same flow exercised end-to-end on a PROD-typed env — behavior
identical
- [x] `pnpm run typecheck --filter webapp` passes
## ✅ Checklist
- [x] I have followed every step in the [contributing
guide](https://github.com/triggerdotdev/trigger.dev/blob/main/CONTRIBUTING.md)
- [x] The PR title follows the convention.
- [x] I ran and tested the code works
---
## Summary
Adds a server-side gate that detects deploy attempts from v3 CLI
versions (i.e. `trigger.dev@3.x`) at the `POST /api/v1/deployments`
entry point and, when enabled, rejects them with a clear upgrade
message. v4 CLI deploys are completely unaffected.
The last 3.x CLI release was `3.3.7`, which we can't update. This
approach short-circuits the deploy before any DB writes, image-ref
generation, S2 stream creation, or queue enqueue — no side effects in
either mode.
## How v3 vs v4 are distinguished
I pulled the published CLI tarballs for `trigger.dev@3.3.7`, `4.0.0`,
`4.0.1`, `4.0.5`, `4.1.0`, `4.2.0`, and the current `4.4.4` in the repo.
The cleanest, most reliable signal is the request body to `POST
/api/v1/deployments`:
| Field on initialize | v3.3.7 CLI | v4.x CLI |
|---|---|---|
| `type` | **never sent** | always sent — `"MANAGED"` (run_engine_v2) or
`"V1"` |
| `isNativeBuild` / `gitMeta` / `triggeredVia` / `runtime` | not sent |
sent |
| `registryHost` / `namespace` | sent (v3-only; stripped by current Zod
schema) | not sent |
Every v4 call site I inspected sets `type: features.run_engine_v2 ?
"MANAGED" : "V1"` unconditionally. `payload.type` is `undefined` if and
only if the client is a 3.x CLI.
## Behavior
- Detection always runs and emits `logger.warn("Detected deploy from
deprecated v3 CLI", { environmentId, projectId, organizationId, enforced
})`, which lets us watch how many v3 deploys are still happening before
enforcement is flipped.
- Enforcement is gated behind `DEPRECATE_V3_CLI_DEPLOYS_ENABLED`
(default `"0"`, off). When `"1"`, the server returns `400` with:
> The trigger.dev CLI v3 is no longer supported for deployments. Please
upgrade your project to v4: https://trigger.dev/docs/migrating-from-v3
The v3 CLI surfaces this verbatim as `Failed to start deployment:
<message>` because `zodfetch` throws `ApiError` for non-retryable 4xx
(400/422) and `deploy.js` in 3.3.7 prints `error.message`.
## Out of scope (intentionally)
- `api.v1.deployments.$deploymentId.finalize.ts` /
`FinalizeDeploymentService` /
`createDeploymentBackgroundWorkerV3.server.ts` are V1-engine paths, not
the v3 CLI gate. Leaving them alone per review.
- Container-side `createDeploymentBackgroundWorker` call in
`managed-index-controller.ts` is still used by v4's in-image indexer.
Not touched.
- v3 `trigger dev` flow (different code path) — separate deprecation
if/when needed.
## Testing
- Ran `pnpm run typecheck --filter webapp` locally — passes.
- Verified v4 tarballs (4.0.0, 4.0.1, 4.0.5, 4.1.0, 4.2.0, 4.4.4) all
include `type:` in the `initializeDeployment` call site, so none will be
accidentally blocked.
- Verified v3.3.7 tarball's `initializeDeployment` payload has no `type`
field.
Rollout plan after merge:
1. Deploy with `DEPRECATE_V3_CLI_DEPLOYS_ENABLED` unset → watch
`Detected deploy from deprecated v3 CLI` log volume.
2. When comfortable, set `DEPRECATE_V3_CLI_DEPLOYS_ENABLED=1` to
enforce.
---
## Changelog
Detect v3 CLI deploys on `/api/v1/deployments` and, when
`DEPRECATE_V3_CLI_DEPLOYS_ENABLED=1`, reject them with an upgrade
message pointing at https://trigger.dev/docs/migrating-from-v3. v4 CLI
deploys are unaffected.
Link to Devin session:
https://app.devin.ai/sessions/b242c11bd86e4099aeec8b59bab62143
Requested by: @ericallam
Example cURL call using an admin user PAT (replace with a real one):
```sh
curl -X PUT https://cloud.trigger.dev/admin/api/v1/environments/<environmentId>/burst-factor \
-H "Authorization: Bearer tr_pat_1234" \
-H "Content-Type: application/json" \
-d '{"burstFactor": 1.5}'
```
Matt Aitken