refactor(webapp): extract shared TaskRunsList for both task pages

## Summary

When a `chat.agent` (or `chat.createSession`) turn's model stream fails
mid-response (e.g. a transport timeout like `UND_ERR_BODY_TIMEOUT`), the
assistant output that already streamed was dropped: `onTurnComplete`
fired with `responseMessage: undefined`, and the manual loop's
`turn.complete()` rethrew without keeping the partial. Apps that
register `hydrateMessages` are hit hardest, since boot-time tail-replay
recovery is off by design.

This preserves the streamed-so-far assistant output while still
reporting the turn as errored, so persistence and recovery keep it.

## Scope of behavior change

Only the **error path** changes. Successful turns are unaffected: the
same chunks stream to the client in the same order, and
backpressure/cancel behave as before. Everything here is a correctness
improvement on a turn that hit a source-stream failure.

## What it does

Follow-up to #4304 (`chat.pipeAndCapture`), extending the same
partial-recovery to the two loops that lacked it:

- **`chat.agent`**: taps the response stream (via a `TransformStream`,
so pass-through backpressure and cancel are preserved) to buffer chunks,
and on a source-stream failure reconstructs the partial (preferring the
`onFinish` message). It's surfaced on the error-path `onTurnComplete`
(`responseMessage`, `rawResponseMessage`, `uiMessages`, `newUIMessages`,
`newMessages`) and committed to the accumulator so the next turn and the
reboot snapshot keep it.
- **`chat.createSession` / `turn.complete()`**: the reconstructed
partial is accumulated (so `turn.uiMessages` reflects it and the caller
can persist after catching) before `turn.complete()` rethrows.

`onBeforeTurnComplete` stays skipped on the error path (it hands out a
writer for a stream that has already broken).

## Correctness properties (each covered by a regression test)

Each test below was confirmed to fail without its fix:

- The recovered partial reaches `onTurnComplete` and the next turn's
accumulated messages.
- An already-committed (possibly enriched) response is not overwritten
if a post-response hook then throws.
- Incomplete tool parts are cleaned from the recovered partial (text
kept), so the UI and model views agree and the next turn isn't poisoned.
- A prior turn's model-only compaction survives an errored turn (append
only the new tail, don't reconvert the full history).
- A reconstructed fragment that reuses an existing message id does not
clobber the complete message.
- Queued `chat.response` data parts are folded into the recovered
partial, matching the success path.
- `newMessages` (model delta) stays symmetric with `newUIMessages`.

## Tests

New `chat-agent-source-stream-error.test.ts` covers the cases above. The
full `@trigger.dev/sdk` unit suite passes and the package build is green
across all supported runtimes (Node 20 to 26, Bun, Deno, Cloudflare
Workers).

.changeset/chat-agent-preserve-partial-on-stream-failure.md

@@ -0,0 +1,5 @@
+---
+"@trigger.dev/sdk": patch
+---
+
+Preserve the partial assistant message when a chat turn's model stream fails mid-response. `chat.agent` now passes the recovered partial to `onTurnComplete`, and `chat.createSession`'s `turn.complete()` keeps it before rethrowing, instead of dropping the streamed-so-far output.

.changeset/sync-env-vars-is-secret.md

@@ -1,5 +0,0 @@
----
-"@trigger.dev/build": patch
----
-
-You can now mark environment variables synced via the `syncEnvVars` build extension as secrets. Return `{ name, value, isSecret: true }` from your callback and those variables are stored redacted in the dashboard, just like manually created secret env vars.

.claude/rules/legacy-v3-code.md

@@ -3,31 +3,24 @@ paths:
   - "apps/webapp/app/v3/**"
 ---
 
-# Legacy V1 Engine Code in `app/v3/`
+# v3 (engine V1) has been removed
 
-The `v3/` directory name is misleading - most code here is actively used by the current V2 engine. Only the specific files below are legacy V1-only code.
+The v3 engine (RunEngineVersion `V1`: MarQS queue + Graphile worker) is end-of-life and its execution code has been removed from the webapp. The `app/v3/` directory name is historical: everything under it now serves the current V2 engine (`@internal/run-engine` + `@trigger.dev/redis-worker`).
 
-## V1-Only Files - Never Modify
+There is no `V1` execution path anymore. If you find a `RunEngineVersion` branch, the `V1` arm should only reject or finalize gracefully (for example, mark a historical run cancelled in the DB), never run V1 work. Do not reintroduce MarQS, the graphile worker, or the v3 socket.io namespaces.
 
-- `marqs/` directory (entire MarQS queue system: sharedQueueConsumer, devQueueConsumer, fairDequeuingStrategy, devPubSub)
-- `legacyRunEngineWorker.server.ts` (V1 background job worker)
-- `services/triggerTaskV1.server.ts` (deprecated V1 task triggering)
-- `services/cancelTaskRunV1.server.ts` (deprecated V1 cancellation)
-- `authenticatedSocketConnection.server.ts` (V1 dev WebSocket using DevQueueConsumer)
-- `sharedSocketConnection.ts` (V1 shared queue socket using SharedQueueConsumer)
+## The deprecation boundary (keep this)
 
-## V1/V2 Branching Pattern
+Requests from clients still on v3 (old SDK/CLI) or historical V1 runs must return a clean 4xx, never a 5xx. The boundary lives in:
 
-Some services act as routers that branch on `RunEngineVersion`:
-- `services/cancelTaskRun.server.ts` - calls V1 service or `engine.cancelRun()` for V2
-- `services/batchTriggerV3.server.ts` - uses marqs for V1 path, run-engine for V2
+- `engineDeprecation.server.ts` - the `V3_TRIGGER_DEPRECATION_MESSAGE` / `V3_DEV_DEPRECATION_MESSAGE` / `V3_MIGRATION_URL` upgrade messages.
+- `engineVersion.server.ts` - `determineEngineVersion()` still detects a V1 project/run so callers can reject it.
+- `services/triggerTask.server.ts`, `services/cancelTaskRun.server.ts`, `services/rescheduleTaskRun.server.ts` - the `V1` arm rejects or finalizes gracefully instead of executing.
+- `services/initializeDeployment.server.ts` - the `DEPRECATE_V3_CLI_DEPLOYS_ENABLED`-gated v3 CLI deploy rejection.
+- `handleWebsockets.server.ts` - the legacy `trigger dev` websocket closes with the upgrade message.
 
-When editing these shared services, only modify V2 code paths.
+## V2 modern stack
 
-## V2 Modern Stack
-
-- **Run lifecycle**: `@internal/run-engine` (internal-packages/run-engine)
-- **Background jobs**: `@trigger.dev/redis-worker` (not graphile-worker/zodworker)
-- **Queue operations**: RunQueue inside run-engine (not MarQS)
-- **V2 engine singleton**: `runEngine.server.ts`, `runEngineHandlers.server.ts`
-- **V2 workers**: `commonWorker.server.ts`, `alertsWorker.server.ts`, `batchTriggerWorker.server.ts`
+- **Run lifecycle**: `@internal/run-engine` (`runEngine.server.ts`, `runEngineHandlers.server.ts`)
+- **Background jobs**: `@trigger.dev/redis-worker` (`commonWorker.server.ts`, `alertsWorker.server.ts`, `batchTriggerWorker.server.ts`; `legacyRunEngineWorker.server.ts` still hosts the live batch-completion jobs)
+- **Queue operations**: RunQueue inside run-engine (`runQueue.server.ts`), not MarQS

.claude/rules/server-apps.md

@@ -14,10 +14,12 @@ area: webapp
 type: fix
 ---
 
-Brief description of what changed and why.
+Fix pages occasionally loading unstyled during deploys. The dashboard now recovers automatically.
 EOF
 ```
 
 - **area**: `webapp` | `supervisor`
 - **type**: `feature` | `fix` | `improvement` | `breaking`
 - If the PR also touches `packages/`, just the changeset is sufficient (no `.server-changes/` needed).
+
+The body ships **verbatim in user-facing release notes**. Keep it to 1–2 short sentences, non-technical, written for a dashboard user: describe what changed for them, never the implementation (no header names, endpoints, middleware, storage mechanisms, internal tools). See `.server-changes/README.md` for full guidance.

.configs/tsconfig.base.json

@@ -26,7 +26,6 @@
     "esModuleInterop": true,
     "emitDecoratorMetadata": false,
     "experimentalDecorators": false,
-    "downlevelIteration": true,
     "isolatedModules": true,
     "noUncheckedIndexedAccess": true,
 

.env.example

@@ -2,11 +2,12 @@
 SESSION_SECRET=abcdef1234
 MAGIC_LINK_SECRET=abcdef1234
 ENCRYPTION_KEY=ae13021afef0819c3a307ad487071c06 # Must be a random 16 byte hex string. You can generate an encryption key by running `openssl rand -hex 16` in your terminal
+MANAGED_WORKER_SECRET=abcdef1234 # Must match the supervisor's MANAGED_WORKER_SECRET
 LOGIN_ORIGIN=http://localhost:3030
 DATABASE_URL=postgresql://postgres:postgres@localhost:5432/postgres?schema=public
 # This sets the URL used for direct connections to the database and should only be needed in limited circumstances
 # See: https://www.prisma.io/docs/reference/api-reference/prisma-schema-reference#fields:~:text=the%20shadow%20database.-,directUrl,-No
-DIRECT_URL=${DATABASE_URL}
+DIRECT_URL=postgresql://postgres:postgres@localhost:5432/postgres?schema=public
 # Dedicated run-ops database (@internal/run-ops-database). Only needed to run prisma commands
 # against it or to enable the run-ops split; start it with `docker compose --profile runops up`.
 RUN_OPS_DATABASE_URL=postgresql://postgres:postgres@localhost:5434/postgres?schema=public
@@ -166,4 +167,4 @@ POSTHOG_PROJECT_KEY=
 # Uncomment these to send metrics to the local Prometheus via OTEL Collector:
 # INTERNAL_OTEL_METRIC_EXPORTER_ENABLED=1
 # INTERNAL_OTEL_METRIC_EXPORTER_URL=http://localhost:4318/v1/metrics
-# INTERNAL_OTEL_METRIC_EXPORTER_INTERVAL_MS=15000
+# INTERNAL_OTEL_METRIC_EXPORTER_INTERVAL_MS=15000

.github/workflows/changesets-pr.yml

@@ -36,7 +36,7 @@ jobs:
       - name: Setup node
         uses: actions/setup-node@48b55a011bda9f5d6aeb4c2d9c7362e8dae4041e # v6.4.0
         with:
-          node-version: 22.23.1
+          node-version: 24.18.0
           cache: "pnpm"
 
       - name: Install dependencies

.github/workflows/claude.yml

@@ -44,7 +44,7 @@ jobs:
       - name: ⎔ Setup node
         uses: actions/setup-node@48b55a011bda9f5d6aeb4c2d9c7362e8dae4041e # v6.4.0
         with:
-          node-version: 22.23.1
+          node-version: 24.18.0
           cache: "pnpm"
 
       - name: 📥 Download deps

.github/workflows/code-quality.yml

@@ -25,7 +25,7 @@ jobs:
       - name: ⎔ Setup node
         uses: actions/setup-node@48b55a011bda9f5d6aeb4c2d9c7362e8dae4041e # v6.4.0
         with:
-          node-version: 22.23.1
+          node-version: 24.18.0
           cache: "pnpm"
 
       - name: 📥 Download deps

.github/workflows/dashboard-agent-deploy.yml

@@ -51,7 +51,7 @@ jobs:
       - name: Setup node
         uses: actions/setup-node@48b55a011bda9f5d6aeb4c2d9c7362e8dae4041e # v6.4.0
         with:
-          node-version: 22.23.1
+          node-version: 24.18.0
           cache: "pnpm"
 
       - name: Install + build the CLI and the agent's deps

.github/workflows/e2e-webapp-auth-full.yml

@@ -85,7 +85,7 @@ jobs:
       - name: ⎔ Setup node
         uses: actions/setup-node@48b55a011bda9f5d6aeb4c2d9c7362e8dae4041e # v6.4.0
         with:
-          node-version: 22.23.1
+          node-version: 24.18.0
           cache: "pnpm"
 
       - name: 🐳 Login to DockerHub

.github/workflows/e2e-webapp.yml

@@ -14,7 +14,7 @@ on:
 jobs:
   e2eTests:
     name: "🧪 E2E Tests: Webapp"
-    runs-on: warp-ubuntu-latest-x64-8x
+    runs-on: warp-ubuntu-latest-x64-16x
     timeout-minutes: 20
     env:
       DOCKERHUB_USERNAME: ${{ secrets.DOCKERHUB_USERNAME }}
@@ -59,7 +59,7 @@ jobs:
       - name: ⎔ Setup node
         uses: actions/setup-node@48b55a011bda9f5d6aeb4c2d9c7362e8dae4041e # v6.4.0
         with:
-          node-version: 22.23.1
+          node-version: 24.18.0
           cache: "pnpm"
 
       # ..to avoid rate limits when pulling images

.github/workflows/e2e.yml

@@ -20,7 +20,7 @@ jobs:
     strategy:
       fail-fast: false
       matrix:
-        os: [warp-ubuntu-latest-x64-4x, warp-windows-latest-x64-4x]
+        os: [warp-ubuntu-latest-x64-4x, warp-windows-latest-x64-8x]
         package-manager: ["npm", "pnpm"]
     steps:
       - name: ⬇️ Checkout repo
@@ -37,7 +37,7 @@ jobs:
       - name: ⎔ Setup node
         uses: actions/setup-node@48b55a011bda9f5d6aeb4c2d9c7362e8dae4041e # v6.4.0
         with:
-          node-version: 22.23.1
+          node-version: 24.18.0
 
       - name: 📥 Download deps
         run: pnpm install --frozen-lockfile --filter trigger.dev...

.github/workflows/helm-prerelease.yml

@@ -52,12 +52,14 @@ jobs:
 
       - name: Lint Helm Chart
         run: |
-          helm lint ./hosting/k8s/helm/
+          helm lint ./hosting/k8s/helm/ \
+            --values ./hosting/k8s/helm/ci/lint-values.yaml
 
       - name: Render templates
         run: |
           helm template test-release ./hosting/k8s/helm/ \
             --values ./hosting/k8s/helm/values.yaml \
+            --values ./hosting/k8s/helm/ci/lint-values.yaml \
             --output-dir ./helm-output
 
       - name: Validate manifests

.github/workflows/pr_checks.yml

@@ -56,6 +56,7 @@ jobs:
               - '.github/workflows/pr_checks.yml'
               - '.github/workflows/unit-tests-webapp.yml'
               - '.github/workflows/e2e-webapp.yml'
+              - '.github/workflows/runops-guard.yml'
               - '.configs/**'
               - 'package.json'
               - 'pnpm-lock.yaml'
@@ -111,6 +112,11 @@ jobs:
     if: needs.changes.outputs.code == 'true' || needs.changes.outputs.typecheck_self == 'true'
     uses: ./.github/workflows/typecheck.yml
 
+  runops-guard:
+    needs: changes
+    if: needs.changes.outputs.webapp == 'true'
+    uses: ./.github/workflows/runops-guard.yml
+
   webapp:
     needs: changes
     if: needs.changes.outputs.webapp == 'true'
@@ -161,6 +167,7 @@ jobs:
       - changes
       - code-quality
       - typecheck
+      - runops-guard
       - webapp
       - e2e-webapp
       - packages

.github/workflows/preview-packages.yml

@@ -59,7 +59,7 @@ jobs:
       - name: ⎔ Setup node
         uses: actions/setup-node@48b55a011bda9f5d6aeb4c2d9c7362e8dae4041e # v6.4.0
         with:
-          node-version: 22.23.1
+          node-version: 24.18.0
           cache: "pnpm"
 
       - name: 📥 Install dependencies

.github/workflows/publish-worker-v4.yml

@@ -13,6 +13,13 @@ on:
         type: string
         required: false
         default: ""
+    outputs:
+      version:
+        description: The published image tag
+        value: ${{ jobs.build.outputs.version }}
+      image_repo:
+        description: The image repository the build was published to (without tag)
+        value: ${{ jobs.build.outputs.image_repo }}
   push:
     tags:
       - "re2-test-*"
@@ -38,6 +45,11 @@ jobs:
       matrix:
         package: [supervisor]
     runs-on: warp-ubuntu-latest-x64-2x
+    # Single-entry matrix, so these job outputs are unambiguous (consumed by the
+    # scan-supervisor job in publish.yml).
+    outputs:
+      version: ${{ steps.get_tag.outputs.tag }}
+      image_repo: ${{ steps.set_tags.outputs.image_repo }}
     env:
       DOCKER_BUILDKIT: "1"
     steps:
@@ -81,6 +93,7 @@ jobs:
           fi
 
           echo "image_tags=${image_tags}" >> "$GITHUB_OUTPUT"
+          echo "image_repo=${ref_without_tag}" >> "$GITHUB_OUTPUT"
         env:
           IMAGE_REGISTRY: ${{ inputs.image_registry || vars.IMAGE_REGISTRY || format('ghcr.io/{0}', github.repository_owner) }}
           STEPS_GET_REPOSITORY_OUTPUTS_REPO: ${{ steps.get_repository.outputs.repo }}

.github/workflows/publish.yml

@@ -97,10 +97,19 @@ jobs:
     permissions:
       contents: read
       packages: read # pull the just-published image from GHCR
-    uses: ./.github/workflows/trivy-image-webapp.yml
+    uses: ./.github/workflows/trivy-image.yml
     with:
       image-ref: ${{ needs.publish-webapp.outputs.image_repo }}:${{ needs.publish-webapp.outputs.version }}
 
+  scan-supervisor:
+    needs: [publish-worker-v4]
+    permissions:
+      contents: read
+      packages: read # pull the just-published image from GHCR
+    uses: ./.github/workflows/trivy-image.yml
+    with:
+      image-ref: ${{ needs.publish-worker-v4.outputs.image_repo }}:${{ needs.publish-worker-v4.outputs.version }}
+
   # Announce the freshly published mutable `main` webapp image to subscriber
   # repos via repository_dispatch, handing them a digest-pinned ref to build or
   # deploy from. The repo, ref prefix, and dispatch target all default to the

.github/workflows/release-helm.yml

@@ -47,12 +47,14 @@ jobs:
 
       - name: Lint Helm Chart
         run: |
-          helm lint ./hosting/k8s/helm/
+          helm lint ./hosting/k8s/helm/ \
+            --values ./hosting/k8s/helm/ci/lint-values.yaml
 
       - name: Render templates
         run: |
           helm template test-release ./hosting/k8s/helm/ \
             --values ./hosting/k8s/helm/values.yaml \
+            --values ./hosting/k8s/helm/ci/lint-values.yaml \
             --output-dir ./helm-output
 
       - name: Validate manifests

.github/workflows/release.yml

@@ -48,7 +48,7 @@ jobs:
 
   release:
     name: 🚀 Release npm packages
-    runs-on: ubuntu-latest
+    runs-on: ubuntu-latest # this cannot run on non-GH runner
     environment: npm-publish
     permissions:
       contents: write
@@ -90,7 +90,7 @@ jobs:
       - name: Setup node
         uses: actions/setup-node@48b55a011bda9f5d6aeb4c2d9c7362e8dae4041e # v6.4.0
         with:
-          node-version: 22.23.1
+          node-version: 24.18.0
           cache: "pnpm"
 
       # npm v11.5.1 or newer is required for OIDC support
@@ -281,7 +281,7 @@ jobs:
   # The prerelease job needs to be on the same workflow file due to a limitation related to how npm verifies OIDC claims.
   prerelease:
     name: 🧪 Prerelease
-    runs-on: ubuntu-latest
+    runs-on: ubuntu-latest # this cannot run on non-GH runner
     environment: npm-publish
     permissions:
       contents: read
@@ -303,7 +303,7 @@ jobs:
       - name: Setup node
         uses: actions/setup-node@48b55a011bda9f5d6aeb4c2d9c7362e8dae4041e # v6.4.0
         with:
-          node-version: 22.23.1
+          node-version: 24.18.0
           cache: "pnpm"
 
       # npm v11.5.1 or newer is required for OIDC support

.github/workflows/runops-guard.yml

@@ -0,0 +1,38 @@
+name: "🛡️ Run-ops Legacy Guard"
+
+on:
+  workflow_call:
+
+permissions:
+  contents: read
+
+jobs:
+  runops-guard:
+    runs-on: warp-ubuntu-latest-x64-16x
+
+    steps:
+      - name: ⬇️ Checkout repo
+        uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
+        with:
+          fetch-depth: 0
+          persist-credentials: false
+
+      - name: ⎔ Setup pnpm
+        uses: pnpm/action-setup@fc06bc1257f339d1d5d8b3a19a8cae5388b55320 # v5.0.0
+        with:
+          version: 10.33.2
+
+      - name: ⎔ Setup node
+        uses: actions/setup-node@48b55a011bda9f5d6aeb4c2d9c7362e8dae4041e # v6.4.0
+        with:
+          node-version: 24.18.0
+          cache: "pnpm"
+
+      - name: 📥 Download deps
+        run: pnpm install --frozen-lockfile
+
+      - name: 📀 Generate Prisma Client
+        run: pnpm run generate
+
+      - name: 🛡️ Run-ops legacy guard
+        run: pnpm --filter webapp run guard:runops-legacy -- --check

.github/workflows/sdk-compat.yml

@@ -70,7 +70,7 @@ jobs:
       - name: ⎔ Setup node
         uses: actions/setup-node@48b55a011bda9f5d6aeb4c2d9c7362e8dae4041e # v6.4.0
         with:
-          node-version: 22.23.1
+          node-version: 24.18.0
           cache: "pnpm"
 
       - name: 🥟 Setup Bun
@@ -112,7 +112,7 @@ jobs:
       - name: ⎔ Setup node
         uses: actions/setup-node@48b55a011bda9f5d6aeb4c2d9c7362e8dae4041e # v6.4.0
         with:
-          node-version: 22.23.1
+          node-version: 24.18.0
           cache: "pnpm"
 
       - name: 🦕 Setup Deno
@@ -158,7 +158,7 @@ jobs:
       - name: ⎔ Setup node
         uses: actions/setup-node@48b55a011bda9f5d6aeb4c2d9c7362e8dae4041e # v6.4.0
         with:
-          node-version: 22.23.1
+          node-version: 24.18.0
           cache: "pnpm"
 
       - name: 📥 Download deps

.github/workflows/trivy-image.yml

@@ -1,7 +1,7 @@
-name: Trivy Image Scan (webapp)
+name: Trivy Image Scan
 
-# OS-level CVE scan of a published webapp image. Called by the publish pipeline
-# (publish.yml) to scan each build right after it's pushed to GHCR — so every
+# OS-level CVE scan of a published image. Called by the publish pipeline
+# (publish.yml) to scan each image right after it's pushed to GHCR — so every
 # main build and every release is scanned, not rebuilt. Also runnable ad-hoc
 # via workflow_dispatch against any image ref.
 #
@@ -27,7 +27,7 @@ on:
 permissions: {}
 
 concurrency:
-  group: trivy-image-webapp-${{ inputs.image-ref }}
+  group: trivy-image-${{ inputs.image-ref }}
   cancel-in-progress: true
 
 jobs:
@@ -59,7 +59,7 @@ jobs:
           ignore-unfixed: true
           severity: HIGH,CRITICAL
           format: table
-          output: trivy-image-webapp.txt
+          output: trivy-image.txt
 
       - name: Job summary
         if: always()
@@ -67,9 +67,9 @@ jobs:
           IMAGE_REF: ${{ inputs.image-ref }}
         run: |
           {
-            echo "## Trivy Image Scan (webapp) — \`${IMAGE_REF}\`"
+            echo "## Trivy Image Scan — \`${IMAGE_REF}\`"
             echo '```'
             # GitHub step summary is capped at 1 MiB; truncate large reports.
-            head -c 900000 trivy-image-webapp.txt 2>/dev/null || echo "(no report produced)"
+            head -c 900000 trivy-image.txt 2>/dev/null || echo "(no report produced)"
             echo '```'
           } >> "$GITHUB_STEP_SUMMARY"

.github/workflows/typecheck.yml

@@ -8,7 +8,7 @@ permissions:
 
 jobs:
   typecheck:
-    runs-on: warp-ubuntu-latest-x64-8x
+    runs-on: warp-ubuntu-latest-x64-16x
 
     steps:
       - name: ⬇️ Checkout repo
@@ -25,7 +25,7 @@ jobs:
       - name: ⎔ Setup node
         uses: actions/setup-node@48b55a011bda9f5d6aeb4c2d9c7362e8dae4041e # v6.4.0
         with:
-          node-version: 22.23.1
+          node-version: 24.18.0
           cache: "pnpm"
 
       - name: 📥 Download deps

.github/workflows/unit-tests-internal.yml

@@ -14,17 +14,14 @@ on:
 jobs:
   unitTests:
     name: "🧪 Unit Tests: Internal"
-    runs-on: warp-ubuntu-latest-x64-8x
-    strategy:
-      # one flaky shard shouldn't cancel its siblings - lets us re-run only the failed shard
-      fail-fast: false
-      matrix:
-        shardIndex: [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12]
-        shardTotal: [12]
+    # Single big machine instead of a 12-job matrix: the internal suites are serial
+    # (fileParallelism: false) and container-wait-bound, so 12 in-machine shard processes
+    # fit comfortably in 32 vCPUs while paying the setup cost (install, prisma generate,
+    # image pulls) once instead of 12 times.
+    runs-on: warp-ubuntu-latest-x64-32x
     env:
       DOCKERHUB_USERNAME: ${{ secrets.DOCKERHUB_USERNAME }}
-      SHARD_INDEX: ${{ matrix.shardIndex }}
-      SHARD_TOTAL: ${{ matrix.shardTotal }}
+      SHARD_TOTAL: 12
     steps:
       - name: 🔧 Disable IPv6
         run: |
@@ -66,7 +63,7 @@ jobs:
       - name: ⎔ Setup node
         uses: actions/setup-node@48b55a011bda9f5d6aeb4c2d9c7362e8dae4041e # v6.4.0
         with:
-          node-version: 22.23.1
+          node-version: 24.18.0
           cache: "pnpm"
 
       # ..to avoid rate limits when pulling images
@@ -108,8 +105,34 @@ jobs:
       - name: 📀 Generate Prisma Client
         run: pnpm run generate
 
-      - name: 🧪 Run Internal Unit Tests
-        run: pnpm run test:internal --reporter=default --reporter=blob --shard=${{ matrix.shardIndex }}/${{ matrix.shardTotal }} --passWithNoTests
+      - name: 🏗️ Build test dependencies
+        # Build once up-front so the parallel shard runs below (turbo --only) never race
+        # to build or cache-restore the same outputs concurrently.
+        run: pnpm exec turbo run build --filter "@internal/*..."
+
+      - name: 🧪 Run Internal Unit Tests (${{ env.SHARD_TOTAL }} in-machine shards)
+        run: |
+          # Same shard partitioning as the old 12-job matrix (DurationShardingSequencer
+          # keys off --shard=i/N), but as parallel local processes. --only skips the
+          # ^build dependency handled by the step above.
+          status=0
+          declare -a pids
+          for i in $(seq 1 "$SHARD_TOTAL"); do
+            pnpm exec turbo run test --only --concurrency=1 --filter "@internal/*" -- \
+              --run --reporter=default --reporter=blob --shard="$i/$SHARD_TOTAL" --passWithNoTests \
+              > "/tmp/internal-shard-$i.log" 2>&1 &
+            pids[i]=$!
+          done
+          for i in $(seq 1 "$SHARD_TOTAL"); do
+            if ! wait "${pids[i]}"; then
+              status=1
+              echo "::error::internal unit test shard $i/$SHARD_TOTAL failed"
+            fi
+            echo "::group::🧪 shard $i/$SHARD_TOTAL"
+            cat "/tmp/internal-shard-$i.log"
+            echo "::endgroup::"
+          done
+          exit "$status"
 
       - name: Gather all reports
         if: ${{ !cancelled() }}
@@ -118,44 +141,6 @@ jobs:
           find . -type f -path '*/.vitest-reports/blob-*.json' \
             -exec bash -c 'src="$1"; basename=$(basename "$src"); pkg=$(dirname "$src" | sed "s|^\./||;s|/\.vitest-reports$||;s|/|_|g"); cp "$src" ".vitest-reports/${pkg}-${basename}"' _ {} \;
 
-      - name: Upload blob reports to GitHub Actions Artifacts
+      - name: 📊 Merge reports
         if: ${{ !cancelled() }}
-        uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
-        with:
-          name: internal-blob-report-${{ matrix.shardIndex }}
-          path: .vitest-reports/*
-          include-hidden-files: true
-          retention-days: 1
-
-  merge-reports:
-    name: "📊 Merge Reports"
-    if: ${{ !cancelled() }}
-    needs: [unitTests]
-    runs-on: warp-ubuntu-latest-x64-2x
-    steps:
-      - name: ⬇️ Checkout repo
-        uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
-        with:
-          fetch-depth: 1
-          persist-credentials: false
-
-      - name: ⎔ Setup pnpm
-        uses: pnpm/action-setup@fc06bc1257f339d1d5d8b3a19a8cae5388b55320 # v5.0.0
-        with:
-          version: 10.33.2
-
-      - name: ⎔ Setup node
-        uses: actions/setup-node@48b55a011bda9f5d6aeb4c2d9c7362e8dae4041e # v6.4.0
-        with:
-          node-version: 22.23.1
-          # no cache enabled, we're not installing deps
-
-      - name: Download blob reports from GitHub Actions Artifacts
-        uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1
-        with:
-          path: .vitest-reports
-          pattern: internal-blob-report-*
-          merge-multiple: true
-
-      - name: Merge reports
         run: pnpm dlx vitest@4.1.7 run --merge-reports --pass-with-no-tests

.github/workflows/unit-tests-packages.yml

@@ -66,7 +66,7 @@ jobs:
       - name: ⎔ Setup node
         uses: actions/setup-node@48b55a011bda9f5d6aeb4c2d9c7362e8dae4041e # v6.4.0
         with:
-          node-version: 22.23.1
+          node-version: 24.18.0
           cache: "pnpm"
 
       # ..to avoid rate limits when pulling images
@@ -146,7 +146,7 @@ jobs:
       - name: ⎔ Setup node
         uses: actions/setup-node@48b55a011bda9f5d6aeb4c2d9c7362e8dae4041e # v6.4.0
         with:
-          node-version: 22.23.1
+          node-version: 24.18.0
           # no cache enabled, we're not installing deps
 
       - name: Download blob reports from GitHub Actions Artifacts

.github/workflows/unit-tests-webapp.yml

@@ -14,13 +14,18 @@ on:
 jobs:
   unitTests:
     name: "🧪 Unit Tests: Webapp"
-    runs-on: warp-ubuntu-latest-x64-8x
+    # 10 shards on 16x machines: webapp test throughput is limited per-machine (one
+    # docker daemon + disk absorbing all the per-file Postgres/ClickHouse container
+    # spin-up), so many machines beats few big ones - fewer/bigger (3x32) measured
+    # SLOWER than 10x8. The 16x (vs 8x) gives the fork pool the CPU headroom the 8x
+    # runners lacked. Setup overhead per machine is ~1 min on warm runners.
+    runs-on: warp-ubuntu-latest-x64-16x
     strategy:
       # one flaky shard shouldn't cancel its siblings - lets us re-run only the failed shard
       fail-fast: false
       matrix:
-        shardIndex: [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]
-        shardTotal: [10]
+        shardIndex: [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12]
+        shardTotal: [12]
     env:
       DOCKERHUB_USERNAME: ${{ secrets.DOCKERHUB_USERNAME }}
       SHARD_INDEX: ${{ matrix.shardIndex }}
@@ -66,7 +71,7 @@ jobs:
       - name: ⎔ Setup node
         uses: actions/setup-node@48b55a011bda9f5d6aeb4c2d9c7362e8dae4041e # v6.4.0
         with:
-          node-version: 22.23.1
+          node-version: 24.18.0
           cache: "pnpm"
 
       # ..to avoid rate limits when pulling images
@@ -155,7 +160,7 @@ jobs:
       - name: ⎔ Setup node
         uses: actions/setup-node@48b55a011bda9f5d6aeb4c2d9c7362e8dae4041e # v6.4.0
         with:
-          node-version: 22.23.1
+          node-version: 24.18.0
           # no cache enabled, we're not installing deps
 
       - name: Download blob reports from GitHub Actions Artifacts

.gitignore

@@ -76,3 +76,6 @@ apps/**/public/build
 ailogger-output.log
 # per-package vitest timing capture (transient; merged into root test-timings.json)
 .vitest-timing.json
+
+# local git worktree checkouts (not source) — keeps oxfmt/oxlint from descending into them
+.worktrees/

.nvmrc

@@ -1 +1 @@
-v22.23.1
+v24.18.0

.oxlintrc.json

@@ -1,7 +1,10 @@
 {
   "$schema": "./node_modules/oxlint/configuration_schema.json",
   "plugins": ["typescript", "import", "react"],
-  "jsPlugins": ["./oxlint-plugins/no-thrown-unawaited-redirect.mjs"],
+  "jsPlugins": [
+    "./oxlint-plugins/no-thrown-unawaited-redirect.mjs",
+    "./oxlint-plugins/runops-residency.mjs"
+  ],
   "ignorePatterns": [
     "**/dist/**",
     "**/build/**",
@@ -34,5 +37,21 @@
     "react-hooks/exhaustive-deps": "off",
     "react-hooks/rules-of-hooks": "off",
     "trigger/no-thrown-unawaited-redirect": "error"
-  }
+  },
+  "overrides": [
+    {
+      "files": ["apps/webapp/app/**/*.ts", "apps/webapp/app/**/*.tsx"],
+      "rules": {
+        "trigger-runops/no-control-plane-run-graph-access": "error",
+        "trigger-runops/no-control-plane-in-runops-slot": "error"
+      }
+    },
+    {
+      "files": ["apps/webapp/app/**/*.test.ts", "apps/webapp/app/**/*.test.tsx"],
+      "rules": {
+        "trigger-runops/no-control-plane-run-graph-access": "off",
+        "trigger-runops/no-control-plane-in-runops-slot": "off"
+      }
+    }
+  ]
 }

.server-changes/batch-list-created-at-index.md

@@ -0,0 +1,6 @@
+---
+area: webapp
+type: fix
+---
+
+Speed up the Batches list page for environments with a large number of batches, which could previously time out while loading.

.server-changes/cap-list-endpoint-page-size.md

@@ -0,0 +1,6 @@
+---
+area: webapp
+type: improvement
+---
+
+List API endpoints now clamp the page size to a maximum of 100. Requests asking for a larger page size return up to 100 items and keep paginating, rather than pulling an unbounded page.

.server-changes/clearer-whitelist-login-error.md

@@ -1,6 +0,0 @@
----
-area: webapp
-type: improvement
----
-
-Clearer login error when an email address is blocked by the WHITELISTED_EMAILS setting: the message now explains the address isn't allowed on this instance instead of the ambiguous "This email is unauthorized".

.server-changes/consume-stale-login-errors.md

@@ -1,6 +0,0 @@
----
-area: webapp
-type: fix
----
-
-Fixed stale login errors: an error from a previous login attempt (for example a rejected email address) no longer keeps reappearing on the login page and no longer makes later, successful attempts look like they failed.

.server-changes/default-billing-alerts.md

@@ -0,0 +1,6 @@
+---
+area: webapp
+type: improvement
+---
+
+Organizations without billing alerts now get default spend alert thresholds, so you're notified before usage grows unexpectedly. The billing limit page no longer pre-selects an option before you've set a limit and prompts you to configure one. Alert previews now update immediately after you change your billing limit.

.server-changes/entrypoint-no-log-db-connection-strings.md

@@ -0,0 +1,6 @@
+---
+area: webapp
+type: fix
+---
+
+Container startup no longer prints database and ClickHouse connection strings (with credentials) to the logs.

.server-changes/pat-token-reveal-last-chars.md

@@ -0,0 +1,6 @@
+---
+area: webapp
+type: improvement
+---
+
+When you create a Personal Access Token, the generated token now shows its first and last few characters instead of being fully hidden, so you can confirm you copied the right value.

.server-changes/realtime-emission-fanout-metrics.md

@@ -0,0 +1,6 @@
+---
+area: webapp
+type: improvement
+---
+
+Add metrics to the realtime backend that measure how often a single changed run is served to multiple subscriptions in one batch.

.server-changes/supervisor-outbound-request-metrics.md

@@ -0,0 +1,6 @@
+---
+area: supervisor
+type: improvement
+---
+
+Improved supervisor observability: it now reports metrics for its outbound requests, making failed calls to upstream services easier to monitor.

.server-changes/task-page-runs-live-updating.md

@@ -0,0 +1,6 @@
+---
+area: webapp
+type: improvement
+---
+
+The runs list on a task's page now updates live — run statuses change and newly triggered runs appear without a manual refresh, matching the main Runs page.

.server-changes/trql-json-extract-on-output.md

@@ -1,6 +0,0 @@
----
-area: webapp
-type: fix
----
-
-Query page: extracting fields from a run's output with JSON functions (such as JSONExtractString or JSONExtractInt) no longer fails with an "illegal type: JSON" error.

AGENTS.md

@@ -138,11 +138,10 @@ User API call -> Webapp routes -> Services -> RunEngine -> Redis Queue -> Superv
 - **internal-packages/redis**: Redis client creation utilities (ioredis)
 - **internal-packages/testcontainers**: Test helpers for Redis/PostgreSQL containers
 - **internal-packages/schedule-engine**: Durable cron scheduling
-- **internal-packages/zod-worker**: Graphile-worker wrapper (DEPRECATED - use redis-worker)
 
-### Legacy V1 Engine Code
+### v3 (engine V1) removed
 
-The `apps/webapp/app/v3/` directory name is misleading - most code there is actively used by V2. Only specific files are V1-only legacy (MarQS queue, triggerTaskV1, cancelTaskRunV1, etc.). See `apps/webapp/CLAUDE.md` for the exact list. When you encounter V1/V2 branching in services, only modify V2 code paths. All new work uses Run Engine 2.0 (`@internal/run-engine`) and redis-worker.
+v3 (engine V1: MarQS + Graphile worker) is end-of-life and its execution code has been removed. The `apps/webapp/app/v3/` directory name is historical - everything there now serves V2 (Run Engine 2.0, `@internal/run-engine` + redis-worker). There is no V1 execution path: a `RunEngineVersion` `V1` branch only rejects or finalizes gracefully so v3 clients get a clean 4xx, never a 5xx. Do not reintroduce V1. See `apps/webapp/CLAUDE.md` and `.claude/rules/legacy-v3-code.md`.
 
 ### Documentation
 

CONTRIBUTING.md

@@ -29,7 +29,7 @@ branch are tagged into a release periodically.
 
 ### Prerequisites
 
-- [Node.js](https://nodejs.org/en) version 22.23.1
+- [Node.js](https://nodejs.org/en) version 24.18.0
 - [pnpm package manager](https://pnpm.io/installation) version 10.33.2
 - [Docker](https://www.docker.com/get-started/)
 - [protobuf](https://github.com/protocolbuffers/protobuf)
@@ -49,7 +49,7 @@ branch are tagged into a release periodically.
    ```
    cd trigger.dev
    ```
-3. Ensure you are on the correct version of Node.js (22.23.1). If you are using `nvm`, there is an `.nvmrc` file that will automatically select the correct version of Node.js when you navigate to the repository.
+3. Ensure you are on the correct version of Node.js (24.18.0). If you are using `nvm`, there is an `.nvmrc` file that will automatically select the correct version of Node.js when you navigate to the repository.
 
 4. Run `corepack enable` to use the correct version of pnpm (`10.33.2`) as specified in the root `package.json` file.
 
@@ -181,7 +181,7 @@ pnpm exec trigger dev --log-level debug
 
 6. Navigate to the `hello-world` project in your local dashboard at localhost:3030 and you should see the list of tasks.
 
-7. Go to the "Test" page in the sidebar and select a task. Then enter a payload and click "Run test". You can tell what the payloads should be by looking at the relevant task file inside the reference project's `src/trigger` folder. Many of them accept an empty payload.
+7. On the Tasks page, open a task and press the "Test" button to open its test page. Then enter a payload and click "Run test". You can tell what the payloads should be by looking at the relevant task file inside the reference project's `src/trigger` folder. Many of them accept an empty payload.
 
 8. Feel free to add additional files in the reference project's `src/trigger` dir to test out specific aspects of the system, or add in edge cases.
 

ai/references/repo.md

@@ -23,7 +23,6 @@ This is a pnpm 10.33.2 monorepo that uses turborepo @turbo.json. The following w
 - <root>/internal-packages/run-engine is the `@internal/run-engine` package that is "Run Engine 2.0" and handles moving a run all the way through it's lifecycle
 - <root>/internal-packages/redis is the `@internal/redis` package that exports Redis types and the `createRedisClient` function to unify how we create redis clients in the repo. It's not used everywhere yet, but it's the preferred way to create redis clients from now on.
 - <root>/internal-packages/testcontainers is the `@internal/testcontainers` package that exports a few useful functions for spinning up local testcontainers when writing vitest tests. See our [tests.md](./tests.md) file for more information.
-- <root>/internal-packages/zodworker is the `@internal/zodworker` package that implements a wrapper around graphile-worker that allows us to use zod to validate our background jobs. We are moving away from using graphile-worker as our background job system, replacing it with our own redis-worker package.
 
 ## References
 

apps/supervisor/.env.example

@@ -1,8 +1,8 @@
 # This needs to match the token of the worker group you want to connect to
 TRIGGER_WORKER_TOKEN=
 
-# This needs to match the MANAGED_WORKER_SECRET env var on the webapp
-MANAGED_WORKER_SECRET=managed-secret
+# Must match the webapp's MANAGED_WORKER_SECRET. Generate with: openssl rand -hex 16
+MANAGED_WORKER_SECRET=
 
 # Point this at the webapp in prod
 TRIGGER_API_URL=http://localhost:3030

apps/supervisor/.nvmrc

@@ -1 +1 @@
-v22.23.1
+v24.18.0

apps/supervisor/CLAUDE.md

@@ -7,7 +7,7 @@ Node.js app that manages task execution containers. Receives work from the platf
 - `src/services/` - Core service logic
 - `src/workloadManager/` - Container orchestration abstraction (Docker or Kubernetes)
 - `src/workloadServer/` - HTTP server for workload communication (heartbeats, snapshots)
-- `src/clients/` - Platform communication (webapp/coordinator)
+- `src/clients/` - Platform communication (webapp)
 - `src/env.ts` - Environment configuration
 
 ## Architecture

apps/supervisor/Containerfile

@@ -1,13 +1,13 @@
-FROM node:22-alpine@sha256:9bef0ef1e268f60627da9ba7d7605e8831d5b56ad07487d24d1aa386336d1944 AS node-22-alpine
+FROM node:24.18.0-alpine@sha256:a0b9bf06e4e6193cf7a0f58816cc935ff8c2a908f81e6f1a95432d679c54fbfd AS node-24-alpine
 
 WORKDIR /app
 
-FROM node-22-alpine AS pruner
+FROM node-24-alpine AS pruner
 
 COPY --chown=node:node . .
 RUN npx -q turbo@2.10.0 prune --scope=supervisor --docker
 
-FROM node-22-alpine AS base
+FROM node-24-alpine AS base
 
 RUN apk add --no-cache dumb-init
 

apps/supervisor/src/env.ts

@@ -14,8 +14,17 @@ export const Env = z
 
     // Required settings
     TRIGGER_API_URL: z.string().url(),
-    TRIGGER_WORKER_TOKEN: z.string(), // accepts file:// path to read from a file
+    TRIGGER_WORKER_TOKEN: z.string().min(1), // accepts file:// path to read from a file
     MANAGED_WORKER_SECRET: z.string(),
+
+    // Deployment token: sign a token into TRIGGER_DEPLOYMENT_ID at pod creation and verify it on
+    // inbound workload calls. "disabled" = off; "log" = mint + verify + metrics only; "enforce" =
+    // also reject invalid tokens.
+    WORKLOAD_TOKEN_SECRET: z.string().optional(),
+    WORKLOAD_TOKEN_ENFORCEMENT: z.enum(["disabled", "log", "enforce"]).default("disabled"),
+    // Absolute expiry for minted deployment tokens. Deterministic (no wall-clock issued-at) so every
+    // pod of a deployment carries an identical token; bump before this date. Must outlive any run.
+    WORKLOAD_TOKEN_EXP: z.string().datetime().default("2032-01-01T00:00:00.000Z"),
     OTEL_EXPORTER_OTLP_ENDPOINT: z.string().url(), // set on the runners
 
     // Workload API settings (coordinator mode) - the workload API is what the run controller connects to
@@ -103,6 +112,7 @@ export const Env = z
 
     // Optional services
     TRIGGER_WARM_START_URL: z.string().optional(),
+    TRIGGER_WARM_START_DISPATCH_URL: z.string().optional(),
     TRIGGER_CHECKPOINT_URL: z.string().optional(),
     TRIGGER_METADATA_URL: z.string().optional(),
 
@@ -365,6 +375,14 @@ export const Env = z
         path: ["TRIGGER_WORKLOAD_API_DOMAIN"],
       });
     }
+    if (data.WORKLOAD_TOKEN_ENFORCEMENT !== "disabled" && !data.WORKLOAD_TOKEN_SECRET) {
+      ctx.addIssue({
+        code: z.ZodIssueCode.custom,
+        message:
+          "WORKLOAD_TOKEN_SECRET is required when WORKLOAD_TOKEN_ENFORCEMENT is not disabled",
+        path: ["WORKLOAD_TOKEN_SECRET"],
+      });
+    }
     if (
       data.TRIGGER_DEQUEUE_BACKPRESSURE_ENABLED &&
       !data.TRIGGER_DEQUEUE_BACKPRESSURE_REDIS_HOST

apps/supervisor/src/index.ts

@@ -22,11 +22,12 @@ import {
   isKubernetesEnvironment,
 } from "@trigger.dev/core/v3/serverOnly";
 import { createK8sApi, createApiserverMetricsFetcher } from "./clients/kubernetes.js";
-import { collectDefaultMetrics, Gauge, Histogram } from "prom-client";
+import { collectDefaultMetrics, Counter, Gauge, Histogram } from "prom-client";
 import { register } from "./metrics.js";
 import { PodCleaner } from "./services/podCleaner.js";
 import { FailedPodHandler } from "./services/failedPodHandler.js";
 import { getWorkerToken } from "./workerToken.js";
+import { mintDeploymentToken } from "./workloadToken.js";
 import { OtlpTraceService } from "./services/otlpTraceService.js";
 import {
   WarmStartVerificationService,
@@ -59,6 +60,21 @@ const workloadCreateDuration = new Histogram({
   registers: [register],
 });
 
+const outboundRequestsTotal = new Counter({
+  name: "supervisor_outbound_request_total",
+  help: "Count of outbound HTTP requests from the supervisor, by target name, method, response status, and outcome (ok, http_error, invalid_response, network_error).",
+  labelNames: ["name", "method", "status", "outcome"],
+  registers: [register],
+});
+
+const outboundRequestDuration = new Histogram({
+  name: "supervisor_outbound_request_duration_seconds",
+  help: "Duration of outbound HTTP requests from the supervisor, by target name and outcome. Includes the HTTP client's internal retries and backoff.",
+  labelNames: ["name", "outcome"],
+  buckets: [0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1, 2, 5, 10, 11, 12.5, 15, 20, 30, 60],
+  registers: [register],
+});
+
 class ManagedSupervisor {
   private readonly workerSession: SupervisorSession;
   private readonly metricsServer?: HttpServer;
@@ -79,6 +95,8 @@ class ManagedSupervisor {
 
   private readonly isKubernetes = isKubernetesEnvironment(env.KUBERNETES_FORCE_ENABLED);
   private readonly warmStartUrl = env.TRIGGER_WARM_START_URL;
+  private readonly warmStartDispatchUrl =
+    env.TRIGGER_WARM_START_DISPATCH_URL ?? env.TRIGGER_WARM_START_URL;
 
   private readonly wideEventOpts: WideEventOptions = {
     service: "supervisor",
@@ -96,6 +114,7 @@ class ManagedSupervisor {
       COMPUTE_GATEWAY_AUTH_TOKEN,
       DOCKER_REGISTRY_PASSWORD,
       TRIGGER_DEQUEUE_BACKPRESSURE_REDIS_PASSWORD,
+      WORKLOAD_TOKEN_SECRET,
       ...envWithoutSecrets
     } = env;
 
@@ -120,6 +139,7 @@ class ManagedSupervisor {
       snapshotPollIntervalSeconds: env.RUNNER_SNAPSHOT_POLL_INTERVAL_SECONDS,
       additionalEnvVars: env.RUNNER_ADDITIONAL_ENV_VARS,
       dockerAutoremove: env.DOCKER_AUTOREMOVE_EXITED_CONTAINERS,
+      checkpointsEnabled: !!env.TRIGGER_CHECKPOINT_URL,
     } satisfies WorkloadManagerOptions;
 
     this.resourceMonitor = env.RESOURCE_MONITOR_ENABLED
@@ -289,8 +309,10 @@ class ManagedSupervisor {
       });
     }
 
+    const workerToken = getWorkerToken();
+
     this.workerSession = new SupervisorSession({
-      workerToken: getWorkerToken(),
+      workerToken,
       apiUrl: env.TRIGGER_API_URL,
       instanceName: env.TRIGGER_WORKER_INSTANCE_NAME,
       managedWorkerSecret: env.MANAGED_WORKER_SECRET,
@@ -317,6 +339,10 @@ class ManagedSupervisor {
       runNotificationsEnabled: env.TRIGGER_WORKLOAD_API_ENABLED,
       heartbeatIntervalSeconds: env.TRIGGER_WORKER_HEARTBEAT_INTERVAL_SECONDS,
       sendRunDebugLogs: env.SEND_RUN_DEBUG_LOGS,
+      onHttpRequestComplete: ({ name, method, status, outcome, durationMs }) => {
+        outboundRequestsTotal.inc({ name, method, status, outcome });
+        outboundRequestDuration.observe({ name, outcome }, durationMs / 1000);
+      },
       preDequeue: async () => {
         // Synchronous, hot-path-safe cached read; false when no monitors are active.
         const skipForBackpressure = this.backpressureMonitors.some((m) => m.shouldSkipDequeue());
@@ -568,6 +594,7 @@ class ManagedSupervisor {
       checkpointClient: this.checkpointClient,
       computeManager: this.computeManager,
       tracing: this.tracing,
+      snapshotCallbackSecret: workerToken,
       wideEventOpts: this.wideEventOpts,
       wideEventsNoisyRoutes: this.wideEventsNoisyRoutes,
     });
@@ -602,6 +629,15 @@ class ManagedSupervisor {
         throw new Error("Image is missing");
       }
 
+      const deploymentToken = await mintDeploymentToken({
+        deployment: message.deployment.friendlyId,
+        deployment_version: message.backgroundWorker.version,
+        environment_id: message.environment.id,
+        environment_type: message.environment.type,
+        org_id: message.organization.id,
+        project_id: message.project.id,
+      });
+
       await this.workloadManager.create({
         dequeuedAt: message.dequeuedAt,
         dequeueResponseMs: timings.dequeueResponseMs,
@@ -615,6 +651,8 @@ class ManagedSupervisor {
         projectId: message.project.id,
         deploymentFriendlyId: message.deployment.friendlyId,
         deploymentVersion: message.backgroundWorker.version,
+        runtime: message.backgroundWorker.runtime,
+        deploymentToken,
         runId: message.run.id,
         runFriendlyId: message.run.friendlyId,
         version: message.version,
@@ -662,7 +700,7 @@ class ManagedSupervisor {
       return false;
     }
 
-    const warmStartUrlWithPath = new URL("/warm-start", this.warmStartUrl);
+    const warmStartUrlWithPath = new URL("/warm-start", this.warmStartDispatchUrl);
 
     const headers: Record<string, string> = {
       "Content-Type": "application/json",
@@ -675,6 +713,18 @@ class ManagedSupervisor {
       headers.traceparent = traceparent;
     }
 
+    const requestStart = performance.now();
+    const record = (
+      status: string,
+      outcome: "ok" | "http_error" | "invalid_response" | "network_error"
+    ) => {
+      outboundRequestsTotal.inc({ name: "warm_start", method: "POST", status, outcome });
+      outboundRequestDuration.observe(
+        { name: "warm_start", outcome },
+        (performance.now() - requestStart) / 1000
+      );
+    };
+
     try {
       const res = await fetch(warmStartUrlWithPath.href, {
         method: "POST",
@@ -683,8 +733,10 @@ class ManagedSupervisor {
       });
 
       if (!res.ok) {
+        record(String(res.status), "http_error");
         this.logger.error("Warm start failed", {
           runId: dequeuedMessage.run.id,
+          statusCode: res.status,
         });
         return false;
       }
@@ -693,6 +745,7 @@ class ManagedSupervisor {
       const parsedData = z.object({ didWarmStart: z.boolean() }).safeParse(data);
 
       if (!parsedData.success) {
+        record(String(res.status), "invalid_response");
         this.logger.error("Warm start response invalid", {
           runId: dequeuedMessage.run.id,
           data,
@@ -700,8 +753,11 @@ class ManagedSupervisor {
         return false;
       }
 
+      record(String(res.status), "ok");
+
       return parsedData.data.didWarmStart;
     } catch (error) {
+      record("none", "network_error");
       this.logger.error("Warm start error", {
         runId: dequeuedMessage.run.id,
         error,

apps/supervisor/src/services/computeSnapshotService.test.ts

@@ -20,13 +20,26 @@ function createService() {
     snapshot,
   } as unknown as ComputeWorkloadManager;
 
+  const submitSuspendCompletion = vi.fn(async () => ({ success: true }));
+
   const service = new ComputeSnapshotService({
     computeManager,
-    workerClient: {} as SupervisorHttpClient,
+    workerClient: { submitSuspendCompletion } as unknown as SupervisorHttpClient,
     wideEventOpts: { service: "supervisor-test", env: {}, enabled: false },
+    snapshotCallbackSecret: "test-secret",
   });
 
-  return { service, snapshot };
+  return { service, snapshot, submitSuspendCompletion };
+}
+
+function dispatchedMetadata(snapshot: {
+  mock: { calls: Array<Array<{ metadata?: Record<string, string> }>> };
+}) {
+  const metadata = snapshot.mock.calls[0]?.[0]?.metadata;
+  if (!metadata) {
+    throw new Error("Snapshot was not dispatched");
+  }
+  return metadata;
 }
 
 function delayedSnapshot(runnerId = "runner-1") {
@@ -38,6 +51,24 @@ function delayedSnapshot(runnerId = "runner-1") {
 }
 
 describe("ComputeSnapshotService", () => {
+  it("refuses to construct with an empty callback secret", () => {
+    const computeManager = {
+      snapshotDelayMs: DELAY_MS,
+      snapshotDispatchLimit: 1,
+      snapshot: vi.fn(async () => true),
+    } as unknown as ComputeWorkloadManager;
+
+    expect(
+      () =>
+        new ComputeSnapshotService({
+          computeManager,
+          workerClient: {} as SupervisorHttpClient,
+          wideEventOpts: { service: "supervisor-test", env: {}, enabled: false },
+          snapshotCallbackSecret: "",
+        })
+    ).toThrow();
+  });
+
   it("dispatches a scheduled snapshot after the delay", async () => {
     const { service, snapshot } = createService();
     try {
@@ -46,7 +77,12 @@ describe("ComputeSnapshotService", () => {
       await vi.waitFor(() => expect(snapshot).toHaveBeenCalledTimes(1), { timeout: 2_000 });
       expect(snapshot).toHaveBeenCalledWith({
         runnerId: "runner-1",
-        metadata: { runId: "run_1", snapshotFriendlyId: "snapshot_1" },
+        metadata: expect.objectContaining({
+          runId: "run_1",
+          snapshotFriendlyId: "snapshot_1",
+          snapshotCallbackNonce: expect.any(String),
+          snapshotCallbackToken: expect.any(String),
+        }),
       });
     } finally {
       service.stop();
@@ -121,10 +157,86 @@ describe("ComputeSnapshotService", () => {
       expect(snapshot).toHaveBeenCalledTimes(1);
       expect(snapshot).toHaveBeenCalledWith({
         runnerId: "runner-1",
-        metadata: { runId: "run_1", snapshotFriendlyId: "snapshot_2" },
+        metadata: expect.objectContaining({
+          runId: "run_1",
+          snapshotFriendlyId: "snapshot_2",
+          snapshotCallbackNonce: expect.any(String),
+          snapshotCallbackToken: expect.any(String),
+        }),
       });
     } finally {
       service.stop();
     }
   });
+
+  it("accepts a snapshot callback with the dispatched token", async () => {
+    const { service, snapshot, submitSuspendCompletion } = createService();
+    try {
+      service.schedule("run_1", delayedSnapshot());
+
+      await vi.waitFor(() => expect(snapshot).toHaveBeenCalledTimes(1), { timeout: 2_000 });
+      const metadata = dispatchedMetadata(snapshot);
+
+      const result = await service.handleCallback({
+        status: "completed",
+        instance_id: "instance_1",
+        snapshot_id: "compute_snapshot_1",
+        metadata,
+      });
+
+      expect(result).toEqual({ ok: true, status: 200 });
+      expect(submitSuspendCompletion).toHaveBeenCalledWith({
+        runId: "run_1",
+        snapshotId: "snapshot_1",
+        body: {
+          success: true,
+          checkpoint: {
+            type: "COMPUTE",
+            location: "compute_snapshot_1",
+          },
+        },
+      });
+    } finally {
+      service.stop();
+    }
+  });
+
+  it("rejects a snapshot callback without a valid token", async () => {
+    const { service, submitSuspendCompletion } = createService();
+    try {
+      const result = await service.handleCallback({
+        status: "completed",
+        instance_id: "instance_1",
+        snapshot_id: "compute_snapshot_1",
+        metadata: { runId: "run_1", snapshotFriendlyId: "snapshot_1" },
+      });
+
+      expect(result).toEqual({ ok: false, status: 401 });
+      expect(submitSuspendCompletion).not.toHaveBeenCalled();
+    } finally {
+      service.stop();
+    }
+  });
+
+  it("rejects a snapshot callback whose token is for a different snapshot", async () => {
+    const { service, snapshot, submitSuspendCompletion } = createService();
+    try {
+      service.schedule("run_1", delayedSnapshot());
+
+      await vi.waitFor(() => expect(snapshot).toHaveBeenCalledTimes(1), { timeout: 2_000 });
+      const metadata = dispatchedMetadata(snapshot);
+
+      const result = await service.handleCallback({
+        status: "completed",
+        instance_id: "instance_1",
+        snapshot_id: "compute_snapshot_1",
+        metadata: { ...metadata, snapshotFriendlyId: "snapshot_2" },
+      });
+
+      expect(result).toEqual({ ok: false, status: 401 });
+      expect(submitSuspendCompletion).not.toHaveBeenCalled();
+    } finally {
+      service.stop();
+    }
+  });
 });

apps/supervisor/src/services/computeSnapshotService.ts

@@ -1,3 +1,4 @@
+import { createHmac, randomBytes, timingSafeEqual } from "node:crypto";
 import pLimit from "p-limit";
 import { SimpleStructuredLogger } from "@trigger.dev/core/v3/utils/structuredLogger";
 import { parseTraceparent } from "@trigger.dev/core/v3/isomorphic";
@@ -16,6 +17,13 @@ import {
   type WideEventOptions,
 } from "../wideEvents/index.js";
 
+const SNAPSHOT_CALLBACK_NONCE_METADATA_KEY = "snapshotCallbackNonce";
+const SNAPSHOT_CALLBACK_TOKEN_METADATA_KEY = "snapshotCallbackToken";
+
+// Domain-separation label so the callback-signing key is derived from, rather
+// than equal to, the secret used for other protocols. Bump the suffix to rotate.
+const SNAPSHOT_CALLBACK_KEY_INFO = "compute-snapshot-callback-v1";
+
 type DelayedSnapshot = {
   runnerId: string;
   runFriendlyId: string;
@@ -34,6 +42,7 @@ export type ComputeSnapshotServiceOptions = {
   workerClient: SupervisorHttpClient;
   tracing?: OtlpTraceService;
   wideEventOpts: WideEventOptions;
+  snapshotCallbackSecret: string;
 };
 
 export class ComputeSnapshotService {
@@ -48,6 +57,7 @@ export class ComputeSnapshotService {
   private readonly workerClient: SupervisorHttpClient;
   private readonly tracing?: OtlpTraceService;
   private readonly wideEventOpts: WideEventOptions;
+  private readonly snapshotCallbackKey: Buffer;
 
   constructor(opts: ComputeSnapshotServiceOptions) {
     this.computeManager = opts.computeManager;
@@ -55,6 +65,18 @@ export class ComputeSnapshotService {
     this.tracing = opts.tracing;
     this.wideEventOpts = opts.wideEventOpts;
 
+    // Reject an empty secret up front: an empty HMAC key would make callback
+    // tokens forgeable by anyone. Guarding here (rather than only at env parse)
+    // also covers the case where the secret is read from an empty file.
+    if (!opts.snapshotCallbackSecret) {
+      throw new Error("snapshotCallbackSecret must not be empty");
+    }
+    // Derive a dedicated key by domain separation so the raw secret is never
+    // used directly as a MAC key for this protocol.
+    this.snapshotCallbackKey = createHmac("sha256", opts.snapshotCallbackSecret)
+      .update(SNAPSHOT_CALLBACK_KEY_INFO)
+      .digest();
+
     this.dispatchLimit = pLimit(this.computeManager.snapshotDispatchLimit);
     this.timerWheel = new TimerWheel<DelayedSnapshot>({
       delayMs: this.computeManager.snapshotDelayMs,
@@ -146,15 +168,29 @@ export class ComputeSnapshotService {
       instanceId: body.instance_id,
       status: body.status,
       error: body.status === "failed" ? body.error : undefined,
-      metadata: body.metadata,
+      runId,
+      snapshotFriendlyId,
       durationMs: body.duration_ms,
     });
 
     if (!runId || !snapshotFriendlyId) {
-      this.logger.error("Snapshot callback missing metadata", { body });
+      this.logger.error("Snapshot callback missing metadata", {
+        status: body.status,
+        instanceId: body.instance_id,
+        metadataKeys: Object.keys(body.metadata ?? {}),
+      });
       return { ok: false as const, status: 400 };
     }
 
+    if (!this.#verifyCallbackToken(body.metadata, runId, snapshotFriendlyId)) {
+      this.logger.error("Snapshot callback failed token verification", {
+        runId,
+        snapshotFriendlyId,
+        instanceId: body.instance_id,
+      });
+      return { ok: false as const, status: 401 };
+    }
+
     this.#emitSnapshotSpan(runId, body.duration_ms, snapshotId);
 
     if (body.status === "completed") {
@@ -266,11 +302,18 @@ export class ComputeSnapshotService {
         },
       },
       async () => {
+        const callbackNonce = randomBytes(16).toString("hex");
         const result = await this.computeManager.snapshot({
           runnerId: snapshot.runnerId,
           metadata: {
             runId: snapshot.runFriendlyId,
             snapshotFriendlyId: snapshot.snapshotFriendlyId,
+            [SNAPSHOT_CALLBACK_NONCE_METADATA_KEY]: callbackNonce,
+            [SNAPSHOT_CALLBACK_TOKEN_METADATA_KEY]: this.#createCallbackToken(
+              callbackNonce,
+              snapshot.runFriendlyId,
+              snapshot.snapshotFriendlyId
+            ),
           },
         });
 
@@ -281,6 +324,51 @@ export class ComputeSnapshotService {
     );
   }
 
+  #createCallbackToken(nonce: string, runFriendlyId: string, snapshotFriendlyId: string): string {
+    return createHmac("sha256", this.snapshotCallbackKey)
+      .update(nonce)
+      .update("\0")
+      .update(runFriendlyId)
+      .update("\0")
+      .update(snapshotFriendlyId)
+      .digest("hex");
+  }
+
+  /**
+   * Verify that a callback carries a token this supervisor issued for the given
+   * run and snapshot. The token binds only the identifiers known at dispatch
+   * time (nonce, run, snapshot); it intentionally does not cover result fields
+   * such as the snapshot location or status/error, which are produced by the
+   * gateway after the snapshot and so cannot be signed in advance. Verification
+   * is also stateless, so a token is not single-use.
+   *
+   * This closes the primary risk (a caller that can merely reach the endpoint
+   * cannot mint a valid token, so cannot forge a result for an arbitrary run).
+   * It does not defend against an attacker who can observe a genuine callback
+   * and then replay it or alter its unsigned result fields - that relies on the
+   * gateway->supervisor callback channel being authenticated and encrypted.
+   */
+  #verifyCallbackToken(
+    metadata: Record<string, string> | undefined,
+    runFriendlyId: string,
+    snapshotFriendlyId: string
+  ): boolean {
+    const nonce = metadata?.[SNAPSHOT_CALLBACK_NONCE_METADATA_KEY];
+    const token = metadata?.[SNAPSHOT_CALLBACK_TOKEN_METADATA_KEY];
+
+    if (!nonce || !token) {
+      return false;
+    }
+
+    const expected = this.#createCallbackToken(nonce, runFriendlyId, snapshotFriendlyId);
+    const expectedBuffer = Buffer.from(expected, "hex");
+    const tokenBuffer = Buffer.from(token, "hex");
+
+    return (
+      expectedBuffer.length === tokenBuffer.length && timingSafeEqual(expectedBuffer, tokenBuffer)
+    );
+  }
+
   #emitSnapshotSpan(runFriendlyId: string, durationMs?: number, snapshotId?: string) {
     if (!this.tracing) return;
 

apps/supervisor/src/workloadManager/compute.ts

@@ -151,7 +151,9 @@ export class ComputeWorkloadManager implements WorkloadManager {
       TRIGGER_DEQUEUED_AT_MS: String(opts.dequeuedAt.getTime()),
       TRIGGER_POD_SCHEDULED_AT_MS: String(Date.now()),
       TRIGGER_ENV_ID: opts.envId,
-      TRIGGER_DEPLOYMENT_ID: opts.deploymentFriendlyId,
+      TRIGGER_DEPLOYMENT_ID: opts.deploymentToken ?? opts.deploymentFriendlyId,
+      // Plain friendlyId for telemetry (worker.id), so it isn't the opaque token in DEPLOYMENT_ID.
+      TRIGGER_DEPLOYMENT_FRIENDLY_ID: opts.deploymentFriendlyId,
       TRIGGER_DEPLOYMENT_VERSION: opts.deploymentVersion,
       TRIGGER_RUN_ID: opts.runFriendlyId,
       TRIGGER_SNAPSHOT_ID: opts.snapshotFriendlyId,

apps/supervisor/src/workloadManager/docker.ts

@@ -72,7 +72,9 @@ export class DockerWorkloadManager implements WorkloadManager {
       `TRIGGER_DEQUEUED_AT_MS=${opts.dequeuedAt.getTime()}`,
       `TRIGGER_POD_SCHEDULED_AT_MS=${Date.now()}`,
       `TRIGGER_ENV_ID=${opts.envId}`,
-      `TRIGGER_DEPLOYMENT_ID=${opts.deploymentFriendlyId}`,
+      `TRIGGER_DEPLOYMENT_ID=${opts.deploymentToken ?? opts.deploymentFriendlyId}`,
+      // Plain friendlyId for telemetry (worker.id), so it isn't the opaque token in DEPLOYMENT_ID.
+      `TRIGGER_DEPLOYMENT_FRIENDLY_ID=${opts.deploymentFriendlyId}`,
       `TRIGGER_DEPLOYMENT_VERSION=${opts.deploymentVersion}`,
       `TRIGGER_RUN_ID=${opts.runFriendlyId}`,
       `TRIGGER_SNAPSHOT_ID=${opts.snapshotFriendlyId}`,

apps/supervisor/src/workloadManager/kubernetes.test.ts

@@ -0,0 +1,40 @@
+import { describe, expect, it } from "vitest";
+import {
+  BLOCK_IO_URING_SECCOMP_PROFILE,
+  withBlockIoUringSeccompProfile,
+} from "./kubernetesPodSpec.js";
+
+const basePodSpec = {
+  restartPolicy: "Never" as const,
+  automountServiceAccountToken: false,
+  securityContext: {
+    runAsNonRoot: true,
+    runAsUser: 1000,
+    fsGroup: 1000,
+  },
+};
+
+describe("withBlockIoUringSeccompProfile", () => {
+  it("adds the Localhost io_uring profile for node-24 and above, preserving pod security defaults", () => {
+    for (const runtime of ["node-24", "node-26", "node-30", "experimental-node-24"]) {
+      const podSpec = withBlockIoUringSeccompProfile(basePodSpec, runtime);
+
+      expect(podSpec).toMatchObject({
+        ...basePodSpec,
+        securityContext: {
+          ...basePodSpec.securityContext,
+          seccompProfile: {
+            type: "Localhost",
+            localhostProfile: BLOCK_IO_URING_SECCOMP_PROFILE,
+          },
+        },
+      });
+    }
+  });
+
+  it("leaves the pod spec unchanged for runtimes that do not create io_uring fds", () => {
+    for (const runtime of ["node", "node-22", "bun", undefined, null, ""]) {
+      expect(withBlockIoUringSeccompProfile(basePodSpec, runtime)).toEqual(basePodSpec);
+    }
+  });
+});

apps/supervisor/src/workloadManager/kubernetes.ts

@@ -14,6 +14,7 @@ import { PlacementTagProcessor } from "@trigger.dev/core/v3/serverOnly";
 import { env } from "../env.js";
 import { type K8sApi, createK8sApi, type k8s } from "../clients/kubernetes.js";
 import { getRunnerId } from "../util.js";
+import { withBlockIoUringSeccompProfile } from "./kubernetesPodSpec.js";
 
 type ResourceQuantities = {
   [K in "cpu" | "memory" | "ephemeral-storage"]?: string;
@@ -105,6 +106,11 @@ export class KubernetesWorkloadManager implements WorkloadManager {
     const runnerId = getRunnerId(opts.runFriendlyId, opts.nextAttemptNumber);
 
     try {
+      const basePodSpec = this.addPlacementTags(this.#defaultPodSpec, opts.placementTags);
+      const podSpec = this.opts.checkpointsEnabled
+        ? withBlockIoUringSeccompProfile(basePodSpec, opts.runtime)
+        : basePodSpec;
+
       await this.k8s.core.createNamespacedPod({
         namespace: this.namespace,
         body: {
@@ -119,7 +125,7 @@ export class KubernetesWorkloadManager implements WorkloadManager {
             },
           },
           spec: {
-            ...this.addPlacementTags(this.#defaultPodSpec, opts.placementTags),
+            ...podSpec,
             affinity: this.#getAffinity(opts),
             tolerations: this.#getScheduleTolerations(this.#isScheduledRun(opts)),
             terminationGracePeriodSeconds: 60 * 60,
@@ -152,6 +158,11 @@ export class KubernetesWorkloadManager implements WorkloadManager {
                   },
                   {
                     name: "TRIGGER_DEPLOYMENT_ID",
+                    value: opts.deploymentToken ?? opts.deploymentFriendlyId,
+                  },
+                  {
+                    // Plain friendlyId for telemetry (worker.id), not the opaque token in DEPLOYMENT_ID.
+                    name: "TRIGGER_DEPLOYMENT_FRIENDLY_ID",
                     value: opts.deploymentFriendlyId,
                   },
                   {

apps/supervisor/src/workloadManager/kubernetesPodSpec.ts

@@ -0,0 +1,33 @@
+import type { k8s } from "../clients/kubernetes.js";
+
+/**
+ * Relative path (kubelet seccomp root) of the profile blocking only io_uring
+ * syscalls. Must match the profile deployed to worker nodes.
+ */
+export const BLOCK_IO_URING_SECCOMP_PROFILE = "profiles/block-io-uring.json";
+
+/**
+ * Node >= 24 always creates io_uring fds, which can't be checkpointed. Blocking
+ * io_uring_setup makes libuv fall back to epoll. Other runtimes don't need this,
+ * so the profile is only applied for node-24+. Tolerates an "experimental-" prefix.
+ */
+export function withBlockIoUringSeccompProfile(
+  podSpec: Omit<k8s.V1PodSpec, "containers">,
+  runtime: string | null | undefined
+): Omit<k8s.V1PodSpec, "containers"> {
+  const match = runtime ? /^(?:experimental-)?node-(\d+)$/.exec(runtime) : null;
+  if (!match || Number(match[1]) < 24) {
+    return podSpec;
+  }
+
+  return {
+    ...podSpec,
+    securityContext: {
+      ...podSpec.securityContext,
+      seccompProfile: {
+        type: "Localhost",
+        localhostProfile: BLOCK_IO_URING_SECCOMP_PROFILE,
+      },
+    },
+  };
+}

apps/supervisor/src/workloadManager/types.ts

@@ -16,6 +16,8 @@ export interface WorkloadManagerOptions {
   snapshotPollIntervalSeconds?: number;
   additionalEnvVars?: Record<string, string>;
   dockerAutoremove?: boolean;
+  // Whether CRIU checkpoint/restore is enabled for this deployment
+  checkpointsEnabled?: boolean;
 }
 
 export interface WorkloadManager {
@@ -40,6 +42,10 @@ export interface WorkloadManagerCreateOptions {
   projectId: string;
   deploymentFriendlyId: string;
   deploymentVersion: string;
+  // Canonical runtime identifier (e.g. "node", "node-22", "node-24")
+  runtime?: string;
+  // When set, overrides the TRIGGER_DEPLOYMENT_ID value the runner forwards as its identity header.
+  deploymentToken?: string;
   runId: string;
   runFriendlyId: string;
   snapshotId: string;

apps/supervisor/src/workloadServer/index.ts

@@ -25,6 +25,11 @@ import { type Namespace, Server, type Socket } from "socket.io";
 import { z } from "zod";
 import { env } from "../env.js";
 import { register } from "../metrics.js";
+import {
+  verifyDeploymentIdHeader,
+  workloadTokenEnforced,
+  workloadTokensEnabled,
+} from "../workloadToken.js";
 import {
   ComputeSnapshotService,
   type RunTraceContext,
@@ -86,6 +91,7 @@ type WorkloadServerOptions = {
   checkpointClient?: CheckpointClient;
   computeManager?: ComputeWorkloadManager;
   tracing?: OtlpTraceService;
+  snapshotCallbackSecret: string;
   wideEventOpts: WideEventOptions;
   /** When true, high-frequency HTTP routes also emit wide events. */
   wideEventsNoisyRoutes: boolean;
@@ -136,6 +142,7 @@ export class WorkloadServer extends EventEmitter<WorkloadServerEvents> {
         workerClient: opts.workerClient,
         tracing: opts.tracing,
         wideEventOpts: this.wideEventOpts,
+        snapshotCallbackSecret: opts.snapshotCallbackSecret,
       });
     }
 
@@ -169,6 +176,34 @@ export class WorkloadServer extends EventEmitter<WorkloadServerEvents> {
     return this.headerValueFromRequest(req, WORKLOAD_HEADERS.PROJECT_REF);
   }
 
+  /**
+   * Verify the deployment token from the workload deployment-id header and return the verified
+   * environment_id to forward upstream. The env id is only forwarded in enforce mode: in log mode
+   * we still verify + record metrics but attach no header (so the platform never scopes). Only
+   * enforce fails a request, and only for a present-but-invalid token; absent and legacy ids pass.
+   */
+  private async authorizeWorkloadRequest(
+    req: IncomingMessage
+  ): Promise<{ ok: true; environmentId?: string } | { ok: false }> {
+    if (!workloadTokensEnabled) {
+      return { ok: true };
+    }
+
+    const result = await verifyDeploymentIdHeader(this.deploymentIdFromRequest(req), "http");
+
+    if (result.outcome === "jwt_invalid" && workloadTokenEnforced) {
+      return { ok: false };
+    }
+
+    return {
+      ok: true,
+      environmentId:
+        workloadTokenEnforced && result.outcome === "jwt_valid"
+          ? result.claims.environment_id
+          : undefined,
+    };
+  }
+
   /**
    * Sets common route meta on the wide-event state from URL params.
    */
@@ -250,11 +285,17 @@ export class WorkloadServer extends EventEmitter<WorkloadServerEvents> {
               "POST",
               async () => {
                 const { req, reply, params, body } = ctx;
+                const auth = await this.authorizeWorkloadRequest(req);
+                if (!auth.ok) {
+                  reply.empty(401);
+                  return;
+                }
                 const startResponse = await this.workerClient.startRunAttempt(
                   params.runFriendlyId,
                   params.snapshotFriendlyId,
                   body,
-                  this.runnerIdFromRequest(req)
+                  this.runnerIdFromRequest(req),
+                  auth.environmentId
                 );
 
                 if (!startResponse.success) {
@@ -286,6 +327,11 @@ export class WorkloadServer extends EventEmitter<WorkloadServerEvents> {
               "POST",
               async () => {
                 const { req, reply, params, body } = ctx;
+                const auth = await this.authorizeWorkloadRequest(req);
+                if (!auth.ok) {
+                  reply.empty(401);
+                  return;
+                }
                 const runnerId = this.runnerIdFromRequest(req);
 
                 // A completion attempt invalidates any pending delayed snapshot
@@ -304,7 +350,8 @@ export class WorkloadServer extends EventEmitter<WorkloadServerEvents> {
                   params.runFriendlyId,
                   params.snapshotFriendlyId,
                   body,
-                  runnerId
+                  runnerId,
+                  auth.environmentId
                 );
 
                 if (!completeResponse.success) {
@@ -336,6 +383,11 @@ export class WorkloadServer extends EventEmitter<WorkloadServerEvents> {
               "POST",
               async () => {
                 const { req, reply, params, body } = ctx;
+                const auth = await this.authorizeWorkloadRequest(req);
+                if (!auth.ok) {
+                  reply.empty(401);
+                  return;
+                }
                 const heartbeatResponse = await this.workerClient.heartbeatRun(
                   params.runFriendlyId,
                   params.snapshotFriendlyId,
@@ -373,6 +425,11 @@ export class WorkloadServer extends EventEmitter<WorkloadServerEvents> {
               "GET",
               async () => {
                 const { reply, params, req } = ctx;
+                const auth = await this.authorizeWorkloadRequest(req);
+                if (!auth.ok) {
+                  reply.empty(401);
+                  return;
+                }
                 const runnerId = this.runnerIdFromRequest(req);
                 const deploymentVersion = this.deploymentVersionFromRequest(req);
                 const projectRef = this.projectRefFromRequest(req);
@@ -469,6 +526,11 @@ export class WorkloadServer extends EventEmitter<WorkloadServerEvents> {
               "GET",
               async () => {
                 const { req, reply, params } = ctx;
+                const auth = await this.authorizeWorkloadRequest(req);
+                if (!auth.ok) {
+                  reply.empty(401);
+                  return;
+                }
                 this.logger.debug("Run continuation request", { params });
 
                 // Cancel any pending delayed snapshot for this run
@@ -477,7 +539,8 @@ export class WorkloadServer extends EventEmitter<WorkloadServerEvents> {
                 const continuationResult = await this.workerClient.continueRunExecution(
                   params.runFriendlyId,
                   params.snapshotFriendlyId,
-                  this.runnerIdFromRequest(req)
+                  this.runnerIdFromRequest(req),
+                  auth.environmentId
                 );
 
                 if (!continuationResult.success) {
@@ -511,10 +574,16 @@ export class WorkloadServer extends EventEmitter<WorkloadServerEvents> {
               "GET",
               async () => {
                 const { req, reply, params } = ctx;
+                const auth = await this.authorizeWorkloadRequest(req);
+                if (!auth.ok) {
+                  reply.empty(401);
+                  return;
+                }
                 const sinceSnapshotResponse = await this.workerClient.getSnapshotsSince(
                   params.runFriendlyId,
                   params.snapshotFriendlyId,
-                  this.runnerIdFromRequest(req)
+                  this.runnerIdFromRequest(req),
+                  auth.environmentId
                 );
 
                 if (!sinceSnapshotResponse.success) {
@@ -585,9 +654,18 @@ export class WorkloadServer extends EventEmitter<WorkloadServerEvents> {
               const { req, reply, params, body } = ctx;
               reply.empty(204);
 
+              // Redact TRIGGER_DEPLOYMENT_ID before relaying to the platform.
+              const sanitizedBody =
+                body.properties && "TRIGGER_DEPLOYMENT_ID" in body.properties
+                  ? {
+                      ...body,
+                      properties: { ...body.properties, TRIGGER_DEPLOYMENT_ID: "[redacted]" },
+                    }
+                  : body;
+
               await this.workerClient.sendDebugLog(
                 params.runFriendlyId,
-                body,
+                sanitizedBody,
                 this.runnerIdFromRequest(req)
               );
             },
@@ -681,7 +759,31 @@ export class WorkloadServer extends EventEmitter<WorkloadServerEvents> {
         return;
       }
 
-      this.logger.debug("[WS] auth success", socket.data);
+      if (workloadTokensEnabled) {
+        const result = await verifyDeploymentIdHeader(socket.data.deploymentId, "ws");
+
+        if (result.outcome === "jwt_invalid" && workloadTokenEnforced) {
+          this.logger.error("[WS] deployment token verification failed", {
+            runnerId: socket.data.runnerId,
+          });
+          socket.disconnect(true);
+          return;
+        }
+
+        // Re-source the deployment id from the verified claim; the raw header may be an opaque token.
+        // A legacy bare id is itself the friendlyId, so it's safe to keep.
+        socket.data.deploymentFriendlyId =
+          result.outcome === "jwt_valid"
+            ? result.claims.deployment
+            : result.outcome === "legacy_bare"
+              ? socket.data.deploymentId
+              : undefined;
+      }
+
+      this.logger.debug("[WS] handshake complete", {
+        runnerId: socket.data.runnerId,
+        deploymentFriendlyId: socket.data.deploymentFriendlyId,
+      });
 
       next();
     });
@@ -693,7 +795,7 @@ export class WorkloadServer extends EventEmitter<WorkloadServerEvents> {
 
       const getSocketMetadata = () => {
         return {
-          deploymentId: socket.data.deploymentId,
+          deploymentId: socket.data.deploymentFriendlyId ?? socket.data.deploymentId,
           runId: socket.data.runFriendlyId,
           snapshotId: socket.data.snapshotId,
           runnerId: socket.data.runnerId,
@@ -712,8 +814,9 @@ export class WorkloadServer extends EventEmitter<WorkloadServerEvents> {
           populate: (state) => {
             state.extras.event = event;
             setMeta(state, "run_id", friendlyId);
-            if (socket.data.deploymentId) {
-              setMeta(state, "deployment_id", socket.data.deploymentId);
+            const deploymentId = socket.data.deploymentFriendlyId ?? socket.data.deploymentId;
+            if (deploymentId) {
+              setMeta(state, "deployment_id", deploymentId);
             }
             if (socket.data.runnerId) setMeta(state, "runner_id", socket.data.runnerId);
             state.extras.socket_id = socket.id;
@@ -725,6 +828,33 @@ export class WorkloadServer extends EventEmitter<WorkloadServerEvents> {
       const runConnected = (friendlyId: string) => {
         socketLogger.debug("runConnected", { ...getSocketMetadata() });
 
+        // Only the owning runner may (re)bind a run. A live socket from a *different*
+        // runner keeps its binding so an unrelated connection can't hijack the run. But
+        // the newest socket for the *same* runner is a legitimate reconnection/handoff and
+        // is allowed to take over even while the stale socket still reports connected -
+        // otherwise, during a reconnect race the fresh socket would silently stay unbound
+        // (missing continue/cancel/suspend notifications) until the dead socket times out.
+        const existing = this.runSockets.get(friendlyId);
+        if (existing && existing.id !== socket.id && existing.connected) {
+          const sameRunner =
+            !!socket.data.runnerId && existing.data.runnerId === socket.data.runnerId;
+
+          if (!sameRunner) {
+            socketLogger.warn("runConnected: run already bound to another socket", {
+              ...getSocketMetadata(),
+              friendlyId,
+              existingSocketId: existing.id,
+            });
+            return;
+          }
+
+          socketLogger.debug("runConnected: replacing stale socket for same runner", {
+            ...getSocketMetadata(),
+            friendlyId,
+            existingSocketId: existing.id,
+          });
+        }
+
         // If there's already a run ID set, we should "disconnect" it from this socket
         if (socket.data.runFriendlyId && socket.data.runFriendlyId !== friendlyId) {
           socketLogger.debug("runConnected: disconnecting existing run", {
@@ -744,6 +874,22 @@ export class WorkloadServer extends EventEmitter<WorkloadServerEvents> {
       const runDisconnected = (friendlyId: string, reason: string) => {
         socketLogger.debug("runDisconnected", { ...getSocketMetadata() });
 
+        // A newer socket may have taken over this run (same-runner reconnect race). If the
+        // run is now bound to a different socket, this stale socket must not clear the fresh
+        // binding or emit a spurious disconnect - just drop its own reference and bail.
+        const bound = this.runSockets.get(friendlyId);
+        if (bound && bound.id !== socket.id) {
+          socketLogger.debug("runDisconnected: run rebound to another socket, skipping", {
+            ...getSocketMetadata(),
+            friendlyId,
+            boundSocketId: bound.id,
+          });
+          if (socket.data.runFriendlyId === friendlyId) {
+            socket.data.runFriendlyId = undefined;
+          }
+          return;
+        }
+
         // The run is gone from this runner (crash, exit, or replaced by a new
         // run), so a pending delayed snapshot for it is stale. Genuine
         // waitpoint suspensions keep the socket connected, so this doesn't

apps/supervisor/src/workloadServer/workloadServer.auth.test.ts

@@ -0,0 +1,107 @@
+import { mintWorkloadDeploymentToken } from "@trigger.dev/core/v3";
+import { WORKLOAD_HEADERS } from "@trigger.dev/core/v3/workers";
+import { afterAll, beforeAll, describe, expect, it, vi } from "vitest";
+
+// Set enforce mode + secret before env.ts parses (vi.mock is hoisted above imports, so the secret
+// must be a literal here). SECRET below mirrors it for use in the test body.
+vi.mock("std-env", () => ({
+  env: {
+    TRIGGER_API_URL: "http://localhost:3030",
+    TRIGGER_WORKER_TOKEN: "test-token",
+    MANAGED_WORKER_SECRET: "test-secret",
+    OTEL_EXPORTER_OTLP_ENDPOINT: "http://localhost:4318",
+    WORKLOAD_TOKEN_SECRET: "integration-test-secret",
+    WORKLOAD_TOKEN_ENFORCEMENT: "enforce",
+  },
+}));
+
+const SECRET = "integration-test-secret";
+const EXP = Math.floor(Date.UTC(2032, 0, 1) / 1000);
+
+const { WorkloadServer } = await import("./index.js");
+
+const PORT = 18732;
+const BASE = `http://127.0.0.1:${PORT}`;
+
+function claims(environmentId = "env_test_123") {
+  return {
+    deployment: "deployment_test",
+    deployment_version: "20260710.1",
+    environment_id: environmentId,
+    environment_type: "PRODUCTION",
+    org_id: "org_1",
+    project_id: "proj_1",
+  };
+}
+
+// Records the args each relay method is called with so we can assert the forwarded claim.
+const calls: { getSnapshotsSince: any[][] } = { getSnapshotsSince: [] };
+
+const workerClient = {
+  getSnapshotsSince: vi.fn(async (...args: any[]) => {
+    calls.getSnapshotsSince.push(args);
+    return { success: true as const, data: { snapshots: [] } };
+  }),
+} as any;
+
+let server: InstanceType<typeof WorkloadServer>;
+
+beforeAll(async () => {
+  server = new WorkloadServer({
+    port: PORT,
+    workerClient,
+    snapshotCallbackSecret: "snapshot-callback-secret",
+    wideEventOpts: { service: "supervisor", env: { nodeId: "test" }, enabled: false },
+    wideEventsNoisyRoutes: false,
+  });
+  await server.start();
+});
+
+afterAll(async () => {
+  await server.stop();
+});
+
+function snapshotsSince(deploymentIdHeader?: string) {
+  const headers: Record<string, string> = {
+    [WORKLOAD_HEADERS.RUNNER_ID]: "runner_1",
+  };
+  if (deploymentIdHeader !== undefined) {
+    headers[WORKLOAD_HEADERS.DEPLOYMENT_ID] = deploymentIdHeader;
+  }
+  return fetch(`${BASE}/api/v1/workload-actions/runs/run_1/snapshots/since/snap_1`, { headers });
+}
+
+describe("WorkloadServer auth (enforce mode)", () => {
+  it("allows a valid token and forwards the verified environment_id", async () => {
+    const token = await mintWorkloadDeploymentToken(claims("env_forwarded_42"), SECRET, EXP);
+    const res = await snapshotsSince(token);
+
+    expect(res.status).toBe(200);
+    const lastCall = calls.getSnapshotsSince.at(-1)!;
+    // getSnapshotsSince(runId, snapshotId, runnerId, environmentId)
+    expect(lastCall[3]).toBe("env_forwarded_42");
+  });
+
+  it("rejects a token signed with the wrong secret (401) and does not relay", async () => {
+    const before = calls.getSnapshotsSince.length;
+    const badToken = await mintWorkloadDeploymentToken(claims(), "wrong-secret", EXP);
+    const res = await snapshotsSince(badToken);
+
+    expect(res.status).toBe(401);
+    expect(calls.getSnapshotsSince.length).toBe(before);
+  });
+
+  it("allows a legacy bare friendlyId and forwards no environment_id", async () => {
+    const res = await snapshotsSince("deployment_legacy_bare");
+
+    expect(res.status).toBe(200);
+    expect(calls.getSnapshotsSince.at(-1)![3]).toBeUndefined();
+  });
+
+  it("allows an absent token and forwards no environment_id", async () => {
+    const res = await snapshotsSince(undefined);
+
+    expect(res.status).toBe(200);
+    expect(calls.getSnapshotsSince.at(-1)![3]).toBeUndefined();
+  });
+});

apps/supervisor/src/workloadServer/workloadServer.authLog.test.ts

@@ -0,0 +1,87 @@
+import { mintWorkloadDeploymentToken } from "@trigger.dev/core/v3";
+import { WORKLOAD_HEADERS } from "@trigger.dev/core/v3/workers";
+import { afterAll, beforeAll, describe, expect, it, vi } from "vitest";
+
+// Log mode: mint + verify + metrics, but the platform must NOT be scoped, so no environment_id is
+// forwarded even for a valid token. (vi.mock is hoisted; secret literal here, mirrored below.)
+vi.mock("std-env", () => ({
+  env: {
+    TRIGGER_API_URL: "http://localhost:3030",
+    TRIGGER_WORKER_TOKEN: "test-token",
+    MANAGED_WORKER_SECRET: "test-secret",
+    OTEL_EXPORTER_OTLP_ENDPOINT: "http://localhost:4318",
+    WORKLOAD_TOKEN_SECRET: "integration-test-secret",
+    WORKLOAD_TOKEN_ENFORCEMENT: "log",
+  },
+}));
+
+const SECRET = "integration-test-secret";
+const EXP = Math.floor(Date.UTC(2032, 0, 1) / 1000);
+
+const { WorkloadServer } = await import("./index.js");
+
+const PORT = 18733;
+const BASE = `http://127.0.0.1:${PORT}`;
+
+const claims = {
+  deployment: "deployment_test",
+  deployment_version: "20260710.1",
+  environment_id: "env_should_not_forward",
+  environment_type: "PRODUCTION",
+  org_id: "org_1",
+  project_id: "proj_1",
+};
+
+const calls: { getSnapshotsSince: any[][] } = { getSnapshotsSince: [] };
+
+const workerClient = {
+  getSnapshotsSince: vi.fn(async (...args: any[]) => {
+    calls.getSnapshotsSince.push(args);
+    return { success: true as const, data: { snapshots: [] } };
+  }),
+} as any;
+
+let server: InstanceType<typeof WorkloadServer>;
+
+beforeAll(async () => {
+  server = new WorkloadServer({
+    port: PORT,
+    workerClient,
+    snapshotCallbackSecret: "snapshot-callback-secret",
+    wideEventOpts: { service: "supervisor", env: { nodeId: "test" }, enabled: false },
+    wideEventsNoisyRoutes: false,
+  });
+  await server.start();
+});
+
+afterAll(async () => {
+  await server.stop();
+});
+
+describe("WorkloadServer auth (log mode)", () => {
+  it("allows a valid token but forwards no environment_id", async () => {
+    const token = await mintWorkloadDeploymentToken(claims, SECRET, EXP);
+    const res = await fetch(`${BASE}/api/v1/workload-actions/runs/run_1/snapshots/since/snap_1`, {
+      headers: {
+        [WORKLOAD_HEADERS.RUNNER_ID]: "runner_1",
+        [WORKLOAD_HEADERS.DEPLOYMENT_ID]: token,
+      },
+    });
+
+    expect(res.status).toBe(200);
+    expect(calls.getSnapshotsSince.at(-1)![3]).toBeUndefined();
+  });
+
+  it("does not reject an invalid token in log mode", async () => {
+    const badToken = await mintWorkloadDeploymentToken(claims, "wrong-secret", EXP);
+    const res = await fetch(`${BASE}/api/v1/workload-actions/runs/run_1/snapshots/since/snap_1`, {
+      headers: {
+        [WORKLOAD_HEADERS.RUNNER_ID]: "runner_1",
+        [WORKLOAD_HEADERS.DEPLOYMENT_ID]: badToken,
+      },
+    });
+
+    expect(res.status).toBe(200);
+    expect(calls.getSnapshotsSince.at(-1)![3]).toBeUndefined();
+  });
+});

apps/supervisor/src/workloadToken.ts

@@ -0,0 +1,95 @@
+import {
+  classifyDeploymentIdHeader,
+  mintWorkloadDeploymentToken,
+  type WorkloadDeploymentTokenClaims,
+  type WorkloadDeploymentTokenInput,
+} from "@trigger.dev/core/v3";
+import { Counter, Gauge } from "prom-client";
+import { env } from "./env.js";
+import { register } from "./metrics.js";
+
+const secret = env.WORKLOAD_TOKEN_SECRET;
+
+// Absolute expiry (epoch seconds) shared by every mint, so tokens stay byte-deterministic per
+// deployment regardless of when/where a pod is created.
+const tokenExpSeconds = Math.floor(new Date(env.WORKLOAD_TOKEN_EXP).getTime() / 1000);
+
+/** Mint + verify run in "log" (dry-run) and "enforce"; the env superRefine guarantees a secret then. */
+export const workloadTokensEnabled = env.WORKLOAD_TOKEN_ENFORCEMENT !== "disabled";
+
+/** Only "enforce" rejects a present-but-invalid token; "log" observes and always allows. */
+export const workloadTokenEnforced = env.WORKLOAD_TOKEN_ENFORCEMENT === "enforce";
+
+const mintCounter = new Counter({
+  name: "workload_token_minted_total",
+  help: "Deployment tokens minted and injected into TRIGGER_DEPLOYMENT_ID at pod creation",
+  labelNames: ["env_type"] as const,
+  registers: [register],
+});
+
+export type WorkloadAuthTransport = "http" | "ws";
+export type WorkloadAuthOutcome = "jwt_valid" | "jwt_invalid" | "legacy_bare" | "token_absent";
+
+const verifyCounter = new Counter({
+  name: "workload_auth_verify_total",
+  help: "Runner-boundary token verification outcomes at the supervisor workload server",
+  labelNames: ["outcome", "transport", "env_type"] as const,
+  registers: [register],
+});
+
+// Exports the active mode (value 1 for the current WORKLOAD_TOKEN_ENFORCEMENT) so dashboards can show
+// disabled/log/enforce at a glance — the counters alone don't distinguish log from enforce.
+const enforcementModeGauge = new Gauge({
+  name: "workload_token_enforcement_mode",
+  help: "Active runner-boundary auth mode: value 1 for the label matching WORKLOAD_TOKEN_ENFORCEMENT",
+  labelNames: ["mode"] as const,
+  registers: [register],
+});
+enforcementModeGauge.set({ mode: env.WORKLOAD_TOKEN_ENFORCEMENT }, 1);
+
+export async function mintDeploymentToken(
+  claims: WorkloadDeploymentTokenInput
+): Promise<string | undefined> {
+  if (!workloadTokensEnabled || !secret) {
+    return undefined;
+  }
+
+  const token = await mintWorkloadDeploymentToken(claims, secret, tokenExpSeconds);
+  mintCounter.inc({ env_type: claims.environment_type });
+  return token;
+}
+
+export type VerifiedDeploymentHeader =
+  | { outcome: "jwt_valid"; claims: WorkloadDeploymentTokenClaims }
+  | { outcome: "jwt_invalid" | "legacy_bare" | "token_absent"; claims?: undefined };
+
+/**
+ * Verify the deployment-id header value and record the outcome. "jwt_valid" returns the claims so the
+ * caller can forward the verified environment_id upstream; other outcomes carry no trusted data.
+ */
+export async function verifyDeploymentIdHeader(
+  value: string | undefined,
+  transport: WorkloadAuthTransport
+): Promise<VerifiedDeploymentHeader> {
+  const result = await classify(value);
+  verifyCounter.inc({
+    outcome: result.outcome,
+    transport,
+    env_type: result.outcome === "jwt_valid" ? result.claims.environment_type : "unknown",
+  });
+  return result;
+}
+
+async function classify(value: string | undefined): Promise<VerifiedDeploymentHeader> {
+  if (!value || !secret) {
+    return { outcome: "token_absent" };
+  }
+
+  const result = await classifyDeploymentIdHeader(value, secret);
+
+  if (result.outcome === "jwt_valid" && result.claims) {
+    return { outcome: "jwt_valid", claims: result.claims };
+  }
+
+  return { outcome: result.outcome === "jwt_valid" ? "jwt_invalid" : result.outcome };
+}

apps/webapp/CLAUDE.md

@@ -91,24 +91,14 @@ Background job workers use `@trigger.dev/redis-worker`:
 - `app/v3/alertsWorker.server.ts`
 - `app/v3/batchTriggerWorker.server.ts`
 
-Do NOT add new jobs using zodworker/graphile-worker (legacy).
-
 ## Real-time
 
 - Socket.io: `app/v3/handleSocketIo.server.ts`, `app/v3/handleWebsockets.server.ts`
 - Electric SQL: Powers real-time data sync for the dashboard
 
-## Legacy V1 Code
+## v3 (engine V1) removed
 
-The `app/v3/` directory name is misleading - most code is actively used by V2. Only these specific files are V1-only legacy:
-- `app/v3/marqs/` (old MarQS queue system)
-- `app/v3/legacyRunEngineWorker.server.ts`
-- `app/v3/services/triggerTaskV1.server.ts`
-- `app/v3/services/cancelTaskRunV1.server.ts`
-- `app/v3/authenticatedSocketConnection.server.ts`
-- `app/v3/sharedSocketConnection.ts`
-
-Some services (e.g., `cancelTaskRun.server.ts`, `batchTriggerV3.server.ts`) branch on `RunEngineVersion` to support both V1 and V2. When editing these, only modify V2 code paths.
+v3 (engine V1: MarQS + Graphile worker) is end-of-life and its execution code is gone. The `app/v3/` directory name is historical; everything under it now serves V2. There is no V1 execution path: a `RunEngineVersion` `V1` branch (e.g. in `triggerTask.server.ts`, `cancelTaskRun.server.ts`) only rejects/finalizes gracefully so v3 clients get a clean 4xx, never a 5xx. Do not reintroduce V1. See `.claude/rules/legacy-v3-code.md` for the deprecation boundary.
 
 ## Performance: Trigger Hot Path
 
@@ -125,6 +115,16 @@ The `triggerTask.server.ts` service is the **highest-throughput code path** in t
 
 - **Always use `findFirst` instead of `findUnique`.** Prisma's `findUnique` has an implicit DataLoader that batches concurrent calls into a single `IN` query. This batching cannot be disabled and has active bugs even in Prisma 6.x: uppercase UUIDs returning null (#25484, confirmed 6.4.1), composite key SQL correctness issues (#22202), and 5-10x worse performance than manual DataLoader (#6573, open since 2021). `findFirst` is never batched and avoids this entire class of issues.
 
+## Transactions
+
+- **Always use the `$transaction` helper from `~/db.server`, never `prisma.$transaction` (or `$replica.$transaction`) directly.** The helper wraps the raw call with tracing (an OTEL span + an `isolation_level` attribute) and boundary logging for infrastructure errors (e.g. `PrismaClientInitializationError`) that the raw client swallows. Signature: `$transaction(prisma, name?, async (tx) => { ... }, options?)`.
+- Pass the isolation level via options as a string: `{ isolationLevel: "Serializable" }`. Reach for `Serializable` when a read-then-write must be atomic against concurrent transactions (e.g. a count-then-delete invariant); the loser of a race fails and can retry, which is the right trade for rare, correctness-critical paths.
+- The helper returns `R | undefined` — guard the result (`if (!result) throw ...`) when callers need a definite value.
+
+## PAT-authenticated API routes
+
+- **A PAT route must resolve its target org/project scoped to the caller's membership** (`members: { some: { userId } }`, or a helper like `findProjectByRef` / `resolveOrganizationForApiUser`). A PAT is user-scoped and can name any org/project by id/slug, and the OSS RBAC fallback ability is permissive — so `ability.can(...)` alone does NOT reject a non-member on self-hosted. The RBAC `authorization` gate enforces the *role*; the membership-scoped query is the *tenant* floor. Skipping it opens cross-org access on OSS.
+
 ## React Patterns
 
 - Only use `useCallback`/`useMemo` for context provider values, expensive derived data that is a dependency elsewhere, or stable refs required by a dependency array. Don't wrap ordinary event handlers or trivial computations.

apps/webapp/app/assets/icons/AvatarCircleIcon.tsx

@@ -1,4 +1,4 @@
-export function AvatarCircleIcon({ className }: { className?: string }) {
+function AvatarCircle({ className, strokeWidth }: { className?: string; strokeWidth: number }) {
   return (
     <svg
       className={className}
@@ -8,13 +8,28 @@ export function AvatarCircleIcon({ className }: { className?: string }) {
       fill="none"
       xmlns="http://www.w3.org/2000/svg"
     >
-      <circle cx="12" cy="12" r="9" stroke="currentColor" strokeWidth="2" />
-      <circle cx="12" cy="9.5" r="2.5" stroke="currentColor" strokeWidth="2" />
+      <circle cx="12" cy="12" r="9" stroke="currentColor" strokeWidth={strokeWidth} />
+      <circle cx="12" cy="9.5" r="2.5" stroke="currentColor" strokeWidth={strokeWidth} />
       <path
         d="M6 19C7.00156 16.6478 9.32233 15 12.0254 15C14.6837 15 16.9724 16.5938 18 18.884"
         stroke="currentColor"
-        strokeWidth="2"
+        strokeWidth={strokeWidth}
       />
     </svg>
   );
 }
+
+/** User avatar placeholder with a 2px stroke (the default). */
+export function AvatarCircleIcon({ className }: { className?: string }) {
+  return <AvatarCircle className={className} strokeWidth={2} />;
+}
+
+/** Thinner 1.5px-stroke variant of {@link AvatarCircleIcon}. */
+export function AvatarCircleIconThin({ className }: { className?: string }) {
+  return <AvatarCircle className={className} strokeWidth={1.5} />;
+}
+
+/** Thinnest 1.25px-stroke variant of {@link AvatarCircleIcon}. */
+export function AvatarCircleIconExtraThin({ className }: { className?: string }) {
+  return <AvatarCircle className={className} strokeWidth={1.25} />;
+}

apps/webapp/app/assets/icons/ChainLinkIcon.tsx

@@ -0,0 +1,27 @@
+export function ChainLinkIcon({ className }: { className?: string }) {
+  return (
+    <svg
+      className={className}
+      width="24"
+      height="24"
+      viewBox="0 0 24 24"
+      fill="none"
+      xmlns="http://www.w3.org/2000/svg"
+    >
+      <path
+        d="M10 19.0004L9.82843 19.1719C8.26634 20.734 5.73368 20.734 4.17158 19.1719L3.82843 18.8288C2.26634 17.2667 2.26633 14.734 3.82843 13.1719L7.17158 9.8288C8.73368 8.2667 11.2663 8.2667 12.8284 9.8288L13.1716 10.1719C13.8252 10.8256 14.2053 11.6491 14.312 12.5004"
+        stroke="currentColor"
+        strokeWidth="2"
+        strokeLinecap="round"
+        strokeLinejoin="round"
+      />
+      <path
+        d="M9.68799 12.5004C9.79463 13.3516 10.1748 14.1752 10.8284 14.8288L11.1715 15.1719C12.7336 16.734 15.2663 16.734 16.8284 15.1719L20.1715 11.8288C21.7336 10.2667 21.7336 7.73404 20.1715 6.17194L19.8284 5.8288C18.2663 4.2667 15.7336 4.2667 14.1715 5.8288L14 6.00037"
+        stroke="currentColor"
+        strokeWidth="2"
+        strokeLinecap="round"
+        strokeLinejoin="round"
+      />
+    </svg>
+  );
+}

apps/webapp/app/assets/icons/LeftSideMenuCollapsedIcon.tsx

@@ -0,0 +1,22 @@
+export function LeftSideMenuCollapsedIcon({ className }: { className?: string }) {
+  return (
+    <svg
+      className={className}
+      width="24"
+      height="24"
+      viewBox="0 0 24 24"
+      fill="none"
+      xmlns="http://www.w3.org/2000/svg"
+    >
+      <rect x="4" y="4" width="16" height="16" rx="3" stroke="currentColor" strokeWidth="2" />
+      <rect x="6" y="6" width="2" height="12" rx="1" fill="currentColor" />
+      <path
+        d="M12 14.5L14.5 12L12 9.5"
+        stroke="currentColor"
+        strokeWidth="2"
+        strokeLinecap="round"
+        strokeLinejoin="round"
+      />
+    </svg>
+  );
+}

apps/webapp/app/assets/icons/LeftSideMenuIcon.tsx

@@ -0,0 +1,44 @@
+import { motion } from "framer-motion";
+import { useState } from "react";
+
+export function LeftSideMenuIcon({
+  className,
+  hovered: controlledHovered,
+}: {
+  className?: string;
+  /** Drives the animation when provided (e.g. parent hover); otherwise the icon uses its own hover. */
+  hovered?: boolean;
+}) {
+  const [internalHovered, setInternalHovered] = useState(false);
+  const isControlled = controlledHovered !== undefined;
+  const hovered = isControlled ? controlledHovered : internalHovered;
+
+  return (
+    <svg
+      className={className}
+      width="24"
+      height="24"
+      viewBox="0 0 24 24"
+      fill="none"
+      xmlns="http://www.w3.org/2000/svg"
+      onMouseEnter={isControlled ? undefined : () => setInternalHovered(true)}
+      onMouseLeave={isControlled ? undefined : () => setInternalHovered(false)}
+    >
+      <rect x="4" y="4" width="16" height="16" rx="2" stroke="currentColor" strokeWidth="2" />
+      {/* Animate a transform (scaleX), not the SVG `width` attr — framer snaps the first animation
+          of an idle SVG geometry attribute. Left origin collapses the panel right-to-left. */}
+      <motion.rect
+        x="6"
+        y="6"
+        width="5"
+        height="12"
+        rx="1"
+        fill="currentColor"
+        initial={false}
+        style={{ originX: 0 }}
+        animate={{ scaleX: hovered ? 0.2 : 1 }}
+        transition={{ duration: 0.3, ease: "easeInOut" }}
+      />
+    </svg>
+  );
+}

apps/webapp/app/components/AskAI.tsx

@@ -11,11 +11,12 @@ import { useSearchParams } from "@remix-run/react";
 import DOMPurify from "dompurify";
 import { motion } from "framer-motion";
 import { marked } from "marked";
-import { useCallback, useEffect, useRef, useState } from "react";
+import { type ReactNode, useCallback, useEffect, useRef, useState } from "react";
 import { useTypedRouteLoaderData } from "remix-typedjson";
 import { AISparkleIcon } from "~/assets/icons/AISparkleIcon";
 import { SparkleListIcon } from "~/assets/icons/SparkleListIcon";
 import { useFeatures } from "~/hooks/useFeatures";
+import { useShortcutKeys } from "~/hooks/useShortcutKeys";
 import { type loader } from "~/root";
 import { Button } from "./primitives/Buttons";
 import { Callout } from "./primitives/Callout";
@@ -38,6 +39,104 @@ function useKapaWebsiteId() {
   return routeMatch?.kapa.websiteId;
 }
 
+/** Open/close state for the Ask AI dialog, including the `?aiHelp=` deep-link handling. */
+function useAskAIState() {
+  const [isOpen, setIsOpen] = useState(false);
+  const [initialQuery, setInitialQuery] = useState<string | undefined>();
+  const [searchParams, setSearchParams] = useSearchParams();
+
+  const openAskAI = useCallback((question?: string) => {
+    if (question) {
+      setInitialQuery(question);
+    } else {
+      setInitialQuery(undefined);
+    }
+    setIsOpen(true);
+  }, []);
+
+  const closeAskAI = useCallback(() => {
+    setIsOpen(false);
+    setInitialQuery(undefined);
+  }, []);
+
+  // Handle URL param functionality
+  useEffect(() => {
+    const aiHelp = searchParams.get("aiHelp");
+    if (aiHelp) {
+      // Delay to avoid hCaptcha bot detection
+      window.setTimeout(() => openAskAI(aiHelp), 1000);
+
+      // Clone instead of mutating in place
+      const next = new URLSearchParams(searchParams);
+      next.delete("aiHelp");
+      setSearchParams(next);
+    }
+  }, [searchParams, openAskAI]);
+
+  return { isOpen, setIsOpen, initialQuery, openAskAI, closeAskAI };
+}
+
+/**
+ * Hosts Ask AI (Kapa provider, ⌘I shortcut, dialog) for a menu that renders its own trigger. Wrap
+ * it around the popover, not inside, so the dialog and shortcut survive the popover closing.
+ * `children` receives the open function, or undefined when Ask AI is unavailable (self-hosted, no
+ * Kapa website id, or SSR).
+ */
+export function AskAIRoot({
+  children,
+}: {
+  children: (openAskAI: (() => void) | undefined) => ReactNode;
+}) {
+  const { isManagedCloud } = useFeatures();
+  const websiteId = useKapaWebsiteId();
+
+  if (!isManagedCloud || !websiteId) {
+    return <>{children(undefined)}</>;
+  }
+
+  return (
+    <ClientOnly fallback={<>{children(undefined)}</>}>
+      {() => <AskAIRootProvider websiteId={websiteId}>{children}</AskAIRootProvider>}
+    </ClientOnly>
+  );
+}
+
+function AskAIRootProvider({
+  websiteId,
+  children,
+}: {
+  websiteId: string;
+  children: (openAskAI: () => void) => ReactNode;
+}) {
+  const { isOpen, setIsOpen, initialQuery, openAskAI, closeAskAI } = useAskAIState();
+
+  useShortcutKeys({
+    shortcut: { modifiers: ["mod"], key: "i", enabledOnInputElements: true },
+    action: () => openAskAI(),
+  });
+
+  return (
+    <KapaProvider
+      integrationId={websiteId}
+      callbacks={{
+        askAI: {
+          onQuerySubmit: () => openAskAI(),
+          onAnswerGenerationCompleted: () => openAskAI(),
+        },
+      }}
+      botProtectionMechanism="hcaptcha"
+    >
+      {children(() => openAskAI())}
+      <AskAIDialog
+        initialQuery={initialQuery}
+        isOpen={isOpen}
+        onOpenChange={setIsOpen}
+        closeAskAI={closeAskAI}
+      />
+    </KapaProvider>
+  );
+}
+
 export function AskAI({ isCollapsed = false }: { isCollapsed?: boolean }) {
   const { isManagedCloud } = useFeatures();
   const websiteId = useKapaWebsiteId();
@@ -72,37 +171,7 @@ type AskAIProviderProps = {
 };
 
 function AskAIProvider({ websiteId, isCollapsed = false }: AskAIProviderProps) {
-  const [isOpen, setIsOpen] = useState(false);
-  const [initialQuery, setInitialQuery] = useState<string | undefined>();
-  const [searchParams, setSearchParams] = useSearchParams();
-
-  const openAskAI = useCallback((question?: string) => {
-    if (question) {
-      setInitialQuery(question);
-    } else {
-      setInitialQuery(undefined);
-    }
-    setIsOpen(true);
-  }, []);
-
-  const closeAskAI = useCallback(() => {
-    setIsOpen(false);
-    setInitialQuery(undefined);
-  }, []);
-
-  // Handle URL param functionality
-  useEffect(() => {
-    const aiHelp = searchParams.get("aiHelp");
-    if (aiHelp) {
-      // Delay to avoid hCaptcha bot detection
-      window.setTimeout(() => openAskAI(aiHelp), 1000);
-
-      // Clone instead of mutating in place
-      const next = new URLSearchParams(searchParams);
-      next.delete("aiHelp");
-      setSearchParams(next);
-    }
-  }, [searchParams, openAskAI]);
+  const { isOpen, setIsOpen, initialQuery, openAskAI, closeAskAI } = useAskAIState();
 
   return (
     <KapaProvider

apps/webapp/app/components/BlankStatePanels.tsx

@@ -748,11 +748,7 @@ export function PromptsNone() {
       iconClassName="text-aiPrompts"
       panelClassName="max-w-lg"
       accessory={
-        <LinkButton
-          to={docsPath("prompt-management")}
-          variant="docs/small"
-          LeadingIcon={BookOpenIcon}
-        >
+        <LinkButton to={docsPath("ai/prompts")} variant="docs/small" LeadingIcon={BookOpenIcon}>
           Prompts docs
         </LinkButton>
       }

apps/webapp/app/components/GlobalShortcuts.tsx

@@ -0,0 +1,40 @@
+import { useNavigate, useSubmit } from "@remix-run/react";
+import { useEffect } from "react";
+import { useIsImpersonating } from "~/hooks/useOrganizations";
+import { useOptionalUser } from "~/hooks/useUser";
+import { adminPath } from "~/utils/pathBuilder";
+
+/** App-wide keyboard shortcuts, mounted once at the root so they work everywhere. Renders nothing. */
+export function GlobalShortcuts() {
+  const user = useOptionalUser();
+  const isImpersonating = useIsImpersonating();
+  const navigate = useNavigate();
+  const submit = useSubmit();
+
+  const isAdmin = Boolean(user?.admin) || isImpersonating;
+
+  useEffect(() => {
+    if (!isAdmin) return;
+
+    const onKeyDown = (event: KeyboardEvent) => {
+      // Admin escape hatch: Cmd+Option+A (Ctrl+Alt+A on Windows) opens the admin dashboard, or stops
+      // impersonating. Avoids Escape — Chrome/macOS never delivers a keydown for Escape+modifier (why
+      // the old Cmd+Esc did nothing). Matched on `event.code`, not `event.key`, because Option makes
+      // "A" report "å" (so a raw listener, not the `event.key`-based useShortcutKeys hook).
+      if (event.code !== "KeyA" || !event.altKey || !(event.metaKey || event.ctrlKey)) {
+        return;
+      }
+      event.preventDefault();
+      if (isImpersonating) {
+        submit(null, { action: "/resources/impersonation", method: "delete" });
+      } else {
+        navigate(adminPath());
+      }
+    };
+
+    document.addEventListener("keydown", onKeyDown);
+    return () => document.removeEventListener("keydown", onKeyDown);
+  }, [isAdmin, isImpersonating, navigate, submit]);
+
+  return null;
+}

apps/webapp/app/components/LoginPageLayout.tsx

@@ -36,7 +36,14 @@ const quotes: QuoteType[] = [
   },
 ];
 
-export function LoginPageLayout({ children }: { children: React.ReactNode }) {
+export function LoginPageLayout({
+  children,
+  rightContent,
+}: {
+  children: React.ReactNode;
+  /** Replaces the default testimonials panel on the right (e.g. a promo highlight). */
+  rightContent?: React.ReactNode;
+}) {
   const [randomQuote, setRandomQuote] = useState<QuoteType | null>(null);
   useEffect(() => {
     const randomIndex = Math.floor(Math.random() * quotes.length);
@@ -62,23 +69,27 @@ export function LoginPageLayout({ children }: { children: React.ReactNode }) {
         </div>
       </div>
       <div className="hidden grid-rows-[1fr_auto] pb-6 lg:grid">
-        <div className="flex h-full flex-col items-center justify-center px-16">
-          <Header3 className="relative text-center text-2xl font-normal leading-8 text-text-dimmed transition before:relative before:right-1 before:top-0 before:text-6xl before:text-charcoal-750 before:content-['❝'] lg-height:text-xl md-height:text-lg">
-            {randomQuote?.quote}
-          </Header3>
-          <Paragraph className="mt-4 text-text-dimmed/60">{randomQuote?.person}</Paragraph>
-        </div>
-        <div className="flex flex-col items-center gap-4 px-8">
-          <Paragraph>Trusted by developers at</Paragraph>
-          <div className="flex w-full flex-wrap items-center justify-center gap-x-6 gap-y-3 text-text-faint xl:justify-between xl:gap-0">
-            <LyftLogo className="w-11" />
-            <UnkeyLogo />
-            <MiddayLogo />
-            <AppsmithLogo />
-            <CalComLogo />
-            <TldrawLogo />
-          </div>
-        </div>
+        {rightContent ?? (
+          <>
+            <div className="flex h-full flex-col items-center justify-center px-16">
+              <Header3 className="relative text-center text-2xl font-normal leading-8 text-text-dimmed transition before:relative before:right-1 before:top-0 before:text-6xl before:text-charcoal-750 before:content-['❝'] lg-height:text-xl md-height:text-lg">
+                {randomQuote?.quote}
+              </Header3>
+              <Paragraph className="mt-4 text-text-dimmed/60">{randomQuote?.person}</Paragraph>
+            </div>
+            <div className="flex flex-col items-center gap-4 px-8">
+              <Paragraph>Trusted by developers at</Paragraph>
+              <div className="flex w-full flex-wrap items-center justify-center gap-x-6 gap-y-3 text-text-faint xl:justify-between xl:gap-0">
+                <LyftLogo className="w-11" />
+                <UnkeyLogo />
+                <MiddayLogo />
+                <AppsmithLogo />
+                <CalComLogo />
+                <TldrawLogo />
+              </div>
+            </div>
+          </>
+        )}
       </div>
     </main>
   );

apps/webapp/app/components/Shortcuts.tsx

@@ -1,8 +1,8 @@
 import { KeyboardIcon } from "~/assets/icons/KeyboardIcon";
 import { useState } from "react";
 import { useShortcutKeys } from "~/hooks/useShortcutKeys";
-import { Button } from "./primitives/Buttons";
 import { Header3 } from "./primitives/Headers";
+import { SideMenuItemButton } from "./navigation/SideMenuItem";
 import { Paragraph } from "./primitives/Paragraph";
 import { Sheet, SheetContent, SheetHeader, SheetTitle, SheetTrigger } from "./primitives/SheetV3";
 import { ShortcutKey } from "./primitives/ShortcutKey";
@@ -11,19 +11,12 @@ export function Shortcuts() {
   return (
     <Sheet>
       <SheetTrigger asChild>
-        <Button
-          variant="small-menu-item"
-          LeadingIcon={KeyboardIcon}
-          leadingIconClassName="text-text-dimmed group-hover/button:text-text-bright"
+        <SideMenuItemButton
+          icon={KeyboardIcon}
+          name="Shortcuts"
           data-action="shortcuts"
-          fullWidth
-          textAlignLeft
-          shortcut={{ modifiers: ["shift"], key: "?", enabled: false }}
-          className="gap-x-0 pl-1.5"
-          iconSpacing="gap-x-1.5"
-        >
-          Shortcuts
-        </Button>
+          trailing={<ShortcutKey shortcut={{ modifiers: ["shift"], key: "?" }} variant="medium" />}
+        />
       </SheetTrigger>
       <ShortcutContent />
     </Sheet>

apps/webapp/app/components/StaleAssetRecovery.test.ts

@@ -0,0 +1,56 @@
+// @vitest-environment jsdom
+import { afterEach, beforeEach, describe, expect, it, vi } from "vitest";
+import { staleAssetRecoveryScript } from "./StaleAssetRecovery";
+
+// Each staleAssetRecoveryScript() call models a fresh page load: it reads the shared
+// sessionStorage budget and returns its own `recover`. We drive recover() directly rather
+// than dispatching resource-error events, so accumulated window listeners never fire.
+describe("staleAssetRecoveryScript", () => {
+  let reload: ReturnType<typeof vi.fn>;
+
+  beforeEach(() => {
+    sessionStorage.clear();
+    reload = vi.fn();
+    vi.stubGlobal("location", { reload });
+    vi.stubGlobal("navigator", { onLine: true });
+  });
+
+  afterEach(() => {
+    vi.unstubAllGlobals();
+    vi.restoreAllMocks();
+  });
+
+  it("reloads on a recovery", () => {
+    staleAssetRecoveryScript().recover();
+    expect(reload).toHaveBeenCalledTimes(1);
+  });
+
+  it("reloads only once per page even if several assets fail (re-entrancy guard)", () => {
+    const { recover } = staleAssetRecoveryScript();
+    recover();
+    recover();
+    recover();
+    expect(reload).toHaveBeenCalledTimes(1);
+  });
+
+  it("stops reloading once the budget is spent across reloads", () => {
+    staleAssetRecoveryScript().recover(); // reload 1
+    staleAssetRecoveryScript().recover(); // reload 2
+    staleAssetRecoveryScript().recover(); // budget spent -> no reload
+    expect(reload).toHaveBeenCalledTimes(2);
+  });
+
+  it("does not reload when offline", () => {
+    vi.stubGlobal("navigator", { onLine: false });
+    staleAssetRecoveryScript().recover();
+    expect(reload).not.toHaveBeenCalled();
+  });
+
+  it("does not reload when sessionStorage is unavailable", () => {
+    vi.spyOn(Storage.prototype, "setItem").mockImplementation(() => {
+      throw new Error("blocked");
+    });
+    staleAssetRecoveryScript().recover();
+    expect(reload).not.toHaveBeenCalled();
+  });
+});

apps/webapp/app/components/StaleAssetRecovery.tsx

@@ -0,0 +1,96 @@
+// Recovers from a rolling deploy rotating the content-hashed /assets files out from
+// under a page. Each image serves only its own build and hard-404s unknown hashes, so
+// a client can request a hash the serving replica doesn't have and get missing styles
+// or a failed asset load. On such an asset load failure we do a bounded full document
+// reload: the fresh document (and, under sticky routing, all of its assets) lands on a
+// single live build, so the asset resolves. Bounded via sessionStorage so it can never
+// loop; when the budget is spent it stops rather than reloading forever.
+//
+// Deliberately minimal — no fetch interception, no build-version polling, no server
+// build-id contract, no form snapshot, no blocking overlay.
+
+// The recovery logic runs as an inline <script> injected before <Links /> (see the
+// component below), so it must execute before the app bundle and before the stylesheet
+// can fail to load. It is authored as a normal, type-checked and lint-checked function
+// and serialized with .toString() at render time — NOT hand-written into a string — so
+// the logic is real code the compiler and linter can see. Because it is serialized, it
+// must stay fully self-contained: no imports, no references to module scope, and plain
+// ES that the bundler won't rewrite to reach a hoisted helper. It returns its `recover`
+// closure purely so the unit test can drive the logic directly (the inline IIFE that
+// runs in the browser ignores the return value).
+export function staleAssetRecoveryScript() {
+  var KEY = "trigger:assetReload";
+  var MAX_RELOADS = 2;
+  var WINDOW_MS = 300000;
+  var recovering = false;
+
+  function budgetAllows() {
+    try {
+      var raw = sessionStorage.getItem(KEY);
+      var state = raw ? (JSON.parse(raw) as { n: number; t: number }) : { n: 0, t: 0 };
+      if (Date.now() - state.t > WINDOW_MS) state = { n: 0, t: 0 };
+      if (state.n >= MAX_RELOADS) return false;
+      sessionStorage.setItem(KEY, JSON.stringify({ n: state.n + 1, t: Date.now() }));
+      return true;
+    } catch {
+      // Storage blocked (private mode / quota): can't bound reloads, so don't auto-reload.
+      return false;
+    }
+  }
+
+  function recover() {
+    // One recovery per page: a broken load fails several hashed assets at once and each
+    // fires its own error event before location.reload() commits — without this guard a
+    // single incident would burn the entire reload budget.
+    if (recovering) return;
+    recovering = true;
+    // Don't reload into the browser's offline error page.
+    if (navigator.onLine === false) return;
+    if (budgetAllows()) location.reload();
+  }
+
+  // Non-bubbling resource load failures (stylesheet, modulepreload, entry <script>) at
+  // document load — the failure class nothing else covers. Capture phase is required.
+  window.addEventListener(
+    "error",
+    function (event) {
+      var el = event.target as Element | null;
+      if (!el || typeof el.tagName !== "string") return; // window/global errors have no tagName
+      var url =
+        el.tagName === "LINK"
+          ? (el as HTMLLinkElement).href
+          : el.tagName === "SCRIPT"
+            ? (el as HTMLScriptElement).src
+            : null;
+      // Match the pathname, not the full URL — a query string or third-party
+      // URL containing /assets/ must not burn the reload budget.
+      if (url && new URL(url, location.href).pathname.indexOf("/assets/") !== -1) recover();
+    },
+    true
+  );
+
+  // Raw dynamic import() failures in app code. (Remix reloads its own route chunks, so
+  // that path rarely reaches here.) The message URL isn't reliable cross-browser, so
+  // match the chunk-load error shape; the once-guard + bounded budget make a rare stray
+  // reload harmless.
+  window.addEventListener("unhandledrejection", function (event) {
+    var message = (event.reason && event.reason.message) || "";
+    if (
+      /dynamically imported module|Importing a module script failed|ChunkLoadError/i.test(message)
+    ) {
+      recover();
+    }
+  });
+
+  return { recover };
+}
+
+export function StaleAssetRecovery({ isProduction }: { isProduction: boolean }) {
+  if (!isProduction) {
+    return null;
+  }
+
+  return (
+    <script dangerouslySetInnerHTML={{ __html: `(${staleAssetRecoveryScript.toString()})()` }} />
+  );
+}

apps/webapp/app/components/UserProfilePhoto.tsx

@@ -1,31 +1,62 @@
-import { UserCircleIcon } from "@heroicons/react/24/solid";
+import {
+  AvatarCircleIcon,
+  AvatarCircleIconExtraThin,
+  AvatarCircleIconThin,
+} from "~/assets/icons/AvatarCircleIcon";
 import { useOptionalUser } from "~/hooks/useUser";
 import { cn } from "~/utils/cn";
 
-export function UserProfilePhoto({ className }: { className?: string }) {
+/** Stroke width (px) of the placeholder avatar icon shown when there is no photo. */
+type AvatarStrokeWidth = 1.25 | 1.5 | 2;
+
+const PLACEHOLDER_BY_STROKE_WIDTH = {
+  1.25: AvatarCircleIconExtraThin,
+  1.5: AvatarCircleIconThin,
+  2: AvatarCircleIcon,
+} as const;
+
+export function UserProfilePhoto({
+  className,
+  strokeWidth = 2,
+}: {
+  className?: string;
+  strokeWidth?: AvatarStrokeWidth;
+}) {
   const user = useOptionalUser();
-  return <UserAvatar avatarUrl={user?.avatarUrl} name={user?.name} className={className} />;
+  return (
+    <UserAvatar
+      avatarUrl={user?.avatarUrl}
+      name={user?.name}
+      className={className}
+      strokeWidth={strokeWidth}
+    />
+  );
 }
 
 export function UserAvatar({
   avatarUrl,
   name,
   className,
+  strokeWidth = 2,
 }: {
   avatarUrl?: string | null;
   name?: string | null;
   className?: string;
+  strokeWidth?: AvatarStrokeWidth;
 }) {
-  return avatarUrl ? (
-    <div className={cn("grid aspect-square place-items-center", className)}>
-      <img
-        className={cn("aspect-square rounded-full p-[7%]")}
-        src={avatarUrl}
-        alt={name ?? "User"}
-        referrerPolicy="no-referrer"
-      />
-    </div>
-  ) : (
-    <UserCircleIcon className={cn("aspect-square text-text-dimmed", className)} />
-  );
+  if (avatarUrl) {
+    return (
+      <div className={cn("grid aspect-square place-items-center", className)}>
+        <img
+          className={cn("aspect-square rounded-full p-[7%]")}
+          src={avatarUrl}
+          alt={name ?? "User"}
+          referrerPolicy="no-referrer"
+        />
+      </div>
+    );
+  }
+
+  const PlaceholderIcon = PLACEHOLDER_BY_STROKE_WIDTH[strokeWidth];
+  return <PlaceholderIcon className={cn("aspect-square text-text-dimmed", className)} />;
 }

apps/webapp/app/components/admin/debugRun.tsx

@@ -10,7 +10,6 @@ import { useEffect } from "react";
 import { Spinner } from "../primitives/Spinner";
 import * as Property from "~/components/primitives/PropertyTable";
 import { ClipboardField } from "../primitives/ClipboardField";
-import { MarQSShortKeyProducer } from "~/v3/marqs/marqsKeyProducer";
 
 export function AdminDebugRun({ friendlyId }: { friendlyId: string }) {
   const hasAdminAccess = useHasAdminAccess();
@@ -69,26 +68,13 @@ function DebugRunContent({ friendlyId }: { friendlyId: string }) {
 
 function DebugRunData(props: UseDataFunctionReturn<typeof loader>) {
   if (props.engine === "V1") {
-    return <DebugRunDataEngineV1 {...props} />;
+    return <DebugRunDataEngineV1 run={props.run} />;
   }
 
   return <DebugRunDataEngineV2 {...props} />;
 }
 
-function DebugRunDataEngineV1({
-  run,
-  environment,
-  queueConcurrencyLimit,
-  queueCurrentConcurrency,
-  envConcurrencyLimit,
-  envCurrentConcurrency,
-  queueReserveConcurrency,
-  envReserveConcurrency,
-}: UseDataFunctionReturn<typeof loader>) {
-  const keys = new MarQSShortKeyProducer("marqs:");
-
-  const withPrefix = (key: string) => `marqs:${key}`;
-
+function DebugRunDataEngineV1({ run }: { run: UseDataFunctionReturn<typeof loader>["run"] }) {
   return (
     <Property.Table>
       <Property.Item>
@@ -98,247 +84,9 @@ function DebugRunDataEngineV1({
         </Property.Value>
       </Property.Item>
       <Property.Item>
-        <Property.Label>Message key</Property.Label>
-        <Property.Value className="flex items-center gap-2">
-          <ClipboardField
-            value={withPrefix(keys.messageKey(run.id))}
-            variant="tertiary/small"
-            iconButton
-          />
-        </Property.Value>
-      </Property.Item>
-      <Property.Item>
-        <Property.Label>GET message</Property.Label>
-        <Property.Value className="flex items-center gap-2">
-          <ClipboardField
-            value={`GET ${withPrefix(keys.messageKey(run.id))}`}
-            variant="tertiary/small"
-            iconButton
-          />
-        </Property.Value>
-      </Property.Item>
-      <Property.Item>
-        <Property.Label>Queue key</Property.Label>
-        <Property.Value className="flex items-center gap-2">
-          <ClipboardField
-            value={withPrefix(
-              keys.queueKey(environment, run.queue, run.concurrencyKey ?? undefined)
-            )}
-            variant="tertiary/small"
-            iconButton
-          />
-        </Property.Value>
-      </Property.Item>
-      <Property.Item>
-        <Property.Label>Get queue set</Property.Label>
-        <Property.Value className="flex items-center gap-2">
-          <ClipboardField
-            value={`ZRANGE ${withPrefix(
-              keys.queueKey(environment, run.queue, run.concurrencyKey ?? undefined)
-            )} 0 -1`}
-            variant="tertiary/small"
-            iconButton
-          />
-        </Property.Value>
-      </Property.Item>
-      <Property.Item>
-        <Property.Label>Queue current concurrency key</Property.Label>
-        <Property.Value className="flex items-center gap-2">
-          <ClipboardField
-            value={withPrefix(
-              keys.queueCurrentConcurrencyKey(
-                environment,
-                run.queue,
-                run.concurrencyKey ?? undefined
-              )
-            )}
-            variant="tertiary/small"
-            iconButton
-          />
-        </Property.Value>
-      </Property.Item>
-
-      <Property.Item>
-        <Property.Label>Get queue current concurrency</Property.Label>
-        <Property.Value className="flex items-center gap-2">
-          <ClipboardField
-            value={`SMEMBERS ${withPrefix(
-              keys.queueCurrentConcurrencyKey(
-                environment,
-                run.queue,
-                run.concurrencyKey ?? undefined
-              )
-            )}`}
-            variant="tertiary/small"
-            iconButton
-          />
-        </Property.Value>
-      </Property.Item>
-      <Property.Item>
-        <Property.Label>Queue current concurrency</Property.Label>
-        <Property.Value className="flex items-center gap-2">
-          <span>{queueCurrentConcurrency ?? "0"}</span>
-        </Property.Value>
-      </Property.Item>
-      <Property.Item>
-        <Property.Label>Queue reserve concurrency key</Property.Label>
-        <Property.Value className="flex items-center gap-2">
-          <ClipboardField
-            value={withPrefix(
-              keys.queueReserveConcurrencyKeyFromQueue(
-                keys.queueKey(environment, run.queue, run.concurrencyKey ?? undefined)
-              )
-            )}
-            variant="tertiary/small"
-            iconButton
-          />
-        </Property.Value>
-      </Property.Item>
-
-      <Property.Item>
-        <Property.Label>Get queue reserve concurrency</Property.Label>
-        <Property.Value className="flex items-center gap-2">
-          <ClipboardField
-            value={`SMEMBERS ${withPrefix(
-              keys.queueReserveConcurrencyKeyFromQueue(
-                keys.queueKey(environment, run.queue, run.concurrencyKey ?? undefined)
-              )
-            )}`}
-            variant="tertiary/small"
-            iconButton
-          />
-        </Property.Value>
-      </Property.Item>
-      <Property.Item>
-        <Property.Label>Queue reserve concurrency</Property.Label>
-        <Property.Value className="flex items-center gap-2">
-          <span>{queueReserveConcurrency ?? "0"}</span>
-        </Property.Value>
-      </Property.Item>
-      <Property.Item>
-        <Property.Label>Queue concurrency limit key</Property.Label>
-        <Property.Value className="flex items-center gap-2">
-          <ClipboardField
-            value={withPrefix(keys.queueConcurrencyLimitKey(environment, run.queue))}
-            variant="tertiary/small"
-            iconButton
-          />
-        </Property.Value>
-      </Property.Item>
-      <Property.Item>
-        <Property.Label>GET queue concurrency limit</Property.Label>
-        <Property.Value className="flex items-center gap-2">
-          <ClipboardField
-            value={`GET ${withPrefix(keys.queueConcurrencyLimitKey(environment, run.queue))}`}
-            variant="tertiary/small"
-            iconButton
-          />
-        </Property.Value>
-      </Property.Item>
-      <Property.Item>
-        <Property.Label>Queue concurrency limit</Property.Label>
-        <Property.Value className="flex items-center gap-2">
-          <span>{queueConcurrencyLimit ?? "Not set"}</span>
-        </Property.Value>
-      </Property.Item>
-      <Property.Item>
-        <Property.Label>Env current concurrency key</Property.Label>
-        <Property.Value className="flex items-center gap-2">
-          <ClipboardField
-            value={withPrefix(keys.envCurrentConcurrencyKey(environment))}
-            variant="tertiary/small"
-            iconButton
-          />
-        </Property.Value>
-      </Property.Item>
-      <Property.Item>
-        <Property.Label>Get env current concurrency</Property.Label>
-        <Property.Value className="flex items-center gap-2">
-          <ClipboardField
-            value={`SMEMBERS ${withPrefix(keys.envCurrentConcurrencyKey(environment))}`}
-            variant="tertiary/small"
-            iconButton
-          />
-        </Property.Value>
-      </Property.Item>
-      <Property.Item>
-        <Property.Label>Env current concurrency</Property.Label>
-        <Property.Value className="flex items-center gap-2">
-          <span>{envCurrentConcurrency ?? "0"}</span>
-        </Property.Value>
-      </Property.Item>
-      <Property.Item>
-        <Property.Label>Env reserve concurrency key</Property.Label>
-        <Property.Value className="flex items-center gap-2">
-          <ClipboardField
-            value={withPrefix(keys.envReserveConcurrencyKey(environment.id))}
-            variant="tertiary/small"
-            iconButton
-          />
-        </Property.Value>
-      </Property.Item>
-      <Property.Item>
-        <Property.Label>Get env reserve concurrency</Property.Label>
-        <Property.Value className="flex items-center gap-2">
-          <ClipboardField
-            value={`SMEMBERS ${withPrefix(keys.envReserveConcurrencyKey(environment.id))}`}
-            variant="tertiary/small"
-            iconButton
-          />
-        </Property.Value>
-      </Property.Item>
-      <Property.Item>
-        <Property.Label>Env reserve concurrency</Property.Label>
-        <Property.Value className="flex items-center gap-2">
-          <span>{envReserveConcurrency ?? "0"}</span>
-        </Property.Value>
-      </Property.Item>
-      <Property.Item>
-        <Property.Label>Env concurrency limit key</Property.Label>
-        <Property.Value className="flex items-center gap-2">
-          <ClipboardField
-            value={withPrefix(keys.envConcurrencyLimitKey(environment))}
-            variant="tertiary/small"
-            iconButton
-          />
-        </Property.Value>
-      </Property.Item>
-      <Property.Item>
-        <Property.Label>GET env concurrency limit</Property.Label>
-        <Property.Value className="flex items-center gap-2">
-          <ClipboardField
-            value={`GET ${withPrefix(keys.envConcurrencyLimitKey(environment))}`}
-            variant="tertiary/small"
-            iconButton
-          />
-        </Property.Value>
-      </Property.Item>
-      <Property.Item>
-        <Property.Label>Env concurrency limit</Property.Label>
-        <Property.Value className="flex items-center gap-2">
-          <span>{envConcurrencyLimit ?? "Not set"}</span>
-        </Property.Value>
-      </Property.Item>
-      <Property.Item>
-        <Property.Label>Shared queue key</Property.Label>
-        <Property.Value className="flex items-center gap-2">
-          <ClipboardField
-            value={`GET ${withPrefix(keys.envSharedQueueKey(environment))}`}
-            variant="tertiary/small"
-            iconButton
-          />
-        </Property.Value>
-      </Property.Item>
-      <Property.Item>
-        <Property.Label>Get shared queue set</Property.Label>
-        <Property.Value className="flex items-center gap-2">
-          <ClipboardField
-            value={`ZRANGEBYSCORE ${withPrefix(
-              keys.envSharedQueueKey(environment)
-            )} -inf ${Date.now()} WITHSCORES`}
-            variant="tertiary/small"
-            iconButton
-          />
+        <Property.Label>Engine</Property.Label>
+        <Property.Value>
+          Engine V1 (v3) is retired. Queue debug data is no longer available for V1 runs.
         </Property.Value>
       </Property.Item>
     </Property.Table>
@@ -352,7 +100,7 @@ function DebugRunDataEngineV2({
   envConcurrencyLimit,
   envCurrentConcurrency,
   keys,
-}: UseDataFunctionReturn<typeof loader>) {
+}: Extract<UseDataFunctionReturn<typeof loader>, { engine: "V2" }>) {
   return (
     <Property.Table>
       <Property.Item>

apps/webapp/app/components/admin/debugTooltip.tsx

@@ -1,4 +1,5 @@
 import { ShieldCheckIcon } from "@heroicons/react/20/solid";
+import { CopyableText } from "~/components/primitives/CopyableText";
 import * as Property from "~/components/primitives/PropertyTable";
 import {
   Tooltip,
@@ -25,7 +26,10 @@ export function AdminDebugTooltip({ children }: { children?: React.ReactNode })
         <TooltipTrigger>
           <ShieldCheckIcon className="size-5" />
         </TooltipTrigger>
-        <TooltipContent className="max-h-[90vh] overflow-y-auto">
+        {/* The copy controls below pass `hideTooltip` so their own tooltips don't fire
+            Radix's global close and dismiss this panel. `pr-8` leaves room for the
+            copy button, which is absolutely positioned to the right of each value. */}
+        <TooltipContent className="max-h-[90vh] overflow-y-auto pr-8">
           <Content>{children}</Content>
         </TooltipContent>
       </Tooltip>
@@ -44,23 +48,31 @@ function Content({ children }: { children: React.ReactNode }) {
       <Property.Table>
         <Property.Item>
           <Property.Label>User ID</Property.Label>
-          <Property.Value>{user.id}</Property.Value>
+          <Property.Value>
+            <CopyableText value={user.id} asChild hideTooltip />
+          </Property.Value>
         </Property.Item>
         {organization && (
           <Property.Item>
             <Property.Label>Org ID</Property.Label>
-            <Property.Value>{organization.id}</Property.Value>
+            <Property.Value>
+              <CopyableText value={organization.id} asChild hideTooltip />
+            </Property.Value>
           </Property.Item>
         )}
         {project && (
           <>
             <Property.Item>
               <Property.Label>Project ID</Property.Label>
-              <Property.Value>{project.id}</Property.Value>
+              <Property.Value>
+                <CopyableText value={project.id} asChild hideTooltip />
+              </Property.Value>
             </Property.Item>
             <Property.Item>
               <Property.Label>Project ref</Property.Label>
-              <Property.Value>{project.externalRef}</Property.Value>
+              <Property.Value>
+                <CopyableText value={project.externalRef} asChild hideTooltip />
+              </Property.Value>
             </Property.Item>
           </>
         )}
@@ -68,7 +80,9 @@ function Content({ children }: { children: React.ReactNode }) {
           <>
             <Property.Item>
               <Property.Label>Environment ID</Property.Label>
-              <Property.Value>{environment.id}</Property.Value>
+              <Property.Value>
+                <CopyableText value={environment.id} asChild hideTooltip />
+              </Property.Value>
             </Property.Item>
             <Property.Item>
               <Property.Label>Environment type</Property.Label>
@@ -81,7 +95,7 @@ function Content({ children }: { children: React.ReactNode }) {
           </>
         )}
       </Property.Table>
-      <div className="pt-2">{children}</div>
+      {children && <div className="pt-2">{children}</div>}
     </div>
   );
 }

apps/webapp/app/components/billing/BillingLimitConfigSection.tsx

@@ -7,6 +7,7 @@ import { z } from "zod";
 import { getBillingLimitMode } from "~/components/billing/billingAlertsFormat";
 import { formatGracePeriodMs } from "~/components/billing/billingLimitFormat";
 import { AnimatedCallout } from "~/components/primitives/AnimatedCallout";
+import { Callout } from "~/components/primitives/Callout";
 import { Button } from "~/components/primitives/Buttons";
 import { CheckboxWithLabel } from "~/components/primitives/Checkbox";
 import { Fieldset } from "~/components/primitives/Fieldset";
@@ -51,10 +52,14 @@ type BillingLimitActionData = {
 
 export function isBillingLimitFormDirty(input: {
   billingLimit: BillingLimitResult;
-  mode: "none" | "plan" | "custom";
+  mode: "" | "none" | "plan" | "custom";
   customAmount: string;
   cancelInProgressRuns: boolean;
 }): boolean {
+  if (input.mode === "") {
+    return false;
+  }
+
   const needsInitialSave = !input.billingLimit.isConfigured;
   const savedMode = getBillingLimitMode(input.billingLimit);
   const savedCustomAmount =
@@ -75,7 +80,7 @@ export function isBillingLimitFormDirty(input: {
 
 export function getBillingLimitFormLastSubmission(
   submission: BillingLimitActionData["submission"] | undefined,
-  mode: "none" | "plan" | "custom",
+  mode: "" | "none" | "plan" | "custom",
   isDirty: boolean
 ) {
   if (!isDirty || !submission) {
@@ -111,17 +116,20 @@ export function BillingLimitConfigSection({
       : "";
   const savedCancelInProgressRuns = billingLimit.isConfigured && billingLimit.cancelInProgressRuns;
 
-  const [mode, setMode] = useState<"none" | "plan" | "custom">(savedMode);
+  // Unconfigured limit starts with nothing selected.
+  const resetMode: "" | "none" | "plan" | "custom" = billingLimit.isConfigured ? savedMode : "";
+
+  const [mode, setMode] = useState<"" | "none" | "plan" | "custom">(resetMode);
   const [customAmount, setCustomAmount] = useState(savedCustomAmount);
   const [cancelInProgressRuns, setCancelInProgressRuns] = useState(savedCancelInProgressRuns);
   const customAmountInputRef = useRef<HTMLInputElement>(null);
   const formRef = useRef<HTMLFormElement>(null);
 
   useEffect(() => {
-    setMode(savedMode);
+    setMode(resetMode);
     setCustomAmount(savedCustomAmount);
     setCancelInProgressRuns(savedCancelInProgressRuns);
-  }, [savedMode, savedCustomAmount, savedCancelInProgressRuns]);
+  }, [resetMode, savedCustomAmount, savedCancelInProgressRuns]);
 
   function handleModeChange(value: string) {
     const nextMode = value as typeof mode;
@@ -183,6 +191,13 @@ export function BillingLimitConfigSection({
         </Paragraph>
       </div>
 
+      {!billingLimit.isConfigured && (
+        <Callout variant="warning" className="mb-3">
+          Configure a monthly billing limit below to cap your spend, or set no limit to let runs
+          keep going.
+        </Callout>
+      )}
+
       <Form method="post" {...getFormProps(form)} ref={formRef}>
         <input type="hidden" name="intent" value="billing-limit" />
         <Fieldset>
@@ -283,7 +298,7 @@ export function BillingLimitConfigSection({
             </div>
           </RadioGroup>
 
-          {mode !== "none" && (
+          {(mode === "plan" || mode === "custom") && (
             <CheckboxWithLabel
               className="mt-4"
               name="cancelInProgressRuns"
@@ -295,14 +310,16 @@ export function BillingLimitConfigSection({
               onChange={setCancelInProgressRuns}
             />
           )}
-          <FormButtons
-            className={isDirty ? undefined : "invisible"}
-            confirmButton={
-              <Button type="submit" variant="primary/small" disabled={!isDirty}>
-                Save billing limit
-              </Button>
-            }
-          />
+          {mode !== "" && (
+            <FormButtons
+              className={isDirty ? undefined : "invisible"}
+              confirmButton={
+                <Button type="submit" variant="primary/small" disabled={!isDirty}>
+                  Save billing limit
+                </Button>
+              }
+            />
+          )}
         </Fieldset>
       </Form>
     </div>

apps/webapp/app/components/billing/FreePlanUsage.tsx

@@ -27,11 +27,11 @@ export function FreePlanUsage({ to, percentage }: { to: string; percentage: numb
       )}
     >
       <div className="flex items-center justify-between gap-2">
-        <div className="flex items-center gap-1">
-          <ArrowUpCircleIcon className="h-5 w-5 text-text-dimmed" />
-          <span className="text-2sm text-text-bright">Free Plan</span>
+        <div className="flex min-w-0 items-center gap-1">
+          <ArrowUpCircleIcon className="h-5 w-5 shrink-0 text-text-dimmed" />
+          <span className="truncate text-2sm text-text-bright">Free Plan</span>
         </div>
-        <Link to={to} className="text-2sm text-text-link focus-custom">
+        <Link to={to} className="shrink-0 text-2sm text-text-link focus-custom">
           Upgrade
         </Link>
       </div>

apps/webapp/app/components/billing/billingAlertsFormat.ts

@@ -208,6 +208,11 @@ export function isLegacyDollarAmountField(
     return false;
   }
 
+  // The exact $1 absolute base marker always wins, even with levels below 100 (e.g. a $5 alert).
+  if (rawAmount === ABSOLUTE_ALERT_BASE_CENTS) {
+    return false;
+  }
+
   if (!Number.isFinite(rawAmount) || rawAmount < 10) {
     return false;
   }
@@ -315,8 +320,9 @@ export function getAlertPreviewLimitCents(
   planLimitCents: number
 ): number {
   const amountCents = getSavedAlertAmountCents(alerts);
+  // Percentages always apply to the current limit, not the base stored at last save.
   if (amountCents > 0 && percentageAlertLevelsToUiThresholds(alerts.alertLevels).length > 0) {
-    return amountCents;
+    return effectiveLimitCents;
   }
   if (percentageAlertAmountMatches(amountCents, effectiveLimitCents, planLimitCents)) {
     return amountCents;

apps/webapp/app/components/environments/EnvironmentLabel.tsx

@@ -81,12 +81,15 @@ export function EnvironmentLabel({
   tooltipSideOffset = 34,
   tooltipSide = "right",
   disableTooltip = false,
+  truncate = true,
 }: {
   environment: Environment;
   className?: string;
   tooltipSideOffset?: number;
   tooltipSide?: "top" | "right" | "bottom" | "left";
   disableTooltip?: boolean;
+  /** When false, the label clips without an ellipsis (side menu fades it in place). Defaults true. */
+  truncate?: boolean;
 }) {
   const spanRef = useRef<HTMLSpanElement>(null);
   const [isTruncated, setIsTruncated] = useState(false);
@@ -113,7 +116,12 @@ export function EnvironmentLabel({
   const content = (
     <span
       ref={spanRef}
-      className={cn("truncate text-left", environmentTextClassName(environment), className)}
+      className={cn(
+        truncate ? "truncate" : "overflow-hidden whitespace-nowrap",
+        "text-left",
+        environmentTextClassName(environment),
+        className
+      )}
     >
       {text}
     </span>

apps/webapp/app/components/navigation/AccountSideMenu.tsx

@@ -7,7 +7,6 @@ import {
   personalAccessTokensPath,
   rootPath,
 } from "~/utils/pathBuilder";
-import { AskAI } from "../AskAI";
 import { LinkButton } from "../primitives/Buttons";
 import { HelpAndFeedback } from "./HelpAndFeedbackPopover";
 import { SideMenuHeader } from "./SideMenuHeader";
@@ -34,7 +33,7 @@ export function AccountSideMenu({ user }: { user: User }) {
           <span className="text-text-bright">Back to app</span>
         </LinkButton>
       </div>
-      <div className="mb-6 flex grow flex-col overflow-y-auto px-1 pt-2 scrollbar-thin scrollbar-track-transparent scrollbar-thumb-surface-control">
+      <div className="mb-6 flex grow flex-col overflow-y-auto pl-2.5 pr-0 pt-2 scrollbar-gutter-stable scrollbar-thin scrollbar-track-transparent scrollbar-thumb-surface-control">
         <SideMenuHeader title="Account" />
         <SideMenuItem
           name="Profile"
@@ -45,6 +44,7 @@ export function AccountSideMenu({ user }: { user: User }) {
         />
         <SideMenuItem
           name="Personal Access Tokens"
+          nameClassName="tracking-[-0.04em]"
           icon={ShieldIcon}
           activeIconColor="text-text-bright"
           to={personalAccessTokensPath()}
@@ -60,7 +60,6 @@ export function AccountSideMenu({ user }: { user: User }) {
       </div>
       <div className="flex w-full items-center justify-between border-t border-grid-bright p-1">
         <HelpAndFeedback />
-        <AskAI />
       </div>
     </div>
   );

apps/webapp/app/components/navigation/EnvironmentSelector.tsx

@@ -35,18 +35,26 @@ import { V4Badge } from "../V4Badge";
 import { type SideMenuEnvironment, type SideMenuProject } from "./SideMenu";
 import { Badge } from "../primitives/Badge";
 
+// Size this Env popover's items to match the Project popover (SIDE_MENU_POPOVER_ITEM_* in
+// SideMenu.tsx). Only at these call sites, so shared EnvironmentLabel/EnvironmentCombo defaults stay.
+const ENV_POPOVER_ITEM_ICON = "size-5";
+const ENV_POPOVER_ITEM_LABEL = "text-[0.90625rem] font-medium tracking-[-0.01em]";
+
 export function EnvironmentSelector({
   organization,
   project,
   environment,
   className,
   isCollapsed = false,
+  isDragging = false,
 }: {
   organization: MatchedOrganization;
   project: SideMenuProject;
   environment: SideMenuEnvironment;
   className?: string;
   isCollapsed?: boolean;
+  /** True while the side menu is being drag-resized; keeps the row in its expanded arrangement. */
+  isDragging?: boolean;
 }) {
   const { isManagedCloud } = useFeatures();
   const [isMenuOpen, setIsMenuOpen] = useState(false);
@@ -73,42 +81,58 @@ export function EnvironmentSelector({
         button={
           <PopoverTrigger
             className={cn(
-              "group flex h-8 items-center rounded pl-1.75 transition-colors hover:bg-background-hover",
-              isCollapsed ? "justify-center pr-0.5" : "justify-between pr-1",
+              "group flex h-8 items-center rounded pl-1.75 hover:bg-background-hover focus-custom",
+              // Expanded arrangement also applies mid-drag (resting classes flip only on release).
+              isDragging || !isCollapsed ? "justify-between pr-1" : "justify-center pr-0.5",
               className
             )}
           >
             <span className="flex min-w-0 flex-1 items-center gap-1.5 overflow-hidden">
               <EnvironmentIcon environment={environment} className="size-5 shrink-0" />
+              {/*
+                In the side menu, opacity + max-width follow --sm-label-opacity (1 → 0): the label
+                fades in place and scales its width to 0 so it never holds width mid-drag. The
+                selector is also reused outside the side menu (BlankStatePanels, limits) where the var
+                is unset — the 0.2 max-width fallback pins a ~200px cap (0.2 * 1000px) so long names
+                ellipsis-truncate there instead of widening the control, while opacity stays 1.
+              */}
               <span
-                className={cn(
-                  "flex min-w-0 items-center overflow-hidden transition-all duration-200",
-                  isCollapsed ? "max-w-0 opacity-0" : "max-w-[200px] opacity-100"
-                )}
+                className="flex min-w-0 items-center overflow-hidden"
+                style={{
+                  maxWidth: "calc(var(--sm-label-opacity, 0.2) * 1000px)",
+                  opacity: "var(--sm-label-opacity, 1)",
+                }}
               >
                 <EnvironmentLabel
                   environment={environment}
-                  className="text-[0.90625rem] font-medium tracking-[-0.01em]"
+                  className="text-ellipsis text-[0.90625rem] font-medium tracking-[-0.01em]"
                   disableTooltip
+                  truncate={false}
                 />
               </span>
             </span>
+            {/*
+              Chevron's 16px width follows --sm-label-opacity so an invisible span never holds width
+              mid-drag and pushes the row's clip edge into the icon.
+            */}
             <span
-              className={cn(
-                "overflow-hidden transition-all duration-200",
-                isCollapsed ? "max-w-0 opacity-0" : "max-w-[16px] opacity-100"
-              )}
+              className="overflow-hidden opacity-0 group-hover:opacity-100"
+              style={{ maxWidth: "calc(var(--sm-label-opacity, 1) * 16px)" }}
             >
-              <DropdownIcon className="size-4 min-w-4 text-text-dimmed transition group-hover:text-text-bright" />
+              <DropdownIcon className="size-4 min-w-4 text-text-dimmed group-hover:text-text-bright" />
             </span>
           </PopoverTrigger>
         }
-        content={environmentFullTitle(environment)}
+        content={`${environmentFullTitle(environment)} environment`}
         side="right"
         sideOffset={8}
+        // Tooltip only on the collapsed rail (expanded shows the label; this selector is also reused
+        // outside the side menu, where a hover tooltip is unwanted).
         hidden={!isCollapsed}
+        delayDuration={0}
         buttonClassName="h-8!"
         asChild
+        tabbable
         disableHoverableContent
       />
       <PopoverContent
@@ -144,7 +168,13 @@ export function EnvironmentSelector({
                 <PopoverMenuItem
                   key={env.id}
                   to={urlForEnvironment(env)}
-                  title={<EnvironmentCombo environment={env} className="mx-auto grow text-2sm" />}
+                  title={
+                    <EnvironmentCombo
+                      environment={env}
+                      className={cn("mx-auto grow", ENV_POPOVER_ITEM_LABEL)}
+                      iconClassName={ENV_POPOVER_ITEM_ICON}
+                    />
+                  }
                   isSelected={env.id === environment.id}
                 />
               );
@@ -162,8 +192,12 @@ export function EnvironmentSelector({
                 )}
                 title={
                   <div className="flex w-full items-center justify-between">
-                    <EnvironmentCombo environment={{ type: "STAGING" }} className="text-2sm" />
-                    <span className="text-indigo-500">Upgrade</span>
+                    <EnvironmentCombo
+                      environment={{ type: "STAGING" }}
+                      className={ENV_POPOVER_ITEM_LABEL}
+                      iconClassName={ENV_POPOVER_ITEM_ICON}
+                    />
+                    <span className={cn("text-indigo-500", ENV_POPOVER_ITEM_LABEL)}>Upgrade</span>
                   </div>
                 }
                 isSelected={false}
@@ -176,8 +210,12 @@ export function EnvironmentSelector({
                 )}
                 title={
                   <div className="flex w-full items-center justify-between">
-                    <EnvironmentCombo environment={{ type: "PREVIEW" }} className="text-2sm" />
-                    <span className="text-indigo-500">Upgrade</span>
+                    <EnvironmentCombo
+                      environment={{ type: "PREVIEW" }}
+                      className={ENV_POPOVER_ITEM_LABEL}
+                      iconClassName={ENV_POPOVER_ITEM_ICON}
+                    />
+                    <span className={cn("text-indigo-500", ENV_POPOVER_ITEM_LABEL)}>Upgrade</span>
                   </div>
                 }
                 isSelected={false}
@@ -199,10 +237,6 @@ function Branches({
   branchEnvironments: SideMenuEnvironment[];
   currentEnvironment: SideMenuEnvironment;
 }) {
-  const organization = useOrganization();
-  const project = useProject();
-  const environment = useEnvironment();
-  const { urlForEnvironment } = useEnvironmentSwitcher();
   const navigation = useNavigation();
   const [isMenuOpen, setMenuOpen] = useState(false);
   const timeoutRef = useRef<NodeJS.Timeout | null>(null);
@@ -234,23 +268,6 @@ function Branches({
     }, 150);
   };
 
-  const activeBranches = branchEnvironments.filter((env) => env.archivedAt === null);
-  const state =
-    branchEnvironments.length === 0
-      ? "no-branches"
-      : activeBranches.length === 0
-        ? "no-active-branches"
-        : "has-branches";
-
-  // Only surface the active environment's archived-branch item in the submenu it
-  // actually belongs to. Both Development and Preview render this component, so
-  // without the parent check an archived dev branch would leak into the Preview
-  // submenu (and vice-versa).
-  const currentBranchIsArchived =
-    environment.archivedAt !== null && environment.parentEnvironmentId === parentEnvironment.id;
-
-  const envTextClassName = environmentTextClassName(parentEnvironment);
-
   return (
     <Popover onOpenChange={(open) => setMenuOpen(open)} open={isMenuOpen}>
       <div onMouseEnter={handleMouseEnter} onMouseLeave={handleMouseLeave} className="flex">
@@ -263,7 +280,11 @@ function Branches({
             textAlignLeft
             fullWidth
           >
-            <EnvironmentCombo environment={parentEnvironment} className="mx-auto grow text-2sm" />
+            <EnvironmentCombo
+              environment={parentEnvironment}
+              className={cn("mx-auto grow", ENV_POPOVER_ITEM_LABEL)}
+              iconClassName={ENV_POPOVER_ITEM_ICON}
+            />
           </ButtonContent>
         </PopoverTrigger>
         <PopoverContent
@@ -276,88 +297,135 @@ function Branches({
           onMouseEnter={handleMouseEnter}
           onMouseLeave={handleMouseLeave}
         >
-          <div className="flex flex-col gap-1 p-1">
-            {currentBranchIsArchived && (
-              <PopoverMenuItem
-                key={environment.id}
-                to={urlForEnvironment(environment)}
-                title={
-                  <>
-                    <span className={cn("block w-full", envTextClassName)}>
-                      {environment.branchName}
-                    </span>
-                    <Badge variant="extra-small">Archived</Badge>
-                  </>
-                }
-                icon={
-                  <BranchEnvironmentIconSmall className={cn("size-4 shrink-0", envTextClassName)} />
-                }
-                isSelected={environment.id === currentEnvironment.id}
-              />
-            )}
-            {state === "has-branches" ? (
-              <>
-                {branchEnvironments
-                  .filter((env) => env.archivedAt === null)
-                  .map((env) => (
-                    <PopoverMenuItem
-                      key={env.id}
-                      to={urlForEnvironment(env)}
-                      title={
-                        <span className={cn("block w-full", envTextClassName)}>
-                          {env.branchName ?? DEFAULT_DEV_BRANCH}
-                        </span>
-                      }
-                      icon={
-                        <BranchEnvironmentIconSmall
-                          className={cn("size-4 shrink-0", envTextClassName)}
-                        />
-                      }
-                      isSelected={env.id === currentEnvironment.id}
-                    />
-                  ))}
-              </>
-            ) : state === "no-branches" ? (
-              <div className="flex max-w-sm flex-col gap-1 p-2">
-                <div className="flex items-center gap-1">
-                  <BranchEnvironmentIconSmall className={cn("size-4", envTextClassName)} />
-                  <Header2>Create your first branch</Header2>
-                </div>
-                <Paragraph spacing variant="small">
-                  Branches are a way to test new features in isolation before merging them into the
-                  main environment.
-                </Paragraph>
-                <Paragraph variant="small">
-                  Branches are only available when using <V4Badge inline /> or above. Read our{" "}
-                  <TextLink to={docsPath("upgrade-to-v4")}>v4 upgrade guide</TextLink> to learn
-                  more.
-                </Paragraph>
-              </div>
-            ) : (
-              <div className="flex max-w-sm flex-col gap-1 p-2">
-                <Paragraph variant="extra-small">All branches are archived.</Paragraph>
-              </div>
-            )}
-          </div>
-          <div className="border-t border-grid-bright p-1">
-            {parentEnvironment.type === "DEVELOPMENT" ? (
-              <PopoverMenuItem
-                to={branchesDevPath(organization, project, environment)}
-                title="Manage dev branches"
-                icon={<Cog8ToothIcon className="size-4 text-text-dimmed" />}
-                leadingIconClassName="text-text-dimmed"
-              />
-            ) : (
-              <PopoverMenuItem
-                to={branchesPath(organization, project, environment)}
-                title="Manage preview branches"
-                icon={<Cog8ToothIcon className="size-4 text-text-dimmed" />}
-                leadingIconClassName="text-text-dimmed"
-              />
-            )}
-          </div>
+          <BranchesPopoverContent
+            parentEnvironment={parentEnvironment}
+            branchEnvironments={branchEnvironments}
+            currentEnvironment={currentEnvironment}
+          />
         </PopoverContent>
       </div>
     </Popover>
   );
 }
+
+/**
+ * Inner content of the branches popover (list, empty states, "Manage branches" footer). Shared by
+ * the `Branches` hover submenu and the side-menu Preview popover.
+ */
+export function BranchesPopoverContent({
+  parentEnvironment,
+  branchEnvironments,
+  currentEnvironment,
+}: {
+  parentEnvironment: SideMenuEnvironment;
+  branchEnvironments: SideMenuEnvironment[];
+  currentEnvironment: SideMenuEnvironment;
+}) {
+  const organization = useOrganization();
+  const project = useProject();
+  const environment = useEnvironment();
+  const { urlForEnvironment } = useEnvironmentSwitcher();
+
+  const activeBranches = branchEnvironments.filter((env) => env.archivedAt === null);
+  const state =
+    branchEnvironments.length === 0
+      ? "no-branches"
+      : activeBranches.length === 0
+        ? "no-active-branches"
+        : "has-branches";
+
+  // Show the archived-branch item only in the submenu it belongs to: both Development and Preview
+  // render this, so without the parent check an archived dev branch leaks into Preview (and vice-versa).
+  const currentBranchIsArchived =
+    environment.archivedAt !== null && environment.parentEnvironmentId === parentEnvironment.id;
+
+  const envTextClassName = environmentTextClassName(parentEnvironment);
+
+  return (
+    <>
+      <div className="flex flex-col gap-1 p-1">
+        {parentEnvironment.type === "DEVELOPMENT" ? (
+          <PopoverMenuItem
+            to={branchesDevPath(organization, project, environment)}
+            title="Manage dev branches"
+            icon={<Cog8ToothIcon className={cn(ENV_POPOVER_ITEM_ICON, "text-text-dimmed")} />}
+            leadingIconClassName="text-text-dimmed"
+            className={ENV_POPOVER_ITEM_LABEL}
+          />
+        ) : (
+          <PopoverMenuItem
+            to={branchesPath(organization, project, environment)}
+            title="Manage preview branches"
+            icon={<Cog8ToothIcon className={cn(ENV_POPOVER_ITEM_ICON, "text-text-dimmed")} />}
+            leadingIconClassName="text-text-dimmed"
+            className={ENV_POPOVER_ITEM_LABEL}
+          />
+        )}
+      </div>
+      <div className="flex flex-col gap-1 border-t border-grid-bright p-1">
+        {currentBranchIsArchived && (
+          <PopoverMenuItem
+            key={environment.id}
+            to={urlForEnvironment(environment)}
+            title={
+              <>
+                <span className={cn("block w-full", envTextClassName, ENV_POPOVER_ITEM_LABEL)}>
+                  {environment.branchName}
+                </span>
+                <Badge variant="extra-small">Archived</Badge>
+              </>
+            }
+            icon={
+              <BranchEnvironmentIconSmall
+                className={cn(ENV_POPOVER_ITEM_ICON, "shrink-0", envTextClassName)}
+              />
+            }
+            isSelected={environment.id === currentEnvironment.id}
+          />
+        )}
+        {state === "has-branches" ? (
+          <>
+            {branchEnvironments
+              .filter((env) => env.archivedAt === null)
+              .map((env) => (
+                <PopoverMenuItem
+                  key={env.id}
+                  to={urlForEnvironment(env)}
+                  title={
+                    <span className={cn("block w-full", envTextClassName, ENV_POPOVER_ITEM_LABEL)}>
+                      {env.branchName ?? DEFAULT_DEV_BRANCH}
+                    </span>
+                  }
+                  icon={
+                    <BranchEnvironmentIconSmall
+                      className={cn(ENV_POPOVER_ITEM_ICON, "shrink-0", envTextClassName)}
+                    />
+                  }
+                  isSelected={env.id === currentEnvironment.id}
+                />
+              ))}
+          </>
+        ) : state === "no-branches" ? (
+          <div className="flex max-w-sm flex-col gap-1 p-2">
+            <div className="flex items-center gap-1">
+              <BranchEnvironmentIconSmall className={cn("size-4", envTextClassName)} />
+              <Header2>Create your first branch</Header2>
+            </div>
+            <Paragraph spacing variant="small">
+              Branches are a way to test new features in isolation before merging them into the main
+              environment.
+            </Paragraph>
+            <Paragraph variant="small">
+              Branches are only available when using <V4Badge inline /> or above. Read our{" "}
+              <TextLink to={docsPath("upgrade-to-v4")}>v4 upgrade guide</TextLink> to learn more.
+            </Paragraph>
+          </div>
+        ) : (
+          <div className="flex max-w-sm flex-col gap-1 p-2">
+            <Paragraph variant="extra-small">All branches are archived.</Paragraph>
+          </div>
+        )}
+      </div>
+    </>
+  );
+}

apps/webapp/app/components/navigation/HelpAndFeedbackPopover.tsx

@@ -1,25 +1,27 @@
 import { ArrowUpRightIcon } from "@heroicons/react/20/solid";
 import { motion } from "framer-motion";
 import { Fragment, useState } from "react";
+import { AISparkleIcon } from "~/assets/icons/AISparkleIcon";
 import { BookIcon } from "~/assets/icons/BookIcon";
 import { BulbIcon } from "~/assets/icons/BulbIcon";
+import { DropdownIcon } from "~/assets/icons/DropdownIcon";
 import { EnvelopeIcon } from "~/assets/icons/EnvelopeIcon";
 import { QuestionMarkIcon } from "~/assets/icons/QuestionMarkIcon";
 import { RadarPulseIcon } from "~/assets/icons/RadarPulseIcon";
 import { StarIcon } from "~/assets/icons/StarIcon";
 import { useShortcutKeys } from "~/hooks/useShortcutKeys";
-import { sanitizeHttpUrl } from "~/utils/sanitizeUrl";
 import { useCurrentPlan } from "~/routes/_app.orgs.$organizationSlug/route";
 import { useRecentChangelogs } from "~/routes/resources.platform-changelogs";
 import { cn } from "~/utils/cn";
+import { sanitizeHttpUrl } from "~/utils/sanitizeUrl";
+import { AskAIRoot } from "../AskAI";
 import { Feedback } from "../Feedback";
 import { Shortcuts } from "../Shortcuts";
-import { Button } from "../primitives/Buttons";
 import { Paragraph } from "../primitives/Paragraph";
 import { Popover, PopoverContent, PopoverTrigger } from "../primitives/Popover";
 import { ShortcutKey } from "../primitives/ShortcutKey";
 import { SimpleTooltip } from "../primitives/Tooltip";
-import { SideMenuItem } from "./SideMenuItem";
+import { SideMenuItem, SideMenuItemButton } from "./SideMenuItem";
 
 export function HelpAndFeedback({
   disableShortcut = false,
@@ -49,135 +51,161 @@ export function HelpAndFeedback({
     <motion.div
       layout="position"
       transition={{ duration: 0.2, ease: "easeInOut" }}
-      className={isCollapsed ? undefined : "flex-1"}
+      className={isCollapsed ? undefined : "min-w-0 flex-1"}
     >
-      <Popover open={isHelpMenuOpen} onOpenChange={setHelpMenuOpen}>
-        <SimpleTooltip
-          button={
-            <PopoverTrigger
-              className={cn(
-                "group flex h-8 items-center gap-1.5 rounded pl-1.75 pr-2 transition-colors hover:bg-background-hover focus-custom",
-                isCollapsed ? "w-full" : "w-full justify-between"
-              )}
-            >
-              <span className="flex items-center gap-1.5 overflow-hidden">
-                <QuestionMarkIcon className="size-5 min-w-5 shrink-0 text-success" />
-                <span
+      {/* AskAIRoot hosts the Ask AI dialog + ⌘I shortcut outside the popover, so both survive the
+          popover closing; the popover just renders the trigger. */}
+      <AskAIRoot>
+        {(openAskAI) => (
+          <Popover open={isHelpMenuOpen} onOpenChange={setHelpMenuOpen}>
+            <SimpleTooltip
+              button={
+                <PopoverTrigger
                   className={cn(
-                    "overflow-hidden whitespace-nowrap text-2sm text-text-bright transition-all duration-150",
-                    isCollapsed ? "max-w-0 opacity-0" : "max-w-[150px] opacity-100"
+                    "group flex h-8 items-center gap-1.5 rounded pl-1.75 pr-2 hover:bg-background-hover focus-custom",
+                    isCollapsed ? "w-full" : "w-full justify-between"
                   )}
                 >
+                  <span className="flex min-w-0 items-center gap-1.5 overflow-hidden">
+                    <QuestionMarkIcon className="size-5 min-w-5 shrink-0 text-success" />
+                    {/*
+                      Width + opacity follow --sm-label-opacity so the label tracks a drag both
+                      directions (no CSS transition — it would lag the per-frame writes).
+                    */}
+                    <span
+                      className="min-w-0 overflow-hidden whitespace-nowrap text-[0.90625rem] font-medium tracking-[-0.01em] text-text-dimmed group-hover:text-text-bright"
+                      style={{
+                        maxWidth: "calc(var(--sm-label-opacity, 1) * 150px)",
+                        opacity: "var(--sm-label-opacity, 1)",
+                      }}
+                    >
+                      Help & Feedback
+                    </span>
+                  </span>
+                  {/*
+                    Hover chevron, only when expanded. Its 16px width follows --sm-label-opacity so
+                    an invisible chevron never holds width mid-drag and clips the help icon.
+                  */}
+                  {!isCollapsed && (
+                    <span
+                      className="overflow-hidden opacity-0 group-hover:opacity-100"
+                      style={{ maxWidth: "calc(var(--sm-label-opacity, 1) * 16px)" }}
+                    >
+                      <DropdownIcon className="size-4 min-w-4 text-text-dimmed group-hover:text-text-bright" />
+                    </span>
+                  )}
+                </PopoverTrigger>
+              }
+              content={
+                <span className="flex items-center gap-1">
                   Help & Feedback
+                  <ShortcutKey shortcut={{ key: "h" }} variant="medium/bright" />
                 </span>
-              </span>
-              <ShortcutKey
-                className={cn(
-                  "size-4 flex-none transition-all duration-150",
-                  isCollapsed ? "hidden" : ""
+              }
+              side="right"
+              sideOffset={8}
+              delayDuration={isCollapsed ? 0 : 500}
+              buttonClassName="h-8! w-full"
+              asChild
+              tabbable
+              disableHoverableContent
+            />
+            <PopoverContent
+              className="min-w-56 divide-y divide-grid-bright overflow-y-auto p-0 scrollbar-thin scrollbar-track-transparent scrollbar-thumb-surface-control"
+              side={isCollapsed ? "right" : "top"}
+              sideOffset={isCollapsed ? 8 : 4}
+              align="start"
+            >
+              <Fragment>
+                {openAskAI !== undefined && (
+                  <div className="flex flex-col gap-1 p-1">
+                    <SideMenuItemButton
+                      icon={AISparkleIcon}
+                      name="Ask AI"
+                      data-action="ask-ai"
+                      trailing={
+                        <ShortcutKey shortcut={{ modifiers: ["mod"], key: "i" }} variant="medium" />
+                      }
+                      onClick={() => {
+                        setHelpMenuOpen(false);
+                        openAskAI();
+                      }}
+                    />
+                  </div>
                 )}
-                shortcut={{ key: "h" }}
-                variant="medium/bright"
-              />
-            </PopoverTrigger>
-          }
-          content={
-            <span className="flex items-center gap-1">
-              Help & Feedback
-              <ShortcutKey shortcut={{ key: "h" }} variant="medium/bright" />
-            </span>
-          }
-          side="right"
-          sideOffset={8}
-          hidden={!isCollapsed}
-          buttonClassName="h-8! w-full"
-          asChild
-          disableHoverableContent
-        />
-        <PopoverContent
-          className="min-w-56 divide-y divide-grid-bright overflow-y-auto p-0 scrollbar-thin scrollbar-track-transparent scrollbar-thumb-surface-control"
-          side={isCollapsed ? "right" : "top"}
-          sideOffset={isCollapsed ? 8 : 4}
-          align="start"
-        >
-          <Fragment>
-            <div className="flex flex-col gap-1 p-1">
-              <SideMenuItem
-                name="Documentation"
-                icon={BookIcon}
-                trailingIcon={ArrowUpRightIcon}
-                trailingIconClassName="text-text-dimmed"
-                to="https://trigger.dev/docs"
-                data-action="documentation"
-                target="_blank"
-              />
-            </div>
-            <div className="flex flex-col gap-1 p-1">
-              <SideMenuItem
-                name="Status"
-                icon={RadarPulseIcon}
-                trailingIcon={ArrowUpRightIcon}
-                trailingIconClassName="text-text-dimmed"
-                to="https://status.trigger.dev/"
-                data-action="status"
-                target="_blank"
-              />
-              <SideMenuItem
-                name="Suggest a feature"
-                icon={BulbIcon}
-                trailingIcon={ArrowUpRightIcon}
-                trailingIconClassName="text-text-dimmed"
-                to="https://feedback.trigger.dev/"
-                data-action="suggest-a-feature"
-                target="_blank"
-              />
-              <Shortcuts />
-              <Feedback
-                button={
-                  <Button
-                    variant="small-menu-item"
-                    className="pl-2"
-                    LeadingIcon={EnvelopeIcon}
-                    leadingIconClassName="pr-1 text-text-dimmed group-hover/button:text-text-bright"
-                    data-action="contact-us"
-                    fullWidth
-                    textAlignLeft
-                  >
-                    Contact us…
-                  </Button>
-                }
-              />
-            </div>
-            <div className="flex flex-col gap-1 p-1">
-              <Paragraph className="pb-1 pl-1.5 pt-1.5 text-xs">What's new</Paragraph>
-              {changelogs.map((entry) => (
-                <SideMenuItem
-                  key={entry.id}
-                  name={entry.title}
-                  icon={GrayDotIcon}
-                  trailingIcon={ArrowUpRightIcon}
-                  trailingIconClassName="text-text-dimmed"
-                  inactiveIconColor="text-text-dimmed"
-                  activeIconColor="text-text-dimmed"
-                  to={sanitizeHttpUrl(entry.actionUrl) ?? "https://trigger.dev/changelog"}
-                  target="_blank"
-                />
-              ))}
-              <SideMenuItem
-                name="Full changelog"
-                icon={StarIcon}
-                trailingIcon={ArrowUpRightIcon}
-                trailingIconClassName="text-text-dimmed"
-                inactiveIconColor="text-text-dimmed"
-                activeIconColor="text-text-dimmed"
-                to="https://trigger.dev/changelog"
-                data-action="full-changelog"
-                target="_blank"
-              />
-            </div>
-          </Fragment>
-        </PopoverContent>
-      </Popover>
+                <div className="flex flex-col gap-1 p-1">
+                  <SideMenuItem
+                    name="Documentation"
+                    icon={BookIcon}
+                    trailingIcon={ArrowUpRightIcon}
+                    trailingIconClassName="text-text-dimmed"
+                    to="https://trigger.dev/docs"
+                    data-action="documentation"
+                    target="_blank"
+                  />
+                </div>
+                <div className="flex flex-col gap-1 p-1">
+                  <SideMenuItem
+                    name="Status"
+                    icon={RadarPulseIcon}
+                    trailingIcon={ArrowUpRightIcon}
+                    trailingIconClassName="text-text-dimmed"
+                    to="https://status.trigger.dev/"
+                    data-action="status"
+                    target="_blank"
+                  />
+                  <SideMenuItem
+                    name="Suggest a feature"
+                    icon={BulbIcon}
+                    trailingIcon={ArrowUpRightIcon}
+                    trailingIconClassName="text-text-dimmed"
+                    to="https://feedback.trigger.dev/"
+                    data-action="suggest-a-feature"
+                    target="_blank"
+                  />
+                  <Shortcuts />
+                  <Feedback
+                    button={
+                      <SideMenuItemButton
+                        icon={EnvelopeIcon}
+                        name="Contact us…"
+                        data-action="contact-us"
+                      />
+                    }
+                  />
+                </div>
+                <div className="flex flex-col gap-1 p-1">
+                  <Paragraph className="pb-1 pl-1.5 pt-1.5 text-xs">What's new</Paragraph>
+                  {changelogs.map((entry) => (
+                    <SideMenuItem
+                      key={entry.id}
+                      name={entry.title}
+                      icon={GrayDotIcon}
+                      trailingIcon={ArrowUpRightIcon}
+                      trailingIconClassName="text-text-dimmed"
+                      inactiveIconColor="text-text-dimmed"
+                      activeIconColor="text-text-dimmed"
+                      to={sanitizeHttpUrl(entry.actionUrl) ?? "https://trigger.dev/changelog"}
+                      target="_blank"
+                    />
+                  ))}
+                  <SideMenuItem
+                    name="Full changelog"
+                    icon={StarIcon}
+                    trailingIcon={ArrowUpRightIcon}
+                    trailingIconClassName="text-text-dimmed"
+                    inactiveIconColor="text-text-dimmed"
+                    activeIconColor="text-text-dimmed"
+                    to="https://trigger.dev/changelog"
+                    data-action="full-changelog"
+                    target="_blank"
+                  />
+                </div>
+              </Fragment>
+            </PopoverContent>
+          </Popover>
+        )}
+      </AskAIRoot>
     </motion.div>
   );
 }

apps/webapp/app/components/navigation/NotificationPanel.tsx

@@ -121,7 +121,7 @@ export function NotificationPanel({
 
   return (
     <Popover>
-      <div className={isCollapsed ? "p-1" : "p-2"}>
+      <div className={isCollapsed ? "p-1" : "p-2 pt-0"}>
         {isCollapsed ? (
           <SimpleTooltip
             asChild

apps/webapp/app/components/navigation/OrganizationSettingsSideMenu.tsx

@@ -1,5 +1,6 @@
-import { ArrowLeftIcon, LinkIcon } from "@heroicons/react/24/solid";
+import { ArrowLeftIcon } from "@heroicons/react/24/solid";
 import { BellIcon } from "~/assets/icons/BellIcon";
+import { ChainLinkIcon } from "~/assets/icons/ChainLinkIcon";
 import { CreditCardIcon } from "~/assets/icons/CreditCardIcon";
 import { PadlockIcon } from "~/assets/icons/PadlockIcon";
 import { UsageIcon } from "~/assets/icons/UsageIcon";
@@ -34,7 +35,6 @@ import { useCurrentPlan } from "~/routes/_app.orgs.$organizationSlug/route";
 import { Paragraph } from "../primitives/Paragraph";
 import { Badge } from "../primitives/Badge";
 import { useHasAdminAccess } from "~/hooks/useUser";
-import { AskAI } from "../AskAI";
 
 export type BuildInfo = {
   appVersion: string | undefined;
@@ -79,11 +79,19 @@ export function OrganizationSettingsSideMenu({
           <span className="text-text-bright">Back to app</span>
         </LinkButton>
       </div>
-      <div className="mb-6 flex grow flex-col gap-4 overflow-y-auto px-1 pt-2 scrollbar-thin scrollbar-track-transparent scrollbar-thumb-surface-control">
+      <div className="mb-6 flex grow flex-col gap-4 overflow-y-auto pl-2.5 pr-0 pt-2 scrollbar-gutter-stable scrollbar-thin scrollbar-track-transparent scrollbar-thumb-surface-control">
         <div className="flex flex-col">
           <div className="mb-1">
             <SideMenuHeader title="Organization" />
           </div>
+          <SideMenuItem
+            name="Settings"
+            icon={SlidersIcon}
+            activeIconColor="text-text-bright"
+            inactiveIconColor="text-text-dimmed"
+            to={organizationSettingsPath(organization)}
+            data-action="settings"
+          />
           {isManagedCloud && (
             <>
               <SideMenuItem
@@ -130,7 +138,7 @@ export function OrganizationSettingsSideMenu({
           {featureFlags.hasPrivateConnections && (
             <SideMenuItem
               name="Private Connections"
-              icon={LinkIcon}
+              icon={ChainLinkIcon}
               activeIconColor="text-text-bright"
               inactiveIconColor="text-text-dimmed"
               to={v3PrivateConnectionsPath(organization)}
@@ -155,21 +163,8 @@ export function OrganizationSettingsSideMenu({
               inactiveIconColor="text-text-dimmed"
               to={organizationSsoPath(organization)}
               data-action="sso"
-              badge={
-                currentPlan?.v3Subscription?.plan?.code === "enterprise" ? undefined : (
-                  <Badge variant="extra-small">Enterprise</Badge>
-                )
-              }
             />
           )}
-          <SideMenuItem
-            name="Settings"
-            icon={SlidersIcon}
-            activeIconColor="text-text-bright"
-            inactiveIconColor="text-text-dimmed"
-            to={organizationSettingsPath(organization)}
-            data-action="settings"
-          />
         </div>
         <div className="flex flex-col">
           <div className="mb-1">
@@ -241,7 +236,6 @@ export function OrganizationSettingsSideMenu({
       </div>
       <div className="flex w-full items-center justify-between border-t border-grid-bright p-1">
         <HelpAndFeedback organizationId={organization.id} />
-        <AskAI />
       </div>
     </div>
   );

apps/webapp/app/components/navigation/SideMenuHeader.tsx

@@ -40,15 +40,8 @@ export function SideMenuHeader({
       <h2 className="text-xs whitespace-nowrap">
         {visiblePart}
         {fadingPart && (
-          <motion.span
-            initial={false}
-            animate={{
-              opacity: isCollapsed ? 0 : 1,
-            }}
-            transition={{ duration: 0.15, ease: "easeOut" }}
-          >
-            {fadingPart}
-          </motion.span>
+          // --sm-label-opacity morphs "Project" → "Proj" as the menu narrows (unset elsewhere → 1).
+          <span style={{ opacity: "var(--sm-label-opacity, 1)" }}>{fadingPart}</span>
         )}
       </h2>
       {children !== undefined ? (

apps/webapp/app/components/navigation/SideMenuItem.tsx

@@ -1,4 +1,9 @@
-import { type AnchorHTMLAttributes, type ReactNode } from "react";
+import {
+  type AnchorHTMLAttributes,
+  type ButtonHTMLAttributes,
+  forwardRef,
+  type ReactNode,
+} from "react";
 import { Link } from "@remix-run/react";
 import { motion } from "framer-motion";
 import { usePathName } from "~/hooks/usePathName";
@@ -14,6 +19,7 @@ export function SideMenuItem({
   trailingIcon,
   trailingIconClassName,
   name,
+  nameClassName,
   to,
   badge,
   target,
@@ -30,18 +36,14 @@ export function SideMenuItem({
   trailingIcon?: RenderIcon;
   trailingIconClassName?: string;
   name: string;
+  nameClassName?: string;
   to: string;
   badge?: ReactNode;
   target?: AnchorHTMLAttributes<HTMLAnchorElement>["target"];
   isCollapsed?: boolean;
   action?: ReactNode;
   disableIconHover?: boolean;
-  /**
-   * Visually indented variant — same item, just pushed further from
-   * the left edge so it reads as a child of the row above. Used for
-   * grouped sub-items like the Tasks > (Agents / Standard / Scheduled)
-   * cluster. The indent is only applied when the side menu is expanded.
-   */
+  /** Indented variant for grouped sub-items; only applied when the menu is expanded. */
   indented?: boolean;
   "data-action"?: string;
 }) {
@@ -56,7 +58,7 @@ export function SideMenuItem({
       target={target}
       data-action={dataAction}
       className={cn(
-        "group/menulink flex h-8 items-center gap-2 overflow-hidden rounded pl-1.75 pr-2",
+        "group/menulink flex h-8 items-center gap-2 overflow-hidden rounded pl-1.75 pr-2 focus-custom",
         isIndented ? "min-w-0 flex-1" : "w-full",
         isActive
           ? "bg-tertiary text-text-bright"
@@ -75,32 +77,39 @@ export function SideMenuItem({
         )}
       />
       <motion.div
-        className="flex min-w-0 flex-1 items-center justify-between overflow-hidden"
+        className="min-w-0 flex-1 overflow-hidden"
         initial={false}
         animate={{
           width: isCollapsed ? 0 : "auto",
-          opacity: isCollapsed ? 0 : 1,
         }}
         transition={{ duration: 0.2, ease: "easeOut" }}
       >
-        <span className="select-none truncate text-[0.90625rem] font-medium tracking-[-0.01em]">
-          {name}
-        </span>
-        {badge && !isCollapsed && (
-          <motion.div
-            className="ml-1 flex shrink-0 items-center gap-1"
-            initial={false}
-            animate={{
-              opacity: 1,
-            }}
-            transition={{ duration: 0.15, ease: "easeOut" }}
+        {/*
+          Label opacity follows --sm-label-opacity so it fades as the menu narrows (unset
+          elsewhere → 1, fully visible).
+        */}
+        <div
+          className="flex w-full min-w-0 items-center justify-between"
+          style={{ opacity: "var(--sm-label-opacity, 1)" }}
+        >
+          <span
+            className={cn(
+              "select-none overflow-hidden whitespace-nowrap text-[0.90625rem] font-medium tracking-[-0.01em]",
+              nameClassName
+            )}
           >
-            {badge}
-          </motion.div>
-        )}
-        {trailingIcon && !isCollapsed && (
-          <Icon icon={trailingIcon} className={cn("ml-1 size-4 shrink-0", trailingIconClassName)} />
-        )}
+            {name}
+          </span>
+          {badge && !isCollapsed && (
+            <div className="ml-1 flex shrink-0 items-center gap-1">{badge}</div>
+          )}
+          {trailingIcon && !isCollapsed && (
+            <Icon
+              icon={trailingIcon}
+              className={cn("ml-1 size-4 shrink-0", trailingIconClassName)}
+            />
+          )}
+        </div>
       </motion.div>
     </Link>
   );
@@ -125,9 +134,11 @@ export function SideMenuItem({
           buttonClassName="h-8! block w-full"
           hidden={!isCollapsed}
           asChild
+          tabbable
           disableHoverableContent
         />
         {!isCollapsed && (
+          // Fades with the labels via --sm-label-opacity (unset → fully visible).
           <div
             className={cn(
               "absolute bottom-1 right-1 top-1 flex aspect-square items-center justify-center rounded",
@@ -135,6 +146,7 @@ export function SideMenuItem({
                 ? "group-hover/menuitem:bg-tertiary"
                 : "group-hover/menuitem:bg-background-hover"
             )}
+            style={{ opacity: "var(--sm-label-opacity, 1)" }}
           >
             {action}
           </div>
@@ -152,7 +164,35 @@ export function SideMenuItem({
       buttonClassName="h-8! block w-full"
       hidden={!isCollapsed}
       asChild
+      tabbable
       disableHoverableContent
     />
   );
 }
+
+/** Button styled to match {@link SideMenuItem}, for entries that open a dialog rather than navigate. */
+export const SideMenuItemButton = forwardRef<
+  HTMLButtonElement,
+  { icon: RenderIcon; name: string; trailing?: ReactNode } & ButtonHTMLAttributes<HTMLButtonElement>
+>(function SideMenuItemButton({ icon, name, trailing, className, type, ...props }, ref) {
+  return (
+    <button
+      ref={ref}
+      type={type ?? "button"}
+      className={cn(
+        "group/menuitem flex h-8 w-full items-center gap-2 overflow-hidden rounded pl-1.75 pr-2 text-left text-text-dimmed hover:bg-background-hover hover:text-text-bright focus-custom",
+        className
+      )}
+      {...props}
+    >
+      <Icon
+        icon={icon}
+        className="size-5 shrink-0 text-text-dimmed group-hover/menuitem:text-text-bright"
+      />
+      <span className="min-w-0 flex-1 select-none truncate text-[0.90625rem] font-medium tracking-[-0.01em]">
+        {name}
+      </span>
+      {trailing && <span className="flex shrink-0 items-center gap-1">{trailing}</span>}
+    </button>
+  );
+});

apps/webapp/app/components/navigation/SideMenuSection.tsx

@@ -1,5 +1,5 @@
 import { AnimatePresence, motion } from "framer-motion";
-import React, { useCallback, useState } from "react";
+import React, { useCallback, useEffect, useRef, useState } from "react";
 import { ToggleArrowIcon } from "~/assets/icons/ToggleArrowIcon";
 
 type Props = {
@@ -14,9 +14,7 @@ type Props = {
   headerAction?: React.ReactNode;
 };
 
-/** A collapsible section for the side menu
- * The collapsed state is passed in as a prop, and there's a callback when it's toggled so we can save the state.
- */
+/** A collapsible section for the side menu. Collapsed state is controlled via props + a toggle callback. */
 export function SideMenuSection({
   title,
   initialCollapsed = false,
@@ -27,6 +25,7 @@ export function SideMenuSection({
   headerAction,
 }: Props) {
   const [isCollapsed, setIsCollapsed] = useState(initialCollapsed);
+  const contentRef = useRef<HTMLDivElement>(null);
 
   const handleToggle = useCallback(() => {
     const newIsCollapsed = !isCollapsed;
@@ -34,22 +33,37 @@ export function SideMenuSection({
     onCollapseToggle?.(newIsCollapsed);
   }, [isCollapsed, onCollapseToggle]);
 
+  // Collapsed items stay in the DOM (height 0) for the animation, so `inert` removes them from the
+  // tab order and a11y tree (it doesn't affect layout). Set the DOM property directly — React 18's
+  // `inert` prop handling is unreliable.
+  useEffect(() => {
+    if (contentRef.current) {
+      contentRef.current.inert = isCollapsed;
+    }
+  }, [isCollapsed]);
+
   return (
     <div className="w-full overflow-hidden">
       {/* Header container - stays in DOM to preserve height */}
       <div className="relative w-full">
-        {/* Header - fades out when sidebar is collapsed */}
-        <motion.div
-          className="group/section flex cursor-pointer items-center justify-between overflow-hidden rounded-sm py-1 pl-1.5 pr-1 transition hover:bg-background-hover"
-          initial={false}
-          animate={{
-            opacity: isSideMenuCollapsed ? 0 : 1,
-          }}
-          transition={{ duration: 0.15, ease: "easeOut" }}
+        {/*
+          Header fades out as the menu narrows via --sm-label-opacity (falls back to 1 unset). Hover
+          background and text color snap (no transition), matching the nav items.
+        */}
+        <button
+          type="button"
+          // A real button for native keyboard toggle + focus ring. Out of the tab order when the
+          // menu is collapsed (the header is hidden and can't be toggled).
+          className="group/section flex w-full cursor-pointer items-center justify-between overflow-hidden rounded-sm py-1 pl-1.5 pr-1 hover:bg-background-hover focus-custom"
           onClick={isSideMenuCollapsed ? undefined : handleToggle}
-          style={{ cursor: isSideMenuCollapsed ? "default" : "pointer" }}
+          tabIndex={isSideMenuCollapsed ? -1 : undefined}
+          aria-expanded={!isCollapsed}
+          style={{
+            opacity: "var(--sm-label-opacity, 1)",
+            cursor: isSideMenuCollapsed ? "default" : "pointer",
+          }}
         >
-          <div className="flex items-center gap-1 text-text-dimmed transition group-hover/section:text-text-bright">
+          <div className="flex items-center gap-1 text-text-dimmed group-hover/section:text-text-bright">
             <h2 className="whitespace-nowrap text-xs">{title}</h2>
             <motion.div
               initial={isCollapsed}
@@ -60,19 +74,18 @@ export function SideMenuSection({
             </motion.div>
           </div>
           {headerAction && <div className="flex items-center">{headerAction}</div>}
-        </motion.div>
-        {/* Divider - absolutely positioned, visible when sidebar is collapsed but section is expanded */}
-        <motion.div
+        </button>
+        {/*
+          Divider fades in via --sm-collapse (0 → 1) as the header fades out. Only while expanded.
+        */}
+        <div
           className="absolute left-2 right-2 top-1 h-px bg-surface-control"
-          initial={false}
-          animate={{
-            opacity: isSideMenuCollapsed && !isCollapsed ? 1 : 0,
-          }}
-          transition={{ duration: 0.15, ease: "easeOut" }}
+          style={{ opacity: isCollapsed ? 0 : "var(--sm-collapse, 0)" }}
         />
       </div>
       <AnimatePresence initial={false}>
         <motion.div
+          ref={contentRef}
           className="w-full"
           initial={isCollapsed ? "collapsed" : "expanded"}
           animate={isCollapsed ? "collapsed" : "expanded"}

apps/webapp/app/components/primitives/Avatar.tsx

@@ -10,7 +10,6 @@ import {
 } from "@heroicons/react/20/solid";
 import type { Prisma } from "@trigger.dev/database";
 import { z } from "zod";
-import { logger } from "~/services/logger.server";
 import { cn } from "~/utils/cn";
 
 export const AvatarType = z.enum(["icon", "letters", "image"]);
@@ -45,7 +44,7 @@ export function parseAvatar(json: Prisma.JsonValue, defaultAvatar: Avatar): Avat
   const parsed = AvatarData.safeParse(json);
 
   if (!parsed.success) {
-    logger.error("Invalid org avatar", { json, error: parsed.error });
+    console.error("Invalid org avatar", { json, error: parsed.error });
     return defaultAvatar;
   }
 

apps/webapp/app/components/primitives/ClipboardField.tsx

@@ -60,9 +60,43 @@ const variants = {
   },
 };
 
+const SECURE_MASK = "••••••••••••••••";
+
+/**
+ * Builds the masked display string, optionally revealing the first/last few
+ * characters in cleartext so users can confirm a copied value. A custom mask
+ * string (when `secure` is a string) is always shown as-is.
+ */
+function maskValue(
+  value: string,
+  secure: boolean | string,
+  revealStart: number,
+  revealEnd: number
+) {
+  if (typeof secure === "string") {
+    return secure;
+  }
+
+  const start = Math.max(0, revealStart);
+  const end = Math.max(0, revealEnd);
+
+  // Nothing to reveal, or revealing would leak the whole value: fully mask.
+  if ((start === 0 && end === 0) || start + end >= value.length) {
+    return SECURE_MASK;
+  }
+
+  const revealedStart = start > 0 ? value.slice(0, start) : "";
+  const revealedEnd = end > 0 ? value.slice(-end) : "";
+  return `${revealedStart}${SECURE_MASK}${revealedEnd}`;
+}
+
 type ClipboardFieldProps = {
   value: string;
   secure?: boolean | string;
+  /** When masked, reveal this many of the first characters in cleartext. */
+  secureRevealStart?: number;
+  /** When masked, reveal this many of the last characters in cleartext. */
+  secureRevealEnd?: number;
   variant: keyof typeof variants;
   className?: string;
   icon?: React.ReactNode;
@@ -73,6 +107,8 @@ type ClipboardFieldProps = {
 export function ClipboardField({
   value,
   secure = false,
+  secureRevealStart = 0,
+  secureRevealEnd = 0,
   variant,
   className,
   icon,
@@ -87,6 +123,8 @@ export function ClipboardField({
     setIsSecure(secure !== undefined && secure);
   }, [secure]);
 
+  const maskedValue = maskValue(value, secure, secureRevealStart, secureRevealEnd);
+
   return (
     <span className={cn(container, fullWidth ? "w-full" : "max-w-fit", className)}>
       {icon && (
@@ -100,7 +138,7 @@ export function ClipboardField({
       <input
         type="text"
         ref={inputIcon}
-        value={isSecure ? (typeof secure === "string" ? secure : "••••••••••••••••") : value}
+        value={isSecure ? maskedValue : value}
         readOnly={true}
         className={cn(
           "shrink grow select-all overflow-x-auto",

apps/webapp/app/components/primitives/CopyableText.tsx

@@ -11,12 +11,19 @@ export function CopyableText({
   className,
   asChild,
   variant,
+  hideTooltip,
 }: {
   value: string;
   copyValue?: string;
   className?: string;
   asChild?: boolean;
   variant?: "icon-right" | "text-below";
+  /**
+   * Hide the "Copy"/"Copied" hint tooltip. Use when this is rendered inside another
+   * Radix tooltip (e.g. the admin debug panel): the nested tooltip would otherwise
+   * fire Radix's global "one tooltip open at a time" close and dismiss the parent.
+   */
+  hideTooltip?: boolean;
 }) {
   const [isHovered, setIsHovered] = useState(false);
   const { copy, copied } = useCopy(copyValue ?? value);
@@ -24,6 +31,24 @@ export function CopyableText({
   const resolvedVariant = variant ?? "icon-right";
 
   if (resolvedVariant === "icon-right") {
+    const iconButton = (
+      <span
+        className={cn(
+          "ml-1 flex size-6 items-center justify-center rounded border border-border-bright bg-background-hover",
+          asChild && "p-1",
+          copied
+            ? "text-green-500"
+            : "text-text-dimmed hover:border-border-bright hover:bg-background-raised hover:text-text-bright"
+        )}
+      >
+        {copied ? (
+          <ClipboardCheckIcon className="size-3.5" />
+        ) : (
+          <ClipboardIcon className="size-3.5" />
+        )}
+      </span>
+    );
+
     return (
       <span
         className={cn("group relative inline-flex h-6 items-center", className)}
@@ -38,29 +63,17 @@ export function CopyableText({
             isHovered ? "flex" : "hidden"
           )}
         >
-          <SimpleTooltip
-            button={
-              <span
-                className={cn(
-                  "ml-1 flex size-6 items-center justify-center rounded border border-border-bright bg-background-hover",
-                  asChild && "p-1",
-                  copied
-                    ? "text-green-500"
-                    : "text-text-dimmed hover:border-border-bright hover:bg-background-raised hover:text-text-bright"
-                )}
-              >
-                {copied ? (
-                  <ClipboardCheckIcon className="size-3.5" />
-                ) : (
-                  <ClipboardIcon className="size-3.5" />
-                )}
-              </span>
-            }
-            content={copied ? "Copied!" : "Copy"}
-            className="font-sans"
-            disableHoverableContent
-            asChild={asChild}
-          />
+          {hideTooltip ? (
+            iconButton
+          ) : (
+            <SimpleTooltip
+              button={iconButton}
+              content={copied ? "Copied!" : "Copy"}
+              className="font-sans"
+              disableHoverableContent
+              asChild={asChild}
+            />
+          )}
         </span>
       </span>
     );

apps/webapp/app/components/primitives/SimpleSelect.tsx

@@ -1,143 +0,0 @@
-"use client";
-
-import * as SelectPrimitive from "@radix-ui/react-select";
-import { Check, ChevronDown } from "lucide-react";
-import * as React from "react";
-import { cn } from "~/utils/cn";
-
-const sizes = {
-  "secondary/small":
-    "text-xs h-6 bg-tertiary border border-tertiary group-hover:text-text-bright hover:border-border-bright pr-2 pl-1.5",
-  medium: "text-sm h-8 bg-tertiary border border-tertiary hover:border-border-bright px-2.5",
-  minimal: "text-xs h-6 bg-transparent hover:bg-tertiary pl-1.5 pr-2",
-};
-
-export type SelectProps = {
-  size?: keyof typeof sizes;
-  width?: "content" | "full";
-};
-
-const Select = SelectPrimitive.Root;
-const SelectGroup = SelectPrimitive.Group;
-const SelectValue = SelectPrimitive.Value;
-const SelectTrigger = React.forwardRef<
-  React.ElementRef<typeof SelectPrimitive.Trigger>,
-  React.ComponentPropsWithoutRef<typeof SelectPrimitive.Trigger> & SelectProps
->(({ className, children, width = "content", size = "secondary/small", ...props }, ref) => {
-  const sizeClassName = sizes[size];
-  return (
-    <SelectPrimitive.Trigger
-      ref={ref}
-      className={cn(
-        "ring-offset-background group flex items-center justify-between gap-x-1 rounded text-text-dimmed transition placeholder:text-text-dimmed hover:text-text-bright focus-visible:focus-custom disabled:cursor-not-allowed disabled:opacity-50",
-        width === "full" ? "w-full" : "w-min",
-        sizeClassName,
-        className
-      )}
-      {...props}
-    >
-      {children}
-      <SelectPrimitive.Icon asChild>
-        <ChevronDown
-          className={cn(
-            "size-4 text-text-dimmed transition group-hover:text-text-bright group-focus:text-text-bright"
-          )}
-        />
-      </SelectPrimitive.Icon>
-    </SelectPrimitive.Trigger>
-  );
-});
-SelectTrigger.displayName = SelectPrimitive.Trigger.displayName;
-
-const SelectContent = React.forwardRef<
-  React.ElementRef<typeof SelectPrimitive.Content>,
-  React.ComponentPropsWithoutRef<typeof SelectPrimitive.Content>
->(({ className, children, position = "popper", ...props }, ref) => (
-  <SelectPrimitive.Portal>
-    <SelectPrimitive.Content
-      ref={ref}
-      className={cn(
-        "relative z-50 min-w-max overflow-hidden rounded-md border border-grid-bright bg-background-dimmed text-text-bright shadow-md animate-in fade-in-40",
-        position === "popper" && "translate-y-1",
-        className
-      )}
-      position={position}
-      {...props}
-    >
-      <SelectPrimitive.Viewport
-        className={cn(
-          "space-y-0.5 px-1 py-1",
-          position === "popper" &&
-            "h-(--radix-select-trigger-height) w-full min-w-(--radix-select-trigger-width)"
-        )}
-      >
-        {children}
-      </SelectPrimitive.Viewport>
-    </SelectPrimitive.Content>
-  </SelectPrimitive.Portal>
-));
-SelectContent.displayName = SelectPrimitive.Content.displayName;
-
-const SelectLabel = React.forwardRef<
-  React.ElementRef<typeof SelectPrimitive.Label>,
-  React.ComponentPropsWithoutRef<typeof SelectPrimitive.Label>
->(({ className, ...props }, ref) => (
-  <SelectPrimitive.Label
-    ref={ref}
-    className={cn(
-      "-ml-1 -mr-1 mb-1 bg-background-deep py-1.5 pl-2 pr-2 font-sans text-xxs font-normal uppercase leading-normal tracking-wider text-text-dimmed first-of-type:mt-0",
-      className
-    )}
-    {...props}
-  />
-));
-SelectLabel.displayName = SelectPrimitive.Label.displayName;
-
-type SelectItemProps = React.ComponentPropsWithoutRef<typeof SelectPrimitive.Item> & {
-  contentClassName?: string;
-};
-
-const SelectItem = React.forwardRef<React.ElementRef<typeof SelectPrimitive.Item>, SelectItemProps>(
-  ({ className, children, contentClassName, ...props }, ref) => (
-    <SelectPrimitive.Item
-      ref={ref}
-      className={cn(
-        "relative flex w-full cursor-default select-none items-center rounded-sm py-1.5 pl-2 pr-12 text-sm outline-hidden transition data-disabled:pointer-events-none data-disabled:opacity-50 hover:bg-background-hover focus:bg-background-hover/50",
-        className
-      )}
-      {...props}
-    >
-      <span className="absolute right-2 flex h-3.5 w-3.5 items-center justify-center">
-        <SelectPrimitive.ItemIndicator>
-          <Check className="h-4 w-4" />
-        </SelectPrimitive.ItemIndicator>
-      </span>
-
-      <SelectPrimitive.ItemText>{children}</SelectPrimitive.ItemText>
-    </SelectPrimitive.Item>
-  )
-);
-SelectItem.displayName = SelectPrimitive.Item.displayName;
-
-const SelectSeparator = React.forwardRef<
-  React.ElementRef<typeof SelectPrimitive.Separator>,
-  React.ComponentPropsWithoutRef<typeof SelectPrimitive.Separator>
->(({ className, ...props }, ref) => (
-  <SelectPrimitive.Separator
-    ref={ref}
-    className={cn("bg-muted -mx-1 my-1 h-px", className)}
-    {...props}
-  />
-));
-SelectSeparator.displayName = SelectPrimitive.Separator.displayName;
-
-export {
-  Select,
-  SelectContent,
-  SelectGroup,
-  SelectItem,
-  SelectLabel,
-  SelectSeparator,
-  SelectTrigger,
-  SelectValue,
-};

apps/webapp/app/components/primitives/Tooltip.tsx

@@ -63,6 +63,7 @@ function SimpleTooltip({
   buttonClassName,
   buttonStyle,
   asChild = false,
+  tabbable = false,
   sideOffset,
   open,
   onOpenChange,
@@ -78,6 +79,9 @@ function SimpleTooltip({
   buttonClassName?: string;
   buttonStyle?: React.CSSProperties;
   asChild?: boolean;
+  /** Set when the trigger wraps an interactive element that should stay tabbable; default removes
+   * it from the tab order (decorative tooltips add no tab stops). */
+  tabbable?: boolean;
   sideOffset?: number;
   open?: boolean;
   onOpenChange?: (open: boolean) => void;
@@ -88,7 +92,7 @@ function SimpleTooltip({
       <Tooltip open={open} onOpenChange={onOpenChange} delayDuration={delayDuration}>
         <TooltipTrigger
           type={asChild ? undefined : "button"}
-          tabIndex={-1}
+          tabIndex={tabbable ? undefined : -1}
           className={cn(!asChild && "h-fit", buttonClassName)}
           style={buttonStyle}
           asChild={asChild}

apps/webapp/app/components/runs/v3/TaskRunsList.tsx

@@ -0,0 +1,117 @@
+import { useLocation, useNavigation, useRevalidator } from "@remix-run/react";
+import { type MutableRefObject, useEffect } from "react";
+import { Button } from "~/components/primitives/Buttons";
+import { PulsingDot } from "~/components/primitives/PulsingDot";
+import { useEnvironment } from "~/hooks/useEnvironment";
+import { useOrganization } from "~/hooks/useOrganizations";
+import { useProject } from "~/hooks/useProject";
+import { useSearchParams } from "~/hooks/useSearchParam";
+import type { NextRunList } from "~/presenters/v3/NextRunListPresenter.server";
+import { useRunsLiveReload } from "~/routes/_app.orgs.$organizationSlug.projects.$projectParam.env.$envParam.runs._index/useRunsLiveReload";
+import { TaskRunsTable } from "./TaskRunsTable";
+
+/**
+ * Compact "N new runs" button, shown in a task page's header to the left of the
+ * time filter when the live-reload hook has detected newer runs.
+ */
+export function NewRunsButton({ count, onClick }: { count: number; onClick: () => void }) {
+  return (
+    <span className="flex duration-150 animate-in fade-in-0">
+      <Button
+        variant="secondary/small"
+        className="text-text-bright"
+        onClick={onClick}
+        LeadingIcon={<PulsingDot className="h-2 w-2" />}
+        tooltip="Refresh to see new runs"
+        aria-label="New runs created. Refresh to see new runs."
+      >
+        {count >= 100 ? "99+ new runs" : `${count} new ${count === 1 ? "run" : "runs"}`}
+      </Button>
+    </span>
+  );
+}
+
+/**
+ * Runs table with live updating, shared by the standard and scheduled task
+ * landing pages. Mirrors the Runs list page: active rows are patched in place
+ * (status/timing/cost). The "N new runs" count is surfaced to the top-bar
+ * button via `onNewRunsCountChange` (count drives visibility) and
+ * `showNewRunsRef` (the latest click action), since the button lives outside
+ * this deferred boundary. The task lives in the route path rather than a
+ * `tasks` filter, so we pass `taskSlug` to scope new-run detection to this task.
+ */
+export function TaskRunsList({
+  list,
+  taskSlug,
+  onNewRunsCountChange,
+  showNewRunsRef,
+}: {
+  list: NextRunList;
+  taskSlug: string;
+  onNewRunsCountChange: (count: number) => void;
+  showNewRunsRef: MutableRefObject<() => void>;
+}) {
+  const organization = useOrganization();
+  const project = useProject();
+  const environment = useEnvironment();
+  const navigation = useNavigation();
+  const location = useLocation();
+  const { has, replace } = useSearchParams();
+  const revalidator = useRevalidator();
+
+  // Loading a new version of this same page (time filter / pagination change).
+  const isLoading =
+    navigation.state === "loading" &&
+    navigation.location !== undefined &&
+    navigation.location.pathname === location.pathname &&
+    navigation.location.search !== location.search;
+
+  const { visibleRuns, newRunsCount, dismissNewRuns, childrenStatusesBasePath } = useRunsLiveReload(
+    {
+      runs: list.runs,
+      hasAnyRuns: list.hasAnyRuns,
+      isLoading,
+      organizationSlug: organization.slug,
+      projectSlug: project.slug,
+      environmentSlug: environment.slug,
+      taskSlug,
+    }
+  );
+
+  const onClickShowNewRuns = () => {
+    const isPaginated = has("cursor") || has("direction");
+    dismissNewRuns();
+    if (isPaginated) {
+      replace({ cursor: undefined, direction: undefined });
+      return;
+    }
+    revalidator.revalidate();
+  };
+
+  // Surface the banner to the top-bar button rendered by the page: keep the
+  // ref's action current, mirror the count up, and clear it when this boundary
+  // unmounts (e.g. the table re-suspends on a filter change).
+  useEffect(() => {
+    showNewRunsRef.current = onClickShowNewRuns;
+  }, [onClickShowNewRuns, showNewRunsRef]);
+  useEffect(() => {
+    onNewRunsCountChange(newRunsCount);
+  }, [newRunsCount, onNewRunsCountChange]);
+  useEffect(() => () => onNewRunsCountChange(0), [onNewRunsCountChange]);
+
+  return (
+    <div className="h-full overflow-y-auto scrollbar-thin scrollbar-track-transparent scrollbar-thumb-surface-control">
+      <TaskRunsTable
+        total={visibleRuns.length}
+        hasFilters={list.hasFilters}
+        filters={list.filters}
+        runs={visibleRuns}
+        childrenStatusesBasePath={childrenStatusesBasePath}
+        isLoading={isLoading}
+        variant="dimmed"
+        showTopBorder={false}
+        stickyHeader
+      />
+    </div>
+  );
+}

apps/webapp/app/db.server.ts

@@ -25,6 +25,7 @@ import {
   assertSplitRealtimeInterlock,
 } from "./v3/runOpsMigration/splitMode.server";
 import { computeRunOpsSplitReadEnabled } from "./v3/runOpsMigration/runOpsSplitReadGate";
+import { assertControlPlaneCoresidencyAdvisory } from "./v3/runOpsMigration/controlPlaneCoresidencySentinel.server";
 import { DATASOURCE_CONTEXT_KEY, startActiveSpan } from "./v3/tracer.server";
 import type { Span } from "@opentelemetry/api";
 import { context, trace } from "@opentelemetry/api";
@@ -188,13 +189,20 @@ export type RunOpsTopology = {
 export type SelectRunOpsTopologyConfig = {
   splitEnabled: boolean;
   legacyUrl?: string;
+  legacyReplicaUrl?: string;
   newUrl?: string;
   newReplicaUrl?: string;
+  // When true, legacy reuses the control-plane client instead of opening its own pool. Defaults to false.
+  legacySharesControlPlane?: boolean;
 };
 export type RunOpsClientBuilders = {
   controlPlane: RunOpsClients;
   buildNewWriter: (url: string, clientType: string) => RunOpsPrismaClient;
   buildNewReplica: (url: string, clientType: string) => RunOpsPrismaClient;
+  // Legacy builders return the same PrismaClient/PrismaReplicaClient types as the control plane (no
+  // RunOpsPrismaClient double-cast needed): the legacy DB carries the full control-plane schema.
+  buildLegacyWriter: (url: string, clientType: string) => PrismaClient;
+  buildLegacyReplica: (url: string, clientType: string) => PrismaReplicaClient;
 };
 
 // Pure run-ops client selector. No env, no isSplitEnabled() — those
@@ -220,7 +228,17 @@ export function selectRunOpsTopology(
     return { newRunOps: cpFallback, legacyRunOps: controlPlane, controlPlane };
   }
 
-  const legacyRunOps = controlPlane;
+  // Same-DB legacy reuses the control-plane pool; only build a separate pool once the DSNs diverge.
+  let legacyRunOps: RunOpsClients;
+  if (config.legacySharesControlPlane) {
+    legacyRunOps = controlPlane;
+  } else {
+    const legacyWriter = builders.buildLegacyWriter(config.legacyUrl, "run-ops-legacy-writer");
+    const legacyReplica: PrismaReplicaClient = config.legacyReplicaUrl
+      ? builders.buildLegacyReplica(config.legacyReplicaUrl, "run-ops-legacy-reader")
+      : legacyWriter;
+    legacyRunOps = { writer: legacyWriter, replica: legacyReplica };
+  }
 
   const newWriter = builders.buildNewWriter(config.newUrl, "run-ops-new-writer");
   const newReplica: RunOpsPrismaClient = config.newReplicaUrl
@@ -246,12 +264,32 @@ const runOpsTopology: RunOpsTopology = singleton("runOpsTopology", () => {
   // Gate on the opt-in flag too: the distinct-DB sentinel only runs when the flag is on.
   const splitEnabled = env.RUN_OPS_SPLIT_ENABLED && !!newUrl && !!env.RUN_OPS_LEGACY_DATABASE_URL;
 
+  // Alias legacy onto the control-plane pool when both roles resolve to the same DB (replica URLs
+  // fall back to their writer, matching how the clients themselves fall back).
+  const cpWriterUrl = env.CONTROL_PLANE_DATABASE_URL ?? env.DATABASE_URL;
+  const cpReplicaUrl = env.CONTROL_PLANE_DATABASE_READ_REPLICA_URL ?? env.DATABASE_READ_REPLICA_URL;
+  const legacySharesControlPlane =
+    sameDatabaseTarget(env.RUN_OPS_LEGACY_DATABASE_URL, cpWriterUrl) &&
+    sameDatabaseTarget(
+      env.RUN_OPS_LEGACY_DATABASE_READ_REPLICA_URL ?? env.RUN_OPS_LEGACY_DATABASE_URL,
+      cpReplicaUrl ?? cpWriterUrl
+    );
+
+  // Only meaningful for an independent legacy pool; a shared pool routes reads through $replica.
+  if (splitEnabled && !legacySharesControlPlane && !env.RUN_OPS_LEGACY_DATABASE_READ_REPLICA_URL) {
+    logger.warn(
+      "RUN_OPS_LEGACY_DATABASE_READ_REPLICA_URL is unset while split is enabled; legacy reads will hit the legacy primary"
+    );
+  }
+
   return selectRunOpsTopology(
     {
       splitEnabled,
       legacyUrl: env.RUN_OPS_LEGACY_DATABASE_URL,
+      legacyReplicaUrl: env.RUN_OPS_LEGACY_DATABASE_READ_REPLICA_URL,
       newUrl,
       newReplicaUrl: env.RUN_OPS_DATABASE_READ_REPLICA_URL,
+      legacySharesControlPlane,
     },
     {
       controlPlane: { writer: prisma, replica: $replica },
@@ -268,6 +306,18 @@ const runOpsTopology: RunOpsTopology = singleton("runOpsTopology", () => {
             tagDatasourceRunOps("replica", buildRunOpsReplicaClient({ url, clientType }))
           )
         ),
+      // Legacy client shares the exact control-plane wrapper stack (the legacy DB carries the full
+      // control-plane schema); markReadReplicaClient only on a real replica URL, as with the NEW replica.
+      buildLegacyWriter: (url, clientType) =>
+        captureInfrastructureErrors(
+          tagDatasource("writer", buildWriterClient({ url, clientType }))
+        ),
+      buildLegacyReplica: (url, clientType) =>
+        markReadReplicaClient(
+          captureInfrastructureErrors(
+            tagDatasource("replica", buildReplicaClient({ url, clientType }))
+          )
+        ),
     }
   );
 });
@@ -281,8 +331,17 @@ export const runOpsNewPrisma: PrismaClient = runOpsTopology.newRunOps
   .writer as unknown as PrismaClient;
 export const runOpsNewReplica: PrismaReplicaClient = runOpsTopology.newRunOps
   .replica as unknown as PrismaReplicaClient;
+// Track 2: under split-on these point at the INDEPENDENT legacy client (its own DSN); under split-off
+// or missing URLs they still alias the control-plane client, so single-DB installs are unchanged.
 export const runOpsLegacyPrisma: PrismaClient = runOpsTopology.legacyRunOps.writer;
 export const runOpsLegacyReplica: PrismaReplicaClient = runOpsTopology.legacyRunOps.replica;
+// Branded legacy handles typed as RunOpsPrismaClient for the run-store boundary — same underlying
+// legacy writer/replica as runOpsLegacyPrisma/runOpsLegacyReplica above, but carrying the run-ops
+// brand so the guard classifies provably-legacy access as `runops`, not `cp`.
+export const runOpsLegacyPrismaClient: RunOpsPrismaClient = runOpsTopology.legacyRunOps
+  .writer as unknown as RunOpsPrismaClient;
+export const runOpsLegacyReplicaClient: RunOpsPrismaClient = runOpsTopology.legacyRunOps
+  .replica as unknown as RunOpsPrismaClient;
 
 export const runOpsSplitReadEnabled: boolean = computeRunOpsSplitReadEnabled({
   newReplica: runOpsNewReplicaClient,
@@ -295,8 +354,8 @@ export const runOpsSplitReadEnabled: boolean = computeRunOpsSplitReadEnabled({
 
 // Boot-time interlock: if the flag is on but the distinct-DB sentinel does not
 // confirm two physically-distinct run-ops DBs, refuse to enable split (data-loss
-// interlock). Async, so it cannot live in the synchronous singleton factory —
-// call it from the eager-boot path before any run-ops routing is wired.
+// interlock). Async, so it cannot live in the synchronous singleton factory — called
+// fire-and-forget from the eager-boot path (routing is wired synchronously at module load).
 export async function assertRunOpsSplitSentinel(): Promise<void> {
   if (!env.RUN_OPS_SPLIT_ENABLED) return;
   // Realtime interlock (synchronous): Electric replicates only from the control-plane
@@ -312,6 +371,9 @@ export async function assertRunOpsSplitSentinel(): Promise<void> {
       "RUN_OPS_SPLIT_ENABLED is on but the distinct-DB sentinel did not confirm two physically-distinct run-ops DBs; refusing to enable split (data-loss interlock)."
     );
   }
+  // Advisory-only (T2.3): observe legacy vs control-plane co-residency. Emits a metric + log and only
+  // throws when RUN_OPS_EXPECT_CONTROL_PLANE_SPLIT is on AND co-residency is positively confirmed.
+  await assertControlPlaneCoresidencyAdvisory();
 }
 
 function getClient() {
@@ -662,7 +724,10 @@ function buildRunOpsReplicaClient({
   clientType: string;
 }): RunOpsPrismaClient {
   const replicaUrl = extendQueryParams(url, {
-    connection_limit: env.DATABASE_CONNECTION_LIMIT.toString(),
+    // The new run-ops replica connects unpooled, so allow capping it independently of the writer.
+    connection_limit: (
+      env.RUN_OPS_DATABASE_READ_REPLICA_CONNECTION_LIMIT ?? env.DATABASE_CONNECTION_LIMIT
+    ).toString(),
     pool_timeout: env.DATABASE_POOL_TIMEOUT.toString(),
     connection_timeout: env.DATABASE_CONNECTION_TIMEOUT.toString(),
     application_name: env.SERVICE_NAME,
@@ -705,6 +770,25 @@ function buildRunOpsReplicaClient({
   return client;
 }
 
+// True when two DSNs point at the same database (host/port/dbname/user), ignoring query params and
+// password. Parse failure or a missing URL returns false, so an unrecognized DSN just isn't aliased.
+export function sameDatabaseTarget(a: string | undefined, b: string | undefined): boolean {
+  if (!a || !b) return false;
+  try {
+    const ua = new URL(a);
+    const ub = new URL(b);
+    const port = (u: URL) => u.port || "5432";
+    return (
+      ua.hostname.toLowerCase() === ub.hostname.toLowerCase() &&
+      port(ua) === port(ub) &&
+      ua.pathname === ub.pathname &&
+      ua.username === ub.username
+    );
+  } catch {
+    return false;
+  }
+}
+
 function extendQueryParams(hrefOrUrl: string | URL, queryParams: Record<string, string>) {
   const url = new URL(hrefOrUrl);
   const query = url.searchParams;

apps/webapp/app/entry.server.tsx

@@ -7,7 +7,6 @@ import { parseAcceptLanguage } from "intl-parse-accept-language";
 import isbot from "isbot";
 import { renderToPipeableStream } from "react-dom/server";
 import { PassThrough } from "stream";
-import * as Worker from "~/services/worker.server";
 import { initMollifierDrainerWorker } from "~/v3/mollifierDrainerWorker.server";
 import { initMollifierStaleSweepWorker } from "~/v3/mollifierStaleSweepWorker.server";
 import { initBillingLimitWorker } from "~/v3/billingLimitWorker.server";
@@ -19,7 +18,6 @@ import { assertRunOpsSplitSentinel, Prisma } from "./db.server";
 import { env } from "./env.server";
 import { eventLoopMonitor } from "./eventLoopMonitor.server";
 import { logger } from "./services/logger.server";
-import { resourceMonitor } from "./services/resourceMonitor.server";
 import { singleton } from "./utils/singleton";
 import { remoteBuildsEnabled } from "./v3/remoteImageBuilder.server";
 import {
@@ -56,6 +54,12 @@ export default function handleRequest(
 ) {
   const url = new URL(request.url);
 
+  // Stale documents reference /build asset hashes that 404 after a deploy —
+  // always revalidate HTML. Route-set headers win.
+  if (!responseHeaders.has("Cache-Control")) {
+    responseHeaders.set("Cache-Control", "no-cache");
+  }
+
   if (url.pathname.startsWith("/login")) {
     responseHeaders.set("X-Frame-Options", "SAMEORIGIN");
     responseHeaders.set("Content-Security-Policy", "frame-ancestors 'self'");
@@ -227,10 +231,6 @@ export const handleError = wrapHandleErrorWithSentry((error, { request }) => {
   }
 });
 
-Worker.init().catch((error) => {
-  logError(error);
-});
-
 initMollifierDrainerWorker();
 initMollifierStaleSweepWorker();
 initBillingLimitWorker();
@@ -241,10 +241,6 @@ bootstrap().catch((error) => {
 
 function logError(error: unknown, request?: Request) {
   console.error(error);
-
-  if (error instanceof Error && error.message.startsWith("There are locked jobs present")) {
-    console.log("⚠️  graphile-worker migration issue detected!");
-  }
 }
 
 process.on("uncaughtException", (error, origin) => {
@@ -304,6 +300,7 @@ singleton("SentryTenantContextProcessor", () => {
 
 export { apiRateLimiter } from "./services/apiRateLimit.server";
 export { engineRateLimiter } from "./services/engineRateLimit.server";
+export { otlpRateLimiter } from "./services/otlpRateLimit.server";
 export { runWithHttpContext } from "./services/httpAsyncStorage.server";
 export { tenantContextMiddleware } from "./services/tenantContextResolver.server";
 export { socketIo } from "./v3/handleSocketIo.server";
@@ -318,7 +315,3 @@ if (remoteBuildsEnabled()) {
 } else {
   console.log("🏗️  Local builds enabled");
 }
-
-if (env.RESOURCE_MONITOR_ENABLED === "1") {
-  resourceMonitor.startMonitoring(1000);
-}

apps/webapp/app/env.server.ts

@@ -85,6 +85,29 @@ const S2EnvSchema = z.preprocess(
   ])
 );
 
+// Previously published secret values must never be accepted, including when
+// an existing deployment or external secret manager still supplies one.
+const INSECURE_SECRET_VALUES = [
+  "managed-secret",
+  "2818143646516f6fffd707b36f334bbb",
+  "44da78b7bbb0dfe709cf38931d25dcdd",
+  "f686147ab967943ebbe9ed3b496e465a",
+  "447c29678f9eaf289e9c4b70d3dd8a7f",
+];
+
+// Escape hatch for deployments that can't rotate a published default yet (e.g.
+// ENCRYPTION_KEY protects existing data). Read raw: a refine can't see the
+// sibling parsed flag.
+const allowInsecureDefaultSecrets = ["true", "1"].includes(
+  (process.env.ALLOW_INSECURE_DEFAULT_SECRETS ?? "").toLowerCase().trim()
+);
+
+const isNotInsecureSecret = (value: string) =>
+  allowInsecureDefaultSecrets || !INSECURE_SECRET_VALUES.includes(value);
+
+const INSECURE_SECRET_MESSAGE =
+  "must not be a known-insecure published default; set a strong, unique value. If you cannot rotate it yet (e.g. it protects existing encrypted data or active sessions), set ALLOW_INSECURE_DEFAULT_SECRETS=1 to boot while you migrate.";
+
 const EnvironmentSchema = z
   .object({
     NODE_ENV: z.union([z.literal("development"), z.literal("production"), z.literal("test")]),
@@ -113,6 +136,8 @@ const EnvironmentSchema = z
     // agent dark; flip to "1" to enable it for everyone at GA. Per-org overrides
     // (org featureFlags) win regardless.
     DASHBOARD_AGENT_ENABLED: z.string().default("0"),
+    // Gates the create-org management API endpoint (default off).
+    ORG_CREATION_API_ENABLED: z.string().default("0"),
     // "1" gives admins/impersonators an everywhere-preview (default off),
     // separate from the per-org rollout flag above.
     DASHBOARD_AGENT_ADMIN_PREVIEW: z.string().default("0"),
@@ -138,8 +163,10 @@ const EnvironmentSchema = z
       .string()
       .refine(isValidDatabaseUrl, "RUN_OPS_DATABASE_URL is invalid")
       .optional(),
-    // The LEGACY run-ops DB (the control-plane DB during the transition). When unset, legacy
-    // run-ops reuses the existing DATABASE_URL (legacy run-ops == control-plane DB initially).
+    // The LEGACY run-ops DB. Now a CONNECTED DSN (Track 2): when split is on and this is set it builds
+    // an INDEPENDENT legacy Prisma client, no longer an alias of the control-plane client (nor merely
+    // the sentinel's probe target). Unset -> legacy reuses the control-plane client / DATABASE_URL, so
+    // single-DB and self-host installs boot byte-identical.
     RUN_OPS_LEGACY_DATABASE_URL: z
       .string()
       .refine(isValidDatabaseUrl, "RUN_OPS_LEGACY_DATABASE_URL is invalid")
@@ -151,6 +178,24 @@ const EnvironmentSchema = z
       .string()
       .refine(isValidDatabaseUrl, "RUN_OPS_DATABASE_READ_REPLICA_URL is invalid")
       .optional(),
+    // The LEGACY run-ops DB read replica (Track 2). Unset -> the legacy replica handle falls back to the
+    // legacy WRITER (as $replica does with no CP replica). Set in production so legacy reads hit the reader.
+    RUN_OPS_LEGACY_DATABASE_READ_REPLICA_URL: z
+      .string()
+      .refine(isValidDatabaseUrl, "RUN_OPS_LEGACY_DATABASE_READ_REPLICA_URL is invalid")
+      .optional(),
+    // Optional cap for the unpooled new run-ops read replica. Unset falls back to DATABASE_CONNECTION_LIMIT.
+    RUN_OPS_DATABASE_READ_REPLICA_CONNECTION_LIMIT: z.coerce.number().int().optional(),
+    // Direct DSN for applying the full @trigger.dev/database migrations to the LEGACY run-ops DB, keeping
+    // its schema current after the control plane moves off it. Direct, not pooled — migrations never run
+    // over a pooler. Optional; unset -> the entrypoint's legacy migrate step is skipped.
+    RUN_OPS_LEGACY_DIRECT_URL: z
+      .string()
+      .refine(isValidDatabaseUrl, "RUN_OPS_LEGACY_DIRECT_URL is invalid")
+      .optional(),
+    // Advisory control-plane co-residency sentinel enforcement (Track 2, T2.3). Default OFF; the advisory
+    // arm always emits its metric, this only turns a still-co-resident pair into a hard boot failure.
+    RUN_OPS_EXPECT_CONTROL_PLANE_SPLIT: BoolEnv.default(false),
     // --- Control-plane datasource repoint. Additive-only. ---
     // Optional control-plane DB. Unset (self-host/single-DB) -> getClient()/getReplicaClient() fall back to
     // DATABASE_URL/DATABASE_READ_REPLICA_URL, so boot is byte-identical. When set, these point at the
@@ -166,14 +211,15 @@ const EnvironmentSchema = z
     // Control-plane cache relax knobs. Unset -> defaults (DEFAULT_CP_CACHE_TTL_MS / _MAX_ENTRIES).
     CONTROL_PLANE_CACHE_TTL_MS: z.coerce.number().int().optional(),
     CONTROL_PLANE_CACHE_MAX_ENTRIES: z.coerce.number().int().optional(),
-    SESSION_SECRET: z.string(),
-    MAGIC_LINK_SECRET: z.string(),
+    SESSION_SECRET: z.string().min(1).refine(isNotInsecureSecret, INSECURE_SECRET_MESSAGE),
+    MAGIC_LINK_SECRET: z.string().min(1).refine(isNotInsecureSecret, INSECURE_SECRET_MESSAGE),
     ENCRYPTION_KEY: z
       .string()
       .refine(
         (val) => Buffer.from(val, "utf8").length === 32,
         "ENCRYPTION_KEY must be exactly 32 bytes"
-      ),
+      )
+      .refine(isNotInsecureSecret, INSECURE_SECRET_MESSAGE),
     WHITELISTED_EMAILS: z
       .string()
       .refine(isValidRegex, "WHITELISTED_EMAILS must be a valid regex.")
@@ -224,9 +270,6 @@ const EnvironmentSchema = z
     PLAIN_CUSTOMER_CARDS_SECRET: z.string().optional(),
     PLAIN_CUSTOMER_CARDS_KEY: z.string().optional(),
     PLAIN_CUSTOMER_CARDS_HEADERS: z.string().optional(),
-    WORKER_SCHEMA: z.string().default("graphile_worker"),
-    WORKER_CONCURRENCY: z.coerce.number().int().default(10),
-    WORKER_POLL_INTERVAL: z.coerce.number().int().default(1000),
     // How often each replica reloads the global flags snapshot from the DB.
     // Sets kill/ramp propagation latency.
     GLOBAL_FLAGS_RELOAD_INTERVAL_MS: z.coerce.number().int().min(1000).default(5000),
@@ -381,6 +424,8 @@ const EnvironmentSchema = z
 
     // Master switch for the native realtime backend; off = Electric serves everything, publishes no-op.
     REALTIME_BACKEND_NATIVE_ENABLED: z.string().default("0"),
+    // Default backend when an org has no `realtimeBackend` override and no global flag row is set.
+    REALTIME_BACKEND_DEFAULT: z.enum(["electric", "native", "shadow"]).default("electric"),
     // Live long-poll backstop hold (ms); matches Electric's ~20s cadence.
     REALTIME_BACKEND_NATIVE_LIVE_POLL_TIMEOUT_MS: z.coerce.number().int().default(20_000),
     // Jitter ratio on the live-poll hold (0.15 = ±15%) to avoid synchronized refetch herds.
@@ -526,9 +571,22 @@ const EnvironmentSchema = z
     API_RATE_LIMIT_JWT_WINDOW: z.string().default("1m"),
     API_RATE_LIMIT_JWT_TOKENS: z.coerce.number().int().default(60),
 
-    //v3
-    PROVIDER_SECRET: z.string().default("provider-secret"),
-    COORDINATOR_SECRET: z.string().default("coordinator-secret"),
+    // Per-IP rate limit for the unauthenticated OTLP ingestion endpoints
+    // (/otel/*). Bounds unauthenticated request rates. Opt-in
+    // (disabled by default): because it keys on the source IP, it is only
+    // safe to enable when each client presents a distinct IP through a proxy
+    // that appends the real client IP to X-Forwarded-For. Enabling it where
+    // many clients share one egress IP (e.g. behind NAT or a shared proxy)
+    // would collapse that traffic into a single bucket and could throttle
+    // legitimate telemetry. Set OTLP_RATE_LIMIT_ENABLED=1 to enable, then tune
+    // OTLP_RATE_LIMIT_MAX / OTLP_RATE_LIMIT_WINDOW for expected volume.
+    OTLP_RATE_LIMIT_ENABLED: z.string().default("0"),
+    OTLP_RATE_LIMIT_WINDOW: z
+      .string()
+      .regex(/^\d+ ?(?:ms|s|m|h|d)$/)
+      .default("1m"),
+    OTLP_RATE_LIMIT_MAX: z.coerce.number().int().positive().default(3000),
+
     DEPOT_TOKEN: z.string().optional(),
     DEPOT_ORG_ID: z.string().optional(),
     DEPOT_REGION: z.string().default("us-east-1"),
@@ -616,16 +674,6 @@ const EnvironmentSchema = z
     // log-only mode before enforcement.
     DEPRECATE_V3_CLI_DEPLOYS_ENABLED: z.string().default("0"),
 
-    // Master switch for the v3 engine (RunEngineVersion.V1) shutdown. When
-    // enabled it: rejects triggers that resolve to V1 (single, batch, schedule,
-    // replay, triggerAndWait) with a graceful error pointing at the v4 migration
-    // guide; closes the legacy `trigger dev` websocket used by v3 CLIs; and turns
-    // the V1 run-lifecycle background jobs (heartbeat timeout, TTL expiry, retry,
-    // resume, scheduled fires) into no-ops so abandoned V1 runs stop generating
-    // database load. v4 (V2) is never affected (every gate also checks the run is
-    // V1). Defaults to off so self-hosted instances still on V1 keep working.
-    DEPRECATE_V3_ENABLED: z.string().default("0"),
-
     // Verify the deploy image exists before promoting. Disable for out-of-band/air-gapped push. ECR only.
     DEPLOY_IMAGE_VERIFICATION_ENABLED: BoolEnv.default(true),
 
@@ -659,13 +707,19 @@ const EnvironmentSchema = z
     EVENTS_MEMORY_PRESSURE_THRESHOLD: z.coerce.number().int().default(5000),
     EVENTS_LOAD_SHEDDING_THRESHOLD: z.coerce.number().int().default(100000),
     EVENTS_LOAD_SHEDDING_ENABLED: z.string().default("1"),
-    SHARED_QUEUE_CONSUMER_POOL_SIZE: z.coerce.number().int().default(10),
-    SHARED_QUEUE_CONSUMER_INTERVAL_MS: z.coerce.number().int().default(100),
-    SHARED_QUEUE_CONSUMER_NEXT_TICK_INTERVAL_MS: z.coerce.number().int().default(100),
-    SHARED_QUEUE_CONSUMER_EMIT_RESUME_DEPENDENCY_TIMEOUT_MS: z.coerce.number().int().default(1000),
-    SHARED_QUEUE_CONSUMER_RESOLVE_PAYLOADS_BATCH_SIZE: z.coerce.number().int().default(25),
 
-    MANAGED_WORKER_SECRET: z.string().default("managed-secret"),
+    MANAGED_WORKER_SECRET: z.string().min(1).refine(isNotInsecureSecret, INSECURE_SECRET_MESSAGE),
+
+    // Allow booting with a known-insecure published default secret. Temporary
+    // bridge for deployments that can't rotate yet; rotate as soon as possible.
+    ALLOW_INSECURE_DEFAULT_SECRETS: BoolEnv.default(false),
+
+    // Tenant scoping on worker actions is header-driven (folded into the engine snapshot read) and
+    // needs no flag. This is only the no-header fallback: when "1", a worker action on a run created
+    // after WORKLOAD_TOKEN_CUTOFF without a verified env header is rejected; runs on or before the
+    // cutoff pass (grandfathered). Default off = no run-row read, byte-for-byte today's behavior.
+    WORKLOAD_CREATED_AT_GATE_ENABLED: z.string().default("0"),
+    WORKLOAD_TOKEN_CUTOFF: z.string().datetime().optional(),
 
     // Development OTEL environment variables
     DEV_OTEL_EXPORTER_OTLP_ENDPOINT: z.string().optional(),
@@ -783,50 +837,9 @@ const EnvironmentSchema = z
 
     LOOPS_API_KEY: z.string().optional(),
     ATTIO_API_KEY: z.string().optional(),
-    MARQS_DISABLE_REBALANCING: BoolEnv.default(false),
-    MARQS_VISIBILITY_TIMEOUT_MS: z.coerce
-      .number()
-      .int()
-      .default(60 * 1000 * 15),
-    MARQS_SHARED_QUEUE_LIMIT: z.coerce.number().int().default(1000),
-    MARQS_MAXIMUM_QUEUE_PER_ENV_COUNT: z.coerce.number().int().default(50),
-    MARQS_DEV_QUEUE_LIMIT: z.coerce.number().int().default(1000),
-    MARQS_MAXIMUM_NACK_COUNT: z.coerce.number().int().default(64),
-    MARQS_CONCURRENCY_LIMIT_BIAS: z.coerce.number().default(0.75),
-    MARQS_AVAILABLE_CAPACITY_BIAS: z.coerce.number().default(0.3),
-    MARQS_QUEUE_AGE_RANDOMIZATION_BIAS: z.coerce.number().default(0.25),
-    MARQS_REUSE_SNAPSHOT_COUNT: z.coerce.number().int().default(0),
-    MARQS_MAXIMUM_ENV_COUNT: z.coerce.number().int().optional(),
-    MARQS_SHARED_WORKER_QUEUE_CONSUMER_INTERVAL_MS: z.coerce.number().int().default(250),
-    MARQS_SHARED_WORKER_QUEUE_MAX_MESSAGE_COUNT: z.coerce.number().int().default(10),
-
-    MARQS_SHARED_WORKER_QUEUE_EAGER_DEQUEUE_ENABLED: z.string().default("0"),
-    MARQS_WORKER_ENABLED: z.string().default("0"),
-    MARQS_WORKER_COUNT: z.coerce.number().int().default(2),
-    MARQS_WORKER_CONCURRENCY_LIMIT: z.coerce.number().int().default(50),
-    MARQS_WORKER_CONCURRENCY_TASKS_PER_WORKER: z.coerce.number().int().default(5),
-    MARQS_WORKER_POLL_INTERVAL_MS: z.coerce.number().int().default(100),
-    MARQS_WORKER_IMMEDIATE_POLL_INTERVAL_MS: z.coerce.number().int().default(100),
-    MARQS_WORKER_SHUTDOWN_TIMEOUT_MS: z.coerce.number().int().default(60_000),
-    MARQS_SHARED_WORKER_QUEUE_COOLOFF_COUNT_THRESHOLD: z.coerce.number().int().default(10),
-    MARQS_SHARED_WORKER_QUEUE_COOLOFF_PERIOD_MS: z.coerce.number().int().default(5_000),
 
     PROD_TASK_HEARTBEAT_INTERVAL_MS: z.coerce.number().int().optional(),
 
-    VERBOSE_GRAPHILE_LOGGING: z.string().default("false"),
-    V2_MARQS_ENABLED: z.string().default("0"),
-    V2_MARQS_CONSUMER_POOL_ENABLED: z.string().default("0"),
-    V2_MARQS_CONSUMER_POOL_SIZE: z.coerce.number().int().default(10),
-    V2_MARQS_CONSUMER_POLL_INTERVAL_MS: z.coerce.number().int().default(1000),
-    V2_MARQS_QUEUE_SELECTION_COUNT: z.coerce.number().int().default(36),
-    V2_MARQS_VISIBILITY_TIMEOUT_MS: z.coerce
-      .number()
-      .int()
-      .default(60 * 1000 * 15),
-    V2_MARQS_DEFAULT_ENV_CONCURRENCY: z.coerce.number().int().default(100),
-    V2_MARQS_VERBOSE: z.string().default("0"),
-    V3_MARQS_CONCURRENCY_MONITOR_ENABLED: z.string().default("0"),
-    V2_MARQS_CONCURRENCY_MONITOR_ENABLED: z.string().default("0"),
     /* Usage settings */
     USAGE_EVENT_URL: z.string().optional(),
     PROD_USAGE_HEARTBEAT_INTERVAL_MS: z.coerce.number().int().optional(),
@@ -834,7 +847,6 @@ const EnvironmentSchema = z
     CENTS_PER_RUN: z.coerce.number().default(0),
 
     EVENT_LOOP_MONITOR_ENABLED: z.string().default("1"),
-    RESOURCE_MONITOR_ENABLED: z.string().default("0"),
     MAXIMUM_LIVE_RELOADING_EVENTS: z.coerce.number().int().default(1000),
     MAXIMUM_TRACE_SUMMARY_VIEW_COUNT: z.coerce.number().int().default(25_000),
     MAXIMUM_TRACE_DETAILED_SUMMARY_VIEW_COUNT: z.coerce.number().int().default(10_000),
@@ -1166,55 +1178,6 @@ const EnvironmentSchema = z
     /** The CLI should connect to this for dev runs */
     DEV_ENGINE_URL: z.string().default(process.env.APP_ORIGIN ?? "http://localhost:3030"),
 
-    LEGACY_RUN_ENGINE_WORKER_ENABLED: z.string().default(process.env.WORKER_ENABLED ?? "true"),
-    LEGACY_RUN_ENGINE_WORKER_CONCURRENCY_WORKERS: z.coerce.number().int().default(2),
-    LEGACY_RUN_ENGINE_WORKER_CONCURRENCY_TASKS_PER_WORKER: z.coerce.number().int().default(1),
-    LEGACY_RUN_ENGINE_WORKER_POLL_INTERVAL: z.coerce.number().int().default(1000),
-    LEGACY_RUN_ENGINE_WORKER_IMMEDIATE_POLL_INTERVAL: z.coerce.number().int().default(50),
-    LEGACY_RUN_ENGINE_WORKER_CONCURRENCY_LIMIT: z.coerce.number().int().default(50),
-    LEGACY_RUN_ENGINE_WORKER_SHUTDOWN_TIMEOUT_MS: z.coerce.number().int().default(60_000),
-    LEGACY_RUN_ENGINE_WORKER_LOG_LEVEL: z
-      .enum(["log", "error", "warn", "info", "debug"])
-      .default("info"),
-
-    LEGACY_RUN_ENGINE_WORKER_REDIS_HOST: z
-      .string()
-      .optional()
-      .transform((v) => v ?? process.env.REDIS_HOST),
-    LEGACY_RUN_ENGINE_WORKER_REDIS_READER_HOST: z
-      .string()
-      .optional()
-      .transform((v) => v ?? process.env.REDIS_READER_HOST),
-    LEGACY_RUN_ENGINE_WORKER_REDIS_READER_PORT: z.coerce
-      .number()
-      .optional()
-      .transform(
-        (v) =>
-          v ?? (process.env.REDIS_READER_PORT ? parseInt(process.env.REDIS_READER_PORT) : undefined)
-      ),
-    LEGACY_RUN_ENGINE_WORKER_REDIS_PORT: z.coerce
-      .number()
-      .optional()
-      .transform(
-        (v) => v ?? (process.env.REDIS_PORT ? parseInt(process.env.REDIS_PORT) : undefined)
-      ),
-    LEGACY_RUN_ENGINE_WORKER_REDIS_USERNAME: z
-      .string()
-      .optional()
-      .transform((v) => v ?? process.env.REDIS_USERNAME),
-    LEGACY_RUN_ENGINE_WORKER_REDIS_PASSWORD: z
-      .string()
-      .optional()
-      .transform((v) => v ?? process.env.REDIS_PASSWORD),
-    LEGACY_RUN_ENGINE_WORKER_REDIS_TLS_DISABLED: z
-      .string()
-      .default(process.env.REDIS_TLS_DISABLED ?? "false"),
-    LEGACY_RUN_ENGINE_WORKER_REDIS_CLUSTER_MODE_ENABLED: z.string().default("0"),
-
-    LEGACY_RUN_ENGINE_WAITING_FOR_DEPLOY_BATCH_SIZE: z.coerce.number().int().default(100),
-    LEGACY_RUN_ENGINE_WAITING_FOR_DEPLOY_BATCH_STAGGER_MS: z.coerce.number().int().default(1_000),
-    LEGACY_RUN_ENGINE_WAITING_FOR_DEPLOY_DISABLED: z.string().default("0"),
-
     COMMON_WORKER_ENABLED: z.string().default(process.env.WORKER_ENABLED ?? "true"),
     COMMON_WORKER_CONCURRENCY_WORKERS: z.coerce.number().int().default(2),
     COMMON_WORKER_CONCURRENCY_TASKS_PER_WORKER: z.coerce.number().int().default(10),
@@ -1399,6 +1362,9 @@ const EnvironmentSchema = z
     // claim TTL), how long a waiter blocks before timing out, and the
     // waiter poll interval.
     TRIGGER_MOLLIFIER_CLAIM_TTL_SECONDS: z.coerce.number().int().positive().default(30),
+    // Pipeline floor: the claim never shrinks below this even for a short customer key TTL, so it
+    // can't expire mid-pipeline and let a loser re-claim (cross-DB duplicate under the split).
+    TRIGGER_MOLLIFIER_CLAIM_MIN_TTL_SECONDS: z.coerce.number().int().positive().default(5),
     TRIGGER_MOLLIFIER_CLAIM_WAIT_MS: z.coerce.number().int().positive().default(5_000),
     TRIGGER_MOLLIFIER_CLAIM_POLL_MS: z.coerce.number().int().positive().default(25),
 
@@ -1727,6 +1693,14 @@ const EnvironmentSchema = z
     RUN_REPLICATION_DISABLE_PAYLOAD_INSERT: z.string().default("0"),
     RUN_REPLICATION_DISABLE_ERROR_FINGERPRINTING: z.string().default("0"),
 
+    // Connection URL for the LEGACY runs-replication source (the runs-CDC slot on the legacy runs DB, plus
+    // the admin recovery route). Direct, not pooled: replication can't run over a pooler. Optional; unset ->
+    // falls back to DATABASE_URL, so nothing changes today.
+    RUN_REPLICATION_LEGACY_DATABASE_URL: z
+      .string()
+      .refine(isValidDatabaseUrl, "RUN_REPLICATION_LEGACY_DATABASE_URL is invalid")
+      .optional(),
+
     // --- Run-ops DB split — second replication source (the NEW dedicated run-ops DB). ---
     // Cloud-only; only consulted when isSplitEnabled() is true. Self-host never sets these.
     // Connection URL for the run-ops DB used by the runs-replication source. Required when the split is
@@ -1753,6 +1727,10 @@ const EnvironmentSchema = z
     RUN_OPS_MINT_ENABLED: BoolEnv.default(false),
     RUN_OPS_MINT_FLAG_CACHE_TTL_MS: z.coerce.number().int().default(30_000),
     RUN_OPS_MINT_FLAG_CACHE_MAX_ENTRIES: z.coerce.number().int().default(10_000),
+    // Deterministic wall-clock cutover after a runOpsMintKind flip. Must exceed the sum
+    // of RUN_OPS_MINT_FLAG_CACHE_TTL_MS and the control-plane cache TTL so every process
+    // (stale or fresh) resolves to the same kind for the whole window. See mintFlipGrace.ts.
+    RUN_OPS_MINT_FLIP_GRACE_MS: z.coerce.number().int().default(90_000),
 
     // Session replication (Postgres → ClickHouse sessions_v1). Shares Redis
     // with the runs replicator for leader locking but has its own slot and
@@ -1763,6 +1741,12 @@ const EnvironmentSchema = z
     SESSION_REPLICATION_PUBLICATION_NAME: z
       .string()
       .default("sessions_to_clickhouse_v1_publication"),
+    // Connection URL for the sessions-replication slot. Direct, not pooled: replication can't run over a
+    // pooler. Optional; unset -> falls back to DATABASE_URL, so nothing changes today.
+    SESSION_REPLICATION_DATABASE_URL: z
+      .string()
+      .refine(isValidDatabaseUrl, "SESSION_REPLICATION_DATABASE_URL is invalid")
+      .optional(),
     SESSION_REPLICATION_MAX_FLUSH_CONCURRENCY: z.coerce.number().int().default(1),
     SESSION_REPLICATION_FLUSH_INTERVAL_MS: z.coerce.number().int().default(1000),
     SESSION_REPLICATION_FLUSH_BATCH_SIZE: z.coerce.number().int().default(100),
@@ -1788,6 +1772,11 @@ const EnvironmentSchema = z
 
     // Clickhouse
     CLICKHOUSE_URL: z.string(),
+    // Optional read replica endpoint. Read-only clients (logs, query, admin, runsList,
+    // engine, realtime) default to this when their own URL is unset; writes always stay on
+    // CLICKHOUSE_URL. Events reads opt in separately via EVENTS_READER_CLICKHOUSE_URL (no
+    // fallback here). Must share storage with the CLICKHOUSE_URL warehouse.
+    CLICKHOUSE_READER_URL: z.string().optional(),
     CLICKHOUSE_KEEP_ALIVE_ENABLED: z.string().default("1"),
     CLICKHOUSE_KEEP_ALIVE_IDLE_SOCKET_TTL_MS: z.coerce.number().int().optional(),
     CLICKHOUSE_MAX_OPEN_CONNECTIONS: z.coerce.number().int().default(10),
@@ -1850,13 +1839,13 @@ const EnvironmentSchema = z
     LOGS_CLICKHOUSE_URL: z
       .string()
       .optional()
-      .transform((v) => v ?? process.env.CLICKHOUSE_URL),
+      .transform((v) => v ?? process.env.CLICKHOUSE_READER_URL ?? process.env.CLICKHOUSE_URL),
 
     // Query page ClickHouse limits (for TSQL queries)
     QUERY_CLICKHOUSE_URL: z
       .string()
       .optional()
-      .transform((v) => v ?? process.env.CLICKHOUSE_URL),
+      .transform((v) => v ?? process.env.CLICKHOUSE_READER_URL ?? process.env.CLICKHOUSE_URL),
     QUERY_CLICKHOUSE_MAX_EXECUTION_TIME: z.coerce.number().int().default(10),
     QUERY_CLICKHOUSE_MAX_MEMORY_USAGE: z.coerce.number().int().default(1_073_741_824), // 1GB in bytes
     QUERY_CLICKHOUSE_MAX_AST_ELEMENTS: z.coerce.number().int().default(4_000_000),
@@ -1875,12 +1864,14 @@ const EnvironmentSchema = z
     ADMIN_CLICKHOUSE_URL: z
       .string()
       .optional()
-      .transform((v) => v ?? process.env.CLICKHOUSE_URL),
+      .transform((v) => v ?? process.env.CLICKHOUSE_READER_URL ?? process.env.CLICKHOUSE_URL),
 
     EVENTS_CLICKHOUSE_URL: z
       .string()
       .optional()
       .transform((v) => v ?? process.env.CLICKHOUSE_URL),
+    // Events read replica (traces/spans/logs). No CLICKHOUSE_READER_URL fallback by design: this write-capable client opts in explicitly.
+    EVENTS_READER_CLICKHOUSE_URL: z.string().optional(),
     EVENTS_CLICKHOUSE_KEEP_ALIVE_ENABLED: z.string().default("1"),
     EVENTS_CLICKHOUSE_KEEP_ALIVE_IDLE_SOCKET_TTL_MS: z.coerce.number().int().optional(),
     EVENTS_CLICKHOUSE_MAX_OPEN_CONNECTIONS: z.coerce.number().int().default(10),
@@ -1893,7 +1884,7 @@ const EnvironmentSchema = z
     RUN_ENGINE_CLICKHOUSE_URL: z
       .string()
       .optional()
-      .transform((v) => v ?? process.env.CLICKHOUSE_URL),
+      .transform((v) => v ?? process.env.CLICKHOUSE_READER_URL ?? process.env.CLICKHOUSE_URL),
     RUN_ENGINE_CLICKHOUSE_KEEP_ALIVE_ENABLED: z.string().default("1"),
     RUN_ENGINE_CLICKHOUSE_KEEP_ALIVE_IDLE_SOCKET_TTL_MS: z.coerce.number().int().optional(),
     RUN_ENGINE_CLICKHOUSE_MAX_OPEN_CONNECTIONS: z.coerce.number().int().default(5),
@@ -1905,7 +1896,7 @@ const EnvironmentSchema = z
     REALTIME_BACKEND_NATIVE_CLICKHOUSE_URL: z
       .string()
       .optional()
-      .transform((v) => v ?? process.env.CLICKHOUSE_URL),
+      .transform((v) => v ?? process.env.CLICKHOUSE_READER_URL ?? process.env.CLICKHOUSE_URL),
     REALTIME_BACKEND_NATIVE_CLICKHOUSE_KEEP_ALIVE_ENABLED: z.string().default("1"),
     REALTIME_BACKEND_NATIVE_CLICKHOUSE_KEEP_ALIVE_IDLE_SOCKET_TTL_MS: z.coerce
       .number()
@@ -1916,6 +1907,20 @@ const EnvironmentSchema = z
       .enum(["log", "error", "warn", "info", "debug"])
       .default("info"),
     REALTIME_BACKEND_NATIVE_CLICKHOUSE_COMPRESSION_REQUEST: z.string().default("1"),
+    // Dedicated ClickHouse pool for the runs list (dashboard + API). Lets us point
+    // the highest-traffic read path at a read replica without moving ingest/replication
+    // writes off CLICKHOUSE_URL. Falls back to CLICKHOUSE_URL when unset.
+    RUNS_LIST_CLICKHOUSE_URL: z
+      .string()
+      .optional()
+      .transform((v) => v ?? process.env.CLICKHOUSE_READER_URL ?? process.env.CLICKHOUSE_URL),
+    RUNS_LIST_CLICKHOUSE_KEEP_ALIVE_ENABLED: z.string().default("1"),
+    RUNS_LIST_CLICKHOUSE_KEEP_ALIVE_IDLE_SOCKET_TTL_MS: z.coerce.number().int().optional(),
+    RUNS_LIST_CLICKHOUSE_MAX_OPEN_CONNECTIONS: z.coerce.number().int().default(10),
+    RUNS_LIST_CLICKHOUSE_LOG_LEVEL: z
+      .enum(["log", "error", "warn", "info", "debug"])
+      .default("info"),
+    RUNS_LIST_CLICKHOUSE_COMPRESSION_REQUEST: z.string().default("1"),
     EVENTS_CLICKHOUSE_BATCH_SIZE: z.coerce.number().int().default(1000),
     EVENTS_CLICKHOUSE_FLUSH_INTERVAL_MS: z.coerce.number().int().default(1000),
     METRICS_CLICKHOUSE_BATCH_SIZE: z.coerce.number().int().default(10000),
@@ -1933,10 +1938,18 @@ const EnvironmentSchema = z
       .enum(["postgres", "clickhouse", "clickhouse_v2"])
       .default("postgres"),
     EVENT_REPOSITORY_DEBUG_LOGS_DISABLED: BoolEnv.default(false),
+    EVENT_REPOSITORY_POSTGRES_WRITES_DISABLED: BoolEnv.default(false),
     EVENTS_CLICKHOUSE_MAX_TRACE_SUMMARY_VIEW_COUNT: z.coerce.number().int().default(25_000),
     EVENTS_CLICKHOUSE_MAX_TRACE_DETAILED_SUMMARY_VIEW_COUNT: z.coerce.number().int().default(5_000),
     EVENTS_CLICKHOUSE_MAX_LIVE_RELOADING_SETTING: z.coerce.number().int().default(2000),
 
+    // OTLP ingest transform worker pool (opt-in). When enabled, decode/convert/enrich run in a
+    // worker_threads pool instead of the request event loop; the single consolidated insert path
+    // is unchanged.
+    OTEL_TRANSFORM_WORKER_POOL_ENABLED: BoolEnv.default(false),
+    OTEL_TRANSFORM_WORKER_POOL_SIZE: z.coerce.number().int().optional(),
+    OTEL_TRANSFORM_WORKER_PATH: z.string().optional(),
+
     // Organization data stores registry
     ORGANIZATION_DATA_STORES_RELOAD_INTERVAL_MS: z.coerce
       .number()
@@ -2072,9 +2085,22 @@ const EnvironmentSchema = z
     // Force RBAC to not use the plugin
     RBAC_FORCE_FALLBACK: BoolEnv.default(false),
 
+    // Per-process pool sizes for an RBAC plugin that owns its own database
+    // client (the fallback queries through Prisma and ignores these). Writes
+    // are rare role mutations; reads run on the per-request auth hot path.
+    RBAC_DATABASE_WRITER_CONNECTION_LIMIT: z.coerce.number().int().default(2),
+    RBAC_DATABASE_READER_CONNECTION_LIMIT: z.coerce.number().int().default(5),
+
     // Force SSO to not use the plugin (contributors without the cloud
     // plugin installed can opt in to a clean OSS-only experience).
     SSO_FORCE_FALLBACK: BoolEnv.default(false),
+
+    // Per-process pool sizes for an SSO plugin that owns its own database
+    // client (the fallback queries through Prisma and ignores these). Writes
+    // are rare config mutations and webhook processing; reads run on the
+    // login path.
+    SSO_DATABASE_WRITER_CONNECTION_LIMIT: z.coerce.number().int().default(2),
+    SSO_DATABASE_READER_CONNECTION_LIMIT: z.coerce.number().int().default(5),
     // Emit a console.log when the SSO fallback is selected because no
     // plugin is installed. Default off so OSS deployments stay quiet.
     SSO_LOG_FALLBACK: BoolEnv.default(false),
@@ -2115,3 +2141,24 @@ const EnvironmentSchema = z
 
 export type Environment = z.infer<typeof EnvironmentSchema>;
 export const env = EnvironmentSchema.parse(process.env);
+
+if (env.ALLOW_INSECURE_DEFAULT_SECRETS) {
+  const insecure = (
+    [
+      ["SESSION_SECRET", env.SESSION_SECRET],
+      ["MAGIC_LINK_SECRET", env.MAGIC_LINK_SECRET],
+      ["ENCRYPTION_KEY", env.ENCRYPTION_KEY],
+      ["MANAGED_WORKER_SECRET", env.MANAGED_WORKER_SECRET],
+    ] as const
+  )
+    .filter(([, value]) => INSECURE_SECRET_VALUES.includes(value))
+    .map(([name]) => name);
+
+  if (insecure.length > 0) {
+    console.warn(
+      `⚠️  ALLOW_INSECURE_DEFAULT_SECRETS is enabled and these secrets still use a known-insecure published default: ${insecure.join(
+        ", "
+      )}. This is insecure - rotate them as soon as you can.`
+    );
+  }
+}

apps/webapp/app/models/admin.server.ts

@@ -130,7 +130,6 @@ export async function adminGetOrganizations(userId: string, { page, search }: Se
       id: true,
       slug: true,
       title: true,
-      v2Enabled: true,
       isActivated: true,
       deletedAt: true,
       members: {