Merge remote-tracking branch 'origin/main' into fix/waitpoint-completion-db-error-mislabel

# Conflicts:
#	internal-packages/run-engine/src/engine/systems/waitpointSystem.ts

.changeset/experimental-node-runtimes.md

@@ -1,6 +0,0 @@
----
-"@trigger.dev/core": patch
-"trigger.dev": patch
----
-
-Add experimental Node.js 24 and 26 task runtimes. Set `runtime` to `experimental-node-24` or `experimental-node-26` in `trigger.config.ts`.

.changeset/project-default-region-response.md

@@ -1,5 +0,0 @@
----
-"@trigger.dev/core": patch
----
-
-Add `defaultRegion` to the project GET and list API responses; null when unset.

.claude/REVIEW.md

@@ -56,8 +56,8 @@ Every PR gets a performance pass — not just the ones that look perf-sensitive.
 ## Always check
 
 - **Tests use testcontainers, not mocks.** Vitest with `redisTest` / `postgresTest` / `containerTest` from `@internal/testcontainers`. Any new `vi.mock(...)` on Redis, Postgres, BullMQ, or other infra is wrong here — 🔴 if added in production-path tests, 🟡 if isolated unit test.
-- **Public-package changes have a changeset.** `pnpm run changeset:add` produces `.changeset/*.md`. Required for any edit under `packages/*`. Missing → 🟡; missing on a breaking change → 🔴.
-- **Server-only changes have `.server-changes/*.md`.** Required for `apps/webapp/`, `apps/supervisor/` edits with no public-package change. Body should be 1-2 sentences (it has to fit as one bullet in a future changelog). Missing → 🟡.
+- **User-facing public-package changes have a changeset.** `pnpm run changeset:add` produces `.changeset/*.md`. Changesets are user-facing release notes, not a catalog of every change: required when a `packages/*` or `integrations/*` change is something a user would notice or act on, skipped for internal-only changes, refactors, chores, and packages not consumed independently (e.g. `@trigger.dev/redis-worker`). Missing on a user-facing change → 🟡; missing on a breaking change → 🔴. Do not flag a missing note when the change is not user-facing.
+- **User-facing server-only changes have `.server-changes/*.md`.** Required for user-facing `apps/webapp/`, `apps/supervisor/` edits in a PR with no package or integration change that requires a changeset; skip internal-only or admin-only changes, refactors, and chores. Body should be 1-2 sentences (it has to fit as one bullet in a future changelog). Missing on a user-facing change → 🟡.
 - **Lua script naming.** Coexisting scripts use behavior-descriptive suffixes (`Tracked`), never `V2`. Old name must keep working until the next deploy clears it.
 - **RunQueue payload shape.** V2 run-queue payload's `projectId` is consumed by `workerQueueResolver` for override matching. If a PR drops it from the payload, 🔴.
 - **`safeSend` scope.** Defensive IPC wrappers belong on loop / interval / handler contexts, not one-shot terminal sends. If the PR adds `safeSend` to a single terminal call for consistency, 🟡 with a "remove this" suggestion.

.claude/agents/code-reviewer.md

@@ -0,0 +1,13 @@
+---
+name: code-reviewer
+description: Adversarially verifies one landed packet against its requirement; read-only.
+model: opus
+---
+
+You are an adversarial code reviewer for one landed packet. READ-ONLY: never modify code, never commit, never push, never post to GitHub.
+
+- Try to refute that the change answers its stated requirement; look for the failure scenario, not confirmation.
+- Check the diff for unrelated drift, dead code, broken semantics of neighbors, and whether tests prove the actual invariant (would the test fail if the fix were subtly wrong?).
+- Check the change landed in the correct PR/branch of the stack.
+- Distinguish fact from inference; cite exact file:line evidence.
+- Return: verdict (approve / needs-changes) with evidence per concern, and the exact minimal correction when needs-changes.

.claude/agents/code-writer.md

@@ -0,0 +1,16 @@
+---
+name: code-writer
+description: Implements exactly one work packet — minimal diff, targeted checks, own-paths-only commits.
+model: opus
+---
+
+You are a code writer. Implement exactly the one work packet in your prompt.
+
+- Minimal diff; match surrounding style and idiom.
+- Prefer no comment at all; comment only a non-obvious constraint, max 2 short lines. All texts (comments, commit messages) short, clear, simple.
+- Verify the packet's own diagnosis against the code before applying; if it is wrong, STOP without committing and report why.
+- Run only the targeted checks for your packet: the relevant vitest files, `pnpm run typecheck --filter <pkg>` when the change warrants it. Never full suites unless asked.
+- `pnpm run format` on touched files before committing.
+- Stage and commit ONLY your packet's files. Conventional commit message. NO Claude attribution, no Co-Authored-By.
+- Push only if the packet explicitly says to.
+- Return: what changed, evidence (test output), commit SHA, and anything contradicting the diagnosis.

.claude/agents/software-architect.md

@@ -0,0 +1,12 @@
+---
+name: software-architect
+description: Resolves contested design questions against the specs; decision + rationale, never code.
+model: opus
+---
+
+You are a software architect. Resolve exactly the contested design question in your prompt against the given specs/contracts. READ-ONLY.
+
+- Ground the decision in the actual code and the project's design contracts (GUIDEBOOK, Linear specs) — not in generic best practice.
+- Weigh stack boundaries: which PR owns the change, what merges independently.
+- Prefer the smallest decision that unblocks the packet; flag speculative architecture rather than endorsing it.
+- Return: the decision, its rationale, rejected alternatives (one line each), and exactly what the dependent packet should do.

.claude/rules/server-apps.md

@@ -5,7 +5,7 @@ paths:
 
 # Server App Changes
 
-When modifying server apps (webapp, supervisor, etc.) with **no package changes**, add a `.server-changes/` file instead of a changeset:
+`.server-changes/` files are user-facing release notes, not a catalog of every change. When a user-facing server app change (webapp, supervisor, etc.) is in a PR with **no package or integration change that requires a changeset**, add a `.server-changes/` file instead of a changeset. Skip it for internal-only or admin-only changes, refactors, and chores:
 
 ```bash
 cat > .server-changes/descriptive-name.md << 'EOF'
@@ -20,6 +20,6 @@ EOF
 
 - **area**: `webapp` | `supervisor`
 - **type**: `feature` | `fix` | `improvement` | `breaking`
-- If the PR also touches `packages/`, just the changeset is sufficient (no `.server-changes/` needed).
+- If the PR also touches `packages/` or `integrations/` and that change needs a changeset, the changeset covers it (no `.server-changes/` needed). If the package or integration change is internal and needs no changeset, still add a `.server-changes/` file for the user-facing server change.
 
 The body ships **verbatim in user-facing release notes**. Keep it to 1–2 short sentences, non-technical, written for a dashboard user: describe what changed for them, never the implementation (no header names, endpoints, middleware, storage mechanisms, internal tools). See `.server-changes/README.md` for full guidance.

.claude/skills/errors-api-e2e/SKILL.md

@@ -154,7 +154,7 @@ PASS: one run, `run_<RID>` (status maps to `FAILED`). Proves `filter[error]` ->
 ### 6. Attribution — `mint-token` -> JWT exchange records the acting user
 
 ```bash
-TOKEN=$(cli mint-token --profile $PROFILE --client errors-api-e2e 2>/dev/null)            # UAT
+TOKEN=$(cli mint-token --profile $PROFILE --client errors-api-e2e --cap read:errors,write:errors 2>/dev/null)            # UAT
 ENVJWT=$(curl -sS -X POST "$B/api/v1/projects/$REF/dev/jwt" -H "Authorization: Bearer $TOKEN" \
   -H 'Content-Type: application/json' -d '{"claims":{"scopes":["read:errors","write:errors"]}}' \
   | python3 -c "import sys,json;print(json.load(sys.stdin)['token'])")

.configs/tsconfig.base.json

@@ -26,7 +26,6 @@
     "esModuleInterop": true,
     "emitDecoratorMetadata": false,
     "experimentalDecorators": false,
-    "downlevelIteration": true,
     "isolatedModules": true,
     "noUncheckedIndexedAccess": true,
 

.env.example

@@ -2,6 +2,7 @@
 SESSION_SECRET=abcdef1234
 MAGIC_LINK_SECRET=abcdef1234
 ENCRYPTION_KEY=ae13021afef0819c3a307ad487071c06 # Must be a random 16 byte hex string. You can generate an encryption key by running `openssl rand -hex 16` in your terminal
+MANAGED_WORKER_SECRET=abcdef1234 # Must match the supervisor's MANAGED_WORKER_SECRET
 LOGIN_ORIGIN=http://localhost:3030
 DATABASE_URL=postgresql://postgres:postgres@localhost:5432/postgres?schema=public
 # This sets the URL used for direct connections to the database and should only be needed in limited circumstances
@@ -22,6 +23,8 @@ NODE_ENV=development
 CLICKHOUSE_URL=http://default:password@localhost:8123
 RUN_REPLICATION_CLICKHOUSE_URL=http://default:password@localhost:8123
 RUN_REPLICATION_ENABLED=1
+# LOGS_SEARCH_PROJECTOR_ENABLED=1
+# LOGS_SEARCH_PROJECTOR_PREVIEW_ENABLED=1
 # Store task run spans/traces in ClickHouse so the dashboard trace view is
 # populated in local dev. The local stack is ClickHouse-backed (see above), so
 # leaving this unset falls back to the "postgres" store and dev run traces show

.gitattributes

@@ -0,0 +1,5 @@
+# Generated, not hand-written: collapsed in diffs and excluded from language stats.
+internal-packages/dashboard-agent-db/drizzle/meta/*.json linguist-generated=true
+internal-packages/dashboard-agent-db/drizzle/meta/** linguist-generated=true
+**/__snapshots__/*.snap linguist-generated=true
+pnpm-lock.yaml linguist-generated=true

.github/VOUCHED.td

@@ -24,4 +24,10 @@ jrossi
 ThullyoCunha
 ConProgramming
 saasjesus
-brentshulman-silkline
+brentshulman-silkline
+Leafgard
+Rohan170603
+NERLOE
+Jakub-Vacek
+gtremper
+wuweiweiwu

.github/workflows/base-images.yml

@@ -0,0 +1,272 @@
+name: 🐳 Deploy base images
+
+# Publishes the deploy base images (see base-images/README.md) to Docker Hub.
+# Tags are mutable and rebuilt in place; the CLI pins digests, so consumers
+# only move when a release bumps its pins.
+
+on:
+  workflow_dispatch:
+    inputs:
+      debian_snapshot:
+        description: "Debian snapshot timestamp (YYYYMMDDTHHMMSSZ). Defaults to yesterday 00:00 UTC."
+        required: false
+        type: string
+  push:
+    branches: [main]
+    paths:
+      - "base-images/**"
+      - ".github/workflows/base-images.yml"
+  pull_request:
+    paths:
+      - "base-images/**"
+      - ".github/workflows/base-images.yml"
+
+concurrency:
+  group: base-images-${{ github.ref }}
+  cancel-in-progress: false
+
+permissions: {}
+
+jobs:
+  setup:
+    runs-on: ubuntu-latest
+    timeout-minutes: 10
+    permissions:
+      contents: read
+    outputs:
+      images: ${{ steps.config.outputs.images }}
+      packages: ${{ steps.config.outputs.packages }}
+      build_packages: ${{ steps.config.outputs.build_packages }}
+      suite: ${{ steps.config.outputs.suite }}
+      snapshot: ${{ steps.config.outputs.snapshot }}
+      publish_id: ${{ steps.config.outputs.publish_id }}
+      source_date_epoch: ${{ steps.config.outputs.source_date_epoch }}
+      push: ${{ steps.config.outputs.push }}
+    steps:
+      - uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
+        with:
+          persist-credentials: false
+
+      - name: Read image matrix and resolve snapshot
+        id: config
+        env:
+          SNAPSHOT_INPUT: ${{ inputs.debian_snapshot }}
+          EVENT_NAME: ${{ github.event_name }}
+          REF: ${{ github.ref }}
+          SHA: ${{ github.sha }}
+        run: |
+          PACKAGES="$(jq -er '.packages' base-images/images.json)"
+          BUILD_PACKAGES="$(jq -er '.buildPackages' base-images/images.json)"
+          SUITE="$(jq -er '.suite' base-images/images.json)"
+
+          # Values land in build args and shell lines; keep them boring.
+          # NUL-delimited whole-record match so multi-line values can't sneak through
+          printf '%s\0' "$PACKAGES" | grep -zqxE '[a-z0-9][a-z0-9 .+:=~-]*' || { echo "invalid packages value"; exit 1; }
+          printf '%s\0' "$BUILD_PACKAGES" | grep -zqxE '[a-z0-9][a-z0-9 .+:=~-]*' || { echo "invalid buildPackages value"; exit 1; }
+          printf '%s\0' "$SUITE" | grep -zqxE '[a-z]+' || { echo "invalid suite value"; exit 1; }
+          jq -e '.images | length > 0 and all((.repo | test("^[a-z0-9-]+$")) and (.tag | test("^[a-z0-9.-]+$")) and (.base | test("^[a-zA-Z0-9./:@-]+$")))' base-images/images.json > /dev/null \
+            || { echo "invalid images entries"; exit 1; }
+
+          SNAPSHOT="$SNAPSHOT_INPUT"
+          if [ -z "$SNAPSHOT" ]; then
+            SNAPSHOT="$(date -u -d yesterday +%Y%m%dT000000Z)"
+          fi
+          printf '%s\0' "$SNAPSHOT" | grep -zqxE '[0-9]{8}T[0-9]{6}Z' || { echo "invalid debian_snapshot: $SNAPSHOT"; exit 1; }
+
+          # Snapshot-derived timestamps: reproducible, with a real created date
+          EPOCH="$(date -u -d "${SNAPSHOT:0:4}-${SNAPSHOT:4:2}-${SNAPSHOT:6:2} ${SNAPSHOT:9:2}:${SNAPSHOT:11:2}:${SNAPSHOT:13:2}Z" +%s)"
+          # Future snapshots resolve to "latest" and break mtime normalization
+          [ "$EPOCH" -le "$(date -u +%s)" ] || { echo "debian_snapshot is in the future: $SNAPSHOT"; exit 1; }
+
+          # Pull requests and branch dispatches build without pushing
+          if [ "$EVENT_NAME" = "pull_request" ] || [ "$REF" != "refs/heads/main" ]; then
+            PUSH=false
+          else
+            PUSH=true
+          fi
+
+          {
+            echo "images=$(jq -c '.images' base-images/images.json)"
+            echo "packages=$PACKAGES"
+            echo "build_packages=$BUILD_PACKAGES"
+            echo "suite=$SUITE"
+            echo "snapshot=$SNAPSHOT"
+            echo "publish_id=${SNAPSHOT:0:8}-${SNAPSHOT:9:6}-${SHA:0:7}"
+            echo "source_date_epoch=$EPOCH"
+            echo "push=$PUSH"
+          } >> "$GITHUB_OUTPUT"
+
+  publish:
+    needs: setup
+    runs-on: ubuntu-latest
+    timeout-minutes: 60
+    permissions:
+      contents: read
+      id-token: write
+      attestations: write
+    strategy:
+      fail-fast: false
+      matrix:
+        image: ${{ fromJSON(needs.setup.outputs.images) }}
+    env:
+      DOCKERHUB_USERNAME: ${{ secrets.DOCKERHUB_USERNAME }}
+      DOCKER_BUILD_SUMMARY: "false"
+      DOCKER_BUILD_RECORD_UPLOAD: "false"
+    steps:
+      - uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
+        with:
+          persist-credentials: false
+
+      # Before any pull so rate limits are authenticated; fork PRs skip (no secrets)
+      - name: 🐳 Login to Docker Hub
+        if: env.DOCKERHUB_USERNAME != ''
+        uses: docker/login-action@650006c6eb7dba73a995cc03b0b2d7f5ca915bee # v4.2.0
+        with:
+          username: ${{ secrets.DOCKERHUB_USERNAME }}
+          password: ${{ secrets.DOCKERHUB_TOKEN }}
+
+      - name: 🐳 Set up QEMU
+        uses: docker/setup-qemu-action@96fe6ef7f33517b61c61be40b68a1882f3264fb8 # v4.2.0
+        with:
+          image: docker.io/tonistiigi/binfmt:latest@sha256:400a4873b838d1b89194d982c45e5fb3cda4593fbfd7e08a02e76b03b21166f0
+
+      - name: 🐳 Set up Docker Buildx
+        uses: docker/setup-buildx-action@bb05f3f5519dd87d3ba754cc423b652a5edd6d2c # v4.2.0
+
+      # Build both targets before pushing either so the tag pair can't skew
+      - name: 🐳 Build both targets (no push)
+        uses: docker/build-push-action@53b7df96c91f9c12dcc8a07bcb9ccacbed38856a # v7.3.0
+        with:
+          context: base-images
+          file: base-images/Dockerfile
+          target: build
+          platforms: linux/amd64,linux/arm64
+          provenance: false
+          outputs: type=image,push=false,rewrite-timestamp=true
+          tags: triggerdotdev/${{ matrix.image.repo }}:${{ matrix.image.tag }}-build
+          build-args: |
+            BASE_IMAGE=${{ matrix.image.base }}
+            DEBIAN_SNAPSHOT=${{ needs.setup.outputs.snapshot }}
+            DEBIAN_SUITE=${{ needs.setup.outputs.suite }}
+            PACKAGES=${{ needs.setup.outputs.packages }}
+            BUILD_PACKAGES=${{ needs.setup.outputs.build_packages }}
+            SOURCE_DATE_EPOCH=${{ needs.setup.outputs.source_date_epoch }}
+          labels: |
+            org.opencontainers.image.source=https://github.com/${{ github.repository }}
+            org.opencontainers.image.revision=${{ github.sha }}
+            dev.trigger.debian-snapshot=${{ needs.setup.outputs.snapshot }}
+
+      - name: 🐳 Push runtime image
+        id: build_runtime
+        if: needs.setup.outputs.push == 'true'
+        uses: docker/build-push-action@53b7df96c91f9c12dcc8a07bcb9ccacbed38856a # v7.3.0
+        with:
+          context: base-images
+          file: base-images/Dockerfile
+          target: runtime
+          platforms: linux/amd64,linux/arm64
+          provenance: false
+          outputs: type=image,push=true,rewrite-timestamp=true
+          # The dated tag is immutable and keeps every published digest
+          # tag-referenced forever; shipped CLI releases pin these digests
+          tags: |
+            triggerdotdev/${{ matrix.image.repo }}:${{ matrix.image.tag }}
+            triggerdotdev/${{ matrix.image.repo }}:${{ matrix.image.tag }}-${{ needs.setup.outputs.publish_id }}
+          build-args: |
+            BASE_IMAGE=${{ matrix.image.base }}
+            DEBIAN_SNAPSHOT=${{ needs.setup.outputs.snapshot }}
+            DEBIAN_SUITE=${{ needs.setup.outputs.suite }}
+            PACKAGES=${{ needs.setup.outputs.packages }}
+            SOURCE_DATE_EPOCH=${{ needs.setup.outputs.source_date_epoch }}
+          labels: |
+            org.opencontainers.image.source=https://github.com/${{ github.repository }}
+            org.opencontainers.image.revision=${{ github.sha }}
+            dev.trigger.debian-snapshot=${{ needs.setup.outputs.snapshot }}
+
+      - name: 🐳 Push build-variant image
+        id: build_toolchain
+        if: needs.setup.outputs.push == 'true'
+        uses: docker/build-push-action@53b7df96c91f9c12dcc8a07bcb9ccacbed38856a # v7.3.0
+        with:
+          context: base-images
+          file: base-images/Dockerfile
+          target: build
+          platforms: linux/amd64,linux/arm64
+          provenance: false
+          outputs: type=image,push=true,rewrite-timestamp=true
+          tags: |
+            triggerdotdev/${{ matrix.image.repo }}:${{ matrix.image.tag }}-build
+            triggerdotdev/${{ matrix.image.repo }}:${{ matrix.image.tag }}-build-${{ needs.setup.outputs.publish_id }}
+          build-args: |
+            BASE_IMAGE=${{ matrix.image.base }}
+            DEBIAN_SNAPSHOT=${{ needs.setup.outputs.snapshot }}
+            DEBIAN_SUITE=${{ needs.setup.outputs.suite }}
+            PACKAGES=${{ needs.setup.outputs.packages }}
+            BUILD_PACKAGES=${{ needs.setup.outputs.build_packages }}
+            SOURCE_DATE_EPOCH=${{ needs.setup.outputs.source_date_epoch }}
+          labels: |
+            org.opencontainers.image.source=https://github.com/${{ github.repository }}
+            org.opencontainers.image.revision=${{ github.sha }}
+            dev.trigger.debian-snapshot=${{ needs.setup.outputs.snapshot }}
+
+      # An auto-created private repo would publish green while customer pulls fail
+      - name: 🔎 Verify anonymous pullability
+        if: needs.setup.outputs.push == 'true'
+        env:
+          IMAGE_REPO: ${{ matrix.image.repo }}
+          RUNTIME_DIGEST: ${{ steps.build_runtime.outputs.digest }}
+          BUILD_DIGEST: ${{ steps.build_toolchain.outputs.digest }}
+        run: |
+          for digest in "$RUNTIME_DIGEST" "$BUILD_DIGEST"; do
+            TOKEN="$(curl -fsS --connect-timeout 10 --max-time 60 "https://auth.docker.io/token?service=registry.docker.io&scope=repository:triggerdotdev/$IMAGE_REPO:pull" | jq -r .token)"
+            curl -fsS --connect-timeout 10 --max-time 60 -o /dev/null               -H "Authorization: Bearer $TOKEN"               -H "Accept: application/vnd.oci.image.index.v1+json, application/vnd.docker.distribution.manifest.list.v2+json, application/vnd.oci.image.manifest.v1+json, application/vnd.docker.distribution.manifest.v2+json"               "https://registry-1.docker.io/v2/triggerdotdev/$IMAGE_REPO/manifests/$digest"               || { echo "triggerdotdev/$IMAGE_REPO@$digest is not anonymously pullable; is the repo private?"; exit 1; }
+          done
+
+      # Builds are reproducible, so re-running a red publish re-pushes the
+      # same digests and re-attests them
+      - name: 🔏 Attest runtime image provenance
+        if: needs.setup.outputs.push == 'true'
+        uses: actions/attest-build-provenance@a2bbfa25375fe432b6a289bc6b6cd05ecd0c4c32 # v4.1.0
+        with:
+          subject-name: index.docker.io/triggerdotdev/${{ matrix.image.repo }}
+          subject-digest: ${{ steps.build_runtime.outputs.digest }}
+          push-to-registry: false
+
+      - name: 🔏 Attest build-variant image provenance
+        if: needs.setup.outputs.push == 'true'
+        uses: actions/attest-build-provenance@a2bbfa25375fe432b6a289bc6b6cd05ecd0c4c32 # v4.1.0
+        with:
+          subject-name: index.docker.io/triggerdotdev/${{ matrix.image.repo }}
+          subject-digest: ${{ steps.build_toolchain.outputs.digest }}
+          push-to-registry: false
+
+      - name: 📋 Record digests
+        if: needs.setup.outputs.push == 'true'
+        env:
+          IMAGE_REPO: ${{ matrix.image.repo }}
+          IMAGE_TAG: ${{ matrix.image.tag }}
+          RUNTIME_DIGEST: ${{ steps.build_runtime.outputs.digest }}
+          BUILD_DIGEST: ${{ steps.build_toolchain.outputs.digest }}
+          SNAPSHOT: ${{ needs.setup.outputs.snapshot }}
+        run: |
+          {
+            echo "### triggerdotdev/$IMAGE_REPO:$IMAGE_TAG"
+            echo '```'
+            echo "runtime: $RUNTIME_DIGEST"
+            echo "build:   $BUILD_DIGEST"
+            echo "debian snapshot: $SNAPSHOT"
+            echo '```'
+          } >> "$GITHUB_STEP_SUMMARY"
+
+  results:
+    needs: [publish]
+    if: always()
+    runs-on: ubuntu-latest
+    timeout-minutes: 5
+    permissions: {}
+    steps:
+      - name: Fail if any image build failed
+        env:
+          RESULT: ${{ needs.publish.result }}
+        run: |
+          [ "$RESULT" = "success" ] || { echo "one or more image builds failed: $RESULT"; exit 1; }

.github/workflows/check-review-md.yml

@@ -38,7 +38,7 @@ jobs:
         with:
           anthropic_api_key: ${{ secrets.ANTHROPIC_API_KEY }}
           use_sticky_comment: true
-          allowed_bots: "devin-ai-integration[bot]"
+          allowed_bots: "devin-ai-integration[bot],claude[bot]"
 
           claude_args: |
             --max-turns 30

.github/workflows/claude-md-audit.yml

@@ -40,7 +40,7 @@ jobs:
         with:
           anthropic_api_key: ${{ secrets.ANTHROPIC_API_KEY }}
           use_sticky_comment: true
-          allowed_bots: "devin-ai-integration[bot]"
+          allowed_bots: "devin-ai-integration[bot],claude[bot]"
 
           claude_args: |
             --max-turns 25

.github/workflows/code-quality.yml

@@ -36,3 +36,6 @@ jobs:
 
       - name: 🔎 Lint
         run: pnpm exec oxlint .
+
+      - name: ✂️ Check unused code and dependencies
+        run: pnpm run knip

.github/workflows/dashboard-agent-deploy.yml

@@ -3,9 +3,19 @@ name: "🤖 Deploy dashboard agent"
 # Deploys the @internal/dashboard-agent chat.agent to its Trigger.dev project
 # with --skip-promotion, so a deploy never becomes "current" on its own. The
 # consuming app cuts over by pinning DASHBOARD_AGENT_VERSION to the new version.
-# Runs a leg per environment (staging + prod), each gated by its own environment;
-# a push to main that touches the agent or its store triggers both. Version
-# numbers are per-environment, so pin each environment to its own leg's version.
+# Runs a leg per environment (staging + prod); a push to main that touches the
+# agent or its store deploys both. Version numbers are per-environment, so pin
+# each environment to its own leg's version.
+#
+# The deploy lands dormant, so it doesn't need a reviewer gate: nothing goes live
+# until DASHBOARD_AGENT_VERSION is flipped. The `environment:` below is kept only
+# to scope the deploy token per environment; its required-reviewers rule is
+# removed in repo settings so pushes deploy unattended. workflow_dispatch takes an
+# optional ref (SHA, branch, or tag) to deploy a specific commit instead of head.
+#
+# The deployed ref must be an ancestor of main, so only reviewed, merged code ever
+# runs with the deploy token (the checked-out build + trigger.config.ts execute
+# with it). A push is always on main; a dispatched ref is checked before deploy.
 
 on:
   push:
@@ -14,6 +24,11 @@ on:
       - "internal-packages/dashboard-agent/**"
       - "internal-packages/dashboard-agent-db/**"
   workflow_dispatch:
+    inputs:
+      ref:
+        description: "Commit SHA, branch, or tag to deploy. Defaults to the ref the workflow runs from."
+        required: false
+        type: string
 
 permissions: {}
 
@@ -27,9 +42,15 @@ jobs:
       max-parallel: 1
       matrix:
         environment: [staging, prod]
-    # Per-environment reviewer gate + source of the scoped deploy PAT.
+    # Kept to scope the deploy token per environment. The required-reviewers rule
+    # on these environments is removed in repo settings, so this no longer gates.
     environment: dashboard-agent-${{ matrix.environment }}
     concurrency:
+      # Queue a superseding deploy behind an in-flight one; do NOT cancel it.
+      # Cancelling the runner wouldn't stop the remote build (it finishes
+      # server-side), and a second concurrent deploy of the same project would
+      # race the indexer. Deploys are short now the gate is gone, so a brief queue
+      # is fine and can't pile up.
       group: dashboard-agent-deploy-${{ matrix.environment }}
       cancel-in-progress: false
     permissions:
@@ -41,8 +62,34 @@ jobs:
       - name: Checkout
         uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
         with:
+          # push: the pushed commit. workflow_dispatch: the input ref if given,
+          # otherwise the head of the ref the run was launched from.
+          ref: ${{ github.event.inputs.ref || github.sha }}
+          # Full history so the ancestor-of-main check below can find a merge base.
+          fetch-depth: 0
           persist-credentials: false
 
+      - name: Require the ref to be an ancestor of main
+        # The deploy token runs the checked-out code, so refuse anything that
+        # hasn't landed on main. A push is main's tip (ancestor of itself); this
+        # only ever rejects a dispatched, unmerged ref.
+        #
+        # NOTE: this in-file check only constrains WHICH commit is deployed. It
+        # can't protect the token on its own, because workflow_dispatch runs the
+        # workflow file from the selected ref. The real guard is the deployment
+        # branch policy on the dashboard-agent-* environments (main only), set in
+        # repo settings, which GitHub enforces server-side against GITHUB_REF.
+        run: |
+          set -euo pipefail
+          # An explicit `ref:` checkout doesn't create remote-tracking branches,
+          # so fetch main before comparing against it.
+          git fetch --no-tags --quiet origin +refs/heads/main:refs/remotes/origin/main
+          if ! git merge-base --is-ancestor HEAD origin/main; then
+            echo "::error::Refusing to deploy $(git rev-parse HEAD): not an ancestor of origin/main. Only merged code can be deployed."
+            exit 1
+          fi
+          echo "$(git rev-parse --short HEAD) is an ancestor of origin/main"
+
       - name: Setup pnpm
         uses: pnpm/action-setup@fc06bc1257f339d1d5d8b3a19a8cae5388b55320 # v5.0.0
         with:

.github/workflows/dependabot-critical-alerts.yml

@@ -1,87 +0,0 @@
-name: Dependabot Critical Alerts
-
-on:
-  schedule:
-    - cron: "0 8 * * *" # Daily 08:00 UTC
-  workflow_dispatch:
-    inputs:
-      severity:
-        description: "Severity to alert on"
-        type: choice
-        options:
-          - critical
-          - high
-          - medium
-          - low
-        default: critical
-
-concurrency:
-  group: ${{ github.workflow }}
-  cancel-in-progress: false
-
-permissions:
-  contents: read
-
-jobs:
-  alert:
-    name: Post critical alerts
-    # Set the ENABLE_DEPENDABOT_ALERTS repository variable to 'false' to turn off
-    # the Dependabot alert/summary notifiers — e.g. forks/mirrors that lack the
-    # DEPENDABOT_ALERTS_TOKEN / SLACK_BOT_TOKEN secrets. Defaults to enabled.
-    if: ${{ vars.ENABLE_DEPENDABOT_ALERTS != 'false' }}
-    runs-on: warp-ubuntu-latest-x64-2x
-    environment: dependabot-summary
-    env:
-      SEVERITY: ${{ inputs.severity || 'critical' }}
-    steps:
-      - name: Fetch alerts
-        id: alerts
-        env:
-          GH_TOKEN: ${{ secrets.DEPENDABOT_ALERTS_TOKEN }}
-          REPO: ${{ github.repository }}
-        run: |
-          set -euo pipefail
-          gh api -X GET "/repos/$REPO/dependabot/alerts" \
-            -F state=open -F severity="$SEVERITY" --paginate > pages.json
-          jq -s 'add' pages.json > alerts.json
-          TOTAL=$(jq 'length' alerts.json)
-          echo "total=$TOTAL" >> "$GITHUB_OUTPUT"
-          if [ "$TOTAL" = "0" ]; then
-            exit 0
-          fi
-          LIST=$(jq -r '
-            map("• <\(.html_url)|#\(.number)> *\(.dependency.package.name)* - \(.security_advisory.summary)")
-            | join("\n")
-          ' alerts.json)
-          {
-            echo "list<<EOF"
-            echo "$LIST"
-            echo "EOF"
-          } >> "$GITHUB_OUTPUT"
-
-      - name: Build Slack payload
-        if: steps.alerts.outputs.total != '0'
-        env:
-          REPO: ${{ github.repository }}
-          CHANNEL: ${{ vars.SLACK_CHANNEL_ID }}
-          TOTAL: ${{ steps.alerts.outputs.total }}
-          LIST: ${{ steps.alerts.outputs.list }}
-        run: |
-          jq -n \
-            --arg channel "$CHANNEL" \
-            --arg repo "$REPO" \
-            --arg total "$TOTAL" \
-            --arg list "$LIST" \
-            --arg severity "$SEVERITY" \
-            '{
-              channel: $channel,
-              text: ":bufo-alarma: `\($repo)` - *\($total) open \($severity) alert(s)*\n\($list)\n\n<https://github.com/\($repo)/security/dependabot?q=is%3Aopen+severity%3A\($severity)|View \($severity) alerts>"
-            }' > payload.json
-
-      - name: Post Slack alert
-        if: steps.alerts.outputs.total != '0'
-        uses: slackapi/slack-github-action@45a88b9581bfab2566dc881e2cd66d334e621e2c # v3.0.3
-        with:
-          method: chat.postMessage
-          token: ${{ secrets.SLACK_BOT_TOKEN }}
-          payload-file-path: payload.json

.github/workflows/dependabot-weekly-summary.yml

@@ -1,210 +0,0 @@
-name: Dependabot Weekly Summary
-
-on:
-  schedule:
-    - cron: "0 8 * * 1" # Mon 08:00 UTC
-  workflow_dispatch:
-
-# Single-purpose monitoring workflow; serialise on workflow name only - we never
-# want two concurrent summary runs racing to post the same digest.
-concurrency:
-  group: ${{ github.workflow }}
-  cancel-in-progress: false
-
-permissions:
-  contents: read # gh CLI baseline
-  pull-requests: read # gh pr list (open dependabot PRs)
-  actions: read # gh run list / view (parse latest dependabot run logs)
-
-jobs:
-  summary:
-    name: Post weekly Dependabot summary
-    # Set the ENABLE_DEPENDABOT_ALERTS repository variable to 'false' to turn off
-    # the Dependabot alert/summary notifiers — e.g. forks/mirrors that lack the
-    # DEPENDABOT_ALERTS_TOKEN / SLACK_BOT_TOKEN secrets. Defaults to enabled.
-    if: ${{ vars.ENABLE_DEPENDABOT_ALERTS != 'false' }}
-    runs-on: warp-ubuntu-latest-x64-2x
-    environment: dependabot-summary
-    env:
-      # Severities surface in the actions list when their remaining TTR drops
-      # below this many days. Override via repo/env var ACTION_THRESHOLD_DAYS.
-      THRESHOLD_DAYS: ${{ vars.ACTION_THRESHOLD_DAYS || '7' }}
-    steps:
-      - name: Fetch alerts and compute summaries
-        id: alerts
-        env:
-          GH_TOKEN: ${{ secrets.DEPENDABOT_ALERTS_TOKEN }}
-          REPO: ${{ github.repository }}
-        run: |
-          if ! gh api -X GET "/repos/$REPO/dependabot/alerts" --paginate > pages.json 2> err.txt; then
-            echo "total=?" >> "$GITHUB_OUTPUT"
-            ERR=$(head -c 200 err.txt | tr '\n' ' ')
-            echo "by_severity=:x: _failed to fetch alerts: ${ERR}_" >> "$GITHUB_OUTPUT"
-            echo "actions=:x: _alerts unavailable_" >> "$GITHUB_OUTPUT"
-            exit 0
-          fi
-          jq -s '[.[][] | select(.state == "open")]' pages.json > open.json
-
-          TOTAL=$(jq 'length' open.json)
-          echo "total=$TOTAL" >> "$GITHUB_OUTPUT"
-
-          if [ "$TOTAL" = "0" ]; then
-            echo "by_severity=:white_check_mark: No open alerts." >> "$GITHUB_OUTPUT"
-            echo "actions=_None_" >> "$GITHUB_OUTPUT"
-            exit 0
-          fi
-
-          # Severity breakdown - real newlines so jq --arg in the payload
-          # builder encodes them as proper \n in JSON (Slack renders as breaks).
-          BY_SEV=$(jq -r '
-            group_by(.security_advisory.severity)
-            | map({sev: .[0].security_advisory.severity,
-                   count: length,
-                   weight: ({"critical":0,"high":1,"medium":2,"low":3}[.[0].security_advisory.severity])})
-            | sort_by(.weight)
-            | map("• *\(.count)* \(.sev)")
-            | join("\n")
-          ' open.json)
-          {
-            echo "by_severity<<EOF"
-            echo "$BY_SEV"
-            echo "EOF"
-          } >> "$GITHUB_OUTPUT"
-
-          # Actions: alerts within THRESHOLD_DAYS of their TTR (P0=7d, P1=30d, P2=90d, P3=no deadline)
-          # Grouped by (package, severity); shows earliest deadline per group.
-          ACTIONS=$(jq -r --argjson threshold "$THRESHOLD_DAYS" '
-            [.[]
-              | (.security_advisory.severity) as $sev
-              | ({"critical":7,"high":30,"medium":90,"low":null}[$sev]) as $ttr
-              | select($ttr != null)
-              | ((now - (.created_at | fromdateiso8601)) / 86400 | floor) as $age
-              | {pkg: .dependency.package.name, sev: $sev, remaining: ($ttr - $age)}
-            ]
-            | group_by([.pkg, .sev])
-            | map({pkg: .[0].pkg, sev: .[0].sev, count: length, min_remaining: ([.[].remaining] | min)})
-            | map(select(.min_remaining < $threshold))
-            | sort_by(.min_remaining)
-            | if length == 0 then "_None_"
-              else (map(
-                "• *\(.pkg)* (\(.sev))" +
-                (if .count > 1 then " ×\(.count)" else "" end) + " - " +
-                (if .min_remaining < 0 then "*OVERDUE* by \(-.min_remaining)d"
-                 else "\(.min_remaining)d remaining" end)
-              ) | join("\n"))
-              end
-          ' open.json)
-          {
-            echo "actions<<EOF"
-            echo "$ACTIONS"
-            echo "EOF"
-          } >> "$GITHUB_OUTPUT"
-
-      - name: Fetch open dependabot PRs
-        id: prs
-        env:
-          GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
-          REPO: ${{ github.repository }}
-          REPO_URL: https://github.com/${{ github.repository }}
-        run: |
-          if ! PR_JSON=$(gh pr list --repo "$REPO" --state open --author "app/dependabot" --json number,title 2> err.txt); then
-            ERR=$(head -c 200 err.txt | tr '\n' ' ')
-            echo "list=:x: _failed to fetch PRs: ${ERR}_" >> "$GITHUB_OUTPUT"
-            exit 0
-          fi
-          LIST=$(echo "$PR_JSON" | jq -r --arg url "$REPO_URL" '
-            if length == 0 then "_None_"
-            else (map("• <\($url)/pull/\(.number)|#\(.number)> \(.title)") | join("\n"))
-            end
-          ')
-          {
-            echo "list<<EOF"
-            echo "$LIST"
-            echo "EOF"
-          } >> "$GITHUB_OUTPUT"
-
-      - name: Find latest npm dependabot run
-        id: latest
-        env:
-          GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
-          REPO: ${{ github.repository }}
-        run: |
-          # Repos without a dependabot.yml have no "Dependabot Updates" workflow;
-          # treat the lookup failure as "no recent run found" rather than failing.
-          if ! RUN_ID=$(gh run list --repo "$REPO" --workflow "Dependabot Updates" --status success --limit 30 --json databaseId,name --jq 'first(.[] | select(.name | startswith("npm_and_yarn")) | .databaseId) // empty' 2>/dev/null); then
-            RUN_ID=""
-          fi
-          echo "run_id=$RUN_ID" >> "$GITHUB_OUTPUT"
-
-      - name: Extract stuck deps (only if actions pending)
-        id: stuck
-        env:
-          GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
-          REPO: ${{ github.repository }}
-          RUN_ID: ${{ steps.latest.outputs.run_id }}
-          ACTIONS: ${{ steps.alerts.outputs.actions }}
-        run: |
-          # Skip the stuck section entirely when nothing in the actions list
-          # - keeps the digest tidy when there's nothing to actually act on.
-          if [ "$ACTIONS" = "_None_" ]; then
-            echo "section=" >> "$GITHUB_OUTPUT"
-            exit 0
-          fi
-          HEADER=$'\n\n*Couldn\'t auto-fix (need manual `pnpm.overrides`):*\n'
-          if [ -z "$RUN_ID" ]; then
-            {
-              echo "section<<EOF"
-              echo "${HEADER}_(no recent npm run found)_"
-              echo "EOF"
-            } >> "$GITHUB_OUTPUT"
-            exit 0
-          fi
-          gh run view "$RUN_ID" --repo "$REPO" --log > log.txt 2>&1 || true
-          STUCK=$(grep -oE "No update possible for [^[:space:]]+ [0-9][^[:space:]]*" log.txt | sed 's/No update possible for //' | sort -u || true)
-          if [ -z "$STUCK" ]; then
-            {
-              echo "section<<EOF"
-              echo "${HEADER}_None_"
-              echo "EOF"
-            } >> "$GITHUB_OUTPUT"
-            exit 0
-          fi
-          LIST=$(echo "$STUCK" | awk 'NR>1{printf "\n"} {printf "• *%s* %s", $1, $2}')
-          {
-            echo "section<<EOF"
-            echo "${HEADER}${LIST}"
-            echo "EOF"
-          } >> "$GITHUB_OUTPUT"
-
-      - name: Build Slack payload
-        env:
-          REPO: ${{ github.repository }}
-          CHANNEL: ${{ vars.SLACK_CHANNEL_ID }}
-          TOTAL: ${{ steps.alerts.outputs.total }}
-          BY_SEVERITY: ${{ steps.alerts.outputs.by_severity }}
-          PRS_LIST: ${{ steps.prs.outputs.list }}
-          ACTIONS: ${{ steps.alerts.outputs.actions }}
-          STUCK: ${{ steps.stuck.outputs.section }}
-        run: |
-          # Build payload via jq so PR titles or error strings containing
-          # quotes/backslashes/newlines can't break the JSON.
-          jq -n \
-            --arg channel "$CHANNEL" \
-            --arg repo "$REPO" \
-            --arg total "$TOTAL" \
-            --arg by_severity "$BY_SEVERITY" \
-            --arg prs_list "$PRS_LIST" \
-            --arg actions "$ACTIONS" \
-            --arg stuck "$STUCK" \
-            --arg threshold "$THRESHOLD_DAYS" \
-            '{
-              channel: $channel,
-              text: ":calendar: *Weekly Dependabot summary* - `\($repo)`\n\n*Open alerts (\($total)):*\n\($by_severity)\n\n*Open Dependabot PRs:*\n\($prs_list)\n\n*Actions needed (<\($threshold)d remaining):*\n\($actions)\($stuck)\n\n<https://github.com/\($repo)/security/dependabot|Dependabot alerts>"
-            }' > payload.json
-
-      - name: Post Slack summary
-        uses: slackapi/slack-github-action@45a88b9581bfab2566dc881e2cd66d334e621e2c # v3.0.3
-        with:
-          method: chat.postMessage
-          token: ${{ secrets.SLACK_BOT_TOKEN }}
-          payload-file-path: payload.json

.github/workflows/e2e-webapp-auth-full.yml

@@ -99,11 +99,10 @@ jobs:
         run: echo "DockerHub login skipped because secrets are not available."
 
       - name: 🐳 Pre-pull testcontainer images
-        if: ${{ env.DOCKERHUB_USERNAME }}
         run: |
           docker pull postgres:14
           docker pull redis:7.2
-          docker pull testcontainers/ryuk:0.11.0
+          docker pull testcontainers/ryuk:0.14.0
 
       - name: 📥 Download deps
         run: pnpm install --frozen-lockfile

.github/workflows/e2e-webapp.yml

@@ -15,9 +15,16 @@ jobs:
   e2eTests:
     name: "🧪 E2E Tests: Webapp"
     runs-on: warp-ubuntu-latest-x64-16x
-    timeout-minutes: 20
+    timeout-minutes: 30
+    strategy:
+      fail-fast: false
+      matrix:
+        shardIndex: [1, 2]
+        shardTotal: [2]
     env:
       DOCKERHUB_USERNAME: ${{ secrets.DOCKERHUB_USERNAME }}
+      SHARD_INDEX: ${{ matrix.shardIndex }}
+      SHARD_TOTAL: ${{ matrix.shardTotal }}
     steps:
       - name: 🔧 Disable IPv6
         run: |
@@ -57,7 +64,7 @@ jobs:
           version: 10.33.2
 
       - name: ⎔ Setup node
-        uses: actions/setup-node@48b55a011bda9f5d6aeb4c2d9c7362e8dae4041e # v6.4.0
+        uses: WarpBuilds/setup-node@bc639b444d583175926b588962199c247d23e8d3 # v6
         with:
           node-version: 24.18.0
           cache: "pnpm"
@@ -73,17 +80,52 @@ jobs:
         if: ${{ !env.DOCKERHUB_USERNAME }}
         run: echo "DockerHub login skipped because secrets are not available."
 
-      - name: 🐳 Pre-pull testcontainer images
-        if: ${{ env.DOCKERHUB_USERNAME }}
+      - name: 📥 Prepare deps and testcontainer images
         run: |
-          echo "Pre-pulling Docker images with authenticated session..."
-          docker pull postgres:14
-          docker pull redis:7.2
-          docker pull testcontainers/ryuk:0.11.0
-          echo "Image pre-pull complete"
+          # Pull images concurrently with dependency installation. Retry each pull because
+          # registry timeouts are a recurring transient CI flake.
+          pull() {
+            for attempt in 1 2 3; do
+              docker pull "$1" && return 0
+              echo "::warning::docker pull $1 failed (attempt ${attempt}/3); retrying in 10s"
+              sleep 10
+            done
+            echo "::error::docker pull $1 failed after 3 attempts"
+            return 1
+          }
 
-      - name: 📥 Download deps
-        run: pnpm install --frozen-lockfile
+          pull_images() {
+            local pids=()
+            local failed=0
+            for image in \
+              postgres:14 \
+              redis:7.2 \
+              testcontainers/ryuk:0.14.0 \
+              ghcr.io/s2-streamstore/s2:0.40.0@sha256:b26249e2ede0949755f5af8028185dc2bcfc3aa2db21eb9610543d144eb6ee9d \
+              minio/minio:latest
+            do
+              pull "$image" &
+              pids+=("$!")
+            done
+            for pid in "${pids[@]}"; do
+              if ! wait "$pid"; then
+                failed=1
+              fi
+            done
+            return "$failed"
+          }
+
+          echo "Installing dependencies and pre-pulling Docker images..."
+          pull_images &
+          pull_pid=$!
+          install_status=0
+          pnpm install --frozen-lockfile || install_status=$?
+          pull_status=0
+          wait "$pull_pid" || pull_status=$?
+          if (( install_status != 0 || pull_status != 0 )); then
+            exit 1
+          fi
+          echo "Dependency install and image pre-pull complete"
 
       - name: 📀 Generate Prisma Client
         run: pnpm run generate
@@ -91,7 +133,10 @@ jobs:
       - name: 🏗️ Build Webapp
         run: pnpm run build --filter webapp
 
+      - name: 🎭 Install Playwright Chromium
+        run: cd apps/webapp && pnpm exec playwright install chromium
+
       - name: 🧪 Run Webapp E2E Tests
-        run: cd apps/webapp && pnpm exec vitest run --config vitest.e2e.config.ts --reporter=default
+        run: cd apps/webapp && pnpm exec vitest run --config vitest.e2e.config.ts --reporter=default --shard=${{ matrix.shardIndex }}/${{ matrix.shardTotal }}
         env:
           WEBAPP_TEST_VERBOSE: "1"

.github/workflows/fk-cascade-guard.yml

@@ -0,0 +1,35 @@
+name: "🛡️ FK Cascade Index Guard"
+
+on:
+  workflow_call:
+
+permissions:
+  contents: read
+
+jobs:
+  fk-cascade-guard:
+    runs-on: warp-ubuntu-latest-x64-16x
+
+    steps:
+      - name: ⬇️ Checkout repo
+        uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
+        with:
+          fetch-depth: 0
+          persist-credentials: false
+
+      - name: ⎔ Setup pnpm
+        uses: pnpm/action-setup@fc06bc1257f339d1d5d8b3a19a8cae5388b55320 # v5.0.0
+        with:
+          version: 10.33.2
+
+      - name: ⎔ Setup node
+        uses: actions/setup-node@48b55a011bda9f5d6aeb4c2d9c7362e8dae4041e # v6.4.0
+        with:
+          node-version: 24.18.0
+          cache: "pnpm"
+
+      - name: 📥 Download deps
+        run: pnpm install --frozen-lockfile
+
+      - name: 🛡️ FK cascade index guard
+        run: pnpm --filter webapp run guard:fk-cascade-index -- --check

.github/workflows/helm-prerelease.yml

@@ -52,12 +52,14 @@ jobs:
 
       - name: Lint Helm Chart
         run: |
-          helm lint ./hosting/k8s/helm/
+          helm lint ./hosting/k8s/helm/ \
+            --values ./hosting/k8s/helm/ci/lint-values.yaml
 
       - name: Render templates
         run: |
           helm template test-release ./hosting/k8s/helm/ \
             --values ./hosting/k8s/helm/values.yaml \
+            --values ./hosting/k8s/helm/ci/lint-values.yaml \
             --output-dir ./helm-output
 
       - name: Validate manifests

.github/workflows/observability-map.yml

@@ -0,0 +1,386 @@
+name: 🗺️ Observability Map
+
+on:
+  # No paths filter, deliberately. GitHub evaluates one per workflow, so a pull request whose diff
+  # stops matching does not start the workflow at all: the resolved state cannot fire and a comment
+  # from an earlier push stands for ever showing findings that are no longer in the diff. Verified on
+  # a throwaway pull request whose only route change was reverted, and the realistic case is worse
+  # than that empty diff, because a pull request touching a route and other files, whose author
+  # reverts the route change and keeps the rest, still has a non-empty diff that no longer matches.
+  # The gating moved into the jobs below instead, where it can read whether a comment exists.
+  pull_request:
+    types: [opened, synchronize, reopened]
+  # The corpus job below is gated to this package's own paths, so a scheduled run is what still
+  # scans the tree as it drifts. Nightly rather than per route pull request: a new route can make a
+  # known laundering shape start paying, but that is a property of the tree accumulating, not of any
+  # one pull request, and it does not need catching within five minutes of the merge.
+  schedule:
+    - cron: "0 3 * * *"
+  workflow_dispatch:
+
+concurrency:
+  group: observability-map-${{ github.event.pull_request.number || github.ref }}
+  cancel-in-progress: true
+
+permissions:
+  contents: read
+
+jobs:
+  # The whole cost of a pull request that touches nothing this workflow watches: a checkout, a paths
+  # filter and one comment lookup. Everything expensive is gated on this job's outputs, and the
+  # lookup is here rather than in the report job so that gate can read it and the report job need
+  # never start.
+  changes:
+    name: 🔍 What moved
+    # Only the pull request path reads this job's output. On a schedule the action has no base to
+    # diff, warns that `before` is missing and reports the files in the last commit on main, which
+    # nothing then consults. Skipping it there keeps the nightly off a job it does not need.
+    if: github.event_name == 'pull_request'
+    runs-on: warp-ubuntu-latest-x64-2x
+    permissions:
+      contents: read
+      # Reading the pull request's comments, to find one an earlier push left. Read only: the write
+      # stays on the report job, which is the only job that posts.
+      pull-requests: read
+    outputs:
+      # The corpus job's gate. Narrower than the report's on purpose: what the corpus measures is
+      # the tool's resistance to laundering, which only an edit to the tool can weaken.
+      package: ${{ steps.filter.outputs.package }}
+      # The report job's gate, the union: a route change moves the report as well.
+      report: ${{ steps.filter.outputs.package == 'true' || steps.filter.outputs.routes == 'true' }}
+      # The id of a marker comment an earlier push left, empty if there is none, and the one source
+      # both the render and upsert steps read it from.
+      comment: ${{ steps.comment.outputs.id }}
+      # Set only by a lookup that finished cleanly, so anything else, retries exhausted or the step
+      # dying somewhere unforeseen, reads as "do not touch this pull request's comments".
+      lookup: ${{ steps.comment.outputs.ok }}
+    steps:
+      - uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
+        with:
+          fetch-depth: 1
+          persist-credentials: false
+      - uses: dorny/paths-filter@fbd0ab8f3e69293af611ebaee6363fc25e6d187d # v4.0.1
+        id: filter
+        with:
+          filters: |
+            package:
+              - 'internal-packages/observability-map/**'
+              - '.github/workflows/observability-map.yml'
+            routes:
+              - 'apps/webapp/app/routes/**'
+
+      # Looked up here because the report job's gate needs it: with the watched paths unmoved, a
+      # pull request that already has a comment gets a resolved state rather than being left with
+      # findings that no longer exist, and one that does not gets no job at all.
+      #
+      # On a failure that outlasts the retries this reports nothing, and the report job's gate reads
+      # that as "post nothing this run". Guessing is worse than silence: this step is the only thing
+      # that knows which comment to PATCH, so a guess of "no comment exists" POSTs, which either
+      # adds a second marker comment beside the stale one or says "the findings an earlier push
+      # reported are gone" on a pull request that never had findings. Worst case now is no comment
+      # this run, which the next push fixes.
+      - name: 🔍 Look for a comment from an earlier push
+        id: comment
+        continue-on-error: true
+        env:
+          GH_TOKEN: ${{ github.token }}
+          PR_NUMBER: ${{ github.event.pull_request.number }}
+        run: |
+          found=""
+          ok=""
+          for attempt in 1 2 3; do
+            # Matched by login, not .user.type == "Bot": other bots and apps on the same PR are
+            # also type Bot, and login is the exact identity this token's own comments carry.
+            if found=$(gh api "repos/${GITHUB_REPOSITORY}/issues/${PR_NUMBER}/comments" --paginate \
+              --jq '[.[] | select(.user.login == "github-actions[bot]" and ((.body // "") | startswith("<!-- observability-map-report -->")))][0].id // empty'); then
+              ok=1
+              break
+            fi
+            echo "comment lookup attempt ${attempt} failed" >&2
+            sleep $((attempt * 5))
+          done
+
+          if [ -z "$ok" ]; then
+            echo "comment lookup failed after 3 attempts; this run posts nothing" >&2
+            exit 0
+          fi
+
+          # --paginate runs the jq once per page, so a marker comment on more than one page yields
+          # one id per page. Unhandled, that puts a newline in the PATCH url and the step dies under
+          # continue-on-error. The oldest wins: it is the one the upsert has been updating.
+          count=$(printf '%s\n' "$found" | grep -c '[0-9]' || true)
+          if [ "$count" -gt 1 ]; then
+            echo "warning: ${count} marker comments on this pull request; updating the oldest" >&2
+          fi
+          {
+            echo "id=$(printf '%s\n' "$found" | awk 'NF { print $1; exit }')"
+            echo "ok=ok"
+          } >> "$GITHUB_OUTPUT"
+
+  # The tree-scale mutation corpus: every known laundering shape applied to the whole route tree,
+  # asserting the score does not rise. 53 entries, a couple of minutes of a runner and a good deal
+  # longer on a laptop, which is why it is gated out of the package's default `pnpm test` and run
+  # here instead. Unlike the report job
+  # below it has no token to lose, so it runs for fork PRs too, and unlike the report job it is
+  # allowed to fail the build.
+  #
+  # Gated to this package's own paths rather than running on every route pull request. What the
+  # corpus measures is the TOOL's resistance to laundering, and only an edit to the tool can weaken
+  # that, so a routes-only change was paying a couple of minutes of a 4x runner for a result that
+  # could not differ from the last one. It was also the worst kind of job to spend that on: a red x
+  # that fires on a large share of webapp pull requests, is allowed to fail, and gates nothing, which is
+  # the shape people learn to scroll past.
+  #
+  # What this gives up is real and small. A route landing a shape no corpus entry has seen can make
+  # a known laundering mutation start paying, and that is now caught by the nightly rather than by
+  # the pull request that caused it. Tree drift accrues over months, so a day is the right
+  # granularity for it; the tool's own regressions, which are the ones a single commit can cause,
+  # still gate per pull request.
+  #
+  # Nothing in this repo watches whether the nightly itself succeeds: no Slack webhook and no
+  # issue-on-failure step here or in e2e-webapp-auth-full.yml, the only other scheduled workflow, so
+  # there is no house pattern to follow. A broken corpus fails quietly on the 3am cron, red only in
+  # the Actions tab,
+  # for as long as nobody checks it. Wiring up a real notification needs infrastructure (a Slack
+  # webhook secret, at minimum) that does not exist here yet, so this is a known, unfixed gap
+  # rather than a fixed one.
+  mutation-corpus:
+    name: 🧬 Mutation corpus
+    needs: changes
+    # `!cancelled()` is here for the nightly, not for tidiness. `needs` carries an implicit
+    # success() on the job it names, and that implicit test outranks the `||` below: with a plain
+    # condition, a `changes` job that failed or was skipped skips this one, so the nightly would
+    # stop scanning for tree drift and report nothing about having stopped. A status-check function
+    # in the `if` is what drops the implicit success(), so the event test below decides alone.
+    # `!cancelled()` rather than `always()` because `cancel-in-progress` above is a real path and a
+    # superseded run should not finish this job.
+    #
+    # Pull request behaviour is deliberately unchanged: on a PR a failed `changes` leaves
+    # `needs.changes.outputs.package` empty, so the corpus still skips. The nightly is the backstop
+    # for that, which is the same trade the paths gate already makes for routes-only pull requests.
+    if: >-
+      !cancelled() &&
+      (github.event_name != 'pull_request' || needs.changes.outputs.package == 'true')
+    runs-on: warp-ubuntu-latest-x64-4x
+    steps:
+      - name: ⬇️ Checkout repo
+        uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
+        with:
+          fetch-depth: 1
+          persist-credentials: false
+
+      - name: ⎔ Setup pnpm
+        uses: pnpm/action-setup@fc06bc1257f339d1d5d8b3a19a8cae5388b55320 # v5.0.0
+        with:
+          version: 10.33.2
+
+      - name: ⎔ Setup node
+        uses: WarpBuilds/setup-node@bc639b444d583175926b588962199c247d23e8d3 # v6
+        with:
+          node-version: 24.18.0
+          cache: "pnpm"
+
+      - name: 📥 Download deps
+        run: pnpm install --frozen-lockfile
+
+      - name: 🧬 Run the corpus
+        env:
+          OBS_MAP_MUTATION_CORPUS: "1"
+        run: |
+          pnpm --filter @internal/observability-map exec vitest run \
+            src/mutationCorpus.test.ts --disable-console-intercept
+
+  # The package's own tests are NOT run here. They gate through pr_checks.yml, which is the only
+  # workflow the all-checks aggregate can see, so a job in this file would report a result nobody
+  # is required to wait for. See unit-tests-observability-map.yml and the obsmap filter.
+  report:
+    needs: changes
+    runs-on: warp-ubuntu-latest-x64-4x
+    # Only this job comments, so only this job gets the write.
+    permissions:
+      contents: read
+      pull-requests: write
+    # Fork PRs get a read-only token, so the comment cannot post. Skipping the job beats a red x.
+    # The event test is what keeps this job off the nightly, which has no pull request to comment on
+    # and only exists for the corpus job above.
+    #
+    # The two output tests are what the workflow-level paths filter used to do, plus the thing it
+    # could not do. The report has to run when the watched paths moved, and ALSO when they did not
+    # but a marker comment is already on the pull request, because that comment is the one showing
+    # findings that have left the diff. Reconciling it needs no scan, so the steps below are gated
+    # again on the same output.
+    #
+    # `needs` carries an implicit success() and that is wanted here: a `changes` job that failed
+    # knows neither which paths moved nor whether a comment exists, and a report job that ran anyway
+    # could only guess. Same reason the lookup test is positive rather than a check for a failure
+    # sentinel: retries exhausted, or the lookup step dying anywhere unforeseen, both leave the
+    # output unset and both mean the same thing, so neither can be read as "no comment exists" by
+    # one step and "a comment exists" by another. That disagreement is what the sentinel pair this
+    # replaces got wrong once already.
+    if: >-
+      github.event_name == 'pull_request' &&
+      github.event.pull_request.head.repo.full_name == github.repository &&
+      needs.changes.outputs.lookup == 'ok' &&
+      (needs.changes.outputs.report == 'true' || needs.changes.outputs.comment != '')
+    steps:
+      - name: ⬇️ Checkout repo
+        uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
+        with:
+          fetch-depth: 0
+          persist-credentials: false
+
+      - name: ⎔ Setup pnpm
+        uses: pnpm/action-setup@fc06bc1257f339d1d5d8b3a19a8cae5388b55320 # v5.0.0
+        with:
+          version: 10.33.2
+
+      - name: ⎔ Setup node
+        uses: WarpBuilds/setup-node@bc639b444d583175926b588962199c247d23e8d3 # v6
+        with:
+          node-version: 24.18.0
+          cache: "pnpm"
+
+      - name: 📥 Download deps
+        run: pnpm install --frozen-lockfile
+
+      # Guarded rather than allowed to fail: this job must never block a pull request. The failure
+      # is not swallowed either, the render step below turns a missing head report into a comment
+      # saying so, because a swallowed failure with no comment is the outcome nobody wants.
+      #
+      # `--out` rather than a stdout redirect, so nothing a tool decides to print can end up inside
+      # the document `prCommentCli` parses. `pnpm --filter` takes its recursive path and some
+      # versions announce `Scope: N of M workspace projects` on the way; that line landing in
+      # head.json would fail the parse and degrade every run to the stale-report comment, which is
+      # a permanent quiet failure rather than a loud one. It does not reproduce on the 10.33.2
+      # pinned above, so this closes the class rather than a reproduction: the file is written by
+      # the process that owns it and stdout is left to be log output. Held by
+      # `it("let the scanner write its own report rather than capturing stdout")` in
+      # `internal-packages/observability-map/src/integration.test.ts`.
+      #
+      # `-s` keeps the partial dance honest now the redirect no longer creates the file: a scanner
+      # that exits 0 without writing takes the else branch and the stale-report comment, instead of
+      # failing the `mv` and turning the job red.
+      #
+      # Gated: this is the expensive half, and the reconcile run has nothing to compare. The steps
+      # above it are not gated because the renderer is TypeScript in this repo, so reconciling still
+      # needs the checkout and the install. That is the cost of the reconcile run and it is paid only
+      # by a pull request that has a comment and no longer matches the paths.
+      - name: 🔎 Scan head
+        if: needs.changes.outputs.report == 'true'
+        run: |
+          if pnpm --filter @internal/observability-map exec tsx src/cli.ts \
+            --out=/tmp/head.json.partial && [ -s /tmp/head.json.partial ]; then
+            mv /tmp/head.json.partial /tmp/head.json
+          else
+            rm -f /tmp/head.json /tmp/head.json.partial
+            echo "head scan failed; the comment will say the report is stale for this run" >&2
+          fi
+
+      # base.sha, not a merge base, and two reviewers have now read that as a bug. The checkout
+      # above is the default for a pull_request event, so the working tree is GitHub's test merge
+      # commit, whose parents are base.sha and the PR head. The head tree therefore already contains
+      # the base branch up to base.sha, and diffing it against base.sha is what isolates this pull
+      # request's own work. A merge base would leave the intervening base-branch commits in the head
+      # tree and out of the base tree, and blame the pull request for all of them.
+      - name: 🔎 Scan base with the head's scanner
+        if: needs.changes.outputs.report == 'true'
+        run: |
+          if git worktree add /tmp/base-tree ${{ github.event.pull_request.base.sha }} \
+            && pnpm --filter @internal/observability-map exec tsx src/cli.ts \
+              --routes=/tmp/base-tree/apps/webapp/app/routes --out=/tmp/base.json \
+            && [ -s /tmp/base.json ]; then
+            :
+          else
+            echo "-" > /tmp/base.json || true
+            echo "base scan failed or the worktree could not be added; falling back to no base" >&2
+          fi
+
+      # continue-on-error for the same reason as the scan: a rendering bug must not turn the job
+      # red. An empty /tmp/comment.md means there is nothing to post, which is a decision
+      # prCommentCli makes, not this shell.
+      #
+      # Both shas are forwarded so every comment this job posts says which commit it was rendered
+      # for, which a sticky comment edited in place across pushes otherwise never tells you. They go
+      # through the CLI as data: the renderer builds no URL and reads no environment.
+      - name: 📝 Render comment
+        continue-on-error: true
+        env:
+          SCANNED: ${{ needs.changes.outputs.report }}
+          EXISTING_COMMENT: ${{ needs.changes.outputs.comment }}
+          HEAD_SHA: ${{ github.event.pull_request.head.sha }}
+          COMPARE_URL: ${{ github.server_url }}/${{ github.repository }}/compare/${{ github.event.pull_request.base.sha }}...${{ github.event.pull_request.head.sha }}
+        run: |
+          rm -f /tmp/comment.md
+          # `--out` rather than a stdout redirect, for the reason the scan steps above give, and with a
+          # worse failure mode than theirs: the marker has to be the comment's first line for the
+          # lookup to find it, so a line printed ahead of the document makes every push post a new
+          # comment instead of updating the one already there. Held by
+          # `it("let the renderer write its own comment rather than capturing stdout")`.
+          render() {
+            pnpm --filter @internal/observability-map exec tsx src/report/prCommentCli.ts \
+              --commit-sha="$HEAD_SHA" --commit-url="$COMPARE_URL" --out=/tmp/comment.md.partial "$@"
+          }
+
+          # Every write goes through this, so a renderer that exits non-zero never leaves a 0-byte
+          # comment.md for the upsert to skip in silence.
+          emit() {
+            rm -f /tmp/comment.md.partial
+            if render "$@"; then
+              mv /tmp/comment.md.partial /tmp/comment.md
+              return 0
+            fi
+            rm -f /tmp/comment.md.partial
+            return 1
+          }
+
+          # Nothing this workflow watches moved, so nothing was scanned and there is no delta to
+          # compute. The job's gate only lets that case through when a comment from an earlier push
+          # is on the pull request, so there is exactly one thing left to say: what it shows is not
+          # in this diff any more.
+          if [ "$SCANNED" != "true" ]; then
+            emit --resolved || echo "could not render the resolved comment" >&2
+            exit 0
+          fi
+
+          if [ ! -s /tmp/head.json ]; then
+            emit --scan-failed || echo "could not render the stale-report comment either" >&2
+            exit 0
+          fi
+
+          base=/tmp/base.json
+          if [ ! -s /tmp/base.json ] || [ "$(cat /tmp/base.json)" = "-" ]; then
+            base="-"
+          fi
+
+          flags=()
+          if [ -n "$EXISTING_COMMENT" ]; then
+            flags=(--existing-comment)
+          fi
+
+          if ! emit /tmp/head.json "$base" "${flags[@]}"; then
+            echo "render failed; falling back to the stale-report comment" >&2
+            emit --scan-failed || echo "could not render the stale-report comment either" >&2
+          fi
+
+      # continue-on-error for the same reason: a transient gh api failure (rate limit, network)
+      # must not fail the job either. Worst case, the PR gets no comment this run.
+      #
+      # The id comes from the same job output the render step read, so the two cannot disagree about
+      # whether a comment exists. A lookup that did not finish cleanly never reaches either of them:
+      # the job's gate stops it.
+      - name: 💬 Upsert PR comment
+        continue-on-error: true
+        env:
+          GH_TOKEN: ${{ github.token }}
+          PR_NUMBER: ${{ github.event.pull_request.number }}
+          EXISTING_COMMENT: ${{ needs.changes.outputs.comment }}
+        run: |
+          if [ ! -s /tmp/comment.md ]; then
+            echo "nothing to post: this pull request does not move the report"
+            exit 0
+          fi
+          if [ -n "$EXISTING_COMMENT" ]; then
+            gh api -X PATCH "repos/${GITHUB_REPOSITORY}/issues/comments/${EXISTING_COMMENT}" -F body=@/tmp/comment.md
+          else
+            gh api "repos/${GITHUB_REPOSITORY}/issues/${PR_NUMBER}/comments" -F body=@/tmp/comment.md
+          fi

.github/workflows/pr_checks.yml

@@ -22,6 +22,7 @@ jobs:
       webapp: ${{ steps.filter.outputs.webapp }}
       packages: ${{ steps.filter.outputs.packages }}
       internal: ${{ steps.filter.outputs.internal }}
+      obsmap: ${{ steps.filter.outputs.obsmap }}
       cli: ${{ steps.filter.outputs.cli }}
       sdk: ${{ steps.filter.outputs.sdk }}
     steps:
@@ -57,6 +58,7 @@ jobs:
               - '.github/workflows/unit-tests-webapp.yml'
               - '.github/workflows/e2e-webapp.yml'
               - '.github/workflows/runops-guard.yml'
+              - '.github/workflows/fk-cascade-guard.yml'
               - '.configs/**'
               - 'package.json'
               - 'pnpm-lock.yaml'
@@ -81,6 +83,38 @@ jobs:
               - 'pnpm-lock.yaml'
               - 'pnpm-workspace.yaml'
               - 'turbo.json'
+            # The whole webapp app tree, not just its routes, and that is the whole reason this
+            # filter exists. Two tests in @internal/observability-map read it: integration.test.ts
+            # scans the live route tree, and webappSymbols.test.ts walks all of apps/webapp/app and
+            # fails when a guard, sensitive or audit symbol stops resolving. Routes-only was this
+            # filter's own bug: renaming e.g. requireUserId in app/services/session.server.ts
+            # matched `webapp` and nothing else, so no job ran the suite and the break landed on
+            # main, or on the next unrelated internal-packages PR.
+            #
+            # The cost of the wider set, measured over the last 400 commits on main: 31% touch
+            # routes, 52% touch apps/webapp/app, so the job goes from firing on roughly a third of
+            # PRs to roughly a half. It is the cheap one -- a single 4x runner, no containers, no
+            # database, no prisma generate -- which is what makes that affordable.
+            #
+            # observability-map.yml is here because integration.test.ts asserts on its text and no
+            # other filter watches it, so editing the report workflow alone ran nothing at all.
+            #
+            # Deliberately NOT here: this package's own paths, and packages/plugins/src and
+            # internal-packages/rbac/src, the other two trees webappSymbols.test.ts reads.
+            # `internal` above already matches `internal-packages/**` and `packages/**`, and
+            # `unit-tests-internal.yml` runs `turbo run test --filter "@internal/*"`, which picks up
+            # @internal/observability-map and runs the same vitest suite. Listing them here as well
+            # ran the suite twice on every PR touching them, which was this filter's own doing.
+            #
+            # Also deliberately NOT here: pr_checks.yml, package.json, pnpm-lock.yaml,
+            # pnpm-workspace.yaml. `internal` already lists all four, so a PR touching only one of
+            # them ran this suite twice for the same reason as above. Editing pr_checks.yml no
+            # longer runs this job live as a result; integration.test.ts still asserts on its text
+            # via the `internal` job.
+            obsmap:
+              - 'apps/webapp/app/**'
+              - '.github/workflows/unit-tests-observability-map.yml'
+              - '.github/workflows/observability-map.yml'
             cli:
               - 'packages/cli-v3/**'
               - 'packages/build/**'
@@ -117,6 +151,11 @@ jobs:
     if: needs.changes.outputs.webapp == 'true'
     uses: ./.github/workflows/runops-guard.yml
 
+  fk-cascade-guard:
+    needs: changes
+    if: needs.changes.outputs.webapp == 'true'
+    uses: ./.github/workflows/fk-cascade-guard.yml
+
   webapp:
     needs: changes
     if: needs.changes.outputs.webapp == 'true'
@@ -149,6 +188,11 @@ jobs:
       DOCKERHUB_USERNAME: ${{ secrets.DOCKERHUB_USERNAME }}
       DOCKERHUB_TOKEN: ${{ secrets.DOCKERHUB_TOKEN }}
 
+  obsmap:
+    needs: changes
+    if: needs.changes.outputs.obsmap == 'true'
+    uses: ./.github/workflows/unit-tests-observability-map.yml
+
   e2e:
     needs: changes
     if: needs.changes.outputs.cli == 'true'
@@ -168,10 +212,12 @@ jobs:
       - code-quality
       - typecheck
       - runops-guard
+      - fk-cascade-guard
       - webapp
       - e2e-webapp
       - packages
       - internal
+      - obsmap
       - e2e
       - sdk-compat
     if: always()

.github/workflows/publish-worker-v4.yml

@@ -13,6 +13,13 @@ on:
         type: string
         required: false
         default: ""
+    outputs:
+      version:
+        description: The published image tag
+        value: ${{ jobs.build.outputs.version }}
+      image_repo:
+        description: The image repository the build was published to (without tag)
+        value: ${{ jobs.build.outputs.image_repo }}
   push:
     tags:
       - "re2-test-*"
@@ -38,6 +45,11 @@ jobs:
       matrix:
         package: [supervisor]
     runs-on: warp-ubuntu-latest-x64-2x
+    # Single-entry matrix, so these job outputs are unambiguous (consumed by the
+    # scan-supervisor job in publish.yml).
+    outputs:
+      version: ${{ steps.get_tag.outputs.tag }}
+      image_repo: ${{ steps.set_tags.outputs.image_repo }}
     env:
       DOCKER_BUILDKIT: "1"
     steps:
@@ -81,6 +93,7 @@ jobs:
           fi
 
           echo "image_tags=${image_tags}" >> "$GITHUB_OUTPUT"
+          echo "image_repo=${ref_without_tag}" >> "$GITHUB_OUTPUT"
         env:
           IMAGE_REGISTRY: ${{ inputs.image_registry || vars.IMAGE_REGISTRY || format('ghcr.io/{0}', github.repository_owner) }}
           STEPS_GET_REPOSITORY_OUTPUTS_REPO: ${{ steps.get_repository.outputs.repo }}

.github/workflows/publish.yml

@@ -97,10 +97,19 @@ jobs:
     permissions:
       contents: read
       packages: read # pull the just-published image from GHCR
-    uses: ./.github/workflows/trivy-image-webapp.yml
+    uses: ./.github/workflows/trivy-image.yml
     with:
       image-ref: ${{ needs.publish-webapp.outputs.image_repo }}:${{ needs.publish-webapp.outputs.version }}
 
+  scan-supervisor:
+    needs: [publish-worker-v4]
+    permissions:
+      contents: read
+      packages: read # pull the just-published image from GHCR
+    uses: ./.github/workflows/trivy-image.yml
+    with:
+      image-ref: ${{ needs.publish-worker-v4.outputs.image_repo }}:${{ needs.publish-worker-v4.outputs.version }}
+
   # Announce the freshly published mutable `main` webapp image to subscriber
   # repos via repository_dispatch, handing them a digest-pinned ref to build or
   # deploy from. The repo, ref prefix, and dispatch target all default to the

.github/workflows/release-helm.yml

@@ -47,12 +47,14 @@ jobs:
 
       - name: Lint Helm Chart
         run: |
-          helm lint ./hosting/k8s/helm/
+          helm lint ./hosting/k8s/helm/ \
+            --values ./hosting/k8s/helm/ci/lint-values.yaml
 
       - name: Render templates
         run: |
           helm template test-release ./hosting/k8s/helm/ \
             --values ./hosting/k8s/helm/values.yaml \
+            --values ./hosting/k8s/helm/ci/lint-values.yaml \
             --output-dir ./helm-output
 
       - name: Validate manifests

.github/workflows/trivy-image.yml

@@ -1,7 +1,7 @@
-name: Trivy Image Scan (webapp)
+name: Trivy Image Scan
 
-# OS-level CVE scan of a published webapp image. Called by the publish pipeline
-# (publish.yml) to scan each build right after it's pushed to GHCR — so every
+# OS-level CVE scan of a published image. Called by the publish pipeline
+# (publish.yml) to scan each image right after it's pushed to GHCR — so every
 # main build and every release is scanned, not rebuilt. Also runnable ad-hoc
 # via workflow_dispatch against any image ref.
 #
@@ -27,7 +27,7 @@ on:
 permissions: {}
 
 concurrency:
-  group: trivy-image-webapp-${{ inputs.image-ref }}
+  group: trivy-image-${{ inputs.image-ref }}
   cancel-in-progress: true
 
 jobs:
@@ -59,7 +59,7 @@ jobs:
           ignore-unfixed: true
           severity: HIGH,CRITICAL
           format: table
-          output: trivy-image-webapp.txt
+          output: trivy-image.txt
 
       - name: Job summary
         if: always()
@@ -67,9 +67,9 @@ jobs:
           IMAGE_REF: ${{ inputs.image-ref }}
         run: |
           {
-            echo "## Trivy Image Scan (webapp) — \`${IMAGE_REF}\`"
+            echo "## Trivy Image Scan — \`${IMAGE_REF}\`"
             echo '```'
             # GitHub step summary is capped at 1 MiB; truncate large reports.
-            head -c 900000 trivy-image-webapp.txt 2>/dev/null || echo "(no report produced)"
+            head -c 900000 trivy-image.txt 2>/dev/null || echo "(no report produced)"
             echo '```'
           } >> "$GITHUB_STEP_SUMMARY"

.github/workflows/typecheck.yml

@@ -23,11 +23,19 @@ jobs:
           version: 10.33.2
 
       - name: ⎔ Setup node
-        uses: actions/setup-node@48b55a011bda9f5d6aeb4c2d9c7362e8dae4041e # v6.4.0
+        uses: WarpBuilds/setup-node@bc639b444d583175926b588962199c247d23e8d3 # v6
         with:
           node-version: 24.18.0
           cache: "pnpm"
 
+      - name: Restore Turbo cache
+        uses: WarpBuilds/cache@40f3443ae7b70e568d6e2070ea897f3df94d7553 # v1
+        with:
+          path: node_modules/.cache/turbo
+          key: turbo-typecheck-${{ runner.os }}-${{ hashFiles('pnpm-lock.yaml') }}-${{ github.sha }}
+          restore-keys: |
+            turbo-typecheck-${{ runner.os }}-${{ hashFiles('pnpm-lock.yaml') }}-
+
       - name: 📥 Download deps
         run: pnpm install --frozen-lockfile
 

.github/workflows/unit-tests-internal.yml

@@ -78,7 +78,6 @@ jobs:
         run: echo "DockerHub login skipped because secrets are not available."
 
       - name: 🐳 Pre-pull testcontainer images
-        if: ${{ env.DOCKERHUB_USERNAME }}
         run: |
           # Retry each pull - DockerHub registry timeouts are a recurring transient CI flake.
           pull() {
@@ -96,7 +95,6 @@ jobs:
           pull clickhouse/clickhouse-server:26.2.19.43-alpine@sha256:c6ad6a7eb2fb5999df3adfb8b69a0c7222c68fa9b8f6b04a088564ebbc959251
           pull redis:7.2
           pull testcontainers/ryuk:0.14.0
-          pull electricsql/electric:1.2.4
           echo "Image pre-pull complete"
 
       - name: 📥 Download deps

.github/workflows/unit-tests-observability-map.yml

@@ -0,0 +1,43 @@
+name: "🧪 Unit Tests: Observability Map"
+
+permissions:
+  contents: read
+
+# Its own workflow rather than a job inside observability-map.yml, because that workflow is not
+# reachable from pr_checks.yml's all-checks aggregate and so gates nothing. Called from there
+# instead, behind a paths filter, which is how every other test suite in this repo is gated.
+on:
+  workflow_call:
+
+jobs:
+  unitTests:
+    name: "🧪 Unit Tests: Observability Map"
+    # No containers and no database: the package is a static analyser over source text, so the
+    # suite is CPU bound on parsing the route tree and needs nothing the runner does not have.
+    runs-on: warp-ubuntu-latest-x64-4x
+    steps:
+      - name: ⬇️ Checkout repo
+        uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
+        with:
+          fetch-depth: 1
+          persist-credentials: false
+
+      - name: ⎔ Setup pnpm
+        uses: pnpm/action-setup@fc06bc1257f339d1d5d8b3a19a8cae5388b55320 # v5.0.0
+        with:
+          version: 10.33.2
+
+      - name: ⎔ Setup node
+        uses: WarpBuilds/setup-node@bc639b444d583175926b588962199c247d23e8d3 # v6
+        with:
+          node-version: 24.18.0
+          cache: "pnpm"
+
+      - name: 📥 Download deps
+        run: pnpm install --frozen-lockfile
+
+      # This suite reads apps/webapp/app (the route tree for the scan, the whole app tree for the
+      # symbol check) and the report workflow's text, which is why the filter that gates this
+      # workflow watches all of those and not only the routes folder.
+      - name: 🧪 Run tests
+        run: pnpm --filter @internal/observability-map run test

.github/workflows/unit-tests-packages.yml

@@ -81,7 +81,6 @@ jobs:
         run: echo "DockerHub login skipped because secrets are not available."
 
       - name: 🐳 Pre-pull testcontainer images
-        if: ${{ env.DOCKERHUB_USERNAME }}
         run: |
           # Retry each pull - DockerHub registry timeouts are a recurring transient CI flake.
           pull() {
@@ -98,7 +97,7 @@ jobs:
           pull clickhouse/clickhouse-server:26.2.19.43-alpine@sha256:c6ad6a7eb2fb5999df3adfb8b69a0c7222c68fa9b8f6b04a088564ebbc959251
           pull redis:7.2
           pull testcontainers/ryuk:0.14.0
-          pull electricsql/electric:1.2.4
+          pull otel/opentelemetry-collector-k8s:0.158.0@sha256:c09130a633196a5becee164411473a0932ecf223f94fda6dab5f22798ff9f376
           echo "Image pre-pull complete"
 
       - name: 📥 Download deps

.github/workflows/unit-tests-webapp.yml

@@ -14,18 +14,18 @@ on:
 jobs:
   unitTests:
     name: "🧪 Unit Tests: Webapp"
-    # 10 shards on 16x machines: webapp test throughput is limited per-machine (one
-    # docker daemon + disk absorbing all the per-file Postgres/ClickHouse container
-    # spin-up), so many machines beats few big ones - fewer/bigger (3x32) measured
-    # SLOWER than 10x8. The 16x (vs 8x) gives the fork pool the CPU headroom the 8x
-    # runners lacked. Setup overhead per machine is ~1 min on warm runners.
+    # Webapp test throughput is limited per-machine (one docker daemon + disk absorbing
+    # all the per-file Postgres/ClickHouse container spin-up), so many machines beats
+    # few big ones - fewer/bigger (3x32) measured slower than 10x8. The 16x (vs 8x)
+    # gives the fork pool the CPU headroom the 8x runners lacked.
     runs-on: warp-ubuntu-latest-x64-16x
     strategy:
       # one flaky shard shouldn't cancel its siblings - lets us re-run only the failed shard
       fail-fast: false
       matrix:
-        shardIndex: [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12]
-        shardTotal: [12]
+        shardIndex:
+          [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24]
+        shardTotal: [24]
     env:
       DOCKERHUB_USERNAME: ${{ secrets.DOCKERHUB_USERNAME }}
       SHARD_INDEX: ${{ matrix.shardIndex }}
@@ -69,7 +69,7 @@ jobs:
           version: 10.33.2
 
       - name: ⎔ Setup node
-        uses: actions/setup-node@48b55a011bda9f5d6aeb4c2d9c7362e8dae4041e # v6.4.0
+        uses: WarpBuilds/setup-node@bc639b444d583175926b588962199c247d23e8d3 # v6
         with:
           node-version: 24.18.0
           cache: "pnpm"
@@ -85,10 +85,10 @@ jobs:
         if: ${{ !env.DOCKERHUB_USERNAME }}
         run: echo "DockerHub login skipped because secrets are not available."
 
-      - name: 🐳 Pre-pull testcontainer images
-        if: ${{ env.DOCKERHUB_USERNAME }}
+      - name: 📥 Prepare deps and testcontainer images
         run: |
-          # Retry each pull - DockerHub registry timeouts are a recurring transient CI flake.
+          # Pull images concurrently with dependency installation. Retry each pull because
+          # DockerHub registry timeouts are a recurring transient CI flake.
           pull() {
             for attempt in 1 2 3; do
               docker pull "$1" && return 0
@@ -98,17 +98,41 @@ jobs:
             echo "::error::docker pull $1 failed after 3 attempts"
             return 1
           }
-          echo "Pre-pulling Docker images with authenticated session..."
-          pull postgres:14
-          pull clickhouse/clickhouse-server:26.2.19.43-alpine@sha256:c6ad6a7eb2fb5999df3adfb8b69a0c7222c68fa9b8f6b04a088564ebbc959251
-          pull redis:7.2
-          pull testcontainers/ryuk:0.14.0
-          pull electricsql/electric:1.2.4
-          pull minio/minio:latest
-          echo "Image pre-pull complete"
 
-      - name: 📥 Download deps
-        run: pnpm install --frozen-lockfile
+          pull_images() {
+            local pids=()
+            local failed=0
+            for image in \
+              postgres:14 \
+              postgres:17 \
+              clickhouse/clickhouse-server:26.2.19.43-alpine@sha256:c6ad6a7eb2fb5999df3adfb8b69a0c7222c68fa9b8f6b04a088564ebbc959251 \
+              redis:7.2 \
+              testcontainers/ryuk:0.14.0 \
+              electricsql/electric:1.2.4@sha256:20da3d0b0e74926c5623392db67fd56698b9e374c4aeb6cb5cadeb8fea171c36 \
+              minio/minio:latest
+            do
+              pull "$image" &
+              pids+=("$!")
+            done
+            for pid in "${pids[@]}"; do
+              if ! wait "$pid"; then
+                failed=1
+              fi
+            done
+            return "$failed"
+          }
+
+          echo "Installing dependencies and pre-pulling Docker images..."
+          pull_images &
+          pull_pid=$!
+          install_status=0
+          pnpm install --frozen-lockfile || install_status=$?
+          pull_status=0
+          wait "$pull_pid" || pull_status=$?
+          if (( install_status != 0 || pull_status != 0 )); then
+            exit 1
+          fi
+          echo "Dependency install and image pre-pull complete"
 
       - name: 📀 Generate Prisma Client
         run: pnpm run generate

.gitignore

@@ -76,3 +76,17 @@ apps/**/public/build
 ailogger-output.log
 # per-package vitest timing capture (transient; merged into root test-timings.json)
 .vitest-timing.json
+
+# local git worktree checkouts (not source) — keeps oxfmt/oxlint from descending into them
+.worktrees/
+
+# local planning/design docs, not committed
+**/docs/superpowers/
+
+# observability-map CLI output artifact, not committed
+observability-map.json
+
+.claude/worktrees/
+
+# CPU benchmark artifacts (profiles + summaries)
+.bench/

.oxfmtrc.json

@@ -21,6 +21,7 @@
     "**/storybook-static/",
     "**/.changeset/",
     "**/dist/",
+    "**/.worktrees/",
     "internal-packages/tsql/src/grammar/",
     "internal-packages/llm-model-catalog/src/defaultPrices.ts",
     "internal-packages/llm-model-catalog/src/modelCatalog.ts",

.oxlintrc.json

@@ -1,13 +1,18 @@
 {
   "$schema": "./node_modules/oxlint/configuration_schema.json",
-  "plugins": ["typescript", "import", "react"],
+  "categories": {
+    "correctness": "error"
+  },
+  "plugins": ["typescript", "import", "react", "jsx-a11y"],
   "jsPlugins": [
     "./oxlint-plugins/no-thrown-unawaited-redirect.mjs",
-    "./oxlint-plugins/runops-residency.mjs"
+    "./oxlint-plugins/runops-residency.mjs",
+    "./oxlint-plugins/prisma-in-filter.mjs"
   ],
   "ignorePatterns": [
     "**/dist/**",
     "**/build/**",
+    "**/.worktrees/**",
     "**/*.d.ts",
     "**/seed.js",
     "**/seedCloud.ts",
@@ -29,29 +34,170 @@
     ],
     "no-empty-pattern": "off",
     "no-control-regex": "off",
-    "typescript/no-non-null-asserted-optional-chain": "off",
-    "no-unused-expressions": ["warn", { "allowShortCircuit": true, "allowTernary": true }],
+    "typescript/no-non-null-asserted-optional-chain": "error",
+    "no-unused-expressions": [
+      "error",
+      {
+        "allowShortCircuit": true,
+        "allowTernary": true
+      }
+    ],
     "typescript/consistent-type-imports": "error",
     "import/no-duplicates": "error",
     "import/namespace": "off",
-    "react-hooks/exhaustive-deps": "off",
-    "react-hooks/rules-of-hooks": "off",
-    "trigger/no-thrown-unawaited-redirect": "error"
+    "react/exhaustive-deps": "error",
+    "react/rules-of-hooks": "off",
+    "guard-for-in": "error",
+    "symbol-description": "error",
+    "no-unneeded-ternary": "error",
+    "prefer-object-has-own": "error",
+    "no-redeclare": "error",
+    "no-multi-assign": "error",
+    "prefer-object-spread": "error",
+    "react/jsx-no-target-blank": "error",
+    "react/jsx-fragments": "error",
+    "react/self-closing-comp": "error",
+    "react/jsx-no-constructed-context-values": "error",
+    "react/no-children-prop": "error",
+    "react/no-danger-with-children": "error",
+    "react/no-direct-mutation-state": "error",
+    "react/no-find-dom-node": "error",
+    "react/no-is-mounted": "error",
+    "react/no-render-return-value": "error",
+    "react/no-string-refs": "error",
+    "react/no-unsafe": "error",
+    "react/no-will-update-set-state": "error",
+    "react/require-render-return": "error",
+    "react/style-prop-object": "error",
+    "react/void-dom-elements-no-children": "error",
+    "react/error-boundaries": "off",
+    "react/globals": "off",
+    "react/immutability": "off",
+    "react/incompatible-library": "off",
+    "react/preserve-manual-memoization": "off",
+    "react/purity": "off",
+    "react/refs": "off",
+    "react/set-state-in-effect": "off",
+    "react/set-state-in-render": "off",
+    "react/static-components": "off",
+    "react/unsupported-syntax": "off",
+    "react/use-memo": "off",
+    "react/void-use-memo": "off",
+    "react/checked-requires-onchange-or-readonly": "error",
+    "react/forward-ref-uses-ref": "error",
+    "react/iframe-missing-sandbox": "error",
+    "react/no-unknown-property": "error",
+    "jsx-a11y/alt-text": "error",
+    "jsx-a11y/aria-role": "error",
+    "jsx-a11y/click-events-have-key-events": "error",
+    "jsx-a11y/control-has-associated-label": [
+      "error",
+      {
+        "depth": 4,
+        "ignoreElements": ["audio", "canvas", "embed", "input", "textarea", "tr", "td", "video"]
+      }
+    ],
+    "jsx-a11y/label-has-associated-control": "error",
+    "jsx-a11y/no-autofocus": "off",
+    "jsx-a11y/no-noninteractive-element-interactions": "error",
+    "jsx-a11y/no-static-element-interactions": "error",
+    "jsx-a11y/prefer-tag-over-role": "off",
+    "jsx-a11y/anchor-ambiguous-text": "error",
+    "jsx-a11y/anchor-has-content": "error",
+    "jsx-a11y/anchor-is-valid": "error",
+    "jsx-a11y/aria-activedescendant-has-tabindex": "error",
+    "jsx-a11y/aria-props": "error",
+    "jsx-a11y/aria-proptypes": "error",
+    "jsx-a11y/aria-unsupported-elements": "error",
+    "jsx-a11y/autocomplete-valid": "error",
+    "jsx-a11y/heading-has-content": "error",
+    "jsx-a11y/html-has-lang": "error",
+    "jsx-a11y/iframe-has-title": "error",
+    "jsx-a11y/img-redundant-alt": "error",
+    "jsx-a11y/media-has-caption": "error",
+    "jsx-a11y/no-access-key": "error",
+    "jsx-a11y/no-aria-hidden-on-focusable": "error",
+    "jsx-a11y/no-distracting-elements": "error",
+    "jsx-a11y/no-redundant-roles": "error",
+    "jsx-a11y/role-has-required-aria-props": "error",
+    "jsx-a11y/role-supports-aria-props": "error",
+    "jsx-a11y/scope": "error",
+    "jsx-a11y/tabindex-no-positive": "error",
+    "no-lone-blocks": "error",
+    "typescript/prefer-function-type": "error",
+    "typescript/prefer-for-of": "error",
+    "trigger/no-thrown-unawaited-redirect": "error",
+    "trigger-prisma/no-unbounded-list-filter": "error",
+    "trigger-prisma/no-unbounded-list-filter-in-args-helper": "error"
   },
   "overrides": [
     {
       "files": ["apps/webapp/app/**/*.ts", "apps/webapp/app/**/*.tsx"],
       "rules": {
+        "react/button-has-type": "error",
+        "react/jsx-no-useless-fragment": "error",
+        "react/no-unstable-nested-components": "error",
+        "react/error-boundaries": "error",
+        "react/globals": "error",
+        "react/hooks": "error",
+        "react/immutability": "error",
+        "react/incompatible-library": "error",
+        "react/memo-dependencies": "error",
+        "react/no-deriving-state-in-effects": "error",
+        "react/preserve-manual-memoization": "error",
+        "react/purity": "error",
+        "react/refs": "error",
+        "react/set-state-in-effect": "error",
+        "react/set-state-in-render": "error",
+        "react/static-components": "error",
+        "react/unsupported-syntax": "error",
+        "react/use-memo": "error",
+        "react/void-use-memo": "error",
+        "react/rules-of-hooks": "error",
         "trigger-runops/no-control-plane-run-graph-access": "error",
         "trigger-runops/no-control-plane-in-runops-slot": "error"
       }
     },
+    {
+      "files": ["packages/react-hooks/src/**/*.ts", "packages/react-hooks/src/**/*.tsx"],
+      "rules": {
+        "react/rules-of-hooks": "error"
+      }
+    },
+    {
+      "files": ["**/*.ts", "**/*.tsx"],
+      "rules": {
+        "no-redeclare": "off"
+      }
+    },
     {
       "files": ["apps/webapp/app/**/*.test.ts", "apps/webapp/app/**/*.test.tsx"],
       "rules": {
         "trigger-runops/no-control-plane-run-graph-access": "off",
         "trigger-runops/no-control-plane-in-runops-slot": "off"
       }
+    },
+    {
+      "files": ["**/*.test.ts", "**/*.test.tsx", "**/test/**", "**/tests/**", "**/e2e/**"],
+      "rules": {
+        "trigger-prisma/no-unbounded-list-filter": "off",
+        "trigger-prisma/no-unbounded-list-filter-in-args-helper": "off"
+      }
+    },
+    {
+      "files": ["internal-packages/tsql/**"],
+      "rules": {
+        "prefer-object-has-own": "off"
+      }
+    },
+    {
+      "files": [
+        "apps/webapp/app/components/primitives/charts/Chart.tsx",
+        "apps/webapp/app/components/primitives/Timeline.tsx"
+      ],
+      "rules": {
+        "react/jsx-no-constructed-context-values": "off"
+      }
     }
   ]
 }

.server-changes/README.md

@@ -4,11 +4,13 @@ This directory tracks changes to server-only components (webapp, supervisor, etc
 
 ## When to add a file
 
-**Server-only PRs**: If your PR only changes `apps/webapp/`, `apps/supervisor/`, or other server components (and does NOT change anything in `packages/`), add a `.server-changes/` file.
+These entries are **user-facing release notes**, not a catalog of every change. The test is "would a user or customer care about this change?", not "did I touch a server app?". Add one only when a server-only change is something a user would notice, act on, or want to hear about (a feature, a bug fix they could have hit, a behavior or performance change they would feel). Skip it for internal-only or admin-only changes, refactors, test-only changes, chores, and performance tuning with no user-visible effect. Anyone who wants the exact history reads the commits. When in doubt, ask a maintainer rather than adding a note by default.
 
-**Mixed PRs** (both packages and server): Just add a changeset as usual. No `.server-changes/` file needed — the changeset covers it.
+**Server-only PRs**: If your PR only changes `apps/webapp/`, `apps/supervisor/`, or other server components (and does NOT change anything in `packages/`) AND the change is user-facing, add a `.server-changes/` file.
 
-**Package-only PRs**: Just add a changeset as usual.
+**Mixed PRs** (both packages and server): the changeset covers it, so no `.server-changes/` file is needed. If the package change is internal and needs no changeset but the server change is user-facing, add a `.server-changes/` file for it.
+
+**Package-only PRs**: Just add a changeset as usual, when the change is user-facing.
 
 ## File format
 

.server-changes/appearance-themes-and-options.md

@@ -0,0 +1,6 @@
+---
+area: webapp
+type: feature
+---
+
+The dashboard has two new themes, Black and White, plus appearance options for stronger colors and underlined links.

.server-changes/attio-sync-transient-retry.md

@@ -1,6 +0,0 @@
----
-area: webapp
-type: improvement
----
-
-Transient internal sync failures are now retried quietly instead of surfacing as errors.

.server-changes/clickhouse-read-replica.md

@@ -1,6 +0,0 @@
----
-area: webapp
-type: improvement
----
-
-Optionally route ClickHouse read traffic to a read replica while writes stay on the primary. Set `CLICKHOUSE_READER_URL` to move all reads, or target the busiest paths with `RUNS_LIST_CLICKHOUSE_URL` (runs list) and `EVENTS_READER_CLICKHOUSE_URL` (traces, spans, logs). All optional; unset keeps current behavior.

.server-changes/fix-batch-idempotency-per-item-keys.md

@@ -1,6 +0,0 @@
----
-area: webapp
-type: fix
----
-
-Fix batchTrigger requests that set a per-item idempotency key failing with an error instead of creating and deduplicating the runs

.server-changes/fix-batch-idempotency-point-lookups.md

@@ -1,6 +0,0 @@
----
-area: webapp
-type: fix
----
-
-Speed up idempotency checks on `batchTrigger` calls that use idempotency keys. Large batches against a task with a big run history no longer degrade to multi-second lookups.

.server-changes/reduce-webapp-cpu-on-worker-routes.md

@@ -0,0 +1,6 @@
+---
+area: webapp
+type: improvement
+---
+
+Cut webapp CPU usage by about a quarter on the routes that workers call most, freeing headroom at the same request rate. Detailed event-loop blocking traces are no longer recorded by default, because producing them was itself a large part of that cost.

.server-changes/remove-deprecated-realtime-stream-action.md

@@ -1,6 +0,0 @@
----
-area: webapp
-type: improvement
----
-
-Remove the deprecated realtime stream write endpoint used by retired v3 task clients.

.server-changes/run-ops-legacy-shared-pool.md

@@ -1,6 +0,0 @@
----
-area: webapp
-type: fix
----
-
-Avoid opening a redundant database connection pool when the legacy and primary databases are the same server, preventing connection usage from doubling.

.server-changes/runs-list-column-customization.md

@@ -0,0 +1,6 @@
+---
+area: webapp
+type: feature
+---
+
+Customize the runs list: show, hide, and reorder columns, and add smart columns that pull a value straight out of a run's payload, metadata, or output. Your column choices are saved in the page URL, so you can share a view, bookmark it, or save it straight to your favorites.

.server-changes/stale-deploy-asset-recovery.md

@@ -1,6 +0,0 @@
----
-area: webapp
-type: fix
----
-
-Fix pages occasionally loading unstyled or failing to load during a deploy. The dashboard now reloads automatically to recover.

.server-changes/vercel-version-skew-protection.md

@@ -0,0 +1,6 @@
+---
+area: webapp
+type: improvement
+---
+
+New Vercel connections now get version skew protection turned on automatically, so each run uses the task version its deployment shipped with. Automatic atomic deployments are deprecated and no longer offered when you connect a project, but stay available in your Vercel integration settings.

AGENTS.md

@@ -81,6 +81,18 @@ pnpm run lint:fix    # oxlint — auto-fixes lint violations
 pnpm run lint        # oxlint — check only (no fixes)
 ```
 
+### Dead code
+
+We use knip to control unused dependencies and code. It is enforced by CI `code-quality`.
+
+Scan your code before pushing with:
+
+```bash
+pnpm run knip
+```
+
+If there are false positives, edit ./knip.json so that it passes.
+
 ### Imports
 
 **Prefer static imports over dynamic imports.** Only use dynamic `import()` when:
@@ -92,7 +104,21 @@ Dynamic imports add unnecessary overhead in hot paths and make code harder to an
 
 ## Changesets and Server Changes
 
-When modifying any public package (`packages/*` or `integrations/*`), add a changeset:
+Changesets and `.server-changes/` files are **user-facing release notes**. They ship verbatim into the changelog that customers read to decide what to upgrade for or pay attention to. They are not a catalog of every change: anyone who wants the exact history reads the commits. So the question is not "did I touch a public package or a server app?" but **"would a user or customer care about this change?"**
+
+**Add one** when the change is something a user would notice, act on, or want to hear about: a new feature, a bug fix they could have hit, a behavior or performance change they would feel, a breaking change.
+
+**Skip it** (no changeset, no `.server-changes/` file) when the change is not worth communicating to users, even if it touches a public package or a server app. For example:
+
+- internal-only or admin-only changes, refactors, test-only changes, chores
+- performance or query tuning with no user-visible behavior change
+- changes to a public package that is not consumed independently (e.g. `@trigger.dev/redis-worker`), where a version bump means nothing to a user
+
+When in doubt, ask a maintainer rather than adding a note by default. An unnecessary entry is noise in the changelog, not a safe default.
+
+### How to add one
+
+When a **public package** (`packages/*` or `integrations/*`) change is user-facing, add a changeset:
 
 ```bash
 pnpm run changeset:add
@@ -102,9 +128,9 @@ pnpm run changeset:add
 - Confirm with maintainers before selecting **minor** (new features)
 - **Never** select major without explicit approval
 
-When modifying only server components (`apps/webapp/`, `apps/supervisor/`, etc.) with no package changes, add a `.server-changes/` file instead. See `.server-changes/README.md` for format and documentation.
+When a **server-only** change (`apps/webapp/`, `apps/supervisor/`, etc., with no package changes) is user-facing, add a `.server-changes/` file instead. See `.server-changes/README.md` for format and documentation.
 
-**Write the description for users, not maintainers.** Both changesets and `.server-changes/` notes ship verbatim in user-visible release notes. Lead with what changed *for the user* - one plain sentence describing behavior, not implementation, and never naming internal tools or infra. The full writing guidance in `.server-changes/README.md` applies to changesets too.
+**Write the description for users, not maintainers.** Both changesets and `.server-changes/` notes ship verbatim in user-visible release notes. Lead with what changed *for the user*: one plain sentence describing behavior, not implementation, and never naming internal tools or infra. The full writing guidance in `.server-changes/README.md` applies to changesets too.
 
 ## Dependency Pinning
 

CHANGESETS.md

@@ -4,11 +4,13 @@ Trigger.dev uses [changesets](https://github.com/changesets/changesets) to manag
 
 ## Adding a changeset (package changes)
 
-To add a changeset, use `pnpm run changeset:add` and follow the instructions [here](https://github.com/changesets/changesets/blob/main/docs/adding-a-changeset.md). Please only ever select one of our public packages when adding a changeset.
+Changesets and `.server-changes/` files are user-facing release notes, not a catalog of every change. Add one only when the change is something a user would notice or act on. Skip it for internal-only changes, refactors, chores, and packages that are not consumed independently (e.g. `@trigger.dev/redis-worker`). Anyone who wants the exact history reads the commits.
+
+To add a changeset, use `pnpm run changeset:add` and follow the [Changesets adding-a-changeset guide](https://github.com/changesets/changesets/blob/main/docs/adding-a-changeset.md). Please only ever select one of our public packages when adding a changeset.
 
 ## Adding a server change (server-only changes)
 
-If your PR only changes server components (`apps/webapp/`, `apps/supervisor/`, etc.) and does NOT change any published packages, add a `.server-changes/` file instead of a changeset:
+If your PR only changes server components (`apps/webapp/`, `apps/supervisor/`, etc.), does NOT change any published packages, AND the change is user-facing, add a `.server-changes/` file instead of a changeset:
 
 ```sh
 cat > .server-changes/fix-batch-queue-stalls.md << 'EOF'
@@ -24,17 +26,19 @@ EOF
 - `area`: `webapp` | `supervisor`
 - `type`: `feature` | `fix` | `improvement` | `breaking`
 
-For **mixed PRs** (both packages and server): just add a changeset. No `.server-changes/` file needed.
+For **mixed PRs** (both packages and server): the changeset covers it, so no `.server-changes/` file is needed. If the package change is internal and needs no changeset but the server change is user-facing, add a `.server-changes/` file for it.
 
 See `.server-changes/README.md` for full documentation.
 
 ## When to add which
 
+Only for user-facing changes. Skip the note entirely for internal-only or admin-only changes, refactors, and chores.
+
 | PR changes | What to add |
 |---|---|
-| Only packages (`packages/`) | Changeset (`pnpm run changeset:add`) |
-| Only server (`apps/`) | `.server-changes/` file |
-| Both packages and server | Just the changeset |
+| Only packages (`packages/` or `integrations/`) | Changeset (`pnpm run changeset:add`), if the package change is user-facing |
+| Only server (`apps/`) | `.server-changes/` file, if the server change is user-facing |
+| Both packages and server | The changeset covers it; if the package change needs no changeset but the server change is user-facing, add a `.server-changes/` file |
 
 ## Release instructions (CI)
 

CONTRIBUTING.md

@@ -214,6 +214,19 @@ pnpm exec trigger dev --log-level debug
 5. Commit the generated migration files as well as the changes to `schema.prisma`.
 6. If you're using VSCode you may need to restart the TypeScript server in the webapp to get updated type inference. Open a TypeScript file, then open the Command Palette (View > Command Palette) and run `TypeScript: Restart TS server`.
 
+## Git hooks (lefthook)
+
+We use [lefthook](https://lefthook.dev) for local git hooks, configured in `lefthook.yml` (the source of truth for what runs and when). Today that's a pre-push hook mirroring the CI `code-quality` checks; the set may grow, so check `lefthook.yml` rather than this guide.
+
+Hooks install automatically on `pnpm install`. A failing hook prints exactly what to run to fix it.
+
+**Opting out**
+
+- GitButler skips hooks on `but push` unless you enable **Run hooks** in the project settings (off by default).
+- Plain git: `LEFTHOOK=0 git push` / `--no-verify` to skip once; `pnpm exec lefthook uninstall` to remove.
+
+This never affects correctness — CI enforces the same checks on every PR; the hooks just give you faster feedback.
+
 ## Making a pull request
 
 **If you get errors, be sure to fix them before committing.**
@@ -225,10 +238,11 @@ pnpm exec trigger dev --log-level debug
 1. **Always open your PR in draft status first.** Do not mark it as "Ready for Review" until the steps below are complete.
 2. **Run format and lint locally before pushing:**
    ```bash
-   pnpm run format      # auto-fixes formatting (oxfmt)
-   pnpm run lint:fix    # auto-fixes lint violations (oxlint)
+   pnpm run format
+   pnpm run lint
+   pnpm run knip
    ```
-   Both are enforced by CI — the `code-quality` check will fail if either produces a diff or errors.
+   These are enforced by CI — the `code-quality` check will fail if either produces a diff or errors.
 3. **Address all CodeRabbit code review comments.** Our CI runs an automated code review via CodeRabbit. Go through each comment and either fix the issue or resolve it with a comment explaining why no change is needed.
 4. **Wait for all CI checks to pass.** Do not mark the PR as "Ready for Review" until every check is green.
 5. **Then mark the PR as "Ready for Review"** so a maintainer can take a look.
@@ -247,7 +261,7 @@ If your change touches core infrastructure, modifies widely-used code paths, or
 
 We use [changesets](https://github.com/changesets/changesets) to manage our package versions and changelogs. If you've never used changesets before, first read [their guide here](https://github.com/changesets/changesets/blob/main/docs/adding-a-changeset.md).
 
-If you are contributing a change to any packages in this monorepo (anything in either the `/packages` or `/integrations` directories), then you will need to add a changeset to your Pull Requests before they can be merged.
+Changesets are user-facing release notes, not a catalog of every change. If you are contributing a **user-facing** change to a package in this monorepo (anything in `/packages` or `/integrations` that a user would notice or act on), add a changeset to your Pull Request before it can be merged. Skip the changeset for internal-only changes, refactors, chores, and packages that are not consumed independently (e.g. `@trigger.dev/redis-worker`), where a version bump means nothing to a user.
 
 To add a changeset, run the following command in the root of the repo
 
@@ -265,7 +279,7 @@ Most of the time the changes you'll make are likely to be categorized as patch r
 
 ## Adding server changes
 
-Changesets only track published npm packages. If your PR only changes server components (`apps/webapp/`, `apps/supervisor/`, etc.) with no package changes, add a `.server-changes/` file so the change appears in release notes.
+Changesets only track published npm packages. If your PR only changes server components (`apps/webapp/`, `apps/supervisor/`, etc.) with no package changes AND the change is user-facing, add a `.server-changes/` file so the change appears in release notes. Skip it for internal-only or admin-only changes, refactors, and chores.
 
 Create a markdown file with a descriptive name:
 
@@ -286,13 +300,13 @@ EOF
 
 The body text (below the frontmatter) is a one-line description of the change. Keep it concise — it will appear in release notes.
 
-**When to add which:**
+**When to add which** (only for user-facing changes; skip the note entirely for internal-only or admin-only changes, refactors, and chores):
 
 | PR changes | What to add |
 |---|---|
-| Only packages (`packages/`) | Changeset |
-| Only server (`apps/`) | `.server-changes/` file |
-| Both packages and server | Just the changeset |
+| Only packages (`packages/` or `integrations/`) | Changeset (if the package change is user-facing) |
+| Only server (`apps/`) | `.server-changes/` file (if the server change is user-facing) |
+| Both packages and server | The changeset covers it; if the package change needs no changeset but the server change is user-facing, add a `.server-changes/` file |
 
 See `.server-changes/README.md` for more details.
 

ONBOARDING.md

@@ -0,0 +1,221 @@
+# Onboarding: taking over the hosted webhooks PR (#4344) for design
+
+You are picking up **PR #4344 "hosted webhooks, agent channels, and human-in-the-loop"** to own the UX and front-end. This doc gets you from a clean machine to a running dashboard with realistic webhook data you can screenshot, restyle, and iterate on.
+
+The feature is built and green (all backend plumbing, all four dashboard surfaces, the in-app test console). Your job is the visual and interaction design of the dashboard surfaces, not the backend. Everything below is oriented around that.
+
+---
+
+## 1. What you are designing
+
+Hosted webhooks let a Trigger.dev user receive and verify a provider's webhooks (Stripe, GitHub, and so on) as a task, with no ingress or verification code of their own. A `webhook()` handler in their project gets a hosted URL; deliveries to that URL are verified, recorded, and routed to their `onEvent` handler.
+
+The dashboard has **four surfaces you own**, all under the "Webhooks" nav section (teal icon):
+
+| Surface | Route (under `/orgs/:org/projects/:project/env/:env`) | What it shows |
+| --- | --- | --- |
+| **Deliveries list** | `/webhooks` | Every delivery across all endpoints in the environment. Runs-style filter bar (Status, Webhook, Created, plus a More-filters menu for Delivery ID / Run ID), applied-filter pills, a Webhook column linking to the handler. This is the main screen. |
+| **Delivery detail** | `/webhooks/deliveries/:deliveryParam` | One delivery. Main panel is a tabbed view (Event payload / Request headers) rendered as JSON. Sidebar property table (status badge, webhook + run links, external delivery id, idempotency key, timestamps, computed duration, error). Also has a friendly "not available / retained for N days" empty state for expired or bogus links. |
+| **Handler detail + Console** | `/webhooks/:webhookParam` | The handler (the `webhook()` in the user's code). Tabs: Deliveries, Runs, Endpoints. This page also hosts the **Webhook Console / Composer** (see section 5), the tool you will lean on for data. |
+| **Endpoint detail** | `/webhooks/endpoints/:endpointParam` | One endpoint. Left: scoped deliveries. Right: a **Connect** card (webhook URL, signing secret set/rotate/generate, provider setup rendered from the verifier config), Routing, Scope, Metadata. |
+
+The status vocabulary, badges, and colors live in `components/webhookDeliveries/v1/DeliveryStatus.tsx` and `components/webhookEndpoints/v1/EndpointStatus.tsx`. The nav accent color is a Tailwind token `--color-webhooks` (teal), used via `text-webhooks`.
+
+---
+
+## 2. Get the code
+
+You need the PR branch, `feat/hosted-webhook-ingress`.
+
+```bash
+git clone https://github.com/triggerdotdev/trigger.dev.git
+cd trigger.dev
+gh pr checkout 4344    # lands you on feat/hosted-webhook-ingress
+```
+
+If you plan to push design changes back to this branch, coordinate with Eric first: the branch is rebased and force-pushed periodically, so agree on timing or work on a child branch and open a follow-up.
+
+Toolchain: pnpm 10.33.2 via corepack, Node 22+. Use `corepack pnpm` (a bare `pnpm` can be an old global that wipes `node_modules`).
+
+```bash
+corepack enable
+corepack pnpm install
+```
+
+---
+
+## 3. Bring the stack up
+
+Four services and the webapp. Run from the repo root.
+
+```bash
+# 1. Core dev services: Postgres, Redis, Electric, MinIO, ClickHouse, s2-lite
+corepack pnpm run docker
+
+# 2. Config
+cp .env.example .env
+```
+
+Now edit `.env` and add the two webhook-delivery replication lines (they are NOT in `.env.example`, and without them the Deliveries list looks empty even after you send webhooks, see section 5):
+
+```bash
+# webhook deliveries replication (required for the Deliveries list/detail to populate)
+WEBHOOK_DELIVERIES_REPLICATION_CLICKHOUSE_URL=http://default:password@localhost:8123
+WEBHOOK_DELIVERIES_REPLICATION_ENABLED=1
+```
+
+Then migrate, seed, build, and run:
+
+```bash
+corepack pnpm run db:migrate
+corepack pnpm run db:seed        # creates the References org + hello-world project
+
+# Build the pieces you will run (do these sequentially, not with db:seed running)
+corepack pnpm run build --filter webapp --filter trigger.dev --filter "@trigger.dev/sdk"
+
+# Run the webapp (http://localhost:3030)
+corepack pnpm run dev --filter webapp
+curl -s http://localhost:3030/healthcheck   # verify
+```
+
+**Log in (dev):** open http://localhost:3030, submit the email `local@trigger.dev`. Dev auto-verifies the magic link (watch the webapp log for `/magic?token=`). That seeded user is an org admin, which matters for the next step.
+
+---
+
+## 4. Turn the feature on
+
+The dashboard is gated by a feature flag, `hasWebhooksAccess` (default off).
+
+- The seeded dev user `local@trigger.dev` is an **admin**, and admins bypass the flag, so on a fresh seed the Webhooks nav section is already visible to you. Nothing to do.
+- If you use a non-admin user, set `featureFlags.hasWebhooksAccess = true` on the `Organization` row to reveal the nav section. (A global `FeatureFlag` row with key `hasWebhooksAccess` makes the pages reachable by URL, but the left nav reads only org-level flags, so the section stays hidden for non-admins.)
+
+If the "Webhooks" section is missing from the left nav, this flag is why.
+
+---
+
+## 5. Get nice data (the part that matters)
+
+Delivery rows are what make these screens interesting: a spread of providers, statuses, payloads, timestamps. Here is how the data flows and how to produce it.
+
+### The pipeline (why an empty list is usually a setup issue, not a bug)
+
+`ingest -> engine (verify, filter, route) -> Postgres WebhookDelivery rows -> replication -> ClickHouse`. The Deliveries **list orders and paginates from ClickHouse**, then hydrates every visible field from Postgres. So if replication is off (section 3), you can create deliveries and still see an empty list. Enable the two replication env vars and restart the webapp.
+
+One caveat baked into the design: replication starts streaming from the moment it is enabled, so deliveries written **before** you turned it on will not appear. Turn replication on first, then generate data.
+
+### Fastest path: the seed script
+
+There is a seed script that inserts a full, stable dataset directly into both stores (Postgres and ClickHouse), so you get realistic screens on a fresh DB with no workers, no `trigger dev`, and no signing secrets to set:
+
+```bash
+corepack pnpm --filter webapp run db:seed:webhooks
+# optional: deliveries per endpoint (default 45)
+corepack pnpm --filter webapp run db:seed:webhooks -- 60
+```
+
+It creates six endpoints across different providers and verifier schemes (Stripe, GitHub, Slack, Svix, Discord, and a custom shared-secret one, with a mix of active/inactive and secret-set/not-set), then a spread of deliveries over the last two weeks covering **every** delivery status (SUCCEEDED, FAILED, FILTERED, PENDING, PROCESSING), realistic per-provider payloads and headers, and a mix of test and live. It attaches to the first DEVELOPMENT environment your local user can see (set `WEBHOOK_SEED_PROJECT="<project name>"` to target a specific one), and prints the exact Deliveries URL when it finishes. Re-running clears and reseeds that environment, so you always get the same clean dataset. The script is `apps/webapp/seed-webhook-deliveries.ts`; edit the `ENDPOINTS` array or the status weights to shape the data to whatever you are designing.
+
+Because it writes the ClickHouse rows directly, seeded data shows up **without** the replication setup in section 3. That replication env is only needed for the live and Composer paths below. (The seed uses `WEBHOOK_DELIVERIES_REPLICATION_CLICKHOUSE_URL` if set, otherwise `CLICKHOUSE_URL`, which is already in `.env.example`.)
+
+This is the recommended way to get data. The interactive paths below are for exercising the live pipeline (real verification, real routed runs) or the in-app test console.
+
+### Interactive: create an endpoint (one-time)
+
+The Composer sends to an endpoint, and endpoints only exist once a Trigger project that declares a `webhook()` has been dev-run or deployed. Quickest path: a tiny demo project.
+
+```ts
+// demo/src/trigger/demo-webhook.ts
+import { webhook, webhooks } from "@trigger.dev/sdk";
+
+export const demoWebhook = webhook({
+  id: "demo-webhook",
+  source: webhooks.custom<{ message: string }>({ /* generic HMAC */ }),
+  onEvent: async ({ event, headers, ctx }) => {
+    // event is the parsed body, headers is a Web Headers object
+  },
+});
+
+// A real provider, for realistic payloads:
+export const stripeWebhook = webhook({
+  id: "stripe-webhook",
+  source: webhooks.stripe(),
+  onEvent: async ({ event }) => {},
+});
+```
+
+Link that demo project to your local build and run `trigger dev` (see `AGENTS.md` "Testing with the hello-world Reference Project" for linking; the `triggerdotdev/references` repo has ready-made projects). Running `trigger dev` registers the `webhook()` handlers, which creates their endpoints. Set each endpoint's signing secret from the **endpoint detail Connect card** (Generate or paste).
+
+### Interactive: fire deliveries with the Webhook Console
+
+Open the handler detail page (`/webhooks/:webhookParam`). It hosts the **Composer** (`components/webhookConsole/WebhookComposer.tsx`). It has four source tabs and four signature modes, and it injects the delivery straight through the engine in-process, so it is fast and does not consume any real rate budget:
+
+- **Sample tab**: pick a real provider event from the built-in catalog (`@internal/webhook-sources`, six first-class providers plus a large sample manifest). This is the fastest way to get realistic Stripe / GitHub / Svix / Square / Discord payloads with correct-looking headers.
+- **Body tab**: hand-write any JSON.
+- **Replay tab**: re-send a prior delivery.
+- **AI tab**: generate a payload with a prompt.
+- **Signature modes** `signed | unsigned | tampered | simulate`: this is how you produce a **spread of delivery statuses**. `signed` (with a secret set) verifies and routes to a SUCCEEDED delivery; `unsigned` and `tampered` produce failed/rejected deliveries. Send a mix to populate every status badge you need to design.
+
+To get SUCCEEDED deliveries whose **runs** also complete (nicest end-to-end data), keep the demo project's `trigger dev` running so the routed task actually executes.
+
+### Interactive: a real provider (most realistic)
+
+For genuine payloads and headers, point the Stripe CLI at an endpoint: `stripe listen --forward-to http://localhost:3030/webhooks/v1/ingest/<opaqueId>`, set that endpoint's `whsec` via the Connect card, then `stripe trigger payment_intent.succeeded`.
+
+---
+
+## 6. Where the front-end code lives
+
+| Area | Path |
+| --- | --- |
+| Routes (pages) | `apps/webapp/app/routes/_app.orgs.$organizationSlug.projects.$projectParam.env.$envParam.webhooks*` |
+| Deliveries list / detail components | `apps/webapp/app/components/webhookDeliveries/v1/` (`DeliveriesTable`, `DeliveryStatus`, `WebhookDeliveryFilters`, `DeliveryTimeline`, `useDeliveriesLiveReload`) |
+| Endpoint components | `apps/webapp/app/components/webhookEndpoints/v1/` (`EndpointsTable`, `EndpointStatus`) |
+| Console / Composer | `apps/webapp/app/components/webhookConsole/` (`WebhookComposer`, `SampleSourcePicker`, `ReplaySourcePicker`) |
+| Data (presenters, read-only from your side) | `apps/webapp/app/presenters/v3/WebhookDeliveriesListPresenter.server.ts`, `WebhookDeliveryDetailPresenter.server.ts`, `WebhookDetailPresenter.server.ts`, `webhookComposerEndpoints.server.ts` |
+| Nav entry | `apps/webapp/app/components/navigation/SideMenu.tsx` (the `staticSections` "webhooks" push) |
+| Path builders | `apps/webapp/app/utils/pathBuilder.ts` (`v3WebhooksPath`, `v3WebhookDeliveryPath`, `v3WebhookEndpointPath`, `v3WebhookTaskPath`) |
+| Accent color token | `apps/webapp/app/tailwind.css` (`--color-webhooks`, used as `text-webhooks`) |
+| Data seed script | `apps/webapp/seed-webhook-deliveries.ts` (run via `db:seed:webhooks`) |
+
+**Styling:** the webapp is on Tailwind v4 (CSS-first `@theme` in `apps/webapp/app/tailwind.css`, there is no `tailwind.config.js`). Add or change design tokens there.
+
+**Design language to match:** these screens deliberately reuse the Runs page primitives (the filter bar is built from `RunFilters` / `SharedFilters`, the tables mirror the Runs table cells). Match the Runs and Sessions pages, not a new visual system.
+
+---
+
+## 7. Iterating
+
+- **HMR vs restart:** editing a component (`.tsx`) hot-reloads. Editing a `.server.ts` file makes the Remix dev server restart the app (a brief connection refused, then it comes back). Editing Tailwind tokens hot-reloads.
+- **Screenshots:** capture from the running dashboard at http://localhost:3030. Save shots outside the repo or to a scratch folder so they do not get committed.
+- **Typecheck after non-trivial changes:** `corepack pnpm run typecheck --filter webapp` (about 1 to 2 minutes). For small style tweaks, trust it and let CI catch anything.
+- **One boundary gotcha that the dev server will NOT catch:** route files must not leak server-only imports into the client bundle. The dev server tolerates it, but the production build fails. If you touch a route file and import anything server-only, run `corepack pnpm --filter webapp run build:remix` before pushing. Pure component and style edits are unaffected.
+
+---
+
+## 8. Shipping your changes
+
+Follow the repo PR workflow:
+
+- Format and lint before committing: `corepack pnpm run format` (oxfmt) and `corepack pnpm run lint:fix` (oxlint). CI enforces both.
+- Commit style is Conventional Commits, for example `feat(webapp): redesign webhook deliveries table`. No emoji, no attribution footer.
+- The PR is a **draft** awaiting an AI review pass, then a human review, before it flips to ready. Do not flip it to ready yourself; push your commits and let Eric coordinate the review and any rebase onto `main`.
+- CI to expect: `code-quality` (oxfmt + oxlint), `typecheck`, webapp unit shards, and the Playwright `e2e-webapp` job. Style-only changes usually only risk `code-quality`.
+
+---
+
+## 9. Quick reference
+
+- **Webapp:** http://localhost:3030 (port comes from `REMIX_APP_PORT`, falling back to `PORT`/3030).
+- **Default docker services:** Postgres 5432, Redis 6379, ClickHouse HTTP 8123 (`default:password`), MinIO, Electric, s2-lite.
+- **Feature flag:** `hasWebhooksAccess` (admins bypass).
+- **Seed data:** `corepack pnpm --filter webapp run db:seed:webhooks` (append `-- <n>` for deliveries per endpoint).
+- **Must-set env for data to show:** `WEBHOOK_DELIVERIES_REPLICATION_ENABLED=1` and `WEBHOOK_DELIVERIES_REPLICATION_CLICKHOUSE_URL=http://default:password@localhost:8123`.
+- **Login:** `local@trigger.dev`, magic link auto-verifies in dev.
+- **Feature docs:** `docs/webhooks/` (overview, sources, connect, deliveries, channels, human-in-the-loop). Read `overview.mdx` and `deliveries.mdx` first for the mental model behind the screens.
+- **PR:** https://github.com/triggerdotdev/trigger.dev/pull/4344
+
+---
+
+## 10. Mental model in one paragraph
+
+A user writes a `webhook()` in their project. On deploy (or `trigger dev`) that handler gets one or more hosted endpoints, each with a signing secret. A provider POSTs to the endpoint's URL; the engine verifies the signature, optionally filters, records a `WebhookDelivery`, and triggers the routed task run. The dashboard reads those deliveries: the list orders them out of ClickHouse and hydrates the rest from Postgres, the detail page reads Postgres directly (it holds the only copy of the event payload and headers). Everything you design sits on top of that delivery record and the endpoint that produced it.

apps/supervisor/.env.example

@@ -1,8 +1,8 @@
 # This needs to match the token of the worker group you want to connect to
 TRIGGER_WORKER_TOKEN=
 
-# This needs to match the MANAGED_WORKER_SECRET env var on the webapp
-MANAGED_WORKER_SECRET=managed-secret
+# Must match the webapp's MANAGED_WORKER_SECRET. Generate with: openssl rand -hex 16
+MANAGED_WORKER_SECRET=
 
 # Point this at the webapp in prod
 TRIGGER_API_URL=http://localhost:3030

apps/supervisor/package.json

@@ -21,7 +21,7 @@
     "ioredis": "~5.6.0",
     "p-limit": "^6.2.0",
     "prom-client": "^15.1.0",
-    "socket.io": "4.7.4",
+    "socket.io": "4.8.3",
     "std-env": "^3.8.0",
     "zod": "3.25.76"
   },

apps/supervisor/src/backpressure/backpressureMetrics.ts

@@ -8,6 +8,8 @@ export class BackpressureMetrics {
   readonly dryRun: Gauge<string>;
   /** Dequeue attempts the gate skipped - or would have, in dry-run (labelled). */
   readonly skipsTotal: Counter<string>;
+  /** Verdict source reads that failed (threw). */
+  readonly readFailuresTotal: Counter<string>;
 
   constructor(opts: { register: Registry; prefix?: string }) {
     const prefix = opts.prefix ?? "supervisor_backpressure";
@@ -30,5 +32,11 @@ export class BackpressureMetrics {
       labelNames: ["dry_run"],
       registers: [opts.register],
     });
+
+    this.readFailuresTotal = new Counter({
+      name: `${prefix}_read_failures_total`,
+      help: "Verdict source reads that threw",
+      registers: [opts.register],
+    });
   }
 }

apps/supervisor/src/backpressure/backpressureMonitor.test.ts

@@ -89,6 +89,60 @@ describe("BackpressureMonitor", () => {
     monitor.stop();
   });
 
+  it("holds an engaged verdict while reads fail, then releases past the max age", async () => {
+    let call = 0;
+    const source: BackpressureSignalSource = {
+      read: async () => {
+        call++;
+        if (call === 1) {
+          return { engaged: true, ts: Date.now() };
+        }
+        throw new Error("signal source unreachable");
+      },
+    };
+    const monitor = new BackpressureMonitor({
+      enabled: true,
+      source,
+      refreshIntervalMs: 1000,
+      maxVerdictAgeMs: 15_000,
+    });
+
+    monitor.start();
+    await vi.advanceTimersByTimeAsync(0);
+    expect(monitor.shouldSkipDequeue()).toBe(true);
+
+    await vi.advanceTimersByTimeAsync(5000);
+    expect(monitor.shouldSkipDequeue()).toBe(true); // read failing, verdict held
+
+    await vi.advanceTimersByTimeAsync(11_000);
+    expect(monitor.shouldSkipDequeue()).toBe(false); // past max age, released
+
+    monitor.stop();
+  });
+
+  it("releases immediately on an explicit null even when a grace window is configured", async () => {
+    let engaged: boolean | null = true;
+    const source: BackpressureSignalSource = {
+      read: async () => (engaged === null ? null : { engaged, ts: Date.now() }),
+    };
+    const monitor = new BackpressureMonitor({
+      enabled: true,
+      source,
+      refreshIntervalMs: 1000,
+      maxVerdictAgeMs: 15_000,
+    });
+
+    monitor.start();
+    await vi.advanceTimersByTimeAsync(0);
+    expect(monitor.shouldSkipDequeue()).toBe(true);
+
+    engaged = null;
+    await vi.advanceTimersByTimeAsync(1000);
+    expect(monitor.shouldSkipDequeue()).toBe(false); // null is an answer, not a failure
+
+    monitor.stop();
+  });
+
   it("fails open when the source reports unknown (null)", async () => {
     const { source } = countingSource(null);
     const monitor = new BackpressureMonitor({ enabled: true, source, refreshIntervalMs: 1000 });
@@ -292,6 +346,7 @@ describe("BackpressureMonitor", () => {
     const logs: Array<{ message: string; meta?: Record<string, unknown> }> = [];
     const logger = {
       info: (message: string, meta?: Record<string, unknown>) => logs.push({ message, meta }),
+      error: (message: string, meta?: Record<string, unknown>) => logs.push({ message, meta }),
     };
     const monitor = new BackpressureMonitor({
       enabled: true,

apps/supervisor/src/backpressure/backpressureMonitor.ts

@@ -1,7 +1,8 @@
 import type { BackpressureMetrics } from "./backpressureMetrics.js";
 
-export interface BackpressureLogger {
+interface BackpressureLogger {
   info(message: string, meta?: Record<string, unknown>): void;
+  error(message: string, meta?: Record<string, unknown>): void;
 }
 
 export type BackpressureVerdict = {
@@ -11,9 +12,10 @@ export type BackpressureVerdict = {
 };
 
 /**
- * Source of the current backpressure verdict. `read()` returns `null` when the
- * verdict is unknown (missing/unreadable) - the monitor treats unknown as
- * "not engaged" (fail-open).
+ * Source of the current backpressure verdict. `read()` returns `null` when the source
+ * answered but there is no verdict - the monitor treats that as "not engaged"
+ * (fail-open). A thrown error is different: the read itself failed, so the monitor
+ * keeps the previous verdict until it ages past `maxVerdictAgeMs`.
  */
 export interface BackpressureSignalSource {
   read(): Promise<BackpressureVerdict | null>;
@@ -24,8 +26,9 @@ export type BackpressureMonitorOptions = {
   source: BackpressureSignalSource;
   refreshIntervalMs?: number;
   /**
-   * If set, a cached verdict older than this is treated as unknown (fail-open).
-   * Guards against the source silently going stale (e.g. hanging reads).
+   * If set, an engaged verdict older than this is released (fail-open), bounding how
+   * long a dead source can hold the brake. Reads that fail keep the last verdict, so
+   * this doubles as the grace window for riding out a transient source outage.
    */
   maxVerdictAgeMs?: number;
   /**
@@ -54,6 +57,7 @@ export class BackpressureMonitor {
   private refreshInFlight = false;
   private wasEngaged = false;
   private releasedAt?: number;
+  private readFailing = false;
 
   constructor(private readonly opts: BackpressureMonitorOptions) {
     this.opts.metrics?.dryRun.set(this.opts.dryRun ? 1 : 0);
@@ -152,12 +156,31 @@ export class BackpressureMonitor {
   }
 
   private async refresh(): Promise<void> {
+    let next: BackpressureVerdict | null = null;
+    let readError: unknown;
     try {
-      this.verdict = await this.opts.source.read();
-    } catch {
-      // Fail-open: a dead/unreachable source must never pin the brake. Treat as
-      // unknown (no verdict) so dequeue resumes as if backpressure were off.
-      this.verdict = null;
+      next = await this.opts.source.read();
+    } catch (error) {
+      readError = error;
+    }
+
+    if (readError === undefined) {
+      this.verdict = next; // an explicit null means "no pressure", so honour it
+      this.readFailing = false;
+    } else {
+      const held = this.opts.maxVerdictAgeMs !== undefined;
+      if (!held) {
+        this.verdict = null; // unbounded hold could pin the brake forever
+      }
+      this.opts.metrics?.readFailuresTotal.inc();
+      if (!this.readFailing) {
+        this.readFailing = true; // log once per outage, not once per tick
+        this.opts.logger?.error("backpressure read failed", {
+          reason: String(readError),
+          heldPreviousVerdict: held,
+          engaged: this.computeEngaged(),
+        });
+      }
     }
 
     // Track the engaged→released transition to anchor the resume ramp. Use the

apps/supervisor/src/backpressure/k8sPodCountSignalSource.test.ts

@@ -1,52 +1,49 @@
 import { describe, it, expect } from "vitest";
-import { parsePodCount, K8sPodCountSignalSource } from "./k8sPodCountSignalSource.js";
+import { K8sPodCountSignalSource } from "./k8sPodCountSignalSource.js";
+import { podCountFromList, withTimeout } from "../clients/kubernetes.js";
 
-describe("parsePodCount", () => {
-  it("reads the pods object count", () => {
-    const text = [
-      "# HELP apiserver_storage_objects Number of stored objects",
-      "# TYPE apiserver_storage_objects gauge",
-      'apiserver_storage_objects{resource="pods"} 8421',
-      'apiserver_storage_objects{resource="configmaps"} 17',
-    ].join("\n");
-    expect(parsePodCount(text)).toBe(8421);
+describe("podCountFromList", () => {
+  it("returns items.length when the list is not truncated", () => {
+    expect(podCountFromList({ items: [{}], metadata: {} })).toBe(1);
   });
 
-  it("is tolerant of extra labels in any order", () => {
-    const text = 'apiserver_storage_objects{group="",resource="pods",extra="x"} 12';
-    expect(parsePodCount(text)).toBe(12);
+  it("returns zero for an empty namespace", () => {
+    expect(podCountFromList({ items: [], metadata: {} })).toBe(0);
   });
 
-  it("parses scientific notation", () => {
-    const text = 'apiserver_storage_objects{resource="pods"} 1.2e+04';
-    expect(parsePodCount(text)).toBe(12000);
+  it("adds remainingItemCount when the list is truncated", () => {
+    const list = { items: [{}], metadata: { _continue: "tok", remainingItemCount: 24492 } };
+    expect(podCountFromList(list)).toBe(24493);
   });
 
-  it("throws when the pods metric is absent", () => {
-    const text = 'apiserver_storage_objects{resource="configmaps"} 17';
-    expect(() => parsePodCount(text)).toThrow(/not found/);
+  it("throws when truncated but remainingItemCount is absent", () => {
+    const list = { items: [{}], metadata: { _continue: "tok" } };
+    expect(() => podCountFromList(list)).toThrow(/remainingItemCount/);
   });
 
-  it("throws on a non-finite value (e.g. 1e999)", () => {
-    const text = 'apiserver_storage_objects{resource="pods"} 1e999';
-    expect(() => parsePodCount(text)).toThrow();
-  });
-
-  it("throws on a negative value", () => {
-    const text = 'apiserver_storage_objects{resource="pods"} -5';
-    expect(() => parsePodCount(text)).toThrow();
+  it("throws when truncated but remainingItemCount is negative", () => {
+    const list = { items: [{}], metadata: { _continue: "tok", remainingItemCount: -1 } };
+    expect(() => podCountFromList(list)).toThrow(/remainingItemCount/);
   });
 });
 
-function metrics(count: number): string {
-  return `apiserver_storage_objects{resource="pods"} ${count}`;
-}
+describe("withTimeout", () => {
+  it("rejects once the deadline passes", async () => {
+    await expect(withTimeout(new Promise(() => {}), 10, "pod count list")).rejects.toThrow(
+      /timed out/
+    );
+  });
+
+  it("passes a value through when it settles first", async () => {
+    await expect(withTimeout(Promise.resolve(7), 1000, "pod count list")).resolves.toBe(7);
+  });
+});
 
 describe("K8sPodCountSignalSource", () => {
   it("engages at the engage threshold and reports the count", async () => {
     const counts: number[] = [];
     const source = new K8sPodCountSignalSource({
-      fetchMetrics: async () => metrics(10000),
+      fetchPodCount: async () => 10000,
       engageThreshold: 10000,
       releaseThreshold: 5000,
       reportPodCount: (c) => counts.push(c),
@@ -59,7 +56,7 @@ describe("K8sPodCountSignalSource", () => {
 
   it("does not engage below the engage threshold", async () => {
     const source = new K8sPodCountSignalSource({
-      fetchMetrics: async () => metrics(9999),
+      fetchPodCount: async () => 9999,
       engageThreshold: 10000,
       releaseThreshold: 5000,
     });
@@ -69,7 +66,7 @@ describe("K8sPodCountSignalSource", () => {
   it("stays engaged in the hysteresis band, releases only below release threshold", async () => {
     let count = 10000;
     const source = new K8sPodCountSignalSource({
-      fetchMetrics: async () => metrics(count),
+      fetchPodCount: async () => count,
       engageThreshold: 10000,
       releaseThreshold: 5000,
     });
@@ -82,9 +79,9 @@ describe("K8sPodCountSignalSource", () => {
     expect((await source.read()).engaged).toBe(false); // band again -> stays off
   });
 
-  it("propagates scrape failures (monitor fails open on throw)", async () => {
+  it("propagates fetch failures (monitor fails open on throw)", async () => {
     const source = new K8sPodCountSignalSource({
-      fetchMetrics: async () => {
+      fetchPodCount: async () => {
         throw new Error("connection refused");
       },
       engageThreshold: 10000,

apps/supervisor/src/backpressure/k8sPodCountSignalSource.ts

@@ -1,22 +1,7 @@
 import type { BackpressureSignalSource, BackpressureVerdict } from "./backpressureMonitor.js";
 
-// Reads the apiserver's stored-pod-object count from a Prometheus /metrics scrape.
-const POD_COUNT_RE = /^apiserver_storage_objects\{[^}]*resource="pods"[^}]*\}\s+([0-9.eE+]+)/m;
-
-export function parsePodCount(metricsText: string): number {
-  const match = metricsText.match(POD_COUNT_RE);
-  if (!match) {
-    throw new Error('apiserver_storage_objects{resource="pods"} not found in metrics');
-  }
-  const value = Number(match[1]);
-  if (!Number.isFinite(value)) {
-    throw new Error(`unparseable pod count: ${match[1]}`);
-  }
-  return value;
-}
-
 export type K8sPodCountSignalSourceOptions = {
-  fetchMetrics: () => Promise<string>;
+  fetchPodCount: () => Promise<number>;
   engageThreshold: number;
   releaseThreshold: number;
   reportPodCount?: (count: number) => void;
@@ -29,8 +14,7 @@ export class K8sPodCountSignalSource implements BackpressureSignalSource {
   constructor(private readonly opts: K8sPodCountSignalSourceOptions) {}
 
   async read(): Promise<BackpressureVerdict> {
-    const text = await this.opts.fetchMetrics();
-    const count = parsePodCount(text);
+    const count = await this.opts.fetchPodCount();
     this.opts.reportPodCount?.(count);
 
     if (this.engaged) {

apps/supervisor/src/clients/kubernetes.ts

@@ -2,9 +2,8 @@ import * as k8s from "@kubernetes/client-node";
 import type { Informer, KubernetesObject, ListPromise } from "@kubernetes/client-node";
 import { assertExhaustive } from "@trigger.dev/core/utils";
 import { SimpleStructuredLogger } from "@trigger.dev/core/v3/utils/structuredLogger";
-import * as https from "node:https";
 
-export const RUNTIME_ENV = process.env.KUBERNETES_PORT ? "kubernetes" : "local";
+const RUNTIME_ENV = process.env.KUBERNETES_PORT ? "kubernetes" : "local";
 
 const logger = new SimpleStructuredLogger("kubernetes-client");
 
@@ -54,55 +53,90 @@ function getKubeConfig() {
 export { k8s };
 
 /**
- * Builds a function that scrapes the apiserver's Prometheus /metrics endpoint.
- * One lightweight aggregate read - not a pod listing. Requires the service
- * account to be granted GET on the /metrics non-resource URL.
+ * createPodCountFetcher sizes a namespace's pod collection with a single `limit=1`
+ * list: one pod transferred, no informer, no watch cache.
+ *
+ * This is an ESTIMATE, not an exact count. Kubernetes documents `remainingItemCount`
+ * as intended for estimating collection size and reserves the right not to set it or
+ * make it exact. Counting exactly would mean paginating the whole collection, which is
+ * what this deliberately avoids. Treat the value as a tight estimate from a quorum read
+ * at request time, and set thresholds with that in mind.
+ *
+ * Two request-shape constraints, both load-bearing. A label or field selector makes
+ * the apiserver omit `remainingItemCount` entirely, and setting `resourceVersion`
+ * serves a cached count instead of a quorum read - so neither is passed.
  */
-export function createApiserverMetricsFetcher(timeoutMs: number): () => Promise<string> {
-  const kubeConfig = getKubeConfig();
+export function createPodCountFetcher(
+  api: K8sApi,
+  namespace: string,
+  timeoutMs: number
+): () => Promise<number> {
+  const serverTimeoutSeconds = Math.max(1, Math.floor(timeoutMs / 1000));
+  let pending: Promise<unknown> | undefined;
 
   return async () => {
-    const cluster = kubeConfig.getCurrentCluster();
-    if (!cluster) {
-      throw new Error("no current cluster in kubeconfig");
+    if (pending) {
+      throw new Error("pod count list still in flight from a previous tick");
     }
-    const url = new URL(`${cluster.server}/metrics`);
-    const opts: https.RequestOptions = {
-      method: "GET",
-      protocol: url.protocol,
-      hostname: url.hostname,
-      port: url.port,
-      path: url.pathname,
-    };
-    // applyToHTTPSOptions sets the cluster CA, client cert/key, and auth headers
-    // (incl. exec plugins) on the request - so TLS verifies against the cluster
-    // CA, not the system store. The fetch-options path attaches the CA as an
-    // https.Agent, which global fetch (undici) ignores.
-    await kubeConfig.applyToHTTPSOptions(opts);
 
-    return new Promise<string>((resolve, reject) => {
-      const req = https.request(opts, (res) => {
-        const status = res.statusCode ?? 0;
-        let body = "";
-        res.setEncoding("utf8");
-        res.on("data", (chunk) => {
-          body += chunk;
-        });
-        res.on("end", () => {
-          if (status >= 200 && status < 300) {
-            resolve(body);
-          } else {
-            reject(new Error(`apiserver /metrics scrape failed: ${status}`));
-          }
-        });
-      });
-      // Without this a hung connect/TLS/read never settles, and the monitor's
-      // refreshInFlight guard would freeze the source (silent fail-open).
-      req.setTimeout(timeoutMs, () => {
-        req.destroy(new Error(`apiserver /metrics scrape timed out after ${timeoutMs}ms`));
-      });
-      req.on("error", reject);
-      req.end();
+    const request = api.core.listNamespacedPod({
+      namespace,
+      limit: 1,
+      timeoutSeconds: serverTimeoutSeconds,
     });
+
+    pending = request
+      .catch(() => {})
+      .finally(() => {
+        pending = undefined;
+      });
+
+    return podCountFromList(await withTimeout(request, timeoutMs, "pod count list"));
   };
 }
+
+/**
+ * podCountFromList turns a `limit=1` pod list into a population estimate.
+ *
+ * `remainingItemCount` is only set when the list is truncated, so `_continue` is the
+ * truncation signal: absent means the returned page is the whole collection and its
+ * length is exact. When truncated the total leans on `remainingItemCount`, which is
+ * documented as an estimate - so the result is an estimate too. Truncated without a
+ * usable count is unknowable, so it throws rather than returning a low number the
+ * caller would act on.
+ */
+export function podCountFromList(list: {
+  items: unknown[];
+  metadata?: { _continue?: string; remainingItemCount?: number };
+}): number {
+  if (!list.metadata?._continue) {
+    return list.items.length;
+  }
+
+  const remaining = list.metadata.remainingItemCount;
+  if (typeof remaining !== "number" || !Number.isFinite(remaining) || remaining < 0) {
+    throw new Error("pod list truncated but remainingItemCount absent or invalid");
+  }
+
+  return list.items.length + remaining;
+}
+
+/**
+ * withTimeout rejects if `promise` outlives `timeoutMs`, so a hung request cannot
+ * freeze the caller. It cannot cancel: the k8s client threads no AbortSignal through to
+ * fetch, so an abandoned request keeps running. Callers must therefore also bound the
+ * request server-side (`timeoutSeconds`) and refuse to start a second one while the
+ * first is pending, or a blackholed connection accumulates one socket per attempt.
+ */
+export function withTimeout<T>(promise: Promise<T>, timeoutMs: number, what: string): Promise<T> {
+  let timer: NodeJS.Timeout;
+  const deadline = new Promise<never>((_resolve, reject) => {
+    timer = setTimeout(
+      () => reject(new Error(`${what} timed out after ${timeoutMs}ms`)),
+      timeoutMs
+    );
+    timer.unref();
+  });
+
+  return Promise.race([promise, deadline]).finally(() => clearTimeout(timer));
+}

apps/supervisor/src/env.ts

@@ -1,7 +1,13 @@
 import { randomUUID } from "crypto";
 import { env as stdEnv } from "std-env";
 import { z } from "zod";
-import { AdditionalEnvVars, BoolEnv } from "./envUtil.js";
+import {
+  AdditionalEnvVars,
+  BoolEnv,
+  NodeLabelValue,
+  OrgPlacementOverrides,
+  Tolerations,
+} from "./envUtil.js";
 
 export const Env = z
   .object({
@@ -14,8 +20,18 @@ export const Env = z
 
     // Required settings
     TRIGGER_API_URL: z.string().url(),
-    TRIGGER_WORKER_TOKEN: z.string(), // accepts file:// path to read from a file
+    TRIGGER_WORKER_TOKEN: z.string().min(1), // accepts file:// path to read from a file
     MANAGED_WORKER_SECRET: z.string(),
+
+    // Deployment token: sign a token into TRIGGER_DEPLOYMENT_ID at pod creation and verify it on
+    // inbound workload calls. "disabled" = off; "log" = mint + verify + metrics only; "enforce" =
+    // also reject invalid tokens.
+    WORKLOAD_TOKEN_SECRET: z.string().optional(),
+    WORKLOAD_TOKEN_ENFORCEMENT: z.enum(["disabled", "log", "enforce"]).default("disabled"),
+    DELETE_CHECKPOINTS_ON_COMPLETION: BoolEnv.default(false), // irreversible; enable per cluster
+    // Absolute expiry for minted deployment tokens. Deterministic (no wall-clock issued-at) so every
+    // pod of a deployment carries an identical token; bump before this date. Must outlive any run.
+    WORKLOAD_TOKEN_EXP: z.string().datetime().default("2032-01-01T00:00:00.000Z"),
     OTEL_EXPORTER_OTLP_ENDPOINT: z.string().url(), // set on the runners
 
     // Workload API settings (coordinator mode) - the workload API is what the run controller connects to
@@ -70,7 +86,7 @@ export const Env = z
       .number()
       .int()
       .positive()
-      .default(15_000), // Stale verdict → fail-open (treat as not engaged)
+      .default(120_000), // Grace window: held verdict older than this → fail-open
     TRIGGER_DEQUEUE_BACKPRESSURE_REDIS_HOST: z.string().optional(),
     TRIGGER_DEQUEUE_BACKPRESSURE_REDIS_PORT: z.coerce.number().int().optional(),
     TRIGGER_DEQUEUE_BACKPRESSURE_REDIS_USERNAME: z.string().optional(),
@@ -103,6 +119,7 @@ export const Env = z
 
     // Optional services
     TRIGGER_WARM_START_URL: z.string().optional(),
+    TRIGGER_WARM_START_DISPATCH_URL: z.string().optional(),
     TRIGGER_CHECKPOINT_URL: z.string().optional(),
     TRIGGER_METADATA_URL: z.string().optional(),
 
@@ -163,11 +180,14 @@ export const Env = z
     // Kubernetes settings
     KUBERNETES_FORCE_ENABLED: BoolEnv.default(false),
     KUBERNETES_NAMESPACE: z.string().default("default"),
-    KUBERNETES_WORKER_NODETYPE_LABEL: z.string().default("v4-worker"),
+    KUBERNETES_WORKER_NODETYPE_LABEL: NodeLabelValue.default("v4-worker"),
     KUBERNETES_IMAGE_PULL_SECRETS: z.string().optional(), // csv
     KUBERNETES_EPHEMERAL_STORAGE_SIZE_LIMIT: z.string().default("10Gi"),
     KUBERNETES_EPHEMERAL_STORAGE_SIZE_REQUEST: z.string().default("2Gi"),
     KUBERNETES_STRIP_IMAGE_DIGEST: BoolEnv.default(false),
+    KUBERNETES_IMAGE_REGISTRY_REWRITE_FROM: z.string().optional(),
+    KUBERNETES_IMAGE_REGISTRY_REWRITE_TO: z.string().optional(),
+    KUBERNETES_RUN_POD_PRIORITY_CLASS_NAME: z.string().optional(),
     KUBERNETES_CPU_REQUEST_MIN_CORES: z.coerce.number().min(0).default(0),
     KUBERNETES_CPU_REQUEST_RATIO: z.coerce.number().min(0).max(1).default(0.75), // Ratio of CPU limit, so 0.75 = 75% of CPU limit
     KUBERNETES_MEMORY_REQUEST_MIN_GB: z.coerce.number().min(0).default(0),
@@ -193,6 +213,16 @@ export const Env = z
 
     KUBERNETES_MEMORY_OVERHEAD_GB: z.coerce.number().min(0).optional(), // Optional memory overhead to add to the limit in GB
     KUBERNETES_SCHEDULER_NAME: z.string().optional(), // Custom scheduler name for pods
+    KUBERNETES_RUNNER_SECCOMP_PROFILE_PATH: z
+      .string()
+      .trim()
+      .min(1)
+      .default("profiles/block-io-uring.json"),
+    KUBERNETES_RUNNER_SECCOMP_PROFILE_RUNTIMES: z
+      .enum(["none", "node-24-plus", "all"])
+      .default("node-24-plus"),
+    KUBERNETES_RUNNER_SECURITY_CONTEXT: z.enum(["off", "baseline", "restricted"]).default("off"),
+    KUBERNETES_RUNNER_RUN_AS_USER: z.coerce.number().int().min(1).default(1000),
 
     // Pod DNS config — override the cluster default ndots to `KUBERNETES_POD_DNS_NDOTS`.
     // Default k8s ndots is 5: any name with fewer than 5 dots (e.g. `api.example.com`, 2 dots) is first walked
@@ -246,65 +276,13 @@ export const Env = z
       .max(100)
       .default(20),
 
-    // Schedule toleration settings - scheduled runs tolerate taints on the dedicated pool
-    // Comma-separated list of tolerations in the format: key=value:effect
-    // For Exists operator (no value): key:effect
-    KUBERNETES_SCHEDULED_RUN_TOLERATIONS: z
-      .string()
-      .transform((val, ctx) => {
-        const tolerations = val
-          .split(",")
-          .map((entry) => entry.trim())
-          .filter((entry) => entry.length > 0)
-          .map((entry) => {
-            const colonIdx = entry.lastIndexOf(":");
-            if (colonIdx === -1) {
-              ctx.addIssue({
-                code: z.ZodIssueCode.custom,
-                message: `Invalid toleration format (missing effect): "${entry}"`,
-              });
-              return z.NEVER;
-            }
+    KUBERNETES_RUNNER_TOLERATIONS: Tolerations.optional(), // every run pod
+    KUBERNETES_SCHEDULED_RUN_TOLERATIONS: Tolerations.optional(), // schedule-tree runs only
 
-            const effect = entry.slice(colonIdx + 1);
-            const validEffects = ["NoSchedule", "NoExecute", "PreferNoSchedule"];
-            if (!validEffects.includes(effect)) {
-              ctx.addIssue({
-                code: z.ZodIssueCode.custom,
-                message: `Invalid toleration effect "${effect}" in "${entry}". Must be one of: ${validEffects.join(
-                  ", "
-                )}`,
-              });
-              return z.NEVER;
-            }
-
-            const keyValue = entry.slice(0, colonIdx);
-            const eqIdx = keyValue.indexOf("=");
-            const key = eqIdx === -1 ? keyValue : keyValue.slice(0, eqIdx);
-
-            if (!key) {
-              ctx.addIssue({
-                code: z.ZodIssueCode.custom,
-                message: `Invalid toleration format (empty key): "${entry}"`,
-              });
-              return z.NEVER;
-            }
-
-            if (eqIdx === -1) {
-              return { key, operator: "Exists" as const, effect };
-            }
-
-            return {
-              key,
-              operator: "Equal" as const,
-              value: keyValue.slice(eqIdx + 1),
-              effect,
-            };
-          });
-
-        return tolerations;
-      })
-      .optional(),
+    // Per-org placement overrides, JSON keyed by the internal org id
+    // (the `org` label on run pods):
+    // {"<orgId>": {"nodeSelector": {"<key>": "<value>"}, "tolerations": "<csv or array>"}}
+    KUBERNETES_ORG_PLACEMENT_OVERRIDES: OrgPlacementOverrides,
 
     // Placement tags settings
     PLACEMENT_TAGS_ENABLED: BoolEnv.default(false),
@@ -351,6 +329,22 @@ export const Env = z
         path: ["TRIGGER_DEQUEUE_BACKPRESSURE_POD_COUNT_RELEASE"],
       });
     }
+    if (data.KUBERNETES_LARGE_MACHINE_AFFINITY_ENABLED && data.KUBERNETES_ORG_PLACEMENT_OVERRIDES) {
+      // Non-large presets carry a hard NotIn on the large-machine pool, so an org
+      // pinned to that pool could never schedule its non-large runs.
+      for (const [orgId, override] of Object.entries(data.KUBERNETES_ORG_PLACEMENT_OVERRIDES)) {
+        const pinnedPool =
+          override.nodeSelector?.[data.KUBERNETES_LARGE_MACHINE_AFFINITY_POOL_LABEL_KEY];
+
+        if (pinnedPool === data.KUBERNETES_LARGE_MACHINE_AFFINITY_POOL_LABEL_VALUE) {
+          ctx.addIssue({
+            code: z.ZodIssueCode.custom,
+            message: `Org "${orgId}" pins run pods to the large-machine pool, but non-large presets are required to stay off it, so those runs would never schedule. Use a different pool or disable KUBERNETES_LARGE_MACHINE_AFFINITY_ENABLED.`,
+            path: ["KUBERNETES_ORG_PLACEMENT_OVERRIDES"],
+          });
+        }
+      }
+    }
     if (data.COMPUTE_SNAPSHOTS_ENABLED && !data.TRIGGER_METADATA_URL) {
       ctx.addIssue({
         code: z.ZodIssueCode.custom,
@@ -365,6 +359,22 @@ export const Env = z
         path: ["TRIGGER_WORKLOAD_API_DOMAIN"],
       });
     }
+    if (data.WORKLOAD_TOKEN_ENFORCEMENT !== "disabled" && !data.WORKLOAD_TOKEN_SECRET) {
+      ctx.addIssue({
+        code: z.ZodIssueCode.custom,
+        message:
+          "WORKLOAD_TOKEN_SECRET is required when WORKLOAD_TOKEN_ENFORCEMENT is not disabled",
+        path: ["WORKLOAD_TOKEN_SECRET"],
+      });
+    }
+    if (data.DELETE_CHECKPOINTS_ON_COMPLETION && data.WORKLOAD_TOKEN_ENFORCEMENT === "disabled") {
+      ctx.addIssue({
+        code: z.ZodIssueCode.custom,
+        message:
+          "DELETE_CHECKPOINTS_ON_COMPLETION needs WORKLOAD_TOKEN_ENFORCEMENT set to log or enforce: the tenancy it deletes by comes from the deployment token, so with tokens disabled it would silently reclaim nothing",
+        path: ["DELETE_CHECKPOINTS_ON_COMPLETION"],
+      });
+    }
     if (
       data.TRIGGER_DEQUEUE_BACKPRESSURE_ENABLED &&
       !data.TRIGGER_DEQUEUE_BACKPRESSURE_REDIS_HOST

apps/supervisor/src/envUtil.test.ts

@@ -1,5 +1,11 @@
 import { describe, it, expect } from "vitest";
-import { BoolEnv, AdditionalEnvVars } from "./envUtil.js";
+import {
+  BoolEnv,
+  AdditionalEnvVars,
+  NodeLabelValue,
+  OrgPlacementOverrides,
+  Tolerations,
+} from "./envUtil.js";
 
 describe("BoolEnv", () => {
   it("should parse string 'true' as true", () => {
@@ -78,3 +84,250 @@ describe("AdditionalEnvVars", () => {
     });
   });
 });
+
+describe("NodeLabelValue", () => {
+  it("should keep a clean value untouched", () => {
+    expect(NodeLabelValue.parse("v4-worker")).toBe("v4-worker");
+  });
+
+  it("should trim surrounding whitespace, which Kubernetes would reject", () => {
+    expect(NodeLabelValue.parse(" v4-worker ")).toBe("v4-worker");
+    expect(NodeLabelValue.parse("\tv4-worker\n")).toBe("v4-worker");
+  });
+
+  it("should treat a whitespace-only value as the empty off-switch", () => {
+    expect(NodeLabelValue.parse("")).toBe("");
+    expect(NodeLabelValue.parse("   ")).toBe("");
+  });
+
+  it("should still apply a default only when unset", () => {
+    const withDefault = NodeLabelValue.default("v4-worker");
+    expect(withDefault.parse(undefined)).toBe("v4-worker");
+    expect(withDefault.parse("")).toBe("");
+  });
+
+  it("should reject a value Kubernetes would reject, rather than 422 every pod create", () => {
+    for (const invalid of ["my worker", "-bad-", "bad.", "a".repeat(64)]) {
+      expect(NodeLabelValue.safeParse(invalid).success).toBe(false);
+    }
+  });
+});
+
+describe("Tolerations", () => {
+  it("should parse key=value entries as Equal", () => {
+    expect(Tolerations.parse("dedicated=runs:NoSchedule")).toEqual([
+      { key: "dedicated", operator: "Equal", value: "runs", effect: "NoSchedule" },
+    ]);
+  });
+
+  it("should parse entries without a value as Exists", () => {
+    expect(Tolerations.parse("scheduled-runs:NoExecute")).toEqual([
+      { key: "scheduled-runs", operator: "Exists", effect: "NoExecute" },
+    ]);
+  });
+
+  it("should keep an empty value as an exact match for a valueless taint", () => {
+    expect(Tolerations.parse("dedicated=:NoSchedule")).toEqual([
+      { key: "dedicated", operator: "Equal", value: "", effect: "NoSchedule" },
+    ]);
+
+    expect(Tolerations.parse("dedicated:NoSchedule")).toEqual([
+      { key: "dedicated", operator: "Exists", effect: "NoSchedule" },
+    ]);
+  });
+
+  it("should parse an empty string as no tolerations", () => {
+    expect(Tolerations.parse("")).toEqual([]);
+    expect(Tolerations.parse("  ")).toEqual([]);
+  });
+
+  it("should skip blank entries and trim whitespace", () => {
+    expect(Tolerations.parse(" a=b:NoSchedule , ,")).toEqual([
+      { key: "a", operator: "Equal", value: "b", effect: "NoSchedule" },
+    ]);
+  });
+
+  it("should reject a missing effect, an unknown effect, and an empty key", () => {
+    for (const invalid of ["dedicated=runs", "dedicated=runs:Nope", "=runs:NoSchedule"]) {
+      expect(Tolerations.safeParse(invalid).success).toBe(false);
+    }
+  });
+
+  it("should accept a hyphenated key, a digit-suffixed key, and every effect", () => {
+    expect(
+      Tolerations.parse("capacity-1=true:PreferNoSchedule,spot:NoExecute,gpu=a10:NoSchedule")
+    ).toEqual([
+      { key: "capacity-1", operator: "Equal", value: "true", effect: "PreferNoSchedule" },
+      { key: "spot", operator: "Exists", effect: "NoExecute" },
+      { key: "gpu", operator: "Equal", value: "a10", effect: "NoSchedule" },
+    ]);
+  });
+
+  it("should accept a DNS-subdomain prefixed key", () => {
+    expect(
+      Tolerations.parse("node.cluster.x-k8s.io/machinepool=scheduled-runs:NoSchedule")
+    ).toEqual([
+      {
+        key: "node.cluster.x-k8s.io/machinepool",
+        operator: "Equal",
+        value: "scheduled-runs",
+        effect: "NoSchedule",
+      },
+    ]);
+  });
+
+  it("should reject a key or value that Kubernetes would reject at pod create", () => {
+    for (const invalid of [
+      "dedicated=prod runs:NoSchedule",
+      "ded icated=runs:NoSchedule",
+      "dedicated=-runs:NoSchedule",
+      `dedicated=${"r".repeat(64)}:NoSchedule`,
+      `${"a".repeat(64)}=runs:NoSchedule`,
+      `example.com/${"a".repeat(64)}=runs:NoSchedule`,
+      "a/b/c=runs:NoSchedule",
+      "Example.com/pool=runs:NoSchedule",
+    ]) {
+      expect(Tolerations.safeParse(invalid).success).toBe(false);
+    }
+  });
+
+  it("should bound the prefix and the name separately, as Kubernetes does", () => {
+    const longestPrefix = `${"a".repeat(63)}.${"b".repeat(63)}.${"c".repeat(63)}.${"d".repeat(61)}`;
+    expect(longestPrefix.length).toBe(253);
+
+    expect(Tolerations.parse(`${longestPrefix}/${"n".repeat(63)}=runs:NoSchedule`)).toHaveLength(1);
+    expect(Tolerations.safeParse(`${longestPrefix}a/pool=runs:NoSchedule`).success).toBe(false);
+  });
+
+  it("should tolerate whitespace around the separators", () => {
+    expect(Tolerations.parse("dedicated = runs : NoSchedule")).toEqual([
+      { key: "dedicated", operator: "Equal", value: "runs", effect: "NoSchedule" },
+    ]);
+  });
+
+  it("should reject a stray extra effect instead of folding it into the value", () => {
+    expect(Tolerations.safeParse("dedicated=runs:NoSchedule:NoExecute").success).toBe(false);
+  });
+});
+
+describe("OrgPlacementOverrides", () => {
+  it("should parse a full override with nodeSelector and tolerations", () => {
+    expect(
+      OrgPlacementOverrides.parse(
+        JSON.stringify({
+          org_123: {
+            nodeSelector: { "node.cluster.x-k8s.io/machinepool": "dedicated-pool" },
+            tolerations: "dedicated=pool:NoSchedule",
+          },
+        })
+      )
+    ).toEqual({
+      org_123: {
+        nodeSelector: { "node.cluster.x-k8s.io/machinepool": "dedicated-pool" },
+        tolerations: [{ key: "dedicated", operator: "Equal", value: "pool", effect: "NoSchedule" }],
+      },
+    });
+  });
+
+  it("should allow either half to be omitted", () => {
+    expect(
+      OrgPlacementOverrides.parse(JSON.stringify({ org_123: { nodeSelector: { pool: "a" } } }))
+    ).toEqual({ org_123: { nodeSelector: { pool: "a" } } });
+
+    expect(
+      OrgPlacementOverrides.parse(JSON.stringify({ org_123: { tolerations: "spot:NoExecute" } }))
+    ).toEqual({
+      org_123: { tolerations: [{ key: "spot", operator: "Exists", effect: "NoExecute" }] },
+    });
+
+    expect(OrgPlacementOverrides.parse(JSON.stringify({ org_123: {} }))).toEqual({ org_123: {} });
+  });
+
+  it("should reject invalid JSON at startup rather than silently skipping the override", () => {
+    for (const invalid of ["not json", "[]", '"org_123"', "{"]) {
+      expect(OrgPlacementOverrides.safeParse(invalid).success).toBe(false);
+    }
+  });
+
+  it("should treat a blank or missing value as no overrides, like the sibling settings", () => {
+    expect(OrgPlacementOverrides.parse(undefined)).toBeUndefined();
+    expect(OrgPlacementOverrides.parse("")).toBeUndefined();
+    expect(OrgPlacementOverrides.parse("   ")).toBeUndefined();
+  });
+
+  it("should accept tolerations as an array of entries, matching the Helm list shape", () => {
+    expect(
+      OrgPlacementOverrides.parse(
+        JSON.stringify({
+          org_123: { tolerations: ["dedicated=pool:NoSchedule", "spot:NoExecute"] },
+        })
+      )
+    ).toEqual({
+      org_123: {
+        tolerations: [
+          { key: "dedicated", operator: "Equal", value: "pool", effect: "NoSchedule" },
+          { key: "spot", operator: "Exists", effect: "NoExecute" },
+        ],
+      },
+    });
+  });
+
+  it("should coerce scalar node selector values to strings, as Kubernetes labels are", () => {
+    expect(
+      OrgPlacementOverrides.parse(
+        JSON.stringify({ org_123: { nodeSelector: { paid: true, replicas: 3 } } })
+      )
+    ).toEqual({ org_123: { nodeSelector: { paid: "true", replicas: "3" } } });
+  });
+
+  it("should trim whitespace around node selector keys and values", () => {
+    expect(
+      OrgPlacementOverrides.parse(
+        JSON.stringify({ org_123: { nodeSelector: { " pool ": " a " } } })
+      )
+    ).toEqual({ org_123: { nodeSelector: { pool: "a" } } });
+  });
+
+  it("should reject blank or padded org keys, since the lookup is exact", () => {
+    for (const key of [" ", " org_123", "org_123 "]) {
+      expect(OrgPlacementOverrides.safeParse(JSON.stringify({ [key]: {} })).success).toBe(false);
+    }
+  });
+
+  it("should reject an empty node selector value instead of pinning the org to nothing", () => {
+    for (const value of ["", "   "]) {
+      expect(
+        OrgPlacementOverrides.safeParse(
+          JSON.stringify({ org_123: { nodeSelector: { pool: value } } })
+        ).success
+      ).toBe(false);
+    }
+  });
+
+  it("should reject an unknown field, so a typo cannot silently drop an override", () => {
+    expect(
+      OrgPlacementOverrides.safeParse(
+        JSON.stringify({ org_123: { toleration: "dedicated=pool:NoSchedule" } })
+      ).success
+    ).toBe(false);
+  });
+
+  it("should reject a node selector key or value Kubernetes would reject", () => {
+    for (const invalid of [
+      { org_123: { nodeSelector: { "bad key": "a" } } },
+      { org_123: { nodeSelector: { pool: "bad value" } } },
+      { org_123: { nodeSelector: { "a/b/c": "a" } } },
+      { org_123: { nodeSelector: { pool: "v".repeat(64) } } },
+    ]) {
+      expect(OrgPlacementOverrides.safeParse(JSON.stringify(invalid)).success).toBe(false);
+    }
+  });
+
+  it("should reject an invalid toleration inside an override", () => {
+    expect(
+      OrgPlacementOverrides.safeParse(
+        JSON.stringify({ org_123: { tolerations: "dedicated=pool:Nope" } })
+      ).success
+    ).toBe(false);
+  });
+});

apps/supervisor/src/envUtil.ts

@@ -16,6 +16,232 @@ export const BoolEnv = baseBoolEnv as Omit<typeof baseBoolEnv, "default"> & {
   default: (value: boolean) => z.ZodDefault<typeof baseBoolEnv>;
 };
 
+const QUALIFIED_NAME = /^[A-Za-z0-9]([-A-Za-z0-9_.]*[A-Za-z0-9])?$/;
+const DNS_SUBDOMAIN = /^[a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*$/;
+const LABEL_VALUE = /^(([A-Za-z0-9][-A-Za-z0-9_.]*)?[A-Za-z0-9])?$/;
+const QUALIFIED_NAME_MAX = 63;
+const DNS_SUBDOMAIN_MAX = 253;
+const LABEL_VALUE_MAX = 63;
+
+/**
+ * isLabelValue mirrors the Kubernetes label value rules. Empty is valid upstream.
+ */
+function isLabelValue(value: string): boolean {
+  return value.length <= LABEL_VALUE_MAX && LABEL_VALUE.test(value);
+}
+
+/**
+ * isQualifiedName mirrors the Kubernetes qualified name rules used for taint and
+ * label keys: an optional DNS subdomain prefix before the slash, then the name.
+ * The two halves have different length limits and different case rules, so a
+ * single pattern with one overall bound gets both ends wrong.
+ */
+function isQualifiedName(key: string): boolean {
+  const slashIdx = key.indexOf("/");
+
+  if (slashIdx === -1) {
+    return key.length <= QUALIFIED_NAME_MAX && QUALIFIED_NAME.test(key);
+  }
+
+  const prefix = key.slice(0, slashIdx);
+  const name = key.slice(slashIdx + 1);
+
+  return (
+    prefix.length <= DNS_SUBDOMAIN_MAX &&
+    DNS_SUBDOMAIN.test(prefix) &&
+    name.length <= QUALIFIED_NAME_MAX &&
+    QUALIFIED_NAME.test(name)
+  );
+}
+
+/**
+ * A node label value. Trimmed because Kubernetes rejects surrounding whitespace
+ * outright, so a padded value fails every pod create. Deliberately no `min(1)`:
+ * empty is the off-switch, and the Helm chart ships empty by default.
+ */
+export const NodeLabelValue = z.string().trim().refine(isLabelValue, {
+  message:
+    "Must be a Kubernetes label value: alphanumeric, with dashes, underscores and dots inside, at most 63 characters",
+});
+
+/**
+ * Comma-separated pod tolerations in the format `key=value:effect`, or `key:effect`
+ * for the Exists operator. Keys and values are checked against the Kubernetes
+ * naming rules here so a typo fails at startup, rather than 422ing every single
+ * pod create with the cause buried in an API server message.
+ */
+export const Tolerations = z.string().transform((val, ctx) => {
+  return val
+    .split(",")
+    .map((entry) => entry.trim())
+    .filter((entry) => entry.length > 0)
+    .map((entry) => {
+      const colonIdx = entry.lastIndexOf(":");
+      if (colonIdx === -1) {
+        ctx.addIssue({
+          code: z.ZodIssueCode.custom,
+          message: `Invalid toleration format (missing effect): "${entry}"`,
+        });
+        return z.NEVER;
+      }
+
+      const effect = entry.slice(colonIdx + 1).trim();
+      const validEffects = ["NoSchedule", "NoExecute", "PreferNoSchedule"];
+      if (!validEffects.includes(effect)) {
+        ctx.addIssue({
+          code: z.ZodIssueCode.custom,
+          message: `Invalid toleration effect "${effect}" in "${entry}". Must be one of: ${validEffects.join(
+            ", "
+          )}`,
+        });
+        return z.NEVER;
+      }
+
+      const keyValue = entry.slice(0, colonIdx);
+      const eqIdx = keyValue.indexOf("=");
+      const key = (eqIdx === -1 ? keyValue : keyValue.slice(0, eqIdx)).trim();
+
+      if (!key) {
+        ctx.addIssue({
+          code: z.ZodIssueCode.custom,
+          message: `Invalid toleration format (empty key): "${entry}"`,
+        });
+        return z.NEVER;
+      }
+
+      if (!isQualifiedName(key)) {
+        ctx.addIssue({
+          code: z.ZodIssueCode.custom,
+          message: `Invalid toleration key "${key}" in "${entry}". Must be a Kubernetes taint key, optionally prefixed with a DNS subdomain.`,
+        });
+        return z.NEVER;
+      }
+
+      if (eqIdx === -1) {
+        return { key, operator: "Exists" as const, effect };
+      }
+
+      const value = keyValue.slice(eqIdx + 1).trim();
+      if (!value) {
+        logger.warn(
+          'Toleration has an empty value, so it matches only a taint whose value is also empty. Drop the "=" to tolerate any value of this key.',
+          { entry, key }
+        );
+      }
+
+      if (!isLabelValue(value)) {
+        ctx.addIssue({
+          code: z.ZodIssueCode.custom,
+          message: `Invalid toleration value "${value}" in "${entry}". Must be a Kubernetes label value: alphanumeric, with dashes, underscores and dots inside.`,
+        });
+        return z.NEVER;
+      }
+
+      return {
+        key,
+        operator: "Equal" as const,
+        value,
+        effect,
+      };
+    });
+});
+
+/**
+ * Scalar values are coerced: YAML/JSON easily produce `true` or `3` where a label
+ * value is meant, and Kubernetes label values are always strings. An empty value
+ * is rejected rather than passed through - as a selector it matches only nodes
+ * carrying a literal empty-valued label, which pins the org to nothing.
+ */
+const NodeSelector = z
+  .record(z.string(), z.union([z.string(), z.number(), z.boolean()]))
+  .transform((selector, ctx) => {
+    const result: Record<string, string> = {};
+
+    for (const [rawKey, rawValue] of Object.entries(selector)) {
+      const key = rawKey.trim();
+      const value = String(rawValue).trim();
+
+      if (!isQualifiedName(key)) {
+        ctx.addIssue({
+          code: z.ZodIssueCode.custom,
+          message: `Invalid node selector key "${rawKey}". Must be a Kubernetes label key, optionally prefixed with a DNS subdomain.`,
+        });
+        continue;
+      }
+
+      if (!value) {
+        ctx.addIssue({
+          code: z.ZodIssueCode.custom,
+          message: `Empty node selector value for key "${key}". Remove the key instead of blanking the value.`,
+        });
+        continue;
+      }
+
+      if (!isLabelValue(value)) {
+        ctx.addIssue({
+          code: z.ZodIssueCode.custom,
+          message: `Invalid node selector value "${value}" for key "${key}". Must be a Kubernetes label value: alphanumeric, with dashes, underscores and dots inside, at most 63 characters.`,
+        });
+        continue;
+      }
+
+      result[key] = value;
+    }
+
+    return result;
+  });
+
+/**
+ * Per-organization placement overrides for run pods, as JSON keyed by the
+ * internal org id (the `org` label on run pods):
+ * `{"<orgId>": {"nodeSelector": {"<key>": "<value>"}, "tolerations": "<csv>"}}`.
+ * Tolerations use the same CSV format as `Tolerations`, or an array of such
+ * entries. Everything is validated at startup for the same reason as
+ * tolerations above: a typo would otherwise reject every pod create for that
+ * org, with the cause buried in API errors. A blank value means no overrides.
+ */
+export const OrgPlacementOverrides = z
+  .string()
+  .optional()
+  .transform((val, ctx) => {
+    if (val === undefined || val.trim() === "") {
+      return undefined;
+    }
+
+    try {
+      return JSON.parse(val) as unknown;
+    } catch {
+      ctx.addIssue({
+        code: z.ZodIssueCode.custom,
+        message: "Invalid org placement overrides: not valid JSON",
+      });
+      return z.NEVER;
+    }
+  })
+  .pipe(
+    z
+      .record(
+        z
+          .string()
+          .min(1)
+          .refine((key) => key === key.trim() && key.trim().length > 0, {
+            message:
+              "Org override keys must not be blank or padded with whitespace; the lookup is exact",
+          }),
+        z
+          .object({
+            nodeSelector: NodeSelector.optional(),
+            tolerations: z
+              .union([z.string(), z.array(z.string())])
+              .transform((val) => (Array.isArray(val) ? val.join(",") : val))
+              .pipe(Tolerations)
+              .optional(),
+          })
+          .strict()
+      )
+      .optional()
+  );
+
 export const AdditionalEnvVars = z.preprocess((val) => {
   if (typeof val !== "string") {
     return val;

apps/supervisor/src/index.ts

@@ -21,12 +21,13 @@ import {
   CheckpointClient,
   isKubernetesEnvironment,
 } from "@trigger.dev/core/v3/serverOnly";
-import { createK8sApi, createApiserverMetricsFetcher } from "./clients/kubernetes.js";
-import { collectDefaultMetrics, Gauge, Histogram } from "prom-client";
+import { createK8sApi, createPodCountFetcher } from "./clients/kubernetes.js";
+import { collectDefaultMetrics, Counter, Gauge, Histogram } from "prom-client";
 import { register } from "./metrics.js";
 import { PodCleaner } from "./services/podCleaner.js";
 import { FailedPodHandler } from "./services/failedPodHandler.js";
 import { getWorkerToken } from "./workerToken.js";
+import { mintDeploymentToken } from "./workloadToken.js";
 import { OtlpTraceService } from "./services/otlpTraceService.js";
 import {
   WarmStartVerificationService,
@@ -59,6 +60,21 @@ const workloadCreateDuration = new Histogram({
   registers: [register],
 });
 
+const outboundRequestsTotal = new Counter({
+  name: "supervisor_outbound_request_total",
+  help: "Count of outbound HTTP requests from the supervisor, by target name, method, response status, and outcome (ok, http_error, invalid_response, network_error).",
+  labelNames: ["name", "method", "status", "outcome"],
+  registers: [register],
+});
+
+const outboundRequestDuration = new Histogram({
+  name: "supervisor_outbound_request_duration_seconds",
+  help: "Duration of outbound HTTP requests from the supervisor, by target name and outcome. Includes the HTTP client's internal retries and backoff.",
+  labelNames: ["name", "outcome"],
+  buckets: [0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1, 2, 5, 10, 11, 12.5, 15, 20, 30, 60],
+  registers: [register],
+});
+
 class ManagedSupervisor {
   private readonly workerSession: SupervisorSession;
   private readonly metricsServer?: HttpServer;
@@ -79,6 +95,8 @@ class ManagedSupervisor {
 
   private readonly isKubernetes = isKubernetesEnvironment(env.KUBERNETES_FORCE_ENABLED);
   private readonly warmStartUrl = env.TRIGGER_WARM_START_URL;
+  private readonly warmStartDispatchUrl =
+    env.TRIGGER_WARM_START_DISPATCH_URL ?? env.TRIGGER_WARM_START_URL;
 
   private readonly wideEventOpts: WideEventOptions = {
     service: "supervisor",
@@ -96,6 +114,7 @@ class ManagedSupervisor {
       COMPUTE_GATEWAY_AUTH_TOKEN,
       DOCKER_REGISTRY_PASSWORD,
       TRIGGER_DEQUEUE_BACKPRESSURE_REDIS_PASSWORD,
+      WORKLOAD_TOKEN_SECRET,
       ...envWithoutSecrets
     } = env;
 
@@ -257,14 +276,16 @@ class ManagedSupervisor {
       // RELEASE < ENGAGE is enforced in env.ts (superRefine), so it's valid here.
       const podCountGauge = new Gauge({
         name: "supervisor_cluster_pod_count",
-        help: "Total pod objects stored in the cluster, scraped for backpressure",
+        help: "Pod objects in the workload namespace, counted for backpressure",
         registers: [register],
       });
       this.backpressureMonitors.push(
         new BackpressureMonitor({
           enabled: true,
           source: new K8sPodCountSignalSource({
-            fetchMetrics: createApiserverMetricsFetcher(
+            fetchPodCount: createPodCountFetcher(
+              createK8sApi(),
+              env.KUBERNETES_NAMESPACE,
               env.TRIGGER_DEQUEUE_BACKPRESSURE_POD_COUNT_SCRAPE_TIMEOUT_MS
             ),
             engageThreshold: env.TRIGGER_DEQUEUE_BACKPRESSURE_POD_COUNT_ENGAGE,
@@ -290,8 +311,10 @@ class ManagedSupervisor {
       });
     }
 
+    const workerToken = getWorkerToken();
+
     this.workerSession = new SupervisorSession({
-      workerToken: getWorkerToken(),
+      workerToken,
       apiUrl: env.TRIGGER_API_URL,
       instanceName: env.TRIGGER_WORKER_INSTANCE_NAME,
       managedWorkerSecret: env.MANAGED_WORKER_SECRET,
@@ -318,6 +341,10 @@ class ManagedSupervisor {
       runNotificationsEnabled: env.TRIGGER_WORKLOAD_API_ENABLED,
       heartbeatIntervalSeconds: env.TRIGGER_WORKER_HEARTBEAT_INTERVAL_SECONDS,
       sendRunDebugLogs: env.SEND_RUN_DEBUG_LOGS,
+      onHttpRequestComplete: ({ name, method, status, outcome, durationMs }) => {
+        outboundRequestsTotal.inc({ name, method, status, outcome });
+        outboundRequestDuration.observe({ name, outcome }, durationMs / 1000);
+      },
       preDequeue: async () => {
         // Synchronous, hot-path-safe cached read; false when no monitors are active.
         const skipForBackpressure = this.backpressureMonitors.some((m) => m.shouldSkipDequeue());
@@ -569,6 +596,7 @@ class ManagedSupervisor {
       checkpointClient: this.checkpointClient,
       computeManager: this.computeManager,
       tracing: this.tracing,
+      snapshotCallbackSecret: workerToken,
       wideEventOpts: this.wideEventOpts,
       wideEventsNoisyRoutes: this.wideEventsNoisyRoutes,
     });
@@ -603,6 +631,15 @@ class ManagedSupervisor {
         throw new Error("Image is missing");
       }
 
+      const deploymentToken = await mintDeploymentToken({
+        deployment: message.deployment.friendlyId,
+        deployment_version: message.backgroundWorker.version,
+        environment_id: message.environment.id,
+        environment_type: message.environment.type,
+        org_id: message.organization.id,
+        project_id: message.project.id,
+      });
+
       await this.workloadManager.create({
         dequeuedAt: message.dequeuedAt,
         dequeueResponseMs: timings.dequeueResponseMs,
@@ -617,6 +654,7 @@ class ManagedSupervisor {
         deploymentFriendlyId: message.deployment.friendlyId,
         deploymentVersion: message.backgroundWorker.version,
         runtime: message.backgroundWorker.runtime,
+        deploymentToken,
         runId: message.run.id,
         runFriendlyId: message.run.friendlyId,
         version: message.version,
@@ -664,7 +702,7 @@ class ManagedSupervisor {
       return false;
     }
 
-    const warmStartUrlWithPath = new URL("/warm-start", this.warmStartUrl);
+    const warmStartUrlWithPath = new URL("/warm-start", this.warmStartDispatchUrl);
 
     const headers: Record<string, string> = {
       "Content-Type": "application/json",
@@ -677,6 +715,18 @@ class ManagedSupervisor {
       headers.traceparent = traceparent;
     }
 
+    const requestStart = performance.now();
+    const record = (
+      status: string,
+      outcome: "ok" | "http_error" | "invalid_response" | "network_error"
+    ) => {
+      outboundRequestsTotal.inc({ name: "warm_start", method: "POST", status, outcome });
+      outboundRequestDuration.observe(
+        { name: "warm_start", outcome },
+        (performance.now() - requestStart) / 1000
+      );
+    };
+
     try {
       const res = await fetch(warmStartUrlWithPath.href, {
         method: "POST",
@@ -685,8 +735,10 @@ class ManagedSupervisor {
       });
 
       if (!res.ok) {
+        record(String(res.status), "http_error");
         this.logger.error("Warm start failed", {
           runId: dequeuedMessage.run.id,
+          statusCode: res.status,
         });
         return false;
       }
@@ -695,6 +747,7 @@ class ManagedSupervisor {
       const parsedData = z.object({ didWarmStart: z.boolean() }).safeParse(data);
 
       if (!parsedData.success) {
+        record(String(res.status), "invalid_response");
         this.logger.error("Warm start response invalid", {
           runId: dequeuedMessage.run.id,
           data,
@@ -702,8 +755,11 @@ class ManagedSupervisor {
         return false;
       }
 
+      record(String(res.status), "ok");
+
       return parsedData.data.didWarmStart;
     } catch (error) {
+      record("none", "network_error");
       this.logger.error("Warm start error", {
         runId: dequeuedMessage.run.id,
         error,

apps/supervisor/src/services/computeSnapshotService.test.ts

@@ -20,13 +20,26 @@ function createService() {
     snapshot,
   } as unknown as ComputeWorkloadManager;
 
+  const submitSuspendCompletion = vi.fn(async () => ({ success: true }));
+
   const service = new ComputeSnapshotService({
     computeManager,
-    workerClient: {} as SupervisorHttpClient,
+    workerClient: { submitSuspendCompletion } as unknown as SupervisorHttpClient,
     wideEventOpts: { service: "supervisor-test", env: {}, enabled: false },
+    snapshotCallbackSecret: "test-secret",
   });
 
-  return { service, snapshot };
+  return { service, snapshot, submitSuspendCompletion };
+}
+
+function dispatchedMetadata(snapshot: {
+  mock: { calls: Array<Array<{ metadata?: Record<string, string> }>> };
+}) {
+  const metadata = snapshot.mock.calls[0]?.[0]?.metadata;
+  if (!metadata) {
+    throw new Error("Snapshot was not dispatched");
+  }
+  return metadata;
 }
 
 function delayedSnapshot(runnerId = "runner-1") {
@@ -38,6 +51,24 @@ function delayedSnapshot(runnerId = "runner-1") {
 }
 
 describe("ComputeSnapshotService", () => {
+  it("refuses to construct with an empty callback secret", () => {
+    const computeManager = {
+      snapshotDelayMs: DELAY_MS,
+      snapshotDispatchLimit: 1,
+      snapshot: vi.fn(async () => true),
+    } as unknown as ComputeWorkloadManager;
+
+    expect(
+      () =>
+        new ComputeSnapshotService({
+          computeManager,
+          workerClient: {} as SupervisorHttpClient,
+          wideEventOpts: { service: "supervisor-test", env: {}, enabled: false },
+          snapshotCallbackSecret: "",
+        })
+    ).toThrow();
+  });
+
   it("dispatches a scheduled snapshot after the delay", async () => {
     const { service, snapshot } = createService();
     try {
@@ -46,7 +77,12 @@ describe("ComputeSnapshotService", () => {
       await vi.waitFor(() => expect(snapshot).toHaveBeenCalledTimes(1), { timeout: 2_000 });
       expect(snapshot).toHaveBeenCalledWith({
         runnerId: "runner-1",
-        metadata: { runId: "run_1", snapshotFriendlyId: "snapshot_1" },
+        metadata: expect.objectContaining({
+          runId: "run_1",
+          snapshotFriendlyId: "snapshot_1",
+          snapshotCallbackNonce: expect.any(String),
+          snapshotCallbackToken: expect.any(String),
+        }),
       });
     } finally {
       service.stop();
@@ -121,10 +157,86 @@ describe("ComputeSnapshotService", () => {
       expect(snapshot).toHaveBeenCalledTimes(1);
       expect(snapshot).toHaveBeenCalledWith({
         runnerId: "runner-1",
-        metadata: { runId: "run_1", snapshotFriendlyId: "snapshot_2" },
+        metadata: expect.objectContaining({
+          runId: "run_1",
+          snapshotFriendlyId: "snapshot_2",
+          snapshotCallbackNonce: expect.any(String),
+          snapshotCallbackToken: expect.any(String),
+        }),
       });
     } finally {
       service.stop();
     }
   });
+
+  it("accepts a snapshot callback with the dispatched token", async () => {
+    const { service, snapshot, submitSuspendCompletion } = createService();
+    try {
+      service.schedule("run_1", delayedSnapshot());
+
+      await vi.waitFor(() => expect(snapshot).toHaveBeenCalledTimes(1), { timeout: 2_000 });
+      const metadata = dispatchedMetadata(snapshot);
+
+      const result = await service.handleCallback({
+        status: "completed",
+        instance_id: "instance_1",
+        snapshot_id: "compute_snapshot_1",
+        metadata,
+      });
+
+      expect(result).toEqual({ ok: true, status: 200 });
+      expect(submitSuspendCompletion).toHaveBeenCalledWith({
+        runId: "run_1",
+        snapshotId: "snapshot_1",
+        body: {
+          success: true,
+          checkpoint: {
+            type: "COMPUTE",
+            location: "compute_snapshot_1",
+          },
+        },
+      });
+    } finally {
+      service.stop();
+    }
+  });
+
+  it("rejects a snapshot callback without a valid token", async () => {
+    const { service, submitSuspendCompletion } = createService();
+    try {
+      const result = await service.handleCallback({
+        status: "completed",
+        instance_id: "instance_1",
+        snapshot_id: "compute_snapshot_1",
+        metadata: { runId: "run_1", snapshotFriendlyId: "snapshot_1" },
+      });
+
+      expect(result).toEqual({ ok: false, status: 401 });
+      expect(submitSuspendCompletion).not.toHaveBeenCalled();
+    } finally {
+      service.stop();
+    }
+  });
+
+  it("rejects a snapshot callback whose token is for a different snapshot", async () => {
+    const { service, snapshot, submitSuspendCompletion } = createService();
+    try {
+      service.schedule("run_1", delayedSnapshot());
+
+      await vi.waitFor(() => expect(snapshot).toHaveBeenCalledTimes(1), { timeout: 2_000 });
+      const metadata = dispatchedMetadata(snapshot);
+
+      const result = await service.handleCallback({
+        status: "completed",
+        instance_id: "instance_1",
+        snapshot_id: "compute_snapshot_1",
+        metadata: { ...metadata, snapshotFriendlyId: "snapshot_2" },
+      });
+
+      expect(result).toEqual({ ok: false, status: 401 });
+      expect(submitSuspendCompletion).not.toHaveBeenCalled();
+    } finally {
+      service.stop();
+    }
+  });
 });

apps/supervisor/src/services/computeSnapshotService.ts

@@ -1,3 +1,4 @@
+import { createHmac, randomBytes, timingSafeEqual } from "node:crypto";
 import pLimit from "p-limit";
 import { SimpleStructuredLogger } from "@trigger.dev/core/v3/utils/structuredLogger";
 import { parseTraceparent } from "@trigger.dev/core/v3/isomorphic";
@@ -16,6 +17,13 @@ import {
   type WideEventOptions,
 } from "../wideEvents/index.js";
 
+const SNAPSHOT_CALLBACK_NONCE_METADATA_KEY = "snapshotCallbackNonce";
+const SNAPSHOT_CALLBACK_TOKEN_METADATA_KEY = "snapshotCallbackToken";
+
+// Domain-separation label so the callback-signing key is derived from, rather
+// than equal to, the secret used for other protocols. Bump the suffix to rotate.
+const SNAPSHOT_CALLBACK_KEY_INFO = "compute-snapshot-callback-v1";
+
 type DelayedSnapshot = {
   runnerId: string;
   runFriendlyId: string;
@@ -34,6 +42,7 @@ export type ComputeSnapshotServiceOptions = {
   workerClient: SupervisorHttpClient;
   tracing?: OtlpTraceService;
   wideEventOpts: WideEventOptions;
+  snapshotCallbackSecret: string;
 };
 
 export class ComputeSnapshotService {
@@ -48,6 +57,7 @@ export class ComputeSnapshotService {
   private readonly workerClient: SupervisorHttpClient;
   private readonly tracing?: OtlpTraceService;
   private readonly wideEventOpts: WideEventOptions;
+  private readonly snapshotCallbackKey: Buffer;
 
   constructor(opts: ComputeSnapshotServiceOptions) {
     this.computeManager = opts.computeManager;
@@ -55,6 +65,18 @@ export class ComputeSnapshotService {
     this.tracing = opts.tracing;
     this.wideEventOpts = opts.wideEventOpts;
 
+    // Reject an empty secret up front: an empty HMAC key would make callback
+    // tokens forgeable by anyone. Guarding here (rather than only at env parse)
+    // also covers the case where the secret is read from an empty file.
+    if (!opts.snapshotCallbackSecret) {
+      throw new Error("snapshotCallbackSecret must not be empty");
+    }
+    // Derive a dedicated key by domain separation so the raw secret is never
+    // used directly as a MAC key for this protocol.
+    this.snapshotCallbackKey = createHmac("sha256", opts.snapshotCallbackSecret)
+      .update(SNAPSHOT_CALLBACK_KEY_INFO)
+      .digest();
+
     this.dispatchLimit = pLimit(this.computeManager.snapshotDispatchLimit);
     this.timerWheel = new TimerWheel<DelayedSnapshot>({
       delayMs: this.computeManager.snapshotDelayMs,
@@ -146,15 +168,29 @@ export class ComputeSnapshotService {
       instanceId: body.instance_id,
       status: body.status,
       error: body.status === "failed" ? body.error : undefined,
-      metadata: body.metadata,
+      runId,
+      snapshotFriendlyId,
       durationMs: body.duration_ms,
     });
 
     if (!runId || !snapshotFriendlyId) {
-      this.logger.error("Snapshot callback missing metadata", { body });
+      this.logger.error("Snapshot callback missing metadata", {
+        status: body.status,
+        instanceId: body.instance_id,
+        metadataKeys: Object.keys(body.metadata ?? {}),
+      });
       return { ok: false as const, status: 400 };
     }
 
+    if (!this.#verifyCallbackToken(body.metadata, runId, snapshotFriendlyId)) {
+      this.logger.error("Snapshot callback failed token verification", {
+        runId,
+        snapshotFriendlyId,
+        instanceId: body.instance_id,
+      });
+      return { ok: false as const, status: 401 };
+    }
+
     this.#emitSnapshotSpan(runId, body.duration_ms, snapshotId);
 
     if (body.status === "completed") {
@@ -266,11 +302,18 @@ export class ComputeSnapshotService {
         },
       },
       async () => {
+        const callbackNonce = randomBytes(16).toString("hex");
         const result = await this.computeManager.snapshot({
           runnerId: snapshot.runnerId,
           metadata: {
             runId: snapshot.runFriendlyId,
             snapshotFriendlyId: snapshot.snapshotFriendlyId,
+            [SNAPSHOT_CALLBACK_NONCE_METADATA_KEY]: callbackNonce,
+            [SNAPSHOT_CALLBACK_TOKEN_METADATA_KEY]: this.#createCallbackToken(
+              callbackNonce,
+              snapshot.runFriendlyId,
+              snapshot.snapshotFriendlyId
+            ),
           },
         });
 
@@ -281,6 +324,51 @@ export class ComputeSnapshotService {
     );
   }
 
+  #createCallbackToken(nonce: string, runFriendlyId: string, snapshotFriendlyId: string): string {
+    return createHmac("sha256", this.snapshotCallbackKey)
+      .update(nonce)
+      .update("\0")
+      .update(runFriendlyId)
+      .update("\0")
+      .update(snapshotFriendlyId)
+      .digest("hex");
+  }
+
+  /**
+   * Verify that a callback carries a token this supervisor issued for the given
+   * run and snapshot. The token binds only the identifiers known at dispatch
+   * time (nonce, run, snapshot); it intentionally does not cover result fields
+   * such as the snapshot location or status/error, which are produced by the
+   * gateway after the snapshot and so cannot be signed in advance. Verification
+   * is also stateless, so a token is not single-use.
+   *
+   * This closes the primary risk (a caller that can merely reach the endpoint
+   * cannot mint a valid token, so cannot forge a result for an arbitrary run).
+   * It does not defend against an attacker who can observe a genuine callback
+   * and then replay it or alter its unsigned result fields - that relies on the
+   * gateway->supervisor callback channel being authenticated and encrypted.
+   */
+  #verifyCallbackToken(
+    metadata: Record<string, string> | undefined,
+    runFriendlyId: string,
+    snapshotFriendlyId: string
+  ): boolean {
+    const nonce = metadata?.[SNAPSHOT_CALLBACK_NONCE_METADATA_KEY];
+    const token = metadata?.[SNAPSHOT_CALLBACK_TOKEN_METADATA_KEY];
+
+    if (!nonce || !token) {
+      return false;
+    }
+
+    const expected = this.#createCallbackToken(nonce, runFriendlyId, snapshotFriendlyId);
+    const expectedBuffer = Buffer.from(expected, "hex");
+    const tokenBuffer = Buffer.from(token, "hex");
+
+    return (
+      expectedBuffer.length === tokenBuffer.length && timingSafeEqual(expectedBuffer, tokenBuffer)
+    );
+  }
+
   #emitSnapshotSpan(runFriendlyId: string, durationMs?: number, snapshotId?: string) {
     if (!this.tracing) return;
 

apps/supervisor/src/wideEvents/index.ts

@@ -7,18 +7,14 @@
  * Off by default behind a kill switch - the dispatch hotpath runs at high
  * QPS, so logging pressure must be cleanly removable.
  */
-export { type Env, isValidRequestId, newState, type NewStateOptions } from "./new.js";
-export { emit, EmitMessage } from "./emit.js";
-export { parseTraceId } from "./traceparent.js";
-export { fromContext, wideEventStorage } from "./context.js";
-export { type PhaseOpt, recordPhase, recordPhaseSince, timePhase } from "./record.js";
+export { fromContext } from "./context.js";
+export { recordPhaseSince } from "./record.js";
 export {
   emitOneShot,
   runWideEvent,
   setExtra,
   setMeta,
-  type WideEventLifecycleOptions,
   type WideEventOptions,
 } from "./middleware.js";
-export type { ErrorInfo, PhaseRecord, State } from "./state.js";
+export type { State } from "./state.js";
 export { encodeBaggage } from "./baggage.js";

apps/supervisor/src/wideEvents/state.ts

@@ -76,7 +76,7 @@ export type PhaseRecord = {
 };
 
 /** Top-level error summary for a failed operation. */
-export type ErrorInfo = {
+type ErrorInfo = {
   code: string;
   message: string;
   /** Coarse classification - "client" | "upstream" | "internal" | "timeout". */

apps/supervisor/src/workloadManager/compute.ts

@@ -151,7 +151,9 @@ export class ComputeWorkloadManager implements WorkloadManager {
       TRIGGER_DEQUEUED_AT_MS: String(opts.dequeuedAt.getTime()),
       TRIGGER_POD_SCHEDULED_AT_MS: String(Date.now()),
       TRIGGER_ENV_ID: opts.envId,
-      TRIGGER_DEPLOYMENT_ID: opts.deploymentFriendlyId,
+      TRIGGER_DEPLOYMENT_ID: opts.deploymentToken ?? opts.deploymentFriendlyId,
+      // Plain friendlyId for telemetry (worker.id), so it isn't the opaque token in DEPLOYMENT_ID.
+      TRIGGER_DEPLOYMENT_FRIENDLY_ID: opts.deploymentFriendlyId,
       TRIGGER_DEPLOYMENT_VERSION: opts.deploymentVersion,
       TRIGGER_RUN_ID: opts.runFriendlyId,
       TRIGGER_SNAPSHOT_ID: opts.snapshotFriendlyId,

apps/supervisor/src/workloadManager/docker.ts

@@ -72,7 +72,9 @@ export class DockerWorkloadManager implements WorkloadManager {
       `TRIGGER_DEQUEUED_AT_MS=${opts.dequeuedAt.getTime()}`,
       `TRIGGER_POD_SCHEDULED_AT_MS=${Date.now()}`,
       `TRIGGER_ENV_ID=${opts.envId}`,
-      `TRIGGER_DEPLOYMENT_ID=${opts.deploymentFriendlyId}`,
+      `TRIGGER_DEPLOYMENT_ID=${opts.deploymentToken ?? opts.deploymentFriendlyId}`,
+      // Plain friendlyId for telemetry (worker.id), so it isn't the opaque token in DEPLOYMENT_ID.
+      `TRIGGER_DEPLOYMENT_FRIENDLY_ID=${opts.deploymentFriendlyId}`,
       `TRIGGER_DEPLOYMENT_VERSION=${opts.deploymentVersion}`,
       `TRIGGER_RUN_ID=${opts.runFriendlyId}`,
       `TRIGGER_SNAPSHOT_ID=${opts.snapshotFriendlyId}`,

apps/supervisor/src/workloadManager/imageRegistry.test.ts

@@ -0,0 +1,42 @@
+import { describe, expect, it } from "vitest";
+import { rewriteImageRegistry } from "./imageRegistry.js";
+
+const FROM = "123456789012.dkr.ecr.us-east-1.amazonaws.com";
+const TO = "123456789012.dkr.ecr.eu-central-1.amazonaws.com";
+
+describe("rewriteImageRegistry", () => {
+  it("rewrites the registry host and keeps the rest of the reference", () => {
+    expect(rewriteImageRegistry(`${FROM}/deployments/proj_abc:20260818.1`, FROM, TO)).toBe(
+      `${TO}/deployments/proj_abc:20260818.1`
+    );
+  });
+
+  it("preserves a digest", () => {
+    expect(rewriteImageRegistry(`${FROM}/deployments/proj_abc@sha256:abc123`, FROM, TO)).toBe(
+      `${TO}/deployments/proj_abc@sha256:abc123`
+    );
+  });
+
+  it("is a no-op unless both ends are configured", () => {
+    const ref = `${FROM}/deployments/proj_abc:tag`;
+
+    expect(rewriteImageRegistry(ref, undefined, TO)).toBe(ref);
+    expect(rewriteImageRegistry(ref, FROM, undefined)).toBe(ref);
+    expect(rewriteImageRegistry(ref, undefined, undefined)).toBe(ref);
+  });
+
+  it("leaves other registries alone", () => {
+    const ref = "ghcr.io/triggerdotdev/something:tag";
+    expect(rewriteImageRegistry(ref, FROM, TO)).toBe(ref);
+  });
+
+  it("only matches on a host boundary", () => {
+    const lookalike = `${FROM}.evil.example.com/deployments/proj_abc:tag`;
+    expect(rewriteImageRegistry(lookalike, FROM, TO)).toBe(lookalike);
+  });
+
+  it("does not rewrite a host that merely contains the source", () => {
+    const ref = `registry.example.com/${FROM}/proj_abc:tag`;
+    expect(rewriteImageRegistry(ref, FROM, TO)).toBe(ref);
+  });
+});

apps/supervisor/src/workloadManager/imageRegistry.ts

@@ -0,0 +1,15 @@
+export function rewriteImageRegistry(
+  imageRef: string,
+  from: string | undefined,
+  to: string | undefined
+): string {
+  if (!from || !to) {
+    return imageRef;
+  }
+
+  if (!imageRef.startsWith(`${from}/`)) {
+    return imageRef;
+  }
+
+  return `${to}${imageRef.slice(from.length)}`;
+}

apps/supervisor/src/workloadManager/kubernetes.test.ts

@@ -1,7 +1,10 @@
 import { describe, expect, it } from "vitest";
 import {
-  BLOCK_IO_URING_SECCOMP_PROFILE,
-  withBlockIoUringSeccompProfile,
+  nodetypeNodeSelector,
+  runPodTolerations,
+  runnerSecurityContext,
+  withRunnerSeccompProfile,
+  withNodeSelector,
 } from "./kubernetesPodSpec.js";
 
 const basePodSpec = {
@@ -14,27 +17,172 @@ const basePodSpec = {
   },
 };
 
-describe("withBlockIoUringSeccompProfile", () => {
-  it("adds the Localhost io_uring profile for node-24 and above, preserving pod security defaults", () => {
-    for (const runtime of ["node-24", "node-26", "node-30", "experimental-node-24"]) {
-      const podSpec = withBlockIoUringSeccompProfile(basePodSpec, runtime);
-
-      expect(podSpec).toMatchObject({
-        ...basePodSpec,
-        securityContext: {
-          ...basePodSpec.securityContext,
-          seccompProfile: {
-            type: "Localhost",
-            localhostProfile: BLOCK_IO_URING_SECCOMP_PROFILE,
-          },
-        },
-      });
+describe("nodetypeNodeSelector", () => {
+  it("omits the nodeSelector entirely when the label is empty or unset", () => {
+    for (const label of ["", undefined]) {
+      expect(nodetypeNodeSelector(label)).toEqual({});
     }
   });
 
-  it("leaves the pod spec unchanged for runtimes that do not create io_uring fds", () => {
+  it("pins to nodetype=<label> when set", () => {
+    expect(nodetypeNodeSelector("v4-worker")).toEqual({ nodeSelector: { nodetype: "v4-worker" } });
+  });
+});
+
+describe("runPodTolerations", () => {
+  const worker = [{ key: "dedicated", operator: "Equal", value: "runs", effect: "NoSchedule" }];
+  const scheduled = [{ key: "scheduled-runs", operator: "Exists", effect: "NoSchedule" }];
+
+  it("leaves tolerations unset when neither is configured", () => {
+    expect(runPodTolerations(undefined, undefined, false)).toBeUndefined();
+    expect(runPodTolerations(undefined, undefined, true)).toBeUndefined();
+    expect(runPodTolerations([], [], true)).toBeUndefined();
+  });
+
+  it("applies the worker tolerations to every run", () => {
+    expect(runPodTolerations(worker, undefined, false)).toEqual(worker);
+    expect(runPodTolerations(worker, undefined, true)).toEqual(worker);
+  });
+
+  it("applies the scheduled-run tolerations on their own, as before this option existed", () => {
+    expect(runPodTolerations(undefined, scheduled, true)).toEqual(scheduled);
+    expect(runPodTolerations(undefined, scheduled, false)).toBeUndefined();
+    expect(runPodTolerations([], scheduled, true)).toEqual(scheduled);
+  });
+
+  it("adds the scheduled-run tolerations only for scheduled runs", () => {
+    expect(runPodTolerations(worker, scheduled, false)).toEqual(worker);
+    expect(runPodTolerations(worker, [], true)).toEqual(worker);
+    expect(runPodTolerations(worker, scheduled, true)).toEqual([...worker, ...scheduled]);
+  });
+
+  it("appends the org tolerations regardless of run type", () => {
+    const org = [{ key: "dedicated", operator: "Equal", value: "org-pool", effect: "NoSchedule" }];
+
+    expect(runPodTolerations(undefined, undefined, false, org)).toEqual(org);
+    expect(runPodTolerations(worker, undefined, false, org)).toEqual([...worker, ...org]);
+    expect(runPodTolerations(worker, scheduled, true, org)).toEqual([
+      ...worker,
+      ...scheduled,
+      ...org,
+    ]);
+    expect(runPodTolerations(undefined, undefined, false, [])).toBeUndefined();
+  });
+});
+
+describe("withNodeSelector", () => {
+  const podSpec = { ...basePodSpec, nodeSelector: { nodetype: "v4-worker", paid: "true" } };
+
+  it("returns the pod spec untouched when there is nothing to merge", () => {
+    expect(withNodeSelector(podSpec, undefined)).toBe(podSpec);
+    expect(withNodeSelector(podSpec, {})).toBe(podSpec);
+  });
+
+  it("merges extra entries with existing ones", () => {
+    expect(withNodeSelector(podSpec, { machinepool: "dedicated-pool" })).toEqual({
+      ...podSpec,
+      nodeSelector: { nodetype: "v4-worker", paid: "true", machinepool: "dedicated-pool" },
+    });
+  });
+
+  it("lets the extra entries win on key collision", () => {
+    expect(withNodeSelector(podSpec, { nodetype: "other" }).nodeSelector).toEqual({
+      nodetype: "other",
+      paid: "true",
+    });
+  });
+
+  it("adds a nodeSelector to a spec that had none", () => {
+    expect(withNodeSelector(basePodSpec, { machinepool: "dedicated-pool" })).toEqual({
+      ...basePodSpec,
+      nodeSelector: { machinepool: "dedicated-pool" },
+    });
+  });
+});
+
+describe("withRunnerSeccompProfile", () => {
+  const base = {
+    profilePath: "profiles/example.json",
+    runtimes: "node-24-plus" as const,
+    runtime: "node-24",
+    checkpointsEnabled: true,
+  };
+
+  const withProfile = {
+    ...basePodSpec,
+    securityContext: {
+      ...basePodSpec.securityContext,
+      seccompProfile: { type: "Localhost", localhostProfile: "profiles/example.json" },
+    },
+  };
+
+  it("applies the profile to node-24 and above under the default scope", () => {
+    for (const runtime of ["node-24", "node-26", "node-30", "experimental-node-24"]) {
+      expect(withRunnerSeccompProfile(basePodSpec, { ...base, runtime })).toMatchObject(
+        withProfile
+      );
+    }
+  });
+
+  it("skips older runtimes under the default scope", () => {
     for (const runtime of ["node", "node-22", "bun", undefined, null, ""]) {
-      expect(withBlockIoUringSeccompProfile(basePodSpec, runtime)).toEqual(basePodSpec);
+      expect(withRunnerSeccompProfile(basePodSpec, { ...base, runtime })).toBe(basePodSpec);
+    }
+  });
+
+  it("applies the profile to every runtime under the all scope", () => {
+    for (const runtime of ["node", "node-22", "bun", "node-24", undefined]) {
+      expect(
+        withRunnerSeccompProfile(basePodSpec, { ...base, runtimes: "all", runtime })
+      ).toMatchObject(withProfile);
+    }
+  });
+
+  it("applies nothing under the none scope, whatever the runtime", () => {
+    for (const runtime of ["node-24", "bun", "node-22"]) {
+      expect(withRunnerSeccompProfile(basePodSpec, { ...base, runtimes: "none", runtime })).toBe(
+        basePodSpec
+      );
+    }
+  });
+
+  it("applies nothing when checkpoints are disabled", () => {
+    for (const runtimes of ["none", "node-24-plus", "all"] as const) {
+      expect(
+        withRunnerSeccompProfile(basePodSpec, { ...base, runtimes, checkpointsEnabled: false })
+      ).toBe(basePodSpec);
+    }
+  });
+});
+
+describe("runnerSecurityContext", () => {
+  it("sets nothing when off", () => {
+    expect(runnerSecurityContext("off", 1000, "node-24")).toBeUndefined();
+  });
+
+  it("drops all capabilities and blocks escalation at baseline", () => {
+    expect(runnerSecurityContext("baseline", 1000, "node-24")).toEqual({
+      allowPrivilegeEscalation: false,
+      capabilities: { drop: ["ALL"] },
+    });
+  });
+
+  it("pins the configured uid when restricted", () => {
+    expect(runnerSecurityContext("restricted", 1000, "node-24")).toEqual({
+      allowPrivilegeEscalation: false,
+      capabilities: { drop: ["ALL"] },
+      runAsNonRoot: true,
+      runAsUser: 1000,
+    });
+  });
+
+  it("pins bun's own uid, which differs from node's", () => {
+    expect(runnerSecurityContext("restricted", 1000, "bun")?.runAsUser).toBe(1001);
+  });
+
+  it("falls back to the configured uid when the runtime is unknown", () => {
+    for (const runtime of [undefined, null, "", "node", "node-22", "node-26"]) {
+      expect(runnerSecurityContext("restricted", 1000, runtime)?.runAsUser).toBe(1000);
     }
   });
 });

apps/supervisor/src/workloadManager/kubernetes.ts

@@ -14,7 +14,14 @@ import { PlacementTagProcessor } from "@trigger.dev/core/v3/serverOnly";
 import { env } from "../env.js";
 import { type K8sApi, createK8sApi, type k8s } from "../clients/kubernetes.js";
 import { getRunnerId } from "../util.js";
-import { withBlockIoUringSeccompProfile } from "./kubernetesPodSpec.js";
+import {
+  nodetypeNodeSelector,
+  runPodTolerations,
+  runnerSecurityContext,
+  withRunnerSeccompProfile,
+  withNodeSelector,
+} from "./kubernetesPodSpec.js";
+import { rewriteImageRegistry } from "./imageRegistry.js";
 
 type ResourceQuantities = {
   [K in "cpu" | "memory" | "ephemeral-storage"]?: string;
@@ -65,6 +72,12 @@ export class KubernetesWorkloadManager implements WorkloadManager {
         domain: opts.workloadApiDomain,
       });
     }
+
+    if (env.KUBERNETES_ORG_PLACEMENT_OVERRIDES) {
+      this.logger.info("[KubernetesWorkloadManager] Org placement overrides enabled", {
+        orgIds: Object.keys(env.KUBERNETES_ORG_PLACEMENT_OVERRIDES),
+      });
+    }
   }
 
   private addPlacementTags(
@@ -106,10 +119,30 @@ export class KubernetesWorkloadManager implements WorkloadManager {
     const runnerId = getRunnerId(opts.runFriendlyId, opts.nextAttemptNumber);
 
     try {
-      const basePodSpec = this.addPlacementTags(this.#defaultPodSpec, opts.placementTags);
-      const podSpec = this.opts.checkpointsEnabled
-        ? withBlockIoUringSeccompProfile(basePodSpec, opts.runtime)
-        : basePodSpec;
+      const orgOverride = env.KUBERNETES_ORG_PLACEMENT_OVERRIDES?.[opts.orgId];
+      const taggedPodSpec = this.addPlacementTags(this.#defaultPodSpec, opts.placementTags);
+      const basePodSpec = withNodeSelector(taggedPodSpec, orgOverride?.nodeSelector);
+
+      if (orgOverride?.nodeSelector) {
+        const replacedKeys = Object.keys(orgOverride.nodeSelector).filter(
+          (key) =>
+            taggedPodSpec.nodeSelector?.[key] !== undefined &&
+            taggedPodSpec.nodeSelector[key] !== orgOverride.nodeSelector?.[key]
+        );
+
+        if (replacedKeys.length > 0) {
+          this.logger.warn(
+            "[KubernetesWorkloadManager] Org placement override replaces node selector keys",
+            { orgId: opts.orgId, replacedKeys }
+          );
+        }
+      }
+      const podSpec = withRunnerSeccompProfile(basePodSpec, {
+        profilePath: env.KUBERNETES_RUNNER_SECCOMP_PROFILE_PATH,
+        runtimes: env.KUBERNETES_RUNNER_SECCOMP_PROFILE_RUNTIMES,
+        runtime: opts.runtime,
+        checkpointsEnabled: this.opts.checkpointsEnabled,
+      });
 
       await this.k8s.core.createNamespacedPod({
         namespace: this.namespace,
@@ -127,18 +160,27 @@ export class KubernetesWorkloadManager implements WorkloadManager {
           spec: {
             ...podSpec,
             affinity: this.#getAffinity(opts),
-            tolerations: this.#getScheduleTolerations(this.#isScheduledRun(opts)),
+            tolerations: this.#getTolerations(this.#isScheduledRun(opts), orgOverride?.tolerations),
             terminationGracePeriodSeconds: 60 * 60,
             containers: [
               {
                 name: "run-controller",
-                image: this.stripImageDigest(opts.image),
+                image: rewriteImageRegistry(
+                  this.stripImageDigest(opts.image),
+                  env.KUBERNETES_IMAGE_REGISTRY_REWRITE_FROM,
+                  env.KUBERNETES_IMAGE_REGISTRY_REWRITE_TO
+                ),
                 ports: [
                   {
                     containerPort: 8000,
                   },
                 ],
                 resources: this.#getResourcesForMachine(opts.machine),
+                securityContext: runnerSecurityContext(
+                  env.KUBERNETES_RUNNER_SECURITY_CONTEXT,
+                  env.KUBERNETES_RUNNER_RUN_AS_USER,
+                  opts.runtime
+                ),
                 env: [
                   {
                     name: "TRIGGER_DEQUEUED_AT_MS",
@@ -158,6 +200,11 @@ export class KubernetesWorkloadManager implements WorkloadManager {
                   },
                   {
                     name: "TRIGGER_DEPLOYMENT_ID",
+                    value: opts.deploymentToken ?? opts.deploymentFriendlyId,
+                  },
+                  {
+                    // Plain friendlyId for telemetry (worker.id), not the opaque token in DEPLOYMENT_ID.
+                    name: "TRIGGER_DEPLOYMENT_FRIENDLY_ID",
                     value: opts.deploymentFriendlyId,
                   },
                   {
@@ -324,13 +371,12 @@ export class KubernetesWorkloadManager implements WorkloadManager {
             schedulerName: env.KUBERNETES_SCHEDULER_NAME,
           }
         : {}),
-      ...(env.KUBERNETES_WORKER_NODETYPE_LABEL
+      ...(env.KUBERNETES_RUN_POD_PRIORITY_CLASS_NAME
         ? {
-            nodeSelector: {
-              nodetype: env.KUBERNETES_WORKER_NODETYPE_LABEL,
-            },
+            priorityClassName: env.KUBERNETES_RUN_POD_PRIORITY_CLASS_NAME,
           }
         : {}),
+      ...nodetypeNodeSelector(env.KUBERNETES_WORKER_NODETYPE_LABEL),
       ...(env.KUBERNETES_POD_DNS_NDOTS_OVERRIDE_ENABLED
         ? {
             dnsConfig: {
@@ -552,12 +598,16 @@ export class KubernetesWorkloadManager implements WorkloadManager {
     };
   }
 
-  #getScheduleTolerations(isScheduledRun: boolean): k8s.V1Toleration[] | undefined {
-    if (!isScheduledRun || !env.KUBERNETES_SCHEDULED_RUN_TOLERATIONS?.length) {
-      return undefined;
-    }
-
-    return env.KUBERNETES_SCHEDULED_RUN_TOLERATIONS;
+  #getTolerations(
+    isScheduledRun: boolean,
+    orgTolerations?: k8s.V1Toleration[]
+  ): k8s.V1Toleration[] | undefined {
+    return runPodTolerations(
+      env.KUBERNETES_RUNNER_TOLERATIONS,
+      env.KUBERNETES_SCHEDULED_RUN_TOLERATIONS,
+      isScheduledRun,
+      orgTolerations
+    );
   }
 
   #getProjectPodAffinity(projectId: string): k8s.V1PodAffinity | undefined {

apps/supervisor/src/workloadManager/kubernetesPodSpec.ts

@@ -1,33 +1,128 @@
 import type { k8s } from "../clients/kubernetes.js";
 
 /**
- * Relative path (kubelet seccomp root) of the profile blocking only io_uring
- * syscalls. Must match the profile deployed to worker nodes.
+ * An empty label is the documented off-switch, leaving the pod unpinned. The Helm
+ * chart ships an empty value, so don't collapse this into a fallback default -
+ * that would pin every chart install to a label its nodes don't carry.
  */
-export const BLOCK_IO_URING_SECCOMP_PROFILE = "profiles/block-io-uring.json";
+export function nodetypeNodeSelector(
+  label: string | undefined
+): Pick<k8s.V1PodSpec, "nodeSelector"> {
+  return label ? { nodeSelector: { nodetype: label } } : {};
+}
 
 /**
- * Node >= 24 always creates io_uring fds, which can't be checkpointed. Blocking
- * io_uring_setup makes libuv fall back to epoll. Other runtimes don't need this,
- * so the profile is only applied for node-24+. Tolerates an "experimental-" prefix.
+ * Tolerations for a run pod: the cluster-wide set, plus the scheduled-run set when the
+ * run came from a schedule tree, plus the org's own set when a placement override
+ * matches. Not reconciled - Kubernetes matches tolerations as an any-match set, so a
+ * broad entry in one set can subsume a narrower one in another.
+ * Returns undefined rather than an empty array to leave the field unset.
  */
-export function withBlockIoUringSeccompProfile(
+export function runPodTolerations(
+  runnerTolerations: k8s.V1Toleration[] | undefined,
+  scheduledRunTolerations: k8s.V1Toleration[] | undefined,
+  isScheduledRun: boolean,
+  orgTolerations?: k8s.V1Toleration[]
+): k8s.V1Toleration[] | undefined {
+  const tolerations = [
+    ...(runnerTolerations ?? []),
+    ...(isScheduledRun ? (scheduledRunTolerations ?? []) : []),
+    ...(orgTolerations ?? []),
+  ];
+
+  return tolerations.length > 0 ? tolerations : undefined;
+}
+
+/**
+ * Merges extra node selector entries into a pod spec. Later entries win on key
+ * collision, so an override can retarget a key set by an earlier stage.
+ */
+export function withNodeSelector(
   podSpec: Omit<k8s.V1PodSpec, "containers">,
-  runtime: string | null | undefined
+  nodeSelector: Record<string, string> | undefined
 ): Omit<k8s.V1PodSpec, "containers"> {
-  const match = runtime ? /^(?:experimental-)?node-(\d+)$/.exec(runtime) : null;
-  if (!match || Number(match[1]) < 24) {
+  if (!nodeSelector || Object.keys(nodeSelector).length === 0) {
     return podSpec;
   }
 
+  return {
+    ...podSpec,
+    nodeSelector: {
+      ...podSpec.nodeSelector,
+      ...nodeSelector,
+    },
+  };
+}
+
+export type RunnerSeccompProfileOptions = {
+  profilePath: string;
+  runtimes: "none" | "node-24-plus" | "all";
+  runtime: string | null | undefined;
+  checkpointsEnabled: boolean | undefined;
+};
+
+/**
+ * Applies the runner seccomp profile, which is a node-local file installed outside
+ * this repo - pointing a pod at a profile its node doesn't have fails pod creation,
+ * so every condition for skipping it lives here.
+ *
+ * "node-24-plus" matches the original rollout: node >= 24 always creates io_uring
+ * fds, which can't be checkpointed, and blocking io_uring_setup makes libuv fall
+ * back to epoll. Tolerates an "experimental-" prefix. "bun" matches only under "all".
+ */
+export function withRunnerSeccompProfile(
+  podSpec: Omit<k8s.V1PodSpec, "containers">,
+  options: RunnerSeccompProfileOptions
+): Omit<k8s.V1PodSpec, "containers"> {
+  if (!options.checkpointsEnabled || options.runtimes === "none") {
+    return podSpec;
+  }
+
+  if (options.runtimes === "node-24-plus") {
+    const match = options.runtime ? /^(?:experimental-)?node-(\d+)$/.exec(options.runtime) : null;
+    if (!match || Number(match[1]) < 24) {
+      return podSpec;
+    }
+  }
+
   return {
     ...podSpec,
     securityContext: {
       ...podSpec.securityContext,
       seccompProfile: {
         type: "Localhost",
-        localhostProfile: BLOCK_IO_URING_SECCOMP_PROFILE,
+        localhostProfile: options.profilePath,
       },
     },
   };
 }
+
+const BUN_RUN_AS_USER = 1001;
+
+/**
+ * runnerSecurityContext maps a configured level onto the run container's security
+ * context. "baseline" drops the capability bounding set and blocks setuid
+ * escalation; "restricted" additionally pins the container to a non-root uid.
+ *
+ * The uid is set explicitly rather than read from the image: the kubelet cannot
+ * verify `runAsNonRoot` against an image that declares a named user, and fails
+ * the container instead. Bun images carry their user at a different uid to
+ * node's, so the runtime selects which uid is pinned.
+ */
+export function runnerSecurityContext(
+  level: "off" | "baseline" | "restricted",
+  runAsUser: number,
+  runtime: string | null | undefined
+): k8s.V1SecurityContext | undefined {
+  if (level === "off") {
+    return undefined;
+  }
+
+  return {
+    allowPrivilegeEscalation: false,
+    capabilities: { drop: ["ALL"] },
+    ...(level === "restricted"
+      ? { runAsNonRoot: true, runAsUser: runtime === "bun" ? BUN_RUN_AS_USER : runAsUser }
+      : {}),
+  };
+}

apps/supervisor/src/workloadManager/types.ts

@@ -44,6 +44,8 @@ export interface WorkloadManagerCreateOptions {
   deploymentVersion: string;
   // Canonical runtime identifier (e.g. "node", "node-22", "node-24")
   runtime?: string;
+  // When set, overrides the TRIGGER_DEPLOYMENT_ID value the runner forwards as its identity header.
+  deploymentToken?: string;
   runId: string;
   runFriendlyId: string;
   snapshotId: string;

apps/supervisor/src/workloadServer/index.ts

@@ -23,8 +23,16 @@ import EventEmitter from "node:events";
 import type { IncomingMessage, ServerResponse } from "node:http";
 import { type Namespace, Server, type Socket } from "socket.io";
 import { z } from "zod";
+import { tryCatch } from "@trigger.dev/core/utils";
+import { Counter } from "prom-client";
 import { env } from "../env.js";
 import { register } from "../metrics.js";
+import {
+  verifyDeploymentIdHeader,
+  workloadTokenEnforced,
+  workloadTokensEnabled,
+} from "../workloadToken.js";
+import type { WorkloadDeploymentTokenClaims } from "@trigger.dev/core/v3";
 import {
   ComputeSnapshotService,
   type RunTraceContext,
@@ -45,6 +53,20 @@ interface DefaultEventsMap {
   [event: string]: (...args: any[]) => void;
 }
 
+const checkpointDeleteRequests = new Counter({
+  name: "checkpoint_delete_requests_total",
+  help: "Checkpoint delete requests attempted at run completion, by outcome",
+  labelNames: ["result"],
+  registers: [register],
+});
+
+const checkpointCancelRequests = new Counter({
+  name: "checkpoint_cancel_requests_total",
+  help: "Checkpoint cancel requests attempted when a run continues, by outcome",
+  labelNames: ["result"],
+  registers: [register],
+});
+
 const WorkloadActionParams = z.object({
   runFriendlyId: z.string(),
   snapshotFriendlyId: z.string(),
@@ -86,6 +108,7 @@ type WorkloadServerOptions = {
   checkpointClient?: CheckpointClient;
   computeManager?: ComputeWorkloadManager;
   tracing?: OtlpTraceService;
+  snapshotCallbackSecret: string;
   wideEventOpts: WideEventOptions;
   /** When true, high-frequency HTTP routes also emit wide events. */
   wideEventsNoisyRoutes: boolean;
@@ -136,6 +159,7 @@ export class WorkloadServer extends EventEmitter<WorkloadServerEvents> {
         workerClient: opts.workerClient,
         tracing: opts.tracing,
         wideEventOpts: this.wideEventOpts,
+        snapshotCallbackSecret: opts.snapshotCallbackSecret,
       });
     }
 
@@ -169,6 +193,127 @@ export class WorkloadServer extends EventEmitter<WorkloadServerEvents> {
     return this.headerValueFromRequest(req, WORKLOAD_HEADERS.PROJECT_REF);
   }
 
+  /**
+   * Verify the deployment token from the workload deployment-id header and return the verified
+   * environment_id to forward upstream. The env id is only forwarded in enforce mode: in log mode
+   * we still verify + record metrics but attach no header (so the platform never scopes). Only
+   * enforce fails a request, and only for a present-but-invalid token; absent and legacy ids pass.
+   *
+   * `claims` are returned on any valid token, for local use only - never to scope the platform,
+   * which is why environmentId stays gated on enforce.
+   */
+  private async authorizeWorkloadRequest(
+    req: IncomingMessage
+  ): Promise<
+    { ok: true; environmentId?: string; claims?: WorkloadDeploymentTokenClaims } | { ok: false }
+  > {
+    if (!workloadTokensEnabled) {
+      return { ok: true };
+    }
+
+    const result = await verifyDeploymentIdHeader(this.deploymentIdFromRequest(req), "http");
+
+    if (result.outcome === "jwt_invalid" && workloadTokenEnforced) {
+      return { ok: false };
+    }
+
+    return {
+      ok: true,
+      environmentId:
+        workloadTokenEnforced && result.outcome === "jwt_valid"
+          ? result.claims.environment_id
+          : undefined,
+      claims: result.outcome === "jwt_valid" ? result.claims : undefined,
+    };
+  }
+
+  /**
+   * reclaimCheckpoints asks the checkpoint service to delete a finished run's checkpoint storage.
+   * Must be called after the reply is sent: it never delays the runner.
+   */
+  private async reclaimCheckpoints(
+    req: IncomingMessage,
+    runFriendlyId: string,
+    attemptStatus: string,
+    claims: WorkloadDeploymentTokenClaims | undefined
+  ): Promise<void> {
+    if (!env.DELETE_CHECKPOINTS_ON_COMPLETION) {
+      checkpointDeleteRequests.inc({ result: "disabled" });
+      return;
+    }
+
+    if (!this.checkpointClient) {
+      checkpointDeleteRequests.inc({ result: "no_client" });
+      return;
+    }
+
+    if (this.snapshotService) {
+      checkpointDeleteRequests.inc({ result: "not_applicable" });
+      return;
+    }
+
+    if (attemptStatus !== "RUN_FINISHED" && attemptStatus !== "RUN_PENDING_CANCEL") {
+      checkpointDeleteRequests.inc({ result: "not_terminal" });
+      return;
+    }
+
+    if (!claims) {
+      checkpointDeleteRequests.inc({ result: "no_claims" });
+      return;
+    }
+
+    const projectRef = this.projectRefFromRequest(req);
+    if (!projectRef) {
+      checkpointDeleteRequests.inc({ result: "no_project_ref" });
+      this.logger.error("Cannot reclaim checkpoints without a project ref", { runFriendlyId });
+      return;
+    }
+
+    const [error, accepted] = await tryCatch(
+      this.checkpointClient.deleteCheckpoints({
+        runFriendlyId,
+        body: {
+          orgId: claims.org_id,
+          envId: claims.environment_id,
+          deploymentVersion: claims.deployment_version,
+          projectRef,
+        },
+      })
+    );
+
+    if (error || !accepted) {
+      checkpointDeleteRequests.inc({ result: "http_error" });
+      this.logger.error("Failed to request checkpoint reclaim", { runFriendlyId, error });
+      return;
+    }
+
+    checkpointDeleteRequests.inc({ result: "sent" });
+  }
+
+  private async cancelCheckpointsAfterReply(runFriendlyId: string): Promise<void> {
+    if (!this.checkpointClient) {
+      checkpointCancelRequests.inc({ result: "no_client" });
+      return;
+    }
+
+    if (this.snapshotService) {
+      checkpointCancelRequests.inc({ result: "not_applicable" });
+      return;
+    }
+
+    const [error, accepted] = await tryCatch(
+      this.checkpointClient.cancelCheckpoints({ runFriendlyId })
+    );
+
+    if (error || !accepted) {
+      checkpointCancelRequests.inc({ result: "http_error" });
+      this.logger.error("Failed to request checkpoint cancel", { runFriendlyId, error });
+      return;
+    }
+
+    checkpointCancelRequests.inc({ result: "sent" });
+  }
+
   /**
    * Sets common route meta on the wide-event state from URL params.
    */
@@ -250,11 +395,17 @@ export class WorkloadServer extends EventEmitter<WorkloadServerEvents> {
               "POST",
               async () => {
                 const { req, reply, params, body } = ctx;
+                const auth = await this.authorizeWorkloadRequest(req);
+                if (!auth.ok) {
+                  reply.empty(401);
+                  return;
+                }
                 const startResponse = await this.workerClient.startRunAttempt(
                   params.runFriendlyId,
                   params.snapshotFriendlyId,
                   body,
-                  this.runnerIdFromRequest(req)
+                  this.runnerIdFromRequest(req),
+                  auth.environmentId
                 );
 
                 if (!startResponse.success) {
@@ -286,6 +437,11 @@ export class WorkloadServer extends EventEmitter<WorkloadServerEvents> {
               "POST",
               async () => {
                 const { req, reply, params, body } = ctx;
+                const auth = await this.authorizeWorkloadRequest(req);
+                if (!auth.ok) {
+                  reply.empty(401);
+                  return;
+                }
                 const runnerId = this.runnerIdFromRequest(req);
 
                 // A completion attempt invalidates any pending delayed snapshot
@@ -304,7 +460,8 @@ export class WorkloadServer extends EventEmitter<WorkloadServerEvents> {
                   params.runFriendlyId,
                   params.snapshotFriendlyId,
                   body,
-                  runnerId
+                  runnerId,
+                  auth.environmentId
                 );
 
                 if (!completeResponse.success) {
@@ -317,6 +474,13 @@ export class WorkloadServer extends EventEmitter<WorkloadServerEvents> {
                 }
 
                 reply.json(completeResponse.data satisfies WorkloadRunAttemptCompleteResponseBody);
+
+                await this.reclaimCheckpoints(
+                  req,
+                  params.runFriendlyId,
+                  completeResponse.data.result.attemptStatus,
+                  auth.claims
+                );
                 return;
               }
             ),
@@ -336,6 +500,11 @@ export class WorkloadServer extends EventEmitter<WorkloadServerEvents> {
               "POST",
               async () => {
                 const { req, reply, params, body } = ctx;
+                const auth = await this.authorizeWorkloadRequest(req);
+                if (!auth.ok) {
+                  reply.empty(401);
+                  return;
+                }
                 const heartbeatResponse = await this.workerClient.heartbeatRun(
                   params.runFriendlyId,
                   params.snapshotFriendlyId,
@@ -373,6 +542,11 @@ export class WorkloadServer extends EventEmitter<WorkloadServerEvents> {
               "GET",
               async () => {
                 const { reply, params, req } = ctx;
+                const auth = await this.authorizeWorkloadRequest(req);
+                if (!auth.ok) {
+                  reply.empty(401);
+                  return;
+                }
                 const runnerId = this.runnerIdFromRequest(req);
                 const deploymentVersion = this.deploymentVersionFromRequest(req);
                 const projectRef = this.projectRefFromRequest(req);
@@ -469,6 +643,11 @@ export class WorkloadServer extends EventEmitter<WorkloadServerEvents> {
               "GET",
               async () => {
                 const { req, reply, params } = ctx;
+                const auth = await this.authorizeWorkloadRequest(req);
+                if (!auth.ok) {
+                  reply.empty(401);
+                  return;
+                }
                 this.logger.debug("Run continuation request", { params });
 
                 // Cancel any pending delayed snapshot for this run
@@ -477,7 +656,8 @@ export class WorkloadServer extends EventEmitter<WorkloadServerEvents> {
                 const continuationResult = await this.workerClient.continueRunExecution(
                   params.runFriendlyId,
                   params.snapshotFriendlyId,
-                  this.runnerIdFromRequest(req)
+                  this.runnerIdFromRequest(req),
+                  auth.environmentId
                 );
 
                 if (!continuationResult.success) {
@@ -494,6 +674,8 @@ export class WorkloadServer extends EventEmitter<WorkloadServerEvents> {
                 }
 
                 reply.json(continuationResult.data as WorkloadContinueRunExecutionResponseBody);
+
+                await this.cancelCheckpointsAfterReply(params.runFriendlyId);
               }
             ),
         }
@@ -511,10 +693,16 @@ export class WorkloadServer extends EventEmitter<WorkloadServerEvents> {
               "GET",
               async () => {
                 const { req, reply, params } = ctx;
+                const auth = await this.authorizeWorkloadRequest(req);
+                if (!auth.ok) {
+                  reply.empty(401);
+                  return;
+                }
                 const sinceSnapshotResponse = await this.workerClient.getSnapshotsSince(
                   params.runFriendlyId,
                   params.snapshotFriendlyId,
-                  this.runnerIdFromRequest(req)
+                  this.runnerIdFromRequest(req),
+                  auth.environmentId
                 );
 
                 if (!sinceSnapshotResponse.success) {
@@ -585,9 +773,18 @@ export class WorkloadServer extends EventEmitter<WorkloadServerEvents> {
               const { req, reply, params, body } = ctx;
               reply.empty(204);
 
+              // Redact TRIGGER_DEPLOYMENT_ID before relaying to the platform.
+              const sanitizedBody =
+                body.properties && "TRIGGER_DEPLOYMENT_ID" in body.properties
+                  ? {
+                      ...body,
+                      properties: { ...body.properties, TRIGGER_DEPLOYMENT_ID: "[redacted]" },
+                    }
+                  : body;
+
               await this.workerClient.sendDebugLog(
                 params.runFriendlyId,
-                body,
+                sanitizedBody,
                 this.runnerIdFromRequest(req)
               );
             },
@@ -681,7 +878,31 @@ export class WorkloadServer extends EventEmitter<WorkloadServerEvents> {
         return;
       }
 
-      this.logger.debug("[WS] auth success", socket.data);
+      if (workloadTokensEnabled) {
+        const result = await verifyDeploymentIdHeader(socket.data.deploymentId, "ws");
+
+        if (result.outcome === "jwt_invalid" && workloadTokenEnforced) {
+          this.logger.error("[WS] deployment token verification failed", {
+            runnerId: socket.data.runnerId,
+          });
+          socket.disconnect(true);
+          return;
+        }
+
+        // Re-source the deployment id from the verified claim; the raw header may be an opaque token.
+        // A legacy bare id is itself the friendlyId, so it's safe to keep.
+        socket.data.deploymentFriendlyId =
+          result.outcome === "jwt_valid"
+            ? result.claims.deployment
+            : result.outcome === "legacy_bare"
+              ? socket.data.deploymentId
+              : undefined;
+      }
+
+      this.logger.debug("[WS] handshake complete", {
+        runnerId: socket.data.runnerId,
+        deploymentFriendlyId: socket.data.deploymentFriendlyId,
+      });
 
       next();
     });
@@ -693,7 +914,7 @@ export class WorkloadServer extends EventEmitter<WorkloadServerEvents> {
 
       const getSocketMetadata = () => {
         return {
-          deploymentId: socket.data.deploymentId,
+          deploymentId: socket.data.deploymentFriendlyId ?? socket.data.deploymentId,
           runId: socket.data.runFriendlyId,
           snapshotId: socket.data.snapshotId,
           runnerId: socket.data.runnerId,
@@ -712,8 +933,9 @@ export class WorkloadServer extends EventEmitter<WorkloadServerEvents> {
           populate: (state) => {
             state.extras.event = event;
             setMeta(state, "run_id", friendlyId);
-            if (socket.data.deploymentId) {
-              setMeta(state, "deployment_id", socket.data.deploymentId);
+            const deploymentId = socket.data.deploymentFriendlyId ?? socket.data.deploymentId;
+            if (deploymentId) {
+              setMeta(state, "deployment_id", deploymentId);
             }
             if (socket.data.runnerId) setMeta(state, "runner_id", socket.data.runnerId);
             state.extras.socket_id = socket.id;
@@ -725,6 +947,33 @@ export class WorkloadServer extends EventEmitter<WorkloadServerEvents> {
       const runConnected = (friendlyId: string) => {
         socketLogger.debug("runConnected", { ...getSocketMetadata() });
 
+        // Only the owning runner may (re)bind a run. A live socket from a *different*
+        // runner keeps its binding so an unrelated connection can't hijack the run. But
+        // the newest socket for the *same* runner is a legitimate reconnection/handoff and
+        // is allowed to take over even while the stale socket still reports connected -
+        // otherwise, during a reconnect race the fresh socket would silently stay unbound
+        // (missing continue/cancel/suspend notifications) until the dead socket times out.
+        const existing = this.runSockets.get(friendlyId);
+        if (existing && existing.id !== socket.id && existing.connected) {
+          const sameRunner =
+            !!socket.data.runnerId && existing.data.runnerId === socket.data.runnerId;
+
+          if (!sameRunner) {
+            socketLogger.warn("runConnected: run already bound to another socket", {
+              ...getSocketMetadata(),
+              friendlyId,
+              existingSocketId: existing.id,
+            });
+            return;
+          }
+
+          socketLogger.debug("runConnected: replacing stale socket for same runner", {
+            ...getSocketMetadata(),
+            friendlyId,
+            existingSocketId: existing.id,
+          });
+        }
+
         // If there's already a run ID set, we should "disconnect" it from this socket
         if (socket.data.runFriendlyId && socket.data.runFriendlyId !== friendlyId) {
           socketLogger.debug("runConnected: disconnecting existing run", {
@@ -744,6 +993,22 @@ export class WorkloadServer extends EventEmitter<WorkloadServerEvents> {
       const runDisconnected = (friendlyId: string, reason: string) => {
         socketLogger.debug("runDisconnected", { ...getSocketMetadata() });
 
+        // A newer socket may have taken over this run (same-runner reconnect race). If the
+        // run is now bound to a different socket, this stale socket must not clear the fresh
+        // binding or emit a spurious disconnect - just drop its own reference and bail.
+        const bound = this.runSockets.get(friendlyId);
+        if (bound && bound.id !== socket.id) {
+          socketLogger.debug("runDisconnected: run rebound to another socket, skipping", {
+            ...getSocketMetadata(),
+            friendlyId,
+            boundSocketId: bound.id,
+          });
+          if (socket.data.runFriendlyId === friendlyId) {
+            socket.data.runFriendlyId = undefined;
+          }
+          return;
+        }
+
         // The run is gone from this runner (crash, exit, or replaced by a new
         // run), so a pending delayed snapshot for it is stale. Genuine
         // waitpoint suspensions keep the socket connected, so this doesn't

apps/supervisor/src/workloadServer/workloadServer.auth.test.ts

@@ -0,0 +1,107 @@
+import { mintWorkloadDeploymentToken } from "@trigger.dev/core/v3";
+import { WORKLOAD_HEADERS } from "@trigger.dev/core/v3/workers";
+import { afterAll, beforeAll, describe, expect, it, vi } from "vitest";
+
+// Set enforce mode + secret before env.ts parses (vi.mock is hoisted above imports, so the secret
+// must be a literal here). SECRET below mirrors it for use in the test body.
+vi.mock("std-env", () => ({
+  env: {
+    TRIGGER_API_URL: "http://localhost:3030",
+    TRIGGER_WORKER_TOKEN: "test-token",
+    MANAGED_WORKER_SECRET: "test-secret",
+    OTEL_EXPORTER_OTLP_ENDPOINT: "http://localhost:4318",
+    WORKLOAD_TOKEN_SECRET: "integration-test-secret",
+    WORKLOAD_TOKEN_ENFORCEMENT: "enforce",
+  },
+}));
+
+const SECRET = "integration-test-secret";
+const EXP = Math.floor(Date.UTC(2032, 0, 1) / 1000);
+
+const { WorkloadServer } = await import("./index.js");
+
+const PORT = 18732;
+const BASE = `http://127.0.0.1:${PORT}`;
+
+function claims(environmentId = "env_test_123") {
+  return {
+    deployment: "deployment_test",
+    deployment_version: "20260710.1",
+    environment_id: environmentId,
+    environment_type: "PRODUCTION",
+    org_id: "org_1",
+    project_id: "proj_1",
+  };
+}
+
+// Records the args each relay method is called with so we can assert the forwarded claim.
+const calls: { getSnapshotsSince: any[][] } = { getSnapshotsSince: [] };
+
+const workerClient = {
+  getSnapshotsSince: vi.fn(async (...args: any[]) => {
+    calls.getSnapshotsSince.push(args);
+    return { success: true as const, data: { snapshots: [] } };
+  }),
+} as any;
+
+let server: InstanceType<typeof WorkloadServer>;
+
+beforeAll(async () => {
+  server = new WorkloadServer({
+    port: PORT,
+    workerClient,
+    snapshotCallbackSecret: "snapshot-callback-secret",
+    wideEventOpts: { service: "supervisor", env: { nodeId: "test" }, enabled: false },
+    wideEventsNoisyRoutes: false,
+  });
+  await server.start();
+});
+
+afterAll(async () => {
+  await server.stop();
+});
+
+function snapshotsSince(deploymentIdHeader?: string) {
+  const headers: Record<string, string> = {
+    [WORKLOAD_HEADERS.RUNNER_ID]: "runner_1",
+  };
+  if (deploymentIdHeader !== undefined) {
+    headers[WORKLOAD_HEADERS.DEPLOYMENT_ID] = deploymentIdHeader;
+  }
+  return fetch(`${BASE}/api/v1/workload-actions/runs/run_1/snapshots/since/snap_1`, { headers });
+}
+
+describe("WorkloadServer auth (enforce mode)", () => {
+  it("allows a valid token and forwards the verified environment_id", async () => {
+    const token = await mintWorkloadDeploymentToken(claims("env_forwarded_42"), SECRET, EXP);
+    const res = await snapshotsSince(token);
+
+    expect(res.status).toBe(200);
+    const lastCall = calls.getSnapshotsSince.at(-1)!;
+    // getSnapshotsSince(runId, snapshotId, runnerId, environmentId)
+    expect(lastCall[3]).toBe("env_forwarded_42");
+  });
+
+  it("rejects a token signed with the wrong secret (401) and does not relay", async () => {
+    const before = calls.getSnapshotsSince.length;
+    const badToken = await mintWorkloadDeploymentToken(claims(), "wrong-secret", EXP);
+    const res = await snapshotsSince(badToken);
+
+    expect(res.status).toBe(401);
+    expect(calls.getSnapshotsSince.length).toBe(before);
+  });
+
+  it("allows a legacy bare friendlyId and forwards no environment_id", async () => {
+    const res = await snapshotsSince("deployment_legacy_bare");
+
+    expect(res.status).toBe(200);
+    expect(calls.getSnapshotsSince.at(-1)![3]).toBeUndefined();
+  });
+
+  it("allows an absent token and forwards no environment_id", async () => {
+    const res = await snapshotsSince(undefined);
+
+    expect(res.status).toBe(200);
+    expect(calls.getSnapshotsSince.at(-1)![3]).toBeUndefined();
+  });
+});

apps/supervisor/src/workloadServer/workloadServer.authLog.test.ts

@@ -0,0 +1,87 @@
+import { mintWorkloadDeploymentToken } from "@trigger.dev/core/v3";
+import { WORKLOAD_HEADERS } from "@trigger.dev/core/v3/workers";
+import { afterAll, beforeAll, describe, expect, it, vi } from "vitest";
+
+// Log mode: mint + verify + metrics, but the platform must NOT be scoped, so no environment_id is
+// forwarded even for a valid token. (vi.mock is hoisted; secret literal here, mirrored below.)
+vi.mock("std-env", () => ({
+  env: {
+    TRIGGER_API_URL: "http://localhost:3030",
+    TRIGGER_WORKER_TOKEN: "test-token",
+    MANAGED_WORKER_SECRET: "test-secret",
+    OTEL_EXPORTER_OTLP_ENDPOINT: "http://localhost:4318",
+    WORKLOAD_TOKEN_SECRET: "integration-test-secret",
+    WORKLOAD_TOKEN_ENFORCEMENT: "log",
+  },
+}));
+
+const SECRET = "integration-test-secret";
+const EXP = Math.floor(Date.UTC(2032, 0, 1) / 1000);
+
+const { WorkloadServer } = await import("./index.js");
+
+const PORT = 18733;
+const BASE = `http://127.0.0.1:${PORT}`;
+
+const claims = {
+  deployment: "deployment_test",
+  deployment_version: "20260710.1",
+  environment_id: "env_should_not_forward",
+  environment_type: "PRODUCTION",
+  org_id: "org_1",
+  project_id: "proj_1",
+};
+
+const calls: { getSnapshotsSince: any[][] } = { getSnapshotsSince: [] };
+
+const workerClient = {
+  getSnapshotsSince: vi.fn(async (...args: any[]) => {
+    calls.getSnapshotsSince.push(args);
+    return { success: true as const, data: { snapshots: [] } };
+  }),
+} as any;
+
+let server: InstanceType<typeof WorkloadServer>;
+
+beforeAll(async () => {
+  server = new WorkloadServer({
+    port: PORT,
+    workerClient,
+    snapshotCallbackSecret: "snapshot-callback-secret",
+    wideEventOpts: { service: "supervisor", env: { nodeId: "test" }, enabled: false },
+    wideEventsNoisyRoutes: false,
+  });
+  await server.start();
+});
+
+afterAll(async () => {
+  await server.stop();
+});
+
+describe("WorkloadServer auth (log mode)", () => {
+  it("allows a valid token but forwards no environment_id", async () => {
+    const token = await mintWorkloadDeploymentToken(claims, SECRET, EXP);
+    const res = await fetch(`${BASE}/api/v1/workload-actions/runs/run_1/snapshots/since/snap_1`, {
+      headers: {
+        [WORKLOAD_HEADERS.RUNNER_ID]: "runner_1",
+        [WORKLOAD_HEADERS.DEPLOYMENT_ID]: token,
+      },
+    });
+
+    expect(res.status).toBe(200);
+    expect(calls.getSnapshotsSince.at(-1)![3]).toBeUndefined();
+  });
+
+  it("does not reject an invalid token in log mode", async () => {
+    const badToken = await mintWorkloadDeploymentToken(claims, "wrong-secret", EXP);
+    const res = await fetch(`${BASE}/api/v1/workload-actions/runs/run_1/snapshots/since/snap_1`, {
+      headers: {
+        [WORKLOAD_HEADERS.RUNNER_ID]: "runner_1",
+        [WORKLOAD_HEADERS.DEPLOYMENT_ID]: badToken,
+      },
+    });
+
+    expect(res.status).toBe(200);
+    expect(calls.getSnapshotsSince.at(-1)![3]).toBeUndefined();
+  });
+});

apps/supervisor/src/workloadToken.ts

@@ -0,0 +1,94 @@
+import {
+  classifyDeploymentIdHeader,
+  mintWorkloadDeploymentToken,
+  type WorkloadDeploymentTokenClaims,
+  type WorkloadDeploymentTokenInput,
+} from "@trigger.dev/core/v3";
+import { Counter, Gauge } from "prom-client";
+import { env } from "./env.js";
+import { register } from "./metrics.js";
+
+const secret = env.WORKLOAD_TOKEN_SECRET;
+
+// Absolute expiry (epoch seconds) shared by every mint, so tokens stay byte-deterministic per
+// deployment regardless of when/where a pod is created.
+const tokenExpSeconds = Math.floor(new Date(env.WORKLOAD_TOKEN_EXP).getTime() / 1000);
+
+/** Mint + verify run in "log" (dry-run) and "enforce"; the env superRefine guarantees a secret then. */
+export const workloadTokensEnabled = env.WORKLOAD_TOKEN_ENFORCEMENT !== "disabled";
+
+/** Only "enforce" rejects a present-but-invalid token; "log" observes and always allows. */
+export const workloadTokenEnforced = env.WORKLOAD_TOKEN_ENFORCEMENT === "enforce";
+
+const mintCounter = new Counter({
+  name: "workload_token_minted_total",
+  help: "Deployment tokens minted and injected into TRIGGER_DEPLOYMENT_ID at pod creation",
+  labelNames: ["env_type"] as const,
+  registers: [register],
+});
+
+export type WorkloadAuthTransport = "http" | "ws";
+
+const verifyCounter = new Counter({
+  name: "workload_auth_verify_total",
+  help: "Runner-boundary token verification outcomes at the supervisor workload server",
+  labelNames: ["outcome", "transport", "env_type"] as const,
+  registers: [register],
+});
+
+// Exports the active mode (value 1 for the current WORKLOAD_TOKEN_ENFORCEMENT) so dashboards can show
+// disabled/log/enforce at a glance — the counters alone don't distinguish log from enforce.
+const enforcementModeGauge = new Gauge({
+  name: "workload_token_enforcement_mode",
+  help: "Active runner-boundary auth mode: value 1 for the label matching WORKLOAD_TOKEN_ENFORCEMENT",
+  labelNames: ["mode"] as const,
+  registers: [register],
+});
+enforcementModeGauge.set({ mode: env.WORKLOAD_TOKEN_ENFORCEMENT }, 1);
+
+export async function mintDeploymentToken(
+  claims: WorkloadDeploymentTokenInput
+): Promise<string | undefined> {
+  if (!workloadTokensEnabled || !secret) {
+    return undefined;
+  }
+
+  const token = await mintWorkloadDeploymentToken(claims, secret, tokenExpSeconds);
+  mintCounter.inc({ env_type: claims.environment_type });
+  return token;
+}
+
+export type VerifiedDeploymentHeader =
+  | { outcome: "jwt_valid"; claims: WorkloadDeploymentTokenClaims }
+  | { outcome: "jwt_invalid" | "legacy_bare" | "token_absent"; claims?: undefined };
+
+/**
+ * Verify the deployment-id header value and record the outcome. "jwt_valid" returns the claims so the
+ * caller can forward the verified environment_id upstream; other outcomes carry no trusted data.
+ */
+export async function verifyDeploymentIdHeader(
+  value: string | undefined,
+  transport: WorkloadAuthTransport
+): Promise<VerifiedDeploymentHeader> {
+  const result = await classify(value);
+  verifyCounter.inc({
+    outcome: result.outcome,
+    transport,
+    env_type: result.outcome === "jwt_valid" ? result.claims.environment_type : "unknown",
+  });
+  return result;
+}
+
+async function classify(value: string | undefined): Promise<VerifiedDeploymentHeader> {
+  if (!value || !secret) {
+    return { outcome: "token_absent" };
+  }
+
+  const result = await classifyDeploymentIdHeader(value, secret);
+
+  if (result.outcome === "jwt_valid" && result.claims) {
+    return { outcome: "jwt_valid", claims: result.claims };
+  }
+
+  return { outcome: result.outcome === "jwt_valid" ? "jwt_invalid" : result.outcome };
+}

apps/webapp/.gitignore

@@ -7,6 +7,9 @@ node_modules
 /cypress/screenshots
 /cypress/videos
 
+# Output of `pnpm run agent-ui:screenshots`
+/screenshots
+
 /app/styles/tailwind.css
 
 # Ensure the .env symlink is not removed by accident
@@ -20,4 +23,4 @@ storybook-static
 
 /prisma/seed.js
 /prisma/populate.js
-.memory-snapshots
+.memory-snapshots

apps/webapp/app/assets/icons/AnimatedHourglassIcon.tsx

@@ -1,6 +1,6 @@
 import { useAnimate } from "framer-motion";
 import { HourglassIcon } from "lucide-react";
-import { useEffect } from "react";
+import { useEffect, useRef } from "react";
 
 export function AnimatedHourglassIcon({
   className,
@@ -10,18 +10,21 @@ export function AnimatedHourglassIcon({
   delay?: number;
 }) {
   const [scope, animate] = useAnimate();
+  const initialDelay = useRef(delay);
 
   useEffect(() => {
-    animate(
+    const controls = animate(
       [
         [scope.current, { rotate: 0 }, { duration: 0.7 }],
         [scope.current, { rotate: 180 }, { duration: 0.3 }],
         [scope.current, { rotate: 180 }, { duration: 0.7 }],
         [scope.current, { rotate: 360 }, { duration: 0.3 }],
       ],
-      { repeat: Infinity, delay }
+      { repeat: Infinity, delay: initialDelay.current }
     );
-  }, []);
+
+    return () => controls.stop();
+  }, [animate, scope]);
 
   return <HourglassIcon ref={scope} className={className} />;
 }

apps/webapp/app/assets/icons/AvatarCircleIcon.tsx

@@ -1,4 +1,4 @@
-export function AvatarCircleIcon({ className }: { className?: string }) {
+function AvatarCircle({ className, strokeWidth }: { className?: string; strokeWidth: number }) {
   return (
     <svg
       className={className}
@@ -8,13 +8,28 @@ export function AvatarCircleIcon({ className }: { className?: string }) {
       fill="none"
       xmlns="http://www.w3.org/2000/svg"
     >
-      <circle cx="12" cy="12" r="9" stroke="currentColor" strokeWidth="2" />
-      <circle cx="12" cy="9.5" r="2.5" stroke="currentColor" strokeWidth="2" />
+      <circle cx="12" cy="12" r="9" stroke="currentColor" strokeWidth={strokeWidth} />
+      <circle cx="12" cy="9.5" r="2.5" stroke="currentColor" strokeWidth={strokeWidth} />
       <path
         d="M6 19C7.00156 16.6478 9.32233 15 12.0254 15C14.6837 15 16.9724 16.5938 18 18.884"
         stroke="currentColor"
-        strokeWidth="2"
+        strokeWidth={strokeWidth}
       />
     </svg>
   );
 }
+
+/** User avatar placeholder with a 2px stroke (the default). */
+export function AvatarCircleIcon({ className }: { className?: string }) {
+  return <AvatarCircle className={className} strokeWidth={2} />;
+}
+
+/** Thinner 1.5px-stroke variant of {@link AvatarCircleIcon}. */
+export function AvatarCircleIconThin({ className }: { className?: string }) {
+  return <AvatarCircle className={className} strokeWidth={1.5} />;
+}
+
+/** Thinnest 1.25px-stroke variant of {@link AvatarCircleIcon}. */
+export function AvatarCircleIconExtraThin({ className }: { className?: string }) {
+  return <AvatarCircle className={className} strokeWidth={1.25} />;
+}

apps/webapp/app/assets/icons/ChainLinkIcon.tsx

@@ -0,0 +1,27 @@
+export function ChainLinkIcon({ className }: { className?: string }) {
+  return (
+    <svg
+      className={className}
+      width="24"
+      height="24"
+      viewBox="0 0 24 24"
+      fill="none"
+      xmlns="http://www.w3.org/2000/svg"
+    >
+      <path
+        d="M10 19.0004L9.82843 19.1719C8.26634 20.734 5.73368 20.734 4.17158 19.1719L3.82843 18.8288C2.26634 17.2667 2.26633 14.734 3.82843 13.1719L7.17158 9.8288C8.73368 8.2667 11.2663 8.2667 12.8284 9.8288L13.1716 10.1719C13.8252 10.8256 14.2053 11.6491 14.312 12.5004"
+        stroke="currentColor"
+        strokeWidth="2"
+        strokeLinecap="round"
+        strokeLinejoin="round"
+      />
+      <path
+        d="M9.68799 12.5004C9.79463 13.3516 10.1748 14.1752 10.8284 14.8288L11.1715 15.1719C12.7336 16.734 15.2663 16.734 16.8284 15.1719L20.1715 11.8288C21.7336 10.2667 21.7336 7.73404 20.1715 6.17194L19.8284 5.8288C18.2663 4.2667 15.7336 4.2667 14.1715 5.8288L14 6.00037"
+        stroke="currentColor"
+        strokeWidth="2"
+        strokeLinecap="round"
+        strokeLinejoin="round"
+      />
+    </svg>
+  );
+}

apps/webapp/app/assets/icons/CircleFilledIcon.tsx

@@ -0,0 +1,16 @@
+/** Solid circle. Paired with {@link CircleOutlineIcon} by the Black and White
+ *  theme options — the filled disc reads as the opposite of the active theme. */
+export function CircleFilledIcon({ className }: { className?: string }) {
+  return (
+    <svg
+      className={className}
+      width="24"
+      height="24"
+      viewBox="0 0 24 24"
+      fill="none"
+      xmlns="http://www.w3.org/2000/svg"
+    >
+      <circle cx="12" cy="12" r="9" fill="currentColor" />
+    </svg>
+  );
+}

apps/webapp/app/assets/icons/CircleOutlineIcon.tsx

@@ -0,0 +1,16 @@
+/** Hollow circle. Paired with {@link CircleFilledIcon} by the Black and White
+ *  theme options, which show the active theme's background through the ring. */
+export function CircleOutlineIcon({ className }: { className?: string }) {
+  return (
+    <svg
+      className={className}
+      width="24"
+      height="24"
+      viewBox="0 0 24 24"
+      fill="none"
+      xmlns="http://www.w3.org/2000/svg"
+    >
+      <circle cx="12" cy="12" r="8" stroke="currentColor" strokeWidth="2" />
+    </svg>
+  );
+}

apps/webapp/app/assets/icons/ColumnsIcon.tsx

@@ -0,0 +1,9 @@
+export function ColumnsIcon({ className }: { className?: string }) {
+  return (
+    <svg className={className} viewBox="0 0 24 24" fill="none" xmlns="http://www.w3.org/2000/svg">
+      <rect x="3" y="4" width="18" height="16" rx="2" stroke="currentColor" strokeWidth="2" />
+      <line x1="9" y1="19" x2="9" y2="5" stroke="currentColor" strokeWidth="2" />
+      <line x1="15" y1="19" x2="15" y2="5" stroke="currentColor" strokeWidth="2" />
+    </svg>
+  );
+}

apps/webapp/app/assets/icons/CrossIcon.tsx

@@ -0,0 +1,14 @@
+export function CrossIcon({ className }: { className?: string }) {
+  return (
+    <svg
+      className={className}
+      width="24"
+      height="24"
+      viewBox="0 0 24 24"
+      fill="none"
+      xmlns="http://www.w3.org/2000/svg"
+    >
+      <path d="M6 6L18 18M18 6L6 18" stroke="currentColor" strokeWidth="2" strokeLinecap="round" />
+    </svg>
+  );
+}

apps/webapp/app/assets/icons/EditPencilIcon.tsx

@@ -0,0 +1,22 @@
+/** Pencil over a couple of text lines — editing a value in place. */
+export function EditPencilIcon({ className }: { className?: string }) {
+  return (
+    <svg
+      className={className}
+      width="24"
+      height="24"
+      viewBox="0 0 24 24"
+      fill="none"
+      xmlns="http://www.w3.org/2000/svg"
+    >
+      <path
+        d="M18.7573 3.6275L20.3732 5.24335C21.1542 6.0244 21.1542 7.29073 20.3732 8.07178L9.72032 18.7246C9.57777 18.8671 9.3957 18.9631 9.19759 19.0002L4.03377 19.9669L5.00052 14.8031C5.03765 14.6051 5.1336 14.4229 5.27604 14.2804L15.9289 3.6275C16.71 2.84645 17.9763 2.84645 18.7573 3.6275Z"
+        stroke="currentColor"
+        strokeWidth="2"
+      />
+      <line x1="17.6464" y1="10.3536" x2="13.6464" y2="6.35355" stroke="currentColor" />
+      <path d="M13 21L21 21" stroke="currentColor" strokeWidth="2" strokeLinecap="round" />
+      <path d="M18 17L21 17" stroke="currentColor" strokeWidth="2" strokeLinecap="round" />
+    </svg>
+  );
+}

apps/webapp/app/assets/icons/EyeClosedIcon.tsx

@@ -0,0 +1,35 @@
+export function EyeClosedIcon({ className }: { className?: string }) {
+  return (
+    <svg
+      className={className}
+      width="24"
+      height="24"
+      viewBox="0 0 24 24"
+      fill="none"
+      xmlns="http://www.w3.org/2000/svg"
+    >
+      <path
+        d="M10.7424 5.08581C14.6841 4.54668 18.7922 6.54985 21.4978 11.0954C21.8296 11.6529 21.8298 12.3468 21.498 12.9043C21.124 13.5326 20.7233 14.1123 20.3 14.6434"
+        stroke="currentColor"
+        strokeWidth="2"
+        strokeLinecap="round"
+        strokeLinejoin="round"
+      />
+      <path
+        d="M3.70711 2.29289C3.31658 1.90237 2.68342 1.90237 2.29289 2.29289C1.90237 2.68342 1.90237 3.31658 2.29289 3.70711L3.70711 2.29289ZM20.2929 21.7071C20.6834 22.0976 21.3166 22.0976 21.7071 21.7071C22.0976 21.3166 22.0976 20.6834 21.7071 20.2929L20.2929 21.7071ZM2.29289 3.70711L20.2929 21.7071L21.7071 20.2929L3.70711 2.29289L2.29289 3.70711Z"
+        fill="currentColor"
+      />
+      <path
+        d="M10.3327 10.8948C10.6385 10.4349 10.5136 9.81416 10.0537 9.50837C9.59377 9.20259 8.97305 9.32753 8.66727 9.78743L10.3327 10.8948ZM14.2126 15.3328C14.6725 15.027 14.7974 14.4063 14.4916 13.9463C14.1858 13.4864 13.5651 13.3615 13.1052 13.6673L14.2126 15.3328ZM12 14C10.8954 14 10 13.1046 10 12H8C8 14.2092 9.79086 16 12 16V14ZM10 12C10 11.5897 10.1225 11.211 10.3327 10.8948L8.66727 9.78743C8.24565 10.4216 8 11.1836 8 12H10ZM13.1052 13.6673C12.789 13.8775 12.4103 14 12 14V16C12.8164 16 13.5785 15.7544 14.2126 15.3328L13.1052 13.6673Z"
+        fill="currentColor"
+      />
+      <path
+        d="M6.12815 7C4.77316 7.99438 3.53535 9.35957 2.50209 11.0955C2.17024 11.6531 2.17115 12.3487 2.50305 12.9062C6.05251 18.8681 12.0149 20.4553 16.8492 17.6681"
+        stroke="currentColor"
+        strokeWidth="2"
+        strokeLinecap="round"
+        strokeLinejoin="round"
+      />
+    </svg>
+  );
+}

apps/webapp/app/assets/icons/EyeOpenIcon.tsx

@@ -0,0 +1,27 @@
+export function EyeOpenIcon({ className }: { className?: string }) {
+  return (
+    <svg
+      className={className}
+      width="24"
+      height="24"
+      viewBox="0 0 24 24"
+      fill="none"
+      xmlns="http://www.w3.org/2000/svg"
+    >
+      <path
+        d="M15 12C15 13.6569 13.6569 15 12 15C10.3431 15 9 13.6569 9 12C9 10.3431 10.3431 9 12 9C13.6569 9 15 10.3431 15 12Z"
+        stroke="currentColor"
+        strokeWidth="2"
+        strokeLinecap="round"
+        strokeLinejoin="round"
+      />
+      <path
+        d="M21.4974 11.0946C16.66 2.9684 7.33998 2.96849 2.50257 11.0947C2.17069 11.6523 2.17069 12.3479 2.50257 12.9054C7.33998 21.0316 16.66 21.0315 21.4974 12.9053C21.8293 12.3477 21.8293 11.6521 21.4974 11.0946Z"
+        stroke="currentColor"
+        strokeWidth="2"
+        strokeLinecap="round"
+        strokeLinejoin="round"
+      />
+    </svg>
+  );
+}

apps/webapp/app/assets/icons/LeftSideMenuCollapsedIcon.tsx

@@ -0,0 +1,22 @@
+export function LeftSideMenuCollapsedIcon({ className }: { className?: string }) {
+  return (
+    <svg
+      className={className}
+      width="24"
+      height="24"
+      viewBox="0 0 24 24"
+      fill="none"
+      xmlns="http://www.w3.org/2000/svg"
+    >
+      <rect x="4" y="4" width="16" height="16" rx="3" stroke="currentColor" strokeWidth="2" />
+      <rect x="6" y="6" width="2" height="12" rx="1" fill="currentColor" />
+      <path
+        d="M12 14.5L14.5 12L12 9.5"
+        stroke="currentColor"
+        strokeWidth="2"
+        strokeLinecap="round"
+        strokeLinejoin="round"
+      />
+    </svg>
+  );
+}

apps/webapp/app/assets/icons/LeftSideMenuIcon.tsx

@@ -0,0 +1,44 @@
+import { motion } from "framer-motion";
+import { useState } from "react";
+
+export function LeftSideMenuIcon({
+  className,
+  hovered: controlledHovered,
+}: {
+  className?: string;
+  /** Drives the animation when provided (e.g. parent hover); otherwise the icon uses its own hover. */
+  hovered?: boolean;
+}) {
+  const [internalHovered, setInternalHovered] = useState(false);
+  const isControlled = controlledHovered !== undefined;
+  const hovered = isControlled ? controlledHovered : internalHovered;
+
+  return (
+    <svg
+      className={className}
+      width="24"
+      height="24"
+      viewBox="0 0 24 24"
+      fill="none"
+      xmlns="http://www.w3.org/2000/svg"
+      onMouseEnter={isControlled ? undefined : () => setInternalHovered(true)}
+      onMouseLeave={isControlled ? undefined : () => setInternalHovered(false)}
+    >
+      <rect x="4" y="4" width="16" height="16" rx="2" stroke="currentColor" strokeWidth="2" />
+      {/* Animate a transform (scaleX), not the SVG `width` attr — framer snaps the first animation
+          of an idle SVG geometry attribute. Left origin collapses the panel right-to-left. */}
+      <motion.rect
+        x="6"
+        y="6"
+        width="5"
+        height="12"
+        rx="1"
+        fill="currentColor"
+        initial={false}
+        style={{ originX: 0 }}
+        animate={{ scaleX: hovered ? 0.2 : 1 }}
+        transition={{ duration: 0.3, ease: "easeInOut" }}
+      />
+    </svg>
+  );
+}

apps/webapp/app/assets/icons/MachineIcon.tsx

@@ -48,20 +48,25 @@ export function MachineDefaultIcon({ className }: { className?: string }) {
 
 function MachineIconNoMachine({ className }: { className?: string }) {
   return (
-    <svg className={className} viewBox="0 0 20 20" fill="none" xmlns="http://www.w3.org/2000/svg">
+    <svg
+      className={cn("text-[#D7D9DD] light:text-charcoal-500", className)}
+      viewBox="0 0 20 20"
+      fill="none"
+      xmlns="http://www.w3.org/2000/svg"
+    >
       <path
         d="M16.1875 13C16.6362 13 17 13.3638 17 13.8125V16.1875C17 16.6362 16.6362 17 16.1875 17H6.18945C6.25882 16.9453 6.32663 16.8872 6.39062 16.8232L10.2139 13H16.1875Z"
-        fill="#D7D9DD"
+        fill="currentColor"
         fillOpacity="0.4"
       />
       <path
         d="M4.84961 12H3.8125C3.36377 12 3 11.6362 3 11.1875V8.8125C3 8.36377 3.36377 8 3.8125 8H8.84961L4.84961 12ZM16.1875 8C16.6362 8 17 8.36377 17 8.8125V11.1875C17 11.6362 16.6362 12 16.1875 12H11.2139L15.2139 8H16.1875Z"
-        fill="#D7D9DD"
+        fill="currentColor"
         fillOpacity="0.4"
       />
       <path
         d="M9.85059 7H3.8125C3.36377 7 3 6.63623 3 6.1875V3.8125C3 3.36377 3.36377 3 3.8125 3H13.8506L9.85059 7Z"
-        fill="#D7D9DD"
+        fill="currentColor"
         fillOpacity="0.4"
       />
       <line
@@ -69,7 +74,7 @@ function MachineIconNoMachine({ className }: { className?: string }) {
         y1="16.9998"
         x2="17.0018"
         y2="2.99805"
-        stroke="#D7D9DD"
+        stroke="currentColor"
         strokeOpacity="0.4"
         strokeWidth="2"
         strokeLinecap="round"
@@ -83,17 +88,17 @@ function MachineIconMicro({ className }: { className?: string }) {
     <svg className={className} viewBox="0 0 20 20" fill="none" xmlns="http://www.w3.org/2000/svg">
       <path
         d="M16.1875 13C16.6362 13 17 13.3638 17 13.8125V16.1875C17 16.6362 16.6362 17 16.1875 17H3.8125C3.36377 17 3 16.6362 3 16.1875V13.8125C3 13.3638 3.36377 13 3.8125 13H16.1875ZM5 14C4.44772 14 4 14.4477 4 15C4 15.5523 4.44772 16 5 16C5.55228 16 6 15.5523 6 15C6 14.4477 5.55228 14 5 14Z"
-        fill="#D7D9DD"
+        fill="currentColor"
         fillOpacity="0.4"
       />
       <path
         d="M16.1875 8C16.6362 8 17 8.36377 17 8.8125V11.1875C17 11.6362 16.6362 12 16.1875 12H3.8125C3.36377 12 3 11.6362 3 11.1875V8.8125C3 8.36377 3.36377 8 3.8125 8H16.1875ZM5 9C4.44772 9 4 9.44772 4 10C4 10.5523 4.44772 11 5 11C5.55228 11 6 10.5523 6 10C6 9.44772 5.55228 9 5 9Z"
-        fill="#D7D9DD"
+        fill="currentColor"
         fillOpacity="0.4"
       />
       <path
         d="M16.1875 3C16.6362 3 17 3.36377 17 3.8125V6.1875C17 6.63623 16.6362 7 16.1875 7H3.8125C3.36377 7 3 6.63623 3 6.1875V3.8125C3 3.36377 3.36377 3 3.8125 3H16.1875ZM5 4C4.44772 4 4 4.44772 4 5C4 5.55228 4.44772 6 5 6C5.55228 6 6 5.55228 6 5C6 4.44772 5.55228 4 5 4Z"
-        fill="#D7D9DD"
+        fill="currentColor"
         fillOpacity="0.4"
       />
     </svg>
@@ -109,12 +114,12 @@ function MachineIconSmall1x({ className }: { className?: string }) {
       />
       <path
         d="M16.1875 8C16.6362 8 17 8.36377 17 8.8125V11.1875C17 11.6362 16.6362 12 16.1875 12H3.8125C3.36377 12 3 11.6362 3 11.1875V8.8125C3 8.36377 3.36377 8 3.8125 8H16.1875ZM5 9C4.44772 9 4 9.44772 4 10C4 10.5523 4.44772 11 5 11C5.55228 11 6 10.5523 6 10C6 9.44772 5.55228 9 5 9Z"
-        fill="#D7D9DD"
+        fill="currentColor"
         fillOpacity="0.4"
       />
       <path
         d="M16.1875 3C16.6362 3 17 3.36377 17 3.8125V6.1875C17 6.63623 16.6362 7 16.1875 7H3.8125C3.36377 7 3 6.63623 3 6.1875V3.8125C3 3.36377 3.36377 3 3.8125 3H16.1875ZM5 4C4.44772 4 4 4.44772 4 5C4 5.55228 4.44772 6 5 6C5.55228 6 6 5.55228 6 5C6 4.44772 5.55228 4 5 4Z"
-        fill="#D7D9DD"
+        fill="currentColor"
         fillOpacity="0.4"
       />
     </svg>
@@ -130,12 +135,12 @@ function MachineIconSmall2x({ className }: { className?: string }) {
       />
       <path
         d="M16.1875 8C16.6362 8 17 8.36377 17 8.8125V11.1875C17 11.6362 16.6362 12 16.1875 12H3.8125C3.36377 12 3 11.6362 3 11.1875V8.8125C3 8.36377 3.36377 8 3.8125 8H16.1875ZM5 9C4.44772 9 4 9.44772 4 10C4 10.5523 4.44772 11 5 11C5.55228 11 6 10.5523 6 10C6 9.44772 5.55228 9 5 9Z"
-        fill="#D7D9DD"
+        fill="currentColor"
         fillOpacity="0.4"
       />
       <path
         d="M16.1875 3C16.6362 3 17 3.36377 17 3.8125V6.1875C17 6.63623 16.6362 7 16.1875 7H3.8125C3.36377 7 3 6.63623 3 6.1875V3.8125C3 3.36377 3.36377 3 3.8125 3H16.1875ZM5 4C4.44772 4 4 4.44772 4 5C4 5.55228 4.44772 6 5 6C5.55228 6 6 5.55228 6 5C6 4.44772 5.55228 4 5 4Z"
-        fill="#D7D9DD"
+        fill="currentColor"
         fillOpacity="0.4"
       />
     </svg>
@@ -155,7 +160,7 @@ function MachineIconMedium1x({ className }: { className?: string }) {
       />
       <path
         d="M16.1875 3C16.6362 3 17 3.36377 17 3.8125V6.1875C17 6.63623 16.6362 7 16.1875 7H3.8125C3.36377 7 3 6.63623 3 6.1875V3.8125C3 3.36377 3.36377 3 3.8125 3H16.1875ZM5 4C4.44772 4 4 4.44772 4 5C4 5.55228 4.44772 6 5 6C5.55228 6 6 5.55228 6 5C6 4.44772 5.55228 4 5 4Z"
-        fill="#D7D9DD"
+        fill="currentColor"
         fillOpacity="0.4"
       />
     </svg>
@@ -175,7 +180,7 @@ function MachineIconMedium2x({ className }: { className?: string }) {
       />
       <path
         d="M16.1875 3C16.6362 3 17 3.36377 17 3.8125V6.1875C17 6.63623 16.6362 7 16.1875 7H3.8125C3.36377 7 3 6.63623 3 6.1875V3.8125C3 3.36377 3.36377 3 3.8125 3H16.1875ZM5 4C4.44772 4 4 4.44772 4 5C4 5.55228 4.44772 6 5 6C5.55228 6 6 5.55228 6 5C6 4.44772 5.55228 4 5 4Z"
-        fill="#D7D9DD"
+        fill="currentColor"
         fillOpacity="0.4"
       />
     </svg>

apps/webapp/app/assets/icons/MonitorIcon.tsx

@@ -0,0 +1,21 @@
+/** Monitor on a stand — the System theme, which follows the OS appearance. */
+export function MonitorIcon({ className }: { className?: string }) {
+  return (
+    <svg
+      className={className}
+      width="24"
+      height="24"
+      viewBox="0 0 24 24"
+      fill="none"
+      xmlns="http://www.w3.org/2000/svg"
+    >
+      <path
+        d="M21 13H3M11 17H13L14 21H10L11 17ZM5 17H19C20.1046 17 21 16.1046 21 15V6C21 4.89543 20.1046 4 19 4H5C3.89543 4 3 4.89543 3 6V15C3 16.1046 3.89543 17 5 17Z"
+        stroke="currentColor"
+        strokeWidth="2"
+        strokeLinecap="square"
+        strokeLinejoin="round"
+      />
+    </svg>
+  );
+}

apps/webapp/app/assets/icons/MoonIcon.tsx

@@ -0,0 +1,21 @@
+/** Crescent moon — the dark theme. */
+export function MoonIcon({ className }: { className?: string }) {
+  return (
+    <svg
+      className={className}
+      width="24"
+      height="24"
+      viewBox="0 0 24 24"
+      fill="none"
+      xmlns="http://www.w3.org/2000/svg"
+    >
+      <path
+        d="M20.9638 12.7674C19.8361 13.5447 18.4693 13.9998 16.9961 13.9998C13.1301 13.9998 9.99609 10.8657 9.99609 6.99975C9.99609 5.52667 10.4511 4.15987 11.2283 3.03223C6.61911 3.42277 3 7.28768 3 11.9979C3 16.9674 7.0286 20.996 11.9981 20.996C16.7084 20.996 20.5734 17.3767 20.9638 12.7674Z"
+        stroke="currentColor"
+        strokeWidth="2"
+        strokeLinecap="round"
+        strokeLinejoin="round"
+      />
+    </svg>
+  );
+}

apps/webapp/app/assets/icons/PadlockRoundedIcon.tsx

@@ -0,0 +1,34 @@
+export function PadlockRoundedIcon({ className }: { className?: string }) {
+  return (
+    <svg
+      className={className}
+      width="24"
+      height="24"
+      viewBox="0 0 24 24"
+      fill="none"
+      xmlns="http://www.w3.org/2000/svg"
+    >
+      <path
+        d="M5 12C5 10.8954 5.89543 10 7 10H17C18.1046 10 19 10.8954 19 12V19C19 20.1046 18.1046 21 17 21H7C5.89543 21 5 20.1046 5 19V12Z"
+        stroke="currentColor"
+        strokeWidth="2"
+        strokeLinecap="round"
+        strokeLinejoin="round"
+      />
+      <path
+        d="M16 9.5V7C16 4.79086 14.2091 3 12 3C9.79086 3 8 4.79086 8 7V9.5"
+        stroke="currentColor"
+        strokeWidth="2"
+        strokeLinecap="round"
+        strokeLinejoin="round"
+      />
+      <path
+        d="M12 14V17"
+        stroke="currentColor"
+        strokeWidth="2"
+        strokeLinecap="round"
+        strokeLinejoin="round"
+      />
+    </svg>
+  );
+}

apps/webapp/app/assets/icons/RenameIcon.tsx

@@ -0,0 +1,27 @@
+export function RenameIcon({ className }: { className?: string }) {
+  return (
+    <svg
+      className={className}
+      width="24"
+      height="24"
+      viewBox="0 0 24 24"
+      fill="none"
+      xmlns="http://www.w3.org/2000/svg"
+    >
+      <path
+        d="M11 4H7.2C6.0799 4 5.51984 4 5.09202 4.21799C4.71569 4.40973 4.40973 4.71569 4.21799 5.09202C4 5.51984 4 6.0799 4 7.2V16.8C4 17.9201 4 18.4802 4.21799 18.908C4.40973 19.2843 4.71569 19.5903 5.09202 19.782C5.51984 20 6.0799 20 7.2 20H16.8C17.9201 20 18.4802 20 18.908 19.782C19.2843 19.5903 19.5903 19.2843 19.782 18.908C20 18.4802 20 17.9201 20 16.8V13"
+        stroke="currentColor"
+        strokeWidth="2"
+        strokeLinecap="round"
+        strokeLinejoin="round"
+      />
+      <path
+        d="M9 14.9999V12.4142C9 12.1489 9.10536 11.8946 9.29289 11.707L17.3358 3.66416C18.1168 2.88311 19.3832 2.88311 20.1642 3.66416L20.3358 3.83573C21.1168 4.61678 21.1168 5.88311 20.3358 6.66416L12.2929 14.707C12.1054 14.8946 11.851 14.9999 11.5858 14.9999H9Z"
+        stroke="currentColor"
+        strokeWidth="2"
+        strokeLinecap="square"
+        strokeLinejoin="round"
+      />
+    </svg>
+  );
+}

apps/webapp/app/assets/icons/ResetIcon.tsx

@@ -0,0 +1,20 @@
+export function ResetIcon({ className }: { className?: string }) {
+  return (
+    <svg className={className} viewBox="0 0 24 24" fill="none" xmlns="http://www.w3.org/2000/svg">
+      <path
+        d="M7 3L4 6L7 9"
+        stroke="currentColor"
+        strokeWidth="2"
+        strokeLinecap="round"
+        strokeLinejoin="round"
+      />
+      <path
+        d="M5 6H13.5C17.0899 6 20 8.91015 20 12.5C20 16.0899 17.0899 19 13.5 19H6"
+        stroke="currentColor"
+        strokeWidth="2"
+        strokeLinecap="round"
+        strokeLinejoin="round"
+      />
+    </svg>
+  );
+}